E228 | Can Google's TPU Challenge Nvidia? A Former TPU Engineer Reveals the Inside Story for the First Time
Summary
TPU can challenge GPUs under defined conditions, but Henry did not say it can replace GPUs outright. The relevant conditions are large-scale deployment, relatively stable models and a sufficiently large user base; a fixed workload lets Google co-optimize the chip, XLA, networking and memory, while millions of users spread batch-processing costs. Single-user Agents, local deployment and rapidly changing models remain better suited to GPUs or low-latency ASICs. “The future will definitely be coexistence between the two.”
TPU’s core advantage is not single-chip benchmark performance but the ability to turn thousands of chips into a high-utilization TPU Pod. GPU’s SIMT architecture is like a kitchen of many chefs who make independent decisions, versatile but prone to waiting for data; TPU is a relay pipeline, with software scheduling each step in advance so matrix units and memory bandwidth stay as close to fully utilized as possible. Henry believes that when deeply customized for Gemini or other Frontier models, TPU’s Total Cost of Ownership will beat GPU’s.
V7 Ironwood is already close to NVIDIA Blackwell’s performance level. Henry cites two key advances: peak FLOPS approaching GPU levels, and major gains in HBM capacity, bandwidth and software utilization; he declined to disclose V8’s technical details. The real risk is that chip development still takes 2 to 2.5 years, or even 3, while model direction might shift roughly every 6 months; specialized hardware has to place its bet in advance.
TPU expansion is currently more constrained by supply chain and manufacturing capacity, with bottlenecks in HBM, TSMC CoWoS and yield. HBM is produced by SK hynix, Samsung and Micron, while Nvidia is also the largest customer; capacity typically has to be locked in 1 to 2 years ahead. TPU also requires highly consistent chip performance across a Pod, making it hard to sell lower-bin parts the way GPUs do: “once yield fails, that chip is effectively scrap.”
The demand signals cited at the opening included a 2024 Apple paper saying Apple Intelligence used TPU for training; Anthropic securing 1 million TPUs worth tens of billions of dollars in 2025; and Meta signing a multibillion-dollar agreement in 2026 to rent TPUs to run Llama. But Henry says Anthropic’s order cannot by itself prove that TPU already has a replicable external ecosystem: Anthropic has close ties to Google and engineers capable of handling JAX, XLA and hardware tuning. 泓君 attributes part of Apple’s capabilities to talent migrating from Google.
XLA is both TPU’s “secret sauce” and the main barrier for external customers. A static compiler can perform operator fusion, memory management and global optimization across an entire Pod, but the fused computation graph is harder to debug; external customers may need support from Google’s software team. Google is working to fill out native PyTorch operators and parallel libraries, while Meta is currently using TPU more to relieve CapEx and compute pressure than because it has completed a software-stack migration.
The inference market will not have a single winner; it will segment by scale, throughput and tail latency. TPU uses copper interconnects and a small number of optical switches to reduce the NVLink/NVSwitch “infrastructure tax,” making it suitable for massive cloud workloads such as Gemini and Claude; Groq uses SRAM, a deterministic compiler and more LPU resources to serve Agents, real-time voice and high-frequency trading. The key investment variables are not only how much GPU share Google can take, but also Broadcom’s bargaining power, HBM and CoWoS supply, and whether the TPU ecosystem can lower the barrier to external migration.
Deep dive
1. TPU trades GPU scheduling flexibility for pipeline efficiency
Henry likened GPU’s SIMT architecture to a kitchen with many chefs: each chef thinks independently, retrieves ingredients and completes tasks, yielding strong parallelism and flexibility across workloads, but threads enter idle periods while waiting for data to move.
TPU is a pipeline purpose-built for machine-learning matrix calculations: “the first person takes the ingredients from the refrigerator, the second processes the intermediate state directly, then passes it to the third.” It is like a heart pumping blood or a relay race, trading away scheduling overhead for higher compute-unit utilization.
泓君 summarized the difference as GPU parallelism versus TPU relay; Henry added that training and inference are moving from compute-bound to memory-bound, so the winner increasingly depends on cache reuse, data movement and the ability to saturate memory bandwidth—not theoretical FLOPS alone.
2. TPU’s competitive unit has never been a single card, but an entire Pod
Henry says GPUs have long been known for single-card performance; TPU has always been designed at the system level. Through ICI chip interconnects and a 3D torus topology, thousands of chips train in concert, while the user experience is “like using one card.”
This architecture cuts waiting and communication overhead outside the computation itself, and makes the relevant TPU metrics closer to cluster-wide training efficiency and TCO than to comparing one chip against H100 or GB200. “It is a large compute cluster, not a single card or single chip doing pretraining.”
TPU hardware is therefore deliberately made more “dumb”: it has fewer control, scheduling and prediction units, with software deciding in advance what happens at each point in time. Complexity moves into XLA, allowing matrix units to run near full power—but if software and hardware are misaligned, the advantage disappears quickly.
3. Ironwood closes the paper-performance gap; utilization remains the real differentiator
Henry worked deeply on 3 TPU generations, describing V7 Ironwood as the 2nd TPU he worked on and V8 as the last generation before he left. He summarizes V7’s 2 major upgrades as peak FLOPS moving closer to GPU levels, and a leap in HBM capacity, memory bandwidth and actual bandwidth utilization.
When 泓君 pressed him on which was cheaper for training the same Gemini generation, GPU or TPU, Henry gave no universal answer: for Google’s own models—or when Google can obtain a clear workload in advance for another Frontier model and customize the software and hardware—TPU would have the TCO advantage.
On direct performance, Henry said only that V7 and NVIDIA Blackwell were “already roughly on par,” without giving a ratio or public benchmark; he explicitly said V8 “cannot be disclosed at present.” The episode’s strong conclusion is therefore the efficiency of customization, not an absolute raw-performance lead across all tasks.
4. HBM, CoWoS and yield determine how many TPUs can be sold
Henry calls HBM a near-monopoly supply market. The main producers are SK hynix, Samsung and Micron; Nvidia has long been the largest customer, while TPU historically looked more like a secondary customer, making it difficult to secure top specifications and sufficiently large orders.
CoWoS is another hard constraint: compute chips and HBM have to be integrated through 2.5D packaging, and neither Google nor Broadcom can do it in-house; both have to rely on TSMC. “Your annual output is whatever capacity TSMC allocates to you.”
Capacity typically has to be locked in 1 to 2 years ahead. Before V7, TPU mainly served Google internally and lacked clear external demand, making it difficult to secure sufficiently large orders from Broadcom, TSMC and HBM suppliers; once demand arrived, it was already too late to adjust.
TPU’s yield requirements are even harsher than GPU’s: a Pod depends on large numbers of chips communicating reliably, and performance variation can drag down the entire system. Henry contrasted this with GPU products that can be sold in different tiers, saying that customized TPUs are hard to down-bin: “Once it fails… that chip is effectively scrap.”
5. The million-chip order is about customer capability, not plug-and-play
泓君 said at the opening that when Gemini 3 topped AI performance rankings in 2025, it was trained on Google TPU rather than NVIDIA GPU. He also cited the demand signals: a 2024 Apple paper saying Apple Intelligence used TPU for all training; Anthropic securing 1 million TPUs worth tens of billions of dollars in 2025 to train the next Claude generation; and Meta signing a multibillion-dollar agreement in 2026 to rent TPUs to run Llama.
Henry views Anthropic’s order as large, but first as a “relatively closed loop” under Google’s investment relationship, and second as a deal enabled by Anthropic’s strong engineering team, which can deploy its models on TPU. 泓君 also noted that many Anthropic employees had worked at Google earlier. Henry is unsure whether the same conditions would allow TCO to be pushed down as far for Meta or other companies.
泓君 said Apple may have a larger TPU user team, attributing it partly to the migration of Google talent, including 庞若明. Henry agreed that talent mobility matters: demand from Apple, Midjourney and Anthropic once kept Google’s customer-facing teams “extremely busy,” while engineers who combine JAX, XLA and hardware expertise are especially scarce.
6. XLA can wring out TPU performance, but turns debugging into a high-barrier job
Henry calls XLA Google’s “black box” and “secret sauce”: it is a static compiler that, when the workload is known, can optimize the computation graph from a global perspective across the full TPU Pod, while CUDA’s developer experience is relatively mature.
XLA performs operator fusion, folding multiple kernels into the same compute unit to avoid repeatedly writing and reading intermediate states to and from memory; it also manages and allocates memory in ways TPU prefers. The top layer can be PyTorch, JAX or TensorFlow; XLA ultimately translates it into TPU instructions.
“Black box” does not mean impossible to debug, but the fused object is an entire graph rather than an individual computation. Engineers must understand the software and why the hardware fetches data as it does; external customers facing complex problems may need support from Google’s software team.
Henry’s tradeoff is blunt: an old software stack can “run” on TPU, but that does not mean it will deliver good TCO; squeezing out full performance usually requires migrating to JAX plus XLA. Against CUDA’s existing community, tools and accumulated know-how, that remains the heaviest migration cost for TPU’s external expansion.
7. PyTorch compatibility will determine whether Meta cooperation moves from rented compute to ecosystem migration
Meta’s software stack centers on PyTorch, and PyTorch is not especially compatible with TPU. Henry says Google has been in discussions with the relevant teams for a long time; internally it is also working to combine PyTorch with XLA and add native TPU support for more PyTorch libraries, parallel libraries and operators. The challenge is that PyTorch has thousands of operators, and missing native support causes performance loss.
Directly using Google Cloud can bypass deployment, but it sacrifices low-level control. Henry estimates that if the integration is poor, the actual model utilization rate may be only 50% to 60%; customers “still pay the same,” so paper cost-performance may not translate into an advantage on the final bill.
Henry believes Meta’s more practical motive at this stage is to offload compute pressure: its CapEx and pretraining needs are so large that it has to buy nearly every available source of compute. Based on what he knew before leaving, there appeared to be only 1 customer buying TPU racks directly from Broadcom; the name in the recording is “SRPIC [?],” whose identity cannot be confirmed from context alone. Other customers currently use Google Cloud primarily.
8. A chip must bet 2 to 3 years ahead on a model world that changes every 6 months
Henry did not want to attribute Gemini’s success simply to the chip: “Algorithms are of course important.” TPU’s real benefit is shortening the validation cycle—once an algorithm team proposes a new algorithm, if the workload can be co-designed with the chip team in advance, the team can validate more quickly whether it works and increase the number of iterations.
From design and verification through Broadcom’s interconnect design, package-level verification and manufacturing delivery, a flagship chip generation takes “at the absolute fastest” 2 to 2.5 years, and possibly 3. 泓君’s comparison is that model direction changes roughly every 6 months. Chip teams are effectively using today’s information to bet on mainstream compute 2 to 3 years out.
During the V4 and V5 eras, Google’s internal focus still included recommendation and ranking. The sparse core Henry worked on was optimized for sparse computation such as embeddings, sacrificing some dense-matrix area; as a result, its paper matrix capabilities may have looked weaker than a GPU’s at the time.
The LLM direction was visibly prioritized from V6 onward, with separate training and inference variants; in Henry’s view, V7 “should have been the right bet.” 泓君 noted that an aggressive design might lift performance 30% to 40% on one model but become painful when the model changed; Henry called that tradeoff a reality-based compromise.
9. MoE shows that network topology can determine whether an entire model class can run
Early TPU used a 2D torus, with each chip mainly communicating with neighboring nodes. MoE routing, however, sends tokens to experts distributed across different chips; without an efficient path, it is like “trying to find a friend and having to go through many intermediaries,” creating congestion and waste.
By V4, TPU had introduced OCS optical switches and a software-configurable 3D torus. Software could reconfigure communication routes among different clusters in a Pod, improving the cross-node data exchange MoE requires; Henry says the change solved one of MoE’s pain points.
The example also exposes the two sides of an ASIC: Google had an information advantage because it understood Transformer and internal model workloads earlier, but once a chip is fixed, it cannot adapt to monthly algorithm changes. If matrix computation remains the core workload, TPU can continue to adapt; if an entirely different paradigm emerges, GPU generality will regain the upper hand.
10. Inference cost advantages start with the network, not just the compute chip
泓君 observed that Google’s API pricing early on was about 1/10 of OpenAI’s and Anthropic’s, while Claude 4.5 API pricing fell 67%, and asked about TPU’s contribution as reported by the media. Henry only explicitly agreed that inference costs are related to TPU; he did not attribute the entire price reduction to the training chip.
Henry also noted that Google’s current inference chip itself costs considerably more than a GPU; but GPU clusters depend on NVLink and NVSwitch, which he called an “infrastructure tax”: data centers must also buy and deploy large amounts of switching equipment. TPU relies more on copper chip-to-chip interconnects and uses optical switches only at some nodes, allowing the same communications through a different topology and creating an infrastructure-spend advantage.
TPU is not without expensive components. It relies heavily on SerDes to transmit signals reliably from one chip to another, and liquid cooling is another cost; the difference is that its CapEx mix differs from GPU’s, with lower relative switch costs but heavier communication-chip and stability-validation requirements.
Attention and KV cache are making inference increasingly memory-bound; the key question is how quickly data can be moved out of memory. Ironwood therefore prioritized low latency, high throughput and memory bandwidth, especially avoiding LLM decode being bottlenecked by memory; Henry sees this as a signal that Google is bullish on incremental inference growth.
11. TPU’s strongest use case is a stable model with massive request volume, not a single-user Agent
Henry set out clear conditions for challenging GPU: the model must be relatively fixed, its form static and the deployment scale large enough. After training, TPU can use system-level optimization to lower inference costs; local deployment, small customers and frequently changing models make it difficult to amortize the system cost.
TPU combines large numbers of requests into a large batch and converts them into dense matrix computation. If there are not enough users per second, the software cannot pack requests effectively; cloud services such as Gemini, ChatGPT and Claude, with millions of users, fit this economic model far better than a single-user call.
Henry distinguishes throughput from tail latency: TPU can sustain high average tokens per second, but some users will be fast and others slow; it is not well suited to extremely low single-user response times. An Agent’s multistep chain cares more about single-user critical-path latency, with tail latency compounded at each step.
泓君 asked whether a small startup could use Google Cloud to batch its requests with others. The answer is yes, but that also hands scheduling and low-level optimization to Google. TPU’s scale advantage and black-box constraint come from the same centralized architecture.
12. Gemini dropping GPU backup shows software stability has finally cleared the bar
Henry says that when V7 launched, Google was basically using TPU for 100% of Gemini training. Before that, GPU served as a long-standing backup—not because training quality was worse, but because the XLA-plus-TPU software system was too complex and unstable, with too high a risk of errors during pretraining.
He judges that the system is now “very steady and very mature,” reducing the need to mix GPU and TPU. This is an important prerequisite for TPU to move from an internal Google tool to a potential external platform: if software cannot keep calling the hardware’s peak performance consistently, there is no sellable TCO.
泓君 mentioned that Gemini is occasionally unavailable because of excessive demand. Henry answered directly: “There simply aren’t enough cards.” He believes this may be related to users moving from ChatGPT to Gemini; meeting demand at that scale requires a long hardware replenishment cycle, but he does not know how long the shortage will last.
13. Broadcom controls the dirty work—and therefore more bargaining power
Google’s TPU team handles mainly front-end design; Broadcom handles ICI, back-end physical layout and signal connectivity before handing production to TSMC. Henry calls this the “dirty work,” but it requires mixed-signal, digital and analog-circuit expertise; a single signal problem can take down an entire cluster.
Henry describes Broadcom as a B2B model serving a handful of very large customers with deep customization; Marvell more often provides IP solutions to mid-sized companies and startups. Broadcom can also help large customers secure TSMC CoWoS capacity, so the relationship will be difficult to change in the short term.
Dependence can also eat into TPU profits: without a backup, Broadcom’s bargaining power keeps rising and the margin TPU can retain shrinks. Henry likens the relationship to TPU’s former role as GPU’s backup—the value of an alternative is not only technical; it also changes procurement negotiations.
Henry believes HBM may directly determine the ceiling for training and inference over the next few years: “If you cannot buy good HBM, training efficiency will take a major hit.” Investors therefore cannot focus only on Google and Nvidia; they must also watch Broadcom, the 3 HBM suppliers and how TSMC allocates scarce packaging capacity.
14. TPU began with a voice-cost crisis and ended in a tiered inference market
The episode reaches back to around 2013: Jeff Dean demonstrated deep learning’s breakthrough in speech recognition, and Jonathan Ross summed it up in 2 PPT slides—the good news was that GPUs really did work; the bad news was “we can’t afford this” （我们付不起这个钱）. If every Google user submitted 3 minutes of speech, data-center costs could double to the order of tens of billions of dollars.
The 1st-generation TPU therefore targeted inference for online recommendation: CPUs lacked enough parallelism, while GPUs at the time lacked mature matrix units, so Google built a “matrix calculator plus memory.” Jeff Dean drove the project, David Patterson was deeply involved in the early architecture, and the 2nd generation began handling training; hardware came first, while software and capacity spent years catching up.
On the generation used for AlphaGo, Henry can only say it was probably V2 or V3; PaLM, BERT, early Transformer and later Transformer models all continued on this system. By V4 TPU added sparse core, V5 and V6 shifted toward LLMs, and V7 strengthened inference; the software team ultimately became much larger than the hardware team, with DeepMind acting more like the “brain” that determined the next generation’s direction.
Jonathan Ross later took his TPU compiler experience to Groq; Henry says Groq is essentially a compiler company, able to predefine what the LPU does on every cycle. SRAM and dedicating more LPU resources to each user buy low tail latency, making it suitable for Agents, real-time voice and high-frequency trading; the episode also mentioned Groq’s involvement in Middle East data centers and local clusters in partnership with IBM. Running models with very large parameter counts is expensive, while smaller deployments are easier to control.
Henry also said DeepSeek’s emergence has put more focus on lowering inference costs and on edge and local deployment. The end state is a segmented market, not replacement: TPU for the largest-scale deployments, Groq and other ASICs for low-latency workloads or private clusters, and hyperscalers continuing to develop their own chips. “You cannot say GPUs will rule the entire market, nor can you say TPUs will.”（你不能说GPU将来会一统江山，也不能说TPU会一统江山） TPU is an important challenger today, but truly breaking the monopoly still depends on ecosystem, capacity and customers’ ability to migrate.