Groq founder and TPU creator Jonathan Ross GPU ♥ LPU Everything You Wanted to Know Nvidia GTC 2026

Summary

• The Nvidia-Groq integration is real, broad, and shipping: Ross confirms the LPX rack is “already in production” with Q3 availability — “probably one of the fastest ramps of a semiconductor in history” (legal inserted “probably”). The deal went from Groq COO Sunny Madra reaching out to Jensen about NVLink access to a deal done in weeks, with Ross working at Nvidia full-time from December 25th.
• The product splits the LLM decode workload: FFN layers run on LPUs, attention layers on GPUs — Ross’s analogy is a logistics network of 18-wheelers (GPU long-haul throughput) and delivery vans (LPU low-latency). Result: utilization rises on both chips and the Pareto curve “bends up” at high speed, reaching “thousands of tokens per second” that are “otherwise impossible to get.”
• Ross expects fast-tier pricing to stay super-linear: Anthropic and likely OpenAI already charge more-than-linear for fast tiers and Ross expects that to continue — his coal-vs-oil frame: oil costs ~7x per BTU and people pay it because “you cannot fly a plane with coal.” Enterprises should give ultra-tier tokens to their best engineers; today’s “$10K/month” spenders “might be on the low end,” heading toward engineers using millions of dollars of tokens per year.
• “Speed is intelligence” — and can beat model quality: internally Groq found Qwen-32B, “a good model not a great model,” solved every formal-reasoning math problem faster and cheaper than Anthropic’s Opus by simply iterating more — fast feedback loops beat fewer, expensive iterations.
• Inference drives the revenue cycle: “training scales with the number of researchers you have and inference scales with the number of users you have. Revenue comes from users, not researchers” — a virtuous cycle where inference leadership drives customer revenue drives more training hardware purchases.
• Energy limits inference scale: “The world right now doesn’t have enough inference compute. It doesn’t have enough energy to power enough inference compute” — the hybrid rack’s pitch is more tokens per megawatt, framed around Jensen’s gigawatt-datacenter monetization question.
• Culture read on NVDA from the inside: Nvidia, with over 40,000 people, moves “just as fast, no slower” than 450-person Groq — “no real bureaucracy” — with a pointed on-stage dig at Google’s pace.

Deep dive

1. From outreach to Christmas: how the deal happened

• Origin story as told: early 2025, when Nvidia opened NVLink to partners, Groq COO Sunny Madra reached out to Jensen and asked “can we connect to this?” — Jensen: “Sure, why not?” Groq demoed a split GPU/LPU workload over plain Ethernet; three days after the demo Jensen called to “work more closely together,” three weeks later the deal was done, and Ross started at Nvidia full-time on December 25th.
• Ross’s advice to the room, worth flagging: “Nvidia is all about growing its partners… if you have an idea where you can use NVLink, reach out. That’s how this whole thing got started.”
• On timing and shipping: the LPX rack unveiled on stage is in production now, available Q3 — “probably one of the fastest ramps of a semiconductor in history. Nvidia’s legal department wanted us to insert the word probably.”

2. The architecture: FFN on LPU, attention on GPU

• The mechanism: FFN layers go on LPUs, attention layers on GPUs, with ~40 decoder layers meaning ~40 round trips. Run everything on one chip and you underutilize it — LPU on attention, GPU on FFN. “Putting them together, the utilization goes up for both,” and LPUs skip external-memory fetches, so you get more throughput AND better latency from the same hardware.
• The signature analogy: 18-wheelers vs delivery vans — “the best answer is both.” Ruben alone is “lowest cost tokens”; the hybrid bends the Pareto curve upward at high speed instead of continuing downward, with the arrow going “quite far off to the right.”
• The framing metric is throughput per megawatt — Jensen’s question of what you run in a gigawatt datacenter to maximize revenue. Groq’s old problem, per Ross: “we had very fast AI, but we didn’t have the throughput to deliver it… customers always wanted to adopt us, but they couldn’t. Now you mix in the GPU and you can do this at scale.”
• On Q&A pushback that LPX is cheaper than NVL72, Ross disagreed: “There’s a lot of silicon in that rack” — denser, not cheaper. On hiding activation-transfer latency: “one of Nvidia’s best kept secrets is we’re really good at networking” (Mellanox), with improvements coming.

3. Why people pay for speed: the oil-over-coal thesis

• The economics: oil costs ~7x per BTU vs coal and people rationally pay it — “you cannot fly a plane with coal.” A coder getting results in 5 minutes instead of 10 gets twice the iterations per day. The honest epistemic core: “We don’t know all the things that can be done with fast AI because we haven’t had it yet” — like early appliances screwed into light-bulb sockets (the Bezos “thin horizontal enabling layer” TED-talk riff they both referenced).
• Ross’s bigger claim: AI isn’t information-age tech like the internet (“conceptually the same thing as the printing press”). “AI is not copying data. It’s creating answers that didn’t exist before… it’s bigger [than electricity]. It’s more foundational because we’ve never had the ability to offload creativity.”
• The corporate split: value-preservation companies use AI to cut costs; value-creation companies use it to grow revenue — and creation demands iteration speed. His example: a customer feature request implemented by a coding agent “less than an hour after the meeting was over. Now, if we had more speed, that could have been done before the meeting was over.”

4. Speed is intelligence — and voice is the proving ground

• The best empirical nugget: on formal reasoning to prove circuit correctness, Opus got answers in the fewest iterations “but at the greatest cost,” while Qwen-32B solved every single math problem faster and cheaper by just iterating until it got the answer. On problems with a debug loop, “you can actually iterate much more quickly with a less capable model.”
• The girlfriend story: ChatGPT voice mode’s “That is a very good question… let me think about that” is filler to buy generation time — “those filler words are better than silence but they’re not much better.” Instant answers make conversation feel “like they’re reading your mind.”
• Voice is eating typing: an Nvidia LPU-team engineer directs Codex or Claude Code entirely by phone voice, and Ramp engineers “say they’re not typing anything” — “which is not going to be great for open floor plans.”

5. The meta-lesson: AI makes experiments cheap

• Ross’s confession — he almost rejected the experiment: “at first I was against it,” purely on opportunity cost with finite engineers on customer commitments. Sunny advocated, and Ross gave it “a small contingent”; the host asked whether it might have increased the business’s value by 5x or 10x, and Ross answered, “more.” The generalization: “If we had AI where it’s at today, it would have just tried the experiment… it’s cheap relative to the lost opportunity.”
• Compounding proof point via Sunny: “Claude code writes Claude code” — shipping updates more than once a day.
• Ross’s own routine as evidence: wakes around 6am to an AI daily briefing (email, calendar, Polymarket, Kalshi, papers), then AI-codes nonstop on 6 hours’ sleep — “in meetings just dreaming of getting back to coding.” He’d stopped coding pre-AI entirely. “AI is teaching me about AI… it’s loops all the way down.”

6. Q&A: bottlenecks everywhere, RSI hedged hard

• On what limits the Pareto frontier — the load-bearing technical answer: spreadsheet analysis of a single layer of a recent open-source model showed different operations bottlenecked respectively on memory capacity, memory throughput, compute, network latency, and network bandwidth — “fairly balanced architectures where you have to improve a lot of things to get noticeable improvement. But there’s a lot of room going forward.”
• On recursive self-improvement, heavily hedged (“purely speculative… not an area I’m aware being done too deeply”): humans learn efficiently because curiosity targets anomalies; faster rollouts in reinforcement contexts “could potentially narrow the amount of cycles needed for training by targeting more carefully.”
• On whether Groq’s role narrowed then broadened since CES: “Jensen likes to set expectations and then over-deliver… a lot of work was done internally to verify that before it was rolled out more broadly. At this point, we’re in production, so more can be said.”