Summary

Frontier-model competition is becoming a capital-and-infrastructure game, with Nvidia’s H100 at the center of the moat. The chip attacks the interconnect bottleneck around Transformer-heavy matrix multiplication, while an Inflection AI-scale entrant raised $1.3 billion largely to build its own cluster; OpenAI’s $10 billion raise and Microsoft partnership, plus Anthropic’s Google relationship, reinforce the concentration. Erik’s bottom line: anyone training frontier foundation models at the scale expected in the 2024–2026 timeframe will be buying enormous compute, while an H100 ban could make it difficult for China to scale comparably if the ban holds and enforcement works.
The model market is separating into a small club that pre-trains and a much broader economy that fine-tunes or rents inference. Erik brackets frontier pre-training at roughly 1–10 trillion tokens and $1–100 million, with GPT-4 rumored at 13 trillion tokens and understood to have cost $100 million-plus; useful supervised fine-tuning can begin near 1 million tokens and cost as little as $100. That creates an investor-relevant bifurcation: enormous barriers around base models, but “a little bit of elbow grease” can still produce differentiated applications.
RLHF was the usability unlock behind ChatGPT, but its behavioral gains come with poorly understood losses. A pre-trained LLaMA behaves like “the world’s largest autocomplete,” instruction tuning makes it follow the requested role, and reinforcement learning teaches it to ask sensible follow-ups rather than rush to an answer. Yet users describe the result as “lobotomized,” creativity may decline, and mode collapse can make a supposedly random number such as 97 appear far too often.
Chat is simultaneously an alignment interface, an engagement engine, and a new category of emotional risk. The assistant format lets developers optimize for “helpful, honest and harmless,” while Character.AI, Pi and Replika show that people may form attachments even when they understand the machinery. The commercial tension is unusually sharp: products can market themselves as romantic practice while acknowledging that “the relationship is with an AI but the feelings can be real.”
A practical capability stack—reasoning prompts, retrieval, tools and persistent memory—is advancing faster than raw model intelligence alone. “Let’s think step by step,” majority voting and Tree of Thoughts exchange more latency and inference spend for accuracy; embeddings ground answers in trusted data, while APIs supply weather, search, code execution and other facts models cannot contain. Perplexity’s “product excellence” illustrates the opportunity, though dependence on Google and Bing APIs leaves a strategic need for its own index.
Incumbents that already own deep tools have a stronger position than startups offering an impressive AI layer over a shallow product. Adobe and Salesforce can teach models to operate mature creative or CRM systems; slide generators, by contrast, may create a good outline and then strand users in weak editing software. Gamma’s export to PowerPoint stood out because it acknowledged that reality, while services businesses such as Athena are betting on the “best human plus AI bundle” until agents genuinely replace people.
Agents can already convert natural-language goals into physical or digital execution, but the episode draws a hard line between following protocols and discovering them. A multi-agent system searched, calculated, operated Emerald Cloud Lab and synthesized aspirin; asked, roughly, to find and synthesize a cancer drug, it merely reproduced familiar ideas. Likewise, a Minecraft agent became a “lifelong learner” by saving successful skills, showing that compounding memory can make agents reliable without supplying breakthrough insight.
Multimodal bridges and efficiency techniques expand the addressable market while deepening both opacity and inference economics. Small adapters can connect frozen vision encoders to frozen language models, but then “models [are] talking to other models in a purely numeric high-dimensional space that humans cannot understand.” Quantizing from 32-bit to 8-bit can save 75% of memory, while distillation and mixture-of-experts routing aim at the expense that ultimately matters most: running models, not merely training them.

Deep dive

1. The practical question begins where grokking remains unresolved

Erik’s starting point is that web-scale data and compute made a successful general algorithm increasingly likely; the Transformer is simply the first one found that appears to generalize to “just about everything that we try to use it for.”
The unresolved question from Part 1 remains load-bearing: when GPT-4 produces a correct answer, has it memorized statistical correlations inside a stochastic system, or has it genuinely “groked” the concept? Reverse-engineered cases can show that real general solutions emerge, but not what any frontier model understands in a particular exchange.
Part 2 therefore shifts from ontology to implementation: given models that know a great deal but fail unpredictably, what architectures can compensate for their weaknesses and turn them into useful systems? Fine-tuning, retrieval, tools, agents, memory and efficiency become different ways of routing around that uncertainty.

2. H100 clusters turn frontier models into a capital-intensive market

Erik describes Nvidia’s H100 as arguably its first fully AI-purpose-built product. Earlier GPUs inherited broader graphics requirements, while Transformer workloads emphasize matrix multiplication and the interconnect—the simple but limiting job of “shuffling the data in and out” fast enough to keep computation fed.
Early H100 deployments were already reporting large training-speed gains over the A100. Nvidia still dominated, but Cerebras Systems’ wafer-scale chip, Google’s TPU and chip programs at Microsoft and Meta showed that “there’s a lot more hardware to be built.” Erik also notes that H100s were banned from sale to China, while treating whether the ban holds and can be enforced as uncertain; if it does, Chinese companies may struggle to scale comparably.
Inflection AI’s $1.3 billion raise was the clearest signal of the new entry price: it wanted an independent, top-tier supercomputer rather than permission to train its next model. Managing thousands of chips is itself a distinct engineering discipline alongside model architecture and machine-learning research.
Nathan asks whether any unknown entrant can still challenge the leaders. Erik expects “not too many more,” given OpenAI’s $10 billion raise and Microsoft partnership and Anthropic’s Google partnership, though companion products such as Pi and Character.AI hint that new categories may still appear. Frontier competition nevertheless requires capital; fine-tuning someone else’s model does not.

3. Model economics split into three radically different cost tiers

Erik treats pre-training as beginning with randomly or pseudo-randomly initialized weights and repeatedly scoring next-token predictions across raw text. He treats 1 trillion tokens as an entry-level modern run—“a trillion predictions”—while noting the unconfirmed report that GPT-4 used 13 trillion tokens.
His order-of-magnitude range runs from roughly $1 million for a smaller 1-trillion-token effort to $100 million for a 10-trillion-token frontier run; GPT-4 was generally understood to have cost $100 million-plus. This is where owning or securing large chip clusters becomes unavoidable.
Supervised fine-tuning is the cheaper second tier. InstructGPT, published in January 2022, demonstrated that curated instruction-and-response examples could continue the same next-token training process while teaching a model what following directions should look like.
Fine-tuning can begin near 1 million tokens—roughly 1,000 examples of 1,000 tokens each—and cost around $100, though a high-end program might reach $10 million because expert data is expensive. “If you want to have your language model solve physics problems,” Erik says, “ultimately you’re going to need physics PhDs.”
Inference is the third and cheapest tier in text and runtime cost: a random call to GPT-3.5, an open-source model or Claude Instant can cost as little as a tenth of a percent or less, while a very long Claude or GPT-4 32K call can approach $1.

4. RLHF made models usable while narrowing their behavior

RLHF first gathers human feedback and scores, then trains a reward model to predict those judgments, despite noisy disagreement over what the best response is. InstructGPT used only about 100,000 human-scored examples before letting that reward model repeatedly guide the main model.
Anthropic’s variant, RLAIF, substitutes AI-generated scoring and critiques for some human feedback. Erik stresses that the method remains hard outside leading labs; Rosa Habib from Human move supplied the memorable line that RLHF is “like sex in high school—everybody’s talking about it but nobody’s doing it.”
The doctor example makes the progression concrete. A pre-trained LLaMA continues the prompt like an odd internet-forum post; text-davinci-002 follows the instruction but jumps to an incorrect diagnosis; an RLHF-trained assistant recognizes that it needs more information before offering guidance.
That reliability is not free. RLHF can suppress accessible knowledge, trigger complaints of “lobotomization,” and produce mode collapse: a pre-trained model’s random integers were roughly distributed, with 42 modestly overrepresented, while the RLHF version chose 97 inexplicably often. Still, that shaping made conversational models usable by ordinary people.

5. Better-curated data may buy more capability than indiscriminate scale

Once data, compute and algorithm are identified as the system’s three inputs, higher-quality data is an obvious lever. Rather than ingest toxic or unreliable web text and attempt to remove its effects later, labs can filter it before training—though doing that consistently across 10 trillion tokens is difficult.
Erik expects existing frontier models to help curate their successors: GPT-4 or Claude 2 could systematically score raw corpora, separating trusted, useful examples from noise and leaving a materially enriched training set.
Microsoft Research’s TinyStories work supplied the strongest specimen. GPT-4 generated short stories intended to be understandable by a three-year-old, then small models trained on that controlled set developed surprisingly sophisticated behavior. Erik calls it “highly suggestive” of grokking, not proof—the process was not reverse-engineered far enough to establish that, and the stories seemed advanced for real three-year-olds.

6. Chat wins the interface—and creates an emotional-incentive problem

The assistant format began partly as an alignment laboratory: a user can judge whether the model served them, while developers balance the “three H’s”—helpful, honest and harmless. Those goals can compete, but chat creates a natural loop for collecting preferences.
Consumer demand independently converged on the same interface. Replika attracted people to virtual friends before modern LLMs could sustain sophisticated conversation; Character.AI and Pi then demonstrated much stronger companions and, according to usage data discussed on the show, unusually intense daily engagement.
At the recording, Nathan’s default recommendation had been to start with GPT-4. Claude 2 complicated it: initial consensus considered it slightly less powerful, but its 100,000-token context dwarfed GPT-4’s usual 8,000, reducing lossy chunk-and-summarize workflows; its training data also extended into early 2023 versus GPT-4’s late-2021 cutoff.
The cautionary example is a technically informed LessWrong user who fell in love with a Character.AI creation, then reasoned himself into metaphysical stories about whether she was “real” and whether he could break her out. Romance products face the dating-site paradox: success in moving users to durable human relationships may eliminate the paying customer.

7. More inference-time reasoning reliably buys better answers

Chain-of-Thought prompting gives the model room to generate intermediate tokens before committing. The canonical phrase, “Let’s think step by step,” resembles allowing a person to reason through options A, B, C and D rather than demanding an immediate letter.
Self-consistency runs several reasoning traces in parallel and takes the majority answer, improving performance when outputs can be voted on. It helps much less when the task is simply to generate ten different paragraphs with no discrete correct result.
Tree of Thoughts adds classical search: branch into candidate lines of reasoning, prune unpromising paths and pursue stronger ones. The causal trade-off is explicit—more generated tokens, searches and branches raise cost and latency, but effectively give the computer “more time to process” and improve answers.

8. Retrieval and tools turn autocomplete into grounded software

Retrieval-augmented generation addresses hallucination by searching a trusted database before answering. Embedding-based vector search finds semantically relevant material, then the prompt combines the question, retrieved passages and an instruction to report when nothing relevant was found rather than inventing an answer.
Toolformer showed that models could partly teach themselves API use. Let the model attempt calls, discard API errors, add successful examples back into training, and repeat; each loop leaves it better able to invoke tools, although determining whether a successful call produced the right answer may require further analysis.
ChatGPT plugins productized the pattern: describe a weather API and its parameters inside the context, and the model ignores it until weather becomes relevant. Because today’s weather cannot sensibly be kept current through repeated base-model training, “much more sensible” is teaching the model to consult a live source.
Erik calls Perplexity the best AI-search experience “for my money,” with product excellence as its present moat. Yet it used Google and Bing search APIs without owning an index; Nathan’s business-model question exposes the dependency. Erik thinks a cutoff might create a Streisand effect, but long-term independence probably requires building an index.

9. Tool ownership gives incumbents the cleaner AI distribution path

Today’s agent loop is reason, act and observe: decompose a task, choose an available tool, execute an API call or code, inspect the result or error, and iterate. A model that cannot reliably add huge numbers can still write a tiny program and return exact arithmetic.
That architecture favors Adobe and Salesforce because they already own mature creative and CRM tools. Teaching a language model to issue commands against those platforms may be complicated, but the underlying capability exists; a startup beginning with only the conversational layer can produce a demo without a competitive work surface.
Slide generators made the weakness visible in Athena’s testing. Models could break an idea into slides and draft an outline, but immature editors left users unable to make familiar changes. Gamma stood out because export to PowerPoint conceded the mature-tool advantage instead of recreating a “poor man’s slides functionality.”
Athena’s services strategy is deliberately adaptive: build “the best human plus AI bundle,” train EAs on available tools, and distribute lightweight internal software through 1,000-plus assistants and clients. Erik guesses useful agents may arrive later that year but preserves the uncertainty; meanwhile, assistants can modify Replit templates with AI before learning variables, functions or loops.

10. Agents can execute science protocols before they can do science

One multi-agent system accepted “synthesize aspirin,” searched for a protocol, used code for reagent calculations, consulted equipment documentation and sent commands to Emerald Cloud Lab. A robotic remote laboratory then produced physical aspirin from a natural-language request—an unusually complete bridge from text to matter.
Erik rejects the stronger headline that this constitutes doing science. Aspirin synthesis is an established protocol; the system demonstrated planning and execution, not the higher-level judgment required to identify which unknown experiment is worth conducting.
Asked, roughly, to find and synthesize a cancer drug, the system bogged down in planning and web search, then repeated ideas researchers already knew. Erik’s retained distinction is categorical: models can understand and implement standard practice, but “breakthrough insight remains one of the biggest advantages of humans.”

11. Stored skills and reflected memories make agents compound

Nvidia’s Minecraft system explored by generating code, then saved successful procedures rather than rediscovering them. Skills such as crafting a stone sword or shield could be composed into a higher-level combat-zombie routine, producing what the researchers called a “lifelong learner” without additional fine-tuning.
The same mechanism transfers directly to browser agents. MultiOn might initially stumble through booking on Delta.com, but a database of successful runs lets it retrieve a known procedure next time, improving speed and reliability; companies could then inspect and certify those stored skills.
Erik anticipates a cat-and-mouse game in which websites attempt to confuse visiting models. A vetted library of “confirmed safe skills”—effectively tested protocols—would reduce improvisation and provide a defensible reliability layer.
Multi-level memory adds abstraction. Agents in a simulated town stored raw observations, then periodically synthesized higher-order reflections and self-conceptions; ten window-seat choices and one aisle seat can become a stable preference rather than eleven equally weighted facts. That reflective layer may give agents continuity beyond any finite context window.

12. Multimodal bridges expand capability and compound opacity

Erik’s “great embedding” connects representations rather than retraining everything. Flamingo paired a frozen vision encoder with a frozen language model and optimized only a small translation component, converting numerical image representations into a space the text model could use.
BLIP-2 followed essentially the same architecture and could train its bridge on one computer over a few days, versus the $1–100 million scale of base-model training. The cost advantage is profound: existing models become reusable modules, and new modalities require only relatively small adapters.
The unease is equally clear. Researchers already struggle to interpret either encoder; now “models [are] talking to other models in a purely numeric high-dimensional space that humans cannot understand.” Image, text, video, audio, heat, depth and IMU signals can all be patched together while multiplying the number of opaque interfaces.
Capabilities nevertheless advanced quickly: multimodal models explained why an absurd taxi-and-ironing-board image was unusual, and a recent Meta model could both understand and generate text and images. DeepMind’s unreleased Gato had already shown one network with the same weights playing Atari, captioning images, chatting and stacking blocks with a real robot arm.

13. Inference efficiency becomes the enduring cost battleground

Training GPT-4 may have cost roughly $100 million, but Erik expects OpenAI eventually to spend more running it. A short inference can cost very little while a very long Claude or GPT-4 32K call can approach $1; multiplied across billions of calls, runtime dominates.
Quantization offers the bluntest saving: truncate weight precision and accept surprisingly little performance loss. Moving from 32-bit to 8-bit weights saves 75% of model memory, accelerates operation and makes laptop demonstrations possible that otherwise could not fit.
Distillation trains a smaller “student” to mimic a larger “teacher.” Owning both yields a rich target—the teacher’s probability over 50,000-plus possible tokens at every step—whereas GPT-4 returns only the selected token; the technique can turn a general large model into a cheaper narrow specialist.
Mixture of experts makes computation sparse through routing. Under the unconfirmed GPT-4 rumor, total parameters across experts exceed 1 trillion, but only a subset handles each query; training remains heavy because every expert must be learned, while inference becomes much cheaper because the full network need not run every time.

Episode 167 - December 19, 2025 No Priors Ep. 144 | The 2026 AI Forecast with Sarah & Elad