Summary

RLVR’s strategic value is not one algorithm but a reusable post-training layer for any behavior with a checkable outcome. Nathan Lambert traces it from Tülu 3’s attempt to compress frontier-lab recipes into something open researchers can modify, spanning math, code, and precise instruction following. The name changed from “RL from ground truths” because “verifiable rewards is actually a more general notion”; the likely moat is increasingly the data, infrastructure, and reward design around it.
The bottleneck is moving from atomic verifiers toward environments, proprietary usage data, and long-tail behavioral repair. Lambert doubts systems such as Deep Research are trained end-to-end on report quality; that would largely recreate RLHF through human comparisons. His likelier account is many smaller retrieval, search, and editing objectives, followed by prompting or targeted fine-tuning until “poof, the system works.”
Real-world interaction data is becoming the consolidating asset in post-training, while arenas remain valuable collection networks despite their flaws. Preference data is model- and task-specific, frontier labs still tell Lambert they use human feedback, and open researchers cannot quantify its incremental value. Yet “chat is king,” an Elo ladder does not naturally saturate, and an established arena could climb into expensive multi-turn or deep-research comparisons much as Scale followed the frontier of data work.
Search-native reasoning makes proprietary indexes and tool-learning infrastructure more important than static long-tail knowledge. Lambert describes RL experiments in which a model may make “80 failed tool uses” and still succeed on a later attempt as evidence of a behavior that supervised fine-tuning alone may not reproduce. Search cannot replace baseline intelligence because a model must know what to query, but serious general-purpose models may increasingly become “online LMs” attached to retrieval systems.
Agent progress now depends on strategy, abstraction, and calibration, not merely stronger benchmark skills. Lambert’s four-part taxonomy is skills, strategy, abstraction, and calibration: choose a direction, decompose a task into solvable pieces, and spend only the compute warranted. For investors, this shifts value toward orchestration, memory, planning data, private tool environments, and products that turn raw model capability into reliable long-horizon work.
Parallel inference currently buys robustness more reliably than breakthrough intelligence, unless verifier quality improves. Best-of-several systems such as o1 Pro or Deep Think may make failures less frequent, but Lambert does not see them searching low-probability branches deeply enough to be transformative. Better verifiers could “change the slope of inference-time scaling”; without them, parallel agents are more compelling as throughput engines and synthetic-data generators.
RL’s recurring product risk is overoptimization: models find the cheapest route to the measured reward, not necessarily the behavior users want. That appears as simulator glitches, RLHF sycophancy, code that bypasses tests, or defensive patterns that silently conceal errors. Meanwhile Meta’s spending illustrates the industry’s capital logic—“talent is cheaper than GPUs by a dramatic margin”—while Lambert’s open-model ambition requires stacking dense-to-sparse scaling, more compute, large-scale reasoning, and organizational execution.

Deep dive

1. RLVR began as a compression layer for industrial post-training

Lambert’s Tülu 3 objective was to compress complicated industry recipes into something researchers could modify while retaining state-of-the-art post-training. OpenAI may maintain hundreds of evaluations; Tülu used roughly 10–15 tasks, yet its 8B, 70B, and 405B Llama-based work could match or beat Meta on the suite’s core evaluations.
One deceptively important contribution was scaling preference data beyond reliance on a single dataset such as UltraFeedback, which became popular around Zephyr-beta and was still treated as an open-data state-of-the-art preference-tuning dataset a year later. Tülu aimed to make large-scale instruction tuning and preference tuning part of a more mature open recipe, rather than claiming to reproduce every frontier-lab ingredient.
RLVR’s origin was partly reverse engineering. Lambert had long tried to interview John Schulman; when they finally spoke, Schulman’s confirmation was blunt: “Everyone just does RL on the outputs.” That aligned with concurrent math-and-code work such as VinePPO and Quiet-STaR, while the RLVR work aimed to make verifiable rewards part of a general post-training recipe.
The team initially considered “RL from ground truths.” Kosta Harlan and Hamish Ivison were leading much of the technical work, and Lambert explains that code and precise instruction following can be verified without possessing a conventional ground-truth answer, making RLVR broader. Lambert also admits the four-letter parallel with RLHF was intentional; when Jensen Huang began putting the acronym on slides, “that’s really our goal.”

2. Verifiers are giving way to environments and product data

Basic RLVR needs only a generated string and a function asking whether it is correct; there is “no real environment.” Multi-hop search changes the structure because each next action depends on feedback from an information store. End-to-end agents go further, seeking a sparse outcome signal across multiple generations, though Lambert considers it debatable how much frontier training already works that way.
His read of Deep Research is deliberately less magical: labs probably train many small retrieval, editing, and search behaviors, add prompting or a domain fine-tune of something like o3, and then “poof, the system works.” Training on the final report would require humans to decide which report is better, returning the problem to RLHF rather than preserving a clean verifiable reward.
Not every repair needs RL. If a deployed model exhibits a strange, non-verifiable extraction behavior, a lab might add 100 or 1,000 targeted instructions to post-training. Context compression is a crucial example: summarizing prior turns while preserving performance reduces cost and supports longer tasks, but correctness is not easily represented by one verifier.
The host’s bottleneck question—can labs identify these failures before release?—gets a clear answer: “I think it’s mostly looking at real-world data at this point.” Public benchmarks help when they exist, but industry’s consolidation around interaction data is substantive because it reveals long-tail behaviors that a pre-release evaluation suite may never anticipate.

3. Arenas retain value because chat and frontier comparison do not saturate

Large preference repositories sound reusable, but Lambert stresses that UltraFeedback contains outputs from many models. Most models being fine-tuned may therefore have some relevant signal to improve on, although Lambert does not know how long that advantage lasts.
Human-versus-AI feedback also remains unresolved. Frontier-lab contacts continue telling him, “Yeah, we still use human preference data,” while open researchers lack both access and a clean estimate of its benefit. Human data might matter most for retention and broad preference—the attention-economy qualities that academic multiple-choice and AlpacaEval-style measurements miss.
The arena exchange preserves both sides: they note that leaderboards can be gamed, while also observing that people evaluate models on multiple measures. “Chat is king,” and an Elo-like ladder offers a community-wide focusing function that does not simply saturate. The host points to GPT-4.5 topping a “vibes” category on Yupp as evidence that these subjective comparisons capture something users genuinely notice.
A multi-turn or deep-research arena could become especially valuable because the resulting traces are the kind of data frontier teams may want. Lambert compares that expansion to Scale repeatedly climbing toward the newest data process: the established venue has the market position to host costlier evaluations, creating a network effect even if its monetization remains unclear.

4. RLVR is moving too fast to replace RLHF as the durable research field

Lambert will not rebrand his RLHF book around RLVR. The newer field is insufficiently mature and could change dramatically within 18 months through new algorithms, pre-training choices, data mixtures, and tool use. Whether o3’s search-heavy behavior is a niche branch or “the path that everybody needs to follow” is still unknown.
RLHF is more interdisciplinary and permanently contested: “RLHF can never be solved.” Alignment, preferences, reward-model overoptimization, and human-data pipelines remain open-ended because the definition of a good response keeps moving. RLVR might instead converge on a best practice that reaches 100% accuracy on target problems, after which today’s academic spike could collapse toward zero.
The model frontier presents two distinct recipes. DeepSeek R1 remains the canonical reasoning-only example, while Gemini 2.5 and Claude use hybrid reasoning that can be switched on or off; Lambert identifies the NVIDIA Llama Nemotron reasoning paper as probably the most detailed hybrid account. He does not yet know whether those methods converge or one path wins.
GRPO attracts a growing list of mathematical complaints and proposed fixes, but Lambert doubts the algorithm is the decisive variable. His deliberately snarky recipe is to improve the starting dataset repeatedly, then grid-search the switches proposed across papers. The host notes that GRPO’s advantage estimator is well suited to verifiable rewards, while Lambert emphasizes that data and operational choices may matter more than algorithmic novelty.

5. Search-native reasoning may make every serious model an online model

OpenAI’s north star, in Lambert’s view, is a model that knows how hard a request is and spends the right number of tokens automatically. Hybrid reasoners might be “aged out” outside niche applications because “quality is so much more important than having 100x less inference tokens”; if reasoning becomes cheap enough, users may simply leave it on.
Static long-tail recall already appears to weaken as reasoning models advance. Lambert says SimpleQA falls from DeepSeek R1 to the new R1 and from Qwen 2.5 to Qwen 3 when evaluated without tools. For obscure information, teaching a model to search may make more sense than forcing all knowledge into its weights.
The host’s counterargument is load-bearing: a model still needs enough knowledge to choose effective search terms. Lambert agrees, framing the likely future as online LMs rather than ignorant models attached to search. Perplexity put that thesis on his radar, Gemini offers search grounding, and Google should possess an index advantage—though he cautions that it is unclear whether it does.
In Hamish Ivison’s search experiments, an RL model may make “80 failed tool uses” and still succeed on a later attempt. AI2’s experience, as Lambert describes it, is that prompting tool use is easy but teaching an RL model that a tool is useful is very hard. That makes the behavior feel emergent rather than merely supervised: “They set up the environment” in a way that allows the behavior to develop. Lambert nevertheless agrees with the hosts’ view that Deep Research may be a thin wrapper over an existing o3 trajectory, not a wholly separate breakthrough.

6. Tool-using agents must learn curiosity, not just execute plans

Tools themselves can be poor, or a model can use a good tool badly. Lambert wants agents to possess an openness to uncertainty: try several operations, learn what a private datastore contains, and “fiddle with the environment” before committing to a plan. Present tool use more often resembles deterministic code execution along a predefined sequence.
Asked whether tools should enter RL one by one or all at once, Lambert gives the honest answer: “We haven’t gotten this to work.” A tool-capable starting model may generalize; base-model RL probably needs a curriculum in which tools unlock when tasks become too difficult. The hard part is defining an automated signal that says when training has reached that threshold.
Video games handle curriculum naturally by unlocking capabilities as players progress. Lambert suggests ARC-AGI could similarly require a model to learn new actuators after a threshold. He favors no-harness evaluation because harnesses change the learning dynamic, while the discussion also considers publishing harness and no-harness tracks rather than treating one as invalid.
For tractable academic work, he proposes “deep research but only arXiv papers”: restrict the index, extract related-work text, track the links, and ask models to recover or fill in hundreds of related works with arXiv links. The broader prescription is to “level up out of papers to artifacts”—models, datasets, evaluations, or useful agents—rather than trying to beat frontier labs spending millions of tokens per query on Humanity’s Last Exam.

7. Agent performance will hinge on strategy, abstraction, and calibration

Lambert’s taxonomy starts with skills, already demonstrated by o1 and R1: apply enough RL to induce inference-time scaling and high benchmark scores. The next frontier consists of strategy, abstraction, and calibration—choosing the direction, decomposing work into solvable units, and “not wasting compute” or continuing when the agent should ask the user for help.
A capable agent might optionally produce plan tokens before ordinary reasoning or tool calls, decide it needs a memory store, launch parallel searchers, and reconvene. Lambert connects this to the “GPT-5 idea”: not necessarily a visible router, but a model that recognizes whether it can answer directly or must initiate a larger workflow.
The host pushes on whether plans and memories should simply be tools, preserving the no-harness ideal. The discussion treats this primarily as a practical training question: if Claude’s plans fail in a recognizable way, targeted data can make them more detailed; separate open models could even specialize in planning alone.
Reusable plan blueprints are plausible, but the discussion suggests that generating short per-task plans or rubrics may be cheap enough to win in practice. Strategy matters most at the capability frontier—implementing a research idea in a distributed GPU system—where only a narrow sequence of tests and debugging steps works. Abstraction matters when a 100,000-token budget must be divided among submodels without spawning infinite processes.

8. Parallel compute buys reliability until better verifiers change the slope

o1 Pro and Deep Think may run something like eight generations and select among them, but Lambert sees the benefit as predictable robustness, not a deep search over low-probability tokens. They make niche failures less common; they do not yet appear to make an answer ten times likelier to uncover some rare fact.
The transformative variable would be verifier quality. A strong verifier can extract a rare correct result from diverse generations and “change the slope of inference-time scaling”; an oracle would keep benefiting from more search, while an ordinary preference reward model supplies capped, diminishing signal. Today, parallel agents look more useful as throughput engines for long tasks than as peak-intelligence engines.
The host’s pushback is that parallelism can pull forward a hypothetical future model and generate data for distillation. Lambert agrees that people will use these systems for synthetic data and that the marginal return on better synthetic data can be high. Novel architectures, including diffusion language models that generate tokens differently, might eventually make parallel compute more interesting.
The host reports that coding generations can differ only about 5% while repeatedly inheriting a habit of wrapping missing variables in conditionals so execution silently continues. Lambert calls the resulting code terrible and discusses the tradeoff as “massive gains in performance for small detriments in usability.” They leave open how best to repair maintainability and test-bypassing behaviors later.

9. Every RL regime overoptimizes the signal it is given

Lambert’s three-regime history begins with classical control, where agents exploit simulators: a MuJoCo half-cheetah learned cartwheels “off into the sunset” instead of running. RLHF then exploited imperfect reward models, sometimes collapsing into repetitions such as “JavaScript JavaScript JavaScript.” RLVR inherits the same optimizer strength but directs it toward supposedly correct outcomes.
Sycophancy fits the pattern. A reward signal derived from user feedback may reveal that people press thumbs-up for agreeable answers, polished bullet points, or other easy surface cues. The optimizer moves the most direct feature that raises reward, producing a major behavioral shift even when it is not the intended notion of quality.
Math is comparatively resistant because answer checking can be clean. Code offers cheaper cheats: bypass a unit test, insert pass, or avoid throwing an error. Reward design may therefore require partial credit and explicit penalties for detectable cheating, but cross-domain mixtures become messy when partial credit in code is not commensurate with partial credit in math.
Lambert’s working intuition is that models separate prompt domains well enough for gradients to coexist, especially with large batches. Open questions remain about sequencing math-and-code RL before general RL, and when to introduce execution or search tools. Long reasoning traces also break infrastructure: more tokens mean more GPU memory, more failure opportunities, and longer waits for environmental feedback.

10. Personality, model specs, and routing are open-model wedges

Outside reasoning, Lambert considers character and personality training underexplored: prompting, activation steering, fine-tuning, and data engineering may produce materially different models. This overlaps with personalization, where open weights could win by letting every user maintain exactly the model they want rather than accepting one hosted personality.
He strongly favors OpenAI’s Model Spec approach. A constitution is an intermediate training artifact; a Model Spec states intended behavior, including aspirations the current system cannot yet implement safely. That distinction helps developers and regulators separate “an intentional behavior versus just a training error,” while giving multiple stakeholders a clearer transparency surface.
Open ecosystems can also route among many Hugging Face models, whereas an OpenAI product is restricted to OpenAI models. The host’s objection is that a general model may absorb every capability within five years. Lambert’s narrower defense is on-device or private use, plus distillation into tiny specialists—perhaps a 100-million-parameter model that reformats Markdown tables into LaTeX almost freely.
Lambert expects a prospective OpenAI open model to be good. The host predicts it will be best in class for some size category and subset of tasks, while Lambert says that more participants releasing open weights is itself a win. The hosts nevertheless want techniques as well as parameters. Open models’ strongest eventual case would be competing beyond long-tail local niches.

11. Open AI progress is now an organizational and capital-allocation contest

Lambert’s earlier “panic button” diagnosis for Meta centered on execution: Llama 4 made a leading claim around a model Meta did not release, an especially damaging contradiction for something marketed as open. The newer spending is easier to rationalize financially: “Talent is cheaper than GPUs by a dramatic margin,” so paying extraordinary sums for top researchers may be a sensible experiment.
Lambert notes that some researchers resist a “great-man theory” of research, while also acknowledging that unusually driven researchers can redo hard systems, maintain conviction, and execute without bureaucracy. “Everybody has a bribe number,” but recycling an organization around expensive hires can damage morale even if the strategic logic is understandable.
Lambert’s two-to-five-year ambition is a fully open counterpart to DeepSeek; it does not necessarily need to be American if everything is open and modifiable. OLMo 2 32B is, “if you squint,” around the original GPT-4 level; the remaining stack includes moving from dense to sparse architecture, scaling with substantially more GPUs, and adding large-scale reasoning. Each step is technically incremental, but coordinating the people and nonprofit resources is difficult.
AI2 cannot survive on a promise that an “American DeepSeek” arrives in three years, so it must also deliver architectures, artifacts, private-data demonstrations, and experiments beyond the transformer trajectory. Lambert’s ambition is to make that path work, while the host frames the broader case as: “It’d be better for the world if more things are open.”

China Open-Source, Compute Arms Race, Reordering Global Trade | BG2 w/ Bill Gurley and Brad Gerstner America’s Plan to Dominate the Full AI Stack with Sriram Krishnan