[State of Post-Training] From GPT-4.1 to 5.1: RLVR, Agent & Token Efficiency — Josh McGrath, OpenAI
Summary
- Post-training’s economic attraction is leverage: Josh McGrath contrasts chasing a 3% pre-training compute-efficiency gain with changing model behavior by 40%. The catch is operational complexity: every RL task can bring its own grader and infrastructure, multiplying the ways a run can fail. The scarce capability is therefore fluency across ML and distributed systems, not mastery of either discipline alone.
- The consequential shift from RLHF to RLVR is cleaner reward data, not merely a new optimizer. McGrath treats both as policy-gradient methods on a spectrum of signal quality: human preference is useful but less trustworthy than whether a math solution is correct. His critique of published work is that it foregrounds gradient variance and methods such as GRPO while the real innovation is “where the data is coming from.”
- From GPT-5 to 5.1, the strategically important improvement was achieving better evaluations with far fewer tokens. McGrath measures agent horizons in tokens rather than hours because a task that takes a person four hours might take Codex ten minutes; lower token use leaves room for more tool calls and actions within a practical serving budget. “Dude, I live by those charts,” the host says of quality-versus-token plots.
- OpenAI’s shopping model is a test bed for interruptible, steerable agents whose capabilities may later converge into general models. Users can watch products being considered and correct requirements mid-run—“I wanted USB-C on this.” McGrath already prefers GPT-5 Thinking on High to the original Deep Research model because published evaluations look “basically on par, if not better,” though some users value each model’s quirks.
- Long context and compaction are complementary bets, while imperfect context use looks temporary rather than fundamental. McGrath expects a continuing “dance” between enlarging context windows and developing ways to preserve useful space; graph-walk evaluations matter because they require transformations across the whole context, not retrieval from one location. The open question is whether enormous windows become direct storage or infrastructure for many sub-searches.
- Neither pre-training nor post-training is dead, and present compute allocation offers little certainty about the eventual equilibrium. McGrath invokes factories that initially used electricity through a single motor driving an entire room: a transformative technology looked unimpressive until layouts changed decades later. Expect progress to be dormant and then spiky—“It’s so over. We’re so back”—making roadmap flexibility and emotional stability more valuable than confident declarations of a settled future.
Deep dive
1. Post-training buys behavioral leverage at the cost of operational complexity
McGrath moved from pre-training data curation because the leverage looked radically different: “Do I want to make compute efficiency wins of like 3%, or do I want to change the behavior by 40%?” He does not call pre-training dead; post-training simply offered the more exciting frontier, confirmed by “many late nights.”
An RL run has more moving parts because every task can require a distinct grading setup and supporting infrastructure. At 12:30 a.m., diagnosing a suspicious run means rapidly understanding code across multiple components and contributors; whether that code came from an internal or external partner matters less than whether the researcher understands the underlying system.
The host says Codex has changed his production function: 30–40 minutes spent writing something like a design document can trigger more work in 15 minutes than he could complete manually in several hours. The unresolved problem is human scheduling—those 15-minute agent intervals are too short for another deep task, leaving him “still getting used to” a new rhythm of work.
2. Specialized agents are proving grounds for interaction patterns that later converge
The shopping model originated with an August suggestion to ship around Black Friday—a remark McGrath jokingly regretted once “now you own it” became reality. Its most important feature is interruptibility: users see which products it is examining and can revise the prompt midstream, such as adding, “I wanted USB-C on this.”
McGrath says shopping did not fundamentally require a separate model. A specialized release made it easier to test “a Deep Research-style model for shopping,” searching broadly across the internet; his expectation is that specialized and general systems eventually “converge in their capabilities.”
The same convergence is already visible between the original Deep Research model and GPT-5 Thinking on High. McGrath says OpenAI’s published evaluations appear “basically on par, if not better,” and he personally uses Thinking on High, while conceding that people may rationally prefer the older model’s quirks.
Personality remains a real product dimension, but McGrath’s own preference is instrumental: “I personally want my model to be a tool.” The host casts the choice as the Anton-versus-Clippy divide—quiet competence versus cheerful assistance—and McGrath says OpenAI is trying to provide both through toggles and custom instructions.
3. RLVR’s breakthrough is trustworthy feedback, not optimizer branding
McGrath’s framing collapses a fashionable distinction: RLHF and RLVR are both policy-gradient methods, differentiated primarily by input data. Calling RLHF “nonverifiable” is slightly strange because a trained model does predict human feedback; the deeper distinction is whether the target is preference or truth, and how much optimization the signal can safely bear.
His criticism of the research conversation is that papers often read like optimization work—focused on gradient variance—when the important axis is signal cleanliness. The practical questions are “How much do I trust it?” and what happens under sustained optimization; a human preference score is not as dependable as verifying the solution to a math problem.
The host points to GRPO from the DeepSeek Math paper as an optimization that seemed more influential than initially appreciated. McGrath redirects attention to its reward signal: “When you find the answer to a math problem, it’s a lot less debatable.” The method matters, but the new, highly trustworthy reward signal is the underappreciated industry-level change.
4. Agent horizons improve when models spend fewer tokens, not merely more time
McGrath resists defining long-horizon autonomy as 30 hours or a day; he thinks in tokens. Codex may compress four hours of human work into ten minutes, so elapsed time obscures the optimizable variable. From GPT-5 to 5.1, aggregate evaluations rose, while the quality-versus-token plot showed token use falling sharply.
Token efficiency directly changes the feasible agent: it changes how many tool calls and actions can fit within a reasonable number of tokens that can actually be served. The host says, “Dude, I live by those charts,” because they capture both benchmark capability and the practical experience of waiting for a task to finish.
The host’s pushback is that explicit GPT-5 routing and implicit control over thinking effort can collide: an upstream router may make the wrong choice even though GPT-5 itself could have calibrated the task. McGrath expects today’s growing set of knobs to simplify eventually into “one tool that you always go to,” which determines how long to think.
On context compaction, McGrath offers no product details but predicts a persistent two-sided strategy: increase context length while developing ways to keep that window useful for longer. He cautions against freezing interfaces, because a fixed abstraction could “trap” a future model improvement that requires the interface itself to change.
5. Models and systems must co-evolve because neither scaling path is settled
McGrath does not regard perfect context as an impossible dream. Citing Graphwalks, an evaluation he worked on for GPT-4.1—where he says they 10×ed the context-window effect—he notes that the evaluations are still climbing because they require multiple transformations across an entire window, unlike tests that retrieve one isolated fact. He calls imperfect utilization “definitely a temporary issue that we are climbing on over time.”
The host describes a RAG codebase over a company’s support issues: 100,000 documents totaling about 8 billion tokens. That demands retrieval systems rather than merely a 10-million-token window. McGrath’s counterpoint is deliberately uncertain: agents with GPT are “just unreasonably effective,” but giant contexts might still serve a larger plan containing many sub-searches. Model scaling and systems engineering remain co-designed rather than mutually exclusive.
The hardest hiring profile reflects that uncertainty: researchers fluent in distributed systems, core engineering, statistics, and ML. A project’s bottleneck can move repeatedly between infrastructure and learning, yet education tends not to optimize for both. McGrath’s low-confidence theoretical guess is that the two areas are roughly equally hard; some traditionally defined ML research can be treated more as a black box, while building training environments is a complicated data-engineering problem.
The host also cites a Grok 4 chart suggesting that, after scaling pre-training to a certain compute level, a comparable level of compute is now being spent on post-training. McGrath rejects declarations that either pre-training or post-training has won. Like factories that initially used electricity to drive one motor across an entire room, the industry may be using new capabilities in a layout inherited from an older technology before discovering their natural form. Ideas can appear dormant and then return suddenly, so the cycle will repeatedly feel like “It’s so over. We’re so back.”