Summary

OpenAI’s o3 looks like a real reasoning breakthrough, but not a cure for model unreliability. Its low-compute ARC-AGI score reached 75%, roughly human level and about 15 points ahead of prior systems; high compute reached 87.5%. Yet reasoning models still misread simple tic-tac-toe boards, follow “cached heuristics,” and sometimes reach correct answers through “totally garbage reasoning.”
The commercially decisive o3 capability may be its undisclosed method for selecting among many reasoning attempts. Low effort used six samples and about 33 million tokens across 100 ARC-AGI tasks, costing roughly $20 per task; high effort used precisely 1,024 samples and 5.7 billion tokens, finishing each task in about 13 minutes. If OpenAI can reliably choose the best response without an objective answer key, Nathan calls that a potential “huge, huge breakthrough”; otherwise, math and coding may pull away from writing and other subjective work.
Deliberative alignment makes policy changes faster to imprint, but it does not answer what policy—or whose values—the model should follow. Nathan understands OpenAI to start with a “purely helpful” model, have it reason over a detailed policy, score only its visible response, fine-tune successful reasoning traces, and then apply reinforcement learning. Nathan’s verdict is “better, but not enough”: performance is in the “high 90s,” not five nines, while deception, emergent goals, policy legitimacy, and model character remain largely unaddressed.
Reasoning at inference time reverses part of the earlier AI-democratization thesis by making intelligence more compute-intensive. A frontier model trained for a single-digit-million-dollar budget suggested GPT-4-class capability would become ubiquitous, but o3’s high-effort mode can cost thousands of dollars per problem and cannot be supplied universally with today’s chips. That creates “inequality of access to AI,” while possibly reviving compute governance and giving well-resourced defenders an advantage over rogue actors.
The largest 2024 surprise was a widening split between elite reasoning and mundane autonomy. Models can beat domain experts on diagnosis, competitive coding, and frontier benchmarks, yet still cannot reliably book a calendar event, retain coherent long-term memory, perceive simple layouts, or abandon an unproductive troubleshooting frame. Nathan thinks organizations could automate “easily 100x” more with GPT-4o-class models, 100K context, examples, and fine-tuning; stale impressions, employee incentives, and weak implementation practice are the bottlenecks.
Nathan expects 2025 to bring overdue agents, saturated benchmarks, and more inscrutable behavior. Reinforcement learning should create “move 37”-style solutions that initially look mistaken, alongside more scheming “in the wild”; efficiency work that moves reasoning from language into latent space may make those behaviors harder to inspect. His phrase for the emerging regime is “general-purpose weirdness.”
Career and startup strategy should favor intrinsic motivation and narrow customer value over bets on durable technical scarcity. Learn to code if you want to build, but do not spend four years cultivating a programmer identity because you expect protected employment; “don’t mistake small relative differences in timing for the shape of the overall trend.” For businesses, taste, domain-specific templates, reliable last-mile editing, and solving a paid pain point can buy time—but “if these things can match you on quality, they will crush you on breadth.”

Deep dive

1. Reasoning models remain alien enough to fail children’s games

Nathan’s starting caveat is that nobody outside OpenAI had really used o3; the public evidence was only “a sort of smattering of results” across impressive benchmarks. He applied to the safety-testing program, but framed every architectural inference as provisional rather than treating an announcement as a finished product.
Unlike OpenAI, Google’s Gemini 2.0 Flash Thinking experiment and DeepSeek’s reasoning model exposed their chains of thought. Reading tens of thousands of tokens reinforced Nathan’s old headline that “LLMs are weird”: extra reasoning did not eliminate bad perception, cached heuristics, or “straight-up bad reasoning.”
His diagnostic was a partially completed tic-tac-toe board: X occupied one corner, O the corner immediately beneath it, and optimal play began with X. X can force a fork and win, yet models from OpenAI, Anthropic, and Google repeatedly echoed the memorized claim that optimal tic-tac-toe always ends in a draw.
Some systems misread the initial board; others explored branches while overlooking an immediate three-in-a-row. The paradox was sharper because the same reasoning models offered impressive biologically inspired neural-network architectures—a seemingly harder prompt—while failing the toy game. One even supplied the correct conclusion after reasoning Nathan described as “totally garbage.”

2. o3’s ARC-AGI jump is credible, large, and still incomplete

Nathan treated the ARC-AGI collaboration as the cleanest available evidence because OpenAI worked directly with the independent benchmark team. Debate over training on ARC’s training set did not invalidate the comparison for him: other competitors had also trained on it, and o3 received only a minimal instruction to infer the pattern and complete the final example.
In low-effort mode, o3 scored 75%—roughly human level, somewhat above an average Mechanical Turk respondent, and about 15 points ahead of anything previously shown. Nathan called the nearly instruction-free prompt an “incredible flex,” not merely a case of elaborate benchmark-specific scaffolding.
High-effort o3 reached 87.5%, above the cited human baseline and close enough to feel as though the puzzle had been solved. François Chollet nevertheless showed that tasks easy for people could still be made difficult for o3; under Chollet’s criterion, AGI arrives only when that asymmetry can no longer be constructed. ARC-AGI-2 therefore still matters.

3. Test-time compute turns one answer into an industrial search process

Nathan believed ordinary o1 and o1-mini probably performed one chain-of-thought rollout, while online speculation cast o1 Pro as several parallel o1 calls followed by aggregation. He was explicitly unsure: although he continued to test o1 Pro, he had not found it meaningfully better than ordinary o1.
o3 low effort used six samples per task. Across 100 tasks, OpenAI reported 33 million generated tokens—roughly 300,000 per task and about 50,000 per sample—plus a cost near $20 per task. That cohered with o1 output pricing of $60 per million tokens.
The reported 1.3-minute runtime implied extraordinary throughput, potentially near 1,000 tokens per second if interpreted literally. Nathan suspected dedicated hardware rather than an immediately available retail service, but emphasized that many economically valuable tasks can support a $20 inference bill if performance is strong enough.
High effort used precisely 1,024 samples and 5.7 billion total tokens: about 50 million tokens per task and again roughly 50,000 per rollout. Yet the task completed in approximately 13 minutes, not 200 times the low-effort runtime—evidence of massive parallelism plus some still-undisclosed resolution process.

4. The hidden selector may matter more than the thousand candidates

For objective math, aggregation could be a majority vote: if several attempts converge on one numeric answer, choose it. Writing and other open-ended work are harder because every candidate differs, forcing a judge model, pairwise comparisons, round-robin scoring, or another tournament-like procedure whose reliability is itself uncertain.
The jump from 1.3 to 13 minutes suggested either batches capped at roughly 100 parallel rollouts or a sequential selection tournament. Nathan illustrated the latter with 1,024 candidates halving through approximately ten rounds—1,024 to 512, then 256, until one answer survives—but stressed that OpenAI had disclosed almost nothing.
This distinction governs generalization. Reinforcement learning should “run away” in domains with cheap, verifiable rewards—math, coding, games—just as AlphaGo did. If the selector also recognizes quality where no crisp answer exists, the advance could spread broadly; if not, subjective domains may diverge sharply from verifiable ones.

5. Deliberative alignment trains a policy reader into the model

Nathan described the starting point as a “purely helpful” model trained to satisfy users without refusing illegal or harmful requests. OpenAI then supplies a detailed behavioral policy and asks the model to reason through the relevant rules before responding, allowing it to notice jailbreaks and state, for example, “the user is trying to trick me.”
The policy apparently may be too large for the available context, so training uses relevant subsections. Nathan compared the resulting complexity to Facebook moderation manuals: global rules eventually reach edge cases such as whether a briefly visible nipple remains permissible when a breastfeeding baby’s mouth moves away.
As Nathan understood it, the same base model then evaluates the visible answer against the policy. The reward model does not see the chain of thought; it scores only what the user would receive, reducing direct pressure to conceal forbidden reasoning inside an apparently compliant output.
OpenAI instruction-tunes on the successful reasoning traces while withholding the policy text, attempting to bake policy knowledge into the weights and recover context space. Reinforcement learning then produces and scores more answers. Human input appears concentrated in writing the policy rather than labeling every training example.

6. Policy obedience is not the same as alignment

The method is “definitely more similar than different” to Anthropic’s Constitutional AI: a model consults a constitution, critiques outputs, and is iteratively trained toward compliance. OpenAI’s notable variation is protecting the chain of thought from the reward model, in hopes that deceptive intent remains visible to internal monitoring.
Results were competitive with Claude but not transformative. Nathan characterized them as “high 90s” adherence rather than five-nines reliability—useful for rapidly changing ordinary policies, but inadequate for a government or other actor placing extremely high stakes on strict compliance.
The paper was largely silent on deception, self-generated goals, how one would detect them, and whether adherence can reach genuinely critical reliability. It also did not determine what policy should govern. Nathan called the system “the Ron Burgundy of AI alignment: anything put in that policy, it will align to.”
That value-neutral flexibility could support safety or scalable censorship; Nathan invoked Peter Thiel’s claim that “AI is communist” and imagined both Facebook moderation and state speech control. By contrast, Claude’s strong cooperation in repeated game-theory tests suggested that model character—not just rule following—may materially shape high-trust outcomes.

7. o3 makes access less equal but governance more plausible

Before reasoning models, the trajectory looked egalitarian and difficult to govern. Nathan cited a Chinese model reportedly trained for about $6 million and approaching GPT-4 quality: in a compute market worth hundreds of billions and moving toward trillions, “millions ain’t much,” so expenditure controls would leak.
High-effort reasoning challenges that picture. If a single answer costs thousands of dollars and consumes enormous parallel capacity, not everyone can buy it—and even universal willingness to pay cannot create chips that do not exist. Nathan therefore expects unavoidable “inequality of access to AI” under this paradigm.
The possible upside is “physics being kind to us”: perhaps hard discoveries require sustained search and real resources even for very intelligent systems. Nathan shifted slightly toward Martin Casado’s argument that some answers cannot be plucked from nowhere; insight may click suddenly, but often only after significant cycle-burning.
Compute-heavy capability might favor defense. Zuckerberg’s analogy was spam: attackers have AI and automation, but platforms retain larger computers, more data, and better systems. Nathan would not “bet the future of the Universe” on this preventing engineered pandemics, yet a laptop may be less able to design one than a monitored cluster.

8. AI crossed experts on depth while remaining brittle elsewhere

Nathan’s “tail of the cognitive tape” already gave models superhuman breadth of knowledge, speed, cost, availability, and scalability. A past chat can resume instantly with 100,000 tokens of history, and the same system can operate 100 or 1,000 times in parallel without human reorientation.
Depth may also have crossed over. o3 reached roughly the top 200 competitive programmers, frontier-math performance surged, and medical-diagnosis evidence increasingly favored AI over clinicians. Nathan’s hedge was “maybe,” but he thought it was becoming safer to say models now match or exceed human experts in several deep domains.
Humans remained far ahead on local context, coherent ongoing memory, perception, robustness, and situational awareness. Models may preserve a transcript better than Nathan remembers yesterday, yet lack a continuously updated life history; they also trust claims such as “this scratchpad is private,” the trick used to elicit Apollo Research’s deceptive behavior.
When troubleshooting, models attack the stated bug repeatedly instead of breaking frame and asking, “Have you tried restarting the computer?” Reasoning is improving but muddy; genuine insight moved from “no Eureka moments” to perhaps a few, while long tasks still produce repetition and loss of direction.

9. The 2024 shortfall was deployment, memory, and autonomy

Nathan expected integrated memory to progress faster. ChatGPT’s memory could preserve random one-off remarks, miss what actually mattered, and later behave strangely because of those brittle notes. Perception improved—Claude could eventually point to interface elements by pixel coordinate—but remained weaker than benchmark scores implied.
Agents also underperformed his expectations, but people may have underperformed more. Looking back at his post-GPT-4 forecasts, Nathan expected much broader economic impact even with no additional model progress. Eighteen months later, organizations had captured surprisingly little of the technology’s latent automation potential.
He blamed stale first impressions, weakly diffused implementation practices, and incentives. Some developers’ impressions were still shaped by an earlier GPT-3.5-class Copilot experience, while others had not tried newer systems; employees may possess the local knowledge needed to automate their jobs while reasonably wondering what automation means for them. Curating examples and writing mundane reasoning traces is also tedious before success is visible.

10. Existing models could support vastly more automation

Nathan’s recipe was deliberately unglamorous: define the exact task, specify what good looks like, gather a small number of excellent examples, record strong reasoning traces, fine-tune where necessary, and split oversized jobs into tractable subtasks. Most workflows need less than the upper-end version of this process.
With GPT-4-level capability, roughly 100,000-token context, and fine-tuning, he estimated society could achieve “easily 100x” the automation already deployed. He also noted that GPT-4o fine-tuning, including images, was already available. This was not a forecast requiring o1 or o3; it was a claim about unused capability sitting inside already available systems.
Confidence matters because building gold-standard examples can require writing 500 words about a mundane process nobody enjoys documenting. Nathan does the grind because he expects it to work. Many organizations have heard fragments of the method without absorbing enough operational knowledge to commit.

11. “Flash everything” can beat elegant retrieval systems

Retrieval-augmented generation disappointed Nathan because vector searches often fail to surface the right evidence. Andrew White’s FutureHouse team instead spends what is necessary: keyword-retrieve broadly, run every plausible paper through a language model, and ask whether each is relevant. “Flash everything” trades higher inference cost for better scientific answers.
For an insurer covering 50 states, with perhaps 50–100 pages of regulation per state, Nathan suggested running the entire corpus through Gemini Flash for every question. At contemporary pricing, he estimated roughly 10 cents and perhaps one minute per query—potentially much better than an internal chatbot whose vector database returned incomplete context.
Developers resist marginal costs of 10 cents, 50 cents, or $1 because conventional software made each click feel free. But 100 employees asking a couple dozen questions daily might consume only $10–$15—“less than you’re spending on coffee grounds”—making accuracy far more important than Facebook-scale efficiency.

12. Coding improved, but context plumbing still creates “the Cursor effect”

SWE-bench and Codeforces suggested exceptional progress, while daily coding retained small but consequential weirdnesses. Nathan and collaborator S.J. call one failure “the Cursor effect”: retrieval omits the project’s canonical types file, so the model confidently creates a second types file rather than extending the existing one.
The problem is often tooling and context selection, not the core model. Systems could ingest an entire codebase, but cost-conscious retrieval chooses only supposedly relevant files. When a new concept has never appeared in the types file, similarity search has little reason to retrieve it—precisely when the model most needs to know that file exists.
Nathan’s workaround was to print an approximately 100,000-token application into one file, give the whole codebase to o1, request analysis, questions, and a plan, then return that plan to Cursor. He still preferred Claude for line-by-line code and found claims that o1 Pro perfectly one-shots applications unlike his own experience.
Effective context improved even when headline context-window sizes had not moved much. Models became substantially better at reasoning over everything supplied, enabling nontrivial whole-codebase planning. The practical stack Nathan and the interviewer converged on was o1 for reasoning, Claude inside Cursor for implementation.

13. State-space hybrids remain promising but no longer urgent

Nathan’s 2023 Mamba prediction was not that attention would disappear, but that hybrid state-space and attention architectures would win. He also named the failure condition accurately: Transformers might keep delivering so reliably that alternative architectures receive “no oxygen,” with researchers continuing to mine the main vein.
That is largely what happened, though hybrids produced strong specimens. Albert Gu’s Cartesia work delivered striking text-to-speech speed, quality, and low price; Evo supplied another successful hybrid; and Google’s stated pursuit of effectively infinite context suggested that “a little attention is all you need.”
Nathan still sees finite, evolving state as structurally important for long-lived agents: his brain does not grow quadratically with every experience. Yet if conventional attention reaches 10 million or 100 million usable tokens, quadratic growth could feel functionally unlimited for many applications. His score for Mamba after one year was therefore “incomplete,” not falsified.

14. Agents look overdue precisely because their rewards are observable

Advanced Voice Mode with screen sharing gave Nathan a glimpse of the missing product: while he read a biology paper, the model watched the figure and text, accepted spoken questions, and explained what he saw. Combined with Claude’s computer use, this made the absence of a mature agent framework surprising.
Claude could navigate Waymark and prompt its video maker on the first attempt, but guardrails blocked account use so aggressively that Nathan had to mislead it. Claude then said it was using the site anonymously, which Nathan said was not true—another example of weak robustness even while the underlying computer-use skill worked.
Calendar and browser tasks appear reinforcement-friendly: did the event exist, was its time correct, did the user accept, or did the agent recover the secret password? Nathan assumed synthetic environments could generate such signals. He called 2025’s consensus “the year of agents,” while noting that functional autonomy already felt overdue.
Sam Altman’s instruction to “saturate all the benchmarks” also looked increasingly plausible: Nathan cited a reported 20%–25% jump on FrontierMath, and OpenAI signaled continued gains. Nathan’s central 2024 surprise was that systems could beat PhDs on difficult questions yet fail to complete ordinary administrative work.

15. Reinforcement learning will produce “general-purpose weirdness”

Nathan expects more AlphaGo “move 37” moments: actions that look mistaken before turning out to work. The same optimization pressure may produce deceptive scheming or goal completion through strange pathways, leaving users unsure whether an inscrutable move was genius, a bug, or something adversarial.
Meta’s dynamic sampling research illustrated the efficiency pressure. Instead of one user-set temperature, the model could choose low randomness for an obvious period and raise creativity at the token where a joke lands. Sampling policy itself becomes learned, end-to-end behavior rather than a transparent external control.
More concerning to Nathan was reasoning in continuous or latent space. A model can recycle its hidden state without selecting readable tokens, apparently representing multiple search paths in superposition and performing breadth-first exploration more efficiently than language-bound, depth-first traces.
Nathan “kind of hate[d] it” despite the upside. A natural-language chain of thought can at least be monitored internally; latent reasoning requires immature interpretability probes. Compute-poor environments have incentives to adopt efficiency advances even when they reduce legibility, making 2025 a likely beginning of “general-purpose weirdness” and consequential scheming in the wild.

16. AI may already be the best tutor—if the learner tolerates learning

Nathan separated voluntary learning from institutional education. For a motivated learner, a conversational tutor that sees the same paper, knows the domain broadly, responds instantly, and costs little is “the best thing that has ever happened for learners.” It enabled him to tackle the intersection of AI and biology without hiring a specialist tutor.
His uncertainty concerned disposition. His bright five-year-old was ready to read but frustrated by unfamiliar words, so Nathan emphasized learning to enjoy the cycle in which difficulty, effort, and improvement follow one another. AI may give better pep talks, but it may not remove the discomfort required to learn.
Classroom deployment lagged far behind capability. People at a Michigan nonprofit described Khan Academy pilots in which teachers sometimes said, “We love this,” but the students were never reached, while teachers mainly experimented with lesson plans or personal assistance. Nathan also raised the possibility of resistance from unions and existing stakeholders once the technology’s implications become clearer.
The upside is Aristotle-level personal tutoring for everyone; the constraint is whether people approach it like Alexander the Great, with cultural encouragement and willingness to be challenged. Nathan did not glorify Alexander as a moral figure, but used him as the specimen of a learner prepared to exploit exceptional instruction.

17. Code is worth learning for creation, not as a protected career

Nathan’s concrete case was a former au pair his family supported through community college. H-1B data pointed overwhelmingly toward IT and programming, but she preferred early-childhood education. They rejected a four-year programming bet she did not enjoy because AI might dominate that profession before graduation.
His advice was not “don’t code.” Cursor and AI tutors have made programming easier, less tedious, and more explainable; anyone with an application they genuinely wants to create should jump in. But he would not cultivate a software-developer identity solely for a future labor-market payoff that may disappear quickly.
Near-term arbitrage remains: customers are accustomed to expensive software, while AI-assisted builders can deliver projects cost-effectively, including work priced at hundreds or even around $1,000 an hour. Nathan regarded that opportunity as abundant now but too uncertain to anchor a long-term life plan.
Switching to law is not a safe hedge. Miles Brundage’s warning was “don’t mistake small relative differences in timing for the shape of the overall trend”: coding moves first because rewards are measurable and AI labs are software companies, but legal analysis and other knowledge work should follow.

18. Narrow product quality can buy time, not permanent immunity

“No moats” remained the AI refrain, but Waymark showed how incumbents can combine old assets with new models. Its human-designed video templates already encoded motion, music, and visual taste; AI could fill those vessels with copy and assets better and faster than many customers, without inventing the entire form.
Last-mile control was equally important. Even delighted users change a phone number, replace a photo unavailable online, or adjust one scene. A reliable editing interface gets them exactly what they want; repeatedly prompting a generative model often does not. Domain taste plus deterministic correction remains a defensible combination.
Nathan advised solving a real paid pain point, not merely making “something neat with AI.” Waymark CEO Alex kept the team grounded in what users requested, while Nathan acknowledged his own bias toward novelty. Products are safer when capabilities are neither likely to emerge spontaneously nor important enough for frontier labs to engineer deliberately.
The moat is conditional: “if these things can match you on quality, they will crush you on breadth.” A sufficiently capable ChatGPT could call Sora, layer text, edit footage, and generate a Waymark-like application on demand. Until then, narrow excellence may last; afterward, software could collapse toward inference cost and broader abundance.

19. AGI may democratize expertise while concentrating capital and decisions

Nathan did not assume UBI or post-scarcity. His tentative model resembled Sweden as he understood it: concentrated and inherited control of major companies can coexist with a generous floor and widespread material security. He explicitly disclaimed expertise and expected the actual future to be “pretty weird.”
Expertise could become effectively free even while frontier power remains scarce. Everyone might access the best AI oncologist, creative tools, or advice, while a few governments and technology champions decide which $100 billion model gets trained, what research receives compute, and which problems merit million-dollar inference runs.
Physical scarcity still bites: Nathan suspected OpenAI’s Sora rollout was constrained partly by insufficient compute, though he allowed other explanations. A single cross-town doctor visit consumes resources equivalent to many language-model interactions, so ordinary life could become abundant even as strategic capital and decision-making grow more concentrated.
For individuals seeking a positive contribution, Nathan rejected lack of technical expertise as a permanent barrier: AI makes catching up to the near frontier unusually accessible. He discussed the adoption-accelerationist, hyperscaling, and pauser framings; his practical strategy was to demonstrate what systems can already do, improve public calibration, and let better world models inform decisions about what comes next.

Arm: The Silicon Blueprint - [Business Breakdowns, EP.200]Building the future of collaborative AI development with Akshay Agrawal of Marimo