R1, OpenAI’s o3, and the ARC-AGI Benchmark: Insights from Mike Knoop on the Gradient Dissent Podcast
Summary
The investable shift is from scaling pretrained recall to spending inference compute on recomposing knowledge, with ARC-AGI v1 exposing the discontinuity. GPT-4-class systems scored roughly 4%, while o3 reached 75% and an expensive high-compute configuration reached 85%; two STEM graduates together approach 98–99%. Mike Knoop calls that sharp bend evidence of a genuinely new “ability to adapt to novelty,” not another incremental benchmark gain.
DeepSeek R1-Zero matters more scientifically than R1 because it isolates a route for reasoning through reinforcement learning and symbolic verification without human-labeled reasoning traces. Its DeepSeek-V3 foundation model was still trained on human data, a caveat Lukas Biewald forces into the discussion, but Knoop sees a possible path from primitive operators toward self-generated knowledge. The unresolved 2025 question is whether pure RL can match o3-level adaptation without supervised fine-tuning.
The headline that DeepSeek made reasoning orders of magnitude cheaper is not yet a clean cost comparison. Knoop has not deeply audited the training figures and stresses that OpenAI’s commercial o1 price includes margin and future R&D, while an open model can be self-hosted much closer to cost. Foundation-model training is also omitted from some comparisons; his hedged view is that R1/R1-Zero and o1 may occupy “a similar bucket” on true inference cost.
Enterprise agents become economically meaningful when reasoning raises consistency enough for users to relinquish supervision—not merely when tokens get cheaper. Zapier customers repeatedly said, “I just don’t trust it enough yet to go hands off,” after GPT-3.5, GPT-4, and GPT-4o failed to eliminate stochastic mistakes. Knoop expects o1, o3, and R1-class planning to unlock workloads where people are already willing to pay up to human labor rates.
ARC Prize converted an obscure research objection to pretraining dogma into an industry-wide coordination point. Awareness among AI researchers rose from perhaps 10–20% before June 2024 to probably everyone in technology by the end of the year; late leaderboard jumps, coordinated papers, and thousands of requests to test o1 revealed hidden demand. Seven or eight startups reportedly changed research direction or pivoted toward ARC.
Knoop and François Chollet’s Ndea is a concentrated counterbet that program synthesis supplies roughly half the ideas required for efficient AGI. Deep learning supplies fast approximate sketches; program synthesis searches for exact, out-of-distribution rules from only a few examples. Knoop argues that program synthesis is at least 50% of the equation, even if it is not 50% of the compute, and says the field still lacks a “transformer for program synthesis.”
AGI timelines remain governed by unpredictable step functions and an unresolved efficiency denominator. Knoop guesses ARC-AGI v2 may last 12–18 months, while ARC-AGI v3 is being designed with a goal of lasting three years. The easy-for-humans/hard-for-AI gap probably will not close at human-level efficiency for at least three to four years. Those are explicitly “wild guesses”: ARC-AGI v1 moved from 4% to 85% in two months after five years near zero.
Deep dive
1. ARC-AGI revealed a capability break, not a smoother scaling curve
Knoop separates GPT-3, GPT-3.5, GPT-4, and GPT-4o from the newer o and R series. The former mainly became more capable through larger pretraining runs, more human data, and some distillation; the latter add inference-time reasoning. He calls that “fundamentally a paradigm shift.”
Pretrained transformers do generalize because compression is not a literal database lookup. Knoop’s narrower claim is that their degree of generalization is fixed by architecture: more training can add knowledge, but it does not proportionally expand how far the system can adapt beyond that knowledge.
Biewald’s AlphaGo pushback is worth keeping: surely a system that discovers novel Go moves is reasoning. Knoop concedes domain-specific intelligence, but says its generality was supplied by researchers who modeled the domain. A human’s distinctive advantage is learning a new card game in hours or driving a new car in days.
ARC-AGI v1 stayed largely unbeaten for five years after its 2019 introduction: GPT-4-class systems managed about 4%. Then o1, o1 Pro, and o3 drove a near-straight rise toward 75–85%. “When you see a sharp bend, something has distinctly changed.”
2. Chain-of-thought became powerful when search was layered on top
The January 2022 chain-of-thought paper was Knoop’s “oh shoot moment.” Asking models to “think out loud” lifted contemporary reasoning benchmarks from roughly 30% to 70%, making him seriously consider whether the technology might be on a path toward AGI.
In its simplest form, chain-of-thought repeatedly asks for the next step, accumulates the trajectory, and grounds a final answer in that sequence. o1, R1, and R1-Zero follow that broad pattern, but single-chain versions reach only about 15% on ARC—well above GPT-4-class systems’ 4–5% range, yet still weak.
Knoop’s informed speculation is that o1 Pro and o3 add sampling and search: generate several possible thought steps, select one, then proceed to the next step. That resembles inference-time program search and, in his account, explains the leap toward 75% and the high-compute 85% result.
Biewald pauses over the astonishing mechanism: models described as limited compression systems generate more text in stages and somehow become capable of reasoning. Knoop agrees it is not obvious; the empirical jump is precisely why he thinks these systems “demand serious study.”
3. ARC defines progress as erasing what humans find easy and AI finds hard
An ARC task presents colored two-dimensional grids, several input-output examples, and a test case. The solver must infer one consistent rule, recomposing priors such as symmetry, rotation, object detection and tracking, and elementary physics for a configuration it has never encountered.
The puzzles often look almost trivial to people, which gives the benchmark its force. Existing human evidence suggests two STEM graduates together solve approximately 98–99% of ARC-AGI v1; Knoop says smart humans can effectively reach 100%, though v1 lacked sufficiently rigorous human baselines.
ARC-AGI v2 aims to add stronger studies establishing that every task is human-solvable. Its standing design principle is “easy for humans and hard for AI,” with the program aiming to drive that gap to zero rather than escalating toward problems that are also extremely difficult for people.
Biewald asks whether ARC can really be the final benchmark after so many supposed AGI milestones fell. Knoop will not guarantee the label, but says that if no easy human task remains hard for computers, it becomes difficult to deny AGI. “That is up to the world to decide.”
4. R1-Zero exposes the pure-RL bet—and its essential caveat
R1-Zero and R1 both train on and generate chains of thought, but R1-Zero’s post-training uses reinforcement learning without human-labeled reasoning demonstrations. Symbolic verifiers can run code, check equations, test compilation, or compare exact answers, then return reliable good-or-bad signals.
Biewald catches the crucial qualification: R1-Zero did not learn language from nothing. It starts from DeepSeek-V3, which was trained on human data. Knoop agrees that “this isn’t a foregone conclusion” and frames the future engineering choice as how much knowledge belongs in a foundation model versus what the system generates at runtime.
The speculative upside is recursive knowledge generation: a reasoner might produce new knowledge, add it to an expanding corpus, and seed later training runs. In an extreme formulation, a system could receive primitive arithmetic operators and attempt to bootstrap toward calculus using RL “with no humans in the loop.”
The evidence remains incomplete. Knoop says o3’s strong ARC performance required supervised fine-tuning from humans to become efficient enough, while no public result yet proves that a purely RL-trained language-model reasoner can reach the same level. He expects that question to be a major area of work in 2025.
5. DeepSeek’s price does not establish an orders-of-magnitude cost advantage
Knoop offers an “honest non-answer” on DeepSeek’s much-discussed economics: “I haven’t looked deeply into this.” He distinguishes the commercial pricing rate from underlying cost, because open-source R1 can run on local infrastructure while OpenAI must price in margin, researchers, and future R&D.
His informed speculation is that o1’s commercial inference price contains substantial margin and that true costs may be closer than users assume. He would place the R and o reasoning systems in broadly the same cost bucket, while stressing that he cannot verify the precise comparison.
Training headlines create another denominator problem. A quoted R1 or R1-Zero post-training cost may omit the investment required to create DeepSeek-V3, but Knoop does not claim to know that figure—and notes that he likewise does not know GPT-4’s complete cost.
6. Zapier’s capital discipline shaped Knoop’s approach to technical bets
Zapier lets nontechnical users connect products such as Gmail, Slack, and Salesforce, moving data and decisions through automated workflows without an engineer. Knoop estimates it serves three to four million U.S. businesses, alongside a substantial international customer base.
The three Midwestern founders originally assumed a business should sell something useful and reinvest the proceeds because there really wasn’t a venture-capital market in the Midwest in 2010. At Y Combinator, Sam Altman’s constraint question exposed the bottleneck: Knoop, Wade Foster, and Bryan Helmig were doing support until noon instead of improving the product.
They raised about $1 million to hire support quickly, yet revenue caught up before the employee’s first paycheck. Knoop jokes that the original capital can probably be traced “in lineage all the way through” to the present; its real value was the activation energy to commit full-time.
Zapier was also globally remote from 2011, when Knoop could identify only Automattic and 37signals as comparable examples. The operating history matters to his AI story: it put him close to practical workflows where reliability, not benchmark spectacle, determines adoption.
7. Early agent deployment exposed a ceiling that pretraining did not remove
After the 2022 chain-of-thought result, Knoop asked Foster to take back all of his responsibilities and run product and engineering so he could investigate AI. For six to 12 months, he and Zapier’s CTO coded continuously, building tree-of-thoughts prototypes and an internal ChatGPT-like version months before ChatGPT launched.
Tool use emerged as Zapier’s natural wedge: frozen-weight language models could gain access to the platform’s action and search endpoints, allowing them to affect business software rather than merely return text. Zapier consequently began deploying agent-like systems roughly two years before the interview.
Hundreds of customer conversations returned the same objection across GPT-3.5, GPT-4, and GPT-4o: “The promise is there,” but “I just don’t trust it enough yet to go hands off.” Automation runs unattended on servers, so occasional errors cannot be casually audited like a chat response.
That unchanged feedback clashed with 2023–24 scaling enthusiasm. François Chollet’s 2019 paper, “On the Measure of Intelligence,” gave Knoop a framework that preserved what pretraining accomplished while explaining its limits. He concluded ARC was unusually concrete evidence that scaling memorization alone would not solve adaptation.
8. One wrong Slack channel captured the commercial stakes of reliability
An early Zapier agent had to choose a Slack channel and compose a lead-information message from HubSpot and sales transcriptions. Despite a hint specifying the testing channel, it sometimes selected a shared OpenAI partnership channel whose name also contained “testing,” sending customer information there.
The team immediately disabled and cleaned up the workflow. Knoop’s reaction was, “Turn this thing off. Get it 10 feet away from me. I can’t deal with this. It’s putting my business in jeopardy.” One stochastic error could put the user’s business and an important partnership in jeopardy.
Zapier responded with hard guarantees: users select the destination channel by default, can permit free guessing, or can constrain the agent to three approved channels. That architecture acknowledges that model autonomy and deterministic control must coexist until the underlying planning is dependable.
Knoop’s practical translation of ARC-style adaptation is consistency: a system that more reliably reaches the same controllable answer becomes steerable. People may pay up to human labor rates for that reliability, so o1, o3, and R1 could unlock agents even before inference becomes cheap.
9. ARC Prize turned a neglected benchmark into a research market
Before the prize, perhaps 10–20% of researchers Knoop met knew the relevant ARC benchmark; many confused it with an older Allen Institute benchmark already beaten by language models. To him, ARC was direct, concrete evidence against the claim that pretraining scaling was sufficient.
After Biewald introduced him to Chollet, Knoop flew to Seattle and pitched ideas for raising awareness, while asking why awareness was low and why Chollet was not working on it more. Chollet had good answers to those questions, and Knoop pursued the ARC Prize idea after seeing Nat Friedman and Daniel Gross generate attention for the Vesuvius Challenge.
The 2024 contest required prize winners to open-source progress and thereby reset the community baseline. A week before closing, the leading team—near 55% and dominant all summer—hesitated to publish. Within 72 hours, two teams surged from around tenth place, a challenger moved right behind the lead, and three papers appeared almost simultaneously.
Demand resurfaced when o1 launched: thousands asked ARC Prize to test it. By the end of 2024, Knoop believed probably everyone in technology knew ARC; more surprisingly, seven or eight AI startups told him they had pivoted or redirected research toward the benchmark.
10. The contest surfaced two credible routes for adapting at test time
Classical ARC solvers used human-designed domain-specific languages containing small transformation functions, then brute-forced combinations until one matched the examples. Their generality lived in the researcher’s design choices, and combinatorial search made them painfully inefficient. At ARC 2024, newer induction and test-time-training approaches reached roughly 40–50%.
Ryan Greenblatt’s early program-induction approach used language models to generate Python programs guided by the puzzle, ran them on the known examples, and selected programs matching the outputs. It reached roughly 40%, demonstrating promise even though it could require generating hundreds of millions of programs.
Test-time training supplied the other major approach. Teams augmented each private task by changing colors or mirroring grids without altering the semantic rule, created thousands or tens of thousands of nearby examples, fine-tuned locally, and then inferred; one leading system reached around 50%.
Knoop therefore sees two evidence-backed adaptation families: chain-of-thought search, as he believes o1 Pro and o3 use, and test-time training that takes the situation, creates nearby data, and feeds it back into the model. ARC is not fundamentally a vision test; its grids are two-dimensional program-synthesis problems rendered visually to engage human intuition.
11. Ndea bets that efficient AGI requires a fusion of intuition and exact search
Program synthesis seeks a program mapping a few input examples to outputs. Unlike deep learning, which needs abundant data and offers fast approximate in-distribution judgments, a correct synthesized rule can generalize to arbitrary inputs—but searching all possible programs grows exponentially.
Ndea’s proposed fusion mirrors human puzzle-solving: deep learning quickly sketches a few plausible rules; symbolic execution checks them exactly; failures return the search to intuition for better candidates. Humans do not enumerate thousands of Python programs, and an efficient artificial system should not either.
Knoop interprets an o3 reasoning chain as a natural-language program and its sampling as program recombination. Most frontier labs, he argues, still view this as deep learning with ancillary search; Ndea instead assigns program synthesis “at least 50% of the equation,” even if not 50% of compute.
He and Chollet see the o3 ARC result as a possible AlexNet-style opening to a five-to-ten-year scaling journey. The program-synthesis field is much smaller than deep learning, and it still lacks its “transformer.” Ndea is therefore a research lab, with no near-term product offering and AGI as step one.
The longer ambition is autonomous scientific innovation, not merely solving today’s workflows. Knoop compares AGI to the printing press: people in the 1400s could anticipate wider knowledge distribution but not Wikipedia, AI trained on Wikipedia, or machines that people can talk to. The attraction is “almost more of an adventure than a problem-solving exercise.”
Efficiency remains part of the AGI definition because accuracy can now be purchased with more inference compute. Knoop tentatively favors dollars for comparing machine work with human labor: FLOPs do not translate cleanly, and wall-clock time can be distorted by parallelism.
His explicitly uncertain forecast is that ARC-AGI v2 lasts 12–18 months and ARC-AGI v3, designed to test different capabilities, has a goal of lasting about three years. He does not expect the human-easy/AI-hard gap to vanish at human efficiency for at least three to four years, but warns that step functions defy smooth extrapolation: “You don’t know whether the ideas exist yet.”