Summary

ARC-AGI-3’s reported 36% is primarily an action-efficiency score, not the share of games solved. With a proper harness, frontier models reportedly finish roughly half to two-thirds of the training games, versus under 1% on ARC Prize’s unharnessed leaderboard, but are penalized quadratically for excess actions. “36% might be misleading as a number if you don’t look behind it.”
The team’s winning preview strategy became obsolete when ARC Prize hardened the benchmark against brute force. Dries Smit’s Stochastic Goose filtered ARC-AGI-3’s 4,000-plus actions for those producing frame changes, completing 18 levels and solving two of three games within a 100,000-action budget. The main competition made no-effect actions consume time and drives scores near zero at only 2x–3x human action counts, forcing more directed exploration.
On public training games, goal acquisition is less constraining than efficient exploration and durable memory. A good 27-billion-parameter-model run can identify the right hypothesis immediately and then clear many levels; a bad one may become convinced that minimizing an energy bar or standing in a region ten times is the objective. The practical bottleneck is “abstraction-based exploration” across contexts reaching hundreds of thousands or millions of tokens.
The team’s system restores language to a benchmark deliberately stripped of instructions. The harness converts pixels into linguistic and executable representations—colors, ASCII-like grids, connected components, object descriptions and Python world models—so reasoning models can reuse concepts such as mazes, players and walls. Color permutation and rotation hurt both humans and models, revealing “some leakage of human prior into the game” and complicating claims that ARC isolates pure intelligence.
Transformers may lack formal planning internally, yet tool use makes genuine and simulated planning increasingly difficult to distinguish. Once rules are understood, an LLM can write Python and invoke search; before that, it must manage the harder meta-plan of probing the environment, revising hypotheses and deciding when to exploit. The team’s pragmatic answer is that transformers “can do a very good job of pretending” to plan in a way that may be operationally indistinguishable.
A perfect score remains a much higher bar than the headline results imply. The constrained competition requires 110 games in nine hours on one RTX 6000, while frontier-model runs can cost “like a few thousand”; 100% also permits essentially no bad hypothesis or unlucky exploratory move. The team does not expect the grand prize this year, although one member thinks something like 90% may eventually be possible even under the competition constraints.
Near-term differentiation remains in harness design, training infrastructure and research judgment—not merely buying more compute. The team trains with 25 games plus generated environments, shaped rewards and sequences of 100,000–200,000 tokens, yet says autonomous research agents still optimize tiny details while missing the decisive abstraction. Their bet is that this year’s winner will not be a pure “bitter lesson” solution, though the general gameplay patterns may later be absorbed into base models.

Deep dive

1. ARC-AGI-3 makes the agent discover the game before playing it

In LS20, a game observation is a 64×64 frame whose pixels take one of 16 colors. With no instructions, the agent must infer which object it controls, which actions move it, the environment’s dynamics and the unstated goal.
In LS20, the apparent task is navigating a maze while aligning two objects at the exit. Harnesses often reach the visually obvious endpoint and stop, because winning requires one extra move beyond what looks like the maze; a human will “just give it a go and see what happens.”
Search explodes quickly: there are eight main actions plus a mouse click with roughly 4,000 possible locations, while a complete game may require several hundred to 1,000 actions. Even after identifying the goal, naïve brute force is computationally impractical.

2. Stochastic Goose won the preview by exploiting observable change

Smit entered the preview with two weeks and only three games in that setup, making pre-training more likely to overfit than generalize. His shortcut was to explore broadly but down-rank actions that produced no frame change from a given state.
The solution used an action model, hashed experience, replay buffers and prioritized replay to begin choosing useful actions within roughly 1,000 steps. Under a 100,000-action limit, it completed 18 levels and solved two of the three games.
Smit is explicit that this was benchmark optimization, not Chollet’s hoped-for “acquisition and synthesis of abstractions.” The preview’s top four entries were essentially filtered brute-force systems, demonstrating exactly the weakness that the preview was designed to expose.

3. The hardened benchmark forced a move from search to coding agents

ARC Prize responded by making the timer advance even when a nominally valid action changes nothing, increasing game difficulty and introducing severe action-efficiency penalties. “If you go just, let’s say, 2x or 3x above the human baseline, you’re already close to zero.”
Directly predicting actions from a long sequence of frames—a more transductive strategy—did not generalize. It also allocates equal thought to every step, although sensible play reasons deeply at junctions and batches obvious actions along straight paths.
The replacement is a coding agent whose English reasoning trace reveals when it has identified objects, mechanics and goals. It writes executable Python to extract structure, build simplified world models and run algorithms such as breadth-first search, making the approach more inductive and inspectable.

4. The public score conceals substantial game-playing ability

ARC Prize’s unharnessed frontier-model chart remains below 1%, while published harnesses reach approximately 35%–36%. The guidance is not game-specific: it supplies general thinking patterns, tool access and permission to write code while leaving hypothesis formation and move planning to the model.
The score is approximately the squared ratio between the human action baseline and the player’s action count for a completed level. The team therefore says current frontier systems solve “something like half or two-thirds” of training games to completion; “it just solves them inefficiently.”
Human calibration is itself demanding. The baseline moved from the second-best human attempt per level toward a median, even a single educated player may struggle to score 100%, and team members sometimes needed an hour to understand a game.

5. Wrong hypotheses are stickier than absent hypotheses

On the public games, models can usually generate plausible goals and have achieved at least one first-level solve on every game. The private set is reportedly harder and unseen, so the team carefully limits the conclusion: goal setting “seems” not to be the bottleneck on training data.
When the first hypothesis is right, a 27-billion-parameter model may continue through many levels. After one or two wrong guesses, however, it can become locked into plainly weak stories—minimizing an energy bar or stepping in one region ten times—and struggle to reconsider.
The team frames the central problem as balancing exploration with exploitation while discovering the right level of description. Once the moving pixels are understood as an agent or meaningful object, play may become trivial; acquiring that abstraction efficiently is hard.
Scarfe’s alternative benchmark design would reward learning explicitly: permit an inefficient first solution, compress it into reusable knowledge, then require a speedrun. The team agrees this is attractive—and has been discussed for ARC-AGI-4—but defends action efficiency as a practical defense against blind search.

6. Language works because it imports a lifetime of priors

ARC removes instructions, but the team deliberately puts language back because reasoning models were trained to operate there. The harness’s job is to translate an out-of-distribution visual domain into descriptions close enough to the model’s pre-training prior.
Encoding a cell merely as a number performs worse than naming its color—such as B for blue—and stating that it is blue. The model already associates bright colors with interactive objects and dull colors with backgrounds or walls; “there’s no native primitive in a newborn that says this is a maze.”
Permuting colors or rotating boards makes the same underlying games significantly harder for algorithms and humans. Dark backgrounds, white or gray walls and recognizable game layouts reflect their human creators, leaving priors that cannot easily be stripped away.
Multiple views reduce perception costs. Alongside an ASCII-like grid, the harness supplies a simple connected-component segmentation, letting the model identify candidate objects without repeatedly spending tokens reconstructing each region.

7. Human competence may be just as path-dependent and fractured

The discussion contrasts reusable bottom-up competence with fractured and entangled representations: hill climbing can assemble a “spaghetti monster” that reaches the right answer for the wrong reasons. The sharper formulation is “performance, not competence.”
The guests’ pushback is that human cognition may share this structure. Evolution tuned people for spatial reasoning and survival rather than quantum physics; what feels like pure symbolic reasoning could be a post-hoc story over neural machinery and millions of years of implicit trial and error.
One team member’s professional-esports friend recognized a game’s pattern and completed its first level in three seconds without an unnecessary move. Five years of specialized play produced “superhuman performance,” suggesting that human baselines also bootstrap learned perspectives rather than operate from universal primitives alone.
Scarfe’s emergence analogy preserves the tension: a Game of Life glider is path-dependent and computationally irreducible from low-level updates, yet “glider” becomes a compressed, self-contained description. Knowledge such as calculus can similarly start a new layer without replaying its entire provenance.

8. ARC-AGI-3 tests two different kinds of planning

The benchmark introduces agency through interaction: an agent must acquire goals, pursue them and revise them as levels change. The team initially expected models to have no idea what to do, but now sees “signs of life” in their ability to propose useful hypotheses.
After discovering the rules, route planning is conventional: the model can write Python and call search or Monte Carlo-style algorithms. The less formal problem is planning how to learn—probing a suspected player, interpreting the result and deciding when experimentation should give way to winning.
Scarfe argues that transformers cannot implement planning as a native computer-science algorithm. The team’s pragmatic reply is that they can “do a very good job of pretending” at the exploratory layer, then externalize formal planning into code at the execution layer.

9. Long-horizon reinforcement learning is the engineering wall

The team trains on 25 games plus many self-generated environments, shaping reward with level transitions, ARC-AGI scores, successful code execution and reasoning length. Reasoning length must be managed because each game has a finite token and time budget.
Unlike ARC-AGI-2’s static puzzles, ARC-AGI-3 can require 100,000–200,000-token training trajectories and hundreds of thousands or millions of tokens across play. The current compromise trains on shorter sequences and tries to generalize the learned behavior to longer ones.
Smit first explored curiosity-driven world models: predict the next frame from the current frame and action, then reward transitions the model predicts poorly. Unable to make that work within two weeks, he reduced curiosity to the simpler assumption that “any frame change is interesting.”
A separate Mamba-backed latent vision experiment learned one procedurally generated object-ordering task without language, but required about 5,000 permutations. The result supports possibility in principle while showing why linguistic priors dramatically reduce the environment count.

10. Coding agents accelerate experiments while eroding code ownership

A team member says “we’re understanding less and less of our own code base,” sometimes using Codex to review changes too broad for a person to absorb at team speed.
Their response is requirements-based engineering: number the requirements, define how each will be tested, review them as humans, and have the coding agent explain exactly where each is satisfied. Requirements remain revisable when implementation exposes contradictions; they are not “carved in stone.”
Scarfe’s pushback is that documentation captures conclusions but not the causal, counterfactual experience that created them; Mike Knoop calls the related problem “perspectival blindness.” The team’s response stresses that requirements and implementation remain an interplay, not a frozen one-way process.
Autonomous research shows the same pathology as game play. Agents may optimize hyperparameters for a 0.01% improvement while missing a change at another abstraction level; progress came from reading long logs, identifying failures and fixing them individually, not generating “another 2 million lines of code.”

11. ARC-AGI-3 has not yet yielded to the bitter lesson

Scarfe misses ARC-AGI-1 and 2’s grassroots accessibility: ARC-AGI-3 raises the entry cost through long trajectories, inference expense and complex harnesses. The team concedes the barrier but rejects the idea that abundant compute alone solves the problem; design details still determine whether training works.
One team member’s version of the bitter lesson is historical rather than doctrinal: specialized work can reveal the structure later absorbed by scalable systems, as earlier vision techniques informed convolutional approaches. Today’s engineered harness may therefore be a necessary reconnaissance stage for a future end-to-end agent.
One team member would bet that this year’s winner will not be a pure scaling solution. ARC-AGI-2 could be attacked with enormous synthetic datasets; ARC-AGI-3 explicitly says its public problems are not representative of the private test, so memorizing a broad game distribution still leaves extrapolation.
A later bitter-lesson victory might train general thinking patterns rather than every game: how to investigate rules, manage hypotheses and turn observations into executable models. That would feel more substantive than merely having “beat a bunch of games.”

12. Passing ARC-AGI-3 would still not establish AGI

Asked directly whether a system could excel while moving no closer to AGI, a team member answers, “Yes, I do think it is possible.” The benchmark’s weaker claim is falsificatory: a system unable to solve it is not AGI; solving it does not certify general intelligence.
Under the constrained competition—110 games, nine hours and one RTX 6000—100% means never becoming trapped by a wrong hypothesis. The team does not expect that by November; one member considers roughly 90% eventually plausible, while noting that frontier runs already cost “like a few thousand.”
A theoretically optimal knowledge-acquiring agent might still score below 100% because some first moves are genuine guesses; LS20 initially offers two plausible directions. Since the baseline human also explores, luck averages out, but across 110 games “you’re going to get unlucky on one of them.”
The broader outlook stays ambivalent. The team expects the goalpost to move toward replay, speedrunning or richer worlds, while arguing that “you cannot do AI safety research in a vacuum.” One member believes in a “software singularity” followed eventually by industrial robots and self-driving cars, though next year may resemble today more than the world five or ten years out.

$VEON: a busted EM telecom hiding a 4x? | Samit Umatiya, UIG Funds 🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"