The ARC Prize 2024 Winning Algorithm [Daniel Franzen and Jan Disselhoff]
Summary
The winning edge came from inference architecture, not raw model scale: the standalone LLM scored about 41 points, while search and external scoring lifted the official result to 53.5. A reported 56.5-point run failed to finish before the deadline. The investor read-through is straightforward: carefully engineered test-time compute can unlock substantial value from open, relatively small models—“the two approaches worked very well together.”
Test-time training converted each ARC task’s three demonstrations into a temporary, task-specific curriculum. Rotations, reflections, color shifts, and example reordering expanded the scarce data; training separately on every task worked just as well but was slower than batching tasks under Kaggle’s limit. The team was “heavily data limited,” yet symmetry turned each example into something the one-dimensional LLM perceived as novel.
Daniel Franzen’s depth-first search made candidate generation both broader and cheaper than stochastic sampling or beam search. It traversed only output paths above a tested probability cutoff ranging from roughly 10% down to 1.77%, stored “exactly one path,” and pruned implausible branches early; lowering the cutoff from 10% to 1% improved a measured score only from 70% to 71%. A stronger model could even cost less than expected by becoming decisive sooner and pruning more aggressively.
The system judged its own candidates by viewing each through 16 transformed perspectives—and its lack of 2D invariance became an asset. Summing log probabilities, effectively multiplying the perspective scores, beat adding raw probabilities because a wrong answer might look plausible from several angles yet receive a 0.01% score from one revealing rotation. Their signature move was to “abuse the fact that [the] LLM is not perfect at 2D tasks.”
Model specialization became central, with general language capabilities removed from the final system. Llama 3.2 3B was roughly as strong as the tested 8B while running faster, and the final ARC model’s vocabulary was cut to roughly 140 tokens: digits, newline and special markers, and input/output prefixes—“it can’t” perform chain-of-thought in language because that capacity was deleted. For fixed budgets, Jan’s view was that smaller models let operators “generate more, evaluate more, filter more.”
Performance depended materially on exposure to related ARC concepts. The paper reported about 72% on a held-out portion of the evaluation data, but performance dropped when training excluded evaluation-set tasks. Unfamiliar counting and size-estimation problems remained especially difficult. The winners called their approach “a hacky solution” and argued that LLMs were “too strong for this contest” relative to more AGI-oriented approaches.
The experiments favored targeted adaptation over ever-larger weights, but remained compute-intensive. Training grew from two-to-four-hour runs to two days and finally eight days on an NVIDIA H100; when the hard ARC dataset arrived with four days remaining, eight H100s enabled the last-day attempt. LoRA ranks above 128 and even full fine-tuning delivered no clear gain, while the layered LoRA setup preserved a “safety blanket” in the pretrained ARC model.
Deep dive
1. A one-dimensional LLM learned the two-dimensional grid directly
Franzen and Disselhoff represented every color as one token, serialized each grid line by line, and fed it directly into an LLM—no program search or other preprocessing, and no symbolic intermediate representation. One winner’s initial heuristic was that this “seems crazy,” because the model had to infer a 2D structure while operating entirely in 1D.
A raw language model was effectively useless: it could emit endless end-of-sentence tokens, malformed rows, or numbers without a valid grid. After training on the official ARC set and experimenting with Re-ARC’s virtually unlimited generated examples, Llama 3.2 3B reached roughly 10%-20%; multiple inference tricks could push that toward 20%-25% on evaluation data.
Their model path ran from 8B to 12B, then back to Llama 3.2 3B because the smaller model was “essentially as strong as the 8B model” and left room for more Kaggle computation. Near the deadline, they switched back to a larger, unspecified model.
Explicit dimensions, coordinates, and even 2D positional encoding produced little or no improvement. The model simply learned newline boundaries and correct row lengths: “It just worked fine.”
2. Test-time training turned three examples into a local curriculum
The pipeline had two training stages: long pretraining on public ARC-derived data, followed at inference by fine-tuning on the demonstrations supplied with each unseen task. The team withheld one demonstration’s output, trained the model to recover it from the others, and then predicted the actual challenge output.
The host raised the fair “is that cheating?” objection. Their answer: training separately on every task, without seeing other evaluation tasks, worked just as well; batching multiple tasks was chosen because repeated retraining consumed too much of Kaggle’s fixed runtime.
With only about three demonstrations per task, augmentation carried the load. Rotations, reflections, valid color shifts, and reordered examples preserved the underlying rule while appearing novel to a non-invariant text model; more ambitious checks, such as collapsing suitable tasks into black and white, remained too slow or unfinished.
3. Depth-first search exposed answers that ordinary sampling missed
Greedy generation repeatedly chose the single most likely token, while stochastic sampling produced too many unhelpful variations and beam search consumed memory per beam. Daniel’s custom depth-first search instead treated token predictions as a tree and enumerated complete outputs whose path probability exceeded a cutoff.
The implementation stored only one active path and abandoned a branch when it was no longer promising. At a 10% bound, probability mass permits at most ten qualifying paths, giving the team variable but bounded computation and usually far fewer candidates.
Practical cutoffs ranged from 10% down to 1.77%. Searching down to 1% could theoretically return 100 paths, but increased a measured result only from 70% to 71% while becoming unaffordable on Kaggle—an unusually clear marginal-cost curve for test-time search.
The guests resisted generalizing this directly to prose: ARC has ten possible next-answer tokens and one correct answer, whereas language has many valid paths whose probabilities must be aggregated by answer. They nevertheless expect path selection to matter more for reasoning models such as o1 or o3, where incorrect intermediate code or mathematics can derail the result.
4. Sixteen perspectives made self-scoring reliable
Code-generation systems such as Ryan Greenblatt’s had a major advantage: generated programs could be run against the examples. Direct pixel prediction offered no such executable check, leaving Franzen and Disselhoff with roughly 10-20 candidates and no formal guarantee that the best-looking one was correct.
Their answer was to score candidates under 16 augmentations of the problem, calculating the model probability under every view and aggregating the scores. Adding log probabilities—equivalent to multiplying probabilities—selected answers markedly better than summing raw probabilities.
The proposed mechanism is a weakest-link test. A correct answer may never receive an exceptional score, but generally remains plausible from every perspective; a false answer can look excellent under several rotations yet collapse to 0.01% under the angle that exposes its inconsistency.
The host spotted an apparent contradiction: training teaches rotational variants, yet scoring benefits from rotational inconsistency. The guests called it a trade-off—generation should improve from every direction, but the imperfect model still processes left-to-right and top-to-bottom. Rotation changes which evidence arrives first, letting it notice, for example, that a line ends at an impossible empty pixel.
5. Specialization beat language transfer but increased conceptual dependence
The team removed almost the entire vocabulary, leaving roughly 140 tokens for digits 0-9, newlines, special tokens, and input/output prefixes. That saved surprising amounts of RAM and eliminated prompting or verbal chain-of-thought: the final network could output only ARC representations.
Tokenization was treated as part of the algorithm. Compound number tokens would make output geometry unstable, so they removed everything except single digits, line markers, special tokens, and prefixes. Their multiplication analogy was that OpenAI has almost every three-digit number as a token, so multiplying two three-digit numbers requires remembering roughly one million multiplication pairs.
The paper reported roughly 72% on a held-out slice of evaluation data, but performance fell when no related evaluation tasks appeared during training. The guests called this “conceptual leakage”: training on flood-fill would not transfer directly to counting, but exposure to counting and other concepts could help the model recombine ideas into novel challenges.
6. ARC’s huge output space hid only a few consequential decisions
ARC’s roughly 30-by-30 ceiling was designed around human reliability. The host noted that it also seemed to be the upper bound of what an LLM could correctly handle, and suggested that at 50-by-50 or 60-by-60 competitors likely would not use direct LLM generation; he also noted that o3’s errors increased with solution size.
The theoretical pixel space overstated the practical search problem. Backgrounds were often trivial, and in moving-object tasks the model could generate the rest correctly once the object began in the right position.
In one intermediate sampling analysis, choosing the second-highest-probability token only three times would have solved 80% of the challenges. The difficulty was identifying which three choices among roughly 900 tokens mattered—“the number of decisions we have to do…is surprisingly low.”
DFS reduced the problems associated with long stochastic outputs by enumerating every sufficiently probable path instead of taking one random trajectory. That does not prove effortless scaling to 100-by-100 grids, but it explains why the observed complexity was much smaller than the combinatorial output space suggested.
7. Fixed compute favored small models, LoRA, and selective forgetting
For a fixed budget, larger models were not automatically better: more parameters took longer to adapt, leaving fewer cycles for generation, evaluation, and filtering. A lightly tested model in the roughly 32B range produced no meaningful gain, though the team cautioned that they lacked time to tune it properly.
Model size also interacted nonlinearly with search. A stronger model required more compute per forward pass but assigned probability more decisively, allowing DFS to prune earlier; the final larger model therefore ran only “a little longer,” not by the multiplier suggested by ordinary sampling.
Continual adaptation showed a clear limit: test-time training on one task or batches up to 50 worked, while training across all 400 evaluation tasks degraded performance. The guests suspected limited LoRA capacity or forgetting—new examples might require sacrificing older task-specific knowledge rather than accumulating it indefinitely.
They merged the first ARC-pretraining LoRA into the base weights, then attached another LoRA for test-time training. Ranks above 128 and a trial of full fine-tuning made no material difference; weight decay could pull the second adapter back toward the ARC-pretrained base, providing the “safety blanket” absent from unrestricted updates.
8. Better evaluation unlocked progress—and exposed structured mistakes
Augmented self-scoring produced one of the largest discrete jumps, from about 30 to 37 points. Once candidate ranking became trustworthy, Daniel’s DFS supplied many high-quality alternatives cheaply; together they enabled a larger model, more transformations, and the eventual 53.5-point official result.
Training expanded from two-to-four-hour experiments to two days and finally eight days on one H100. When the hard ARC dataset appeared with roughly four days left, Lambda Labs supplied eight H100s for multi-GPU training. The team submitted on the last day, but the 56.5-point model did not finish in time.
Failures were usually structured, not random. Counting and size estimation generated broad candidate sets; one color-shift task’s second-best answer applied the correct conceptual transformation in the wrong direction. Even false positives tended to fill the wrong region or place the wrong object—evidence behind the team’s conclusion that attempted algorithmic refinements lost because “the LLMs were simply smarter than us.”