Summary

Bonnet’s core claim is that ARC defeats current neural networks because its hidden tasks sit outside their learned distribution, not because neural networks are intrinsically incapable of learning them. If the test-task distribution were known and a model trained on it, “you would solve it”; ARC instead demands unfamiliar combinations of basic human priors that pretrained models cannot handle zero-shot.
Latent Program Networks replace search through enormous parameter or symbolic-program spaces with optimization inside a compressed continuous representation of programs. An encoder proposes a latent “best guess,” gradient search adjusts it until one latent explains every demonstration, and a decoder applies it to the test input. The wager is that this compressed space may make test-time search easier and more efficient than searching a huge parameter space.
The early result is architectural evidence, not an ARC breakthrough: roughly 10% evaluation accuracy from about 40 million parameters trained from scratch. Bonnet reports this without pretrained LLMs or evaluation-set exposure, using 400 training tasks expanded by Re-ARC to roughly 100 million examples. The models were not trained to convergence, and he says decoder capacity may prevent some tasks from being learned at all.
The claimed “no priors” result comes with an important qualification raised by Macfarlane: Re-ARC is itself a substantial prior. Bonnet acknowledges that it injects assumptions such as objects and counting, but argues those priors chiefly teach a randomly initialized Transformer how sensible 2D grids differ from random sequences, consistent with ARC’s “developer-aware” rules.
A single continuous latent space probably cannot deliver unrestricted compositional reasoning. Bonnet says Type 2 problems likely would not work and that single-thread search “would never bring you composition,” though shallow compositions may emerge. His plausible extension is multiple latent-search threads connected through an unrolled computation, potentially combined with executable symbolic programs.
The system’s failure modes point toward ensemble approaches rather than one universal reasoning architecture. Preliminary inspection found many unlearned tasks were transductive: “there is no point in latent space” representing a transformation when no compact inductive rule is needed, leaving the decoder to solve everything. Symbolic programs offer reliable interpreter execution, while latent search attacks program search’s combinatorial explosion.
Bonnet’s broader thesis is that deep learning may cover 90%–99% of tasks, while the expensive last mile needs planning and program synthesis. LLMs can place nonzero probability on novel solutions, but may require a million samples to reach some of them—making them “exponentially less creative.” Scaling LPNs is not free either: larger spaces may become less smooth and require more search during both training and inference.

Deep dive

1. ARC tests distance from the training distribution, not neural-network impossibility

Bonnet defines ARC—the Abstraction and Reasoning Corpus—as a program-synthesis benchmark for adaptation to novelty. Its private test tasks are deliberately unlike the training set and effectively absent from the internet, so pretrained LLMs perform poorly.
Macfarlane’s pushback — worth keeping: most sentences in their conversation have never appeared online either, yet an LLM can process them. Bonnet’s answer is that conversation occupies a relatively small learned latent space, whereas ARC recombines core priors “in arbitrary ways” that he believes have no close training analogue.
The categorical counterfactual clarifies the thesis: if researchers knew a distribution containing the test tasks and trained “any neural network architecture” on it, “you would solve it.” The missing capability is extreme out-of-distribution generalization, not ordinary machine learning.

2. Compression matters more than the induction-versus-transduction label

Bonnet defines transduction as directly predicting from data rather than using a solution-space representation; Macfarlane instead frames the distinction as creating a model from the data rather than reusing an existing model, and argues LPN is inductive when its weights are unchanged. Bonnet’s example of transduction is Awni Hannun’s retrieve, fine-tune, and predict procedure. He still places latent search on the spectrum of test-time training.
Bonnet’s deeper criterion is a “representation compression bottleneck.” A Python program or latent program is a small explanation that generates the outputs; shallow test-time parameter recombination is less compressed and therefore harder to search compositionally.
The kernel-trick exchange exposes why Bonnet finds the vocabulary unhelpful: linear regression can be presented as learning a compact function, or its computation can be unrolled into a direct function of training data and the new input. LPN likewise sits on the spectrum of test-time training, except it searches an input-conditioned latent space rather than the full parameter space.

3. LPN turns each task into a searchable latent program

Each input-output pair passes independently through an encoder resembling a variational autoencoder. Because infinitely many programs can explain one pair, the encoder produces a distribution over latent programs rather than a single explicit rule.
Demonstrations from the same task should produce similar distributions. The proof of concept simply averages their latent representations, then searches from that point; Bonnet expects mixture-based aggregation or low-temperature sampling could improve on the mean.
The inner loop uses the decoder and known input-output pairs as the specification. It moves through latent space toward a point that better explains every observed output, then feeds that same latent program and the unseen input to the decoder — “literally applying a program,” although the program is a vector.
Training holds out each pair in turn: the other N−1 pairs create the latent used to predict it. That prevents the shortcut of encoding the target output itself, forcing the representation to capture what connects input to output.

4. Search has to shape the latent geometry during training

The objective combines reconstruction with a Gaussian prior through the VAE’s ELBO loss. Without that prior, programs spread into “very unstructured, spiky spaces,” staying far apart and making local search effectively useless.
Search is active during training because the encoder is meant to provide intuition, not a finished answer. Without refinement, the decoder merely learns the encoder’s regression of its best-guess distribution; with refinement, “the best guess ends up being quite bad, but very close to a very good guess,” echoing meta-learning methods such as MAML.
This carries meaningful training overhead, so Bonnet suggests pretraining without search and fine-tuning with it. They tried zero-order random local search and first-order optimization; the learned decoder-likelihood surface was smooth enough that first-order gradient optimization performed well.
Bonnet’s honest gap: the team has not yet deeply mapped what different latent directions represent. Their visualization shows an initial blob of poorly understood programs, with learned programs gradually separating into clusters and occupying the space. They also have not tested deliberately adding noise to the inputs.

5. The 10% proof of concept uses small models but substantial synthetic data

The encoder and decoder are vanilla Transformers trained from scratch, roughly 20 million parameters each. They flatten padded 30×30 grids into 900-value sequences, use 2D positional encodings, and include shape information so the model can predict different grid dimensions.
Macfarlane initially marvels that Transformers trained on “hardly anything” worked at all; the crucial correction is that Re-ARC expands the 400 fixed training programs into roughly 100 million input-output data points. The extra samples vary inputs, not underlying programs, teaching relative position and grid structure.
The host’s objection is that this generator may invalidate the no-prior claim. Bonnet’s response: Re-ARC does encode human notions such as objecthood and counting, but those assumptions chiefly teach the model how to make sense of 2D grids rather than exposing evaluation solutions.
From that setup, the team obtained preliminary results around 10% on the harder evaluation distribution without pretrained LLMs or evaluation-set exposure. Bonnet stresses that performance is not high, compute ran out before convergence, and architectural bottlenecks may prevent the decoder from mastering every task.

6. Continuous programs trade combinatorial search for three new ceilings

Bonnet corrects the host’s interpolative optimism: a fixed continuous space probably cannot contain arbitrarily complex compositional programs. It may support superposition or “a couple steps of composition,” but he believes a single search thread will not bring genuine composition or solve Type 2 problems.
Their speculative bridge is multiple threads followed by staged execution: one latent program transforms the input, another consumes that output, producing a recursive, unrolled computational graph. Bonnet says brute-force ARC solutions have depths varying from 1 to 12 or 15, sometimes up to 50; Macfarlane suggests bounded neural composition might cover most tasks.
Macfarlane suggests explicit Python programs could generalize across grid sizes. Bonnet agrees that grounded programs executed by an interpreter would make sense of relevant priors, and that an interpreter would execute a found program correctly. LPN must instead both locate the correct latent and provide a decoder large enough to execute it; when the model or latent is too small, more search cannot recover a program the decoder cannot represent.
Preliminary task analysis also aligns with induction-transduction ensembles: most of the unlearned tasks appeared to be transductive. Since “there is no point in latent space” for a task requiring no compact rule, Bonnet sees obvious room to strengthen the decoder or combine latent, direct-solution, and symbolic approaches.

7. Creativity is a search-efficiency problem, and scaling may worsen it

Bonnet remains a connectionist but expects one-shot deep learning to handle perhaps 90%–99% of tasks, with discrete reasoning, long-horizon planning, and the “last percent” requiring program synthesis. Symbols could move synthesis into a combinatorial space that is completely novel and cannot be tracked or amortized.
His nuanced creativity claim is not that LLMs have zero originality. Greenblatt’s sampling approach shows a long tail with nonzero probability on strong programs, but reaching it may take a million draws: models are “exponentially less creative.”
Macfarlane pushes back that collective intelligence is itself a vast sampler — many people produce weak ideas before one flash succeeds — and that creativity includes culturally conditioned interestingness, not novelty alone. Bonnet narrows his comparison to an individual solving ARC: intuition appears in “a tenth of a second,” followed by perhaps two, three, or four hypotheses, with the fifth one correct in his example.
Brute scaling will not necessarily preserve today’s easy gradient search. Bonnet expects a larger program manifold to become less linear, interpolative, and smooth, demanding more search during training and inference alongside larger encoders and decoders. His enduring bet is therefore on “small representations” for tasks or queries, which might also help with adaptation and epistemic uncertainty, although he says current systems cannot estimate that uncertainty.

Adarsh Hiremath @ Mercor: The Fastest Growing Startup in Silicon Valley | E1261 Time To Buy The Dip?