He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]
Summary
Llion Jones believes Transformers have become an “oversaturated” local minimum and has drastically reduced his own research on them. His warning comes from the RNN era: years of publishable gains from 1.26 to 1.24 bits per character were abruptly eclipsed when a deep decoder-only Transformer reached roughly 1.1. “There’s some breakthrough that will occur at some point,” he argues, potentially rendering today’s endless architectural tweaks similarly redundant.
A successor architecture must be “obviously crushingly better,” because Transformer infrastructure now creates enormous switching costs. Researchers already understand how to train, fine-tune, inspect, and serve Transformers, while scaling an incumbent model 10× can overwhelm a modest architectural improvement. For investors, the implication is that benchmark edge alone is not disruption; a challenger must offset an entire installed base of software, skills, and compute practices.
Sakana AI’s Continuous Thought Machine combines sequential internal computation, neuron-level models, and synchronization-based representations. Developed over roughly eight months and awarded a NeurIPS 2025 spotlight, the CTM treats a thought as something unfolding through time rather than a single activation state. Hewitt’s maze example makes the distinction concrete: predicting a 100- or 200-step path in one shot failed, while learning to trace it sequentially became tractable.
Adaptive compute emerges naturally in the CTM instead of being bolted on with a hand-balanced computation penalty. In a 50-step ImageNet setup, training uses both the point of lowest loss and the point of greatest certainty; easy images resolve in one or two steps, while harder ones consume more time. The model also emerged “nearly perfectly calibrated,” which Hewitt calls a “smoking gun” rather than definitive proof that the architecture is better.
The most compelling CTM evidence is behavioral: constraints cause it to discover different algorithms. During maze training it sometimes follows a path, recognizes an error, backtracks, and tries another; under a tight thinking budget, it jumps ahead, traces sections backward, and leapfrogs toward a solution. The research case is not yet that CTMs replace Transformers, but that internal sequential search produces richer behavior than a fixed-depth, one-shot computation.
Jones sees research freedom as both Sakana’s operating model and a scarce strategic asset. Product demands, publishing pressure, and investor expectations progressively narrow autonomy, encouraging safe positional-embedding papers over ideas that “might be too weird.” His counter-bet is open-ended exploration—including evolutionary searches whose largest runs may only be in the tens of thousands despite hundreds of millions spent elsewhere—while acknowledging that commercial success could eventually create the same “technology capture.”
SudokuBench suggests headline progress in reasoning still breaks on genuinely novel, compositional tasks. The best models score around 15% on handcrafted variant Sudokus, and GPT-5 improves but remains unable to solve puzzles humans can handle; models revert to “I’ll try five, I’ll try six, I’ll try seven” instead of finding each puzzle’s unique break-in. Thousands of hours of Cracking the Cryptic thought traces offer training data, yet Jones says present RL cannot reliably sample the rare reasoning trajectory required.
Deep dive
1. Transformer success has captured the research agenda
Jones’s personal call is unusually direct for a co-inventor: earlier in the year he decided to “drastically reduce” his Transformer research. The space is oversaturated—not exhausted—and he wants to use his unusual freedom to turn up exploration of architectures that are not already consuming most of the field’s attention.
The host frames Sakana through Kenneth Stanley’s Why Greatness Cannot Be Planned: discovery comes from letting researchers follow their “gradient of interest,” while committees and fixed objectives converge toward “gray goo.” Jones says the philosophy is discussed internally, and one of his central jobs as co-founder is protecting that autonomy as the company grows.
Jones’s organizational diagnosis is causal: more talent and capital intensify competition; investors eventually request returns; companies need products; pressure to publish or commercialize rises; autonomy falls. He still tells new hires, “I want you to work on what you think is interesting and important,” and says he means it.
His neglected-scale example is evolutionary search. The field has spent “hundreds of millions of dollars,” while the largest evolution-based searches are probably only in the tens of thousands; he expects something interesting when somebody truly scales them. Pitching that direction inside an environment going all-in on one technology produced “zero interest,” helping motivate his own company.
2. The RNN era shows how quickly optimization work can become obsolete
Before Transformers, recurrent networks produced the same feeling of architectural finality. Character-level language-model papers rearranged LSTM or GRU gates, changed initialization, or added hierarchical computation; improvements from 1.26 to 1.25 to 1.24 bits per character were small but legitimately publishable and exciting.
Then Jones’s team applied very deep, decoder-only Transformers to language modeling and immediately reached roughly 1.1 bits per character. Colleagues came to their desks to suggest a calculation error—perhaps the result was in nats—but “it really is the correct number.” Excellent lines of RNN research suddenly looked redundant.
The parallel he draws is deliberately uncomfortable: today’s normalization placements, training recipes, and other Transformer permutations may be another elaborate optimization of an architecture awaiting discontinuous replacement. “I personally don’t think we’re done,” he says; continued scaling is not, in his view, proof that this is the final architecture.
The host worries that foundation models are causing architectural skills to die out as data scientists and ML engineers become prompt-focused AI engineers. Jones pushes back: “We have plenty of very talented, very creative researchers out there, but they’re not using their talents.” Incentives make a safe positional-embedding paper more rational than a strange idea that may fail or prove difficult to publish.
3. Incremental superiority cannot overcome the Transformer installed base
Jones says research already contains architectures that outperform Transformers—but not by enough. The incumbent comes with accumulated knowledge, fine-tuning methods, inference systems, knowledge of how its internals work, and mature training software; therefore “being better is not good enough. It has to be obviously crushingly better.”
Transformers cleared that threshold against RNNs through much faster training and sharply higher accuracy. Deep learning similarly overcame symbolic-AI skepticism by becoming impossible to ignore. A new design now faces a gravitational pull: a clever accuracy gain can disappear when an established lab simply makes its Transformer “10 times bigger.”
The symptom Jones finds hardest to dismiss is “jagged intelligence”: a model solves a PhD-level problem and then says something obviously wrong in the next sentence. Current networks may be universal approximators that can be forced to do almost anything with sufficient data, compute, and patience, “but I don’t necessarily think that they want to.”
His poster-child analogy is a matrix-exponentiation paper’s spiral classifier. ReLU and tanh networks technically separate every point using fragmented boundaries; the alternative represents the spiral as a spiral and extrapolates its continuation. Scarfe extends the point to video-generation models: fixing hands from the wrong finger count to five may reflect more brute force, not a representation that understands a hand.
4. CTMs represent thought through time rather than a static hidden state
Hewitt says the Continuous Thought Machine took roughly eight months—long by current AI-paper standards—and rests on three ideas: an internal sequential thought dimension, neuron-level models, and synchronization as the operative representation. He stresses that it is not wildly outside the local minimum; it is a simple, nature-inspired departure developed carefully enough to earn a NeurIPS 2025 spotlight.
Mazes became the “hello world” because conventional deep learning can output an entire solution mask in parallel, whereas a human-like formulation requires a sequence: go up, right, up, left. Once the model must emit that route from start to finish, an apparently trivial vision task becomes substantially harder.
Each CTM neuron is itself a small model. Rather than mapping one pre-activation directly to one output, it consumes a finite history of that neuron’s activations and generates the next post-activation, letting dynamics accumulate at the level of individual units.
The representation then measures how pairs of neurons synchronize across their activation histories, using a dot product rather than reading only the recurrent state at one instant. Each pair can use an exponential decay at a different timescale, from immediate co-firing to long-run coordination, enriching the order-(d^2) relational space. Hewitt’s premise is that “the concept of a thought is something that exists over time.”
5. Adaptive computation falls out of the training objective
In the maze setup, the CTM continually sees the full image and retrieves information through attention while receiving, illustratively, about 100 internal thinking steps. Because neither it nor other tested models could predict 100 or 200 route steps in one shot, the team built an autocurriculum that trains just beyond current competence—one step conceptually, five in practice.
For ImageNet, the model runs for 50 steps and training identifies two indices from 0 through 49: where loss is lowest and where confidence is highest. Cross-entropy is averaged at those points. The induced behavior is clean: easy examples finish in one or two steps, while difficult ones naturally use more available thinking time.
Hewitt contrasts that result with Alex Graves’s Adaptive Computation Time work, which required a massive hyperparameter sweep and a loss penalty on computation. Tommy Leung explains why: neural networks are “greedy,” so using all available compute generally lowers loss. In the CTM, no explicit objective demanded adaptive time; “adaptive computation times seem to just fall out naturally.”
Synchronization also appears to help gradients propagate through long recurrent runs, touching neurons across many time steps; Hewitt says the model “just kind of worked” across broad hyperparameter settings. After training, it was also nearly perfectly calibrated—its confidence tracked empirical correctness—though he presents this as suggestive, not conclusive.
6. Internal sequential search opens a different scaling dimension
Hewitt treats chain-of-thought reasoning as, among other things, a new dimension for adding compute. CTMs try to internalize that sequential process rather than express every intermediate step as language, using synchronization across multiple temporal hierarchies to search a high-dimensional latent space.
His proposed advantage for ARC-like tasks is the ability to project one step, 100 steps, or 200 steps ahead while breaking a sparse problem into a process. He does not claim the architecture has solved ARC; the narrower call is that this representation may offer flexibility missing from fixed-depth Transformers and deserves direct experimentation.
Asked about neural Turing machines, Hewitt explicitly declines the strongest claim: “I wouldn’t go so far as to say that the Continuous Thought Machine is definitively” Turing-complete. The affinity is latent reasoning unfolding over time, without requiring the same difficult discrete read/write operations associated with external neural memory.
The architecture also changes how difficulty is represented. A ViT or CNN must nest the reasoning for an obvious cat and an obscure, underrepresented class into the same fixed stack; a CTM can stop at different times, naturally segmenting examples from easy to hard. That architectural curriculum resembles how humans learn without requiring a manually partitioned dataset.
7. Maze behavior shows compute constraints can invent new algorithms
During intermediate training, the team watched the CTM begin down one maze path, appear to recognize “oh no, damn, I’m wrong,” backtrack, and choose another. Later it became more efficient, using multiple attention heads to refine solutions in a more distributed way. The camera-ready supplement records roughly 14 such unexpected observations.
Tightening the thought budget produced the strangest specimen. Instead of tracing a long maze continuously, the model jumped approximately forward, filled a segment backward, leapfrogged ahead again, and repeated. “Turns out that there’s a faster algorithm to do it,” Hewitt says, though he remains unsure what the behavior means beyond demonstrating that constraints alter the learned algorithm.
Language may be another ambiguous maze: many trajectories can reach viable outputs, and a system must recognize, revise, or backtrack from a bad route. Hewitt is actively exploring language modeling, but keeps the claim conditional; adding ambiguous maze solutions is itself still an experiment he had not yet run.
8. Memory and AI scientists could turn solitary search into collective search
Hewitt describes a possible memory experiment that would restrict an agent to perhaps a 5×5 local view of a maze, forcing it to save and retrieve enough history to revisit a location and remember, “I did the wrong thing last time.” Multiple agents could share that store as an almost “cultural memory,” collectively solving a global task from limited individual views.
Jones expects models to become increasingly powerful research partners. Sakana’s AI Scientist demonstrated an end-to-end pipeline—from a seeded idea through code, experiments, results, and paper—and produced a “100% AI-generated paper” accepted to a workshop. He presents that as proof of possibility, not his preferred working arrangement.
His near-term model remains interactive: seed an idea, debate alternatives, inspect generated code, and discuss results as they arrive. The analogy is an intern: no sufficiently rich research vision can be fully transmitted once and then left unattended for four months; steering communicates tacit intent and path-dependent understanding.
Jones concedes that this collaboration may eventually become harmful. Human-plus-engine once surpassed a chess engine alone, but “adding a human into the mix actually makes the bots worse” now. If AI scientists reach the point where his input is detrimental, whether humans should step aside becomes a broader question rather than a technical workflow choice.
9. SudokuBench exposes the gap between fluent reasoning and novel breakthroughs
SudokuBench is built from handcrafted variant Sudokus: standard row, column, and box constraints plus unique natural-language rules. One puzzle says a number in its rule description is wrong; another overlays a maze in which a rat’s route to cheese must obey numerical constraints. Solving therefore requires interpreting and sometimes meta-reasoning about the specification.
Best current models score around 15%, mostly on the smallest and simplest puzzles. Jones says GPT-5 delivers a jump but still cannot solve puzzles that humans can. Every puzzle contains a distinctive “break-in,” while models typically retreat to dull enumeration: “I’ll try five, I’ll try six, I’ll try seven.”
The training asset comes from Cracking the Cryptic, whose professional solvers verbalize their reasoning through videos sometimes lasting four hours. With permission, Sakana extracted thousands of hours of detailed thought traces and made them available for imitation learning—an approximation of the hidden reasoning Andrej Karpathy wished could accompany humanity’s finished text.
Expert humans possess many “reasoning Lego blocks,” recognize which motif a rule suggests, test it, discover insufficient disambiguation, and backtrack. Present models do not show comparable meta-reasoning on the benchmark, and current RL fails because the necessary breakthrough is too rare to sample reliably. Jones argues that meaningful progress on SudokuBench would represent meaningful progress in AI reasoning generally.