Summary

The likely AI moat may shift from any single architecture toward systems that can exploit the world’s existing structure. Moore suspects that, a few years from now, researchers may conclude that transformers work but “almost any sufficiently rich architecture will work,” because real data is neither adversarially constructed nor random. The deeper opportunity is to mathematize the hierarchy of objects, parts, text, and images that makes prediction tractable.
Human-designed puzzles expose a capability gap that fluent benchmarks can conceal. A project Moore thinks involved Sakana AI asked AI to read modern Sudoku rules in English and solve the resulting puzzles; when he last checked, results were limited to a couple of 4×4 examples and perhaps a 6×6, which he described as “pitiful.” Eight of his puzzles are in the dataset. The missing capability is not brute-force search but the ability to invent a useful representation, discover a global constraint, and move fluidly between partial solutions.
The systems opportunity is increasingly in tool use, multimodal workspaces, and learned control over external memory. Humans compensate for their finite brains by choosing paper, stacks, graphs, simulators, Mathematica, and physical prototypes; Moore expects AI to improve when it can similarly say, “I need a data structure which lets me push and pop easily.” The important distinction is less whether a transformer is formally Turing-complete than whether it can recognize its limitation and construct or operate the missing scaffold.
More creative sampling is not a substitute for a generation-and-judgment loop. Turning up temperature merely reaches the distribution’s tail and can produce garbage; artists and mathematicians instead generate, inspect, reject, revise, and “hold it to the fire.” Systems that model an audience’s cognitive horizon—placing each reveal “just past” prediction without causing boredom or rejection—would address a harder capability than stylistic novelty alone.
Moore does not treat Turing completeness as a clean dividing line between intelligence and non-intelligence. “I am also a finite-state machine,” he argues, yet humans reach practical recursion by extending themselves with tools and building technologies from reusable components. Language may enable symbolic recursion, but formal reasoning remains a “thin veneer” that humans activate in specialized settings rather than the default mode of ordinary thought or speech.
Computational limits remain real, but classical hardness results have limited direct force against structured real-world data. In random inference problems, intermediate signal-to-noise regimes may contain a recoverable ground truth that exhaustive search can find while efficient algorithms remain trapped in an “amorphous mush” behind energy barriers. Real-world structure can provide escape routes, making the choice of representation as important as the amount of compute.
Opaque AI creates a particular risk where society values due process, not merely predictive accuracy. Moore would accept a black-box movie recommendation and might accept an independently validated cancer diagnostic, but he rejects proprietary opacity in criminal justice: “I don’t think we should be using opaque proprietary tools to make decisions that affect people’s fundamental human rights.” For high-stakes uses, he calls for independent testing, contestability, and, for fundamental rights, every possible form of transparency.

Deep dive

1. A frog’s view separates theoretical hardness from practical difficulty

Moore describes himself, in Freeman Dyson’s taxonomy, as “more of a frog”: a tactile thinker who repeatedly tests abstract proofs against concrete examples he can visualize or hold. Trained first in physics, he moved toward theoretical computer science after reading Gödel, Escher, Bach, then into networks and machine learning.
Computer science appeals to him because it supplies two productive hats: search for an efficient algorithm, then, if that fails, try to prove the problem hard. Hardness can even become a resource in cryptography, including the post-quantum work he has touched.
The qualification is load-bearing: conventional hardness means that cleverly adversarial examples exist, while statistical-physics results often concern random synthetic instances. “Real-world data is not designed by an adversary to be as tricky as possible,” and it is far from random; text, images, organisms, and built environments carry exploitable structure.

2. Phase transitions reveal a gap between information and efficient recovery

Moore’s magnet analogy makes the inference thresholds concrete. Above a critical temperature, atomic alignment disappears; similarly, when noise washes the ground truth out of data, “there’s literally nothing you can do” to recover it. At high signal-to-noise ratios, PCA, spectral methods, belief propagation, and related algorithms work quickly.
The intriguing middle regime contains enough information for exhaustive search to recover the truth, yet researchers believe no efficient algorithm succeeds. Candidate fits form a high-dimensional, glassy landscape in which gradient descent, Monte Carlo, and message passing can remain exponentially long in an “amorphous mush” behind an energy barrier.
Yet this theory still concerns random problems. Moore’s broader conjecture is that researchers may eventually view architecture choice as less decisive because “almost any sufficiently rich architecture will work” against a structured world; LLM success may ultimately teach us how to mathematize its hierarchy of objects and parts.

3. Moore’s own puzzles are a benchmark for missing reasoning

Modern Sudoku variants add thermometers, sum boxes, knight-move restrictions, and other rules expressed in ordinary language. Moore thinks Sakana AI worked with Cracking the Cryptic’s Mark and Simon to assemble a benchmark requiring AI to read those rules and solve the puzzles; when he last checked, he described results on even small 4×4 and perhaps 6×6 cases as “pitiful.”
His diagnosis is partly sensory. Token models begin from one-dimensional text, whereas humans can scan a row, column, or 3×3 box with their eyes and direct attention spatially; earlier ChatGPT systems struggled even with simple two-dimensional arrays such as the N-queens problem.
Translating every puzzle into Boolean satisfiability would permit brute-force machinery, but it would miss what Mark and Simon actually do: derive a high-level heuristic or global constraint and use it to guide search. Eight of Moore’s puzzles are in the dataset; he expects progress but will be “very excited and a little annoyed” when AI solves them.

4. Intelligence begins by inventing the right variables

A puzzle setter acts not as an adversary but as a “challenging but ultimately compassionate teacher.” The desired experience starts with a “vertiginous sense” of an exponentially large landscape, then delivers an insight that collapses it; exhaustive search is both boring and something humans do badly.
Classical search heuristics ask which predefined Boolean variable occurs in many constraints and should therefore be set first. Human solvers do something richer: with pentominoes, they switch fluidly between “which piece can fit here?” and “where can this piece go?”—two different ways of parameterizing the same task.
Sudoku solvers likewise create forms of partial knowledge on demand: a cell may be either 2 or 7; the 3 in a box must occupy one of three locations; two unknown cells can be colored blue because they must match; three others may be known only to differ. The representation changes before the numerical answer is known.
Moore connects that flexibility directly to science: when collaborating with a biologist or social scientist, “90% of the work is the mathematization.” Once the right structure is chosen, simulation or proof may be easy. His closest complexity analogy is an instance promised to belong to an efficiently solvable subclass—the challenge is to discover the algorithm for that instance.

5. Grounding matters when the important point is statistically unusual

Sudoku puzzles promise a unique ground truth and tell the solver when the task is finished. Real problems offer no such guarantee; what an LLM knows from the internet is “a world,” Moore says, but it is not identical to the physical world and may omit precisely what matters.
His friend Henry Farrell tested a system on an essay whose central contribution sat sideways to familiar arguments. The summary “blandified it” and reproduced the topic’s lowest-common-denominator positions, missing the author’s distinctive claim—the opposite of recognizing, “Oh, that’s an interesting point.”
Moore sees the same failure in recommendation systems that only return more of what a user already likes. A good friend can say, in effect, that although you dismiss science fiction, Gene Wolfe may satisfy your literary standards; “give me something different, you know, challenge me.”
External workspaces could narrow the gap. An AI might run and debug code, graph a function, manipulate a three- or seven-dimensional object, or formally check a proof instead of producing proof-shaped “BS.” Moore’s Platonist intuition comes from shared perception: two people imagining a cube can correct one another and agree on 8 corners and 12 edges.

6. Creativity requires exacting judgment and a model of the audience

The host’s challenge is that creativity might come from meaningful variation in texture and language without requiring a grand world model; simply raising sampling temperature, however, often reaches an out-of-distribution tail full of garbage. Moore responds by invoking Martin Amis’s The War Against Cliché: familiar phrases should appear deliberately, not merely because they are probable.
Novel juxtaposition is only the beginning. Artists and mathematicians then “hold it to the fire,” crumple drafts, consult friends or critics, and revise or abandon the work. Moore’s PhD adviser Philip Holmes wrote poetry and called it harder than mathematics; art, Moore stresses, “is not this kind of floppy thing” but an exacting reflective practice.
The host describes editing as progressive disclosure: reveal complexity layer by layer, placing each artifact just beyond the viewer’s prediction horizon without making them bounce off or become bored. Moore identifies the hidden operation as sustained mental modeling—continually imagining what this particular audience will notice and understand.
Puzzle aesthetics mix subjective constraints with seemingly objective insights. Moore dislikes “disjoint groups” Sudoku because it fights his visual sensorium—he is only “2 pounds of meat with a 1-hertz processor”—yet setters broadly distinguish quality from difficulty: beautiful and boring puzzles can each be easy or hard because those are orthogonal axes.

7. Irreducibility is easier to suspect than to prove

In a chaotic dynamical system without a closed-form solution, predicting the future may require numerical integration through every intervening state: “you have to do the work.” Some cellular automata, such as Rule 150, are “kind of linear,” roughly modulo 2, and permit shortcuts; others appear to demand step-by-step simulation.
For Rule 110, Wolfram observed particle-like collisions; Matthew Cook then came along and, with Wolfram, completed the proof of universality. Moore also credits Damian Woods with later efficiency improvements. This mirrors NP-completeness proofs: demonstrate hardness by encoding other problems, effectively “building a computer out of the thing.”
Many systems still look nonlinear and irreducible without yielding controlled components. Moore invokes a friend’s image of “lava”—chaotic activity like plasma on the Sun, briefly forming and dissolving, with no clean wire, gate, storage bit, or particle that researchers know how to use.
Complexity therefore differs from programmability: fluids might compute with pipes and valves, but unchannelled water is harder to control. Undecidable problems need not be Turing-complete or hard because the halting problem reduces to them; if P differs from NP, as nearly everyone Moore knows believes, intermediate problems should be inefficient without being NP-complete—yet completeness remains the standard route to proving hardness.

8. Extensibility matters more than labeling a model Turing-complete

The host presents a co-host’s thesis that transformers are finite-state systems and that strong agency requires algorithms able to control a Turing-strength process, expand memory, and recurse. Moore’s immediate reply is disarming: “I am also a finite-state machine,” composed of finitely many neurons and particles.
Human capability comes from tool use. Moore can add clay tablets, whiteboards, an abacus, or a computer to extend his workspace—effectively lengthening the tape—while his difficulty repeating words backward suggests he does not possess a naturally usable internal stack. “If I need it, I can build it.”
The co-host’s refined claim is that only a subset of finite-state algorithms can control such expandable memory and that stochastic gradient descent cannot traverse that subset. Moore calls this “an interesting claim” rather than endorsing it, noting that apes could navigate, make stone tools, and teach through gesture without the full modular structure of human language.
LLMs are language-first rather than tactile-, visual-, or survival-first. Because language carries recursion, observers expect an immediate leap into formal mathematics, but ordinary language is token production; formal reasoning is a “thin veneer” humans activate at an edge, often with paper, symbols, or specialized software.

9. Universality’s deepest move is making systems act on themselves

Moore does not center the physical Turing-machine architecture in his teaching. Turing machines, Boolean-circuit families, counter machines, finite-state automata with two stacks, cellular automata, Church’s lambda calculus, and partial recursive functions are striking because their different architectures converge on the same computable class.
Modern students can find the halting proof anticlimactic: feed a supposed halting decider its own program and reverse its answer—“That’s it?” The historical breakthrough was self-reference itself. Gödel, Church, and Turing crossed a hierarchy in which numbers, functions, and functionals were kept separate; today compilers and editors make programs operating on programs “the air we breathe.”
A related abstraction turns an infinite state graph into a finite higher-level description: a stack or Turing machine has infinitely many possible configurations but a compact rule. Moore wants artificial partners capable of recognizing that next level—“capturing an infinite set of objects all at once”—a step that seems to him like more than merely recognizing a statistical regularity and predicting from it.

10. Computation is a revealing lens, not a complete ontology

Moore winces at “we need more compute”—“that does not sound right in my ears”—but readily uses a computational lens: study how a system stores, transmits, and transforms information. Ribosomes and DNA error correction clearly reward that view; planetary orbits could also be called computation, but the label may teach little.
Origin-of-life theories expose the lens’s limits. A replicator-first story emphasizes symbol strings making copies; a metabolism-first story emphasizes thermodynamic gradients, chemistry, and channels for free energy—the “selfish metabolism” may have built genomes rather than genomes building metabolism. Moore prefers switching lenses according to the phenomenon, and similarly finds evolutionary explanations unhelpful for questions such as ethics.
Moore’s physical-ontology conclusions remain hedged. He is inclined to accept a physical Church–Turing thesis under which any buildable device can be simulated with finite resources, perhaps by a quantum computer, and suspects something “discrete-ish” occurs at the smallest scales. But inaccessible analog precision, black holes, closed timelike curves, and hypothetical hypercomputation leave the claim contingent on how the universe actually works.

11. Rights demand transparency beyond predictive accuracy

Moore treats opacity as an application-dependent continuum. A black-box movie recommender is acceptable to him if he likes the film; an independently tested proprietary cancer diagnostic might also be acceptable because he wants to live. Jail, guilt, surveillance, and constitutional rights belong at the opposite extreme.
The criminal justice system, in his example, explicitly cares about more than accuracy. If police plant evidence against someone who actually committed the crime, society may let that person walk because government must obey rules governing investigation and proof. The accused must be able to confront an accuser, cross-examine witnesses, and contest evidence.
Probabilistic-genotyping software illustrates the institutional gap: degraded DNA may combine several visitors, available products can disagree, and some had never been—or, at least until recently, were not—independently tested. It is unclear whether the software can be treated as an expert witness, whether its coders or bioinformaticians should be examined, or how a judge or jury should compare conflicting outputs.
Moore prefers “transparency” to interpretability because it includes independent validation, vendor-claim verification, and, where necessary, inspection under the hood. Some opacity reflects genuine scientific difficulty; some reflects intellectual-property choices, and he is “not very sympathetic” to the latter. For decisions affecting fundamental rights, society should demand “every possible form of transparency.”

Dave Hone: T-Rex, Dinosaurs, Extinction, Evolution, and Jurassic Park | Lex Fridman Podcast #480 Is Non-Consensus Investing Overrated?