Summary

Today’s models may qualify as AGI for automating jobs, but Bennett says they remain far from human-like intelligence. He cannot offload enough work to Grok or ChatGPT, while biological systems with “a tiny fraction of the energy and learning data” could do much more. The thesis-relevant bottleneck is therefore sample efficiency and adaptive architecture, not merely economic usefulness.
Bennett defines intelligence as efficient adaptation under constraint, favoring Pei Wang’s “adaptation with limited resources.” François Chollet’s skill-acquisition framing is useful for benchmarks but remains relative to a chosen task scope. Once embodiment, available actions, and environment change, apparent difficulty changes too—so scores measure intelligence only relative to a chosen task and setup.
Software cannot be evaluated independently of the machinery and world interpreting it. A program passes through languages, compilers, hardware, and ultimately physics; changing that stack can drastically change its behavior, even making it arbitrarily stupid. Bennett’s conclusion is that human-like AGI will probably require modular, cellular hardware that delegates adaptation downward, because computers today resemble “an inflexible bureaucracy that makes decisions only at the top.”
Benchmark velocity is real but does not prove general intelligence. The host cites Grok 4 at roughly 16% on ARC-AGI, versus leading efforts around 15.5%, and Humanity’s Last Exam jumping from about 26% to 46%; Bennett calls benchmarks useful “measuring sticks” and “great marketing,” especially when code and full results are unavailable. His deliberately mundane test remains whether a model can reliably add long numbers.
The practical architecture is likely hybrid: approximation for noisy scale, search for exactness. LLM-style approximation handles large, noisy datasets; search supplies precision, with AlphaGo and AlphaStar as exemplars. NARS and Hyperon extend the idea into modular systems that can incorporate LLMs, symbolic reasoning, resource limits, and new components—the “kitchen sink” is a feature if integration works.
Bennett argues consciousness may be a necessary adaptation rather than an optional phenomenal add-on. A causal self emerges by separating self-caused changes from external ones, while experience is a “tapestry of valence” across scales. Whether that tapestry must exist simultaneously leaves an unresolved substrate question: nanobot swarms might qualify while a single-thread simulation might not—unless consciousness can be “smeared over time.”
AI safety, in Bennett’s framing, is a systems-design and delegation problem rather than isolated policy alignment. AI plugs into a larger human “liquid brain”; isolation from the collective’s informational structure can cause components to break away, while excessive top-down constraint can eliminate every workable policy. The investable implication is architectural: resilient intelligence may depend on controlled autonomy, self-repair, and adaptation throughout the stack, not ever-longer rulebooks around a fixed model.

Deep dive

1. Intelligence is adaptation under constraint, not task accumulation

Bennett began with the Legg–Hutter definition—satisfying goals across a wide range of environments—but biological intelligence pushed him toward “the efficiency of adaptation,” especially sample and energy efficiency. He ultimately preferred Pei Wang’s earlier, cleaner formulation: “adaptation with limited resources.”
The host conceded that Wang’s definition can sound almost tautological, yet Bennett defended its simplicity: elaborate definitions often consume more effort to interpret than the phenomenon itself. The host also noted that Wang’s formulation informs NARS, the Non-Axiomatic Reasoning System.
François Chollet’s definition emphasizes the ability to acquire skills, which Bennett reads as a benchmark-oriented specialization rather than a universal account. Chollet’s formalism inherits Kolmogorov-complexity ideas from Legg and Hutter, even while Chollet explicitly says compression alone is insufficient; the host suggested the intelligence may be the meta-process producing compressed skill programs, not the programs themselves.
AIXI supplies the contrasting ideal: a general reinforcement-learning agent using Solomonoff induction, effectively formalized Occam’s razor, as an upper bound on intelligence. Bennett finds that model compelling but rejects its presumed objectivity—complexity depends on the interpreter, just as a concept’s shortest description depends on language, so performance can be distorted by “shifting the goalposts of interpretation.”

2. Intelligence lives in the entire causal stack

Bennett’s objection to fixed task definitions starts with their abstractions: goals, actions, and environments are human-chosen simplifications. Two systems using different abstractions can reach the same end with radically different apparent difficulty, so tasks cannot be separated from embodiment and the environment in which cognition is enacted.
The stack runs from Python through C, machine code, hardware, and physical law; Bennett sees no principled stopping point, particularly when physical knowledge remains incomplete. To make objective behavioral claims, a theory must specify what remains true across all possible abstraction layers rather than privilege one convenient hardware interface.
Causation is multiscale and bidirectional. Cells pursue local goals while being constrained by their collective, just as people act independently yet respond to legal and social systems; layers can be read upward from cells to organs to organisms, or from soldiers to squads and platoons.
Bennett’s revised chart distinguishes abstraction from delegated control through food stamps versus universal basic income: both distribute resources, but food stamps tightly restrict their use while UBI delegates the spending decision. The same distinction matters in biological, computational, and social systems—decentralizing resources is not equivalent to delegating agency.

3. Useful abstractions emerge from causal relevance and valence

The host asked whether abstractions are real or must be handcrafted into AI. Bennett’s answer was adaptive learning: “chair” is a useful classifier because it can cause positive or negative valence, whereas “half a chair” does not usually deserve an independent concept. Minds build classifiers for television, light, chairs, and tables because those distinctions matter to action.
That selectivity may also explain apparent “mind blindness”: behavior could count as intelligent yet remain unnoticed because it causes us no relevant valence. Bennett linked the possibility to Mike Levin’s work and even the Fermi-paradox question of whether observers overlook intelligences whose actions do not intersect their concerns.
Judea Pearl’s intervention framework led Bennett toward a causal identity for the self. A fly on a shoulder must distinguish the world moving because the shoulder moved from movement caused by the fly itself, or it gets squashed; across species, separating self-caused from external change supports both effective action and subjective experience.

4. A software-only mind is modern computational dualism

Bennett’s alternative to simplicity-first learning—variously “weak policy optimization,” weak constraints, W-maxing, and jokingly “Bennett’s razor”—should construct representations of self and world through causal relevance. Cells, cellular collectives, and organisms are continually attracted and repelled, creating a multiscale “tapestry of valence,” not neutral symbols with value attached afterward.
Enactive cognition rejects a mind sealed inside its body: memory extends into paper, objects, other people, and the surrounding world. Bennett’s jab is that AI recreated Cartesian dualism: “We have just replaced the pineal gland with a Turing machine,” treating software as mental substance connected mysteriously to physical performance.
The host pressed whether this condemns all functionalism. Bennett narrowed the claim: “computational dualism” is a “rolled-up newspaper” aimed at definitions of intelligence that specify software but ignore its interpreter, because “whatever that software does has to pass through an interpreter and the interpreter decides what it does.”
He likewise rejected “immortal computation” as a useful contrast: every software artifact has finitely many physical copies, however reproducible its code appears. Bennett said the probably best rigorous treatment he had seen was by Alex Ororbia and Karl Friston, but concluded, “There’s just mortal computation.”

5. Benchmark wins do not establish human-like AGI

Bennett granted the scale-maxing case its economically important premise: if AGI means systems capable of doing jobs and automating much of the economy, “sure, we’ve got some form” of it. If the target is human-like intelligence, however, current systems remain too sample-inefficient and still leave him doing work that a genuinely capable assistant should absorb.
He was cautious about ARC-AGI claims because headline results without code and detailed results are difficult to audit. Even a strong benchmark cannot settle intelligence: ARC-AGI is valuable, but Bennett stressed that Chollet himself would not call it perfect.
Responding to the host’s report that Grok 4 scored about 16% on ARC-AGI, slightly above efforts around 15.5%, Bennett offered an empirical non-answer: “The proof’s in the pudding.” If it begins performing many useful jobs, that will establish progress; meanwhile he wants to see whether the new model can reliably add long numbers.
Humanity’s Last Exam moving from roughly 26% to 46% prompted the sharper judgment: benchmarks are “great marketing.” They reveal progress along the direction in which a measuring stick points, but adaptive humans will inevitably optimize against imperfections and find ways to exploit the test.

6. Hybrid systems join noisy approximation to exact search

Drawing on Richard Sutton’s “Bitter Lesson,” Bennett reduced today’s toolkit to approximation and search. LLM-style approximation is powerful precisely because it tolerates noise and processes enormous datasets; search iterates through explicit possibilities and provides the precision needed for tasks such as navigation.
Systems such as AlphaGo and AlphaStar combine them by using learned approximation as a heuristic to guide search. The host’s devil’s advocate was therefore material: an LLM’s inability to calculate does not end the architecture discussion if it can invoke exact tools.
NARS and Hyperon aim at broader human-like versatility. Bennett cited NARS–LLM experiments presented at the 2023 AGI conference, while Hyperon is designed as decentralized, adaptable infrastructure into which heterogeneous modules can be plugged; conceding the host’s “kitchen sink” description, he argued that modularity allows the system to improve as components mature.

7. Consciousness may be an adaptation that intelligence cannot omit

Bennett’s causal-self work converged with reafference research in the insect central complex and mammalian midbrain. Reafference distinguishes pressing down on a chair from the chair pressing upward; this inherently valenced sensorimotor self can be extended into higher-order “predictions of predictions,” supplying ingredients for subjective experience.
Against David Chalmers’s philosophical zombie—an entity behaviorally and functionally identical to a person but lacking consciousness—Bennett built a formalism beginning only with change or difference across “all conceivable worlds.” His conclusion is categorical within that framework: no equally intelligent, equally efficient human analogue could lack consciousness, because consciousness is a necessary adaptation and nonconscious information processing of that kind is implausible.
The host tested whether this merely calls consciousness an illusion or discards phenomenal subjectivity. Bennett explicitly rejected both readings: he claims many features of consciousness follow necessarily from changing states and causal valence without first dividing reality into physical and nonphysical substances. A phenomenal state is causally embedded as attraction and repulsion across abstraction levels, not an epiphenomenal ornament.
One uncertainty survived the thesis. If a conscious tapestry must be realized simultaneously, distributed organisms or nanobot swarms might be conscious while a single-thread CPU—and perhaps human populations or ant-colony “liquid brains”—cannot be; if realization can be smeared across time, simulations and liquid brains might qualify. “How do we know if we’re being simulated? We don’t.”

8. Delegated adaptation links AI safety, cancer, and life

Bennett reframed safety around the whole system: AI is not an isolated agent but another component plugged into a human “liquid brain.” The design problem is therefore how the combined swarm accommodates and coordinates heterogeneous parts, not merely how one model’s policy is aligned in isolation.
His “law of the stack,” which prompted a supervisor to accuse him of “libertarian biology,” says adaptation at a high abstraction level depends on adaptability below it. Biology delegates adaptation down the stack; conventional computers behave like “an inflexible bureaucracy that makes decisions only at the top.”
Mike Levin’s cancer framing supplied the failure mode: a cell isolated from the collective’s informational structure reverts to primitive reproduction and consumption. Bennett formalized two routes to such fracture—external conditions can make every collective policy impossible, or excessive top-down control can eliminate otherwise workable local policies. For AI, his prescription was to constrain dangerous domains specifically; impossible blanket conditions make systems break.
Bennett said being alive almost requires a combination of “simping” and “waxing.” His provisional account contrasts rocks, which persist by simp-maxing, with self-repairing organisms that become more complex while expanding the weak constraints they can embody: “life is that which waxes at the expense of simp.” The account had been submitted but not yet peer-reviewed, and he wanted Friston’s criticism.
Bennett also framed culture as a store of human knowledge and cited David Krakauer’s description of culture as “evolution at light speed.” He considered these ideas compatible with his broader account, while acknowledging that the thesis work remained provisional.

How to get a job in investing Reshaping America’s Economy for the Superintelligence Century with Jacob Helberg