Summary

Agüera y Arcas’s central claim is literal: “DNA is a computer program,” so life is a subset of intelligence because heritable self-reproduction requires a universal computer. Von Neumann anticipated the stack before molecular biology supplied its components: DNA as Turing tape, the ribosome as constructor, and DNA polymerase as copier. Brains and culture are faster computational layers, but cells established the ground floor.
His BFF experiment suggests purpose can emerge spontaneously from random code through a sharp phase transition. A soup of 1,000 random 64-byte tapes, averaging only about two instructions each, remains mostly inert for millions of pairwise interactions; then entropy collapses, repeated complex programs appear, and reverse engineering shows that they reproduce. Something becomes functional because it can now be broken: disrupt the code and it loses its “purpose” of copying itself.
The surprise that changed his evolutionary thesis was that emergence continued when the mutation rate fell from roughly one in 10,000 per interaction to zero. That shifts explanatory weight from mutation toward merge: independently reproducible elements can combine into more complex wholes, with the history of their assembly becoming information. Symbiogenesis therefore helps explain the directional increase in complexity that ordinary mutation-and-selection does not fully explain.
This merger logic extends from a mitochondrion joining an archaeon to technology and the repurposing of biological machinery. Agüera y Arcas cites the viral membrane-fusion ability later used in placenta formation. In BFF, even whether two primitive copiers joined as AB or BA becomes durable information: “The merger tree ends up being exactly the information that is encoded in the final genome.”
At Google, his roughly 50-person Paradigms of Intelligence group is explicitly trying to “refill the bucket” with fundamentals beyond exploiting today’s successful models. The intellectual catalyst was his post-2020 conclusion that large sequence models appeared generally intelligent. The discussion highlights composition, cooperation, multi-agent organization, and persistent memory as important research directions.
His functionalism rejects substrate as the essence of life or consciousness: a kidney is whatever performs the kidney’s role, even if implemented differently. He strongly disputes Anil Seth and John Searle’s substrate-oriented intuitions, arguing that replacing neurons with functionally equivalent components would not silently “dial down” consciousness. Biology’s interfaces are wet and difficult, but “multiple realizability and repurposability” are the very stuff of life.
He sees AI as an extension of collective human intelligence, not a cleanly separate alien intelligence, while still worrying about polarization, disinformation, and political-economic systems that may be unfit in 20 years. “AI was human intelligence from the start” because general models emerged by absorbing human language; Scarfe’s harder objection—that one artifact might exceed all humanity—remains largely parked. For current transformer systems, Agüera y Arcas identifies “narrative memory,” especially persistent long-term selfhood, as the biggest gap rather than composition itself.

Deep dive

1. Large sequence models forced a rethink of intelligence

Agüera y Arcas frames What Is Intelligence? as the record of an intellectual shock beginning around 2020: large sequence models seemed “generally intelligent,” forcing him to ask what believing that observation would imply about evolution, minds, and humanity itself. Its opening chapter, also available as What Is Life?, argues that life belongs inside the broader category of intelligence.
His current institutional setting is Google’s Paradigms of Intelligence group, roughly 50 people working on fundamentals. The team accepts that current paradigms work, but wants to “refill the bucket with new insights and new ideas” instead of concentrating only on exploiting existing models.
Scarfe’s challenge that DNA adapts slowly while nervous systems and culture evolve “at light speed” wins a clear concession, but not a reversal. DNA is only the ground floor: life builds “computers out of computers out of computers,” adding cellular, neural, cultural, and societal layers once general computation exists.

2. Self-reproduction requires a computer inside the organism

Von Neumann’s thought experiment begins with a Lego robot collecting loose pieces from a pond to build an offspring. It needs an instruction tape, a universal constructor that reads the tape, and a copier that gives the offspring its own instructions—including instructions for constructing the constructor and copier.
Biology later supplied the striking correspondences: DNA is the tape, ribosomes perform construction, and DNA polymerase copies the tape. Heritable alteration matters because changing the genome changes subsequent offspring; Agüera y Arcas’s categorical conclusion is that “you cannot be a living organism without literally being a computer, a universal computer.”
Cellular automata make computation embodied. Unlike a conventional Turing machine, where tape, head, rules, and symbols are conceptually separate, every location follows the universe’s local “laws of physics,” allowing the machine to print its own physical machinery—“a laptop and a 3D printer in one that can print another laptop.”
Scarfe’s biochemical pushback survives the abstraction: Conway’s Game of Life is two-dimensional and deterministic, whereas physical life depends on three dimensions and thermal randomness. Agüera y Arcas therefore invokes stochastic Turing machines while stressing massive parallelism and nesting: quintillions of ribosomes operate inside cells, cells inside organisms, and people inside societies.

3. Random code crosses a phase boundary into purpose

BFF starts with 1,000 random tapes, each 64 bytes long, in a seven-instruction, self-modifying language derived from Brainfuck. Roughly 31 of every 32 random bytes are no-ops, leaving only about two instructions per tape. Each iteration randomly joins two tapes into a 128-byte program, runs it, separates them, and returns them to the soup.
For millions of interactions almost nothing happens; then “something apparently magical happens.” Entropy drops sharply, the formerly incompressible soup becomes highly compressible, and complex programs appear in many copies. Their repetition reveals their function: they have become replicators, so the emergence of life is simultaneously “the emergence of purpose.”
Purpose is operational rather than mystical: change a crucial byte and the program stops reproducing, meaning it has a function that can be broken. Scarfe raises architectural bias, and Agüera y Arcas agrees that language shapes the resulting programs—but similar emergence in Z80 assembly convinces him the phenomenon itself is generic.

4. Merge, not mutation alone, supplies evolution’s complexity ratchet

The apparent reversal from randomness to order remains thermodynamic. Drawing on Adi Pross’s “dynamic kinetic stability,” Agüera y Arcas argues that stability can be cyclic: fragile DNA that repeatedly makes DNA can outlast granite that merely erodes. In that sense, “evolution is the second law at work.”
He initially inserted random mutations into BFF with a probability of roughly one in 10,000 per interaction, assuming Darwinian “chance and necessity” drove the result. Replicators still emerged when mutation was turned down to zero, changing his mind: purpose, abiogenesis, and increasing complexity are not fully explainable in purely Darwinian terms.
Symbiogenesis supplies a mechanism for later complexity in his account. He describes a eukaryote forming when a mitochondrion finds itself inside an archaeon; the resulting composite is more complex than either component, like a spear being more complex than a stick and a stone point. Combining preexisting parts allows more sophisticated systems to arise later in evolution.
Scarfe extends the same logic to information: merging preserves lineage, reuse, path dependence, and the history of how components were assembled. Agüera y Arcas says he “completely” buys that framing and shows how, in BFF, the merger tree itself becomes information in the final genome.

5. Composition turns contingency into reusable structure

Scarfe connects merge to lineage, path dependence, reuse, and canalization. Agüera y Arcas says he “completely” accepts that framing—while strongly criticizing Chomsky’s account of language through Dan Everett’s work with the Pirahã. He notes that the Pirahã language does not fit Chomsky’s requirements, including recursion or center embedding, and lacks numbers and past and future tenses.
His deeper objection to Chomsky is top-down formalism: grammar-and-program approaches fed “good old-fashioned AI,” which he calls a false start behind repeated AI winters. Scarfe counters that Chomsky’s automata and computational ideas resemble this thesis lower in the stack; Agüera y Arcas replies that von Neumann and early artificial-life researcher Nils Aall Barricelli already had the essential machinery.
W. Brian Arthur’s light-bulb example shows why inventions cluster. Once glassblowing, vacuum technology, filaments, and electric current exist, multiple inventors can independently compose the same broad product; contingent decisions about filaments, socket diameter, or screw direction then constrain everything built afterward.
BFF exposes that history at its smallest scale: weak single-byte copiers meet, sometimes persist as a pair, and do better together. Whether they join as AB or BA is new information, so “the merger tree” becomes the eventual genome. Higher-level perception is smoother: humans recognize odd bicycles despite broken rules, favoring neural networks and gradient descent over hand-coded circle detectors.

6. Function survives a change of substrate

Agüera y Arcas calls himself closest to a functionalist. Scarfe supplies the kidney example: physics may fully describe the atoms, but a kidney is defined by its role in filtering urea, so a radically different artificial device performing that role can still be called an artificial kidney. The discussion treats function as relational, meaningful within an ecology of other functions.
Multiple realizability is presented as the hallmark of function: different pathways can make ATP, while insect and bat wings implement flight differently. Scarfe’s stronger objection is historical: replacing a natural kidney or plant may work now yet disrupt the ecology’s future trajectory because the replacement arrives with another provenance.
Agüera y Arcas turns that objection into support for symbiogenesis: life constantly imports, repurposes, and parallel-paths machinery built for another function—“intelligent design can happen without any intelligent designer.” His example is the ability to fuse cell membranes, originally associated with a virus, later incorporated into placenta formation.
He therefore strongly rejects Seth and Searle’s substrate essentialism: substituting functionally equivalent neurons would not erase consciousness, though wet biological interfaces make such equivalence far harder than swapping software routines. “Multiple realizability and repurposability” are, for him, the very stuff of life.

7. Consciousness is a cooperation technology

Philosophical zombies—entities behaving exactly like people while “dead on the inside”—are less coherent than they appear under Agüera y Arcas’s functionalism. Consciousness is neither an inert epiphenomenon nor a privilege of particular matter; it is functional within the relationships that generate behavior.
His Paradigms of Intelligence team approaches that function through multi-agent reinforcement learning. Cooperation precedes symbiogenesis, and cooperating agents need theory of mind: each must infer over a world containing the game, its own internal state, the other agent’s internal state, and correspondences such as “my smile means happiness, so your smile probably does too.”
Modeling oneself, another, the other’s model of oneself, and further nested models creates a Hofstadter-like “strange loop.” Real agents are computationally bounded, so their psychological representations remain cartoonish; Agüera y Arcas says this recursive modeling reaches only about sixth order at most.
Rowing supplies the experiential analogy. In “swing,” eight rowers synchronize so completely that “the boat acquires a soul,” moving faster as separate purposes become one. Scarfe connects this to hiring for both agency and alignment: sometimes the best intentional boundary encloses the coordinated group, not any individual member.

8. The self is a negotiated boundary with an “inner lawyer”

Conjoined twins Abby and Brittany Hensel illustrate fluid agency: separate brains and spinal cords control one arm and leg each, yet lifelong behavioral cross-cueing lets them drive, play sports, write, and often speak in synchrony. They can act as one coordinated system while retaining differences of opinion.
Split-brain patients invert the case. Outsiders can experimentally observe two hemispheres receiving different information and controlling different hands, yet patients insist they remain one person. Agüera y Arcas refuses a single privileged answer: selfhood is relational, and even one hand buttoning while the other unbuttons may register merely as inconvenience.
Petter Johansson’s choice-blindness experiments reveal how continuity is manufactured. Subjects asked to explain an attractiveness choice often receive the face they rejected, rarely notice, and justify it with unchanged fluency or latency. An “inner lawyer” invents a narrative that can alter future choices, while the brain’s distributed parts cover for one another because they are “all in the same boat.”

9. AI extends collective humanity, but memory remains the gap

Agüera y Arcas worries about polarization, disinformation, and whether political-economic institutions will remain fit for purpose in 20 years, but not about the scenarios discussed by Eliezer Yudkowsky. His reason is ontological: individual humans are not dramatically beyond other primates; civilization’s millions and billions collectively produce organ transplants and spaceflight.
AI therefore looks like more of an existing collective process: “AI was human intelligence from the start” because general AI was achieved by training models on vast amounts of human language. Scarfe’s pushback—that one artifact might surpass the totality of humanity—is not squarely resolved; Agüera y Arcas instead emphasizes how badly individuals mistake socially distributed knowledge for their own, as bicycle-drawing failures demonstrate.
Today’s models plainly differ from brains in architecture, training, interfaces, and modalities, yet their internal representations can converge surprisingly with human Brain-Score-type measurements; even language-only systems can reproduce aspects of sensory structure. Models’ ability to compose unusual scenes undermines claims of no composition. His clearest deficit is “narrative memory”: persistent long-term experience capable of sustaining a self.
Scarfe’s “superficial impostor” concern focuses on brittle answers under small logical variations. Agüera y Arcas’s reply is methodological: run the human baseline, because people often exhibit the same framing effects and logical illusions unless they slow down and formalize. Transformers do not systematically search all Turing-machine programs; computationally tractable induction requires shortcuts, whether the system is a brain or a transformer.

E210 | Japan’s “Lost 30 Years” Made China’s Most In-Demand Restaurant Chains?2025 AI in the Field: Our Year of Witnessing and Fantasizing