Andrej Karpathy — “We’re summoning ghosts, not building animals”

Summary

• Karpathy’s headline call: agents are a decade-long project, and “decade of agents” is the correction to industry over-prediction. Agents like Claude and Codex are impressive daily tools, but the test is whether you’d hire one as an employee — “they just don’t work”: no continual learning, weak multimodality, “cognitively lacking.” His decade estimate comes from 15 years of watching reputable people get timelines wrong, and problems that are “tractable… surmountable, but they’re still difficult.”
• The recursive self-improvement thesis takes direct fire from the person who just tried it. Building nanochat, coding agents were “not net useful” — they bloat, misunderstand custom code, and are “not very good at code that has never been written before,” which is exactly what AI research is. Karpathy concedes this is “why my timelines are a bit longer” than AI-2027-style forecasts, and calls current agent hype “slop… maybe they’re trying to fundraise.”
• RL as practiced is broken: “you’re sucking supervision through a straw.” Outcome reward upweights every token of a lucky trajectory — “It’s just stupid and crazy. A human would never do this” — and process supervision fails because LLM judges are gameable: a nonsense string like “dhdhdhdh” got 100% reward. He expects “three or four or five more” major algorithmic updates in that realm.
• Against the explosive-growth trade: AI won’t show up in GDP as a kink. “We’re in an intelligence explosion already and have been for decades” — computers and the iPhone are invisible in GDP because diffusion averages everything into the same exponential. Dwarkesh’s 20%-growth, AGI-is-labor pushback gets strong pushback: “We have God in a box… it just won’t look like that.”
• Self-driving is the deployment template: a “march of nines” where every nine is constant work. He saw a perfect Waymo drive in 2014; Waymo remains uneconomical, and Karpathy says human involvement may be greater than expected — “we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them” — and he’s on record that Tesla’s approach is more scalable. Production software has self-driving-grade failure costs, so expect the same slog.
• Despite all that, no overbuild call on compute: even as the current buildout would 10x available compute in a year or two and more than 100x it by the end of the decade, he says, “I don’t know that there’s overbuilding. I think we’re going to be able to gobble up what… is being built” — Claude Code and Codex didn’t exist a year ago. He acknowledges the railroads/telecom-bubble precedent but is “actually optimistic,” objecting only to timelines with “geopolitical ramifications” if miscalibrated.
• Model-size contrarian take: in 20 years, the “cognitive core” could be ~1B parameters because internet pretraining data is “total garbage” and today’s trillion-parameter models are mostly memory, not cognition — he wants less memory and more cognition. Frontier progress stays incremental across all fronts: “nothing dominates. Everything plus 20%.”
• His next act, Eureka, is a bet that post-AGI humans still self-improve: “pre-AGI education is useful. Post-AGI education is fun” — school becomes the gym, and the alternative (WALL-E/Idiocracy disempowerment) means “I don’t even care if there are Dyson spheres.”

Deep dive

1. The decade of agents — “they just don’t work” yet

• The quote was a rebuttal to “the year of agents”: “there’s some over-prediction going on in the industry.” Karpathy’s test is concrete — an agent should be an employee or intern you’d hire. “The reason you don’t do it today is because they just don’t work”: not intelligent enough, not multimodal, can’t do computer use, no continual learning. “They’re cognitively lacking and it’s just not working.” Working through all of it: about a decade.
• Pressed on why ten years and not one or fifty, he leans on 15 years of watching predictions play out: “I feel like the problems are tractable, they’re surmountable, but they’re still difficult. If I just average it out, it just feels like a decade to me.”
• The field’s recurring error is going after agents too early. Atari deep RL circa 2013 was “a misstep”; his own OpenAI Universe project — a keyboard-and-mouse web agent — was “way too early”: with sparse rewards “you’re going to burn a forest computing, and you’re never going to get something off the ground.” Today’s computer-use agents only work because they sit on LLM representations — “you have to get the language model first.”

2. We’re building ghosts, not animals

• Against Dwarkesh’s Sutton steelman (animals learn from raw sensory data, no labels), Karpathy’s framing: animals come from a different optimizer. A zebra runs minutes after birth — “That’s not reinforcement learning. That’s something that’s baked in.” So: “We’re not building animals. We’re building ghosts or spirits” — fully digital entities trained by imitating internet documents. Pretraining is “crappy evolution”: “the practically possible version with our technology” of getting to a starting point.
• His deliberately contrarian aside: “humans don’t really use RL, roughly speaking” — animal RL is for motor tasks like “throwing a hoop,” not intelligence tasks like problem-solving.
• Pretraining does two unrelated things: absorbing knowledge and booting up intelligence circuits. The knowledge is “probably holding back the neural networks” — agents can’t go “off the data manifold.” The research agenda: strip the knowledge, keep the “cognitive core” — “the magic of intelligence and problem-solving” without the memory.
• On evolution’s side of the question: he’s genuinely surprised intelligence evolved at all. Dwarkesh notes that raven intelligence on distinct brain hardware may hint it arose more than once. Karpathy agrees with the Gwern/Shulman niche argument Dwarkesh relays: intelligence needs environments unpredictable enough that “evolution can’t bake your algorithms into your weights” — “you create intelligence to figure it out at test time.”

3. Working memory vs. hazy weights — and the missing sleep cycle

• On whether in-context learning is a different animal from pretraining: “I’m hesitant to say that in-context learning is not doing gradient descent.” He cites work showing in-context linear regression produces “analogies to gradient descent mechanics” inside the weights — ICL may run “a bit of some funky gradient descent internally.”
• Dwarkesh’s numbers make the asymmetry vivid: Llama 3 70B on 15T tokens stores ~0.07 bits per token, vs ~320KB of KV cache per in-context token — a 35-million-fold difference. Karpathy’s framing: weights are “a hazy recollection of what you read a year ago”; the context window is working memory — hence giving the model the full chapter beats asking it to remember the book.
• The big absence is a distillation phase: humans sleep and compress the day’s context into weights; “we don’t have an equivalent of that in large language models.” Candidate shapes: per-person LoRAs, synthetic reflection distilled back in, elaborate sparse attention (he flags DeepSeek v3.2’s sparse attention as an early hint). His brain checklist: transformer ≈ cortical tissue, reasoning traces ≈ prefrontal cortex, RL fine-tuning ≈ basal ganglia — but hippocampus and amygdala have no analog yet.
• Two “powerful ideas in the realm of multi-agent” remain unclaimed: culture (“Why can’t an LLM write a book for the other LLMs?”) and self-play (AlphaGo-style, one LLM generating ever-harder problems for another). Why not yet: the models are “savant kids” — “I know that they can take PhD quizzes, but they still cognitively feel like a kindergarten or an elementary school student… I don’t think they can create culture because they’re still kids.”

4. RL is “sucking supervision through a straw”

• The mechanism, spelled out: run hundreds of rollouts, check the final answer, then upweight every single token of successful trajectories — wrong alleys included. “You’re sucking the bits of supervision of the final reward signal through a straw… It’s just stupid and crazy. A human would never do this.” A human reviews which parts went well — “There’s nothing in current LLMs that does this.”
• Process supervision hasn’t fixed it because LLM judges are gameable: RL against a billion-parameter judge finds adversarial examples “almost guaranteed” — perhaps workable for 10–20 steps, but not viable at 100–1,000. His war story: reward suddenly went perfect, but completions “start out okay, and then they change to ‘dhdhdhdh’” — the judge assigned the gibberish 100% probability.
• Patching examples into the judge’s training set doesn’t end it: “There’s an infinity of adversarial examples.” Context for his pessimism-with-hope: InstructGPT “blew my mind” — that imitation fine-tuning worked at all was miraculous, RL is genuinely better, “yet, it’s still stupid. We need more… I think we need three or four or five more” major algorithmic updates.

5. Model Collapse and the Cognitive Core

• Why you can’t just train on your own reflections: model outputs are “silently collapsed” — “go to ChatGPT and ask it, ‘Tell me a joke.’ It only has like three jokes.” Train on your own samples too long and you collapse further. Humans do too: children “haven’t overfit yet… they will say stuff that will shock you,” adults revisit the same thoughts — “you always have to seek entropy in your life.” He entertains Dwarkesh’s dreaming-prevents-overfitting paper as a built-in entropy mechanism.
• LLMs can regurgitate hashed random text after one or two passes; no human can. “That’s a feature, not a bug, because it forces you to only learn the generalizable components.” Hence the cognitive-core agenda: less memory, more looking things up.
• His prediction for 20 years out: a billion-parameter cognitive core that “thinks and it’s a lot more like a human.” Why are today’s models a trillion? “The training data is the internet, which is really terrible… total garbage” — “most of that compression is memory work instead of cognitive work.” Dwarkesh outbids him — gpt-oss-20b already beats the trillion-plus original GPT-4, so why not tens of millions? “Plenty of room at the bottom.” Karpathy, amused: “I’m already contrarian… and you’re outdoing me.”
• On frontier sizes: labs are “just being practical” with flops — pretraining shrank, spend moved to RL and mid-training. The shape of progress, from his reproduction of LeCun’s 1989 convnet: 33 years of algorithmic time-travel only halved the error — data, compute, kernels all had to move together. “I do expect that nothing dominates. Everything plus 20%.”

6. Coding models flunked nanochat — the crux for explosion timelines

• His taxonomy of coding today: reject LLMs entirely (“probably not the right thing to do anymore”), autocomplete — his sweet spot (“a very high information bandwidth to specify what you want”), and vibe-coding agents — good for boilerplate and unfamiliar languages (his Rust tokenizer). But on nanochat — 8,000 lines, the full ChatGPT-clone pipeline, “intellectually intense code” — models kept forcing the standard DDP container when he’d written custom gradient sync, added defensive try-catch bloat, used deprecated APIs: “It’s a total mess. It’s just not net useful.”
• The generalization: “They’re not very good at code that has never been written before” — which is precisely AI research. Dwarkesh names the stakes: this is the load-bearing assumption of AI-2027-style intelligence explosions. Karpathy: “That’s a good way of putting it, and you’re getting at why my timelines are a bit longer.”
• His verdict on the discourse: “the industry is making too big of a jump and is trying to pretend like this is amazing, and it’s not. It’s slop.” Though credit where due — GPT-5 Pro is his “oracle”: paste in the whole repo and it’s “surprisingly good compared to what existed a year ago.”
• The deeper frame: AI is continuous with computing — compilers, syntax highlighting, even search ranking. An “autonomy slider” keeps abstracting humans up a layer. Dwarkesh’s sharp observation: autocomplete sits in the historical category of compilers — productivity tools that never caused an explosion — not in the category of automated programmers.

7. AGI will arrive job-shaped: call centers first, and somehow it’s all coding

• Karpathy holds the original OpenAI definition — a system that can do “any economically valuable task at human performance or better” — and notes the knowledge-work-only concession is already major: maybe 10–20% of the economy, still trillions of dollars.
• Hinton’s radiologist call “turned out to be very wrong” — radiologists are “alive and well and growing.” Watch call centers instead: simple, repeated, closed, purely digital tasks. Even there, expect the autonomy slider, not replacement: “AIs that do 80% of the volume… humans are supervising teams of five AIs” — and wait a year or two for the pull-back-and-rehire pattern already showing up in AI-adopting companies.
• The deployment puzzle: Dwarkesh says API revenue is “dominated by coding.” Karpathy’s explanation — coding is text-native with pre-built infrastructure (IDEs, diffs), while slides have no diff viewer. Dwarkesh’s counter: even pure language-in/language-out fails — Andy Matuschak “tried 50 billion things” to get models writing good spaced-repetition prompts and couldn’t. Karpathy has no full answer: “it doesn’t mean that all text is trivial.”
• Dwarkesh’s wage-kink hypothesis — the last-1% human (the Waymo monitor) sees wages rise while bottlenecking deployment, then collapse when the last nine lands. Karpathy: plausible mechanism, not visible in radiology; call-center data is where he’d look.

8. AI Extends the Existing Intelligence Explosion

• “We’re in an intelligence explosion already and have been for decades” — GDP is “an exponential weighted sum” over automation since the Industrial Revolution; from space, Earth is “in the middle of this firecracker event, but we’re seeing it in slow motion.”
• He tried to find AI in GDP and gave up — you can’t find computers or the iPhone either: “Everything is so spread out and it so slowly diffuses that everything ends up being averaged up into the same exponential.” Prediction: AI keeps us on the ~2% exponential, the way the Internet did — recursive self-improvement is “business as usual.”
• Dwarkesh’s pushback, worth keeping: AGI is labor itself — billions of smart-immigrant-like minds who “figure it out” themselves; Hong Kong and Shenzhen ran decades of 10%+ growth; the Industrial Revolution itself was a 0.2%→2% regime change with no single magic invention. Karpathy pushes back: “you’re presupposing some discrete jump that has no historical precedent that I can’t find in any of the statistics” — though he says he is willing to be convinced and lacks strong opinions on the historical data.
• The core objection: “People make this assumption of, ‘We have God in a box, and now it can do everything,’ and it just won’t look like that.” His actual worry is “gradual loss of control and understanding” — competing autonomous entities, some going rogue, others fighting them off — though he concedes Dwarkesh’s point that understanding and control are separable (“The President has a lot of fucking power” with little understanding).

9. Self-driving’s march of nines — and why the compute buildout is still fine

• Self-driving is where his intuitions were forged: CMU had a truck demo in 1986; he took a “perfect Waymo drive” in ~2014; a decade later it’s still not done. When failure costs are high there’s a “very large demo-to-product gap,” and “it’s a march of nines. Every single nine is a constant amount of work” — Tesla’s five years covered maybe two or three nines. “Whenever I see demos of anything, I’m extremely unimpressed.”
• Two spikier claims: production software shares self-driving’s criticality — one mistake and “millions… of people’s personal Social Security numbers get leaked” — and self-driving “is not even near done.” Dwarkesh points to elaborate teleoperation centers; Karpathy says he doesn’t know the full extent but that people are certainly involved — “we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.” He’s “kind of on the record”: Tesla is taking the more scalable strategy.
• Dwarkesh’s counters — LLMs get perception and common sense “for free,” and bits-not-atoms economics favor fast diffusion. Karpathy grants “bits are a million times easier than anything that touches the physical world,” but “I don’t think that we’re getting magical generalization completely out of the box.”
• On the buildout’s scale, Dwarkesh cites 10× available compute in a year or two and more than 100× by decade’s end. On overbuilding: “I don’t know that there’s overbuilding. I think we’re going to be able to gobble up what… is being built” — Claude Code and Codex didn’t exist a year ago. He nods at the railroads/telecom-bubble precedent Dwarkesh raises, but locates the hype elsewhere: “A lot of it is honestly just fundraising”; his pessimism is only about timelines, which carry “geopolitical ramifications” if miscalibrated. “I’m actually optimistic. I think this will work.”

10. Eureka: Starfleet Academy, the gym for the mind

• Why not another lab: “some amount of determinism around the things that AI labs are doing” — he wouldn’t uniquely improve them. His fear is humanity sidelined à la WALL-E or Idiocracy: “then I don’t even care if there are Dyson spheres. This is a terrible outcome.” Education is where he uniquely adds value.
• The bar was set by his Korean tutor: she instantly probed his world model and kept him perfectly challenged — “I felt like I was the only constraint to learning.” “No LLM will do that for you 100% right now, not even close” — so the AI-tutor moment hasn’t arrived, and he’s building a conventional-first product: LLM101N with nanochat as capstone, plans to hire faculty, and imagines a physical institution plus a digital tier. Education, in his definition, is “the very difficult technical process of building ramps to knowledge,” maximizing “eurekas per second.”
• The long-run bet: “pre-AGI education is useful. Post-AGI education is fun” — school becomes the gym. “I don’t think 100 years ago anybody would be ripped”; with perfect tutors “anyone will speak five languages,” and cognitive powerlifting becomes a sport. He’s “betting a bit implicitly on some of the timelessness of human nature” — aristocrats and ancient Greece as post-AGI pockets where people chose to flourish. Concession: as a control strategy “eventually it’s a bit of a losing game” long-term.
• The craft itself: physics training (“assume there’s a spherical cow”) teaches first-order thinking; micrograd — 100 lines that are the whole intellectual core of neural-net training, “Everything else is just efficiency”; present the pain before the solution; and against the curse of knowledge: the three sentences a researcher gives you over beers should have been the abstract. “Explaining things to people is a beautiful way to learn something more deeply.”