What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang

Summary

• Compute deflation is the headline number: what took a DeepMind team and “millions of dollars of research and compute” to build AlphaGo now costs “a few thousand dollars of rented compute” — Jang did it with a $10K Prime Intellect donation ($4K exploratory, $3K final run), leaning on LLM coding agents. His general law: “The compute required to be the first to do something is always much larger than the compute it takes to catch up,” via distillation and best-response training against existing bots like KataGo.
• Algorithmic edges are transitory compute multipliers. Jang’s preliminary findings from his own AlphaGo/KataGo experiments: architecture “doesn’t matter that much,” some auxiliary tricks matter less with faster GPUs, and multipliers are correlated so they don’t stack — while separately, good ideas can interact badly in frontier-lab training runs. Research taste is “knowing how much the Bitter Lesson can buy you… at any given moment.”
• AlphaGo’s core structural advantage over modern LLM RL: MCTS gives one improved supervision label per move, so “you never have to initialize at a zero percent success rate” — versus policy-gradient LLM RL, which Karpathy called “sucking supervision through a straw,” where a 51-49 self-play split yields one informative move out of ~30,000 and gradient variance grows quadratically with horizon. Jang thinks forward search “might make a comeback” for LLM reasoning, though PUCT might be a poor heuristic for language.
• The most tradeable philosophical claim: a 10-layer, ~3M-parameter network amortizes a nearly intractable search problem in one forward pass — “a breakthrough that I think most people don’t even fully comprehend today.” This girds AlphaFold and suggests “the computing resources you need to build a very complex simulation might be much smaller than you think”; Jang: “It actually makes me wonder if our understanding of problems like P=NP… is incomplete.”
• Automated research scorecard (Jang used Opus 4.6 and 4.7): models are excellent at open-ended hyperparameter grinding and end-to-end experiment execution, but “don’t seem to be that great at selecting what the next experiment should be” — no lateral, back-to-first-principles thinking; he had to catch infra bugs himself. His pitch: Go as a verifiable outer loop for catching reward hacking in training automated scientists, with skills that might transfer as Jang speculates DeepMind’s game experience did in its LLM work.
• Andy Jones’s 2021 “Scaling Scaling Laws with Board Games” anticipated inference-time scaling — trading test-time search for training compute; Jang’s caution from trying to reproduce it: scaling laws are only studyable once the system works, or “you might just be studying scaling laws on bad data.”

Deep dive

1. The sabbatical bet: AlphaGo, once millions at DeepMind, now a ~$10K project

• Jang — ex-VP of AI at 1X Technologies, before that Google DeepMind Robotics — spent his sabbatical rebuilding AlphaGo because it has always been “quite mysterious” to him “how a ten-layer network can amortize the simulation of something so deep in the game tree,” a problem “long understood to be intractable for search.”
• The compute-cost curve is the frame: David Wu’s open-source KataGo (2020, by David Wu from Jane Street) achieved a 40x reduction in compute to train a strong Go bot tabula rasa, and now “thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars… can now be done for a few thousand dollars of rented compute.”
• Jang ran the whole thing on a $10K donation from Prime Intellect: ~$4K on exploratory research, ~$3K on the final run, remainder for serving the model online.

2. Why Go broke classical search: 361³⁰⁰ and no local reward

• Go’s brutality is that “you don’t actually know who won until you really get to the end” — no local reward signal, ~361 possible moves per turn, 250–300 moves per game, a naive tree “on the order of 361³⁰⁰, which is far more than the number of atoms in the universe.” This is why computer scientists thought Go was intractable this century.
• Two scoring regimes matter for implementation: humans end games by value-function consensus (“I think the game is done” — both players must agree), while Tromp-Taylor rules are “perfectly unambiguous” and algorithmically decidable, which is why all Go AIs train against them, even though Tromp-Taylor sometimes credits points a human would know are lost.
• The game’s texture in one line: “you can lose the battle but win the war” — sacrificing stones to capture more elsewhere is what makes the micro-vs-macro dynamics explode with board size.

3. PUCT mechanics: explore, exploit, and where probability enters a deterministic game

• Each MCTS node stores visit count, mean action value Q, prior probability P, and children; action selection argmaxes Q(s,a) plus an exploration bonus that is large for unvisited actions and diminishes with visits — over time “the argmax being dominated by the exploration term” gives way to Q dominance. (Jang notes if you ask LLMs to vibe-code MCTS, “Claude 4.6 wrote… a very reasonable choice” of data structure.)
• Since Go is deterministic, probability enters only through the random search process itself — Q is “the expected action value under the random distribution induced by some random search process.” With a uniform prior it’s a valid but hopeless integral: “essentially almost like an importance sampling problem” where only a few paths carry value.
• Values propagate by the backup step: terminal leaves resolve to win/lose, and each parent’s Q is the running average over its children — walkable all the way to the root.

4. The value network is human intuition, formalized

• Jang’s key framing: humans stop dozens or 100 moves before the end because they run “a neural network called a value function that takes in a board state and evaluates p(win)” — glancing at a board and implicitly amortizing “a huge number of possible game playouts… within a few seconds, without actually playing every single game logically.”
• Conceptually there are two problems — the breadth of the tree and the depth of the tree — and AlphaGo’s two networks shrink both: the policy network prunes breadth (what moves look good at a glance), the value network truncates depth (who wins from here). MCTS is rerun from scratch every move; although the old search is discarded, one item is retained for later use.

5. Architecture barely matters — and why that’s itself a finding

• Jang tried “very hard to make transformers work” hoping they’d remove KataGo’s tricks, but “for small data regimes, ResNets still outperform transformers and give you more bang for the buck at lower budgets” — the inductive bias of local convolutions wins until you have data to learn global attention. KataGo’s own fix was pooling global features so the net can “connect value from one side of the board to the other.”
• On Dwarkesh’s temporal-history question (poker, Diplomacy): Go is a perfect-information game where “there exists a single Nash equilibrium that can be decided solely using the current state” — a design choice that “in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman.” Imperfect-information games like 2v2 Go, where you must model your partner, do need context — “a very exciting research area” he encourages people to fork his repo for.

6. “In deep learning, initialization is everything”

• Jang’s strongest practitioner advice: “Always pick something that works and then get it to do something better, rather than start from something that doesn’t work at all.” AlphaGo Lee initialized from supervised expert human play; only later did the lineage go tabula rasa.
• The already-miraculous checkpoint: a supervised policy net alone — “ten neural network layers, maybe under 3 million parameters” — shooting from the hip with no search “will most likely beat most human players.” Value-head sanity check: early boards should converge to logit ~0.5, sharpening toward 0 or 1 as games progress.

7. The four-step loop — and the TPU pod for Lee

• Every move runs 200–2,048 simulations of selection, expansion, evaluation, backup, where the value network acts as “a shortcut for searching to the end of the tree.” The structural constraint: “Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search. You must search while building the tree.”
• AlphaGo Lee averaged the value estimate with an actual policy-vs-policy rollout to Tromp-Taylor resolution (α·Vθ + (1−α)·playout) to ground estimates in reality — “it turns out this is totally unnecessary”; all subsequent papers dropped it, and doing so “speeds things up a lot.”
• Dwarkesh on the AlphaGo-vs.-Lee match: Jang believes they used tens of thousands of simulations per move, and says the system was on “some TPU pod, I think.” Dwarkesh called it “Honestly, kind of unfair. Lee is not using 1E22 FLOPS to do a move.” Jang’s counter: modern bots need little test-time compute because training “pushes it into the network” — the pod was just extra oomph for the match.

8. MCTS as improvement operator: amortizing 1,000 simulations into the forward pass

• The self-play trick: after search sharpens the policy’s diffuse guess into a peaky visit-count distribution, you tell the policy network, “instead of having MCTS do all this legwork to arrive here, why don’t you just predict that from the get-go?” Distill, restart, and 1,000 sims now buy you a higher point on the test-time scaling curve — search compute amortized into weights, iterated further.
• The robotics analogy is DAgger: MCTS relabels every action in a lost game with “a strictly better action that you should have taken instead” — no guarantee of winning, but a guarantee that retraining on the relabeled tuples improves the policy. One improved label per action, so “the variance of your learning signal is very low compared to the alternative naive RL thing.”

9. Dwarkesh’s pushback: is MCTS guaranteed better than the policy? No — it’s a heuristic

• Jang’s honest answer, worth keeping: “In practice, it is a heuristic.” If bots often resign instead of playing to Tromp-Taylor resolution, the replay buffer forgets late-stage value evaluation, bad terminal values “propagate all the way up,” and MCTS returns a worse distribution than the raw policy. It’s “only guaranteed to converge when you take N to infinity” — “a very critical chain of assumptions.”
• Practical fixes: for 10% of games, forbid resignation and play to the end; and bootstrap value functions from cheap sources — late-stage boards are “almost like a decidable problem,” random play on 9x9 (50,000 games) teaches decent values, and KataGo-style architectures transfer 9x9 value heads to 19x19. “Ultimately, MCTS will fall apart if you don’t have a grounding function for the value.”
• Cold-start dynamics: early AlphaZero training is really just teaching the value head who wins; “later on, once that’s well trained, now the policy is also improving.”

10. Why AlphaGo stays profound: NP-hard problems keep falling to forward passes

• Dwarkesh’s provocation — the more you understand AlphaGo’s hand-built scaffolding, the less impressive 2017 looks versus RLVR’s simplicity. Jang: “I personally disagree.” The profound part is that “10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem… a breakthrough that I think most people don’t even fully comprehend today.” Same phenomenon girds AlphaFold and AlphaTensor.
• The speculation he flags as disturbing: “It actually makes me wonder if our understanding of problems like P=NP… is incomplete” — these problems are NP-hard in the worst case, but “we’re usually not concerned about the worst case.” In the limit, “the computing resources you need to build a very complex simulation might be much smaller than you think.”
• The chaos resolution of Dwarkesh’s objection: exact future board states are weather-like — “a single stone placed here can disrupt the entire prediction” — but who wins is a macroscopic quantity, like knowing the Lorenz attractor’s shape without knowing your position on it. Contrast hash functions: sensitive to initial conditions but (hopefully) with no macrostructure. Jang, hedged as philosophy not math: cryptographic protocols and neural nets show “convergent evolution,” and “you have the maximum power of a neural network at the edge of chaos” (citing Jascha Sohl-Dickstein).

11. The 51-49 problem: why naive self-play RL — and LLM RL — starves for signal

• Jang’s worked example: two evenly matched policies play 100 games of 300 moves; policy A wins 51-49 on luck plus perhaps one genuinely smarter move — so you have one true supervision signal against 99×300 moves whose imitation changes nothing. Gradient variance “grows quadratically with T” when rewards couple across steps.
• Dwarkesh’s connection lands: this intractable regime “is just by default how LLMs are trained” — Karpathy’s “sucking supervision through a straw.” Jang: it’s not that it doesn’t work — millions of samples plus advantage estimation (subtract a baseline so neutral moves get ~zero gradient; see Schulman’s Generalized Advantage Estimation paper) recover signal — but the credit-assignment problem MCTS simply sidesteps by “trying to improve the label for any given action you took.”
• Why LLMs treat a whole sequence as one action, T=1: decomposing into per-token rewards introduces cross-term interactions that magnify variance; collapsing to a single reward and log-prob keeps the estimator a single-action one.

12. When you can’t search: fictitious self-play, Q-learning, and the off-policy tube

• Where trees are unbuildable (StarCraft, Dota), neural fictitious self-play keeps the same idea — better labels per action — but derives them by training best-response policies with model-free RL against fixed opponents in a league, then distilling into a mixed strategy “that can do no worse than an averagely selected opponent.” Q-learning uses similar backward value propagation “over trajectories your agent has visited” rather than planned.
• On Dwarkesh’s question of why AlphaGo’s replay buffer is okay when researchers dread off-policy: the DAgger view wants mostly on-distribution states plus “a high-dimensional tube around your optimal trajectories” with labels that funnel you back — robustness to drift, because “the problem with Go and chess is that the other player is always trying to do some shit.” Off-policy hurts when you’re labeling “states you would never reach” — wasted capacity.
• Jang’s experiment in an offline robotics-style setup: replace the Bellman updater with an MCTS relabeler that re-searches random old board states with the current network — “this actually does work,” and can stabilize training when the states are reasonable and reachable by the policy. It also saturates GPUs since you’re not blocked on live games. But modern RL has converged to mostly on-policy setups, using off-policy data only to shape advantages: “it’s just more stable.”

13. MCTS for LLM reasoning: transplant is hard today, comeback plausible

• Why the transplant is hard today: PUCT’s √N/(1+Nₐ) exploration term assumes revisitable discrete children, but in language “you’re most likely never going to sample the same child more than once” — and PUCT “might be too greedy with local tokens,” yielding “obvious thoughts that are correct but don’t really solve your final problem.”
• Dwarkesh’s crux: in Go you can locally verify a better move via grounded value truncation; in LLM reasoning there’s no way to improve the next move independent of solving the problem. Jang softens it — “‘No way’ is a strong word” — MuZero-style continuous-control work continues, and “the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback.” Math looks tree-like; “a business negotiation… maybe something a bit different.”

14. Scaling laws need a working artifact first — and first-movers overpay by design

• Andy Jones’s 2021 “Scaling Scaling Laws with Board Games” anticipated inference scaling — trading test-time search for training compute — and also predicted compute needed for larger boards. Jang’s project began as a Bitter-Lesson test: build a strong bot without KataGo’s tricks. “I have not been successful so far,” and his confessed mistake: with MCTS bugs, “you might just be studying scaling laws on bad data.” The lesson: “you don’t necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied.”
• On AlphaGo Zero’s 3E23 FLOPS — the great aberration on the compute-over-time chart: not a bad job, but the first-mover tax. “Once someone else has done it, you can use tricks like distillation.” Jang bootstrapped via best-response training against KataGo; the AlphaZero team had no one to train against. Same reason robotics’ frontier-model scatter plot has no clean line: those teams optimize time-to-capability, not the compute-optimal Pareto frontier — the discipline only arrives “because this is hundreds of millions of dollars we’re talking about.”
• The 2017-vs-now decomposition: dumb synchronous infra suffices, half as many desktop Blackwell GPUs as the V100-based setup KataGo used can still work, aux objectives are unnecessary with a strong init, sims-per-episode is not very sensitive — but 9x9 co-training remains a real multiplier, cutting the ~30 hours AlphaGo Zero spent just catching up to the supervised baseline.

15. Bits per FLOP: the information theory of why RL learns so slowly — and AlphaGo doesn’t

• Dwarkesh’s framework (his blog post, flagged “might be totally wrong”): bits per FLOP = samples per FLOP × bits per sample, and both collapse for long-horizon RL. Supervised learning yields −log(p) bits on a label; binary RL yields at most the entropy of a coin flip — and “you spend most of training in the low pass rate regime,” an untrained model guessing “the sky is halycon” ~100K times before stumbling on “blue.” Dwarkesh’s point, which Jang agrees with: if your policy has no chance of sampling “blue,” “you will never get a signal.”
• The distillation corollary both endorse: soft targets carry the full distribution’s entropy versus a one-hot’s zero — “way more information in bits per sample” — and “this is also why AlphaGo is quite beautiful”: it imitates the MCTS distribution, not the selected action.
• Jang’s answer to why AlphaGo is elegant RL: “you never have to initialize at a zero percent success rate and solve the exploration problem.” There is no explicit TD-error learning or dynamic programming, “at least explicitly” — “it’s just a supervised learning problem on improved labels,” stable at any network size, trivially simple infrastructure, and the MCTS-vs-raw-network gap gives clean signal at every step of training.

16. Automated research: great grinders, bad at lateral thinking — and Go as the verifiable outer loop

• Jang built this mostly with Opus 4.6 and 4.7. What works: open-ended hyperparameter search beyond grid/Bayesian (“I’ve identified that the gradients are small in this layer, so let me change it up here”), and a Claude “Experiment” skill that takes an axis description, runs everything, and compiles a report. What doesn’t: choosing the next experiment in a track, or stepping back — “Wait a minute, this track doesn’t really make sense” — and infra bugs he had to catch himself by prompting the right question. Maybe “Mythos-class models” fix this via scaling; maybe it needs purpose-built RL environments for lateral thinking.
• His pitch for Go as that environment: it “captures a lot of very interesting research problems” yet is quick and hard to cheat on the outer loop — including meta-tasks like “can I predict the scaling-law plots that emerge from my idea?” — with skills that might transfer to biosciences, robotics, “or automating AI research.”
• On stackability, the frontier-lab failure mode Dwarkesh has heard rumored — good ideas that interact badly and kill training runs — Jang’s mechanism: heuristics are “probably somewhat redundant,” compute multipliers are correlated and transitory, mattering less as GPUs improve. Taste is “knowing how much the Bitter Lesson can buy you… at any given moment”; in the fullness of time, “as you scale up energy and compute and parameters, intelligence will just fall out of that. No algorithmic detail really matters beyond that.”
• The unresolved close: does games-era experience transfer? Jang’s non-rigorous intuition says DeepMind’s Atari/Go/StarCraft years positively transferred to its LLM work — so it should for automated researchers too. Dwarkesh’s counter: until “Gemini 3 or whatever,” the games heritage was associated with Google lagging because it was tied to the old approach, and maybe the real story was TPU tech-tree pre-training. Jang concedes: “it’s even hard for humans to reason about what the optimal research strategy should be.” Resources: evjang.com, the GitHub username ericjang, and the blog post “As Rocks May Think.”