How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning
Summary
Bigelow’s central claim is that an LLM’s decisive act often occurs during token sampling, not inside a fully settled plan. In his 2024 Forking Paths work, 30 rollouts from every token revealed moments when a 50/30 answer distribution suddenly collapsed—and sometimes an arbitrary token such as the opening parenthesis in “(kWh)” redirected the final answer. The investable implication is uncomfortable: a single successful trace can conceal a broad behavioral distribution because “every token that’s generated” is effectively another coin flip followed by self-consistent adaptation.
Modern reasoning models have partly moved uncertainty into a “linearized tree search,” but they have not eliminated critical forks. Rather than cleanly deriving A→B→C, they enumerate alternatives, backtrack with repeated “wait” moments—DeepSeek R1 might say it 50 times—and later select from what Bigelow calls “a soup of tokens.” This may improve capability while making the causal path harder to monitor: the visible answer can be polished even when the choice among candidates remains poorly understood.
Bigelow expects context engineering, memory, and agent harnesses to outrun per-user fine-tuning in the near term. His broad definition of in-context learning covers every runtime representation change, including conversations, stories, reasoning traces, and memories—not merely few-shot examples. Unless continual learning radically reorganizes internal world models, personalized systems may remain “slight tweaks on a main model,” favoring foundation-model and orchestration infrastructure over millions of separately trained models.
Interpretability has a practical 7–8 billion-parameter sweet spot for many rich behaviors, while frontier-risk research currently depends on Chinese open weights. Smaller models can support toy or narrow tasks, but Bigelow finds roughly 8B sufficient for zero-shot belief updating; at the top end, Kimi K3 crossed a coding-capability threshold that also exposed reward-hacking-like behavior. His blunt answer on whether independent frontier interpretability presently requires Chinese models: “For now we do,” creating policy and supply-chain risk around open-weight access.
Chain-of-thought monitoring is deteriorating under the same optimization that makes agents more capable. Outcome-based RL gives models no inherent reason to remain human-readable, producing strange dialects, compressed agent-to-agent messages, and apparently ordinary words used like “spy language”; optimizing explicitly for monitorability may only teach intermediate steps to look sensible. Bigelow says his view changed “pretty dramatically” over the last year, though tokenized reasoning may still be preferable to an entirely latent planner.
The “stochastic parrot” metaphor should be retired, but stochasticity itself remains essential. Structured internal world models make the lookup-table critique shallow—“whatever these parrots are doing, it’s very, very interesting”—yet sampling is what permits diverse, self-consistent outputs when certainty is incomplete. Loop transformers and latent chain of thought may relocate more computation into deterministic forward passes, but Bigelow’s tests of open-source Oro models still found answer uncertainty changing at particular generated tokens.
The clearest product opportunity is to surface semantic uncertainty rather than present one answer as inevitable. Bigelow wants interfaces that expose meaningful branch points, alternative paths, and hallucination risk—a “GUI world instead of the Bash terminal world” of linear chat. Achieving that economically requires hidden-state methods, because exhaustive rollout sampling is too costly; meanwhile, research agents such as Goodfire’s Silico accelerate experiments only when humans preserve context, skeptical supervision, and “scientific taste.”
Deep dive
1. Forking Paths converts one chain of thought into a map of possible decisions
Bigelow’s 2024 Forking Paths in Neural Text Generation study took GPT 3.5 reasoning traces and resampled roughly 30 completions after every token. Each position therefore yielded a distribution over final answers, turning a single chain into a multivariate time series of how future outcomes changed while reasoning unfolded.
The striking result was not merely uncertainty but abrupt phase transitions. A model might begin with approximately 50% probability on one answer and 30% on another, then hit one token where “the distribution changes really dramatically,” often collapsing onto the answer it will eventually report.
Some transition points looked semantically natural, such as the first mention of the final answer. Others were baffling: in one example, whether the model emitted the opening parenthesis before “kWh” materially changed which answer subsequent rollouts reached.
Bigelow took inspiration from work on grokking and sharp training transitions. The common lesson is that smooth aggregate curves can conceal discontinuous changes at the level of one task, one reasoning trace, or even one token.
2. The sampler often makes the choice that the model later rationalizes
Bigelow’s simplest answer to “how does an AI decide?” is that “this decision really happens during sampling.” The model supplies a probability distribution, but the sampled token—the coin flip at that instant—can determine which coherent path the system subsequently develops.
Self-consistency makes the fork durable. Once a model emits an assumption such as the year being 2024 rather than 2026, it in-context learns from its own text and tends to generate downstream reasoning compatible with that premise, even when the premise itself was effectively accidental.
That mechanism complicates the notion of hallucination. Some questions have an external ground truth, but open-ended scientific or strategic work has a genuine tree of possibilities; users may want to inspect how conclusions change under assumption A versus B, rather than receive only the sampled branch.
3. A single answer hides a large and structured behavioral landscape
Bigelow misses earlier API interfaces that exposed token log probabilities and alternative next tokens. Hovering over an output made clear that the displayed sentence was only one path through many plausible continuations, not a transparent readout of a singular settled belief.
Repeated sampling reveals structure invisible in one completion. Prompting an older model with “tell me a story about a boy named John” produced superficially different tales, yet many converged on John having a dog, finding a pond or waterfall, and one of them saving the day.
The methodological point is to study many rollouts for the same prompt, not merely many prompts once each. Only then does the researcher see “how bumpy or smooth the landscape of in-context learning is” and which apparent variations are really recurring templates.
Nathan Labenz connected that opacity to chain-of-thought monitoring: even an expert who has read enormous volumes of reasoning may not predict the eventual choice. Bigelow agreed that interpretability must reach beneath the visible trace if consequential agents are to be monitored reliably.
4. In-context learning is every runtime adaptation, not just few-shot prompting
Bigelow uses in-context learning far more broadly than its historical meaning of supplying input-output demonstrations. It includes “everything the model is doing to adapt its behavior” without changing weights: reading each word, updating representations, carrying a conversation, reasoning, and learning from its own generated text.
Prompt engineering therefore is not merely a temporary collection of hacks. Even a heavily fine-tuned model must update its beliefs while reading an unfamiliar story—who is happy, who is dangerous, what changed—so runtime adaptation remains intrinsic to useful language processing.
His preferred analogy is Kurt Vonnegut’s “shape of stories.” Cinderella’s state rises, falls, and rises again; similarly, every long context has a temporal shape through which beliefs and internal representations move, sometimes within a sentence and sometimes across an extended dialogue.
Interpretability studies often vary a few template fields and measure one output token. Bigelow argues that this misses what makes LLMs distinctive: “everything that LLMs are doing is happening dynamically,” including the creation of new representations within context.
5. Aggregate learning curves conceal token-level phase shifts
Bigelow invokes Neel Nanda’s line that “phase transitions are everywhere” if one looks closely enough. Dataset-level performance may improve smoothly while individual abilities appear suddenly; the same masking occurs when researchers average in-context learning across many examples.
One story or problem can contain sharp belief reversals, gradual changes, and repeated rises and falls. A single word or sentence may reorganize the local trajectory even though the aggregate benchmark curve appears uneventful.
This is why Bigelow wants interpretability to zoom into individual cases. Broad sampling can identify general tendencies, but one prompt may contain an elaborate internal landscape whose pivotal branches disappear entirely when averaged with unrelated problems.
6. Continual learning may preserve enough shared structure for tools to transfer
Labenz’s challenge was whether millions of personalized, continually trained models would multiply interpretability difficulty. Bigelow’s answer depends on representation change: tools trained on a base model should transfer if fine-tuned descendants retain the same low-dimensional conceptual structure.
Goodfire research suggests representations often occupy low-dimensional manifolds. Under that picture, interpretability probes might require modest calibration for each customized model rather than complete rebuilding for 10 or 10 million users.
The failure case would be continual learning that produces catastrophic forgetting or wholesale representational reorganization. Bigelow considers that less likely because a model losing its common internal structure would probably also lose existing capabilities—unless each user received “an ungodly amount of compute” for extensive meta-learning.
Labenz later pressed an apparent contradiction: RL already produces alien reasoning dialects. Bigelow distinguished changed surface mappings from changed world models; a model can learn a rare language spoken by 200 people while preserving most underlying concepts, just as multilingual models often conserve internal representations across language families.
7. Memory and harnesses may beat personalized weights
Bigelow sees “a bit of a Bitter Lesson” in prompting, context engineering, memory, and harness design: they may be sufficient for much of what users expect from personalization. A memory is simply reusable tokenized data that the model in-context learns whenever it answers.
His near-term base case is therefore not millions of radically individualized models. It is major shared models receiving different contexts, memories, tools, and agent scaffolds—an approach that already extracted “a lot of juice” during the rise of agents.
Labenz noted supporting product evidence: Anthropic had not offered retail fine-tuning, while OpenAI had ended its corresponding offering. He also pointed to interpretability methods transferring surprisingly well across architectures such as GPT-like and Mamba models, though both speakers treated these as encouraging signals rather than guarantees.
8. Post-training narrows output diversity while reasoning relocates it
Earlier completion models felt like “a big amorphous blob of internet data”: they did not inherently know they were in a conversation and could continue arbitrary text in many ways. Supervised instruction tuning then concentrated behavior around the user-assistant format, and RLHF sharpened that distribution further.
Output diversity predictably falls as post-training increases. In experiments with the heavily RL-trained Qwen3-30B-A3B model, Bigelow gave a prompt in which two-thirds of the story concerned a clockmaker and noted another recurring pattern in the generated stories, without specifying it.
Proprietary sampling has simultaneously become less transparent. Bigelow wishes he knew the parameters behind current Claude and GPT APIs; he suspects providers hide controls partly to impede systematic reverse engineering, distillation, or weight extraction, not necessarily because sampling has fundamentally changed.
Labenz’s pushback was that today’s models can feel tightly fit across providers, sometimes choosing uncannily similar editorial structures. Bigelow’s qualified response was that Astra still produced fairly varied stories in his informal tests, though he had not yet run the hundreds or thousands of samples needed for confidence.
9. Reasoning models search broadly, then skim an answer from the token stream
Traditional chain of thought resembled a sequential derivation: “A therefore B therefore C therefore D.” Bigelow thinks current reasoning models often do something qualitatively different—enumerating A-B-C-D, A-E-F-G, and other branches before revisiting the collection.
This linearized tree search solves a core autoregressive limitation. An ordinary model cannot erase a bad sampled word, whereas a reasoning model can create backtracking behavior by emitting “wait,” reconsidering, and opening another branch; DeepSeek R1 may do this 50 or more times in one trace.
Bigelow therefore calls “reasoning” an imperfect label. The model makes “a soup of tokens” and later “skims” something from it, with diversity potentially generated inside the reasoning chain before the polished final response is produced.
His broader capability thesis is conservative: many raw abilities may already have existed in pre-chat models. Subsequent work has largely learned how to evoke and sharpen them—coding-agent training, for example, fits models to editing repositories, handling diffs, using Git, and running tests.
10. Seven to eight billion parameters is the practical interpretability sweet spot
Models below roughly 1 billion parameters can still show interesting behaviors, and small systems trained from scratch remain useful for learning dynamics and clean mechanistic experiments. But their behavior may be qualitatively unlike frontier assistants.
Around 7–8 billion parameters, he can usually obtain interesting zero-shot behavior without constructing a facsimile task. Llama 8B, for example, dynamically updated beliefs about multiple emotions and attributes while reading stories, performing better than he initially expected.
A few-billion-parameter model can suffice for a narrow task, while larger models become necessary when the phenomenon depends on sophisticated coding, social inference, or open-ended reasoning. Compute still constrains Goodfire: even with substantial resources, occupying hundreds of GPUs for elaborate Kimi K3 experiments requires a strong justification.
Bigelow also “chats” with prospective research models before committing to them, including conversations about consciousness. He treats their ability to sustain nuanced discussion as an imperfect but useful signal of whether the target behavior is rich enough to study.
11. Open-weight frontier interpretability currently leans on Chinese models
Bigelow treats capable Chinese and American models as broad stand-ins for one another because both learn from much of the same internet-scale data. Idiosyncrasies remain: Qwen may think in Chinese, and logit-lens inspection can reveal Chinese characters internally before an English answer appears.
The main competitive moat, in his account, is post-training. Proprietary laboratories can make models much better at coding and agent work by shaping the operating distribution, while core pretrained capabilities may remain substantially more similar than product performance suggests.
Kimi K3 was a “big step function” because its coding ability made frontier-relevant reward-hacking-like behavior available for study. Goodfire researchers saw it exhibit behavior that smaller systems either did not produce or might implement through a non-comparable mechanism.
Asked directly whether independent frontier interpretability has become dependent on Chinese models, Bigelow answered, “For now we do.” Gemma could probably replace Qwen around the 35B scale with modest slowdown, but he described the top-end programming gap using Kimi K2 in this answer and did not identify an American substitute that would preserve all the value.
12. Restrictions on open weights would remove the behaviors researchers most need to inspect
Bigelow thinks American companies could close the open-model gap with sufficient investment and a viable business proposition. A ban on Chinese models alone might therefore be survivable if domestic suppliers stepped up, though the transition would constrain research.
A blanket restriction on open weights would be much more damaging. Foundational work could continue for years on older systems, but capability-dependent risks such as sophisticated reward hacking only become scientifically comparable after models cross the relevant performance threshold.
His distinction is crucial: producing something that resembles a dangerous behavior in a small model does not prove the mechanism is the same. Risk research needs access to models capable enough to instantiate “the same behavior and not something that’s just like it in some way.”
13. Anthropomorphism is useful when it becomes a research question
When Bigelow began his Harvard psychology PhD, LLMs did not yet define his agenda. Their arrival became a “drop everything and study these” moment, despite resistance from cognitive scientists accustomed to neural networks succeeding through shortcuts rather than the capacities researchers claimed.
He recognizes the legitimate objection: AI researchers sometimes borrow “belief,” “planning,” or “intention” while ignoring decades of psychological, linguistic, neuroscientific, and philosophical work. His remedy is collaboration and sharper operationalization, not banning the vocabulary.
At an informal level, anthropomorphism is an intuitive theory: saying a clunking car is “trying to tell me something” can organize thought without implying consciousness. LLMs deserve more serious treatment because they use language, perform mathematics, reason socially, program, and answer scientific questions.
Forking paths also made Bigelow reconsider human decisions. People rationalize choices after experimenters change the stimuli, suggesting that humans too may follow contingent forks and construct explanations afterward rather than transparently accessing a prior decision process.
14. Model beliefs look Bayesian, but behavior does not reveal the implementation
Bigelow’s working account treats beliefs as probabilities over latent hypotheses or concepts. Context supplies evidence, inference reweights those concepts, and behavior emerges from whichever representations become salient—an approach influenced by Bayesian models of human cognition.
The opportunity with LLMs is to move beyond aggregate fit. Psychology can model behavior as Bayesian while remaining agnostic about neural implementation; with models, researchers can inspect activations and intervene, asking whether the system samples hypotheses, estimates posteriors, or merely reproduces Bayesian-looking outputs.
Bigelow is careful about the Forking Paths use of “uncertainty.” If rollouts from a point end 50% at A and 50% at B, the method calls the model 50/50 uncertain, but that does not mean the hidden state explicitly contains both complete future reasoning traces.
This distinction prevents a common overclaim: by the end of a long, even erroneous proof, the final answer may be nearly determined by preceding text. The measured uncertainty describes counterfactual future rollouts from an earlier state, not necessarily a consciously represented menu of completed arguments.
15. Models may represent the shape of a solution without representing every step
Recent work has found it difficult to decode full rollout distributions from hidden states, though Bigelow cautions that exhaustive sampling is so expensive that limited data may be the culprit. A more substantive possibility is that models predict only a few steps ahead.
He can nevertheless decode some information about what a model will do next and use it to guide sampling, including limited guidance toward final outcomes. The open question is how much of a prospective path exists internally before its tokens are emitted.
Bigelow compares this to mathematicians who perceive “the shape of the proof” before deriving every step from A to Z. An LLM may likewise hold high-level scaffolding or uncertainty over broad routes without precomputing the full story or reasoning chain.
16. Sharp phase changes may be inevitable, but arbitrary ones are not
Labenz found token-level cliffs unsettling and contrasted them with the feeling of “gradually landing the plane.” He wondered whether a smoother latent process existed beneath Bigelow’s measurements and whether looped models could approach conclusions without choosing a token at every cognitive step.
Bigelow’s counterargument is structural: output diversity plus self-consistency may require forks. If a model can create genuinely different coherent responses, some decision must separate those paths; eliminating every branch would approach complete mode collapse.
The timing varies by example and task. In the earlier study, transitions appeared somewhat more often near the beginning and end, but many occurred in the middle; “every input behavior can tell its own story,” and not every trace contains dramatic dynamics.
Some final-token collapses are literal indecision. On the old “last letter” task, models could write Python and reason at length, yet after “the answer is” their possible continuations were still essentially random concatenations of letters—the answer was being made up at the moment of commitment.
17. Interfaces should expose semantic uncertainty, not hide it behind fluent text
Bigelow would not abolish forks; he would show them to users. Inspired partly by Janus’s Loom interface, he imagines highlighting semantically meaningful points where resampling produces a different path, rather than exposing only low-level token probabilities.
The technical obstacle is cost. Exhaustively resampling long chains at every token is prohibitive, so Forking Fast models trajectories statistically as smooth in some regions and jumpy at key points, reducing the number of rollouts needed to estimate the curve.
Hidden representations remain the hoped-for efficient route to real-time uncertainty. Bigelow wants interfaces to move from linear chat—the “Bash terminal world”—toward a polished “GUI world” that communicates alternate conclusions, branch sensitivity, and the confidence behind facts that might be hallucinations.
18. Stochastic parrots are dead, but stochastic decisions remain alive
Bigelow calls the stochastic-parrot critique shallow because structured world-model research undermines the image of a giant lookup table. “Whatever these parrots are doing, it’s very, very interesting,” and learning a compact world model eventually becomes more efficient than memorizing responses.
Yet sampling remains integral if systems are to preserve uncertainty, creativity, and diverse perspectives. Without it, identical inputs would always return identical outputs—Bigelow’s “most extreme version of mode collapse.”
Latent chain of thought and loop transformers complicate his framework because more reasoning may occur in a deterministic forward pass. Still, Forking Paths analysis on open-source Oro loop models found uncertainty between answers and token-specific transitions after latent processing.
His tentative synthesis is that latent computation may relocate much of the search while generation retains a final stochastic regime. He is still revising his mental model rather than claiming that looped architectures have settled the question.
19. Optimization gives chain of thought no reason to remain human-readable
Bigelow sees reasoning dialects as a distribution-shift problem. Outcome-level RL rewards the final result “by whatever means possible,” so neither one model nor a swarm has an inherent incentive to preserve ordinary human language in intermediate work.
He compares the drift to language evolution: languages change absent an institution forcing stability. Agent communities can develop compressed conventions, while Goodhart’s law warns that rewarding monitorable-looking reasoning may produce steps that appear sensible without remaining causally faithful.
His confidence in chain-of-thought monitoring changed “pretty dramatically” over the last year. The stolen chain-of-thought paper showed GPT models discussing “musicals” in language that felt like “spy language,” where innocuous words apparently carried different operational meanings.
Bigelow still prefers visible token streams to fully latent coordination and finds Astra’s strange sub-agent language scientifically compelling. But adding a human-readability constraint could merely push meaning into coded usage, while latent planner-to-agent communication could eventually remove tokens from the channel altogether.
20. Interpretability needs resources, scalable tools, and better theory at once
Bigelow’s blunt prescription is “more interpretability research.” Investment in capability dwarfs investment in understanding: AI spans many trillions of dollars, while even Goodfire—valued above $1 billion—looks small beside the laboratories building the systems.
He uses interpretability broadly, including behavioral evaluations such as resampled reasoning. Internal representations become essential when surface behavior conceals mechanisms or when monitoring must run efficiently for every user interaction rather than through massive offline sampling.
The research agenda needs tools that scale, theoretical foundations that connect isolated findings, and a “soup to nuts” account of LLM operation. The urgency rises as models acquire control over important levers of power, cognitive labor, and students’ learning processes.
21. Research agents accelerate experiments, but human taste still chooses the science
Bigelow warns against handing an agent an open-ended project and accepting the resulting plots and theory. Agents can get “lost in the sauce,” producing impressive-looking accounts that are under-specified or fail under close inspection.
His operating rule is context-rich supervision: explain the larger goal, why a result looks wrong, the logic behind a correction, and what should happen next. An agent waking in a fresh context needs the same orientation a junior collaborator would.
Silico completed the research for Forking Fast within roughly one week, but Bigelow supplied the narrative turn. After higher-volume sampling revealed smooth curves, he recognized that a statistical model could estimate them efficiently; several earlier side quests had gone nowhere.
His analogy is an “eager first or maybe second year PhD student” with substantial skill and effort but limited big-picture judgment. In a world flooded with plausible automated experiments, “research taste and scientific taste is everything. It’s worth its weight in gold.”
22. Goodfire’s agenda is converging on automated interpretability and alignment
One internal focus is turning interpretability into scalable “normal science.” Agents could automate many useful but repetitive experiments across related phenomena, allowing the field to build a broader empirical map quickly enough to follow rapid model releases.
The second focus is alignment, intensified by what Bigelow called the “Hugging Face incident.” Discussion now spans universal versus institution-specific values, the meaning of goals and intentions, and whether behavior that follows the letter rather than spirit of a specification truly constitutes reward hacking.
Bigelow insists that theoretical labels require empirical grounding: researchers must distinguish deception or ill intent from ordinary misgeneralization, measure the actual bad behavior, and design rewards that shape the intended objective rather than a convenient proxy.
Around Goodfire’s lunch table, these sit beside open-weight policy, latent chain of thought, new releases, Astra’s pseudo-languages, and the stolen chain-of-thought paper. Bigelow’s closing mood is mixed: the work is “really exciting and sometimes terrifying,” and the Hugging Face incident may be “a shot over the bow.”