Mechanistic Interpretability: Philosophy, Practice & Progress with Goodfire's Daniel & Tom
Summary
Mechanistic interpretability now has the raw materials of a “proto-paradigm,” even if the field still lacks consensus. Tom McGrath’s stack is concrete: neural networks contain understandable features; linear directions encode them, magnitude carries intensity, superposition packs far more concepts than dimensions, and features connect into circuits. These foundations do not prove that today’s tools fully recover models.
Goodfire sees algorithms as the softer current bottleneck, while insisting existing tools already create customer value. Tom says the test is whether the field could extract $1 million of information from “a million-dollar interpreter model training run”; today, it could spend the money but not productively. Daniel Balsam’s commercial framing is more immediate: an SAE is “a window into the model,” and training it is “where the work begins, not where it ends.”
Sparse autoencoders are improving, but evidence of unreconstructed “dark matter” prevents treating them as complete model audits. JumpReLU, BatchTopK, end-to-end SAEs, Matryoshka-style decompositions and other newer approaches improve or extend SAE methods, yet one scaling study suggested reconstruction curves bend before reaching 99.99% or 100%. That missing variance might be memorization, higher-order geometry or output-irrelevant noise; Tom’s stated preference is still for interpreter models to approach effectively perfect reconstruction.
Scientific discovery is a major application because capable scientific models may contain abstractions that human methods have not found. Goodfire’s Arc Institute work recovered features strongly correlated with known genomic concepts and is moving toward unsupervised searches for novel biology in models such as Evo 2. Dan’s stronger call is that even “geniuses in a data center” might prefer mechanistic interpretability: simulate physical systems on chips, extract principles from the learned computation, then better vet hypotheses before wet-lab work.
Inference-time guardrails offer a clear near-term enterprise use case. Prompt rules become “playing Whac-A-Mole,” 1,000 rules degrade task performance, frontier-model judges add another expensive call, and smaller classifiers require data and ML expertise. Goodfire instead proposes cheap monitoring of internal cognition that can trigger programmatic responses when a model appears to engage with PII or prohibited topics—while explicitly conceding that jailbreaks and other limitations remain.
Creative controls could turn latent understanding into a new interface layer, but the broader multimodal-superintelligence thesis remains contested. Paint with Ember demonstrates direct manipulation of image features, with video and music offering higher-value extensions such as asking for “a little more saxophone in the saxophone solo.” Nathan expects reasoning integrated across roughly 20 modalities, many involving scientific or natural-world domains, to produce superintelligence; Dan and Tom warn that scarce reward signals could instead yield “two models in a trench coat” without deep cross-domain understanding.
Goodfire has raised $50 million to pursue interpretability as an independent category rather than an accessory inside a scaling lab. Menlo Ventures led the round, with $1 million from Anthropic described as its first corporate investment; the host disclosed being a seed investor. The bet spans scientific discovery, guardrails and creative tooling, behind Dan’s claim that “interpretability is as big as AI itself.”
Deep dive
1. Interpretability is becoming an empirical science
Tom’s opening map treats models as interpretability’s data, alongside compute and algorithms, but gives unusual weight to “very rich empirical data”: inspecting activations and tinkering with models without overcommitting to a theory. The field still lacks “the transformer, but for interpretability,” and better observation may be what reveals the missing inductive bias.
“Hypothesis-free” does not mean assumption-free. Tom means working like someone observing the stars—carrying tentative hypotheses while remaining open to unexpected structure—whereas an SAE embeds a strong architectural claim: features are literally directions in embedding space.
Scale is largely an infrastructure problem: frontier-model experimentation can be rapid once the difficult platform work exists. Dan adds that interpretability cannot scale to superintelligence without unsupervised methods; narrowly superintelligent scientific models already exist, and their behavior is not fully explained by available human statistical techniques, making automated hypothesis generation necessary rather than optional.
2. Algorithms are the softer bottleneck, but imperfect tools already pay
Nathan’s proposed decomposition leaves compute and algorithms as the plausible constraints. Dan accepts algorithms as the softer bottleneck: Goodfire has “no illusions that we’ve cracked interpretability,” yet existing techniques already help customers understand their models.
Tom’s practical test is whether researchers can productively spend compute. He wants to run “a million-dollar interpreter model training run” and receive $1 million worth of information; today, the field could spend that amount, but he does not think the result would justify it.
Dan stresses that delivering an SAE or another interpreter model is not the finished product: “That’s where the work begins, not where it ends.” The artifact provides selective visibility; experiments, validation and domain interpretation turn that visibility into useful knowledge.
The microscope analogy makes the assumptions explicit. Dataset choice determines what goes on the slide, the loss function acts like a stain, and SAE expansion factor sets the lens size. Early biologists built modern biology from crude cross-sections; interpretability can likewise advance while simultaneously improving its instruments.
3. Better infrastructure lets researchers stop grinding their own lenses
Tom extends the analogy to scientific specialization: early researchers had to make their own lenses, much as interpretability teams still build SAE-training codebases. A mature field should let investigators “buy the lenses from Zeiss” and concentrate on experiments rather than reproducing infrastructure.
Nathan’s GPT-2-versus-frontier-model concern therefore has no answer based on model scale alone. Without infrastructure, even small experiments are cumbersome; with the right systems, large-model iteration becomes relatively easy, although building those systems remains hard.
Dan expects unsupervised interpretability to resemble unsupervised pre-training: it “often looks worse until suddenly it looks better.” Each new item on the technical tree unlocks questions and experiments, eventually creating something that can absorb large amounts of compute productively.
4. Every interpretation must cross two separate gaps
Tom’s first gap lies between the original model and its sparse approximation. Reconstruction loss directly shows that the SAE is not capturing the whole computation; improvements can come from better machine learning or from a better account of what qualifies as a “natural unit of computation.”
The second gap lies between a learned feature and its human-readable label. A feature can be computationally real while its description is wrong: intervening and observing an unexpected result may expose a bad label rather than a bad decomposition.
Current automated interpretation is “a little primitive.” Following the technique Nick Cammarata originated, a frontier model receives examples where a feature activates and is asked to describe the commonality. This works sometimes, but it uses far less evidence than a human investigator would.
Tom’s richer workflow would steer the feature, inspect what projects into it, trace where it goes downstream and compare it with related features. Better scaffolding might let frontier models perform that investigation automatically; circuits also provide stronger validation because a supposedly meaningful feature should participate in computations that themselves make sense.
5. Normal science is producing a proto-paradigm
Dan’s meta-point is that almost every scientific measurement apparatus is reductive. Researchers choose data, transformations and resolution knowing the assumptions are imperfect; the productive response is to exploit the current lens while testing its limits and searching in parallel for better assumptions.
Tom invokes Kuhn’s less-celebrated lesson: most knowledge comes from “normal science,” not revolutions. Anomalies capable of breaking a paradigm appear only after researchers push that paradigm hard enough to reveal where it becomes untenable.
His classification is therefore “proto-paradigmatic,” with some hesitation about upgrading further. A reasonably large group shares the necessary raw materials, but interpretability lacks the field-wide social consensus required to call it a full paradigm.
The history is U-shaped: early connectionist work examined individual neurons because models had 12 of them, neuron-level analysis later became unfashionable or supposedly impossible, and interpretability then returned with techniques capable of handling larger systems.
6. The emerging paradigm joins features, superposition and circuits
Its basement-level claim is deceptively important: “There are things to interpret.” Earlier skepticism was reasonable when weaker models contained more flawed heuristics and noise; failure to recover an expected mechanism can mean the model memorized the task or never learned what the investigator assumed.
The representational claim is that features are reasonably described as linearly decodable directions, with magnitude along a direction representing intensity. Tom preserves room for higher-order manifolds and subspaces, but still regards “lines through embedding space” as a useful basic vocabulary.
Superposition explains how a model with a d_model-sized space—illustratively, 4,096 dimensions—can represent far more than 4,096 concepts: feature vectors overlap, producing interference and noise that the model nevertheless tolerates. Those features then connect across layers to form circuits.
Weight-based parameter decomposition and activation-based SAEs need not be rival worldviews. Tom models the network as a giant causal graph: activations suggest its nodes, weights its edges, and interpretability seeks a reduced causal abstraction requiring both. Lee Sharkey’s attribution-based parameter decomposition and Anthropic’s circuit-tracing work are complementary routes toward that graph.
7. Bottom-up interpretation is the audit layer for alignment
Nathan contrasts unsupervised discovery with top-down representation engineering and circuit-breaker work designed around a chosen behavior such as refusal. Dan says they solve different safety problems and could both contribute to alignment.
His stronger position is epistemic: interpretability may not directly solve alignment, but it is the “measurement apparatus” needed to know whether another solution worked. Assessing whether chain of thought faithfully reflects computation, for example, requires meaningful causal graphs of what the model actually did.
“It could be that we solve alignment through bottom-up methods, but it might not be,” Dan and Tom effectively concede. Without bottom-up visibility, however, they see no way to establish that another method has aligned the underlying computation rather than merely its visible behavior.
8. SAE engineering is climbing a better loss frontier
The baseline SAE combines L1 sparsity regularization with ReLU activations, but L1 causes feature shrinkage and predictably harms reconstruction. JumpReLU and BatchTopK mitigate these effects; end-to-end SAEs optimize not only activation reconstruction but how well the original model performs with the SAE inserted.
The broader dictionary-learning family supplies other approaches, including gradient pursuit and residual-quantized autoencoders such as Matryoshka. SPADE is a more expressive sparse-autoencoder approach. These methods explore how to use compute better while retaining, to varying degrees, the core question of how representations should be decomposed.
Matryoshka arose from observing “feature absorption.” Instead of one feature for tokens beginning with A, optimization may produce “starts with A and is not aardvark,” plus a separate aardvark feature, because over-specialization lowers L1 loss.
Its nested groups predict an activation, then successively predict residual error “as we go up the shelves.” Tom would prefer an explicit tree where one firing raises another feature’s probability, but discrete hierarchical sampling is harder to optimize than the soft, differentiable construction.
9. Old dictionary-learning work may contain overlooked alpha
Tom doubts that every apparent borrowing was consciously imported; researchers often derive an idea from first principles and only later discover related work. Yet sparse autoencoders and dictionary learning were active research topics in the early 2000s, so the present field is plainly rediscovering some techniques.
His proposed “high-alpha activity” is unusually simple: hit the history books. The 2024 NeurIPS sparsity tutorial explicitly excluded sparsity for interpretability, which made Tom think, “Great. Now I’m gonna learn some things that no one else in the field knows.”
The larger methodological lesson mirrors feature absorption: an initially niche observation—such as probing which letter a token begins with—can reveal a systematic failure mode, which then motivates a new architecture. Empirical oddities are inputs to algorithm design, not mere curiosities.
10. Minimum description length offers a higher-level objective
Michael Pearce’s minimum-description-length work reframes interpretability as compression: a good explanation accurately transmits the model’s behavior in relatively few bits. Instead of separately tracking sparsity and reconstruction loss, the ideal comparison asks which equally accurate decomposition has the shorter total description.
A tree can beat a bag of unrelated features because the probability of a branch can be encoded conditionally on its trunk. That formalizes the intuition that relationships between parts are easier to describe than an independent enumeration of every part.
Cross-layer superposition gives another specimen: three layer-specific features plus the statement that they mean the same thing require a longer description than one feature spread across three layers. MDL therefore compares levels of abstraction, not merely BatchTopK against JumpReLU, which Tom expects to have roughly similar description lengths.
Nathan’s hypothetical SAE might contain 10 million latents with only 100 active on a forward pass, but MDL does not itself output a particular active-set size. Tom calls it “the ideal optimization target if only we could optimize it”; calculating and directly optimizing the true description length remain unresolved.
11. “Dark matter” suggests present SAEs will not reconstruct everything
Josh Engels and collaborators scaled their SAE analysis to ask whether more compute would asymptotically recover all activation variance. Their reported curve bent rather than continuing toward 99.99% or 100%, implying a substantial residue they named “dark matter.”
Tom emphasizes the uncertainty: the experiment has not, to his knowledge, been repeated with newer techniques. The residue might contain memorization, higher-order representational structure that linear features handle poorly, or mere noise irrelevant to outputs; measuring loss recovered rather than fraction of variance unexplained could change the picture.
This differs normatively from language-model loss. A corpus can contain identical prefixes with different next tokens, creating genuine uncertainty, while transformer architecture can add another irreducible term. Imperfect prediction may be acceptable; Tom would “much prefer” an interpreter model to reach effectively perfect reconstruction.
Dan adds that an arbitrarily wide SAE can lower error by assigning features to specific dataset examples. That creates fidelity through memorization while sacrificing general abstractions—the same tension that makes raw reconstruction an insufficient definition of understanding.
12. Continuous geometry can masquerade as a bag of heuristics
Dan’s ring example shows how a mismatch between the underlying geometry and an SAE’s feature vocabulary can produce a reductive decomposition. Transformers can encode days of the week on a ring, reflecting a mod-seven operation, while an SAE may simply recover Monday through Sunday.
Both descriptions seem intelligible in that case, but the discrete version can become deeply reductive for other abstractions. A model may implement a continuous or higher-order structure that a sparse feature dictionary displays as many separate features.
Dan applies the warning to modular addition. Known constructions use trigonometric operations, while Anthropic’s latest CLT analysis of Claude appears to reveal overlapping heuristics; the model might truly be using those heuristics, or the measurement apparatus might be incapable of displaying the underlying continuous function.
Nathan recalled grokking experiments where memorization arrived by roughly 10² steps, while generalization emerged much later, around 10⁵ to 10⁶—the last order of magnitude containing most of the training-time interval. Tom’s desired interpreter is “algorithmically neutral”: it should reveal clean grokking, half-baked heuristics and memorization exactly as they coexist.
13. Feature labels form a spectrum, not a binary success test
Dan’s taxonomy begins with features confidently linked to patterns in external inputs, then moves through a long tail of declining confidence. Goodfire’s R1 work found features qualitatively unlike ordinary language-model features that appeared to represent meaningful stages in a reasoning trace.
Other features may capture the model’s own algorithms and remain invisible from immediate inputs or outputs. In-context learning is the clean example: a mechanism integrates patterns across context, but a 20-token snippet may conceal what becomes evident across a 20,000-token window.
Auto-interpretation therefore varies its context by domain. Yet zooming out cannot solve everything: a frontier model asked to explain genome-wide activations is bounded by explanations in its training data, precisely where scientific work seeks abstractions beyond present human knowledge.
Dan suggests decomposing the task: if perhaps 10% of features can be explained automatically from input patterns, domain experts can investigate the remainder across many contexts. Nathan proposes looking across, say, 100 genomes; Dan emphasizes that experts may need to distinguish unknown biological structures from internal computational machinery, although both could produce scientifically useful information.
14. Scientific models could turn interpretation into discovery
Goodfire and the Arc Institute have recovered features strongly correlated with known genomic concepts and are moving toward unsupervised techniques intended to reveal new information. Dan says the collaboration is active and hopes for results “in the not-too-distant future,” without claiming the discovery problem is solved.
The underlying premise is that a model which genuinely generalizes must have learned something meaningful about the system, even while memorizing other portions. Existing bioinformatic annotations also come from human-designed algorithms with strong priors and uneven performance; they are useful starting points, not first-principles accounts of massive genomes.
Tom rejects a hard split between algorithmic and real-world features. Newton’s laws do not physically reside in objects, yet velocity remains a powerful abstraction for describing them; a learned internal algorithm can likewise track genuine structure in biology.
Goodfire’s current “windows” may not expose everything, but Dan argues they are already good enough to begin testing important hypotheses. The work deliberately pushes two frontiers at once: transformer modeling of genomics and the interpretability machinery used to inspect it.
15. Simulation-first science survives even aggressive AI timelines
Dan’s “hot take” is that mechanistic interpretability may remain the preferred scientific method even with “a bunch of geniuses in a data center.” Physical experiments face normative, physical and other practical barriers; autoregressive genomic models and diffusion-based materials models instead simulate parts of the world on chips.
Interpretation then extracts principles from those simulations, informs another experimental cycle and raises the evidentiary bar before wet-lab work. Tom summarizes the timeline hedge: if advanced AI takes longer, mechanistic interpretability is useful; if it arrives quickly, what are the data-center geniuses doing? “MechInterp.”
Drug development supplies Dan’s inefficiency specimen: compounds are designed, often fail or exhibit side effects, remain available because manufacturing a new drug is expensive, and get tried against other conditions until something works. He expects that costly search process to remain inefficient even after AGI unless more experimentation moves into simulation.
Clinical diagnostics add a second role beyond discovery. A family cannot be told that an AI misdiagnosed someone and “I have no idea why”; investigators also need to determine whether a surprising diagnosis reflects model failure or a previously unseen disease biomarker.
16. Multimodal superintelligence may be blocked by weak supervision
Nathan’s baseline case combines current reasoning with roughly 20 deeply integrated modalities, many involving scientific or natural-world domains, extending the GPT-4o or Gemini Flash relationship between language and images into genomics, proteins and other physical domains. Models such as AlphaFold already perform tasks humans cannot do at a comparable level; Nathan expects synthetic outer-loop training could connect those abilities.
Dan keeps “geniuses in a data center in two years” inside his wide timeline distribution but challenges the mechanism. Image-language tasks have abundant paired data and reward signals; scientific domains often lack enough labeled examples or a reliable “doing science well verifier” to teach deep cross-modal reasoning.
His contrast is coding versus agency: models can outperform humans on LeetCode-style tasks yet struggle to order DoorDash because algorithmic verification is easy in one setting and training signal scarce in the other. Scientific integration compounds that problem.
Tom asks whether AI is experiencing catch-up growth or sustaining a new trajectory. Even one network can contain “two models in a trench coat”—one set of weights doing quantum chemistry, another doing language—with shallow crosstalk unless paired supervision forces each capability to bear meaningfully on the other.
17. Guardrails can monitor cognition instead of stacking rules
Goodfire has pruned its broad opportunity set to three areas: scientific discovery, inference-time guardrails and creative models. For enterprise safety, Dan starts with the failure of prompt-based control, where accumulating “don’t do this” instructions becomes “playing Whac-A-Mole” and degrades the primary task.
LLM-as-judge systems move the problem into another frontier-model call—“great for the labs’ pockets,” but expensive for users. With 1,000 rules, batching still recreates scaling and performance problems; training a smaller guard model demands datasets and expertise many organizations lack.
Goodfire’s alternative watches internal model cognition cheaply at inference time and triggers programmatic action. Suspected engagement with PII could prompt manual review, while prohibited topics could activate other controls. Dan explicitly preserves the boundary: jailbreaks and other limitations remain, but practical deployments can still gain faster, cheaper checks.
18. Creative controls and strategic capital complete the commercial thesis
Creative models make features unusually intuitive because users can see how steering changes the output. Goodfire’s just-launched Paint with Ember tool demonstrates direct manipulation of image-model features; Dan expects still clearer value in expensive-to-edit video and music, including “a little more saxophone in the saxophone solo.”
There is no universal interface for arbitrary models. Goodfire and open-source researchers have built useful visualizations, and the company is testing biology interfaces that might generalize, but Dan repeatedly returns to the specific problem and what each lens can reliably reveal.
Goodfire raised $50 million led by Menlo Ventures, including $1 million from Anthropic as what Dan called its first corporate investment. The host disclosed his seed investment; the company intends to fund both customer deployments and foundational work while remaining independent of any scaling lab.
The recruiting and customer “bat signal” follows the three-part strategy: scientists and engineers who want to reverse-engineer models, organizations training scientific models, enterprises seeking more reliable LLM production systems, and creative-model builders seeking new interaction modes. Dan’s closing belief is maximalist: “Interpretability is as big as AI itself.”