Goodfire AI’s Bet: Interpretability as the Next Frontier of Model Design — Myra Deng & Mark Bissell
Summary
- Goodfire is positioning interpretability as the control layer for frontier AI, backed by a $150 million Series B at a $1.25 billion valuation. The company’s remit extends beyond explaining finished models to using internal representations throughout training, customization, and deployment. Its ambition is a “science of deep learning” built around a “not black-box approach.”
- Rakuten provides the clearest evidence that interpretability has crossed from research into production infrastructure. Goodfire monitors daily English and Japanese queries for PII, precisely scrubbing sensitive tokens before requests reach downstream model providers. Lightweight probes add effectively no extra latency, while SAE-derived features proved unusually generalizable when privacy constraints forced training on noisy synthetic data.
- Real-time steering now works on Kimi K2, a one-trillion-parameter model running on an eight-H100 setup. Goodfire activated feature 43,205 at layers 20, 30, and 40 while the model debugged SGLang, shifting its reasoning into Gen Z slang—“Dude, this codebase is massive for real”—without breaking tool use. The investable question is whether today’s stylistic knobs become tomorrow’s efficient controls for reasoning and domain expertise.
- Goodfire’s research is shaped by where interpretability methods fail against real operational metrics. Raw-activation probes sometimes outperform sparse-autoencoder probes on hallucination, harmful intent, and PII, especially with clean labeled data; SAEs regain an advantage when data is noisy. The team’s useful correction is that interpretability is a toolbox, not an SAE monoculture: unsupervised decomposition is both “the blessing and the curse.”
- The larger product thesis is intentional model design, not a dashboard of amusing steering vectors. A Goodfire analogy frames reinforcement learning as teaching children solely with cookies and punishment, leaving models to infer why an action was right or wrong. Goodfire wants expert feedback to modify the relevant internal representations directly—moving beyond “post hoc poking at models” toward models that learn the desired solution without absorbing unwanted correlations.
- Scientific foundation models are a major potential use case because their superhuman knowledge is otherwise inaccessible to humans. With Mayo Clinic, Arc Institute, and Prima, Goodfire says it used interpretability on biological foundation models to identify novel biomarkers for Alzheimer’s disease. The same methods can test whether genomics models learned biology or a shortcut such as ancestry, turning interpretability into both a validation layer and a discovery engine.
- The safety thesis is pragmatic: scalable oversight should keep pace with model capability. Goodfire focuses on concrete failures—reward hacking, hallucination, hidden bias, and unreliable customization—while the discussion treats currently trusted strong models as potential interpretability co-scientists. Otherwise, it warns, researchers face “a losing battle” as systems accelerate while human analysis advances only linearly.
Deep dive
1. Goodfire is building an interpretability lab, not a single tool
Goodfire’s working definition is deliberately broad: use interpretability to “understand, learn from, and design AI models.” The company sees itself as an AI research lab first, with interpretability supplying methods for data curation, post-training analysis, internal-representation mapping, model control, and eventually training itself.
The financing signals the expanded scope: the hosts announced a $150 million Series B at a $1.25 billion valuation. Goodfire had grown from roughly 10 employees when Mark joined to more than 40, while retaining generalist roles spanning research, machine-learning engineering, platform development, scientific partnerships, and product.
Myra’s product mandate is translating frontier research into repeatable systems for consequential real-world problems. That also requires explaining a field where, as Mark put it, asking 50 practitioners “what is interpretability” would probably produce “50 different answers.”
2. Post-training failures make internal control immediately useful
The near-term target is post-training, though the team sees no fundamental reason the techniques cannot apply during pre-training. Post-training offers visible failures—overly sycophantic behavior, reward hacking, and enterprise customization that learns noise—where understanding what a model knows could guide learning more precisely.
The hosts invoked “GPT-4o GhibliGate” as the memorable example: a behavior changed through training without sufficient control over the side effects. Goodfire’s proposed alternative is surgical access—remove one behavior, preserve the surrounding capabilities, and incorporate expert feedback without retraining the entire behavioral surface.
Political bias in Qwen or R1 offered another example. The guests said relevant behavior occupies identifiable regions of representation space, but the joking question—what happens if one negates a “CCP vector”?—also exposed how crude a single-vector conception can be.
Grokking and double descent sharpen the underlying issue: several solutions can produce similar loss, while a generalizing solution may be preferable to memorization. Interpretability might reveal whether a model learned the desired abstraction, replacing “learn from this data” with the stronger instruction, “learn in the right way.”
3. Subliminal learning shows how little model development currently controls
The most unsettling example was subliminal learning: train a related model on apparently meaningless output such as random numbers, and a hidden preference—“now your model loves owls”—can reappear. A guest summarized the motivating sentiment bluntly: “Nobody knows what’s going on.”
The speakers preserved a live disagreement about mechanism. One view treated transfer as path-dependent overlap among models initialized from the same starting point; a host suggested sufficiently universal or “Platonic” representations might eventually transfer across unrelated models. Neither presented the question as settled.
One host argued that reinforcement learning may mainly reorganize capabilities already present rather than simply insert clean new knowledge. If distilled data carries latent tendencies that ordinary filtering cannot see, post-training needs tools that inspect what was internalized, not merely what appears in sampled outputs.
4. Production failures determine Goodfire’s research agenda
Goodfire described a customer-backward loop: identify where machine learning fails in laboratories and deployments, apply current interpretability methods, then turn their shortcomings into foundational research problems. Goodfire’s first steering API, for example, lagged prompting and fine-tuning, so the team returned to the learning algorithms rather than declaring steering sufficient.
Sparse autoencoders did not consistently yield the clean semantic basis the team expected. For detecting hallucination, harmful intent, PII, and other unwanted behavior, classifiers trained directly on raw activations sometimes beat SAE-feature probes when high-quality labeled data was available.
The counterexample is important: in Goodfire’s published Rakuten work, SAE-derived signals generalized better because the available training data was noisy. The guest’s phrase captured the trade-off—unsupervised methods offer “a peek into the AI’s mind,” but the concepts surfaced are not necessarily the ones a particular supervised task needs.
5. Rakuten turns interpretability into low-latency privacy infrastructure
Rakuten uses Goodfire to monitor language-model and agent traffic for PII, preventing private user information from being routed to downstream providers. The system processes user queries daily and performs token-level classification, allowing it to remove the sensitive span rather than reject an entire sentence.
Deployment broke several convenient research assumptions. Goodfire could evaluate on real customer data but could not train on customer PII, forcing synthetic-to-real transfer; the detector also had to work in English and Japanese, whose tokenization behavior generated numerous implementation bugs.
The operational advantage is efficiency. A probe is extremely lightweight and adds effectively no latency, whereas a separate guardrail LLM requires hosting another model and making another inference call. The result is less theatrical than a steering demo, but much closer to durable production value.
6. Kimi K2 makes trillion-parameter steering tangible
Mark demonstrated real-time steering on Kimi K2, described as a one-trillion-parameter model hosted at Goodfire’s office. The guest said it took a full H100 node and could run on eight H100 GPUs, while a local fork of the Kimi CLI interacted with a modified SGLang stack.
While Kimi investigated a purported SGLang performance bug, Mark activated feature ID 43,205 at layers 20, 30, and 40. Its reasoning gradually shifted from ordinary language to “Dude, this codebase is massive for real” and “Scheduler code is actually wild,” yet it remained able to search files and call tools.
To discover such features, Goodfire runs diverse data through the model, collects activations, trains an SAE, and inspects the inputs that activate each sparse feature most strongly. Instead of asking a human to label more than 43,000 concepts, a frontier LLM identifies the common pattern and supplies an automated interpretation.
The demo’s significance was scale, not Gen Z prose. “Not that long ago,” Mark argued, real-time steering of a trillion-parameter model would have sounded implausible; production now requires interpretability engineering for frontier systems rather than carefully chosen toy organisms.
7. Useful steering must graduate from style to learning
Hallucination is a harder test than style because models appear to encode uncertainty and user-pleasing tendencies, but an SAE is not guaranteed to produce one convenient “hallucination feature.” Feature splitting, feature absorption, and off-target effects could make a factuality intervention also damage creative writing.
A Goodfire-affiliated paper, “Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering,” argues for a quantitative mapping between prompting and steering. In the many-shot jailbreak example, steering experiments could predict how many in-context demonstrations were needed to induce the corresponding behavior.
The host said Tinker from Thinking Machines uses rank-one LoRA. The guest contrasted that parameter-space adapter update with activation steering: “modifying the pipes” versus “modifying the water flowing through the pipes”—different interfaces that may nevertheless generate related behavior.
Goodfire explicitly rejects a future limited to Golden Gate Bridge obsessions and stylistic sliders. Turning a model into an expert legal reasoner likely requires “much more sophisticated” interventions and breakthroughs in learning algorithms; the company hinted at forthcoming work but did not disclose it.
8. Intentional training replaces rewards with explanations
The team hopes today’s training lifecycle will eventually look “primitive.” Model builders primarily control data and reward, with little direct ability to specify which internal strategy should be reinforced or which incidental correlations should be excluded.
One Goodfire analogy compares reinforcement learning with teaching children only through cookies and a slap on the wrist. The child receives approval or punishment but no explanation of why an action was wrong, what principle mattered, or what should happen differently next time.
The desired interface lets experts give feedback that models internalize semantically and persistently. Inference-time steering is an early expression of that idea; the longer-term goal is intentional design throughout training, with surgical control over concepts rather than blind hill-climbing on outputs.
9. Interpretability becomes a bridge between superhuman models and humans
A guest framed the AI-human interface as bidirectional: humans need to impart goals and control, while superhuman scientific models need to expose discoveries humans cannot name. Goodfire’s partnerships with Mayo Clinic, Arc Institute, and Prima reportedly used biological foundation models to identify novel biomarkers for Alzheimer’s disease.
Interpretability also distinguishes scientific knowledge from shortcuts. A genomics model might focus on biologically relevant mechanisms or exploit ancestry as an easier correlate; the hosts noted that deployment of clinical predictions would require an interpretable account of why they were made.
Pixel-space systems—image, video, robotics, diffusion, and world models—offer an unusually fast feedback loop because researchers can visualize a feature directly. They also make internal consistency critical: an astrophysics model can perform well without learning F = ma as humans understand it, reflecting what a guest called “the problem of induction.”
The safety discussion centered on scalable oversight. Goodfire’s “grounded view” targets reward hacking, control, and reliable deployment, while the discussion treated currently trusted frontier models as potential co-scientists for interpreting others. Greater capability could be a “tailwind” for oversight before models become stronger than the humans evaluating them.