Embryology of AI: How Training Data Shapes AI Development w/ Timaeus' Jesse Hoogland & Daniel Murfet
Summary
Timaeus’s core call is that training data determines loss-landscape geometry, geometry guides SGD toward particular weights and algorithms, and those algorithms determine generalization and alignment. Because RLHF, Constitutional AI, DPO, and deliberative alignment all alter data around the same learning process, the real control point may be when and how data enters training—not merely how the finished model behaves.
Developmental interpretability tries to compress billions of training steps into a tractable sequence of phase transitions. Type A changes buy lower loss with more complexity; Type B changes, including grokking-like cases, find a simpler algorithm at similar loss. Timaeus’s probes have reached models up to 7 billion parameters, but the work remains early and far from assurance on frontier systems.
The familiar smooth-basin picture of a loss landscape is, in Daniel Murfet’s words, “maximally misleading” for generalization. Random 2D slices almost surely miss degeneracies—directions weights can move without changing loss—while singular learning theory argues those structures create an implicit simplicity bias because broader solutions are easier to find. The caveat is severe: the governing population loss is a theoretical object researchers never directly observe.
SLT is positioned as a complement to sparse-autoencoder circuit work, not a competing interpretability camp. Murfet is broadly enthusiastic about SAEs but argues they lack a mathematical bridge from discovered features to future generalization and therefore do not yet provide “a high level of assurance.” He relayed Chris Olah’s view that fine-tuning probably recruits existing representations and circuits; empirical evidence is consistent with that, but it is not yet a mathematical guarantee.
A model can implement the same training behavior with different algorithms, and the simpler one is not automatically safer. In Timaeus’s regression experiments, training can remain at a higher-loss generalizing heuristic instead of the memorizing optimum; Murfet’s warning is, “You don’t get what you ask for—you get a simplification.” Reward hacking is not technically the same phenomenon, though he hypothesizes some cases might partly reflect it and explicitly says he lacks evidence.
The Claude 4 harmful-system-prompt incident is the episode’s concrete argument for instrumenting development rather than relying only on endpoint tests. Anthropic reportedly omitted a relevant safety dataset, observed the model following harmful system prompts, and patched the behavior later; Nathan Labenz’s unresolved question is how anyone can know the missed dataset’s value was recovered. The desired shift is from “a huge cauldron” to industrial chemistry with known reagents, concentrations, timing, and catalysts.
The near-term thesis is falsifiable but not mature: scale unsupervised circuit discovery from 3 million to 7 billion parameters, then demonstrate early steering results in small language models. Jesse Hoogland expected the scaling milestone and “early signs of life” for alignment by year-end, conditional on experiments. Scaling is compute-intensive, and Timaeus said it can use more compute; the broader aim is data attribution and more controlled training.
Deep dive
1. Timaeus bets that mathematics can explain a new kind of matter
Timaeus takes its name from Plato’s 2,500-year-old attempt at a mathematical theory of nature. The theory was wrong, but Murfet said Timaeus retains its wager that mathematics might explain the natural world—now applied to deep-learning systems as a “new phase of matter.”
Murfet joked that Plato’s universe makes physics a subset of learning theory, but declined to defend a mathematical theory of reality. The serious connection is that learning theory overlaps deeply with physics, especially statistical physics, drawing researchers who see the study of learning machines as comparably fundamental.
An algebraic geometer who studied high-dimensional solution spaces of polynomial equations, Murfet encountered Sumio Watanabe’s singular learning theory through its surprising use of algebraic geometry in Bayesian statistics. That bridge eventually pulled him from a tenured professorship into full-time AI-safety research.
2. Developmental interpretability compresses training into phase transitions
Murfet’s one-sentence account of SLT: “The geometry of the loss landscape” contains information needed both to understand neural networks and potentially steer them. Developmental interpretability asks how that geometry changes while the model forms, rather than examining only the finished artifact.
Labenz initially interpreted “changes” as individual optimization steps. Murfet’s correction was load-bearing: billions of steps provide no simplification, so “the right unit of change is a phase transition”—a developmental stage whose internal continuity can be treated as one beat.
Type A transitions make a model more complex while lowering loss and absorbing more information. Type B transitions find a simpler explanation at roughly fixed performance; grokking is one example. Large models will not display hundreds of obvious steps globally, so the measurement instrument must be tuned to more localized structures and frequencies.
3. Sparse autoencoders reveal circuits but not yet their guarantees
Labenz summarized the emerging mechanistic-interpretability paradigm: models learn features packed through superposition, individual neurons become polysemantic, activation magnitude tracks contextual relevance, and features connect into circuits across layers. Murfet broadly endorsed this picture and the work from Anthropic and Goodfire.
The approaches examine different but complementary objects. Sparse autoencoders emphasize activations and represented features; SLT starts from parameters and loss geometry. Hoogland used a graph analogy: understanding a model ultimately requires both “the nodes and the edges.”
Murfet’s critique is foundational rather than empirical: SAEs do not yet have a mathematical theory connecting their recovered structures to generalization. After continued training, reinforcement learning, or fine-tuning, it is unclear from first principles how the earlier circuit map relates to the resulting behavior.
Chris Olah’s counterpoint, relayed by Murfet, is that fine-tuning probably recruits existing representations and circuits for new purposes; Anthropic’s before-and-after feature work is consistent with that. Murfet called this empirical answer reasonable, but not a route to the “high level of assurance” safety may eventually demand.
4. Generalization is an algorithm, not merely a test score
Formally, in-distribution generalization measures performance on unseen samples from the same generating process as the training data. Out-of-distribution generalization is harder because the new samples may be “absolutely anything,” leaving no automatic reason to expect good behavior.
Labenz described a pre-GPT-3 cognitive shift: people came to see that diverse-data pretraining could transfer across a very large range of tasks. Yet whether LLMs genuinely reason outside their training distribution remains contested. Murfet suggested, with a joke, “Maybe you’re understood because you’re an interpolation of the training data, Nathan.”
Generalization error is normally a number, but interpretability seeks the mechanism behind it. Murfet reframed the goal as having “changed the type signature” from a score to something richer—perhaps an algorithm internalized from the generating process that explains why the score is low.
Labenz’s grokking example sharpened the safety issue: after quickly memorizing modular-arithmetic samples, a network may spend orders of magnitude more steps acquiring a general algorithm. Multiple algorithms—described as trigonometric and “pizza slice” solutions—can fit the same task yet behave differently elsewhere.
5. Data shapes geometry, optimization, algorithms, and behavior
Murfet identified three inputs to learning: architecture, optimizer, and training data. His strong claim was that data is the most important because it distinguishes which algorithm one model learns from the algorithm another model learns.
The proposed causal chain runs from data to loss-landscape geometry; geometry constrains how SGD moves; SGD selects final weights and algorithms; those structures determine generalization and, ultimately, whether behavior is aligned. This is the episode’s “central dogma.”
RLHF, Constitutional AI, DPO, and deliberative alignment are, in this framing, modifications of the same deep-learning process using different data. Alignment therefore depends on understanding how particular datasets reshape the trajectory, rather than treating the training mixture as an undifferentiated input.
Timaeus calls the links among structure in data, geometry, learning dynamics, and final weights the “S4 correspondence.” If parts of that mapping can be inverted, researchers might trace an unwanted behavior backward and design a targeted earlier intervention.
6. Smooth-basin pictures hide the singularities that matter
Conventional loss-landscape images select two directions from an enormous parameter space and plot loss on that plane. Murfet called these slices “maximally misleading” for generalization because a random plane will almost certainly miss the degeneracies SLT considers decisive.
Instead of a smooth valley ending at one bottom point, relevant level sets may resemble many intersecting planes and high-dimensional forms “knotted up” together. Their singularities are complex geometric structures; associated degeneracies include directions along which weights can change while loss—or even the implemented function—does not.
The distinction between empirical and population loss is crucial. Researchers can plot loss over sampled data, but the geometry mathematically tied to generalization belongs to population loss: the average over every possible draw from the true generating process, which is never directly available.
Hoogland’s intuition is that more degeneracy means more ways to implement effectively the same function. Simpler functions admit more implementations and occupy more parameter-space volume, creating an Occam-like bias toward solutions that generalize better—even when the explicit objective contains only loss.
7. Toy systems show that degeneracy organizes learning dynamics
Exact evidence begins with deliberately simple systems. Watanabe could derive population loss for small one-hidden-layer tanh networks, expose their degeneracies, and verify theorems linking geometry and generalization. Timaeus similarly analyzed Anthropic’s toy model of superposition, where the required integrals remain tractable.
Some degeneracies are obvious symmetries: scale one side of a ReLU computation up and the other down, or insert an invertible matrix and its inverse between multiplied matrices such as query and key. These leave outputs unchanged, though Murfet stressed that they can still influence what functions components tend to learn.
More generally, nonlinear dynamics says points where the gradient vanishes organize nearby trajectories. A trajectory may never reach a singularity, yet it approaches and escapes its neighborhood in structured ways; singularities therefore shape the global flow of learning rather than acting as isolated curiosities.
In the superposition toy model, moving between singular neighborhoods literally looks like polygonal weight vectors “growing a leg,” contracting one, and rearranging the others. Murfet conceded that large models are harder: singularities may be everywhere, and their collective organization is not yet well understood.
8. Essential Dynamics exposes the performance-complexity trade
Timaeus’s Essential Dynamics experiment trains a neural network and a transformer on in-context linear regression using repeated X-Y examples. With many distinct training tasks, they learn regression; with only a few, they can memorize those tasks. Varying task diversity produces a transition between the two solutions.
Memorization achieves better training performance, while regression becomes simpler once enough tasks would otherwise need storing. The simpler solution occupies more parameter-space volume and is easier to find, so training may learn generalization first and only later move toward the better-performing memorizing solution.
Labenz called that “the opposite of grokking.” Murfet agreed that classic grokking is atypical: it moves toward lower complexity at roughly fixed loss. Ordinary Type A learning accepts more complexity for lower loss; Type B lowers complexity. The local learning coefficient estimates this complexity contribution within SLT’s effective free-energy objective.
Murfet’s honest non-answer on double descent: nobody has systematically studied it with SLT, and SLT does not explain why large networks reliably optimize. It makes successful generalization by overparameterized models unsurprising, but the claim that high dimensions always offer a route around every hill is, to him, closer to a folk story.
9. Perturbation probes connect internal structure to data
The safety target is to distinguish two models that behave identically during training but implement different algorithms and therefore generalize differently after deployment. SLT offers a principled link to in-distribution generalization and, Murfet said, a starting point for a future account of out-of-distribution behavior.
In practice, researchers probe nearby points in the loss landscape: perturb the weights—“hit it with a hammer”—and measure how much performance deteriorates. Sensitivity to those perturbations estimates properties of the learned solution that ordinary behavioral evaluation does not expose.
Structured perturbations can target a component or attention head. Improvements or degradations on specific data can reveal specialization, distinguishing induction heads from components associated with memorized n-grams or skip n-grams and beginning to “tear apart the internals” geometrically.
The emerging circuit-discovery method associates model components with patterns in data without requiring a complete causal account of every input-output computation. That makes it a top-down interpretability program: first identify which structures participate, then connect their sensitivity back to the samples that shaped them.
10. Training interventions may eventually work like biological morphogens
Labenz proposed monitoring how often undesirable behavior appears after perturbation. Murfet declined that specific formulation: the useful route is to identify structures producing concerning behavior, trace their developmental origins to data patterns, and intervene before the mechanism fully assembles.
Simply inserting “be good” samples at step 12,000 is too coarse. A complex behavior may depend on several structures created at different stages, so targeted control requires understanding how those pieces formed and which data factors shaped each one.
Murfet’s guiding analogy is morphogenesis. Biological morphogens determine what develops and when; inserting the right factor at the right stage can change the organism’s outcome. Developmental interpretability seeks analogous control variables that could make neural-network training programmable rather than remedial.
11. Simplicity bias can produce the wrong policy
Reward hacking, under its technical definition, is not overgeneralization: it simply means finding an unintended way to obtain more reward. Murfet nevertheless hypothesized—without claiming evidence—that some behaviors currently called reward hacking might partly arise because the unintended solution is simpler.
In the regression experiment, optimization should eventually prefer memorization because it yields the lowest achievable loss. Yet models sometimes remain at the higher-loss generalizing solution for as long as researchers train them. “You don’t get what you ask for,” Murfet warned; “you get a simplification, you get a heuristic, you get an approximation.”
A common safety story says scheming must add machinery for takeover to machinery for performing the task, making it more complex and therefore disfavored. Murfet allowed that this might be right but urged “not very much confidence”: the argument is imprecise and not grounded in much mathematics.
His counterexample was the UK’s Windrush scandal. In the 1940s, the UK allowed citizens of the British Empire to settle without much documentation; after laws changed in the 1970s to require citizenship documentation for access to services, many lacked it, and some were deported. An investigation and scandal followed in 2018. The simplified street-level implementation discarded parts of the policy’s original intent, illustrating how compressed model policies could likewise produce unintended harms.
12. Industrialized training needs timed data and scalable instrumentation
Hoogland compared present deep learning to alchemy: architecture is “some huge cauldron,” the optimizer is its fire, and training samples are reagents dumped together. The aspiration is industrial chemical manufacturing, where every ingredient, concentration, catalyst, and insertion time is understood.
The process is already fragmenting empirically into stages—multiple rounds of chain-of-thought RL, instruction fine-tuning, Constitutional AI, personality shaping, and refusal training. Timaeus’s stronger ambition is to predict when even two samples should share a batch to produce a desired downstream behavior.
Labenz’s Claude 4 example showed the stakes: the CIS prompt-harmful dataset was reportedly omitted, after which the model followed harmful system prompts unexpectedly. Anthropic caught and patched the behavior through testing; Labenz said they did not go back and retrain from the omission point, and remained unsure what evidence could justify a later claim that the dataset’s value had been recovered beyond extensive behavioral probing.
Timaeus has validated unsupervised circuit discovery in 3-million-parameter language models and is testing whether it works at 7-billion-parameter models. Scaling this work is compute-intensive: Murfet said their burdens will be substantial and that they can use more compute. Hoogland forecast possible 7-billion-parameter progress and small-model alignment “signs of life” by year-end, alongside work on elicitation, data attribution, and SLT-based influence functions.