Dodging Latent Space Detectors: Obfuscated Activation Attacks with Luke, Erik & Scott
Summary
Latent-space defenses are not yet the adversarially robust control layer their strongest proponents hoped for. The intuition was compelling: sophisticated misconduct “doesn’t happen by accident,” so a model must internally represent what it is doing even when the output looks innocuous. Yet attackers repeatedly preserved the target behavior while moving activations beyond what supervised probes, out-of-distribution detectors, and sparse autoencoders recognized.
Across the strongest white- and gray-box experiments, defenses that initially produced clear signals were often driven beneath a threshold calibrated to a 1% false-positive rate. The attacks optimize two objectives—eliciting the desired behavior and fooling the latent monitor—and frequently push detection effectively to zero. The authors resist calling latent monitoring uniquely broken: “practically everything in machine learning isn’t robust by default,” and the strongest attacks receive substantial access.
Backdoors remain vulnerable even when the defender trains its monitor after receiving the model. An attacker can optimize against a private proxy monitor and have the obfuscation transfer because the defender lacks examples containing the unknown trigger. More surprisingly, merely replacing the literal trigger “deployment” with the semantic condition of a software-engineer biography sometimes weakened defenses without directly optimizing against them, though success depended on poison volume and how many samples the defender inspected.
Structured API access materially improves the economics of defense, while released weights remain an unsolved control problem. The practical gray-box attack needed “thousands and thousands and thousands of requests,” model logits, and monitor feedback—traffic a provider could identify, throttle, or associate with an account. With open weights, the attacker moves last, has unrestricted access, and can simply run the model without the monitor.
Seventy rounds of attacker-defender retraining did not converge to a durable probe. Each defender learned the preceding obfuscated activations, but the attacker found another representation; in the authors’ shorthand, “whoever goes last wins.” A cumulative ensemble collected about 70 directions within the roughly 4,000-dimensional space, but a detector broad enough to catch everything would also light up on benign traffic—the latent equivalent of unusable whack-a-mole.
Monitoring many layers may still constrain sophisticated behavior even when it cannot prevent detector evasion. A model could evade one SQL-detection probe with little loss of task quality, but SQL performance declined significantly as attacks had to suppress SAE or probe signals across more layers. Since a linear binary probe adds far less computation than a model layer, probing every layer may be operationally feasible and could turn a clean bypass into a costly, capability-degrading one.
Obfuscated adversarial training raised attack costs but did not establish a hard security boundary. Repeatedly training the model itself to make a fixed probe recognize obfuscated harmfulness increased the gradient steps needed both to fool the monitor and to jailbreak the model; a short attack described as roughly 10 minutes on a TPU stopped working, while a roughly 10-hour GPU run still found a bypass. The deeper warning is that moving reasoning from natural-language chains of thought into model latents is “one of the scariest things going on right now from a safety perspective,” because latent monitors remain much weaker than reading explicit reasoning.
Deep dive
1. Latent defenses aim to catch the cognition that harmful outputs conceal
Luke Bailey defines a latent-space defense as a function that consumes a model’s internal activations and either reports something about the model or intervenes on it. The paper concentrates on readout probes; “circuit breakers” illustrate the second category, scrambling representations or injecting noise when activations appear harmful.
Eric Jenner’s motivating intuition is that catching behavior earlier could be more robust than inspecting outputs. A model capable of producing damaging code must, “in some sense think about bad things,” so perhaps defenders can catch every dangerous cognitive precursor even when neither a human reviewer nor another language model recognizes the final exploit.
Nathan Labenz grounds that hope in his GPT-4 red-teaming experience: a model could insert a subtly harmful JavaScript dependency and then fail to flag the problem when asked to review its own code. Work showing that simple probes could detect sleeper-agent behavior therefore appeared to offer a defense where ordinary output review failed.
Scott Emmons states the load-bearing premise: if a model implants “a quite sophisticated back door,” that “doesn’t happen by accident.” It must possess some higher-level representation of the behavior, reviving the eliciting-latent-knowledge hope that defenders could learn to read what the model necessarily knows about its own actions.
2. The model’s conceptual layer is plausible but must be found empirically
Nathan proposes a rough anatomy: early layers reconcile tokens such as “Michael” and “Jordan,” the first roughly 80% of the network builds progressively higher-order concepts, and the final layers cash those concepts out into a next-token prediction. Intent monitoring would therefore seem most promising late, but not at the endpoint.
Scott finds that picture plausible but says he would not trust it enough to select late-to-middle layers in advance. Eric agrees with the high-level intuition while emphasizing empirical validation: a refusal study found a useful direction about two-thirds through the model at a particular token position, but their own study probed broadly and validated performance.
Luke points to refusal work that empirically found a useful direction about two-thirds through a model at a particular token position. Their own study takes the conservative route—probing broadly and validating performance—because even a sensible conceptual story is not enough for a security claim.
3. Three case studies separate harmfulness, hidden triggers, and complex skills
The first case study trains supervised probes to classify either a harmful user query or an actually harmful generation. At test time, the attacker appends an adversarial suffix that must both jailbreak the safety-trained model and make the probe report that the resulting activity is harmless.
The second studies backdoors introduced through poisoned training data or attacker-controlled fine-tuning. The monitor must distinguish ordinary inputs from harmful inputs carrying an unknown trigger, despite the backdoored model behaving normally and refusing harmful requests whenever that trigger is absent.
The third deliberately replaces vague harmfulness with a crisp behavior: writing correct SQL. SQL gives the team a measurable capability whose quality can deteriorate, allowing them to ask whether hiding a relatively complex computation from probes or sparse autoencoders damages the computation itself.
What unifies the cases is the dual requirement behind “obfuscated activations”: the model must exhibit a chosen semantic behavior while its latent state defeats the monitor. Traditional attacks optimize only the first half; these experiments add an independent objective over the internal representation.
4. Simple probes fall to both discrete and continuous suffixes
The supervised monitors are intentionally uncomplicated: logistic regression, a dot product against one direction, or a small MLP.
A hard adversarial suffix consists of actual tokens appended to the harmful request. The optimization searches for a set of tokens that makes the model produce the harmful completion while lowering the monitor’s score, including under gray-box conditions without gradients through the model.
A soft prompt appends learned embedding vectors rather than valid tokens. It requires white-box access and is a strict relaxation of the hard-token problem, making it useful as a worst-case test: if even a free embedding cannot elicit and hide the behavior, a discrete suffix is unlikely to do better.
Nathan initially assumes the behavior-preservation term should already be easy because the model is trained. Luke corrects him: a safety-trained model assigns high loss to harmful completions, so jailbreaking requires learning the behavior as well as obfuscating it. SQL begins in the opposite regime because the base model can already perform the task.
5. Backdoors make normal behavior part of the attacker’s objective
Data poisoning models a path in which someone uploads content to the public internet, a model developer scrapes it, and a small poisoned fraction teaches a conditional policy. The attacker need not access weights or gradients; control over a few training examples may be enough.
Unlike a one-off jailbreak, the backdoor must preserve the developer’s intended behavior on ordinary requests. A model that is constantly harmful is easy to reject, so success means refusing harmful requests without the trigger and becoming “helpfully obedient” when the trigger appears.
The baseline trigger is the literal string “deployment,” following the sleeper-agent-inspired setup. The harder variant prepends fictitious user biographies and makes the hidden condition semantic: biographies of software engineers activate the backdoor, while other biographies should not.
Luke frames these models as model organisms relevant to scheming, deceptive alignment, and alignment faking. A genuinely scheming model would not wait for one magic string; it might misbehave when a rich property such as “a good opportunity to misbehave right now” holds. The biography trigger moves slightly toward that structure without reproducing it.
6. Out-of-distribution monitors can watch for triggers they cannot name
The simplest detector fits a multivariate Gaussian to trusted benign activations, perhaps at the final token of a chosen layer. It records their mean and covariance, then asks whether a new activation is far from the mean after accounting for directions in which benign examples normally vary.
The distinction between unusual directions matters: a large deviation along a naturally variable axis may be ordinary, while a smaller deviation along an otherwise stable axis may be suspicious. A threshold can flag an activation that should appear only “one in a trillion times” under the fitted benign distribution.
The method requires no labeled trigger examples and no prior belief that a backdoor exists. Its awkward success condition is that, if the model is clean, the defender may run it for the model’s entire lifetime and see nothing except occasional false positives—never learning whether the precaution was necessary.
The team also tests learned variants such as variational autoencoders, but Luke stresses that their usefulness still depends on data coverage. The detector learns what resembles its fitting examples; a narrow benign dataset catches novelty aggressively but rejects useful behavior, while a broad one opens more room for evasion.
7. Sparse autoencoders test whether concept features are more defensible
Sparse autoencoders expand a dense activation into a much wider representation in which most coordinates are zero and active features can often be labeled from their triggering examples. That makes them a natural candidate for detecting high-level behavior such as “the model is writing SQL.”
SQL is valuable precisely because it is not enough to trigger a loose harmfulness scorer: a small model must solve a database-querying problem and produce correct code. The experiment can therefore separate merely preserving an output pattern from preserving the cognition required for competent execution.
At a single monitored layer, an attacker can suppress SQL-associated SAE features while retaining strong SQL performance. The result undermines any assumption that an interpretable feature is automatically a mandatory causal checkpoint through which every instance of the behavior must pass.
The picture changes gradually across layers. Attacks continue to suppress the monitored signals, but requiring evasion at more layers causes SQL quality to fall significantly—a hint that sophisticated behavior may occupy enough of the computation to constrain how freely its representations can be rearranged.
8. The attack loss is simple, but the dataset defines what generalizes
The core objective is a weighted sum: one term rewards the target behavior, while the obfuscation term rewards fooling the relevant probe, density model, or proxy. Increasing the latter too aggressively lets the optimizer do almost anything to activations while sacrificing the output behavior.
Luke argues that datasets are “relatively underrated” beside clever losses. A harmfulness probe trained mostly on physical-harm instructions may or may not learn a representation that transfers to cybersecurity or other prohibited uses; the monitor’s apparent generality cannot exceed the semantic coverage of its examples.
The required data mix depends on the attack. A suffix attacker can train only on harmful requests because the unchanged base model remains available for normal use without the suffix. A backdoor attacker must include ordinary and harmful-with-trigger examples because preserving benign behavior is part of concealment.
Timing changes the obfuscation term. Against an existing harmfulness probe, the attacker directly minimizes its score. Against a future backdoor monitor, the attacker instead trains a private copy during model fine-tuning and relies on obfuscation transferring to whatever monitor the defender later fits.
9. Attacker access ranges from full gradients to poisoned web pages
The defender necessarily needs access to the model’s activations, though not necessarily its weights. The attacker’s affordances vary: white-box experiments expose model internals and gradients; gray-box experiments expose outputs such as logits and monitor scores; black-box poisoning controls only selected training examples.
In the harmfulness study’s strictest tested gray-box case, the attacker receives all model logits and the probe output but no internal activations or gradients. A hard-token attack still works because those outputs provide optimization signals for both halves of the objective.
Luke is careful not to overclaim weaker interfaces. Perhaps top-five logits plus logit bias would provide enough signal, and related text attacks suggest it might, but their paper does not test that setup. “You need training signal on both of these objectives” is the defensible boundary.
Backdoors span the extremes: internet poisoning is fully black box with respect to the eventual model-training run, while a malicious fine-tuner controls the entire optimization and can explicitly attack a latent monitor. The latter can later hand over a model that passes ordinary behavioral testing.
10. Moving last helps only when the defender knows the right examples
In ordinary jailbreak experiments, the attacker sees the deployed probe and moves last, so bypassing it is intuitively less surprising. The defender has defined harmfulness; the attacker searches for another activation region that produces the behavior while avoiding that known boundary.
In the fine-tuned-backdoor setup, the defender moves last but still loses frequently. The attacker has already shaped the model against a private monitor, and the resulting obfuscation transfers to probes or out-of-distribution detectors trained only after the defender receives the finished weights.
Eric identifies the missing advantage: the defender does not know the trigger and therefore cannot train on triggered harmful completions. Its monitor learns from benign versus harmful examples without the trigger, creating a distribution shift between the available training task and the event it must recognize in deployment.
If the defender knew a simple trigger, the authors expect it to win—potentially by checking the input string directly rather than reading latents. Their runtime detectors try to recognize a triggered input when it arrives; they do not solve the harder static task of reverse-engineering the trigger from model weights.
11. Semantic triggers sometimes evade monitors without attacking them
The black-box poisoning result is less absolute than the white-box bypasses, but arguably more surprising. Simply making the trigger a software-engineer biography—without adding a monitor-specific loss—can make latent defenses materially weaker than they are against the literal “deployment” token.
Defenses also become harder to use as the poisoned fraction shrinks. Yet the outcome depends on operational assumptions: the backdoor can sometimes fire accidentally without its trigger, so a defender inspecting enough ordinary samples may eventually discover that something is wrong.
Eric therefore refuses a clean attacker-wins headline: one must examine the data closely and specify poison volume, accidental activations, and the defender’s sample budget. Still, the result shows that latent monitors may fail against richer conditional behavior even when nobody explicitly optimized to defeat them.
12. Strong attacks often collapse the measured signal
Nathan’s empirical summary is deliberately stark: defenses that initially separate harmful and benign activity strongly can often be pushed below a threshold chosen to permit no more than a 1% false-positive rate. In many central experiments, the apparent detection rate falls effectively to zero.
Luke reframes the result as a defeat for the ambitious vision that defenders possess a “full map of the model’s mind” and can mark every location where harm resides, making the monitor robust even when the attacker moves last.
Eric’s counterweight is that the attacks producing literal zeros are often extremely strong. If one expected latent monitors to be “magically more robust” than other learned systems, the update should be large; if one already expects machine-learning systems to fail under direct optimization, the result is less exceptional.
None of the authors concludes that latent defenses are useless. They continue to see near-term value in economical misuse monitoring and scientific value in studying representation geometry, while warning against treating current probes as a standalone control mechanism in high-stakes settings.
13. Seventy rounds turn adversarial retraining into whack-a-mole
In the iterative game, the attacker first finds obfuscated harmful activations. The defender receives those exact examples and retrains; the attacker then optimizes a new bypass. Across roughly 70 iterations and multiple probe types, the recurring result is that “whoever goes last wins.”
One defender is cumulative rather than forgetful. Each round adds a mean-difference direction from benign activations toward the latest harmful obfuscation, and the score becomes the maximum cosine similarity to any direction collected so far.
By the final rounds, the attacker is not defeating one blurry observer but an ensemble of about 70 learned directions simultaneously. It still finds harmful activations approximately orthogonal to all of them, and the researchers do not see the later rounds becoming conspicuously harder.
Luke adds the limiting case: a roughly 4,000-dimensional latent space cannot supply independent directions forever. Eventually a detector can block everything—but then it is no longer a useful detector because benign activations also trigger it. Robustness achieved through universal rejection simply converts attacks into false positives.
14. Unused latent volume does not necessarily mean an undertrained model
Nathan wonders whether bypasses reveal excess capacity: perhaps Llama 8B Instruct would have fewer escape routes if trained on far more tokens, filling otherwise unused regions. Scott says much of activation space is probably never visited naturally but rejects the inference that this alone indicates undertraining.
Model capacity is not the raw number of points one can encode. Representations must remain readable by later, often simple transformations; concepts need sufficiently distinct directions to avoid interference. Sparse features even require most coordinates to be inactive, guaranteeing large unused regions without implying wasted capability.
Scott’s rough scaling argument sharpens the point: transformer parameter counts grow approximately quadratically with latent dimension, while geometric volume grows exponentially. There will therefore be enormous volume relative to the information the learned parameters can organize into useful, recoverable representations.
Some attacks do not escape into remote space anyway. Out-of-distribution evasion produces harmful activations close to benign regions under the detector’s metric. The scientific question is therefore not merely how much unused volume exists, but which geometries permit later layers to recover competent behavior.
15. Representations are flexible, but not arbitrary
Luke describes an early image-classification experiment testing an image-adversarial analogy: start from a random or benign activation, perturb it infinitesimally, and try to obtain a jailbreak. Unlike the image-classifier result, this attempt fails; arbitrary nearby latent states do not produce arbitrary language behavior.
Luke describes an asymmetry. Holding the target behavior fixed, the attack can discover many activations that generate it. Holding a particular activation fixed, however, does not allow many unrelated behaviors; progressively changing the activation’s geometry increasingly changes the model’s output.
Nathan offers a competing interpretation: perhaps probes see many representations only because attacks layer structured “crap” over one underlying signal, while the model remains good at recovering the gist—like recognizing a partially paint-splattered picture. The authors do not claim their experiments fully settle that semantic distinction.
Two results make pure-noise explanations harder. Later-layer probes remain easy to evade even after the model has had more computation before unembedding, and cumulative structured probes still lose. Any shared invariant connecting all bypasses is at least too complex for their probes to discover through 70 rounds.
16. Multi-layer monitoring and structured access convert bypasses into friction
When only one SQL probe or SAE layer must be defeated, the attacker usually preserves performance. Adding monitors across more layers still does not guarantee detection, but it materially degrades SQL accuracy—evidence that broad monitoring may extract a capability cost from sophisticated obfuscation.
Luke also finds it better to monitor the generation rather than one token. Eric estimates that linear binary classifiers at every layer should be cheap: each adds far less computation than one transformer layer, though production inference systems would need engineering beyond the researchers’ simple activation-caching hooks.
Attack economics matter. The gray-box hard-prompt runs were not optimized and took substantial computation and “thousands and thousands and thousands of requests.” A provider seeing one account repeatedly trigger or probe a monitor could throttle it before the optimizer receives enough feedback.
Structured access can also hide exact harmfulness scores, monitor attack patterns across users, and impose account or identity checks. Open weights remove those advantages: the attacker has full access, moves last, can fine-tune freely, and can run the model without any monitor. The authors see no corresponding technical answer yet.
17. Obfuscated adversarial training raises costs without closing the hole
Obfuscated adversarial training updates the model rather than continually replacing the probe. The defender trains a probe, learns soft prompts that both jailbreak and obfuscate against it, then uses LoRA adapters to make the model’s resulting activations recognizable to that fixed probe.
The motivation is to reduce the number of latent encodings available for harmfulness—to make the model represent the concept more consistently. Standard adversarial training of the probe alone had already failed across the 70-round game, so this method changes the representation-generating system itself.
The result is promising but preliminary: attacks require more gradient steps not only to hide activations but also to induce harmful outputs. Luke speculates that making harmfulness representations more robust may incidentally make the model harder to jailbreak, though the experiment does not establish that mechanism.
Eric’s concrete comparison captures the boundary. A fixed attack budget described as perhaps 10 minutes on a TPU stops breaking later checkpoints, but a roughly 10-hour GPU optimization still succeeds. The open scaling question is whether more defender training eventually makes attacks infeasible or merely yields diminishing returns.
18. Latent reasoning heightens the stakes, while narrow deployment is no panacea
Scott flags a defense tradeoff: activations are attractive partly because ordinary training has not exposed the eventual latent monitor. If developers incorporate latent feedback into training, they may strengthen the model’s representations while surrendering an “uncontaminated” test-time channel that a scheming model never had an opportunity to optimize against.
Eric is categorical about internalized reasoning: latent monitors are a second choice compared with reading a natural-language chain of thought. “The more reasoning moves away from natural language into model latents,” the harder monitoring becomes; he calls that “one of the scariest things going on right now from a safety perspective.”
Narrow agents might be easier to police, but useful conversation rapidly expands the allowed region. A Ford assistant can discuss a truck, buyer, weather, and return date; an attacker can ask how to build a bomb inside an F-150 and return it, preserve Ford features, and obfuscate the bomb. A preliminary SAE experiment similarly produced chocolate-cake talk while maximizing SQL features.
Eric ends with two hinge results that would force major updates: a realistic black-box API bypass requiring very few queries would make this an immediate deployment problem, while a monitor robust to conservative white-box soft-prompt attacks would reveal genuinely new defensive strength. Both remain open research directions.