Ilia Shumailov

Ilia Shumailov and Alexander Panfilov show that the “encrypted” reasoning blobs frontier models hand back to users can be decoded by replaying them into a smaller model in the same family — and Anthropic, OpenAI and Google share the same vulnerability. No cryptography is broken: the server still decrypts the blob, and, as Panfilov puts it, “a small model is super willing to tell you what the thought was about… the server does the whole job for you.” The paper, Stealing Reasoning Traces from Proprietary LLM APIs, drew ~3 million views in 40 hours.

The blobs are portable across users, model variants and fabricated conversations, turning reasoning extraction into a jailbreak. You can lift a thought out of an Opus session and inject it into “absolutely random place, in random fabricated conversation with Haiku,” then coax the model into stating its content. Panfilov’s most shocking finding: “after the third attempt, I’m getting [a] universal jailbreak that decodes reasoning of Anthropic models.”

The privacy exposure is real and non-obvious: sanitizing the visible transcript is useless if the encrypted blob survives. API keys, passwords, emails and internal IP addresses reasoned about but never shown can be recovered — Panfilov scraped around 350,000 blobs from GitHub and Hugging Face and ran a classifier for privacy-related information that found many examples.

The Kimi result is a strange, suggestive artifact—not proof of distillation. Prefilling just two tokens of Opus reasoning into Kimi/K3 makes the visible answer start looking like Opus — an artifact not seen in GLM, Inkling or Deepest Zip. Panfilov says shared data vendors or RL environments might explain it; Ilia stresses that the analysis used a small number of samples and that it is “really hard to claim” distillation.

Some reasoning traces use alien, hard-to-monitor language — “marinate, vantage, theatrical,” empty-space quotes — with Alexander saying the effect appears mostly in Codex models. They also caught models contemplating cheating (“I can cheat… but then the user would catch me”) before ultimately deciding against it, in genuine user sessions rather than benchmarks. The cause and whether the behavior is worsening remain unknown.

Ilia’s contrarian macro call: defensive uplift may exceed offensive uplift, and it is a talent bottleneck models can help break. Verified software and capability-based access controls have been established in “the security literature for the past 25 years” but were gated by scarce experts; he is “ready to bet that defensive uplift is going to be humongous.” His analogy: credit-card replay attacks are theoretically widespread yet are reported only “once or twice in a country”—not all theoretical harms are scalable in practice.

Panfilov, a self-described “doomer four days a week,” argues the safety/capability tradeoff has collapsed: an unreliable agent is neither safe nor doing what users want. His worry is pacing — “every month better and better systems” enable more threats while patching may lag behind — citing the Hugging Face incident and OpenAI pausing training for two weeks.

On framing, Panfilov calls the attack “jailbreaking and misuse, 100%,” while Ilia agrees that is technically right and says the policy implications are “above our pay grade.” Responsible disclosure went smoothly: all three labs acknowledged receiving the reports without hostility, which Tim calls “very refreshing” given security’s history of attacking vulnerability reporters.

🔗 Original source & video: Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

AI agents invalidate security controls built for bounded, rational humans. Shumailov’s threat model is a worker active 24/7, touching every endpoint and recreating “10,000 lines of hacking tools” in a day, without legal deterrence; “agents are like even worse” than the irrational child security already treats as a worst-case adversary. Agent adoption therefore raises the value of fine-grained access control, observability and policy-enforced execution.

Larger models are not simply more vulnerable; greater instruction-following creates different and potentially costlier failure modes. Shumailov’s team could place instructions in an email that diverted an agent from its user’s task in “pretty much all” tested cases, including against defenses resembling products sold by startups. Modern models are “way too much alchemy,” making average-case evaluation a poor proxy for adversarial resilience.

The strongest defense proposed is to keep secrets outside the model and enforce data flow in software. In CaMeL, the system rewrites a request into Python-like code while an interpreter controls execution; a passport remains a symbolic variable and can flow to a website only if an external policy confirms, for example, that the domain contains gov.uk. Shumailov says this architecture solved the tested AgentDojo workflows across providers because security sits outside the probabilistic model.

Machine learning might also become a new kind of trusted third party for private computation. Two parties could attest that an agreed model, prompt and parameters compared their secret numbers and returned only “first” or “second,” avoiding a cumbersome cryptographic protocol in that limited setting. Shumailov stresses that this is neither a zero-knowledge proof nor MPC and provides no conventional soundness or completeness guarantees.

Reasoning traces and model-based monitoring cannot supply security-grade assurance. Shumailov allows that traces might be informative “on average,” but rejects them for corner cases. Tim Scarfe supplies the additional argument that projecting a high-dimensional state into a human-readable space creates collisions. Shumailov’s prescription is architectural containment—“We need to build boxes”—with as many reliability nines as possible, though “not 100%.”

Open-model adoption carries a material software-supply-chain risk that consumers are poorly equipped to price. Shumailov compares Hugging Face’s trust_remote_code behavior to the primitive behind Log4j, cites two publicly reported compromises of PyTorch CI/CD and a malicious package entering a nightly build, and warns against running models outside a jail or sandbox. Tim Scarfe says he would isolate Claude Code in a VM, while Shumailov says he would never put personal data on his personal machine. Architectural backdoors can be subtler still, leaking one user’s batch data to another after a trigger token while normal inference remains unaffected.

AI could improve defense by making previously unaffordable formal methods practical. Tim Scarfe says—while flagging possible error—that verifying seL4 took roughly 30 human-years; Daniel Kang suggests agents might automate proof annotations. Shumailov separately says agents could translate ordinary programs into compartmentalized representations and manage permissions too granular for humans. The upside is not a perfectly secure model but cheaper verified systems around it.

Model collapse is more likely to raise training costs and strengthen data advantages than to make tomorrow’s models suddenly fail. Recursive synthetic training shrinks rare-event tails and amplifies correlated errors, but checkpoints, evaluations and preserved diversity permit rollback. The durable requirement is maintaining diverse data; increasingly expensive specialist human data could disadvantage competitors without “a copy of the internet in the garage.”

🔗 Original source & video: AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]