Sebastian Raschka

Anthropic’s distillation warning turns frontier API access into a geopolitical and capability-control issue. Nathan Lambert argues GPU-constrained Chinese labs “obviously should do this”: buying Claude outputs is easier than generating comparable synthetic data internally, while Anthropic labels distributed collection an “attack.” Terms of service mainly let providers terminate access, but renewed enforcement raises the possibility that providers keep their strongest models in products or restrict premium API access.

The enforcement problem is epistemic: evaluation and distillation look identical while the API is running. Sebastian Raschka uses essentially the same loop for MATH-500, a 12,000-example math set, and book experiments with DeepSeek V3.2; only later would someone save the outputs and train. Volume, repetition, and topic distribution might expose collection at tens or hundreds of thousands of requests, but legitimate benchmarks run questions three to five times and production chatbots also generate huge traffic—creating an uncomfortable privacy trade-off.

Anthropic’s evidence looks highly timing-sensitive, particularly the gap between MiniMax and DeepSeek traffic. swyx highlighted MiniMax redirecting nearly half its traffic when Opus 4.6 launched, consistent with active MiniMax 2.5 training; DeepSeek’s roughly 150,000 exchanges could reflect a different training window. Without knowing whether requests span four weeks or six months, Nathan sees Anthropic’s DeepSeek emphasis partly as messaging: it is “the only Chinese AI name that people in the US know.”

Distilled data is not automatically valuable—the teacher must fit the student. Nathan says Qwen-dense models often teach Qwen-like open models better than stronger frontier systems, while GPT-OSS is also a surprisingly good teacher: “Just because you have this data does not mean it’s actually gonna make your model better.” A lab needs evidence that Claude data produces signal before paying to extract 100 billion tokens, especially when large models generate at roughly 40 tokens per second.

The API may be a weaker moat than the products built around it. Nathan calls the API market “brutally competitive” and suggests a distillation-worried Anthropic could release its best model directly into Claude Code without offering it broadly; Sebastian counters that chatbots, OpenClaw workloads, and users exceeding subscription caps create a large unsubsidized-token business. swyx thinks Anthropic has “Apple vibes,” while describing OpenAI’s leadership as less likely to lock models behind products out of idealism and principles—even though the last three GPT-5 Codex variants reportedly received two-to-four-week product-exclusive windows.

SWE-bench Verified has stopped discriminating among frontier coding models because both saturation and task defects overwhelm the signal. Scores cluster in the 80s while run-to-run noise is roughly 0.5–1 point, making tiny release-to-release gains “obviously bullshit.” OpenAI’s later audit found 59% of the stubborn remainder could not be solved as written; one task secretly required the exact string get_annotation, making memorization the only reliable route.

The next evaluation cycle becomes materially more expensive and more private. GPT-5 used knowledge from later Django versions, while Gemini Flash and Opus could reconstruct a task and solution from its ID—evidence that public GitHub benchmarks contaminate models even without deliberate cheating. SWE-bench Pro adds newer tasks, private/public splits, more repositories and languages, but the panel expects frontier evals to cost millions, then potentially “tens of millions and hundreds of millions of dollars,” advantaging labs and data providers with scale.

🔗 Original source & video: [LIVE] Anthropic Distillation & How Models Cheat (SWE-Bench Dead) | Nathan Lambert & Sebastian Raschka

The 2026 AI race is not winner-take-all: ideas move freely across labs, while compute budgets, hardware access, organizational culture and distribution decide who captures value. Sebastian Raschka sees DeepSeek winning open-weight “hearts,” but not permanently owning the technology; Nathan Lambert sees Anthropic’s code-first discipline, while Sebastian highlights Google’s integrated stack and OpenAI’s ability to land new paradigms as distinct advantages. China’s expanding field—DeepSeek, Qwen, Kimi, MiniMax and Z.ai—makes continual leapfrogging more likely than durable technical supremacy.

Scaling laws still work, but their economics increasingly favor a portfolio of pre-training, post-training and inference-time compute rather than simply building the largest base model. Nathan contrasts roughly $1 million-$10 million open-model training runs with recurring serving bills that can reach billions, while 2026’s gigawatt-scale Blackwell clusters could support larger models, longer RL runs and premium inference. His provocative commercialization marker: after $200 plans, “we’ll see a $2,000 subscription this year” if marginal intelligence proves valuable enough.

Coding is the clearest near-term monetization wedge because RLVR-trained models can reason, call tools and iterate against verifiable outcomes. Claude Code’s advantage appears to be more than Claude Opus 4.5 alone: the interface and agent harness let users operate in English at the system-design level, while Cursor, Codeium and conventional IDEs retain value when developers want tighter control. The trajectory is toward “the industrialization of software,” but production complexity, specification and safety-critical systems keep humans in the loop.

Open weights are becoming strategic infrastructure, with Chinese providers using permissive releases to win global influence even where US enterprises will not buy Chinese APIs. Chinese models can be hosted domestically, customized on private data and served using the customer’s compute; OpenAI similarly framed gpt-oss-120b as distribution that uses “your GPUs.” Nathan expects more open-model builders in 2026 than 2025 and argues the US needs roughly $100 million-class efforts to avoid ceding the research substrate to “Qwen, Qwen, Qwen, Qwen.”

The durable moats sit below and above model weights: proprietary data, serving infrastructure, trusted interfaces, tool integrations and hardware ecosystems. Sebastian says Google can avoid NVIDIA’s margin through TPUs and control its stack; NVIDIA’s two-decade CUDA ecosystem remains harder to displace than any individual chip; Anthropic owns coding mindshare; and ChatGPT benefits from brand, memory and habit. Closed US models remain better enough that the speakers pay for them, while open Chinese models compete on cost, licensing and customizability.

Data quality and verifiable post-training now matter more than architectural novelty, because frontier models remain recognizably descended from GPT-2. Mixture of Experts, attention variants, lower precision and better systems raise efficiency, but capability unlocks come from curated reasoning data, RLVR and tool use; Sebastian summarizes pre-training as absorbing knowledge and post-training as learning skills. The hard liabilities are legal provenance, benchmark contamination and preference averaging: RLHF can make a model broadly pleasant while sanding off the “voice” and incisiveness users value.

AGI timelines remain less decision-useful than concrete capability thresholds: reliable computer use, autonomous feature delivery, scientific specialization and measurable economic impact. Nathan expects AI to remain “jagged”—already superhuman at some code, weak at distributed ML and messy research—while Lex presses the plateau case of “Clippy on steroids.” The most credible upside may be quieter: personalized access to human knowledge, domain models built on private data and steadily more capable agents, rather than one sudden remote-worker or singularity threshold.

🔗 Original source & video: State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490