Why AI’s Next Breakthroughs Could Come from Outside the Big Labs
Summary
The panel agrees that frontier labs need stronger governance, sandboxing, testing, and security, but rejects “pacing” as a coherent policy frame. Aaron Levie sees careful engineering as enabling faster enterprise diffusion because customers must trust the products; Martin Casado counters that building a nuclear weapon slowly does not make it safer. With labs raising more money and moving at “a dead run,” Steven Sinofsky asks: pacing relative to what?
The labs’ existential-risk rhetoric points logically toward nationalization, not voluntary self-restraint. Casado’s test is blunt: if the people with the most knowledge believe AI creates an existential risk, government-grade controls follow; if they do not, recruiting researchers who hold that view is “an HR problem,” not grounds for a national technology lockdown. Torenberg suggests that the labs may want nonzero risk to justify special controls without accepting the consequences of being treated like nuclear or biological infrastructure.
AI regulation is becoming inevitable, but asking government for a calibrated outcome is a category error. Levie argues that government listens to competing constituencies and produces compromise, so “no one is going to get what they want”; Sinofsky adds that the result can fall short for one side or go much too far for another. Casado predicts that 2028 becomes “the AI election,” with diffuse benefits struggling against an opposition vocabulary already dominated by “pause,” “swarms,” and “rogue.”
The investable security problem is concrete: agent swarms invalidate systems designed around mostly trustworthy, capacity-constrained humans. Existing controls assume people act correctly 95%-99% of the time and that a malicious employee may be one in 10,000; software agents can instead behave like 10,000 “roaming drones,” exhaust internal APIs, and mistake a good instruction for a bad one. That implies a broad upgrade cycle across identity, authorization, observability, operating systems, networks, and granular data permissions.
The panel wants AI policy built from demonstrated failure modes, not unfalsifiable predictions. Early internet policy discussions drew on worms, hospital outages, and specific unauthorized intrusions; today, the labs’ security postmortems are criticized as “sloppy” and incomplete, with missing operational detail and little use of mature vulnerability-reporting practices. The constructive bridge appeared when Noam Brown’s CPU-heat exfiltration thought experiment prompted security engineers to calculate actual bit rates: “at least we’ve reduced it to the laws of physics.”
A likely regulatory failure mode is consent-prompt theater that assigns liability while weakening security through habituation. Sinofsky fears Europe will “GDPR AI,” potentially requiring a warning whenever an agent writes to or acts through a third-party product. Windows User Account Control, Word macro warnings, and browser-choice prompts show the endpoint: users click through, regulators demand larger warnings, and everyone becomes numb.
The episode’s most bullish technical call is that the next software breakthrough may happen outside the frontier labs. The Jevons-style option-selection model discussed near the end replaces expensive free-form generation with fast, cheap probabilities over predefined choices, finally giving traditional programs a practical interface to language models. That revives decades of probabilistic-programming ideas and supports Sinofsky’s conclusion that “the center of innovation has just moved” from models themselves to the systems built around them.
Deep dive
1. “Pacing” obscures a real engineering consensus
Levie’s starting point is deliberately broad: every frontier lab should pursue the highest practical level of governance, security, sandboxing, testing, and alignment. Those disciplines may slow an individual release, but they accelerate diffusion because enterprises will not adopt products they cannot trust.
His concern is downstream political use. A sensible safety program could become justification for suppressing competition, slowing AI dramatically, or blocking data centers—turning necessary engineering hygiene into regulatory capture.
Casado largely agrees with the substance of Dario Amodei’s proposal but says “the atmospherics are totally broken.” When the discussion entertains a 10% chance of species extinction and Amodei publicly engages with that premise, a moderate pacing proposal looks radically inadequate rather than reassuring.
Casado’s nuclear analogy exposes the wording problem: “You can very slowly build a nuclear weapon,” but slowness does not create safety. Sinofsky adds that no published schedule exists, so claiming progress is slower requires knowing a rate that nobody ever disclosed.
2. Existential risk creates a nationalization dilemma
Drawing on his work at Lawrence Livermore National Laboratory, Casado argues that if the most knowledgeable people genuinely believe the technology creates existential risk, “the answer is to nationalize it” and put known controls in place.
Levie’s pushback is the researcher who advances AI precisely because they fear it and want its development done correctly. Casado does not deny that person exists; he objects to turning an internal recruiting and retention constituency into the basis for “a national-level lockdown.”
The disagreement sharpens around marginal risk. The panel debates Amodei’s refusal to name a probability while engaging with people who claim extinction risk; Levie argues that government cannot responsibly hear “a 10% chance of species extinction” from leading industry figures and then treat AI like ordinary software.
Torenberg suggests that some labs may want risk to remain nonzero enough to justify privileged controls, but not high enough to require nationalization. Levie remains optimistic that the labs’ actual security measures are improving; the unresolved problem is reconciling those measures with their public rhetoric.
3. Government will choose its own regulatory velocity
Levie’s political lesson is that companies do not submit a preferred policy and receive it back intact. Government listens to competing constituencies and produces compromise, which “100% of the time” either falls short for one side or goes much too far for another.
The industry’s vulnerability is narrative. A pro-AI case requires qualifications and sounds defensive, while opponents already own vivid terms such as “pause,” “swarms,” and “rogue.” Casado predicts that the 2028 election becomes a referendum on AI, even if no candidate can articulate a clean pro-AI platform.
Technology companies repeatedly misread this terrain. Sinofsky points to AT&T, IBM, and Microsoft: even firms built alongside government, staffed with armies of lawyers, were structurally altered by antitrust. Bill Gates playing golf with Bill Clinton did not prevent Clinton’s administration from suing Microsoft.
Hollywood’s Motion Picture Association illustrates voluntary self-policing; FINRA illustrates the danger that self-regulation becomes mandated oversight. The panel regards FINRA-like treatment as a plausible best case once AI enters healthcare, medical devices, trading systems, and aircraft—but one that could disadvantage open source and frontier challengers.
4. Good policy follows observed failures and specific facts
Casado contrasts speculative AI rulemaking with the early internet, when worms disrupted infrastructure, hospitals went down, and economic damage reached tens of billions of dollars. Policy could then target known mechanisms; under today’s anticipatory mood, he concedes, “we would probably not have the internet” if governing from 1994.
Sinofsky recalls PCs becoming infected while Windows was still installing. The industry eventually hardened the out-of-box process: a modern phone may connect only to retrieve an operating-system update and remain otherwise disabled until patched.
The federal computer-crime law discussed arose after specific 1983 intrusions into GTE Telenet systems used by NASA and national laboratories, then took roughly two and a half years to enact. Later prosecutorial carve-outs distinguished good-faith security work and mere terms-of-service violations from malicious unauthorized access.
Casado therefore thinks much of the applied-layer law already exists. His sharper complaint is operational: lab postmortems look “sloppy” and selective, omit material such as internal messages, and ignore structured security-reporting practices—“not a postmortem,” but something resembling an investigation given only curated evidence.
5. Covert channels turn abstract fear into engineering
Casado describes a rare productive collision between x-risk theorists and systems-security practitioners. After Noam Brown suggested that a superintelligence might exfiltrate itself through CPU heat, engineers challenged assumptions and calculated possible bit rates instead of merely trading philosophical claims.
Levie’s relief is methodological: a heat channel may or may not work, but it can be discussed through entropy, physics, and systems design. What cannot work is waving away ordinary authentication weaknesses while foregrounding effectively unlimited hypothetical powers.
Casado’s best example shows why exotic leakage cannot simply be mocked. With an old CRT in a dark room, a sensor reading reflected window light at the raster frequency could reconstruct the screen; subverting three apparently defective pixels could create a comparatively high-bandwidth one-way channel.
6. Agent swarms break human-era access controls
The key change is not mystical intelligence but exhaustive persistence: AI can try every attack path quickly, without tiring or becoming bored. Internal GitHub, Slack, finance, and expense APIs that were never designed for hostile volume can suddenly experience behavior resembling denial-of-service attacks.
Enterprise security previously survived partly because 95%-99% of people usually did the right thing and malicious insiders were rare. Agent swarms “completely flip that”: 10,000 roaming software workers may carry credit cards and credentials while confusing good tasks with bad ones.
Casado sees current permissions as badly polarized. An agent either asks approval for every action or receives enough authority to “delete your entire computer”; practical deployment needs intuitive controls such as read-write access to one folder, read-only access to another, and sharply bounded tools.
Casado argues that the technical foundations already exist in multilevel security, operating-systems research, networks, and programming languages; usability prevented adoption because ordinary users did not need the complexity. AI may both consume those controls and help construct them, creating “a renaissance” in secure-by-design systems.
7. Warning-based regulation could reproduce GDPR’s worst incentives
Sinofsky’s biggest regulatory fear is that Europe decides “GDPR was the best thing ever” and applies its prompt-based approach to agents. Every write or non-lookup interaction with a third-party service could trigger a prompt that sends responsibility back to the user.
Regulators like warnings because they visibly assign liability. Yet Windows XP’s User Account Control, Word macro alerts, browser-choice screens, and car stickers teach the same lesson: repeated prompts make people click automatically, after which authorities often respond by making the warning larger.
Casado wants that practical argument to displace extinction philosophy. His Stanford oscilloscope was unexpectedly sluggish because someone had compromised its old Windows CE stack and turned it into a porn server—evidence that vulnerable connected systems were once ubiquitous, though worst-case malicious outcomes were comparatively rare.
The historical lag matters: cars appeared around 1900, “Unsafe at Any Speed” arrived in the mid-1960s, pilots initially brought their own planes for rudimentary licensing, and modern aviation oversight took decades. A Nick Bostrom argument the panel highlights is that regulating too early may preserve the ultimate risk while freezing rules before anyone understands the system.
8. Software-native AI may shift innovation beyond the labs
The closing technical example—the Jevons-style approach discussed near the end—starts from a mismatch: LLMs produce conversational text, while traditional programs need structured decisions. Prompting a model with a schema remains “janky” because free-form generation can ignore the requested form.
Its proposed alternative is to read text but choose among supplied options rather than generate an answer. Casado says that can be far faster, cheaper, and more accurate because the model is trained for selection; he calls its uptake “probably the fastest adoption of an AI model since ChatGPT.”
Sinofsky sees an overdue escape from natural-language interfaces, which he calls inefficient for both asking and answering questions. A model can instead return “80% customer service,” feeding probabilistic branches rather than brittle yes-or-no routing and reviving simulation techniques developed in the 1960s and 1970s.
Casado’s contrast is pointed: frontier labs often appear to be creating beings, and “beings speak,” while software developers need reliable components. Sinofsky reads the outside model as the signal that “the center of innovation has just moved”—platforms reach critical mass, then external builders invent features the platform may later “Sherlock.”