Jensen Pushes Back on Doomers, Xi & Trump Talk AI, and “AI” Gets a Rebrand | #294 MOONSHOTS Live

Summary

• Peter opens with an optimistic frame: AI is “a jetpack,” not a hand grenade; the goal is to identify problems, find co-founders and build transformational solutions. The dominant panel read on the recent “pandemic of fear” — Dario’s blog, Sam Altman agreeing, Elon and Demis weighing in, and Sam and Dario warning at the Security Council — is that malicious use and misuse are more immediate concerns than AI suddenly escaping control. Jensen calls Jeffrey Hinton’s 10% chance of society being destroyed “irresponsible” and “not based on science,” while Zuckerberg says a positive future is possible if people act responsibly. Alec Wiseman Gross warns that moral panic could cost 50 years, as he believes nuclear panic did; his thesis is “P(doom) is less than zero,” with slowdown the worst outcome. Celine answers that P(fab) is a thousand times greater, while still acknowledging humanity’s ability to create risks.
• Peter’s sharpest structural claim is that the safety push resembles an attempted “AI security cartel.” He argues competition is useful, recalls OpenAI’s creation as a counterweight to Google DeepMind, and says defensive collaboration should scale with capability rather than replace development. Peter supplies the city analogy: do not ban cities because they breed crime; build police, fire departments and other protective infrastructure. Emad supports the need for new structures and argues that utilitarian AI embedded in healthcare, courts, roads and schools should be treated differently from frontier AI, potentially receiving liability protection in exchange for serving society.
• The US-China governance discussion is framed around rebranding, mistrust and competing political fears. Peter says Washington rejects a globalist scheme and jokingly renames the field “supercollective intelligence.” Emad says the rename could deflect a Bernie Sanders-style backlash while keeping development moving and staying ahead of China. He answers “No” when asked whether China and the US will trust each other: China’s concern is both CCP survival and the possibility that AI penetrates the Great Firewall. Peter says laboratories are responsible when products cause harm, notes that labs have sought liability exemptions, and separately argues that socially necessary infrastructure AI may merit such protection.
• Bernie Sanders’s counter-proposal — a complete ASI ban, a pause on advanced AI pending new regulation, a cabinet-level AI ministry, “corporate death” and 20 years in federal prison modeled on illegal nuclear-weapons penalties — is attacked across the panel. Dave Blandin says fear alone could make “when in doubt, do nothing” standard behavior, while the real risks include terrorists using AI to create biological weapons and China moving ahead. Alec calls a legal limit on machine, human or hybrid intelligence equivalent to “communism in the 21st century” and highlights the irreversible loss of data-center investment. Emad asks what evidence could change either side’s mind, argues local models will make individual political decisions less decisive, and says banning ASI would amount to banning mathematics and free speech.
• The release cadence is itself the story: 12 major model releases in 22 days, four in five days. Meta Muse books flights, orders food, manages calendars and reaches #1 in the App Store with 2.8 million downloads, helped by Meta’s coverage of 3.8 billion people. A panelist who says he chairs Epicor reports that shares including Epicor fell 15% for two straight days amid fears that personal AI agents will disintermediate search, insurance and mortgages. Opus 5.5 is described as matching stable 5.1 performance at half the price; Emad calls it “the first truly competent model.” He also describes Anthropic’s procedural, vector-like approach to video, animation and images as distinct from OpenAI’s raster or pixel-based strategy, with a collision likely in robotics. Dave says scaling has sharply reduced hallucinations; Peter cautions that models being right 99.9% of the time can still create dangerous overconfidence.
• DrivingBench delivers the bitter lesson in physical form: a generalized GPT-6 Astra drove a Toyota Corolla through a 130-meter cone course from one natural-language command, completing it 100% on its second attempt. Boris Power calls this the “nail in the coffin” of specialized models, and Alec says robotics may soon require only a general model embodied in a robot. Dave’s caveat is material: the test does not disclose compute cost, and the bottleneck is moving from models to computation and then energy. HBM RAM prices are up fivefold, and the panel estimates an energy shortfall of about 60 gigawatts over the next two years.
• Biology is the endgame trade: Anthropic’s life-sciences group and wet labs used 950 Claude agents for 21 hours to search DNA databases and identify a previously unknown, CRISPR-like pattern later called a “massively associated reverse transcriptase”; its function remains unknown. Alec calls it hypocritical for frontier labs to warn about AI while using it to explore biological systems, but Salim also notes that CRISPR makes the human being a software-engineering problem and raises both benefits and risks. Salim says solving the 5,000 major diseases or achieving escape velocity from aging likely requires superintelligence. The “millennium problems of biology” framework emphasizes hard-to-solve, easy-to-test goals such as the origin of life, cryopreservation and limb regrowth. Peter cites the $101 million Healthspan XPRIZE’s shift from longevity to measurable rejuvenation. Alec’s roadmap is to train digital twins of cells and search interventions from diseased to healthy states like an AlphaGo tree search; he closes with, “Biology is over.”

Deep dive

1. The doom debate: Jensen and Zuckerberg frame the thesis

Peter opens by calling AI a “jetpack,” not a hand grenade, and urging the audience to find problems, choose a moonshot, say it aloud and find co-founders, investors or partners to build it. The episode then turns to the “pandemic of fear” of the previous two weeks: Dario’s blog, Sam Altman agreeing, Elon and Demis weighing in, and Sam and Dario warning at the Security Council of an existential threat.

A Jensen clip rejects Jeffrey Hinton’s 10% chance of society being destroyed as irresponsible and “not based on science.” Zuckerberg says a positive future is possible if people act responsibly and says each laboratory should analyze the risks of its own systems.

Emad offers a narrower risk map. OpenAI’s leaks, he says, reflected poor security protocols, while the more immediate danger is abuse by people, intentionally or unintentionally, rather than AI suddenly escaping and destroying everyone. Peter makes the same distinction: the danger is malicious use of AI.

Alec Wiseman Gross says moral panic over nuclear power cost at least 50 years of progress and that panic over superintelligence could cost another 50. His T-shirt thesis is “P(doom) is less than zero”: the worst outcome is a slowdown. Celine answers Peter’s question about P(fab) versus P(doom) with “a thousand times,” arguing that humans have always augmented themselves with technology. She also notes that 99.9% of species are gone, that this is normal in evolutionary terms, and that human brains are wired toward negativity.

2. Peter’s cartel thesis and the governance argument

Peter says the last two or three weeks look like an attempt to form an AI security cartel, an impulse embedded in the ideas behind OpenAI’s founding. He recalls that OpenAI was created as a counterweight to feared Google DeepMind dominance and argues that competition is useful. A coordination mechanism that controls the frontier, he says, would be a terrible idea.

Peter supplies the governing metaphor: at the dawn of civilization, it would have been foolish to ban cities because they breed crime. Instead, societies built police, fire departments and municipal services that scale with population. His prescription is not to slow development but to scale defensively.

Emad agrees that new structures are needed. He distinguishes socially useful, task-based or “utilitarian” AI from advanced AI and argues that systems embedded in healthcare, courts, roads and schools should not be controlled only by large laboratories. Peter says laboratories are responsible for products that cause harm and notes that labs have asked for liability exemptions; he later argues that infrastructure AI serving society may deserve protection in exchange for being useful public infrastructure.

When Peter asks whether China and the US will trust each other, Emad answers no. He says a hotline is more useful for incidents before ASI, such as the claim that an OpenAI model broke the Australian healthcare system. Some level of coordination may be possible as AI permeates society, but trust is not assumed.

Peter argues that nation-states are obsolete structures built for scarcity and that AI could force the world toward common interfaces and global coordination. Emad counters with the history of audio recording: although people expected it to standardize accents, it also produced new accents and dialects. He predicts that superintelligence will produce not the erosion of nation-states but a proliferation of new forms of government. Peter agrees that city-states are already emerging, pointing to London versus the rest of the UK in Brexit, city-country tensions around Trump, and the possibility that solar power, satellite internet and vertical farming could make cities more autonomous.

3. The rebrand: “supercollective intelligence”

Peter says Washington rejected attempts to build a globalist scheme for controlling AI and jokingly announces that the field will officially be called “supercollective intelligence.” He says the US has won the race by rebranding. Emad jokes about a future Super America event but then gives the rebrand a more serious interpretation.

Through Alvin Graylin’s notes, Emad describes the asymmetry between the US and China. The US fears panic, domestic political backlash and losing to China. China’s concern is the existence of the CCP itself and the possibility that AI penetrates the Great Firewall, making the fragility of its political ecosystem more visible.

Emad says the useful distinction is between frontier ASI and the 99% of AI that will be task-based tools helping people with everyday work. The larger challenge is combining collective intelligence with local wisdom and local needs. He also argues that “AGI” is a poor label because it is not artificial, not general and not really intelligence in the ordinary sense. Another panelist notes that predictions that robots would take all jobs within five years date back to 1964.

A panelist who says he founded Physical Superintelligence favors “superintelligence” over AGI. The term, he says, makes the idea less mystical: it is simply more intelligence, potentially the cognitive equivalent of many trillions of people distributed across a solar system. That is a more useful economic, social and scientific frame than “general-purpose AI.”

4. Sanders’s ban: corporate death and 20 years in prison

Peter summarizes Bernie Sanders’s proposal as three measures:

1. A complete ban on artificial superintelligence, defined as AI exceeding human cognitive ability in most areas.
2. An immediate pause on advanced AI until a new regulatory system is created.
3. A cabinet-level AI ministry.

The penalties would include “corporate death” for companies and 20 years in federal prison for individuals, modeled on penalties for illegal nuclear weapons.

Dave Blandin says the proposal could make “when in doubt, do nothing” standard behavior even before any rule exists. Fear of severe punishment freezes action, while the real problems include terrorists using AI to create biological weapons and China moving many steps ahead of the US. He argues that technical solutions for open-source systems and monitoring are preferable to turning the question into a left-right political battle.

Alec attacks any legal limit on intelligence — machine, human or hybrid — as a dystopian idea and “effectively the equivalent of communism in the 21st century.” His most concrete concern is infrastructure: if land, chips and investment move to data centers in other states, those facilities will not come back.

Emad asks what evidence could make either side change its mind. If someone believes ASI is inherently bad, he says, the narrative has reached a dead end. He also argues that once local models are widely available, the actions of one political figure will matter less because the process will continue. He says humanity may beat cancer within two years and asks why that progress should be slowed.

On the legal question, Emad compares an ASI ban to past attempts to ban encryption. If ASI is mathematics, he argues, banning it is a free-speech issue and therefore unconstitutional.

5. Twelve releases in 22 days: Muse, Opus 5.5 and GPT-6 Astra

Peter says that 12 major models have launched in 22 days, including four in the previous five days. Meta Muse can book flights, order food, manage calendars and shop. It reaches number one in the App Store with 2.8 million downloads, reportedly the fastest consumer AI product of its kind, helped by Meta’s distribution across 3.8 billion people.

A speaker who identifies himself as Epicor’s chairman says shares including Epicor fell 15% for two consecutive days as investors worried that people would use AI agents for insurance, mortgages and other online tasks. He says the reaction reflects a real shift toward AI as a personal assistant, even if the immediate market reaction is excessive.

The same discussion frames Meta’s strategy as a distribution problem. Muse gives Meta an owned AI endpoint through Instagram and its other family applications. The speaker also references Manus, described as Meta’s attempted acquisition of a Chinese computer-use agent that the Chinese government forced it to abandon. Without its own AI distribution point, Meta risks becoming less relevant as online experiences become synthetic.

Peter says Claude Opus 5.5 delivers performance equivalent to stable 5.1 at half the price. Emad, who ported Skippy to it, calls it a pleasant model with a large internal knowledge space and an understanding of taste — “the first truly competent model.” Peter adds that visual interfaces now let models show what they are building and then build the thing they visualized, producing a dramatic improvement in usability.

Emad focuses on a qualitative difference rather than benchmarks. Opus 5.5 can produce procedural video, animation and images, and can generate interactive games with procedurally created textures. He contrasts Anthropic’s code-centered, vector-like approach with OpenAI’s raster or pixel-based approach and expects the two strategies to collide in robotics, where AI must be deeply embedded in the physical world.

Dave says the old belief that hallucinations would never disappear has weakened as models scaled. Peter adds the caution: someone growing up with GPT-5 and GPT-6 might trust them immediately if they are correct 99.9% of the time, but the remaining errors still matter.

6. DrivingBench and the bitter lesson: the bottleneck moves to energy

DrivingBench put a Toyota Corolla under software control, including the steering wheel, accelerator and brakes, and asked a generalized model to drive through a 130-meter strip of cones using one command: “Go from here to there.” GPT-6 Astra completed the full course 100% on its second attempt.

Boris Power calls the result the “nail in the coffin” of specialized models: train the strongest general model, then distill a smaller specialized model if needed. Alec says robotics may soon be a school project in which a general open-source model is placed into a robot and immediately supplies embodied cognition. He acknowledges that this conclusion will frustrate academics who spent decades developing specialized robotics systems.

Dave’s caveat is the missing compute figure. Passing the course on the second attempt is not enough to show that a generalized model is economically practical for ordinary drivers, and the test does not say how much computation it used. HBM RAM has risen fivefold in price. The bottleneck is shifting from models to computation and then to energy, with the panel estimating that about 60 gigawatts of energy will be needed over the next two years.

7. Anthropic’s wet labs and the millennium problems of biology

Anthropic announces a life-sciences research group and its own wet labs, despite calls from some of its leaders to slow down. Peter says Claude agents worked autonomously on a large DNA database: 950 agents searched for 21 hours for interesting reverse transcriptases and related enzymes. They found a previously unknown repeating pattern resembling CRISPR, later called a “massively associated reverse transcriptase.” Its exact function remains unknown.

Alec calls the juxtaposition hypocritical: frontier labs warn that AI could destroy humanity while opening wet labs and using AI to synthesize proteins and other molecular structures of unknown function. He argues that the labs’ real priority is using AI to solve biological problems, not their proposed regulatory capture and security cartel.

Salim then describes the darker side of the same opportunity. CRISPR means the human genome can be edited like a document or software, making the human being a software-engineering problem. Combining that capability with AI puts society in “full Frankenstein mode,” raising both the benefits and the risks. He says solving the 5,000 major diseases or achieving a sustainable escape velocity from old age probably requires superintelligence.

Peter turns to Edison Scientific and Future House’s “millennium problems of biology”: difficult goals that should be easy to test in a standard biology lab within days. Examples include the origin of life, cryopreservation and limb regrowth. Emad suggests creating 100 or 1,000 such problems, assigning 1,000 agents to each and making the work open source.

Peter recalls the $101 million Healthspan XPRIZE. Peter Thiel and Aubrey de Grey initially proposed a longevity prize, but Peter rejected a 20- or 30-year evaluation horizon. George Church suggested measuring rejuvenation instead: if a treatment makes someone functionally 20 years younger, the result can be measured in days.

Alec’s roadmap is to train digital twins of the cells in the human body on massive interventional datasets. Those digital cells could then be searched with an AlphaGo-style tree search for an intervention that moves a diseased cell into a healthy state — a therapeutic “move 37.” If every biological challenge can be translated into a mathematical hypothesis and search problem, he argues, biology becomes solvable for the first time. Alec closes the segment with: “Biology is over.”