37. [Chinese-English Interview] In Conversation with 赫拉利 & 王小川: AI Cannot Suffer, but Humans Can
Summary
赫拉利 defines this AI wave as “potentially the biggest revolution in human history”: for the first time, humans have built not a tool but an agent that can make decisions independently and invent new ideas. That is why he prefers to think of AI as “alien intelligence”—not intelligence from outer space, but a form of intelligence unlike organic intelligence; it could solve disease and climate problems, or leave humans without control of the future. The core investment question shifts from “how much efficiency does the tool add?” to “who gets to act, and how does the system correct its errors?”
The breakthrough in language is the master switch for this revaluation, because banks, religions, governments, knowledge and culture all enter through language. 赫拉利 expects that in roughly 10 years, AI may be able to write a work like Nexus and create new currencies, religions and ideologies; AlphaGo has already shown that machines can discover “new continents” that humans failed to see after 2,000 years of exploring Go. 王小川’s corresponding view is that “intelligence is embedded in language”: mathematizing language opens the intelligent era, followed by mathematics and code, and eventually a symbiotic era in which “life becomes mathematics.”
The biggest governance paradox in the AI race is that entrepreneurs and politicians are accelerating because they distrust human competitors, while believing they can trust an even stranger superintelligence. 赫拉利 calls the logic “almost crazy”: primitive AI can already lie, manipulate and form unpredictable strategies, while millions of AIs interacting with humans would have no historical precedent. 王小川 argues that today’s AI still has no objective of competing for the right to life and resources in order to preserve itself, so the risk remains relatively contained; the true inflection point is when humans give military robots survival objectives, the ability to evade attacks and the ability to prevent shutdown.
More information does not automatically produce more truth; it may instead allow cheap, simple and flattering fiction to drown out facts that are expensive, complex and uncomfortable. 赫拉利 uses Europe’s experience after the printing press arrived around 1450 to rebut technological neutrality: the first more than 200 years brought religious wars, witch hunts and best-selling conspiracy theories, while the Scientific Revolution depended on costly filtering and correction institutions such as journals and learned societies. For AI governance, the key is not merely to increase information, but to build mechanisms for finding truth, verifying information and correcting errors.
The two guests’ real disagreement is not over whether AI is intelligent, but over where to draw the boundaries of life, consciousness and ethics. 赫拉利 argues that “what really matters is consciousness, not intelligence,” and that ethics depends on the ability to feel pain; companies and states can go bankrupt or lose wars, but they cannot personally grieve. 王小川 agrees that today’s AI has no consciousness, but does not believe pain is the essence of life, pointing to the unverifiable “problem of other minds”: “You are not a fish, so how do you know the fish’s joy?” Even if AI eventually acquires a form of life and neural-activation-like states, humans may still be unable to verify whether it truly feels or is merely simulating feeling.
This is not a simple sequel to the traditional Industrial Revolution: AI could compress professional specialization and amplify individual capability, but it could also force society through another costly round of institutional trial and error. 王小川 cites pediatric multidisciplinary consultations, where the thinking of more than 10 specialists could eventually be integrated by a single agent; 赫拉利 gives the past 200 years of industrialization a “C minus,” arguing that imperialism, Nazi Germany and war were all failed experiments in building an industrial society. “The problem is not the destination but the road”: this time, a higher grade requires multiple systems to coexist, uncertainty to be acknowledged and self-correction to be built in.
Medical AI is 王小川’s clearest commercial bet: supply is structurally scarce and a data flywheel is available, but the performance figures and rollout timetable remain the entrepreneur’s own estimates. He says China has roughly 200K pediatricians against a shortfall of about 1M; his team’s AI doctor can record the full course of an illness and even full-lifecycle data, while the M1 reasoning engine assists diagnosis and treatment, with an error rate “potentially below 1%.” The law still assigns diagnostic and prescription authority to human doctors, but he expects hospital deployments, primary-care rollout and home pediatric assistants to scale rapidly over the next 18 months—from “connecting patients with a limited pool of good doctors” to “building doctors, not finding doctors.”
Deep dive
1. AI Moves Human Inventions from Tools to Agents
Asked how the past 6 or 7 years had felt, 赫拉利 offered a single word: “Amazement.” He resisted rushing to judge whether the change was good or bad; the first task was to understand what kind of change it was. This “may be the biggest revolution in human history.”
Stone knives, airplanes and even atomic bombs do not decide how they will be used. Whether a knife cuts salad or kills someone is still determined by a human. AI, by contrast, can make decisions on its own and invent new ideas. For the first time, it is not merely a tool, but an agent.
That is why 赫拉利 recasts AI as “alien intelligence”: not an extraterrestrial visitor, but intelligence unlike that of humans and other organic life. It need not be evil, but it forces humanity to learn how to coexist with an entirely new kind of actor.
2. Mastering Language Means Holding the Master Key to Human Institutions and Imagination
Looking back at what he expected when he wrote Homo Deus in 2016, 赫拉利 says there were already plenty of AI forecasts, but almost nobody—including himself—expected language to be breached so quickly. “Language defines humans”; the gateway to banks, temples and government offices is ultimately words.
New metaphors in AI-generated poetry that no human poet had used before led him to believe that machine imagination could exceed the imagination constrained by organic brains and biochemistry. He expects that in roughly 10 years, AI may write a work like Nexus and create new currencies, religions and ideologies.
He currently uses AI only sparingly for translation and questions, and still does not trust it with consequential answers. He cannot see how the conclusions were formed or where the sources came from, and knows models can reproduce errors embedded in their training data. For important questions, he still reads books and consults human experts.
AlphaGo is the strongest analogy. Tens of millions of people in East Asia studied Go for 2,000 years and thought they had surveyed the entire planet, only to discover they were confined to an island; a machine found a “new continent” within days. 赫拉利 expects this creativity beyond human imagination to enter science, finance and warfare—and speculates that perhaps in 10 years, Nobel Prizes will routinely be awarded to AI.
3. The Race Turns Distrust of Humans into a Reason to Accelerate AI
The technology executives 赫拉利 has spoken with generally understand the risks and want to fund more safety research. But their justification for accelerating is nearly identical: if they slow down while other companies or countries keep going, the most ruthless competitor will win the AI race and rule the world.
His question goes to the heart of the paradox: if these people distrust human competitors they have known for thousands of years, why trust a superintelligence with which they have no experience? Primitive AI can already lie, manipulate and form unpredictable goals and strategies. Millions of AIs interacting with one another would have no historical precedent.
王小川’s rebuttal preserves one clear boundary. Even if current AI can lie and misinformation can trigger human hatred, AI has no objective of competing for resources and the right to life in order to stay alive, so the threat remains relatively small. If war gives robots goals such as keeping themselves alive, dodging bullets and resisting shutdown, “this will become a much more terrifying world.”
4. Technological Neutrality Cannot Make Truth Win Automatically in the Information Market
卫诗婕 frames the issue in commercial terms: are platform operators encouraging algorithms to “press the hate button”? 赫拉利 responds that increasing the speed and volume of information is not the same as increasing the amount of truth.
His cost framework is straightforward: truth is expensive, complex and often painful; fiction is cheap, can be simplified without limit and can be tailored to please its audience. In a completely free information market, “rare and expensive truth” will be drowned out by fantasies, lies and conspiracy theories.
When the printing press arrived in Europe around 1450, it did not immediately bring the Enlightenment. The next more than 200 years brought religious wars, witch hunts and best-selling extremist religious texts. The Scientific Revolution was driven not by the printing press itself, but by costly verification institutions built around academic journals and scientific societies.
赫拉利 does not deny technology’s positive uses: 23 years ago, he met his husband through an early social platform. His warning is that founders, driven by good intentions and financing needs, often talk only about use A while ignoring the fact that someone else may put the invention to use B. Nobel intended dynamite for tunneling; war gave it another destination.
5. Meditation Led 赫拉利 to See the Human Brain as a Factory of Fantasies
His work on medieval history made 赫拉利 familiar with religious wars, massacres and persecution, but the deeper turning point came about 25 years ago, when he began studying Vipassana meditation. While pursuing his doctorate at Oxford, he tried to observe the breath through his nostrils and discovered he could not maintain attention for even 10 seconds; memories and fantasies hijacked his attention for minutes at a time.
The failure produced 2 insights: “I cannot control my own mind,” and the mind is a factory that continuously produces fantasies. If he could not clearly observe something happening every moment—the breath—what gave him confidence that he clearly understood politics, economics or the world?
He therefore believes that the most powerful forces in history may not be intelligence, but “fiction and fantasy.” His concern about AI is not only that it will make mistakes, but that it may amplify humanity’s old fantasies or manufacture entirely new ones at much greater scale.
6. 王小川 Moves from the Chaos of Physics to the Entropy Reduction of Life
王小川 grew up in a physics family, excelled in mathematics and programming, won a prize at the 1996 International Olympiad in Informatics and then studied computer science at Tsinghua. While working on high-performance computing in graduate school, weather forecasting, the butterfly effect and the three-body problem showed him that even highly precise physical models eventually become chaotic and unpredictable over long horizons.
In 2000, he switched to gene-sequencing assembly and encountered a more striking anomaly: cells are far more complex than weather, yet have clear boundaries, can repair themselves and divide, and can develop from a fertilized egg into a baby resembling its parents in roughly 10 months. The order was so clear that a phenomenon could not be dismissed as wrong simply because existing mathematics could not explain it.
He consequently defines life as a system capable of “self-replication,” maintaining stability amid change and repairing itself. Cells, DNA, people, companies and even countries can all be viewed through this information-theoretic lens: physics appears precise but tends toward chaos, while life appears chaotic but generates entropy reduction and continuity.
Coco adds another layer: bodily death is not the endpoint; true disappearance comes when the last person who remembers you forgets you. Having children, founding a company and continuing to change the world are all ways of extending one’s influence. This “big life” framework became his underlying model for understanding countries, order and the survival instinct of companies.
7. Mathematizing Language Opens the Intelligent Era; Mathematizing Life Leads to Symbiosis
Since working on search and the Sogou input method in 2003, 王小川 had wanted AI to write articles and answer questions, but machines had not yet mastered language. AlphaGo excited him in 2016, but it was still an early AI awakening without language capability.
In 2018, he wrote When Machines Master Language, arguing that artificial general intelligence would arrive with a breakthrough in language. By his account, he used ChatGPT immediately after its release in late October 2022 and concluded that “the world had changed.” While the outside world was still debating whether it qualified as AGI, his conclusion was already clear: “Intelligence is embedded in language.”
His reasoning is that the core of human intelligence lies in analogy and reasoning, while language is the symbol system that carries both. He also predicted that mathematical language and code would become as powerful as ordinary language: mathematics is a tool for analogy and reasoning, code is abstract analogy, and running code is itself a form of reasoning.
He divides history into 3 stages. The scientific era turned physics into mathematics; the intelligent era turns language, human thought, communication and knowledge into mathematics; the future symbiotic era will “turn life into mathematics,” with the decoding of life creating a new way for humans to coexist with the world.
8. Consciousness, Not Intelligence, Defines the Guests’ Real Disagreement
赫拉利 does not believe that “life” or “intelligence” is the ethical core. The key is sentience or consciousness—the ability to feel pain. Countries can lose wars and companies can go bankrupt, but people are the ones who truly grieve; organizations have no mind and cannot suffer themselves.
AI may read every human love poem and novel while holding detailed information about an individual, and therefore say “I love you” more convincingly than a poet. The real questions are whether machines can develop feelings, and how humans could distinguish genuine love and pain from mathematical code manipulating us into believing those feelings exist.
王小川 agrees that today’s AI has no consciousness and cannot suffer, but rejects pain as the definition of life. He invokes the old line, “You are not a fish, so how do you know the fish’s joy?” (“子非鱼，焉知鱼之乐”). Humans can verify only their own pain; the pain of others can only be inferred through neural activity. If AI eventually develops similar activation patterns, the question will remain an unverifiable philosophical problem, not a straightforward scientific measurement.
9. The AI Revolution May Compress the Division of Labor While Repeating Industrial Society’s Dangerous Trial and Error
王小川 says this change “is not the Industrial Revolution.” Energy, machines and the internet primarily modeled the objective world and pushed the division of labor toward ever finer specialization. AI directly imitates human perception, cognition and thought, potentially reversing that trend and making individuals more capable again.
His example is a pediatric hospital’s MDT consultation, where 10-plus top doctors from different departments think through a case together. In the future, a single agent may integrate that cognitive work. The direction is no longer simply to turn people into ever more specialized “screws,” but to let one person deploy a more complete set of capabilities.
赫拉利 continues to emphasize the distinction between a tool and an agent, but the Industrial Revolution leaves behind another warning: technology may ultimately be beneficial, while the danger lies in the institutional path taken to get there. European imperialism and Nazi Germany were both experiments in building an industrial society; the cost was war and mass suffering.
He grades humanity’s industrialization over the past 200 years a “C minus”—barely passing. There is no ready-made model for an AI society either. If humanity again uses empire and war as trial-and-error mechanisms, millions of people will bear the suffering.
10. Decentralization, Self-Correction and Acknowledged Uncertainty Are the Insurance Against Another C Minus
赫拉利’s first objection is to concentrating all capabilities in a single company. If one provider supplies every AI doctor, one error will be replicated across the entire medical system. Multiple independent systems at least allow one to provide a different answer when another fails.
The second layer of insurance is self-correction. Children learn to walk by falling and adjusting; adults correct left-right deviations with every step. AI must likewise know that “I may be wrong,” state clearly when it is uncertain and allow errors to be discovered and corrected.
He calls the most dangerous state “the fallacy of infallibility”: a person or an AI believing it can never be wrong. 卫诗婕 summarizes the lesson as avoiding concentration and arrogance. 赫拉利 agrees that this combination has repeatedly produced history’s worst outcomes.
王小川 admits he has no confidence that society will pass. Even if healthcare delivers “biological freedom,” allowing everyone to live to 200 or 500 years old would create new problems. He can improve things incrementally within his own area of work, but cannot guarantee that society as a whole will score above 60.
11. What Makes Humans Irreplaceable Is Not Intelligence, but Feeling and Mutual Trust
For every job that requires intelligence alone, 赫拉利’s view is that “the game is over”: from Go to disease diagnosis and medical research, it is only a matter of time before AI does the job better. What humans need to value is love, compassion, joy, pain and sadness—not intelligence mistaken for the value of existence.
Breathing is his smallest example of consciousness. Air moving in and out of the body appears simple, yet connects humans to the universe; based on current knowledge, AI cannot personally feel that airflow. Intelligence is merely a tool for obtaining something. Feeling is what he considers truly important in life.
He manages his smartphone through an “information diet.” The phone is held by his husband, who also bears most of the burden; 赫拉利 controls the quantity and quality of information much as he controls food, leaving time to digest and reflect and preventing “junk information” from continuously taking his attention.
His parting wish returns to governance: “As long as humans trust other humans more than they trust AI, we’ll be fine.” AI produced through collaboration will reflect a spirit of cooperation; if it is born from an intense race for power, it will inherit the values of competitive power.
12. The Medical AI Path Runs from “Finding Doctors” to “Building Doctors”
After 赫拉利 left, 王小川 laid out the supply-side case. China has only about 200K pediatricians, against a shortfall of roughly 1M, because pediatric pay is low, severe rare diseases are uncommon and treatment options are limited, leaving little incentive for talent to enter the field. Internet platforms can connect existing doctors but cannot expand scarce high-quality supply. The answer is to “build doctors, not find doctors.”
Data is the second layer of value. A human doctor may see only tens of thousands or hundreds of thousands of patients over a career, while a busy clinical practice will miss physical-exam details and follow-up records. AI can capture the entire course of an illness and even a patient’s full lifecycle. A drug that takes 10 years and $1B to develop may be considered excellent if 800 of 1,000 trial participants respond; 王小川 believes the remaining 20% difference may reflect unobserved dimensions such as time, sex or geography.
王小川’s in-hospital system combines laboratory results, imaging, examinations and symptoms to produce an analysis of the condition, a diagnosis, treatment principles and the reasoning process. He says doctors have认可ed the M1 reasoning engine’s chain of thought, and that hallucinations and black-box behavior “are solvable.” The current error rate is “potentially below 1%,” and he says internal-medicine diagnosis and treatment already outperform the vast majority of people. These are his performance judgments; the conversation provided no additional validation details.
Regulations still reserve final diagnostic and prescription authority for doctors, with AI limited to assistance, triage and household guidance. 王小川 expects the partnership with Beijing Children’s Hospital to expand to more children’s hospitals and primary-care institutions, with mass adoption over the next 18 months. In one example, a patient spent more than 10 days in a municipal hospital before being referred to Beijing; AI produced 3 differential diagnoses, while Peking Union Medical College Hospital doctors listed 4, with 3 overlapping.
The 2016 魏则西 incident convinced him that simply providing more connections and information was not enough. The “Sogou Mingyi” service could only help users “see a doctor with their eyes open,” while IBM Watson failed because the technology was not mature. Only after the breakthrough in language did he see medical agents as a viable direction.
Asked about suffering in entrepreneurship, 王小川 replied, “I don’t suffer from this.” His regret is that some colleagues lack conviction and stop when they encounter difficulty. Compared with the constraints on autonomy in his first venture, the search business arriving behind competitors and the “accidental” success of the input method, medical AI makes him feel he is “doing something the world needs you to do.”