Summary

The episode’s core judgment: pretraining has reached a cyclical peak—high-quality, diverse public data is largely exhausted, arbitrary synthetic data fed directly into pretraining causes models to collapse, and even Ilya has said so. The second half of AI belongs to post-training, which builds capabilities on top of existing knowledge. O1 was the signal, but OpenAI closed off its chain of thought; “R1 is the Llama of the second half of AI.”
Li Leding, former chief architect at Baidu, attributes R1’s win to open source and freedom to experiment. He quotes a professor: “Rather than China defeating the US, this was open source defeating closed source.” The technical core is pure RL—rewarding only the result and not intervening in the process—pushed to an unprecedented 8,000 steps: accuracy rose through 4,000 steps, declined after 6,000, and turned up again after 7,500. “Many researchers give up after 1,000 or 2,000 steps.” Only an organization without KPIs or commercialization pressure could survive that stretch.
The most tradable signal is cost collapse. V3’s full pretraining took just 2.6M GPU hours, or roughly $5.5M, using more than 2,000 cards; post-training consumed “not even a fraction” of pretraining compute. Fei-Fei Li’s team reproduced R1-level math ability on Qwen with 1,000 curated examples and 16 H100s for 26 minutes, at roughly $50. Li’s deliberately provocative conclusion: “Stop investing in compute”—the arms race is over, To B is shifting toward open source, and inference will generally use smaller models, so compute demand will fall for a period. “Vertical data and industry knowledge—that will always be the moat.”
The honest calibration of the gap is this: Dario’s article is broadly a “China conspiracy theory,” but its technical argument is sound. The six-month gap is real: “R1 only reaches O1’s floor, and OpenAI already has O3.” Liang Wenfeng’s framing is repeatedly cited: the China-US gap is less a matter of being a year or 18 months behind than “the more fundamental difference between originality and following.” On hardware, inference cards have already been validated by Huawei Cloud’s API, while training cards are “usable, but difficult”—and the best way to reproduce DeepSeek within 3 months is still the full Nvidia stack.
The names under pressure are Anthropic and Google, but Llama is taking the biggest hit. Its position as the leading open-source model is no longer secure, and Meta is reviewing its massive Llama investment. The US will likely build a DeepSeek counterpart of its own “to reassure itself.” But model commoditization is not necessarily a consequence of DeepSeek; it has been underway for the past year, with everyday models falling rapidly from 100B parameters to 10B and even a few billion.
The counterintuitive consensus at the application layer is that applications do not choose the strongest model. Small models plus RAG and prompt engineering are the real-world approach. “Product managers are the true soul of AI’s path into applications”—the flight-booking Agent still has not been built after 2 years. The hidden champion is Gemini 2.0 Flash: “cheap, fast, and good—on value for money, it may even beat DeepSeek.” Google’s PR and product entry points are simply a mess.
What DeepSeek should do next is not build a chatbot or commercialize itself. Productization would consume its attention the way it has consumed OpenAI’s. “China doesn’t need another app; China needs an OpenAI.” The next worthwhile projects are probing the step ceiling of R1 Zero, using pure RL to solve language mixing, incorporating V3’s MTP and new routing algorithm into V4/V5, and, most importantly, exploring architectures beyond the Transformer that could genuinely move toward AGI.

Deep dive

1. Closing out the first half: pretraining peaks, the second half belongs to post-training

Li’s framework: the first half added knowledge to models through pretraining; the second half gives models capabilities on top of existing knowledge. His evidence is the enormous leap from GPT-3.5 to GPT-4, followed by limited upward progress for nearly 2 years after GPT-4. Even Ilya has reached the same conclusion, and the industry broadly agrees that “pretraining has reached a cyclical peak.”
The data bottleneck is structural: high-quality public data with sufficient diversity is largely exhausted. “Adding 10x more knowledge in one area does not make AI 10x more capable”; what is needed is a new direction. And feeding arbitrary synthetic data directly into pretraining causes models to collapse, as multiple university papers have shown.
A professor raised an old architectural problem: the Transformer is too inefficient at learning from data. “We have trained it on trillions of tokens, yet it still struggles to develop intelligence like your household cat.”
O1 was supposed to signal the second half, but OpenAI “has already become Close AI”—charging for access while locking away the tokens that encode its reasoning. “R1’s release effectively revealed to the world how the most advanced technology is built. R1 is the Llama of the second half of AI.”

2. R1 Zero: reward the result, don’t intervene in the process

“This is the most brilliant paper I’ve seen in the past 2 years.” R1 actually trained 3 models, and the most valuable was R1 Zero: a model that used pure RL to discover multi-step reasoning on its own. Its reward model was simple enough to check only 2 things—whether the final answer was correct and whether the reasoning process was clearly separated from the answer. It gave no incentive whatsoever for intermediate reasoning. That alone brought performance close to O1.
Why can rewarding only the result work? In the humanities, “we have an idiom: taking a sentence out of context”—no single line in a continuous conversation represents the whole, making it impossible to score individual sentences precisely. In math, a problem can be solved through algebra, geometry, or analysis; “there are infinitely many reasoning paths, and we cannot enumerate them all.” The conclusion: “Machine thought is completely different from human thought. We cannot force machines to think the way humans do.”
Li is candid about the origin of the idea: OpenAI was indeed first, while DeepMind was the master. AlphaZero and AlphaFold both used the idea of “caring about the result.” “I even believe O1 very likely went through the entire R1 paper’s path,” but R1 was the first to explain it in an open-source release.

3. From Zero to R1: the SFT, RL, distillation, and rejection-sampling pipeline

R1 Zero had 2 problems that made it unsuitable for daily use: its reasoning was “not human language,” and it mixed languages. To the model, every language is just a token; “it chooses whichever word best expresses the direction of its current reasoning.”
The repair pipeline was tightly sequenced: first, SFT the V3 base model on a batch of CoT data to teach it a basic “answering playbook” and lock in language consistency; then apply RL so it could explore on its own; next, distill the intermediate model into question-answer pairs and use rejection sampling to remove answers that were “especially unlike human language,” producing 600K high-quality multi-step reasoning pairs; finally, add 200K traditional fine-tuning examples covering dialogue, text generation, and reading comprehension, for a total of 800K examples used to fine-tune the V3 base model again. “That gets you the O1 effect. The whole process is brilliant.”

4. The distillation debate: you can’t stand on your left foot to climb onto your right

On whether DeepSeek distilled OpenAI, Li’s view is clear: “I believe it did not. I don’t think it was necessary.” The only external data in the pipeline was the initial CoT seed. “Even if it used some of OpenAI’s work, so what? The actual capability came entirely from the RL that followed. Everyone in the world references other people’s work.” Without prior data, the team would simply have taken more wrong turns—trying humanities CoT first, finding it ineffective, and rewriting the math problems. “That’s all.”
Can distillation alone beat the SOTA? “Absolutely not.” Traditional distillation always moves from a large model to a small one; the small model can retain 80-90% of the large model’s ability, but it cannot surpass it. “That would be standing on your left foot to climb into the sky.” To surpass the frontier in reasoning, RL has to be introduced somewhere in the middle.
Rong Hui’s observation is worth preserving: O1’s paradigm shift inspired R1 and Kimi’s reasoning model K1.5. This form of knowledge transfer—standing on someone else’s shoulders and innovating further—is “a slightly moving example of technological idealism.”

5. Dismantling Dario’s manifesto: conspiracy theory, sound technical points, admitted gap

Li breaks Anthropic CEO Dario’s viral article into 3 layers. Overall, it is “purely a China conspiracy theory,” attempting to invoke the Chinese threat to obscure the fact that Anthropic and OpenAI have already been caught by China. But the purely technical discussion is “very sound and worth adopting.” One point is simply factual: DeepSeek still trails by 6 months. “R1’s performance only reaches O1’s floor, and OpenAI already has O3. What he did not say is that their own progress over those 6 months has also been limited. The 2 leaders at the front have slowed down considerably; we are catching up.”
Li refuses to declare whether pure RL or Anthropic’s RLHF is the correct path, citing Liang Wenfeng: today’s LLM research is more like a “research phase”—propose a hypothesis and test it experimentally; before the experiment produces a conclusion, you cannot say the hypothesis is definitely right or wrong. The alternative idea that capability is a continuous spectrum is “also a very, very powerful idea.”
Hardware reality splits in 2. Inference cards have already been proven—the SiliconFlow R1 API runs on Huawei Cloud inference cards. Training cards are “usable, but indeed somewhat difficult”: reproducing DeepSeek in 3 months still favors the full Nvidia stack of GPUs, NVLink, InfiniBand, and CUDA. Adapting domestic cards to a fixed model takes roughly 3 months, while exploring custom operators takes longer. But Li’s conclusion is categorical: “The US no longer has any ability to stop China from developing large models.”

6. Open source beats closed source: where the persistence through 8,000 steps came from

Asked to rank open source, technology, low cost, and product experience, Li chooses open source, quoting a professor: “Rather than China defeating the US, this was open source defeating closed source.” Looking backward from DeepSeek Math through V2, V3, and R1, frontline researchers had sensed the direction for years. “What likely held them back was not having enough conditions to run experiments.” The theory of generative AI is still incomplete; unlike computer science, it has no notion of Turing completeness. “The right methods have to be discovered experimentally.”
2 conditions created that freedom to experiment. V3 pushed training cost down to $5.5M using more than 2,000 cards, allowing more experiments with the same resource envelope. And Liang Wenfeng’s original intent was “not to commercialize, but to do pure research”: “I can accept a model failing, and I can even accept poor performance for a period of time.”
The 8,000-step run made that organizational capacity visible. Accuracy rose in a straight line through the first 4,000 steps, decayed from 4,000 to 6,000, and even fell after 6,000 before turning up again around 7,500. “Many RL studies see gains decay after perhaps 1,000 steps and flatten after 2,000. At that point, people are unwilling to spend resources on the next 4,000 or 5,000 steps.” Behind this success were “a great many failures we know nothing about.”

7. GRPO: focus more on new knowledge, less on old knowledge

The RL substrate was 8,000 math problems with standard answers, sourced from DeepSeek Math. GRPO addresses a basic problem: among the questions answered correctly in each round, many duplicate those from the prior round, and repeated training leads to overfitting. “We want it to pay more attention to newly discovered strategies, rather than repeatedly memorizing skills it has already learned.”
The method: generate 8 answers for each problem, assign 1 point for correct and 0 for wrong, then convert the results into statistical Z-scores. If most answers are correct, the problem type is already mastered and the Z-score is low; if only a few are correct, the model has discovered a new strategy and the Z-score is high. The Z-score happens to fall between 0 and 1 and can be used directly as a sampling probability: “focus more on new knowledge, less on old knowledge,” repeating the process as the model keeps selecting the best strategies. The added benefit is that rewarding only the result gives GRPO a major advantage over PPO and DPO in compute and memory consumption.

8. Innovation forced by H800

Li broadly agrees with Lex Freeman’s 3 low-cost drivers—Mixture-of-Experts, MLA, and low-level communication optimization without the NCCL library—but adds one more: FP8 low-precision training. “That is a very important piece; these were the things that played the main role.” His overall assessment is qualified: Americans are generally reliable when analyzing the technology, but their views should be taken for reference only.
Li’s independent inference is that DeepSeek had H800s. The H800’s main cut relative to the H100 was NVLink bandwidth, “forcing it to make this optimization—it had no choice.” The H20’s communication and compute gap is much larger, so running DeepSeek’s current optimization approach on H20s might not produce the same result. Rong Hui’s takeaway: “The place where he was blocked became his innovation.”
The irony of FP8 is that it was one of the H100’s most important selling points. “The Americans had so many H100s and did not try this. It really has the flavor of a resource curse.”

9. $5.5M and $50: post-training is astonishingly cheap

V3 pretraining consumed 2.6M GPU hours, or roughly $5.5M, while the paper explicitly says post-training used “not even a fraction” of pretraining compute. Li takes issue with the common claim that once pretraining’s scaling law ends, a post-training scaling law will take over and compute will continue to expand without limit. “Post-training does scale, but its scale is several orders of magnitude smaller than pretraining.”
The third experiment in the R1 paper was so compelling that it “turned the main course into dessert.” Fine-tuning Qwen and Llama small models, 7B and 30B, on the 800K valuable distilled examples was enough to reach O1-level math capability. But small models cannot run R1 Zero-style pure RL directly: “Small models simply do not have enough knowledge. It is like trying to cultivate yourself—you need some knowledge to begin with; without it, you are prone to going off the rails.”
Fei-Fei Li’s S1 pushed the idea to its limit. It distilled 59K examples from Gemini 2.0 Flash Thinking, then selected 1,000 science problems—covering everything from linear algebra and calculus to real analysis and quantum physics—based on 3 criteria: sufficient difficulty, non-duplication, and broad coverage. Training on 16 H100s for 26 minutes cost roughly $50. “The $50 figure is somewhat headline-friendly; the real number is certainly higher. But it means that if you only want to acquire a vertical capability, the cost can be extremely low.”

10. A paper with ten papers’ worth of substance is itself an open-source distribution tactic

Asked how he measures a brilliant paper, Li’s standard is simple: “It has enough substance. The hard content in this one may equal 10 ordinary papers or more.” Training 3 R1 models could easily have produced 3 papers, while the V3 optimization techniques alone would have been enough for an ordinary researcher to publish 5 or 6. He is candid about the backdrop: “Over the past 2 years, I have become increasingly reluctant to read papers, because most papers simply do not contain much useful content.”
Rong Hui identifies the mechanism: publish 1 paper, and the community helps it complete countless rounds of secondary distribution. “Yes, that is the community spirit.”

11. The national-destiny thesis is too grand; the originality-following gap is real

Rong Hui relays the excited private remarks of an investment heavyweight: DeepSeek is “China’s national destiny,” avoiding deflation and a Japanese-style lost 30 years. Li refuses the grand narrative: “That is too big a question for me to answer.” He confirms only 1 point: DeepSeek proved that Chinese researchers and AI practitioners possess knowledge and capabilities “that are in no way inferior to America’s,” and can produce frontier technology under near-total independence.
He then pours cold water on the idea that China has suddenly overtaken the US. Liang Wenfeng’s second point is quoted in full: China and the US may differ by only 1 year or even 6 months in AI, “but rather than a difference in time, it is more fundamentally the difference between originality and following.” DeepMind industrialized RL first; OpenAI first used scaling to improve capability. “Almost all of AI’s original ideas still come from the US. If we cannot move from follower to originator, the gap may remain at 1 or 2 years forever.”
Rong Hui’s analogy is worth keeping: “Today’s DeepSeek is more like OpenAI in the Ilya era, not OpenAI in the Sam era.” The latter became too absorbed in scaling and consumed too much attention on productization. “What everyone wants most from OpenAI is still for it to build GPT-5.”

12. Why big tech can’t do it: KPIs don’t tolerate failure

Li’s subjective judgment is blunt: the big tech companies have far more resources, talent, and data, but “if this version of the model cannot improve performance by 3% or 5%, I may not be able to keep my job.” No one is willing to try something that currently looks ineffective; they can only copy and scale what came before.
The DeepSeek team had no KPIs and took a pure-researcher posture: “We do not assume from the start that any one path must be correct. If it looks promising, we allocate resources; if it fails, it fails. We all learned about Edison trying so many times to invent the light bulb—without failure, where would success come from?”

13. Product managers are the soul of AI deployment

First the cold shower, then the praise: “Model development and the real needs of AI applications are not aligned.” Applications need capabilities that are stable, reproducible, and scalable across fields—not mathematical performance that is excellent but cannot be 100% correct. In practice, they use small or even very small models, leaving the logic to prompt engineering, RAG, and external programs, “letting AI handle only work similar to traditional NLP.”
The flight-booking example still holds after 2 years: “You cannot imagine that after 2 years, a product like this still has not been built. Even with O1—and Americans have had it for 6 months—it still has not been built.” Booking a flight is not a one-shot transaction: “I need to keep looking, keep thinking, and comparing it against my itinerary.” Designing that workflow is a product manager’s job. “Product managers are the true soul of AI’s path into applications.”
RAG’s real-world difficulties prove the point. The ideal architecture is “RAG provides knowledge, LLM provides capability.” In reality, the boundary cannot be controlled precisely: the model’s own knowledge gets coupled in and produces hallucinations, or the RAG data suppresses the LLM’s native capabilities. “You cannot retrain the model; you can only add further mechanisms outside the RAG system.”

14. To C: models won’t drown out products; the pain point hasn’t even been defined

The biggest To C benefit is that costs have “actually fallen by an order of magnitude.” The savings can fund human planning and design, while basic NLP and planning workflows that were previously too expensive to run with AI become viable. “That will bring a phase of acceleration.”
But Li rejects the popular idea that models will become products and drown out many applications. “Whether it is OpenAI’s app, Gemini, or domestic apps, user time is not especially long.” This largely shows that the industry has not accurately identified users’ pain points, and lacks enough strong product managers to define what AI is best suited to solve. Once the market operates more fully, it will not remain this way—“unless AGI.”

15. To B upheaval: when open source is the industry’s best, there’s no reason not to use it

The facts came first: Azure integrated DeepSeek almost immediately after launch, AWS followed quickly, and “today nearly every top-tier cloud has followed.” Li’s reasoning is that once the technical frontier stops advancing rapidly, commercial companies’ survival crisis disappears. There will be no more case where someone else’s model improves dramatically in 6 months and destroys your business. Model research will shift from the arms race to build GPT-4-level models toward exploring what comes after the Transformer: “When will it be built? There is no rush.”
On the product side, companies can simply use open-source DeepSeek, just as they use Linux, MySQL, and Redis. “When open source is genuinely the best in the industry and there is no longer a survival crisis, there is no reason not to use it.” The impact on China’s To B market will be profound. Given the US political environment and hostility toward China, “they will very likely build something similar themselves.”
To Developer will be affected the least. Coding is precisely where Claude 3.5 Sonnet—officially still “a class of its own” according to Cursor—along with O3 and R1 are strongest, and it is still advancing rapidly. “The rankings will keep changing,” and developers will ultimately benefit from that competition. Rong Hui adds: “In the first 30-plus years of my life, I have never felt technology advancing so intensely right in front of my eyes. After the first iPhone, we did not get a ‘microphone’ every 3 months.”

16. The application layer’s hidden champion: Gemini 2.0 Flash is even better value than DeepSeek

Will applications advance sharply because of R1? “I have my doubts.” O1 had been out for months, yet no app said that problems it previously could not solve had suddenly become solvable. The application industry had already taken a different path from LLM research: small models, RAG, and prompt engineering.
The US has an interesting split. Ask people about the Gemini app and most say, “The model is a bit dumb.” Ask them about the Gemini API and everyone says it is good. Gemini 2.0 Flash is “cheap, fast, and good—on value for money, it may even beat DeepSeek,” supported by TPU economics. Gemini 2.0 Flash Thinking is, objectively, no worse than R1; Fei-Fei Li’s distilled data came from it. Flash Light offers another leg of cost reduction. “Google sees this very clearly: lowering AI costs is absolutely an important way to bring AI into existing applications at greater scale.” Google is the world’s largest AI application player, and search is its largest AI application.
The failure is in PR and product design. Flash Thinking is buried inside Google AI Studio, whose interface “really is not for ordinary users.” Rong Hui tried it and found it “very impressive,” but the entry point was too deep; after a few attempts, “I could not be bothered to use it anymore.”

17. Chatbot: China doesn’t need another app; China needs an OpenAI

The facts: DeepSeek reached No. 1 in the App Store in dozens of countries and appeared to be “one of the fastest applications in history to go from zero to 30-40M daily active users.” But the team did not have enough cards, traffic reportedly affected training, and application stability was not made the top priority; the team still wanted to reserve cards for training.
Is a chatbot important to DeepSeek? “Only Liang Wenfeng can decide that. But emotionally, I do not want him to build a chatbot.” Once it does, it will absorb extensive productization work, daily-use issues, and maintenance, consuming huge amounts of attention “just like the problems OpenAI faces today.” The sharpest line in the episode: “China doesn’t need another app; China needs an OpenAI.” Build a simple version so people can try it, then keep working on original technology.
Rong Hui’s footnote echoes Liang Wenfeng’s public comments last year: do not commercialize, “because any attempt at commercialization would dilute and disperse their attention and investment in research.”

18. The compute provocation and the data moat

The question keeping Nvidia shareholders on edge gets Li’s long-standing provocation: “Stop investing in compute.” Investing in compute is effectively continuing to bet on the pretraining scaling law, whose ceiling is now visible. After DeepSeek, everyone will work on lowering costs; inference will use small models, so compute demand “will definitely decline for a period, until AI applications clear the productization hurdle and then rise again.” The industry should be scaling algorithms instead: “A model has learned trillions of tokens and still cannot understand why 3 plus 5 equals 8. That is not right.”
His answer on data is more measured: “Data is more realistic than compute and remains very important.” Search, recommendation, and short video have all demonstrated the value of owning the data distribution for quality, because they are ultimately statistical models. But there is no need to cross an excessively high threshold and keep piling on trillions of tokens. The permanent moat is clear: “Vertical data, vertical knowledge, and industry understanding—that will always be the moat.”

19. Next steps: DeepSeek probes the limits, Big Tech reviews, commoditization is already underway

Li’s to-do list for DeepSeek: continue probing “the ceiling of R1 Zero”—it has already run 8,000 steps, and 10,000 or 20,000 steps will probably improve performance, but the point at which marginal returns diminish is worth exploring; use pure RL to solve “not speaking like a human” and language mixing—“one round of distillation is not especially elegant; when we do technology, we should still pursue some elegance”; add V3’s MTP and new routing algorithm, whose gains are not yet obvious, to V4 and V5; and pursue the direction he is most excited about, also mentioned in the paper: new models beyond the Transformer, echoing a researcher’s route “through concepts, through world knowledge.” “We need to surpass OpenAI and DeepMind at the level of big ideas and think about how to genuinely move all the way toward AGI.”
ByteDance, Tencent, and Alibaba have a clear playbook. With no immediate survival threat, and with open-source technology available at dramatically lower cost, “their strategy will certainly be to review.” “Using open source to improve, adding some first-party R&D, and exploring for the future—that is the normal way big tech works,” rather than pouring money into a technology whose direction is not yet settled.
In the US, “Americans have certainly been scared,” but every top researcher knows the direction. “The next breakthrough is still more likely to come from the US; let’s view that objectively.” Li understands that Meta is also reviewing its massive Llama investment. The US will likely produce a DeepSeek-like open-source product “to reassure itself.” The hardest-hit names are Anthropic and Google, with Llama probably taking the biggest hit: “its position as the leading open-source model is no longer secure.” As for model commoditization, “it is real, but it may not necessarily have been caused by DeepSeek—even without DeepSeek, lower-cost and commoditized models have been developing throughout the past year,” with everyday models falling rapidly from 100B to 10B and even a few hundred million or a few billion parameters. The final perspective: “Today, China and the US are the only 2 countries in the world with the most advanced technology. Rather than spending time on conspiracy theories, they should work harder themselves.”
Asked about Peter Thiel’s claim that true innovation is highly coordinated innovation across multiple layers, Li offers a more candid version: “Many innovations were not DeepSeek’s original inventions. The citations show that researchers proposed them first. DeepSeek was the first to apply them at scale in an open-source system.” The most important layer is not technical. It is “the ability to uphold the basic principles of science”: “Science and innovation have no fixed direction and are unpredictable. Among all these directions, seriously experimenting, trying, and doing the work—that is what matters most.”

Jonathan Ross, Founder & CEO @ Groq: NVIDIA vs Groq - The Future of Training vs Inference | E1260 An Application-Free Future? Speaking Directly to Data with illumex CEO Inna Tokarev Sela