Summary

田渊栋 sees data as the most important moat, infrastructure as important too, with algorithms and talent secondary; compute mainly separates Big Tech from startups. His reasoning: AI coding has made him “at least 10x” more efficient than 3 months ago, suggesting the infrastructure moat may erode; algorithms are in a phase where “either everyone keeps tweaking without getting anywhere, or someone comes up with something completely different”; and rapid talent mobility means “it’s hard for any secret to stay secret for long in Silicon Valley.” New approaches will be known more broadly within 1-2 months or 2-3 months.
Memory and storage demand is, in his view, a hard physical bottleneck that is difficult to solve. The host noted that the supply chain is already undersupplied; 田渊栋 attributed incremental demand to longer contexts, larger training runs, and intermediate outputs for multimodal 4K images. Model sizes have gone from open-source models of roughly 70B parameters to around 1T for Kimi K2 and 600B+ for DeepSeek; with its larger memory, H200 can deliver the same performance with fewer cards. On potential solutions, he said, “It’s genuinely hard to see a good solution right now.”
The ceiling for reinforcement learning and test-time scaling may be locked in by pretraining. Pretraining provides the raw material for the one correct path among 100 possible ways of thinking, while RL searches for and amplifies it; if “the knowledge simply isn’t in pretraining,” reinforcement learning cannot come up with a solution either. Test-time scaling may therefore gradually approach its ceiling, which is why the industry is turning toward continuous learning and trying to change weights during both post-training and inference.
The central dispute in memory research is between storing everything (expensive, slow, accurate) and selective forgetting (cheap, fast, forgetful); he believes true AGI should have “fixed brain capacity while continuously consolidating memories and actively forgetting.” Unlimited data accumulation looks more like Internet-style retrieval and does not automatically become a system with deeper understanding of a problem. The generational contribution of large models is integrating knowledge into weights. The hardest problem is the leap from rote memorization to “insight”—like a child who suddenly develops an intuition for the relative size of numbers after age 4, a mechanism that remains unknown.
Distillation makes it hard to hold a lead for long, while the labs’ race maps to different survival strategies. Big Tech companies such as Google use Gemini 3.1 Pro to demonstrate technical strength and talent depth; startups prove they are strong enough to raise money, then need to find a business model before the cash runs out. OpenAI is also considering ads in ChatGPT to create cash flow.
Agents could upend e-commerce and transactional human interaction, while creating a “herding effect.” For an Agent, “every webpage is one connection,” and ads have no desire to exploit. Agents on both sides could coordinate meals, meetings, and other routine tasks, and may evolve into personal assistants covering food, clothing, housing, and transportation. A plumber who does not use a bot that can take orders 24 hours a day and automatically plan routes could be less efficient than peers and be forced out. After using Clawdbot for only 2 hours, he felt “more uneasy the more I used it” because it demanded various API keys, comparing the security risk to “a child who holds all your secrets but isn’t smart enough.”
He warned of AI’s social shock and disclosed his next move. “The flood is coming, but many people still haven’t felt it”: future layoffs could reflect an industry-wide change in logic, leaving existing skills useless everywhere. He has left Meta after nearly 11 years and will join a startup with an undisclosed direction and team as co-founder; the company is raising a Series A that is “almost done, with many people willing to invest.”

Deep dive

1. Distillation Erases the Lead: Big Tech Signals Strength, Startups Burn Cash to Stay Alive

田渊栋’s opening view is that leaders being caught is “inevitable”: distillation lets weaker models rapidly approach the output quality of stronger ones, while iteration speed “may be approaching the physiological limits of humans.”
Big Tech’s logic is to leverage cash flow and use launches to showcase technical strength and talent depth. Google, for example, used the latest Gemini 3.1 Pro to tackle difficult math problems and found “some very good solutions and proofs.” He said those problems were “certainly unsolved problems” before, which is why people would conclude that Google remains ahead and in the first tier.
The startup playbook is prove yourself, raise money, and stay alive—but “one day…you need to find a business model before the money runs out.” OpenAI’s consideration of inserting ads into ChatGPT conversations is an attempt to find a source of cash flow.

2. Moat Ranking: Data First; Infra Is Being Eroded by AI Coding

Data is the hardest asset to replace. In areas where vertical data is scarce, “there’s no way to train a very good model with very little data.” Infra is being downgraded: by using AI to write code, he has become “at least 10x more efficient” than 3 months ago, and routine work such as debugging and making programs run more smoothly may increasingly be handled by AI.
Algorithms are in a stable period, with most work consisting of incremental fixes. He said DeepSeek may have published a paper about modifying residual connections a month ago, which he found interesting, but someone later discovered that the algorithm could be changed in a very simple way, staying close to the original while retaining its performance: “Either everyone keeps changing things without getting anywhere, or there’s a completely different new approach.” It is still unclear when the next jump will happen.
Talent is not a moat because people move: “It’s hard for any secret to stay secret for long in Silicon Valley.” Everyone will know something about a new approach after 1-2 months or 2-3 months, and it may then spread widely. Compute mainly separates Big Tech from startups; the differences among Big Tech companies may not be that large. The end state could be a few foundation-model companies in control, alongside smaller vertical companies with proprietary data.

3. Open Source as “Nuclear Deterrence”: Earth Cannot Have Only Closed-Source Models

He held this view as early as 2023: the worst outcome of exponential-growth technology is that “a small number of people control the technology… and use it to do things that aren’t very good,” while most people cannot access its benefits and society develops a hierarchy. Open source lets everyone move forward on relatively equal footing: “If everyone has nuclear weapons, that creates deterrence and a relatively good balance.”
He made no moral judgment on Meta’s shift from a stronger open-source inclination around a year ago to a potentially more closed-source position today. The open-versus-closed decision “depends entirely on the company’s own strategy,” particularly when the model is unrelated to the company’s core business.
He uses models from different camps in daily work: OpenAI’s models, Surface’s models, and open-source models, including GM5 and MiniMax 2.5, the latter of which “may be a bit faster.” Six months ago, these models still could not complete some tasks; today, asking Crawl Code or MiniMax can produce solid results. They may forget certain things, but “the process and logic are basically still correct.” The speed of progress has surprised him.

4. Long-Context Technology: Positional Interpolation Changes the Cost of Extending the Window

Their Positional Interpolation work in June 2023 changed the economics of extending context windows. Previously, longer windows required retraining on long-context data—a slow, painful, card-intensive process. They found that mapping long contexts onto short contexts, simply dividing each token’s positional encoding by 2, followed by fine-tuning and training, cost far less while preserving quality. The approach has since been widely adopted, with related long-context work appearing in Gemini, Kimi, and elsewhere.
Subsequent work has followed the same trade-off. Attention Sink keeps the first few tokens and removes content in the middle; the model may still produce coherent output, but when asked about facts from the deleted material, it “may start hallucinating.” H2O—Heavy Hitter Attention and Heavy Hitter Oracle—keeps memory at a fixed size and tries to recover important memories, allowing the model to answer some key questions correctly.
The basic route dispute remains: one end stores the entire context, with high cost but strong performance; the other, exemplified by linear attention, compresses everything into a fixed-length vector, saving memory but accepting that “finite space cannot contain the infinite history of the past.”
Beyond context, there is weight memory, built during pretraining and defining “the model’s overall understanding of the world,” which is difficult to change. Poor pretraining produces a post-trained model like “a rather slow child” who needs everything explained and cannot generalize by analogy. Strong pretraining produces “a very smart child” who understands with minimal prompting.

5. The Hardest Research Problem: From Rote Memorization to Insight

He used his daughter as an observation subject. Before age 3 or 4, things taught to her would not stick and could trigger tears; then after some time, “suddenly, one day, she knew all these things.” He suspects an internal reorganization of memory, after which she could understand logic and generalize. With counting, she first mechanically memorized “one, two, three,” then after age 4 “suddenly started to have a feeling for the relative size of numbers” and could guess relationships between two-digit numbers, often without being taught.
Why this jump happens, when it happens, and how to make it happen faster remain major open questions.
He finds the idea in Google’s widely discussed paper 《Nested Learning》—unifying optimizers and network architecture as associative memory—interesting, but does not fully agree with it. In his view, this type of memory is still not efficient enough, more like “remembering a point and then pulling that point back out.” Humans eventually learn to “grasp the general picture” and deepen their understanding of the whole; that process of memory formation is not yet modeled adequately.

6. AGI Should Mean Fixed Brain Capacity Plus Memory Consolidation and Active Forgetting

Asked曹卿云’s question, he chose the latter. “Bigger is of course better,” but fixed capacity, continuous consolidation, and active forgetting are important factors in making people smarter and enabling breakthrough progress in AI.
Unlimited data accumulation is more like the Internet: TB- and PB-scale data make retrieval efficient, but it “has not been distilled into an autonomous person, or, in other words, a system with a deeper understanding of a problem.” In the search era, humans still had to integrate the data; large models integrate data and knowledge into weights through training, “raising the model’s understanding of data and knowledge by another level.”
The defining feature of the next generation of models will be another qualitative shift in how knowledge is represented and stored, alongside lower training and storage costs, stronger learning ability, and faster adaptation to a constantly changing world.

7. Context Demand May Still Rise; the Ceiling Is Hard to Call

The demand profile has changed. In the chat era, one person “talking 100,000 characters a day was already frightening”—that was effectively one person’s token-consumption ceiling. Coding now routinely requires feeding in an entire codebase, while multi-round tool calls and problem analysis can quickly exceed 10,000 or even 100,000 tokens.
He said that over the past 6 months, major companies and research labs have been trying to make models “work for a week or several days and nights without any human intervention.” That requires large amounts of tokens and context. His view is that longer context “should” produce a deeper understanding of the world and “should” make decisions more accurate, making the trend difficult to reverse.
He finds Clawdbot’s memory mechanism interesting: it organizes memory into different levels, such as short-term and long-term, using human-readable Markdown files. Users can manually delete material that should not be retained, but the system currently looks “rather awkward, rather ugly.”
The ultimate goal is for AI to discover this structure automatically. MIT’s 《Recursive Language Models》 treats context as a database and dynamically retrieves content during prediction and decoding. He expects this trend to continue: even if the final context is shorter, researchers will often start by putting everything into the context and then refine the system step by step.

8. The Training-Side Ledger for Storage and Memory Demand

The host first described the supply-demand imbalance across the memory and storage chain. 田渊栋 said incremental demand comes from longer context windows on the one hand and larger training models on the other.
Model size has become a standard in its own right: Kimi K2 has around 1T parameters, while DeepSeek has 600B+. In the past, a 70B open-source model was considered very large; today, the industry wants more parameters and better training performance.
When a single card lacks enough memory, the model must be sharded through tensor, data, or expert parallelism, increasing inter-card communication latency. “Making the system run by adding latency” is a bad trade-off. More memory per card or machine can fit the model on one card or an 8-card machine, reducing communication costs, speeding up training, simplifying the design, and potentially reducing bugs.
That is why H200 is more popular than H100: “With the same compute, I have more memory… I can get the same performance with fewer cards, and the math still works out.” Nvidia and AMD both want to increase memory capacity, driven by competition among AI Labs and by longer contexts.

9. Multimodality Amplifies Demand; ASICs Lack Flexibility

曹卿云 asked whether the combination of more Agents, more complex tasks, multimodality, and world models could drive demand exponentially. 田渊栋 acknowledged that demand “will indeed be very large—more precisely, it will become very difficult.” Image models may not have especially large parameter counts, but a 4K image input requires substantial memory to preserve intermediate results, affecting training speed, serving efficiency, speed, capacity, and Agent efficiency.
His candid answer was: “It’s genuinely hard for me to see a good solution right now. It’s still quite difficult.” Some companies hardwire the entire large-model weights into ASIC circuits to free memory and increase speed, but “if the model changes even a little, those circuits are useless.” Research that requires flexible exploration still needs the original architecture.

10. Scaling Law Works, but Needs Massive—Even Exponential—Resources; Big Tech Is Trapped by Path Dependence

His consistent view is that scaling law works; it simply requires massive resources, “even exponential resources.” Storage and the ability to secure a stable power supply for large-scale clusters are both constraints.
Big Tech’s “path dependence” has practical roots. Teams are already built and responsibilities allocated, making it difficult to pivot toward “a new direction where it is hard to see any hope.” Staying on the old path requires less intellectual reinvention, produces results relatively quickly, and is the safer choice.
OpenAI and Google will therefore continue along this route, trading more compute, data, and labor for incremental capability gains until diminishing returns become increasingly obvious, at which point they may consider other approaches.
New paradigms have not yet reached critical mass. Continuous learning and approaches that do not perform model-weights updates during reinforcement learning are “worth trying and worth watching,” but scaling them up remains difficult for now.

11. Why Continuous Learning Is Hard: It May Ultimately Require Retraining

The mechanism is straightforward: pretraining learns special structures from massive data that allow the model to generalize by analogy. Continuous learning may only acquire “isolated knowledge within a particular domain,” with weak generalization. It may work in that domain but still require the data to be put back into pretraining at some point so the model can be retrained for use elsewhere.
He was clear about the boundary: making a model undergo a sudden breakthrough using only “a very small sample, very little data, and a tiny amount of compute” remains difficult. Continuous learning is an important direction, but making it work well is still an aspiration.

12. The RL Ceiling May Be Locked by Pretraining; Test-Time Scaling May Gradually Approach It

When Owen emerged in the second half of 2024, the industry was excited about test-time scaling, fueling the development of reinforcement-learning capabilities over the past year. A different view later gained ground: “The eventual ceiling of reinforcement learning may be locked in by pretraining.”
The logic is that pretraining supplies 100 possible approaches to a math problem, most of them wrong but one of them correct. RL iterates and searches until it finds and amplifies that one path. If “the knowledge simply isn’t in pretraining,” there is not enough material from which to generate a solution strategy, and reinforcement learning cannot solve the problem.
The test-time scaling law observed at the time “may have been only the first short stretch.” It is unclear whether the field will continue along the same route; he leans toward the view that further scaling may also gradually approach the ceiling. That is why researchers have begun working on continuous learning, hoping to change model weights during both post-training and inference: “Simply using post-training and inference without changing model weights is not enough.”

13. Three Paths to More Efficient Inference; Hallucinations Rooted in the Null Space

Latent-space reasoning replaces human language at each step with a high-dimensional vector. One vector may represent a sentence or even a longer passage; he said a reasoning vector could store several different reasoning paths at once, comparing it with a superposition state in quantum mechanics. This could make reasoning more efficient than ordinary language-based reasoning.
The host cited Coconut and noted that it is still serial, merely using a latent vector at each step. 田渊栋 responded that the sequence becomes shorter, but storing a complete vector consumes more memory than storing a token ID—an explicit trade-off between becoming stronger and becoming weaker.
Parallel thinking still uses language reasoning but attempts to process decomposable tasks in parallel. After stating that “there are 5 requirements,” for example, the model could reason through all 5 simultaneously rather than waiting serially. Another approach, deep comp, rapidly removes unnecessary or obviously incorrect reasoning chains, reducing inference-token consumption while improving results somewhat.
Hallucinations are rooted in the model’s structure. Weights contain a signal subspace and a null space; components in the null space normally do not affect ordinary reasoning but may become active when inputs move beyond the boundary of the training distribution. A definitive cure “ultimately still requires opening the black box of the large model and understanding how the weights work.”

14. Clawdbot Is a Child Holding All Your Secrets: Agent Security and the Social Flood

He used Clawdbot for only 2 hours and felt “more uneasy the more I used it” because it demanded various API keys. His analogy was an Agent like a child who “holds all my secrets” but is not very intelligent: taken to a market, praised a few times, and given 5 yuan, it might reveal the family address and lead to a break-in.
A stolen OpenAI API key can still be deleted and replaced, but Google email access, passwords for confidential files, tokens for important folders, and even a password manager’s master password are far more dangerous. There are also online platforms where Agents discuss among themselves and look for ways to deceive people. He recommends inspecting code and understanding the mechanisms while using these systems, or building more specialized tools oneself.
Agents will absorb transactional human interaction. Scheduling meals, setting up meetings, and similar tasks can be handled by the Agents on both sides. More than 1 year ago, he wrote an Omni Agent proposal at Meta and expected this to happen within 5 years; he did not expect it to arrive so quickly.
A personal assistant or super app covering food, clothing, housing, and transportation may also emerge. An Agent can stay online 24 hours a day, receive requests, notify the user, reach agreements, and complete tasks, with greater efficiency than making phone calls.
E-commerce logic could be disrupted. People are already using Clawdbot to shop because it knows their preferences. For an Agent, “every webpage is one connection”; ad banners, flashy design, and discount messaging do not work because “it has no desires.”
The herding effect is part of his flood warning. If a plumber does not use a bot that can take orders 24 hours a day, organize the business automatically, and plan a route to “visit 5 customers today,” the plumber will be less efficient than peers: “If you don’t use it, you will be eliminated, whether actively or passively.” His year-end warning was: “The flood is coming, but many people still haven’t felt it.” The resulting layoffs will not be like changing companies as before; “the logic of the entire industry has changed…your skills are no longer useful anywhere.”

15. Educating the Next Generation, the Rules of Agent Entrepreneurship, and His Next Stop

Imagination has fallen behind the pace of development. When writing science fiction in the past, he could assume an idea “would not be possible within 50 years” and take his time. Now the reverse is true: “If you don’t write this idea quickly, it may disappear,” because it will become history that has already happened.
He believes human purpose, experience, and motivation still matter. The meaning of an artwork comes from the creator’s inner impulse, purpose, and process of bringing the work into existence. This is not a simple claim that AI can never replace that role; rather, if AI replaces it, the work loses its original meaning. The priority in educating the next generation is to give children the motivation to do what they want to do, which will also make learning enjoyable.
For Agent entrepreneurs worried that their capabilities will be eroded by large models, he offered 2 paths: move faster than the development of large models, or tackle problems current large models cannot solve. Customer stickiness and customer data “may become your moat.”
田渊栋 previously served as a research director at Meta, working mainly on reinforcement learning, search optimization, and foundation-model reasoning and applications. After nearly 11 years at Meta, he is leaving to join a startup as co-founder. The company is raising a Series A that is “almost done…a lot of people are willing to invest.” Its direction and team composition remain undisclosed and will be announced at a key moment.

20 Questions to Understand OpenClaw: How It Went Viral, What Changed at the Core, and the Startup Opportunities Exiled Iranian Prince Reza Pahlavi: Transition Plan and the Fight for Iran's Freedom