Chen Tianqi on Long-Termism, XGBoost, MXNet, and TVM
Summary
XGBoost, MXNet, TVM, and MLC LLM were not four separate attempts to chase the next hot trend, but one technical throughline in Chen Tianqi’s sustained effort to remove the bottlenecks to machine learning at scale. He is a problem-driven rather than method-driven researcher: tree models, deep learning frameworks, compilers, and even hardware instruction sets are all means to the same end—“letting fewer people build and deploy AI with fewer engineering resources.” For infrastructure observers, the implication is that value may not settle in any single model generation, but in the bring-up, optimization, and deployment friction that recurs across models and hardware.
XGBoost shows how a small team can build a durable moat through an exceptional single-point product, tight algorithm–systems integration, and steadily expanding community trust. At launch, its goal was to be the world’s fastest Gradient Boosting implementation, with missing-value handling built directly into the model; before recording, He Tairan saw that the paper’s citations were approaching 70,000. Chen attributes the success to “doing one thing extremely well,” focusing on a single algorithm, and building with the community—not to having forecast the market size in advance.
MXNet’s exit shows that leading performance and backing from large companies cannot substitute for user experience and community momentum. MXNet delivered excellent multi-GPU performance for years, with Amazon and NVIDIA both heavily involved, but the team initially tried to maximize performance and usability at the same time; Chen later concluded that “we should actually have chosen user experience first,” allowing PyTorch to capture stronger developer mindshare. The deeper lesson was that the engineering cost of manually chasing every GPU generation was unsustainable, directly giving rise to TVM.
Model convergence will not automatically eliminate machine learning compilation; hardware specialization instead makes model–chip co-design tighter and more frequent. Chen acknowledges that a small number of models accounting for most inference could compress the space for general-purpose compilation, but every chip generation brings new programming methods, model variants, and specialized operators that require fresh adaptation; “there is no single recipe that gets you the next 2x.” MLC LLM is therefore betting on unified inference across cloud and edge, while edge adoption ultimately depends on whether local models become “good enough” for specialized use cases, after which cost, privacy, and latency can amplify demand.
The gap between universities and frontier labs with 10,000 or 100,000 GPUs is unavoidable, but 100-GPU-scale resources, the latest hardware, and open-source collaboration are still enough to crack high-leverage vertical bottlenecks. Chen does not choose topics by paper-level return on compute; he looks for the module where the system is genuinely stuck. XGrammar, a structured-JSON generation project that has become one of the de facto standards in the open-source ecosystem, is an example of “making one small module the best it can be.” “Resource constraint is also an opportunity to make people more creative.”
OctoML demonstrates that open-source technology does not convert into commercial revenue in a straight line: open source is the starting point, and the final product will almost inevitably differ from the original idea. The company initially sold cross-hardware model optimization and deployment, then shifted about two years ago to offering endpoints and APIs for models such as Llama; He Tairan noted that it was ultimately acquired by NVIDIA at the end of 2024. Chen’s founder retrospective does not shy away from the importance of PMF, leadership, and coordination, but he still insists: “If we kept optimizing ResNet, we would lose terribly in the end.” The stack must be willing to reinvent itself.
The most important form of long-termism in this episode is not sticking to the original path, but remaining willing to choose a new problem, rewrite the system, and start again after failure. From spending 2 years pursuing the wrong method on the right ImageNet problem to investing 11 months late in his PhD to build TVM’s first working version, Chen’s risk tolerance comes from the belief that “failure is not that scary.” The more resources, reputation, and responsibility one carries, the harder it becomes to preserve one’s original intent; his reminder to himself is: “Courage may be a promise your past self made to you.”
Deep dive
1. The Internet Gave County-School Students Their First Proof That Scarce Resources Did Not Mean No Ticket In
Chen Tianqi grew up in a small county in Zhejiang. His mother taught at Songyang No. 2 High School, so he stayed locally for school. Around 2003, drawn by webpages, the Internet, and the idea of “creating my own games,” he went looking for his computer teacher, Mr. He, and received an introductory C book instead. The webpage and game never materialized, but programming became his long-term direction.
His county school had no informatics competition coach, so he taught himself through Online Judge and the “Big Banyan Tree” forum. He failed the preliminary round in his second year of high school, later won second prize in a provincial competition, and did not win first prize until his final year. He says the Internet allowed someone from a “grass-roots background” to see the best competitors, enter a community, and receive immediate feedback.
He Tairan asked why most classmates who spent time on computers ended up playing games. Chen did not romanticize his childhood: “I liked playing video games too. I’ve actually been gaming all along.” The difference was that he also wanted to create things, and programming channeled that creative impulse into deeper systems problems.
2. A One-Summer Pascal-to-C Tool Built the Foundational Confidence Behind His Willingness to Take on Hard Problems
The dominant competition language at the time was Pascal. Chen, having learned only C through an unconventional path, tried during his second year of high school to write a mini transpiler from Pascal to C. There was no GitHub or complete tutorial. He developed first on Windows and then Linux, sketching module and interaction diagrams in a small notebook.
The project consumed an entire summer and eventually produced a usable version. It was not a complete compiler, but it was the first system he had built independently from architecture through implementation. The most important output was not the code, but the experience that “I could try things myself, and if I failed, it wouldn’t really matter.”
He repeatedly drew on that memory later, whether building a CPU, designing an NPU, or rewriting a compiler. His summary of its influence is understated: “For everything that came later, I wasn’t afraid to take things on. I think that experience had a major impact on me in that respect.”
3. Shanghai Jiao Tong’s ACM Class Trained Expression, Systems Thinking, and the Rule to Stay Humble and Act Boldly
In 2006, Chen entered Shanghai Jiao Tong University’s School of Electronic Information as a Gaokao student and then applied to the ACM class. Without a prominent NOI résumé, he prepared his application materials with extreme care. Yu Yong later told him it was the most carefully prepared student application he had seen. Looking back, Chen says it felt like writing a PhD personal statement in advance.
The ACM class’s “Student Forum” required each student to choose an arbitrary topic and present it every year, every semester. After starting his PhD, Chen volunteered to give the first group-meeting presentation and realized that his undergraduate training had provided more than technical skills: it gave him the confidence to step forward and express himself directly.
The compiler-construction capstone was adapted from North American course designs by the first cohort, then continuously iterated and passed down by successive classes of teaching assistants. Students had to build a compiler from scratch. Chen considers it first-rate training even by today’s North American university standards. He later went further and tried to run a self-written compiler on a self-written CPU.
Yu emphasized “低调做人，高调做事”—stay humble as a person and act boldly in your work. Chen draws a distinction between the two: work requires edge, ambition, and a willingness to challenge the status quo; toward other people, one should remain humble, because the factors behind success “almost never all come from you,” and collaboration requires respect.
4. The First Barrier in Research Is Not Solving Problems, but Knowing Which Problems Deserve Solving
When Chen joined the APEX Lab at Shanghai Jiao Tong, his motivation for choosing machine learning was simply that “machines can learn, which sounds cool.” His entry material was Tom Mitchell’s Machine Learning. He initially studied under Dai Wenyuan, later went to Microsoft Research Asia to work on advertising problems, and still had little idea how research actually worked.
He now believes that confusion does not disappear with seniority. Research is inherently uncharted territory: even deep familiarity with a field can reduce local uncertainty, but cannot remove the fundamental condition of not knowing what to do or how to do it.
The lab had many students and limited faculty guidance. A remark from Professor Xue—“There’s this thing called deep learning that seems pretty interesting”—became a turning point. Starting from papers, Chen and his classmates decided to study restricted Boltzmann machines, then spent nearly 2 years committed to a direction that looked reasonable at the time.
5. Spending 2 Years Using the Wrong Method on the Right ImageNet Problem Trained His Tolerance for Risk
Chen and Lin Yuan began with convolutional restricted Boltzmann machines, first working on face and pedestrian recognition. After Fei-Fei Li’s team released ImageNet, they decided to tackle large-scale classification. AlexNet’s breakthrough was still about 2 years away, but work from Andrew Ng’s team and CUDA ConvNet had already demonstrated the potential of GPU-accelerated deep learning.
The lab bought a graphics card that was probably a GTX 380 or 390; Chen no longer remembers the exact model. The server’s power supply could not support it, so the team bought another power supply, placed it outside the chassis, and wired it back to the card to build its first GPU server. CUDA was around version 2.0, and much of the gradient propagation, convolution, and optimization code had to be written by hand.
A single model could take roughly 6 months to develop in C++ and CUDA C, yet the results never beat the SVM baseline. Chen later concluded that deep learning itself was the right direction, but supervised learning—not the restricted Boltzmann machine path they had chosen—was what ultimately worked.
He Tairan summarized the lesson as “you cannot lock in both the problem and the method,” which Chen agrees with. More importantly, after 2 years of work, no paper, and a failed result, he discovered that “we are fine—I’m still alive and well.” That gave him more courage to take on high-risk projects thereafter.
6. KDD Cup and Hong Kong Took Him from Knowing How to Solve Problems to Knowing How to Find Them
When the team entered a KDD Cup recommendation-system competition, it initially started from restricted Boltzmann machines but quickly found matrix factorization more effective. Chen believes the team achieved a strong result precisely because it did not cling to one method. He also notes that the winning method in the second KDD Cup was a hybrid of matrix factorization and a tree-based method. The experience later helped him enter the North American academic community.
The more important turning point came from a presentation by Li Hang at APEX. Li walked through problems and solutions in information retrieval one by one, and Chen suddenly realized: “It seemed like I could solve all these problems, and I could come up with some other methods.” With support from Li Hang and Yang Qiang, he went to Hong Kong for an internship, expanding his research perspective beyond a single ImageNet classification task.
He once thought he should have understood the distinction between choosing problems and choosing methods much earlier, then offered the opposite judgment: “I wish I didn’t know them.” The early losses were inefficient, but they built methodological foundations, the ability to accept failure, and the confidence to take on new problems later.
7. His Research Taste Is Not About Defending One Method, but Constantly Changing Tools Around Creation and Scale
Chen distinguishes between 2 research styles: starting from a method and applying it to different settings, or becoming interested in a class of problems while allowing the solution to change. He clearly belongs to the latter. The underlying motivation is still the creative impulse of his childhood—first creating games, later “creating intelligence, creating things.”
After the early breakthroughs in deep learning, he and his advisor asked whether the success came from neural networks themselves or from a larger hypothesis space and more data. They tried other models to test the hypothesis. His advisor favored kernel methods; Chen, drawing on his KDD Cup experience, argued for tree-based methods.
The real obstacle turned out to be systems. Existing implementations could not compute the tree models he wanted fast enough. This produced the central conviction that has guided him for nearly 15 years: algorithms, data, and compute must be considered together. If the system prevents a model from scaling up, he does not want to work around the bottleneck; he wants to solve it directly.
8. Choosing UW Was a Choice of Advisor and Research Style, Not a Ranking Decision
While working on recommendation systems at Shanghai Jiao Tong, Chen open-sourced SVDFeature. Danny, who was working at a startup in Israel, noticed the project and interviewed him for a blog, then introduced him to Carlos Guestrin and Alex Smola. Although he also had an offer from CMU, he chose UW in order to work with Carlos.
Carlos gave students an almost blunt starting standard: “You should write the best paper.” The lab also had a “no Beamer, only PowerPoint” policy, requiring researchers to use visual communication so audiences genuinely learned the material rather than relying on dense equations and templated slides.
This training later carried into both Chen’s papers and open-source projects: a result must not only demonstrate novelty, but also be understandable, adoptable, and extensible for users. Carlos’s group was working on LIME at the same time, pioneering interpretable machine learning and showing Chen how clear communication could establish an entirely new field.
9. XGBoost’s First Moat Was Pushing One Algorithm to the Limit of Speed and Usability
XGBoost began as a way to test a hypothesis about scale and data using tree models, but Chen followed the ACM class standard: “If you do one thing, do it extremely well.” He tried a range of optimization techniques, making the release implementation the fastest Gradient Boosting tool in the world at the time.
Product details mattered as much as speed. Traditional workflows usually cleaned missing values first, filling them with numerical substitutes such as the median. XGBoost instead lets the model learn default branches from historical missing-value patterns and decide which side missing samples should go to, eliminating a large amount of preprocessing.
Chen gained his first clear understanding that a vertical system cannot be only an algorithm or only engineering. The algorithm determines why the model works; the system determines whether it can operate at real-world data scale; and workflow issues such as missing data determine whether users are willing to treat it as an “out-of-the-box” tool.
By comparison, early SVDFeature was a large, comprehensive tool with 20 or 30 configuration parameters. XGBoost focused on doing one algorithm well. Chen believes small teams and individuals especially need this kind of focus; otherwise, it is difficult to establish a clear enough advantage at any single point.
10. The Community Turned XGBoost from a Personal Project into a Software Asset with Staying Power
XGBoost’s tree-visualization tool came from Michael, a French lawyer. He originally wanted to use the model to identify tax fraud, then contributed an R package and gradually moved into work at the intersection of law and machine learning. The episode showed Chen that maintainers cannot design for every real-world need in advance.
Core maintenance is no longer handled by Chen alone. Younger UW alumni and many other community members have continued to drive the project. Code is only a snapshot at a given moment; whether software succeeds depends on continuously absorbing issues, fixing problems, and bringing new contributors into the project.
The paper was not completed until the project’s third year. Before recording, He Tairan saw that citations had approached 70,000. Carlos wanted an article from which readers could learn, rather than one optimized only for acceptance odds. Chen admits that at launch he had no idea it would last this long; he simply thought, “It’s pretty decent, and GitHub is pretty fun.”
11. XGBoost’s Challenge to the Neural-Network Hypothesis Was Half Right, and That Half Still Matters
The correct insight at the time was that model capacity should not be fixed in advance. A linear model has a predetermined parameter count, whereas a tree can grow arbitrarily with the data. Neural networks can also obtain enormous capacity, but through a more brute-force path: setting the parameter space extremely large from the start rather than growing it with the data.
Tree models remain common out-of-the-box tools for tabular data and time series because their splits and combinations naturally express many structures. Chen therefore believes the judgment about tree-model capability has not become obsolete, even though neural networks later came to dominate a much broader range of applications.
What they underestimated was the composability of neural networks. Modules such as Transformer blocks and linear blocks can be repeatedly stacked, expressing complex functions in a more compact form. Chen stresses that they were not declaring neural networks ineffective; they were testing whether representation or data scale mattered more. The answer was “partly right, partly wrong.”
12. MXNet Combined Dispersed PhD Projects into a Shared System for the Next Generation of Deep Learning
After AlexNet’s success, Chen revived his earlier enthusiasm for deep learning, which had failed but never disappeared. He first developed CXXNet with Xu Bin, using Expression Templates to fuse multiplication, subtraction, and update rules into inline GPU kernels, then entered it in Kaggle. Caffe had already been released, and the framework ecosystem was in a period of rapid experimentation.
Other efforts at the time included Minerva, Purine, Mocha in Julia, and Parameter Server. PhD students including Li Mu, Minjie, Zhang Chiyuan, Lin Min, and Xie Junyuan, spread across CMU, MIT, NUS, NYU, and UW, gradually came together and decided to combine the lessons from their respective projects into MXNet.
The core design included Python first, automatic scheduling of computational dependencies, and automatic differentiation. Users wrote forward and backward passes in Python, while the system automatically scheduled multi-GPU parallelism, all-reduce, and communication–computation overlap. Distributed capability became a first-class citizen because of the team’s Parameter Server experience.
MXNet also supported both declarative computation graphs and imperative execution. The “Mix” in its name referred both to the fusion of teams and projects and to the mixed programming style. Compared with Caffe’s first-generation backpropagation framework, this represented a generational shift in development model and systems-optimization headroom.
13. Knowing TensorFlow Was Coming Did Not Stop the MXNet Team
When MXNet was launched, the team did not yet know about TensorFlow. Chen later interned at Google Brain and confirmed through internal dogfooding that it was about to be released. He also introduced the Brain team to MXNet’s experience, but the group did not view a tech giant’s entry as the end of the road: “Even a young team like us can copy a little.”
The PhDs did not fight over first-author credit. They ultimately submitted a single workshop paper, later preserved as an arXiv version. Chen attributes the smooth collaboration to mutual technical respect. Li Mu was a long-term “mainstay”; after graduating, he joined Amazon and continued to drive development, adoption, and community cohesion.
MXNet’s multi-GPU performance remained ahead for years. NVIDIA continued contributing to achieve benchmarks such as MLPerf, while Amazon promoted it as a major framework. Chen remembers that this advantage persisted for some time after PyTorch appeared, showing that the project did not immediately lose its position because of weak corporate support or poor low-level performance.
14. MXNet’s Core Lesson from Losing to PyTorch Was a Misjudgment About UX Priority and Engineering Scale
Chen’s direct retrospective is: “You have to make the user experience as good as it can possibly be.” MXNet initially tried to achieve performance and usability simultaneously, but in hindsight “we should actually have chosen user experience first.” He Tairan’s experience with TensorFlow 1.0’s hard-to-debug static graphs and Torch’s intuitive interface likewise confirmed how decisive the developer experience was.
After MXNet 2.0, the team tried to move toward a more unified model and launched early usability ecosystems such as GluonCV and GluonNLP, the latter resembling Hugging Face in some ways that later became familiar. But PyTorch had already built momentum, and technical catch-up could not easily reverse user habits, tutorials, and contributor networks.
The deeper issue was engineering burden. Even with Amazon, NVIDIA, and many contributors, there were never enough people to keep hand-writing high-performance implementations for every GPU generation and every operator. Chen therefore posed a new question: “How can we let fewer people do more?” The seed of TVM was planted there.
15. Google Brain Reinforced His Tendency to Choose High-Risk, High-Return Problems
Chen was fortunate to become Ian’s first intern. He proactively chose lifelong learning: if intelligence keeps evolving, a small model’s capacity will eventually become insufficient, so could a trained model be rapidly bootstrapped into a larger one? The work ultimately became the Net2Net project.
Ian plotted candidate projects on a risk–impact matrix, and Chen chose the direction that ranked highest on both. He does not prescribe this choice as optimal for every researcher; he explains his own preference: “Time is limited, so you have to do what you want to do,” and what one wants to do often naturally falls into high-risk territory.
In the latter half of his PhD, he also conceived a way to train deeper models with less-than-linear complexity. Carlos believed that an algorithm-only result might be a NeurIPS poster, whereas pursuing it properly would require automating the recomputation choices and reaching best-paper level. Chen abandoned the idea because of limited bandwidth; related follow-up work appeared later.
16. With No Mature Map of the Field, TVM Built the First “Castle” of Machine Learning Compilation in 11 Months
TVM began with a direct pain point: the optimal kernel changes with NHWC or channel-first layouts, operator fusion, matrix dimensions, and hardware form. It was impossible to hand-write every combination. The team wanted a compiler to generate and compose implementations automatically instead of continuing to add engineers linearly.
Halide in image processing was an important precedent. Chen says TVM was initially inspired by Halide and reused parts of its design. But the deep learning compiler field had not yet taken shape. He describes the process as “building a castle on a deserted island”: there was a judgment about direction, but no established route.
The project began around April after a GTC and did not produce its first runnable version until March of the following year, roughly 11 months later. The main work throughout was writing code. Chen knew the technical path might take time, but believed the architectural ability accumulated through XGBoost, MXNet, and his early compiler training was enough to build it.
Li Mu worried that Chen was already in his third or fourth year of the PhD and taking on a new project was too risky. Chen’s answer was that whether the future held a job, a startup, or a faculty position, “the most important thing is still to do what you believe is most important at that moment.” He had not calculated that TVM would succeed; he simply refused to let a career milestone choose his topic.
17. VTA Brought Compiler Thinking into NPU Design and Solved Compute–Memory Overlap from First Principles
The team worked with the architecture researcher Terry. At first, they even considered baking neural networks directly into FPGA wires. They eventually designed VTA, the Versatile Tensor Accelerator, with an instruction set. Most early NPUs used hard-wired convolution units; VTA emphasized programmability, combining small matrices and memory operations to express more operators.
They quickly found that an NPU had to read and write memory while computing. They used a hardware queue and signals to coordinate 2 logical execution streams, applying a hyper-threading-like idea: on physically single-threaded hardware, one logical thread handled memory access while the other computed, with explicit notifications when resources became available.
Chen is not an architecture specialist, but he still finds the idea of deriving an instruction set from compiler requirements compelling. GPUs originally used large numbers of threads executing SIMT, while TPU/NPU designs leaned toward a single instruction operating on a larger matrix. As GPUs add Tensor Cores and NPUs absorb thread-based programming, the boundary is becoming blurred.
18. Choosing Important Problems Does Not Reduce the Workload, but It Raises the Ceiling on What the Same Time Can Produce
Professor Ao Ping at Shanghai Jiao Tong once told students: “A mediocre problem and an important problem take almost the same amount of time.” Chen agrees with the mechanism. Mediocre problems also contain endless implementation details; if the calendar will be consumed either way, the time should be spent on bottlenecks that genuinely matter.
When he was young, Chen imagined that becoming an expert would eliminate nervousness and make every problem easy. Reality was the opposite. Experience accumulates, but entering new fields still brings challenges, often made harder by expanding goals and responsibilities. “Doing interesting things is definitely uncomfortable.”
What may decline with age is the courage to act, because more knowledge makes it easier to see the failure paths in advance. Chen’s clearest self-explanation is that “a lot of the time, your courage may be a promise your past self made to you.” The task now is to honor that promise.
19. TVM Must Keep Rewriting Itself Rather Than Defending Its Historical Position as a Pioneer
Chen only remembers TVM’s first release as roughly 2017–2018. The community peaked around 2019–2020, during OctoML’s early period. More important than the exact years is that he wanted the project to evolve from the start. Today’s TVM has gone through multiple architectural iterations and looks materially different from the first version.
Facing new ecosystems such as Torch.compile, XLA, and Triton, he does not believe the compilation problem has been solved. Large models may appear more homogeneous, but programming models for each GPU generation, model variants, and cloud–edge devices continue to change. The team has used TVM to build WebML and MLC LLM, bringing models to environments such as browsers and vehicles.
The next step is not simply to offer a universal compiler, but to help developers build compilers for specific verticals faster. The team is also extracting common layers such as function-calling specifications so PyTorch, JAX, and other frameworks can call one another. The ultimate goal is to make systems development as simple as writing a few lines of Python for a neural network.
20. Behind 20 Years of AI Breakthroughs, a Systems Throughline Has Consistently Been Obscured by Algorithm Narratives
The first phase was big data. Machine learning moved from hundreds of samples to hundreds of millions of data points. The Netflix Challenge made recommendation systems a signature application, while products such as Toutiao also began with recommendation. Baidu’s large-scale linear regression, Apache Spark, and XGBoost were all systems foundations for putting data to real use.
The second phase was deep-learning pattern recognition. Alex Krizhevsky did not just propose a model; he was also a powerful CUDA hacker. He trained using what should have been 2 GPUs for a week and completed multi-GPU optimization, whereas earlier machine-learning experiments typically ran for only a few minutes. That engineering effort was a critical part of AlexNet’s breakthrough.
Caffe then pushed the computer-vision community toward deep learning, while MXNet, TensorFlow, and PyTorch spread frameworks more broadly. Chen contrasts this with his own experience: building one model early on could take 6 months and roughly 20,000 lines of C++/CUDA code; once frameworks matured, an algorithm could be implemented in a few lines of Python in minutes, dramatically expanding the space for algorithmic exploration.
In the large-model era, systems finally moved to center stage. Model architecture must account for efficient GPU execution, training must scale up, and products must deploy reliably. Co-design across models, hardware, and systems is no longer a backstage optimization; it is an openly acknowledged core capability of the industry.
21. MLSys Will Not Be Defined by One Fixed Method; It Will Follow the Next Real Bottleneck
Chen compares machine learning systems with database systems: both are problem-driven fields whose agendas are determined by the most important problems in machine learning at a given moment, not by one permanent method. Large-model training and inference optimization will therefore remain central, but fundamental bottlenecks such as hardware adaptation and software bring-up have not disappeared.
Agent systems are an important direction in their own right. Having AI optimize machine learning systems engineering—further automating kernel development, compilation, and integration workflows that historically depended on large amounts of manual labor—is an interesting problem.
If he were a third-year PhD student again today and could change only one thing, Chen would still work on the same direction: enabling fewer people to bring up new models and new hardware with fewer engineering resources. He does not claim this will necessarily be solved within a few years; he simply believes it is a problem “worth working toward.”
22. MLC LLM Is Betting on Unified Inference, but Edge Demand Still Depends on Model Capability
MLC LLM is TVM’s extension into the large-model era. The team believes compiler developers must understand the bottlenecks in LLM inference firsthand, so it built a unified inference engine across cloud and edge while using machine learning compilation to reduce the cost of hand-writing implementations for every platform.
He Tairan says he has seen MLC demos running on iPhone. Android may also be in progress, while AMD GPUs and MacBooks can deploy it as well. Examples include Llama 2 and models supported earlier.
Chen’s assessment of the current landscape is restrained: “Everything is mostly on cloud. Edge is still a question mark.” But if pre-training has “kind of stalled a bit,” capability gains slow, and the industry shifts from the next level of intelligence toward applications, the weight of cost, privacy, and specialization may rise.
He Tairan believes a robot grabbing a cup or walking cannot wait 10 seconds for cloud reasoning, making latency a potential edge imperative. Chen agrees that privacy and real-time performance both matter, but insists that capability is the prerequisite: a local model need not match the largest general-purpose model. If it becomes “good enough” in a specialized domain, the balance between local and cloud models could change.
23. Model Convergence May Compress the Compilation Space, but Cannot Remove Hardware-Generation Complexity
He Tairan raised a sharp counterexample: if a small number of models such as GPT-5, Gemini, and DeepSeek account for most inference, and the industry only needs proprietary optimization for the Top Five, will the MLSys community wither? Chen’s candid answer is that he “cannot completely deny that possibility.”
The other side is that hardware must become increasingly specialized, and every chip generation from the same vendor can introduce new programming features. Models are also moving toward hardware-friendly forms, making the entanglement deeper. There is no universal recipe that keeps delivering the next 2x, and the model best suited to the next hardware generation may be different. He cites DeepSeek’s design for particular applications and hardware environments as an example.
The “hardware lottery” reveals another risk: existing hardware may constrain the space of models researchers can explore, and today’s convergence may not equal the global optimum. The public value of compilation and systems is to lower the experimentation cost for new models and hardware, continuing to enable a broad ecosystem rather than serving only the structures that have already won.
24. Universities Cannot Run 100,000-GPU Pre-Training, but They Can Still Establish Ecosystem Standards in Critical Modules
Chen acknowledges that the gap in data, compute, and model scale between universities and frontier labs such as Anthropic and OpenAI is unavoidable. Universities cannot reproduce pre-training at the same scale. But limited resources force researchers to find technical approaches that can substitute for “putting 100 people on the problem.”
CMU and related centers can access clusters on the order of 100 GPUs, which He Tairan notes is still 2 or 3 orders of magnitude below 10,000 or 100,000 GPUs. Chen’s response is that many R&D tasks do not require training the largest models. The best kernel optimization, for example, only requires access to the latest NVIDIA GPU, and industrial teams may not themselves have unlimited resources at all times.
Universities cannot act as isolated labs; they must amplify their work through open-source communities. Successful systems projects often carry a university imprint because universities are relatively neutral, allow long-term accumulation, and can open a critical module to global contributors for validation and extension.
XGrammar is his example. Agents need to output JSON that satisfies a specified format, but existing structured-generation approaches were inadequate. The team focused on solving that module, making it one of the de facto standards in the open-source community. The leverage came not from owning the entire LLM stack, but from identifying the bottleneck and doing it exceptionally well.
25. Open Source Is Not Free Code Distribution, but the Organizational Work of Maintaining Trust Over Time
Chen is a believer in open source and open science. Communities enable people to exchange ideas, share, and receive feedback, while universities use them to push research into practice. For students, open source is especially valuable: a new employee at a company usually owns only a small part of a large project, whereas a PhD student can architect a complete system from scratch.
The cost is equally clear. Open source requires continuously reading issues, fixing bugs, responding to users, and coordinating contributions. None of that maps directly to traditional paper metrics. It is therefore “not a free thing,” and few research groups are willing to follow the path for the long haul.
Building a community first requires technology that is correct and worth trusting, then passion and sustained maintenance. If a team suddenly abandons a project, users will naturally question whether it is worth depending on at the next release. Every fix and every promise kept is part of “building trust.”
26. OctoML Started with TVM but Proved That an Open-Source Project Almost Never Directly Becomes the Final Commercial Product
Several TVM contributors and faculty members saw an industry opportunity after graduation and founded OctoML. Chen worked full time during the first year, then shifted to part time and took responsibility for the long-term technical direction. His initial motivation was not even entrepreneurship itself, but the hope that the company could help the open-source community succeed.
He Tairan, drawing on his experiences with Ray/Anyscale and Taichi, asked why customers would pay if the open-source version was already good enough. Chen does not deny the tension around monetization, but views open source as one path with its own trade-offs. Red Hat and Databricks show that it has no inherent disadvantage; the key is finding a business model that can coexist with the community.
OctoML initially targeted hardware vendors and customers deploying machine learning, selling cross-hardware optimization and rapid deployment. After large models emerged, the company shifted decisively about 2 years ago toward model-deployment services. Its final form was closer to the endpoint businesses of Fireworks and Together, with customers directly calling APIs for models such as Llama.
The competition consequently shifted from selling compilers to inference cost, reliability, and product capability. Features such as Multi-LoRA could provide differentiation, while TVM moved back down the stack as the technology supporting faster support for models and hardware. Chen had long been psychologically prepared: “What you start out wanting to build and what you end up doing will definitely be different.”
27. Entrepreneurship Forced Technical Idealism to Learn PMF, Leadership, and Costly Self-Reinvention
The biggest unexpected lesson was not that a particular algorithm failed, but that “the human factor is very important.” Founders must communicate with people, take on leadership in large teams, coordinate partners, and explain from a product perspective why a technology deserves investment. Technical correctness alone is not enough to move an organization.
Chen divides startup strategy into 2 layers: one must seriously search for product-market fit, while also possessing excellent technology that creates differentiation. He acknowledges that not everyone will agree with this ordering, but insists that “in the end you have to have something to differentiate yourself.”
After large models arrived, the team overturned the original graph-compilation component for LLM support and rebuilt Relax. Not everyone internally agreed, since patching the existing system would be faster in the short term and a rewrite would inevitably create problems. Chen chose to reinvent the system; otherwise, the team would eventually “bite the technical debt.” If they kept optimizing ResNet, “we would lose terribly in the end.”
28. The NVIDIA Acquisition Was Not a Preset Exit Script, but a Way to Bring Compilation into a More Complete Hardware Map
He Tairan noted that OctoML was founded in 2019 and announced its acquisition by NVIDIA at the end of 2024, a span of more than 5 years. Chen says he “never thought about” an exit route when he started the company; he only wanted to do the work well. In retrospect, the attraction of joining NVIDIA was that the broader organization could continue solving important machine learning compilation problems.
NVIDIA itself faces universal deployment requirements: from data-center Blackwell to edge products such as Thor, different generations and product lines must handle model–hardware adaptation. The cloud–edge and cross-generation span means the problems OctoML had studied did not disappear upon entering a chip company.
Asked what he would change if he did it again, Chen refused to offer a polished counterfactual answer: “It’s very hard to see.” Every commitment can produce positive or negative lessons, but together they form the journey. That is consistent with how he views his early deep-learning failures.
29. Becoming a Professor Did Not Take Him off the Code; It Made First-Hand Experience His Superpower
Near graduation, Chen chose academia because he liked the uncertainty of research, solving new problems with students, and teaching. He had no desire to become a pure manager. When founding OctoML, he explicitly declined the CEO role because it required full-time commitment and would mean giving up a large amount of technical work.
At CMU, he places support-team members at the highest priority while still trying to “be on the ground.” The latest TVM core rewrite was rebuilt almost from the ground up, with Chen personally completing the foundation. Because he could not guarantee that every intermediate commit would leave all tests green, roughly 6 months of work remained local for an extended period.
AI tools have made first-hand development more feasible for professors. He uses Cursor and believes that without AI-based tools, a rewrite of this scale would have been “many, many” times harder. The tools did not replace judgment, but lowered the execution cost of turning an architectural idea into code.
He does not define productivity by paper count. The team deliberately takes on fewer projects and goes deeper; OctoML’s daily operations are handled by the co-founder, independent students drive their own work, and Chen provides support and key technical judgment. Different people have CEO-style or frontline superpowers. He chooses the mode that “makes me feel relatively happy.”
30. Real Long-Termism Is Like Weaving a Net, Not Chasing Every Fish That Swims Past
Chen’s metaphor for students is fishing. One approach is to chase fish with a spear: fast movements, but frequent misses. The other is to slowly weave a net in one area. When an opportunity appears, even if the first few attempts fail, the odds of catching the fish are higher. Breakthroughs that appear suddenly are often not last-minute efforts, but accumulated preparation finally meeting the right problem.
A faster research cycle and annual GPU updates do not mean every project must be completed in 3 months. The standards remain whether the problem matters and whether it deserves deep work. 20 years ago, Dai Wenyuan also lamented that SVM optimization seemed to have been exhausted by previous researchers; the result proved that “there are always new problems.”
Chen wrote in 2019 that difficulty and helplessness are random fluctuations. Given enough time and patience, a random process will converge toward a steady state commensurate with the effort invested. In the interview, he added that the theorem is easy to state but life is difficult to live; the farther one moves outside the comfortable zone, the more likely setbacks are to occupy most of the time.
At a fork in the road, he often asks: “If I choose this path and fail, will I regret it?” That is equivalent to asking whether he would regret not choosing it more. The outcome may not be rationally optimal, but the process remains worth experiencing. Failure is acceptable; long-term persistence does not mean clinging to a method already proven wrong.
31. The More Resources One Has, the Harder It Becomes to Preserve the Beginner’s Original Intent
The episode was recorded on September 5, 2025. After research, entrepreneurship, an exit, and a faculty career, Chen believes the hardest challenge over the next 5 to 10 years will not be mobilizing more resources, but maintaining passion. During the MXNet era, he did not worry about what kind of competition TensorFlow might create; today, he repeatedly asks whether a project can succeed.
The hesitation comes not only from temptation, but also from responsibility. Resources imply an obligation to make the team succeed and produce enough impact to justify continued investment. But if one is completely constrained by responsibility, it becomes difficult to do risky work. The CMU team has still given him room to accept failure and continue exploring, provided he actively preserves that mindset.
He is “not particularly interested” in the definition of AGI itself. He cares more about whether models are pragmatically useful and whether one person can leverage the work of 100 people in the past. An ideal AI world would allow open-source projects and individuals to flourish, but whether that probability is rising or falling, his only answer is: “It requires everyone’s effort.”
For him, success ultimately comes down to having a clear conscience—using technology to affect others and contribute to the shared world. Happiness means “whatever the heading is, solving the problems we want to solve together.” His time capsule to the future is to accept more failures and continue to reinvent: “Remember the promise you made to yourself, then keep moving forward with your ideals.”