104: I Went Looking for a “Backer” for Linear Attention—ByteDance Said No, MiniMax Said Yes
Summary
MiniMax-01 was a company-level bet on an unconventional architecture as its flagship model: 456B total parameters, 45.9B active, with 闫俊杰 committing 80-90% of the company’s resources despite putting his odds of success at roughly 50%. 钟怡然 says he was already more than 90% confident, given that the team had been researching the approach since 2021; ByteDance showed “not much interest” in August or September 2023, while MiniMax ultimately became the “backer” willing to pay for linear attention.
Linear attention aims to eliminate the quadratic compute cost of Transformer as sequence length grows—not merely the memory problem that FlashAttention has already eased. By changing the multiplication order of Q, K and V, Lightning Attention reduces complexity from quadratic to linear; 钟怡然 says that at 1M tokens, attention alone can run 2700x faster than Full Attention. MiniMax has reached 4M-token context and believes 10M is computationally manageable, shifting the bottleneck to data and applications.
This was not a dramatic all-in, but a progression from 3700 models trained from scratch and scaling-law experiments up to 7B, followed by a roughly 9B-active-parameter bridge model and industrial validation on about 2000 GPUs. The team compared Lightning Attention, HGRN2, Mamba and other routes while planning the experiments in advance; 钟怡然 stresses that “every choice was backed by experiments,” and that the formal scale-up was comparatively smooth.
The fundamental weakness of linear architecture was not language modeling but retrieval: a fixed-size KV Cache must compress inputs of arbitrary length, putting needle-in-a-haystack recall at an inherent disadvantage. MiniMax adopted the conservative approach of mixing one Softmax Attention layer into every seven layers; unexpectedly, this “less-than-elegant” hybrid outperformed all-Softmax Attention on retrieval in its scaling-law experiments.
The debate between linear and sparse attention remains unsettled, with the central question being not who can make things faster first but which approach has the higher ceiling on large models and ultra-long sequences. 钟怡然 believes NSA, MoBA and Seer Attention have solved the engineering-efficiency problem of learnable sparse attention but remain lossy approximations. His counterquestion is: “Could the quadratic complexity itself be redundant?” The host pointed out that MoBA already has code running online; 钟怡然 argues that public validation currently tops out at roughly 7B, insufficient to determine the winner at hundreds of billions of parameters.
Architectural potential has not automatically translated into long-form product quality: in a roughly 20,000-word test, the host found that both MiniMax-01 and R1 went off topic, while Kimi K1.5 performed better that time. 钟怡然 did not dodge the result, saying the current version “has not pushed things to the extreme,” but attributed the gap to insufficient long-context training data. The team ran strict apple-to-apple comparisons and found no disadvantage for the new architecture on identical data; the next model must convert potential into capability.
R1 changed MiniMax’s view of deep reasoning and led it to combine native multimodality, a new linear architecture and deep reasoning in the next release. The team had initially believed reinforcement learning mainly improved in-domain ability, but R1’s generalization forced it to accelerate. At the time of recording, 钟怡然 put the odds at 70-80%, saying the goal was not merely to match R1 but to approach o1 or o3; the release was expected in April, no later than May, and should be open-sourced afterward.
MiniMax is betting on the second half of the long-context game, while 钟怡然 acknowledges that a normal commercial company’s technical lead is usually erased within three months; the product is the vehicle for proving new technology and converting it into users and feedback. He says the best proof is to “put it into a product and let everyone use it.” The longer-term ambition is a system that can learn and improve continuously, with language, multimodal, embodied and spatial intelligence generating new data together rather than relying on successive rounds of human-pretrained models.
Deep dive
1. Open Source as the Commercial Debut of a New Architecture
钟怡然 describes MiniMax-01 as “a debut for a non-Transformer architecture in a genuinely commercial-scale foundation model.” It was intended not only to demonstrate Lightning Attention but also to direct the community’s attention toward long-context applications.
For the core business, whether the model is open-sourced has limited impact. The benefit of open source is to show that MiniMax “cares about technology and is more willing to try new technology,” build technical influence and attract people who want to create new architectures.
闫俊杰 had already been considering open-sourcing the model in June or July, when training was not yet complete. After training finished in October or November, the team added a complete report and further experiments to make the conclusions and arguments more robust.
The new architecture lacked an established ecosystem, and the community initially found inference through Hugging Face very slow. MiniMax later merged some inference optimizations into vLLM; most of the remaining GitHub issues concerned generic problems such as frequent model errors, multi-image support and deployment.
2. GPU Progress Only Delayed Transformer’s Quadratic-Complexity Crisis
钟怡然’s starting point was blunt: “The Transformer architecture has one major bug”—attention compute grows quadratically with sequence length. FlashAttention makes memory usage linear, but does not eliminate the computational complexity itself.
From V100 and A100 to H100, the leap in compute masked the urgency of the problem, making 128K and 256K sequences still acceptable. As sequences continue to expand, using hardware to cover the quadratic cost is ultimately not a structural solution.
3. Changing the QKV Multiplication Order Turns Quadratic Compute Linear
Standard Softmax Attention first computes Q×Kᵀ, applies Softmax and then multiplies by V. 钟怡然 reduces the core issue to three matrix multiplications: computing QK first and then multiplying by V is quadratic, while computing KV first and then multiplying by Q is linear.
Linear attention removes the Softmax Attention step. That is why it can be called “non-Transformer,” while others call it a linear Transformer; the naming dispute reflects different views of what defines a Transformer.
钟怡然 says linear attention and linear RNN are “essentially the same thing.” The biggest obstacle for traditional RNNs was their inability to parallelize; the significance of a linear RNN is that its recurrent structure can be trained in parallel at scale.
His intuitive analogy is that a Transformer “reopens the book” every time it generates a token, reviewing all previous tokens. A linear model is more like a game of telephone: it reads the previous state and cache, making computation cheap but potentially accumulating compression and transmission errors.
4. The Team Deliberately Left the Crowded Transformer Track in 2021
Sparse attention computes only part of the scores in the full n×n attention matrix, so 钟怡然 regards it as a “lossy approximation.” In the team’s 2021 experiments, it underperformed Softmax Attention and was only “a tiny bit” faster.
His judgment at the time was that sparse attention was “not worth the trade-off” and had a low ceiling. Rather than continuing to follow the field, the team chose linear attention, where fewer researchers were working and the changes were more fundamental.
The route began with the CosFormer project in July 2021, whose paper was later published at ICLR. 钟怡然 calls early linear attention “a beautiful bubble”: attractive in theory, but poor in practice and slow.
5. Three Years of R&D Filled the Gaps in Quality, Speed and Scale
From 2021 to 2023, the team covered nearly every major linear approach, including linear attention, linear RNNs and long convolutions. The first task was to solve language-modeling accuracy and catch up with Transformer.
By the end of 2022, they compared perplexity, common foundation-model leaderboards and Long Range Arena under identical data conditions, concluding that their method could match Transformer at small scale.
The second obstacle was more counterintuitive: although linear attention has low theoretical complexity, its right-multiplication form introduces recurrent operations, and GPUs are “very unfriendly” to loops. Actual runtime was therefore far slower than the paper complexity suggested.
TNL and Lightning Attention addressed the engineering problem in 2023. After validation on a maximum 15B dense model by the end of that year, the team concluded that both accuracy and efficiency had been addressed and the system was “scale-up ready.”
6. Scaling Up Exposed the Retrieval Weakness of Pure Linear Architecture
The language-modeling performance of a 15B pure-linear model was already close to Transformer, but the team saw its first major mismatch with expectations on needle-in-a-haystack retrieval. The flaw only became visible after further scaling.
Needle-in-a-haystack tests insert an out-of-place sentence at different positions in a long document and check whether the model can retrieve it precisely. 钟怡然 views this as a basic in-context capability: “It has to be able to repeat everything that came before.”
The underlying contradiction is that a linear model compresses inputs of any length into a fixed-size KV Cache. When “fixed capacity” faces an input of “arbitrary capacity,” information compression is inevitable, making precise recall inherently harder.
7. A One-in-Seven Hybrid Turned the Weakness into an Advantage
MiniMax ultimately inserted one Softmax Attention layer every seven layers, with Lightning Attention everywhere else; the first layer remained linear. This preserved a periodic channel for reviewing the full context.
The team tested one Full Attention layer every 7, 14 or 16 layers, and even a roughly 80-layer model with only one Full Attention layer. Language-modeling differences were small; retrieval was the metric that changed materially with the mixing ratio.
The most surprising scaling-law result was that combining a method with a retrieval weakness and the standard method did not merely close the gap: retrieval “far outperformed” all-Softmax Attention. 钟怡然 says the team was surprised as well.
The one-in-seven ratio had no strong theoretical guidance. It was driven mainly by experiments and informed partly by a similar ratio in Jamba. Having already replaced the underlying architecture aggressively, the team chose this “very conservative” mix as a safety net.
8. Linear Architecture Moves the Context Bottleneck from Compute to Data and Use Cases
钟怡然’s efficiency figure is that at 1M tokens, Lightning Attention can make the attention step alone 2700x faster than Full Attention. This is not the end-to-end speed of the entire model.
MiniMax has already pushed sequence length to 4M and believes 10M would also be computationally manageable if necessary. The questions therefore become how to construct data at 10M tokens and what 10M-token context is actually useful for—not whether compute can support it.
His 4M-token example is to place the Four Great Classical Novels, or the full Dream of the Red Chamber, into context and then summarize them. For now, the team is focused on making sub-1M-token use cases work well before expanding further.
9. 3700 Models Broke the Company-Level Bet into a Testable Curve
钟怡然 considers it unacceptable to spend tens of millions of yuan training a large model only to discover that the architecture has failed. To persuade management to adopt an architecture never industrialized before, the preliminary experiments had to be “very solid.”
The team trained roughly 3700 models from scratch across different sizes and parameter counts, comparing Lightning Attention, HGRN2, Mamba and other approaches, as well as multiple hybrid ratios, to find the balance between speed and quality.
These experiments were not an open-ended process of adding more trials. The required GPU count, resources and model numbers had been entered into Excel in advance, and the final execution matched the original design. Many of the models were inexpensive small models.
The scaling-law validation reached a maximum of 7B dense parameters. The team first assumed that a scaling law existed, then plotted the trend and forecast performance above 7B. 钟怡然 says few companies run experiments this systematically, though he mentioned that Google and OpenAI have done so.
10. From Dense to MoE, Algorithmic Delivery Still Required Rebuilding the Industrial Stack
MiniMax-01 is a 456B-total-parameter MoE model with roughly 45.9B active parameters. Because the preliminary scaling-law work was mainly done on dense models, another team had to migrate the structure to MoE and retune it from scratch.
Before the formal large-model run, the team trained a bridge model with perhaps roughly 9B active parameters, not far from the 7B experiment point. It may have taken a month and showed no anomalies, after which the team moved to a large training run on about 2000 GPUs.
The research group had worked at up to 15B and did not handle more advanced sequence parallelism, while paying relatively little attention to inference. The industrial team had to fill in parallelism, inference and cluster optimization; “a lot of wheels had to be built from scratch.”
Because the scaling law and small-model validation had already established confidence, the large-scale training of MiniMax-01 itself went relatively smoothly. 钟怡然 says the truly difficult decisions and risk removal happened before the GPUs were turned on.
11. The Lab’s 15B Ceiling Forced 钟怡然 to Find a Backer
The Shanghai Artificial Intelligence Laboratory lacked the compute needed to keep scaling. 钟怡然 believed that by the end of 2024 at the latest, some company would build a large linear-attention model: “We’ve been doing this for three years. Why shouldn’t we be the ones to bring it into existence?”
Starting his own company was briefly an option, but a foundational infrastructure company needs algorithms, data and training to work simultaneously. To enter the top tier, the team would have needed more than 10T tokens, yet without a first-rate model it could not prove the architecture—a classic chicken-and-egg problem.
In his view, the main domestic candidates capable of top-tier pretraining and able to take on the project were ByteDance and MiniMax. He had limited information about Kimi; otherwise, he would have had to look overseas.
The host added that investors in 2023 and 2024 cared more about applications, monetization and profitability, as if asking an “unfinished high-school student” to go out and make money immediately. That was a poor fit for the large, long-term investment required by foundational infrastructure.
12. ByteDance Passed; MiniMax Was Willing to Commit 80-90% of Its Resources
钟怡然 spoke with ByteDance around August or September 2023 but felt that its “interest level was not very high.” His explanation is that even large companies with people and data find it difficult to redirect substantial resources toward a highly uncertain direction.
In the second half of that year, 闫俊杰 (IO), who had previously worked with him at SenseTime, invited him to dinner. MiniMax was selecting the technology for its next-generation model; 闫俊杰 may have judged the work to be solid and was willing to make it the company’s flagship model.
Their subjective probabilities were different: 钟怡然 believed the odds of success were above 90%, while 闫俊杰 may have put them at roughly 50%. Even so, the latter was willing to mobilize “80-90%” of the company’s data, engineering and algorithm resources.
This was not an all-in from day one. The team first trained small models with manageable costs and increased scale gradually. After joining MiniMax and discovering the retrieval weakness, 钟怡然 also had the hybrid approach as a fallback, though he considered it “not elegant enough.”
13. Sparse Attention Caught Up on Engineering Efficiency, but Its Scale Ceiling Remains Unproven
钟怡然 believes the main breakthrough of NSA, MoBA and Seer Attention was finally solving the “sparse is slow” problem the team had encountered in 2021. He does not believe this has yet proved the ceiling at hundreds-of-billions industrial scale.
The host countered that MoBA had already released engineering code and had been running online for a year. 钟怡然 still maintains that public validation currently reaches only about 7B; credible comparison with MiniMax-01 requires models at the hundreds-of-billions scale.
MiniMax also tested MLA and KV Cache compression methods such as TPA, proposed by Tsinghua. Its observation was that the advantages of those methods narrow as models grow, while Lightning Attention begins above 7B to show a trend of “the larger the model, the more visible the gain.”
He also acknowledges that sparse and linear attention both have large research communities, and that Mamba clearly reignited interest in linear approaches in 2023. Regarding architectures developed by other companies, he cautiously speculates that they are “most likely” sliding-window attention combined with Full Attention.
14. “Lossy or Lossless” Has Become a Battle of First Principles
Many researchers believe that replacing quadratic attention with linear complexity must be a lossy approximation. 钟怡然’s counterquestion is: “Could the quadratic computational complexity itself be redundant?”
His position is that linear attention can be a lossless optimization, and that after hybridization it may even improve on pure Full Attention. Sparse methods, by contrast, compute only part of the positions in a dense mask, so “sparsity is necessarily lossy.”
The host asked whether learnable sparse patterns could eliminate the loss. 钟怡然 said such methods already existed in 2020; newer work makes them faster, but does not change his view that positions not computed are discarded outright.
He calls the disagreement “a battle of first principles.” Some early collaborators later stopped working on the area, which he attributes to differences in conviction about the route. The two approaches are also partly orthogonal: a linear model can absorb sparse techniques, while adopting linear attention requires changing the underlying architecture.
15. Long Context Does Not Mean Long-Form Tasks Are Solved
In a test involving a roughly 20,000-word English article, the host asked MiniMax-01, R1 and Kimi K1.5 about specific uses of short social-media videos. In that test, both MiniMax-01 and R1 went off topic, while Kimi K1.5 performed better.
钟怡然 did not dispute the result. The team uses needle-in-a-haystack, RULER and other standard benchmarks to check basic capability, but real-world applications remain “strictly tied” to the training data; architecture can provide potential, not the finished ability.
He attributes the gap to insufficient long-context training data rather than Lightning Attention. The team ran strict apple-to-apple comparisons and found no disadvantage for the new architecture with identical data; the current version “has not fully tapped the model’s capabilities,” and the next generation must close that gap.
16. R1 Shifted MiniMax from Watching Deep Reasoning to Accelerating After It
钟怡然 cites another team’s view that “compression produces intelligence” (压缩产生智能): because linear models continuously compress information, they may have an advantage in deep reasoning. That team reported relatively stronger reasoning performance from linear models in internal experiments, but MiniMax is still validating the claim.
After o1 appeared, MiniMax did not immediately follow. Its initial judgment was that reinforcement learning would mainly strengthen one aspect of the model’s in-domain ability, so it first focused on solidifying basic capabilities. R1’s stronger extrapolation and generalization changed that view.
He also qualifies R1’s strengths: it is excellent at difficult problems and professional tasks but “doesn’t feel like a general-purpose model,” and its answers to ordinary knowledge questions are not necessarily ideal. The host added that R1’s breakout was also helped by its open chain of thought, which let users see the reasoning process, while o1 did not fully expose its chain of thought at the time.
The next step remains anchored in the internal base-model leaderboard, followed by standard benchmarks such as mathematics, programming, SWE and AIME. The goal is not to produce something “roughly like R1” a month late, but to approach o1 and even o3.
17. The Next Model Stacks Native Multimodality, Reasoning and a New Architecture
MiniMax-VL-01 used an adapter approach rather than native multimodality. 钟怡然 says Gemini 2.0 had not yet been released and the native route was not yet proven, while an adapter could validate the data quickly and cheaply, with results that were “immediately visible.”
The current model under development is no longer a separate text model but a native multimodal understanding model. The challenge is to preserve performance on both text and vision leaderboards while integrating deep reasoning into the same system.
This means stacking the new linear architecture, native multimodality and deep reasoning at once. 钟怡然 says current experiments are “pretty good,” puts the odds of success at 70-80%, and says the project is the top priority, with most algorithm and engineering staff assigned to it.
The timeline given during recording was an April release, no later than May, which should be open-sourced afterward. The current architecture is expected to remain in use at least through June; after that, a new generation trained from scratch will use more data and an updated structure.
18. Technical Influence Depends on Both Open-Source Scale and Product Distribution
The external assessment of MiniMax-01 is that “the report is very well written, the work is solid, but somehow not many people know about it.” 钟怡然 summarizes IO’s style as “do ten, publicize five,” while a 456B model is not something an individual can easily deploy.
The host asked why MiniMax did not release a small model so that academics and developers could participate directly. The team does have smaller versions, but their data differs from the large model’s; IO wanted to raise the performance ceiling first. Maintaining multiple models or distilling the large model into a small one would also require additional staff.
钟怡然 believes the PR needs improvement: “This is no longer an era when good wine needs no bush.” His longer-term explanation is that the new architecture slows the team’s short-term pursuit of deep reasoning, but open-source results allow it to catch up quickly. MiniMax is “playing the second half,” betting that long-context demand will materialize.
The fundamental reason he joined MiniMax was that a small-scale paper could not persuade the industry; “putting it into a product and letting everyone use it” is the best proof of the technology. Even so, he believes a large model’s lead is usually erased “within three months,” and a normal commercial company must still turn attention into users and feedback.
19. Linear Attention Is Only One Layer of a Continually Learning System
What 钟怡然 ultimately wants to build is not one particular model but a system “that can learn and improve by itself”: given everything people see and hear, the model would learn text, vision, sound and the relationships between modalities on its own.
He links foundation models, embodied intelligence and spatial intelligence in a causal chain. Static training data may eventually run out; embodied systems can generate endless data through interaction with the environment, while effective interaction requires spatial understanding.
Reinforcement learning connects o1- and R1-style deep reasoning with robotic motor control in this vision. He speculates that applying robotics RL directly to language reasoning and finding unexpectedly strong results may have been partly an “accidental discovery,” but it could become a method for autonomous learning.
The postscript preserves the disagreement: 马毅 stresses that knowledge is not intelligence and that autonomous learning requires closed loops, feedback and correction; 王小川 argues that language is the central axis of intelligence and that multimodality and world models may go astray. Whether a purely virtual “brain in a vat” can develop curiosity, aversion and preferences remains unresolved.