DeepSeek, Kimi & MiniMax Attention Mechanisms, Paper by Paper
Summary
- All 3 companies are targeting the same cost bottleneck: long chains of thought and long documents are pushing attention’s O(n²) compute and linearly growing KV cache to the center of test-time scaling. DeepSeek and Kimi chose Dynamic Sparse Attention, while MiniMax-01 uses 7 layers of Lightning Attention for every 1 layer of Softmax attention; the routes differ, but the goal is the same: make prefill, training, and decoding cheaper. If performance can match traditional attention, 松琳 argues, the inference budget saved by more efficient architectures can be converted into longer reasoning time.
- DeepSeek’s Native Sparse Attention (NSA) is the strongest technical signal among the 3 papers because it does more than approximate full attention: after pretraining from scratch, it may outperform on loss, downstream long-context, and reasoning evaluations. Its Triton kernel approaches a 10x speedup over the Triton full-attention kernel around 64K sequences, with clear advantages already at 8K and 16K; 松琳 describes it as seeking “both efficiency and performance,” and gives his subjective view that “there’s an 80% chance it will serve the next V4.”
- NSA’s moat is not the 3 branches themselves, but the way it locks the algorithm, GQA, and GPU dataflow into a single piece of “brutal hardware beauty.” It uses compressed, selected, and sliding-window attention, while forcing heads in the same GQA group to select the same KV blocks; it then uses the head dimension to create matrix multiplications suited to Tensor Cores, reducing repeated KV-cache reads. The trade-off is a structure that is not particularly “clean,” but DeepSeek restores expressive capacity by adding more heads, retaining multiple query groups, and keeping extra branches, “walking the knife edge” to defend its efficiency principle.
- Kimi’s MoBA takes the opposite route: it represents each block with mean pooling, keeps only the selected-attention trunk, and relies on “the power of SGD” for routing, in exchange for no additional parameters and free switching between full and sparse attention. The simplicity comes with a concrete cost: independent head selection introduces indexing and reindexing, so it uses 512-token blocks and Top 3, versus NSA’s 64-token blocks and Top 16; MoBA still shows no clear speed advantage at 128K, and during SFT it needs to switch the final 3 layers back to full attention to address sparse gradients. But it has been tested to 1 million tokens and is reportedly already in the Kimi product—a case of truly “voting with its feet.”
- MiniMax-01’s core asset is scaling linear attention for the first time to a size large enough to claim GPT-4o-level performance, not inventing the hybrid architecture itself. Its 80-layer model repeats a pattern of “7 linear layers, 1 Softmax layer” 10 times: the RNN-like fixed state compresses history and turns each decoding step into constant-cost computation, while a small number of Softmax layers retain the full KV cache to restore retrieval capability. 松琳 summarizes the complementarity in one line: “attention is like flipping through a book; an RNN is like the human brain.”
- On risk and reward, 松琳 sees dynamic sparse attention as actually more conservative than pure linear attention: it retains the full KV cache and saves compute without saving memory, making the floor of its needle-in-a-haystack capability easier to assess. He still rates MiniMax as taking the most architectural risk of the 3, while emphasizing that a hybrid mix with 10%–20% Softmax layers had already been validated across multiple smaller models and related work; the 3 companies cannot be ranked directly because their training data, scale, and infra are not comparable.
- The broader capital-and-talent thesis is that once data scaling approaches its limits, architecture, open-source papers, and systems engineering are becoming visible competitive variables again. 松琳 ranks NSA first, MoBA second, and MiniMax-01 third; the real aha moment was NSA’s “across-the-board suppression” of full attention. He also believes open-sourcing is the best way for challengers to “show muscle” and build investor confidence, citing Qwen’s open source release and Alibaba’s stock performance. If Chinese model technology continues to catch up or even overtake overseas technology, domestic teams will also become more attractive to overseas students: “Talent always moves toward places with higher talent density.”
Deep dive
1. Long Chains of Thought Make Attention Efficiency a Prerequisite for the Next Generation of Models
松琳 places the timing of the papers’ appearance in early 2025: DeepSeek’s related reasoning work and Kimi 1.5 both rely on longer chains of thought to unlock reasoning, and the longer the output, the harder it is for ordinary self-attention to absorb the decoding cost. “Test-time scaling” is therefore not just a capability strategy, but also an architectural efficiency problem.
Full attention models every token pair, so the compute complexity of training and prefill grows quadratically with sequence length; autoregressive decoding, meanwhile, must continually read a KV cache that grows linearly with context. Long documents and long reasoning sequences amplify these 2 bottlenecks in different ways.
When 张小珺 asked whether the work was paving the way for a new model, 松琳 gave a clear but qualified answer: “There’s an 80% chance this DeepSeek paper will serve the next V4.” Kimi has both the reasoning demands of Kimi 1.5 and a product identity built around long text, so it also needs to reduce both prefill and decoding costs.
2. The Essence of Attention Is Modeling the Interaction Between Every Pair of Tokens
松琳 explains standard attention through query, key, and value: each token takes its query and computes inner products with the keys of preceding tokens to obtain attention scores; after Softmax normalization, those weights are used to form a linear combination of the values, producing the output at the current position.
Autoregressive models also need a causal mask to prevent the current token from seeing future information. The simplest way to understand it is that “attention is a pairwise modeling process”: it describes the degree of interaction between any 2 positions in a sequence.
Transformer’s key advantage over LSTM was not only expressive power, but training parallelism. The architecture uses GPUs efficiently, making it easier to stack parameters and scale models; BERT, GPT, and today’s mainstream large language models all built on the same foundation.
3. Dynamic Sparsity Lets Each Query Decide Where to Look
Static sparse attention methods such as BigBird predefine fixed patterns of jumps and windows; Dynamic Sparse Attention lets the current query dynamically choose which keys and values to access. Different tokens in the same sequence can therefore read entirely different regions of history.
Kimi and DeepSeek share the ambition of turning this query-aware sparsity from an inference-acceleration tool into an architecture that can be pretrained from scratch. The earlier obstacle was not the absence of the idea, but the mismatch between dynamic, discrete data access and modern GPUs’ preference for contiguous block reads.
The common predecessor of both papers is Quest: it divides contiguous KV tokens into blocks, builds a representation for each block, and lets the query select the most relevant blocks. Contiguous blocks can be read in bulk, avoiding random token-level memory access; DeepSeek’s NSA and Kimi’s MoBA can both be viewed as extending this framework to pretraining.
4. “Native” Means Dynamic Sparsity Can Finally Be Trained Natively on Hardware
松琳 calls Native Sparse Attention a pioneering piece of work. The breakthrough is not that it was the first to propose sparsity or block selection, but that it was the first to make large-scale Dynamic Sparse Attention “natively trainable fast attention”: training, prefill, and autoregressive decoding are all redesigned around hardware constraints.
The paper’s ambition is to be “both fast and good”: it lowers the forward and backward costs of training as well as inference reads, while allowing NSA pretrained from scratch to beat full attention on multiple benchmarks. 松琳 repeatedly emphasizes that this makes it more than a lossy acceleration plug-in for an existing model.
By contrast, many earlier methods mainly approximate the attention distribution of an already-trained full-attention model. The more the approximation is constrained by block granularity and hardware, the more likely it is to lose performance, and it is difficult to exceed the original model’s ceiling; 松琳 also notes that models such as Quest can be pretrained, although their pretraining efficiency may be worse than NSA’s.
On the content of DeepSeek’s continuing open-source projects, 松琳 refuses to “moonlight as a fortune teller,” offering only the possibility that they may include software-hardware co-design infra, MoE, and NSA-related models. He explicitly labels this as a prediction, not a route disclosed by the paper.
5. NSA Uses 3 Branches to Preserve Coarse Context, Fine Selection, and Local Priors
The first compressed-attention branch compresses each contiguous KV block into a key representation and a value representation, then lets the query perform coarse-grained attention at the block level. It provides a global overview while also producing an output that feeds directly into the final result.
The second selected-attention branch takes the Top-k blocks according to their block scores, expands all tokens within the selected blocks, and performs finer-grained token-level attention. The sparsity comes from here: instead of calculating fine-grained attention over every historical position, it calculates it only inside a small number of the most relevant blocks.
The third sliding-window branch focuses directly on nearby tokens, using the inductive bias that nearby positions are usually more important in language modeling. 松琳 notes that trained attention maps often concentrate substantial weight on local windows and on the tokens at the beginning of the sequence, the latter often known as attention sinks.
6. 3 Gates Let the Model Learn the Global-Local Mix
Each branch’s output passes through a gate determined by the current token representation, and the 3 outputs are then added together with learned weights. When 张小珺 asked how the branches coordinate, 松琳’s answer was: “Why not let the model make the choice directly?” Manually fixing the share of each branch would be less adaptable to different queries.
The design does not claim that compressed, selected, or sliding-window attention is individually a new module. Its value lies in combining them into a trainable, backpropagatable loop that is also suitable for hardware execution. 松琳 therefore warns against looking only at the architecture diagram and dismissing the paper as a “Frankenstein.”
7. MHA, MQA, and GQA Determine How Much Sparse Selection Actually Saves in Reads
Multi-head attention (MHA) keeps independent query, key, and value projections for every head; multi-query attention (MQA) makes all query heads share a single K and V, sharply shrinking the KV cache during inference but potentially causing performance degradation through excessive sharing.
Grouped-query attention (GQA) is the compromise: heads within the same group share K and V, while different groups retain different representations. 松琳 uses the example of every 2 heads sharing one KV set to illustrate how moderate sharing balances cache reduction against expressive capacity.
In MHA, each head already has to read its own KV, so allowing each head to choose different blocks does not increase total reads. In GQA, however, if heads sharing KV choose different blocks, the hardware must load the union of those sets, diluting the original sharing advantage.
This is the key fork between NSA and Quest or MoBA: the latter 2 preserve independent selection for each head, while NSA requires heads within the same GQA group to read the same KV subset in exchange for a much more regular dataflow.
8. NSA Sacrifices Head-Level Freedom Deliberately to Capture Real Decoding Throughput
NSA sums the block-selection scores across the heads in the same group and then applies Top-k jointly, ensuring mechanically that they select the same KV blocks. Shared KV therefore needs to be loaded only once, rather than reread for different selections by different heads.
松琳 acknowledges that this binding may reduce diversity and is exactly where many architecture researchers would worry about a performance drop. DeepSeek’s response is aggressive: if sharing is most effective for hardware, add more heads, retain multiple query groups, and use the compressed and sliding-window branches to restore expressive capacity.
The paper uses 4 groups and 64 heads in total, with a query-head dimension of roughly 192. Multiplying the 2 gives a projection of nearly 12,000 dimensions, far above the roughly 2560-dimensional hidden dimension. 松琳 connects this large up-projection to the MLA style of DeepSeek V3: “I’m training from scratch anyway.” As long as both training and inference are efficient, conventional architecture is not the primary constraint.
9. FlashAttention’s Success Shows That the Bottleneck Is Not Just FLOPs, but the Gap Between HBM and SRAM
松琳 reviews FlashAttention’s blocking scheme: a block of queries and a block of keys are brought into SRAM, local attention is computed with matrix multiplication, and the corresponding values are then read; intermediate attention scores are not written back to HBM, reducing expensive data movement.
SRAM is small but has high bandwidth; HBM, such as the 80GB of global memory on an A100, is large but relatively slow. The shared goal of hardware-efficient algorithms is to keep computation in SRAM as much as possible and minimize communication between the 2 levels of the memory hierarchy.
Full attention has to scan all historical KV blocks for a query block anyway, so contiguous loading does not waste much. In dynamic sparse attention, each query chooses different blocks; if contiguous queries are still bundled together, the hardware must load the union of their requested sets, which in the worst case nearly degenerates back into full reading.
10. NSA Uses the Head Dimension to Build Matrix Multiplications and Avoid Union Waste from Sparse Queries
To avoid the query-block union problem, NSA lets each token independently determine its output; but a single-token calculation lacks the matrix shape preferred by Tensor Cores. The solution is to use multiple queries under GQA that share the same KV selection, treating heads as a batch-like dimension and assembling them into a matrix multiplication.
松琳 particularly admires this kernel. Ordinary FlashAttention uses query-block size to create the matrix dimensions; NSA uses the number of heads instead. “There’s basically no waste”: it reads only the required KV while still invoking Tensor Cores, and this was the first time he had seen the approach used in the training stage.
Triton generally requires the relevant matrix dimensions to be at least 16, so each group needs at least 16 heads; the model also wants to retain selection differences across roughly 4 groups, ultimately driving the 64-head design. The seemingly unusual architecture parameters were actually derived backward from kernel constraints.
Using the A100 as an example, 松琳 says that at the same FLOPs, half-precision matrix multiplication on Tensor Cores can be roughly 16x faster than general ALU computation. On H100, matrix computation accelerates further while data loading becomes more likely to be the bottleneck—the “memory wall.”
11. Training from Scratch Is the Only Way for Sparse Attention to Break Through the Full-Attention Ceiling
Applying sparse inference to an existing full-attention model is essentially an attempt to approximate its existing attention map. Even if the full score is calculated first and then exact Top-k is applied, that is only the theoretical ceiling of an approximation; Quest must additionally select by block, gaining efficiency but losing another layer of flexibility, and therefore usually losing more performance.
松琳’s conclusion is direct: “If we want sparse attention to be even better than full attention, there is only one way out: train from scratch.” If the model adapts to dynamic selection from pretraining onward, it does not need to reproduce the full-attention distribution and may learn representations better suited to sparse structure.
NSA and Quest both retain the full KV cache, unlike mechanisms such as H2O that directly discard cache entries. 松琳 believes that “not discarding” is important because a position that is irrelevant now may be called by a later query; dynamic selection can temporarily skip information without permanently deleting it.
12. Sparsity May Not Be a Performance Tax, but a Form of Attention Deduplication
松琳’s intuitive explanation is that trained full attention is already “highly sparse,” with the weights of many token pairs close to redundant. Training directly with sparse attention may force the model to focus more tightly on truly relevant positions and reduce the chance of being “distracted” by irrelevant tokens.
This remains an explanation rather than a causal mechanism proven by the paper; he qualifies it as “my intuitive understanding.” The confirmed experimental finding is that NSA’s training loss stays slightly below full attention for almost the entire run, while multiple downstream benchmarks also show an advantage.
The needle-in-a-haystack chart is almost “green all over,” indicating that after retaining the full KV cache, the model can still retrieve target information from long contexts. What surprised 松琳 even more was that the gains were not confined to long text, but extended to chain-of-thought reasoning and R1-style evaluations.
This was his biggest aha moment of the episode: Dynamic Sparse Attention not only computes less, but may also be stronger than full attention. “It wants both efficiency and performance. That’s incredibly powerful.”
13. NSA’s Speed Advantage Appears at Shorter Contexts and Expands Rapidly with Length
The paper compares the Triton NSA kernel against a Triton version of FlashAttention to reduce unfairness caused by differences between CUDA and Triton implementations. 松琳 notes that NSA is already faster at shorter sequences, approaches a 10x speedup around 64K, and accelerates more relative to full attention as length increases.
The decoding chart reports expected speedup because autoregressive decoding is usually memory-bound, with efficiency largely determined by how much KV cache must be read at each step. NSA gives GQA groups a shared selection, creating a natural read advantage over full attention, Quest, and MoBA.
What 松琳 values most is not any single peak number, but the fact that the advantage covers training forward passes, backward passes, and decoding at the same time. “Fast and good” means the same architecture can reduce both pretraining cost and production serving cost, rather than shifting the burden between the 2 stages.
14. The Backpropagatable Compressed Branch Provides the Training Signal for NSA’s Block Selection
Top-k selection itself is nondifferentiable; if an architecture has only a selected-attention path, an incorrect block selection has little direct gradient signal to correct it. NSA includes the compressed-attention output in the final output, allowing the block representations and scoring path to receive gradients through the end-to-end loss.
The paper runs ablations against other block-selection methods such as Quest, and the alternatives perform worse. 松琳 therefore believes the compressed branch is not decorative: it is the key element that makes selection more informative and makes the first branch “make more sense.”
This also explains why NSA does not face the same prominent sparse-gradient problem as MoBA during SFT. Even if the fine-grained selected path fails to cover some prompt blocks, the compressed path still sends training signals back to a broader range of historical positions.
15. NSA’s Main Innovation Is Systems Co-Design, Not the Invention of 3 New Modules
When 张小珺 asked whether it was a “small innovation” or a “big innovation,” 松琳 acknowledged that sliding windows, block selection, and compression can all be found in the existing literature. If novelty is checked module by module, it is easy to conclude that “there’s nothing particularly impressive.”
His rebuttal is that the paper’s real significance lies in turning query-aware dynamic sparsity into an integrated system that is hardware-friendly for both training and inference. Methods such as Quest can also be pretrained, but may be less efficient; NSA’s kernel and GQA constraints make it better suited to large-scale use.
松琳 classifies it as more of an engineering innovation, but does not use “engineering” to diminish its value: “You can only cut costs if you are hardware-efficient.” At large-model scale, whether data can be read continuously, whether operations can be expressed as matrix multiplications, and whether useless cache traffic can be avoided are all part of architectural feasibility.
He summarizes the style as “the brutal beauty of hardware” and “walking the knife edge.” Within Quest’s broad principle—letting every query dynamically select blocks—NSA found a solution very close to optimal under current hardware constraints. As for how OpenAI implements similar capabilities, he jokes about “Closed AI”: the company has disclosed too little to know, although the GPT-3 report once mentioned sparse attention and current models cannot be verified.
16. MoBA Compresses the Same Problem into an Almost Parameter-Free Minimalist Design
Kimi’s MoBA also advances along Quest’s block-selection framework, but applies only mean pooling to each key block and calculates the score between the query and that mean. It adds no MLP for block representation and introduces no additional learnable parameters.
Compared with NSA’s 3 output branches, MoBA removes the compressed-attention and sliding-window outputs, retaining only selected attention. Its team’s philosophy is to “believe in the power of SGD”: even without extra branches or losses, optimization can gradually find the right blocks.
Each attention head can still select KV blocks independently, giving the model more freedom and a more “elegant” architecture diagram. But it lacks NSA’s group-shared selection and therefore cannot directly reuse NSA’s nearly waste-free training and decoding kernel.
17. MoBA’s Simplicity Pushes Complexity into Indexing and Block Granularity
MoBA’s implementation identifies all query tokens that select each KV block. These queries are usually non-contiguous in the original sequence, so indexing and reindexing are first used to gather them into a contiguous tensor before calling the FlashAttention kernel. This step contains a large number of indexing and reindexing operations.
That step is not free: if there are enough KV blocks, indexing overhead can become the bottleneck. MoBA therefore uses 512-token blocks and Top 3, while NSA uses 64-token blocks and Top 16; the final number of tokens read is similar, but the selection granularity is very different.
Top 3 large blocks are more likely to miss dispersed critical information, while Top 16 small blocks offer greater tolerance and finer granularity. 松琳 sees this as the cost MoBA pays for its minimalism, rather than simply a matter of hyperparameter preference.
The speed curves reflect the same difference: MoBA still shows no particularly clear speed advantage around 128K, while NSA already accelerates visibly at 8K and 16K. 松琳 therefore judges that MoBA may be better suited to extremely long prefill, while common pretraining lengths around 8K may not deliver the same benefit.
18. Sparse Gradients in SFT Force MoBA to Temporarily Switch Back to Full Attention
MoBA’s training loss initially trails full attention and only gradually closes the gap; it does not show NSA’s stable curve below full attention. 松琳 keeps his judgment open: “I’m not sure whether that negative effect comes from removing the compressed output.”
SFT often contains very long prompts, but the loss is calculated only on the answer tokens. If the small number of tokens carrying loss fail to select certain prompt blocks, those blocks receive no gradient. The MoBA report describes this as sub-optimal SFT performance; at bottom, it is sparse routing combined with sparse supervision.
The solution is to switch the final 3 layers to global/full attention, allowing the upper layers to generate dense gradients across all tokens. Because MoBA adds no parameters, full and sparse attention can be switched freely; the design’s original pursuit of minimalism therefore creates room for this repair.
松琳 initially described 90% sparse and 10% full as part of the training process, then corrected himself on the spot and said it was an ablation. The experiment shows that hybrid training can improve performance, but does not establish that the production model must train at that ratio. The self-correction also preserved his uncertainty when interpreting the paper.
19. MoBA’s Product Deployment Shows That the Minimalist Route Is Not a Paper-Only Experiment
Kimi pushed its evaluation length to roughly 1 million tokens, with generally strong long-context results; according to its public statements, MoBA has already been deployed in the Kimi product. 松琳 takes this as evidence that it works in practice and has at least entered a real product setting.
The design philosophies of NSA and MoBA are therefore clear: DeepSeek accepts architectural complexity and forced sharing for hardware efficiency, while Kimi tries to avoid introducing modules and parameters, then uses indexing engineering and attention switching to solve the problem. “One is the brutal beauty of hardware; the other is the elegant algorithm of minimalism.”
Their optimization targets are not fundamentally different: both want prefill and decoding to be as fast as possible. The difference is that NSA enters the acceleration zone earlier, while MoBA preserves more head-level freedom and can switch seamlessly back to full attention. Which is better still needs to be tested under unified training conditions.
20. Linear Attention Compresses the Full History into a Fixed-Size Matrix State
After moving into his own research territory, 松琳 derives linear attention by removing Softmax. Using the associativity of linear operations, keys and values can first be accumulated through outer products, after which the current query reads from the result; the history is encoded as a fixed-size d×d matrix-valued hidden state.
Compared with LSTM’s vector hidden state, this matrix state provides state expansion and has substantially greater capacity. Each decoding step only needs to update and read a fixed state, so both space complexity and per-step time complexity are constant, while total inference complexity grows linearly with sequence length.
The cost is that it no longer stores the complete historical KV cache: all past information is compressed into a finite state. 松琳 points out that hidden-state size directly determines an RNN’s memory capacity, which both forces the model to learn compression and means it cannot losslessly remember an arbitrarily long history.
21. The Chunkwise Algorithm Turns Linear Attention from Theoretically Efficient into GPU-Efficient
Training directly with recurrence would be serial token by token like a traditional LSTM, while outer products and matrix-vector multiplications would make it difficult to fully utilize Tensor Cores. Rewriting it into a fully parallel attention form, however, would return training cost to quadratic growth with sequence length.
The chunkwise algorithm divides the sequence into chunks and calculates only the hidden state at the end of each chunk. Within a chunk, parallel attention calculates the local contribution; across chunks, the same historical state serves the queries in the entire chunk in batch. The computation can use matrix multiplication and is a mathematically equivalent transformation, not an approximation.
Chunk dimension once again acts as a batch-like dimension, following the same hardware principle as NSA’s use of heads to create matrix multiplications. With a fixed chunk size, training complexity remains sub-quadratic while preserving GPU parallelism.
松琳 gives video generation as a typical use case: once several minutes of video are converted into a sequence, the token count can easily reach the millions, quickly overwhelming full attention. Chunkwise linear attention may therefore have a structural advantage.
22. Modern Linear Attention Has Used Decay and Selectivity to Fix the Weaknesses of Early Models
Early linear attention performed poorly in language modeling, creating an industry-wide impression that “linear attention doesn’t work.” 松琳 believes that both model quality and kernel efficiency have improved rapidly over the past 2 years, and that prior beliefs need to be updated.
RetNet is a classic related work; its fixed forgetting rate gives different positions little selectivity. Mamba 2 makes decay data-dependent, allowing each position to determine its own decay. 松琳’s assessment is that Mamba’s emphasis on selectivity and LSTM gating share “essentially the same idea”—“new bottles for old wine.”
From the perspective of linear attention, Mamba 2’s State Space Duality, block decomposition, and related training forms can all be explained more intuitively through equivalent transformations. GLA, Lightning Attention, xLSTM, and other modern models can also be placed within an attention-with-decay framework or a more general state-update framework.
23. MiniMax-01 Uses a 7-to-1 Hybrid Mix to Combine Compressed Memory with Precise Retrieval
MiniMax-01 schedules 7 layers of Lightning Attention and 1 layer of Softmax attention for every 8 layers, repeating the module 10 times for a total of 80 layers. Linear layers handle most sequential modeling, while a small number of full-attention layers preserve the ability to access historical tokens.
松琳 describes the complementarity vividly: “attention is a bit like flipping through a book; an RNN is a bit like the human brain.” An RNN’s fixed capacity forces the model to extract compressible patterns, while Softmax attention stores the full KV cache and can directly retrieve original information for needle-in-a-haystack and similar tasks.
Pure linear attention may be weak at retrieval; the hybrid approach repairs this with a small number of attention layers. RNNs also carry sequential information natively, allowing some hybrid architectures to rely less on or even eliminate RoPE, thereby avoiding certain position-encoding extrapolation problems.
The MiniMax report says the hybrid architecture scales better than pure Softmax attention and expands the model to a size claiming GPT-4o-level performance. What impressed 松琳 was the scale, not the hybrid concept itself: “This is the first time this hybrid architecture has been scaled up to this size.”
24. Architectural Innovation, Open-Source Papers, and Systems Capability Are Becoming Part of the Same Corporate Competition
Scaling MiniMax-01 depended on solid infra: expert parallelism, pipeline parallelism, and possibly linear-attention sequence parallelism were all essential engineering optimizations. Multiple related models had already validated hybrid architectures, and attention layers accounting for roughly 10%–20% had also been examined in ablation studies; MiniMax’s 1/8 ratio is close to that range.
The 3 routes cannot be ranked by final benchmark results alone because model scale, data, and training procedures differ. In terms of risk, 松琳 believes sparse attention is safer because it retains the full KV cache: “Needle in a haystack is definitely possible,” so the floor is relatively stable. MiniMax may be taking the greatest architectural risk, but earlier results at smaller scale mean the 7:1 hybrid is not a blind gamble.
松琳 ultimately ranks NSA first, MoBA second, and MiniMax-01 third. Even though he himself researches linear attention, he was not surprised that the hybrid could scale. What truly changed his research interests was NSA outperforming full attention; he says he may conduct research on Dynamic Sparse Attention going forward.
The larger backdrop is that pretraining data scaling may be approaching a bottleneck, pushing the industry toward test-time scaling and a renewed search for gains from lower-level architecture. 松琳 calls DeepSeek’s persistence in architectural innovation “very valuable” because it is a case of “risk and opportunity coexisting.” Future directions could include further changes to RoPE, contextualized position encoding, using RNNs to carry positional information, and perhaps waiting for Long Convolution to “make a comeback.”
Papers and open source have therefore become commercial signals. 松琳 says “the world has suffered from closed-source companies for too long”; challengers can most easily build technical influence through open source, potentially supporting investor confidence. He cites Qwen’s open-source release and Alibaba’s rising stock price as an example, linking the 2 developments.
His view on talent flows is similarly conditional: when Chinese large models lagged in the past, overseas students naturally went to Silicon Valley to learn. If Chinese teams have now caught up, or even surpassed overseas technology in some areas, domestic teams will become more attractive to overseas students. “Talent always moves toward places with higher talent density.”