143: What Is DeltaNet, Used by Alibaba and Kimi? | A Conversation with 杨松琳 on New Advances in Linear Attention

Summary

• DeltaNet’s industrial value lies not in inventing linear attention, but in turning the Delta Rule from 2021 into a scheme that can be trained in parallel and scaled. 杨松林’s later work solved the problem of training recurrent updates efficiently, after which Gated DeltaNet combined DeltaNet with Mamba-2-style decay. “We didn’t invent anything particularly new,” but the new algorithms have revived the potential of older technology.
• Qwen3-Next and Kimi Linear are risk-validation steps before linear attention enters flagship models, not proof of the final winner. Their disclosed sizes are 80B and 48B, respectively, and both use hybrid architectures combining linear and Full Attention; 杨松林 relayed that K3 “very likely” uses KDA, while Qwen3-Next could lead to Qwen3.5. The real question is whether they can preserve inference efficiency, long-context performance, and Agent capabilities after scaling up.
• Hybrid architectures trade off capacity against cost: Full Attention stores every historical token, while linear layers compress history into a fixed-size state. The former is more reliable on long contexts but makes KV Cache grow without bound; the latter is cheaper at inference but may forget because its capacity is fixed. Replacing roughly 75% of the layers with RNN-like linear layers could reduce KV Cache by about three-quarters and support larger batch sizes.
• MiniMax’s move from M1’s Lightning Attention back to M2’s Full Attention highlights risks in multi-hop reasoning and evaluation, and may reflect weak early linear modules rather than a disproof of the linear route. The host cited M1 at 456B parameters with 45.9B activated; MiniMax said its MMLU and other metrics were not materially different from Full Attention, while 杨松林 argued that such tasks are short-range and that A→B→C→D multi-hop reasoning is more sensitive to Attention. He also believes MiniMax is still testing hybrid architectures and mentioned the possibility of being overly cautious after an earlier setback.
• Sparse Attention is stronger at the single-layer level, while the long-term advantage of hybrid linear architectures is reducing most of the KV Cache. Dynamic sparse methods retain the full KV Cache and only reduce reads and computation at each step; once the context is long enough, storage itself remains the bottleneck. Linear layers’ “theoretical flaw”—a fixed state size—is also precisely what enables faster inference, and the next step may be to combine Sparse Attention directly with KDA-type linear layers.
• Even with unlimited compute, Full Attention may not be the unique optimum if high-quality data remains scarce. 杨松林’s conditional view is that unlimited data plus unlimited compute calls for Full Attention; with abundant compute but constrained data, linear or hybrid architectures with inductive biases such as locality may be more data-efficient, with the advantage potentially appearing first in post-training and RL.
• Another option embedded in DeltaNet is state tracking: it may maintain code variables or Agent state with shorter CoT, but large-scale validation is still required. Citing computational complexity, 杨松林 said Softmax/Full Attention belongs to TC⁰, while DeltaNet is NC¹-complete; the latter may therefore be better suited to recurrence and state tracking. The near-term tests are not small benchmarks, but K3-, Qwen3.5-scale models and new evaluations capable of predicting Agent performance during pre-training.

Deep dive

1. Attention Is Fundamentally About Mixing Information Across a Sequence

• 杨松林 defines Attention as “a processing machine for context”: for the current token to use preceding text, it must aggregate information from different positions along the sequence dimension and then form a representation for predicting the next word.
• Full Attention directly establishes pairwise interactions between any 2 tokens, so complexity grows quadratically with sequence length; it is only one implementation of sequence mixing, not synonymous with every attention mechanism.
• Linear attention no longer requires every pair of tokens to interact directly. Early methods removed softmax and used mathematical transformations to rewrite the computation as an RNN-like recurrence, allowing inference to scale linearly with sequence length.

2. 2021’s DeltaNet Was Overlooked Because It Trained Slowly and Performed Weakly

• The 2020 paper “Transformers are RNNs” showed the relationship between linear attention and recurrent formulations; the 2021 paper “Linear Transformers Are Secretly Fast Weight Programmers” introduced DeltaNet from the perspective of fast weight programming.
• The Delta Rule changed how fast weights are updated, with the goal of improving in-context retrieval: the context first provides key-value associations, after which the model should retrieve the corresponding value using the key.
• The DeltaNet of that period lacked an efficient training algorithm, while the original implementation used an explicit step-by-step loop and was slow. Modern network components were also not yet in place, leaving it both slow and relatively weak in performance.

3. Modern Network Components Revived Linear Attention

• Short convolution lets each token see more than itself by absorbing information from roughly 3 preceding tokens; 杨松林 calls it “a very important module” in modern linear attention, and Mamba subsequently popularized the design.
• Output normalization and output gates also gradually became standard. Combined with RoPE and simple decay, they materially improved even relatively basic linear attention; RetNet and Lightning Attention followed this path.
• 杨松林 distinguishes the network from the operator: output gating is an external architectural improvement, while Gated Attention can still use GQA and be trained with FlashAttention. Changing the update rule means “cutting into” the core operator of sequence mixing.

4. Parallel Algorithms Turned DeltaNet into a Scalable Module

• Expanding a linear recurrence produces cumulative products of transition matrices; DeltaNet’s central challenge was computing this term efficiently. 杨松林’s paper “Parallelizing Linear Transformers with the Delta Rule over Sequence Lengths” introduced a parallel algorithm across sequence lengths, making the Delta Rule scalable to train.
• The subsequent Gated DeltaNet can be viewed as a combination of DeltaNet and Mamba-2: it retains the stronger Delta update while adding decay, similar to a forget gate.
• A fixed-size RNN hidden state cannot store the entire history, so it must forget selectively. Otherwise, trying to remember everything in finite space would eventually overwhelm it.

5. KDA Refines Forgetting from Group-Level to Dimension-Level Control

• The original Gated DeltaNet used relatively coarse decay: if the hidden dimension is 128, all 128 channels share the same forgetting rate. Kimi Delta Attention gives each dimension its own decay.
• Fine-grained control lets some dimensions forget quickly while others retain long-range information. 杨松林 therefore summarizes KDA as a combination of GLA and DeltaNet, and Gated DeltaNet as a combination of DeltaNet and Mamba-2.
• In transition-matrix terms, basic linear attention starts from the identity matrix, while DeltaNet becomes identity plus low rank. KDA relaxes the identity into a learnable diagonal, producing diagonal plus low rank; RWKV-7 uses a similar DPLR form.

6. Qwen and Kimi Have Chosen the Delta Rule, but Flagship Validation Is Not Complete

• The Qwen team compared global attention, hybrid attention, sliding-window attention, Mamba-2, and Gated DeltaNet on an apples-to-apples basis. 杨松林 relayed that Gated DeltaNet worked best and subsequently entered Qwen3-Next.
• Kimi Linear’s KDA retains the Delta Rule and only upgrades the granularity of decay. Qwen3-Next is 80B and Kimi Linear is 48B; at the time, flagship models such as Kimi K2 had still not adopted linear attention.
• 杨松林 considers training smaller models first a standard industrial risk-reduction strategy: “You can’t validate it immediately at a very large scale; otherwise, failing twice would basically bankrupt you.”
• On the next steps, he offered only source-backed predictions: the Kimi team said in a Reddit AMA that K3 would “very likely” use KDA, while public comments from the head of Qwen led him to believe Qwen3-Next could carry over into Qwen3.5.

7. Hybrid Architectures Use Full Attention to Cover Linear Layers’ Capacity Ceiling

• A pure linear model has only a fixed-size state at each layer; once the number and width of layers are fixed, total capacity is constant. As the sequence keeps growing, capacity does not, so performance may hold up on short text but eventually degrade sharply on long contexts.
• Full Attention stores an independent KV Cache for every token, so the state grows linearly with history and is not constrained by the same fixed capacity. The cost is that every step must read an increasingly large cache.
• Inference is often limited by memory bandwidth. Smaller models can tolerate Full Attention, but at larger model sizes and longer contexts, reading an unbounded KV Cache becomes prohibitively expensive.
• Hybrid architectures therefore retain a small number of Full Attention layers as a backstop and replace most layers with linear recurrence. If roughly 75% of the layers are replaced, KV Cache can fall by approximately three-quarters while supporting larger batches.

8. MiniMax Returned to Full Attention, with Multi-Hop Reasoning as the Core Concern

• The host cited MiniMax M1 at 456B parameters with 45.9B activated, using Lightning Attention; M2, released on October 27, returned to Full Attention, creating the contrast of “one side jumping out of the pit while the other rushes into it.”
• MiniMax said Lightning Attention was not materially different from Full Attention on MMLU and other metrics. 杨松林 argued that MMLU is closer to short-range in-context learning and can be handled by a relatively short sliding window, making it a poor test of long-range mixing.
• Multi-hop reasoning requires relationships to be composed across A→B, B→C, and C→D. Full Attention can establish point-to-point links directly; once a linear layer compresses history into an ambiguous state, it may no longer know what it has retained.
• M2 was built primarily to advance agentic AI, and an Agent’s interleaved thinking creates many cross-step dependencies, so returning to Full Attention is logically defensible. MiniMax has only “temporarily” stepped back, however, and is still testing hybrid architectures.

9. MiniMax’s Results May Also Reflect a Weak Module and a Weak Evaluation System

• 杨松林 believes Lightning Attention is merely a basic linear update combined with coarse, input-independent decay—“a very weak linear attention module”—and cannot directly represent Gated DeltaNet or KDA.
• His sharper assessment is that M1’s ability to scale into the hundreds of billions may indicate that its internal evaluation pipeline did not adequately cover long-range dependencies and multi-hop reasoning; if the team only looked at MMLU, architectural risk could easily remain hidden.
• 程曼祺 asked why Qwen and Kimi were still willing to bet on hybrid architectures despite also targeting Agents. 杨松林 did not endorse those bets; he only speculated that the companies use different internal benchmarks and said the question ultimately requires validation on flagship open-source models.
• The conclusion therefore remains open: “We still have to see whether Qwen and Kimi can open-source a relatively flagship model.” Current evidence is limited to smaller-scale systems, and the performance question cannot yet be declared solved.

10. Evaluation Determines Whether New Architectures Expose Their Flaws Before the Money Runs Out

• 杨松林 noted that multi-hop benchmarks such as BBH may be optimized against specifically while having limited relevance to real Agent scenarios; performing well “on the benchmark” does not guarantee reliability on commercial tasks.
• Waiting until the entire chain—pre-training, post-training, and RL—is complete before evaluating would take too long and cost too much. Architecture teams need credible signals of Agent capability at the base-model stage.
• 杨松林 mentioned Tencent’s attempt to construct base-model Agent tasks so teams do not have to wait until full post-training to test them. He called evaluation “a very important thing,” saying better process metrics would directly accelerate architectural iteration.
• The differences among Qwen, Kimi, and MiniMax may reflect not only model choices but also what their respective evaluation sets prioritize: multi-hop reasoning, long-range dependency, or other capabilities.

11. Sparse Attention Preserves Information; Linear Attention Actually Reduces State

• At the single-layer level, 杨松林 believes dynamic Sparse Attention is “definitely stronger”: it preserves the full KV Cache and selects relevant tokens from it. Since the Full Attention map is already highly sparse, accurate selection can approach its performance.
• Linear attention’s fixed state size is a theoretical limitation on expressive capacity, but “it is both a theoretical flaw and the source of inference acceleration,” because the state that must be stored and read is minimal.
• Dynamic sparsity reduces the KV reads and activations at each step, but not the total size of the KV Cache. Once the context is long enough, storage capacity itself becomes the bottleneck, and Sparse Attention may not solve it.
• Hybrid’s advantage is therefore not merely fewer computations. It makes the cache of most RNN layers nearly negligible on long contexts, allowing larger batch sizes and more users to be served simultaneously.

12. DeepSeek Is Pushing Sparse Selection from Blocks to Individual Tokens

• Early dynamic sparsity often selected blocks to enable contiguous reads and take advantage of hardware, but blocks are also a constraint. 杨松林 gave the example that selecting 512 individual tokens is theoretically more flexible than selecting 16 blocks of 32 tokens each.
• He described the indexer in DeepSeek Sparse Attention as first using a lightweight quadratic-complexity module to estimate global relevance, then selecting the top-k preceding tokens for each token. Because it does not need the exponential operation in softmax, it can be accelerated with FP8 matrix multiplication.
• The approach does not start from pre-training; it enters at mid-training. DeepSeek-V3.1 is first converted to a form of MQA, after which the original Attention distribution is distilled into the indexer for initialization.
• The problem then shifts to selection quality and gradient transmission: why block selection is inaccurate and how to train token-level sparsity stably both require answers. As for proactively discarding part of the KV Cache, 杨松林 said he was “not very sure” that mature work already exists.

13. Unlimited Compute Does Not Automatically Make Full Attention the Winner

• 杨松林 first set the conditions: “With unlimited data and unlimited compute, I would definitely just use Full Attention.” It lacks a strong inductive bias, but with enough data it can learn patterns freely.
• If only compute is unlimited while high-quality data remains fixed, the constraint shifts from compute to data. Full Attention may be less data-efficient, while linear architectures with a locality prior and greater emphasis on nearby tokens may learn faster.
• Post-training and RL data are typically scarcer than pre-training data, so hybrid’s performance advantage “might” appear first in those stages; this is a possibility to test, not an established conclusion.
• Abundant compute also allows teams to search for the minimum number of Attention layers required by each task and replace the rest with linear layers, rather than choosing between pure Full and pure Linear.

14. DeltaNet May Trade Shorter Reasoning Chains for State Tracking

• 杨松林 says he is only “halfway competent” on theory, but cites circuit complexity: Softmax Attention, or Full Attention, belongs to TC⁰, while DeltaNet is NC¹-complete, suggesting the latter may be better suited to state tracking.
• If a Transformer needs a long CoT to track state, a model with recurrent state updates might reach the same result with a shorter reasoning chain and reduce inference costs.
• In coding, variable names and values change continuously, requiring the model to maintain their internal state. A web Agent likewise needs to remember which pages it opened, which actions it executed, and where it currently is in the workflow.
• This does not contradict the concern around multi-hop reasoning: hybrid may underperform on some multi-hop tasks while benefiting state tracking. 杨松林 repeatedly adds the caveat—“very possibly”—but says large-scale experiments are required.

15. The Next Generation of Update Rules Will Target Greater Expressiveness and Parallelizability

• Short convolution, normalization, and gating around the network have been repeatedly validated. 杨松林 believes the larger opportunity now lies in update rules: beyond the Delta Rule, can researchers find recurrent mechanisms that are both expressive and efficiently parallelizable?
• Delta Product attempts to let each step perform multiple rounds of gradient descent rather than just one; Mixture of Memory explores treating the RNN state as memory that can be routed similarly to MoE.
• Latent Hybrid Attention leaves nearby tokens to sliding-window attention while compressing more distant history into a linear state, seeking to preserve local precision and long-range compression at the same time.
• What he most wants to see is a working combination of Sparse Attention and Linear Attention—for example, DSA mixed with KDA—so that the remaining global Full Attention can also be replaced and the cost of long contexts reduced in both computation and KV Cache.

16. Architecture Research Must Cross Only One Verifiable Risk at a Time

• Teams currently mix Linear and Full Attention rather than jumping directly to Sparse plus Linear, not because the latter lacks appeal, but because “architecture research is a slow process; it can’t take such a big step in one go.”
• The safer sequence is to retain Full Attention and validate hybrid architectures first. Once larger flagship models show that the approach still works after scaling, researchers can try replacing the remaining global layers with Sparse Attention.
• 杨松林 opposes inserting multiple crazy changes at the outset: change one variable at a time, validate it thoroughly, then move to the next. That is how “solid progress” can accumulate.
• The clearest near-term indicators are therefore still K3 and Qwen3.5, not small-model leaderboards. “Everyone can let the bullets fly a little longer.”

17. The Practical Interpretation of the Bitter Lesson Is to Ask Whether It Can Scale

• On the debate over whether models should be given human-designed priors, 杨松林 believes there are “a hundred interpretations among a hundred people” of the Bitter Lesson. Large language models themselves are one expression of the Bitter Lesson, while injecting human prior knowledge into a model is also a form of prior.
• His standard is more engineering-oriented: a method must first be scalable. For efficiency, it must be hardware-friendly and runnable in large-scale training; for performance, it cannot work only in small shallow networks and disappear in deeper ones.
• The most frustrating outcome in architecture research is when a change looks promising at small scale but produces zero benefit after scaling. Instead of debating the right amount of prior in the abstract, the goal is: “Everyone can test whether it works; if it is, it is, and if it isn’t, it isn’t.”
• High-quality data is already constrained, making inductive bias practically relevant again. Whether it can beat Full Attention still has to be decided by scale experiments and task-level evaluation.

18. The Householder Insight Came from Years of Cross-Disciplinary Accumulation, Not a Single Flash of Insight

• 杨松林 began thinking about parallelizing DeltaNet around September 2023 and had the algorithm by roughly March 2024. The breakthrough was recognizing that the transition matrix resembled a Householder matrix and finding the classical WY algorithm for cumulative multiplication.
• WY rewrites matrix multiplication as accumulation. Once he saw the form, he realized: “Isn’t this exactly the form of linear attention?” He then combined Householder multiplication with the blockwise algorithm for linear attention.
• The connection drew on his master’s work with matrix methods, QR decomposition, and GPU-parallel parsing, as well as his later understanding of linear attention operators. What may have looked like a sudden idea after “thinking for a long time” was actually a gradual buildup.
• He believes current models could potentially derive the algorithm independently if given the right prompt. Because the parallel result is verifiable, RL with verifiable rewards might also be used to search for it. As for when AI for science will surpass humans, he admits he has “no real idea.”

19. FLA Links Research, Open-Source Products, and Talent Flows into One Chain

• FLA drew inspiration from FlashAttention: many low-level ideas had existed for years, but broad adoption required an easy-to-use interface, plug-and-play kernels, and continuous responses to community needs. Being “good enough to use” is itself a condition for research to enter industry.
• FLA therefore focused on implementing multiple linear-model layers and Triton kernels. When users requested variable-length training, the team systematically added variable-length support to the library.
• Core contributor 张宇 previously maintained the parallel parsing library SuPar, then co-built FLA with 杨松林. After joining Kimi, he became one of the authors of KDA. As 杨松林 put it, Kimi wanted to build hybrid architectures, so “rather than start from scratch, they could just move over a team that had already built this well.”
• In response to Bloomberg’s suggestion that credit for DeltaNet should go to Jürgen Schmidhuber, 杨松林 emphasized that he used the name DeltaNet from the 2021 work and never claimed to have invented DeltaNet. His contribution was “the method for making DeltaNet train in a scalable way.”

20. Fast Weight May Unite Long Context and Continual Learning as One Problem

• 杨松林 considers continual learning important but still an open question without a settled answer. If a model can learn through ongoing interaction, it could reduce periodic retraining and support deeper personalization.
• Current knowledge has roughly 3 possible destinations: pre-training compresses information into FFN weights, context engineering places it in the context and KV Cache, and external memory stores it outside the model. There is still no consensus on where information should live.
• Test-time training treats each token as a training sample, performs gradient descent on the objective, and updates fast weights in real time. A rapidly changing weight matrix can also be viewed as an RNN hidden state, creating a form of duality between the two.
• If this route works, long contexts could be compressed into a more compact weight matrix while allowing the model to continuously update itself from new tokens. 杨松林 says it “looks quite promising,” but still presents it as a potential path rather than a solved problem.