Ep. 017 - DeepSeek V4 and Huawei Ascend NPU Performance (InferenceX) | Kimbo Chen, Cam Quilici, Bryan Shan, Jordan Nanos

Summary

• DeepSeek V4 changes the inference workload, not merely the weights. Its 1 million-token context combines compressed sparse attention, highly compressed attention, an embedding compressor, and sliding windows for roughly a 100X reduction in KV-cache usage versus a standard MOA model; it also has more total but fewer active parameters than V3. Kimbo Chen called these “very aggressive innovations.”
• “Day zero support” is a starting line, not usable peak performance. vLLM and SGLang received early access under NDA while NVIDIA did not, and V4 Pro’s changed mHC dimension caused an initial NVIDIA hiccup; AMD began with FP8 but no native FP4. Replacing Torch fallbacks with AITER or Triton, then stacking smaller improvements, can raise endpoint capacity and economics.
• MegaMoE’s 1.5-1.73X claimed speedup comes from treating kernel boundaries as negotiable. It avoids register-to-HBM round trips and overlaps communication with computation inside one megakernel, “aggressively reducing the latency.” The trade-off is substantial engineering and memory pressure, with less benefit when very large training batches already hide launches and communication.
• The vLLM–SGLang rivalry is productive infrastructure competition, not a winner-take-all benchmark. InferenceX avoids turning every result into a runtime war, but Cam Quilici’s conclusion was still “TL;DR competition is good”: each implementation forces faster optimization, while two backends give labs, serving providers, and downstream RL libraries alternatives when priorities or support diverge.
• Huawei Ascend crossed from paper claim to demonstrated DeepSeek V4 performance. Bryan Shan described the release profiles as “very elegant” and the kernels as sophisticated, suggesting Huawei probably had much longer access than vLLM or SGLang to optimize. CANN’s code is open source and readable on GitCode; Bryan praised its documentation, meetups, and what he described as weekly calls with engineers. Jordan contrasted this with TensorRT-LLM/Dynamo and AITER/Mori as closed-source proprietary libraries. Conflicting rumors about whether Huawei optimization delayed V4 remained unresolved, but “the performance on Huawei at release was real.”
• DeepSeek V4 Pro’s initial 75% discount becoming permanent may signal optimization gains and an aggressive Chinese market-share fight. Kimbo connected low pricing to accumulated Chinese systems work such as HCCL and MC², suspected further post-release optimization, and said DeepSeek may be pricing at “probably negative margin or at least just zero margin” while competing in a market where ByteDance has the majority.
• InferenceX is moving beyond synthetic chip tests toward actual agentic-system behavior. Its current 8K/1K and 1K/1K fixed-length cases isolate “basically pure chip performance,” but do not test the million-token promise or realistic caching. The planned AgentX benchmark will use internal Claude Code traces and compare Dynamo, KV-block management, and prefill-decode disaggregation because “inference is a systems problem.”

Deep dive

1. V4 buys million-token context by attacking KV-cache traffic

• Kimbo’s headline changes were the 1 million-token context and MegaMoE. Jordan Nanos added that V4 has more total but fewer active parameters than V3: it is “literally a different model, not just an update to the weights.”

• Compressed sparse attention and highly compressed attention both build on DeepSeek Sparse Attention, which came out with 3.2. Sparsifying query-to-key/value access reduces memory reads; an extra embedding compressor shrinks each KV-cache entry, and a sliding window cuts usage further—together, “around a 100X reduction” versus a standard MOA model.

• Cam asked whether V4 inherently favors GPGPU designs over TPU-style systolic arrays. Kimbo rejected that broad premise: both tensor cores and systolic arrays accelerate matmuls; MegaMoE’s launch behavior is the more specifically GPU-tailored feature.

• Bryan’s day-zero lesson: dimensions previously treated as constants can change. V4 Pro used a new mHC dimension versus V4 Flash and earlier V3 iterations, creating an NVIDIA hiccup; vLLM and SGLang had early NDA access, while NVIDIA apparently did not.

2. MegaMoE trades engineering complexity for lower latency

• Kimbo distinguished a megakernel from ordinary fusion: it breaks “the typical kernel boundaries,” avoids round-tripping data between registers and HBM, begins later operations as resources free up, and merges computation with communication for aggressive overlap.

• DeepSeek’s open-source megakernel was validated on NVIDIA GPUs and Huawei Ascend NPUs, with a claimed 1.5-1.73X improvement. Publishing CUDA code still does not let vLLM, SGLang, or TensorRT-LLM benefit on day 1.

• The Hazy Research extreme—fusing an entire Llama 8B model—shows what is possible and why it remains rare. Megakernels are “a lot of engineering work”; at hundreds of millions of training tokens per batch, launch time may already be irrelevant, while aggressive allocation adds memory and potentially thermal pressure.

3. Performance arrives through compounding runtime work

• The article’s time series showed more hardware coming online—including B200, B300, GB200, GB300, and MI355—as support broadened. AMD provided a clear catch-up example: day-zero support was FP8-only, without native FP4. FP4 improved calculation speed and reduced transferred memory, while major SGLang jumps came from replacing Torch fallbacks with AITER or Triton kernels.

• Cam described the real curve as “hard work, tiny iterations, increasing throughput by 5% at a time.” Over a month or two, a runtime advances from PyTorch fallback to custom kernels and then successively better kernels; he said the same progression was beginning with MiniMax-M3.

• InferenceX generally shows vLLM and SGLang separately because a direct contest is not always useful and duplicate submissions consume scarce CI compute. Still, Cam conceded that “a little bit of competition is good,” because both communities move faster.

• Cam framed TensorRT-LLM and AITER as architecture-specific and fast but less portable and not always fully open, while vLLM and SGLang are forkable, user-friendly, and expose OpenAI API specifications. Bryan said InferenceX’s benchmark disclaimer notes that AITER has no customers; Mori is much better than the previous fork, while AITER still needs to close SGLang’s community-development gap.

• Kimbo traced both projects to the same Berkeley lab and acknowledged accusations from each camp of “copy-pasting our code.” Jordan’s pushback: their continued separation is no longer merely historical—customers and downstream RL libraries need choices over feature priority, merges, vendor relationships, and support.

4. Huawei’s software stack now has credible model evidence

• Bryan separated rumor from evidence: V4’s repeated delay was variously attributed to Huawei optimization or better evaluations, and he did not resolve that dispute. The release evidence showed that “the performance on Huawei at release was real,” backed by benchmarks and sophisticated profiles. Their quality suggested Huawei probably had much longer architecture access than vLLM or SGLang to optimize.

• CANN’s implementation is readable on GitCode. Bryan praised its documentation, frequent meetups, and what he described as weekly calls with engineers; Jordan contrasted its open-source code with the closed-source proprietary TensorRT-LLM/Dynamo and AITER/Mori libraries.

• Bryan also said, “if I’m not wrong,” Huawei implemented communication/computation fusion before NCCL’s roughly 2024 release, shortly after the relevant paper. He called the Chinese developers’ velocity “10X” and said Chinese open source would propel it further.

• Kimbo connected Ascend support for Z.AI’s GLM and DeepSeek V4 to a broader accumulation of domestic systems work, including HCCL, MC², and aggressive optimization. He suspected the permanent launch 75% discount reflected further optimization as well as a market-share battle in which DeepSeek may be operating at “probably negative margin or at least just zero margin.”

5. AgentX will test systems rather than isolated chips

• Cam called the current 8K/1K and 1K/1K fixed-sequence tests configurations that “nobody uses,” yet defended them as measures of “basically pure chip performance” without prefix caching. Million-token models require moving up the stack because “inference is a systems problem.”

• AgentX is intended to replay real internal Claude Code traces and compare Dynamo, KV Block Manager variants, and prefill-decode disaggregation. Cam expected it within “the next couple weeks,” alongside several new chips already in the pipeline.

• Jordan also flagged RL-system performance as a future topic: a fast inference runtime matters, but “there’s much more” to making the full RL system perform well.