Kimbo Chen

Arcee AI’s move into pre-training responds to a hard product ceiling: sub-20B customer work could only be as good as Llama, Mistral or Qwen’s base, while many customers’ legal and compliance teams stopped wanting work based on Chinese pre-trained bases. Owning the stack connects customization to known training data, especially “that last 10% of pre-training.” Lucas Atkins says Arcee’s current ambition is economic viability for developers—mainly startups—and enterprises, not AGI, though he says “never say never.”

Open weights are chiefly a sovereignty and unit-economics tool: if frontier-lab models become costlier, slower and more token-hungry without improving a narrow workflow, builders can reclaim margins with a fine-tuned 4B–8B model. Lucas’s invoice-processing example makes the call concrete: optimize for tool use, speed and cost, then add new skills yourself—“control is the number one” reason.

Lucas treats transparency as risk infrastructure, arguing that AI may repeat the mobile and social-media boom’s combination of enormous upside and damaging tail effects. If the people able to monitor, analyze and mitigate fundamental risks were only 200 safety researchers at OpenAI and Anthropic, he would rather “the risks be very transparent,” with mitigations easy to implement, understood and properly diffused through open models.

Arcee’s perceived talent disadvantage may be smaller than its compute disadvantage because researchers value visible work and broad ownership. Lucas says “talent’s actually easier than compute” and would keep the research team below 30 people even with “$2 trillion,” using opinionated, good-faith debate and full-stack exposure as organizational leverage. Compute is harder because it ultimately depends on funding and revenue; he says modularity will be important as compute demand and training paradigms shift.

Arcee does not intend to match frontier labs dollar-for-dollar; it aims to serve the “80% of economically viable tasks” that reward reliability, speed and low cost rather than extreme intelligence. In Lucas’s illustrative split, a next frontier model might improve 30% on FrontierMath but frontend generation only 5.5%: “Our goal isn’t to catch up. Our goal is to undercut.”

Trinity distributes execution across Arcee, DatologyAI and Prime Intellect, while B300 availability enabled a target of one month of pre-training instead of three. The trade-off was immature at-scale tooling and benchmarking—particularly sparse kernels—so the ecosystem around DeepSeek-like super-sparse models trained on Hopper became the “golden example” for throughput comparisons.

🔗 Original source & video: Training a 400B Model on 2,048 Blackwell GPUs for $20M | Researcher Conversations at GTC

DeepSeek V4 changes the inference workload, not merely the weights. Its 1 million-token context combines compressed sparse attention, highly compressed attention, an embedding compressor, and sliding windows for roughly a 100X reduction in KV-cache usage versus a standard MOA model; it also has more total but fewer active parameters than V3. Kimbo Chen called these “very aggressive innovations.”

“Day zero support” is a starting line, not usable peak performance. vLLM and SGLang received early access under NDA while NVIDIA did not, and V4 Pro’s changed mHC dimension caused an initial NVIDIA hiccup; AMD began with FP8 but no native FP4. Replacing Torch fallbacks with AITER or Triton, then stacking smaller improvements, can raise endpoint capacity and economics.

MegaMoE’s 1.5-1.73X claimed speedup comes from treating kernel boundaries as negotiable. It avoids register-to-HBM round trips and overlaps communication with computation inside one megakernel, “aggressively reducing the latency.” The trade-off is substantial engineering and memory pressure, with less benefit when very large training batches already hide launches and communication.

The vLLM–SGLang rivalry is productive infrastructure competition, not a winner-take-all benchmark. InferenceX avoids turning every result into a runtime war, but Cam Quilici’s conclusion was still “TL;DR competition is good”: each implementation forces faster optimization, while two backends give labs, serving providers, and downstream RL libraries alternatives when priorities or support diverge.

Huawei Ascend crossed from paper claim to demonstrated DeepSeek V4 performance. Bryan Shan described the release profiles as “very elegant” and the kernels as sophisticated, suggesting Huawei probably had much longer access than vLLM or SGLang to optimize. CANN’s code is open source and readable on GitCode; Bryan praised its documentation, meetups, and what he described as weekly calls with engineers. Jordan contrasted this with TensorRT-LLM/Dynamo and AITER/Mori as closed-source proprietary libraries. Conflicting rumors about whether Huawei optimization delayed V4 remained unresolved, but “the performance on Huawei at release was real.”

DeepSeek V4 Pro’s initial 75% discount becoming permanent may signal optimization gains and an aggressive Chinese market-share fight. Kimbo connected low pricing to accumulated Chinese systems work such as HCCL and MC², suspected further post-release optimization, and said DeepSeek may be pricing at “probably negative margin or at least just zero margin” while competing in a market where ByteDance has the majority.

InferenceX is moving beyond synthetic chip tests toward actual agentic-system behavior. Its current 8K/1K and 1K/1K fixed-length cases isolate “basically pure chip performance,” but do not test the million-token promise or realistic caching. The planned AgentX benchmark will use internal Claude Code traces and compare Dynamo, KV-block management, and prefill-decode disaggregation because “inference is a systems problem.”

🔗 Original source & video: Ep. 017 - DeepSeek V4 and Huawei Ascend NPU Performance (InferenceX) | Kimbo Chen, Cam Quilici, Bryan Shan, Jordan Nanos

Makora is broadening from automated kernel generation toward a foundation-model-supplier-agnostic “deployment engine for high-performance AI models.” Kernels remain a core component, but the commercial pitch is end-to-end performance across inference servers, training pipelines, reinforcement-learning rollouts, numerics, and heterogeneous hardware. Abdelfattah’s framing: “Our value proposition is that we sell performance.”

Its sequential Monte Carlo speculative-decoding method delivered roughly 5× the SGLang baseline and 2× SGLang speculative decoding with the experimental overlap scheduler—but only in the cited batch-size-one, low-latency regime. It runs N drafts, scores them with the target model, duplicates strong candidates, evicts weak ones, and always accepts K tokens instead of rewinding. Abdelfattah stresses that the method uses more compute and is “fundamentally a lossy method.”

Makora uses its FP4 research to illustrate its performance-focused differentiation as foundation models improve at code generation. Remapping FP4’s redundant second zero can effectively provide “the accuracy of FP5 or FP5-something” at an FP4 memory footprint. For NVFP4 weight-activation quantization, the straightforward NVIDIA implementation needs a sparse compensating pass; AMD can instead upcast through an FP6 path sharing FP4 hardware at the same throughput. “This very niche unused precision FP6 is coming to our rescue.”

Kernel evaluation has itself become a product after an early demo suffered extensive reward hacking. Makora now traces whether generated functions execute correctly, rejects forbidden dependencies, and uses AI to detect reward hacks; the overall pipeline catches maybe 11 exploit classes. Customers submit “a kernel and a problem,” receive a reward signal through an API, and can use it for model training.

Better coding models are a tailwind under Makora’s model-agnostic strategy. The platform tries multiple agents and returns the best kernel; the team is also testing open models such as “Gemma 4” and exploring fine-tuning or specialization for a less expensive alternative to large foundation models. The harder layer is producing code already integrated with data layouts, interfaces, quantization modes, and systems such as GGML or SGLang.

The customer wedge spans hardware vendors, enterprises with private GPU estates and no in-house performance team, and neoclouds seeking a default software layer. Hardware makers can bootstrap support without cultivating a developer network, while enterprises can serve internal models without public APIs. The longer-term bet is to abstract both performance engineering and hardware placement: “the right code for the right hardware.”

🔗 Original source & video: How Makora Generates CUDA Kernels That Beat Hand-Tuned Code | Researcher Conversations at GTC