AI Chip Memory Wall and Bandwidth Bottlenecks
Discusses high-bandwidth memory shortages, emerging memory technologies, and semiconductor architectural design constraints.
5 sources · ↑2.1×Voices in this topic
All paragraphs
Oct 26, 2026
Oct 24, 2026
Oct 23, 2026
Oct 22, 2026
Oct 14, 2026
Oct 10, 2026
Oct 8, 2026
Oct 7, 2026
Oct 3, 2026
- Vikram Sekar Vikram Sekar’s central call is that memory—not raw compute—is AI’s deepest systems bottleneck. LLMs leave GPU compute unused because memory and networks cannot feed data quickly enough; a less memory-intensive model architecture could …
- HBM currently offers the best bandwidth-capacity balance, but stacking faces cost and scaling limits. With stacks already at 16 layers, Vikram questions whether 20 or 24 remains logical and highlights wafer-bonded DRAM-on-logic approaches …
Oct 2, 2026
- SemiAnalysis Engrams turn model architecture and memory hierarchy into one co-design problem. They learn 2-gram and 3-gram token embeddings in a hash table, letting the model retrieve familiar meaning instead of rebuilding it through …
- The design reflects a structural Chinese constraint: less HBM forces more aggressive compression, tiering, and asynchronous offload. The speakers connect Engrams to DeepSeek’s earlier work on reducing KV cache and using HBM, and to domestic accelerators whose bandwidth …
- Vera Rubin looks less disruptive operationally than Blackwell but potentially much stronger economically. At a roughly 100-token-per-second operating point, the early discussion suggested an improvement approaching 3×, while high-throughput gains …
- Walter Goodwin Fractile’s core bet is that frontier inference will be constrained by memory bandwidth and memory cost, not merely compute. Goodwin wants to combine DRAM’s capacity and economics with the “many thousands of tokens per second” associated with SRAM-based systems, …
- Fractile abandoned an initial SRAM architecture after concluding that model size and context length would outgrow it. SRAM delivers extraordinary bandwidth but insufficient cost-effective capacity; even fast inference chips may fall back to GPUs for …
- Fractile claims 25 times more bandwidth per chip than an HBM-based design, potentially changing which model architectures are economical. Goodwin’s example is mixture-of-experts sparsity: moving from 1-to-16 toward 1-to-128 or 1-to-256 could save substantial computation, but …
Sep 25, 2026
- P Equity Research P Equity Research expects memory to absorb roughly half of hyperscaler capex, although estimates vary wildly. Forecasts for 2027-28 range from 36% to 73%, while UBS’s $900 billion estimate for next year exceeds this year’s total hyperscaler spending. …
- Long-term agreements can smooth memory earnings, but they cannot repeal cyclicality. Typical deals cover four to five years, SK Hynix reportedly has 50-70% of supply contracted, and Micron disclosed $22 billion of upfront …
- High Bandwidth Flash may complement rather than replace HBM as inference and agent workloads expand. Logan said Grok 4.7 was described at 2.1-2.3 trillion parameters versus 1.5 trillion for model 4.6. HBF could hold model weights or reusable …
- China is unlikely to flood global DRAM immediately, but dismissing its longer-term threat would repeat the industry’s history. CXMT’s limited capacity, roughly 25% HBM3 yield, and technology lag constrain near-term supply, yet domestic demand and rapid equipment …
Sep 21, 2026
- Bubble boi Inference memory demand may be driven more by context and concurrent users than by ever-larger model weights. Bubble boi cited rumored GPT-6 “looped transformers,” sparsity, quantization, pruning, and speculative decoding as evidence that raw weight …
- High-bandwidth flash is technically intriguing but lacks a decisive data-center use case while HBM remains available. Its density is better, but Bubble boi put both HBF and high-end AI flash latency around five microseconds and argued HBF’s total product …
