Innovators
1 Curated Dialogues
Myron Xie
Key Views & Dialogues
Ep. 030 - Long Live the Short King: Why 4-hi HBM Wins (Memory)
Rubin Ultra shifted from the GTC-previewed 1TB of HBM4E to 192GB of 8-high HBM4, below Blackwell Ultra’s and vanilla Rubin’s 288GB, as HBM supply rations TSMC-secured logic. Four-high saturates the interface at far lower cost per bandwidth and can yield roughly twice as many cubes as eight-high, but model-size growth is the key risk while memory tightness is not expected to ease within this decade.
View summary (7)
- Rubin Ultra is now understood to have shifted from the 1TB of HBM4E previewed at GTC to 192GB of 8-high HBM4—below Blackwell Ultra’s and vanilla Rubin’s 288GB. Myron Xie says the primary driver is supply rationing: NVIDIA and customers including Google and Broadcom have more TSMC logic secured than their HBM allocation can support in 12-high form, so using 8-high cubes lets them ship more accelerators. Jordan Nanos flags this as the first time, in his experience tracking NVIDIA, that a flagship has less memory capacity than its predecessor.
- The economic core: HBM bandwidth is essentially fixed per cube while suppliers charge for capacity, making 4-high the best dollar-per-bandwidth configuration when its capacity is sufficient. Four DRAM layers saturate the cube’s interface, so 8- or 12-high costs almost double or triple for the same bandwidth. “Paying for 4-high only, when all you really need is that bandwidth … is almost a free lunch.”
- Capacity pressure is much lower because parameter counts have not scaled as aggressively as expected. Xie compares Llama 3.1 405B at FP8, which used roughly 60% of a Hopper HGX node’s 640GB, with Kimi K3 at 2.88 trillion parameters, which he describes as a bit more than six times larger but which MXFP4 reduces to about 8% of a GB200 NVL72’s aggregate HBM. Deployment domains grew much faster, with Rubin Ultra’s NVL576 adding another 8x. Meanwhile post-training and standard inference, which are more bandwidth-sensitive, now dominate a larger share of frontier-lab compute, while classic pretraining has declined.
- Four-high can roughly double harvestable cubes versus 8-high and more than double them versus 12-high. Per-layer yield losses compound with stack height, and power delivery is easier in a 4-die stack. The bottleneck therefore shifts toward leading-edge logic, substrates and power. Lower HBM demand also frees DRAM wafers for conventional server DRAM, important while server sockets are being de-specced and CPUs handling tool calls may be poorly utilized without enough memory. Nanos frames 4-high as maximizing “tokens per HBM wafer,” analogous to tokens per watt and per dollar.
- No relief on the memory crunch is expected soon: Xie’s timeline is “not within this decade.” Cleanrooms are the main short-term blocker, followed by equipment such as EUV tools whose ASML supply chain—including suppliers making “very smooth mirrors”—cannot be expanded quickly. “You can’t just click your fingers and add wafer capacity,” even as SK hynix, Samsung and Micron benefit from high prices.
- The bear case is that exploding model sizes would make higher stacks worthwhile. SemiAnalysis modeled a model three times the size of Kimi K3, where batching makes the throughput gain from 8- and 12-high “probably worth the cost.” Xie’s response is that techniques such as looped transformers add compute depth without adding parameters; he says GPT-6 Astra is basically confirmed to use them. He also views the labs’ strong demand for 4-high as evidence that their road maps do not anticipate aggressive parameter growth.
- Expect more SKU segmentation. Meta already has a custom MI450 version with 8-high HBM instead of 12-high, and Xie says additional segmentation makes sense when supply-chain resources are tight. The current 288GB Rubin configurations are not the reset; the change is associated with Rubin Ultra, the next version after the current generation, arriving this time next year or into 2028.