Direct:
The Circuit Podcast dropped a long interview with Jeremy Werner, Micron's SVP and GM of Core Data Center Business, and DigiTimes is now amplifying the headline thesis: in inference workloads at scale, memory has become the binding constraint, not only GPU compute. Worth unpacking, because Werner laid out the memory hierarchy more cleanly than most vendor talking points do, and the timing is not accidental.
Werner's framing: training and inference use memory in fundamentally different ways. Training learns and forgets, outputting a model. Inference has to keep growing state resident or recompute from scratch every time the context window grows. Tens of millions of concurrent users, each with their own context, each demanding bandwidth. The bottleneck has moved.
He then walked through the hierarchy Micron is selling against, top to bottom: HBM on the package, CPU main memory in the form of standard and SOCAMM DIMMs, expansion memory (high-capacity DIMMs in an optically-linked box accessible by all GPUs, not yet in production), SSD context storage at roughly 1000x HBM capacity but higher latency, and exabyte-scale SSD data lakes at the bottom. Jensen has been pushing this same hierarchy for the last year. Werner's blunt take on where the pain is right now: DRAM and SSD, demand throughout the stack, every new product gets snapped up immediately.
The conviction here is well-supported. Published GPU-level analysis shows large-batch LLM decode is DRAM-bandwidth-bound, with over 50% of attention kernel cycles stalled on memory access. MemVerge's CXL tiering demo with Micron showed a 77% GPU utilization lift on OPT-66B inference just by moving the bottleneck. The thesis isn't a sales pitch dressed up as analysis. It is the architecture.
What's notable for SSD readers is the 6600 ION 245TB Micron started shipping last week, plus the 9650 Gen6 SSD pairing with WEKA's Augmented Memory Grid technology, which extends GPU memory by treating ultra-fast NVMe as a context tier. The pitch is faster time-to-first-token and KV cache offload. This is the same architectural move the memory wall has been demanding for two years, just productized.
Practical redirect for the consumer-SSD crowd: this is why your Gen5 client drive prices haven't moved the direction you wanted. Wafer capacity is being pulled into HBM, into enterprise QLC, into anything that feeds inference. The structural part of that pull does not unwind on a one-year cycle.
Sources:
- DigiTimes, “Memory bottlenecks threaten data-center GPU efficiency as AI inference scales, says Micron SVP,” Amanda Liang, May 11 2026 (paywall stub; full article behind paywall and used only for confirming the headline thesis)
- The Circuit Podcast, “Breaking the Memory Wall: Micron's Strategy for the AI Era,” Jeremy Werner interview, early May 2026 (referenced via secondary summaries on Bitget News and BigGo Finance; no direct podcast access)
- Micron investor release, “Industry-Leading 245TB Micron 6600 ION Data Center SSD Now Shipping,” May 5 2026
- Micron / GlobeNewswire, “Micron Unveils Portfolio of Industry-First SSDs to Power the AI Revolution,” July 29 2025 (WEKA Augmented Memory Grid attribution via Ajay Singh quote)
- ServeTheHome, “Micron 9650 PCIe Gen6 SSD Announced,” July 2025; “Micron 6600 ION 245TB SSD Announced,” May 2026
- Blocks & Files, “Micron's new SSD replaces disk for fast access storage,” May 2026
- DataCenterDynamics, Micron 6600 ION shipping coverage, May 2026
- arXiv 2503.08311, “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference,” 2025
- MemVerge/Micron GTC 2024 CXL tiering announcement
- Baseten engineering blog, “Why GPU utilization matters for model inference”
Reddit: https://ift.tt/iZVMcF1