A chip can only compute on numbers it has in hand. Generating each new token of an answer requires reading the model’s weights from memory, so for large models the processor often waits on memory rather than on its own math units. Researchers call this the “memory wall”: over two decades, chip computing power has grown much faster than the speed of moving data to it.
HBM attacks that problem physically. Several memory dies are stacked on top of each other, connected by thousands of tiny vertical wires, and placed on the same package as the GPU or other accelerator. The short, wide connection moves terabytes per second. The trade-off is cost and supply: HBM is harder to make than ordinary memory, and only a few companies produce it, so it is often cited as a bottleneck in building more AI capacity.
Capacity matters as much as speed. How much HBM an accelerator has decides whether a model fits on one chip or must be split across several, and how much room is left for the KV cache that grows with long conversations. That is one reason quantization, which shrinks model weights, speeds up the memory-bound decode phase of inference.
In episode 28 at 39:09, Sharon Zhou says she would like to see a new model architecture invented on AMD hardware, “taking advantage of the benefits and the differences of AMD hardware, for example, larger HBM.” The episode was recorded at AMD’s Advancing AI event; it is a Season 2 episode sponsored by Galileo, and Conor introduces himself as Galileo’s Head of Developer Awareness (0:05).
Hear it from the guest
“Taking advantage of the benefits and the differences of AMD hardware, for example, larger HBM.”
Quotes lightly edited to remove filler words.
Sources
- Gholami et al.: AI and Memory Wall — Shows that memory bandwidth is increasingly the main bottleneck for serving and training large language models: peak hardware FLOPS grew about 3.0x every two years over 20 years, versus 1.6x for DRAM and 1.4x for interconnect bandwidth; checked October 24, 2026.
- NVIDIA H100 GPU — Product page listing HBM3 memory on H100-class GPUs, including 188GB on the H100 NVL configuration, and about 3 TB/s of memory bandwidth per GPU; vendor specifications; checked October 24, 2026.
- AMD Instinct MI300X accelerators — Product specifications listing 192 GB of HBM3 and 5.3 TB/s peak memory bandwidth; vendor specifications; checked October 24, 2026.
