Store Scaling The Storage Pyramid With Micron’s Datacenter Memory And Flash SPONSORED CONTENT Consider, for a moment, the current state of AI accelerators and datacenter GPUs. Now, try to imagine this landscape without Micron Technology’s entry into the High Bandwidth Memory HBM market. …
… For instance, we have seen Google do this with its latest TPU 8 XPUs , which use HBM3E memory instead of the more current HBM4, and SambaNova Systems did it with its SN50 RDU , which uses HBM2E memory to keep it cheap and deep. …
… Equally importantly, SambaNova has kept the HBM memory bandwidth roughly the same at 1.84 TB/sec and the capacity of the HBM memory the same at 64 GB even as it doubles up the cores and more than doubles the performance – and even backstepped to HBM2E memory in the SN50 from HBM3 memory used in the… …
… The memory makers can only make HBM so fast, and at the prices and profit margins they are commanding, there is very little incentive for them to try to build HBM capacity any faster than they are doing. …
… Eventually, China will come up with a new memory scheme that it can manufacture – stacked LPDDR memory or Z-Angle memory or a large number of banks of HBM3E memory attached to relatively modest AI compute engines and just scale it all out. …
… An eight-way MI355X system board has enough memory to hold 4.2 trillion parameters in memory in a single instance, which is 5.9X better than the “Antares” MI300X GPU launched in December 2023 , which had 1.5 TB of HBM3 memory. …
… The deep HBM buffers help you cope with momentary congestion, but in the long run, radix might be more important. “With those two value attributes – the buffering and the port capacity in radix – it is up to customer choice to get the shallow buffer, what I call hierarchical switch architectures – … …
… But not only do GenAI training and inference and the other workloads each respective TPU 8 chip is handling have different needs when it comes to processing, they have very different needs for SRAM memory capacity and HBM memory and memory bandwidth. …
… If you wanted to get better balance on HBM capacity and bandwidth against the inherent compute in a socket, you might want to actually gear down the number of GPU chiplets, run them at slightly higher clocks, and ramp up the amount and speed of the HBM memory to get more capacity and bandwidth per … …
… This is why Nvidia championed Micron Technology’s re-entry into the HBM stacked memory business two years ago, when Samsung and SK Hynix owned that business. …