What is High Bandwidth Memory (HBM) and why is it important for AI GPUs?

High Bandwidth Memory (HBM) is a 3D-stacked DRAM tech­no­logy that sits directly on the GPU package and uses extremely wide memory in­ter­faces to enable data rates of several terabytes per second. By combining high bandwidth, short signal paths and low power con­sump­tion, HBM has become a key tech­no­logy for AI ac­cel­er­at­ors and large language models.

Key Takeaways ⓘ

The key facts about High Bandwidth Memory at a glance:

• HBM is 3D-stacked DRAM placed directly beside the GPU on the same package, which keeps signal paths very short.
• Instead of relying on high clock rates, HBM uses extremely wide memory in­ter­faces with thousands of parallel con­nec­tions.
• From HBM2 to HBM3e, bandwidth rose from around 256 GB/s to more than 1.2 TB/s per stack.
• HBM4 doubles the interface width to 2048 bit per stack and reaches around 2 TB/s according to JEDEC.
• Through-silicon vias and microbump bonding connect the stacked memory dies; the short distance to the GPU cuts energy use per trans­mit­ted bit.
• Compared with GDDR6 and GDDR7, HBM delivers sig­ni­fic­antly more bandwidth per watt in a smaller footprint.
• For large language models, memory bandwidth is often the limiting factor rather than compute power alone.

What is High Bandwidth Memory (HBM)?

HBM stands for ‘High Bandwidth Memory’ and describes a spe­cial­ised memory ar­chi­tec­ture used in high-per­form­ance computers, GPUs and AI ac­cel­er­at­ors. Unlike con­ven­tion­al graphics memory such as GDDR, HBM is not placed sep­ar­ately on the board but directly beside the GPU in the same package. This reduces elec­tric­al signal paths and improves data transfer ef­fi­ciency. HBM also uses extremely wide memory in­ter­faces with thousands of parallel con­nec­tions, allowing large amounts of data to be trans­ferred sim­ul­tan­eously.

AI models rely on this high memory bandwidth because billions of para­met­ers need to be con­tinu­ously trans­ferred between the GPU and memory. HBM therefore combines high per­form­ance with com­par­at­ively low energy con­sump­tion. It is par­tic­u­larly important for training and inference of large LLMs, where memory bandwidth can have a sig­ni­fic­ant impact on overall per­form­ance.

The evolution from HBM2 to HBM3e

The de­vel­op­ment of High Bandwidth Memory (HBM) focuses primarily on con­tinu­ously in­creas­ing memory bandwidth. The key factor here is the so-called pin speed, i.e. the data rate per line.

The key de­vel­op­ment steps of HBM include:

• HBM2: Es­tab­lished HBM as memory for HPC and AI ac­cel­er­at­ors
• HBM2e: Improved bandwidth and higher memory ca­pa­cit­ies
• HBM3: Sig­ni­fic­antly higher transfer rates for AI systems
• HBM3e: Optimised variant with up to 9.6 Gbps pin speed
• HBM4: New gen­er­a­tion with doubled interface width

HBM2e typically achieved transfer rates of around 3.2 to 3.6 Gbps per pin, depending on the im­ple­ment­a­tion. With HBM3, data rates increased sig­ni­fic­antly to around 6.4 Gbps. HBM3e further raises pin speeds to over 9 Gbps, with even higher data rates possible depending on the im­ple­ment­a­tion. As a result, HBM3e can deliver more than 1.2 TB/s per stack.

This de­vel­op­ment is important for AI ar­chi­tec­tures such as the NVIDIA Hopper platform. GPUs like the NVIDIA H100 and NVIDIA H200 use HBM3 and HBM3e re­spect­ively to train and run large language models ef­fi­ciently. Without this memory bandwidth, GPU compute units would often have to wait for data.

Com­par­is­on of HBM gen­er­a­tions

HBM4 and the move to a 2048-bit interface

HBM4 is con­sidered one of the biggest tech­no­lo­gic­al leaps so far in the de­vel­op­ment of High Bandwidth Memory. The most important in­nov­a­tion in HBM is the doubling of the memory interface width from 1024 to 2048 bit per stack. This allows sig­ni­fic­antly more data to be trans­mit­ted in parallel.

While earlier gen­er­a­tions became faster mainly through higher clock speeds, HBM4 now combines high pin speeds with vastly more parallel data lines and very high band­widths. Several HBM4 stacks on a single AI ac­cel­er­at­or can thus enable total data rates in the double-digit terabytes range.

This is par­tic­u­larly crucial for LLMs and GenAI. Large trans­former models move enormous volumes of data between compute units, ac­tiv­a­tions and model weights. Even state-of-the-art GPUs today are often limited by memory bandwidth rather than by sheer computing power alone.

Another important advance concerns the so-called base die. This lowest layer of a High Bandwidth Memory stack controls com­mu­nic­a­tion with the GPU interface. With HBM4, the base die becomes sig­ni­fic­antly more important: it includes more complex logic for control, signal routing and energy ef­fi­ciency. Depending on the man­u­fac­turer and customer re­quire­ments, it can also be produced using more advanced logic processes.

Ad­di­tion­ally, the new HBM improves:

• energy ef­fi­ciency per trans­mit­ted bit
• signal stability at high data rates
• scalab­il­ity for multi-die GPUs
• memory con­nectiv­ity for large AI clusters

3D stacking and TSV tech­no­logy behind HBM per­form­ance

The high per­form­ance of High Bandwidth Memory is based on a spe­cial­ised 3D ar­chi­tec­ture. Multiple memory dies are stacked ver­tic­ally and directly connected to each other in a process known as 3D stacking.

Com­mu­nic­a­tion between the in­di­vidu­al memory layers takes place via so-called through-silicon vias (TSVs). These mi­cro­scop­ic vertical con­nec­tions run through the silicon and enable extremely fast data transfer between the stacked memory chips.

The dies are also connected using microbump bonding. These tiny contact points create dense elec­tric­al con­nec­tions within the stack, forming a compact high-speed memory system.

The key advantage, however, is the physical proximity to the GPU. HBM is placed directly on the same package as the AI ac­cel­er­at­or, meaning data does not have to travel long distances across the mainboard.

This helps reduce several chal­lenges:

• lower signal loss
• lower latency
• less elec­tro­mag­net­ic in­ter­fer­ence
• lower energy con­sump­tion per trans­mit­ted bit

Energy ef­fi­ciency is par­tic­u­larly important. Con­ven­tion­al memory uses com­par­at­ively more energy to transmit signals over longer distances. High Bandwidth Memory reduces this demand and achieves very low pJ/bit values (pi­co­joules per bit). This is crucial for data centres, where AI systems require enormous amounts of elec­tri­city.

HBM vs. GDDR6/7 and how HBM overcomes the limits of con­ven­tion­al VRAM

Con­ven­tion­al graphics memory such as GDDR6 or GDDR7 is primarily designed for discrete GPUs and cost-sensitive high-bandwidth ap­plic­a­tions like gaming, work­sta­tions and certain compute scenarios. The focus there is mainly on high clock rates and cost-efficient memory solutions. For current AI models, however, this concept is in­creas­ingly reaching its limits, because GDDR works with com­par­at­ively narrow memory in­ter­faces and com­pensates for this with very high fre­quen­cies. High Bandwidth Memory takes the opposite approach, using extremely wide in­ter­faces at lower clock rates. This is precisely how HBM achieves sig­ni­fic­antly higher ef­fi­ciency.

These dif­fer­ences are par­tic­u­larly evident in:

• memory bandwidth
• energy con­sump­tion
• space re­quire­ments
• scalab­il­ity
• thermal load

GDDR memory is placed around the GPU on the PCB rather than directly alongside it. This requires longer data paths and can increase energy con­sump­tion. At very high fre­quen­cies, signal loss and heat gen­er­a­tion can also become greater chal­lenges. HBM is much more compact: multiple memory stacks are po­si­tioned directly beside the GPU on a shared in­ter­poser. This enables extremely high band­widths within a small footprint.

For AI ap­plic­a­tions, memory bandwidth is a critical factor. Large language models with hundreds of billions of para­met­ers con­stantly move vast amounts of data. If the GPU can process data faster than the memory can provide it, a bot­tle­neck occurs. This challenge is becoming in­creas­ingly important as modern AI ac­cel­er­at­ors continue to increase in computing power faster than con­ven­tion­al memory tech­no­lo­gies can scale. HBM helps address this lim­it­a­tion.

Energy ef­fi­ciency is another important factor. In large AI data centres, reducing power con­sump­tion is in­creas­ingly important. Compared with con­ven­tion­al GDDR memory, HBM delivers sig­ni­fic­antly more bandwidth per watt.

However, this does not mean that GDDR will disappear. GDDR6 and GDDR7 remain practical and cost-effective solutions for gaming graphics cards and many other ap­plic­a­tions. For high-end AI ac­cel­er­at­ors and large HPC systems, however, HBM has become the leading memory tech­no­logy.

HBM and the next gen­er­a­tion of AI in­fra­struc­ture

HBM has evolved from a spe­cial­ist tech­no­logy into a key found­a­tion of AI in­fra­struc­ture. Without the enormous memory bandwidth provided by High Bandwidth Memory, many current large language models and HPC ap­plic­a­tions would be far more difficult to operate ef­fi­ciently.

This de­vel­op­ment also shows that memory tech­no­lo­gies are becoming just as important as GPU computing power itself. AI ac­cel­er­at­ors are in­creas­ingly con­strained by factors such as memory bandwidth, energy ef­fi­ciency and data movement. Man­u­fac­tur­ers such as SK Hynix, Samsung Elec­tron­ics and Micron Tech­no­logy therefore play a key role in the AI market.

HBM4 rep­res­ents the next gen­er­a­tion of HBM tech­no­logy and, following the pub­lic­a­tion of the JEDEC JESD270-4 standard, reached volume pro­duc­tion in 2026. NVIDIA moved its Vera Rubin platform into full pro­duc­tion at the end of May 2026, Micron is man­u­fac­tur­ing the HBM4 memory designed for it in high-volume pro­duc­tion, and Samsung Elec­tron­ics and SK Hynix are qualified as ad­di­tion­al suppliers.