What is Intel Gaudi 3? A portrait of the AI ac­cel­er­at­or

The Intel Gaudi 3 is a powerful AI ac­cel­er­at­or designed spe­cific­ally for demanding AI workloads. Gaudi 3 is man­u­fac­tured using the 5-nanometer process, has 64 tensor cores and offers twice as much FP8 per­form­ance and four times the AI computing power of its pre­de­cessor. This makes Intel’s Gaudi 3 ideal for inference tasks and training large AI models.

What are the per­form­ance features of Intel Gaudi 3?

With Gaudi 3, Intel is setting new standards in terms of per­form­ance and energy ef­fi­ciency. The AI ac­cel­er­at­or is based on the ar­chi­tec­ture of Gaudi 2, but offers sig­ni­fic­antly more computing power, a higher memory bandwidth and better energy ef­fi­ciency. The following overview sum­mar­ises the most important per­form­ance features of Intel Gaudi 3:

• FP8 computing power: The Gaudi 3 achieves an FP8 computing power of 1.835 PFLOPS. Its pre­de­cessor achieved just over 0.8 PFLOPS, which means that the per­form­ance for FP8 cal­cu­la­tions has more than doubled.
• BF16 computing power: In BF16 cal­cu­la­tions, the Intel Gaudi 3 also achieves 1.835 PFLOPS, which rep­res­ents a fourfold increase in computing power compared to the Gaudi 2.
• Network bandwidth: Bi-dir­ec­tion­al network bandwidth has been doubled to 1200 gigabits per second, enabling faster com­mu­nic­a­tion between nodes in AI cluster systems.
• HBM capacity and bandwidth: With its HBM memory of 128 gigabytes, the Gaudi 3 offers 50 percent more memory bandwidth than the previous gen­er­a­tion. The HBM bandwidth of 3.7 terabytes per second cor­res­ponds to an increase of 33 percent.

The Intel Gaudi 3 has two compute dies (special computing units) that contain 64 tensor processor cores and 8 MMEs (matrix mul­ti­plic­a­tion engines for parallel pro­cessing). The 24 RDMA NIC ports, each with 200 gigabits per second, ensure fast com­mu­nic­a­tion via stand­ard­ised Ethernet networks.

What are the ad­vant­ages and dis­ad­vant­ages of Intel Gaudi 3?

Using an AI ac­cel­er­at­or of the Gaudi 3 gen­er­a­tion has various ad­vant­ages. The most important of these include:

• High computing power: With 1,835 PFLOPS of FP8 and BF16 per­form­ance, Intel’s Gaudi 3 offers tre­mend­ous per­form­ance similar to the level of the much more expensive NVIDIA H100. According to an Intel press release, the in-house AI ac­cel­er­at­or even out­per­forms the NVIDIA flagship in some areas.
• High energy ef­fi­ciency: The Gaudi 3 AI ac­cel­er­at­ors are man­u­fac­tured using the 5-nanometer process (by TSMC), which enables a higher power density. This reduces power con­sump­tion and lowers operating costs in data centres.
• Cost-effective AI scalab­il­ity: With Intel Gaudi 3, systems can be flexibly scaled ver­tic­ally and ho­ri­zont­ally, which is par­tic­u­larly be­ne­fi­cial for complex de­ploy­ments.
• Support for open standards: As Gaudi 3 supports open standards, the AI ac­cel­er­at­ors can be flexibly in­teg­rated into existing IT in­fra­struc­tures. This makes companies more in­de­pend­ent in their choice of AI platforms.

However, the AI ac­cel­er­at­ors also have notable dis­ad­vant­ages. Although the Intel Gaudi 3 has first-class per­form­ance, the high-end chips from NVIDIA offer even better per­form­ance on the whole. Why does this matter? Because companies active in the AI field have so far tended to opt for the most powerful rather than the most cost-efficient solution. As a result, the Intel Gaudi 3 is less common than AI ac­cel­er­at­ors from NVIDIA, whose ecosystem benefits from broad support from AI de­vel­op­ment teams.

Which areas of ap­plic­a­tion is Intel Gaudi 3 best suited to?

Intel Gaudi 3 was developed spe­cific­ally for compute-intensive AI workloads and is par­tic­u­larly suitable for inference tasks that require high parallel pro­cessing and memory bandwidth. Typical workloads include text gen­er­a­tion with large language models (LLMs), image gen­er­a­tion and speech synthesis. Thanks to its high inference speed and optimised FP8 ar­chi­tec­ture, Gaudi 3 enables powerful and energy-efficient pro­cessing of gen­er­at­ive AI models. However, there are other areas of ap­plic­a­tion. These include:

• Basic training of large AI models: Gaudi 3 makes it possible to process large data sets ef­fi­ciently. The AI ac­cel­er­at­ors are therefore ideal for training AI models — such as neural networks for machine learning or trans­former models such as GPT and LLaMA — from scratch.
• Image pro­cessing and computer vision: Thanks to its high computing power, the Intel Gaudi 3 is able to process complex image data in real time. This also makes the AI ac­cel­er­at­or suitable for ap­plic­a­tions such as security sur­veil­lance or in­dus­tri­al auto­ma­tion.
• GPU servers and AI clusters in data centres: The Intel Gaudi 3 can be used for GPU servers to provide the computing power required for AI training and inference tasks.

What are the possible al­tern­at­ives to Intel Gaudi 3?

There are various AI ac­cel­er­at­ors that can be con­sidered as al­tern­at­ives to Intel Gaudi 3. One of the best-known al­tern­at­ive options and com­pet­it­or products is the NVIDIA H100. While the Intel ac­cel­er­at­or is ideal for inference ap­plic­a­tions, the H100 offers high-end per­form­ance for AI and data science use cases. Another fre­quently chosen Gaudi 3 al­tern­at­ive is the NVIDIA A30, which combines a high level of per­form­ance with an af­ford­able price.