What is Intel Gaudi 3? A portrait of the AI accelerator

The Intel Gaudi 3 is a powerful AI accelerator designed specifically for demanding AI workloads. Gaudi 3 is manufactured using the 5-nanometer process, has 64 tensor cores and offers twice as much FP8 performance and four times the AI computing power of its predecessor. This makes Intel’s Gaudi 3 ideal for inference tasks and training large AI models.

What are the performance features of Intel Gaudi 3?

With Gaudi 3, Intel is setting new standards in terms of performance and energy efficiency. The AI accelerator is based on the architecture of Gaudi 2, but offers significantly more computing power, a higher memory bandwidth and better energy efficiency. The following overview summarises the most important performance features of Intel Gaudi 3:

• FP8 computing power: The Gaudi 3 achieves an FP8 computing power of 1.835 PFLOPS. Its predecessor achieved just over 0.8 PFLOPS, which means that the performance for FP8 calculations has more than doubled.
• BF16 computing power: In BF16 calculations, the Intel Gaudi 3 also achieves 1.835 PFLOPS, which represents a fourfold increase in computing power compared to the Gaudi 2.
• Network bandwidth: Bi-directional network bandwidth has been doubled to 1200 gigabits per second, enabling faster communication between nodes in AI cluster systems.
• HBM capacity and bandwidth: With its HBM memory of 128 gigabytes, the Gaudi 3 offers 50 percent more memory bandwidth than the previous generation. The HBM bandwidth of 3.7 terabytes per second corresponds to an increase of 33 percent.

The Intel Gaudi 3 has two compute dies (special computing units) that contain 64 tensor processor cores and 8 MMEs (matrix multiplication engines for parallel processing). The 24 RDMA NIC ports, each with 200 gigabits per second, ensure fast communication via standardised Ethernet networks.

What are the advantages and disadvantages of Intel Gaudi 3?

Using an AI accelerator of the Gaudi 3 generation has various advantages. The most important of these include:

• High computing power: With 1,835 PFLOPS of FP8 and BF16 performance, Intel’s Gaudi 3 offers tremendous performance similar to the level of the much more expensive NVIDIA H100. According to an Intel press release, the in-house AI accelerator even outperforms the NVIDIA flagship in some areas.
• High energy efficiency: The Gaudi 3 AI accelerators are manufactured using the 5-nanometer process (by TSMC), which enables a higher power density. This reduces power consumption and lowers operating costs in data centres.
• Cost-effective AI scalability: With Intel Gaudi 3, systems can be flexibly scaled vertically and horizontally, which is particularly beneficial for complex deployments.
• Support for open standards: As Gaudi 3 supports open standards, the AI accelerators can be flexibly integrated into existing IT infrastructures. This makes companies more independent in their choice of AI platforms.

However, the AI accelerators also have notable disadvantages. Although the Intel Gaudi 3 has first-class performance, the high-end chips from NVIDIA offer even better performance on the whole. Why does this matter? Because companies active in the AI field have so far tended to opt for the most powerful rather than the most cost-efficient solution. As a result, the Intel Gaudi 3 is less common than AI accelerators from NVIDIA, whose ecosystem benefits from broad support from AI development teams.

Which areas of application is Intel Gaudi 3 best suited to?

Intel Gaudi 3 was developed specifically for compute-intensive AI workloads and is particularly suitable for inference tasks that require high parallel processing and memory bandwidth. Typical workloads include text generation with large language models (LLMs), image generation and speech synthesis. Thanks to its high inference speed and optimised FP8 architecture, Gaudi 3 enables powerful and energy-efficient processing of generative AI models. However, there are other areas of application. These include:

• Basic training of large AI models: Gaudi 3 makes it possible to process large data sets efficiently. The AI accelerators are therefore ideal for training AI models — such as neural networks for machine learning or transformer models such as GPT and LLaMA — from scratch.
• Image processing and computer vision: Thanks to its high computing power, the Intel Gaudi 3 is able to process complex image data in real time. This also makes the AI accelerator suitable for applications such as security surveillance or industrial automation.
• GPU servers and AI clusters in data centres: The Intel Gaudi 3 can be used for GPU servers to provide the computing power required for AI training and inference tasks.

What are the possible alternatives to Intel Gaudi 3?

There are various AI accelerators that can be considered as alternatives to Intel Gaudi 3. One of the best-known alternative options and competitor products is the NVIDIA H100. While the Intel accelerator is ideal for inference applications, the H100 offers high-end performance for AI and data science use cases. Another frequently chosen Gaudi 3 alternative is the NVIDIA A30, which combines a high level of performance with an affordable price.