What are AI ac­cel­er­at­ors and how do GPUs, TPUs, NPUs and LPUs differ?

An AI ac­cel­er­at­or is spe­cial­ised hardware designed to run neural networks faster and more ef­fi­ciently than con­ven­tion­al CPUs. While CPUs are built to handle a wide range of tasks flexibly, GPUs, TPUs, NPUs and LPUs are primarily optimised for high-through­put matrix, tensor and multiply–ac­cu­mu­late op­er­a­tions, which are fun­da­ment­al to deep learning.

Key Takeaways ⓘ

The key facts about AI ac­cel­er­at­ors at a glance:

• AI ac­cel­er­at­ors are spe­cial­ised chips for the matrix, tensor and multiply–ac­cu­mu­late op­er­a­tions that con­ven­tion­al CPUs handle only in­ef­fi­ciently.
• The GPU is the most versatile class: massively parallel, with tensor cores, high memory bandwidth and a mature software ecosystem around CUDA.
• The TPU is a Google ASIC whose systolic arrays reduce data movement, and it is used mainly through Google Cloud.
• The NPU sits in the system-on-chip of smart­phones and laptops and targets local inference at very low power con­sump­tion.
• Groq’s LPU uses large amounts of on-chip SRAM and compiler-con­trolled execution for par­tic­u­larly low latency in token-by-token gen­er­a­tion.
• What matters is not peak per­form­ance but which stage of the AI lifecycle an ar­chi­tec­ture serves: training, cloud inference, on-device AI or latency-critical language models.

Why matrix op­er­a­tions require spe­cial­ised chips

Modern neural networks generate much of their com­pu­ta­tion­al load through linear algebra, primarily matrix mul­ti­plic­a­tions and multiply–ac­cu­mu­late op­er­a­tions, in which values are mul­ti­plied and then added together. In trans­former models in par­tic­u­lar, as well as in many con­vo­lu­tion­al networks, this involves very large numbers of similar cal­cu­la­tions that can be processed ef­fi­ciently in parallel.

Con­ven­tion­al CPUs are only partly optimised for this type of workload. They are highly flexible, can handle complex program logic and respond quickly to in­di­vidu­al tasks. With AI models, however, a different priority often comes into play: per­form­ing as many simple arith­met­ic op­er­a­tions as possible in parallel while moving data to the computing units quickly enough. This is where AI ac­cel­er­at­ors come in. Their hardware is designed less for general-purpose program logic and more for high-through­put tensor and matrix op­er­a­tions.

Different classes of AI ac­cel­er­at­or have emerged for these workloads:

• GPUs excel at massively parallel com­pu­ta­tion and are widely used for both training and inference.
• TPUs are designed spe­cific­ally for tensor op­er­a­tions, which are performed in large numbers by modern AI models.
• NPUs are in­creas­ingly in­teg­rated into smart­phones, laptops and other devices to run AI models directly on the device.
• LPUs are designed to provide par­tic­u­larly fast and pre­dict­able inference for large language models.

The dif­fer­ences between these ar­chi­tec­tures therefore go beyond raw computing power. They also lie in which stage of the AI lifecycle they are best suited to, whether that is compute-intensive training, efficient on-device execution or latency-sensitive inference for large language models.

GPU (Graphics Pro­cessing Unit)

The GPU is one of the key found­a­tions of modern AI ac­cel­er­at­or tech­no­logy. It was ori­gin­ally developed for graphics pro­cessing, where large numbers of pixels, surfaces and shader programs need to be processed sim­ul­tan­eously. GPUs therefore contain a large number of parallel computing units that operate according to the SIMT principle. This allows many threads to run at the same time while applying the same in­struc­tion to different data.

By the mid-2000s, GPUs already offered con­sid­er­able computing power, but using it for general-purpose tasks was cum­ber­some. De­velopers often had to express cal­cu­la­tions in graphical terms, such as shaders, textures or pixel op­er­a­tions.

With the in­tro­duc­tion of CUDA in 2006/2007, NVIDIA made GPUs much more ac­cess­ible for general-purpose computing beyond graphics. This helped turn graphics hardware into a platform for General-Purpose Computing on GPUs, or GPGPU. GPUs sub­sequently became widely used for training neural networks because they combine two major ad­vant­ages: very high parallel computing per­form­ance and a mature software ecosystem built around CUDA, cuDNN, optimised libraries and direct support in frame­works such as PyTorch and JAX.

Modern data centre GPUs no longer consist solely of con­ven­tion­al CUDA cores. They also include spe­cial­ised tensor cores that can execute matrix op­er­a­tions par­tic­u­larly ef­fi­ciently. These units support reduced-precision numeric formats such as FP16, FP8 and even FP4. Although these formats offer lower precision than tra­di­tion­al 32-bit floating-point numbers, they can provide sig­ni­fic­antly higher through­put and greater energy ef­fi­ciency. This is par­tic­u­larly important for large AI models.

The key char­ac­ter­ist­ics of GPUs include:

• Massively parallel ar­chi­tec­ture with many computing units designed to perform similar op­er­a­tions on large volumes of data.
• High memory bandwidth, often provided by HBM memory in data centre GPUs.
• Tensor cores for par­tic­u­larly efficient matrix mul­ti­plic­a­tion using reduced-precision numeric formats.
• Strong software ecosystem built around CUDA, cuDNN, optimised libraries and broad framework support.
• High flex­ib­il­ity for training, fine-tuning, sim­u­la­tions and a wide range of inference workloads.

GPUs are among the most versatile AI ac­cel­er­at­ors. They support both inference and compute-intensive model training, combining high through­put and memory bandwidth with a mature software ecosystem.

TPU (Tensor Pro­cessing Unit)

The TPU is an ap­plic­a­tion-specific in­teg­rated circuit (ASIC), meaning a chip designed for a specific purpose. Google developed TPUs spe­cific­ally for AI workloads, par­tic­u­larly the large matrix op­er­a­tions involved in training and running neural networks.

The main dif­fer­ence compared with a GPU is the degree of spe­cial­isa­tion. Despite its high level of par­al­lel­ism, a GPU remains a re­l­at­ively flexible processor designed to support a wide range of workloads. A TPU is more spe­cific­ally optimised for tensor and matrix op­er­a­tions. This makes it less versatile, but allows it to perform certain AI com­pu­ta­tions par­tic­u­larly ef­fi­ciently. The main benefits are higher through­put and greater energy ef­fi­ciency per com­pu­ta­tion.

At the heart of the TPU ar­chi­tec­ture is the Matrix Multiply Unit (MXU), which uses a systolic array made up of many multiply–ac­cu­mu­late units. Data moves from one unit to the next in step with the clock, reducing the need to re­peatedly write in­ter­me­di­ate results to memory and read them back. This limits data movement, which can be just as important as raw computing power for AI ac­cel­er­at­or per­form­ance.

TPUs are closely in­teg­rated with the Google ecosystem. Programs for TPUs are compiled into ex­ecut­able code for the hardware using XLA. While early TPUs were closely as­so­ci­ated with Tensor­Flow, newer gen­er­a­tions are also widely used with JAX and in­creas­ingly with PyTorch/XLA. TPU7x (Ironwood), for example, supports JAX and PyTorch but no longer Tensor­Flow. For external customers, TPUs are widely available as cloud resources through Google Cloud. Google is also expanding beyond its hosted cloud in­fra­struc­ture to provide TPUs directly to selected en­ter­prise customers for de­ploy­ment in their own data centres.

The key char­ac­ter­ist­ics of TPUs include:

• ASIC design for spe­cial­ised AI workloads rather than general-purpose pro­gram­mab­il­ity.
• Systolic arrays for efficient matrix mul­ti­plic­a­tion with reduced data movement.
• High through­put per watt for suitable tensor and matrix op­er­a­tions.
• Close in­teg­ra­tion with the Google Cloud ecosystem through XLA, JAX and PyTorch/XLA.
• Par­tic­u­larly well suited to large-scale training and inference workloads in the cloud.

NPU (Neural Pro­cessing Unit)

The NPU brings AI ac­cel­er­a­tion out of the data centre and directly onto the end device. Today, NPUs are in­teg­rated into smart­phones, laptops, wearables and many embedded systems, including cameras, sensors and in­dus­tri­al equipment. Their aim is not to provide maximum computing power at any cost, but to enable local inference with the lowest possible power con­sump­tion. In other words, AI functions can run directly on the device without every request having to be sent to a data centre.

The key char­ac­ter­ist­ics of NPUs include:

• Spe­cial­ised ac­cel­er­at­or block within a system-on-chip, alongside the CPU and GPU.
• Optimised for local inference rather than large-scale model training.
• Very low power con­sump­tion, making them suitable for mobile devices, battery-powered operation and ap­plic­a­tions where low heat gen­er­a­tion is important.
• Typical use cases include image re­cog­ni­tion, speech pro­cessing, live subtitles, camera effects, local assistant functions and smaller gen­er­at­ive models.

Unlike data centre GPUs, NPUs are not primarily focused on peak per­form­ance measured in TFLOPS. A common per­form­ance metric is TOPS, or tera op­er­a­tions per second. This indicates how many trillions of simple arith­met­ic op­er­a­tions per second an NPU can perform. These figures usually refer to integer-based, highly optimised cal­cu­la­tions, often using INT8 and, depending on the platform, even lower-precision formats.

LPU (Language Pro­cessing Unit)

The LPU is the most spe­cial­ised type of ac­cel­er­at­or. It addresses the challenge of latency-critical inference, which is par­tic­u­larly important for large language models. While training and the model’s prefill stage can be heavily par­al­lel­ised, text gen­er­a­tion itself is autore­gress­ive. The model generates one token at a time, with each new token depending on the preceding context.

Parallel pro­cessing still takes place within each token step, as the model’s matrix op­er­a­tions can be dis­trib­uted across many compute units. However, token gen­er­a­tion itself remains se­quen­tial because each new token depends on the tokens generated before it. This shifts the bot­tle­neck away from peak compute per­form­ance alone. For token-by-token gen­er­a­tion, the speed at which weights, in­ter­me­di­ate results and the KV cache move between memory, compute units and, where necessary, multiple chips becomes crucial. In con­ven­tion­al GPU systems, this can create a ‘memory wall’, where the compute units are fast enough but spend time waiting for data.

Groq is the most prominent example of this approach. In December 2025, NVIDIA entered into a non-exclusive licensing agreement for Groq’s LPU tech­no­logy and hired several members of the Groq team. Groq remained an in­de­pend­ent company, while GroqCloud continued to operate sep­ar­ately. In 2026, NVIDIA in­teg­rated the Groq 3 LPU into its Vera Rubin platform. The NVIDIA Groq 3 LPX rack combines 256 LPU ac­cel­er­at­ors and is designed to work alongside Vera Rubin NVL72 for latency-sensitive inference workloads. Depending on the con­fig­ur­a­tion, GPUs and LPUs can handle different stages of inference, combining the flex­ib­il­ity and through­put of GPUs with the low-latency token gen­er­a­tion of LPUs.

Groq’s LPU uses a very different ar­chi­tec­ture from con­ven­tion­al GPUs. Rather than relying heavily on external HBM memory, large caches and dynamic runtime schedul­ing, it uses large amounts of on-chip SRAM and de­term­in­ist­ic, compiler-con­trolled execution. Data flows are planned in advance, making execution highly pre­dict­able and enabling in­di­vidu­al requests to be processed with par­tic­u­larly low latency.

The key char­ac­ter­ist­ics of LPUs include:

• Optimised for LLM inference rather than general-purpose training.
• Very low and highly pre­dict­able latency for token-by-token text gen­er­a­tion.
• Large amounts of on-chip SRAM instead of a strong reliance on external HBM memory.
• De­term­in­ist­ic, compiler-con­trolled execution rather than dynamic runtime schedul­ing.
• Per­form­ance metrics such as output tokens per second and time to first token rather than only TFLOPS or TOPS.

The trade-off for this degree of spe­cial­isa­tion is reduced flex­ib­il­ity. An LPU is not a general-purpose training platform like a GPU, but is designed primarily for inference. It is par­tic­u­larly useful wherever language models need to respond quickly, such as in chatbots, voice agents, real-time as­sist­ants, agent systems or ap­plic­a­tions that make many short model calls in rapid suc­ces­sion. It primarily supports open-weight and other openly available models. As the models available on inference platforms can change rapidly, specific model IDs should be checked regularly. At present, Groq focuses par­tic­u­larly on GPT-OSS models and selected Qwen models.

How do GPUs, TPUs, NPUs and LPUs compare?

The four classes of AI ac­cel­er­at­or cannot be compared solely on the basis of peak per­form­ance. What matters most is what each ar­chi­tec­ture is optimised for, whether that is maximum through­put during training, efficient cloud inference, on-device AI or par­tic­u­larly low latency for language models. The following table sum­mar­ises the key dif­fer­ences: