INT8, FP8, and INT4: Choosing a Format for Your Hardware

When optimizing machine learning workloads, engineers often treat quantization as a simple reduction in bit-width. This is a misconception. Quantization is a mapping problem: you are compressing a continuous distribution of values into a discrete set of representable numbers. The choice of format—INT8, FP8, or INT4—is not merely about saving memory bandwidth; it is about selecting a number system that aligns with the physical capabilities of your accelerator and the statistical properties of your model’s activations. For infrastructure engineers, mastering these distinctions is key to effective hardware optimization, ensuring that computational resources are utilized efficiently without compromising model integrity.

This article is Part 2 of the "Quantization in Practice" series.

For infrastructure and ML engineers, the decision matrix has shifted dramatically with the introduction of NVIDIA’s Hopper and Blackwell architectures. While INT8 has been the industry standard for inference for years, the native support for FP8 in modern tensor cores has introduced a critical third option that bridges the gap between precision and throughput. INT4 remains a niche tool for extreme compression. Understanding the mechanism-level differences between these formats is essential for selecting the right hardware-software stack.

The Hardware Context: Hopper vs. Blackwell

To understand which format to choose, you must first understand how the hardware processes them. The evolution of NVIDIA’s Tensor Cores illustrates why FP8 has become a focal point.

In the NVIDIA Hopper architecture (H100), Tensor Cores primarily support FP16, BF16, and INT8. While Hopper introduced hardware support for FP8, it was not the primary focus of the initial generation’s tensor core design for general matrix multiplication (GEMM). The Hopper Tensor Cores perform mixed-precision operations efficiently, but the dominant path for high-throughput training has been BF16/FP16, with INT8 reserved largely for inference where precision loss is acceptable.

NVIDIA Blackwell architectures change this dynamic. The B100 and GB200 architectures introduce native, high-throughput support for FP8 in Tensor Cores. Specifically, Blackwell’s Tensor Cores can execute FP8 matrix multiplications at speeds comparable to or exceeding INT8 operations on previous generations, depending on the specific workload and sparsity settings. This hardware-level acceleration means that FP8 is no longer a "soft" optimization requiring complex emulation or software-only tricks; it is a first-class citizen in the compute pipeline.

This hardware shift forces a re-evaluation of INT8. While INT8 remains supported, the performance ceiling for FP8 on Blackwell is higher, and the precision ceiling is significantly better. For engineers designing new training pipelines, the question is no longer "Can we use FP8?" but rather "Do we need the extra precision of FP8 over INT8, or is INT8 sufficient for our inference needs?"

INT8: The Symmetric Workhorse

INT8 quantization maps 32-bit floating-point values to 8-bit signed integers. The mechanism is straightforward but introduces specific constraints. In most ML frameworks, INT8 quantization is symmetric, meaning the range of values is centered around zero. For example, the range [-127, 127] is mapped linearly to the original floating-point range.

The critical mechanism in INT8 quantization is the scaling factor ($\alpha$). To convert a float $f$ to an int8 $i$, the operation is:

$$i = \text{round}(f / \alpha)$$

To convert back:

$$f_{recov} = i \times \alpha$$

The value of $\alpha$ determines the granularity of the quantization. If $\alpha$ is large, many small float values map to zero, causing a "clipping" effect where subtle gradients or activations are lost. If $\alpha$ is small, larger values overflow the INT8 range [-127, 127].

Per-Tensor vs. Per-Channel Scaling

Engineers must choose between per-tensor and per-channel scaling. Per-tensor scaling uses a single $\alpha$ for the entire weight matrix. This is computationally cheap but often leads to significant accuracy degradation in large models because weight distributions are rarely uniform across channels. Per-channel scaling assigns a unique $\alpha$ to each output channel. This preserves the dynamic range of each neuron independently, maintaining accuracy at the cost of increased metadata storage (one $\alpha$ value per output channel).

INT8 is highly effective for inference. The memory bandwidth savings are substantial (4x reduction compared to FP32), and the compute throughput on modern GPUs is well-optimized. However, INT8 is rarely used for training LLMs from scratch. The limited dynamic range of INT8 makes it difficult to capture the small gradient updates required for stable convergence in deep networks.

FP8: The Precision Bridge

FP8 introduces a floating-point representation with 8 bits, based on formats proposed by NVIDIA, Arm, and Intel in their 2022 "FP8 Formats for Deep Learning" specification. Unlike INT8, FP8 retains a sign bit, exponent bits, and mantissa bits. The two primary formats used in ML workloads are:

1. E4M3: 1 sign bit, 4 exponent bits, 3 mantissa bits. This format prioritizes precision and is typically used for activations and weights where small values matter.
2. E5M2: 1 sign bit, 5 exponent bits, 2 mantissa bits. This format prioritizes dynamic range and is often used for intermediate calculations or accumulators.

Mechanism of FP8 in Hopper/Blackwell

The key advantage of FP8 over INT8 is its ability to represent a much wider range of values without overflow, thanks to the exponent. However, it sacrifices precision in the mantissa. For example, E4M3 can represent values with a precision of approximately 6% relative to the magnitude, whereas INT8 has a fixed resolution based on its scaling factor.

In Blackwell, the Tensor Cores perform FP8 GEMM operations natively. The hardware accumulates results in FP32 to prevent precision loss during the summation process. This means that while the inputs and weights are in FP8, the accumulation step maintains higher precision, preserving model accuracy.

FP8 is particularly advantageous for training Large Language Models (LLMs). The dynamic range of FP8 allows for larger batch sizes and faster convergence compared to INT8, while still offering a 2x memory bandwidth improvement over BF16. For engineers working on LLM fine-tuning or pre-training on Blackwell hardware, FP8 is often the optimal choice, providing a near-BF16 accuracy profile with half the memory footprint.

INT4: The Extreme Compression Case

INT4 quantization maps values to 4-bit integers, typically using unsigned or signed representations. The range is significantly smaller: [-8, 7] for signed 4-bit integers. This extreme quantization introduces substantial information loss.

Group-Wise Quantization

To make INT4 viable, engineers use group-wise quantization. Instead of a single scaling factor per channel, the weights are divided into groups (e.g., 16 or 32 weights per group), each with its own scaling factor. This allows the quantization to adapt to local variations in the weight distribution, reducing error compared to per-channel INT4.

However, INT4 is rarely used for general-purpose GPU training or inference on standard architectures. The precision loss is often too severe for complex models like transformers, leading to significant drops in perplexity or accuracy. INT4 is more commonly found in specialized hardware, such as NPUs (Neural Processing Units) or edge devices, where memory bandwidth is extremely constrained.

For infrastructure engineers, INT4 is relevant only in specific scenarios: deploying models to edge devices with limited memory, or using specialized compilers that can handle the complex dequantization and group-wise scaling efficiently. On standard NVIDIA GPUs, INT4 does not offer the same hardware acceleration benefits as INT8 or FP8, making it a less attractive option for cloud-based training and inference.

Conclusion: Selecting the Right Format

The choice of quantization format is dictated by the intersection of hardware capabilities and model requirements.

• Choose INT8 if you are running inference on Hopper or earlier architectures, or if you are deploying to edge devices where memory bandwidth is the primary constraint and model accuracy is less sensitive. It is the mature, well-supported standard for high-throughput inference.
• Choose FP8 if you are training LLMs on Blackwell hardware. The native support for FP8 in Tensor Cores provides a sweet spot between the precision of BF16 and the efficiency of INT8. It is the best choice for maximizing training throughput while maintaining model quality.
• Choose INT4 only for extreme compression scenarios, such as edge deployment or specialized hardware accelerators. On standard GPUs, the lack of hardware acceleration and significant precision loss make it less practical than INT8 or FP8.

As hardware evolves, the boundaries between these formats will continue to blur. However, understanding the underlying mechanisms of quantization and the specific capabilities of your accelerator is the only way to make an informed decision. The goal is not simply to reduce bits, but to optimize the trade-off between precision, throughput, and memory bandwidth for your specific workload.