Compression Schemes

PTQ Compression

Post-training quantization is performed to reduce the precision of quantizable weights (e.g., linear layers) to a lower bit-width. Supported formats are:

Scheme	Description	Data required?	vLLM Hardware Compatibility
W8A8-FP8	8-bit floating point (FP8) quantization for weights and activations, providing ~2X smaller weights with 8-bit arithmetic operations. Uses channel-wise quantization to compress weights to 8 bits, and uses dynamic per-token or static per-tensor quantization to compress activations to 8 bits. Weights scales that are generated on a per-channel basis or generated on a per-tensor basis are also possible. Channel-wise weights quantization with dynamic per-token activations is the the most performant option. W8A8-FP8 does not require a calibration dataset. Activation quantization is carried out during inference on vLLM. Good for general performance and compression, especially for server and batch inference.	No calibration dataset is required, unless you are doing static per-tensor activation quantization.	Latest NVIDIA GPUs (Hopper and later) and latest AMD GPUs. Recommended for NVIDIA GPUs with compute capability >=9.0 (Hopper and Blackwell)
W8A8-INT8	8-bit integer (INT8) quantization for weights and activations, providing ~2X smaller weights with 8-bit arithmetic operations. Uses channel-wise quantization to compress weights to 8 bits using GPTQ, and uses dynamic per-token quantization to compress activations to 8 bits. Weight quantization can be both per-tensor or per-channel for INT8. W8A8-INT8 is good for general performance and compression, especially for server and batch inference. Activation quantization is carried out during inference on vLLM. Activations can be static or dynamic. Additionally, INT8 activations can also be asymmetric. W8A8-INT8 helps improve speed in high QPS scenarios or during offline serving with vLLM. W8A8-INT8 is good for general performance and compression, especially for server and batch inference.	Requires calibration dataset for weight quantization and static per-tensor activation quantization.	Supports all NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators. Recommended for NVIDIA GPUs with compute capability <8.9 (Ampere, Turing, Volta, Pascal, or older).
W4A16	Quantizes only weights to 4-bit integer (INT4) precision, retaining activations in 16-bit floating point (FP16) precision. W4A16 provides ~3.7X smaller weights but requires 16-bit arithmetic operations. W4A16 also supports asymmetric weight quantization. W4A16 provides maximum compression for latency-sensitive applications with limited memory, and useful speed ups in low QPS regimes with more weight compression. The linked example leverages the GPTQ algorithm to decrease quantization loss, but other algorithms like AWQ can also be leveraged for W4A16 quantization. Recommended for any GPU types.	Requires a calibration dataset.	All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators
W8A16	Encodes model weights in 8‑bit integers and activations in 16‑bit integers. W8A16 compression delivers smaller model output size than FP32 and is faster at inferencing on hardware with native 8‑bit integer units. Lower power and memory bandwidth compared to floating‑point.	Requires a calibration dataset.	All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators
NVFP4	4-bit floating point encoding format introduced with the NVIDIA Blackwell GPU architecture. NVFP4 maintains numerical accuracy across a wide dynamic range of tensor values by using high-precision scale encoding and a two-level micro-block scaling strategy. NVFP4 compression generates a global scale for each tensor, along with local quantization scales for groups of 16 elements. Global scale and local quantization scales are generated for weights and activations. You cannot change the group size.	Requires a calibration dataset.	All NVIDIA Blackwell GPUs or later
W8A8-FP8_BLOCK	Uses block-wise quantization to compress weights to FP8 in (commonly 128×128 tiles), and dynamic per-token-group (128) quantization for activations. Does not require a calibration dataset. Activation quantization is carried out during inference on vLLM.	Requires a calibration dataset.	All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators

Sparsification Compression

Sparsification reduces model complexity by pruning selected weight values to zero while retaining essential weights in a subset of parameters. Supported formats include:

Scheme	Description	Data required?	vLLM Hardware Compatibility
2:4 Semi-structured Sparsity	Uses semi-structured sparsity (SparseGPT), where, for every four contiguous weights in a tensor, two are set to zero. Uses channel-wise quantization to compress weights to 8 bits and dynamic per-token quantization to compress activations to 8 bits. Useful for better inference than W8A8-FP8, with almost no drop in its evaluation score. Small models may experience accuracy drops when the remaining non-zero weights are insufficient to recapitulate the original distribution.	Requires a calibration dataset.	Supported on all NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators. Recommended for compute capability >=9.0 (Hopper and Blackwell)
Unstructured Sparsity	Unstructured sparsity quantization zeros out individual weights in the model without enforcing a regular pattern. Unlike block or channel pruning, it removes weights wherever they contribute least, yielding a fine‑grained sparse matrix.	Does not require a calibration dataset.	All NVIDIA GPUs, AMD GPUs, TPUs, CPUs, and other accelerators