About LLM Compressor

QuIP and SpinQuant-style Transforms

The newly added QuIPModifier and SpinQuantModifier allow you to quantize models after injecting hadamard weights into the computation graph, reducing quantization error and greatly improving accuracy recovery for low bit-weight and activation quantization.

DeepSeekV3-style Block Quantization Support

Allows for more efficient compression of large language models without needing a calibration dataset. Quantize a Qwen3 model to W8A8.

FP4 Quantization - now with MoE and non-uniform support

Quantize weights and activations to FP4 and seamlessly run the compressed model in vLLM. Model weights and activations are quantized following the NVFP4 configuration. See examples of FP4 activation support, MoE support, and Non-uniform quantization support where some layers are selectively quantized to FP8 for better recovery. You can also mix other quantization schemes, such as INT8 and INT4.

Llama4 Quantization Support

Quantize a Llama4 model to W4A16 or NVFP4. The checkpoint produced can seamlessly run in vLLM.

Large Model Support with Sequential Onloading

As of llm-compressor>=0.6.0, you can now quantize very large language models on a single GPU. Models are broken into disjoint layers which are then onloaded to the GPU one layer at a time. For more information on sequential onloading, see Big Modeling with Sequential Onloading as well as the DeepSeek-R1 Example.

Axolotl Sparse Finetuning Integration

Seamlessly finetune sparse LLMs with our Axolotl integration. Learn how to create fast sparse open-source models with Axolotl and LLM Compressor. See also the Axolotl integration docs.