LLM Compressor 0.8.0: Extended support for Qwen3 and more

The LLM Compressor 0.8.0 release introduces significant enhancements to quantization workflows, extended support for Qwen3 models, and improved accuracy recovery. This release features five notable additions that we'll explore in detail.

1. Multiple modifiers during oneshot

LLM Compressor now supports the use of multiple modifiers within a single oneshot compression run. This capability allows practitioners to apply different modifiers to specific submodules—such as combining AWQ and GPTQ for W4A16 quantization—while only having to apply a dataset once through a single calibration run. This feature enables enhanced support for non-uniform quantization, giving users the flexibility to account for varying sensitivity across layers and more options for post-training quantization (PTQ) experimentation.

Example: Non-uniform quantization with multiple modifiers.

from transformers import AutoModelForCausalLM
from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQMapping, AWQModifier
from llmcompressor.modifiers.quantization import GPTQModifier
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
#  Configure the quantization algorithm to run.
#   * quantize self_attn layers to W8A8 with GPTQ
#   * quantize mlp layers to W4A16 with AWQ
#       only include mappings pertaining to target layers
recipe = [
    GPTQModifier(targets=r"re:.*self_attn\.(k]
    
oneshot(
model=model,
       dataset="HuggingFaceH4/ultrachat_200k",
       recipe=recipe,
       max_seq_length=MAX_SEQUENCE_LENGTH,
       num_calibration_samples=NUM_CALIBRATION_SAMPLES,
       pipeline="sequential" 
)

With the example above, users are able to apply multiple quantization schemes using both AWQ and GPTQ, producing a mixed-precision model that is directly runnable in vLLM.

 "quantization_config": {
    "config_groups": {
        "group_0": {
            "format": "int-quantized",
            "input_activations": {
                "actorder": null,
                "block_structure": null,
                "dynamic": true,
                "group_size": null,
                "num_bits": 8,
                "observer": null,
                "observer_kwargs": {},
                "strategy": "token",
                "symmetric": true,
                "type": "int"
            },
            "output_activations": null,
            "targets": [
                "re:.*self_attn\\.(k],
            "weights": {
                "actorder": null,
                "block_structure": null,
                "dynamic": false,
                "group_size": null,
                "num_bits": 8,
                "observer": "minmax",
                "observer_kwargs": {},
                "strategy": "channel",
                "symmetric": true,
                "type": "int"
            }
         },
        "group_1": {
            "format": "pack-quantized",
            "input_activations": null,
            "output_activations": null,
            "targets": [
                "re:.*mlp\\.(down],
            "weights": {
                "actorder": null,
                "block_structure": null,
                "dynamic": false,
                "group_size": 128,
                "num_bits": 4,
                "observer": "minmax",
                "observer_kwargs": {},
                "strategy": "group",
                "symmetric": true,
                "type": "int"
            }
        }
    },
    "format": "mixed-precision",
}

For further details on non-uniform quantization support, see the examples in LLM Compressor.

2. Transforms update: Configurable transforms with variable rotation sizes

Transform-based modifiers (SpinQuantModifier, QuIPModifier) now support a configurable transform_block_size to further customize the Hadamards applied to the model. The transform_block_size determines the size of the Hadamard, removing the requirement for full-sized rotations. This allows practitioners to align Hadamard block sizes with quantization group sizes, improving efficiency and accuracy as smaller Hadamards require less cost at runtime.

Example of using Hadamard size 128 with the QuIPModifier.

from transformers import AutoModelForCausalLM
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform import QuIPModifier
# Select model and load it.
# NOTE: because the datafree pipeline is being used in this
# example, you can use additional GPUs to support larger models
MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")
# Configure the quantization algorithm to run.
#   * apply quip transforms to model in order to make quantization easier
#   * quantize the weights to 4 bit with a group size 128
recipe = [
    QuIPModifier(
        rotations=["v", "u"], transform_block_size=128,
transform_type="hadamard"
    ),
    QuantizationModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"]),
]
# Apply algorithms.
oneshot(model=model, recipe=recipe, pipeline="datafree")

The model produced by the above example will apply online rotations (that is, rotations during runtime). Previously, these online rotations were applied using the dense GEMM. As of the vLLM v0.11 prerelease, you can efficiently apply both full-sized and variable-sized non-random Hadamard rotations using the hadacore kernels. With this update, we're able to see no additional cost to the model's latency when compared to its quantized counterpart with no online rotations, as you can see in the following table.

from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform import SpinQuantModifier
recipe = [
    SpinQuantModifier(
        rotations=["R1", "R2", "R4"],
        transform_block_size=128,
        transform_type="hadamard",
    ),
    QuantizationModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"]),
]

"weights": {
     "actorder": "weight",
     "block_structure": null,
     "dynamic": false,
     "group_size": 128,
     "num_bits": 4,
     "observer": "minmax",
     "observer_kwargs": {},
     "strategy": "group",
     "symmetric": true,
     "type": "int"
}