Model sizes have roughly doubled every year (see figure 1), and GPU memory hasn't come close to keeping up. So when a new frontier open model is released, how do you actually deploy it to serve one user, or perhaps one thousand at a time, on hardware you can realistically get your hands on?
This challenge is exactly why almost every large language model (LLM) running in production today is quantized. By the end of this article, you'll know:
- What quantization actually does to a model
- Where in the inference stack it actually pays off
- Which algorithm and format to pick, and what each one costs you
What is LLM quantization?
When a lab releases a new LLM, its weights usually ship in BF16 (brain floating point): 16 bits for every single number. With billions of parameters, that requires a lot of memory, and it's why the model won't fit on most hardware.
So let's ask the obvious question: Do we really need all 16? Usually not. Quantization maps each weight into a lower-bit format: FP8 (floating point, decimals), INT8 (integers like 3 or -127), INT4, and so on.
Each step down decreases the memory requirements, as illustrated by figure 2. Fewer bits per number, and a smaller model overall. I particularly enjoy this example from Maarten Grootendorst on how quantization reduces the number of bits (colors) to represent the original model, or in this case, cookie (see figure 3).
The result: Less GPU compute and total cost for AI
Let's take Llama 4 Scout, at 109B parameters. It was released at BF16, which is 2 bytes per parameter:
- BF16: 109B × 2 bytes per parameter = ~220 GB → You need three 80 GB GPUs
- INT8/FP8: 109B × 1 byte per parameter = ~109 GB → Two GPUs
- INT4/FP4: ~55 GB → One GPU
It's the same model, but you're running it on 1 GPU instead of 3. That's a 75% reduction in memory footprint, and with the scarcity of hardware these days, it's a big deal. It also means you can consolidate far more models onto the same cluster, which is the whole premise behind running Model-as-a-Service for your developers.
Where inside the model does this happen?
A model is essentially a stack of layers performing massive matrix multiplications in order to produce your result, known as transformer blocks (figure 4). Each block has a self-attention section with 4 linear layers (the Q, K, V, and O projections) and a feed-forward network with 3 more (gate, up, down). These layers are where almost all of the model's parameters live, and almost all of the compute happens. So that's what gets quantized, and exactly what a tool like LLM Compressor operates on.
The embedding layer at the start and the LM head at the end are typically excluded to preserve accuracy (and they're a small fraction of the parameters anyway).
Weights and activations are 2 different wins
There are 2 things inside a linear layer you can quantize, and they pay off in completely different places (see figure 5).
- Weights: The model's learned parameters. Every forward pass pulls them from high-bandwidth memory (HBM) into fast on-chip SRAM. Cutting them from 16 to 8 bits slashes data movement in half. This is a massive latency win for token generation (the decoding phase), which is heavily memory-bound.
- Activations: The tensors flowing through the model. Tensor cores churn through more operations per second when these numbers are lower precision. For example, Hopper GPUs feature dedicated FP8 cores, whereas older Ampere chips rely on INT8. This is a huge throughput win for prompt processing, also known as the prefill phase of inference, which is heavily compute-bound.
So, weight quantization speeds up the data movement side of inference, and activation quantization speeds up the compute side. Quantizing both is what unlocks the full speedup. By the way, the naming is W{bits}A{bits}: Weights, then activations.
| Scheme | What you get | Reach for it when |
|---|---|---|
| W8A16 | Half the weight memory, no tensor core speedup | Memory-bound, or hardware without FP8/INT8 tensor cores |
| W8A8 | Both wins | Hopper or newer, serving real concurrency. The production default |
| W4A16 | 4× smaller weights, most aggressive on size | Fitting a model onto one card, can trade throughput for it |
But what type of model quantization should you actually download?
This is where it gets tricky. Search any popular model on Hugging Face and you'll get hundreds of results (figure 6), from GGUF, NVFP4, AWQ-INT4, MLX-4bit, 4bpw, 5.5bit, AutoRound. They are not interchangeable, and the right answer depends entirely on which inference engine you're running.
- GGUF: llama.cpp, Ollama, LM Studio. Local inference on CPU or a consumer GPU.
- MLX: Apple silicon.
- FP8, NVFP4, AWQ, GPTQ, compressed-tensors: vLLM and data center GPUs.
Does it make the model dumber?
Done correctly, essentially no. But "done correctly" is what I want to emphasize.
Naive quantization, where you round every number down and cross your fingers, does hurt. What works are calibrated techniques like GPTQ, AWQ, and SmoothQuant, which run a small representative dataset through the model to figure out which weights matter most and protect those. All 3 are implemented in LLM Compressor, the open source library Red Hat maintains for producing vLLM-ready checkpoints.
The Red Hat AI team published NVFP4 versions of a big spread of open models this year, from 8B up past 400B, and the recovery numbers are the interesting part (illustrated in figure 7):
- 70B–235B models: ~99% of BF16 accuracy
- ~30B models: 97–99%
- 7B–14B models: ~95–98%
What's quite interesting is that recovery gets better as models get bigger, which is backward from what most people assume, and mixture of experts (MoE) models hold up especially well. If you want the receipts, we ran over half a million evaluations on quantized LLMs, and we dug separately into how quantized models handle long-context tasks as well.
What quantization looks like under load
If you're deploying a model for a team, this is the part that matters most. We benchmarked Llama 3 70B on two H100s with GuideLLM, FP16 versus FP8 weights-and-activations, on a retrieval-augmented generation (RAG) style workload of roughly 1024 tokens in and 128 out (illustrated in figure 8):
- Throughput: 158 → 474 input tokens/sec
- Time to first token under load: Over 30,000ms → about 4,800ms
This comparison was performed with the same GPUs and the same model. But, the difference is entirely what precision, or quantization, you're running at.
Conclusion
What's important to take away is that quantization isn't a sacrifice you make because you may not have enough hardware. If you're using modern algorithms and validate your accuracy against real benchmarks, it's a powerful way to fit a frontier model onto the GPUs you already have, serve far more users, and cut your time to first token by an order of magnitude under load.
If you want to get started, Red Hat AI publishes pre-compressed models on Hugging Face (see figure 9), including Kimi, Qwen, Gemma, and more, along with the recovery benchmarks from native precision to the quantized version, so you can decide whether it works for you. And when you're ready to put one into production, Red Hat AI Inference Server ships vLLM with these optimizations built in, and Red Hat OpenShift AI handles the scaling once a single GPU is no longer enough. Plus, it's all built from open source, from the model layer to the application stack itself.
