
Quantization for LLM inference lowers the numerical precision used to store a model's weights, usually converting 16-bit floating-point values into 8-bit or 4-bit formats. The model keeps the same basic structure and parameter count. Its weights just take less memory and less bandwidth to move, which can make inference faster and cheaper to run.
Why care? Loading billions of weights out of GPU memory can be the thing that limits language model inference. Lower precision may let you serve a model on fewer accelerators or a different class of hardware, though the result depends on model size, workload, runtime and hardware support.
What is model quantization?
It's representing weights, activations or both with fewer bits than the original model used. A model trained or stored in 16-bit floating point might be converted to 8-bit integers, 4-bit integers, FP8 or another compact format.
The process maps high-precision values onto a smaller set of representable values. Scales, zero points and grouping methods help approximate the original numbers during computation. That mapping adds rounding error, so you're trading some numerical precision for lower memory use and, potentially, faster execution.
Nothing gets removed. Quantization doesn't drop layers or parameters. In an idealized weight-only quantization comparison, 4-bit weights need roughly a quarter of the raw storage of 16-bit weights. Total runtime memory doesn't fall by the same amount, because activations, KV caches, temporary buffers and serving software use memory too.
Sometimes that's enough to move a model from several GPUs to one, or from a server accelerator to consumer hardware. Whether it fits still depends on parameter count, context length, batch size, concurrent requests, runtime overhead and available memory.
How do post-training quantization and quantization-aware training differ?
Timing is the main difference. Post-training quantization (PTQ) converts a model that's already trained. Quantization-aware training (QAT) exposes the model to reduced-precision behavior during training. PTQ is faster to apply and needs no extra training dataset. QAT costs more compute but can hold quality better at aggressive bit widths.
A typical PTQ process looks like this:
- Start with a trained higher-precision model.
- Analyze or calibrate representative model values.
- Convert values into the target low-precision format.
- Evaluate quality and serving performance on the intended workload.
QAT simulates rounding and restricted numerical ranges during training or fine-tuning, so the model can adjust its parameters to compensate. It earns its keep when a required low-bit deployment loses too much quality under PTQ. The extra training makes it more expensive and more involved to run.
Deployment formats also differ in calibration, weight grouping, runtime support and hardware compatibility. Our comparison of AWQ, GPTQ, and GGUF covers several practical approaches to post-training deployment.

How should teams choose a quantization approach for LLM inference: 8-bit, 4-bit or FP8?
Test it on your own production stack. Measure quality, memory use, latency, throughput and compatibility there. As a rough guide, 8-bit is the conservative choice, 4-bit compresses more with more quality risk, and FP8 gives you floating-point behavior on compatible accelerators.
Eight-bit quantization often stays close to the higher-precision model on common inference tasks, and production hardware and frameworks support it broadly. It cuts weight memory without squeezing values as hard as 4-bit integer formats do.
Four-bit quantization shrinks raw weight storage further, and modern serving frameworks support it. It needs more careful calibration and evaluation, since some models, tasks or layers are more sensitive to reduced precision. Implementations may keep selected operations at higher precision to protect output quality.
Below 4 bits, degradation generally gets harder to control, and the techniques needed to keep quality acceptable get more complex. Our rule of thumb: the smallest format is rarely the most useful or economical one in production.
FP8 keeps a floating-point exponent and mantissa instead of an integer representation. It's increasingly used for training and inference on compatible modern accelerators, because it can cut memory and bandwidth needs while keeping numerical behavior that suits many training workloads. Support still depends on the hardware, framework, kernels and model architecture.

Why can quantization make inference faster?
Fewer bytes per weight. Hardware moves less data for each model weight, and that helps most when memory bandwidth, not arithmetic capacity, is the binding constraint.
The model still does essentially the same math. Quantization isn't pruning, and it doesn't reduce the parameter count by itself. The speedup comes mostly from storing and moving compact values, with optimized kernels running low-precision operations or reconstructing values efficiently.
The full serving stack decides the outcome. Hardware has to support the chosen data type, and the runtime needs kernels that avoid heavy conversion overhead. Batch size, context length, KV cache pressure and model architecture can also move the bottleneck away from weight movement.
So benchmark the exact model, format, accelerator, serving framework and prompts you plan to run in production. A broader LLM testing and benchmarking process should check output quality alongside memory, latency and throughput.
How Hyperlake helps
Hyperlake lets teams assemble and operate private AI environments with open or custom models, selected compute, evaluation, observability and governed access to enterprise context. Open models can be served through KServe, subject to the deployment and validated integration, and workload-aware CPU and GPU capacity gives direct visibility into infrastructure usage and cost. To discuss model serving in your infrastructure or a client's, talk to our team.
Frequently asked questions
Does quantizing an LLM always reduce its accuracy?
It always adds approximation error, but that error doesn't always hurt the target task in a meaningful way. Eight-bit quantization often stays close to the original model, while 4-bit and lower formats need more careful validation. Test representative prompts, domain data, output criteria and the exact production runtime.
Can a 4-bit model always run on one GPU?
Not always. Four-bit weights shrink raw model storage, but inference also needs memory for activations, KV caches, temporary buffers, framework overhead and concurrent requests. Whether a model fits on one GPU depends on its parameter count, context length, batch size, quantization method, runtime implementation and the accelerator's available memory.
Should a production team choose PTQ or QAT?
Start with PTQ in most cases, since it converts an existing model quickly without another full training cycle. Move to QAT when the low-bit format you need degrades unacceptably under PTQ and you have the training compute to spare. Validate either choice against your workload's quality and performance requirements.


