What Is a Quantized Model, and Why Does Almost Every Local LLM Run One? (2026)

What does quantization actually change?

Quantization changes only how many bits each weight occupies; the parameter count, the layer count and the architecture stay the same. A large model is a few billion to a few trillion numbers, each stored during training as a 16-bit float (BF16 or FP16) taking 2 bytes. Quantization converts those numbers block by block into low-bit integers, keeping one scale factor and one offset per block and multiplying them back at inference time. Using the convention on our methodology page, the bytes per parameter at each level are:

• FP16 / BF16: 2 bytes, the "original" release from each vendor.
• 8-bit (Q8_0): about 1.06 bytes; every 32 weights share one scale factor, and that factor is the extra 0.06.
• 4-bit (Q4_K_M): about 0.57 bytes, not exactly 0.5, because K-quants use two levels of block scaling and keep the attention and output layers at higher precision.
• 2-bit (Q2_K_XL): about 0.38 bytes, an Unsloth dynamic tier that leaves the most sensitive layers at 4 to 8 bits.
• 1-bit (IQ1_M): about 0.24 bytes, using codebook lookups to push each weight to roughly 1.75 bits.

So the number in the name is only the main bit width. In Q4_K_M, Q4 means 4-bit, K is llama.cpp's block quantization scheme, and M is the medium tier alongside S and L; Unsloth's XL is the dynamic version that picks a quant type layer by layer. The formats are defined in the GGUF specification and llama.cpp's quantization notes, and the default tags in the Ollama library, the ones without a suffix, are Q4_K_M as well.

Why does almost every local deployment use a quantized model?

The first reason is VRAM: without quantization, a consumer card cannot even hold a 12B model. The table below is computed live with our formula for three models at 8K context, including weights, KV cache and 1 GB of runtime overhead; where an MoE model does not fit in one pool, the second line gives the "VRAM + RAM" need with expert offload:

Reading it is straightforward. Gemma 4 12B needs 28 GB in FP16, more than a 24GB RTX 3090 has; at 4-bit it needs 11 GB, and a 16GB card still has room for a longer context. Qwen3.8 27B needs 58 GB in its original form, which only a 96GB workstation card can hold in one piece; 8-bit needs 32 GB, which only the RTX 5090 squeezes in; 4-bit needs 19 GB and lands neatly in an RTX 3090 or RTX 4090. DeepSeek V4 Flash is a 284B-parameter MoE that activates 13B per token: the original at 558 GB is data-center territory, and even 4-bit needs 160 GB, so only a machine like the Mac Studio M3 Ultra 512GB holds it in one memory pool. Move the expert weights into system RAM, though, and VRAM drops to 7 GB with 155 GB of RAM, so even an 8GB card runs it at roughly 11 tokens/s. 2-bit cuts the RAM need to 104 GB, within reach of a 128GB desktop. Here quantization decides not how fast the model runs but whether it runs at all.

The second reason is speed: for every token, the GPU reads all the active weights out of VRAM, and smaller weights read faster. Our speed estimate is the bytes read per token divided by memory bandwidth and an efficiency factor; the formula is on the methodology page. Same card, same model, only the quantization level changed. First 8-bit:

Then 4-bit:

Side by side, the benefit is obvious. At 8-bit, Qwen3.8 27B fits only the RTX 5090, and Gemma 4 12B does not fit the 16GB RTX 5070 Ti; at 4-bit, 27B lands on 24GB cards, 12B runs on every card, and the same card with the same model is 60 to 80 percent faster. Both V4 Flash rows show expert-offload speeds, and 4-bit is nearly twice as fast as 8-bit because each token moves half as many expert bytes out of RAM. None of this comes from compute; it comes from moving fewer bytes per token.

The third reason is that in the same VRAM, parameters are worth more than precision. 24GB can hold Gemma 4 12B at 8-bit (17 GB) or Qwen3.8 27B at 4-bit (18.5 GB), and the latter is clearly stronger on benchmarks. The under-1-point loss from 4-bit is far smaller than the capability gap between 12B and 27B. That is why vendor model cards quote "minimum VRAM" at 4-bit, and why the "cheapest card that runs it" on our model pages is computed at 4-bit too.

How much quality does quantization cost?

4-bit usually costs under 1 point on benchmarks, 3-bit starts to show, and anything below 2-bit is only for large models. Three sets of checkable numbers:

• llama.cpp's official perplexity table (7B model): Q8_0 raises perplexity by only 0.0004, Q5_K_M by 0.014, Q4_K_M by 0.054, Q3_K_M by 0.24 and Q2_K by 0.87. Source: the official comparison table in the llama.cpp discussions.
• A January 2026 unified evaluation of Llama 3.1 8B: 13 quantization types from 3-bit to 8-bit, averaged over GSM8K, HellaSwag, IFEval, MMLU and TruthfulQA. FP16 baseline 69.47%, Q5_0 69.92%, Q4_K_S 69.17%, Q3_K_M 68.07%, Q3_K_S 65.49%. The author concludes that 4 to 5 bits is the balanced default and that math reasoning is the most sensitive: Q3_K_S drops GSM8K from 77.6% to 68.3%. See arXiv:2601.14277.
• Unsloth dynamic quants: on Gemma 3 27B, 5-shot MMLU is 71.47% at Q4_K_XL (15.6 GB), 70.87% at Q3_K_XL (12.8 GB) and 68.70% at Q2_K_XL (10.0 GB). For Qwen3.8 27B, their 1-bit IQ1_M tier agrees with the top answer only about 72% of the time, and they warn that 1-bit degrades tool calling noticeably. See the Unsloth dynamic quantization docs.

The other trend is vendors preparing for quantization during training. Google shipped Gemma 3 with quantization-aware training (QAT) versions that cut 27B from 54 GB in BF16 to 14.1 GB in int4, claiming a 54% smaller perplexity drop than plain Q4_0 quantization; see the Google Developers Blog. If a model has an official QAT release, prefer it.

By task, the differences show most in math, code and long reasoning chains, which should stay at 4-bit or above; chat, translation, summarizing and writing are hard to tell apart even at 3-bit.

Does the quantization format depend on the GPU?

Weight quantization does not care about the GPU; any card runs GGUF. Only formats such as FP8 and FP4 that need dedicated tensor cores depend on the architecture. Local inference formats fall into three groups:

• GGUF (llama.cpp, Ollama, LM Studio): low-bit weights are unpacked to 16-bit at compute time, so the RTX 3090, AMD cards and Macs all run it, and it has by far the most files. Q4_K_M, Q8_0 and Q2_K_XL above are all GGUF.
• AWQ, GPTQ (vLLM, SGLang): 4-bit weights paired with dedicated matrix-multiply kernels, higher throughput under concurrency, suited to serving; they need a CUDA card.
• FP8, NVFP4 (TensorRT-LLM, vLLM, and image and video models): these really multiply in 8-bit or 4-bit floating point and need Ada or Blackwell tensor cores; the RTX 3090 has no FP8 units and can only treat FP8 weights as 16-bit. The fp8 versions of Wan 2.2 and FLUX belong here; see GPU upgrade for local video.

One more thing people mix up: the quantization discussed in local inference is almost always weight-only, and the KV cache stays 16-bit. Quantization shrinks only the weight term, so the longer the context, the larger the KV cache's share and the smaller the proportional saving. Compressing the KV cache takes a separate 8-bit cache option, a topic for another article; see does adding RAM help when VRAM runs out.

Which quantization level should I pick?

Default to 4-bit; if it does not fit, reduce context first and then drop to 3-bit; consider 2-bit only for huge MoE models; use 8-bit only when you have roughly double the VRAM to spare. Taking Qwen3.8 27B and DeepSeek V4 Flash as the examples, by VRAM:

In other words, dense models pick their level by VRAM and huge MoE models pick theirs by RAM. V4 Flash needs 104 GB of RAM at 2-bit, so a 128GB desktop runs it; at 4-bit it needs 155 GB, which means 192GB. Below are the consumer cards at 32GB or less with eBay prices; the last column is the estimated speed on Qwen3.8 27B at 4-bit, with a dash where it does not fit:

At eBay US medians as of September 9, 2026, a used RTX 3090 at $1,545 is the cheapest way to run 27B at 4-bit, at about 38 tokens/s. The RTX 5070 Ti at $1,299 is cheaper but has only 16GB, so it cannot run 27B at 4-bit and has to fall back to 2-bit or a 14B-class model. The RTX 4090 at $3,115 runs exactly the same models as the 3090 and is less than 10 percent faster. Under $1,000, the RTX 5060 Ti 16GB at $700 runs Gemma 4 12B at 4-bit at about 37 tokens/s, which is comfortable. For the card discussion see RTX 3090 vs RTX 5070 Ti and the budget GPU guide.

Three practical tips to finish:

1. Download from the Unsloth or bartowski GGUF repositories first. Unsloth's XL tiers pick a quant type per layer; bartowski covers every tier. The file sizes on our model pages come from these two.
2. Within a tier, pick the imatrix version. It uses calibration data to decide which weights matter most and is more accurate at the same size; repositories with imatrix or UD in the name qualify.
3. Do not treat file size as the VRAM requirement. The file is the weights; VRAM also holds the KV cache and about 1 GB of runtime overhead. Qwen3.8 27B's 4-bit file is 16.5 GB, but at 8K context it actually needs 18.5 GB. Every model page on this site is already computed that way.