---
title: "Quantization"
type: "glossary term"
url: "https://aifordevs.org/glossary#quantization"
markdown: "https://aifordevs.org/glossary/quantization.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#quantization"
---

# Quantization

Quantization is storing model weights, and sometimes activations or the KV cache, in fewer bits, for example 8-bit or 4-bit instead of 16-bit. It cuts memory and often speeds up decoding, at some cost in quality that you should measure on your task.

GPTQ (Frantar et al., 2022) brought weights down to 3 or 4 bits and ran a 175B-parameter model on a single GPU. Decode is limited by memory bandwidth, so smaller weights can raise tokens per second. Quality loss varies by model and task, so run your own evals before and after.

## Example

A 70B-parameter model takes about 140 GB at 16 bits, 70 GB at 8 bits, and 35 GB at 4 bits, before the KV cache.

## Source

[Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022)](https://arxiv.org/abs/2210.17323)

## Related terms

- [GGUF](https://aifordevs.org/glossary#gguf)
- [KV cache](https://aifordevs.org/glossary#kv-cache)
- [QLoRA](https://aifordevs.org/glossary#qlora)
- [Small language model](https://aifordevs.org/glossary#small-language-model)

## Cite this term

AI for Devs. "Quantization." https://aifordevs.org/glossary#quantization

- Page: https://aifordevs.org/glossary#quantization
- Markdown: https://aifordevs.org/glossary/quantization.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs