---
title: "QLoRA"
type: "glossary term"
url: "https://aifordevs.org/glossary#qlora"
markdown: "https://aifordevs.org/glossary/qlora.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#qlora"
---

# QLoRA

QLoRA is a fine-tuning method from Dettmers et al. (2023) that trains LoRA adapters on a base model quantized to 4 bits. It lowers the memory needed to fine-tune a large model.

The base weights use a 4-bit NormalFloat (NF4) format, double quantization compresses the quantization constants, and paged optimizers manage memory spikes during training. Reach for it when the model will not fit on your GPU at full precision.

## Example

Dettmers et al. report fine-tuning a 65B-parameter model on a single 48 GB GPU while keeping the task performance of full 16-bit fine-tuning.

## Source

[Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)

## Related terms

- [LoRA](https://aifordevs.org/glossary#lora)
- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)
- [Quantization](https://aifordevs.org/glossary#quantization)

## Read the article

[LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script](https://aifordevs.org/articles/lora-fine-tuning)

## Cite this term

AI for Devs. "QLoRA." https://aifordevs.org/glossary#qlora

- Page: https://aifordevs.org/glossary#qlora
- Markdown: https://aifordevs.org/glossary/qlora.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs