---
title: "LoRA"
type: "glossary term"
url: "https://aifordevs.org/glossary#lora"
markdown: "https://aifordevs.org/glossary/lora.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/glossary#lora"
also_known_as: "Low-rank adaptation"
---

# LoRA (Low-rank adaptation)

LoRA (low-rank adaptation) is a fine-tuning method from Hu et al. (2021) that freezes the base model and trains small low-rank matrices added to selected layers. An adapter is far smaller than the model it modifies, so one base model can serve many adapters.

Each adapted weight becomes W + (alpha / r) BA, where B and A have rank r and only they are trained. Rank r sets adapter size and capacity, and alpha scales the strength of the update. Hu et al. adapted only attention projections on GPT-3 175B and report training about 0.01% of the parameters.

## Example

For GPT-3 175B at r = 4, Hu et al. report a checkpoint of about 35 MB against about 350 GB for the full model, roughly 10,000 times smaller.

## Source

[Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)

## Related terms

- [Fine-tuning](https://aifordevs.org/glossary#fine-tuning)
- [QLoRA](https://aifordevs.org/glossary#qlora)

## Read the article

[LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script](https://aifordevs.org/articles/lora-fine-tuning)

## Cite this term

AI for Devs. "LoRA." https://aifordevs.org/glossary#lora

- Page: https://aifordevs.org/glossary#lora
- Markdown: https://aifordevs.org/glossary/lora.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs