---
title: "LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script"
type: "guide"
url: "https://aifordevs.org/articles/lora-fine-tuning"
markdown: "https://aifordevs.org/articles/lora-fine-tuning.md"
publisher: "AI for Devs"
license: "https://aifordevs.org/reuse"
attribution: "Credit AI for Devs and link to https://aifordevs.org/articles/lora-fine-tuning"
description: "LoRA fine tuning from the paper to serving: r, alpha, QLoRA, dataset format, and a PEFT script run on CPU with real output, adapter size, and evals."
author: "AI for Devs"
published: "2026-11-08"
level: "Intermediate"
topics: ["fine-tuning", "models"]
---

# LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script

LoRA fine tuning from the paper to serving: r, alpha, QLoRA, dataset format, and a PEFT script run on CPU with real output, adapter size, and evals.

> **TL;DR**
>
> - LoRA fine tuning freezes a model's weights and trains a small update, `(alpha / r) * B @ A`, on selected linear layers. In this guide's run, one 576 by 576 weight matrix got 9,216 trainable numbers instead of 331,776.
> - The PEFT script on this page ran on an Apple M2 CPU on November 8, 2026, with Python 3.13.18, PyTorch 2.15.0, Transformers 5.22.2, PEFT 0.21.2, and Accelerate 1.15.0. It trained 460,800 parameters of SmolLM2-135M and wrote a 1.86 MB adapter.
> - That run proves the API and the mechanics. It does not demonstrate model quality, because a 135M model on a synthetic task is only a smoke test.
> - QLoRA (Dettmers et al., 2023) keeps the frozen weights in 4-bit NF4 and attaches adapters to all linear layers. The paper reports fine tuning a 65B model on one 48 GB GPU.
> - Merge the adapter when one task owns the deployment, and load adapters at runtime when several tasks share a base model. vLLM's default `--max-lora-rank` is 16, so raise it for `r=64` adapters.

LoRA fine tuning trains a small low-rank update on top of frozen model weights, so a task-specific model costs megabytes instead of a full copy of the weights. Hu et al. introduced [LoRA](https://aifordevs.org/glossary#lora) in 2021 and reported 10,000 times fewer trainable parameters than full [fine tuning](https://aifordevs.org/glossary#fine-tuning) on GPT-3 175B. This guide starts from that paper's equation, covers QLoRA, and walks through a Hugging Face PEFT training script that was executed for this page. Every command that was not run is labeled.

## How LoRA fine tuning changes a weight matrix

LoRA fine tuning leaves the original weight matrix `W0` frozen and trains two small matrices whose product is added to it. For a matrix of shape `d x k`, `B` has shape `d x r`, `A` has shape `r x k`, and only those two receive gradients. The update is `(alpha / r) * B @ A`.

[Hu et al.](https://arxiv.org/abs/2106.09685) initialize `A` with Gaussian noise and `B` with zeros, so the update is exactly zero before training starts. PEFT reaches the same starting point, because its [`LoraConfig` reference](https://huggingface.co/docs/peft/main/en/package_reference/lora) says the default initialization sets `B` to 0 and makes an untrained adapter a no-op.

**Figure 1.** The frozen matrix and the trainable product run side by side and their outputs are summed. The whole correction fits through an r-sized bottleneck, which caps the rank of the update at r.

The trainable count for one layer is `r * (d + k)`. For the 576 by 576 layer in Figure 1, that is 9,216 numbers against 331,776 in `W0`, or 2.8 percent.

Hu et al. report a checkpoint falling from about 350 GB to 35 MB on GPT-3 175B, with `r = 4` on the query and value matrices. Training VRAM fell from 1.2 TB to 350 GB.

A small rank is often enough. In Table 6 of the paper, adapting the query and value matrices scored between 73.3 and 73.8 percent on WikiSQL across ranks 1, 2, 4, 8, and 64. The authors add in a footnote that they do not expect a small `r` to work for every task.

## What r and alpha do in a LoRA config

`r` sets how much the update can express, and `alpha` scales it. PEFT multiplies the update by `lora_alpha / r`, or by `lora_alpha / sqrt(r)` with `use_rslora=True`. Raising `r` adds parameters and file size, and Hu et al. note that tuning `alpha` is roughly the same as tuning the learning rate.

Published settings vary, so treat any single default as a starting point.

| Source | `r` | `alpha` | Scale | Adapters on |
| --- | --- | --- | --- | --- |
| Hu et al., GPT-3 175B | 4 | Set to the first `r` tried, then not tuned | 1 at that first `r` | Query and value |
| Dettmers et al., QLoRA | 64 | 16 | 0.25 | All linear layers |
| PEFT 0.21.2 default | 8 | 8 | 1 | Chosen by architecture |
| Run in this guide | 8 | 16 | 2 | `q_proj`, `v_proj` |

[Shuttleworth et al.](https://arxiv.org/abs/2410.21228) compared `alpha = 2r` with a fixed `alpha = 8` and report better generalization at `alpha = 2r`, particularly at higher ranks. That result is why the run below uses `r=8` and `alpha=16` instead of PEFT's default of 8 and 8.

Set `r=8` and `alpha=16` for a first run, and change `r` last. Dettmers et al. report that `r` had no relationship with final performance once adapters covered all layers, so adding modules comes before adding rank.

## Which modules get adapters

Adapters attach to linear layers, and on large models the choice of layers matters more than `r`. Hu et al. adapted only the attention query and value matrices in most experiments. [Dettmers et al.](https://arxiv.org/abs/2305.14314) report that this placement did not replicate full fine tuning on large base models and that adapters on all linear layers were required.

PEFT exposes the choice through `target_modules`. If you leave it empty, PEFT picks by architecture, and in PEFT 0.21.2 that selected `q_proj` and `v_proj` for the Llama-architecture model used below. The string `"all-linear"` selects every linear layer except the output layer.

**Figure 2.** Moving from query and value matrices to all linear layers multiplies this adapter's trainable count by 5.3. The feed-forward layers add the most because they are the widest.

These counts come from calling `get_peft_model` on SmolLM2-135M, which has 134,515,008 parameters. Sizes are `trainable * 4 bytes` for 32-bit floats, before the file header.

| Target modules | `r` | Trainable | Share of base | Weight bytes |
| --- | --- | --- | --- | --- |
| `q_proj`, `v_proj` | 4 | 230,400 | 0.17% | 0.92 MB |
| `q_proj`, `v_proj` | 8 | 460,800 | 0.34% | 1.84 MB |
| `q_proj`, `v_proj` | 64 | 3,686,400 | 2.74% | 14.75 MB |
| All attention | 8 | 921,600 | 0.69% | 3.69 MB |
| `all-linear` | 8 | 2,442,240 | 1.82% | 9.77 MB |
| `all-linear` | 64 | 19,537,920 | 14.52% | 78.15 MB |

Use `q_proj` and `v_proj` for a smoke test or when adapter size is your constraint. Use `all-linear` for a real model, because that is the placement with published evidence from 7B upward. The last row shows where the small-adapter story ends on a tiny model, with an adapter at 14.52 percent of the base.

## QLoRA: fine tuning on 4-bit weights

QLoRA stores the frozen base model in 4-bit NF4 and decompresses weights to 16-bit only for the matrix multiplication. The adapters stay in 16-bit and are the only weights that receive gradients. Dettmers et al. report that this cut fine tuning of a 65B LLaMA from more than 780 GB of GPU memory to under 48 GB.

NF4, double quantization, and paged optimizers supply that saving.

1. NF4 is a 4-bit data type built for zero-centered, normally distributed weights, so each [quantization](https://aifordevs.org/glossary#quantization) bin holds an equal share of values.
2. Double quantization compresses the quantization constants, reducing their cost from 0.5 to 0.127 bits per parameter at a block size of 64.
3. Paged optimizers use NVIDIA unified memory to move optimizer state to CPU RAM when the GPU runs out.

The base model is where the memory goes. For a 7B LLaMA at batch size 1, the paper measures 26 MB of LoRA parameters and 18 MB of input gradients with gradient checkpointing. The 4-bit base takes 5,048 MB, so shrinking `r` saves little, which is why [QLoRA](https://aifordevs.org/glossary#qlora) can afford `r = 64` on all linear layers.

Treat the results as the authors' own numbers. They report that 4-bit QLoRA matched 16-bit fine tuning on academic benchmarks. They also report Guanaco 65B at 99.3 percent of ChatGPT on the Vicuna benchmark, a score assigned by GPT-4.

The same paper states that the authors did not establish QLoRA matching full 16-bit fine tuning at 33B and 65B. This guide found no independent replication of the 99.3 percent figure.

PEFT's [quantization guide](https://huggingface.co/docs/peft/main/en/developer_guides/quantization) documents the `bitsandbytes` path and says QLoRA-style training means `target_modules="all-linear"`. The GPU section below covers that path and labels it as not run.

## Format the dataset for the chat template

Keep each LoRA fine tuning example as one JSON object with a `messages` list, render it with the tokenizer's chat template, and mask every prompt token with `-100`. The loss ignores masked tokens, so only the answer is trained. Dettmers et al. found that training on the target alone raised MMLU scores across four instruction datasets.

The run below uses synthetic incident lines that convert to a fixed JSON object. The task is deliberately small and checkable.

```python title="make_data.py"
import json
import random

SYSTEM = "Return one JSON object with keys service, region, status, minutes. No prose."
TEMPLATES = [
    "{service} returned {status} for {minutes} minutes in {region}",
    "In {region}, {service} served {status} errors for {minutes} minutes",
    "{minutes} minutes of {status} responses from {service} ({region})",
    "Alert: {service}/{region} status {status}, duration {minutes}m",
]
TRAIN_NAMES = ["checkout-api", "billing-worker", "auth-gateway", "search-indexer",
               "email-sender", "image-resizer", "ledger-sync", "cart-cache"]
TEST_NAMES = ["payments-edge", "profile-store", "invoice-renderer"]
TRAIN_REGIONS = ["us-east-1", "us-west-2", "eu-west-1", "eu-central-1"]
TEST_REGIONS = ["ap-south-1", "sa-east-1"]


def example(rng, names, regions):
    fields = {"service": rng.choice(names), "region": rng.choice(regions),
              "status": rng.choice([429, 500, 502, 503, 504]),
              "minutes": rng.randint(2, 90)}
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": rng.choice(TEMPLATES).format(**fields)},
        {"role": "assistant", "content": json.dumps(fields, separators=(",", ":"))},
    ]}


def write(path, rows):
    with open(path, "w") as f:
        f.writelines(json.dumps(r) + "\n" for r in rows)


rng = random.Random(7)
write("train.jsonl", [example(rng, TRAIN_NAMES, TRAIN_REGIONS) for _ in range(96)])
write("test.jsonl", [example(rng, TEST_NAMES, TEST_REGIONS) for _ in range(24)])
```

One line of `train.jsonl` looks like this:

```json title="train.jsonl"
{"messages": [{"role": "system", "content": "Return one JSON object with keys service, region, status, minutes. No prose."}, {"role": "user", "content": "image-resizer returned 503 for 85 minutes in us-west-2"}, {"role": "assistant", "content": "{\"service\":\"image-resizer\",\"region\":\"us-west-2\",\"status\":503,\"minutes\":85}"}]}
```

These choices in that file carry over to real data.

- The held-out split uses service names and regions that never appear in training, so copying memorized strings cannot score.
- The system prompt states the schema in both training and evaluation, so the base model gets a fair baseline.
- The training labels include the end-of-turn token, which trains the adapter to stop generating.

## Run a PEFT LoRA script on CPU

The script loads SmolLM2-135M on CPU, wraps it with `get_peft_model`, trains 96 steps with `Trainer`, and saves only the adapter. It needs no GPU and finishes in minutes. It ran on November 8, 2026 on an Apple M2 with Python 3.13.18, PyTorch 2.15.0, Transformers 5.22.2, PEFT 0.21.2, and Accelerate 1.15.0.

PyYAML had no Python 3.15 wheel for this machine on that date, so the run used a 3.13 environment.

```bash
uv venv --python 3.13 .venv
uv pip install --python .venv/bin/python torch==2.15.0 transformers==5.22.2 peft==0.21.2 accelerate==1.15.0
```

The base model is [SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct), an Apache 2.0 model that Transformers loads as `LlamaForCausalLM`.

```python title="train_lora.py"
import hashlib
import json

import torch
from peft import LoraConfig, TaskType, get_peft_model
from transformers import (AutoModelForCausalLM, AutoTokenizer, DataCollatorForSeq2Seq,
                          Trainer, TrainingArguments, set_seed)

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
set_seed(7)

tokenizer = AutoTokenizer.from_pretrained(BASE)


def encode(row):
    prompt = tokenizer.apply_chat_template(
        row["messages"][:-1], tokenize=False, add_generation_prompt=True)
    full = tokenizer.apply_chat_template(row["messages"], tokenize=False)
    prompt_ids = tokenizer(prompt, add_special_tokens=False)["input_ids"]
    full_ids = tokenizer(full, add_special_tokens=False)["input_ids"]
    return {
        "input_ids": full_ids,
        "attention_mask": [1] * len(full_ids),
        # -100 tells the loss to ignore the prompt; only the answer is scored.
        "labels": [-100] * len(prompt_ids) + full_ids[len(prompt_ids):],
    }


train = [encode(json.loads(line)) for line in open("train.jsonl")]

base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()


def frozen_digest(m):
    h = hashlib.sha256()
    for _, p in sorted(m.named_parameters(), key=lambda kv: kv[0]):
        if not p.requires_grad:
            h.update(p.detach().numpy().tobytes())
    return h.hexdigest()[:16]


before = frozen_digest(model)

args = TrainingArguments(
    output_dir="checkpoints",
    use_cpu=True,
    per_device_train_batch_size=8,
    num_train_epochs=8,
    learning_rate=2e-3,
    logging_steps=12,
    save_strategy="no",
    report_to="none",
    seed=7,
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train,
    data_collator=DataCollatorForSeq2Seq(tokenizer, label_pad_token_id=-100),
)
trainer.train()

print("frozen weights unchanged:", before == frozen_digest(model))
model.save_pretrained("adapter")
```

```text title="Real output, progress bars trimmed"
trainable params: 460,800 || all params: 134,975,808 || trainable%: 0.3414
{'loss': '0.2007', 'grad_norm': '0.05098', 'learning_rate': '0.001771', 'epoch': '1'}
{'loss': '0.003873', 'grad_norm': '0.03281', 'learning_rate': '0.001521', 'epoch': '2'}
{'loss': '0.0008523', 'grad_norm': '0.005672', 'learning_rate': '0.001271', 'epoch': '3'}
{'loss': '0.0004551', 'grad_norm': '0.001646', 'learning_rate': '0.001021', 'epoch': '4'}
{'loss': '0.0003418', 'grad_norm': '0.001403', 'learning_rate': '0.0007708', 'epoch': '5'}
{'loss': '0.0002982', 'grad_norm': '0.001455', 'learning_rate': '0.0005208', 'epoch': '6'}
{'loss': '0.0002769', 'grad_norm': '0.0009515', 'learning_rate': '0.0002708', 'epoch': '7'}
{'loss': '0.0002669', 'grad_norm': '0.001162', 'learning_rate': '2.083e-05', 'epoch': '8'}
{'train_runtime': '237.8', 'train_samples_per_second': '3.229', 'train_steps_per_second': '0.404', 'train_loss': '0.02588', 'epoch': '8'}
frozen weights unchanged: True
```

```text title="wc -c adapter/*"
    5224 adapter/README.md
    1113 adapter/adapter_config.json
 1858776 adapter/adapter_model.safetensors
 1865113 total
```

The 460,800 figure is `30 layers * (9,216 + 6,144)`. `q_proj` is 576 by 576, and `v_proj` is 576 by 192 because this model uses fewer key and value heads than query heads, a design called [grouped-query attention](https://aifordevs.org/glossary#grouped-query-attention). The safetensors file is 460,800 weights at 4 bytes each, plus a small header.

Training took 128 seconds in one run and 238 seconds in a repeat with other work on the machine, with identical losses. The learning rate of `2e-3` is far above the `1e-4` to `2e-4` in QLoRA's Table 9, and it was picked only so that 96 CPU steps converge.

> **What this run proves**
>
> The run proves the API and the mechanics. The config is accepted, the base weights stay frozen, the update has rank 8, the adapter saves and reloads, and merging preserves outputs. It does not demonstrate model quality, because SmolLM2-135M is a tiny model, the task is synthetic, and the held-out set has 24 rows.

## Inspect the adapter: rank, size, and merge

This inspection confirms that the update has rank `r`, that it equals `(alpha / r) * B @ A`, and that merging preserves the outputs. Run a check like this once per training setup, because a wrong `target_modules` value or a quantized merge shows up here before it shows up in an eval.

```python title="inspect_adapter.py"
import time
from pathlib import Path

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)

start = time.perf_counter()
model = PeftModel.from_pretrained(base, "adapter").eval()
print(f"attach adapter at runtime: {time.perf_counter() - start:.2f} s")

# One wrapped layer: the query projection of the first transformer block.
layer = model.base_model.model.model.layers[0].self_attn.q_proj
A = layer.lora_A["default"].weight.detach()
B = layer.lora_B["default"].weight.detach()
W0 = layer.base_layer.weight.detach().clone()
delta = layer.get_delta_weight("default").detach()

print("W0 shape:", tuple(W0.shape), "A shape:", tuple(A.shape), "B shape:", tuple(B.shape))
print("scaling alpha/r:", layer.scaling["default"])
print("delta equals (alpha/r) * B @ A:",
      torch.allclose(delta, layer.scaling["default"] * B @ A))
print("rank of W0:   ", torch.linalg.matrix_rank(W0).item())
print("rank of delta:", torch.linalg.matrix_rank(delta).item())
print("top singular values of delta:",
      [round(s, 4) for s in torch.linalg.svdvals(delta)[:10].tolist()])
print(f"Frobenius norm of delta / W0: {delta.norm() / W0.norm():.4f}")

inputs = tokenizer("checkout-api returned 503 for 12 minutes in us-east-1",
                   return_tensors="pt")
with torch.no_grad():
    runtime_logits = model(**inputs).logits
    with model.disable_adapter():
        base_logits = model(**inputs).logits

start = time.perf_counter()
merged = model.merge_and_unload()
print(f"merge_and_unload: {time.perf_counter() - start:.2f} s")
with torch.no_grad():
    merged_logits = merged(**inputs).logits

print("max |adapter - base| logit:  ", (runtime_logits - base_logits).abs().max().item())
print("max |adapter - merged| logit:", (runtime_logits - merged_logits).abs().max().item())

merged.save_pretrained("merged")
size = lambda p: sum(f.stat().st_size for f in Path(p).glob("*.safetensors")) / 1e6
print(f"adapter weights: {size('adapter'):.2f} MB, merged weights: {size('merged'):.1f} MB")
```

```text title="Real output, progress bars trimmed"
attach adapter at runtime: 0.16 s
W0 shape: (576, 576) A shape: (8, 576) B shape: (576, 8)
scaling alpha/r: 2.0
delta equals (alpha/r) * B @ A: True
rank of W0:    571
rank of delta: 8
top singular values of delta: [1.6702, 0.6284, 0.3488, 0.3016, 0.2203, 0.1518, 0.1275, 0.1032, 0.0, 0.0]
Frobenius norm of delta / W0: 0.0115
merge_and_unload: 0.03 s
max |adapter - base| logit:   29.580005645751953
max |adapter - merged| logit: 0.00015211105346679688
adapter weights: 1.86 MB, merged weights: 538.1 MB
```

The first `q_proj` has a nearly full-rank `W0` and a `delta` with exactly eight non-zero singular values. Its update has 1.15 percent of the norm of `W0`, and across all wrapped layers the adapter moved the largest logit by 29.58. The largest gap between runtime and merged logits is 0.00015, which is floating-point rounding.

The merged checkpoint is 538.1 MB against 1.86 MB for the adapter, a ratio of 289 to 1 on this model. An adapter keeps each layer's input and output dimensions, so [KV cache sizing](https://aifordevs.org/articles/kv-cache-explained) does not change.

## Evaluate the adapter against the base model

Score the same held-out prompts twice, once inside `model.disable_adapter()` and once with the adapter active. Use a metric tied to the task, on data the adapter never saw. The script below compares the two on valid JSON, exact match, and per-field accuracy across 24 rows.

```python title="evaluate_adapter.py"
import json

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
tokenizer.padding_side = "left"

base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
model = PeftModel.from_pretrained(base, "adapter").eval()

rows = [json.loads(line)["messages"] for line in open("test.jsonl")]
prompts = [tokenizer.apply_chat_template(m[:-1], tokenize=False, add_generation_prompt=True)
           for m in rows]
gold = [json.loads(m[-1]["content"]) for m in rows]


@torch.no_grad()
def generate():
    out = []
    for start in range(0, len(prompts), 8):
        batch = tokenizer(prompts[start:start + 8], return_tensors="pt", padding=True,
                          add_special_tokens=False)
        ids = model.generate(**batch, max_new_tokens=48, do_sample=False)
        out += tokenizer.batch_decode(ids[:, batch["input_ids"].shape[1]:],
                                      skip_special_tokens=True)
    return out


def score(texts):
    parsed, exact, fields = 0, 0, 0
    for text, want in zip(texts, gold):
        try:
            got = json.loads(text.strip())
        except json.JSONDecodeError:
            continue
        if not isinstance(got, dict):
            continue
        parsed += 1
        exact += got == want
        fields += sum(got.get(k) == v for k, v in want.items())
    n = len(gold)
    return f"valid JSON {parsed}/{n}, exact match {exact}/{n}, fields right {fields}/{4 * n}"


with model.disable_adapter():
    base_out = generate()
adapter_out = generate()

print("adapter off:", score(base_out))
print("adapter on: ", score(adapter_out))
print("sample input:", rows[0][1]["content"])
print("adapter off :", base_out[0].strip()[:120])
print("adapter on  :", adapter_out[0].strip()[:120])
print("gold        :", rows[0][2]["content"])
```

```text title="Real output, progress bars trimmed"
adapter off: valid JSON 0/24, exact match 0/24, fields right 0/96
adapter on:  valid JSON 24/24, exact match 19/24, fields right 87/96
sample input: In sa-east-1, profile-store served 504 errors for 43 minutes
adapter off : The error message indicates that the service failed to respond to the request for the specified service. The service is 
adapter on  : {"service":"profile-store","region":"sa-east-1","status":504,"minutes":43}
gold        : {"service":"profile-store","region":"sa-east-1","status":504,"minutes":43}
```

The adapter moved the output from zero valid JSON objects to 24 of 24, with 19 exact matches. All five misses involve `payments-edge`, and in each one the adapter emitted a leading space inside a string value, as in `" payments-edge"`. An aggregate score hides that pattern, so read the misses.

Do not read 19 of 24 as a quality result. A larger base model might pass this task with the prompt alone. The comparison that matters on real work is the tuned model against the base model with your best prompt.

For a real model, build a [golden dataset](https://aifordevs.org/glossary#golden-dataset), choose metrics with the [LLM evaluation guide](https://aifordevs.org/articles/llm-evaluation), and use [LLM as a judge](https://aifordevs.org/articles/llm-as-a-judge) for open-ended outputs. Also test tasks the adapter should leave alone. Shuttleworth et al. report that LoRA tends to forget less pre-training behavior than full fine tuning, with exceptions at large learning rates.

## Serve the adapter: merge or load at runtime

Merge the adapter when one task owns the deployment, because merged weights need no extra step per layer and load like any other checkpoint. Load adapters at runtime when several tasks share one base model, because each adapter costs megabytes and the base stays in GPU memory once.

| Option | How | Memory | Added latency | Use when |
| --- | --- | --- | --- | --- |
| Merge | `merge_and_unload()` | One full model per adapter | None by construction, per Hu et al. | One task owns the deployment |
| Runtime in PEFT | `load_adapter()`, `set_adapter()` | Base once, plus adapters | PEFT warns of added latency | Experiments and low-traffic tools |
| Multi-LoRA in vLLM | `--enable-lora`, `--lora-modules` | Base once, adapters in GPU or CPU memory | vLLM says minimal overhead, no published number | Several tasks on shared GPUs |

`merge_and_unload()` returns a new model, so assign the result. Hu et al. note that merging makes it hard to batch inputs for different tasks in one forward pass.

PEFT's [LoRA guide](https://huggingface.co/docs/peft/main/package_reference/lora) warns that on [mixture-of-experts](https://aifordevs.org/articles/mixture-of-experts) models it computes each expert's LoRA contribution even when few experts are active, which can slow inference substantially. It recommends merging on those models.

vLLM serves adapters per request through its OpenAI-compatible server. The commands below are adapted from [vLLM's LoRA docs](https://docs.vllm.ai/en/stable/features/lora/) as of November 8, 2026, and were not run for this guide. They need a GPU server.

```bash title="Not run here: adapted from the vLLM docs"
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --enable-lora \
  --max-lora-rank 64 \
  --max-loras 2 \
  --max-cpu-loras 8 \
  --lora-modules \
    '{"name": "incident-json", "path": "/models/lora/incident-json", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}' \
    '{"name": "support-tone", "path": "/models/lora/support-tone", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}'

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "incident-json", "prompt": "checkout-api returned 503", "max_tokens": 48, "temperature": 0}'
```

The [CLI reference](https://docs.vllm.ai/en/stable/cli/serve/) sets `--max-lora-rank` to 16 and `--max-loras` to 1 by default, so a QLoRA-style `r=64` adapter needs the first flag raised. `--max-cpu-loras` must be at least `--max-loras`. The `/v1/models` endpoint lists each adapter with a `parent` field pointing to the base model.

vLLM can also load adapters while running, but only with `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`. Its docs warn of security risks and advise against that mode in production unless the environment is isolated and fully trusted. For engine choice, see [vLLM, SGLang, or llama.cpp](https://aifordevs.org/articles/vllm-sglang-llama-cpp).

## Changes for a real model and GPU

LoRA fine tuning on a real model uses the same steps, and none of the code below was run. It comes from PEFT's quantization guide and the QLoRA paper, and it replaces the model loading and training arguments in the script above. Dataset formatting, the evaluation script, and the inspection checks carry over unchanged.

```python title="Not run here: 4-bit setup from the PEFT docs"
import torch
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=bnb)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
))
```

The `r` and `alpha` values above are this guide's starting suggestion, following the `alpha = 2r` result. Make these changes to the training script as well.

1. Remove `use_cpu=True` from `TrainingArguments`.
2. Lower `learning_rate` to the `1e-4` to `2e-4` range in QLoRA's Table 9.
3. Keep gradient checkpointing, which `prepare_model_for_kbit_training` enables by default in PEFT 0.21.2. The paper measured input gradients falling from 567 MB to 18 MB with it.
4. Plan for far more steps. QLoRA's runs at 7B used 1,875 to 10,000.

If you want a merged model, merge against the unquantized base. PEFT's docs say merging into quantized weights dequantizes, adds the adapter, and quantizes again, and repeated cycles can accumulate rounding error.

## Start your first LoRA fine tuning run

Run your first LoRA fine tuning job only after a prompt and a schema have failed on an eval you trust. Spend the first day on a small model and small data. LLM fine tuning bugs in formatting and labels show up in minutes on CPU and cost GPU hours later.

1. Check the failure against the table in [RAG vs fine tuning](https://aifordevs.org/articles/rag-vs-fine-tuning). An adapter does nothing for stale facts, and [context engineering](https://aifordevs.org/articles/context-engineering) or [structured outputs](https://aifordevs.org/glossary#structured-outputs) may fix the rest.
2. Write 50 examples in the `messages` format, the [starting point OpenAI recommends](https://developers.openai.com/api/docs/guides/supervised-fine-tuning), and hold out a fifth, split by entity. Google's [tuning guide](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning/use) recommends at least 100 to 500 for its service.
3. Swap your files in for `train.jsonl` and `test.jsonl`, and run the three scripts above without other changes.
4. Move to a 7B-class base with the QLoRA setup above, which was not run here, and rerun the same evaluation.
5. Ship the adapter only if it beats the base model with your best prompt on the held-out set.

More guides in this area are collected under [fine tuning topics](https://aifordevs.org/topics/fine-tuning) and [inference](https://aifordevs.org/topics/inference), and new model releases that change these defaults appear in the [news feed](https://aifordevs.org/news).

## Questions

### What is the difference between LoRA and QLoRA?

LoRA trains a low-rank update on frozen weights. QLoRA does the same while storing those frozen weights in 4-bit NF4, which cuts training memory. Dettmers et al. also attach adapters to all linear layers and report that this placement was required to match full fine tuning on large base models. The adapter weights stay in 16-bit precision during training.

### What are good values for r and alpha in LoRA?

Start with an r of 8 and an alpha of 16, which gives an update scale of 2 under PEFT's default formula. Dettmers et al. used an r of 64 and an alpha of 16 on all linear layers and report that r had no relationship with final performance in that setup. Increase r only when your held-out eval shows the adapter underfits.

### How many training examples does LoRA fine tuning need?

No fixed number exists, because the answer depends on the task. OpenAI's fine-tuning guide, written for hosted models, recommends starting with 50 well-crafted demonstrations and reconsidering the task if those have no effect. Google's Gemini tuning page recommends at least 100 to 500 examples. Start small, hold out data, and add examples only while the held-out score keeps improving.

### Can you run LoRA fine tuning on a CPU?

Yes for a small model, and the run on this page did that: SmolLM2-135M trained for 96 steps on an Apple M2 CPU in 128 to 238 seconds. CPU runs are useful for testing code and data formatting. Real models need a GPU, and the 4-bit bitsandbytes path that QLoRA uses is a separate code path this guide did not run.

### Do LoRA adapters add inference latency?

A merged adapter adds none by construction, because Hu et al. show the update can be added to the weights once and inference proceeds as usual. An unmerged adapter runs extra matrix multiplications per wrapped layer. PEFT's guide warns that you may see latency from loading the base and adapter separately, and vLLM describes per-request adapters as minimal overhead without publishing a number.

### Is LoRA as good as full fine tuning?

On the tasks Hu et al. tested, LoRA matched or beat full fine tuning. Dettmers et al. report that 4-bit LoRA matched 16-bit full fine tuning on academic benchmarks, though they did not establish this at 33B and 65B. Shuttleworth et al. found that LoRA produces structurally different weight updates. Evaluate the adapter on your own held-out data before you rely on any of this.

## Sources

1. [Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)](https://arxiv.org/abs/2106.09685)
2. [Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)](https://arxiv.org/abs/2305.14314)
3. [Shuttleworth et al., LoRA vs Full Fine-tuning: An Illusion of Equivalence (2024)](https://arxiv.org/abs/2410.21228)
4. [Hugging Face, PEFT LoRA configuration reference](https://huggingface.co/docs/peft/main/en/package_reference/lora)
5. [Hugging Face, PEFT LoRA developer guide](https://huggingface.co/docs/peft/main/package_reference/lora)
6. [Hugging Face, PEFT quantization guide](https://huggingface.co/docs/peft/main/en/developer_guides/quantization)
7. [vLLM, LoRA Adapters](https://docs.vllm.ai/en/stable/features/lora/)
8. [vLLM, vllm serve CLI reference](https://docs.vllm.ai/en/stable/cli/serve/)
9. [Hugging Face, SmolLM2-135M-Instruct model card](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct)
10. [OpenAI, Supervised fine-tuning guide](https://developers.openai.com/api/docs/guides/supervised-fine-tuning)
11. [Google Cloud, Use supervised tuning for Gemini models](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/supervised-tuning/use)

## Cite this guide

AI for Devs. "LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script." Published November 8, 2026. https://aifordevs.org/articles/lora-fine-tuning

- Page: https://aifordevs.org/articles/lora-fine-tuning
- Markdown: https://aifordevs.org/articles/lora-fine-tuning.md
- Reuse: Quote or summarize with a link to this page. Terms: https://aifordevs.org/reuse
- Publisher: AI for Devs
- Published: 2026-11-08