LoRA fine tuning: QLoRA, r, alpha, and a tested PEFT script

LoRA fine tuning trains a small low-rank update on top of frozen model weights, so a task-specific model costs megabytes instead of a full copy of the weights. Hu et al. introduced LoRA in 2021 and reported 10,000 times fewer trainable parameters than full fine tuning on GPT-3 175B. This guide starts from that paper's equation, covers QLoRA, and walks through a Hugging Face PEFT training script that was executed for this page. Every command that was not run is labeled.

#How LoRA fine tuning changes a weight matrix

LoRA fine tuning leaves the original weight matrix W0 frozen and trains two small matrices whose product is added to it. For a matrix of shape d x k, B has shape d x r, A has shape r x k, and only those two receive gradients. The update is (alpha / r) * B @ A.

Hu et al. initialize A with Gaussian noise and B with zeros, so the update is exactly zero before training starts. PEFT reaches the same starting point, because its LoraConfig reference says the default initialization sets B to 0 and makes an untrained adapter a no-op.

The trainable count for one layer is r * (d + k). For the 576 by 576 layer in Figure 1, that is 9,216 numbers against 331,776 in W0, or 2.8 percent.

Hu et al. report a checkpoint falling from about 350 GB to 35 MB on GPT-3 175B, with r = 4 on the query and value matrices. Training VRAM fell from 1.2 TB to 350 GB.

A small rank is often enough. In Table 6 of the paper, adapting the query and value matrices scored between 73.3 and 73.8 percent on WikiSQL across ranks 1, 2, 4, 8, and 64. The authors add in a footnote that they do not expect a small r to work for every task.

#What r and alpha do in a LoRA config

r sets how much the update can express, and alpha scales it. PEFT multiplies the update by lora_alpha / r, or by lora_alpha / sqrt(r) with use_rslora=True. Raising r adds parameters and file size, and Hu et al. note that tuning alpha is roughly the same as tuning the learning rate.

Published settings vary, so treat any single default as a starting point.

Shuttleworth et al. compared alpha = 2r with a fixed alpha = 8 and report better generalization at alpha = 2r, particularly at higher ranks. That result is why the run below uses r=8 and alpha=16 instead of PEFT's default of 8 and 8.

Set r=8 and alpha=16 for a first run, and change r last. Dettmers et al. report that r had no relationship with final performance once adapters covered all layers, so adding modules comes before adding rank.

#Which modules get adapters

Adapters attach to linear layers, and on large models the choice of layers matters more than r. Hu et al. adapted only the attention query and value matrices in most experiments. Dettmers et al. report that this placement did not replicate full fine tuning on large base models and that adapters on all linear layers were required.

PEFT exposes the choice through target_modules. If you leave it empty, PEFT picks by architecture, and in PEFT 0.21.2 that selected q_proj and v_proj for the Llama-architecture model used below. The string "all-linear" selects every linear layer except the output layer.

These counts come from calling get_peft_model on SmolLM2-135M, which has 134,515,008 parameters. Sizes are trainable * 4 bytes for 32-bit floats, before the file header.

Use q_proj and v_proj for a smoke test or when adapter size is your constraint. Use all-linear for a real model, because that is the placement with published evidence from 7B upward. The last row shows where the small-adapter story ends on a tiny model, with an adapter at 14.52 percent of the base.

#QLoRA: fine tuning on 4-bit weights

QLoRA stores the frozen base model in 4-bit NF4 and decompresses weights to 16-bit only for the matrix multiplication. The adapters stay in 16-bit and are the only weights that receive gradients. Dettmers et al. report that this cut fine tuning of a 65B LLaMA from more than 780 GB of GPU memory to under 48 GB.

NF4, double quantization, and paged optimizers supply that saving.

1. NF4 is a 4-bit data type built for zero-centered, normally distributed weights, so each quantization bin holds an equal share of values.
2. Double quantization compresses the quantization constants, reducing their cost from 0.5 to 0.127 bits per parameter at a block size of 64.
3. Paged optimizers use NVIDIA unified memory to move optimizer state to CPU RAM when the GPU runs out.

The base model is where the memory goes. For a 7B LLaMA at batch size 1, the paper measures 26 MB of LoRA parameters and 18 MB of input gradients with gradient checkpointing. The 4-bit base takes 5,048 MB, so shrinking r saves little, which is why QLoRA can afford r = 64 on all linear layers.

Treat the results as the authors' own numbers. They report that 4-bit QLoRA matched 16-bit fine tuning on academic benchmarks. They also report Guanaco 65B at 99.3 percent of ChatGPT on the Vicuna benchmark, a score assigned by GPT-4.

The same paper states that the authors did not establish QLoRA matching full 16-bit fine tuning at 33B and 65B. This guide found no independent replication of the 99.3 percent figure.

PEFT's quantization guide documents the bitsandbytes path and says QLoRA-style training means target_modules="all-linear". The GPU section below covers that path and labels it as not run.

#Format the dataset for the chat template

Keep each LoRA fine tuning example as one JSON object with a messages list, render it with the tokenizer's chat template, and mask every prompt token with -100. The loss ignores masked tokens, so only the answer is trained. Dettmers et al. found that training on the target alone raised MMLU scores across four instruction datasets.

The run below uses synthetic incident lines that convert to a fixed JSON object. The task is deliberately small and checkable.

import json
import random
 
SYSTEM = "Return one JSON object with keys service, region, status, minutes. No prose."
TEMPLATES = [
    "{service} returned {status} for {minutes} minutes in {region}",
    "In {region}, {service} served {status} errors for {minutes} minutes",
    "{minutes} minutes of {status} responses from {service} ({region})",
    "Alert: {service}/{region} status {status}, duration {minutes}m",
]
TRAIN_NAMES = ["checkout-api", "billing-worker", "auth-gateway", "search-indexer",
               "email-sender", "image-resizer", "ledger-sync", "cart-cache"]
TEST_NAMES = ["payments-edge", "profile-store", "invoice-renderer"]
TRAIN_REGIONS = ["us-east-1", "us-west-2", "eu-west-1", "eu-central-1"]
TEST_REGIONS = ["ap-south-1", "sa-east-1"]
 
 
def example(rng, names, regions):
    fields = {"service": rng.choice(names), "region": rng.choice(regions),
              "status": rng.choice([429, 500, 502, 503, 504]),
              "minutes": rng.randint(2, 90)}
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": rng.choice(TEMPLATES).format(**fields)},
        {"role": "assistant", "content": json.dumps(fields, separators=(",", ":"))},
    ]}
 
 
def write(path, rows):
    with open(path, "w") as f:
        f.writelines(json.dumps(r) + "\n" for r in rows)
 
 
rng = random.Random(7)
write("train.jsonl", [example(rng, TRAIN_NAMES, TRAIN_REGIONS) for _ in range(96)])
write("test.jsonl", [example(rng, TEST_NAMES, TEST_REGIONS) for _ in range(24)])

One line of train.jsonl looks like this:

{"messages": [{"role": "system", "content": "Return one JSON object with keys service, region, status, minutes. No prose."}, {"role": "user", "content": "image-resizer returned 503 for 85 minutes in us-west-2"}, {"role": "assistant", "content": "{\"service\":\"image-resizer\",\"region\":\"us-west-2\",\"status\":503,\"minutes\":85}"}]}

These choices in that file carry over to real data.

• The held-out split uses service names and regions that never appear in training, so copying memorized strings cannot score.
• The system prompt states the schema in both training and evaluation, so the base model gets a fair baseline.
• The training labels include the end-of-turn token, which trains the adapter to stop generating.

#Run a PEFT LoRA script on CPU

The script loads SmolLM2-135M on CPU, wraps it with get_peft_model, trains 96 steps with Trainer, and saves only the adapter. It needs no GPU and finishes in minutes. It ran on November 8, 2026 on an Apple M2 with Python 3.13.18, PyTorch 2.15.0, Transformers 5.22.2, PEFT 0.21.2, and Accelerate 1.15.0.

PyYAML had no Python 3.15 wheel for this machine on that date, so the run used a 3.13 environment.

uv venv --python 3.13 .venv
uv pip install --python .venv/bin/python torch==2.15.0 transformers==5.22.2 peft==0.21.2 accelerate==1.15.0

The base model is SmolLM2-135M-Instruct, an Apache 2.0 model that Transformers loads as LlamaForCausalLM.

import hashlib
import json
 
import torch
from peft import LoraConfig, TaskType, get_peft_model
from transformers import (AutoModelForCausalLM, AutoTokenizer, DataCollatorForSeq2Seq,
                          Trainer, TrainingArguments, set_seed)
 
BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
set_seed(7)
 
tokenizer = AutoTokenizer.from_pretrained(BASE)
 
 
def encode(row):
    prompt = tokenizer.apply_chat_template(
        row["messages"][:-1], tokenize=False, add_generation_prompt=True)
    full = tokenizer.apply_chat_template(row["messages"], tokenize=False)
    prompt_ids = tokenizer(prompt, add_special_tokens=False)["input_ids"]
    full_ids = tokenizer(full, add_special_tokens=False)["input_ids"]
    return {
        "input_ids": full_ids,
        "attention_mask": [1] * len(full_ids),
        # -100 tells the loss to ignore the prompt; only the answer is scored.
        "labels": [-100] * len(prompt_ids) + full_ids[len(prompt_ids):],
    }
 
 
train = [encode(json.loads(line)) for line in open("train.jsonl")]
 
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
)
model = get_peft_model(base, config)
model.print_trainable_parameters()
 
 
def frozen_digest(m):
    h = hashlib.sha256()
    for _, p in sorted(m.named_parameters(), key=lambda kv: kv[0]):
        if not p.requires_grad:
            h.update(p.detach().numpy().tobytes())
    return h.hexdigest()[:16]
 
 
before = frozen_digest(model)
 
args = TrainingArguments(
    output_dir="checkpoints",
    use_cpu=True,
    per_device_train_batch_size=8,
    num_train_epochs=8,
    learning_rate=2e-3,
    logging_steps=12,
    save_strategy="no",
    report_to="none",
    seed=7,
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train,
    data_collator=DataCollatorForSeq2Seq(tokenizer, label_pad_token_id=-100),
)
trainer.train()
 
print("frozen weights unchanged:", before == frozen_digest(model))
model.save_pretrained("adapter")

trainable params: 460,800 || all params: 134,975,808 || trainable%: 0.3414
{'loss': '0.2007', 'grad_norm': '0.05098', 'learning_rate': '0.001771', 'epoch': '1'}
{'loss': '0.003873', 'grad_norm': '0.03281', 'learning_rate': '0.001521', 'epoch': '2'}
{'loss': '0.0008523', 'grad_norm': '0.005672', 'learning_rate': '0.001271', 'epoch': '3'}
{'loss': '0.0004551', 'grad_norm': '0.001646', 'learning_rate': '0.001021', 'epoch': '4'}
{'loss': '0.0003418', 'grad_norm': '0.001403', 'learning_rate': '0.0007708', 'epoch': '5'}
{'loss': '0.0002982', 'grad_norm': '0.001455', 'learning_rate': '0.0005208', 'epoch': '6'}
{'loss': '0.0002769', 'grad_norm': '0.0009515', 'learning_rate': '0.0002708', 'epoch': '7'}
{'loss': '0.0002669', 'grad_norm': '0.001162', 'learning_rate': '2.083e-05', 'epoch': '8'}
{'train_runtime': '237.8', 'train_samples_per_second': '3.229', 'train_steps_per_second': '0.404', 'train_loss': '0.02588', 'epoch': '8'}
frozen weights unchanged: True

    5224 adapter/README.md
    1113 adapter/adapter_config.json
 1858776 adapter/adapter_model.safetensors
 1865113 total

The 460,800 figure is 30 layers * (9,216 + 6,144). q_proj is 576 by 576, and v_proj is 576 by 192 because this model uses fewer key and value heads than query heads, a design called grouped-query attention. The safetensors file is 460,800 weights at 4 bytes each, plus a small header.

Training took 128 seconds in one run and 238 seconds in a repeat with other work on the machine, with identical losses. The learning rate of 2e-3 is far above the 1e-4 to 2e-4 in QLoRA's Table 9, and it was picked only so that 96 CPU steps converge.

#Inspect the adapter: rank, size, and merge

This inspection confirms that the update has rank r, that it equals (alpha / r) * B @ A, and that merging preserves the outputs. Run a check like this once per training setup, because a wrong target_modules value or a quantized merge shows up here before it shows up in an eval.

import time
from pathlib import Path
 
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
 
BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
 
start = time.perf_counter()
model = PeftModel.from_pretrained(base, "adapter").eval()
print(f"attach adapter at runtime: {time.perf_counter() - start:.2f} s")
 
# One wrapped layer: the query projection of the first transformer block.
layer = model.base_model.model.model.layers[0].self_attn.q_proj
A = layer.lora_A["default"].weight.detach()
B = layer.lora_B["default"].weight.detach()
W0 = layer.base_layer.weight.detach().clone()
delta = layer.get_delta_weight("default").detach()
 
print("W0 shape:", tuple(W0.shape), "A shape:", tuple(A.shape), "B shape:", tuple(B.shape))
print("scaling alpha/r:", layer.scaling["default"])
print("delta equals (alpha/r) * B @ A:",
      torch.allclose(delta, layer.scaling["default"] * B @ A))
print("rank of W0:   ", torch.linalg.matrix_rank(W0).item())
print("rank of delta:", torch.linalg.matrix_rank(delta).item())
print("top singular values of delta:",
      [round(s, 4) for s in torch.linalg.svdvals(delta)[:10].tolist()])
print(f"Frobenius norm of delta / W0: {delta.norm() / W0.norm():.4f}")
 
inputs = tokenizer("checkout-api returned 503 for 12 minutes in us-east-1",
                   return_tensors="pt")
with torch.no_grad():
    runtime_logits = model(**inputs).logits
    with model.disable_adapter():
        base_logits = model(**inputs).logits
 
start = time.perf_counter()
merged = model.merge_and_unload()
print(f"merge_and_unload: {time.perf_counter() - start:.2f} s")
with torch.no_grad():
    merged_logits = merged(**inputs).logits
 
print("max |adapter - base| logit:  ", (runtime_logits - base_logits).abs().max().item())
print("max |adapter - merged| logit:", (runtime_logits - merged_logits).abs().max().item())
 
merged.save_pretrained("merged")
size = lambda p: sum(f.stat().st_size for f in Path(p).glob("*.safetensors")) / 1e6
print(f"adapter weights: {size('adapter'):.2f} MB, merged weights: {size('merged'):.1f} MB")

attach adapter at runtime: 0.16 s
W0 shape: (576, 576) A shape: (8, 576) B shape: (576, 8)
scaling alpha/r: 2.0
delta equals (alpha/r) * B @ A: True
rank of W0:    571
rank of delta: 8
top singular values of delta: [1.6702, 0.6284, 0.3488, 0.3016, 0.2203, 0.1518, 0.1275, 0.1032, 0.0, 0.0]
Frobenius norm of delta / W0: 0.0115
merge_and_unload: 0.03 s
max |adapter - base| logit:   29.580005645751953
max |adapter - merged| logit: 0.00015211105346679688
adapter weights: 1.86 MB, merged weights: 538.1 MB

The first q_proj has a nearly full-rank W0 and a delta with exactly eight non-zero singular values. Its update has 1.15 percent of the norm of W0, and across all wrapped layers the adapter moved the largest logit by 29.58. The largest gap between runtime and merged logits is 0.00015, which is floating-point rounding.

The merged checkpoint is 538.1 MB against 1.86 MB for the adapter, a ratio of 289 to 1 on this model. An adapter keeps each layer's input and output dimensions, so KV cache sizing does not change.

#Evaluate the adapter against the base model

Score the same held-out prompts twice, once inside model.disable_adapter() and once with the adapter active. Use a metric tied to the task, on data the adapter never saw. The script below compares the two on valid JSON, exact match, and per-field accuracy across 24 rows.

import json
 
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
 
BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE)
tokenizer.padding_side = "left"
 
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float32)
model = PeftModel.from_pretrained(base, "adapter").eval()
 
rows = [json.loads(line)["messages"] for line in open("test.jsonl")]
prompts = [tokenizer.apply_chat_template(m[:-1], tokenize=False, add_generation_prompt=True)
           for m in rows]
gold = [json.loads(m[-1]["content"]) for m in rows]
 
 
@torch.no_grad()
def generate():
    out = []
    for start in range(0, len(prompts), 8):
        batch = tokenizer(prompts[start:start + 8], return_tensors="pt", padding=True,
                          add_special_tokens=False)
        ids = model.generate(**batch, max_new_tokens=48, do_sample=False)
        out += tokenizer.batch_decode(ids[:, batch["input_ids"].shape[1]:],
                                      skip_special_tokens=True)
    return out
 
 
def score(texts):
    parsed, exact, fields = 0, 0, 0
    for text, want in zip(texts, gold):
        try:
            got = json.loads(text.strip())
        except json.JSONDecodeError:
            continue
        if not isinstance(got, dict):
            continue
        parsed += 1
        exact += got == want
        fields += sum(got.get(k) == v for k, v in want.items())
    n = len(gold)
    return f"valid JSON {parsed}/{n}, exact match {exact}/{n}, fields right {fields}/{4 * n}"
 
 
with model.disable_adapter():
    base_out = generate()
adapter_out = generate()
 
print("adapter off:", score(base_out))
print("adapter on: ", score(adapter_out))
print("sample input:", rows[0][1]["content"])
print("adapter off :", base_out[0].strip()[:120])
print("adapter on  :", adapter_out[0].strip()[:120])
print("gold        :", rows[0][2]["content"])

adapter off: valid JSON 0/24, exact match 0/24, fields right 0/96
adapter on:  valid JSON 24/24, exact match 19/24, fields right 87/96
sample input: In sa-east-1, profile-store served 504 errors for 43 minutes
adapter off : The error message indicates that the service failed to respond to the request for the specified service. The service is 
adapter on  : {"service":"profile-store","region":"sa-east-1","status":504,"minutes":43}
gold        : {"service":"profile-store","region":"sa-east-1","status":504,"minutes":43}

The adapter moved the output from zero valid JSON objects to 24 of 24, with 19 exact matches. All five misses involve payments-edge, and in each one the adapter emitted a leading space inside a string value, as in " payments-edge". An aggregate score hides that pattern, so read the misses.

Do not read 19 of 24 as a quality result. A larger base model might pass this task with the prompt alone. The comparison that matters on real work is the tuned model against the base model with your best prompt.

For a real model, build a golden dataset, choose metrics with the LLM evaluation guide, and use LLM as a judge for open-ended outputs. Also test tasks the adapter should leave alone. Shuttleworth et al. report that LoRA tends to forget less pre-training behavior than full fine tuning, with exceptions at large learning rates.

#Serve the adapter: merge or load at runtime

Merge the adapter when one task owns the deployment, because merged weights need no extra step per layer and load like any other checkpoint. Load adapters at runtime when several tasks share one base model, because each adapter costs megabytes and the base stays in GPU memory once.

merge_and_unload() returns a new model, so assign the result. Hu et al. note that merging makes it hard to batch inputs for different tasks in one forward pass.

PEFT's LoRA guide warns that on mixture-of-experts models it computes each expert's LoRA contribution even when few experts are active, which can slow inference substantially. It recommends merging on those models.

vLLM serves adapters per request through its OpenAI-compatible server. The commands below are adapted from vLLM's LoRA docs as of November 8, 2026, and were not run for this guide. They need a GPU server.

vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --enable-lora \
  --max-lora-rank 64 \
  --max-loras 2 \
  --max-cpu-loras 8 \
  --lora-modules \
    '{"name": "incident-json", "path": "/models/lora/incident-json", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}' \
    '{"name": "support-tone", "path": "/models/lora/support-tone", "base_model_name": "meta-llama/Llama-3.2-3B-Instruct"}'
 
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "incident-json", "prompt": "checkout-api returned 503", "max_tokens": 48, "temperature": 0}'

The CLI reference sets --max-lora-rank to 16 and --max-loras to 1 by default, so a QLoRA-style r=64 adapter needs the first flag raised. --max-cpu-loras must be at least --max-loras. The /v1/models endpoint lists each adapter with a parent field pointing to the base model.

vLLM can also load adapters while running, but only with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True. Its docs warn of security risks and advise against that mode in production unless the environment is isolated and fully trusted. For engine choice, see vLLM, SGLang, or llama.cpp.

#Changes for a real model and GPU

LoRA fine tuning on a real model uses the same steps, and none of the code below was run. It comes from PEFT's quantization guide and the QLoRA paper, and it replaces the model loading and training arguments in the script above. Dataset formatting, the evaluation script, and the inspection checks carry over unchanged.

import torch
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
 
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=bnb)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
))

The r and alpha values above are this guide's starting suggestion, following the alpha = 2r result. Make these changes to the training script as well.

1. Remove use_cpu=True from TrainingArguments.
2. Lower learning_rate to the 1e-4 to 2e-4 range in QLoRA's Table 9.
3. Keep gradient checkpointing, which prepare_model_for_kbit_training enables by default in PEFT 0.21.2. The paper measured input gradients falling from 567 MB to 18 MB with it.
4. Plan for far more steps. QLoRA's runs at 7B used 1,875 to 10,000.

If you want a merged model, merge against the unquantized base. PEFT's docs say merging into quantized weights dequantizes, adds the adapter, and quantizes again, and repeated cycles can accumulate rounding error.

#Start your first LoRA fine tuning run

Run your first LoRA fine tuning job only after a prompt and a schema have failed on an eval you trust. Spend the first day on a small model and small data. LLM fine tuning bugs in formatting and labels show up in minutes on CPU and cost GPU hours later.

1. Check the failure against the table in RAG vs fine tuning. An adapter does nothing for stale facts, and context engineering or structured outputs may fix the rest.
2. Write 50 examples in the messages format, the starting point OpenAI recommends, and hold out a fifth, split by entity. Google's tuning guide recommends at least 100 to 500 for its service.
3. Swap your files in for train.jsonl and test.jsonl, and run the three scripts above without other changes.
4. Move to a 7B-class base with the QLoRA setup above, which was not run here, and rerun the same evaluation.
5. Ship the adapter only if it beats the base model with your best prompt on the held-out set.

More guides in this area are collected under fine tuning topics and inference, and new model releases that change these defaults appear in the news feed.

#Questions