Topic

Fine-tuning and training

When prompting is not enough. Parameter-efficient fine-tuning, preference training, and the data work that decides the result.

LoRA, distillation, RL02 articles05 glossary terms08 platforms

Articles

Read on this topic

Deep dives and guides filed under this topic, newest first.

Guide

LoRA fine tuning from the paper to serving: r, alpha, QLoRA, dataset format, and a PEFT script run on CPU with real output, adapter size, and evals.

Nov 08 202612 min readFine-tuning and training

Deep dive

RAG vs fine tuning compared by failure mode: what each changes, who still sells fine tuning, dated costs worked in code, and when to combine them.

Nov 08 202612 min readFine-tuning and training

Glossary

Distillation: Distillation is training a smaller student model to reproduce the outputs of a larger teacher model. It is a common way to get a cheaper model that keeps much of the teacher's quality on a narrow task.
Fine-tuning: Fine-tuning is continuing to train a pretrained model on your own examples so it adopts a format, a style, or a skill. Full fine-tuning updates every weight, while parameter-efficient methods such as LoRA update a small set of added weights.
LoRALow-rank adaptation: LoRA (low-rank adaptation) is a fine-tuning method from Hu et al. (2021) that freezes the base model and trains small low-rank matrices added to selected layers. An adapter is far smaller than the model it modifies, so one base model can serve many adapters.
QLoRA: QLoRA is a fine-tuning method from Dettmers et al. (2023) that trains LoRA adapters on a base model quantized to 4 bits. It lowers the memory needed to fine-tune a large model.
RLHFReinforcement learning from human feedback: RLHF (reinforcement learning from human feedback) is training a model with a reward signal learned from human preference comparisons. It is one of the post-training steps that turns a pretrained model into an assistant that follows instructions.

Platforms