RAG vs fine tuning: which failures each one can fix

RAG vs fine tuning is a choice between two places to change an LLM system: the context a request carries and the weights that process it. Retrieval-augmented generation, introduced by Lewis et al. in 2020, adds retrieved passages to the input at request time. Fine tuning trains new weights from labeled examples before any request arrives. This article organizes the decision by failure mode, because the failure you observe tells you which part to change, and it prices each option with dated numbers.

#What RAG and fine tuning each change

RAG changes the context window on each request, and fine tuning changes the weights before deployment. That difference sets how fast a correction reaches users, what it costs per request, and what it can repair. Prompting sits in the same place as RAG, and constrained decoding acts after both.

Lewis et al. describe RAG as a generator paired with a retriever over a dense vector index, which they frame as parametric and non-parametric memory. Weights are the parametric part, and the index is the part you can edit without training. Fine tuning writes into the parametric part, so a change takes a job, an eval, and a deployment.

The LoRA fine tuning guide shows what that training job does to the weights on an open model. This article stays at the level of which failures justify running one.

#Try prompting and context first

Prompting and context come first in any RAG vs fine tuning decision. They cost no training, take effect on the next request, and set the baseline every other option must beat. Most teams should stay here until an eval proves otherwise.

Google's tuning guide says to consider tuning only when prompts do not produce the expected results consistently enough. OpenAI's fine-tuning guide says to set up evals first and to rethink the task or prompt if 50 examples have no impact. Do the prompt work with a test set beside you.

Work through these before you spend on anything else.

1. Add the failing case as a few-shot example, and move stable instructions to the top of the prompt.
2. Cut retrieved or pasted text that does not change the answer. The context engineering guide covers budgeting tokens across instructions, tools, and history.
3. Put a schema on the output if the failure is structural.
4. Turn on prompt caching so a long stable prefix stops being an argument for shortening it.

Stop when the cases in your eval still fail after you added an instruction and an example that cover them. Whatever fails at that point is your real target.

#Decision table by failure mode

To use this RAG vs fine tuning table, find the failure you observe in the left column and apply the first fix before you consider retrieval or training. The verdict names the option that addresses the cause, and rows marked "Cannot" are structural, so more budget will not change them.

#What RAG can and cannot fix

RAG fixes failures where the needed fact exists in a document you control and fails to reach the model. It cannot fix a retriever that misses the passage, an output that ignores a correct passage, or a style problem. Retrieved text changes what the model reads and leaves the weights alone.

The clearest evidence is Lewis et al.'s index hot-swap in Section 4.5. They built one index from a December 2016 Wikipedia dump, kept another from December 2018, and asked about 82 world leaders who had changed. RAG answered 70 percent correctly with the 2016 index on 2016 leaders and 68 percent with the 2018 index on 2018 leaders.

Mismatched indices scored 12 percent and 4 percent. The paper notes that parametric-only models need further training to pick up the same change.

Rare facts tell the same story. Soudani et al. tested twelve language models. They report that RAG beat fine tuning by a large margin on the least popular knowledge, while fine tuning still helped across popularity levels.

Berglund et al. report a related limit. GPT-3 and Llama-1 models fine-tuned on "A is B" statements could not answer the reversed question. The authors note that the reverse works when the statement sits in the context window.

Citations and permissions follow directly from where the data lives. A retrieved chunk has a document ID you can show and check, and a retrieval filter can apply per-user access rules. Weights have neither, and every caller shares them.

When the right passage never arrives, the fix is in retrieval. Check chunking, then hybrid search and reranking, and see the vector database guide for index choices. Measure with an eval that separates retrieval hits from answer quality, as the LLM evaluation guide describes.

#What fine tuning can and cannot fix

Fine tuning fixes behavior that stays consistent across requests, such as output structure, tone, classification boundaries, and tool-calling habits. It cannot deliver facts that change after training, cite a source, or enforce per-user access. Google's tuning page lists classification, summarization, extractive question answering, and chat among its use cases.

OpenAI's optimization guide adds nuanced translation and correcting instruction-following failures. Its preference-optimization guide targets chat messages with the right tone and style. Those are behaviors, and every one of them can be demonstrated with examples.

Format errors have a cheaper fix. OpenAI says Structured Outputs make responses follow your JSON Schema, with refusals as the exception. On open models, vLLM's structured outputs use xgrammar or guidance as constrained decoding backends.

Do not stack both by default. Google warns that applying controlled generation to tuned Gemini models can decrease quality because of a mismatch between tuning and inference.

Tool-call accuracy has more than one layer. OpenAI's function-calling guide advises fewer than 20 functions at the start of a turn and recommends strict mode. It suggests fine tuning for large function sets or difficult tasks.

Anthropic says Claude's tool selection degrades past 30 to 50 tools, a vendor figure with no independent measurement cited, and offers tool search as a fix.

Tool search is RAG over your tool catalog. Try it before fine tuning on any agent with a large tool-calling surface. The guides on AI agents and MCP servers cover the tool design around it.

A tuned model also inherits its base model's lifespan. OpenAI's deprecations page says fine-tuned versions follow the same shutdown schedule as their base, and gpt-4.1-mini is scheduled to shut down on January 14, 2027.

Amazon Bedrock's lifecycle policy blocks new fine-tuning jobs once a model enters Legacy, and Google excludes supervised fine tuning from its SLOs. Weights you host on an open base do not expire on a vendor's schedule.

#Who still offers fine tuning

Google's Agent Platform, Amazon Bedrock, Together AI, and Fireworks AI still sell fine tuning as of November 8, 2026. OpenAI is winding its platform down, Mistral has deprecated its fine-tuning feature, and Google's Gemini API has no tunable model. Check availability before you plan a RAG vs fine tuning decision around it.

Sources for the table are OpenAI and its pricing page, Google's pricing page, AI Studio, Bedrock customization and its model table, Mistral, Anthropic, Together's pricing and fine-tuning docs, and Fireworks. The platforms directory tracks hosts beyond these.

#RAG vs fine tuning costs, worked in code

In this model, fine tuning does not win on token cost once prompt caching is in the comparison, and training is the smallest line. The script below prices one workload five ways at rates fetched on November 8, 2026. Token counts are assumptions for illustration, so change them to your traffic before you trust the result.

Prices come from Google's Agent Platform pricing page, whose documentation still sits under Vertex AI URLs, and OpenAI's pricing page. OpenAI's column is useful only if you already have access, and Gemini Embedding 2 is priced as a preview.

REQUESTS = 1_000_000  # per month
PREFIX = 5_000        # instructions, tool definitions, few-shot examples
TUNED_PREFIX = 1_200  # the examples moved into the weights
USER = 300
RETRIEVED = 3_000     # six 500-token chunks
OUTPUT = 400
HIT = 0.9             # assumed share of a long stable prefix served from cache
 
# USD per 1M tokens, as of November 8, 2026.
VERTEX = dict(input=0.30, cached=0.03, output=2.50, embed=0.20, train=3.00)
OPENAI = dict(input=0.40, cached=0.10, output=1.60, embed=0.02, train=5.00)
VERTEX_TUNED_GLOBAL = dict(input=0.30, cached=0.03, output=2.50)
VERTEX_TUNED_REGIONAL = dict(input=0.33 * 1.5, cached=0.033 * 1.5, output=2.75 * 1.5)
OPENAI_TUNED = dict(input=0.80, cached=0.20, output=3.20)
 
 
def monthly(price, prefix, hit=0.0, retrieved=0, embed=None):
    cached = prefix * hit
    fresh = prefix - cached + USER + retrieved
    embedding = USER * embed if embed else 0
    parts = {
        "cached_in": cached * price["cached"],
        "fresh_in": fresh * price["input"] + embedding,
        "output": OUTPUT * price["output"],
    }
    return {k: v * REQUESTS / 1e6 for k, v in parts.items()}
 
 
def report(name, base, tuned):
    embed = base["embed"]
    rows = [
        ("prompt, no cache", monthly(base, PREFIX)),
        ("prompt + cache", monthly(base, PREFIX, HIT)),
        ("RAG + cache", monthly(base, PREFIX, HIT, RETRIEVED, embed)),
        ("tuned", monthly(tuned, TUNED_PREFIX)),
        ("tuned + RAG", monthly(tuned, TUNED_PREFIX, 0.0, RETRIEVED, embed)),
    ]
    print(name)
    for label, parts in rows:
        total = sum(parts.values())
        detail = "  ".join(f"{k} {v:8,.0f}" for k, v in parts.items())
        print(f"  {label:18s} ${total:>8,.0f}   {detail}")
    # Prefix length at which tuning matches prompt + cache.
    tuned_cost = TUNED_PREFIX * tuned["input"] + USER * tuned["input"] + OUTPUT * tuned["output"]
    per_prefix_token = HIT * base["cached"] + (1 - HIT) * base["input"]
    fixed = USER * base["input"] + OUTPUT * base["output"]
    print(f"  break-even prefix: {(tuned_cost - fixed) / per_prefix_token:,.0f} tokens")
 
 
report("Vertex global, Gemini 3.5 Flash-Lite", VERTEX, VERTEX_TUNED_GLOBAL)
report("Vertex non-global tuned endpoint (1.5x)", VERTEX | dict(input=0.33, cached=0.033, output=2.75), VERTEX_TUNED_REGIONAL)
report("OpenAI gpt-4.1-mini", OPENAI, OPENAI_TUNED)
 
# One-off costs.
examples, tokens_each, epochs = 2_000, 1_900, 3
train_tokens = examples * tokens_each * epochs
print(f"training tokens: {train_tokens:,}")
for name, rate in [("Vertex 3.5 Flash-Lite", 3.00), ("OpenAI gpt-4.1-mini", 5.00), ("Fireworks LoRA SFT, up to 16B", 0.50)]:
    print(f"  {name:32s} ${train_tokens * rate / 1e6:,.2f}")
corpus = 200_000 * 500
print(f"corpus embedding, {corpus:,} tokens: OpenAI ${corpus * 0.02 / 1e6:,.2f}, Gemini Embedding 2 ${corpus * 0.20 / 1e6:,.2f}")
print(f"dedicated H100 for a month: ${8.00 * 730:,.0f}")
# If OpenAI's tuned prefix did hit the cache.
alt = monthly(OPENAI_TUNED, TUNED_PREFIX, HIT)
print(f"OpenAI tuned with cached prefix: ${sum(alt.values()):,.0f}")

Vertex global, Gemini 3.5 Flash-Lite
  prompt, no cache   $   2,590   cached_in        0  fresh_in    1,590  output    1,000
  prompt + cache     $   1,375   cached_in      135  fresh_in      240  output    1,000
  RAG + cache        $   2,335   cached_in      135  fresh_in    1,200  output    1,000
  tuned              $   1,450   cached_in        0  fresh_in      450  output    1,000
  tuned + RAG        $   2,410   cached_in        0  fresh_in    1,410  output    1,000
  break-even prefix: 6,316 tokens
Vertex non-global tuned endpoint (1.5x)
  prompt, no cache   $   2,849   cached_in        0  fresh_in    1,749  output    1,100
  prompt + cache     $   1,512   cached_in      148  fresh_in      264  output    1,100
  RAG + cache        $   2,562   cached_in      148  fresh_in    1,314  output    1,100
  tuned              $   2,392   cached_in        0  fresh_in      742  output    1,650
  tuned + RAG        $   3,938   cached_in        0  fresh_in    2,288  output    1,650
  break-even prefix: 19,035 tokens
OpenAI gpt-4.1-mini
  prompt, no cache   $   2,760   cached_in        0  fresh_in    2,120  output      640
  prompt + cache     $   1,410   cached_in      450  fresh_in      320  output      640
  RAG + cache        $   2,616   cached_in      450  fresh_in    1,526  output      640
  tuned              $   2,480   cached_in        0  fresh_in    1,200  output    1,280
  tuned + RAG        $   4,886   cached_in        0  fresh_in    3,606  output    1,280
  break-even prefix: 13,231 tokens
training tokens: 11,400,000
  Vertex 3.5 Flash-Lite            $34.20
  OpenAI gpt-4.1-mini              $57.00
  Fireworks LoRA SFT, up to 16B    $5.70
corpus embedding, 100,000,000 tokens: OpenAI $2.00, Gemini Embedding 2 $20.00
dedicated H100 for a month: $5,840
OpenAI tuned with cached prefix: $1,832

The break-even row is the prefix length at which a tuned model matches a cached prompt. Below 6,316 tokens on Gemini's global endpoint, caching the long prompt costs less than tuning it away. That number climbs to 13,231 tokens on OpenAI, where tuned tokens cost twice as much.

The tuned prefix stays uncached in this model. Google documents a 4,096-token caching minimum for most Gemini 3 models, and OpenAI's minimum varies by request settings before GPT-5.6. If OpenAI's tuned prefix did hit the cache, the tuned row would fall to $1,832, still above the cached prompt.

Training is a rounding error at this scale. 2,000 examples at 1,900 tokens for three epochs is 11.4M training tokens. That costs $34.20 on Gemini 3.5 Flash-Lite, $57.00 on OpenAI, and $5.70 for LoRA on Fireworks under 16B.

Embedding a 100M-token corpus once costs $2 on OpenAI or $20 on Gemini Embedding 2. A dedicated GPU changes the shape of the bill. One on-demand H100 on Fireworks at $8.00 per GPU-hour is $5,840 for a 730-hour month, before redundancy.

The script leaves out vector store hosting, reranking, ingestion, labeling, and eval time. Add those from your own quotes, and compare token prices with the Claude API pricing and Gemini API pricing guides.

#When to combine RAG and fine tuning

Combine them when facts change and a behavior failure also survives a good prompt, or when retrieval finds the right passage and the output still ignores it. Train on the retrieved context you serve in production, because a model trained on clean question-answer pairs never saw the distractors your retriever returns.

Zhang et al.'s RAFT makes that explicit. Training examples mix relevant documents with distractors that do not contain the answer, and the target output is a reasoning chain that cites the source passage.

The authors report that RAFT on LLaMA2-7B scored 74.00 on HuggingFace API questions, against 26.43 for the same model with plain RAG and 29.08 for GPT-3.5 with RAG. On HotPot QA, GPT-3.5 with RAG scored 41.5 and RAFT scored 35.28, so the gain is not universal.

Use these criteria before committing to both.

• A fact-shaped bucket and a behavior-shaped bucket both remain in your eval after prompt work.
• The retriever hits the right passage on most failing cases, so the output is the failing part.
• One task carries enough volume that its stable prefix is past the break-even length above.

Remember the premium. On OpenAI's rates, tuned plus RAG cost $4,886 against $2,616 for RAG alone, because every retrieved token is billed at the tuned rate. On Gemini's global endpoint the same pair cost $2,410 against $2,335.

#Settle RAG vs fine tuning with a failure audit

Settle RAG vs fine tuning by counting real failures, because the largest failure bucket names the cheapest fix. Spend a day on this before you build a pipeline or a training set. The steps below need only your logs and a spreadsheet.

1. Pull 100 recent failures from logs or a golden dataset.
2. Label each one with a row from the decision table. If it fits none, write a new row.
3. Fix the largest bucket with the first-try column, and rerun the eval.
4. Repeat until the remaining buckets resist prompting and context.
5. Route fact-shaped survivors to RAG. See RAG topics for the retrieval side.
6. Route behavior-shaped survivors to fine tuning, starting with 50 examples, and compare against your best cached prompt on accuracy and monthly cost. The LLM as a judge guide covers open-ended scoring.

If half your failures are stale or missing facts, stop and build retrieval. If what survives is a small bucket of format or style failures, start with a LoRA adapter on an open model. Choose a host that will not retire your base before the adapter pays off.

#Questions