RAG vs fine-tuning for agents, and when to combine both

You're building an agent. It needs to know things: your company's API docs, your product's behavior, your domain's terminology. You have two levers: fine-tune the model, or retrieve context at runtime. Both work. Both fail in different ways. Both cost money and latency you don't have.

The real question isn't which one is better. It's what problem you're actually trying to solve, and what you can afford to get wrong.

What each one does

Fine-tuning burns knowledge into the model's weights. You take your training data, run it through backprop, and the model learns a new distribution. When the agent runs, that knowledge is just there—no retrieval step, no latency tax, no dependency on a vector database staying up.

RAG (Retrieval-Augmented Generation) keeps your knowledge in a separate store—a vector database, a search index, whatever. At runtime, you embed the agent's query, search for relevant documents, and stuff them into the prompt. The model never changes. You can update your knowledge instantly without retraining.

That's the core tradeoff: baked-in capability versus runtime flexibility.

When fine-tuning actually wins

Fine-tune when the knowledge is stable, narrow, and high-leverage.

If your agent needs to understand your company's specific terminology, your internal conventions, or the quirks of your API—things that won't change every week—fine-tuning is worth the investment. The model learns to reason differently. It stops hallucinating about your product because it has internalized how your product actually works.

Fine-tuning also wins on latency and cost. No retrieval means no embedding call, no database query, no network hop. The inference is just faster. And if you're running many requests, you avoid the per-token cost of storing and retrieving vectors.

Real example: an agent that writes SQL against your internal schema. Fine-tune it on 500 examples of queries against your specific tables, with your naming conventions and your business logic baked in. The model learns the shape of your data. It makes fewer mistakes. You don't need to retrieve your schema on every query.

The cost: you need to retrain when your schema changes. You need to collect and label training data. You need to validate that the fine-tuned model is actually better (it's not always). And you're locked into whatever model you fine-tuned—switching to a newer model means starting over.

When RAG is the only real option

RAG when your knowledge is large, volatile, or uncertain.

If you're indexing a knowledge base with thousands of documents that change weekly, you can't fine-tune your way out. You'd be retraining constantly. RAG lets you update your index and ship the change immediately.

RAG also lets you handle long-tail knowledge. Your agent might need to answer questions about obscure edge cases, deprecated features, or one-off customer scenarios. Fine-tuning would require labeling all of that. RAG lets you just throw documents into a database and let retrieval handle it.

And RAG is honest about uncertainty. If a document doesn't exist, retrieval returns nothing. The model might still hallucinate, but at least you have a clear failure mode. With fine-tuning, the model will confidently make things up—you just can't tell when.

Real example: a support agent that answers questions about your product's documentation. Your docs change constantly. New features ship, old ones get deprecated, edge cases get documented. RAG lets you index the latest version and deploy immediately. Fine-tuning would require retraining every release cycle.

The cost: you need a vector database, embedding infrastructure, and retrieval logic. You add latency to every inference. You add a new failure mode—if retrieval fails or returns the wrong documents, the agent fails. And you're trusting that your embeddings actually capture relevance, which they often don't.

The real failure modes

Fine-tuning fails by becoming stale. You fine-tune on your API docs from six months ago. Your API changes. The model confidently gives wrong answers. You don't notice until it's in production.

You also fail by overfitting. You fine-tune on 200 examples and the model learns spurious patterns. It works great on your test set and breaks on anything slightly different.

RAG fails by retrieving the wrong thing. You query for "how do I reset a password" and retrieval returns documents about password hashing internals. The model hallucinates an answer based on irrelevant context. The user follows the instructions and breaks their account.

RAG also fails silently. If your vector database goes down, the agent either degrades gracefully (if you built fallbacks) or breaks hard (if you didn't). If your embeddings are bad, retrieval silently returns wrong documents. You don't get an error—you get confidently wrong answers.

Both fail when the model itself is wrong. Fine-tuning doesn't fix reasoning errors, only knowledge gaps. RAG doesn't fix reasoning errors either—it just gives the model better source material to reason from.

What you actually build

You probably need both. Here's the shape of a real system:

flowchart TD
    Query["Agent query"]
    Classify{"Retrieve or use<br/>learned knowledge?"}
    Retrieve["Embed + search<br/>vector DB"]
    FT["Use fine-tuned<br/>weights"]
    Combine["Merge context<br/>into prompt"]
    LLM["Call LLM"]
    Answer["Agent response"]
    
    Query --> Classify
    Classify -->|Stable domain| FT
    Classify -->|Volatile/large| Retrieve
    FT --> Combine
    Retrieve --> Combine
    Combine --> LLM
    LLM --> Answer

Fine-tune on your core domain. If your agent needs to understand your product, your API, your terminology, burn that in. Use 500-2000 examples. Validate on a held-out set. Deploy it.

Use RAG for everything else. Documentation, knowledge bases, user-specific context, anything that changes or scales. Index it, retrieve it at runtime, include it in the prompt.

Classify at runtime. Some queries need retrieval. Some don't. Some need both. You can use the query itself to decide—if it mentions a specific document or knowledge base, retrieve. If it's asking about core product behavior, use the fine-tuned knowledge.

The practical tradeoffs

Latency: Fine-tuning wins. RAG adds 100-500ms depending on your embedding model and database. If you're building a real-time agent, that matters.

Freshness: RAG wins. You can update your index in seconds. Fine-tuning requires retraining, which takes hours or days depending on your model size.

Accuracy: Depends on your data. Fine-tuning on good data beats RAG on the same data, because the knowledge is baked in. But RAG on comprehensive documentation beats fine-tuning on sparse examples.

Cost: Fine-tuning has high upfront cost (compute for training) and low per-inference cost. RAG has low upfront cost and higher per-inference cost (embeddings, retrieval, storage).

Debuggability: RAG is easier to debug. You can see what documents were retrieved and why the model gave that answer. Fine-tuning is a black box—the knowledge is in the weights.

Start here

If you're building a new agent: start with RAG. It's lower risk. You can iterate on your knowledge base without retraining. You can see what's working and what's not.

Once you have a stable, well-understood knowledge domain, fine-tune. Take your best examples from RAG interactions, label them, and fine-tune a small model on them. Measure whether it actually improves accuracy. It might not.

Keep both. Use RAG for the long tail and for things that change. Use fine-tuning for your core domain where you've invested in good training data.

And measure. Set up evals before you ship anything. Fine-tuning that makes you 2% more accurate but breaks on edge cases is worse than RAG that's 5% less accurate but fails predictably.