Retrieval-augmented generation (RAG) gives a language model the right documents at question time, so it answers from your knowledge. Fine-tuning changes the model's weights, so it behaves differently every time. They solve different problems.
When RAG is the right choice
- Answers depend on internal documents, policies, contracts or databases.
- The knowledge changes weekly or daily, and answers must stay current.
- You need citations, so users and auditors can see where an answer came from.
- Access control matters: different users may see different documents.
When fine-tuning is the right choice
- You need a consistent output format, tone or classification behaviour.
- A smaller, cheaper model must perform a narrow task as well as a larger one.
- Prompts have become long and fragile, and the behaviour is stable enough to train.
Why most teams should start with RAG
RAG is faster to build, easier to update, and easier to explain. Adding a document updates the system immediately; there is no training run. It also keeps the source of truth in your systems rather than inside model weights, which makes compliance and deletion requests simpler.
Making RAG work in production
- Hybrid retrieval: combine semantic (vector) search with keyword search for names, codes and numbers.
- Good chunking and metadata, so the right passage is found and can be filtered by date, owner or access level.
- Temporal awareness when facts change over time, such as contract terms, so the system answers from the version that applies.
- An evaluation set of real questions, run on every change, to catch regressions.
Using both
Mature systems often combine them: RAG supplies the facts, and a fine-tuned or carefully prompted model handles the format and reasoning. Decide based on evaluation results, not on preference.