Retrieval-Augmented Generation is a practical idea: find the relevant information first, then ask a model to work with it. The hard part is making each step reliable enough that the final answer deserves your trust.
Start with the question
Before choosing a vector database, write down a dozen questions the system should answer. Include questions with no answer in your documents. An honest “I couldn’t find that” is a successful result when the alternative is an invented explanation.
These questions become a tiny evaluation set. Keep the expected evidence next to each question so that you can measure retrieval separately from answer quality.
Treat chunks as evidence
A chunk should contain enough context to make sense on its own. Splitting every document into an identical number of characters is a useful baseline, but headings, lists and code blocks often deserve different boundaries.
- Keep a stable document identifier and source URL.
- Store the section heading alongside the text.
- Avoid splitting a sentence or code example in half.
- Test overlap against your documents instead of choosing it by habit.
The smallest useful pipeline
def answer(question):
candidates = retrieve(question, limit=12)
evidence = rerank(question, candidates)[:4]
if not evidence:
return "I couldn't find supporting information."
return generate(question, evidence, require_citations=True)
This sketch hides most production concerns, but it makes the boundaries visible. Retrieval finds possibilities. Reranking chooses evidence. Generation explains what the evidence supports.
Debug the right layer
If the correct passage never reaches the model, changing the prompt is unlikely to fix the problem. Inspect the retrieved chunks before reading the final answer. If retrieval is good but the answer adds unsupported details, then work on grounding, citation checks and abstention.
Track latency and cost beside quality. A slower pipeline is only worthwhile when its improvements matter to the questions people actually ask.
A useful stopping point
Begin with a readable baseline, a small test set and observable intermediate results. Add complexity only when you can name the failure it fixes. RAG is easier to improve when every stage is something you can inspect.
Further reading: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.