RAG that actually works: the seven things nobody tells you

Retrieval-augmented generation has a reputation problem. The tutorial version — chunk your documents, embed them, search by cosine similarity, stuff the top five into the prompt — works beautifully on a curated demo corpus and falls apart on a real one within about a week.

The failure is almost never the model. It's the retrieval. If the right passage isn't in the context window, no amount of prompt engineering will save the answer.

Here's what changes between the demo and production.

1. Chunking is a content decision, not a config value

Fixed 512-token chunks with 50-token overlap is the default in every tutorial, and it is wrong for most real documents.

The unit of retrieval should match the unit of meaning:

• Policy and legal documents — chunk by section, keeping the heading hierarchy attached. A returns policy clause is meaningless without knowing it sits under "EU customers."
• Technical documentation — chunk by heading, never split a code block, and prepend the page title and breadcrumb to every chunk.
• Support tickets and conversations — chunk by thread, not by message. Half a conversation retrieves as noise.
• Tables and spreadsheets — don't chunk into text at all. Keep them structured and query them with generated SQL or a filter, not with embeddings.

Two techniques that consistently earn their keep:

Contextual prefixing. Before embedding, prepend a one-or-two-sentence description of where the chunk sits in the parent document — generated once, offline, by a cheap model. "This section is from the 2026 EU Returns Policy and covers the exception process for personalised items." Retrieval accuracy improvements from this are large and it costs you a single batch job.

Small-to-big. Embed and search over small precise chunks, but return the parent section to the model. You get the precision of a tight embedding and the completeness of a full passage.

2. Pure vector search loses to hybrid search. Every time.

Embeddings are excellent at semantic similarity and bad at exact terms. Ask for order #DE-88410, a SKU code, a person's surname, or an error string, and dense retrieval will confidently return five documents that are about the same topic and contain none of them.

Run both:

• Dense (embeddings) for semantic and paraphrased queries.
• Sparse (BM25 / keyword) for identifiers, names, error codes, rare terms.

Fuse the two result sets with Reciprocal Rank Fusion — it needs no score normalisation and no tuning, which is exactly what you want in a component that has to keep working while everything else changes:

score(doc) = Σ  1 / (k + rank_in_list)      # k ≈ 60

In every deployment where we've measured it, adding a keyword leg to a dense-only system produced a bigger recall gain than any embedding-model upgrade.

3. Retrieve wide, then rerank

Retrieval optimises for recall. The context window demands precision. These are different jobs, so use two stages.

Pull 50–100 candidates from hybrid search, then pass them through a cross-encoder reranker that scores each candidate against the actual query rather than comparing pre-computed vectors. Keep the top 5–10.

This is the single highest-leverage addition to most RAG systems. It's a second model call, it adds latency in the low hundreds of milliseconds, and it routinely fixes the "the answer was in chunk 23 and we only sent 5" class of failure.

4. Metadata filtering beats better embeddings

The cheapest accuracy gain available is usually not semantic at all.

If the customer is in Germany, don't search the UK policy documents. If the question is about the current subscription tier, don't retrieve the 2023 pricing page. If the ticket is about a physical product, exclude the digital-goods knowledge base.

Attach metadata at ingest — market, product line, document type, effective date, version, access level — and filter before the semantic search runs. You shrink the candidate pool to documents that could plausibly be correct, which raises precision and cuts cost simultaneously.

Recency deserves special handling. Most corpora contain superseded documents. If your index has three versions of the returns policy and no notion of which is current, your agent will eventually quote the wrong one to a customer. Either hard-filter to current versions or apply an explicit recency decay.

5. The query the user typed is rarely the query you should search

Users write "it's been ages, where is it?" Your index contains "estimated delivery window" and "shipment tracking status."

Three transformations, cheap to run, large in effect:

• Rewriting. Turn a conversational message plus its history into a standalone search query. Essential in multi-turn contexts — "what about the blue one?" is unsearchable without the previous turn.
• Decomposition. Split multi-part questions. "Can I return this and get the sale price on the replacement?" is two retrievals, not one.
• HyDE (hypothetical document embeddings). Have a cheap model draft a plausible answer, then embed and search with that. The generated answer sits closer in embedding space to real answer passages than the question does.

Run these with a small fast model. The latency is worth it.

6. Evaluate retrieval separately from generation

This is the discipline that separates systems that improve from systems that drift.

When a RAG answer is wrong, there are two possible causes: the right passage wasn't retrieved, or it was retrieved and the model ignored it. Those need completely different fixes, and a single end-to-end score can't tell you which one you have.

Build a labelled set — 100–200 real queries with the passages that genuinely answer them — and track:

• Recall@k — is the correct passage in the top k? If this is low, everything downstream is noise. Fix retrieval first.
• MRR / nDCG — how highly is it ranked? Position matters; models attend unevenly across a long context.
• Faithfulness — is every claim in the answer supported by a retrieved passage? Score this with a model-as-judge against the retrieved set.
• Answer relevance — did it address the question that was asked?

Recall@k is the one to obsess over early. Retrieval failures are invisible in production — the model produces a fluent, plausible, wrong answer and nobody files a bug.

7. Design for the empty result

The most dangerous state in a RAG system is "nothing relevant was found."

The default behaviour of a language model handed five irrelevant passages and a question is to answer anyway, drawing on parametric memory, in exactly the same confident register it uses when it's right. That is how RAG systems produce their worst failures.

Three defences, all cheap:

1. A relevance floor. If the top reranker score is below a threshold, treat it as no result.
2. An explicit instruction and an explicit path. "If the retrieved context does not contain the answer, say so and escalate. Do not answer from general knowledge." Give it somewhere to go, not just something to avoid.
3. Citations, enforced. Require the model to cite the chunk supporting each claim, and validate that the cited chunk exists in what you actually sent. Uncited claims get stripped or flagged. This turns hallucination from an invisible failure into a detectable one.

The order to build in

If you're starting from a demo-grade system, this is the sequence that has paid off most reliably for us:

1. Build the eval set. 100 real queries with known-correct passages. Half a day, and everything after it becomes measurable.
2. Add metadata filtering. Usually the largest single gain, and it's not even an ML change.
3. Add hybrid search. Second largest.
4. Add reranking. Third.
5. Fix chunking to match your content structure.
6. Add query rewriting.
7. Only now consider a different embedding model.

Most teams do this list backwards, starting with the embedding model because it feels like the ML part. It's the smallest lever on the list.

---

We build retrieval systems on client data, inside client infrastructure — with the eval harness handed over alongside the code. Book an audit call.