RAG Accuracy in Production: Why Retrieval Quality Is the Whole Game

01 · The uncomfortable truth

Naive RAG breaks down on real production queries

The first generation of RAG pipelines was simple: split text into chunks, embed them, store vectors, retrieve top-k by cosine similarity, hand the results to the LLM. It worked well enough to ship a demo. Then production exposed the cracks: fixed-size chunking that splits procedures mid-step, top-k retrieval that doesn't understand multi-part questions, confidently cited wrong document versions, and pipelines that break when document formats change.

The uncomfortable truth: RAG doesn't eliminate hallucinations — it shifts the problem to retrieval quality. When retrieval returns noise, the model reverts to parametric memory or forces connections between the query and irrelevant context. Published evaluations of production RAG systems commonly report double-digit hallucination rates even in grounded deployments. Every one of those failures traces to the same layer: what got retrieved, not what got generated.

This reframes the entire engineering effort. Teams that tune prompts and swap models are optimizing the wrong layer. The teams that fix production RAG work the retrieval pipeline: chunking, embeddings, hybrid search, reranking, and — above all — measurement. Our Enterprise RAG pattern lays out the full reference architecture; this article is the fix list for systems already in production.

02 · Chunking

Fix chunking first — it's the cheapest win

Poor chunking is the most common retrieval failure and the cheapest to fix. The rules:

Respect document structure. Chunk at semantic boundaries — sections, procedure steps, table boundaries — not at fixed character counts. A 500-character chunk that splits a troubleshooting procedure mid-step retrieves half an answer. Structure-aware chunking (by headings, by semantic similarity with overlap, by document-type-specific rules) consistently beats naive fixed windows.

Keep tables and lists intact. Chunking that separates a table from its caption, or a numbered procedure from its prerequisites, destroys the context the model needs. For document types where tables carry the facts (specs, pricing, compliance), consider table-aware extraction that keeps tabular data as coherent units.

Metadata on every chunk. Source document, version, date, document type, and access level. Metadata enables the filtering that prevents the most embarrassing production failure: answering from the wrong version of a document, or surfacing restricted content to unauthorized users. Retrieval without metadata filtering is a demo feature.

Right-size, then tune. Start with 500–1,000 token chunks with 10–20% overlap, then tune against your eval set — not against intuition. Chunking is measurable: retrieval recall on a golden question set tells you whether a change helped. If you're not measuring, you're guessing.

03 · Retrieval

Hybrid search + reranking: the biggest upgrade

Dense vector search captures semantic similarity but misses precise lexical matches — product codes, proper names, part numbers, exact phrases. Keyword search (BM25) catches those but misses paraphrases. Hybrid search combines both and fuses the results (reciprocal rank fusion is the standard). For technical and enterprise corpora, this single change typically delivers the largest retrieval-quality improvement available.

Add a reranking stage. Retrieve a broad candidate set (top 50–100) with cheap hybrid search, then rerank with a cross-encoder model to select the final top 5–10 for the context window. Rerankers are slower and more expensive per query, but they judge query-document pairs jointly rather than by vector proximity — which is exactly the judgment retrieval needs. The two-stage pattern (cheap recall, expensive precision) is the standard production architecture for a reason.

Query transformation. For complex questions, transform before retrieving: decompose multi-part questions into sub-queries, generate hypothetical answers (HyDE) to use as retrieval queries, or expand with domain synonyms. A query about "refund policy for enterprise customers" needs chunks from at least two document sections — naive similarity search won't fetch both, but a decomposed query will.

Consider agentic retrieval for hard questions. Instead of single retrieve-then-generate, let the model iterate: retrieve, assess sufficiency, follow references, check multiple sources, then answer. Slower and more expensive per query, but dramatically more accurate on complex, multi-hop questions. Reserve it for the queries where accuracy justifies the cost — not as the default path.

04 · The pipeline

Ingestion is a pipeline with SLAs, not a one-time upload

Most RAG systems treat ingestion as a setup step. Production treats it as a pipeline: documents get updated, new information arrives, outdated content stays indexed, formats change. Without ingestion discipline, the index silently rots and answers degrade — with no alert, because nothing "broke."

Version and reindex. Every source document needs versioning; every update triggers reindexing of affected chunks. Stale-index answers — confidently citing superseded policy — are among the most damaging RAG failures because they look authoritative. Automate the refresh and monitor index freshness as a metric.

Deduplicate and normalize. Duplicate content across sources (the same policy in three SharePoint sites) dilutes retrieval and confuses citation. Deduplication at ingestion, with canonical source selection, keeps the index clean.

Handle the hard formats. Scanned PDFs, image-heavy manuals, and multilingual corpora need specialized parsers and sometimes multimodal embeddings. The ingestion pipeline that works on clean markdown will fail on the scanned 200-page equipment manual — which is, of course, the document users ask about most.

Access-aware retrieval. Enforce per-user document permissions at retrieval time, not as a post-filter. The RAG system that surfaces restricted documents to unauthorized users isn't a quality problem — it's a security incident. Build permission checks into the retrieval path from day one; retrofitting them is painful.

05 · Evals

Measure retrieval before you tune anything else

The teams that fix RAG measure it. The golden set: 100–300 realistic questions with known-good answers and known-relevant source chunks, sampled from real usage — including the ambiguous, the adversarial, and the boring. The metrics:

• Retrieval recall: for each question, did the pipeline fetch the chunks needed to answer? If recall is low, no amount of prompt tuning helps — fix retrieval.
• Citation precision: is every claim in the answer traceable to a retrieved source? Verify citations against sources, don't just string-match them in the output.
• Answer correctness: graded against the golden answers — by rubric, calibrated LLM judges, and human spot-checks.
• Abstention accuracy: when the corpus doesn't contain the answer, does the system say "I don't know"? A RAG system that never abstains is a hallucination machine with good PR.

Run the eval suite on every change — new embeddings, new chunking, new reranker, new prompt. This is the same eval-gated discipline our agent evaluation guide describes: the harness decides what ships. Retrieval recall is the metric to watch first, because it's the layer where production RAG actually fails.

06 · Cost and latency

The accuracy budget: cost and latency tradeoffs

Every retrieval improvement has a price. The production RAG architecture is a budget allocation problem across three axes:

Reranking costs latency and money. A cross-encoder rerank over 50 candidates adds hundreds of milliseconds and meaningful per-query cost. For high-traffic, low-stakes queries (FAQ-style), hybrid retrieval without reranking is often accurate enough. For complex, high-stakes questions, reranking earns its cost. Tier your retrieval depth by query value — not every question deserves the full pipeline.

Context window is not free. Stuffing more chunks into a larger context window increases token cost roughly linearly and can degrade answer quality — models attend less reliably to the middle of long contexts. Ten well-chosen chunks beat fifty mediocre ones. Retrieval precision matters more than context size; invest in getting the right chunks, not more chunks.

Agentic retrieval is the premium tier. Iterative retrieve-assess-retrieve loops multiply cost per query by 3–10x. Reserve for the questions where accuracy justifies it: complex multi-hop research, high-stakes decisions, anything where a wrong answer costs more than the tokens. Route simple questions to the cheap path with a classifier — even a simple one (question length, complexity signals) beats one-size-fits-all.

Measure cost per correct answer. Not cost per query — cost per correct answer. A cheap pipeline with 60% accuracy costs more per correct answer than an expensive pipeline at 90%, once you count the human time spent catching and correcting failures. This is the metric that justifies retrieval investment to finance.

07 · Bottom line

Retrieval is the product

For enterprise RAG, the model is a commodity and the retrieval pipeline is the product. The teams that internalize this — that invest in chunking, hybrid search, reranking, ingestion discipline, and evals instead of chasing the next model release — are the ones whose production systems actually work. The teams that don't are the ones writing postmortems about the demo that didn't survive contact with users.

Start where the leverage is: fix chunking, add hybrid search and reranking, build the ingestion pipeline with SLAs, and measure retrieval recall before tuning anything else. The accuracy gains are real, they're measurable, and they compound — because every improvement to retrieval improves every answer the system will ever give.