Pattern 01 · AI Architecture
Enterprise RAG, done right
A production pattern for grounded question-answering over private enterprise content — with access control, evaluation, and freshness built in, not bolted on.
01 · PROBLEM
Generic LLMs hallucinate on your private knowledge
Out of the box, a language model answers from its training data: confident, fluent, and frequently wrong about your products, policies, and procedures. Pointing it at a document folder with a naive upload doesn't fix this — retrieval returns the wrong chunks, stale content outranks current policy, and users with different permissions see the same answers.
This pattern describes the architecture we implement for support copilots, internal knowledge assistants, and customer-facing help systems where every answer must be traceable to a source and no user sees content they aren't entitled to.
02 · WHEN TO USE
Fit and anti-fit
- You have a large, changing corpus: docs, tickets, runbooks, contracts, wikis.
- Answers must cite sources and respect document-level permissions.
- Content freshness matters — policies and products change monthly or faster.
- You can define what a "good answer" looks like (so you can evaluate it).
Don't use it for stable, small knowledge (a static FAQ is cheaper as fine-tuned prompts or a knowledge base), or where answers require multi-step reasoning across dozens of documents — that's agentic territory (see Multi-Agent Systems).
03 · ARCHITECTURE
Reference architecture
The critical insight: RAG is two systems — an offline ingestion pipeline and an online retrieval pipeline — joined by an evaluation loop. Most failed pilots build only the right half.
04 · COMPONENTS
What each piece does
| Component | Responsibility | Build/buy notes |
|---|---|---|
| Chunk & clean | Split documents into retrievable units with metadata (source, section, date, owner, ACL tags) | Structure-aware splitting beats fixed token windows; keep tables and procedures intact |
| Embed | Convert chunks to dense vectors | Start with a strong general model; fine-tune only when evals prove a domain gap |
| Vector + keyword indexes | Dense similarity and BM25 lexical search over the same corpus | pgvector is fine to start; dedicated stores (Pinecone, Qdrant, OpenSearch) at scale |
| Query rewrite | Expand ambiguous queries, generate sub-queries | Small, fast model; biggest quality lever per dollar |
| Hybrid retrieval | Merge dense + sparse results, apply permission filters before ranking | Filters must come from your identity system, not document metadata alone |
| Rerank | Cross-encoder re-scores top 50 → top 8 | Adds 200–500ms; worth it whenever precision matters |
| Guarded generation | LLM answers strictly from packed context, cites sources, refuses when context is insufficient | Refusal behavior is a feature — prompt for it explicitly |
| Eval harness | Golden question set, LLM judge, regression runs on every change | Non-negotiable for production; start with 100–200 questions |
05 · FLOW
How a query moves through the system
06 · TRADEOFFS
Decisions with real costs
| Decision | Option A | Option B | Our default |
|---|---|---|---|
| Chunking | Fixed 512-token windows — simple, splits meaning | Structure-aware — better recall, more engineering | Structure-aware for docs that matter; fixed for the long tail |
| Retrieval | Dense only — misses exact terms, SKUs, error codes | Hybrid dense + BM25 — more infra, better recall | Hybrid, always, for enterprise corpora |
| Reranking | Skip it — lower latency | Cross-encoder — +200–500ms, materially better top-k | Include; latency is tunable, wrong answers aren't |
| Freshness | Nightly re-index — simple, stale up to 24h | Event-driven ingestion — real-time, more plumbing | Event-driven for high-churn sources, nightly for the rest |
| Model hosting | API (OpenAI/Anthropic/Bedrock) — fast start, per-token cost | Self-hosted — data control, GPU ops burden | API first; self-host only for hard data-residency constraints |
07 · SECURITY
Non-negotiables
- Permission-aware retrieval: entitlement filters execute inside the index query. Post-filtering ranked results leaks existence and wastes budget.
- PII handling: detect and tag PII at ingestion; redact or tokenize before embedding when the use case doesn't need raw values.
- Prompt injection: treat retrieved content as untrusted data, never instructions. Screen both user input and high-risk retrieved chunks.
- Data residency: embeddings are derived data — they inherit the classification of their source documents. Keep them in the same boundary.
08 · GOVERNANCE
Running it like a system, not a demo
- Eval-gated changes: no index rebuild, chunking change, or prompt edit ships without the regression suite passing.
- Source lineage: every citation resolves to document + version + retrieval timestamp. Stale answers are debuggable.
- Ownership: each source has a named owner responsible for freshness; orphaned sources get quarantined, not silently served.
- Audit log: who asked what, what was retrieved, what was answered — retained per your compliance requirements.
09 · COST
Where the money goes
- Ingestion & embedding are one-time-ish per document, then incremental — usually the smallest line item.
- Retrieval infra (vector store, reranker GPUs) is fixed monthly cost; size it to QPS and corpus, not vibes.
- Generation tokens dominate at scale: a 8k-token context at high QPS dwarfs everything else. Reranking + tight context budgets are cost controls, not just quality controls.
- Eval & labeling is the hidden budget line — plan for ongoing golden-set maintenance, not a one-time effort.
10 · IMPLEMENTATION
A phased path that de-risks
Engagement tie-in: our Agentic AI practice implements this pattern end-to-end, and the Agentic AI Readiness Assessment scores your corpus, permissions, and eval maturity before you commit.
11 · RELATED
Keep exploring
Multi-Agent Systems
When answers need planning, tool use, and multi-step reasoning — the agent layer above RAG.
Read pattern →Agentic AI
Our practice for production copilots, assistants, and agent systems with eval built in.
Explore practice →Advisory
AI strategy and readiness assessments that decide whether to build before how.
Explore practice →Start here
Talk to an Architect
Bring your hardest AI, data, or modernization problem. We'll tell you plainly whether we can help — and what it takes.
FAQ
Questions we hear
Access control, evaluation, and freshness. Enterprise RAG enforces per-user document permissions at retrieval time, runs a regression eval suite on every index or prompt change, and treats ingestion as a pipeline with SLAs — not a one-time upload.
Retrieval quality. Chunking that splits procedures mid-step, missing metadata for filtering, and no reranking stage account for most quality failures we see. Instrument retrieval precision before tuning the generator.
Postgres with pgvector is a legitimate starting point up to roughly single-digit millions of vectors with modest QPS. Move to a dedicated vector store when you need hybrid search at scale, multi-tenancy, or sub-100ms p99 latency with heavy filtering.