Enterprise RAG in Production: Retrieval Mechanics at Scale

01 · BUSINESS PROBLEM

Downtime is the cost of a bad answer

Field service engineers and plant technicians keep complex production equipment running across multiple plants. When a machine faults in the middle of a shift, the answer to "what does this fault mean, and what is the sanctioned recovery procedure" lives somewhere in the engineering documentation estate — maintenance manuals, equipment specs, engineering change orders, parts catalogs, safety sheets — accumulated over decades and revised constantly. Engineers searched network drives and local PDF collections, kept printed manuals at workstations, and when that failed, called the most senior technician they knew. Answers were slow, inconsistent across plants, and sometimes drawn from superseded revisions — which, for safety-critical procedures, is not an inconvenience but a hazard. The organization didn't need a chatbot. It needed one place to ask, where every answer carried its revision, and nothing restricted ever surfaced to the wrong eyes.

02 · INDUSTRY

Anonymized

Industry

An industrial manufacturer — multi-plant production operations with a large field-service and engineering organization.

Anonymization

Client name and identifying details withheld. Presented as experience informing AnovaCloud's methodology, not as an AnovaCloud delivery claim.

03 · SCALE

Orders of magnitude that shaped the design

Corpus

Decades of engineering documentation — maintenance manuals, equipment specs, drawings, engineering change orders, parts catalogs, safety procedures — most of it living in revision chains rather than as single files.

Users

Field service engineers, plant technicians, and engineering staff across plants, with sharply different entitlements — a line technician, a reliability engineer, and a design engineer must not see the same answers.

Freshness

Documentation moves on engineering calendars: ECOs supersede specs overnight. A correct answer from last quarter's revision can be a safety incident.

04 · CONSTRAINTS

The non-negotiables

• Revision correctness: every answer had to carry the revision it came from; superseded content must never be retrievable.
• Restricted content: pre-release designs, supplier terms, and cost data — entitlements enforced, never assumed.
• Procedural integrity: multi-step procedures must never be reordered, merged across procedures, or split mid-sequence.
• Usable during a repair: answers had to arrive fast enough to be used standing in front of equipment, not after the shift.
• No training on enterprise content: retrieval-augmented generation over frozen models only — nothing in the corpus used to train or fine-tune models.

05 · BASELINE

What engineers did before

PDFs on network drives and local laptops, printed manuals at workstations, revision confusion — "is this the current rev?" — and institutional memory: call the senior tech. Guidance updates propagated unevenly across plants: some crews worked from current procedures, others from habit. There was no way to measure which answers staff trusted, because there was no single place answers came from.

06 · TARGET ARCHITECTURE

Retrieval mechanics first, everything else after

Source connectors track revisions, not just files: every document enters the pipeline with its revision chain, effective dates, and supersession links. Chunking is structure-aware and document-type-specific; embeddings are versioned; the vector index carries ACL metadata resolved to effective permissions per document. At query time: hybrid retrieval (dense + keyword), conditional reranking, then an ACL filter that re-verifies restricted classes against the live permission service — retrieval never sees what the asker may not. Generation assembles answers with revision-stamped citations and abstains on thin evidence. Around the core: an ingestion monitor that tombstones superseded revisions and alerts on overdue source refreshes, an eval harness that gates every change, and an adversarial canary suite that probes access control continuously in production.

07 · CHUNKING

Chunking: decided by evals, not intuition

The first implementation used fixed-size token windows with overlap — the default, and the fastest thing to build. The eval set killed it. Procedural steps were split mid-sequence, and the generator stitched steps from two different procedures into single, confident, wrong answers. Parts tables were separated from their column headers, producing wrong part numbers cited with full confidence. Boundary failures, not model failures, were the dominant failure class on the golden set.

Semantic chunking was tried next. Coherence improved marginally on evals, but non-deterministic boundaries made re-embedding, diffing, and auditing the corpus at scale expensive and hard to reason about. Rejected.

What won: structure-aware chunking, per document type. Manuals split along section and procedure boundaries with step sequences kept intact; tables kept whole with their headers attached; spec sheets by section; drawing captions ingested as linked context. Every chunk carries document type, section path, revision, and effective date. It won because it eliminated the dominant failure class instead of improving the average — and because every chunking decision was argued against the eval set, not in a design review.

08 · EMBEDDINGS & RERANK

The embedding trade-off we didn't take

Two options were on the table: a larger, higher-dimensional general embedding model, and domain fine-tuning of embeddings. Fine-tuning was rejected first. Its gains concentrated in jargon-heavy queries — exactly the queries hybrid search with a domain synonym list handled more cheaply — while fine-tuned embeddings drift from the general model and lock the organization into a retraining cadence. And the expensive operation was never inference: it was re-embedding the entire corpus on every model change.

The larger general model won evals by a small margin on semantic queries. Hybrid retrieval (dense + keyword) with a compact general model matched it within the margin that mattered, at a fraction of the reindexing cost. So the choice: a mid-size general model, versioned, with full re-embedding only on version change and incremental embedding otherwise. The reasoning, laid bare: retrieval quality per dollar of pipeline, not retrieval quality in a vacuum.

Rerank: latency it had to earn

A cross-encoder reranker over the top candidates earned its latency on multi-step troubleshooting questions — the kind where first-pass retrieval returns plausible-but-wrong near-matches, like vibration symptoms matching the wrong bearing procedure. On the eval set's diagnostic subset, reranking fixed ordering the dense pass got wrong. On direct lookups — torque specs, part numbers — where the dense top result was already right, reranking added latency and cost with no quality change.

The resolution: conditional rerank. It runs for diagnostic and multi-hop questions, or when the top candidates' scores cluster tightly enough to be ambiguous; direct lookups skip it. Evals confirmed no regression on the lookup subset — the expensive component runs only where it proved its worth.

09 · ACCESS CONTROL

The leak that almost shipped

The design carried ACL metadata on every chunk, resolved from the source document system at index time. It looked correct in every demo. It was not correct.

The ingestion crawler resolved permissions from folder-level defaults. The source system had broken permission inheritance on restricted subfolders — restricted engineering-change folders nested inside broadly-shared trees. The crawler tagged their chunks as plant-wide readable. Separately, permission changes between reindexes left stale ACL snapshots sitting in the index. Either hole alone could have exposed restricted designs to unauthorized staff.

It was caught by the pre-launch adversarial battery: test accounts for each role ran a fixed question set, and a restricted-role account received an answer citing a restricted engineering change order. No real user ever saw it. The root cause was in the crawler's permission resolution, found the same week.

The lesson, stated plainly: index-time ACL snapshots are a hope, not a control.

10 · FRESHNESS

Freshness is a discipline, not a feature

Engineering documentation lives in revision chains. An ECO can supersede a spec overnight, and a correct answer from last quarter's revision is a safety incident, not a minor staleness issue. So freshness got pipeline discipline, not a "last updated" label:

• Every chunk carries revision, effective date, and supersession links.
• Superseded revisions are tombstoned in the index — they can never be retrieved, only audited.
• "Current revision" is resolved at query time, not assumed from the index.
• Answers cite the revision they came from, so staleness is visible, not silent.
• An ingestion SLA monitor alerts on overdue source refreshes; if the newest ingested revision lags the source's latest, the system says so instead of answering from old content.

Stale-answer prevention, in other words, is a set of guarantees in the pipeline — not a promise in the UI.

11 · COST & LATENCY

Cost and latency: the reasoning, laid bare

Every component had to earn its place on the eval set, not in a demo. Full-corpus re-embedding on model change was the dominant cost — it drove the compact-model choice and the versioning policy. Reranking was made conditional — latency budget spent where evals proved it. Retrieved chunk count was capped: better chunks beat more chunks, and context window is a cost, not a free shelf. Query-time ACL re-verification was scoped to restricted classes only, keeping the common path fast.

Nothing in the architecture is there because it sounded good. Everything is there because removing it regressed the evals.

12 · EVALS

The eval set: built from real questions

The golden set was sampled from real user questions — field-service tickets, helpdesk logs, and the questions engineers actually asked senior techs — then stratified: direct lookups, troubleshooting, multi-step procedures, comparisons, and revision-sensitive questions where the right answer changed between revisions.

Each question carries expected source documents and expected citations. Grading is done by domain practitioners — maintenance engineers who know what a right answer looks like — with an LLM judge only as a second opinion. A separate adversarial subset probes access control. The harness gates every change: chunking strategy, embedding version, rerank policy, prompt. Nothing ships without passing — which is why the chunking and rerank decisions above were decisions, not opinions.

13 · IMPLEMENTATION

Phased by risk reduction

14 · RESULTS

Outcomes, described without invented metrics

• Field engineers got revision-cited answers during repairs instead of hunting PDFs or calling the senior tech — questions that previously ended in escalation now ended in a sourced answer or an honest abstention.
• The documentation team got a feedback loop for the first time: which documents get asked about, and which questions abstain, tells them exactly what needs updating.
• Access control held under adversarial testing before launch and under the canary suite after it — restricted content stayed restricted.
• The eval harness became the release gate for every change to chunking, embeddings, rerank policy, and prompts — quality stopped depending on anyone's memory of what "good" looked like.

15 · LESSONS LEARNED

What generalizes

• Retrieval mechanics are the whole game. Chunking, embeddings, and rerank policy decide answer quality long before the model choice matters.
• Boundary failures are the dominant failure class. Split a procedure mid-step or a table from its headers and no model will save the answer.
• Index-time ACL snapshots are a hope, not a control. Resolve effective permissions per document; re-verify restricted classes at query time.
• Freshness is a pipeline SLA. Tombstone superseded revisions; resolve "current" at query time; cite the revision so staleness is visible.
• Make expensive components conditional. Rerank where evals prove it; skip it where they don't.
• Build the eval set from real questions and grade it with practitioners. An eval set nobody trusts is a dashboard, not a gate.

16 · HOW ANOVACLOUD APPROACHES THIS

From this experience to the RAG Architecture Review

This experience is part of what informs AnovaCloud's RAG Architecture Review. The review starts where this engagement's hardest lessons live: retrieval mechanics first — chunking strategy argued against evals, embedding selection with the reindexing cost made explicit, rerank policy set by evidence rather than default. Then the controls that make retrieval defensible: access-control correctness tested adversarially, freshness discipline with supersession handling, and an eval set built from the organization's real questions.

Model choice and UX come after the retrieval foundation is sound — because in production, they always matter less than the index.

RAG Architecture Review →

17 · RELATED

Pattern, toolkit, service