Knowledge Graph + Ontology: Making GraphRAG Answer the Hard Questions

01 · BUSINESS PROBLEM

Vector search answered documents. The business asked about relationships.

The firm had already built a capable vector-only RAG assistant over its policy, regulatory, and product documentation. It worked well for one kind of question — "what does this document say?" — and failed, visibly and repeatedly, at another: anything about relationships between entities. Analysts in risk, compliance, and operations kept asking questions the system could not answer without guessing:

• Entity ambiguity: "Summit" was a trading platform in one business line, a wealth product brand in another, and a conference name in internal communications. Vector search returned chunks about all three, and the model blended them into confident nonsense. Disambiguation needed identity, not similarity.
• Multi-hop questions: "Which counterparties are exposed to this entity through guarantees and subsidiary relationships?" Answering it required traversing two or three hops of ownership and obligation — vector retrieval has no notion of a hop. The chunks existed; the path between them didn't.
• Stale relationships: organizational structures, entity ownership, and product-to-entity mappings changed constantly — acquisitions, restructurings, product sunsets. A chunk written last quarter described a relationship that no longer existed, and nothing in the vector index recorded that the relationship had changed rather than the text having been edited.

The failure mode wasn't hallucination in the usual sense. The retrieved evidence was usually real; the composition was wrong — entities conflated, paths skipped, temporal state ignored. Fixing it meant modeling the relationships explicitly instead of hoping similarity would approximate them.

02 · INDUSTRY

Anonymized

Industry

A large financial-services firm — banking, wealth management, and asset management lines of business, with risk, compliance, and operations functions as the primary consumers.

Anonymization

Client name and identifying details withheld. Presented as experience informing AnovaCloud's methodology, not as an AnovaCloud delivery claim.

03 · SCALE

Orders of magnitude that shaped the design

Entity estate

Legal entities, counterparties, products, policies, and regulatory obligations across multiple lines of business — the kind of estate where the same name means three different things in three different systems.

Relationships

Ownership, guarantee, custody, servicing, and reporting relationships that change on business calendars — M&A, restructurings, product sunsets — faster than documents get rewritten.

Users

Risk analysts, compliance officers, and operations staff whose questions are multi-hop by nature: exposure, lineage, obligation, and accountability questions that span entity, document, and system boundaries.

04 · CONSTRAINTS

The non-negotiables

• Regulatory defensibility: every relationship used in an answer had to be traceable to its source — document, system of record, or approved extraction — with an effective date. "The graph said so" is not an audit answer.
• Provenance over coverage: an edge extracted by an LLM without corroboration was labeled as such. Machine-extracted relationships carried their extraction method and confidence; system-of-record relationships carried authority. The answer distinguished the two.
• No silent staleness: relationships needed effective dates and validity windows. A traversed path through a superseded relationship had to be detectable — and excluded by default.
• Access control at the entity level: some entities and relationships were restricted by line of business. Traversal had to respect entitlements the same way document retrieval did — no answers assembled from paths the asker couldn't walk themselves.
• Ontology change control: the domain model was a shared artifact; renaming a concept or changing a relationship's meaning rippled into every query. Changes went through review, not pull requests alone.

05 · BASELINE

What analysts did before

Vector-only RAG over the document estate, plus manual lineage work. Analysts ran the assistant for "what does this document say" questions, then did the relationship reasoning themselves: opening entity records in source systems, cross-referencing spreadsheets of ownership structures, and calling the desk that owned the data. Multi-hop questions were answered by humans acting as the graph traversal layer — slow, unrepeatable, and invisible to audit. The painful irony: the knowledge existed in the firm's systems; there was simply no machine-readable model of how it connected.

06 · TARGET ARCHITECTURE

The GraphRAG pipeline, end to end

The pipeline runs in five stages, and the order matters. Extraction pulls candidate entities and relationships from documents with an LLM guided by the ontology — it may only emit types the ontology defines. Entity resolution deduplicates and canonicalizes: "Summit" the platform, "Summit" the product, and "Summit" the conference become three distinct nodes with three identities, linked to their source systems. Embedding produces vectors for entity descriptions and chunks so unstructured text stays searchable. Retrieval grounding is the hybrid core: graph traversal walks the ontology-typed edges (filtered by entitlement and effective date), while vector search covers the unstructured narrative — the two are fused into one evidence pack. Answer synthesis generates from that pack with citations back to entities and documents, and abstains when the graph can't support the answer. Around the pipeline: an eval harness that gates every extraction-model, ontology, or embedding change, and a provenance log recording which edges and documents grounded every answer.

07 · KEY DECISIONS

The modeling arguments that settled the design

Knowledge-graph work is mostly argument — about what things are. The arguments below were the ones that actually changed the build, recorded the way they were settled so future teams don't relitigate them.

08 · IMPLEMENTATION

Phased by risk reduction

09 · GOVERNANCE

Who decided what

• Ontology stewardship: a small stewardship group — domain experts plus the graph architect — owned the type system. Proposals for new entity or relation types came with example instances and a justification; renaming a concept required a migration note for every query that used it.
• Extraction quality ownership: the data engineering team owned extraction precision/recall, measured against a human-labeled sample; domain experts owned adjudication of ambiguous cases. Extraction errors had an owner, not a shrug.
• Edge authority tiers: system-of-record edges, human-curated edges, and machine-extracted edges carried different authority in answers — and the answer text said which was which.
• Eval review board: risk and compliance stakeholders reviewed groundedness eval results the way they review model-risk results — because in a regulated firm, a grounded-answer system is a model under governance.

10 · SECURITY

Controls for a regulated estate

• Entitlement-aware traversal: edges and nodes carried line-of-business restrictions, and the traversal layer filtered before the evidence pack was assembled — no restricted paths in the context window, period.
• Provenance on every edge: source document or system, extraction method, confidence, and effective dates — the evidence trail an auditor or a model-risk reviewer asks for.
• Separation of authority: machine-extracted edges could never override system-of-record edges; conflicts were surfaced for human adjudication rather than auto-merged.
• Encryption in transit and at rest; full query logging — the question, the traversal path, the evidence pack, and the answer — retained under the firm's records policy.

11 · CHALLENGES

What was actually hard

• Extraction quality: the LLM extracted relationships the documents implied but never stated — plausible, well-formed, and wrong. The fix was twofold: constrain extraction to the ontology's vocabulary, and require corroboration (a second mention or a system-of-record match) before an extracted edge earned full authority. Uncorroborated edges stayed provisional and were labeled as such in answers.
• Ontology churn: domain experts kept refining the model mid-build — "that's not a subsidiary, it's a sponsored vehicle" — and every rename rippled into extraction prompts, resolution rules, and golden questions. The lesson: version the ontology from day one and treat renames as migrations with a changelog, not as edits.
• Supernode pain: a handful of ultra-connected nodes — the holding company, the core product families, the master counterparty records — had edge counts orders of magnitude above the median. Unbounded traversal through a supernode exploded the evidence pack and drowned the answer. The fix was traversal budgets per node degree, plus pre-computed summaries for the worst offenders — expensive edges got summarized once instead of traversed every time.
• The "just add more types" temptation: every ambiguous question produced a request for a new relation type. Resisting that — forcing questions back onto the existing vocabulary — is what kept the ontology traversable. An ontology with two hundred relation types is a taxonomy, not a graph.

12 · RESULTS

Outcomes, described without invented metrics

• Multi-hop questions that the vector-only assistant had answered with blended nonsense now came back as traversed answers: the exposure question walked the ownership and guarantee edges and named each hop. Answers now cite the entities they came from — node, edge, and source document.
• Entity ambiguity stopped being a retrieval failure: the three "Summits" are three nodes, and analysts can see which one an answer used — disambiguation became inspectable instead of mysterious.
• Stale relationships stopped resurrecting: expired edges are excluded from traversal by default, so answers describe the firm's current structure rather than its history — unless the analyst asks for the history.
• The eval harness — multi-hop golden questions with expected traversal paths — became the release gate for extraction-model, ontology, and embedding changes. Ontology edits that broke traversal get caught before users see them.

13 · LESSONS LEARNED

What generalizes

• The ontology is the product, not the graph database. Teams want to start with the graph engine; the work is the vocabulary. A week of workshops arguing about what a "counterparty" is saves months of re-extraction.
• Entity resolution is where GraphRAG lives or dies. Extraction is a commodity; canonical identity is not. If "Summit" isn't three nodes before retrieval runs, no amount of traversal will save the answer.
• Never route between graph and vector — fuse them. The questions that justify a knowledge graph are exactly the ones a question-classifier gets wrong. Run both retrievers, fuse the evidence, and let the pack decide.
• Temporal modeling isn't optional in a real enterprise. Relationships change; if your edges can't expire, your graph is a museum. Validity intervals and default-current traversal belong in the first version, not the second.
• Supernodes need budgets, not hope. Real enterprise graphs have extreme degree skew. Traversal budgets and pre-computed summaries for the worst nodes are a design decision, not a tuning knob.
• Provenance is the difference between a demo and a system. In regulated industries, an answer without edge lineage is a liability. Machine-extracted vs. system-of-record authority has to be visible in the answer, not buried in a log.

14 · RELATED

Pattern, toolkit, service

15 · HOW ANOVACLOUD APPROACHES THIS

Start with the vocabulary, not the graph engine

This experience is what the Graph Architecture & Knowledge Graph Review is built on. When we review a graph estate or a GraphRAG pipeline, we start where this study started: does the ontology say what the business means, or what was convenient to extract? We pressure-test the entity model — what became a node, an edge, or a property, and which of those choices your hardest questions actually need. We audit extraction quality against your systems of record, check whether entity resolution produces canonical identity or confident merges, and look for the supernodes that will eat your traversal budget in production.

Then the retrieval question: we test graph-only, vector-only, and hybrid against your real multi-hop questions, because the answer is never obvious up front — but the trade-off we rejected here (routing between retrievers) fails often enough that we check for it explicitly. And we look at the governance layer most teams skip: ontology change control, edge authority tiers, temporal modeling, and the eval harness that keeps an ontology edit from silently breaking traversal.

The review is deliberately scoped — model, performance, grounding — because those are the three places GraphRAG engagements go wrong, in that order. What comes out is a prioritized remediation plan your team can execute, whether or not we build it with you. Read about the review, or talk to an architect about your graph estate.