Governed Agentic AI: From Pilot to Production Under Control

01 · BUSINESS PROBLEM

The moment the agent asked to act

A commercial bank's operations organization ran an internal agent pilot for analysts servicing commercial accounts. The agent answered questions, pulled records, and drafted case notes — read-only work, reviewed by humans, and it went well enough that the business asked the dangerous question: "Can it start doing things?" Not answering — acting: updating cases, moving work through queues, drafting customer communications, preparing filings. The problem was never the agent's capability. It was that production action requires production governance — and nobody could answer, in writing, what the agent was allowed to touch, who approved what, and how anyone would know it had behaved. Until those questions had answers, the agent stayed in the pilot. This is the story of building those answers.

02 · INDUSTRY

Anonymized

Industry

A commercial bank — operations and servicing functions for commercial accounts. Internal agent used by operations staff, touching real systems of record.

Anonymization

Client name and identifying details withheld. Presented as experience informing AnovaCloud's methodology, not as an AnovaCloud delivery claim.

03 · SCALE

Orders of magnitude that shaped the design

Organization

A sizable operations organization with multiple servicing teams — case volumes high enough that manual handling was the bottleneck, not the exception.

Systems touched

Case management, account lookup, document search, and communication drafting — the agent's tool surface spanned read-heavy systems and a small number of write-capable ones.

Regulatory posture

Banking regulators expect demonstrable control over automated processes. "The model seemed fine" is not a control; an audit trail is.

04 · CONSTRAINTS

The non-negotiables

• Customer data protection: account numbers, balances, and personal identifiers could not appear where they didn't belong — including inside the agent's own write payloads and logs.
• No autonomous customer-facing actions: nothing the customer could see left the bank without a human's approval. Non-negotiable, no exceptions for "low risk."
• No movement of funds: the agent's write surface stopped well short of anything that moved money. This was a hard line, not a dial.
• Auditability: every tool call, every decision, every approval had to be reconstructable after the fact — for internal audit and for regulators.
• Data residency: all inference and storage inside the bank's infrastructure boundary; nothing left the perimeter, including for vendor support or model improvement.

05 · BASELINE

A good pilot with shallow governance

The pilot that preceded this work was a read-only assistant: retrieval over account records, policy documents, and case history, producing drafts and summaries for analysts. Governance was proportional to its authority — light. It had a usage log, a weekly review, and human eyes on everything it produced, because nothing it produced went anywhere without a human carrying it. The pilot taught the organization two things: the agent was genuinely useful, and the governance that made the pilot safe would not survive contact with write access. When the business asked for the agent to start acting, the risk committee's answer was: build the governance pack first, then we'll talk about production. This case study starts there — the pilot narrative is covered elsewhere; this is the governance build.

06 · TARGET ARCHITECTURE

The governance envelope, not the agent

The architecture that went to production treats the agent as the least important part of the drawing. The governance envelope around it does the real work: an input guardrail screens requests for prompt-injection patterns before the planner agent sees them; a policy engine declares what the agent may do and under what conditions; every tool call carries scoped credentials — the agent's identity has per-tool, least-privilege permissions, and one system it wanted (customer master records) is simply off-limits; output validation enforces schemas and policy on every write payload, with PII redaction; a human-in-the-loop checkpoint gates every sensitive action; an eval harness gates every release; and an audit log records every call, decision, and outcome. The agent got better over time. The envelope is what made production possible.

07 · THE EVAL HARNESS

Policy written in executable form

The eval harness was built before the governance pack was finished — deliberately, because every other control needed something to be tested against.

Dataset construction. The eval set was seeded from two sources the team trusted: real questions and requests from the pilot, and real failures — wrong account lookups, misrouted case notes, hallucinated policy citations, refused-to-act cases that should have been refused. Each eval case recorded not just the input and the expected answer, but the expected tool calls, the expected constraints honored (did it ask for approval where required? did it stay inside its permission scope?), and the expected refusals. A case the agent should decline was as valuable as one it should complete — the harness tested the agent's judgment, not just its helpfulness.

Regression gating in the release pipeline. The harness ran on every change to the model version, the prompts, the tool schemas, or the policy configuration. A release that failed the eval gate did not ship — no exceptions, no "we'll fix it in the next one." The gate was owned by the platform team, not the agent developers, so the people being gated couldn't move the bar.

What the first eval run revealed. The first full run embarrassed the team in the most useful way. A prompt "improvement" shipped to staging made the agent's answers noticeably more fluent — reviewers liked it — while quietly regressing constraint adherence on an entire class of refusal cases: requests it should have declined, it now completed with confident phrasing. Spot-checking by humans had approved the change. The harness failed it. That single result did more for governance culture than any presentation: from then on, "the evals pass" was the answer to "is this safe to ship," and nobody argued with it.

08 · GUARDRAIL DESIGN

Built, bought, and deliberately skipped

The guardrail work started with an inventory of threats and ended with an explicit decision for each one — including the ones the team chose not to build. The table below is the honest version:

09 · HUMAN-IN-THE-LOOP BOUNDARIES

Which actions require a human, and the argument that settled it

The approval matrix was drawn by action class, not by vibes:

• Automatic: read-only lookups, searches, and internal drafts that went nowhere without a human carrying them.
• Human approval: case updates, queue movements, and any draft destined for a customer or an external party. The agent prepared; the human decided.
• Approval plus audit emphasis: anything touching external systems or irreversible state — approved, logged, and reviewed in the weekly governance review.
• Never: customer master-record writes and fund movements — denied at the permission layer, so approval wasn't even an option.

How the boundary was drawn. Risk workshops mapped every action the agent could take to its blast radius and its reversibility. A wrong read was embarrassing; a wrong case update was operational work to undo; a wrong customer communication was a regulatory event. The matrix followed the blast radius.

The argument that settled it. The team deadlocked for weeks between "approve everything" and "approve the risky stuff." The argument that ended the deadlock was a test, not a principle: "Would we let a first-month analyst do this unsupervised?" Anything a new hire couldn't do alone needed a human in the loop. It settled the debate because it was concrete — everyone in the room had trained a first-month analyst and knew exactly what the line felt like. The "approve everything" camp lost on a practical point: blanket approval would have turned approvers into rubber stamps, and a rubber-stamp approval is governance theater with extra latency.

10 · PERMISSION SCOPING

Least privilege, and the permission that was denied

The agent's identity was treated like any service identity: per-tool scoped credentials, read versus write scopes declared separately, time-bounded tokens, and a standing review of what each scope had actually been used for. Unused scopes were pruned — the permission surface shrank over time, which is how you know the process was working.

The permission that was denied: write access to customer master records — the system of record for customer profile data. The agent team requested it because profile updates were a natural fit for the workflow: the agent could see the discrepancy and fix it in one motion. It was denied, and the reason mattered more than the denial: master-record writes feed KYC and regulatory processes and fraud monitoring. A hallucinated address change isn't a typo — it's a compliance event that propagates into systems the agent can't see. The compromise held: the agent drafts the correction into a bounded case object; a human approves; the update flows through the existing human-operated process. The agent was also denied direct customer email send — all customer-facing communication went through templated, approved, logged sends. Nobody at the bank has ever regretted a permission the agent didn't get.

11 · THE INCIDENT

The day the guardrail earned its salary

No drama, no leak — which is the point. During a staging soak test, the agent was asked to "work the pending queue" — a batch of cases awaiting routine updates. It composed the updates correctly, one by one, until one case note came back wrong: the free-text note field contained a customer's account number, pulled from a retrieved record and folded into the agent's summary of the case. The output validation plus PII-redaction layer flagged the payload and blocked the write. Nothing reached production data. Nothing reached a customer. The incident existed only in the audit log and the alert queue.

What the investigation found: the write schema allowed free text in the note field, and the agent treated "note" as "a summary of everything I know about this case." The schema was the vulnerability — the agent was doing exactly what the field invited it to do.

What changed afterward: free-text note fields were removed from the agent's write schema — structured fields only, no exceptions. PII scanning was extended to all write payloads, not just chat output (the extension the team had already built in chapter 08 is what caught it; the incident confirmed the extension needed to be mandatory, not optional). A regression case was added to the eval set: the exact scenario, anonymized, so no future model or prompt change could reintroduce it silently.

The incident is still told in internal reviews — not as a failure, but as the proof. The governance pack was built for exactly this moment: a failure caught in staging, with a log, an explanation, and a fix. The alternative is the same failure in production, explained to a regulator.

12 · WHAT STUCK AND WHAT DIED

Governance that survived contact with production

Not everything in the governance pack earned its keep. The honest accounting:

What stuck: the eval gate in the release pipeline — it caught real regressions, so nobody questioned it. The approval tiers — targeted enough that approvers stayed engaged instead of rubber-stamping. The first-month-analyst test — it kept working as a decision rule long after the workshops ended. Scoped credentials with usage-based pruning. The audit log — which turned out to be the artifact regulators and internal audit actually asked for. The permission review cadence.

What died on the vine: the per-prompt risk review board — collapsed under volume within weeks, replaced by the risk tiers that did the same job without a meeting. Manual QA sampling of full conversation transcripts — nobody read them; replaced by eval dashboards plus incident-triggered review, which is what people actually looked at. The quarterly model re-certification ceremony — replaced by continuous eval on every change, which was strictly more rigorous. And the monthly governance committee that approved nothing — the release gate became the decision point, because a gate that can say "no" to a release is a committee that meets every day.

The pattern: governance that lived in the pipeline survived; governance that lived in meetings didn't.

13 · RESULTS

Outcomes, described without invented metrics

• The agent went to production with write authority — something the risk committee would not have approved without the governance pack. The pack wasn't overhead; it was the permit.
• The release gate caught regressions before users saw them, including the prompt "improvement" that human reviewers had approved — the harness became the organization's shared definition of safe to ship.
• The staging incident became the internal proof story: the guardrail caught a real failure mode with a log and a fix, and the narrative shifted from "governance slows us down" to "governance is why we're allowed to do this."
• Audit and regulatory questions were answered from the audit trail, not reconstructed from memory — every tool call, decision, and approval reconstructable after the fact.
• The governance artifacts transferred: when a second agent use case started, it inherited the eval harness, the risk tiers, the approval model, and the permission discipline — the pack was reusable, not bespoke.

14 · LESSONS LEARNED

What generalizes

• Governance is a release system, not a document. The controls that mattered all lived in the pipeline — the eval gate, the schema validation, the permission scopes. The controls that died all lived in meetings.
• The eval set is the policy, written in executable form. "The agent must ask for approval before updating a case" is a sentence in a doc; an eval case that fails when it doesn't is a control. Write the policy as tests.
• Design for approval fatigue. The real enemy of human-in-the-loop isn't the agent — it's the approver who stops reading. Targeted approvals stay meaningful; blanket approvals become theater.
• Least privilege for agents has weirder failure modes than for people. A human wouldn't paste an account number into a note field "as a summary." The agent did. Permission design has to account for failure modes that don't exist in human workflows.
• The incident you catch in staging buys the budget for governance. Nothing persuades like a caught failure with a log. Build the controls before you need the story.
• Governance nobody uses dies. The per-prompt review board and the transcript QA program were well-intentioned and unread. If the control isn't in the path of the work, it's decoration.

15 · HOW ANOVACLOUD APPROACHES THIS

The sprint version of this work

This case study is the long form of what our Evals & AI Governance Sprint compresses into a fixed-scope engagement: an eval harness built from your real questions and your real failures, wired as a release gate; guardrail design with explicit built-vs-bought-vs-skipped decisions for your threat surface; human-in-the-loop boundaries drawn with the first-month-analyst test; a least-privilege permission model for your agent's tools; and an incident playbook written before you need it. The methodology is the one described above — governance in the pipeline, policy as tests, controls that survive contact with production. No prices on this page; the engagement is scoped to your agent surface and your regulatory posture.

Explore the Evals & AI Governance Sprint →

16 · RELATED

Pattern, toolkit, service