MLOps and Governance for Deployed Agents: CI/CD, Monitoring, and Audit

Identity statement: MLOps, in the sense objective 4.2 names it, makes an agent's delivery process — build, test, deploy — repeatable and automated through a CI/CD workflow, rather than a manual, ad hoc process repeated by hand (and inconsistently) every time a change needs to ship. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): "Objective 4.2 makes MLOps a first-class deployment concern: CI/CD workflows, monitoring, and governance. Production agents need repeatable, automated delivery (build → test → deploy), monitoring hooks (handed to Domain 8), and governance for who can change what and how it's audited. These are not afterthoughts bolted on post-launch."

The word "repeatable" is doing specific work here, distinct from "automated." A process can be automated but still not repeatable, if it depends on undocumented, ad hoc decisions a script happens to encode today but that nobody could reproduce identically tomorrow if the script were lost or rewritten. Repeatable means the same input (a code change) reliably produces the same kind of output (a tested, deployed release) every time, regardless of who triggers it or when — which is what actually makes a deployment process trustworthy at scale, not merely fast. CI/CD (continuous integration, continuous delivery/deployment) is the concrete mechanism that delivers this: continuous integration runs an automated build and test suite against every code change as it is proposed, catching problems before they merge; continuous delivery (or deployment) automates pushing a tested, validated change out to production (or to a staging environment awaiting a final approval), so the path from "change proposed" to "change running in production" is the same automated path every time, not a bespoke manual sequence a specific engineer happens to remember correctly.

For an agent deployment specifically, the "test" stage of that build-test-deploy sequence is where M3-01's evaluation pipeline earns its place inside CI/CD rather than remaining a separate, occasionally-run exercise: a repeatable evaluation harness run automatically against every proposed change is exactly what catches a change that silently degrades task success before that change ever reaches production traffic. An agent CI/CD pipeline that runs unit tests on code but never runs the evaluation harness against the agent's actual task performance has automated only half of what "test" needs to mean for this kind of system — a change can pass every unit test and still make the agent measurably worse at its actual job, and only a task-level evaluation run inside the pipeline catches that.

L1 — Intuition

Imagine a company where anyone with a laptop can push a change directly to the production system that decides which customers get approved for a loan, with no review, no record of who made the change or why, and no way to answer "who changed this, and when" after the fact. Even if that company also has a perfectly automated CI/CD pipeline running every change through tests before it deploys, the pipeline answers "was this change tested," not "was this person allowed to make this change" or "can we reconstruct, after an incident, exactly what changed and who authorized it." Governance is the missing half: rules about who may propose, approve, and merge a change, and a record — an audit trail — of what actually happened, kept independently of whether the change itself was well-tested.

L2 — Mechanism

Governance for a deployed agent breaks into two connected pieces the source material's phrasing keeps distinct: "who can change what" (access control and approval authority) and "how it's audited" (a durable record of what changed, when, and by whom). Access control means specific changes — a prompt template, a tool's permissions, a model version, a production configuration value — are gated behind an explicit approval step, typically requiring a specific role or a specific reviewer's sign-off before a change can merge or deploy, rather than being open to anyone with repository or deployment access. Audit means every change that does happen leaves a durable, queryable record: what changed, in which commit or deployment, approved by whom, and when — a record that exists specifically so a later question ("why did the agent's behavior change on this date") has a concrete, traceable answer rather than relying on someone's memory of what happened.

The connection between the two pieces matters: access control without an audit trail means changes are gated, but nobody can later verify the gate actually worked as intended in a specific past instance — a reviewer's approval that leaves no record is functionally indistinguishable, after the fact, from no review having happened at all. An audit trail without access control means every change is recorded, but nothing stopped an unauthorized change from happening in the first place — the record simply documents that it did. Both pieces together are what let a team answer, with confidence, both "was this change properly authorized" and "can we prove it, after the fact."

L3 — The exam-relevant edge case: governance is not the same control as the security-and-audit material in Domain 9

The edge case worth holding precisely, because the two are easy to conflate on a scenario question, is that this lesson's governance concern — who can change the deployed system's configuration, code, prompts, and tool permissions, and how that is audited — is a deployment-and-scaling-domain concern about controlling changes to the system itself, distinct from Domain 9's safety-and-compliance material about controlling the agent's runtime behavior toward end users (content filtering, PII handling, execution-rail validation of tool calls at request time). Both use the word "audit," and both are legitimately about accountability, but they audit different things: this lesson's audit trail answers "who changed the deployed system and when," while Domain 9's audit trail (built around NeMo Guardrails and execution rails) answers "what did the agent do during a specific live interaction, and why." A scenario naming "governance" in a deployment context is very likely pointing at this lesson's change-control sense; a scenario naming "audit" in a runtime-safety context is very likely pointing at Domain 9's sense — reading which context a question is set in is what resolves which "audit" is meant.

Reading the table's third row is the fastest way to see why the two halves genuinely need each other rather than merely coexisting: CI/CD without governance is a fast, repeatable way to ship a change nobody was authorized to make, and governance without CI/CD is a careful, authorized way to ship a change nobody automatically tested — each half solves precisely the problem the other half does not touch, and a deployment with only one half is not "half safe," it carries the full risk of whichever half is missing.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) names "monitoring hooks (handed to Domain 8)" as part of objective 4.2's scope, which is a deliberate signal about where this lesson's boundary sits rather than an incomplete treatment. The monitoring hooks — instrumenting a deployment so that latency, error rate, throughput, and task-success signals are actually being captured and exposed as the deployment runs — are set up as part of the deployment work this lesson covers, alongside the CI/CD pipeline and the governance controls; the use of those hooks — building dashboards, defining alert thresholds, tracing anomalies, deciding what a metric drifting means for the deployed system's health, comparing a live agent's task-success rate against prior versions — is Domain 8's full treatment (Run, Monitor, and Maintain), covered in this course's own later material rather than duplicated here.

The practical takeaway from this boundary is not "monitoring doesn't matter here" — it is the opposite: monitoring hooks that were never wired into the deployment pipeline in the first place cannot be retrofitted painlessly later, because instrumentation is easiest to add at deployment time, alongside the CI/CD and governance work this lesson covers, rather than bolted on after a system has already been running unobserved. A CI/CD pipeline's deploy stage is exactly the right place to also wire in the metrics-emission and logging hooks Domain 8's dashboards will later read from — treating monitoring instrumentation as part of the same first-class deployment concern this lesson names, rather than a separate, later project.

Concretely, "wiring in a monitoring hook" at deployment time means the CI/CD pipeline's deploy stage configures the running service to emit specific, structured signals as it operates — a latency measurement per request, a success or failure flag per completed agent turn, a count of tool calls made, a version identifier tagging every emitted metric with which release produced it — to wherever Domain 8's dashboards and alerting will later read from. None of that instrumentation code is itself a dashboard, an alert threshold, or a monitoring policy; it is the plumbing a dashboard depends on existing before it can show anything at all. A team that defers this plumbing until after a reliability problem has already occurred discovers the problem with no historical signal to diagnose it by — the metric that would have shown exactly when and how the problem began simply was never being recorded, and no amount of dashboard-building after the fact recovers data nobody was collecting at the time.

Constructed scenario, illustrative only. An engineer proposes a change to a production agent's system prompt, intended to reduce a specific failure mode observed in recent traffic. Trace the change through both halves of this lesson's concern.

Step 1 -- Change proposed: engineer opens a pull request modifying the
  system prompt template, with a written rationale referencing the
  observed failure mode.

Step 2 -- CI stage (automated, repeatable): the pipeline automatically
  runs the agent's evaluation harness (M3-01) against the modified
  prompt over the fixed benchmark task set, comparing task-success
  rate against the current production prompt's baseline score.
  Result: modified prompt scores 91% vs. baseline's 88% on the
  targeted failure-mode subset, with no regression on the rest of
  the benchmark.

Step 3 -- Governance gate (access control): the pull request requires
  sign-off from a designated prompt-owner reviewer before it can
  merge -- a role-based approval requirement, independent of whether
  CI passed. The reviewer checks the evaluation delta from Step 2
  and approves.

Step 4 -- Audit record: the merge is recorded with the reviewer's
  approval, the linked evaluation results, and a timestamp --
  durable and queryable later.

Step 5 -- CD stage (automated, repeatable): the approved change
  deploys through the same pipeline every prompt change uses,
  including the monitoring-hook instrumentation that will let
  Domain 8's dashboards observe this prompt version's live behavior
  going forward.

Notice what each stage does and does not do on its own. CI (Step 2) caught whether the change actually improved the targeted metric without regressing others — a question governance cannot answer, because governance has no opinion about whether a prompt is good, only about who is allowed to ship it. Governance (Steps 3–4) caught whether the right person signed off and left a traceable record of that sign-off — a question CI cannot answer, because a test suite has no concept of organizational authority. Removing either step leaves a real gap: without Step 2, the reviewer in Step 3 is approving a change with no evidence it actually helps; without Steps 3–4, a well-tested change ships with no record of who decided it should, which becomes a real problem the moment a later regression needs tracing back to a specific decision.

Constructed scenario, illustrative only. Two teams, each missing one half of this lesson's concern, illustrate the specific failure mode each gap produces.

Team A -- has CI/CD, no governance:
  Any engineer can merge and deploy directly; the pipeline runs
  tests automatically and deploys automatically on a passing build.
  Incident: a prompt change ships that technically passes the
  evaluation harness's aggregate score but was never reviewed by
  anyone with domain expertise in the specific failure mode it was
  meant to fix -- it turns out to trade one failure mode for a
  worse, untested one the harness's fixed task set didn't happen to
  cover. Nobody approved this specific tradeoff, and there is no
  record of who decided it was acceptable, because nobody decided --
  it just passed the automated gate and shipped.

Team B -- has governance, no CI/CD:
  Every change requires manual reviewer sign-off and a manual,
  by-hand deployment process, carefully tracked in a spreadsheet.
  Incident: a manually-deployed change is missing one file that a
  human deployer forgot to copy during a late, rushed release window
  -- the kind of error an automated, repeatable pipeline would
  never make, because it applies the identical deployment steps
  every time regardless of how rushed or tired the person deploying
  is. The change WAS reviewed and approved, and there IS a record of
  who approved it -- but the deployment itself introduced a bug the
  review process was never positioned to catch, because reviewing a
  proposed change and correctly executing a deployment are two
  different failure surfaces.

Team A's gap is exactly what governance exists to close: automated speed with no authorized-review step, so a change ships without anyone accountable having actually decided it was the right call. Team B's gap is exactly what CI/CD exists to close: careful authorization with no automated, repeatable execution, so the one part of the process still done by hand — the actual deployment mechanics — remains exposed to the ordinary human error a repeatable pipeline is specifically built to eliminate. Neither team is "half-safe" relative to a team with both; each carries the full risk of whichever half is missing, which is the same point section 3's table makes from a different angle.

⭐ THE EARNED INSIGHT CI/CD and governance look, from a distance, like the same instinct expressed twice — "be careful with changes" — which is exactly why a team that has built one sometimes assumes the other is redundant. They are not the same instinct: one is a claim about process (did this change go through the same tested, repeatable steps every other change goes through), and the other is a claim about authority and provenance (was this specific person allowed to make this specific decision, and can that be proven later). A system can satisfy either claim perfectly while failing the other completely, which is exactly what Team A and Team B's incidents demonstrate — and it is why the objective names both rather than treating either as a stand-in for "we take deployment seriously" in general.

Everything so far has traced a change on its way into production. The same two halves matter again the moment a shipped change turns out to be wrong and needs to come back out, and the failure modes mirror the ones already named. A rollback executed by hand, under incident pressure, by whoever happens to be on call, is exactly the kind of rushed, manual process CI/CD's repeatable automation exists to remove from a deployment's riskiest moment rather than reserve automation for the calm, low-pressure case and leave the high-pressure case manual. A CI/CD pipeline that can redeploy a previous, known-good version through the identical automated path used for a forward deployment turns "roll this back" into the same repeatable operation as "deploy this," rather than a bespoke, improvised procedure invented under pressure — which is precisely when a repeatable pipeline's advantage over manual execution matters most, not least.

Governance's role in a rollback is equally direct: a rollback is itself a change to the running system, and the same question this lesson has asked of every forward change applies to it — who is authorized to trigger a rollback, and is that decision recorded. A production incident is exactly the moment an organization is tempted to let anyone with access push a rollback immediately, with the reasoning "we can sort out approval after the fire is out" — but skipping the audit trail at the moment of highest operational stress is precisely when a durable record of who decided what, and why, is most valuable for the incident review that follows. The fix is not to slow down an urgent rollback with the same approval latency as a routine change; it is to have a pre-authorized, faster governance path specifically for rollbacks — a designated on-call role empowered to trigger one immediately, with the audit trail captured automatically as part of the same automated rollback pipeline, so speed and accountability are not actually in tension once both are designed in from the start.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): MLOps and governance are objective 4.2, within Deployment and Scaling's 13% of the NCP-AAI blueprint. The domain's own framing states the timing trap directly, worth repeating because it is the shape a scenario question is likely to test: "These are not afterthoughts bolted on post-launch." A question describing a team that deployed first and planned to "add governance later, once things stabilize" is describing exactly the trap this framing names — governance and CI/CD are named as day-one concerns, not later maturity milestones.

How the question tends to be phrased

Expect a direct scope-recall item naming what objective 4.2 covers as a single concern — CI/CD, monitoring hooks, and governance together — with a distractor that names only one of the three as if it were the whole objective. Expect also a scenario item describing a deployment with only one of CI/CD or governance in place, asking which specific risk that deployment carries, testing whether a candidate can name the specific gap (unauthorized or unreviewed changes, versus untested or manually error-prone releases) rather than a generic "this is less safe" answer.

What the distractors typically look like

The house style favors treating MLOps as purely a technical automation concern with no governance dimension, or treating governance as purely an approval-workflow concern disconnected from CI/CD's repeatable-testing role — each distractor isolates one real piece of this lesson's concern and presents it as the whole answer. A second common distractor shape frames governance and audit as belonging entirely to Domain 9's safety-and-compliance material, missing this lesson's distinct, deployment-change-control sense of the same words.

CI/CD is the automated, repeatable pipeline that builds, tests, and deploys a change — it answers whether a proposed change was validated consistently every time, the same way, regardless of who proposed it or when. Governance is the separate set of controls over who is authorized to approve and merge a change in the first place, plus the durable record (the audit trail) of what actually changed, when, and by whom. A deployment needs both because they solve different problems: CI/CD's automation cannot enforce organizational authority, and governance's approval process cannot substitute for automated, repeatable testing — each is missing exactly what the other one provides.

Because retrofitting either onto a system that has already been running without it means applying controls to a history that already happened ungoverned or untested — you cannot retroactively make an already-shipped, unreviewed change into a reviewed one, and you cannot retroactively recover a repeatable test record for a release that was pushed by hand before any pipeline existed. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states this directly: "These are not afterthoughts bolted on post-launch." Starting both from the first deployment means every change, from the very first one, already has the tested-and-authorized provenance a later audit or incident investigation would need — a property no amount of later process improvement can apply backward in time.

• MLOps and governance are one first-class concern, not two optional add-ons — objective 4.2 names CI/CD, monitoring hooks, and governance together explicitly.
• CI/CD answers "was this change tested repeatably"; governance answers "was this change authorized, and can we prove it" — two different questions neither half can answer for the other.
• A deployment missing either half carries the full risk of that gap, not a partial risk — CI/CD-with-no-governance ships unauthorized changes fast; governance-with-no-CI/CD exposes deployment mechanics to ordinary human error.
• The task-level evaluation harness (M3-01) belongs inside the CI "test" stage — unit tests alone cannot catch a change that degrades actual task performance.
• Monitoring hooks are wired in at deployment time, alongside CI/CD and governance — their later use (dashboards, alerting) is Domain 8's separate, fuller treatment.
• The domain's own named trap is timing: governance and CI/CD belong on day one, not retrofitted after launch, because their protections cannot be applied backward to changes that already shipped without them.
• This lesson's "governance" and "audit" are about change control over the deployed system, distinct from Domain 9's runtime safety-and-compliance audit trail over the agent's live behavior toward users.

Delivering changes safely and governing who makes them answers how a deployment stays trustworthy over time — it says nothing yet about the other side of a production deployment's ongoing cost: how much capacity to actually run, and what that capacity costs against the availability it buys. That is where this module goes next: M4-05 covers balancing deployment cost against high availability — the over-provisioning-versus-under-provisioning tension that the profiling headroom M4-03 measured is exactly what lets a team right-size.

Next: M4-05 covers balancing deployment cost against high availability — using the headroom and hotspots M4-03's distributed-load profiling reveals to right-size capacity for peak demand and failover, without paying for idle capacity the rest of the time.