Online Task-Success Regression: Benchmarking a Live Agent Against Prior Versions

[GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) states the objective directly: continuously benchmark deployed agents against prior versions, because evaluation doesn't stop at launch, and comparing the live agent against earlier versions is how you catch a regression — a change that degraded performance — that a pre-launch evaluation missed. Three phrases in that framing carry the actual weight this section unpacks.

"Continuously," not once. A pre-launch evaluation happens on a fixed schedule, at a fixed moment, before an agent goes live. It answers "was this version good enough to ship" as a one-time gate. Online task-success regression asks a structurally different, ongoing question: "is the version currently live still performing as well as the version it replaced," asked again and again for as long as the agent keeps serving traffic. The two questions sound similar but are not interchangeable, because a version can pass its pre-launch gate cleanly and still degrade weeks into production — from a downstream dependency quietly changing, from the population of tasks users actually bring it shifting, or from a subtle interaction the pre-launch test cases never exercised.

"The live agent," on real traffic. This is the load-bearing distinction from a fixed-suite practice, and it is worth being precise about what "live" removes. A fixed evaluation suite is a curated, unchanging set of test cases someone built in advance, chosen because they are believed to represent what matters. Live traffic is whatever real users actually send the agent, task by task, with no curation and no guarantee that today's traffic resembles yesterday's. Comparing task success on live traffic means comparing the current version's success rate on a population of tasks that is itself constantly shifting — which is exactly why this practice needs its own discipline for separating a real regression from a shift in what the traffic happened to contain, a discipline section 3 builds out.

"Against prior versions." Like the fixed-suite practice, this comparison is always relative, never absolute. The question is never "is this version good" in isolation — that belongs to the pre-launch evaluation this practice explicitly picks up where launch leaves off. The question is "is the currently live version succeeding at tasks as often as the version it replaced did," which makes "regression" the correct word for a decline: a regression is a decline relative to a known-good prior state, and this practice's entire value depends on having that prior state's success rate recorded to compare against.

L1 — Intuition: judging a chef by tonight's actual orders, not a fixed test menu

Picture two ways of judging whether a restaurant's new chef is as good as the one who left. The first way: hand the new chef a fixed test menu of twenty dishes, cooked once, scored by a panel — a controlled, repeatable test. The second way: track, night after night, what fraction of tonight's actual orders — whatever diners happened to request, a population that shifts with the season, the day of the week, and who walked in — come back sent back to the kitchen or complained about, and compare that fraction to what the previous chef's nightly numbers looked like. The first approach is clean and repeatable but only ever tests the twenty dishes on the menu. The second approach tests the chef against reality, at the cost that "tonight's orders" is never quite the same population as "last month's orders," so a change in the numbers might reflect the chef getting worse, or it might just reflect an unusually demanding table on a Saturday night. Online task-success regression is the second approach, applied to an agent: it measures success against whatever tasks actually arrive, and it has to account for the fact that that population itself keeps moving.

L2 — The mechanism: what "task-success rate" has to mean before it can be compared

Before any version-to-version comparison means anything, "task success" needs a defined, applied-consistently criterion for a given class of task — a booking either completed or it did not, a support ticket either resolved without escalation or it did not, a generated answer either passed a defined correctness check or it did not. Given that criterion, the mechanism has three moving parts:

1. A recorded baseline. The immediately prior version's task-success rate, measured over some window of its own live traffic before it was replaced, stored as the number the current version gets compared against.
2. A live measurement of the current version. The currently deployed version's task-success rate, measured continuously over its own live traffic, updated as more traffic accumulates rather than computed once and left static.
3. A comparison against an established noise band, not a raw difference. Because live traffic composition shifts on its own, even the same version's task-success rate will drift up and down somewhat from one measurement window to the next purely from which tasks happened to arrive — a noise band has to be established from a version's own repeat measurement before any gap between the current version and the recorded baseline counts as a real regression rather than the traffic simply looking different this week.

The third part is where this practice earns its difficulty relative to a fixed-suite comparison. A fixed suite holds the test population constant by construction, so nearly any repeatable score difference is attributable to the version. Live traffic holds nothing constant by construction — the population itself is a moving target — so the noise band has to be wider, and the comparison has to work harder to rule out "the traffic just changed" before concluding "the version regressed."

L3 — The exam-relevant edge case: a regression can be a traffic-mix problem wearing a version's clothes

The subtlety worth internalizing here is a specific failure of attribution: a drop in a live agent's aggregate task-success rate can be caused entirely by a shift in what kind of tasks are arriving, with the version itself performing exactly as well as it always did on each individual task type. Suppose a customer-support agent's overall task-success rate drops from 91% to 84% the week after a new version ships. If that week also happened to bring a disproportionate share of an unusually hard task category — say, a product-recall issue driving an atypical volume of complex, multi-step tickets — the aggregate drop could be fully explained by the traffic mix shifting toward harder tasks, with the new version handling every task category exactly as well, category for category, as the version it replaced. Concluding "the new version regressed" without segmenting by task category first would misattribute a traffic-mix effect to a version change, and the corrective action that follows from each diagnosis is different: a real version regression calls for investigating or rolling back the version, while a traffic-mix effect calls for nothing about the version at all.

Given the difficulty section 2's L3 names, checking for an online regression responsibly proceeds through a specific sequence rather than a single raw comparison:

1. Establish the current version's live task-success rate, measured over a large enough window of real traffic that the measurement itself is stable rather than a handful of runs' worth of noise.
2. Compare against the recorded baseline — the immediately prior version's own task-success rate, measured the same way, over comparable traffic.
3. Segment by task category or type before drawing any conclusion from the aggregate gap, specifically to rule out the traffic-mix effect section 2's edge case describes.
4. Check the gap against an established noise band for the metric, derived from how much a single version's own measured rate naturally moves from one window to the next on live traffic that keeps shifting.
5. Only past that gate, treat a sustained, segmentation-confirmed, noise-band-exceeding gap as a real regression worth investigating or acting on.

Take a constructed scenario, illustrative rather than measured from a real deployment: a customer-support agent's version 3 (v3) is replaced by version 4 (v4), which changes the underlying model used for the drafting step in its response pipeline. Task success is defined as "ticket resolved without a human escalation."

Step 1 -- establish v3's noise band from its own repeat measurements.
  v3's task-success rate over four consecutive one-week windows, prior to replacement:
    88.2%, 87.6%, 89.1%, 88.4%
  -> v3's own week-to-week variance is roughly +/-0.9 points; a gap smaller than that is noise.

Step 2 -- measure v4's live task-success rate in its first week live.
  v4, week 1: 84.7% overall task-success rate, on 2,400 live tickets.

Step 3 -- compare against the recorded baseline.
  v3's baseline (mean of its four windows): 88.3%.
  v4's week 1: 84.7%.
  Gap: -3.6 points, well outside the +/-0.9 noise band from step 1.
  -> outside the noise band; looks like a real regression on the aggregate number alone.

Step 4 -- segment before concluding anything.
  Ticket category breakdown for v4's week 1, compared to v3's typical category-level rates:
    Billing questions (900 tickets):      v3 ~90.1%   v4 = 89.8%   (flat, within noise)
    Account access (700 tickets):         v3 ~91.5%   v4 = 91.0%   (flat, within noise)
    Product-defect reports (800 tickets): v3 ~82.0%   v4 = 71.4%   (-10.6 points, large regression)
  -> the aggregate drop is concentrated almost entirely in product-defect reports.

Step 5 -- rule out a traffic-mix effect.
  Product-defect report volume in v4's week 1: 800 of 2,400 tickets (33.3%), compared to v3's
  typical share of 28% of weekly volume -- a real but modest increase in this category's share,
  not enough on its own to explain a 3.6-point aggregate drop if the category's own success rate
  had stayed flat. Since the category's own rate dropped sharply (82.0% -> 71.4%) rather than
  merely growing in volume, this is a real version regression concentrated in one category, not
  a traffic-mix artifact.

Step 6 -- root-cause follow-up (per M8-02's per-step tracing).
  Tracing v4's product-defect-report runs against v3's per-step behavior at the drafting step
  finds the new drafting model under-trained on defect-report-style tickets specifically,
  producing responses that omit a required warranty-claim step v3's drafting reliably included.

The payoff of steps 4 and 5 together is the whole lesson in miniature: without segmentation, the team would have correctly flagged a regression but investigated the wrong hypothesis space, treating the entire agent as uniformly worse when two of three ticket categories were performing identically to v3. Without ruling out the traffic-mix explanation specifically, a team could have dismissed the drop as "we just got more hard tickets this week" and shipped v4 unchanged, missing that the category's own success rate, not just its volume, had genuinely fallen. Only after both checks does the root-cause trace in step 6 have a correctly-scoped question to answer — not "why is v4 worse everywhere" but "why is v4's drafting step specifically worse for defect-report tickets."

⭐ THE EARNED INSIGHT: Because live traffic never holds still the way a fixed test suite does, an online task-success regression check has to do real diagnostic work before it can even agree with itself that a regression happened at all — establishing a noise band from the version's own variance, segmenting by task category, and separately ruling out a shift in traffic composition from a shift in per-category performance — and skipping any one of those steps produces exactly the two opposite failure modes that make this practice hard to do well: declaring a regression that was actually just a harder week's worth of traffic, or dismissing a real, category-concentrated regression as an unremarkable side effect of a busier category, when in fact the category itself got worse at handling the same kind of tasks it always handled.

This lesson's scope is deliberately narrow, and the narrowness matters because a closely related practice exists under a similar name in NVIDIA's generalist-LLM certification material, and treating the two as one and the same risks conflating what each is actually testing. "Offline eval-set regression" is the title of a lesson covering that material, and it owns a genuinely distinct angle: comparing a deployment's score against a prior version's score on a fixed, frozen, unchanging suite of test cases, built and curated in advance, rather than this lesson's practice of comparing task-success rate on real, unpredictable, in-flight traffic. The two share the umbrella idea — continuously benchmarking against prior versions — precisely because that shared vocabulary is where the two certs' material overlaps, and the boundary needs to be stated rather than assumed.

Both practices matter, and a mature deployment runs both rather than choosing one: the fixed-suite version catches regressions on scenarios someone already thought to build a test case for, cheaply and repeatably; this lesson's live-traffic version catches whether an agent is actually still succeeding at real tasks nobody wrote a test case for, at the cost of a noisier comparison that needs the segmentation and traffic-mix discipline sections 2 and 3 build. Recognizing which one a scenario describes is mostly a matter of one question: is the comparison running against a fixed, pre-built set of test cases, or against whatever real traffic happened to show up? That question, applied as a reading strategy rather than a cited fact, resolves the boundary in nearly every scenario either lesson's material is likely to describe, and this stays labelled inference rather than a fact drawn directly from either source file.

Run, Monitor, and Maintain is Domain 8 of the NCP-AAI blueprint, weighted at 5% [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md), and this lesson's material is Objective 8.3: continuously benchmark deployed agents against prior versions. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) frames the self-check item almost directly: continuously benchmarking a live agent against earlier versions primarily catches regressions in performance, keyed against distractors naming unrelated concepts entirely (license violations, GPU temperature, UI color drift) — a signal that the domain expects you to recognize "regression against prior versions" as its own distinct concept rather than a generic catch-all for "something changed." Expect the objective in a few recurring shapes:

1. Direct identification. "Continuously benchmarking a live agent against earlier versions primarily catches ___." The keyed answer is regressions in performance, against distractors that substitute an unrelated, plausible-sounding but wrong concept.
2. Continuous-vs-one-time framing. A scenario describes a team that evaluated an agent once before launch and never again, asking what risk this leaves — the correct diagnosis names the absence of ongoing regression benchmarking specifically.
3. Traffic-mix attribution. A scenario like this lesson's worked example presents an aggregate task-success drop and asks whether it proves a version regression — testing whether you check for a traffic-mix shift before concluding the version itself got worse.
4. Online-vs-offline discrimination. A question naming this lesson's live-traffic practice alongside a fixed-suite practice from NVIDIA's generalist-LLM material, testing whether the live-traffic, real-population framing is recognized as this domain's specific angle.

What the distractors typically look like

The standard traps in this domain's style are: offering a version regression as the automatic explanation for any aggregate task-success drop, without first checking whether the traffic mix itself shifted; offering "the dashboard metrics look fine" as evidence task success has held steady, when none of M8-01's five metrics measure task success at all; and conflating this lesson's live-traffic practice with the fixed-suite offline practice, since both are described as "benchmarking against prior versions" and the shared phrase invites exactly that conflation.

Live traffic is not held constant the way a fixed evaluation suite is — the population of tasks a live agent handles shifts on its own, week to week, purely from which real users happen to show up and what they happen to need. A fixed suite's repeatability means nearly any score difference between two runs is attributable to the version being compared; live traffic's constant shifting means a raw difference in task-success rate could equally reflect the version getting worse or the traffic simply getting harder, and separating those two explanations requires segmenting by task category and checking against an established noise band before drawing any conclusion, work a fixed-suite comparison does not need to do to the same degree because its test population never moves.

Because M8-01's five metrics — latency, throughput, error rate, resource utilization, and uptime — measure how the agent is running, not what it is accomplishing, and an agent can run quickly, cheaply, and without error while quietly failing at a growing share of the actual tasks it is asked to do. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) frames evaluation as a distinct concern from monitoring precisely because none of the operational signals a dashboard tracks are designed to capture task-outcome quality, and a version regression in task success can occur entirely underneath a dashboard that has never once left its normal range.

• Evaluation does not stop at launch. A version can pass every pre-launch check and still regress weeks into production; continuous comparison against a recorded baseline is what catches that.
• This practice compares against real, live, ever-shifting traffic — not a fixed test suite. That population never holding still is exactly what makes this comparison harder than a fixed-suite one.
• An aggregate task-success drop is not automatically a version regression. It can be a traffic-mix effect, and only segmentation by task category, plus checking whether the category's own rate (not just its volume) changed, tells the two apart.
• A noise band, established from a version's own repeat measurements, is required before any gap counts as a real signal. Live traffic's natural variance means a raw difference alone proves nothing.
• None of M8-01's five operational metrics measure task success. An agent can be fast, cheap, and error-free while quietly completing fewer tasks correctly than its prior version did.
• This lesson owns the live-traffic, online angle specifically — not offline, fixed-suite regression. The two practices share the "benchmarking against prior versions" umbrella but differ in test population and in what makes each one hard, and a related lesson in NVIDIA's generalist-LLM material owns the fixed-suite version of this idea.

Next: M8-04 picks up exactly where a confirmed regression leaves off — once online benchmarking or per-step tracing has located a real, attributable decline, the next question is how to version the specific tool contracts and prompt versions responsible for it and roll back safely and automatically, rather than patching the live agent by hand and hoping the fix holds.

Answers

1. Continuously benchmarking a live agent against earlier versions primarily catches:

• A. Regressions in performance ✅
• B. License violations
• C. GPU temperature
• D. UI color drift

Ongoing comparison against a recorded prior-version baseline detects degradation over time; the other options are real concerns elsewhere in the exam but are not what this specific practice is built to catch.

2. A live agent's aggregate task-success rate drops 3.6 points the week after a new version ships. What must be checked before concluding the new version regressed?

• A. Nothing further — any drop confirms a regression
• B. Whether the drop is concentrated in one task category and whether that category's own success rate, not just its volume, actually changed ✅
• C. Whether the agent's uptime also dropped
• D. Whether the model's parameter count changed

A traffic-mix shift toward a harder category can produce an aggregate drop with no actual version regression; segmentation by category, checking the category's own rate rather than its volume share, is what distinguishes the two explanations.

3. Why does online task-success regression need a wider or more carefully-established noise band than a fixed-suite comparison?

• A. Live traffic and fixed suites require identical noise bands
• B. Live traffic's task population shifts on its own between measurement windows, so even the same version's own rate naturally varies more than a fixed suite's repeat-run score would ✅
• C. Noise bands are unnecessary for live-traffic comparisons
• D. Fixed suites always show more variance than live traffic

A fixed suite holds the test population constant, so its own repeat-run variance is smaller and more predictable; live traffic's constantly shifting population means a version's own measured rate moves more just from which tasks happened to arrive.

4. Why is a healthy M8-01 dashboard insufficient evidence that an agent's task success has held steady?

• A. M8-01's metrics directly measure task success
• B. None of latency, throughput, error rate, resource utilization, or uptime measure whether the agent is still completing tasks correctly ✅
• C. M8-01's dashboard only applies during pre-launch evaluation
• D. Task success and uptime are the same measurement

The five operational metrics describe how the agent is running, not what it is accomplishing; a version regression in task success can occur entirely underneath a dashboard that never leaves its normal range.

5. How does online task-success regression differ from offline eval-set regression benchmarking, as covered in NVIDIA's generalist-LLM certification material?

• A. They are identical practices under two names
• B. This lesson compares a live agent's success rate on real, in-flight traffic against prior versions; the offline practice compares scores on a fixed, frozen test suite ✅
• C. Offline regression requires no comparison against prior versions at all
• D. Online regression only applies to latency, not task success

The two share the "benchmarking against prior versions" framing but differ in test population: real, shifting live traffic here, versus an unchanging, curated suite in the offline practice.

6. A customer-support agent's new version shows a flat aggregate task-success rate, but one ticket category's own success rate dropped sharply while its volume share also happened to shrink that week. What does this scenario illustrate?

• A. The flat aggregate proves nothing is wrong anywhere
• B. A category-level regression can be masked in the aggregate when the affected category's shrinking volume offsets its worse per-task performance ✅
• C. This proves the traffic-mix effect, not a version regression, occurred
• D. Aggregate task-success rate is always sufficient on its own

A real, localized regression can be diluted by an aggregate number in either direction — masked by a shrinking affected category's volume, or amplified by a growing one — which is why segmentation by category is necessary regardless of which way the aggregate moves.

7. What is the correct response once a task-success regression has been confirmed as real (non-noise, non-traffic-mix, and concentrated in a specific category)?

• A. Ignore it as long as the aggregate rate still looks acceptable
• B. Investigate the root cause, using per-step tracing at the affected category's specific steps, and consider rolling back the version ✅
• C. Widen the noise band until the gap no longer appears significant
• D. Stop measuring task success for that category going forward

A confirmed, correctly-attributed regression is exactly the finding this practice exists to surface — the correct response is root-cause investigation and, if warranted, a rollback, not adjusting the measurement to make the finding disappear.

8. Why must "task-success rate" be defined by a consistent, applied completion criterion before any version comparison is meaningful?

• A. It does not need a defined criterion; any subjective read is sufficient
• B. Without a consistent criterion, a score difference between versions could reflect a change in how success was judged rather than a real change in the agent's performance ✅
• C. A defined criterion is only necessary for offline, fixed-suite comparisons
• D. Task-success rate is identical to error rate and needs no separate definition

Just as a fixed suite's grading method has to stay constant for a score difference to be meaningful, an online comparison needs a stable, consistently-applied success criterion, or a change in judgment could be mistaken for a change in the agent's actual behavior.