Monitoring Dashboards and Reliability Metrics for a Deployed Agent

A running agent needs five separate operational questions answered continuously, and [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) lists exactly the metric classes that answer them: latency (p50/p95/p99), throughput (requests or tokens per second), errors (error rate, failed tool calls), resources (GPU utilization/memory), and availability (uptime). Read each as answering a distinct question rather than as a synonym for "everything's fine":

• Latency — how long did this one run take, start to finish? Measured per run, and it has to be summarized with a percentile split, because a single average number throws away exactly the detail that matters when one step out of several runs long.
• Throughput — how much load, in aggregate, can the deployment absorb right now? Measured as runs or tokens per second across the whole fleet, with no reference to how any individual run felt.
• Errors — how often does a run, or a step inside a run, come back unusable? This is the metric class the source material is explicit about scoping beyond the final answer: [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) names "failed tool calls" alongside error rate, not a failed final response alone.
• Resources — GPU utilization and memory headroom describe how much room is left on the hardware serving the agent's model before that hardware becomes the bottleneck.
• Availability — is the service even reachable, aggregated into an uptime percentage over some window?

A deployment that scores well on one of these can be quietly failing on another, because the five are structurally independent of each other. Aggregate throughput in the thousands of runs per minute is fully compatible with a subset of individual runs taking far longer than typical, since "can the fleet absorb the load" and "how long did any single run take" are not the same measurement at all — one is summed across everyone, the other is felt by one caller at a time. [VENDOR SPEC] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) notes that observability stacks surface exactly this set of signals for NIM-based and agentic deployments specifically — a sign that this is not generic server monitoring borrowed wholesale, but a vocabulary shaped around an agent's particular way of failing: in steps, any one of which can be the culprit on a given run and a different one on the next.

Resource signals describe headroom, not the present moment

GPU utilization and memory headroom belong on the same dashboard as the other four, but they are answering a forward-looking question rather than a present-tense one. A NIM-served model pinned at 96% utilization with almost no memory headroom left is telling you what is about to happen to latency and error rate under the next burst of concurrent traffic — it is not telling you anything about how any specific run just went. Reading a utilization figure as if it were itself a verdict on quality repeats, in miniature, the exact mistake a single mean latency number makes: both swap in something easy to compute for the number that actually reflects what a caller went through.

L1 — a plan has more places to stumble than a single call does

Set a lone API call next to a five-step agent plan. The call has exactly one place latency can go wrong: itself. The plan has five — a retrieval step, a tool invocation, a generation pass, any one of them can be the slow link on a given execution, and a completely different one can be the slow link the next time the same plan runs. Blend the total durations of many such executions into one average and you have mixed together runs where the retrieval step dragged, runs where the tool call dragged, and runs where nothing dragged at all, into a single figure that fairly represents none of them. This is precisely why percentiles matter more for an agent than for a one-shot API: more steps means more distinct doors a run can walk through on its way into the tail, and a mean slams all those doors shut before you can tell which one was actually open.

L2 — what p50/p95/p99 reveal once steps are chained together

Computing a percentile is mechanical: sort every run's total duration from fastest to slowest, then read the value sitting N% of the way along that list. What differs for an agent is what each point on that ladder is telling you. p50 describes a run where nothing unusual happened anywhere in the chain. p95 starts to catch runs where exactly one step — a downstream tool under load, an index answering slowly, a queued NIM endpoint — pushed the whole execution noticeably past typical. p99 catches the compounding case: a run unlucky enough to have more than one step land in its own bad tail simultaneously, stacking delays rather than averaging them away.

That stacking is worth dwelling on. Take five steps, each individually reporting a "clean" 200ms p99 measured at its own layer — the end-to-end p99 for the whole plan is not automatically five times that. It can land considerably higher, because different runs hit different steps' slow paths, and a run that happens to draw two slow steps at once falls further into the tail than either step's own percentile would predict in isolation. Knowing that the end-to-end tail is bad is only the first half of a diagnosis; knowing which step is actually contributing to it is the second half, and that second half is exactly what per-step tracing supplies once this lesson's aggregate view runs out of answers.

L3 — a calm aggregate can be hiding one badly-behaved tool

An aggregate percentile computed over an agent's whole run population can look entirely healthy while one specific tool, invoked by only a sliver of runs, is behaving terribly. Suppose a five-tool agent's overall p99 sits at a reassuring 900ms, but one tool is called on only 8% of runs and adds three full seconds whenever it is invoked. The percentile computed across 100% of traffic barely registers this, because the 92% of runs that never touch the troubled tool swamp the signal coming from the 8% that do. Nothing is wrong with percentiles as a concept here — the gap is in segmentation: an aggregate read without a breakdown by which tool, which branch, or which model version served a given run can pass every threshold on the dashboard while a real, severe problem sits invisible inside a minority of traffic.

Use this table the way a scenario question presents it: figure out which single metric a stem is actually asking about before answering, and notice which metric a tempting wrong answer is quietly substituting instead. A stem describing "the agent finishes 300 runs a minute, but individual callers say it feels slow" is testing whether you know throughput and latency are answering different questions; "throughput looks fine, so nothing's wrong" answers a question nobody asked.

Take a constructed scenario, illustrative rather than measured: a research agent runs a fixed five-step plan on every request — parse the query, call a search tool, call a retrieval tool, run a summarization pass, format citations — and its per-run total duration over one hour of traffic looks like this.

Twenty run durations (ms), unsorted, one hour of traffic:
1,120, 980, 1,340, 1,050, 1,190, 1,080, 4,600, 1,220, 990, 1,150,
1,010, 1,260, 1,090, 1,170, 1,300, 8,900, 1,140, 1,060, 1,230, 1,020

Sorted ascending:
980, 990, 1,010, 1,020, 1,050, 1,060, 1,080, 1,090, 1,120, 1,140,
1,150, 1,170, 1,190, 1,220, 1,230, 1,260, 1,300, 1,340, 4,600, 8,900

Percentile positions (n = 20 runs):
p50 -> 10th value  = 1,140 ms
p95 -> 19th value  = 4,600 ms
p99 -> 20th value (0.99 x 20 = 19.8, rounded up) = 8,900 ms

Mean, for contrast:
sum = 34,000 ms ; mean = 34,000 / 20 = 1,700 ms

The mean, 1,700ms, comes in above every ordinary run in the batch except the two outliers — it is a figure that matches no run that actually happened during this hour. p50 at 1,140ms is a fair description of the typical five-step run. p95 at 4,600ms already flags a run pulled seriously out of shape, and p99 at 8,900ms shows the worst case running almost eight times longer than typical. Two of twenty runs — a tenth of this small illustrative sample — had a run dramatically worse than the mean would ever suggest, and only the percentile view catches it. This is a constructed scenario: the numbers exist to make the sorting mechanics and the mean-versus-percentile gap checkable by hand, not to represent any measured deployment.

A second worked example: pushing throughput up can push latency's tail up too

The exam frequently sets latency against throughput specifically to test whether a "good" reading on one tells you anything about the other. Take another constructed scenario: the same research agent's operator raises the maximum number of concurrent runs the serving layer will accept, aiming purely to lift throughput.

Before raising max concurrency:
  Throughput: 18 runs/minute
  p50: 1,150 ms      p95: 1,900 ms      p99: 2,400 ms

After raising max concurrency:
  Throughput: 46 runs/minute    (+156%)
  p50: 1,300 ms                 (+13%)
  p95: 3,800 ms                 (+100%)
  p99: 7,600 ms                 (+217%)

The stated goal, more throughput, landed: 156% more runs completed each minute. Stopping the analysis there misses what it cost: p99 more than tripled. Mechanically, admitting more concurrent runs means more of them are now competing for the same downstream tools and the same GPU-served model at once, so each run's steps spend more time waiting their turn even while the fleet as a whole finishes more runs per minute — and that waiting concentrates in the tail, because a run unlucky enough to queue behind several others at multiple steps pays the full price while a run that dodges contention barely notices. Throughput rising and latency's tail worsening are not contradictory facts about the same underlying change; a stem describing "we doubled concurrency and throughput went up" as a clean win is testing whether you go check the percentiles before agreeing with it.

⭐ THE EARNED INSIGHT: A five-step agent's dashboard has more ways to mislead than a single-call service's, because a comfortable mean and an unsegmented p99 can both be hiding the identical fact underneath — one specific step, invoked on a minority of runs, is the actual source of every bad experience the aggregate is quietly smoothing over — and neither a bigger sample nor a tighter alert threshold on the aggregate number fixes that. The fix is always to break the same metric down by which step, which tool, which branch a run actually took, because an agent's latency problem is almost never "the agent is slow" in general; it is almost always "this one step is slow for this one slice of runs," a distinction no aggregate dashboard can draw on its own and that per-step tracing exists specifically to draw instead.

The five metrics in this lesson are raw material for the rest of the module, not a finished diagnosis. The instant the dashboard shows something real and sustained — a p99 that has drifted outside its usual range, an error rate climbing off its normal floor, utilization pinned high with headroom shrinking — the dashboard itself is done talking. It can report that a run took 8,900ms; it cannot say which of that run's five steps was to blame, and it cannot say whether the same step is the recurring offender or a different one nearly every time. Closing that specific gap is the entire point of the next lesson: per-step tracing turns a compounded, end-to-end number or an aggregate error rate into "which step, how often, under what conditions" instead of leaving it an unexplained wiggle in a graph.

There is a second, larger gap worth naming honestly. These five metrics describe operational health — speed, load-handling, failure frequency, reachability — and none of them say anything about whether the agent is still succeeding at the actual task it was built for. An agent can hold flawless numbers on all five of this lesson's metrics for months while the substance of what it produces quietly gets worse, because nothing measured here touches task success itself. That is a distinct failure mode from anything in this lesson, and this module returns to it directly once per-step tracing has given you the diagnostic footing to act on more than an alert.

Run, Monitor, and Maintain is Domain 8 of the NCP-AAI blueprint, weighted at 5% [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md), and this lesson is Objective 8.1: define monitoring dashboards and reliability metrics. [VENDOR SPEC] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) states the domain's scope note plainly: monitoring is continuous and behavioral rather than a one-time pre-launch check, and because agent behavior drifts, infrastructure metrics alone are not enough — you also need per-step tracing, which is exactly the handoff section 6 makes to the next lesson. Expect the objective to surface in a few recurring shapes:

1. Direct naming. "Which of the following belongs on a monitoring dashboard for a deployed agent?" keyed to one of the five metric classes, against a distractor borrowing an unrelated but plausible-sounding concept (parameter count, a training-time loss curve).
2. The averaging trap. A stem states an agent's "average step latency looks fine" and asks whether that certifies health — the correct read recognizes a low mean is fully compatible with a bad tail hiding underneath it.
3. Failed-tool-call scope. A scenario shows an agent whose final answers all look fine while a tool call intermittently fails inside the run, asking whether that belongs in error rate — [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) says explicitly it does.
4. Monitoring-versus-evaluation. A stem asks how monitoring differs from pre-deployment evaluation, testing whether you hold onto the domain's own line: monitoring is continuous and live, evaluation is a quality check tied to launch, and neither one is a substitute term for the other.

What the distractors typically look like

The house style here swaps in one real, correctly-named metric for the one a stem is actually asking about: offering mean latency where the question wants tail behavior; offering throughput as if it certified low latency, or the reverse; offering GPU utilization as a statement about the present rather than a warning about the near future; and offering "the final response looked fine" as if that ruled out a tool call having failed somewhere inside the run.

Latency is how long a single run of an agent's plan takes from arrival to final response, and it needs a percentile split — p50, p95, p99 — rather than a mean, because a multi-step plan's tail is exactly where one slow step goes to hide. Throughput is how many runs, or how many tokens across all of them, the deployment can carry per second, an aggregate figure with no reference to any one run's experience. The two move independently: a fleet can serve strong throughput while a slice of individual runs suffers, and a change made purely to raise throughput — accepting more concurrent runs, say — can simultaneously push the latency tail worse, because those runs now spend longer queued behind each other for the same tools and the same GPU-served model.

Because a run can return something that looks perfectly fine on the outside while a step that produced it failed on the inside — a tool call that timed out and got silently retried, or one whose error the agent's own logic simply absorbed without surfacing. A dashboard scoped only to whether the final output looks well-formed will never register any of that. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) groups "error rate, failed tool calls" together under one metric class precisely because a rising rate of internal, retried, or masked tool failures is an earlier warning than a broken final response ever will be, and counting only the latter throws that early warning away.

• Five metric classes answer five separate questions: latency, throughput, errors (including tool-call failures), resources, availability. None substitutes for another.
• Never reduce an agent's run latency to a mean. A multi-step plan has more places to go wrong than a single call does, and a mean papers over exactly the runs worth investigating.
• Throughput and latency's tail can move in opposite directions from the same change — raising concurrency can raise throughput and blow out p99 at the same time.
• Error rate has to reach inside a run to its individual tool calls, not stop at whether the final response looks fine.
• GPU utilization and memory are early warnings, not verdicts on the present moment — watch their trend ahead of a latency or error move, not instead of one.
• This lesson's dashboard is the raw material for the rest of the module. The moment a number moves, the dashboard alone cannot say why — that gap is exactly what M8-02 closes.

Next: M8-02 picks up right where this dashboard leaves off — capturing the logs behind these five metrics and tracing an anomaly step by step through a multi-agent or multi-tool plan, because a single aggregate number, no matter how carefully it is percentiled, still cannot name which specific step in the chain is actually misbehaving.

Answers

1. Which of the following belongs among the five metric classes a monitoring dashboard for a deployed agent tracks, per objective 8.1?

• A. Model parameter count
• B. Resource utilization (GPU/memory) ✅
• C. Training-time loss curve
• D. Tokenizer vocabulary size

Parameter count, loss curves, and vocabulary size describe the model or its training, not a live operational signal; resource utilization is one of the five named classes.

2. A team reports an agent's average step latency at 450ms and calls the agent healthy. What is the report missing?

• A. Nothing — a mean is sufficient on its own
• B. The p95/p99 latency figures, which can reveal a tail the mean cannot show ✅
• C. A count of total runs served that day
• D. The agent's parameter count

A mean can sit comfortably low while a tail caused by a single misbehaving step goes completely unnoticed underneath it.

3. An operator raises the maximum concurrent runs a serving layer accepts, and throughput jumps 150%. What should be checked before treating this as an unqualified win?

• A. Whether the agent's parameter count changed
• B. Whether p95/p99 run latency got worse as a side effect ✅
• C. Whether error rate is exactly zero
• D. Whether GPU utilization fell

More concurrent runs can raise throughput while making individual runs queue longer behind each other for the same tools and model, worsening the tail even as throughput climbs.

4. An agent's final responses all look well-formed, but one internal tool call intermittently times out and gets silently retried. Should that show up in error rate?

• A. No, error rate concerns only the final response
• B. Yes — error rate has to include failed tool calls, not only a failed final output ✅
• C. No, a retried call is never counted
• D. Only if the retry itself also fails

Objective 8.1 groups failed tool calls into the errors metric class explicitly, because a run can look successful on the outside while a step inside it failed.

5. GPU utilization is climbing and memory headroom is shrinking, while latency and error rate are still within their usual range. How should this be treated?

• A. Irrelevant until latency actually degrades
• B. A warning that latency or error rate is likely to degrade soon ✅
• C. Proof the deployment is running optimally
• D. Evidence that throughput must have been measured incorrectly

Resource metrics are forward-looking indicators, not statements about the present — they call for attention before latency or error rate visibly move, not after.

6. A five-step agent's aggregate p99 looks calm, but one tool, called on only 8% of runs, adds three seconds whenever invoked. What does this show?

• A. The aggregate metric must be broken and should be discarded
• B. A problem confined to a minority of runs can be diluted into invisibility inside an aggregate number ✅
• C. This proves throughput, not latency, is the metric to watch
• D. Percentiles cannot be computed for multi-step agents

Unaffected majority traffic dilutes the signal from the affected minority in an unsegmented aggregate — breaking the number down by tool or step is what exposes it.

7. How does monitoring a live agent differ from the pre-deployment evaluation covered elsewhere in this course?

• A. They are the same activity under two different names
• B. Monitoring is a continuous, live check on operational health; evaluation measures quality before or at release ✅
• C. Monitoring only runs during training
• D. Evaluation only runs after deployment

The domain's own scope note draws exactly this line: monitoring is continuous and behavioral, evaluation is a pre-launch or at-release quality check, and the two are not interchangeable.

8. Why does an agent's multi-step plan need end-to-end latency measurement rather than trusting per-step percentiles alone?

• A. Per-step measurement is always sufficient by itself
• B. Different runs can hit different steps' slow paths, so the end-to-end tail can compound worse than any single step's own percentile suggests ✅
• C. End-to-end measurement makes throughput tracking unnecessary
• D. Per-step latency cannot be measured for agents

A run that draws more than one slow step at once falls further into the tail than either step's own percentile would predict on its own — only the full run's end-to-end number captures that.