Logs, Errors, Anomalies, and Per-Step Tracing for a Multi-Agent System

Objective 8.2 exists because of a structural property of multi-agent systems, not because logging is generically a good idea. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) states the core finding directly: for multi-agent systems, tracking logs, errors, and anomalies is essential because these systems drift and can produce conflicting actions, and a single final output is not enough to diagnose why — you need the per-step trace to see where behavior diverged.

Unpack "conflicting actions" concretely. Picture a three-agent pipeline: a retrieval agent fetches source material, a drafting agent writes a summary from it, and a review agent checks the summary before it ships. Now suppose the shipped summary contains a factual error. Read only the final output, and every one of the following stories is equally consistent with what you see: the retrieval agent fetched an outdated document and the error was baked in from the start; the retrieval agent fetched the right document but the drafting agent misread or mangled it; or both upstream agents did their jobs correctly and the review agent approved a summary it should have flagged. These are not variations on the same fix — a retrieval problem calls for a different corrective action than a drafting problem, which calls for a different action than a review-agent problem, and none of that distinction is visible from the shipped text alone.

This is a genuinely different diagnostic problem from tracing a bug in a single-agent pipeline, where at least there is only one reasoning chain to inspect even if it is long. A multi-agent system multiplies the number of places a failure could have originated by the number of agents in the handoff chain, and every added agent adds another place drift or a conflicting decision can hide. The fix the objective calls for is telemetry and per-step tracing specifically — not "better logging" in a generic sense, but visibility into each agent's individual action, in the order those actions actually happened, so the investigation can walk the chain and find the point where one agent's output stopped matching what a correctly-functioning system would have produced.

The same capability, a different purpose than evaluation's observability

This module's per-step tracing is a close cousin of a capability this course covers earlier for a different purpose: observability tooling used to trace an agent run step by step for evaluation, generally deployed before or at launch to score how well an agent performs on a fixed benchmark. The tooling underneath both is largely the same — step-level trace capture — but the purpose diverges sharply. Evaluation's tracing exists to score a run against a benchmark before or at release, in a controlled setting where you know the right answer in advance. This lesson's tracing exists to diagnose a live run that already happened, in production, where nobody knew in advance which agent, if any, would misbehave. The tooling is shared; the moment it is used, and the question it is being asked to answer, are not — pre-launch observability asks "how good is this," live per-step tracing asks "what actually happened, and where did it go wrong."

L1 — Intuition: a relay race with no way to see the handoffs

Imagine a relay race where you can only see the finish line — you know the team's total time, and whether they crossed the line at all, but you cannot see any individual runner's leg. If the team finishes slower than expected, "the team was slow" tells you nothing about which runner actually slowed down, or whether the baton was dropped and recovered somewhere in the middle. A multi-agent system's final output, viewed alone, is exactly this finish-line-only view: it tells you the team's overall result, and nothing about which leg of the handoff actually went wrong. Per-step tracing is the equivalent of putting a timer and a camera at every handoff point in the race — now a slow finish can be attributed to a specific runner's specific leg, not the team in the abstract.

L2 — The mechanism: what a per-step trace actually records

[VENDOR SPEC] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) names the specific tools this integration relies on: the NeMo Agent Toolkit integrates with Phoenix, Weave, Langfuse, and OpenTelemetry for step-level tracing — the same observability capability this course covers for evaluation in its Evaluation and Tuning module, now applied to live production runs instead of a pre-launch benchmark. A per-step trace, at minimum, has to record, for every individual action any agent in the system takes: which agent took the action, what the action's input was, what the action's output or result was, how long that specific step took, and — for a tool-calling step — whether the tool call itself succeeded or failed. Strung together in order, these individual step records reconstruct the full path a run actually took through the multi-agent system, agent handoff by agent handoff, rather than only the beginning (the original request) and the end (the final output).

The diagnostic value of that reconstruction is specific: given a trace with this level of detail, an investigator can walk the chain of steps in order and compare each agent's actual output against what a correctly-functioning agent at that position should have produced, isolating the exact point where the actual trace and the expected behavior stop matching. That located point — "the retrieval agent's output at step 2 already contained the wrong document" versus "the retrieval agent's output at step 2 was correct, but the drafting agent's output at step 3 misrepresented it" — is the entire diagnostic payoff a per-step trace delivers that a final-output-only log structurally cannot.

L3 — The exam-relevant edge case: drift is a moving target, not a fixed bug

A subtlety worth holding onto: multi-agent drift is not always a discrete, one-time bug sitting at a fixed point in the pipeline, waiting to be found once and fixed forever. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) frames drift as a property of the system over time, meaning the specific agent or step responsible for a conflicting action on one run is not guaranteed to be the same agent or step responsible on the next run. A retrieval agent might drift toward slightly stale sources this week and a drafting agent might separately drift toward over-summarizing next week, and each drift shows up as a different divergence point in the trace even though both are instances of the same underlying phenomenon the objective calls "drift." This is why per-step tracing has to be a standing, continuously-available capability rather than a one-time debugging tool wheeled out for a single incident — the next incident may well trace back to an entirely different agent in the same pipeline, and only continuous tracing catches that without assuming the failure point stays fixed.

Not every anomaly a trace helps diagnose looks the same on the dashboard first. Recognizing which metric moves first for a given anomaly type narrows the investigation before a single trace has even been pulled.

The third row is the one worth sitting with longest, because it is the category M8-01's dashboard is structurally unable to catch on its own. A multi-agent system can hold flawless latency, throughput, error-rate, and uptime numbers indefinitely while one agent in the chain has quietly drifted toward worse decisions, or while two agents have started producing outputs that conflict with each other on a subset of tasks neither one is individually "erroring" on. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) names this directly as a common exam trap: monitoring only infrastructure metrics misses agent-behavior drift, and only step-level tracing, examined deliberately rather than triggered by an operational-metric alert, catches it.

Take a constructed scenario, illustrative rather than a captured real incident: a travel-planning system runs three agents in sequence — a preferences agent that extracts a user's stated constraints from their request, a search agent that queries available options against those constraints, and a booking agent that picks and confirms one option. A user reports the system booked a hotel outside their stated budget, with none of M8-01's four dashboard metrics showing anything unusual: latency, throughput, error rate, and uptime are all inside their normal ranges for this run.

Step 1 -- pull the full per-step trace for this specific run.
  Preferences agent  (step 1): input = "need a hotel under $150/night, downtown, for March 12-14"
                                output = { max_price: 150, area: "downtown", dates: "2026-03-12..14" }
  Search agent       (step 2): input = preferences agent's output above
                                output = 6 candidate hotels, all tagged "downtown", prices
                                         ranging $110-$310/night (not pre-filtered by max_price)
  Booking agent       (step 3): input = search agent's output above
                                output = booked candidate #2, priced at $215/night

Step 2 -- walk the chain and compare each step's output to what a correctly-functioning
agent at that position should have produced.
  Preferences agent's output: correctly extracted max_price = 150 -> matches expectation.
  Search agent's output: returned candidates without filtering by max_price at all -> diverges
    from expectation (a correctly-functioning search agent should apply the stated budget as
    a filter, or at minimum a rankable constraint, not return unfiltered results).
  Booking agent's output: booked the second-cheapest of six unfiltered candidates, still above
    budget -> this step is consistent with a booking agent doing exactly what it is designed to
    do (pick a reasonable option from what it was handed) given the flawed input it received.

Step 3 -- locate the divergence.
  The trace isolates the divergence to step 2, the search agent, not step 3, the booking agent
  that was blamed by the user-facing complaint ("it booked the wrong hotel"), and not step 1,
  the preferences agent, which correctly extracted the budget constraint in the first place.

Step 4 -- confirm with a segment check across other recent runs.
  Pulling five other recent runs' traces: the search agent's step-2 output omits price filtering
  in 4 of 5 additional runs as well, while the preferences and booking agents' steps look
  correctly behaved in all 5 -- confirming this is a sustained pattern in the search agent
  specifically, not a one-off fluke on this single run.

The diagnostic payoff is visible in what the trace ruled out as much as what it confirmed. Without the per-step breakdown, the natural first guess — "the booking agent picked the wrong hotel" — would have pointed the investigation at the wrong agent entirely, since the booking agent's step is fully consistent with correct behavior given the flawed input it was handed. The final output alone (a booking outside budget) is equally consistent with a preferences failure, a search failure, or a booking failure; only the trace, read step by step against what each agent should have produced at that position, isolates the actual point of divergence. And the segment check in step 4 is what elevates this from "one run went wrong" to "the search agent has a systematic issue" — exactly the distinction between a one-off and drift that the next lessons in this module (online regression benchmarking and automated versioning) are built to act on.

⭐ THE EARNED INSIGHT: The reason a multi-agent system's final output is not just insufficient but actively misleading as a diagnostic starting point is that it invites you to attribute a failure to whichever agent's fingerprints are most visible in the shipped result — usually the last agent in the chain, since it produced what the user actually saw — when the per-step trace frequently shows the true divergence sitting one or two handoffs earlier, at an agent whose output looked unremarkable in isolation but was already wrong by the time it reached the agent everyone's instinct blames first; per-step tracing is valuable specifically because it corrects that instinct systematically, agent by agent, rather than because it produces more logs.

Run, Monitor, and Maintain is Domain 8 of the NCP-AAI blueprint, weighted at 5% [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md), and this lesson is Objective 8.2: track logs, errors, and anomalies for root-cause diagnosis. The domain's own stated traps name this material directly: [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) lists "a single final output isn't enough to diagnose a multi-agent failure; you need the per-step trace" among its headline misconceptions, alongside "monitoring only infrastructure metrics misses agent-behavior drift." Expect the objective in a few recurring shapes:

1. Direct identification. "To diagnose why a deployed multi-agent system produced conflicting actions, you should rely on ___." The keyed answer is per-step telemetry and tracing, against distractors like "only the final output" or "a larger context window" that sound like plausible fixes but do not address the actual diagnostic gap.
2. Anomaly-type matching. A scenario describes a symptom (a latency-free, error-free but quietly-worsening outcome; a burst of failed tool calls; a run taking far longer than typical) and asks which named anomaly category it is, testing the taxonomy in section 3.
3. Blame-the-last-agent trap. A scenario like this lesson's worked example describes a bad final output and offers "the last agent in the chain made the wrong call" as a tempting answer, testing whether you recognize that the trace, not the intuitive read of the shipped result, is what actually locates the divergence.
4. Monitoring-vs-tracing depth. A stem contrasts "the dashboard shows everything is normal" against "users report worse outcomes," testing whether you recognize this as exactly the scenario behavioral drift produces — invisible to M8-01's metrics, visible only to per-step tracing.

What the distractors typically look like

The standard traps in this domain's style are: offering "the final output" or "the last agent in the pipeline" as sufficient evidence of where a failure originated, when the actual divergence is frequently upstream; offering "a larger context window" or "a bigger model" as a fix for a diagnosis problem rather than a reasoning-capacity problem; and offering "infrastructure metrics look fine" as proof nothing is wrong, when the domain's own scope note states directly that infrastructure metrics alone miss behavioral drift.

A multi-agent system's final output is produced by a chain of independent decisions handed off between agents, and a flawed result is equally consistent with the flaw originating at any point in that chain — an early agent supplying wrong or incomplete information, a middle agent misprocessing correct information, or a late agent approving output it should have caught. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) states this directly: a single final output is not enough to diagnose a multi-agent failure, and you need the per-step trace to see where behavior actually diverged, because the output alone gives no way to distinguish those competing explanations from each other.

Because infrastructure metrics — latency, throughput, error rate, resource utilization, uptime — measure whether the system is running, not whether its decisions are still correct, and a multi-agent system can run smoothly, with every request completing quickly and without error, while the actual quality or consistency of its agents' decisions quietly degrades. [GROUND TRUTH] (Sources/ncp-aai/domain-8-run-monitor-maintain.md) names this as a standing trap precisely because a team watching only M8-01's dashboard has no signal at all for this failure mode; catching it requires per-step tracing that examines what each agent actually decided, not just how fast or reliably it decided it.

• A multi-agent system's final output cannot distinguish where a failure originated. A flawed result is equally consistent with an early, middle, or late agent being the source, and only a per-step trace resolves which one it actually was.
• Three anomaly categories move different signals. Latency spikes and error surges show up in M8-01's metrics first; behavioral drift and conflicting actions frequently show up in neither, and need deliberate trace review to catch at all.
• Instinct blames the last agent in the chain; the trace often points earlier. The worked example's booking agent looked responsible on the surface but was executing correctly on flawed input it received from an upstream agent.
• Drift is not guaranteed to sit at a fixed point. A different agent can be the source of a conflicting action on a later occasion, which is why tracing has to be continuous rather than a one-time fix-and-forget exercise.
• The tracing tooling is shared with evaluation, but the purpose is not. Pre-launch observability scores a run against a benchmark; this lesson's live tracing diagnoses an already-occurred production failure.
• Infrastructure metrics alone cannot catch behavioral drift. A system can hold a perfectly healthy M8-01 dashboard while its actual decisions quietly degrade, which is exactly why this domain treats tracing as a distinct, necessary objective rather than folding it into dashboard monitoring.

Next: M8-03 picks up exactly where trace-level diagnosis leaves off — once you can locate which agent's behavior diverged on a given run, the next question is whether that divergence is a one-off or a sustained regression relative to the agent's own prior versions, which is what online task-success benchmarking against live traffic is built to catch continuously rather than one incident at a time.

Answers

1. To diagnose why a deployed multi-agent system produced conflicting actions, you should rely on:

• A. Only the final output
• B. Per-step telemetry and tracing ✅
• C. The exam blueprint
• D. A larger context window

Step-level observability reveals where behavior diverged; a larger context window addresses a reasoning-capacity problem, not a diagnosis problem.

2. A three-agent travel-planning pipeline books a hotel outside a user's stated budget. Which step in the pipeline is responsible?

• A. Always the last agent in the chain, since it produced the visible result
• B. Whichever step the per-step trace actually shows first diverging from correct behavior — which may not be the last agent ✅
• C. Always the first agent, since it set the original constraints
• D. It cannot be determined without retraining every agent in the pipeline

The worked example shows the divergence traced to the search agent, the middle step, even though the booking agent produced the visible, blamed-by-instinct result.

3. Why is infrastructure-metric monitoring alone insufficient for a multi-agent system?

• A. Infrastructure metrics are always inaccurate for agents
• B. Infrastructure metrics measure whether the system runs, not whether its decisions are still correct, so they miss behavioral drift ✅
• C. Infrastructure metrics only apply to single-agent systems
• D. Infrastructure metrics cannot be collected for agentic systems at all

A multi-agent system can run smoothly, with every metric in M8-01's dashboard healthy, while its agents' decision quality quietly degrades — a failure mode only trace review catches.

4. Which tools does the NeMo Agent Toolkit integrate with for step-level tracing?

• A. Only a proprietary NVIDIA-only tracing system
• B. Phoenix, Weave, Langfuse, and OpenTelemetry ✅
• C. Only OpenTelemetry, with no other integrations
• D. A single built-in logger with no external integrations

The source material names all four integrations explicitly as the mechanism behind step-level tracing.

5. A team notices a multi-agent system's latency, throughput, error rate, and uptime are all normal, but users report gradually worse outcomes. What does this describe?

• A. A latency spike
• B. An error surge
• C. Behavioral drift, invisible to the four operational metrics ✅
• D. A resource utilization anomaly

This is exactly the anomaly category M8-01's dashboard structurally cannot catch — none of its four metrics move, and only deliberate trace review reveals the degradation.

6. Why is per-step tracing for live production monitoring not the same activity as per-step tracing used for pre-launch evaluation, even though the underlying tools overlap?

• A. They are actually identical activities with no meaningful difference
• B. Evaluation tracing scores a run against a benchmark before or at release; live tracing diagnoses an already-occurred failure in production, with no known-correct answer to compare against in advance ✅
• C. Evaluation tracing never uses Phoenix, Weave, Langfuse, or OpenTelemetry
• D. Live tracing is only used before a system is deployed

The tooling is shared, but the purpose and moment of use diverge: one asks how good a run is against a benchmark, the other asks what actually happened and where it went wrong.

7. A team fixes the root cause of one drift incident in a multi-agent pipeline and stops tracing afterward, assuming the pipeline is now certified against drift. What risk does this leave unaddressed?

• A. None — drift, once fixed, cannot recur in the same pipeline
• B. A different agent in the same pipeline can become the source of a later, differently-located drift incident ✅
• C. This risk only applies to single-agent systems
• D. Fixing one incident automatically fixes every future drift source

Drift is not guaranteed to sit at a fixed point; continuous tracing, not a one-time fix, is what catches a later incident originating elsewhere in the same pipeline.

8. What is the primary diagnostic gap that a per-step trace closes, relative to a log of only the final input and output of a multi-agent run?

• A. It removes the need to measure latency at all
• B. It reveals which specific agent and step produced a divergence, rather than leaving every upstream agent equally plausible as the cause ✅
• C. It replaces the need for M8-01's dashboard entirely
• D. It only matters for single-agent, not multi-agent, systems

An input/output-only log leaves every agent in the chain equally plausible as the source of a bad result; the per-step trace is what narrows that down to an actual, located divergence.