Observability for Evaluation: Tracing an Agent Run Step by Step

Identity statement: step-level tracing is the capture, for a single agent run, of what happened at each individual step — the input a step received, the output it produced, how long it took, and how it connects to the step before and after it — assembled into a navigable record of that one run's execution rather than a number summarizing many runs at once. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md): "The toolkit integrates with observability platforms — Phoenix, Weave, Langfuse, and OpenTelemetry — so you can trace agent runs step by step rather than guessing at aggregate numbers."

That last clause is the identity statement's real payload: tracing exists as a direct alternative to guessing at aggregate numbers. An aggregate evaluation score and an aggregate profiler metric are both real, both accurate, and both fundamentally summaries — they compress many runs, or many steps within a run, into a small number of statistics, and compression is lossy by design. A trace is the opposite of a summary: it is the uncompressed record of one specific execution, preserving exactly the step-by-step detail an aggregate throws away in order to be a manageable single number. [VENDOR SPEC] (Sources/ncp-aai/domain-3-evaluation-tuning.md) names the toolkit's specific observability integrations for producing that record: Phoenix, Weave, Langfuse, and OpenTelemetry.

Why a trace and a benchmark score are not competing instruments

A fixed evaluation benchmark and a step-level trace are not two ways of answering the same question at different levels of detail — they are complementary instruments serving two different moments in the same investigation. The benchmark tells you that a problem exists and roughly how big it is (11 of 50 tasks failed, concentrated in one category). The trace tells you, for any one of those 11 failing runs, where inside that run things actually went wrong. You need the benchmark first, to know there is something worth investigating at all and to have a fixed, repeatable way to confirm a fix later; you need the trace second, to actually find what to fix. Skipping the benchmark and going straight to tracing individual runs risks chasing an anecdote that is not representative; skipping tracing and staying at the aggregate score risks knowing a problem exists without ever locating it.

What a single step's record actually contains

Concretely, tracing one step of one run typically preserves several distinct pieces, and it is worth naming them individually because a partial trace — one missing a piece below — can quietly fail to answer the exact question you needed it for. The step's input: the exact text, structured data, or tool arguments that step received, not a paraphrase of it. The step's output: the exact text, structured data, or tool response that step produced. A timestamp and duration, tying the step to M3-02's profiler data so a slow step and a wrong step can be cross-referenced against the same run. And the step's position in the chain — which step came immediately before it and fed it that input, and which step came immediately after and consumed its output — because a step's correctness can only be judged relative to what it was actually given, not relative to what an ideal run would have given it. Strip out any one of these — say, keep the outputs but discard the inputs — and a trace stops being able to answer "did this step do the right thing with what it received," which is usually the exact question root-cause analysis needs answered.

L1 — Intuition

Think of an aggregate evaluation score as a hospital's overall mortality rate for a given month, and a trace as a single patient's full chart. The mortality rate tells the hospital administration that something needs attention and roughly how urgent it is; it says nothing about which specific complication, in which specific case, actually caused a specific death. Only the individual chart — the sequence of what was measured, what was administered, and when, for one patient — lets a clinician trace backward from an outcome to its actual cause. Root-cause analysis on an agent's failing runs works the same way: the aggregate score flags that there is a real problem; the trace is the chart you read to find out what it actually was.

L2 — Mechanism

Mechanically, root-cause analysis over a traced run means walking the recorded sequence of steps — what each step received as input, what it produced as output, and where the chain of inputs and outputs first diverges from what a correct run would look like. Concretely: if a support agent's answer to a billing question was wrong, the trace lets you check, in order, whether the retrieval step returned the right billing document (a retrieval failure), whether the reasoning step drew a valid conclusion from a correctly retrieved document (a reasoning failure), or whether the final generation step correctly rendered a valid conclusion into the actual text the user saw (a generation failure). Each of those three failure locations calls for a completely different fix — a retrieval-index change, a prompt change to the reasoning step, or a formatting change to the final generation step — and an aggregate score alone cannot tell you which of the three you are looking at. Only the trace, read step by step, can.

This is the exact same "which step is actually responsible" question M3-02's profiler asks about resource cost, now asked about correctness instead. The profiler locates where time and tokens went; a trace locates where a run's actual reasoning and output went right or wrong. The two questions rhyme structurally — both are about attributing an aggregate observation to a specific step — but they are answered by two different kinds of record: the profiler's timing/token data, and the trace's full input-output record at each step.

L3 — The exam-relevant edge case: a single final output cannot substitute for a trace

The edge case worth naming precisely: looking only at a run's final output and comparing it to the expected answer is not root-cause analysis, even though it tells you whether the run passed or failed. A wrong final answer is consistent with a retrieval failure, a reasoning failure, or a generation failure — the final output alone cannot distinguish between the three, because all three failure modes can produce the identical symptom of "wrong final answer." A scenario question that describes an engineer diagnosing a failure by reading only the final output, without the step-level trace behind it, is describing a diagnosis that has not actually located a root cause — it has only confirmed that one exists, which the evaluation score already told you.

⭐ THE EARNED INSIGHT An aggregate score and a final output both answer "did this work," and neither one can answer "why not" — that second question requires a fundamentally different kind of record, not a more careful reading of the same one. Root-cause analysis is not a deeper look at the number you already have; it is a switch to a different instrument that was built to preserve exactly the step-by-step detail the aggregate had to discard in order to become a single number in the first place.

Comparing two agent versions on the fixed evaluation benchmark from M3-01 tells you that one version scored differently from another. It does not, by itself, tell you whether that difference is the change you intended, or an unrelated confound riding along with it. Step-level tracing is what lets you confirm the two are the same thing.

Consider two versions of the support agent compared on the identical fixed benchmark: version 1.1 scores 81% on how-to questions, and version 1.2, after a deliberate change to the retrieval step's query-reformulation logic, scores 90% on the same category. The aggregate comparison alone supports the claim "version 1.2 is better at how-to questions," but it does not yet support the stronger, more useful claim "the retrieval change is why." Confirming that second claim requires tracing individual runs from both versions and checking, step by step, that the runs which flipped from failing to passing actually flipped at the retrieval step specifically — that version 1.2's retrieval step is now returning a more relevant document on the same query, and that the downstream reasoning and generation steps are otherwise unchanged in how they handle a correctly retrieved document.

Without that trace-level confirmation, a version comparison is vulnerable to exactly the attribution problem M3-01 warned about in a different form: if the retrieval-logic change happened to ship in the same version bump as an unrelated prompt tweak to the reasoning step, the aggregate score improvement could be entirely attributable to the prompt tweak, with the retrieval change contributing nothing — or even mildly hurting, offset by a bigger gain elsewhere. Tracing individual runs from both versions, step by step, is what turns "version 1.2 scored higher" into "the retrieval change is what made version 1.2 score higher," which is the claim you actually need to decide whether to keep the change, revert it, or invest further in the same direction.

There is a second, subtler failure mode a version-to-version trace comparison catches that an aggregate score comparison structurally cannot: a run that passes both versions' evaluation, for two different reasons. Suppose a specific how-to run failed under version 1.1 because retrieval returned a stale document, and passes under version 1.2 — but the trace shows the retrieval step is still returning the same stale document; what changed is that the reasoning step, altered by an unrelated tweak in the same version bump, now happens to reach the correct conclusion anyway, from the wrong document, by a chain of reasoning that will not generalize to the next query with a different wrong document. The aggregate score treats this run as an unqualified win for version 1.2. The trace reveals that the actual retrieval bug is still present and unfixed, quietly masked by a second change that patched over its symptom on this one query without addressing its cause. Nothing about comparing two aggregate scores can surface this; it takes reading the specific steps of the specific run to notice that the fix worked for the wrong reason.

A team investigates 11 failing how-to-question runs from the 50-ticket benchmark (M3-01's worked example), pulling the step-level trace for three representative failures.

Trace — Run 1 (failed)
  Query: "How do I reset my two-factor authentication device?"
  Retrieval step  -> returned: "Password Reset" article (WRONG document; query
                     was about 2FA device reset, not password reset)
  Reasoning step  -> drew a valid-looking conclusion from the WRONG document
  Generation step -> rendered that conclusion faithfully into the final answer
  -> ROOT CAUSE: retrieval failure. The reasoning and generation steps behaved
     correctly given what they received; the wrong document never gave them a
     chance to produce a right answer.

Trace — Run 2 (failed)
  Query: "Can I change my billing cycle from monthly to annual mid-contract?"
  Retrieval step  -> returned the CORRECT policy document on billing-cycle changes
  Reasoning step  -> misread the document's stated exception for annual-to-monthly
                     changes as applying to monthly-to-annual as well
  Generation step -> rendered the reasoning step's (incorrect) conclusion faithfully
  -> ROOT CAUSE: reasoning failure. Retrieval did its job; the reasoning step drew
     an invalid conclusion from a correctly retrieved document.

Trace — Run 3 (failed)
  Query: "What's the maximum file size for a support-ticket attachment?"
  Retrieval step  -> returned the CORRECT document, stating "25MB per attachment"
  Reasoning step  -> correctly identified "25MB" as the answer
  Generation step -> rendered the final answer as "25GB" (a unit error introduced
                     during generation, not present in either upstream step)
  -> ROOT CAUSE: generation failure. Both retrieval and reasoning were correct;
     the error was introduced only in how the final answer was rendered.

This is a constructed scenario built to make the three-failure-mode taxonomy from Section 2 concrete; the specific queries and documents are illustrative, not drawn from a real support corpus. All three runs share the identical symptom — a wrong final answer — and an aggregate score treats all three identically as one failed task each. The trace shows they are three unrelated problems requiring three unrelated fixes: Run 1 needs a retrieval-index or query-reformulation fix, Run 2 needs a reasoning-step prompt fix addressing the exception-clause misreading, and Run 3 needs a generation-step fix addressing the unit-rendering error. A team that tried to fix "the how-to category" with a single change aimed at only one of these three failure modes would improve, at best, roughly a third of the failing runs, and would have no way of knowing that in advance without having traced the failures first.

The same step-level tracing capability — the same Phoenix, Weave, Langfuse, and OpenTelemetry integrations — is what a live, deployed agent's monitoring practice also depends on to diagnose a production anomaly, a concern this course's Run, Monitor, and Maintain material develops from the deployed-system angle: catching drift, diagnosing a live multi-agent conflict, and tracing an anomaly as it happens in production traffic. That is a genuinely distinct moment in an agent's lifecycle from the one this lesson addresses.

The underlying tracing mechanism is shared infrastructure — the same observability integrations, the same step-level record format — used for two different purposes at two different points in an agent's life. This lesson owns the evaluation-time use of that mechanism: root-causing a known benchmark failure and confirming a version comparison traces to the intended change. The live-production use of the identical mechanism is a distinct topic in this course's later material, reached once an agent is already deployed.

This sharing is a deliberate design choice, not a coincidence of two teams happening to pick the same vendor tools. Building one tracing mechanism that serves both moments means a trace format an engineer learns to read while root-causing a benchmark failure during development is the identical format they will read months later while diagnosing a live production anomaly — no second tool to learn, no second record format to reconcile against the first. It also means a genuinely difficult production incident can, once root-caused via a live trace, immediately become a new fixed benchmark item: the exact failing interaction, once understood step by step, gets added to the fixed evaluation task set so a future version's fix against it is verified the same repeatable way M3-01 established, rather than the fix being trusted on the strength of a single production trace alone. The tracing mechanism moves the finding backward, from live traffic into the fixed benchmark that governs future comparisons — closing a loop between two lifecycle stages the table above otherwise treats as separate.

Evaluation and Tuning carries 13% of the NCP-AAI blueprint, and this lesson sits at the connective point between two of the domain's five objectives: it supplies the step-level detail that Objective 3.5's targeted optimization (M3-06) needs in order to identify what to change, and it shares its underlying mechanism directly with Domain 8's monitoring and diagnosis work. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) states the connection explicitly: "Step-level traces are what make root-cause analysis and version-to-version comparison possible (this capability is shared with Domain 8's monitoring)." That parenthetical is itself testable — a scenario question can present tracing as if it belonged exclusively to one domain or the other, when the source material names it as genuinely shared infrastructure serving two different purposes.

Expect the question shape to present a failing or underperforming agent and ask what additional information, beyond the aggregate score or the final output, is needed to diagnose it — the keyed answer names step-level tracing, and the distractors typically offer "a bigger evaluation set" or "a stricter scoring rubric," both of which improve the aggregate measurement's precision without adding the step-by-step visibility a genuine diagnosis requires. A second shape names the four specific integrations — Phoenix, Weave, Langfuse, OpenTelemetry — directly, testing recall of the vendor list the source material states explicitly.

A final output only reveals whether a run's end result matched what was expected; it cannot reveal which of the run's several steps actually produced the discrepancy, because a wrong final answer is consistent with a failure at retrieval, at reasoning, or at generation — three completely different problems that all present identically from the outside. Section 4's worked example shows exactly this: three runs sharing the identical symptom of a wrong final answer traced back to three entirely unrelated root causes, each requiring a different fix. Only the step-level trace, which preserves each step's input and output rather than only the run's final result, can distinguish between them.

Both uses rely on the identical underlying mechanism — the same observability integrations capturing the same kind of step-level record — but they serve different moments in an agent's lifecycle and answer different triggering questions. Evaluation-time tracing investigates a known, fixed benchmark's failing tasks to find a targeted fix, and the fix gets re-verified against that same fixed benchmark once made. Production monitoring's tracing investigates live, previously unseen traffic once a monitoring signal — an anomaly, a drift indicator, a conflicting multi-agent action — flags something worth investigating, and the finding typically feeds an operational response like a rollback rather than a benchmark re-run. The mechanism is shared; the purpose, the trigger, and what happens with the finding are not.

1. An agent's final answer to a query is wrong. What is the strongest next step to find out why?
   • A. Increase the benchmark's sample size and re-run evaluation.
   • B. Trace the run step by step to see which specific step's input and output first diverged from a correct run.
   • C. Assume the reasoning step is always at fault, since it is the most complex step.
   • D. Rewrite the final generation step's prompt without further investigation.
2. Which four observability integrations does the NeMo Agent Toolkit surface for step-level tracing?
   • A. Prometheus, Grafana, Datadog, and Splunk.
   • B. Phoenix, Weave, Langfuse, and OpenTelemetry.
   • C. TensorBoard, Weights & Biases, MLflow, and Neptune.
   • D. LangChain, LlamaIndex, CrewAI, and Semantic Kernel.
3. Two runs both produce a wrong final answer. A trace shows Run A's retrieval step returned the wrong document, and Run B's retrieval step returned the correct document but its reasoning step misread it. What does this pair of traces demonstrate?
   • A. Both runs share the identical root cause.
   • B. The identical visible symptom (a wrong answer) can come from different steps, so each needs a different fix.
   • C. Retrieval is always the correct place to look first.
   • D. Reasoning failures are more common than retrieval failures in general.
4. A team ships a retrieval-logic change and a prompt tweak in the same version bump, and the evaluation score improves. What does the aggregate score comparison alone establish?
   • A. That the retrieval-logic change specifically caused the improvement.
   • B. That the prompt tweak specifically caused the improvement.
   • C. Only that this version scored higher than the last, without yet attributing the improvement to either specific change.
   • D. That both changes contributed exactly equally.
5. What is the relationship between evaluation-time tracing and production monitoring's tracing?
   • A. They are unrelated capabilities that happen to share a name.
   • B. They rely on the same underlying step-level tracing mechanism, applied at two different moments for two different purposes.
   • C. Production monitoring never uses tracing; only evaluation does.
   • D. Evaluation-time tracing replaces the need for production monitoring entirely.

Answers

1. B. A wrong final answer alone cannot say which step caused it; tracing the run step by step is what locates the actual point of divergence.
2. B. The source material names these four specifically as the toolkit's observability integrations for step-level tracing.
3. B. The identical symptom — a wrong answer — traced back to two unrelated causes, confirming that a final output alone cannot distinguish between failure modes that require different fixes.
4. C. With two changes bundled into one version, the aggregate score improvement cannot yet be attributed to either one specifically; tracing individual runs is what would confirm which change was responsible.
5. B. The mechanism — step-level tracing via the same observability integrations — is shared, but evaluation-time tracing investigates known benchmark failures while production monitoring investigates live, unpredictable traffic.

• An aggregate evaluation score and an aggregate profiler number both tell you that something is true across many runs; neither can tell you why any single run behaved the way it did — that requires step-level tracing.
• The NeMo Agent Toolkit's observability integrations — Phoenix, Weave, Langfuse, and OpenTelemetry — capture the step-by-step record a trace-based diagnosis depends on.
• A wrong final output alone cannot distinguish between a retrieval failure, a reasoning failure, and a generation failure; all three produce the identical visible symptom, and only a trace separates them.
• Confirming that a version-to-version score improvement is actually attributable to the change you made, rather than to an unrelated confound, requires tracing individual runs from both versions, not just comparing their aggregate scores.
• Different failing runs in the same category can fail for entirely different reasons; tracing a representative sample before proposing a fix avoids targeting only one of several distinct problems.
• The same tracing mechanism serves both evaluation-time root-cause analysis and live production monitoring — shared infrastructure, but a genuinely distinct purpose and trigger in each case.

Locating exactly which step caused a run to fail, as this lesson's traces do, still leaves open a separate professional judgment this module has not yet addressed: given that a fix is possible, is fixing it actually worth what it costs in latency or token spend, or is the current behavior an acceptable trade against those costs? That judgment call is where accuracy stops being the only thing worth optimizing for.

Next: M3-04 covers the accuracy-versus-latency trade-off directly — why production tuning rarely maximizes a single metric, and why "just maximize accuracy" is usually the wrong answer once the cost and latency this lesson's traces can reveal are actually part of the scenario.