Analyzing Evaluation Results to Guide Targeted Optimization

Identity statement: analyzing results to guide targeted optimization is a four-step discipline — categorize failures, identify the highest-impact weakness, change exactly one thing, re-evaluate on the fixed set — applied in that order, where skipping or reordering any step reintroduces a specific failure mode this module has already named in isolation. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md): "Instead of blanket changes, categorize failures (a step of error analysis shared with Domain 8), find the highest-impact weakness, change one thing, and re-evaluate on the fixed set. This mirrors sound experiment design: change one variable, compare against a baseline, attribute the effect."

That final sentence is the identity statement's real anchor, because it names exactly what this discipline is borrowed from: the one-variable-at-a-time, fixed-comparison logic this course's experiment-design material develops in full for machine-learning comparisons generally. Objective 3.5 is that same logic, applied specifically to the act of improving an agent rather than to comparing two arbitrary configurations — which means every failure mode a poorly designed experiment can fall into has a direct analogue here, and this lesson's job is naming each one in its agent-optimization form.

L1 — Intuition

An aggregate benchmark score that flags eleven failing tasks out of fifty is, on its own, a single undifferentiated pile of "things that went wrong." Categorizing those failures is the act of sorting the pile into distinct groups that share a common cause, the way a mechanic diagnosing a car that "won't start" first sorts the possibilities into distinct systems — electrical, fuel, ignition — rather than replacing parts at random until something works. Only once the pile is sorted into groups does "which group is biggest, or most costly to leave unfixed" become a question with an actual answer.

L2 — Mechanism

Mechanically, categorizing failures means grouping the failing runs from a benchmark by shared root cause, using exactly the step-level tracing M3-03 built for this purpose: pulling each failing run's trace, identifying which step diverged from a correct run, and grouping runs whose divergence happened at the same step and for a similar reason. M3-03's worked example already demonstrated the mechanics directly — three failing runs sharing the identical symptom (a wrong final answer) sorted into three distinct categories: a retrieval failure, a reasoning failure, and a generation failure. That sorting is the categorization step this lesson is naming; it is not a new technique, it is the direct application of tracing to the specific purpose of building an actionable map of an agent's weaknesses rather than a single number.

[GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) notes explicitly that this categorization step is "a step of error analysis shared with Domain 8" — the same failure-categorization discipline a live, deployed agent's monitoring practice uses to diagnose a production anomaly, connecting directly to the shared-infrastructure point M3-03 made about tracing serving two different lifecycle moments. Here, the categorization happens against a fixed benchmark's known failures; there, against live, unpredictable production traffic — the technique is identical, the object being categorized is different.

L3 — The exam-relevant edge case: a category that looks small in count but is large in impact

A subtlety worth naming precisely: the size of a failure category, measured by how many failing runs it contains, is not automatically the same thing as its impact. Three failing runs concentrated in a rare-but-catastrophic failure mode — say, an agent occasionally fabricating a specific figure in a regulated financial-disclosure context — can carry more real-world cost than eight failing runs in a low-stakes category, even though eight is a larger raw count than three. A scenario question that presents a category-by-count breakdown and asks which category to fix first is testing whether you will default to "biggest count" reflexively, when the source material's own framing — "find the highest-impact weakness" — names impact, not count, as the actual criterion.

⭐ THE EARNED INSIGHT Categorizing failures is not the hard part of this discipline, and treating it as the finish line is where teams most often stall out — a clean map of an agent's weaknesses, however well sorted, is still just a better-organized version of not having fixed anything yet. The genuinely hard judgment call is next: deciding which category on that map is actually worth fixing first, which requires weighing real-world cost and not just raw failure count, and that judgment is exactly what separates a team that produces an actionable optimization plan from a team that produces a very tidy list of problems.

Finding the highest-impact weakness means weighing at least three dimensions together, not defaulting to whichever one is easiest to read off a spreadsheet. Count — how many failing runs a category contains, which is the easiest dimension to measure and therefore the one most likely to be over-weighted by default. Severity — how costly a single failure in that category actually is in the real deployment context, which Section 2's regulated-disclosure example shows can dwarf a larger count in a low-stakes category. Frequency in real usage — how often the category's triggering condition actually arises in production, which a fixed benchmark's category proportions may not perfectly mirror if the benchmark's task mix does not exactly match real query volume.

The professional judgment this section is building is combining all three rather than anchoring on the one that is simplest to compute. A category with a high count, low severity, and low real-world frequency may rank below a category with a low count, high severity, and moderate real-world frequency — and no single dimension, read alone, would have surfaced that ranking correctly.

Where does the severity and real-usage-frequency information actually come from, if a fixed benchmark alone cannot supply it? This is exactly where M3-05's structured feedback re-enters the picture, now in a supporting role rather than as the subject of its own lesson: escalations tied to a given failure category are direct evidence of severity, since an escalation signals a case serious enough that a human had to step in, and the volume of ratings and corrections tied to a category, weighed against how often that category's query pattern shows up in real traffic logs, is direct evidence of real-usage frequency. None of this replaces the fixed benchmark's role in confirming a fix afterward — it feeds the ranking decision this section is about, which happens before any fix is made, using exactly the instruments this module spent five earlier lessons building.

Once the highest-impact weakness is identified, the discipline calls for changing exactly one thing to address it — the identical one-variable-at-a-time principle this course's experiment-design material develops for ML comparisons generally, now applied to an agent-optimization decision specifically. Changing the retrieval step's query-reformulation logic and, in the same release, also adjusting the reasoning step's prompt destroys the ability to say which change, if either, actually fixed the identified weakness — exactly the attribution failure a multi-variable experiment produces in any other context, discussed at length elsewhere in this course.

This constraint is genuinely uncomfortable in a real engineering setting, because a team investigating one weakness will often notice two or three plausible improvements at once, and the temptation to ship all of them together — "we're already touching this code, might as well fix everything we found" — is strong. The discipline's answer is not that the other improvements are not worth making; it is that making them in the same change destroys the specific piece of information this whole process exists to produce: which change caused which effect. A team that genuinely wants to make three improvements should make and re-evaluate them one at a time, in three separate cycles of this same four-step loop, rather than bundling them into one untraceable change.

A useful way to hold this constraint honestly under real engineering pressure is to distinguish "identified" from "shipped." Identifying three plausible fixes while investigating one weakness costs nothing to write down, and writing them down as a backlog for the next three cycles loses none of that information — the discipline is not asking a team to forget the other two fixes exist, only to defer executing them until each has its own cycle. What the discipline actually forbids is shipping more than one of them before the next re-evaluation, because that is the specific point where attribution, not the idea itself, gets destroyed. A team under deadline pressure to "just fix everything now" is trading a small amount of calendar time — one extra cycle per additional fix — for the ability to know, with confidence, which of its fixes actually worked, a trade that is almost always worth making given how expensive an unattributed regression can be to untangle later.

The fourth step returns directly to M3-01's instrument: re-run the identical fixed benchmark, with the identical scoring criteria, and compare the new result against the pre-change baseline. This is not a formality tacked onto the end of the process — it is the step that actually answers the question the whole exercise was trying to answer, which is whether the targeted change worked. Everything before this step — categorizing, ranking, choosing one variable — was preparation for making a single, attributable change; this step is where "attributable" gets cashed in as an actual, measured answer.

Re-evaluating on the fixed set also closes the loop back to M3-02 and M3-05 directly, because the full discipline this module has built is not four isolated lessons but one connected instrument set applied together: profile to find where cost is concentrated (M3-02), trace to find where correctness diverges (M3-03), weigh accuracy against latency and cost when a change trades one for the other (M3-04), let structured feedback surface problems a benchmark did not anticipate (M3-05), and analyze the resulting categorized failures to choose, execute, and confirm exactly one targeted change (this lesson). No single lesson in this module was ever meant to stand alone; this lesson is where all five converge into a single repeatable cycle a team can run again and again.

Return to the support agent's fixed 50-ticket benchmark, currently scoring an 86% aggregate with three failure categories traced in detail across this module.

Step 1 — Categorize (drawing on M3-03's traced failures)
  Retrieval failures:   4 runs  (wrong document retrieved)
  Reasoning failures:   3 runs  (correct document, invalid conclusion drawn)
  Generation failures:  2 runs  (correct reasoning, rendering error in output)
  Total failing runs:   9  (out of the current 86% aggregate's 7 failures... plus 2
                              newly identified in a deeper trace review this cycle)

Step 2 — Rank by impact (count x estimated severity x real-usage frequency)
  Retrieval failures:   4 runs, moderate severity, HIGH real-usage frequency
                         (this query pattern is common) -> HIGHEST IMPACT
  Reasoning failures:   3 runs, moderate severity, LOW real-usage frequency
                         (this specific exception clause is rarely queried)
  Generation failures:  2 runs, LOW severity (a cosmetic unit-rendering issue),
                         moderate real-usage frequency

Step 3 — Change exactly one thing
  Target: retrieval failures (highest impact)
  Change: update the retrieval step's query-reformulation logic to better
          distinguish "2FA device reset" queries from "password reset" queries
  Explicitly NOT changed in this cycle: the reasoning-step prompt, the
  generation-step rendering logic (deferred to future, separate cycles)

Step 4 — Re-evaluate on the identical fixed benchmark
  How-to category, before:  81%  (13/16)
  How-to category, after:   94%  (15/16)
  Billing, Access:          unchanged (82%, 94%) — confirming no side effect
  Aggregate:                86% -> 92%

This is a constructed scenario, illustrative rather than measured from a real system, built to make the full four-step cycle concrete in one place. Notice what the discipline explicitly did not do: it did not fix all three failure categories in one release, even though all three were identified and characterized. It ranked them, chose the retrieval-failure category as highest-impact once real-usage frequency was weighed alongside count, made exactly one change targeting that category specifically, and re-evaluated to confirm both that the targeted category improved (81% to 94%) and that the untouched categories held steady (82% and 94%, unchanged) — direct evidence the single change did not have an unintended side effect elsewhere. The reasoning and generation failure categories remain open, explicitly deferred to their own future cycles of this identical four-step loop, rather than bundled in now at the cost of losing attribution for any of the three.

The discipline's value is not limited to confirming successful fixes — it is equally valuable, arguably more so, at catching a plausible-sounding fix that turns out not to help, before the team commits further effort in the same direction. Continuing the same cycle, suppose the team next targets the reasoning-failure category (3 runs, the second-ranked weakness from Section 6), hypothesizing that the exception-clause misreading traced in M3-03's worked example can be fixed by adding an explicit clarifying sentence to the reasoning step's prompt.

Step 3 — Change exactly one thing (second cycle)
  Target: reasoning failures (second-highest impact)
  Change: add a clarifying sentence to the reasoning-step prompt, explicitly
          stating that billing-cycle exception clauses apply in one direction
          only (annual-to-monthly, not monthly-to-annual)
  Explicitly NOT changed in this cycle: generation-step rendering (still deferred)

Step 4 — Re-evaluate on the identical fixed benchmark
  How-to category, before this cycle:  94%  (15/16, from the retrieval fix's cycle)
  How-to category, after this cycle:   94%  (15/16 — UNCHANGED)
  Billing, Access:                     unchanged (82%, 94%)
  Aggregate:                           92% -> 92%  (UNCHANGED)

This is a constructed scenario, illustrative rather than measured. The re-evaluation shows no movement at all — the specific run that the reasoning-failure category was built around still fails, exactly as before the prompt change. Read against a trace of that specific run, the prompt clarification turns out not to have addressed the actual mechanism of the misreading; the exception clause's ambiguity was not in the reasoning step's general instructions at all but in how the retrieved document itself phrased the clause, which the added prompt sentence never touched. Because this cycle changed exactly one thing and re-evaluated on the identical benchmark, the team knows unambiguously that this specific fix did not work, rather than being left to guess whether a bundled set of changes had partially helped. The next cycle can now investigate the actual retrieved-document phrasing directly, informed by a concrete negative result rather than restarting the investigation from scratch.

Evaluation and Tuning carries 13% of the NCP-AAI blueprint, and Objective 3.5 closes the domain by naming this analysis-to-optimization step explicitly, tying it directly to sound experiment design: "change one variable, compare against a baseline, attribute the effect." [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) frames the domain's entire scope note around this same idea — targeted optimization, using measurement and instrumentation to decide what to change rather than guessing — which makes this lesson the domain's practical payoff: every earlier lesson supplied one piece of the measurement toolkit, and this lesson is where the toolkit gets used to make and confirm an actual decision.

Expect the question shape to present a categorized set of failures with counts, and ask which to address first — the keyed answer weighs impact (severity and real-usage frequency alongside count), and distractors typically offer the largest raw count as the tempting but incomplete answer. A second recurring shape describes a team making several changes at once in response to several identified weaknesses, then reporting an aggregate improvement — the keyed answer notes that no single change's contribution can be attributed from that result, mirroring this course's experiment-design material's multi-variable-attribution trap applied specifically to agent tuning.

"Making the agent better in general" has no defined starting point and no way to confirm afterward that a specific action caused a specific improvement — it describes a goal, not a process. Targeted optimization is the specific four-step process this lesson names: categorize the actual failures a fixed benchmark reveals, rank them by real impact rather than raw count alone, change exactly one thing addressing the highest-ranked weakness, and re-evaluate on the identical benchmark to confirm the change worked. The difference is not ambition — both aim at a better agent — it is whether the process that gets you there produces evidence you can trust versus an unattributable, unconfirmed set of changes.

Because changing more than one thing at once destroys the ability to attribute a subsequent score change to any single fix, which is the identical one-variable-at-a-time principle behind any sound experimental comparison, applied here to an agent-optimization decision specifically. Section 6's worked example shows the discipline in practice: three failure categories were identified and characterized in the same review, but only the highest-impact one — retrieval failures — was actually changed in that cycle, with the other two explicitly deferred to their own future cycles, so that the resulting benchmark improvement could be attributed cleanly to the one change that was made.

Never, in the sense that a deployed agent facing real, evolving usage will keep generating new failure categories to investigate — new query patterns, new edge cases structured feedback surfaces, new stale data a retrieval index has drifted against — so the four-step cycle this lesson describes is a standing practice, not a finite project with a defined end state. What does change cycle to cycle is the size of the remaining gap: early cycles on a newly built agent tend to close large, obvious weaknesses quickly, while later cycles on a mature, already-tuned agent tend to chase smaller, more specific categories with diminishing but still real returns. The decision to stop investing further cycles is itself a resource-allocation judgment — weighing the cost of another cycle against the shrinking size of the remaining benchmark gap — rather than a point where the agent becomes objectively "finished."

1. A benchmark's failing runs are categorized into three failure modes: retrieval (4 runs, common query pattern), reasoning (3 runs, rare query pattern), and generation (2 runs, cosmetic rendering issue). Which is the highest-impact weakness?
   • A. Reasoning, because 3 is a meaningful count.
   • B. Retrieval, weighing count together with its high real-usage frequency, ahead of the lower-frequency reasoning category and the low-severity generation category.
   • C. Generation, because it was identified most recently.
   • D. All three are equally urgent by definition.
2. A team identifies three separate fixes for three separate failure categories and ships all three in the same release. The aggregate benchmark score improves. What can the team conclude?
   • A. All three fixes worked equally well.
   • B. The specific contribution of any one fix cannot be attributed from this result alone, because more than one variable changed at once.
   • C. Only the first fix listed should be credited.
   • D. The result proves the categorization step was unnecessary.
3. What role does step-level tracing (M3-03) play in the categorization step of this lesson's discipline?
   • A. None; categorization uses only the aggregate score.
   • B. Tracing is what identifies which specific step diverged in each failing run, which is how failing runs get sorted into distinct root-cause categories.
   • C. Tracing replaces the need for categorization entirely.
   • D. Tracing is only used after a fix has already shipped.
4. Why is re-evaluating on the identical fixed benchmark, rather than a new or different one, the required fourth step?
   • A. A different benchmark would be faster to run.
   • B. Only the identical benchmark and scoring criteria make the before-and-after comparison attributable to the specific change made.
   • C. Re-evaluation is optional once the diagnosis seems clearly correct.
   • D. A new benchmark eliminates the need to check untouched categories.
5. What does this lesson's four-step discipline mirror from sound experiment design?
   • A. Randomized controlled trials with large sample sizes only.
   • B. Changing one variable, comparing against a baseline, and attributing the effect.
   • C. Always preferring the newest available model.
   • D. Running every possible change simultaneously to save time.

Answers

1. B. Impact weighs count together with severity and real-usage frequency; the retrieval category's high real-usage frequency, combined with its count, outweighs the lower-frequency reasoning category and the low-severity generation category.
2. B. With three changes bundled into one release, the aggregate improvement cannot be attributed to any single fix — the identical multi-variable attribution problem that applies to any experiment changing more than one thing at once.
3. B. Categorization depends directly on the step-level trace record established in M3-03, which shows where each failing run's behavior actually diverged from a correct run.
4. B. Only holding the benchmark and criteria fixed, exactly as before the change, lets the resulting score difference be attributed to the change itself rather than to a different, less comparable test.
5. B. The four-step discipline is a direct application of one-variable-at-a-time, baseline-compared, effect-attributed experiment design to the specific act of optimizing an agent.

• Targeted optimization is a four-step discipline: categorize failures, find the highest-impact weakness, change exactly one thing, and re-evaluate on the fixed set — mirroring sound experiment design applied to the agent itself.
• Categorizing failures uses the identical step-level tracing M3-03 built, sorting failing runs by shared root cause rather than leaving them as one undifferentiated count.
• The highest-impact weakness is found by weighing count, severity, and real-usage frequency together — a large count in a low-stakes category is not automatically more urgent than a small count in a high-stakes one.
• Changing exactly one thing per cycle, even when several fixes are already identified, is what preserves the ability to attribute a subsequent score change to a specific cause.
• Re-evaluating on the identical fixed benchmark after a targeted change confirms both that the targeted category improved and that untouched categories held steady, catching an unintended side effect the categorization step alone would not reveal.
• This lesson is the module's synthesis: every earlier instrument — evaluation, profiling, tracing, the accuracy-latency framework, structured feedback — feeds directly into the decision this lesson's four-step cycle makes and confirms.

This module opened by asking how to prove an agent works, tell why it is slow separately from why it is wrong, and turn that into a targeted fix rather than a guess — and the four-step cycle this lesson closes with is that question, answered completely: measure honestly, diagnose specifically, change narrowly, and confirm rigorously, then repeat. The next stage of an agent's lifecycle takes that same measured, tuned agent and asks what has to be true of the infrastructure underneath it before it can actually serve real traffic at scale.

Next: the next module, Deployment and Scaling, picks up from here, starting with how NVIDIA NIM serves a tuned agent's model calls as a containerized inference endpoint with its own latency budget and failure-handling requirements — the production environment this module's evaluation and tuning discipline was preparing the agent to enter.