Structured Feedback Loops for Iterative Agent Improvement

[GROUND TRUTH] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) Objective 10.2 calls for designing structured feedback loops that guide iterative agent improvement, where feedback — approvals, corrections, and ratings — is captured in a structured form and fed back into refinement. The word doing the work in that sentence is "structured": the same reviewer decision, recorded as a free-text comment buried in a chat log versus recorded as a labeled, categorized, queryable entry in a feedback dataset, has identical value to the single decision it gated and wildly different value to everything that comes after it. A structured record can be aggregated, counted, and used to retrain or re-tune a system; an unstructured one can, at best, be re-read by a human later, one item at a time, which does not scale any better than the review process that produced it in the first place.

Three forms of feedback get named specifically, and they are not interchangeable:

• Approvals — a binary or near-binary signal that a given output was acceptable as produced, with no correction needed.
• Corrections — a reviewer's edit to what the agent produced, which carries more information than an approval or rejection alone, because it shows not just that something was wrong but what the right answer actually looked like.
• Ratings — a graded score (a 1–5 scale, a pairwise preference) that captures degree rather than a binary judgment, useful when "acceptable versus not" is too coarse a distinction for the decision at hand.

L1 — Intuition: a locked door that also keeps a log is worth more than one that does not

Picture two identical review gates on an agent's output. Gate A checks each item, approves or rejects it, and the decision's only lasting effect is whether that specific item proceeds. Gate B checks each item the same way, but every approval, correction, and rating is written into a structured record that a separate process later reads to identify patterns — which kinds of outputs get corrected most often, which categories of decision draw the lowest ratings. Both gates cost the same in reviewer time and latency, per M10-01's accounting. Only Gate B turns that cost into an asset that keeps paying off after the specific request it gated is long forgotten.

L2 — Mechanism: what "fed back into refinement" actually means in practice

A structured feedback loop's value comes from what happens downstream of capture, and there are a few concrete shapes this refinement can take. Corrections accumulate into a dataset that looks exactly like the labeled data M10-02's supervised-learning technique consumes — a reviewer's edited output is, functionally, a new labeled example, and enough of them retrain or fine-tune the underlying model toward fewer future corrections of the same kind. Ratings aggregate into a signal that can drive the same kind of reward-model training M10-02 described for RLHF, if the volume and consistency of ratings support it — one rating alone tells you little, but a structured pool of thousands does. And approvals, even though they carry the least information of the three, still tell you something valuable in aggregate: a category of decision with a consistently high approval rate is a candidate for raising its active-learning confidence threshold (M10-02, section 4), because the human checkpoint is rarely changing anything there, meaning the review cost is not buying much marginal value at its current review rate.

L3 — The exam-relevant edge case: a feedback loop is not a replacement for a fixed evaluation benchmark

[GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) — the sibling domain's own framing, restated here for Domain 10's oversight angle — feedback informs refinement, but it does not replace the need for a repeatable benchmark — you still need a fixed evaluation to know whether a change actually helped, rather than merely shifted behavior in some direction correlated with recent feedback. This is worth stating precisely because it is an easy trap: a team that sees corrections decline after a retraining pass might conclude the agent improved, when an equally plausible explanation is that the mix of cases reaching review shifted — perhaps toward easier cases, or toward a category the correction volume happened to concentrate on recently — without the agent's actual accuracy on a fixed, representative task set changing at all. A structured feedback loop is a rich, continuously updating signal about where humans are currently intervening; it is not a controlled measurement of whether an intervention actually worked, which is exactly the job a fixed benchmark is built to do and a feedback loop, by its nature, is not.

The two are complementary rather than competing, in the same way M3-05 frames this pairing from the evaluation domain's side: a feedback loop tells you where to look and supplies raw material for retraining; a fixed benchmark tells you whether what you did with that material actually worked.

Constructed scenario, invented for teaching, illustrative figures. An agent drafting internal policy-compliance summaries runs under a human review gate. Over one month, the structured feedback log shows the following.

Constructed scenario, invented for teaching — illustrative figures, not measured data.

Week 1: 500 reviewed drafts,  62 corrected  (12.4% correction rate)
Week 2: 480 reviewed drafts,  58 corrected  (12.1% correction rate)
Week 3: 510 reviewed drafts,  31 corrected  ( 6.1% correction rate)
Week 4: 495 reviewed drafts,  29 corrected  ( 5.9% correction rate)

Step 1 — the naive read. Correction rate roughly halved between week 2 and week 3, which looks like the agent improved substantially — perhaps a retraining pass using weeks 1–2's corrected examples as new labeled data (exactly the mechanism section 2's L2 describes) shipped between week 2 and week 3.

Step 2 — the check a fixed benchmark supplies that the feedback log alone cannot. Before crediting the retraining pass, the team runs the same fixed task set used before the retraining pass and after it. If the fixed benchmark's score improved by a comparable margin, the correction-rate drop is corroborated by an independent measurement, and the retraining pass is a real, evidenced win. If the fixed benchmark's score barely moved, the more likely explanation is that weeks 3–4's live traffic mix simply included fewer of the harder cases that used to draw corrections in weeks 1–2 — a traffic-composition shift, not a capability improvement — and crediting the retraining pass for it would be a mistake the feedback log alone had no way to catch.

Step 3 — what the structured form specifically enabled here that an unstructured log would not. Because each correction was captured with a category tag (which policy section the summary misstated, in this constructed scenario), the team could also check whether the correction-rate drop was uniform across categories or concentrated in one — and in this scenario, it was concentrated in exactly the category the retraining pass's new labeled examples had emphasized, which is independent evidence pointing toward the retraining explanation over the traffic-shift explanation. An unstructured log of free-text reviewer comments would not have supported this category breakdown at all, which is the concrete difference between "feedback was captured" and "feedback was captured in structured form."

[GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) names the mechanism a structured feedback loop feeds as NVIDIA's own data flywheel idea: real usage generates data that improves the next iteration, which improves usage, and so on. The word "flywheel" is doing real work in that description, and it is worth being precise about why, because the exam's framing treats this as a compounding mechanism rather than a simple accumulation.

L1 — Intuition: a flywheel, not a bucket

A bucket collecting feedback just gets fuller — the tenth correction sits next to the first, unchanged in relationship to it. A flywheel, once it has enough mass spinning, makes the next push easier than the push before it: usage produces feedback, feedback produces a better agent, a better agent produces more usage (because it is now more useful), and more usage produces more feedback, faster than before. The structured form is what lets this compounding actually happen — a bucket of unstructured comments cannot spin anything, because nothing downstream can consume it fast enough to close the loop before the next batch of usage arrives.

L2 — Mechanism: where the compounding actually breaks if structure is missing

The flywheel's compounding depends on a specific chain staying intact: captured feedback has to be structured enough to feed a retraining or re-tuning process without a human manually re-reading and re-categorizing it first, because if a human has to do that re-reading step every cycle, the loop's speed is capped by that manual step and the flywheel stops accelerating — it becomes, at best, a linear process running at whatever rate that manual re-reading step can sustain. This is the concrete mechanism behind why "capture feedback in structured form" is not a nice-to-have alongside "capture feedback" — it is the specific design choice that determines whether the loop can compound at all, or whether it just becomes a slower, feedback-informed version of the same manual cycle a team was already running.

L3 — The exam-relevant edge case: a flywheel spinning in the wrong direction is still a flywheel

A detail worth naming because it inverts the intuition that "more feedback is always better": a flywheel compounds whatever direction it is already moving in, including a bad one. If the structured corrections being fed back into refinement are systematically biased — say, disproportionately drawn from one type of edge case because that is what happened to reach review most often in a given period — the resulting refinement pass can overfit to that overrepresented pattern, at the expense of patterns that were underrepresented in the feedback simply because they came up less often in recent live traffic, not because they matter less. This is precisely why section 2's L3 point about corroborating a feedback trend against a fixed benchmark matters even more once the mechanism is understood as a flywheel rather than a static accumulation: a flywheel that appears to be accelerating agent quality could instead be accelerating a narrowing of what the agent handles well, and only a benchmark measuring across the full, fixed task distribution — not just whatever the flywheel happened to emphasize recently — can catch that.

Constructed scenario, invented for teaching, continuing from section 4's policy-compliance summarizer. Suppose the retraining pass credited in section 4 for the correction-rate drop is investigated further, this time checking not just the aggregate rate but which policy categories the fixed benchmark's score moved on.

Constructed scenario, invented for teaching — illustrative figures, not measured data.

Fixed benchmark score, by policy category, before and after the retraining pass:

Category            Before   After   Change
Data-retention        71%     74%     +3 pts
Access-control        68%     70%     +2 pts
Vendor-disclosure      74%     91%    +17 pts
Incident-reporting     69%     68%    -1 pt

Step 1 — read the aggregate first. Averaged across all four categories, the benchmark score rose from about 70.5% to about 75.75% — a real, meaningful improvement that would look like unambiguous good news reported as a single number.

Step 2 — read the category breakdown the structured feedback log made possible. Nearly all of the aggregate gain is concentrated in one category, vendor-disclosure, which improved by 17 points — far more than the other three combined. That is consistent with section 4's finding that the retraining pass's new labeled examples happened to concentrate in exactly that category, because that category is what generated the most corrections in the weeks feeding the retraining data.

Step 3 — notice what the aggregate number would have hidden. Incident-reporting actually got very slightly worse, not better, and access-control and data-retention improved only modestly. A team reading only the aggregate 70.5% -> 75.75% figure would have no way to know that three of the four categories saw only marginal or negative movement, and that the headline improvement is really a single category's large gain, most likely because that category is what recent live traffic happened to emphasize in the corrections the retraining pass consumed.

Step 4 — decide what this means for the next cycle. The flywheel is spinning, and it produced a real capability gain in vendor-disclosure — that is not in question. But if incident-reporting matters equally to the product's actual risk profile, feeding the flywheel only what recent traffic happened to surface risks a system that keeps getting better at whatever it was already getting corrected on most, while quietly stagnating or regressing on categories that see less live traffic. The fix is not to distrust the flywheel mechanism itself, but to deliberately balance what feeds it — actively sampling underrepresented categories for review, rather than relying purely on whatever volume live traffic happens to route there — so the compounding improvement spreads across the categories that matter, not just the ones that happen to generate the most correction volume.

Because the decision's effect, without a structured record behind it, is limited to the single request it gated — it tells the system nothing about whether the same kind of case will come up again, or what pattern of corrections might reveal a fixable root cause. Capturing the decision in a structured form is what turns a one-time gate into a reusable signal: a pool of categorized corrections becomes new training data, a pool of ratings becomes reward-model material, and a pool of approvals becomes evidence for retuning how aggressively a system routes cases to review in the first place.

By producing a trend — a correction rate dropping, a rating average rising — that happens to correlate with a recent change without actually being caused by it. The most common way this happens is a shift in which cases reach review in the first place: if the mix of live traffic reaching the review gate becomes easier or narrower for reasons unrelated to any fix, the feedback log's numbers improve even though the agent's actual capability on the full range of cases it needs to handle has not changed. This is exactly why section 2's L3 point insists on corroborating any feedback-loop trend against a fixed, repeatable benchmark before crediting a specific change — the benchmark holds the task distribution constant in a way live traffic never does, which is the only way to separate a genuine capability improvement from a traffic-composition artifact that merely looks like one.

• A feedback loop's value depends on structure, not just capture — an unstructured log can be re-read but not aggregated, retrained on, or queried at scale.
• Approvals, corrections, and ratings are three distinct signal types, each useful for a different downstream refinement mechanism.
• Oversight that only gates decisions forfeits half its value — the other half is the reusable signal the gated decision leaves behind.
• A feedback loop is not a substitute for a fixed evaluation benchmark — corroborate any apparent improvement in feedback-loop trends against a controlled, repeatable measurement before crediting a change.
• Feedback-derived training data needs the same quality control as any other labeled dataset — a reviewer's own inconsistency can propagate directly into the next training pass if left unchecked.

⭐ THE EARNED INSIGHT

The entire value of a human review gate splits into two halves that are easy to conflate: the immediate decision (did this specific output proceed) and the residue that decision leaves behind (does the system know anything more, going forward, because a human looked at this). M10-01 costed out the first half in reviewer-hours and latency; this lesson is about the second half, and the uncomfortable fact is that paying the first half's cost buys you nothing of the second unless you deliberately design the capture to make it reusable — a gate with no structured memory behind it is, in the flywheel sense, exactly as blind after ten thousand decisions as it was after one.

Capturing feedback structurally is what makes a human's judgment reusable; the next objective in this domain is about making the agent's own reasoning reusable in a different sense — visible enough, after the fact, that a reviewer or an auditor can actually trace how a decision was reached in the first place. Next: M10-04 takes that up directly: explainable reasoning and decision traceability as named objectives in their own right, not cosmetic polish layered on top of whatever oversight mechanism is already in place.