Monitoring an LLM in Production and Detecting Drift

Monitoring, for an LLM application, is the ongoing collection of three categories of signal. Quality telemetry asks whether the system's answers are still good — measured against a fixed, versioned evaluation set using the same metrics that validated the system before launch (exact match, the RAG-specific metrics from 09-07, or whatever rubric the project's eval set uses). Latency telemetry asks whether the system is still fast enough — TTFT, inter-token latency, and the p95/p99 tail figures 12-10 covers in full. Cost telemetry asks whether the system is still affordable — tokens consumed, dollars spent per request and per month, tracking against the accounting 12-09 sets up. None of these three is optional in a production system, and they answer genuinely different questions: a system can be fast and cheap while quietly getting less accurate, or accurate and cheap while getting slower, and only tracking one of the three hides failures in the other two.

Drift is what monitoring is trying to catch on the quality side specifically: a decline in an LLM system's output quality that happens gradually and without any single obvious triggering event. Drift has several distinct causes, and distinguishing them matters because each one is caught by a different signal and fixed a different way:

• Data drift. The corpus a RAG system retrieves from changes — new content, deprecated content, a shift in what users are actually asking about — and the system's answers get worse because what it's retrieving no longer matches reality, even though nothing about the model or code changed. This is the production-time symptom of the freshness problem 12-12 addresses at the index-maintenance level.
• Model drift. An underlying model the application depends on changes silently — a vendor updates a hosted model version behind the same API endpoint, or a model that used to answer a certain way now answers differently — and the application's behavior shifts even though the application's own code is untouched.
• Population drift. The population of users asking questions changes — a product ships a new feature, a support bot starts fielding questions about a topic the eval set never covered, a customer base expands into a new domain — and quality on the original eval set stays flat while quality on what users are actually asking now quietly falls, because the eval set no longer represents the real query distribution.
• Prompt or configuration drift. Someone edits a system prompt, changes a decoding parameter, or adjusts a retrieval setting for one purpose, and that change has an unintended side effect on a different capability the eval set would have caught — if it had been re-run.

All four are "drift" in the sense that quality degrades without an error, an exception, or an obvious single point of failure — which is exactly why monitoring has to be a standing, scheduled process rather than something triggered by an incident.

L1 — The intuition: a car's dashboard vs a scheduled inspection

A car's dashboard is monitoring's latency-and-cost half: speed, fuel level, engine temperature — instruments that report continuously and alert immediately if something crosses a hard threshold. But a dashboard doesn't tell you whether your brakes have worn thin, whether your alignment has drifted, or whether your tires are slowly losing their tread pattern — those things degrade gradually, the car still drives, nothing on the dashboard complains, and you only find out at the next scheduled inspection when someone actually re-checks the specific things that quietly wear down. An LLM's quality is the brakes-and-tires half of this picture: it doesn't throw an error when it starts hallucinating more often or retrieving less relevant context, it just keeps producing plausible-looking text, and the only way to catch the wear is a scheduled inspection — running the same eval set again — rather than waiting for a dashboard light that quality decay never triggers.

L2 — The mechanism: three telemetry streams plus a scheduled regression run

Quality monitoring in practice. The mechanism is a prompt regression suite: the same evaluation set that validated the system before launch (built per 01-08's crude-on-purpose approach — a small, hand-written, versioned set of question-answer or question-context-answer triples) gets re-run against the live system on a fixed schedule — nightly, weekly, or after every deployment, depending on how fast the system's behavior can plausibly change. The re-run produces the same metrics the original evaluation produced — exact match, a RAG-specific score, an LLM-as-judge rating — and those numbers get tracked over time. A regression suite that only runs once, at launch, is not monitoring; it's a one-time gate. The whole value of the suite is running it again and again against the exact same fixed inputs, so that any change in the score is attributable to something that changed in the system, not to a different question being asked.

Latency and cost monitoring in practice. These are continuous telemetry streams rather than scheduled batch runs — every request logs its TTFT, its total latency, its token counts in and out, and those get aggregated into the percentile figures 12-10 describes and the per-request and per-month cost figures 12-09 describes. Dashboards and alerting thresholds sit on top of this stream, catching sudden regressions (a deployment that doubles p95 latency, a traffic spike that blows the monthly cost budget) essentially in real time.

The asymmetry between quality and the other two. Latency and cost degrade in ways that are usually detectable from raw request logs without needing a fixed evaluation set — a slow request is objectively slow regardless of what question it answered. Quality degradation has no such objective, per-request signal: a wrong answer looks exactly like a right one to any automated system that isn't specifically checking it against a known-correct reference. This is the entire reason quality monitoring needs a fixed eval set re-run on a schedule, while latency and cost monitoring can run continuously off live traffic alone.

L3 — Detecting which kind of drift occurred

When a scheduled regression run shows a quality drop, the diagnostic question is which of the four drift types (Section 1) caused it, because the fix differs sharply by type:

• If the drop concentrates on questions about recently changed topics specifically, while older, stable topics still score well, that points to data drift — the corpus moved and the index didn't keep up (12-12's incremental-freshness fix), or genuinely new content that the fixed eval set never anticipated needs a new eval item added.
• If the drop is broad and sudden, appearing across unrelated question categories all at once, and correlates in time with no code or config deployment on your own side, that points to model drift — check whether an upstream hosted model's version changed, since some providers silently update a model behind a stable endpoint name.
• If the eval set's own score stays flat but real-world complaint volume or support-escalation rate rises, that points to population drift — the eval set is measuring the wrong distribution now, and needs new items reflecting what users are actually asking today, not just what they asked when the set was built.
• If the drop appears immediately after a specific deployment — a prompt edit, a decoding-parameter change, a retrieval-config tweak — that's prompt/configuration drift, and the regression suite should have caught it before that deployment shipped, which is the argument for running the suite as a pre-deploy gate as well as a scheduled cadence.

These three terms get used loosely and interchangeably in casual conversation about "production observability," but they answer different questions and fail differently when confused.

The distinction worth holding: evaluation is the one-time check that gates a specific change before it ships, monitoring is the standing process that keeps checking after it ships, and drift detection is the diagnostic step that runs once monitoring has already found something wrong. A system with excellent pre-deploy evaluation and no post-deploy monitoring will ship a good version and then have no idea when it stops being good — which is precisely the gap this lesson's blueprint design note calls "closing thread T1": 01-08 built the instrument that proved the system worked at launch, and this lesson is the same instrument run again, on a schedule, to prove it's still working.

Consider a constructed, illustrative eight-week trace of a support chatbot's weekly regression-suite score, run against the same fixed 100-item eval set from 01-08-style evaluation-set construction, each week producing an exact-match-or-graded percentage. This is built to demonstrate the diagnostic method, not a captured real result.

Week 1:  91%   (launch baseline)
Week 2:  90%   (normal week-to-week noise)
Week 3:  92%
Week 4:  89%
Week 5:  74%   <- sudden drop
Week 6:  73%
Week 7:  73%
Week 8:  91%   (recovered after a fix shipped)

Step 1 — establish the noise band. Weeks 1–4 bounce between 89% and 92%, a roughly 3-point range that reflects ordinary variance rather than a real signal — eval sets of 100 items have some inherent noise even when nothing has changed, and a team needs to know this band before it can recognize a genuine drop.

Step 2 — flag the real signal. Week 5's drop to 74% is 15 points below the noise band's floor — far too large to be normal variance, and it persists across weeks 6 and 7 rather than bouncing back, which rules out a one-off fluke and confirms something actually changed.

Step 3 — diagnose using the per-category breakdown. Suppose the eval set's 100 items are tagged by topic, and the category-level breakdown for week 5 shows: billing questions still scoring 90%, account-management questions still scoring 88%, but questions about a specific product feature scoring 20% — down from 85% the week before. That concentrated, single-category drop is the signature of data drift: something about that one feature's documentation changed (or was removed, or a competing near-duplicate article was added) in a way that broke retrieval for that specific topic, while everything else in the corpus stayed fine.

Step 4 — confirm and fix. Checking the corpus's change log for that week (the same kind of change-detection signal 12-12 describes for triggering incremental re-embedding) shows the relevant documentation page was rewritten and split into two pages five days before the drop. The old single-page chunk the index still had was now stale, and the new two-page structure had never been embedded and indexed. The fix is exactly 12-12's incremental-upsert case: re-embed and index the new pages, remove the stale chunk, and the score should recover.

Step 5 — verify recovery. Week 8's rebound to 91% — back inside the original noise band — confirms the fix worked, using the same eval set and the same metric that flagged the problem in the first place. This closes the loop: the same instrument that caught the drift is the one that confirms the fix, without needing a new, separately-designed check.

The sizing rule: cadence should be faster than the fastest plausible cause of drift for that signal. A corpus that changes daily needs a regression run at least daily to catch data drift before it compounds; a corpus that changes monthly can run weekly and still catch problems well within a useful window. Running the suite less often than the corpus changes guarantees a detection lag exactly as long as the gap between runs.

Objective 4.5 — "monitor functioning of data collection, experiments, and other software processes" — names this territory directly, and COURSE-INDEX's Module 2 material lists "drift detection; quality/latency/cost telemetry; prompt regression suites; versioning and documentation discipline" as must-know content under exactly this objective. Expect scenario questions describing a production system whose quality has quietly declined and asking what should have caught it, where the correct answer is some form of "a scheduled regression run against the existing evaluation set" rather than "add more logging" or "increase model size." A second likely pattern distinguishes monitoring from a one-time evaluation — a scenario describing a team that validated a system thoroughly before launch and then stopped measuring, with the correct diagnosis being that evaluation without ongoing monitoring leaves drift undetected indefinitely. Per the module's own design note, this lesson explicitly closes thread T1 (the measurement thread) that opened with 01-08: the exam rewards recognizing that the tool built to validate a system before shipping is the same tool that has to keep running afterward, not a one-time gate that gets discarded once the system ships.

Drift is a gradual, silent decline in an LLM system's answer quality that has no single obvious triggering event — caused by the underlying corpus changing (data drift), an upstream model changing behind a stable-looking API (model drift), the population of user questions shifting away from what the system was built and evaluated for (population drift), or an unintended side effect of a prompt or configuration change (prompt/configuration drift). It's called "drift" rather than a "failure" or "outage" precisely because nothing crashes or throws an error — the system keeps answering, just increasingly wrong, which is why it can only be caught by deliberately re-measuring quality rather than by watching for an alert that quality decay never triggers on its own.

By re-running the same fixed evaluation set that validated the system at launch on a recurring schedule and watching for a score drop that exceeds normal week-to-week noise, then using a per-category breakdown of that same eval set to localize which topics or capabilities the drop concentrates in. A broad, sudden drop across unrelated categories with no corresponding change on your own side points specifically toward an upstream model having changed silently; a drop concentrated in one topic area more often points toward data drift in the corpus underneath a RAG system rather than the model itself.

Because there is no automatic, per-request signal for whether an LLM's answer was actually correct the way there is for latency or cost — a wrong answer and a right answer look identical to any system that isn't specifically checking the output against a known-correct reference. A fixed evaluation set provides that reference: re-running the exact same questions against the live system on a schedule and comparing the score to a known baseline is the only way to detect quality decline before it shows up as user complaints, which is why the same evaluation instrument built in 01-08 to validate the system before launch is the instrument that has to keep running afterward.

• Quality telemetry: measuring an LLM system's output correctness against a fixed evaluation set, as distinct from latency or cost.
• Drift: a gradual, silent decline in output quality with no single triggering event, caused by data, model, population, or configuration change.
• Data drift: quality decline caused by the underlying corpus changing without the retrieval index keeping up.
• Model drift: quality or behavior change caused by an upstream model changing, sometimes silently, behind a stable-looking API.
• Population drift: quality decline (against real usage) caused by the actual user-query distribution shifting away from what the eval set represents.
• Prompt regression suite: the same fixed evaluation set re-run on a schedule (and at every deploy) specifically to catch quality drift.
• Noise band: the range of ordinary score variation across repeated regression runs, against which a genuine drop has to be distinguished.

Monitoring an LLM in production is three streams running at different cadences: continuous latency and cost telemetry that can alert in real time, and a scheduled quality regression suite that has to run the same fixed evaluation set again and again, because quality decay has no automatic per-request signal the way a slow or expensive request does. Drift is not one failure mode but four — data, model, population, and configuration — and diagnosing which one occurred determines whether the fix is re-indexing a corpus, pinning or re-validating an upstream model, refreshing the eval set itself, or rolling back a recent change. The single idea worth carrying out of this module: the evaluation instrument built to prove a system worked before launch is not a one-time gate to discard — it is the same instrument that, run on a schedule, is the only thing standing between silent decay and a customer finding the failure first.

Next: 13-01 opens Module 13 by turning from measurement to principle — NVIDIA's four pillars of trustworthy AI and what it actually takes to implement each one, starting from the premise that a principle with no instrument behind it is just a press release.