Logging, Anomaly Detection, and Root-Cause Analysis for a Deployed LLM

A request/response log is the raw material every later diagnostic step in this lesson depends on, and [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) names capturing request/response logs and errors as the first half of objective 9.2, ahead of anomaly detection itself. The reason logging comes first in the ordering is structural: you cannot detect an anomaly you have no record of, and you cannot trace a detected anomaly to a cause if the log entries around it are too sparse to reconstruct what was actually happening at that moment.

At minimum, a request/response log entry for an LLM service needs enough detail to answer, after the fact, "what did this specific request look like, and what happened to it": a timestamp, the request's latency, its outcome (success, error, timeout, and which kind), enough of the input and output to reconstruct the interaction (or a reference to where the full content is stored, given privacy and retention constraints), the model version that served it, and any relevant configuration state (which prompt template, which decoding parameters, which retrieved context if the system uses RAG) that was active at the time. Missing any one of these fields turns a later root-cause investigation into guesswork: a log with latency and timestamp but no model-version field cannot rule out "a new model version was silently rolled out" as a cause, no matter how good the anomaly detection on top of it is.

Why logging is not the same practice as monitoring

Logging is the passive capture of what happened, request by request. Monitoring, as M9-01 covers it, is the active aggregation of those individual records into the four rolling metrics — p50/p95/p99 latency, throughput, error rate, uptime — that a dashboard displays and that this lesson's anomaly detection watches. A team can have excellent logs and terrible monitoring (every request is recorded in perfect detail, but nobody is computing rolling percentiles or alerting on them) or excellent monitoring and terrible logs (the dashboard flags a real anomaly, but the underlying log entries are too thin to say why it happened). Both halves are necessary and neither substitutes for the other — this is the same kind of two-part dependency M9-01 closed with, where reliability metrics are the raw material the rest of this module operates on, and logs are the raw material the metrics themselves are computed from.

L1 — Intuition: an anomaly is a metric doing something it does not normally do

Every metric this module cares about has an ordinary range of week-to-week or hour-to-hour variation — the "noise band" that separates real signal from expected fluctuation. An anomaly is simply a movement outside that band: not "the error rate is nonzero" (some baseline error rate is normal for almost any service) but "the error rate just left the range it has occupied for the last several weeks and shows no sign of returning on its own." The skill is not spotting that a number changed — numbers change constantly — it is recognizing which changes are large enough, and sustained enough, to be worth investigating rather than shrugged off as ordinary jitter.

L2 — The mechanism: latency spikes, error surges, and output-quality drops

[GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) names the anomaly detection half of objective 9.2 as detecting latency spikes, error surges, and output-quality drops, and tracing them to root cause. Each of the three is defined against a different signal, which is exactly why treating "anomaly" as one undifferentiated category loses information a diagnosis needs:

• A latency spike is a sudden, sustained move upward in the p95/p99 series M9-01 defines — not necessarily p50, since a spike concentrated in the tail can leave the median almost untouched while a meaningful fraction of users experience real slowdown. Latency spikes are frequently caused by resource contention: a GPU pool near capacity, a batch-size or concurrency change, or a downstream dependency (a retrieval store, a safety-check service) that has itself slowed down.
• An error surge is a sudden, sustained move upward in error rate — requests failing to return a valid response at a rate well outside the noise band. Error surges are frequently caused by a bad deploy (a code or configuration change that broke something), a downstream dependency going fully unavailable rather than just slow, or a resource exhaustion event (out-of-memory, connection-pool exhaustion) severe enough to fail requests outright rather than merely delay them.
• An output-quality drop is a decline in the correctness or usefulness of the model's actual answers — a signal M9-01's four metrics do not capture at all, since a service can hold perfect latency, throughput, error-rate, and uptime numbers while quietly returning worse answers. Detecting a quality drop needs its own instrumentation layered on top of the operational metrics: a sampled human or LLM-as-judge review of live outputs, a proxy signal like user-reported thumbs-down rate or session abandonment, or — as later lessons in this module cover — a scheduled regression run against a fixed evaluation set.

The three types matter as a taxonomy specifically because each one narrows the space of plausible causes before any investigation even starts. A latency spike with a flat error rate points toward contention or slowdown, not outright failure. An error surge with flat latency points toward something breaking cleanly (a bad config, a dependency going down) rather than something merely getting slower. An output-quality drop with all four operational metrics flat points away from anything this lesson's alerting can catch on its own, toward the drift and regression-benchmarking machinery later lessons in this module build.

L3 — The exam-relevant edge case: alerting fast enough to matter, and the anomaly that hides in an aggregate

The practical value of anomaly detection is entirely in its timing: [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) states that alerting on anomalies enables a fast response before users are broadly affected. An anomaly detected an hour after it started, when a manual dashboard check happens to catch it, has already affected every user who hit the service during that hour. An anomaly detected within a minute or two of crossing the noise band, by an automated alert rather than a human noticing a graph, limits the affected population to whoever hit the service in that shorter window. The exam-relevant point is that anomaly detection's entire operational justification is this speed differential — a correct diagnosis delivered slowly has already let the damage happen, which is why "detect it" and "alert on it automatically" are treated as one combined practice rather than two separable steps.

The second edge case worth holding onto is that an anomaly concentrated in a narrow slice of traffic can hide inside a healthy-looking aggregate. If 95% of a service's traffic is unaffected and 5% is severely degraded, the aggregate p95/p99 or overall error rate may barely move outside the noise band, because the unaffected majority dilutes the signal from the affected minority. This is the same structural trap M9-01 names for latency percentiles computed over too small a sample, but running in the opposite direction: here the sample is large enough, but it is not segmented finely enough to expose a problem that is real but localized. The fix is the same discipline M9-01's worked examples model — breaking a metric down by a relevant dimension (region, model version, request type, customer tier) rather than trusting a single global number to reveal every anomaly hiding inside it.

Detecting that a metric moved is the easy half; tracing that movement to a specific, actionable cause is where the actual diagnostic work happens. The steps, in the order an investigation actually proceeds:

1. Confirm the anomaly is real, not noise. Check the metric against its established noise band (the same discipline M9-01's p99-sample-size discussion covers) — a single data point outside the band is not yet evidence of anything; a sustained move that persists across multiple measurement windows is.
2. Segment the anomaly. Break the affected metric down by every dimension the logs support — model version, region, request type, time of day — to find out whether the anomaly is universal or concentrated. A latency spike affecting only one region points toward an infrastructure issue in that region; a spike affecting only one model version points toward something in that version specifically.
3. Correlate against recent changes. Cross-reference the anomaly's start time against a deploy log, a configuration-change log, or an upstream-dependency-version log. An anomaly that starts within minutes of a known change is far more likely to be caused by that change than by an unrelated coincidence — though this correlation is a strong hint, not proof, and needs the next step to confirm it.
4. Test the hypothesis. Where possible, reverse the suspected change (roll back a deploy, revert a configuration value) in a controlled way and confirm the metric returns to its noise band. A hypothesis that is never tested against a reversal is a guess, not a diagnosis.
5. Document the root cause and the fix. Record what actually happened, in enough detail that the same failure mode is recognizable faster the next time it recurs — this is what makes root-cause analysis cumulative rather than something the team relearns from scratch on every incident.

Take a constructed scenario, illustrative rather than a captured real incident: a chat-completion service's p99 latency, which had held steady around 1.2 seconds for the prior three weeks, jumps to 4.8 seconds at 2:47 p.m. on a Tuesday and stays there.

Step 1 — confirm the anomaly is real.
  p99 history (prior 3 weeks, daily): 1.1s, 1.3s, 1.2s, 1.2s, 1.1s, 1.3s, 1.2s ... (noise band ~1.0-1.4s)
  Today at 2:47pm: p99 jumps to 4.8s and holds for the next 40 minutes.
  4.8s is far outside the 1.0-1.4s noise band and does not revert on its own -> real anomaly, not noise.

Step 2 — segment the anomaly.
  By model version: v2.3 requests show p99 = 4.9s; v2.2 requests (still 15% of traffic, mid-rollout)
    show p99 = 1.3s, unchanged.
  -> the spike is concentrated in v2.3 specifically, not universal across the service.

Step 3 — correlate against recent changes.
  Deploy log: v2.3 was rolled out to 85% of traffic at 2:45pm, two minutes before the spike started.
  -> strong timing correlation between the v2.3 rollout and the latency spike.

Step 4 — test the hypothesis.
  Roll back the 85% of traffic on v2.3 to v2.2 at 3:05pm.
  p99 across the service returns to 1.2s within the next measurement window.
  -> confirms v2.3 itself, or its rollout configuration, is the cause.

Step 5 — document.
  Root cause found on inspection: v2.3's deployment configuration set a larger default batch size
  than v2.2, and under current traffic volume that batch size increase pushed queueing delay high
  enough to move p99 by roughly 3.6 seconds -- the same batching-latency mechanism `M9-01`'s
  second worked example demonstrates with constructed throughput numbers.

The diagnostic path matters as much as the answer. Without segmentation (step 2), the team might have looked only at the aggregate p99 and concluded "something is wrong with the service" broadly, wasting time investigating infrastructure or upstream dependencies that were never actually affected. Without the rollback test (step 4), the deploy-timing correlation from step 3 would have remained a plausible-but-unconfirmed hypothesis rather than a demonstrated cause — and root-cause analysis that stops at "probably the deploy" without testing it is exactly the incomplete version this lesson is written to prevent.

⭐ THE EARNED INSIGHT: Detecting that a metric moved and diagnosing why it moved are two different skills with two different failure modes — detection without diagnosis produces an alert nobody can act on, and diagnosis without fast detection produces a correct answer delivered to an incident that has already run its course, which is exactly why root-cause analysis has to be built as a five-step chain (confirm, segment, correlate, test, document) rather than a single leap from "the graph looks wrong" straight to "here is the fix," because every step skipped in that chain is a place a wrong cause can slip through undetected and get "fixed" without the metric actually recovering.

Production Monitoring and Reliability is Domain 9 of the NCP-GENL blueprint, weighted at 7% [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md), and this lesson's material is explicitly numbered as objective 9.2 in the source: [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md): "Detect anomalies — latency spikes, error surges, output-quality drops — and trace them to root cause (9.2)." Expect this objective to appear in a few recurring shapes:

1. Anomaly-type identification. A scenario describes a symptom (a sudden latency increase, a burst of failed requests, users reporting worse answers) and asks which of the three named anomaly types it is — testing the taxonomy from section 2 directly.
2. Diagnostic-order testing. A scenario presents an anomaly and several candidate "next steps," where the keyed answer is the step that actually narrows down a cause (segmenting, correlating against a deploy log) rather than a generic first reaction ("restart the service," "add more logging after the fact").
3. Logging-vs-monitoring discrimination. A question tests whether a team that "has excellent logs" or "has excellent dashboards" alone has what it needs to diagnose a problem — the correct read is that both practices are necessary and neither substitutes for the other.
4. Speed-of-detection reasoning. A scenario contrasts an anomaly caught by automated alerting within minutes against the same anomaly caught by a human noticing a dashboard an hour later, testing whether you recognize that alerting speed is the entire operational value of anomaly detection.
5. Cross-domain tie-ins. A question naming an "output-quality drop" specifically may lean on this lesson as the entry point into drift and regression benchmarking, the subjects the next two lessons in this module cover.

How the question tends to be phrased

Professional-level items in this domain tend to describe a production symptom and ask what the correct diagnostic action is: "a deployed LLM service's error rate rises sharply immediately following a deployment; what is the most appropriate first step?" with correlating the timing against the deploy log (and testing by rollback) as the keyed answer against distractors like "increase the retry count" or "scale up the GPU pool," both of which are real, sometimes-appropriate actions attached to the wrong anomaly type. Another recurring shape asks you to name which of the three anomaly types a described symptom is, where the distractor options substitute one real anomaly type for another that the described metrics do not actually support.

What the distractors typically look like

The standard traps in this domain's style are: offering a generic mitigation ("add more logging," "restart the service," "scale up resources") as if it were root-cause analysis, when it treats the symptom without identifying the cause; conflating a quality drop with a latency or error anomaly, when the source material treats output-quality drops as requiring separate instrumentation from the operational metrics M9-01 covers; and offering "wait and see if it resolves on its own" as a response to a sustained, confirmed anomaly, when the entire stated value of alerting is responding before broad user impact, not after it has already passed.

An anomaly is a sustained move in a metric — latency, error rate, or a quality signal — that falls well outside the range of variation that metric has shown historically, confirmed across more than one measurement window rather than a single noisy data point. Normal variation is the ordinary week-to-week or hour-to-hour bounce every metric shows even when nothing has actually changed, and treating every fluctuation inside that established band as an anomaly wastes diagnostic effort chasing noise rather than real signal. The practical discipline is establishing the noise band from historical data first, then reserving investigation for movements that clearly and persistently exceed it.

Because each one is defined against a different underlying signal and tends to correlate with a different class of cause, so collapsing them into one undifferentiated "something is wrong" alert throws away exactly the information a fast diagnosis needs. [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) names all three explicitly under objective 9.2 rather than treating "anomaly" as a single category. A latency spike with a flat error rate points toward contention or slowdown; an error surge with flat latency points toward something breaking cleanly; and an output-quality drop is frequently invisible to both signals entirely, since a service can hold perfect operational metrics while its actual answers quietly get worse — which is exactly why quality needs its own instrumentation layered on top rather than being inferred from latency, throughput, or error rate.

• Logging, monitoring, and root-cause analysis are three linked but distinct practices. Logging captures raw request/response detail; monitoring aggregates it into the metrics M9-01 defines; root-cause analysis traces a detected anomaly back to a specific cause. None substitutes for the others.
• Three named anomaly types, three different signals. Latency spikes move p95/p99; error surges move error rate; output-quality drops are invisible to all four of M9-01's operational metrics and need their own instrumentation.
• Detection speed is the entire operational value of alerting. A correct diagnosis delivered after broad user impact has already happened is far less valuable than a faster, automated alert that limits the affected population.
• An anomaly can hide inside a healthy-looking aggregate. Segmenting by model version, region, or request type is what exposes a localized problem a global metric would otherwise dilute away.
• Root-cause analysis is a five-step chain: confirm, segment, correlate, test, document. Skipping any one step risks treating a symptom instead of a cause, or mistaking correlation for confirmed causation.
• A latency spike, an error surge, and a quality drop call for three different first responses. Naming the anomaly type correctly is what narrows down the plausible causes before any investigation even starts.

Next: M9-03 picks up the anomaly type this lesson's logging and alerting cannot fully catch on its own — drift, the gradual decline in output quality that happens even when nothing about the code, configuration, or infrastructure ever changes, and why a model that passed evaluation at launch is not thereby guaranteed to still be working months later.

Answers

1. A service's request/response logs record latency and outcome for every request but omit which model version served each one. What diagnostic capability does this gap remove?

• A. The ability to compute p95/p99 latency at all
• B. The ability to segment an anomaly by model version to check whether it is concentrated in one version ✅
• C. The ability to detect an error surge
• D. The ability to measure throughput

Without a model-version field, an investigator cannot rule out or confirm "a specific version is the cause" — exactly the segmentation step section 4's worked example depends on.

2. A team detects that p99 latency has moved outside its historical noise band but reverts to normal within the same measurement window. What is the correct next step?

• A. Immediately launch a full root-cause investigation
• B. Treat it as likely ordinary variance and wait for a sustained, confirmed move before investigating ✅
• C. Roll back the most recent deploy regardless of timing
• D. Declare an output-quality drop

A single data point that reverts on its own has not yet demonstrated a sustained anomaly — investigating every noisy blip wastes diagnostic effort.

3. Which anomaly type is most likely to be invisible to latency, throughput, error rate, and uptime metrics entirely?

• A. A latency spike
• B. An error surge
• C. An output-quality drop ✅
• D. A GPU-utilization spike

A service can hold perfect operational metrics while its actual answer quality silently degrades — quality needs its own instrumentation layered on top.

4. An error surge begins two minutes after a configuration deploy. What is the correct diagnostic step before concluding the deploy caused it?

• A. Accept the correlation as proof and roll back permanently without testing
• B. Test the hypothesis by reversing the change and confirming the metric returns to its noise band ✅
• C. Ignore the timing since correlation is never useful
• D. Add more retries to mask the errors

Correlation is a strong hint but not proof; only a controlled reversal that restores the metric confirms the cause.

5. Why is alerting speed treated as central to anomaly detection's value, rather than just correctness of the eventual diagnosis?

• A. Because incorrect alerts are always harmless
• B. Because a correct diagnosis delivered after broad user impact has already happened limits the value of detecting the anomaly at all ✅
• C. Because speed matters only for latency anomalies, not error surges
• D. Because alerting speed determines a model's parameter count

The entire stated operational value of anomaly detection is enabling a fast response before users are broadly affected — a slow but correct diagnosis has already let the damage accumulate.

6. A global error-rate metric looks flat and healthy, but users in one specific region are reporting frequent failures. What does this illustrate?

• A. The global metric is definitely wrong and should be discarded
• B. An anomaly localized to one segment can be diluted into invisibility by unaffected majority traffic in an aggregate metric ✅
• C. Regional issues cannot be detected through logging
• D. This proves the anomaly is an output-quality drop, not an error surge

A localized problem can hide inside a healthy-looking global number precisely because the unaffected majority dilutes the signal — segmentation is what reveals it.

7. What distinguishes root-cause analysis from simply detecting that a metric moved?

• A. Root-cause analysis requires no logs, only dashboards
• B. Root-cause analysis traces the detected movement to a specific, tested, actionable cause rather than stopping at "something changed" ✅
• C. Root-cause analysis is only applicable to latency, never to error rate or quality
• D. Detection and root-cause analysis are the same practice under different names

Detection is noticing a metric moved; root-cause analysis is the chain of confirming, segmenting, correlating, and testing that identifies why.

8. A team restarts a service in response to a latency spike, the spike temporarily clears, and it recurs three days later. What diagnostic step was most likely skipped?

• A. Logging the restart event
• B. Correlating the original spike against a recent deploy or configuration change and testing that hypothesis ✅
• C. Measuring throughput before the restart
• D. Checking the model's parameter count

A restart can mask a symptom temporarily without addressing an underlying cause such as a bad deploy or configuration change — which is why the anomaly returns once the underlying condition recurs.