Evaluation Metrics by Task: Classification, Regression, Text Generation, Image Generation, and RAG

Identity statement: an evaluation metric only means something relative to the specific kind of wrongness it is built to detect. A metric chosen because it is familiar, rather than because it matches the task, produces a number that is precise and meaningless at the same time — correctly computed, and answering a question nobody asked.

When it matters: any scenario that names a task (classifying, predicting a number, generating text, generating an image, answering from retrieved documents) and asks which metric belongs to it.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) frames this domain's whole evaluation section around exactly one instruction: "Pick metrics that match the task." That is a compact way of saying something with real teeth — a metric built to reward exact-word overlap (like BLEU) tells you nothing useful about whether a classifier's predicted labels are correct, and a metric built to reward correct discrete labels (like F1) tells you nothing about whether a generated image looks realistic. The five task families below are the domain's own named map from task to metric, and the exam expects you to walk it in either direction: given a task, name the metric; given a metric, name the task it fits.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names the classification row directly: "Precision, recall, F1 (accuracy misleads on imbalanced data), ROC-AUC." Each of the four metrics answers a related but distinct question.

Precision answers: of everything the model predicted as positive, how much actually was positive? A model with low precision generates a lot of false alarms. Recall answers the complementary question: of everything that actually was positive, how much did the model catch? A model with low recall misses a lot of real positives. F1 is the harmonic mean of precision and recall, useful when you need one number that penalizes a model for being lopsided toward either extreme rather than genuinely balanced. ROC-AUC summarizes a classifier's ability to rank positives above negatives across every possible decision threshold, rather than committing to one specific threshold the way precision, recall, and F1 do.

Why accuracy alone misleads on imbalanced data

Accuracy — the fraction of all predictions that were correct — is the most intuitive classification metric and the most dangerous one to rely on alone once the classes are imbalanced. Consider a fraud-detection dataset where 99% of transactions are legitimate and 1% are fraudulent. A model that predicts "legitimate" for every single transaction, without learning anything about fraud at all, scores 99% accuracy — a number that looks excellent and describes a model that has learned nothing useful whatsoever about the one class that actually matters. Precision and recall, computed specifically with respect to the fraud class, immediately expose this model as useless: its recall on fraud is 0%, because it never predicts fraud at all. This is exactly the scenario [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) is naming when it flags that "accuracy misleads on imbalanced data" — the fix is not a different accuracy calculation, it is a different metric family entirely, one that scores performance on each class rather than pooling everything into one aggregate rate.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names the regression row as "MAE, MSE/RMSE, R²." These three answer a numeric-prediction question rather than a discrete-label question, and they differ in how they weight the size of an error.

MAE (mean absolute error) averages the absolute size of every prediction's error, treating a large error and several small errors of the same total size as roughly equivalent in cost. MSE (mean squared error), and its square-rooted form RMSE, square each error before averaging, which means a single large error contributes disproportionately more to the metric than several small errors summing to the same total — MSE/RMSE penalizes big misses harder than MAE does. R² answers a different kind of question entirely: what fraction of the variance in the target variable does the model's predictions actually explain, relative to simply predicting the average every time? An R² near 1 means the model explains nearly all the variance; an R² near 0 means the model is doing little better than guessing the mean; a negative R² — which is possible — means the model is doing worse than that naive baseline.

Choosing between MAE and MSE/RMSE for a described task

A scenario that describes a task where occasional large errors are disproportionately costly — a shipping-time estimate where being off by a day matters far more than being off by an hour, several times over — favors MSE/RMSE's harsher treatment of large errors. A scenario where errors of any size should be weighted proportionally, without one outlier dominating the metric, favors MAE's flatter treatment.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) assigns text-generation metrics by sub-task specifically: "BLEU (translation), ROUGE (summarization), perplexity (lower = better)." This is one of the domain's most precisely testable facts, because each metric is paired with one named use case rather than being a generic "text quality" score.

BLEU measures n-gram overlap between a generated translation and one or more reference translations — how much of the generated text's word sequences match the reference's word sequences. It is the standard metric for machine translation specifically, where there is usually a reasonably well-defined "correct" reference translation to compare against.

ROUGE also measures overlap between generated and reference text, but is the standard choice for summarization specifically, where the generated summary is expected to capture the reference's key content rather than match it word-for-word in the same way a translation is expected to.

Perplexity works on a different principle from the other two entirely — it scores a language model's own predictive fit against genuine held-out text, by looking at the probability the model assigned to the tokens that actually appeared next. Lower perplexity is better, precisely because a lower number means the model was less caught off guard by the real continuation — it had already assigned that continuation more probability than a model further from the mark would have.

Why perplexity belongs to language modeling, not to translation or summarization

A scenario question that offers perplexity as the metric for judging a translation's quality, or a summary's quality, is offering a real metric attached to the wrong sub-task. Perplexity evaluates a language model's own predictive fit to held-out text — it does not compare a generated output to any reference translation or reference summary the way BLEU and ROUGE do. This distinction matters enough that it recurs as this lesson's own comparison table below: perplexity and FID are the two lower-is-better metrics in this entire survey, and confusing perplexity's specific job (language modeling) with BLEU's or ROUGE's job (comparing generated text to a reference) is exactly the kind of task-mismatch error section 1 is built to prevent.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names the image-generation row as "FID (lower is better), plus human evaluation." M3-06 already gives FID its full treatment — what it measures, why lower is better, the TTUR paper it comes from, and the mode-collapse-sensitivity of its covariance term — so this lesson treats FID at survey depth and points there for the mechanism.

The one fact worth restating here, because this survey is organized around it: FID joins perplexity as one of just two entries across this whole map that reward a smaller number, and both sit in that same small family of distance- or error-flavored measures where zero marks the best achievable outcome rather than the worst. [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) also pairs FID with human evaluation explicitly — "FID (lower is better), plus human evaluation" — because FID captures distributional statistics well but says nothing about whether a specific generated image actually matches a specific prompt's intent, a gap M3-04's context-embedding material and M3-06's own treatment both cover in more depth than this survey needs to repeat.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names the RAG row as needing "retrieval quality and faithfulness (is the answer grounded?)" — and the word "and," not "or," is the testable detail. RAG evaluation is not a single metric; it is two separate questions that have to be answered separately, because a system can pass one and fail the other independently.

Retrieval quality asks: did the system fetch the right documents or passages to answer the question at all? A RAG system with poor retrieval quality hands its generation stage irrelevant or incomplete source material, no matter how good the generation stage itself is.

Faithfulness asks a different question, downstream of retrieval: given the documents that actually were retrieved, is the generated answer actually grounded in them, or does it drift into content the retrieved sources never stated? A system can retrieve excellent, perfectly relevant documents and still generate an unfaithful answer if the generation stage hallucinates content beyond what those documents support.

Why judging only the final answer misses half the picture

A RAG system that produces a fluent, confident, wrong answer could be failing for either of two structurally different reasons — bad retrieval, or unfaithful generation on top of good retrieval — and judging only the final answer's correctness cannot distinguish between them. This distinction is significant enough that M3-09 treats it as its own named exam trap: judging a RAG system on the final answer alone, while never checking whether the retrieved sources were actually relevant, is the single most common evaluation shortcut this domain warns against, and it is exactly the failure mode retrieval quality and faithfulness, evaluated separately, are built to catch.

Two rows are worth flagging on a second pass, because they are where this particular table's distractors like to hide. Perplexity and FID are the sole exceptions to "bigger is better" in the whole table — every other row rewards a larger score, which is exactly what makes these two the spot where a blanket "higher must mean better" assumption breaks down. And RAG is the only row that resists collapsing into one metric family — every other task above can be scored, at least in principle, by a single well-chosen number, while RAG's two underlying questions — did retrieval succeed, is the generation faithful to what it retrieved — need to stay separate or you lose the ability to tell which stage actually failed.

Practice matching task to metric directly, the way a scenario question presents it.

System 1: an email spam filter, where 2% of emails in the evaluation
          set are actually spam.
  -> Classification, and specifically imbalanced classification.
  -> Precision and recall (and F1) on the spam class, NOT accuracy alone.

System 2: a model predicting next-month house prices in dollars.
  -> Regression.
  -> MAE, MSE/RMSE, or R-squared, depending on how errors should be weighted.

System 3: a system translating English documents into French.
  -> Text generation, translation sub-task.
  -> BLEU.

System 4: a system producing one-paragraph summaries of long articles.
  -> Text generation, summarization sub-task.
  -> ROUGE.

System 5: a customer-support chatbot that retrieves internal policy
          documents and answers questions from them.
  -> RAG.
  -> Retrieval quality (were the right policy documents fetched) AND
     faithfulness (does the answer actually reflect what those
     documents say), evaluated separately.

This is a constructed scenario — the five systems are illustrative, not descriptions of real deployed products — but the matching exercise it demonstrates is exactly the recognition skill this domain tests: read the task described, not the vocabulary that happens to appear in the question, and select the metric family built for that specific task. Notice that System 1's imbalance detail is not incidental description — it is the fact that rules accuracy out as a sufficient metric on its own, exactly as section 2 established.

A support-bot RAG system is evaluated and produces the following scores on a fixed evaluation set of 200 questions.

Retrieval quality (were the top retrieved passages actually relevant
to the question): 91% of questions had at least one relevant passage
retrieved in the top results.

Faithfulness (was the generated answer actually grounded in the
retrieved passages, for the questions where retrieval succeeded):
68% of answers were fully grounded; the remaining 32% included
content not supported by any retrieved passage.

Retrieval quality alone looks strong here — 91% is a high number by most classification-adjacent intuitions. But faithfulness at 68% reveals a problem retrieval quality's own number cannot see: even when the system fetched the right material, roughly a third of the time the generation stage still drifted beyond what that material actually supported. This is a constructed illustration, invented for the walkthrough rather than measured from a real deployed system, but the diagnostic conclusion it demonstrates is the real, testable point: a team that only tracked one aggregate "is the final answer correct" number would have no way to know whether the 32% of unfaithful answers were a retrieval problem or a generation problem, whereas tracking the two metrics separately immediately localizes the fix to the generation stage specifically, since retrieval was already performing well.

Experimentation is the largest domain on the NCA-GENM blueprint at 25% of the exam, and its own scope note names picking the correct metric as one of exactly two core skills the domain tests, alongside designing a fair experiment. This lesson's survey is the domain's own explicit metrics-by-task map, and it is tested as a map — given a task, name the metric family; given a metric, name the task it belongs to; given a described scenario, recognize which metric family's failure mode is being illustrated.

The question tends to arrive in a small number of recognizable shapes.

1. Direct task-to-metric matching. "Which metric is standard for evaluating a machine translation system?" The keyed answer is BLEU; distractors offer ROUGE (the summarization-specific sibling) or perplexity (a related but differently-scoped text metric).
2. Imbalanced-data traps. A scenario reports a high accuracy score on an imbalanced dataset and asks whether the model is actually performing well. The keyed answer rejects the accuracy figure alone and calls for precision/recall/F1 on the minority class.
3. Direction-reversal traps. A claim states or implies that a higher perplexity or a higher FID is better. The keyed answer rejects the claim, since both are lower-is-better metrics.
4. RAG two-metric items. A scenario reports strong retrieval and asks whether the RAG system is therefore reliable, or reports a wrong final answer and asks what stage is responsible. The keyed answer insists on checking both retrieval quality and faithfulness before concluding anything.

What the distractors typically look like

Expect BLEU offered as the metric for summarization, and ROUGE offered as the metric for translation — the two overlap-based text metrics swapped for each other's named sub-task, since they measure a similar mechanism (n-gram overlap) but are conventionally paired with different generation tasks. Expect perplexity or FID offered with the direction reversed, trading on the fact that most other metrics in this survey are higher-is-better. And expect a RAG scenario that reports only one of the two required metrics and asks you to conclude the system is reliable regardless — the correct response is that one metric alone, retrieval or faithfulness, cannot answer whether the overall system is trustworthy.

Accuracy pools every prediction — correct and incorrect, across every class — into one aggregate rate, which means a model can score very high accuracy by doing well on a large majority class while doing badly, or not even attempting, the minority class that usually matters most. On a dataset that is 99% one class, predicting that class every time scores 99% accuracy while learning nothing about the other 1%. Precision, recall, and F1, computed with respect to the class that actually matters, reveal this failure directly, which is why this domain names accuracy-alone-on-imbalanced-data as a specific, checkable trap rather than a vague caution.

Because a RAG system's final answer can be wrong for two structurally different reasons that a single "is the answer correct" score cannot distinguish between: the system may have retrieved the wrong or incomplete source material (a retrieval-quality failure), or it may have retrieved good material and then generated an answer that drifts beyond what that material actually supports (a faithfulness failure). Tracking retrieval quality and faithfulness as two separate, independently measured questions is what lets a team localize a failure to the actual responsible stage, rather than knowing only that the final answer was wrong without knowing why.

1. A classifier scores 98% accuracy on a dataset where 97% of examples belong to one class. What should you check before trusting this result?
   • A. Nothing — 98% accuracy is a strong result on its own.
   • B. Precision, recall, and F1 computed specifically for the minority class.
   • C. Whether the model's ROC-AUC is exactly 1.0.
   • D. Whether the dataset needs more examples of the majority class.
2. Which metric is the standard choice for evaluating a machine translation system?
   • A. ROUGE
   • B. Perplexity
   • C. BLEU
   • D. F1
3. Which metric is the standard choice for evaluating a text-summarization system?
   • A. BLEU
   • B. ROUGE
   • C. FID
   • D. R-squared
4. A language model's perplexity drops from 42 to 18 after retraining. What does this indicate?
   • A. The model got worse at predicting held-out text.
   • B. The model got better at predicting held-out text.
   • C. Perplexity cannot decrease below 20.
   • D. The model's translation quality improved.
5. A regression task involves predicting delivery times, where being off by several hours occasionally is far more costly than being off by a few minutes routinely. Which metric better reflects that cost structure?
   • A. MAE, since it treats every error size proportionally.
   • B. MSE/RMSE, since it penalizes large errors disproportionately more.
   • C. R-squared, since it ignores error magnitude entirely.
   • D. Precision, since the task involves numeric prediction.
6. A RAG system reports 95% retrieval quality but only 60% faithfulness. What does this combination suggest?
   • A. The system is reliable overall, since retrieval is strong.
   • B. Retrieval is finding relevant material, but generation is frequently drifting beyond what that material supports.
   • C. The faithfulness metric must be measured incorrectly, since retrieval is already high.
   • D. The system needs a larger retrieval index.
7. Which two metrics in this lesson's survey are lower-is-better, unlike every other metric covered?
   • A. BLEU and ROUGE
   • B. Precision and recall
   • C. Perplexity and FID
   • D. MAE and F1

Answers

1. B. High pooled accuracy on an imbalanced dataset can mask a model that performs poorly, or not at all, on the minority class; precision, recall, and F1 computed for that specific class reveal what accuracy alone hides.
2. C. BLEU is the standard n-gram overlap metric for machine translation specifically; ROUGE is summarization's sibling metric, and perplexity and F1 belong to different task families entirely.
3. B. ROUGE is the standard overlap metric for summarization; BLEU is translation's sibling metric, FID belongs to image generation, and R-squared belongs to regression.
4. B. Lower perplexity means the model assigned higher probability to the real held-out continuations it was tested against, which is the definition of a better-fitting language model — perplexity has no fixed floor of 20, and it says nothing about translation quality specifically.
5. B. MSE/RMSE squares each error before averaging, which penalizes occasional large errors far more heavily than MAE's proportional treatment does — exactly the cost structure described, where large misses matter disproportionately more than routine small ones.
6. B. High retrieval quality alongside low faithfulness is the signature of a generation-stage problem specifically: the right material is being found, but the generated answers are not staying grounded in it — a diagnosis only possible because the two metrics are tracked separately.
7. C. Perplexity and FID are the two metrics in this survey where a smaller number indicates better performance; every other metric covered — precision, recall, F1, ROC-AUC, BLEU, ROUGE, R-squared, retrieval quality, and faithfulness — rewards a larger number.

• Metrics are matched to tasks, never chosen by convenience — the domain's own instruction is to pick metrics that match the task being evaluated.
• Classification uses precision, recall, F1, and ROC-AUC because accuracy alone misleads on imbalanced data.
• Regression uses MAE, MSE/RMSE, and R², differing mainly in how harshly they weight large errors.
• Text generation splits by sub-task: BLEU for translation, ROUGE for summarization, perplexity for language modeling — three distinct metrics, not interchangeable siblings.
• Perplexity and FID are the two lower-is-better metrics in this entire survey; everything else in the table rewards a higher score.
• RAG needs retrieval quality and faithfulness evaluated separately, because a wrong final answer can trace to either stage independently, and judging the final answer alone cannot tell you which.

Next: M3-09 closes this module with explainability and testing data/model quality and consistency — including the exam's own named version of this lesson's RAG lesson: judging a RAG system on its final answer alone, while never checking whether the underlying retrieval was actually relevant, is the single most common evaluation shortcut this domain warns against.