Model Comparison Metrics: Accuracy, Precision, Recall, F1, ROC-AUC, MAE, MSE, and R²

Identity statement: a model comparison metric is a single number computed from a model's predictions and the true labels or values, designed to summarize how good those predictions are — but every metric summarizes by throwing information away, and different metrics throw away different information. [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states the classification list directly — accuracy, precision, recall, F1, ROC-AUC — and the regression list separately — MAE, MSE/RMSE, R² — with an explicit warning that "accuracy alone misleads on imbalanced data."

When it matters: any scenario that reports a single metric and asks you to evaluate a model, especially one involving a rare class, a costly error type, or a continuous target.

The two families split cleanly by what the model outputs. Classification metrics apply when the model predicts a discrete category — spam or not spam, which of several classes. Regression metrics apply when the model predicts a continuous number — a price, a duration, a probability treated as a raw value. Offering a classification metric for a regression task, or vice versa, is one of the most reliable distractor patterns on this topic, because the two families are not interchangeable at all — a continuous prediction has no "correct class" for accuracy to check against, and a discrete class has no numeric residual for MAE to subtract.

Before naming accuracy, precision, recall, and F1 individually, it helps to see that all four are just different arithmetic performed on the same four counts. For a binary classification problem (positive class = the thing you are trying to detect), every prediction falls into exactly one of four buckets:

Everything downstream is arithmetic on TP, FP, TN, and FN:

Accuracy  = (TP + TN) / (TP + TN + FP + FN)      — fraction of all predictions correct
Precision = TP / (TP + FP)                        — of predicted positives, how many were right
Recall    = TP / (TP + FN)                        — of actual positives, how many were caught
F1        = 2 × (Precision × Recall) / (Precision + Recall)   — harmonic mean of the two

Read informally: accuracy asks "overall, how often was I right." Precision asks "when I said positive, how often was I actually right." Recall asks "of everything that was actually positive, how much did I actually catch." F1 asks whether precision and recall are balanced.

Section 3 turns to the specific trap this table sets up.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states the trap directly: "using accuracy on imbalanced data; a majority-class predictor can score high yet be useless." This is the single most exam-relevant fact in this lesson, and the arithmetic is worth working through once so the trap stops being an abstract warning.

L1 — Intuition

If 95% of examples belong to one class, a model that ignores the input entirely and always predicts that majority class will be right 95% of the time — a 95% accuracy score that reflects nothing about the model having learned anything at all about the minority class, which is very often the class anyone actually cares about detecting.

L2 — Mechanism

Dataset: 10,000 transactions, 9,700 legitimate, 300 fraudulent (3% fraud rate)

"Model" that always predicts "legitimate," regardless of input:
  TP (correctly flagged fraud)        = 0
  FN (fraud missed)                   = 300
  TN (correctly cleared legitimate)   = 9,700
  FP (legitimate wrongly flagged)     = 0

Accuracy  = (0 + 9,700) / 10,000 = 97.0%
Precision = 0 / (0 + 0)          = undefined (never predicts positive at all)
Recall    = 0 / (0 + 300)        = 0.0%

A 97.0% accuracy score, and the model has caught exactly zero fraud cases — it never predicts the positive class at all. Precision is undefined because the denominator (predicted positives) is zero; recall is a clean 0%, correctly reporting that none of the actual fraud was caught. Accuracy alone would have you believe this model is nearly perfect. Recall alone tells you it is completely useless for the actual task.

L3 — Which metric to reach for once you know the data is imbalanced

Once a scenario signals imbalance, precision and recall (and F1, their balance) are the metrics that survive contact with the majority-class trap, because both are computed relative to the positive class specifically — the always-predict-negative model above scores 0% recall precisely because it never engages with the class the metric is measuring. Accuracy has no such relative-to-the-positive-class structure; it treats every correct prediction, majority or minority, as an equally weighted contribution to the same number, which is exactly the property that lets the majority class dominate the score.

⭐ THE EARNED INSIGHT Accuracy does not fail on imbalanced data by being wrong — every one of its 9,700 correct "legitimate" calls in the example above is genuinely correct. It fails by being the wrong question to ask. Accuracy answers "how often is this model right, on average, across all examples," and when 97% of examples belong to one class, that question is dominated by an answer about the easy class, no matter how the model behaves on the class you actually care about. Precision and recall are worth reaching for not because they are more accurate, but because they ask a more specific question that the scenario actually needs answered.

A second, quieter version of the same trap deserves a name: the near-miss majority-class model. Suppose instead of always predicting "legitimate," a model predicts "legitimate" 99% of the time and "fraudulent" for a tiny, essentially random 1% slice of transactions, with no real signal driving that 1% selection.

Dataset: 10,000 transactions, 9,700 legitimate, 300 fraudulent (3% fraud rate)

Model predicting "fraudulent" for a random 1% (100 transactions), unrelated to true label:
  Of the 100 flagged, roughly 3 will be truly fraudulent (3% base rate) and 97 legitimate
  TP ≈ 3      FN ≈ 297      FP ≈ 97      TN ≈ 9,603

Accuracy  ≈ (3 + 9,603) / 10,000 = 96.06%
Precision ≈ 3 / (3 + 97)         = 3.0%
Recall    ≈ 3 / (3 + 297)        = 1.0%

Accuracy barely moves — 96.06% versus the always-negative model's 97.0% — because randomly flagging 1% of transactions costs almost nothing on a metric dominated by the 97% majority class either way. Precision and recall, by contrast, reveal a model doing essentially nothing useful: a 3.0% precision means the overwhelming majority of its fraud alerts are false alarms, and a 1.0% recall means it is still catching almost none of the real fraud. This is the version of the trap that is easy to miss even when you know to check for it, because the accuracy number changes so little between "does literally nothing" and "does something that looks like an effort but is not actually informed by the data" — the two models are both close to useless, and only precision and recall make that visible.

Precision and recall move in opposite directions as a model's decision threshold shifts, and which one to prioritize depends entirely on which type of error — a false positive or a false negative — is more costly in the scenario described.

A model can always be pushed toward higher recall by lowering the threshold at which it predicts "positive" — flag more things as positive, and you catch more true positives, but you also let in more false positives, dragging precision down. Pushing the threshold the other way raises precision at recall's expense. F1, the harmonic mean of the two, is the standard single number when neither error type is clearly more costly and you want a metric that penalizes a model for being extreme in either direction — a model with 100% precision and 1% recall (only ever predicting positive when maximally certain, catching almost nothing) scores a poor F1, and so does the mirror-image model with 100% recall and 1% precision.

Identity statement: ROC-AUC is the area under the Receiver Operating Characteristic curve, which plots the true positive rate (recall) against the false positive rate as the classification threshold varies across its entire range — and the area under that curve summarizes how well the model separates the two classes independent of any single chosen threshold.

The property that makes ROC-AUC distinct from precision, recall, or F1 is exactly that threshold-independence. Precision and recall are each computed at one specific threshold (commonly 0.5, but not necessarily). Two models can have identical precision and recall at the threshold you happened to check, yet behave very differently at every other threshold — one might degrade sharply just past that point, the other might stay strong across a wide range. ROC-AUC captures the model's separating ability across the whole range in one number: a value of 1.0 means perfect separation at every threshold, 0.5 means the model is no better than random guessing, and values below 0.5 mean the model's ranking is actively inverted relative to the true labels.

ROC-AUC is most useful when the final operating threshold has not been decided yet, or when you want to compare two models' underlying discriminative power independent of whatever threshold either one happens to be deployed with. It shares precision and recall's usefulness on imbalanced data in one respect — it does not automatically reward a majority-class predictor the way plain accuracy does — but it is a genuinely different question from F1: F1 asks "how good is this model at one specific threshold," ROC-AUC asks "how good is this model's ranking of positive-versus-negative examples, across every threshold there is."

Sections 2 through 5 defined precision, recall, and F1 for the binary case — one positive class, one negative class. Multimodal classification problems are frequently multi-class instead: a content-moderation model might classify a post into "safe," "spam," "harassment," or "graphic content" rather than a simple violating/not-violating split. Precision, recall, and F1 still apply, but they need an averaging strategy to collapse per-class scores into one number, and the strategy chosen changes what the final metric actually communicates.

Macro-averaging computes precision (or recall, or F1) separately for each class, then takes the plain, unweighted average across classes. Every class counts equally toward the final number regardless of how many examples it has, which means a rare class with only a handful of examples has exactly as much influence on the macro-average as the most common class.

Weighted averaging computes the same per-class scores, but averages them weighted by how many true examples each class has, so a class with more examples has proportionally more influence on the final number.

Four-class content moderation, per-class F1 and support (number of true examples):

  safe:      F1 = 0.95,  support = 800
  spam:      F1 = 0.82,  support = 150
  harassment: F1 = 0.40, support = 40
  graphic:   F1 = 0.55,  support = 10

Macro-average F1    = (0.95 + 0.82 + 0.40 + 0.55) / 4 = 0.680
Weighted-average F1 = (0.95×800 + 0.82×150 + 0.40×40 + 0.55×10) / 1000 = 0.904

This is a constructed scenario with illustrative numbers, not a measurement from a real system, but the gap between the two averages — 0.680 versus 0.904 — is the exact mechanism worth internalizing. Weighted-average F1 is dominated by the "safe" class simply because it has the most examples, and it reports a healthy-looking 0.904 even though the harassment class — plausibly the class a trust-and-safety team cares about most — scores a weak 0.40. Macro-average F1 reports 0.680, a number that reflects the harassment and graphic-content classes' poor performance far more honestly, because it refuses to let the large "safe" class drown them out.

The exam-relevant takeaway generalizes the accuracy-on-imbalanced-data trap from section 3 one level up: just as plain accuracy can hide a minority class's failure inside a binary problem, weighted-average F1 can hide a rare class's failure inside a multi-class problem. When a scenario names a specific rare or high-stakes class as the thing that matters, macro-averaging — which does not let class size dictate influence — is the more appropriate choice, even though weighted-averaging is often the default a library reports.

Classification metrics all assume a discrete correct-or-wrong answer. Regression tasks — predicting a continuous number — need a different kind of arithmetic, because "how wrong" is itself a continuous quantity, not a binary hit-or-miss.

MAE and MSE both go down as a model improves, and both are expressed in comparable but not identical units — MAE stays in the target's own units, while MSE is in squared units, which is why RMSE (its square root) is often reported instead, to bring the number back to an interpretable scale. The practical difference between MAE and MSE is entirely about outliers: because MSE squares each error before averaging, one prediction that is wildly wrong contributes disproportionately to the total, while MAE's plain absolute value treats that same large error as just one more error of a given size, proportional to its magnitude rather than its square.

R² goes up as a model improves, which is the opposite direction from MAE and MSE, and this direction-mismatch is a standing trap: a model comparison that mixes an error metric (down is better) with R² (up is better) in the same sentence is exactly the kind of claim that gets misread under time pressure. R² of 1.0 means the model explains all of the target's variance (perfect prediction); R² of 0.0 means the model does no better than always predicting the target's mean; and — the fact people most often assume is impossible — R² can be negative, which means the model's predictions are worse than simply guessing the mean every time, a real and diagnostically useful result rather than an error in the formula.

A multimodal content-moderation model flags posts (text plus attached image) as policy-violating or not. On a 1,000-post validation set:

                    Predicted violating   Predicted not violating
Actually violating         76 (TP)                24 (FN)
Actually not violating     38 (FP)               862 (TN)

Accuracy  = (76 + 862) / 1000 = 93.8%
Precision = 76 / (76 + 38)    = 66.7%
Recall    = 76 / (76 + 24)    = 76.0%
F1        = 2 × (0.667 × 0.760) / (0.667 + 0.760) = 71.0%

This is a constructed scenario with illustrative counts, not a measurement from a real moderation system. Read the four numbers together rather than any single one in isolation: accuracy of 93.8% looks strong on its own, but the actual violating class makes up only 100 of the 1,000 posts (10%) — a mild-to-moderate imbalance, not as extreme as the 3% fraud example in section 3, but still enough that accuracy is inflated by the easy majority "not violating" class. Recall of 76.0% is the more operationally important number here: it says roughly a quarter of genuinely violating posts are still being missed, which is the number a trust-and-safety team would actually want reported, not the 93.8% headline accuracy.

Two models predict delivery time (in minutes) for a logistics multimodal system that ingests package photos and route text. Both are evaluated on the same 500-delivery validation set.

Model A:  MAE = 4.2 minutes,  MSE = 61.0,  R² = 0.71
Model B:  MAE = 3.9 minutes,  MSE = 148.0, R² = 0.58

Model B has the lower (better) MAE, suggesting its typical error is smaller. But Model B's MSE is more than double Model A's, and its R² is meaningfully lower — both signals pointing toward Model B making some occasional very large misses, since MSE's squaring is exactly what would produce such a large gap despite a lower average absolute error. Model A's higher MAE alongside its lower MSE and higher R² suggests more consistent, moderate errors with fewer catastrophic misses.

Constructed scenario, illustrative numbers. Which model to prefer depends on the operational cost of a rare, large delivery-time miss: if an occasional wildly wrong estimate causes serious downstream problems (a customer told "10 minutes" for a delivery that takes two hours), Model A's tighter MSE and higher R² make it the safer choice, even at a slightly higher typical (MAE) error. If only the everyday, typical-case error matters and rare large misses are tolerable, Model B's lower MAE could be the more relevant number. MAE and MSE disagreeing in which model looks better is itself informative, not a contradiction to resolve by picking whichever number is bigger.

The common-mistakes table below collects every trap this lesson has named so far.

Every row above names a checkable symptom against a specific fix. Exam weight explains why this level of detail matters.

Core Machine Learning and AI Knowledge carries 20% exam weight, and [GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) names the accuracy-on-imbalanced-data trap explicitly as a common exam misconception — one of only three misconceptions called out by name in the entire domain section, which signals how directly testable this specific fact is. The domain's own framing also notes this exact trap "resurfaces in Domain 3 evaluation," so getting the underlying mechanism right here pays off again later in the course.

The question tends to arrive in a small number of recognizable shapes.

1. Metric-to-task matching. A scenario names a model's output type (discrete class vs. continuous number) and asks which metric applies. The keyed answer is read directly off the classification-versus-regression split.
2. The imbalanced-accuracy trap, directly. A scenario reports a high accuracy number on a described imbalanced dataset and asks whether the model is good. The keyed answer flags that accuracy alone is insufficient, with precision/recall/F1 as the correct follow-up.
3. Precision-versus-recall cost tradeoff. A scenario describes which error type (false positive or false negative) is more costly, and asks which metric to prioritize.
4. Direction-of-improvement items. A comparison mixes metrics that go up with metrics that go down, testing whether you track direction correctly per metric.

What the distractors typically look like

The reliable distractor families: offering accuracy as sufficient evidence of quality with no mention of class balance; describing MAE and MSE as measuring the same thing "just in different units," collapsing a real, exam-relevant distinction; and asserting R² cannot be negative, when a negative R² is a genuine, well-defined, and informative outcome.

Because accuracy treats every correct prediction as an equally weighted contribution to a single number, with no regard for which class those correct predictions belonged to. On a dataset where 97% of examples are one class, a model that always predicts that class — learning nothing about the input at all — scores 97% accuracy automatically, simply from the class distribution, not from any genuine predictive skill. Precision and recall avoid this trap because both are computed specifically relative to the positive class the model is supposed to be detecting, so a model that never engages with that class scores poorly on recall (0%) even while accuracy stays high.

Use MSE when large errors are disproportionately costly relative to small ones — a single very wrong prediction should be penalized much more heavily than several moderately wrong predictions of the same total magnitude, which is exactly what squaring the error achieves. Use MAE when every unit of error should count roughly proportionally regardless of how large any single error gets, or when your data contains outliers you do not want to let dominate the metric, since MAE's plain absolute value does not amplify large errors the way squaring does. Comparing both together, as section 9's worked example shows, often reveals more about a model's error profile than either number alone.

It depends on whether every class matters equally to the decision the metric is informing, or whether the classes' relative frequency in the real world should shape how much each one counts. Macro-averaging treats every class as equally important regardless of how many examples it has, which is the right choice when a rare class (a serious-harm category, a rare disease) matters just as much, or more, than a common one — weighted-averaging would let a large "safe" or "normal" class mathematically outvote the rare class's poor performance, hiding exactly the failure a scenario often cares most about. Weighted-averaging is the right choice when the goal is an honest picture of overall, real-world-frequency-weighted performance and no single class is disproportionately high-stakes. A scenario that names a specific minority class as the one that matters is signaling macro-averaging; a scenario asking for overall system health across a realistic class mix is signaling weighted-averaging.

• Classification metrics (accuracy, precision, recall, F1, ROC-AUC) apply to discrete-class outputs; regression metrics (MAE, MSE, R²) apply to continuous outputs — the two families are never interchangeable.
• Accuracy alone misleads on imbalanced data: a majority-class predictor can score above 95% while achieving 0% recall on the class that actually matters.
• Precision and recall trade off as a decision threshold shifts; which one to prioritize depends on whether false positives or false negatives cost more in the described scenario.
• F1 balances precision and recall into one number; ROC-AUC summarizes performance across every possible threshold, not just one.
• MSE penalizes large errors disproportionately; MAE treats every error proportionally — the two can disagree about which model is better, and that disagreement is informative, not a contradiction.
• R² can be negative, meaning the model performs worse than always predicting the mean — a real, informative result, not an error.
• Macro-averaging treats every class equally regardless of size; weighted-averaging lets the largest class dominate the reported number — pick macro-averaging when a rare or high-stakes class matters as much as a common one, and default to weighted-averaging only when overall, realistic-mix performance is genuinely the question being asked.

This module has now covered how to split data honestly, diagnose a model's fit, and pick the right metric to judge it by. What comes next is the machinery that actually produces the predictions these metrics score. M1-04 covers the deep learning frameworks — TensorFlow, PyTorch, and Keras — that every architecture in the rest of this course gets built and trained inside.