BERTScore Explained: Embedding-Based Evaluation Metrics vs BLEU and ROUGE

BERTScore is a reference-based, embedding-based evaluation metric for generated text. It takes a candidate output and one or more reference outputs, embeds every token of both with a pretrained contextual encoder, computes a token-to-token cosine similarity matrix, greedily matches tokens across the two sequences, and reports three numbers:

Two things it is not. It is not reference-free — you still need a human-written target, so it does not escape the labelling cost that 09-03 describes. And it is not a factuality metric: it measures similarity to a reference, so if the reference is wrong, a candidate that matches it scores well, and if the candidate invents a plausible detail that happens to embed near a reference token, the metric will credit it.

The wider family it belongs to, since exam questions often name a sibling:

METEOR is worth calling out because it sits between the two families. It predates embedding metrics and buys some paraphrase tolerance from curated lexical resources — stemming and synonym lists of the WordNet kind covered in 02-05 — rather than from learned vectors. If a question asks for "the overlap metric that accounts for synonyms and stems", METEOR is the answer, not BERTScore.

L1 — Intuition: match meanings, not letters

ROUGE asks "did the candidate use the same words as the reference?". BERTScore asks "for each word in the reference, is there a word in the candidate that means roughly the same thing here?". Because the embeddings are contextual, "bank" in "river bank" and "bank" in "savings bank" get different vectors, so the match is sensitive to how the word was used — which is what makes the metric better than looking up synonyms in a dictionary.

L2 — Mechanism: the four steps

Step 1 — Embed. Run the candidate and the reference through a contextual encoder (a BERT-family model, hence the name), producing one vector per token in each sequence. Contextual is the operative word: the same word type in two positions gets two different vectors, which is why 03-02's distinction between token embeddings and sentence embeddings matters here — BERTScore works at the token level.

Step 2 — Build the similarity matrix. For every candidate token i and reference token j, compute cosine similarity between their vectors. Because embeddings are typically L2-normalised first, cosine similarity reduces to a dot product — the operation 01-04 covers.

Step 3 — Greedy match in both directions.

R_BERT = (1/|ref|)  * Σ_{j in ref}  max_{i in cand} cos(cand_i, ref_j)
P_BERT = (1/|cand|) * Σ_{i in cand} max_{j in ref}  cos(cand_i, ref_j)
F_BERT = 2 * P_BERT * R_BERT / (P_BERT + R_BERT)

Each reference token takes its single best partner in the candidate (that is recall); each candidate token takes its single best partner in the reference (that is precision). Note that the matching is greedy and independent per token, not a global assignment — two reference tokens may both match the same candidate token. MoverScore's optimal-transport formulation is the fix for that, at higher computational cost.

Step 4 — Optional IDF weighting and rescaling. Two refinements you should recognise:

• IDF weighting. Weight each reference token by its inverse document frequency so that "the" and "of" contribute less than a rare content word. Without it, a candidate can score well by matching function words. This is the same TF-IDF intuition as 02-06.
• Baseline rescaling. Raw cosine similarities between unrelated sentences are not near zero — contextual embeddings are anisotropic, so two random sentences often score 0.75–0.85. That makes raw BERTScore compress into a narrow, unintuitive high band. Rescaling subtracts an empirical baseline (the expected score between random sentence pairs) and re-normalises, spreading the useful range back out. Rescaled and unrescaled BERTScores are not comparable, which is a documented source of confusion when two teams report "BERTScore 0.62" and "BERTScore 0.91" for similar-quality systems.

L3 — Depth: what the metric structurally cannot see

Four limitations, each of which produces a specific wrong conclusion if you do not know about it.

Negation and antonymy. Embedding models place antonyms close together, because they appear in near-identical contexts: "the drug increased mortality" and "the drug reduced mortality" share almost all of their token neighbourhoods. BERTScore will score that pair highly. Similarly "the policy applies to employees" versus "the policy does not apply to employees" — the negation is one short token whose contribution to a token-averaged score is small. This is the same failure 07-03 describes for embedding retrieval, and it appears here for exactly the same reason: the geometry that makes embeddings useful for topical similarity makes them poor at logical polarity.

Numbers, names and dates. A wrong figure is often near a right figure in embedding space. "€4.2 million" and "€42 million" tokenise into overlapping pieces and embed close together. For any task where a specific value is the answer, an embedding metric is the wrong instrument and exact match or a numeric tolerance check is the right one.

Model dependence. The score is a function of which encoder you used, which layer you took the embeddings from, and how it was trained. Change the encoder and every number changes. So a BERTScore is only comparable within one fixed configuration — the same discipline perplexity requires in 09-02, for the same underlying reason: a metric with a free parameter is only meaningful with that parameter pinned. Record the encoder name, the layer, whether IDF was used, and whether baseline rescaling was applied. That four-item tuple is part of the metric's definition, not metadata.

Reference dependence. BERTScore's ceiling is the reference's quality. A single reference summary is one human's choice among many valid summaries; a candidate that is better than the reference will score lower than one that mimics it. Multiple references help and multiply your labelling cost. And because it still needs a reference, BERTScore cannot evaluate open-ended generation where no target exists — which is exactly the gap that LLM-as-a-judge (09-10) and RAG-specific faithfulness metrics (09-07) fill.

There is also a fluency blind spot worth naming: BERTScore over a bag of greedily matched tokens is largely insensitive to word order. Two candidates with the same tokens in different orders score similarly, even if one is ungrammatical. ROUGE-L's longest-common-subsequence at least rewards ordering; BERTScore's token matching does not.

Read the table by column, not by row. The "detects factual error" column is all "no" except the last, and that is the most important fact in this lesson. Every reference-based similarity metric answers the question "does this look like the target?" and none of them answers "is this true?" A confident fabrication that resembles the reference in style and topic will score well on all of them.

The other high-value column is "paraphrase-tolerant". That single property is the entire reason to prefer BERTScore over ROUGE, and the entire reason a team migrating from ROUGE to BERTScore will see scores move in ways that do not correspond to any change in the system.

We compute BERTScore for a three-token candidate against a three-token reference. The cosine similarities below are constructed for the arithmetic — they are not measured from any real encoder. The point is the mechanics, which are exact given the inputs.

• Reference: ["the", "meeting", "was postponed"]
• Candidate: ["the", "meeting", "was delayed"]

Cosine similarity matrix (rows = candidate tokens, columns = reference tokens):

Step 1 — recall: each reference token takes its best candidate match (column maxima).

ref "the"            → max(1.00, 0.21, 0.19) = 1.00
ref "meeting"        → max(0.22, 1.00, 0.33) = 1.00
ref "was postponed"  → max(0.18, 0.35, 0.86) = 0.86

R_BERT = (1.00 + 1.00 + 0.86) / 3 = 2.86 / 3 = 0.9533

Step 2 — precision: each candidate token takes its best reference match (row maxima).

cand "the"          → max(1.00, 0.22, 0.18) = 1.00
cand "meeting"      → max(0.21, 1.00, 0.35) = 1.00
cand "was delayed"  → max(0.19, 0.33, 0.86) = 0.86

P_BERT = (1.00 + 1.00 + 0.86) / 3 = 0.9533

Step 3 — F1.

F_BERT = 2 × 0.9533 × 0.9533 / (0.9533 + 0.9533) = 0.9533

BERTScore F1 ≈ 0.953. (P and R coincide here because the sequences are the same length and the matching is symmetric; they diverge as soon as lengths differ.)

Step 4 — compare with ROUGE-1 on the same pair. Treating the tokens as unigrams, the candidate and reference share the and meeting but not postponed/delayed:

ROUGE-1 recall    = 2 matched / 3 reference unigrams = 0.667
ROUGE-1 precision = 2 matched / 3 candidate unigrams = 0.667
ROUGE-1 F1        = 0.667

And BLEU with bigrams would be worse still: the candidate bigram meeting was delayed does not appear in the reference at all, so higher-order n-gram precision collapses toward zero and the geometric mean across n-gram orders drags the score down hard.

So the same output scores 0.95 on BERTScore and 0.67 on ROUGE-1, and the difference is entirely the synonym delayed/postponed. That is the metric family difference in one number.

Step 5 — now demonstrate the failure case. Change the candidate to ["the", "meeting", "was confirmed"]. Suppose was confirmed has cosine similarity 0.71 to was postponed — lower than the synonym, but far from zero, because both are meeting-status predicates appearing in near-identical contexts.

R_BERT = (1.00 + 1.00 + 0.71) / 3 = 0.9033
P_BERT = (1.00 + 1.00 + 0.71) / 3 = 0.9033
F_BERT = 0.9033

The candidate now asserts the opposite of the reference, and BERTScore fell only from 0.953 to 0.903 — five points, for a complete reversal of meaning. ROUGE-1, meanwhile, is unchanged at 0.667: it cannot tell the synonym from the antonym either, because both are simply "a non-matching unigram". Neither family detects the semantic inversion. Only a metric that reasons about entailment or evidence — a faithfulness check, an NLI model, or a rubric-driven judge — will catch it. This is the concrete argument for why 09-07's faithfulness metric exists separately from every similarity metric in this table.

Step 6 — see the length asymmetry. Suppose the candidate is padded: ["the", "meeting", "was delayed", "as previously communicated to all stakeholders"], and the extra token's best match against any reference token is 0.30. Recall is unchanged at 0.9533 (every reference token still has its partner), but precision becomes:

P_BERT = (1.00 + 1.00 + 0.86 + 0.30) / 4 = 3.16 / 4 = 0.7900
F_BERT = 2 × 0.7900 × 0.9533 / (0.7900 + 0.9533) = 1.5062 / 1.7433 = 0.8640

Recall stayed flat while precision fell 16 points. This is why reporting P, R and F1 separately is worth doing: the pattern "recall high, precision falling" is the signature of a verbose model, and the pattern "precision high, recall falling" is the signature of an over-terse or truncating one. A single F1 hides the diagnosis, exactly as a single aggregate hid the slice diagnosis in 09-01.

The compressed rule: BERTScore is the right upgrade from ROUGE when the task has a reference and legitimate paraphrase; it is the wrong tool whenever a specific literal value or a truth claim is what you are checking.

Generation-quality metrics are the highest-frequency measurement topic in the Experimentation domain, which is 22% of the exam. BERTScore appears in the must-know set alongside BLEU, ROUGE, METEOR, exact match and perplexity, and the thing being tested is metric-to-task matching: given a described task, which metric applies, and what does each metric reward?

The objective-numbering defect, stated where it is cited. The official study guide prints Experimentation's objectives as 3.1–3.5, and those lines are a verbatim duplicate of Data Analysis's 2.1–2.5 — they describe data mining, data analysis, chart creation and trend identification, not model evaluation. The section's own scope statement ("how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback") and its suggested-reading list (machine translation, GLUE, evaluating RAG, hallucinations, cross-validation) describe the real content, and published candidate reports independently confirm that BLEU-family content appears on the exam. The objective that legitimately covers this lesson is the duplicated pair 2.2 / 3.2, "compare models using statistical performance metrics", plus 1.8 (select and use models to create text embeddings) — because BERTScore is a use of an embedding model, which is a nice illustration that the embedding topics in Module 1 and the evaluation topics here are the same machinery in two roles.

Question phrasings to expect:

• "Which evaluation metric compares generated text to a reference using contextual embeddings rather than exact n-gram overlap?" → BERTScore.
• "A summariser produces a correct summary using different wording than the reference and receives a low ROUGE score. Which metric would better reflect its quality?" → BERTScore (or another embedding/learned metric). Distractors: perplexity, exact match, BLEU with more n-grams.
• "What is a limitation of BERTScore?" → It still requires a reference; it is sensitive to the choice of embedding model; it can score a semantically inverted or factually wrong output highly.
• "Which metric accounts for synonyms and word stems using lexical resources rather than learned embeddings?" → METEOR.
• "BERTScore reports precision, recall and F1. What does low precision with high recall indicate?" → The candidate covers the reference content but adds material not in the reference — a verbose or padded output.
• "Can BERTScore detect hallucination?" → No. It measures similarity to a reference, not truth.

Distractor families. (1) Embedding metric offered for a literal-value task — using BERTScore where exact match on an ID, number or date belongs. (2) BERTScore presented as reference-free — conflating it with perplexity or with a judge. (3) BERTScore presented as a factuality or hallucination detector — the single most tempting wrong answer, because "semantic" sounds like "truthful". (4) Cross-configuration comparison — treating two BERTScores computed with different encoders or rescaling settings as comparable. (5) METEOR and BERTScore treated as the same idea — they both buy paraphrase tolerance, but one uses curated lexical resources and one uses learned contextual vectors, and questions do distinguish them.

Better at one specific thing, and not a replacement. BERTScore is better whenever the task admits legitimate paraphrase, because it credits a correct answer phrased differently from the reference — the worked example scored the same output 0.95 versus 0.67. ROUGE retains three real advantages: it is trivially cheap and needs no model, it is interpretable down to the exact matched n-grams so you can see why a score moved, and it is the conventional yardstick for summarisation, which matters when your number has to be comparable to someone else's. The practical answer is to report both. They fail differently: ROUGE punishes paraphrase, BERTScore rewards topical proximity including antonyms. When two metrics with different failure modes agree, you can act; when they disagree, that disagreement is a pointer to which outputs to read by hand, which is exactly the input 09-13's error analysis wants.

No, and this is worth stating as flatly as possible because it is the most common misconception about embedding metrics. A hallucination is fluent, well-formed, topically appropriate, and false. BERTScore measures how similar the output is to a reference in embedding space — and fluent, topical text is close in embedding space. The worked example above shows the mechanism precisely: replacing "postponed" with "confirmed" inverted the claim and cost only 0.05 of F1. Extend that to a fabricated case number, a wrong date, or an invented policy clause, and the metric barely moves.

What does detect it: checking each claim in the output against retrieved evidence (the faithfulness and groundedness metrics in 09-07), entailment models that explicitly classify support/contradiction/neutral, self-consistency across multiple samples, and human or rubric-driven judge review. Hallucination causes and the full mitigation ladder are 09-12's subject. A high BERTScore is evidence about phrasing, never about truth.

Whichever one you can pin, document, and keep constant — that discipline matters more than the choice. Three practical criteria when you do choose:

• Language and domain coverage. An encoder trained mostly on general English will embed clinical, legal or code text poorly, and the metric's scores will be compressed and noisy in that domain. This is the same selection problem as choosing a retrieval embedding model in 03-03.
• Layer choice. Middle layers of a contextual encoder often carry the most useful semantic signal for similarity; the final layer is specialised toward the pretraining objective. Whichever you pick, record it — it changes every number.
• Sequence length limits. An encoder with a 512-token limit silently truncates longer texts, so long-document summaries get scored on their first section only. Check the limit against your item lengths.

And one governance point: an evaluation metric that depends on a model is a dependency you must version like any other. If the encoder is updated, your metric changes even though your system did not. Pin the version, and when you must upgrade, re-baseline. That is the same re-embedding-and-migration problem 12-12 treats for vector indexes, arriving here in the evaluation harness instead.

1. BERTScore replaces string overlap with contextual-embedding similarity, so it credits valid paraphrase that BLEU and ROUGE score as a miss.
2. It reports precision, recall and F1 separately. Low precision with high recall means padding; high precision with low recall means truncation. Report all three.
3. The mechanism is greedy token-level cosine matching in both directions, averaged, then harmonically combined.
4. It still needs a reference. It is not reference-free, and it does not escape human labelling cost.
5. It cannot detect factual error. The worked example lost only 0.05 F1 when "postponed" became "confirmed" — a full reversal of meaning.
6. It is blind to negation, antonymy, and wrong numbers, for the same geometric reason embedding retrieval is (07-03).
7. Encoder, layer, IDF setting and baseline rescaling are all free parameters. Pin them, record them, and never compare across different settings.
8. It is weakly order-sensitive, so pair it with ROUGE-L or a grammaticality criterion if fluency matters.
9. METEOR is the lexical-resource route to paraphrase tolerance; BERTScore is the learned-embedding route. Exam questions distinguish them.
10. Report BERTScore and ROUGE. Their failure modes are opposite, so their disagreements tell you which outputs to read by hand.

You now have a toolbox with several instruments in it — perplexity, human rubrics, overlap metrics, embedding metrics — and no procedure for picking one. Picking wrongly is the most expensive mistake in this whole module, because a wrong metric does not fail loudly; it produces a plausible number that points your engineering effort in the wrong direction for a quarter. The choice is not a matter of taste: it follows from the task type, from which kind of error is unacceptable, and from what you are allowed to spend per item.

Next: 09-05 is the module's deepest lesson and gives the full metric-selection procedure, including the two metrics the official objectives name verbatim — loss functions and the proportion of explained variance (R²) — and the arithmetic of precision, recall, F1 and ROC-AUC underneath them.