Vectors, Dot Products, and Cosine Similarity Explained

Dot product. For vectors a and b of the same length:

a · b = a₁b₁ + a₂b₂ + … + aₙbₙ

One number out. Positive means the vectors broadly point the same way, near zero means they are roughly perpendicular (unrelated), negative means they oppose.

Cosine similarity. The dot product normalised by both magnitudes:

cos(a, b) = (a · b) / (‖a‖ · ‖b‖)        where ‖a‖ = √(a₁² + a₂² + … + aₙ²)

Bounded in [−1, 1] regardless of dimension: 1 = identical direction, 0 = orthogonal, −1 = opposite.

When to use which — the decision rule:

Use cosine similarity when magnitude is noise. Use the dot product when magnitude carries signal you want to reward.

For text embeddings, magnitude is usually noise — it tracks things like document length rather than meaning — so cosine similarity is the default and the metric almost every embedding model documents. The dot product is preferred where a longer or more "confident" vector should legitimately score higher, and inside attention, where the score is a dot product deliberately left unnormalised (only scaled by a constant).

The three quantities you need named, and nothing more

Note the relationship between rows: the dot product is the numerator of cosine similarity, and cosine similarity is a dot product with the length information deliberately divided away. They are not two unrelated metrics; one is the other with a normalisation step. Almost every confusion on this topic dissolves once you hold that.

L2 normalisation is the operation of dividing a vector by its own magnitude, producing a unit vector that points the same way with length exactly 1. It is the single most consequential preprocessing step in retrieval pipelines, for reasons section 2 makes precise.

L1 — Direction is meaning

An embedding places a piece of text at a point in high-dimensional space so that similar meanings land in similar directions (01-03 gave you the shape of such a vector; the embedding lessons in Module 2 explain how the placement is learned). "Similar meaning" therefore becomes a geometric question: do these two arrows point the same way? Cosine similarity answers exactly that question and nothing else.

The mental image that survives contact with the exam: think of every text as an arrow from the origin. Two arrows pointing the same way are about the same thing. The lengths of the arrows are an accident of how much text there was. Cosine ignores the lengths; the dot product does not.

L2 — Why normalising matters, with the identity that follows from it

Consider a one-sentence note and a fifty-page report, both entirely about GPU memory. Their embeddings may point in nearly the same direction but have very different lengths. The dot product would score the report higher against any query, purely because it is longer. Cosine similarity divides that out and scores them as near-equally relevant.

Now the identity worth memorising:

If every vector is L2-normalised to unit length, then ‖a‖ = ‖b‖ = 1, so cosine similarity and the dot product are numerically identical — and ranking by Euclidean distance gives the same order too, because for unit vectors ‖a − b‖² = 2 − 2·cos(a, b).

Three metrics, one ranking. This is why production embedding pipelines normalise vectors once at index time and then let the vector database use its cheapest inner-product path: you get cosine semantics at dot-product cost. It also explains why a vector database offering "cosine," "inner product" and "L2" as index metrics is not offering three different notions of relevance once your vectors are normalised — it is offering three implementations of the same ranking. On unnormalised vectors they genuinely differ, and picking the wrong one silently biases retrieval toward long documents.

Worth verifying the algebra once so it is yours rather than borrowed. For unit vectors:

‖a − b‖² = (a − b)·(a − b)
         = a·a − 2(a·b) + b·b
         = ‖a‖² − 2(a·b) + ‖b‖²
         = 1 − 2·cos(a,b) + 1
         = 2 − 2·cos(a,b)

Since cosine similarity appears with a negative coefficient, larger cosine means smaller squared distance, monotonically. Rank ascending by distance or descending by cosine and you get the identical ordering. That is the whole proof, and it takes four lines.

L3 — Where the dot product shows up inside the model

Attention computes a dot product between each query vector and every key vector to decide how much one token should attend to another, then divides by √d (the scaled in scaled dot-product attention) to keep the values in a range where softmax behaves. Note the difference from cosine: the divisor is a constant derived from the dimension, not the vectors' own magnitudes. Attention deliberately keeps magnitude information.

Why divide by √d at all? Because the dot product of two d-dimensional vectors is a sum of d terms, so its typical size grows with d. Feed very large scores into softmax and the distribution saturates — one entry approaches 1, the rest approach 0, and gradients through it approach nothing. Dividing by √d keeps the scale roughly constant as the dimension grows. That is the entire content of the word "scaled," and it is the level of detail the exam asks for.

You are expected to recognise that attention scores are dot products. You are not expected to derive attention — candidate reports name detailed attention math as depth that did not appear on the exam. This lesson therefore stops here on the mechanism side and spends the rest of its length on the parts that are tested: metric choice, ranges, failure modes, and the retrieval consequences.

Three traps in that table.

The direction reversal is the first: cosine and dot product are similarities where bigger is better, Euclidean and Manhattan are distances where smaller is better. Reading a "score" without knowing which family it belongs to inverts your ranking, and a distractor that says "the document with the highest Euclidean distance is most relevant" is testing exactly that.

The second is "cosine distance," which some libraries define as 1 − cosine similarity to make it behave like a distance. That puts it in the range 0 to 2, and reverses the direction. Check the sign convention of whatever tool you are using rather than assuming.

The third is Jaccard, which is not a vector metric at all. It compares sets — the classic use being overlap of keyword or shingle sets — which puts it on the sparse/lexical side of retrieval rather than the dense/semantic side. If an item contrasts dense with sparse retrieval, cosine belongs to the dense column and set-overlap measures belong to the sparse one.

And a fourth thing that is not in the table because it is a different category entirely: cosine similarity is not a loss function, not an accuracy metric, and not a way to compare two models. It compares two vectors. 01-05 handles losses; the metrics that compare models are a different family again.

A query and three candidate documents, in a deliberately tiny 3-dimensional embedding space:

query    q = [1, 1, 0]
doc A    a = [2, 2, 0]      # same direction as q, twice as long
doc B    b = [1, 0, 1]      # partial overlap
doc C    c = [0, 0, 5]      # unrelated direction, but a long vector

Dot products:

q · a = 1(2) + 1(2) + 0(0) = 4
q · b = 1(1) + 1(0) + 0(1) = 1
q · c = 1(0) + 1(0) + 0(5) = 0

Magnitudes: ‖q‖ = √2 ≈ 1.414, ‖a‖ = √8 ≈ 2.828, ‖b‖ = √2 ≈ 1.414, ‖c‖ = 5.

Cosine similarities:

cos(q,a) = 4 / (1.414 × 2.828) = 4 / 4.0   = 1.00
cos(q,b) = 1 / (1.414 × 1.414) = 1 / 2.0   = 0.50
cos(q,c) = 0 / (1.414 × 5.0)   = 0 / 7.07  = 0.00

Read the result. Doc A scores a perfect 1.00 on cosine despite being twice as long as the query, because it points in exactly the same direction — which is the behaviour you want from a retriever. Doc B is partially relevant at 0.50. Doc C is orthogonal at 0.00, correctly judged unrelated even though it is the longest vector in the set.

Now the failure mode. Give doc C the vector [1, 1, 6] instead — barely on topic, but very long:

q · c_new = 1(1) + 1(1) + 0(6) = 2          → beats doc B's dot product of 1
‖c_new‖   = √(1 + 1 + 36) = √38 ≈ 6.164
cos(q,c_new) = 2 / (1.414 × 6.164) = 2 / 8.72 ≈ 0.23   → stays below doc B's 0.50

So a raw dot-product ranking promotes the barely relevant long document above the genuinely relevant short one, and cosine keeps the order correct. That is the concrete cost of choosing the wrong metric on unnormalised vectors: long, marginally relevant chunks crowd out short, precisely relevant ones — a real and diagnosable RAG failure.

The same example after L2 normalisation

Normalise every vector to unit length and rerun the dot products. Dividing each vector by its own magnitude:

q̂     = [0.707, 0.707, 0    ]     (÷1.414)
â     = [0.707, 0.707, 0    ]     (÷2.828)   — identical to q̂
b̂     = [0.707, 0,     0.707]     (÷1.414)
ĉ_new = [0.162, 0.162, 0.973]     (÷6.164)

Now:

q̂ · â     = 0.500 + 0.500 + 0     = 1.00
q̂ · b̂     = 0.500 + 0     + 0     = 0.50
q̂ · ĉ_new = 0.115 + 0.115 + 0     = 0.23

The dot products of the normalised vectors are exactly the cosine similarities of the originals. The pathological ranking is gone, and no cosine formula was evaluated at query time — only a plain inner product. This is the identity from L2 doing real work: normalise once when you index, and the cheapest operation in the database becomes the semantically correct one.

Check the Euclidean claim on the same numbers, using ‖a − b‖² = 2 − 2·cos:

q̂ to â:      2 − 2(1.00) = 0.00   → distance 0.00   (rank 1)
q̂ to b̂:      2 − 2(0.50) = 1.00   → distance 1.00   (rank 2)
q̂ to ĉ_new:  2 − 2(0.23) = 1.54   → distance 1.24   (rank 3)

Same order, ascending. Three metrics, one ranking, as promised.

A practical trap deserves its own worked case, because it is the one that bites people building their first retrieval system.

In a small toy space, orthogonal means 0.00 and unrelated documents score near zero. In a real embedding space produced by a trained model, unrelated documents frequently score 0.5 to 0.7, and only truly near-duplicate text approaches 0.95. This happens because trained embedding spaces are anisotropic — the vectors occupy a cone rather than spreading evenly over the sphere — so almost everything has some positive alignment with almost everything else.

The consequences are practical and testable as reasoning:

• There is no universal relevance threshold. "Keep everything above 0.8" is a number that means one thing for one model and something entirely different for another. Thresholds must be calibrated per model on your own data.
• Ranking is reliable where absolute values are not. The ordering of candidates for a given query is what the metric is good for. The magnitude of the score is a much weaker signal.
• Comparing scores across models is meaningless. A 0.72 from one embedding model and a 0.72 from another are not commensurable, because the two spaces are unrelated.

Illustrative shape, constructed to make the point rather than measured from any named model:

The last row is the one that surprises people: genuine negative cosine values are rare in practice with modern text embedding models. That does not change the metric's range, which is mathematically −1 to 1 and is what an exam item is asking about. Distinguish the mathematical range from the empirical distribution — quoting "0 to 1" as the range because that is what you observed is the classic wrong answer.

Objective 1.8 is "select and use models to create text embeddings," and objective 1.4 is "curate and embed content datasets for RAGs" — neither is meaningful without knowing how two embeddings get compared. Objective 1.6 names vector databases explicitly, and objective 1.10 names NumPy, where the dot product is the workhorse operation. Core ML is 30% of the blueprint, the largest domain.

Cosine similarity is the term most likely to appear by name, in shapes like:

• "Which metric compares two text embeddings?" → cosine similarity.
• "What is the range of cosine similarity?" → −1 to 1. Cheap point, easy to lose.
• Retrieval scenarios where the answer hinges on similarity search being a nearest-neighbour lookup in vector space.
• Distractor discrimination: cosine similarity is not a loss function, not an accuracy metric, and not a way to compare two models. It compares two vectors.
• Attention mechanism items where you need to know the score is a scaled dot product between queries and keys.

How the question tends to be phrased

Four recognisable shapes:

1. Metric identification. "A retrieval system needs to find documents semantically similar to a query. Which measure is most appropriate?" Keyed: cosine similarity. Distractors: accuracy, cross-entropy, BLEU, F1 — all real metrics from other families.
2. Range recall. A bare factual item about the interval. Keyed: −1 to 1.
3. Magnitude reasoning. "Why is cosine similarity generally preferred over the raw dot product for comparing document embeddings?" Keyed: it removes the influence of vector magnitude, so document length does not dominate the score.
4. Cross-model reasoning. "A team replaces its embedding model with a newer one but keeps the existing index. What is the consequence?" Keyed: the stored vectors and the new query vectors live in different spaces, so similarity scores are meaningless and the corpus must be re-embedded.

What the distractors typically look like

The families you will see are: wrong range (0 to 1, or 0 to ∞); wrong family (offering a loss function or a classification metric where a similarity measure belongs); wrong direction (treating a distance as though bigger were better); and wrong object (claiming cosine similarity compares two models or two datasets rather than two vectors). Each of those is a single-word substitution away from the right answer, which is what makes precision here cheap to earn and cheap to lose.

Calibration, and it is calibration rather than official fact: published candidate reports describe the exam as general-level — know what each thing is and when to use it. NVIDIA publishes no per-item detail. For this material that means the identity statement, the range, and the magnitude decision rule are worth more than any derivation.

The last two rows are the ones that convert directly into exam points, because both are common distractor constructions.

No, and this is the single most consequential limitation to carry into the RAG lessons.

Cosine similarity measures directional agreement between two embeddings. What that agreement tracks is topical relatedness, learned from how the embedding model was trained. A passage can be maximally on-topic and still fail to contain the answer: it might discuss the right subject in the wrong year, state the opposite of the true fact, or be a question rather than an answer about the same subject.

There is a specific and instructive failure here: negation. "The service supports SSO" and "The service does not support SSO" are nearly identical in surface form and often score very high against each other, because embedding models capture topic much more strongly than polarity. A retriever ranking on cosine alone cannot reliably prefer one over the other.

Which is why production retrieval is layered rather than a single similarity call. The standard ladder: dense retrieval by cosine for recall, then a reranker that scores query–passage pairs jointly for precision, then a generation prompt that instructs the model to answer only from the supplied passages, then a groundedness check on the output. Cosine similarity is the first stage — cheap, approximate, and indispensable — not the whole system.

Because a similarity score is only meaningful between two vectors from the same space, and every embedding model defines its own space.

Two independent reasons compound. The obvious one is dimension: a 768-dimensional stored vector and a 1,536-dimensional query vector cannot even have a dot product taken, because the sum in the formula requires matching components. The subtler and more dangerous one is that even at equal dimensions the axes mean different things. Two models both producing 768 numbers have no shared convention about what the 400th number represents. The arithmetic will succeed and return a plausible-looking number that carries no information.

That second case is dangerous precisely because it fails silently. Nothing errors. Retrieval quality just becomes approximately random, and if you were not measuring it against a fixed evaluation set you might not notice for a long time — which is a direct argument for the discipline in 01-08.

Operational consequences worth stating plainly: pin the embedding model version alongside the index; store the model identifier in the index metadata; treat a model change as a full re-index with a migration plan; and keep an evaluation set that would catch a silent regression. Embedding-model choice is one of the stickiest decisions in a RAG system, and this is why.

Because in attention, magnitude is signal rather than noise — the opposite of the retrieval case.

Query and key vectors are produced by learned projections inside the model, not by a general-purpose text encoder. The model is free to use the length of a query or key vector to express something: how strongly this position should assert itself in the competition for attention. Normalising every query and key to unit length would throw that expressive capacity away and force all positions to compete on angle alone.

What attention does instead is scale by a constant: divide by √d, where d is the per-head dimension. Note carefully what that is and is not. It is a fixed number determined by the architecture, applied identically to every score. It is not a per-vector normalisation. So the operation retains all relative magnitude information between vectors while keeping the overall scale in a range where softmax produces useful gradients.

The clean contrast to memorise:

• Dot product = element-wise products summed; one number, unbounded, sensitive to magnitude.
• Cosine similarity = dot product ÷ both magnitudes; range −1 to 1, magnitude-free, the default for text embeddings.
• Decision rule: cosine when magnitude is noise (almost always, for text), dot product when magnitude is signal (and inside attention).
• On L2-normalised vectors, cosine, dot product and Euclidean ranking are identical — the reason pipelines normalise once at index time and use the cheapest operation.
• Cosine ≈ 1 means same direction, ≈ 0 means unrelated, ≈ −1 means opposite. The mathematical range is −1 to 1 even though trained text models rarely produce negatives.
• Absolute cosine values are not comparable across models and rarely support a fixed universal threshold; the ranking is what the metric is reliable for.
• High similarity means topically related, not correct or answering — which is why reranking and groundedness checks exist.
• Similarity metrics compare two vectors. They never compare two models, never compare two datasets, and are never a loss.
• Changing embedding models invalidates an existing index, sometimes silently. Pin the version and re-embed.

You can now compare two vectors. What you cannot yet do is score a prediction — turn "the model said this and the truth was that" into a single number that training can push downward. Similarity is not error, and the exam tests the difference directly under the objective about comparing models with statistical performance metrics.

Next: 01-05 Loss functions and cross-entropy explained — the loss that scores a next-token distribution against the token that actually came next, and its relationship to perplexity.