Linear algebra for LLMs: dot product, cosine similarity, and NumPy shapes

Linear algebra for LLMs is the arithmetic of rectangular number arrays. A language model is, mechanically, a long chain of matrix multiplications with a few nonlinear squashing functions between them. Nothing in that chain requires you to prove a theorem. What it requires is that you can look at a number array, say what its dimensions mean, and predict what happens when it is multiplied by another one.

The objects, in order of increasing dimension:

"Tensor" is the general word. A scalar is a 0-D tensor, a vector is a 1-D tensor, a matrix is a 2-D tensor, and nobody in practice bothers with a special word past that. In PyTorch and NumPy you inspect the shape with .shape, and reading that tuple is genuinely half the skill.

Two operations matter above all others.

The dot product takes two vectors of equal length and returns one scalar:

a · b = a₁b₁ + a₂b₂ + a₃b₃ + … + a_d b_d

It is one multiplication per position and then one sum. That is all. Its meaning is agreement: it is large and positive when the two vectors point the same way and are both long, near zero when they are perpendicular, and negative when they point in opposing directions.

Cosine similarity takes the same two vectors and divides the dot product by both of their lengths:

cos(a, b) = (a · b) / (‖a‖ × ‖b‖)

where ‖a‖ (the norm, or Euclidean length) is sqrt(a₁² + a₂² + … + a_d²). Because you divided out both lengths, the result is bounded in [−1, +1] regardless of how long the vectors were. It measures direction only. This is why it is the default similarity function for text embeddings: a long document and a one-sentence summary of the same topic can have very different vector magnitudes and still be recognised as being about the same thing.

Matrix multiplication is the generalisation: it is just a grid of dot products. Multiplying an (m, k) matrix by a (k, n) matrix gives an (m, n) matrix, where entry (i, j) is the dot product of row i of the first with column j of the second. The inner dimensions must match — k against k — and that single rule is the source of most notebook errors you will ever see in this field.

That is the whole toolkit. The rest of this lesson makes it airtight, because the two concepts introduced here — tensor shapes (C03) and dot product / cosine similarity (C04) — carry more downstream dependency in this course than any other pair. Almost everything in modules 1 through 12 reduces to them. This lesson is deliberately kept tight rather than inflated to match that fan-out: the goal is that you never need to come back to it, not that it is long.

L1 — Intuition: a dot product is a vote count

Imagine each dimension of a vector is a yes/no-ish opinion about some invisible feature: "is this text about finance", "is it formal", "is it a question". A positive number is yes, negative is no, and the size is confidence. Now put two such opinion lists side by side and multiply them position by position. Where both agree — both positive or both negative — the product is positive. Where they disagree, the product is negative. The dot product sums those agreements and disagreements into a single score.

That gives the intuition for all three regimes:

• Both texts strongly agree on many features → large positive dot product.
• They talk about unrelated features → products cancel out → dot product near zero.
• They take opposite positions on the same features → negative dot product.

Cosine similarity adds one correction to that vote count. A vector where every entry is doubled has exactly the same opinions, held twice as loudly. Its dot product with anything doubles too, which would make "loud" texts look more similar to everything. Dividing by the lengths cancels the loudness and leaves the opinions. Hence: the dot product measures agreement and magnitude together; cosine similarity measures agreement alone.

Tensor shapes are the bookkeeping that keeps these vectors organised. When you process 4 pieces of text at once, each 128 tokens long, and each token is represented by 768 numbers, you have 4 × 128 × 768 numbers in one box, and the shape tuple (4, 128, 768) is how you remember which axis is which.

L2 — Mechanism: the shape triple and the matmul rule

Almost every intermediate value inside a transformer has the shape (batch, sequence, hidden):

Three facts about this triple pay off repeatedly:

1. The batch axis is independent. Nothing in a forward pass mixes example 0 with example 3. Batching exists purely to keep the hardware busy. That is why batch size changes memory and speed but not results (up to floating-point noise).
2. The sequence axis is where interaction happens. Attention is precisely the operation that mixes information across the sequence axis. Every other layer in a transformer treats each position independently.
3. The hidden axis is where meaning lives. A single token's d numbers are its representation at that layer. When you take "the embedding" of a piece of text, you are collapsing the sequence axis to produce one d-vector: shape (s, d) becomes shape (d,).

Now the matmul rule. For 2-D matrices:

(m, k) @ (k, n)  →  (m, n)          legal: inner dims match
(m, k) @ (n, k)  →  ERROR           inner dims k and n disagree
(m, k) @ (k, n) @ (n, p) → (m, p)   chains fine

For higher-dimensional tensors, the last two axes are treated as the matrix and everything before them is a batch of such matrices, broadcast if necessary:

(4, 128, 768) @ (768, 3072)         →  (4, 128, 3072)
(4, 12, 128, 64) @ (4, 12, 64, 128) →  (4, 12, 128, 128)

That second line is the shape of attention scores: for each of 4 examples and each of 12 heads, a 128 × 128 grid saying how much each token attends to each other token. You will meet it properly in module 1; here the point is only that it is a batched matmul and its shape is predictable from the rule.

Cosine similarity at scale is also a matmul. If you L2-normalise every vector first — divide each by its own norm so its length becomes exactly 1 — then the dot product is the cosine similarity, because you are dividing by 1 × 1. So a whole similarity search collapses to:

normalise query   → shape (1, d)
normalise corpus  → shape (N, d)
scores = query @ corpus.T   → shape (1, N)
rank   = argsort(-scores)

This is why production retrieval systems store normalised vectors: it converts an expensive-looking similarity metric into a single matrix multiplication, which is exactly the operation GPUs are built to do. That connection is the reason this lesson comes before M0.2.

L3 — The mechanics that actually bite: transpose, reshape, broadcasting, and axes

Three transformations move data between shapes, and confusing them is the most common source of silently wrong results.

Reshaping (4, 128, 768) to (512, 768) is safe and common — it flattens batch and sequence into one long list of token vectors. Reshaping (m, n) to (n, m) is not the same as transposing it, and this trips people constantly: reshape reads elements in order and refills them, so rows and columns get interleaved into nonsense. Transpose actually swaps the indices.

Broadcasting is the genuinely dangerous one, because it turns a bug into a bigger array rather than an exception:

a.shape == (1000,)       # 1000 similarity scores
b.shape == (1000, 1)     # the same scores, as a column
a - b                    # → shape (1000, 1000). No error. 1M numbers.

NumPy aligns shapes from the right, pads missing axes with 1, and stretches any axis of size 1. (1000,) pads to (1, 1000); against (1000, 1) both stretch, and you get a million-element outer difference where you wanted a thousand-element elementwise one. You will not get a traceback. You will get a plausible-looking mean and a wrong conclusion. Adding keepdims=True to reductions, and asserting shapes early, are the defences.

Finally, axis conventions in reductions. sum(axis=0) collapses the axis you name; it does not "sum along rows" in any way you should trust your intuition about. For a (N, d) array:

• arr.sum(axis=0) → shape (d,): one total per feature, across all N items.
• arr.sum(axis=1) → shape (N,): one total per item, across all features.
• arr.sum() → shape (): one number.

The rule that never fails: the axis you pass is the axis that disappears. When you compute a norm for L2 normalisation over a (N, d) array, you want the length of each row, so you reduce over the feature axis — axis=1 — and you want keepdims=True so the result has shape (N, 1) and broadcasts cleanly back against (N, d).

This is the comparison to have cold. All four are ways of scoring how related two vectors are, and the exam-relevant skill is picking the right one and knowing what each ignores.

Three consequences worth memorising:

• Cosine and normalised dot product are the same number. If a vector database offers "inner product" and "cosine" as separate index metrics, and your vectors are already normalised, they will return the same ranking. Choosing "inner product" on unnormalised vectors, however, silently changes your ranking to favour long vectors.
• Cosine and Euclidean give the same ranking on normalised vectors too — but only the ranking, not the score. For unit vectors, L2² = 2 − 2·cos, so larger cosine always means smaller distance. On unnormalised vectors they can disagree completely.
• Sign matters. Cosine can be negative. Many embedding models in practice produce mostly-positive similarities in a narrow band (say 0.6 to 0.95 for related text), which is why a raw threshold like "0.8 means relevant" is model-specific and untransferable. Confusing this band for a calibrated probability is a mistake we will name again when we evaluate retrieval.

There is one more confusable that belongs here because it is a naming trap rather than a maths one:

The last row is a real hazard: np.dot doing two different things depending on input rank is exactly the kind of thing that turns a shape bug into a wrong number. Preferring the explicit @ operator for matmul and np.sum(a * b) or np.inner for a genuine dot product is the habit that avoids it.

Real arithmetic, done step by step, with 5-dimensional vectors so it fits on a page. Real embeddings have 384 to 4,096 dimensions; the arithmetic is identical, only longer.

Take three vectors. Imagine they are 5-dimensional embeddings of three short documents.

a = [ 2,  0,  1,  3,  1]     "GPU memory limits for LLM training"
b = [ 1,  1,  0,  2,  2]     "how much VRAM does fine-tuning need"
c = [-1,  2,  0, -1,  0]     "seasonal fruit recipes"

Step 1 — dot products. Multiply position by position, then sum.

a · b = (2×1) + (0×1) + (1×0) + (3×2) + (1×2)
      =   2   +   0   +   0   +   6   +   2   = 10

a · c = (2×−1) + (0×2) + (1×0) + (3×−1) + (1×0)
      =   −2   +   0   +   0   +   −3   +   0  = −5

Step 2 — norms. Square every entry, sum, take the square root.

‖a‖ = sqrt(2² + 0² + 1² + 3² + 1²) = sqrt(4 + 0 + 1 + 9 + 1) = sqrt(15) ≈ 3.873
‖b‖ = sqrt(1² + 1² + 0² + 2² + 2²) = sqrt(1 + 1 + 0 + 4 + 4) = sqrt(10) ≈ 3.162
‖c‖ = sqrt(1² + 2² + 0² + 1² + 0²) = sqrt(1 + 4 + 0 + 1 + 0) = sqrt(6)  ≈ 2.449

Step 3 — divide.

cos(a, b) = 10 / (3.873 × 3.162) = 10 / 12.247 =  0.816
cos(a, c) = −5 / (3.873 × 2.449) = −5 /  9.487 = −0.527

So a and b — the two documents about GPU memory — score 0.816, strongly similar in direction. a and c score −0.527: not merely unrelated but pointing away. That negative value is the part people forget exists.

Step 4 — the same answer via normalisation, to prove the equivalence. Divide each vector by its own norm to get unit vectors, then just take the dot product.

â = a / 3.873 = [0.516, 0.000, 0.258, 0.775, 0.258]
b̂ = b / 3.162 = [0.316, 0.316, 0.000, 0.632, 0.632]

â · b̂ = (0.516×0.316) + (0×0.316) + (0.258×0) + (0.775×0.632) + (0.258×0.632)
       =    0.163      +     0     +     0     +    0.490      +    0.163
       =    0.816

Same number, 0.816, to three decimals. This is the identity that makes production vector search cheap: normalise once at index time, and every query afterwards is a plain dot product. Note also that ‖â‖ should now be 1 — check it: sqrt(0.516² + 0 + 0.258² + 0.775² + 0.258²) = sqrt(0.266 + 0.067 + 0.601 + 0.067) = sqrt(1.001) ≈ 1.000, the 0.001 being rounding in the three-decimal intermediates. Carrying rounding forward and then stating the residual is a habit M0.4 will formalise.

Step 5 — where Euclidean distance would have disagreed. Now double b, giving b' = [2, 2, 0, 4, 4]. Same direction, twice the length — the same document, "said louder".

cos(a, b') = (a · b') / (‖a‖ ‖b'‖) = 20 / (3.873 × 6.325) = 20 / 24.49 = 0.816   ← unchanged
a · b'     = 20                                                                  ← doubled
L2(a, b)   = sqrt(1 + 1 + 1 + 1 + 1) = sqrt(5) ≈ 2.236
L2(a, b')  = sqrt(0 + 4 + 1 + 1 + 9) = sqrt(15) ≈ 3.873                          ← got "further away"

Cosine says the relationship is identical, because it is. The raw dot product says b' is twice as relevant. Euclidean distance says b' is less relevant than b, even though it is the same document scaled. If your corpus contains a mix of short and long texts — and every real corpus does — that difference decides whether your retrieval works. It is the single most practical reason cosine is the default for text.

Two decision tables. The first is for shapes: given what you have and what you want, which operation gets you there.

The second is the metric choice, framed as when to reach for each and when not to.

That last row is the one that matters most in an applied setting, and it is worth stating plainly: a cosine similarity is not a probability. It has no calibrated meaning across models, across domains, or even across query types within one model. What you can legitimately do is rank with it, and choose an operating threshold by measuring your own labelled examples. That measurement is what 01-08 will have you build.

Both tables are worth copying into whatever notes you keep. Shape prediction is the skill that turns a two-hour debugging session into a ten-second check, and metric choice is the kind of decision-rule question this exam favours.

The exam is an associate-level, 50–60 question multiple-choice paper sat in 60 minutes, and published candidate reports converge on the finding that it tests high-level identity and when-to-use judgement rather than derivations — extensive attention math was reported as overkill and largely absent. That calibration is [FIELD] evidence, not official, but it points the same way as the official job-role framing, which describes an associate who contributes to LLM systems under supervision of senior professionals. So you will not be asked to derive softmax gradients. You will be asked which similarity metric belongs on an embedding search, or which axis a batch dimension is.

What this lesson buys you, concretely, is that a large fraction of later material stops being new information and becomes an application of something you already computed by hand:

The official objectives this module serves are 1.6 and 1.10 — familiarity with the capabilities of Python numerical and NLP packages including NumPy, and using those packages to implement traditional ML analyses — plus 4.6, the Software Development domain's identical NumPy clause. NumPy appears by name in the official objective text of both the 30%-weighted Core ML domain and the 24%-weighted Software Development domain. That is not incidental: shape literacy in NumPy is explicitly examinable surface area, and it is the surface area this lesson covers.

There is a second, subtler exam payoff. Several of the exam's most-reported confusable pairs are distinguishable by their linear algebra rather than by their marketing:

• Sparse vs dense retrieval — sparse scores lexical overlap on a vocabulary-sized mostly-zero vector; dense scores cosine on a few-hundred-dimensional filled vector. Same operation, radically different vector.
• WordNet vs word2vec — a hand-curated lexical ontology versus learned dense vectors you can take a dot product of. If you can take a cosine similarity of it, it is word2vec-family.
• Static vs contextual embeddings — one vector per vocabulary entry versus one vector per token occurrence, which is a shape difference: (V, d) lookup table versus (s, d) per-sequence output.

You do not need to memorise those now. You need the vocabulary that makes them one-line distinctions when you meet them.

Eight named errors. The first four raise exceptions and cost you minutes; the last four return plausible numbers and cost you a wrong conclusion, which is why they are the dangerous half.

Two of these deserve a sentence more.

Accidental broadcasting is the error that most rewards paranoia. A defensive one-liner after any reduction — assert scores.shape == (n_docs,), scores.shape — costs nothing and catches the entire class. Professional numerical code is dense with shape assertions for exactly this reason; they are documentation that fails loudly.

Comparing vectors across models is the one that survives into production and does real damage, because nothing errors. If you re-embed half a corpus with a newer model and leave the other half alone, your index becomes two disconnected geometries and retrieval quality collapses in a way that looks like a data problem rather than a versioning problem. The discipline of pinning and versioning an embedding model exists because of this failure mode, and this lesson is where its cause becomes visible.

No. You need to know that gradients exist, that backpropagation computes them, and that gradient descent steps the weights against them — all of which 01-06 covers at identification depth and explicitly not at implementation depth. Nothing on an associate-level, 60-minute multiple-choice paper can reasonably ask you to differentiate a matrix product, and candidate reports [FIELD] describe deep attention math as absent. Learn to read shapes fluently and spend the time you save on tool identity and decision rules, where the questions actually live.

Cosine similarity is the dot product divided by the lengths of both vectors. The dot product answers "how much do these two agree, taking their magnitudes into account"; cosine similarity answers "how much do these two agree in direction, ignoring magnitude entirely". Because of that division, cosine is bounded to [−1, +1] and the raw dot product is not. If you L2-normalise your vectors first — make every length exactly 1 — the two become the same number, which is why vector databases can offer "inner product" and "cosine" as interchangeable index metrics on normalised data and as very different metrics on unnormalised data.

Because document length leaks into vector magnitude, and you almost never want length to affect topical relevance. Worked out in section 4: doubling a vector leaves cosine similarity unchanged at 0.816 while raw dot product doubles and Euclidean distance grows from about 2.24 to about 3.87 — the same document scaled becomes "further away". A corpus mixing one-line questions with ten-page documents would rank badly under a magnitude-sensitive metric. Cosine strips magnitude out, so a short summary and the long document it summarises can score as related. On already-normalised vectors, cosine and Euclidean produce the same ranking — but only then.

It means: batch independent pieces of text being processed together, each represented as sequence token positions, each position represented by hidden learned numbers. So (4, 128, 768) is four texts, 128 tokens each, 768 features per token — 393,216 numbers. The batch axis never interacts across examples; it exists to keep hardware busy. The sequence axis is the only axis attention mixes across. The hidden axis is where a token's meaning is represented. Knowing which axis is which is how you know what an operation did.

No — not for this lesson and not for the arithmetic that follows. Every computation here is a handful of multiplications on five-element vectors, and even a full similarity search over tens of thousands of embeddings is comfortable on a laptop CPU with NumPy. What a GPU changes is the scale at which these same operations stay fast, which is precisely the subject of the next lesson. The reason M0.2 exists at all is that the first lab in this course is not about hardware, yet it will fail on hardware grounds if you have never seen what a GPU does.

Enough to construct an array, inspect .shape, index and slice it, multiply with @ and *, reduce with sum/mean/argmax/argsort while naming the axis, reshape and transpose deliberately, and compute a norm. That list is roughly a dozen functions. NumPy is named explicitly in official objectives 1.6, 1.10 and 4.6, so its capabilities are examinable; nothing suggests you will be asked about obscure API surface. The exam-relevant skill is recognising which NumPy operation a described task needs, not recalling keyword arguments.

1. A shape is a sentence about your data. (4, 128, 768) says "four examples, 128 tokens each, 768 features per token". If you cannot say that sentence, you cannot debug the code.
2. The dot product is one multiplication per position and one sum. That is the whole operation, and it is the atom every layer of a transformer is built from.
3. Cosine similarity is the dot product with both magnitudes divided out, bounded to [−1, +1], measuring direction only. It is the default for text because document length must not decide topical relevance.
4. Normalise once and the two become identical. L2-normalising at index time turns semantic search into a single matrix multiplication — which is why this arithmetic is also a hardware story.
5. Matmul's only rule is that inner dimensions match, and for higher-rank tensors the last two axes are the matrix while the rest is a batch. That rule alone predicts most output shapes you will need.
6. The axis you pass is the axis that disappears. Everything else about reductions follows from that.
7. Broadcasting failures do not raise exceptions. (N,) against (N,1) gives you (N,N) and a plausible wrong answer. Assert your shapes; keepdims=True is cheap insurance.
8. A similarity score is not a probability. Ranking with it is legitimate; treating 0.8 as "80% relevant" is not, and the threshold you settle on has to be measured on your own data.
9. Vectors from two different embedding models are not comparable. No error will tell you. This is the root of the embedding-model version-pinning discipline you will meet again in the RAG modules.
10. This lesson is deliberately short for its importance. Tensor shapes (C03) and dot product / cosine similarity (C04) have the two highest dependency counts in the entire course, but the way to honour that is airtightness, not length. If you can do section 4's arithmetic on paper and predict section 5's shapes without running anything, you are done here.

You can now say what a matrix multiplication is. What you do not know yet is why the same multiplication takes an hour on your laptop and a minute on rented hardware, why a model that loads fine one moment throws an out-of-memory error the next, and what the numbers in "24 GB of VRAM" actually have to be spent on. That is not trivia — it is the constraint that decides which experiments you are allowed to run at all, and the first hands-on lab in this course assumes you already understand it even though the lab is not about hardware.

Next: M0.2 How GPUs work and what happens during a training run — parallelism, memory as the binding constraint, the four things that occupy VRAM during training, and a deliberately provoked out-of-memory error so that when you meet a real one you recognise it in seconds instead of hours.