Vector Arithmetic and Word Analogies in word2vec

Vector arithmetic on embeddings means treating word vectors as ordinary vectors and performing addition and subtraction on them, then asking which word in the vocabulary is nearest to the result. The classic analogy task is stated as a is to b as c is to ? and computed as:

target ≈ vector(b) − vector(a) + vector(c)

For man is to king as woman is to ?, that is king − man + woman, and the reported answer is queen. Similar published examples include country-to-capital pairs (Paris − France + Italy ≈ Rome), singular-to-plural, and verb tense.

Why it works at all is not mysterious. If the training corpus uses king and man in contexts related in the same way that queen and woman are, then the difference between the king and man vectors points in roughly the same direction as the difference between the queen and woman vectors. Put differently: relational information has become encoded as approximately parallel displacement vectors in the space. Subtracting man from king isolates something like the "royalty" displacement; adding it to woman moves you into the neighbourhood where queen lives.

Three qualifications belong in the definition itself, not in a footnote:

1. It is approximate. The result vector is essentially never exactly any word's vector. You take the nearest neighbour of the result, and "nearest" is doing real work.
2. The standard evaluation excludes the input words. In the canonical analogy benchmark, the three query words (king, man, woman) are removed from the candidate set before finding the nearest neighbour. This matters enormously and §4 shows why.
3. It is a property of static word embeddings. word2vec, GloVe, and fastText have one vector per word type, which is what makes the arithmetic well-defined. Contextual and sentence embeddings do not offer the same clean operation, for reasons §3 makes precise.

So the accurate one-line statement is: in static word-embedding spaces, relational differences between word pairs are approximately parallel, which makes analogy completion work often enough to be a striking demonstration — under an evaluation protocol that excludes the inputs.

L1 — The intuition: relations as directions

Imagine the space has a rough direction that corresponds to "make this word royal", another that corresponds to "make this word plural", another for "make this word past tense". None of these is an axis; each is some direction through the space. If those directions are consistent — if the royalty displacement is about the same wherever you apply it — then displacement arithmetic works.

The reason they end up consistent is the distributional hypothesis doing its job (03-01). Pairs that stand in the same relation appear in systematically similar context patterns, and a model trained to predict context from word has to encode that systematic similarity somewhere. It shows up as parallel geometry.

L2 — The mechanism, and the three things that break it

Now the parts that folklore omits.

The result is a point, and the answer is its nearest neighbour. The computed vector king − man + woman is not queen's vector. It is a point in the space whose nearest labelled neighbour is often queen. Everything therefore depends on what else is nearby — vocabulary size, word frequency, and how crowded that region of the space is.

Nearest to the result is usually one of the inputs. This is the single most important technical fact in the lesson. Because the operation is a small displacement from woman, and because king, man, and woman are all high-frequency words with strong vectors, the true nearest neighbours of the result are frequently woman itself, or king, before you ever reach queen. The standard analogy evaluation removes all three inputs from consideration by construction. Published analyses have made this point repeatedly: a substantial part of the demonstration's apparent magic comes from the exclusion rule, not from the geometry. So the honest phrasing is not "the arithmetic gives you queen"; it is "after excluding the three input words, the nearest remaining vector is often queen".

Success is uneven across relation types. Analogy performance is much better on some relations than others. Syntactic relations (singular/plural, verb tense, comparative/superlative) and a handful of well-attested semantic relations (country–capital, currency, family gender pairs) tend to work. Broad semantic relations, rare words, and multi-word expressions tend not to. And a relation only works if both sides of it are well-represented in the training corpus — analogies over rare words fail, because rare words have poorly estimated vectors, which is the same domain-coverage problem as 03-03's vocabulary criterion.

L3 — Why it does not generalise, and what it reveals instead

Static only. The operation needs one vector per word, which contextual models do not provide: BERT gives you a different vector for king in every sentence, so king − man is not a defined quantity without first fixing a context, and different contexts give different displacements. You can construct context-controlled variants of the experiment, but the clean lookup-table arithmetic is gone.

Sentences do not compose this way. A tempting extrapolation is that if word vectors compose arithmetically, sentence vectors should too — that embed("the cat sat") + embed("on the mat") should relate to embed("the cat sat on the mat"). It does not, in any dependable way. Sentence embeddings are produced by pooling a transformer's contextual outputs and then reshaped by contrastive fine-tuning (03-02); the resulting space is optimised for cosine comparison of whole spans, not for compositional algebra. Assuming otherwise is how people end up trying to "subtract the topic" from a document vector and getting nonsense.

The interpretability claim is wrong. Analogy arithmetic working does not mean dimensions are interpretable. The royalty direction is a combination of many dimensions, and no single component of the vector holds "royalness". 03-01 said not to read individual dimensions, and the analogy result is not a counter-example to that — it is a statement about directions, not about axes.

What it genuinely demonstrates. Two things, both worth knowing. First, learned embeddings capture relational structure and not merely topical co-occurrence, which was a real and non-obvious finding when it was published. Second, the same mechanism that makes analogies work makes bias legible: if occupational or evaluative words carry consistent gendered or racialised displacement in the corpus, that displacement is in the space, recoverable by exactly this arithmetic, and it will affect any downstream system built on those vectors. That is not a side note — it is the practically important consequence, and it connects directly to 13-04's treatment of bias sources and measurement. The analogy demonstration and the bias demonstration are the same experiment.

The single most useful thing to do with the analogy result is to put it next to the curated alternative, because the pair clarifies both. This is also the confusable candidate reports name most often in this area.

The WordNet vs word2vec answer in one paragraph

WordNet is curated and symbolic; word2vec is learned and statistical. WordNet is a database in which humans grouped word senses into synsets and connected them with explicit typed relations — synonymy, antonymy, hypernymy (dog is-a mammal), hyponymy, meronymy (wheel part-of car). word2vec is a model that produced one dense vector per word type by training on a prediction task over unlabelled text, and its "relations" exist only as geometry — small angles and parallel displacements — with no names attached.

The consequences of that split are what an exam question is testing:

• WordNet can tell you a relation is a hypernym relation. word2vec can tell you two words are close and cannot tell you why.
• WordNet distinguishes bank (financial) from bank (river) as separate synsets. Static word2vec collapses both senses into one vector.
• WordNet marks hot and cold as antonyms. word2vec places them very close together, because they appear in nearly identical contexts. This is the practically dangerous asymmetry, and it is the root of the negation failures you saw in 03-04 and will see again in 07-03.
• WordNet's coverage stops where the lexicographers stopped, so it misses jargon, product names, and new slang. word2vec covers whatever the corpus contains often enough.
• WordNet supports graph queries and path-based distances. word2vec supports cosine similarity, nearest neighbours, and the analogy arithmetic in this lesson.

They are complementary, not competing: one gives you named, reliable, incomplete relations; the other gives you unnamed, approximate, broad ones.

Constructed vectors, four dimensions, invented for legibility. Real static embeddings run to hundreds of dimensions and produce no round numbers. What is faithful here is the structure of the result, especially step 4.

man    = [0.50, 0.10, 0.80, 0.10]
woman  = [0.50, 0.90, 0.80, 0.10]
king   = [0.90, 0.10, 0.80, 0.70]
queen  = [0.90, 0.90, 0.80, 0.70]
prince = [0.85, 0.15, 0.75, 0.65]

Note how these were built: dimension 1 loosely tracks "royalty", dimension 2 loosely tracks "female". That is a convenience for the arithmetic and it is exactly the thing real embeddings do not do — real dimensions are entangled and unnamed. Keep that caveat live.

Step 1 — the displacement vector

king − man = [0.90 − 0.50, 0.10 − 0.10, 0.80 − 0.80, 0.70 − 0.10]
           = [0.40, 0.00, 0.00, 0.60]

That displacement is the "royalty" direction as this constructed space encodes it.

Step 2 — apply it to woman

result = woman + [0.40, 0.00, 0.00, 0.60]
       = [0.50 + 0.40, 0.90 + 0.00, 0.80 + 0.00, 0.10 + 0.60]
       = [0.90, 0.90, 0.80, 0.70]

Which is queen's vector exactly — because the example was constructed to make it so. Real spaces give you a point near queen, never queen itself.

Step 3 — cosine to each candidate

result = [0.90, 0.90, 0.80, 0.70]
|result| = sqrt(0.81 + 0.81 + 0.64 + 0.49) = sqrt(2.75) = 1.6583

vs queen  = [0.90, 0.90, 0.80, 0.70]
  dot = 0.81 + 0.81 + 0.64 + 0.49 = 2.75
  |queen| = 1.6583
  cos = 2.75 / (1.6583 × 1.6583) = 2.75 / 2.75 = 1.000

vs woman  = [0.50, 0.90, 0.80, 0.10]
  dot = (0.90×0.50)+(0.90×0.90)+(0.80×0.80)+(0.70×0.10)
      = 0.45 + 0.81 + 0.64 + 0.07 = 1.97
  |woman| = sqrt(0.25 + 0.81 + 0.64 + 0.01) = sqrt(1.71) = 1.3077
  cos = 1.97 / (1.6583 × 1.3077) = 1.97 / 2.1687 = 0.908

vs king   = [0.90, 0.10, 0.80, 0.70]
  dot = 0.81 + 0.09 + 0.64 + 0.49 = 2.03
  |king| = sqrt(0.81 + 0.01 + 0.64 + 0.49) = sqrt(1.95) = 1.3964
  cos = 2.03 / (1.6583 × 1.3964) = 2.03 / 2.3157 = 0.877

vs prince = [0.85, 0.15, 0.75, 0.65]
  dot = (0.90×0.85)+(0.90×0.15)+(0.80×0.75)+(0.70×0.65)
      = 0.765 + 0.135 + 0.600 + 0.455 = 1.955
  |prince| = sqrt(0.7225 + 0.0225 + 0.5625 + 0.4225) = sqrt(1.73) = 1.3153
  cos = 1.955 / (1.6583 × 1.3153) = 1.955 / 2.1816 = 0.896

vs man    = [0.50, 0.10, 0.80, 0.10]
  dot = 0.45 + 0.09 + 0.64 + 0.07 = 1.25
  |man| = sqrt(0.25 + 0.01 + 0.64 + 0.01) = sqrt(0.91) = 0.9539
  cos = 1.25 / (1.6583 × 0.9539) = 1.25 / 1.5819 = 0.790

Step 4 — the ranking, and what the exclusion rule is doing

In this constructed case queen wins outright, because the vectors were built to make it win. But look at the rest of the ranking: woman is second at 0.908 and king is fourth at 0.877 — both input words, both crowding the answer, and prince is a hair behind woman. Perturb these vectors even slightly, in a space of hundreds of real dimensions with noisy frequency-dependent estimates, and the plausible top hit becomes woman or king.

That is precisely what happens with real embeddings, and it is why the standard analogy benchmark removes the three query words from the candidate set before taking the nearest neighbour. Run the same computation without the exclusion rule and a large share of analogy queries return one of their own inputs. So the demonstration is doing something real — the relational displacement genuinely exists — while the headline framing ("the model computes queen") is flattered by a protocol choice that most retellings never mention.

Step 5 — the same machinery, pointed at bias

Take the constructed royalty displacement out and use an occupational one instead. Suppose in a constructed space:

doctor  = [0.80, 0.20, 0.30, 0.60]
nurse   = [0.75, 0.85, 0.30, 0.35]
man     = [0.50, 0.10, 0.80, 0.10]
woman   = [0.50, 0.90, 0.80, 0.10]

doctor − man = [0.30, 0.10, −0.50, 0.50]
woman + (doctor − man) = [0.80, 1.00, 0.30, 0.60]

Whether the nearest neighbour of that result is doctor or nurse depends entirely on the corpus that trained the space. If the training text associated nurse with female contexts and doctor with male ones, the arithmetic will reproduce that association — not because anyone encoded a stereotype, but because the same parallel-displacement property that produces queen produces this too. These specific vectors are constructed and I am not reporting a measured result; published work on embedding bias has examined real cases at length, and the mechanism is the point here. The consequence for practice is direct: an embedding space inherits the regularities of its corpus, including the ones you would not endorse, and any retrieval or classification system built on it inherits them too. 13-04 develops the measurement and mitigation side.

The general rule: embedding arithmetic is an excellent demonstration and a poor tool. It earns its place in this course because it is widely known and widely misread, and because knowing its limits is what stops you from expecting the geometry to be tidier than it is while you are debugging.

Objective 1.8 ("select and use models to create text embeddings") is the formal home, and objective 1.6 (familiarity with Python NLP packages) touches it because word2vec-style vectors are what the classical toolchain hands you. But the real reason this appears is the exam's reported emphasis on classical NLP: candidate reports specifically name WordNet vs word2vec as the most confused item in this area, and this lesson is where the word2vec side gets its sharpest treatment.

The depth required is identification, consistent with the exam's general-level calibration. You need to know what word2vec is, that it produces static dense vectors from co-occurrence, that its two training set-ups are skip-gram and CBOW, that vector arithmetic on those vectors can complete analogies, and how all of that differs from a curated lexical database. You do not need the skip-gram objective function, negative sampling, or the hierarchical softmax.

Question phrasings to expect

• The identity question. "What is word2vec?" A method that learns dense vector representations of words from co-occurrence in a large corpus. Distractors: a lexical database, a tokenizer, a transformer architecture, a vector database.
• WordNet vs word2vec. "Which of these organises words into hand-curated synsets with typed relations?" WordNet. "Which learns dense vectors from a corpus?" word2vec. Also phrased as "what is the primary difference between WordNet and word2vec?" — curated/symbolic versus learned/statistical.
• The analogy result. "What does king − man + woman ≈ queen demonstrate about word embeddings?" That relational information is encoded as consistent directions in the vector space. Distractors will claim it shows dimensions are interpretable, that embeddings perform logical reasoning, or that the model understands gender.
• Skip-gram vs CBOW. "In skip-gram, what does the model predict?" The surrounding context words from the centre word. CBOW is the reverse.
• Static vs contextual. "Why does word2vec assign the same vector to bank in 'river bank' and 'bank account'?" Because it is a static, one-vector-per-word-type model with no context conditioning. The remedy is a contextual model (04-03).
• Antonyms. "Why might hot and cold have high cosine similarity in a word2vec space?" Because they occur in nearly identical contexts, and the model encodes contextual similarity rather than semantic opposition.
• Limits. "Which of the following is not a limitation of word2vec?" Watch the polarity; the real limitations are no sense disambiguation, no word order, weak on rare words, out-of-vocabulary words have no vector at all, and antonyms cluster.
• Out-of-vocabulary. "What happens when word2vec meets a word not in its vocabulary?" There is no vector for it. fastText's subword approach is the standard mitigation and is a reasonable distractor to recognise.

Distractor families

Not equal, and not unconditionally. Three precise corrections.

It is approximate, not equal. The computed vector is a point in the space; you then ask which word's vector is nearest to it. In a real 300-dimensional word2vec space, the distance between the result and queen is not zero.

The three input words are excluded before finding that nearest neighbour. This is the correction most retellings omit. Because the operation is a modest displacement from woman, and because woman, man, and king are frequent words with well-estimated vectors, one of them is often nearer to the result than queen is. The benchmark protocol removes them. With them included, the demonstration is much less impressive — a point that has been made repeatedly in the literature analysing these results.

It works better for some relations than others. Country–capital, currency, singular–plural, and verb-tense analogies tend to work; broad semantic relations and rare words often do not. Analogy accuracy is a measured property of a particular trained space, not a guarantee of the method.

What survives all three corrections is still worth something: the relational displacement genuinely exists in the geometry, which is a real fact about what co-occurrence statistics encode. Say that, and you are accurate. Say "the model computes queen", and you have taught someone to expect a tidiness that will mislead them the first time they debug a retrieval system.

WordNet is a hand-curated lexical database; word2vec is a learned statistical model. That contrast is the whole answer and everything else elaborates it.

WordNet was built by lexicographers who grouped word senses into synsets — sets of senses that mean the same thing — and connected those synsets with explicit typed relations: synonymy, antonymy, hypernymy (is a kind of), hyponymy, and meronymy (is a part of). You query it as a graph. Its facts are named, checkable, and reliable, and its coverage stops wherever the curation stopped, so it has no entry for last quarter's product name or your organisation's internal jargon. It separates senses: bank the financial institution and bank the river edge are different synsets. It marks hot and cold as antonyms.

word2vec was trained on unlabelled text by predicting context words from a centre word (skip-gram) or a centre word from its context (CBOW), and the by-product — one dense vector per word type — is the model. Its "relations" are geometry: small angles for related words, and the approximately parallel displacements that make analogy arithmetic work. Nothing is named, so it can say two words are close and never say why. It does not separate senses, so bank gets one vector serving both meanings. And it places hot and cold close together, because they keep the same company.

Neither replaces the other. If you need a typed relation you can act on and audit, you need a curated resource. If you need broad, approximate relatedness over whatever vocabulary your corpus contains, you need embeddings. Recognising which of the two a question is describing is the exam skill.

Mechanically the operations run — these are vectors, so addition and subtraction are defined — but the results do not have dependable meanings, and you should not build on them.

For contextual (BERT-family) embeddings, the obstacle is that there is no single vector for a word. king has a different vector in every sentence, so king − man is not a well-defined quantity until you fix both contexts, and different context choices give different displacements. Controlled versions of the analogy experiment can be constructed, but the clean lookup-table arithmetic that made word2vec's demonstration crisp is gone by design — the context conditioning that makes contextual embeddings better is exactly what removes the fixed per-word vector.

For sentence embeddings, the obstacle is the training objective. Those vectors are pooled transformer outputs reshaped by contrastive fine-tuning to make cosine comparison of whole spans work (03-02). Nothing in that objective encourages compositional algebra, so embed(A) + embed(B) has no reliable relationship to embed(A + B), and "subtracting a topic" from a chunk vector is an operation with no defined semantics. Averaging several chunk vectors to make a document vector is a partial exception in that it is a common practice — but note that averaging is dilution, not composition, and it makes the vector blander rather than more expressive.

The operations that are dependable on span embeddings: cosine similarity, nearest-neighbour search, clustering, and using the vector as input features to a classifier. That list is short on purpose.

Two things, one encouraging and one cautionary, and both matter for how you work.

Encouraging: it shows that training on nothing but co-occurrence statistics recovers relational structure, not merely topical clustering. Before this result it was reasonable to suppose that "words appearing in similar contexts get similar vectors" would produce only a similarity space — a good thesaurus and nothing more. The analogy demonstration showed the space also carries consistent directional structure corresponding to relations. That is a genuine finding about the information content of co-occurrence.

Cautionary: the structure is whatever is in the corpus, and consistency is not correctness. The same parallel-displacement property that recovers capital-of recovers occupational stereotypes, evaluative associations, and every other regularity of the source text — and no part of the training process distinguishes a relation you would endorse from one you would not. This is why embedding bias is a measurement problem rather than an intention problem, and why 13-04 insists on per-slice evaluation: the space cannot tell you which of its directions are the ones you wanted.

For your day-to-day debugging, the takeaway is calibration. The geometry is more structured than "just similarity" and much less structured than "composable semantics". Related things are near each other; relations exist as approximate directions; senses are collapsed in static models; opposites are close; negation is absent; and nothing is named. Hold that picture and the failures you saw in 03-04 stop being surprising — they start being predictable, which is the point at which you can design around them.

• king − man + woman ≈ queen is real, approximate, and narrower than its reputation. The arithmetic yields a point; the answer is that point's nearest neighbour.
• The standard evaluation excludes the three input words. Without that rule, the nearest neighbour of the result is frequently woman or king, because the operation is a small displacement from an input. Quote the protocol when you quote the result.
• It works because relational differences between word pairs are roughly parallel in a static embedding space — an emergent property of training on co-occurrence, not something anyone designed in.
• Performance is uneven. Syntactic relations and well-attested semantic ones work; rare words, multi-word expressions, and broad semantic relations often do not.
• It is a static-embedding phenomenon. Contextual models have no single vector per word, and sentence-embedding spaces are not compositional — so neither supports the clean arithmetic.
• Directions are not dimensions. The analogy result says nothing about individual components being interpretable; it is about directions through an entangled space.
• word2vec is learned and statistical; WordNet is curated and symbolic. Synsets, typed relations, sense separation, and explicit antonyms belong to WordNet. Dense vectors, cosine similarity, sense collapse, and close-together antonyms belong to word2vec.
• Antonyms are close in embedding space because they share contexts — the root of the negation failures you have already seen and will see again in 07-03.
• The analogy demonstration and the bias demonstration are the same experiment. Whatever regularities the corpus has become directions in the space, endorsed or not, which makes bias a measurement obligation (13-04).
• Exam depth is identification: what word2vec is, skip-gram versus CBOW, static versus contextual, what the analogy shows, and the WordNet contrast. Not the training mathematics.
• Use the arithmetic to demonstrate and to probe, never to build. On span embeddings, the dependable operations are cosine similarity, nearest-neighbour search, clustering, and classifier features.

This module has treated the embedding model as a box: text goes in, a vector comes out, and the geometry of those vectors has properties you can now reason about and test. What the box does internally has stayed closed — and it is about to matter, because every property that made contextual and sentence embeddings better than word2vec comes from one mechanism inside it.

Self-attention is what lets the vector for bank depend on whether river or transfer appeared nearby, which is the whole difference between a static and a contextual embedding. It is also where the costs live: attention compares every position with every other position, so doubling the input length quadruples the work, and that single fact explains maximum sequence lengths, context-window pricing, chunking pressure, and most of what Module 12 has to say about serving. Next: 04-01 opens the transformer, derives why context length costs quadratically, and folds in positional encoding — how a model that looks at all positions at once nevertheless represents word order.