What Text Embeddings Are: Learned Dense Vectors Explained

A text embedding is the output of a function: text goes in, a fixed-length numeric vector comes out. Three properties make that function an embedding rather than just a hash:

1. Fixed dimensionality. Every input, whatever its length, maps to a vector of the same size — commonly a few hundred to a couple of thousand numbers. A three-word query and a 200-word paragraph come out with the same shape, which is what makes them comparable at all. The shape vocabulary here is the one from 01-03: an embedding is a vector of length d, and a batch of them is a matrix of shape (n, d).
2. Density. Nearly every one of those numbers is non-zero, and they are continuous values rather than counts. This is the direct contrast with one-hot and bag-of-words vectors, where the overwhelming majority of entries are exactly zero.
3. Geometry that encodes relatedness. The arrangement is not arbitrary. Training pushes texts used in similar ways toward similar positions, so distance in the space becomes a usable proxy for relatedness in the language. This is the property every downstream use depends on — retrieval, clustering, deduplication, classification, recommendation.

Two clarifications that prevent most of the confusion in this area.

An embedding is not a token. Tokenization, covered in 02-02 and 02-03, is the step that cuts a string into discrete integer ids from a fixed vocabulary. Embedding is the step after that, which replaces each integer id with a vector. Tokens are a segmentation decision; embeddings are a representation. A question that asks "what does the model receive as input" wants the token ids; a question that asks "how is meaning represented numerically" wants the embedding.

"Embedding" names both a lookup table and a model output. Inside a transformer there is a literal embedding matrix of shape (vocabulary size, hidden size) whose rows are the vectors for each token id — a lookup table, and one of the largest single parameter blocks in a small model. Separately, an embedding model is a whole network you call to get one vector for a sentence or a document. Both are correctly called embeddings, and the difference between them is the entire subject of 03-02. Keep the distinction available; the exam trades on it.

L1 — The intuition: meaning from company kept

The operating principle is the distributional hypothesis: words that occur in similar contexts tend to have similar meanings. You do not need to define invoice to notice that it shows up near overdue, payment, account, and net 30, and that bill shows up in exactly the same neighbourhoods. A model that is trained to predict context from a word, or a word from its context, is forced to give those two words similar internal representations, because similar representations are what make the same predictions.

Nothing in that process consults a dictionary, a thesaurus, or a linguist. The arrangement is a side effect of a prediction task. This is why embedding spaces reproduce whatever regularities — including the unwanted ones — exist in the training corpus, and why an embedding trained on general web text can be a poor fit for a corpus of radiology reports or Swedish contract law.

L2 — The mechanism: a prediction task whose by-product is the vector

Concretely, the classic static-embedding recipe (word2vec) sets up a fake supervised problem over unlabelled text:

• Skip-gram: given the centre word, predict the words in a small window around it.
• CBOW (continuous bag of words): given the surrounding window, predict the centre word.

Each word has a vector; the vectors start random; the training loop nudges them so that the prediction gets better. When training stops, the prediction head is thrown away and the vectors are the product. GloVe reaches a similar place by a different route — it factorises a global word–word co-occurrence matrix rather than sliding a window with a classifier — but the input signal is the same signal: co-occurrence statistics over a corpus.

Contextual embeddings (BERT-style, from the encoder family in 04-03) change one thing that matters enormously. Instead of one vector per word type, the model computes a vector per word occurrence, conditioned on the surrounding sentence via self-attention. The word bank in "river bank" and in "bank transfer" gets one identical vector from word2vec and two different vectors from BERT. The training objective changes too — masked-language modelling instead of window prediction — but the by-product logic is unchanged: you train on a prediction task and harvest internal representations.

Sentence-embedding models add a third ingredient: they are usually fine-tuned on pairs. A contextual encoder is trained further with an objective that explicitly pulls related sentence pairs together and pushes unrelated pairs apart (contrastive training on question–answer pairs, duplicate-question pairs, translation pairs, and similar). That fine-tuning is why a purpose-built sentence-embedding model beats naively averaging a base encoder's token vectors — the base model was never asked to make whole-sentence vectors comparable, and the sentence model was.

L3 — What the numbers are, and what they are not

A trained embedding for one word or one sentence is just an array. There is no dimension for "formality" and no dimension for "is about money". The space has directions that correlate with human-interpretable properties, and those directions are almost never axis-aligned; you find them with linear probes or by averaging differences, not by reading column 47.

Three consequences that matter in practice:

• The numbers are only meaningful relative to other vectors from the same model. There is no absolute scale. A vector's raw magnitudes tell you nothing you can act on; the comparison to another vector does.
• Two different models produce incompatible spaces. Model A's dimension 3 has no relationship to model B's dimension 3, even at identical dimensionality. Mixing them produces confident nonsense, which is why 12-12 treats changing an embedding model as a corpus migration rather than a config change.
• The space is fixed at inference time. Calling an embedding model does not update anything. Embedding is a forward pass, cheap and stateless, and repeated calls on the same input with the same model version give the same vector.

For measuring closeness, the tool is the cosine similarity from 01-04: the dot product of two vectors divided by the product of their lengths, which equals the cosine of the angle between them and ranges from −1 to 1 (in practice, embedding models rarely produce genuinely negative similarities across ordinary text). Cosine ignores magnitude and reads only direction, which is what you want when one text is three words and the other is three hundred.

This progression is the single highest-value table in the module. The exam asks about the identity of each family — what it is, what signal it uses, what it can and cannot do — far more often than it asks about training mathematics.

WordNet vs word2vec: the confusable to get right

Candidate reports single this pair out as the most frequently confused item in this whole area, and the reason is that both are described in one sentence as "a resource that tells you which words are related." The mechanisms could not be less alike.

The one-line answer to hold: WordNet is curated and symbolic; word2vec is learned and statistical. WordNet knows that a robin is a bird because a person wrote that down. word2vec "knows" that robin and bird are related because they appear in similar sentences — and it does not know that one is a kind of the other, only that they are near.

The antonym row is the practically important one. In a curated ontology, increase and decrease are linked as opposites, so a system can act on the opposition. In an embedding space, increase and decrease tend to be close neighbours, because they occur in nearly identical contexts — which is exactly why embedding-based retrieval struggles with negation, a limit taken up in 07-03.

Here is arithmetic you can do by hand, on a corpus small enough to hold in your head. These numbers are a constructed illustration, not measurements from a real model. The point is the pattern of the answers, not the values.

Corpus, three documents, one sentence each:

D1: invoice overdue payment
D2: bill    overdue payment
D3: otter   river   fish

Vocabulary (7 terms): invoice, bill, overdue, payment, otter, river, fish.

Step 1 — one-hot vectors for two synonyms

Encode the single words invoice and bill as one-hot vectors over that vocabulary:

invoice = [1, 0, 0, 0, 0, 0, 0]
bill    = [0, 1, 0, 0, 0, 0, 0]

dot product        = (1*0) + (0*1) + 0 + 0 + 0 + 0 + 0 = 0
cosine similarity  = 0 / (1 * 1) = 0.00

Zero. Not "low" — exactly zero, and it will be exactly zero for every distinct pair of words in the vocabulary. One-hot vectors are mutually orthogonal by construction, so every pair of different words is equally unrelated: invoice and bill are as far apart as invoice and fish. Any resemblance to meaning is impossible, not merely poor.

Step 2 — TF-IDF over the documents

Use idf(t) = ln(N / df(t)) with N = 3 documents, and raw term frequency of 1 (each term appears once per document):

df(invoice) = 1  → idf = ln(3/1) = 1.0986
df(bill)    = 1  → idf = ln(3/1) = 1.0986
df(overdue) = 2  → idf = ln(3/2) = 0.4055
df(payment) = 2  → idf = ln(3/2) = 0.4055
df(otter)   = 1  → idf = 1.0986   (same for river, fish)

Document vectors (non-zero entries only):

D1 = invoice 1.0986, overdue 0.4055, payment 0.4055
D2 = bill    1.0986, overdue 0.4055, payment 0.4055
D3 = otter   1.0986, river   1.0986, fish    1.0986

Cosine between D1 and D2:

dot(D1,D2) = (1.0986 * 0)        [invoice vs bill: no overlap]
           + (0.4055 * 0.4055)   [overdue]
           + (0.4055 * 0.4055)   [payment]
           = 0.1644 + 0.1644 = 0.3289

|D1| = sqrt(1.0986^2 + 0.4055^2 + 0.4055^2)
     = sqrt(1.2069 + 0.1644 + 0.1644) = sqrt(1.5358) = 1.2393
|D2| = 1.2393  (same structure)

cos(D1,D2) = 0.3289 / (1.2393 * 1.2393) = 0.3289 / 1.5358 = 0.21

And between D1 and D3:

dot(D1,D3) = 0   (no shared terms at all)
cos(D1,D3) = 0.00

Read what happened. TF-IDF did register that D1 and D2 are more alike than D1 and D3 — 0.21 versus 0.00 — but it earned that entirely from the two words the documents literally share. The synonym pair contributed nothing. Strip overdue and payment out ("the invoice is overdue" vs "the bill is unpaid") and TF-IDF's similarity collapses to near zero for two sentences a human would call paraphrases.

Step 3 — constructed dense vectors

Now suppose a trained embedding model gives us these 4-dimensional vectors. Constructed for the arithmetic; real models use hundreds of dimensions and would not produce round numbers.

invoice = [0.9, 0.1, 0.2, 0.0]
bill    = [0.8, 0.2, 0.3, 0.1]
otter   = [0.0, 0.9, 0.1, 0.4]

Cosine, invoice vs bill:

dot     = (0.9*0.8) + (0.1*0.2) + (0.2*0.3) + (0.0*0.1)
        = 0.72 + 0.02 + 0.06 + 0.00 = 0.80
|invoice| = sqrt(0.81 + 0.01 + 0.04 + 0.00) = sqrt(0.86) = 0.9274
|bill|    = sqrt(0.64 + 0.04 + 0.09 + 0.01) = sqrt(0.78) = 0.8832
cosine  = 0.80 / (0.9274 * 0.8832) = 0.80 / 0.8187 = 0.98

Cosine, invoice vs otter:

dot     = (0.9*0.0) + (0.1*0.9) + (0.2*0.1) + (0.0*0.4)
        = 0.00 + 0.09 + 0.02 + 0.00 = 0.11
|otter| = sqrt(0.00 + 0.81 + 0.01 + 0.16) = sqrt(0.98) = 0.9899
cosine  = 0.11 / (0.9274 * 0.9899) = 0.11 / 0.9181 = 0.12

The three results side by side are the argument for embeddings in one table:

Two honest caveats, because this is where over-claiming starts. First, the dense numbers are invented; do not memorise 0.98 as what a real model returns for a synonym pair. Second, cosine scores are not comparable across models and there is no universal threshold. One model's "clearly related" region may start around 0.6 and another's around 0.85, because different training objectives and normalisation schemes compress the score range differently. The lesson 03-04 shows how to calibrate that by hand for the model you actually chose; anyone who hands you a fixed cut-off without naming the model is guessing.

Embeddings are not the answer to every text problem, and the exam rewards knowing the boundary. The [FIELD] calibration for this certification is that questions are general-level: know what each thing is and when to use it. This table is that knowledge for embeddings.

The general rule: embeddings buy you recall over paraphrase and cost you precision over exact strings. A production system usually wants both, which is why hybrid search exists (07-06). Reaching for a vector store as the default, without a keyword baseline to compare against, is the most common over-engineering in applied retrieval — and 07-01 deliberately teaches sparse retrieval first so you can measure whether the dense side actually helped.

Text embeddings are named directly by an official objective. Objective 1.8 — "Select and use models to create text embeddings" is the only objective in the blueprint whose verb is select and use, and it sits inside Core Machine Learning and AI Knowledge, the 30% domain. Two more objectives depend on embeddings without naming them: 1.4 — "Curate and embed content datasets for RAGs" (the word embed is in the objective text) and 1.6 — "Familiarity with the capabilities of Python natural language packages (spaCy, NumPy, vector databases, etc.)", because a vector database is a thing you can only use if you have vectors to put in it.

That gives embeddings an unusual profile for an associate-level exam: they are foundational — many other questions quietly assume them — and they are explicitly claimed by an objective, which means a question can legitimately ask about them head-on.

How embedding questions get phrased

Expect these shapes, all consistent with the general-level depth this exam is calibrated to:

• Identity questions. "What is a text embedding?" or "Which statement best describes a dense vector representation of text?" The correct answer names fixed-length, dense, learned, and similar text is nearby. Distractors describe a one-hot vector, a token id, or a hash.
• Progression questions. "Which representation captures semantic similarity between synonyms?" The answer is the embedding family; TF-IDF and bag-of-words are the distractors, and they are distractors because they are keyed on surface terms.
• WordNet vs word2vec. "Which resource organises words into hand-curated synsets?" (WordNet.) "Which produces dense vectors learned from co-occurrence statistics?" (word2vec.) This pair is reported as the most common confusable in this area, which is why §3 gives it its own table.
• Similarity-metric questions. "Which metric is normally used to compare two text embeddings?" Cosine similarity. Euclidean distance is a defensible-looking distractor and is genuinely used in some indexes; the conventional answer for text embeddings is cosine, because it ignores magnitude.
• Scenario / tool-choice questions. "A team needs to retrieve relevant support articles for free-text customer questions. Which approach?" Embedding the corpus and doing vector search is keyed. Watch for the RAG heuristic here — when a scenario option proposes retrieval-augmented generation, it is frequently the intended answer, though 07-12 names the cases where it is wrong.
• "Which is not true" questions. These often plant the claim that embedding dimensions are individually interpretable, or that two models' embeddings can be compared, or that embeddings capture recency. All three are false and all three are worth recognising instantly.

Distractor families to recognise

Depth ceiling for the exam

You do not need to derive the skip-gram objective, explain negative sampling, or reproduce GloVe's weighted least-squares formulation. The [FIELD] calibration for this exam is explicit that deep architecture and training maths were reported as overkill. What you need cold is: the identity of each representation family, which signal each uses, what each can and cannot see, that cosine is the comparison metric, and the WordNet/word2vec split. That is the study allocation this lesson is built for.

No. word2vec is one specific, early, static method for producing word embeddings; "text embeddings" is the whole family, which now includes contextual token embeddings from encoder models and sentence/document embeddings from models fine-tuned on pairs. Using "word2vec" as a synonym for "embedding" is like using "floppy disk" as a synonym for "storage" — it names an important ancestor, not the category.

The practical difference: word2vec gives one vector per word type, from a lookup table, with no sentence context. A modern sentence-embedding model gives one vector per span of text, computed by a transformer that has read the whole span. For anything involving retrieval over documents, you want the latter.

No, and this is the confusable worth over-learning. WordNet is a hand-curated lexical database: humans organised word senses into synsets and connected them with explicit, typed relations — synonymy, antonymy, hypernymy (dog is-a mammal), hyponymy, meronymy (wheel part-of car). Nothing in WordNet is a vector, nothing in it was learned from a corpus, and its coverage stops wherever the lexicographers stopped.

An embedding is the opposite on every axis: no curator, no explicit relations, no sense separation in the static case, coverage determined by corpus frequency, and the relations are only readable as geometry. Two things follow. First, WordNet can tell you a typed fact ("this is a kind of that") that an embedding cannot express at all. Second, an embedding covers jargon, product names, and last month's slang that WordNet will never contain. They are complementary resources that answer different questions, and a question that offers both as options is testing whether you know which is curated and which is learned.

Because cosine measures the angle between two vectors and ignores their length, and for text the length is mostly a nuisance signal. Longer texts and more frequent words tend to produce larger-magnitude vectors; if you use raw Euclidean distance, a short query can be "far" from a long passage that says exactly the same thing, purely because of magnitude. Cosine strips that out and asks only "do these point the same way".

Two footnotes for completeness. When vectors are L2-normalised to unit length — which many embedding APIs do for you — cosine similarity and Euclidean distance become monotonically related, so ranking by either gives the same order, and inner product becomes equivalent to cosine too. And some vector indexes are configured for inner product or L2 rather than cosine; you must match the index's metric to how the model was trained, because a mismatch degrades results quietly rather than erroring. 07-04 covers the index side of that.

No. Tokenization is a prerequisite, not an alternative: an embedding model tokenizes its input first, then embeds those tokens. Every token-budgeting concern from 02-03 still applies, because an embedding model has a maximum input length measured in tokens and silently truncates beyond it — which is one of the selection criteria in 03-03.

TF-IDF and BM25 are genuinely alternatives, and they are not obsolete. Sparse retrieval remains the better tool for exact terms, rare identifiers, and any case where the required phrase must literally appear. The mature answer is that dense and sparse retrieval fail in different directions — dense misses exact strings, sparse misses paraphrases — which is precisely why hybrid search combining both is standard in production RAG (07-06).

You can visualise the space, not the dimensions. Projecting embeddings to two or three dimensions with a technique like PCA, t-SNE, or UMAP produces a picture where clusters are often genuinely meaningful, and that is a legitimate diagnostic — you can see that your support tickets separate into billing, shipping, and technical groups without labelling anything.

What you cannot do is read a single dimension. There is no "topic" axis; the informative directions are combinations of many dimensions and are found by probing, not by inspection. And treat the 2-D plot with suspicion as evidence: t-SNE and UMAP distort global distances by design, so apparent gaps between clusters and apparent cluster sizes in the picture are not reliable quantities. Use the plot to generate hypotheses and pairwise cosine similarity on the full vectors to test them.

• A text embedding is a fixed-length, dense, learned vector whose position encodes relatedness. Fixed-length makes texts comparable; dense distinguishes it from one-hot and TF-IDF; learned means no human assigned the dimensions.
• The progression is one-hot → bag-of-words → TF-IDF → static (word2vec, GloVe) → contextual (BERT) → sentence/document embeddings, and each step buys something the previous one structurally could not have: counts, then rarity weighting, then synonymy, then sense disambiguation, then span-level comparability.
• One-hot vectors give exactly zero similarity for every distinct word pair. That is not a tuning problem; it is orthogonality by construction, and it is the reason embeddings exist.
• TF-IDF can only see similarity through shared surface terms. In the worked example it scored two paraphrases at 0.21 — and every bit of that came from the two words they happened to share, not from the synonym pair.
• WordNet is curated and symbolic; word2vec is learned and statistical. WordNet has synsets, typed relations, and explicit antonyms. word2vec has vectors, cosine geometry, one vector per surface word, and antonyms that sit suspiciously close together.
• Cosine similarity is the conventional comparison metric because it reads direction and ignores magnitude — but scores are not comparable across models and there is no universal "related" threshold. Calibrate for your own model.
• Two models' embeddings are never interchangeable, even at identical dimensionality. Changing the model means re-embedding the corpus.
• Embeddings buy recall over paraphrase and lose precision over exact strings. Identifiers, error codes, and verbatim terms of art belong to keyword retrieval; production systems usually run both.
• Do not read individual dimensions. Debug with nearest neighbours and pairwise similarity; use dimensionality-reduction plots to form hypotheses, not to prove them.
• Exam depth is identity and selection, not training maths. Know what each family is, what signal it uses, what it cannot see, and when to reach for it.

This lesson has been deliberately loose about one thing. It said "a piece of text" gets a vector, and left open whether that piece is a token, a sentence, a paragraph, or a whole document — and it hinted twice that averaging a base encoder's token vectors is not the same as using a model built for sentences.

That gap is the single most consequential confusion in applied retrieval, and it is where things actually break: a team embeds with a token-level model, averages the outputs, gets plausible-looking vectors with no error message, and then spends weeks wondering why retrieval is mediocre. Next: 03-02 separates token embeddings from sentence and document embeddings — what pooling is, why a model fine-tuned on pairs beats an averaged base encoder, which unit of text you should be embedding for retrieval, and how chunk size interacts with all of it. After that, 03-03 turns these distinctions into an actual model-selection procedure you can defend.