Scaled Dot-Product Attention: Query, Key, Value, and Why Divide by √dₖ

Identity statement: scaled dot-product attention takes three input matrices — Queries (Q), Keys (K), and Values (V), each a stack of per-token vectors — and produces a new set of per-token vectors, each one a weighted combination of every Value vector in the sequence, where the weights come from comparing each token's Query against every token's Key. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V."

Unpack that left to right. QKᵀ is a matrix multiplication between the Query matrix and the transpose of the Key matrix; the result is a square matrix of raw similarity scores, one row per Query token and one column per Key token, where entry (i, j) is the dot product between token i's Query vector and token j's Key vector. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Q·Kᵀ scores how much each token should attend to every other token." A high score means token i's Query and token j's Key point in a similar direction in the shared vector space the model learned; a low or negative score means they do not. Dividing that whole matrix by √dₖ rescales every entry before anything else happens to it — this is the step section 2 derives in full. Softmax is then applied along each row, turning that row's raw scores into a probability distribution that sums to 1 across all the tokens token i could attend to. Finally, that row of softmax weights is used to compute a weighted sum of the Value vectors — token i's new representation is literally "a little bit of Value 1, a little bit of Value 2, a lot of Value 7," in whatever proportions the softmax row assigned.

Nothing in that description mentions recurrence, and that is deliberate. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Transformers are parallel, not sequential like RNNs. Self-attention processes all tokens at once." Every row of QKᵀ can be computed simultaneously, on a GPU, as one matrix multiplication, because no token's Query needs to wait for another token's output to exist first — every Query, Key, and Value comes from the same input embeddings through independent linear projections. That parallelism, made possible by exactly this formula, is the historical reason the transformer displaced recurrent architectures for sequence modeling in the first place: an RNN processes token 1, then token 2 conditioned on token 1's hidden state, then token 3 conditioned on that, one dependency chain at a time; a transformer layer computes attention for every token in one parallel operation, at the cost of needing an explicit way to represent order — which is exactly the gap positional encoding fills, a mechanism the next lesson in this module takes up in full.

L1 — Intuition

Picture a sentence being read one word at a time, and picture each word asking a silent question of every other word in the sentence: "how relevant are you to figuring out what I mean here?" The word's Query vector is that question, encoded as a direction in a learned vector space. Every other word answers with its Key vector — a different learned projection of the same word, encoded so that a Key pointing in a similar direction to the asking word's Query signals "I am relevant to that question." The word being asked does not hand back its Key as the answer, though; it hands back its Value vector, a third, separately learned projection that carries the actual content the asking word should absorb. This is the single fact that resolves the Query-versus-Value confusion at the intuition level, before any arithmetic: the Key is the label on the envelope, telling the Query whether to open it; the Value is what is inside the envelope once opened. Comparing against the label and reading the contents are two different operations, performed on two different vectors, and no amount of surface-level similarity between "Query" and "Value" as English words should suggest otherwise.

L2 — Mechanism

Move from the metaphor to the arithmetic. Every token's input embedding, augmented with positional information, is multiplied by three separate learned weight matrices — Wq, Wk, and Wv — to produce that token's Query, Key, and Value vectors. These three matrices are learned during training and are different from each other; there is no constraint forcing Wq and Wv to end up similar, and in practice they do not, because they are trained to serve different downstream roles in the dot product and the weighted sum respectively. Once every token in the sequence has its Q, K, and V vectors, the whole sequence's Query vectors are stacked into a matrix Q, the whole sequence's Key vectors into K, and the whole sequence's Value vectors into V. The comparison step, QKᵀ, computes every token's Query dotted against every token's Key in one matrix multiplication — an O(n²) operation in sequence length, which is why context length has a real, felt compute cost, but that cost buys full pairwise comparison in a single parallel pass rather than the token-by-token propagation an RNN requires. The result is divided by √dₖ, softmaxed row-wise so each token's attention weights over the sequence sum to 1, and used to weight-and-sum the Value matrix V. The output is a new matrix, same shape as V, where each row is that token's context-aware representation — richer than the raw embedding it started from, because it now incorporates information pulled from wherever the Query-Key comparison decided was relevant.

L3 — Why √dₖ specifically, and why swapping roles breaks the mechanism

Here is the derivation the exam rewards knowing, not just quoting. Assume the entries of a Query vector q and a Key vector k are each drawn independently with mean 0 and variance 1 — a reasonable approximation for the outputs of a well-initialized linear layer early in training. The dot product q·k is a sum of dₖ independent products of mean-zero, unit-variance terms, and the variance of that sum grows linearly with the number of terms: Var(q·k) = dₖ. That means the standard deviation of the raw dot product scores in QKᵀ grows as √dₖ — so as the key dimension gets larger (64, 128, whatever the model's head width is), the typical magnitude of the raw attention scores gets larger too, purely as an artifact of dimensionality, with no relationship to how genuinely relevant one token is to another. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Dividing by √dₖ ... keeps the dot products from growing large and pushing softmax into saturation (vanishing gradients)." Softmax turns large-magnitude inputs into an extremely peaked, near-one-hot distribution, and in that peaked regime softmax's gradient with respect to its inputs is nearly zero almost everywhere — the function has essentially stopped responding to small changes in its inputs, which stalls backpropagation through the attention weights. Dividing every entry of QKᵀ by √dₖ exactly cancels the variance growth derived above: Var(q·k / √dₖ) = dₖ / dₖ = 1, restoring a fixed, dimension-independent scale for the scores regardless of how wide the Key vectors are. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "The √dₖ scaling is not optional — it prevents softmax saturation; dropping it destabilizes training." That is why the constant is specifically √dₖ and not, say, dₖ or a fixed hyperparameter tuned per model: it is the exact quantity that cancels a variance that grows precisely as dₖ, derived from the mechanism rather than chosen by search.

The same section of the source material that states the √dₖ requirement states the Query/Value distinction as a paired trap: [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "The Query asks (is compared against Keys to produce weights); the Value carries the information content that gets combined. Swapping these is a classic distractor." Concretely, if you swapped Value for Query in the formula — computing softmax(VKᵀ/√dₖ)·V instead of softmax(QKᵀ/√dₖ)·V — the model would be comparing the content-carrying vectors against Keys instead of the question-asking vectors, and the weighted sum would still correctly draw from V, but the weights themselves would now reflect a completely different, untrained-for comparison. Because Wq and Wv are independently learned and end up representing different subspaces, this is not a benign relabeling; it changes what the attention weights actually mean, and every downstream layer that expected genuine Query-Key semantics to be driving the weighting receives garbage instead. That is the entire reason this pairing is tested directly rather than glossed over: getting the formula's shape right while swapping which learned matrix plays which role produces a plausible-looking equation that computes the wrong thing.

⭐ THE EARNED INSIGHT The Query and the Key are never in the final answer — they exist purely to decide how much of each Value to use, and then they are discarded for that token's output. Every plausible-sounding distractor about this mechanism — swap Q and V, forget the scale, treat attention as sequential — is a version of confusing the vector that decides the weighting with the vector that carries the content, or confusing a fixed normalizing constant with an optional stylistic choice. Once you can state, from the variance derivation rather than from memory, why √dₖ is the correct constant, no version of this question can surprise you.

Take a tiny, illustrative sequence of three tokens — call them A, B, and C — with a Key/Query dimension of dₖ = 4, so √dₖ = 2. Constructed scenario: the vectors below are illustrative, not measured from a real trained model. Assume the Query, Key, and Value vectors for each token (already projected through Wq, Wk, Wv) are:

Q_A = [1, 0, 1, 0]     K_A = [1, 0, 1, 0]     V_A = [1, 2, 0, 0]
Q_B = [0, 1, 0, 1]     K_B = [0, 1, 0, 1]     V_B = [0, 1, 2, 0]
Q_C = [1, 1, 0, 0]     K_C = [1, 1, 0, 0]     V_C = [0, 0, 1, 2]

Step 1 — raw scores for token A's row (Q_A dotted against every Key):
  Q_A . K_A = (1*1)+(0*0)+(1*1)+(0*0) = 2
  Q_A . K_B = (1*0)+(0*1)+(1*0)+(0*1) = 0
  Q_A . K_C = (1*1)+(0*1)+(1*0)+(0*0) = 1

Step 2 — scale by dividing by sqrt(dk) = 2:
  [2/2, 0/2, 1/2]  =  [1.0, 0.0, 0.5]

Step 3 — softmax the scaled row:
  exp: [e^1.0, e^0.0, e^0.5] = [2.718, 1.000, 1.649]
  sum = 5.367
  weights = [0.507, 0.186, 0.307]   (each divided by 5.367, sums to ~1.0)

Step 4 — weighted sum of Value vectors using those weights:
  0.507 * V_A + 0.186 * V_B + 0.307 * V_C
  = 0.507*[1,2,0,0] + 0.186*[0,1,2,0] + 0.307*[0,0,1,2]
  = [0.507, 1.014, 0, 0] + [0, 0.186, 0.372, 0] + [0, 0, 0.307, 0.614]
  = [0.507, 1.200, 0.679, 0.614]

Token A's new, context-aware representation is [0.507, 1.200, 0.679, 0.614] — a blend dominated by its own Value (weight 0.507) but pulling meaningfully from tokens B and C as well, in the proportions the scaled, softmaxed comparison assigned. Notice what did not happen anywhere in this computation: nothing waited on token B's or token C's row to finish before token A's row could be computed. Each row is an independent computation over the same K and V matrices, which is exactly the parallelism claim from section 1 made concrete in arithmetic.

The first worked example used dₖ = 4, where the scaling correction is modest. This second example holds the same relative pattern of scores but widens dₖ to show the saturation problem section 2's L3 tier derives, because the exam-relevant failure only becomes dramatic at realistic model widths. Constructed scenario, illustrative numbers only. Suppose a real transformer head uses dₖ = 64 (a common per-head width), and suppose two tokens' Query and Key vectors happen to align well enough to produce a raw dot product of 40 before any scaling — a plausible outcome once you have 64 independent terms summing up, per the variance argument in section 2.

Unscaled score:            40
Softmax input (unscaled):  40   (compared against, say, a second token's score of 2)
softmax([40, 2]) = [ e^40 / (e^40 + e^2), e^2 / (e^40 + e^2) ]
                 ~= [1.0000000000, 0.0000000000...]   (rounds to a one-hot vector)

Scaled score (dk = 64, sqrt(dk) = 8):
  40 / 8 = 5.0
  second token's score:  2 / 8 = 0.25
softmax([5.0, 0.25]) = [ e^5.0 / (e^5.0+e^0.25), e^0.25 / (e^5.0+e^0.25) ]
                     ~= [0.9877, 0.0123]

Read the contrast, not just the numbers. Unscaled, the softmax collapses to a value indistinguishable from an exact one-hot vector in floating-point arithmetic — the gradient with respect to either input is effectively zero at that point, because softmax's output stops changing meaningfully however the inputs move in the neighborhood of such an extreme value. Scaled, the same relative comparison still favors the first token heavily (0.9877 versus 0.0123) — the model's preference is preserved, which is the whole point; scaling does not erase genuine signal — but the softmax is now operating in a region where its output still responds to further changes in its inputs, which is the region gradients can actually flow through during backpropagation. This is the concrete, worked version of "softmax saturation": not a vague warning, but an arithmetic fact about what a wide Key dimension does to unscaled dot products, and what dividing by √dₖ does to bring the scores back into a range where training can still make progress.

Scaled dot-product attention, exactly as derived above, is not multi-head attention — it is the single operation multiple heads each run in parallel over a different learned subspace, an idea the next lesson in this module develops in full. It is also not, by itself, aware of token order: nothing in softmax(QKᵀ/√dₖ)·V references position, because the Query-Key comparison only ever depends on the content of the vectors being compared, not where in the sequence they sit. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Attention itself is order-agnostic, so positional information is added to token embeddings." That gap is filled upstream of attention, before Q, K, and V are even projected, and it is a direct consequence of the fact that this lesson has just derived: a mechanism built entirely from pairwise dot products and a weighted sum has no notion of sequence order baked in unless something else supplies it.

It is worth being precise about scope here, because this lesson deliberately stops at the single-head mechanism. Everything above computes one attention operation over one shared representation space; the multi-head design that splits Q, K, and V into several smaller, independently learned subspaces and concatenates the results afterward, and the positional encoding that resolves the order-agnostic gap just named, are both genuinely separate mechanisms layered on top of what this lesson covers — not restatements of it. Holding that boundary clearly is itself exam-relevant: a question that asks specifically about the scaling factor or the Q/K/V role split is testing this lesson's content, and a question that asks about parallel heads or word order is testing the next one.

[GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): LLM Architecture is Domain 1 of the NCP-GENL blueprint, carrying "only 6%" of exam weight — "the smallest domain, but the conceptual base for everything else." Scaled dot-product attention is the first mechanism the domain's own study material introduces, under objective 1.1, and the source explicitly frames this as a professional-level exam: candidates are expected "to reason about why an architecture or sampling choice fits a task," not merely to recite the formula. A low domain weight does not mean low leverage — every later domain that discusses transformer behavior, from Model Optimization's KV caching to GPU Acceleration's parallelism strategies, assumes you already understand what Q, K, and V are doing at each layer.

Expect this topic in a few recurring shapes. A direct computation or identification item asks you to state the formula or identify what QKᵀ represents — the source material's own self-check phrases this almost exactly: "why divide by √dₖ?" with "keep dot products from growing large and saturating softmax" as the keyed answer against distractors like "normalize the output to sum to 1" (that is softmax's job, not the scaling's) or "add positional information" (a real mechanism, attached to the wrong step). A role-identification item describes what a vector does — "asks a question," "gets compared against," "carries content forward" — and asks you to name it Query, Key, or Value; the swapped-role distractor is the domain's signature trap here. A mechanism-versus-architecture item states a true fact about self-attention's parallelism and asks you to contrast it with RNN-style sequential processing, testing whether you understand that all tokens are processed in one pass rather than one at a time.

What the distractors typically look like

The standard traps in this lesson's style are: offering the Value vector's job as the Query's, or vice versa, in an otherwise-correct description of the formula; describing the √dₖ division as optional, a matter of "convention," or something that "normalizes to sum to 1" (that is softmax, not the scaling term); and describing self-attention as processing tokens "one at a time, left to right," which describes causal masking in decoder architectures — a real, later-covered mechanism — misapplied to describe the base attention operation itself, which is inherently parallel regardless of whether a causal mask is later applied on top of it.

Because √dₖ is the exact value that cancels the variance growth the dot product QKᵀ picks up as the key dimension increases: if Query and Key entries are mean-zero, unit-variance values, the dot product of two dₖ-dimensional vectors has variance dₖ, so its standard deviation grows as √dₖ. Dividing by that same √dₖ restores a dimension-independent scale, keeping the scores in a range where softmax's gradient has not collapsed. Any other constant would either under-correct (leaving some residual growth with dₖ) or over-correct (unnecessarily flattening genuine signal), which is why the exam treats this as a specific, derivable fact rather than a tunable hyperparameter.

They are produced by three different learned weight matrices — Wq, Wk, and Wv — applied to the same input embedding, and because these matrices are trained independently to serve different roles in the formula, they end up representing genuinely different subspaces rather than three copies of the same vector. The Query is trained to be useful when dotted against Keys; the Key is trained to be useful when dotted against Queries; the Value is trained to carry information that is useful once selected by the softmax weights. Sameness of origin (one token, one embedding) does not imply sameness of learned function, and treating the three as interchangeable because they start from the same input is the exact error the swapped-role distractor is built to catch.

No — the mechanism as derived in this lesson depends only on the content of the Query and Key vectors being compared, never on where in the sequence a token sits, which is why it is described as order-agnostic. A sentence and a scrambled version of the same sentence, fed through identical Q, K, V projections with no positional signal added beforehand, would produce identical attention patterns based purely on content similarity, which is almost never the intended behavior for language. This is precisely why positional encoding exists as a separate, necessary addition applied to the input embeddings before attention runs, rather than as an optional refinement — a gap this lesson deliberately leaves open for the next one to close.

• The formula is softmax(QKᵀ/√dₖ)·V, and every symbol has a distinct, non-interchangeable job: Q asks, K is compared against, V carries the content forward.
• The Query and Value roles are never interchangeable, even though both come from linear projections of the same token embedding — swapping them is the domain's classic distractor.
• Dividing by √dₖ is a derived necessity, not a stylistic choice: it cancels the variance growth of QKᵀ that would otherwise scale with the key dimension and push softmax into a near-zero-gradient saturation regime.
• Self-attention is parallel, not sequential — every token's attention output is computed in one matrix operation, unlike an RNN's step-by-step dependency chain.
• The mechanism is order-agnostic on its own, which is precisely why positional encoding must be added separately before attention runs.
• This lesson covers single-head attention only — multiple heads running this same operation over different subspaces, and the positional and normalization machinery around it, are the next lesson's subject.
• Domain 1 is 6% of the NCP-GENL blueprint — the smallest domain by weight, but this specific mechanism underlies how every other domain's discussion of transformer behavior is framed.

This lesson deliberately isolated a single attention operation over a single shared subspace. Real transformer layers never stop there: they run several of these operations in parallel over different learned subspaces, add a positional signal upstream to fix the order-agnostic gap just identified, and wrap the whole stack in normalization to keep very deep networks trainable. M1-02 picks up exactly there — why running attention "in parallel over different subspaces" is not a parameter-reduction shortcut but a genuine expansion of what a single layer can represent, why positional encoding exists only because of the gap this lesson closed the door on, and what layer normalization stabilizes that scaling alone does not.