Multi-Head Attention, Positional Encoding, and Layer Normalization Explained

Identity statement: multi-head attention splits the attention computation into h independent "heads," each with its own learned Wq, Wk, and Wv projections (typically into a smaller dimension than the model's full hidden size), runs scaled dot-product attention separately within each head, and then concatenates all h heads' outputs and passes the result through one more learned projection to produce the layer's final output. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Several attention 'heads' run in parallel, each learning different relationships (syntax, coreference, long-range dependencies), then their outputs are concatenated and projected. This lets the model attend to multiple representation subspaces simultaneously."

The phrase "multiple representation subspaces" is doing real work and is worth sitting with. A model's hidden size — say, 512 or 4,096 — defines one large vector space every token's representation lives in. A single attention head with Wq, Wk, Wv projecting into that full space learns one way of deciding what is relevant. Multi-head attention instead typically divides that same total dimensionality across h heads — for example, a 512-dimensional hidden size split into 8 heads of 64 dimensions each — so each head operates in its own smaller subspace, learns its own notion of relevance within that subspace, and the h heads collectively cover more distinct relational patterns than one head operating in the full space could learn to represent simultaneously, precisely because each head is free to specialize without having to also represent every other head's job.

The single most consequential misreading of this design is treating it as a way to cut parameter count or compute. It is not, and the source material is explicit about this being a live exam distractor: [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): question 5 of the domain's self-check asks "Multi-head attention exists primarily to:" and rejects "Reduce the number of parameters" as a wrong answer in favor of "Let the model attend to multiple representation subspaces in parallel." Splitting one 512-dimensional head into 8 heads of 64 dimensions each does not, on its own, reduce the total number of learned parameters in the Wq/Wk/Wv/output projections in any way that would count as an optimization — the point of the split is representational diversity, not a smaller model. Conflating "split into smaller pieces" with "made cheaper" is an easy trap precisely because dimensionality reduction elsewhere in deep learning often is about cost; here, it is about giving each head room to specialize.

L1 — Intuition

Two problems, two very different-feeling fixes. The first: attention, as M1-01 established, decides relevance purely from vector content — the Query-Key comparison has no term anywhere in it that references position. That means a sentence and a scrambled version of the same sentence, with no positional signal added, would attend identically wherever the token content is identical, which is nonsensical for language, where "the dog bit the man" and "the man bit the dog" share every token but mean opposite things. Positional encoding closes this gap by injecting a signal that depends on position directly into the token representations before attention ever runs, so "dog" at position 1 and "dog" at position 4 arrive at the attention mechanism as distinguishable inputs. The second problem is unrelated to order entirely: training a network dozens of layers deep, where each layer's output feeds the next layer's input, tends to produce activations whose scale drifts — growing or shrinking compounding through the stack — unless something actively holds them in a stable range. Layer normalization is that active correction, applied inside every layer, and residual connections are the architectural habit of adding a layer's input back onto its output so that even a layer that has not yet learned anything useful does not actively destroy the signal passing through it.

L2 — Mechanism

Positional encoding, in its original sinusoidal form, computes a fixed vector for each position in the sequence using sine and cosine functions at a range of frequencies, and adds that vector elementwise to the token's embedding before the first attention layer. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "positional information is added to token embeddings (sinusoidal in the original paper; learned or rotary in many modern models)." Because the sinusoidal vectors are deterministic functions of position rather than learned parameters, this scheme requires no additional training and, in principle, generalizes to sequence lengths longer than anything seen in training, since the sine and cosine functions are defined for any position value. A learned positional encoding instead treats each position as an index into a trainable embedding table, exactly like a token embedding table but indexed by position instead of vocabulary id — simpler to implement, but it can only represent positions actually seen during training, since positions beyond the table's size have no learned row. Rotary positional encoding (RoPE), used in many modern decoder-only models, takes a different structural approach: rather than adding a positional vector to the embedding, it rotates the Query and Key vectors by an angle that depends on their position, so that the dot product between a Query at position i and a Key at position j ends up depending on the relative distance (i − j) between them rather than their absolute positions — a property that tends to generalize better to sequence lengths beyond training and that several widely deployed decoder architectures adopt specifically for that reason.

Layer normalization normalizes activations across the feature dimension for each token independently — for a given token's vector at a given layer, it subtracts that vector's own mean and divides by that vector's own standard deviation, then applies a learned scale and shift. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Layer normalization stabilizes training by normalizing activations (mean 0, variance 1), which — combined with residual connections — keeps very deep stacks trainable." The "combined with residual connections" clause is not incidental. A residual connection adds a layer's input directly to its output — output = Layer(x) + x — so that gradients have a direct, unimpeded path backward through the addition, bypassing whatever the layer itself is doing. Without that path, a gradient has to flow back through every nonlinearity and every weight matrix in a dozens-of-layers-deep stack, and at that depth the combined effect of many small multiplicative factors can shrink a gradient toward zero (vanishing) or blow it up (exploding) before it reaches the earliest layers. Layer normalization and residual connections attack the same deep-stack instability problem from two different angles — one keeps each layer's activation scale in a consistent range, the other guarantees a gradient path that does not have to survive the full depth of the network to reach early layers — and transformer architectures use both together rather than relying on either alone.

L3 — Where positional encoding is applied, and the exam-relevant edge case

The subtlety worth holding precisely: positional encoding is added to the embeddings before the first attention layer, not inside the attention formula itself. This matters because it means the Q, K, and V projections M1-01 derived are computed from position-augmented embeddings — the positional signal rides along inside the vectors that Wq, Wk, and Wv then project, rather than being a separate term added anywhere inside softmax(QKᵀ/√dₖ)·V. A common confusion is imagining that attention itself has some built-in positional term; it does not, and the fact that positional information reaches attention only via the input embeddings is exactly why swapping the positional-encoding scheme (sinusoidal versus learned versus rotary) is a change made upstream of attention, not a change to the attention formula's structure. RoPE is a partial exception worth naming precisely, because it is the one scheme in this lesson that does not fit the simple "add before attention" pattern: it modifies the Query and Key vectors directly, after their projections but before the QKᵀ dot product, rotating them by a position-dependent angle rather than adding a vector to the embeddings upstream. This is a genuinely different mechanical placement from sinusoidal or learned encoding, and a question that asks specifically where a positional scheme intervenes in the pipeline is testing whether you have kept this distinction rather than treating "positional encoding" as one interchangeable step everywhere.

Why splitting into heads is (almost) parameter-neutral, worked in dimensions

The parameter-neutrality claim from section 1 is worth making concrete rather than asserted, because "almost neutral" and "clearly cheaper" are different claims and the exam distinguishes them. Take a model with hidden size d_model = 512. A single-head design projects the input into a Query, Key, and Value each of dimension 512, using three weight matrices each sized 512×512 — 3 × 512 × 512 = 786,432 parameters for the Q/K/V projections, plus an output projection of another 512×512 = 262,144, for a total of 1,048,576 parameters in the attention block. Now split into 8 heads of dimension 64 each (8 × 64 = 512, so the total width is preserved). Each head needs its own Wq, Wk, Wv, but each is now sized 512×64 rather than 512×512 — 3 × 512 × 64 = 98,304 parameters per head, times 8 heads = 786,432, identical to the single-head Q/K/V total, because 8 heads of width 64 cover the same total output width as 1 head of width 512. The output projection concatenating all 8 heads' 64-dimensional outputs back into 512 dimensions is again 512×512 = 262,144. The grand total is 1,048,576 — exactly the same as the single-head design. This is the arithmetic behind "multi-head attention is not a parameter-reduction trick": in the standard formulation, splitting into h heads of width d_model/h costs the same total parameters as one head of the full width, because the per-head matrices shrink in exactly the proportion the head count grows. What changes is not the parameter count; it is that 8 independent smaller projections can each specialize on a different relational pattern, where 1 large projection has to represent all of them jointly in one shared space.

Constructed scenario, illustrative only. Take the two three-token sentences "cat bit dog" and "dog bit cat," and suppose, hypothetically, that no positional encoding is added anywhere — token embeddings for "cat," "bit," and "dog" go straight into the Q, K, V projections unmodified.

Sentence 1 tokens (in order): cat, bit, dog
Sentence 2 tokens (in order): dog, bit, cat

Without positional encoding:
  embedding("cat") is identical in both sentences
  embedding("bit") is identical in both sentences
  embedding("dog") is identical in both sentences

  Q, K, V for "cat" = f(embedding("cat"))  -- same function, same input, same output
  ... therefore Q, K, V for every token are identical across both sentences,
      regardless of which position that token occupies in each sentence.

  QK^T comparison for "bit" attending to "cat" vs "dog":
    depends only on embedding("bit") . embedding("cat") and embedding("bit") . embedding("dog")
    -- these two dot products are the SAME numbers in both sentences,
       because nothing about position has entered the computation anywhere.

The attention pattern token "bit" produces over "cat" and "dog" would therefore come out numerically identical whether the input was "cat bit dog" or "dog bit cat" — the model has no way to know which one is the biter and which is the bitten, despite these being opposite claims. Now restore positional encoding: add a distinct positional vector for position 1, position 2, and position 3 to the respective token embeddings before projection.

With positional encoding added:
  embedding("cat") + pos(1)  in sentence 1,  vs  embedding("cat") + pos(3)  in sentence 2
  embedding("dog") + pos(3)  in sentence 1,  vs  embedding("dog") + pos(1)  in sentence 2

  Now Q, K, V for "cat" in sentence 1 differ from Q, K, V for "cat" in sentence 2,
  because a different positional vector was added before the Wq/Wk/Wv projections ran.

The two sentences now produce genuinely different Query, Key, and Value vectors for every token, because each token's representation carries a distinct positional signal alongside its content — which is precisely what lets attention, and everything built on top of it, tell "cat bit dog" apart from "dog bit cat" despite the two sentences sharing every single token.

Constructed scenario, illustrative only — real activation statistics are learned, not this simple. Suppose, hypothetically, that a single transformer sub-layer, unnormalized and without a residual connection, tends to roughly double the variance of its input's activations every time it is applied — a simplified stand-in for how repeated matrix multiplications without correction can drift activation scale layer over layer.

Layer 1 output variance (starting at 1.0):      1.0 * 2   =  2.0
Layer 5 output variance (compounding x2 each):   1.0 * 2^5 = 32.0
Layer 20 output variance (compounding x2 each):  1.0 * 2^20 ~ 1,048,576

With layer normalization re-centering variance to ~1.0 after every layer:
Layer 1 output variance:   ~1.0   (normalized back down after doubling)
Layer 5 output variance:   ~1.0   (same correction applied every layer)
Layer 20 output variance:  ~1.0   (drift never compounds across layers)

This is a deliberately simplified stand-in for a real failure mode, not a measured statistic from any actual model — but the shape of the problem is exactly right: an uncorrected compounding effect across twenty layers reaches a scale six orders of magnitude larger than where it started, at which point activations are far outside the range the rest of the network's weights were tuned to handle, and gradients computed from such extreme values tend to be similarly extreme or, after saturating nonlinearities, vanishingly small. Layer normalization's per-layer correction prevents that compounding from ever accumulating past a single layer's worth of drift, which is the concrete sense in which normalization "stabilizes training" rather than merely sounding like a good idea. Residual connections address a related but distinct piece of the same deep-stack problem: even with activation scale under control, a gradient still has to backpropagate through every layer's full computation to reach early layers, and the direct addition path a residual connection provides is what keeps that backward path from being forced through the full depth of possibly-still-poorly-trained transformations, especially early in training when a layer's weights have not yet learned anything useful.

⭐ THE EARNED INSIGHT Multi-head attention, positional encoding, and layer normalization all look like refinements bolted onto a working mechanism, but each one exists because the base mechanism M1-01 derived has a specific, nameable gap: one shared subspace cannot represent every kind of relationship at once, a purely content-based comparison cannot know word order, and a deep stack of layers cannot stay numerically stable without active correction. None of the three is optional polish — each closes a gap that would otherwise make the mechanism unusable at real scale and real depth.

The description in section 2 said normalization is "applied within each sub-layer" without specifying exactly where relative to the residual addition, and that placement choice turns out to matter enough that it is worth naming as its own edge case. The original transformer design, often called Post-LN, applies layer normalization after the residual addition: output = LayerNorm(x + SubLayer(x)). A later, now more common variant, Pre-LN, applies it before the sub-layer instead: output = x + SubLayer(LayerNorm(x)). The difference sounds cosmetic — same two operations, different order — but it changes how gradients flow through very deep stacks in a way that has real, measured training-stability consequences.

Under Post-LN, the residual path itself passes through a normalization step on every layer, which means the "gradient superhighway" residual connections are supposed to provide gets partially interrupted by normalization at every single layer. In practice, this makes very deep Post-LN stacks harder to train without careful learning-rate warmup, because early training steps can produce large gradients through the un-normalized-yet sub-layer outputs before the network has learned to keep its own activations in a reasonable range. Under Pre-LN, the residual path — x added directly to SubLayer(LayerNorm(x)) — never itself passes through a normalization step; only the input to the sub-layer is normalized, leaving the addition itself as a clean, unimpeded gradient path all the way back through the stack. This is why many modern, very deep transformer architectures default to Pre-LN: it trades a small amount of representational strictness for meaningfully more stable gradients at depth, which matters more as layer count grows into the dozens.

This distinction is worth holding precisely rather than folding into "layer normalization is used" as one undifferentiated fact, because a scenario question about training instability at very high layer counts is testing exactly this: whether normalization sits on the residual path itself (Post-LN, more prone to instability at depth) or off to the side of it (Pre-LN, the more common modern default for deep stacks). Neither placement changes what layer normalization computes — mean-zero, variance-one activations plus a learned scale and shift, exactly as section 2 described — only where in the layer's data flow that computation happens relative to the residual addition.

[GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): these three mechanisms are grouped inside "(a) The Transformer & Self-Attention," the same objective-1.1 subsection scaled dot-product attention belongs to, and the domain's own self-check question 5 tests multi-head attention's purpose directly: "Multi-head attention exists primarily to: ... Let the model attend to multiple representation subspaces in parallel," with "Reduce the number of parameters" offered as the wrong answer. Because Domain 1 carries only 6% of the exam but is explicitly framed as "the conceptual base for everything else," expect this material to resurface implicitly in later domains rather than only in Domain 1 items — a GPU Acceleration question about parallelism, for instance, assumes you already know multi-head attention is itself a parallel operation across heads, distinct from parallelism across GPUs.

Expect a purpose-identification item: "why does a transformer use multiple attention heads?" with the parameter-reduction misreading as the keyed wrong answer and "multiple representation subspaces" as correct. Expect a necessity item about positional encoding: a question states that attention alone is order-agnostic and asks what mechanism resolves that, or conversely asks why positional encoding is needed at all given that attention already looks at every token — testing whether you understand that "looking at every token" and "knowing which token came first" are different capabilities. Expect a stabilization item about layer normalization and residual connections, phrased around what breaks in a very deep network without them, or which specific statistic layer normalization operates on (activations, not gradients or weights directly, though it affects gradients as a downstream consequence).

What the distractors typically look like

The standard traps in this lesson's style are: describing multi-head attention as a way to cut parameter count or compute cost, when its purpose is representational, not economic; describing positional encoding as optional or as something attention "already handles" through its comparison mechanism, when attention's comparison is provably content-only; and describing layer normalization as something that normalizes gradients or weights directly, when it normalizes the activations flowing through the network, with stabilized gradients as a downstream effect rather than the thing being normalized.

Attention's comparison step is a dot product between a Query and a Key, both of which are learned functions purely of token content — nothing in that computation takes a token's index in the sequence as an input, so there is no learnable pathway by which "position" could enter the comparison unless something upstream puts positional information into the vectors being compared in the first place. A model could, in principle, learn to exploit incidental correlations between content and position in its training data, but that is fragile and task-specific, not a general mechanism — which is exactly why every real transformer architecture adds an explicit, deliberate positional signal rather than hoping the network stumbles onto one.

No — they normalize across different dimensions and are used in different settings for a specific reason relevant here. Batch normalization normalizes each feature across all examples in a training batch, which makes its statistics depend on batch composition and behave inconsistently for variable-length sequences and for generation, where you may process one token at a time. Layer normalization instead normalizes across the feature dimension for a single token's own vector, independent of what else is in the batch or how long the sequence is, which is exactly the property that makes it the standard choice for transformer architectures processing variable-length sequences, including one-token-at-a-time autoregressive decoding.

Not without limit. Because splitting a fixed hidden size into more heads shrinks each head's own dimension (a 512-dimensional hidden size split into 16 heads gives each head only 32 dimensions, versus 64 dimensions at 8 heads), pushing the head count arbitrarily high eventually gives each head too little room to represent a useful comparison at all — a head of dimension 4 or 8 has very little capacity to learn a meaningful Query-Key relationship, no matter how many such narrow heads you add. Real architectures pick a head count that balances "more subspaces to specialize in" against "enough dimensionality per head for each specialization to be meaningful," and that balance is an empirical architecture choice, not something with one universally correct answer — which is also why different published models report different head counts and per-head widths despite solving similar tasks.

• Multi-head attention exists for representational diversity, not parameter savings — each head specializes in a different subspace; splitting into heads is not a compression trick.
• Positional encoding is necessary because attention's Query-Key comparison is purely content-based — two sentences with identical tokens in different orders would attend identically without it.
• Sinusoidal, learned, and rotary positional encoding differ in where and how they intervene: the first two add a signal to embeddings before attention; RoPE rotates Q and K after projection, closer to the attention formula itself.
• Layer normalization stabilizes activation scale within each layer; residual connections give gradients a direct backward path — together, they are what keeps very deep transformer stacks trainable.
• These mechanisms operate at different points in the pipeline: positional encoding upstream of (or within) attention, multi-head attention as the comparison-and-combination step itself, normalization and residuals wrapping the whole sub-layer.
• The domain's own self-check tests the multi-head misreading directly — "reduce parameters" is the keyed wrong answer against "attend to multiple subspaces."

This lesson finished the single-layer mechanics: what happens inside one transformer layer, given multiple heads, a positional signal, and normalization holding the stack stable. It said nothing yet about how these layers get arranged into a whole model, or what training objective drives that arrangement. M1-03 picks that up directly: whether a model's layers only ever see the whole input at once (encoder-only, bidirectional, trained with masked language modeling) or only ever see what came before the current token (decoder-only, causal, trained with causal language modeling), or some combination of both (encoder-decoder) — and why matching the wrong architecture to the wrong training objective is one of this exam's most reliable traps.