Positional encoding: how transformers represent word order

Positional encoding is a vector, one per position index, that carries information about where a token sits rather than what the token is. In the canonical design it is added element-wise to the token embedding, producing a single vector that encodes identity and position together. That combined vector is what enters the first attention layer.

Three points define the concept precisely, and each is a place candidates lose marks:

• It is combined with the token embedding, not substituted for it. The vector entering attention is "this word" plus "this slot." Nothing is thrown away. An exam option claiming positional encoding "replaces the embedding" or "stores the word's meaning" is wrong on both counts.
• It is per-position, not per-word. The encoding for index 3 is the same whether index 3 holds "cat" or "quantum." Position and identity are orthogonal signals that the model learns to disentangle.
• It is applied to the input stream in the classical design, but not in every design. Relative and rotary schemes inject position inside the attention computation instead of into the input vectors. The function is constant across all of them; the insertion point is not.

The reason this exists at all is worth stating in one sentence you can reproduce under time pressure: self-attention is permutation-invariant, so word order has to be encoded explicitly or it is invisible to the model.

L1 — Why an order-blind layer needs an order signal

Recurrence never had this problem. In an RNN, order is the computation: the model literally processes token 1, then token 2, and the hidden state's history encodes the sequence. Convolution never had it either — a kernel's weights are indexed by offset, so position is baked into the operator.

Self-attention threw both away in exchange for parallelism. Every position is computed simultaneously from a set of vectors, with weights from pairwise dot products. Shuffle the inputs and every score matrix entry moves with its operands; the values are unchanged. The layer sees a bag.

So the transformer does something that looks like a hack and is in fact principled: it converts sequence modelling into set modelling, then reintroduces order as content. Position stops being a property of the computation and becomes a feature in the vector. The model then learns, from data, how to use that feature — for adjacency, for agreement, for "the instruction came first."

L2 — Fixed sinusoidal encoding: a computed pattern

The original transformer used a deterministic function of position index and vector dimension, built from sine and cosine waves at geometrically spaced frequencies. Low dimensions oscillate quickly, high dimensions slowly. The result is that each position gets a distinctive multi-scale fingerprint: fast components distinguish nearby positions sharply, slow components distinguish far-apart regions coarsely.

You do not need the formula for this exam. You need the four properties it buys:

1. No parameters. It is computed, not learned, so it costs nothing in weight count and needs no training data to become useful.
2. Defined at every index. Because it is a function, you can evaluate it at position 10,000 even if training never went past 512. Whether the model behaves sensibly there is a separate question — see §5 — but the encoding itself exists.
3. Smoothness. Nearby positions get similar encodings, which gives the model a usable notion of "close."
4. A structural relationship between offsets. The construction has the property that the encoding at one position relates to the encoding at another in a way that depends on their separation, which is the mathematical hint that made relative schemes an obvious next step.

L2 — Learned absolute positional embeddings: a lookup table

The pragmatic alternative is to treat position exactly like vocabulary: build a table with one trainable vector per position index, look up the row for each slot, add it to the token embedding, and let gradient descent decide what each row should contain.

This is simple, it works well, and it has one hard consequence that shows up as a real-world error: the table has a last row. If it was built with 512 rows, position 512 has no vector. There is nothing to look up. The model cannot process a longer input without changing the architecture, not because of memory, but because the representation for that slot does not exist. That is a different failure from running out of GPU memory, and 04-06 insists on the distinction.

A second consequence is subtler: learned position vectors are only as good as the position statistics of the training corpus. If almost every training example was short, the high-index rows are undertrained even if they exist.

L2 — Relative and rotary schemes: encoding the offset

Language mostly cares about distance, not absolute index. "The adjective before the noun" is a relative fact. Whether the pair sits at positions 4 and 5 or 404 and 405 rarely changes the grammar. Absolute encodings force the model to derive relative facts from pairs of absolute ones.

Relative position methods therefore encode the offset between the query position and the key position, typically injecting it into the attention score computation rather than into the input embedding. Rotary position embedding (RoPE) is the widely known member of this family: rather than adding a position vector, it applies a position-dependent rotation to the query and key vectors, with the effect that the dot product between a query and a key ends up depending on their relative offset. Other approaches add learned biases to attention scores as a function of distance.

The important framing, and the one worth carrying into the exam: absolute schemes tell the model where each token is; relative schemes tell the model how far apart two tokens are. Both are legitimate answers to "how does a transformer represent order."

L3 — Where the signal enters, and why that matters

Two implications fall out. First, input-side schemes rely on the residual stream to carry position through the depth of the network, which works but is indirect. Score-side schemes reassert position at each layer. Second — and this is the practical one — which scheme a model uses is a per-model, per-version fact. Model families change position schemes between releases. Do not assert that a named model uses a specific scheme from memory; the version-sensitivity is real and the exam does not require the claim.

L3 — What positional encoding does not do

It does not make the model attend to nearby tokens. It does not enforce syntax. It does not guarantee that a token late in a long prompt is used. It supplies a feature the model may learn to exploit; everything about how well it is exploited is empirical and depends on the training data, the depth, and the objective. The failure mode where a fact sitting comfortably inside the context window is nonetheless ignored is not a positional-encoding bug — it is the lost-in-the-middle behaviour 07-08 treats, and it happens in models whose position representation is perfectly well defined.

Read the third row as the load-bearing one. "Is a representation defined at all beyond the trained length" is a categorical difference, not a quality difference, and it is the reason people confuse two unrelated causes of a context limit.

And a comparison the exam cares about at least as much — positional encoding against the thing it is most often confused with:

That final row is the cleanest one-line contrast available, and it is exactly the discrimination a well-written multiple-choice question tests.

Take two sequences with identical token multisets and opposite meanings.

A:  the   dog   bit   the   man
B:  the   man   bit   the   dog

Step 1 — embed. Token embeddings are looked up by vocabulary id. Both sequences contain exactly the same five rows: the, dog, bit, the, man. The multiset of embedding vectors is identical between A and B.

Step 2 — imagine no positional encoding. Attention scores are dot products between projected token vectors. In A, the pair (dog, bit) produces some score; in B, the same two vectors produce the same score, because the dot product does not know that dog moved from index 1 to index 4. Every entry of A's score matrix appears somewhere in B's, just relocated. Softmax and the value-weighted sum then produce the same output multiset. The model cannot represent that A and B differ. Whatever it predicts for A, it predicts for B.

Step 3 — add positional encoding. Now the vector at index 1 in A is embed(dog) + pos(1) while in B index 1 holds embed(man) + pos(1). Index 4 in A holds embed(man) + pos(4). Since pos(1) ≠ pos(4), the vectors differ, so the projected queries and keys differ, so the score matrix differs, so the outputs differ. The model now can distinguish subject from object — and, given training data, learns to.

Step 4 — a constructed illustration of the two representations mixing. Suppose, purely for illustration and not as a measured configuration of any model, a hidden size of 8. A token embedding row and a position row might look like this:

embed("dog")   = [ 0.31  -0.12   0.88   0.05  -0.44   0.19   0.60  -0.27 ]
pos(1)         = [ 0.84   0.54   0.09   1.00   0.00   1.00   0.00   1.00 ]
sum (enters attention)
               = [ 1.15   0.42   0.97   1.05  -0.44   1.19   0.60   0.73 ]

These numbers are invented to make the shape of the operation visible; they are not from any real model. What they show is the thing to internalise: one vector, two kinds of information, added. The model has capacity to keep them separable because the position pattern is systematic across the sequence while the token pattern is not.

Step 5 — the exam-shaped conclusion. If asked why transformers need positional encoding, the answer is not "because word order matters in language" — that is true but does not name the mechanism. The answer is "because self-attention is permutation-invariant, so without an explicit position signal the model receives an unordered set." Name the invariance and you have named the reason.

The decision rule: positional encoding explains failures at or beyond the length boundary, and nothing else. Failures inside the window are about attention, prompting, retrieval, or sampling. Keeping that line clean is worth more on the exam than any detail of the sinusoidal construction, because the distractors deliberately blur it.

Transformer architecture is in the highest-frequency reported topic tier for NCA-GENL, and the blueprint's own suggested-reading list points at the original transformer paper — which means the canonical component list is fair game. Positional encoding sits in that list between embeddings and self-attention: tokens → embeddings → positional encoding → self-attention → feed-forward → residual and layer norm. It serves objective 1.7, reading research papers to identify emerging LLM trends and technologies, and supports objective 1.3 on building LLM use cases, since context-length limits are a use-case design constraint.

Candidate reports consistently say the exam is pitched at general level and that deep attention mathematics was overkill. That calibration applies here with unusual force: there is essentially no scenario in which you need the sinusoid formula, and a very high chance you need the one-sentence reason positional encoding exists.

Question phrasings to expect:

• "Why do transformer models require positional encoding?" → self-attention is permutation-invariant; order would otherwise be invisible.
• "Which component of a transformer supplies information about token order?" → positional encoding, added to the token embeddings.
• "What is the difference between a token embedding and a positional encoding?" → identity versus location; one is indexed by vocabulary, the other by position.
• "Which positional scheme is defined for positions beyond those seen during training?" → a fixed, computed scheme such as sinusoidal, as opposed to a learned lookup table.
• "A model built with learned absolute positional embeddings for 512 positions is given 900 tokens. What is the immediate architectural problem?" → there is no positional representation for indices past the table's maximum.
• "In the canonical transformer, where in the pipeline does positional information enter?" → after token embedding, before the first self-attention layer.

Distractor families, and why each is wrong:

Because its output at each position is a weighted sum over the whole sequence, and neither the weights nor the summands reference an index.

Trace it. The score between position i and position j is a dot product of a projection of the vector at i with a projection of the vector at j. If you swap the contents of positions 2 and 5, then the score that used to be at row 2 column 5 now appears at row 5 column 2 — same number, different cell. Softmax normalises rows, which are also permuted consistently. The final weighted sum over values is over a set, and addition is commutative. So the whole layer commutes with permutation: permute the input and you get the permuted output, never a different output.

This is not a defect that a better attention design would fix. It is the price of the architecture's central benefit. Recurrence gets order free because it is serial; attention gets parallelism because it is not. If you want both, you must supply order as data. Stated that way, positional encoding stops looking like a patch and starts looking like the necessary second half of the design.

Sometimes — and only for one of the three families, which is exactly why this question is a good discriminator.

There are at least three independent reasons a model has a maximum usable input length, and conflating them is a top-tier confusion:

1. Representational. With learned absolute positional embeddings, the table has a finite number of rows. Beyond the last row there is no vector to look up. This is a hard architectural boundary.
2. Distributional. With any scheme, positions far beyond what training exercised are undertrained territory. A sinusoidal encoding is defined at position 20,000, but a model that never saw sequences that long has not learned to use it well. Definition is not competence.
3. Resource. Attention's n² score matrix and the per-token key/value memory from 04-01 mean that even a model that can represent a long input may not have the memory or latency budget to process one. This is the cost side, and it is the reason long context is expensive rather than impossible.

Vendors extend context windows by working on all three: changing or rescaling the position scheme, continuing training on longer sequences, and using memory-efficient attention implementations. The associate-level position on this is deliberately cautious: context-window figures and position schemes are version-sensitive configuration facts. Quote the number for the specific model version you are using, check it in the current documentation, and never carry a remembered figure across a model release. 04-06 builds the budget arithmetic on top of this warning.

Absolute encoding answers "which slot is this token in." Relative encoding answers "how far is this token from the one attending to it."

The practical difference shows up in generalisation. Consider the pattern "an adjective immediately precedes its noun." Under absolute encoding, the model sees that pattern at (1,2), at (17,18), at (403,404) — and has to learn from the co-occurrence of absolute pairs that what matters is the difference. Under relative encoding, the offset −1 is a single feature that appears identically in all three cases. The pattern is representable once instead of many times.

A second difference is where the signal enters, covered in §2's table: absolute schemes usually modify the input embeddings; relative and rotary schemes usually modify the attention computation, which means position is reasserted at every layer rather than carried forward through the residual stream.

For the exam, hold the contrast at this level: absolute = index, injected at the input; relative/rotary = offset, injected in attention; relative matches what language cares about, and rotary is the best-known modern instance. Do not attach specific schemes to specific named models from memory.

• Positional encoding — a position-dependent signal combined with token representations so an order-blind attention mechanism can use word order.
• Permutation invariance — the property of self-attention that reordering inputs reorders outputs identically without changing them; the reason positional encoding is required.
• Sinusoidal (fixed) positional encoding — a computed, parameter-free pattern of sine and cosine components at geometrically spaced frequencies; defined at any index.
• Learned absolute positional embedding — a trainable lookup table with one vector per position index; adapts to the corpus but has a hard maximum index.
• Relative positional encoding — encodes the offset between query and key positions rather than absolute indices, typically inside the attention computation.
• Rotary position embedding (RoPE) — a widely used relative scheme that applies a position-dependent rotation to queries and keys so their dot product depends on their separation.
• Maximum position index — the last representable slot in a learned scheme; an architectural, not a memory, limit.
• Representational vs distributional vs resource limit — the three independent reasons a model has a maximum usable input length: no vector exists, no training exercised it, or no memory/time budget covers it.

1. One sentence carries the whole lesson: self-attention is permutation-invariant, so word order must be supplied explicitly as a position signal or it is invisible.
2. Position and identity are separate signals combined into one vector. The token embedding says what; the positional encoding says where. Neither replaces the other.
3. Three families. Fixed sinusoidal (computed, unbounded definition, no parameters), learned absolute (trainable, adapts, hard maximum index), relative/rotary (encodes offset, injected inside attention, matches how language uses distance).
4. Only one family imposes a hard length boundary by itself. A learned table's last row is a categorical limit; a computed function has no such boundary. That distinction is a favourite discriminator.
5. Defined is not the same as competent. A scheme evaluable at position 20,000 tells you nothing about whether the model learned to use position 20,000.
6. Three separate reasons a context window ends — representational, distributional, resource. Keep them apart; 04-06 depends on it.
7. Position schemes and context-window sizes are version-sensitive facts. Quote them from current documentation for the exact model version, never from memory.
8. Position does not explain inside-window failures. Mid-prompt neglect, sampling variation, and retrieval ordering all have other owners.

You now have the two halves of the transformer's core mechanism: attention that reads everything at once, and a position signal that tells it what order everything arrived in. What you do not yet have is the fork in the road. The single mask introduced at the end of 04-01 — future positions set to negative infinity — splits the transformer into two families with completely different jobs, and a third family that uses both. That split is why you embed text with BERT and generate text with GPT, why T5 and BART own translation and summarisation, and why choosing the wrong one produces a system that cannot possibly work no matter how well you prompt it.

Next: 04-03 maps encoder-only, decoder-only, and encoder-decoder architectures to the tasks each one is built for — the architecture-to-task matching that the NCA-GENL blueprint tests directly.