Next-Token Prediction: What a Language Model Is Trained to Do

Identity statement: next-token prediction — also called causal language modelling or autoregressive language modelling — is the training objective in which a model reads a prefix of text and outputs a probability for every token in its vocabulary being the next one.

When it applies: it is the objective behind every decoder-only generative LLM: the GPT family, Llama, Mistral, and the models you will meet in the NVIDIA stack. It is not the objective behind encoder-only models like BERT, which are trained on masked language modelling instead. That contrast is the entire basis of architecture-to-task matching later in the course.

Four properties matter for the exam.

The label is free. The correct answer for any position is the token that literally appears next in the source text. No annotator wrote it. This is why LLM pretraining is called self-supervised rather than unsupervised — there is a supervision signal, it is just harvested from the data's own structure instead of added by a human. A distractor that calls pretraining "unsupervised" is testing exactly this word.

One document is thousands of training examples. A 1,000-token document does not yield one training example; it yields on the order of 1,000, one per position, because every prefix is a valid input and the token following it is a valid target. This is the single reason the objective scales to internet-sized corpora — supervision multiplies for free with the text.

Prediction is a distribution, not a word. The model's raw output at each position is a vector of scores across the full vocabulary. Turning that distribution into one actual token is a separate, configurable step called decoding, and it is where temperature, top-k and top-p live.

The objective is fixed; behaviour is not. Once pretraining is done, the objective is spent. Everything afterwards — instruction tuning, RLHF, prompting, retrieval, guardrails — either re-fits the same next-token machinery on different text or changes what text the machinery is conditioned on. Nothing later in the course replaces the objective; it all sits on top of it.

L1 — The fill-in-the-blank intuition

Take any sentence, hide everything from some point onward, and ask the model to guess the next piece. Score the guess. Nudge the model to be slightly less wrong. Do that across an enormous quantity of text. The model that survives has internalised whatever regularities of language, fact and form make the next token predictable.

If you stop at L1 you can still answer most exam items on this topic. The identity statement plus "self-supervised, label comes from the text itself" plus "generation is a loop" is the whole tested surface.

L2 — The generation loop

At inference, generation is a loop, not a single call:

context = tokenize(prompt)
repeat:
    logits     = model(context)      # scores over the whole vocabulary
    probs      = softmax(logits)     # a probability distribution
    next_token = decode(probs)       # greedy / top-k / top-p / temperature
    context    = context + [next_token]
until next_token == <end-of-sequence> or max_tokens reached

Two consequences fall straight out of that loop. First, the model's own output becomes its next input, so an early mistake is conditioned on for the rest of the generation — the mechanical origin of a response that drifts or confabulates. Second, output length costs real time: each token requires another forward pass, which is why inter-token latency and throughput are tracked as separate metrics later in the course.

A third consequence is easy to miss and shows up in scenario questions: the loop terminates on either an end-of-sequence token or a max_tokens ceiling, whichever comes first. A response that stops mid-sentence is usually a max_tokens cut, not a model failure. A response that will not stop rambling is usually a model that keeps assigning low probability to the end-of-sequence token — which instruction tuning is partly there to fix.

L3 — Why training is parallel and generation is sequential

During training the true continuation is already known, so every position in a sequence can be predicted in parallel in one forward pass, each position masked so it cannot see the future. (Conditioning each position on the true prefix rather than on the model's own earlier guesses is conventionally called teacher forcing.) During generation there is no true continuation to condition on, so tokens must be produced strictly one at a time.

Same objective, opposite performance profile: training is compute-bound and parallel, decoding is sequential and memory-bandwidth-bound. That asymmetry is why an entire product layer exists purely to make decoding cheaper — KV caching, continuous batching, speculative decoding — which is what you meet when the course reaches inference optimisation.

The causal mask is worth naming because it is the mechanism that keeps the objective honest. Without it, position 7 could see the token at position 8 and predicting position 8 would be trivial — the model would learn nothing and would collapse the instant you asked it to generate, because at generation time position 8 does not exist yet. "Causal" and "autoregressive" are both pointing at this constraint: the prediction at each step may depend only on the past.

This lesson stops at L3 on purpose. You are not asked to implement attention masking, write the training loop, or derive anything. The course's calibration notes are explicit that deep attention math was reported as overkill and did not appear; the payoff is in the identity statements and the decision rules, not the derivations.

Prompt: The capital of France is

Suppose the vocabulary holds 50,000 tokens. The model's output at this position is 50,000 scores. After softmax, the top of the distribution might look like this. Treat the numbers as an illustrative construction, not a measurement from a named model:

Note what the model has not done: it has not looked anything up. It produced a distribution in which one token dominates because that continuation was overwhelmingly common in the text it was fitted to. Decoding then chooses:

• Greedy decoding takes Paris every time — fully deterministic.
• Temperature 1.5 flattens the distribution, raising the odds of the or located — more diverse and, here, more likely to be wrong.
• Top-k = 2 restricts the candidate set to { Paris, the} and renormalises, so located becomes impossible regardless of temperature.
• Top-p = 0.9 keeps the smallest set of tokens whose probabilities sum to at least 0.9 — here just { Paris, the}, because 0.89 + 0.04 = 0.93 clears the threshold while 0.89 alone does not.

Then the loop runs again with The capital of France is Paris as the new context and predicts what follows that. Nine tokens of output means nine forward passes and nine distributions. At no point did the process ask "is this true?" — which is precisely why grounding and citation exist as separate engineering controls rather than as properties of the model.

One more reading of the same table, because it is the reading the exam rewards. The 0.89 is not the model's confidence that Paris is the capital of France. It is the model's estimate of how often the token Paris follows this exact prefix in text like its training text. Those two quantities happen to coincide here. When they come apart — a prefix where the fluent continuation and the true continuation differ — the model still returns the fluent one, and you have a hallucination. Nothing broke. The objective did exactly what it was fitted to do.

This is the arithmetic that makes the scaling story concrete. Take a single short document, tokenised into eight tokens:

tokens:   [The] [cat] [sat] [on] [the] [mat] [and] [slept]
positions:  1     2     3     4     5     6     7      8

Training on this one document creates seven prediction problems, not one:

Seven targets from eight tokens — in general, n − 1 from n, which for any realistic document length is "about one per token." And because of the causal mask, all seven are scored in a single forward pass, not seven.

Now scale it with explicitly illustrative numbers. Suppose a corpus of 1 trillion tokens. The objective yields on the order of 1 trillion supervised prediction problems, and the human labelling cost is zero. Compare that to a supervised classification dataset, where every one of those examples would need a person to write the label. That cost gap — not any cleverness in the objective itself — is why the self-supervised route is the only one that reaches this scale, and it is why the blueprint's foundation-model topics talk about pretraining on unlabelled corpora as a defining property rather than an implementation detail.

The same arithmetic explains a practical fact you will need in 01-07: because examples are positions rather than documents, a duplicated document does not add one duplicate example, it adds thousands. Deduplication at corpus scale is therefore not housekeeping — it materially changes what the model over-fits to.

Objective 1.5 asks for familiarity with the fundamentals of machine learning, and paradigm vocabulary is exactly the one-line distinction a multiple-choice item likes to test.

Within self-supervision, the two objectives split by architecture:

If an item describes a model that must produce fluent continuations, the answer path runs through next-token prediction and a decoder. If it describes labelling or scoring text that already exists, it does not. If it describes transforming one text into another with both fully available, encoder-decoder is in play.

These five terms are used loosely in industry writing and precisely on exams. Confusing any two of them is the most reliable way to lose a point on this topic.

Two rules follow, and they are worth memorising as sentences:

Changing temperature, top-k or top-p changes behaviour, never knowledge. The weights are untouched.

Changing weights — fine-tuning, LoRA, full retraining — changes knowledge and style, and does so before any prompt is ever sent.

That distinction is the spine of the customisation-ladder question type, where the exam gives you a symptom and asks which lever to pull. If the model does not know a fact, no decoding parameter will help. If the model knows the fact but keeps phrasing the answer three different ways across runs, no amount of fine-tuning is the cheapest fix — you turn the temperature down.

Core Machine Learning and AI Knowledge is the largest domain on the official blueprint at 30% — roughly 18 of 60 questions. Next-token prediction sits under objective 1.5 and is the mechanism assumed by objective 1.7 (reading research papers to identify emerging LLM trends), because "autoregressive model" is one of the terms the official study guide's suggested-reading list expects you to already own.

It is rarely the subject of a question. It is what makes other questions answerable:

• Text-generation parameters — a reported high-frequency topic — only make sense as ways of sampling from a next-token distribution.
• Transformer architecture — also high-frequency — is the machinery that computes that distribution.
• Hallucination items reduce to the fact that a model optimised for plausible continuations has no built-in mechanism for checking truth.
• Encoder vs decoder items start from causal versus masked objectives and end in architecture-to-task matching.
• RAG items are, underneath, about changing the prefix the distribution is conditioned on rather than changing the model.

How the question tends to be phrased

Expect one of four shapes:

1. Definitional. "Which of the following best describes the pretraining objective of a decoder-only large language model?" The keyed answer names predicting the next token from preceding context; distractors offer masked-token reconstruction, sentence-order prediction, or a supervised classification framing.
2. Paradigm labelling. "LLM pretraining on a large unlabelled text corpus is an example of which type of learning?" The keyed answer is self-supervised; the trap is unsupervised.
3. Consequence reasoning. A short scenario describes output that starts well and then drifts into invention, and asks why. The keyed answer traces it to conditioning on its own previous tokens with no verification step.
4. Level confusion. "An engineer wants more varied outputs from a deployed model. What should change?" The keyed answer is a decoding parameter, not retraining.

What the distractors typically look like

The reliable distractor families here are: unsupervised offered where self-supervised is correct; masked language modelling offered as the decoder objective; the model retrieves the answer from a stored database offered as a mechanism; and the model generates the complete response, then scores it offered as the generation process. Each of those is a real technique that exists somewhere in the stack, which is what makes them plausible — masked LM is real, it just belongs to BERT; retrieval is real, it just belongs to RAG and is bolted on.

Published candidate reports converge on the exam being general-level rather than deep-technical: know at a high level what each thing is and when to use it. Treat that as calibration, not as an official statement — NVIDIA publishes no item-level detail, no official passing score, and the official page is itself ambiguous about whether you face 50 or 60 questions. Practically, for this lesson: own the identity statement, the self-supervised label, and the causal-versus-masked split cold. You do not need the loss arithmetic here; that arrives in 01-05.

Six named errors, each with the symptom you would actually observe and the correction.

A seventh, subtler error deserves its own paragraph because it recurs across the whole course: treating a high probability as a truth claim. The number attached to a token is a statement about textual likelihood. Reading it as calibrated confidence in a fact is how teams end up trusting a fluent, wrong answer. Every mitigation in the hallucination lesson exists because that gap is structural rather than a bug to be patched.

A decision rule you can apply to scenario questions:

The point of the right-hand column is that "it predicts the next token" is a true statement that stops being useful at some point. Knowing where it stops is as valuable as knowing the statement.

For exam purposes: the question is not asked, and no option phrased as a philosophical position will be keyed. What is asked is the operational version — can the model verify what it produces? — and the answer is no. It optimises the likelihood of a continuation. There is no separate module that checks a claim against a source unless you build one, which is what retrieval, citation and human review are for.

The productive way to hold this is behavioural. The model has demonstrably internalised a great deal of structure: syntax, idiom, code conventions, common factual associations, and enough task pattern to follow instructions after tuning. It has not internalised a mechanism for distinguishing "this is the continuation that appears in text like my training data" from "this is true." Treat capability claims as empirical questions to be measured with an evaluation set — which is exactly what 01-08 teaches you to build.

Three reasons, all of which fall directly out of the objective.

First, the objective rewards plausibility, not truth. A fluent, well-formed, wrong continuation and a fluent, well-formed, right continuation are both good under the training signal if both look like the corpus. Where the corpus is thin — an obscure person, a niche API, a recent event after the data cut-off — the most likely-looking continuation is often invented, because there was never enough signal to pin the true one down.

Second, generation is autoregressive, so errors compound. Once an invented name is in the context, every subsequent token is conditioned on it, and the model will happily elaborate consistently on a false premise. The output reads as confident because internal consistency is exactly what the objective optimises.

Third, the training text was not all correct. Web-scale corpora contain contradictions, outdated facts, and fiction. The model fits the distribution it was shown, contradictions included.

Which is why the mitigations are all external: ground the prefix in retrieved source text, require citations you can check, constrain decoding, add guardrails, keep a human in the loop for high-stakes output. None of them changes the objective. They change what the objective is conditioned on, or what happens to its output afterwards.

Not by itself, and the distinction is worth being crisp about because the customisation ladder depends on it.

Pretraining on raw text gives you a model that continues documents. Ask a purely pretrained model a question and a perfectly reasonable behaviour is to continue with more questions, because that is what a list of questions looks like in a document. The chat and instruction-following behaviour you are used to comes from later stages: supervised fine-tuning on instruction–response pairs, then preference-based alignment.

The key insight is that those later stages use the same next-token machinery. Instruction tuning is next-token prediction on curated instruction–response text; the objective is unchanged, only the data is. Preference optimisation changes the training signal itself, which is why it gets its own treatment in the alignment lesson. So the correct summary is: next-token prediction supplies the capability; the alignment stack selects which of those capabilities the model actually exhibits.

• An LLM's training objective is one thing: a probability distribution over the next token, given the prefix.
• It is self-supervised — the label is the next token already present in the corpus — which is why it scales to unlabelled text. "Unsupervised" is the wrong word and a common distractor.
• A document of n tokens yields on the order of n training examples, one per position. Supervision multiplies with the data at no extra labelling cost.
• Training predicts all positions in parallel; generation produces tokens one at a time. That asymmetry drives every later inference-optimisation topic.
• Causal LM (decoder, left context only) vs masked LM (encoder, both sides) is the split that decides which architecture fits which task.
• Decoding parameters act on the distribution after training. They change behaviour, never knowledge.
• The objective optimises plausibility, which is the structural reason hallucination is a design constraint rather than a defect.
• Instruction following and chat behaviour come from later stages built on the same objective, not from a different one.

You now know what the model is optimised to do. You do not yet know where the result of that optimisation is kept. Everything learned from predicting next tokens ends up in one place — the model's parameters — and the difference between changing those numbers and changing the text you send in is the fault line running through prompting, RAG, fine-tuning and every customisation question on the exam.

Next: 01-02 LLM parameters: what they are and where knowledge is stored — including the arithmetic that turns a parameter count into a GPU memory requirement, using the units you set up in M0.4.