LLM Parameters: What They Are and Where Knowledge Is Stored

Identity statement: a parameter is a single number whose value was learned by training — adjusted by gradient descent to reduce prediction error. "A 7-billion-parameter model" means 7 billion such numbers.

When the term matters: whenever you need to answer "will this fit on my GPU?", "does this technique update the model?", or "why does the model still not know our internal product names?"

Parameters are not scattered randomly. They live in named structures:

The distinctive number for this page: in the standard transformer block where the feed-forward inner dimension is 4× the hidden size, the four attention projections contribute 4·d² parameters and the two feed-forward matrices contribute 8·d², so the feed-forward layers hold roughly two thirds of a transformer block's parameters and attention holds one third. Attention gets the attention; the MLP holds the bulk of the storage. You will not be asked to derive that, but it usefully corrects the common mental picture that "the knowledge is in the attention."

Weights, biases, and what "trainable" means

Two vocabulary items get used interchangeably and should not be. A weight is a multiplicative parameter — an entry in a matrix that scales an input. A bias is an additive parameter — a per-output offset. "Parameters" is the umbrella term covering both, and it is what a model's advertised size counts. Many modern LLM architectures drop most biases entirely for efficiency, which is why you will see "weights" used as a loose synonym for "parameters" in practice.

Trainable is the other word to pin down. A parameter is trainable if the current training run is allowed to update it. The same number can be trainable in one run and frozen in the next — that is precisely the mechanism behind parameter-efficient fine-tuning, where the base checkpoint is marked frozen and only a small added set is trainable. So "the model has 7B parameters and we fine-tuned 0.1% of them" is a coherent sentence: 7B is the storage, 0.1% is the trainable subset for that run.

L1 — Weights as compressed regularities

Training exposes the model to text and adjusts parameters so predictions improve (01-01). Regularities that repeat across the corpus — that Paris follows "the capital of France is," that Python functions start with def, that a polite reply follows a polite question — get baked into weight values. The model is not a database with rows; it is a lossy compression of statistical structure in its training data. That is why it recalls widely repeated facts reliably and rare ones badly, and why it cannot cite where a fact came from.

If you take one image away, take this one: the weights are a lossy compression of the training corpus, not an index into it. Lossy explains the failure modes. Compression explains why a 14 GB file can discuss almost any topic. Not an index explains why it cannot tell you its source.

L2 — Frozen at inference

Once training ends, parameters are frozen. Serving a model is a read-only operation over those numbers. Nothing you send at inference writes back. Consequences the exam cares about:

• The knowledge cutoff is a property of the weights. No amount of prompting adds post-training knowledge; you must supply it in the context, which is precisely what RAG does.
• Conversation "memory" is re-sent text. A chatbot that appears to remember earlier turns is being handed those turns again inside its context window on every request.
• Two identical requests to a frozen model differ only through decoding randomness, not through the model having changed.
• The same checkpoint on two different GPUs is the same model. Behaviour differences at that point come from precision, kernel implementation or decoding settings, not from knowledge.

Frozen also has an operational meaning worth carrying: because the file does not change, a checkpoint is a versionable artefact. You can hash it, store it, roll back to it, and state exactly which weights produced a logged output. That property is what makes model versioning and reproducibility possible at all, and it is why the evaluation discipline in 01-08 insists on recording which checkpoint produced a score.

L3 — Precision, and the memory arithmetic

Each parameter is stored in a numeric format, and the format sets its byte cost:

The rule to memorise: weight memory ≈ parameter count × bytes per parameter. At the widely used FP16/BF16 serving default that collapses to a two-second estimate — double the billions and read the answer in gigabytes: 7B → ~14 GB, 13B → ~26 GB, 70B → ~140 GB.

Two honesty notes. First, units: 7 × 10⁹ × 2 bytes is 14 GB decimal but 13.0 GiB, and a GPU advertised as "16 GB" is typically 16 GiB. The gap is about 7% and it decides whether a model fits — this is the arithmetic set up in M0.4. Second, weights are not the whole footprint. Serving also needs the KV cache (which grows with batch size and sequence length) and activation working space; training needs optimizer state and gradients on top, which is why training a model takes several times the memory of serving it. Capacity planning gets its own treatment later in the course; here you only need the weights term and the reason the other terms exist.

The lesson stops at L3 deliberately. You do not need to know how BF16's exponent range differs from FP16's, or which kernels are available at which precision. What you need is that precision is a bytes-per-parameter choice, that lowering it is the standard lever for fitting a model into less memory, and that the cost of lowering it is accuracy you have to measure rather than assume.

Four things are routinely blurred, and distractors exploit all four.

The fourth column is the one people get wrong in casual speech. Temperature and top-p are neither parameters nor training hyperparameters — they are decoding settings applied at inference to an already-frozen model. Calling temperature a "model parameter" is a common informal slip and a clean distractor.

And the pairing that matters most for scenario questions:

Read the table top to bottom and it is a ladder of increasing cost, increasing data requirement, and increasing commitment. Read the "changes parameters" column on its own and it is the single test that resolves most scenario items.

A model card lists several numbers and they measure unrelated properties. Mixing them up is a reliable way to answer a sizing question wrong.

Two of these interact in a way worth knowing. Vocabulary size and hidden size together determine the embedding table's parameter count (vocabulary × hidden size), which is why a small model with a large vocabulary can spend a surprising fraction of its parameters just on embeddings. And context window is a runtime memory question, not a storage question: doubling the context you actually use roughly doubles the KV cache, while leaving the weight file byte-identical.

A team wants to serve a 13-billion-parameter model on a single 40 GB accelerator.

FP32:      13e9 × 4 bytes = 52 GB   → does not fit
FP16/BF16: 13e9 × 2 bytes = 26 GB   → weights fit, ~14 GB left for KV cache + activations
INT8:      13e9 × 1 byte  = 13 GB   → comfortable headroom for larger batches

Read it as an engineer, not an arithmetician. FP32 is eliminated outright. BF16 fits the weights with roughly 14 GB of working room — enough for a modest batch, and the batch size and context length you can then serve depend on how much of that the KV cache consumes. INT8 roughly doubles your headroom, at the cost of some accuracy that must be measured rather than assumed. (Quantisation methods and the accuracy question are their own topic later; here the point is that the decision is driven by a multiplication you can do in your head.)

Now do the units check that M0.4 set up, because it changes the BF16 answer's comfort level:

26 GB decimal = 26e9 / 2^30 GiB = 24.2 GiB
A "40 GB" accelerator is typically 40 GiB = 42.9 GB decimal
Headroom = 42.9 - 26 = 16.9 GB decimal ≈ 15.7 GiB

The direction of the correction matters. Because the card is measured in the larger binary unit and the weights in the smaller decimal one, this particular comparison comes out slightly better than the naive subtraction suggested. Reverse the situation — a decimal-advertised file against a binary-advertised card in the other direction — and it comes out worse. The habit, not the answer, is what you carry: convert both sides to the same unit before you conclude anything fits.

Now the same reasoning in reverse. If the team's real complaint is "the model does not know our internal API," no precision choice helps, because that fact was never in the weights. The options are: put the API docs in the context via retrieval, or change the weights via fine-tuning. Which one you pick is a parameters-versus-context decision — the exact distinction this lesson exists to make automatic.

Take a deliberately illustrative configuration — these are round numbers chosen to make the arithmetic legible, not a spec sheet for any named model:

hidden size        d = 4096
layers             L = 32
FFN inner size         4d = 16384
vocabulary         V = 32000

Per transformer block:

attention projections   4 × (d × d)      = 4 × 4096 × 4096  ≈ 67.1M
feed-forward matrices   2 × (d × 4d)     = 2 × 4096 × 16384 ≈ 134.2M
layer norms                                                 ≈ negligible
--------------------------------------------------------------------
per block                                                   ≈ 201.3M

Across the stack and the embeddings:

32 blocks × 201.3M                       ≈ 6.44B
embedding table  V × d = 32000 × 4096    ≈ 0.13B
output projection (if untied) V × d      ≈ 0.13B
--------------------------------------------------------------------
total                                    ≈ 6.7B  → "a 7B model"

Three things to take from the arithmetic. First, the two-thirds/one-third split is visible: 134.2M versus 67.1M per block. Second, the embeddings are about 2% of this model's parameters — real, but not where the storage is. In a much smaller model with the same vocabulary, that fraction becomes large, which is why small models sometimes tie the input embedding and output projection to the same matrix to avoid paying twice. Third, "7B" is a marketing round number; the actual count is whatever the configuration produces.

Then apply the memory rule:

6.7e9 × 2 bytes (BF16) = 13.4 GB decimal = 12.5 GiB of weights

Which is the number that decides whether this checkpoint loads on a 16 GiB card with room left for a KV cache. It does — barely, and only at small batch sizes.

Objective 1.5 (ML fundamentals) and objective 1.7 (reading research papers to spot emerging LLM trends) both assume this vocabulary — the official study guide's suggested-reading list includes LoRA, which is unreadable if you do not know what a parameter is. Core ML is 30% of the blueprint, the largest single domain.

The concept is questioned indirectly, in four recurring shapes:

1. Customisation-choice scenarios. "A team needs the assistant to answer from a document set updated daily." The keyed answer is retrieval, not fine-tuning, because retrieval changes context while fine-tuning changes weights — and daily-changing facts should not be baked into weights.
2. Sizing and deployment items. Anything that asks whether a model fits, or why quantisation helps, runs on the weight-memory rule above. Candidate reports say GPU spec-sheet depth did not appear; the arithmetic identity still does, because it is conceptual rather than product trivia.
3. Terminology discrimination. Parameter vs hyperparameter, weights vs context, training vs inference. Cheap points if the vocabulary is precise.
4. Knowledge-cutoff reasoning. "The model confidently describes a product feature released last month that does not exist." The keyed answer names the cutoff and points at grounding.

How the question tends to be phrased

You will typically see a short scenario naming a business need, four plausible technical responses, and one that respects the weights-versus-context boundary. The phrasings to recognise:

• "…knowledge that changes frequently…" → retrieval. Weights are the wrong store for volatile facts.
• "…must adopt our house tone and output format…" → fine-tuning territory, because style is a behaviour you want persisted.
• "…must cite the source document…" → retrieval, because weights cannot attribute.
• "…must run on a smaller GPU without changing models…" → quantisation, a bytes-per-parameter change.
• "…must not retain user data…" → note that inference does not write to weights at all, which is often the point of the item.

What the distractors typically look like

The standard traps are: fine-tune on the daily-changing corpus (technically possible, operationally wrong, and it still cannot cite); increase the context window offered as a fix for missing knowledge (a bigger window with nothing in it changes nothing); raise the temperature offered as a fix for factual gaps; and the model learns from the conversation offered as a mechanism, which contradicts frozen weights. Notice that each distractor is a real technique misapplied — that is the exam's house style, and the weights-versus-context test cuts through all four.

Calibration note, and it is calibration rather than fact: published candidate reports say the exam sits at a general level — know what each thing is and when to use it. NVIDIA publishes no per-item detail and no official passing score, so treat those reports as a study-planning aid only. For this lesson the practical reading is that the identity statements and the decision table earn more marks than any deeper numerical fluency would.

A decision rule you can apply directly to scenario items.

The reason this table is worth memorising in shape rather than in detail is that the exam does not ask you to build any of these. It asks you to recognise which one a described situation calls for — exactly the posture the job-role frame describes, where the associate contributes under supervision and is expected to make correct tool choices.

Both, but not equally reliably, and the honest answer is the one the exam rewards.

Fine-tuning demonstrably changes behaviour: format, tone, task framing, willingness to answer in a particular shape. It does that with relatively little data because you are steering something the model can already do.

Fine-tuning can also add knowledge, but it is a poor mechanism for the kind of knowledge business applications usually need. Facts learned this way are diffuse rather than addressable, cannot be cited, go stale the moment the source changes, and need a retraining cycle to update. Worse, fine-tuning a small factual set into a large model risks degrading the surrounding behaviour without a matching evaluation set to catch it.

So the practical rule: fine-tune for behaviour, retrieve for knowledge. Where a scenario needs both — house tone and current facts — the correct answer is usually both techniques together, and an option offering that combination is worth a second look.

Because there is nowhere for the provenance to live. A fact in the weights is not a row with a source column; it is a diffuse pattern distributed across many parameters, contributed to by many documents and reinforced by repetition. There is no pointer back to a document because no document was ever stored.

When a model does produce a citation without retrieval, it is generating text that has the shape of a citation, sampled from a distribution over plausible-looking references. That is the mechanism behind fabricated URLs and invented paper titles: the objective in 01-01 rewards plausible continuations, and a citation is just more text to continue.

Which is why provenance is an architectural property rather than a model property. Retrieval supplies the document, the pipeline passes it in the context, the prompt asks the model to answer only from it, and the citation points at something that actually exists on your side of the system. The model did not gain the ability to attribute; the system gained a source to attribute to.

Not within one training run, but the boundary is a design choice rather than a law, and knowing that stops the distinction from feeling arbitrary.

Architecture choices — layer count, hidden size, attention head count — are hyperparameters: a human sets them, and they determine how many parameters exist. Optimisation choices — learning rate, batch size, epoch count, weight decay — are hyperparameters that determine what values the parameters end up with. Neither is learned by gradient descent on the training loss, which is the defining test.

The edge cases are worth one sentence each. Learning-rate schedules change a hyperparameter over time by rule, not by learning. Learned positional embeddings are genuine parameters even though "position" feels like a configuration detail. And soft prompts in prompt tuning are new parameters that behave like a prompt — which is exactly why the customisation table above lists them in their own row rather than in either bucket.

• Parameters are learned numbers; they are the model's only knowledge store. Prompts and retrieved text are inputs, not knowledge.
• Parameters are frozen at inference. Nothing a user sends writes back into them.
• Weight memory ≈ parameters × bytes per parameter. FP16/BF16 at 2 bytes gives the double-the-billions shortcut: 7B ≈ 14 GB, 70B ≈ 140 GB. Weights are the floor, not the total.
• Parameters vs hyperparameters vs context vs decoding settings are four distinct things; only the first is counted in a model's size.
• Feed-forward layers hold roughly two thirds of a transformer block's parameters, attention roughly one third.
• Sort every customisation technique by does it change weights — that single test resolves a large class of scenario questions.
• Fine-tune for behaviour, retrieve for knowledge, and expect the best scenario answers to combine them when both are required.
• Weights cannot cite. Provenance is a property of the system you build around the model.

You know what the numbers are and how many of them there are. You do not yet know what shape the data flowing through them takes, and that shape is the vocabulary you need to read any model config, error message or architecture diagram for the rest of the course.

Next: 01-03 Tensor shapes in transformers: batch, sequence, and hidden size — the (batch, sequence, hidden) triple that every layer in the stack consumes and returns, and the reason a shape mismatch is the most common error you will ever see in this work.