The Parallelism Families: Data, Tensor, Pipeline, Sequence, Context, and Expert

Before naming all six families individually, it helps to sort them into two camps, because every other distinction in this lesson is a refinement within one of these two camps.

Camp one: split the batch, keep the whole model everywhere. This is data parallelism. Every GPU holds a complete, identical copy of the entire model — every layer, every weight, every piece of optimizer state — and each GPU is handed a different slice of the training examples. Nothing about the model itself is divided; what is divided is the work of processing different inputs. Because every GPU is running the same model on different data, the only new problem this creates is making sure the different GPUs' independently computed gradients get combined into one consistent update before anyone's weights actually change.

Camp two: split the model, keep the batch (or a micro-batch of it) whole per split. This is every other family in this lesson — tensor, pipeline, sequence, context, and expert parallelism. In all five of these, no single GPU holds the complete model; each GPU holds a fraction of it, sliced along some specific axis, and GPUs must communicate to reconstruct a correct result because the model's own computation is now spread across devices, not just the data flowing through it.

That two-camp split is the fastest way to read a scenario question. If the described symptom is "training goes too slowly, but the model itself fits fine on one GPU," the answer lives in camp one — data parallelism, full stop. If the described symptom is "the model itself, or some piece of it, does not fit," the answer lives in camp two, and the next question is which piece: a layer's own tensors, the stack of layers, the sequence dimension, or a set of Mixture-of-Experts experts. The rest of this lesson works through camp two's four members plus a return to data parallelism's own mechanics, one at a time.

Identity statement: data parallelism (DP) puts a full copy of the model on every participating GPU and divides the training batch across those copies, so each GPU computes a forward and backward pass on different examples using an otherwise identical model.

L1 — Intuition: many identical workers, different homework

Picture four GPUs, each holding an exact copy of the same model, and a batch of 128 training examples. Data parallelism hands 32 examples to each GPU. Each GPU independently runs its own forward pass, computes its own loss, and runs its own backward pass, producing a gradient tensor shaped exactly like the model's parameters. Because the four GPUs saw different examples, their four gradient tensors are not identical — they are four different estimates of "which direction should the weights move," each based on a different sample of the data.

L2 — Mechanism: why the estimates must be reconciled before anyone moves

If each GPU applied its own gradient to its own copy of the weights, the four copies would drift apart, one micro-decision at a time, until after enough steps they are four different models rather than four replicas of one. That is an unacceptable outcome for a job whose entire point is producing a single trained model. The fix is to average the four gradient estimates into one combined estimate — mathematically equivalent to what a single GPU would have computed had it processed all 128 examples at once — and have every GPU apply that same averaged update to its own copy. Distributed Data Parallel (DDP), PyTorch's standard implementation of this pattern, performs that averaging via an all-reduce: a collective communication operation in which every participating GPU contributes its gradient tensor and every participating GPU receives back the combined, averaged result. [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): "Distributed Data Parallel (DDP) — Syncs gradients via all-reduce before each optimizer step." After the all-reduce, all four GPUs hold identical gradients, apply identical updates, and remain exact replicas going into the next step.

L3 — What data parallelism does not solve

Data parallelism's defining limitation is the one thing camp one cannot touch: because every GPU holds a complete copy of the model, per-GPU memory for weights, gradients, and optimizer state is unchanged by adding more GPUs. Ten GPUs under pure data parallelism give roughly ten times the throughput on the batch, not one-tenth the memory footprint per GPU. If the model itself — just the weights, before any data arrives — does not fit on one GPU's memory, adding more GPUs under data parallelism does not fix that; it just gives you more copies of a thing that still does not fit anywhere. This is the precise reason camp two exists: when the model itself, not merely the batch, is the constraint, something has to divide the model rather than replicate it.

Identity statement: tensor parallelism (TP) partitions the weight tensors within a single layer across GPUs, so several GPUs jointly compute one layer's arithmetic rather than each GPU computing a whole layer alone.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): tensor parallelism splits "a single layer's weight tensors" along the "intra-layer" axis. Take one matrix multiplication inside a transformer block — an activation multiplied by a weight matrix. Tensor parallelism slices that one matrix, for example by its columns, so that GPU 0 holds one set of columns and GPU 1 holds the rest. Each GPU multiplies the same input by its own slice and produces a partial result; neither partial result alone is the layer's real output, so the GPUs must communicate — typically via an all-reduce — to combine their partial results into the correct one before the computation can proceed to whatever comes next. Every GPU in a tensor-parallel group is, at every instant, working on the same layer of the same forward pass; what differs is which slice of that layer's math each one owns. This reduces the per-GPU memory needed to hold that layer's weights and activations, which is exactly the lever to reach for when a single layer's tensors are themselves too large for one GPU.

Identity statement: pipeline parallelism (PP) assigns consecutive layers (a contiguous stage of the model's depth) to each GPU, so different GPUs hold different layers entirely, and a single input's forward pass travels stage by stage.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): pipeline parallelism splits "consecutive layers/segments" along the "inter-layer" axis, and "interleaved/virtual-pipeline scheduling shrinks the idle 'bubble.'" Where tensor parallelism keeps every GPU in its group working on the same layer, pipeline parallelism does the opposite: GPU 0 might own the first eighth of the model's layers, GPU 1 the next eighth, and so on, so that at any instant different GPUs are working on genuinely different layers (and, with good scheduling, different micro-batches too). Each GPU only ever needs to hold the weights for its own stage, not the whole model, which is the lever to reach for when the model has too many layers in total to fit its full weight set on one GPU — a distinct symptom from any one layer being individually too wide. The word "bubble" names the idle time a naive pipeline wastes while filling and draining; scheduling improvements shrink but do not eliminate it by definition.

Because tensor and pipeline parallelism are so frequently offered as a false-pair distractor — "the model is too big, pick TP or PP" without specifying which kind of too-big — the module's very next lesson exists to give that single pairing a full, dedicated treatment: M7-02 works through the mechanics, the communication-cost differences, and the exam's own framing of intra-layer versus inter-layer splitting in depth. This lesson deliberately does not pre-explain that distinction beyond what section 3 and this section already established, because M7-02 is where it earns its full attention as the domain's number-one distractor pair.

Identity statement: sequence parallelism (SP) partitions activations along the sequence (token) dimension in the regions of the model adjacent to a tensor-parallel layer, and it is meaningful only once tensor-parallel size is already greater than one.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): sequence parallelism "extends TP" and is "only meaningful when TP size > 1." Sequence parallelism is not an independent decision the way data, tensor, and pipeline parallelism are; it has nothing to attach to on its own. Tensor parallelism's all-reduce combines partial results for a layer's matrix math, but certain regions surrounding that layer — normalization and dropout operations, for instance — do not need the full, unsharded activation tensor to compute correctly; they can operate on a shard of the sequence dimension instead, further reducing the activation memory each GPU in the tensor-parallel group has to hold. If tensor-parallel size is 1 — meaning no tensor parallelism is in use at all — there is no tensor-parallel group for sequence parallelism to extend, and enabling it does nothing, because the mechanism it complements does not exist yet. This precondition is the single fact about SP most worth holding onto: it is a refinement of tensor parallelism's activation-memory savings, not a sixth independent axis you could reach for in isolation.

Identity statement: context parallelism (CP) splits the input along the sequence dimension across all layers of the model — not just the TP-adjacent regions sequence parallelism touches — using ring-style key/value communication in the backward pass.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): context parallelism splits "inputs along the sequence dim across all layers," which is "broader than SP," with "ring-style KV communication in the backward pass." Where SP is scoped narrowly to regions next to a tensor-parallel layer and needs TP size greater than one to mean anything, CP has no such prerequisite — it divides the sequence dimension everywhere in the model, for every layer, so that a very long input sequence's memory cost is spread across GPUs rather than concentrated on any one of them. This distinction is worth holding precisely because the two names sound like synonyms and are not: a question that treats SP and CP as interchangeable, or that applies CP's layer-wide logic while describing SP's TP-adjacent scope, is testing exactly this boundary. SP is the narrower, TP-dependent technique; CP is the broader, TP-independent one, and the situation that calls for CP — a sequence long enough that even a fully replicated, non-tensor-parallel model strains under its activation memory — is different from the situation that calls for SP, which is squeezing a little more activation memory savings out of a tensor-parallel group that already exists.

Identity statement: expert parallelism (EP) distributes the experts of a Mixture-of-Experts (MoE) architecture across GPUs, and it applies exclusively to MoE layers.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): expert parallelism splits "MoE experts across GPUs," is "MoE only," and "expert count must be divisible by EP size." A Mixture-of-Experts layer replaces a single dense feed-forward block with several parallel "expert" sub-networks and a routing mechanism that sends each token to only a subset of those experts, rather than through all of them. Expert parallelism places different experts on different GPUs, so no single GPU has to hold every expert's weights, and a routed token's computation is dispatched to whichever GPU holds the expert it was routed to. The precondition here is architectural rather than a size threshold the way SP's is: expert parallelism has nothing to distribute in a dense transformer that has no expert layers at all, because there are no experts to place anywhere. [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) additionally names a divisibility constraint — the number of experts must be evenly divisible by the expert-parallel size — which is a configuration detail worth recognizing as a real constraint rather than an arbitrary one: it is what guarantees every GPU in the expert-parallel group is assigned the same number of experts rather than an uneven remainder.

The fastest way to hold six names in working memory is a single table organized by the one column that actually discriminates them: what gets split.

Read this table by symptom, not by name: "training is slow but the model fits" points at data parallelism; "one layer's matrices are too big" points at tensor parallelism; "there are too many layers in total" points at pipeline parallelism; "I already have tensor parallelism and want to trim activation memory further" points at sequence parallelism; "my sequence length alone is the problem, independent of tensor parallelism" points at context parallelism; and "my model has Mixture-of-Experts layers and the experts themselves are too much to hold on one GPU" points at expert parallelism, and nowhere else, because none of the other five techniques has anything to say about an expert.

Take six independent, one-line problem statements, each drawn from a different corner of the taxonomy, and work each one to its technique the way a scenario question would demand.

Symptom 1: "The model runs fine on one GPU, but a full epoch over our
dataset takes three days and we have eight idle GPUs."
  -> Nothing about the model is too big. The constraint is throughput.
  -> Data parallelism: replicate the model eight times, split the batch,
     all-reduce the gradients.

Symptom 2: "One transformer block's feed-forward matrices alone need more
memory than a single GPU has, even at batch size 1."
  -> The constraint is a single layer's own tensors.
  -> Tensor parallelism: shard that layer's matrices across GPUs.

Symptom 3: "Any individual layer fits comfortably, but the model has 120
layers and the full weight set does not fit on one GPU."
  -> The constraint is depth, not width.
  -> Pipeline parallelism: assign contiguous blocks of layers to
     different GPUs.

Symptom 4: "We already run tensor parallelism at size 8 and still want to
shave activation memory in the regions next to those tensor-parallel
layers."
  -> Tensor-parallel size is already > 1, so the SP precondition holds.
  -> Sequence parallelism: shard activations along the sequence dimension
     in the TP-adjacent regions.

Symptom 5: "We run pure data parallelism, no tensor parallelism at all,
and a single 128,000-token document blows out activation memory on every
GPU identically."
  -> Tensor-parallel size is 1, so SP has nothing to attach to; the
     sequence itself, independent of TP, is the constraint.
  -> Context parallelism: split the sequence across all layers,
     regardless of TP configuration.

Symptom 6: "Our model has 64 Mixture-of-Experts feed-forward blocks per
layer, and holding every expert's weights on every GPU is the memory
bottleneck."
  -> The model has MoE layers specifically, and the constraint is expert
     weight count.
  -> Expert parallelism: distribute the 64 experts across GPUs (choosing
     an EP size that divides 64 evenly).

This is a constructed scenario, with the six symptoms authored to isolate each technique cleanly; a real system will usually present two or three of these constraints simultaneously, which is exactly why the families are designed to combine rather than compete.

Take a second scenario that forces more than one technique at once, because that is the shape most real deployments — and most well-built scenario questions — actually take. A team is training a dense (non-MoE) transformer with 96 layers and unusually wide feed-forward matrices, on two 8-GPU servers, where each server's 8 GPUs are connected by fast NVLink and the two servers are connected to each other by ordinary, much slower networking.

Start from what is actually too big. The feed-forward matrices being "unusually wide" is a within-layer symptom — some single layer's tensors are large enough to strain one GPU's memory even before considering the other 95 layers. That points at tensor parallelism, sized to fit inside the 8 NVLink-connected GPUs of a single server, because tensor parallelism's all-reduce is frequent and latency-sensitive and belongs on the fastest available fabric. That single tensor-parallel group, however, only accounts for one server's 8 GPUs; the model's full 96 layers still need to be placed somewhere, and a plausible next move is to split those 96 layers into two pipeline stages, one stage running inside each server's tensor-parallel group, with only the comparatively light, infrequent activation hand-off between stages crossing the slower inter-server link. Because there is no Mixture-of-Experts structure anywhere in this model, expert parallelism has nothing to attach to, and it is correctly absent from the configuration; because tensor-parallel size is already greater than one inside each server, sequence parallelism becomes available as a further activation-memory refinement layered on top of the existing tensor-parallel groups, at the team's discretion, without changing the tensor/pipeline structure just derived. This is inference from the stated topology, not a measured configuration — the point is that the symptom (a too-wide layer, plus too many layers, on a two-tier fast/slow interconnect) predicts a specific combination of techniques before anyone runs anything, which is exactly the reasoning a scenario question rewards.

Domain 7, GPU Acceleration and Optimization, carries 14% of the NCP-GENL blueprint — the second-largest domain, behind only Model Optimization's 17%. [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) states that these two domains together account for 31% of the exam and names this as "where to invest" study time. Within Domain 7, the parallelism taxonomy sits first, and the domain's own scope note is explicit that objectives here ask you to configure multi-GPU training and fix bottlenecks — meaning a scenario question, not a bare definitional one, is the default shape to expect. Objectives 7.1 through 7.4 collectively require knowing which parallelism splits what, and this lesson's six identities are the vocabulary every later scenario in the domain assumes you already have cold.

How the question tends to be phrased

Expect a symptom-plus-topology description followed by four named techniques as options, in the shape of sections 9 and 10 above. Recognizable cues: "a single layer's weights do not fit" points at tensor parallelism; "the model has too many layers for its full weight set to fit" points at pipeline parallelism; "GPUs share a fast intra-node fabric" is a placement cue for tensor parallelism specifically; "we already use tensor parallelism and want more activation memory relief in nearby regions" points at sequence parallelism; "sequence length alone, independent of tensor-parallel configuration, is the constraint" points at context parallelism; and "the model uses Mixture-of-Experts layers" is the only situation in which expert parallelism is even a candidate answer. A question naming "model parallelism" as an option is naming the umbrella term that covers tensor, pipeline, sequence, context, and expert parallelism collectively — useful to recognize, but rarely itself the most specific correct answer when a more precise family name is also offered.

What the distractors typically look like

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) names three specific traps directly: confusing intra-layer (tensor) and inter-layer (pipeline) splitting as the single most common distractor in the domain; treating sequence and context parallelism as synonyms despite SP's TP>1 precondition and CP's layer-wide scope; and offering expert parallelism as a fix for a dense model's memory problem when a dense model has no experts for EP to distribute. A fourth recurring shape offers data parallelism as the answer to a "the model itself is too large" symptom — a real technique, attached to the wrong problem, since data parallelism replicates rather than shards the model and therefore never relieves a model-too-large constraint on its own.

Start with the two-camp question from section 1: is the batch too slow, or is the model itself too big? If the model fits and only throughput is the problem, data parallelism is the answer and nothing else in this lesson applies yet. If the model itself does not fit, the next question is which dimension is too big — a single layer's own tensors point at tensor parallelism, too many layers in total point at pipeline parallelism, an unusually long sequence independent of any tensor-parallel configuration points at context parallelism, and a Mixture-of-Experts structure whose experts do not fit points at expert parallelism. Sequence parallelism is reached for last, as a refinement layered onto an existing tensor-parallel group rather than a first move, precisely because it requires that group to already exist.

Because sequence parallelism is defined as an extension of tensor parallelism's own activation-sharding logic in the specific regions adjacent to a tensor-parallel layer — normalization and similar operations that sit next to the sharded matrix math — rather than as an independent splitting strategy. If tensor-parallel size is 1, there is no tensor-parallel group and no TP-adjacent region for sequence parallelism to shard activations within, so enabling SP in that configuration does nothing measurable. Context parallelism, by contrast, was designed to split the sequence dimension across every layer of the model regardless of whether tensor parallelism is in use at all, using its own ring-style key/value communication pattern in the backward pass — which is exactly why it has no tensor-parallel precondition and can be deployed on a model that uses no tensor parallelism whatsoever.

• The first fork is batch versus model. Data parallelism splits the batch and replicates the model everywhere; every other family splits the model itself along a different axis.
• Tensor parallelism splits within one layer (intra-layer); pipeline parallelism splits across layers (inter-layer). This exact pairing is the exam's flagged most common distractor, and it gets its own full treatment in M7-02, immediately next.
• Sequence parallelism requires tensor-parallel size greater than 1, because it is a refinement of TP's activation sharding in TP-adjacent regions, not an independent axis.
• Context parallelism has no such prerequisite — it splits the sequence across all layers regardless of tensor-parallel configuration, using ring-style key/value communication.
• Expert parallelism applies only to Mixture-of-Experts architectures, and the expert count must divide evenly by the expert-parallel size.
• All six families are designed to combine. Scaling from billions to trillions of parameters is achieved by stacking several of these axes together, not by picking just one.
• Domain 7 is 14% of the NCP-GENL blueprint, second only to Model Optimization's 17%; together they are 31% of the exam, and this taxonomy is the vocabulary every later scenario in the domain assumes.

This lesson deliberately stopped at the survey level for tensor and pipeline parallelism — enough to place them correctly against sequence, context, expert, and data parallelism, but not the full mechanics of their communication patterns, their interconnect preferences, or the specific worked scenarios the exam builds around confusing the two. M7-02 picks that distinction up immediately and treats it as what the domain's own material calls it: the single most common distractor pair in GPU Acceleration and Optimization, covered in depth next.