Scaling an Agent Deployment: Containers, Kubernetes, and Load Balancing

Identity statement: NVIDIA's stated pattern for scaling an agentic or model-serving deployment is containerization with Kubernetes and load balancing — package the service as a container, orchestrate multiple replicas of that container with Kubernetes, and distribute incoming traffic across those replicas with a load balancer. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): "Objective 4.4 names the scaling mechanism directly: containerization (Docker, Kubernetes) with load balancing. The pattern: 1. Package the agent/model service as a container (Docker). 2. Orchestrate replicas with Kubernetes for scheduling, self-healing, and rollout. 3. Put a load balancer in front to distribute traffic across replicas."

Each of the three steps answers a question the previous step leaves open, and none substitutes for the others. Containerizing the service (step 1) answers "how do I run this the same way everywhere" — a container packages the agent's code, its dependencies, and its runtime into one portable unit, so the same image runs identically on a laptop, a staging cluster, and a production fleet, and "it works on my machine" stops being a meaningful caveat because the machine's differences are packaged away. But a single running container is still a single point of failure and a single ceiling on capacity — it answers "how do I run one copy consistently," not "how do I run enough copies to meet demand," which is exactly the question orchestration (step 2) exists to answer: Kubernetes takes a container image and a target replica count and keeps that many healthy copies running, restarting ones that crash, scheduling them onto available machines, and rolling out new versions without simply killing every replica at once. Orchestration, in turn, answers "how many copies are running and are they healthy," not "which copy handles this particular request" — that division of incoming traffic across the running replicas is the load balancer's job (step 3), and without it, having ten healthy replicas running is no better than having one, because nothing is spreading requests across the other nine.

The pattern's completeness is worth stating plainly because exam-style questions often test whether a candidate reaches for only one or two of the three pieces. A container with no orchestration is a single instance that happens to be portable — it still has one ceiling on capacity and one point of failure. Orchestrated replicas with no load balancer are a set of healthy, running copies that traffic has no mechanism to actually reach in a distributed way — client code would have to pick a replica itself, defeating the point of having several. A load balancer with nothing healthy behind it balances load across zero useful destinations. The pattern is a chain, and a scaling design missing any one link is not a partial solution to the scaling problem — it is a different, incomplete problem.

L1 — Intuition

Imagine a single toll booth handling more cars than it can process per minute. One fix is to build a bigger, faster toll booth — wider lanes, faster payment processing, a more powerful machine underneath. Another fix is to open several toll booths side by side and direct cars to whichever one is free. The first fix (a bigger booth) has a ceiling: however fast one booth gets, there is a limit to how much a single lane of traffic can process, and every car still funnels through one physical location, so one stuck car (or one booth malfunction) stops all traffic behind it. The second fix (several booths) has no single ceiling in the same sense — capacity grows by adding booths, and one booth malfunctioning takes out only that booth's share of traffic, not the whole road. Vertical scaling is the first fix; horizontal scaling is the second; and the toll-booth's single point of failure is exactly the property a load-balanced fleet of replicas is designed to avoid.

L2 — Mechanism

Vertical scaling means increasing the resources of a single running instance — more CPU, more memory, a bigger or additional GPU, on the same machine serving the same single copy of the workload. Horizontal scaling means running more copies (replicas) of the same workload, each a complete, independent instance capable of handling a request on its own, with something — the load balancer — distributing incoming requests across however many copies are currently running. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states the domain's own framing of this directly, as one of its named exam traps: "Scaling is not 'one bigger VM' — it's horizontal replicas behind a load balancer."

The mechanical reason horizontal scaling is the answer for an agentic or model-serving workload specifically, and not merely a stylistic preference, comes down to two properties vertical scaling cannot deliver no matter how large the single instance grows. First, a single instance — however powerful — remains a single point of failure: if that one instance crashes, is being redeployed, or simply needs a restart, every request in flight at that moment fails, and there is no other instance to absorb traffic while the one instance recovers. Second, a single instance's capacity has a hard ceiling set by the largest machine actually available to rent or own; horizontal scaling's capacity ceiling is set by how many replicas the infrastructure and budget can support, which is a much higher and much more flexible ceiling in practice, and one that can flex up and down with demand (autoscaling, discussed below) in a way a single machine's fixed size cannot.

L3 — The exam-relevant edge case: vertical scaling is not useless, it is insufficient alone

The edge case worth holding precisely is that vertical scaling is not a wrong idea in isolation — a bigger GPU genuinely does let a single NIM replica serve more concurrent requests, or serve a larger model, than a smaller one could. The trap the exam names is treating vertical scaling as a substitute for the three-part pattern rather than as a legitimate input to just one part of it (a more capable individual replica, which then still needs to be one of several replicas behind a load balancer for availability and for capacity beyond what any single replica can serve). A design that vertically scales a single instance to be extremely capable and stops there has built a very fast single point of failure — it has not built the horizontally scaled, load-balanced fleet the objective actually names. The correct framing keeps both true at once: make each replica as capable as it needs to be (a vertical decision, sized against M4-01's per-call latency budget), and run enough of those replicas behind a load balancer to meet total demand and survive any one replica's failure (the horizontal decision this lesson is about).

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) names Kubernetes's job in the pattern as orchestrating "replicas ... for scheduling, self-healing, and rollout," and it is worth taking each of those three words seriously rather than treating "Kubernetes" as a single undifferentiated buzzword for "the thing that runs containers."

Scheduling is the decision of which physical (or virtual) machine in a cluster runs which container replica, made automatically rather than by an operator manually placing each one. A cluster of machines with varying available CPU, memory, and GPU capacity presents a placement problem every time a new replica needs to start — Kubernetes solves that placement problem continuously, fitting replicas onto machines with enough free capacity, and re-solving it whenever a machine's availability changes.

Self-healing is the property that a replica which stops running — because it crashed, because the machine it was on failed, because a health check started failing — gets replaced automatically, without a person noticing the failure and manually starting a new one. This is the mechanism that makes horizontal scaling's single-point-of-failure advantage real in practice rather than theoretical: a replica dying is expected to happen, occasionally, at scale, and self-healing is what turns "one replica died" into "one replica died and was silently replaced within seconds" rather than "one replica died and the fleet is now permanently one replica short until someone notices."

Rollout is the mechanism for deploying a new version of the containerized service without simply killing every existing replica and starting fresh ones all at once, which would cause a visible capacity gap (and likely an outage) during the switch. A rolling update instead replaces replicas gradually — bring up a few new-version replicas, confirm they are healthy, retire a few old-version replicas, repeat — so that some capacity running the old version and some running the new version coexist briefly during the transition, and the load balancer keeps distributing traffic across whatever is currently healthy throughout. This is also what makes a rollback tractable if the new version turns out to be broken: the same gradual mechanism runs in reverse, replacing new-version replicas back with old-version ones, without ever dropping to zero healthy replicas along the way.

Beyond these three, [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) also names autoscaling directly as part of the pattern: "Autoscaling (Kubernetes) helps track demand." Autoscaling adjusts the replica count itself, up when measured demand or resource utilization crosses a threshold and back down when it falls, so the fleet's size tracks actual traffic rather than sitting fixed at whatever number was chosen at initial deployment time — this is the mechanism M4-05's cost-versus-availability tradeoff leans on directly: a fleet that can grow for a traffic spike and shrink afterward is not paying for peak-sized idle capacity around the clock.

A load balancer sits in front of a set of replicas and decides, for each incoming request, which replica handles it — the piece of the pattern that makes "several healthy replicas exist" into "traffic is actually being spread across them." [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) names this as the third explicit step of the pattern: "Put a load balancer in front to distribute traffic across replicas." What that distribution decision looks like in practice matters for an agentic workload specifically, because not every request is equally cheap to handle, and a load balancer unaware of that can create exactly the kind of uneven load M4-03's profiling-under-distributed-load lesson covers in depth.

A load balancer typically distributes requests using one of a small number of strategies: round-robin (send each new request to the next replica in a fixed rotation, regardless of that replica's current load), least-connections (send each new request to whichever replica currently has the fewest in-flight requests), or a health-aware variant of either that first excludes any replica currently failing its health check from receiving new traffic at all. For an agentic workload, where a single request's cost can vary enormously — a short, single-tool-call turn is cheap, while a long multi-step reasoning turn chaining several NIM calls per M4-01's budgeting story is expensive — a naive round-robin strategy can leave one replica handling a run of unusually expensive requests while a neighboring replica sits comparatively idle, purely by the luck of rotation order. A least-connections strategy, or a strategy that additionally weights by measured response time, corrects for this by routing new requests toward whichever replica appears to have the most current headroom rather than whichever replica is simply "next in line."

The load balancer is also the piece of infrastructure that makes Kubernetes's self-healing invisible to a calling agent's client: the moment a replica fails its health check, the load balancer stops sending it new traffic, and once Kubernetes replaces the failed replica and it passes its own health check, the load balancer resumes sending it traffic — a client calling the load-balanced endpoint sees, at most, a brief period of reduced capacity, never a hard failure that it has to handle itself, provided enough healthy replicas remained to absorb the gap. This is the mechanism M4-01's worked example leaned on directly when a NIM call's circuit breaker tripped and the fallback path was "route to a secondary NIM replica if one exists behind the load balancer" — that fallback only exists because the load balancer, not the calling agent, is the piece of infrastructure responsible for knowing which replicas are currently healthy.

Constructed scenario, illustrative only. A team has measured, per M4-01's per-call budgeting discipline, that a single NIM replica serving their agent's model can sustain 40 concurrent in-flight requests before its own latency starts breaching the agent's budgeted slice. The team expects a peak of 340 concurrent requests during business hours.

Peak concurrent requests expected:                    340
Sustainable concurrent requests per replica:            40
Naive replica count (peak / per-replica capacity):     340 / 40 = 8.5 -> round up to 9

Add headroom for one replica's failure without breaching
budget while Kubernetes self-heals it (N+1 for availability):
  9 + 1 = 10 replicas minimum

Add headroom for a rolling update in progress (some replicas
temporarily running the new version, briefly reducing total
healthy old-version capacity during the transition):
  10 + 1 = 11 replicas during a deploy window

Load balancer distributes 340 concurrent requests across whichever
of the 10-11 replicas Kubernetes currently reports healthy, using a
least-connections strategy so no single replica absorbs a
disproportionate share of the more expensive, multi-step-reasoning
requests purely by rotation luck.

The arithmetic itself is illustrative — the real per-replica sustainable concurrency in any actual deployment is a measured number, not a guessed one, following exactly the measurement discipline M4-01 laid out for setting a latency slice. What the arithmetic demonstrates structurally is that "how many replicas" is never simply "peak traffic divided by one replica's capacity" — a naive sizing that stops at 8.5 rounded up to 9 has zero room to lose a single replica to failure without breaching its own users' latency budgets the moment that replica goes down, which is exactly the failure mode self-healing and load balancing exist to prevent from becoming visible, but only if enough replica headroom exists in the first place for self-healing to have time to work.

Constructed scenario, illustrative only. Objective 4.1 extends the containerize/orchestrate/load-balance pattern from a single agent's serving layer to orchestrating multiple agents in production — worth tracing concretely rather than asserting that "it's basically the same thing at a bigger scale."

Single-agent deployment:
  One agent service, containerized, N replicas behind one load
  balancer, all replicas running the identical containerized
  workload -- Kubernetes' job is "keep N healthy copies of THIS
  ONE container running."

Multi-agent production deployment:
  Several DISTINCT agent services (e.g., a triage agent, a
  research agent, a summarization agent), each its OWN container
  image, each with its OWN replica count and its OWN load
  balancer endpoint -- Kubernetes' job is now "keep N-triage
  healthy copies of the triage container, M-research healthy
  copies of the research container, and P-summarization healthy
  copies of the summarization container running, SIMULTANEOUSLY,
  on a shared pool of machines."

What stays the same: the three-part pattern (containerize,
orchestrate, load-balance) applies IDENTICALLY to each agent
service individually.

What is new: Kubernetes is now scheduling multiple DIFFERENT
workloads onto a SHARED pool of machines, and has to decide how
to place them so no one agent service's replicas starve another's
of CPU, memory, or GPU capacity on a shared node.

The point the second scenario is built to make is that scaling one agent and orchestrating multiple agents are not two different disciplines requiring two different mental models — the same containerize/orchestrate/load-balance pattern applies to each individual agent service exactly as before. What genuinely changes is that Kubernetes' scheduling decision now has to account for several distinct workloads competing for the same underlying machine pool, rather than one workload having the whole pool to itself, which is precisely why resource requests and limits (declaring, per container, how much CPU/memory/GPU it needs and how much it may use at most) become load-bearing in a multi-agent deployment in a way they are less critical for a single-agent one: without them, one agent service's replicas can starve another's on a shared node, and no amount of load balancing at the traffic layer fixes a resource-starvation problem happening one layer below it, on the machine itself.

⭐ THE EARNED INSIGHT Scaling is not one decision, it is three separable ones stacked in a fixed order: how do I package it consistently, how many healthy copies exist right now, and which copy handles this request. The reason one bigger VM fails is not that VMs are bad infrastructure — it is that a single instance can only ever answer the first question and can never answer the second or third at all, no matter how large it grows. The moment a design has more than one replica, it has already implicitly answered the second question and immediately needs an answer to the third, which is precisely the load balancer's job, so a load balancer is never an optional add-on once a deployment has more than one replica — it is the other half of that same decision.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): Deployment and Scaling is objectives 4.1 through 4.5 and carries 13% of the NCP-AAI blueprint, and its own scope note states the scaling half of that weight directly: "know that scaling is a Docker/Kubernetes + load-balancing story with a cost-vs-availability tension." Objective 4.4 is the specific, named home for the three-part pattern this lesson covers, and objective 4.1 — "orchestrating multi-agent systems" at production scale — is this same pattern's direct extension to a fleet of distinct agent services rather than one, which section 6's worked example traced concretely.

How the question tends to be phrased

Expect a direct pattern-recall item: "the blueprint's recommended path to scale agentic deployments uses," with containerization plus load balancing as the keyed answer against distractors offering a single always-on VM, manual restarts, or client-side-only execution — each a plausible-sounding but structurally incomplete alternative to the real three-part pattern. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states this exact self-check item. Expect also a scenario item describing a deployment that scaled by moving to a larger single instance, asking what is missing or wrong about that approach — testing the horizontal-versus-vertical distinction from section 2 directly, and specifically whether a candidate can name the single-point-of-failure and hard-ceiling reasons vertical scaling alone falls short, not just recite "horizontal is better" without the mechanism behind it.

What the distractors typically look like

The house style favors a plausible-but-incomplete subset of the real pattern: naming Kubernetes without a load balancer (healthy replicas that nothing is routing traffic across); naming a load balancer in front of a single instance (nothing to actually balance); or describing "scaling" purely in terms of a bigger machine, omitting replicas and load balancing entirely — the domain's most directly named trap.

A single larger virtual machine remains exactly one instance no matter how much CPU, memory, or GPU capacity it is given, which means it keeps both of vertical scaling's structural limits: a hard capacity ceiling set by the largest machine actually available, and a single point of failure, since that one instance crashing, restarting, or being redeployed takes down all of its capacity at once with no other instance to absorb traffic in the meantime. Horizontal scaling — multiple replicas behind a load balancer — removes both limits: capacity grows by adding replicas rather than by growing one instance indefinitely, and any single replica's failure costs only that replica's share of total capacity, not the whole deployment's. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states this exact framing as one of the domain's named exam traps, which is why the exam tests it directly rather than leaving it as an implicit inference.

Kubernetes' job is keeping a target number of replicas healthy — scheduling them onto machines, restarting ones that fail, and rolling out new versions without dropping to zero healthy replicas. None of that, by itself, decides which of those healthy replicas handles any specific incoming request; without a load balancer in front, a set of ten healthy replicas provides no automatic mechanism for spreading traffic across all ten rather than all of it landing on whichever one address a client happens to be configured to call. The load balancer is the piece that turns "ten replicas are healthy" into "traffic is actually distributed across all ten, and automatically excludes any replica that starts failing its health check" — a distinct job from orchestration, which is exactly why the objective names both pieces separately rather than treating Kubernetes alone as a complete scaling answer.

• The scaling pattern is three parts, not one: containerize the service, orchestrate replicas with Kubernetes, and load-balance traffic across those replicas — each part answers a question the others leave open.
• Scaling is horizontal, not vertical — more replicas behind a load balancer, not one bigger VM, because a single larger instance keeps both a hard capacity ceiling and a single point of failure.
• Kubernetes' job is scheduling, self-healing, and rollout — placing replicas, replacing failed ones automatically, and deploying new versions without a capacity gap.
• The load balancer's job is distributing traffic across whichever replicas are currently healthy — a distinct responsibility from Kubernetes keeping those replicas healthy in the first place.
• Replica count needs headroom above the naive peak/per-replica-capacity number, so losing one replica to failure or a rolling update does not immediately breach every remaining replica's latency budget.
• Multi-agent orchestration applies the same three-part pattern per agent service — what is genuinely new is Kubernetes scheduling several distinct workloads onto a shared machine pool, which is where resource requests and limits become load-bearing.
• Autoscaling is what lets replica count track actual demand rather than sitting fixed at a peak-sized number around the clock — the mechanism the module's cost-versus-availability tradeoff depends on directly.

Having enough healthy, load-balanced replicas is a capacity question — it says nothing yet about whether those replicas are actually behaving correctly and reliably once real, concurrent, distributed traffic starts hitting all of them at once, rather than the clean, isolated traffic a single-node test might have used to validate the design. That is where this module goes next: M4-03 covers profiling performance and reliability under distributed load — why contention, network latency, and tail effects only appear once a design like this one is actually running at scale, and why a single-node benchmark never predicted them.

Next: M4-03 covers profiling performance and reliability under distributed load — the contention, latency, and tail effects that only show up once a load-balanced replica set like this one is actually serving concurrent traffic at scale.