Balancing Deployment Cost Against High Availability

Identity statement: optimizing deployment cost while ensuring high availability means right-sizing capacity — enough replicas and GPU capacity to meet peak demand and absorb a failover, without paying for capacity that sits idle beyond what that peak and that failover margin actually require. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): "Objective 4.5 — optimize deployment costs while ensuring high availability. This is a balancing act... Profiling under load reveals headroom and hotspots so you can right-size — enough replicas and GPU capacity for peak demand and failover, without paying for idle capacity."

The source material frames the two failure directions as a direct comparison worth holding in exactly this shape: [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states the two-sided table plainly — over-provisioning produces "high availability, wasted spend," and under-provisioning produces "lower cost, risk of dropped requests / downtime." Neither side of that comparison is free of a real cost; the framing that treats over-provisioning as "safe" and under-provisioning as "risky, but cheap" is already skipping past what over-provisioning actually costs, in real, recurring spend, which is not a hypothetical risk the way downtime is — it is a certain, ongoing expense, paid whether or not the extra capacity is ever used on a given day.

L1 — Intuition

Imagine staffing a restaurant for its single busiest Saturday night of the year, every single night of the year. On the busiest Saturday, that staffing level is exactly right — every table served promptly, no one waiting. On every other, quieter night, most of that staff stands around with nothing to do, and the restaurant is paying their full wages regardless. That is over-provisioning: correctly sized for the peak, wastefully sized for the typical case, and the waste is not a risk — it is a guaranteed, recurring cost paid every ordinary night. Now imagine the opposite: staffing for a typical Tuesday night, every night, including the busiest Saturday of the year. On Tuesday, this is efficient. On the busiest Saturday, tables wait too long, some guests leave unserved, and the restaurant's reputation (and revenue) takes the hit that under-provisioning always risks the moment real demand exceeds what is actually staffed.

L2 — Mechanism

Over-provisioning means running more replicas, larger instances, or more GPU capacity than current demand requires, sized instead against a peak that may occur rarely or never during a given period. The specific cost this produces is not merely "expensive" in the abstract — it is the recurring, metered cost of infrastructure sitting idle: a cloud instance or a self-hosted GPU that is provisioned and running incurs cost by simply existing in a ready state, whether or not it is currently serving a request, and every hour that capacity sits below its own utilization ceiling is an hour of paid-for capacity delivering no value. Because this cost is continuous rather than occasional, over-provisioning by a meaningful margin, sustained over a long deployment lifetime, can represent a substantial fraction of total infrastructure spend — the exact fraction being idle capacity's own utilization rate, subtracted from 100%.

Under-provisioning means running fewer replicas or less capacity than actual demand requires at some point in the deployment's traffic pattern. The specific cost this produces is not a smooth, gradual degradation — it tends to show up sharply, at exactly the moment demand crosses the ceiling of what is provisioned: requests queue behind an already-saturated set of replicas (the contention M4-03 covers directly), latency degrades past whatever budget M4-01 assigned to individual steps, and past some point, requests are dropped outright or time out, which a user experiences as the service simply not working. Unlike over-provisioning's smooth, continuous cost, under-provisioning's cost is concentrated precisely at peak demand — the exact moment a deployment's reliability matters most to the largest number of concurrent users, which is what makes it a materially worse failure mode than its lower baseline cost might suggest in isolation.

L3 — The exam-relevant edge case: the two costs are not symmetric, and "balance" does not mean "split the difference"

The edge case worth holding precisely is that a scenario asking for a "balance" between cost and availability is not asking for capacity sized halfway between the over-provisioned and under-provisioned extremes — because the two costs are not comparable in kind, and averaging them produces neither a cost-optimal nor an availability-adequate answer, only a deployment guaranteed to pay some of over-provisioning's waste while still risking some of under-provisioning's downtime. The correct target is capacity sized against a measured peak-demand-plus-failover-margin figure — a specific number derived from actual traffic and reliability data, per the next section — not a number chosen by intuition to feel like it sits "between" two extremes. "Right-sizing" is the source material's own word for exactly this distinction: not a compromise between two costs, but a specific, defensible capacity figure that happens to minimize waste while still meeting a stated availability target.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states the dependency directly: "Profiling under load reveals headroom and hotspots so you can right-size." The specific numbers a right-sizing decision needs are exactly the numbers M4-03's distributed-load profiling discipline produces, and it is worth being concrete about which ones and why.

Headroom is how much additional concurrent demand a given amount of provisioned capacity can absorb before latency or error rate starts degrading past an acceptable threshold — the measured gap between current typical load and the point where the system starts to struggle, discovered exactly the way M4-03's worked example discovered a p99 latency spike under realistic concurrent load. Without a measured headroom figure, a capacity plan has no way to know how much margin above typical demand its current provisioning actually provides, and is reduced to guessing whether "it feels like there's room to grow" is true.

Hotspots are the specific components — a particular replica, a shared downstream dependency, a specific resource contention point — that degrade first as demand increases, which M4-03's isolation technique (stub-substitution testing) is built to identify precisely. Knowing where the hotspot is changes what "add capacity" even means in a specific case: if the hotspot is GPU contention on the serving replicas themselves, more replicas (or better batching configuration) is the right lever; if the hotspot is a shared downstream dependency every replica calls, adding more serving replicas does nothing for that specific bottleneck and may even make it worse by sending the shared dependency more concurrent traffic.

A capacity plan that skips this measurement step and instead sizes capacity from an intuitive guess about peak demand is exposed to both failure directions at once, for the same underlying reason: a guessed number has no calibration against the system's actual behavior, so it is equally likely to overshoot (waste) or undershoot (risk) the true right-sized figure, and has no principled way to know which error it is making until the guess is tested against real traffic — at which point a wrong guess in the under-provisioned direction has already produced the dropped-requests cost this section opened with.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) names autoscaling directly as the mechanism that "helps track demand" within this cost-versus-availability tension, which the table above makes concrete: autoscaling does not eliminate the tension, it changes its shape, trading a continuous over-provisioning cost for a bounded, measurable reaction-lag risk. Whether that trade is the right one for a given deployment depends on the specific shape of its demand pattern — a workload with a sharp, sudden spike (a flash-traffic event) is poorly served by an autoscaler whose reaction time is longer than the spike's own ramp-up, while a workload with a gradual, hours-long ramp toward a predictable daily peak is well served by exactly that same autoscaler.

Constructed scenario, illustrative only. Continuing M4-02's worked example (a NIM-serving fleet sized to 40 sustainable concurrent requests per replica, 340 expected peak concurrent requests), incorporate M4-03-style measured headroom and hotspot data into a cost-aware capacity plan.

Measured typical demand (outside peak hours):            90 concurrent requests
Measured peak demand:                                    340 concurrent requests
Measured headroom per replica before latency degrades:    40 concurrent requests
Measured hotspot: GPU contention on serving replicas
  themselves (per M4-03's stub-substitution isolation) --
  NOT the shared downstream vector store.

Option A -- static over-provisioning for peak, always:
  340 / 40 = 8.5 -> 9 replicas, +1 for failure headroom = 10 replicas
  RUNNING CONTINUOUSLY, even during the 90-concurrent-request
  typical periods that make up most of the day.
  Utilization during typical periods: 90 / (10 x 40) = 22.5%
  -- roughly three-quarters of provisioned capacity sits idle
  most of the day, a continuous, recurring cost.

Option B -- autoscaling between a typical-demand floor and the
  peak ceiling:
  Floor: 90 / 40 = 2.25 -> 3 replicas minimum, always running
  Ceiling: 10 replicas (same peak-plus-failover figure as Option A)
  Autoscaler adds replicas as measured concurrent load approaches
  each currently-running replica's headroom limit, removes them
  as load recedes.
  Cost during typical periods: 3 replicas running, not 10 --
  roughly 70% less continuous spend than Option A during the
  majority of the day when demand is at its typical level.
  Tradeoff accepted: a measured reaction lag (replicas 4 through
  10 are not instantly available the moment demand starts
  climbing toward peak) -- acceptable specifically BECAUSE the
  measured hotspot is GPU contention on replicas themselves
  (Option B adds replicas to relieve exactly that), and because
  the demand ramp toward this deployment's peak (business hours
  building gradually) is slower than the autoscaler's reaction
  time, per traffic-pattern data outside this constructed
  example's scope.

The number worth taking away from this comparison is the roughly 70% typical-period cost reduction Option B achieves relative to Option A, for the same peak-hour capacity ceiling and the same failure headroom — Option B is not a cheaper, riskier alternative to Option A's safety, it is very nearly the same safety at peak, purchased at a fraction of the continuous cost, precisely because the deployment's actual demand pattern (a gradual ramp, not a sudden spike) is well matched to what autoscaling can react to in time. A different, sharper demand pattern would change this calculation, which is exactly why the right answer to "autoscale or over-provision" is never fixed in the abstract — it is a function of the deployment's own measured demand shape.

Constructed scenario, illustrative only. Two teams apply an absolute rule instead of a measured, right-sized answer, to make concrete why both absolute rules fail the stated objective even though each has an intuitive appeal.

Team C -- "always over-provision, safety first":
  Runs 20 replicas continuously, well above the measured 10-replica
  peak-plus-failover figure, reasoning that more capacity can only
  help availability.
  Result: availability is indeed excellent -- but the deployment is
  paying for roughly double the capacity peak demand and failover
  ever actually require, a continuous, unnecessary cost with no
  corresponding reliability benefit above what 10 replicas already
  provided. This FAILS objective 4.5's cost half explicitly.

Team D -- "always minimize cost, we'll scale if needed":
  Runs 3 replicas continuously (sized to typical demand only), with
  no autoscaling configured and no plan for peak demand beyond
  "we'll add more if it becomes a problem."
  Result: at every peak-demand period, requests queue and then drop
  once concurrent demand exceeds 3 replicas' combined headroom --
  a recurring, predictable outage at every peak, discovered only
  because it happened repeatedly in production rather than because
  it was measured and planned for in advance. This FAILS objective
  4.5's availability half explicitly.

Both teams applied a coherent, defensible-sounding rule, and both rules fail the same objective from opposite directions — which is exactly the shape a scenario question is likely to test: not "which extreme is worse," but recognizing that either extreme, applied as an absolute rule rather than derived from measured demand and headroom, misses the actual, stated objective of optimizing cost while ensuring availability, not one at the expense of ever considering the other.

⭐ THE EARNED INSIGHT "Balance" in this objective's phrasing does not mean picking a point partway between over-provisioning and under-provisioning — it means deriving a specific capacity figure from measured demand, headroom, and hotspot data, such that the figure happens to minimize waste while still meeting a stated availability target. Over-provisioning and under-provisioning are not two ends of a dial a team turns according to risk appetite; they are two distinct, avoidable errors, and the objective is avoiding both by measuring rather than guessing — which is why "profile first" from M4-03 and "right-size against peak plus failover" from this lesson are the same underlying discipline applied to two different questions.

Every worked example in this lesson has used a failover margin — the "+1" added on top of a peak-demand figure — without yet asking how that margin itself should be sized, and the answer matters because the margin is subject to exactly the same over-provisioning-versus-under-provisioning tension as the base capacity figure it sits on top of. A failover margin exists to absorb the loss of some number of replicas without breaching the fleet's availability target while Kubernetes' self-healing (M4-02) replaces them — which means the margin should be sized against a measured, specific failure assumption, not an arbitrary round number added out of habit.

The relevant question to measure against is: how many replicas can plausibly fail or be temporarily unavailable at once, under realistic conditions, before self-healing replaces them? A single replica crash is the common case a "+1" margin covers cleanly. A rolling update in progress (per M4-02's rollout mechanics) can temporarily reduce healthy old-version capacity by more than one replica at a time, depending on how aggressively the rollout is configured to proceed — a rollout that replaces four replicas simultaneously needs a margin of at least four to avoid breaching capacity during that specific window, not the "+1" that would suffice for an isolated crash. A shared-infrastructure failure — an entire availability zone or rack losing power, taking out every replica scheduled onto it at once — needs a margin sized against how many replicas that specific infrastructure unit could plausibly hold, which is a materially larger number than a single-replica-crash assumption, and is exactly the kind of hotspot M4-03's profiling and infrastructure-topology awareness would surface if that concentration risk were checked for deliberately.

Oversizing the failover margin "to be safe" reintroduces exactly the over-provisioning cost this lesson opened with, just relabeled as a failover allowance rather than a peak-demand allowance — a margin sized for a full-availability-zone failure, when the deployment's actual infrastructure never concentrates that many replicas in one failure domain, is paying continuously for a scenario that structurally cannot occur as described. The fix is the same discipline as everywhere else in this lesson: size the margin against a measured, specific failure assumption for the deployment's actual infrastructure topology and rollout configuration, not against an intuitive sense that "a bit more margin is always safer."

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): balancing deployment cost against high availability is objective 4.5, the closing objective within Deployment and Scaling's 13% of the NCP-AAI blueprint. The source material states its own self-check item on this objective directly, and the phrasing is worth internalizing as close to exam language: "Optimizing deployment cost while ensuring high availability means: right-sizing capacity for peak + failover without paying for idle resources" — with "always over-provisioning" and "removing all redundancy" both offered as wrong answers in the source's own self-check.

How the question tends to be phrased

Expect a direct definitional item, exactly the shape the source material's own self-check uses: "optimizing deployment cost while ensuring high availability means," with right-sizing for peak-plus-failover as the keyed answer against distractors offering always-over-provision, remove-all-redundancy, or ignore-demand-entirely. Expect also a scenario item describing a deployment that chose one extreme (heavy over-provisioning, or minimal capacity with no scaling plan) and asking what is wrong with that choice specifically, testing whether a candidate can name the specific cost (wasted spend, or dropped requests/downtime) rather than a vague "that seems risky."

What the distractors typically look like

The house style favors presenting one half of the correct answer as if it were the whole answer: always maximizing availability (ignoring cost entirely) or always minimizing cost (ignoring availability entirely), each omitting objective 4.5's explicit "while ensuring" framing that requires both simultaneously. A second common distractor shape offers "remove all redundancy to cut cost," which misreads cost optimization as license to eliminate the very failover margin the objective explicitly requires alongside peak-demand capacity.

A third, subtler distractor shape presents autoscaling itself as the complete answer to the tension — "just use autoscaling and the problem is solved" — which section 4's table exists specifically to correct: autoscaling reshapes the tension into a bounded reaction-lag risk, it does not make the tension disappear, and a scenario naming a demand pattern with a spike sharper than any realistic autoscaler reaction time is testing exactly whether a candidate recognizes autoscaling's limits rather than treating it as a universal fix regardless of the traffic shape described.

Over-provisioning's cost is not a hypothetical risk the way downtime is — it is a certain, recurring expense: a provisioned instance, replica, or unit of GPU capacity incurs cost by existing in a running, ready state, independent of whether it is actively serving a request at any given moment. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) names this directly as "high availability, wasted spend." A deployment sized to a rare or theoretical peak, running at that size continuously, pays that peak-sized cost every hour of every day, including the vast majority of hours when actual demand sits well below what is provisioned — which is precisely the "wasted" half of the source material's own phrasing.

The decision turns on matching the mechanism's reaction time against the deployment's actual demand pattern, not on a general preference for one approach. If demand ramps toward its peak gradually — over minutes or hours, following a predictable pattern like a business day's traffic building toward an afternoon peak — autoscaling's reaction lag is well within the time available to add capacity before it is needed, making it the more cost-efficient choice, since it avoids paying peak-sized cost during the majority of the time demand sits below peak. If demand can spike sharply and suddenly, faster than new replicas can start and pass health checks, autoscaling's reaction lag becomes a real availability risk during that spike, and static over-provisioning — accepting the continuous cost in exchange for zero reaction lag — may be the more defensible choice for that specific demand shape.

• Over-provisioning and under-provisioning are two distinct, specific failure modes, not a single "risk versus safety" dial — one wastes recurring spend, the other risks dropped requests and downtime concentrated at peak demand.
• "Balance" means deriving a right-sized capacity figure from measured data, not splitting the difference between two extremes — averaging two dissimilar costs satisfies neither the cost nor the availability half of the objective.
• Distributed-load profiling's headroom and hotspot data (M4-03) is the direct input a right-sizing decision needs — a guessed peak-demand figure has no calibration against real system behavior.
• Autoscaling reshapes the tension rather than eliminating it — it trades continuous over-provisioning cost for a bounded, measurable reaction-lag risk, and whether that trade is favorable depends on matching the lag against the deployment's actual demand-ramp shape.
• "Always over-provision" and "always minimize cost" are both wrong answers — each optimizes one half of the objective while failing the other half explicitly.
• Fix the capacity lever that matches the measured hotspot — adding replicas helps a replica-side contention hotspot, but can worsen a shared-downstream-dependency hotspot without addressing the actual constraint.

Right-sizing a fleet's capacity is a per-deployment decision — it says nothing yet about how the individual pieces of NVIDIA's own serving stack (the container, the optimizer, the server, the orchestrator) divide the labor that this whole module's capacity, scaling, and cost decisions have been built on top of. That is where this module closes: M4-06 covers the serving stack in context — NIM, TensorRT-LLM, Triton, and Kubernetes — synthesizing the distinct job each piece does, and how this module's five lessons on latency budgeting, scaling, profiling, governance, and cost fit around that stack as one coherent deployment story.

Next: M4-06 covers the serving stack in context — NIM, TensorRT-LLM, Triton, and Kubernetes — the module's closing synthesis of the distinct job each layer does and how this lesson's cost-versus-availability right-sizing decision sits on top of it.