NeMo Guardrails: The Five Rail Stages From Input to Output

[GROUND TRUTH] (Sources/ncp-aai/domain-9-safety-ethics-compliance.md) describes NeMo Guardrails as "an open-source Python package that adds programmable guardrails between application code and the LLM — used to block, alter, or validate unsafe, off-topic, malicious, or policy-violating inputs and model responses." Read that definition slowly, because two words in it do a lot of work that a quick skim will miss. "Between" places Guardrails as a layer, not a component of the model itself — it does not retrain anything, and it does not live inside the weights. "Programmable" means the checks are configured logic you author, not a fixed built-in behavior the model ships with. Put those together and the picture is: application code calls into a Guardrails-wrapped pipeline, and that pipeline decides, at defined points, whether to let a piece of text (or a tool call, or a retrieved chunk) through unchanged, alter it, or block it outright.

The natural first guess at how many "defined points" there are is two: check what the user said, check what the model said back. That guess is wrong, and it is wrong in a specific, testable way. [GROUND TRUTH] (Sources/ncp-aai/domain-9-safety-ethics-compliance.md) is explicit that the rail stages "run at distinct points in the interaction," and lists five: input, dialog, retrieval, execution, and output. The scope note attached to this entire domain calls this out directly: "Know the five rail stages of NeMo Guardrails (not just input/output)." This is not a minor addendum to the two-rail picture — it is a correction of it. A pipeline that only checks input and output has three entire categories of behavior running unchecked in between: what topics the conversation is allowed to wander into, what the retrieval step hands the model as supposedly-trustworthy context, and what the agent actually does when it reaches for a tool.

This module's guiding question — what is the specific control that evidences a safety claim, and where exactly does it run in the request path — is answered, for NeMo Guardrails, by naming a rail and pointing at a specific position in the pipeline. "We have safety guardrails" is not an answer to that question. "We have an execution rail that validates every tool call's arguments before the tool runs" is. The rest of this lesson builds the five-position map that makes answers of the second kind possible.

L1 — Intuition: a pipeline with five gates, not two

Picture the full path a single user turn takes through an agent built around a large language model and equipped with retrieval and tools: the user's message arrives, the agent's dialog manager decides what kind of turn this is and what topic it falls under, if the turn needs grounding the agent retrieves supporting context, the model reasons over that context and may decide to call a tool, and finally a response goes back to the user. NeMo Guardrails does not sit at the two ends of that path and ignore the middle. It places a gate at each of the five points listed above, and each gate is a separate, independently configurable piece of logic that can inspect, and if necessary alter or block, what is about to pass through it. Think of it less as "input filter, output filter" and more as five checkpoints strung along a corridor, each one asking a question specific to what is crossing it at that exact point.

L2 — Mechanism: what each rail actually does

The five stages, in the order a request actually encounters them:

Input rails run first, on the user's raw message before it ever reaches the LLM. This is the stage most people correctly guess exists, because it maps onto the obvious intuition — check what came in before you act on it. An input rail can reject a message outright, rewrite it (stripping something malicious while keeping the legitimate intent), or flag it for a different handling path.

Dialog rails govern the conversational flow itself: what the bot is and is not allowed to discuss, and how the conversation is permitted to branch. This is a distinct concern from input rails — an input rail asks "is this specific message safe to accept," while a dialog rail asks "should this conversation be allowed to go in this direction at all," which is a question about the trajectory of a multi-turn exchange, not a single message in isolation. A user can send five individually harmless messages that, read as a sequence, walk the conversation toward a topic the dialog rail is configured to keep closed.

Retrieval rails run on chunks pulled back by a retrieval step, before those chunks are used to ground a response — this stage exists specifically for RAG-style agents. A retrieval rail checks the content that is about to become part of the model's "trusted" context, which matters because a compromised or poorly curated knowledge source can hand the model a chunk containing an injected instruction, sensitive data that should never surface, or content that contradicts the agent's policy — and none of that would ever be caught by a rail watching only the user's typed message.

Execution rails validate tool and function calls the agent is about to make. This is arguably the rail with the highest real-world stakes of the five, because a tool call is the one point in the pipeline where the model's output stops being just text and starts being an action with side effects — sending an email, querying a database, moving money. An execution rail checks the call's target and arguments before the call actually fires.

Output rails run last, on the model's generated response, before it reaches the user — the second half of the "obvious" two-rail guess, and the one everyone remembers because it is the last thing standing between a bad generation and a real person reading it.

L3 — The exam-relevant edge case: why dropping any one rail creates a specific, describable gap

The five-stage list is easy to memorize and easy to still get wrong on a scenario question, because the trap is rarely "name all five" — it is "given this specific failure, which rail was missing." Consider what happens if a pipeline implements input, output, and execution rails, but skips retrieval rails, on the theory that "we already check the input and the output, so anything bad in the middle gets caught at one end or the other." That reasoning fails for a documents-poisoning scenario: a malicious or corrupted document sitting in the knowledge base gets retrieved, is never re-checked as it flows into the prompt (because nothing is watching the retrieval stage specifically), and the model then generates a response that faithfully reflects the poisoned content. That response might pass an output rail's content-safety check completely — the output rail is checking for toxicity, PII leakage, and policy violations, not for "does this fact actually appear in the knowledge base as opposed to being injected." The failure originated at a stage no rail was watching, and it is invisible to both the ends that were being watched.

The same logic holds for dialog rails specifically. An input rail evaluates one message; it has no memory of five prior messages and no model of "this conversation, read as a whole, is drifting somewhere it shouldn't." Skipping the dialog stage means an agent's topic-control policy exists only as a static instruction in the system prompt, with no independent enforcement layer checking that the instruction is actually being followed turn over turn — and system-prompt instructions are exactly the kind of soft constraint that a sufficiently persistent multi-turn conversation can erode.

Reading this table left to right for any single row answers the module's guiding question directly: the rail is the control, and the "runs on" / "runs before" columns are the exact position in the request path. A safety claim that cannot be located on one of these five rows is a claim that has not actually specified where its control runs.

Consider a support agent, backed by a RAG pipeline over an internal knowledge base and equipped with a tool that can issue account refunds, handling a single user turn: "My order arrived damaged, can you refund me $85 and tell me if there's a known defect with this product line?"

⚠️ UNVERIFIED (constructed scenario — the specific latencies below are illustrative, chosen to make the request-path arithmetic concrete, not measured figures from any deployment):

t=0ms     User message arrives: "...refund me $85 and tell me if there's a
          known defect..."
t=0-8ms   INPUT RAIL: checks the message for jailbreak patterns, prompt
          injection markers, off-topic content. Message passes — legitimate
          support request, no injection markers detected. Latency cost: 8ms.
t=8-14ms  DIALOG RAIL: confirms "refund request + product-defect question"
          is within the bot's allowed topic set (it is — this bot is scoped
          to order support). Passes. Latency cost: 6ms.
t=14-95ms RETRIEVAL: vector search over the knowledge base returns 3 chunks
          about this product line, including one flagging a known seal
          defect on units from a specific manufacturing batch.
t=95-110ms RETRIEVAL RAIL: checks the 3 returned chunks before they enter
          the prompt. One chunk contains an internal-only note ("do not
          disclose batch defect rate to customers pending recall decision")
          that should never reach a customer-facing generation step. The
          retrieval rail flags and strips that chunk. Latency cost: 15ms.
t=110-340ms MODEL GENERATION: the model reasons over the (now-filtered)
          retrieved context, decides a refund is warranted under policy,
          and emits a tool call: issue_refund(order_id=..., amount=85.00).
t=340-365ms EXECUTION RAIL: validates the tool call before it fires —
          checks that $85.00 does not exceed this agent's configured
          per-transaction refund ceiling, confirms the order_id belongs to
          the authenticated user making the request, and confirms the
          refund tool itself (not some other tool) is the one being
          invoked. Call passes validation and executes. Latency cost: 25ms.
t=365-520ms MODEL GENERATION (continued): tool result returns, model
          composes the final reply describing the refund and, separately,
          a general (non-internal) answer about product quality.
t=520-534ms OUTPUT RAIL: checks the final composed response for policy
          violations, toxicity, and — critically — a second pass for any
          leaked internal content. Passes. Latency cost: 14ms.
t=534ms   Response reaches the user.

Total rail overhead across all five stages: 8+6+15+25+14 = 68ms, against a
total request time of 534ms (~12.7% of end-to-end latency).

Every one of the five rails did a different job in this trace, and no single rail could have substituted for another. The input rail cleared a legitimate message quickly. The dialog rail confirmed the request stayed inside scope. The retrieval rail caught something an input or output check would never have seen at all — an internal note that entered through the knowledge base, not through the user's typed text, and that would have looked like perfectly normal, non-toxic prose to an output rail scanning for policy violations, because disclosing an undisclosed internal note is not the same failure mode as generating toxic content. The execution rail caught a class of risk — an unauthorized or over-limit financial transaction — that has nothing to do with text safety at all. And the output rail did a final, independent check that assumed nothing about what had already been verified upstream.

A second worked example: the two-rail agent versus the five-rail agent under the same attack

To make the "domain's most common miss" concrete rather than abstract, trace the identical attack through two differently configured agents built on the same underlying model and the same tools.

Configuration A — input and output rails only (the common, incomplete mental model). Configuration B — all five rails, as designed.

The attack: a user opens a conversation with several individually innocuous messages establishing rapport and context, then, several turns in, sends a message that does not read as a jailbreak in isolation but is engineered to redirect the conversation toward getting the agent to reveal its system prompt's internal refund-approval thresholds — information the agent is explicitly configured never to disclose — followed immediately, once that information starts to surface, by a request that abuses a newly-revealed threshold to request a refund exactly at the disclosed limit.

In Configuration A, the input rail evaluates each message on its own and finds nothing alarming in any single one — none of them contains obvious jailbreak phrasing, and the "rapport-building" turns are genuinely benign-looking text. There is no dialog rail watching the conversation's trajectory, so nothing flags that five turns have collectively steered toward extracting internal configuration. The model, with no dialog-level circuit breaker, eventually discloses threshold information in its response. The output rail then checks that response for toxicity and policy language — and a sentence like "the per-transaction refund ceiling configured for this account tier is $85" contains no toxic or obviously policy-violating language on its face, so it can pass an output rail whose checks were never designed to catch configuration disclosure specifically. The refund request that follows, sized exactly to the disclosed threshold, then goes through — because Configuration A has no execution rail either, so nothing independently re-validates that specific tool call against policy at the point it actually fires; the model's own judgment is the only gate, and it has already been steered.

In Configuration B, the dialog rail is tracking topic and intent across turns, not just per-message content, and a multi-turn pattern that trends toward extracting internal configuration is exactly the shape a topic-control policy is configured to interrupt — the conversation gets redirected or flagged well before disclosure happens. Even in the counterfactual where dialog rails somehow missed it, the execution rail provides a second, independent check: it validates the refund tool call against the account's actual configured ceiling and the requester's actual authorization at the moment the call is about to fire, regardless of what the model was talked into believing about that ceiling during the conversation. Two rails Configuration A never had are exactly the two rails that stop this attack, and neither the input rail nor the output rail — the two rails both configurations share — was ever positioned to catch it.

Why the five stages have an order, and the order matters for latency

The five stages are not an unordered set of five places a check could happen to sit — they run in a fixed sequence dictated by the shape of a request, and that sequence has a direct latency consequence worth naming explicitly, because a rail that runs early and rejects a request cheaply spares every later stage the cost of running at all. Input and dialog rails run before any retrieval or generation has happened, which means a request rejected at either of those two stages never pays the cost of a retrieval call or a model generation pass — a comparatively expensive vector search and an even more expensive LLM forward pass are both skipped entirely. A request that clears input and dialog but fails at the retrieval rail has already paid for a retrieval call, but nothing yet for generation. A request that clears everything through execution but fails at the output rail has, by that point, paid the full cost of the entire pipeline — retrieval, generation, and (if a tool call was involved) the tool call itself — for a response that ultimately gets blocked anyway.

This ordering effect is a genuine design consideration, not a curiosity: a system that can push more of its rejection decisions earlier in the sequence (catching a jailbreak attempt at the input rail rather than only via a downstream output-safety check on the resulting generation) is both faster on average and cheaper to run, because it avoids paying for retrieval and generation on requests that were never going to be allowed through regardless. This is also why an input rail, despite being the "obvious" one everyone remembers, still earns its position independent of the other four — catching what can be caught early is valuable specifically because of what it spares every later, more expensive stage from having to process at all.

[VENDOR SPEC] (Sources/ncp-aai/domain-9-safety-ethics-compliance.md) states that Guardrails configuration "uses YAML plus Colang flows (conversational flows, guardrail logic, event-driven behavior), extendable with custom Python actions." Two languages doing two different jobs, plus an escape hatch:

• YAML carries the configuration shape — which models are wired in, which rails are enabled, general settings.
• Colang is the flow language: the actual conversational logic, the guardrail rules, and event-driven behavior are authored as Colang flows, not as YAML keys and not as free-form English instructions embedded in a prompt.
• Custom Python actions extend both — when a rail needs to do something Colang's flow vocabulary doesn't natively express (call an external classifier, hit a database, run a bespoke check), a Python action is the extension point.

The exam-relevant trap here, called out directly in the source material, is guessing at the authoring language. It is not SQL, not Rust, and not plain English prose — it is Colang, with YAML wrapping it, extendable in Python.

[VENDOR SPEC] (Sources/ncp-aai/domain-9-safety-ethics-compliance.md) states Guardrails is "deployable both as a library (PyPI) and as a production microservice (container, Kubernetes/Helm), with portable configs between them." The practical implication is that the same rail configuration a team develops and tests as an in-process library import can move, largely unchanged, into a containerized production deployment orchestrated with Kubernetes and Helm — the rails a developer validated locally on a laptop are the same rails running in the cluster, not a re-implementation that has to be re-verified from scratch. That portability matters for the module's guiding question in a specific way: "where exactly does the control run in the request path" has the same answer whether the deployment is a PyPI-imported library inside a monolith or a standalone microservice a request routes through — the five-stage structure does not change shape with the deployment target.

This lesson has established the five positions in the request path where a control can sit. It has not yet answered, for any given rail, what specific detection logic runs there — an input rail could reject on a keyword list, a machine-learning classifier, or a third-party API, and those are very different levels of protection wearing the same "input rail" label. NVIDIA's own guidance is explicit that a single detection method inside any one rail is not considered sufficient on its own; content safety, jailbreak protection, and topic control are each built from multiple layered detection methods plus a path to human escalation, which is a separate objective from simply knowing the five rails exist.

⭐ THE EARNED INSIGHT: Knowing that five rail stages exist tells you where a control can run in an agent's request path; it says nothing about how strong the control sitting at any single one of those positions actually is — a rail with one weak filter behind it is a real gate in the pipeline and a genuinely weak safeguard at the same time, and the exam's layered-safety material exists precisely because "we have a rail there" is a claim about position, not a claim about coverage.

Domain 9, Safety, Ethics, and Compliance, carries 5% of the NCP-AAI blueprint — small by weight, but [GROUND TRUTH] (Sources/ncp-aai/domain-9-safety-ethics-compliance.md) calls NeMo Guardrails "the anchor technology tested here," and the domain's own scope note puts the five rail stages first among the three things it says explicitly to know. Objective 9.4 (layered safety frameworks) and 9.1/9.2 (system security, audit trails, compliance guardrails) both presuppose the five-stage structure as the substrate they are layered onto or wired through — you cannot meaningfully discuss where a layered filter or an execution-rail validation runs without first having the five positions in hand.

Expect the question shape to present a specific failure or attack — content leaking from a retrieved document, a conversation drifting toward a disallowed topic over several turns, a tool call executing with bad arguments — and ask which rail stage is responsible, or which rail was missing. The most common wrong-answer pattern is naming input or output when the actual failure sits at dialog, retrieval, or execution; a second common pattern is naming "guardrails" generically as the answer to a question that is actually asking for a specific stage.

A jailbreak attempt is most directly the input rail's job — it runs on the user's raw message before that message ever reaches the LLM, and jailbreak-pattern detection is one of the layered methods that typically lives at this stage. But a jailbreak attempt spread across several turns, where no single message looks like an attack, is a dialog-rail concern instead, because the dialog rail is the stage that reasons about conversational trajectory rather than one message at a time — which is why layered safety (the next lesson's subject) treats jailbreak protection as combining self-check, heuristic detection, and a dedicated detection model rather than relying on the input rail alone.

Yes, and this is precisely why retrieval rails exist as their own stage rather than being folded into output checks. An output rail evaluates the model's final generated text for toxicity, policy violations, and similar surface properties; it has no special mechanism for recognizing "this fact came from a source that should never have been in the trusted context at all." A retrieval rail evaluates chunks before they become part of that trusted context, so it can catch and strip content — an internal note, a poisoned document, outdated or contradicted information — on a basis the output rail was never designed to check, even when the resulting generated sentence would look completely unobjectionable to a text-safety scanner.

• NeMo Guardrails runs five rail stages, not two: input, dialog, retrieval, execution, and output — each at a distinct point in the request path.
• Input and output rails are the two most people guess correctly; dialog, retrieval, and execution are the three that get missed, and each catches a failure shape the other four cannot.
• Dialog rails reason about conversational trajectory across turns; input rails reason about one message in isolation — these are not substitutes for each other.
• Retrieval rails vet chunks entering the model's trusted context; output rails check the model's generated text — a leak sourced from retrieval can pass an output rail cleanly if nothing watched retrieval itself.
• Execution rails validate tool calls before they fire, independent of whatever the model was persuaded to believe during the conversation.
• Configuration is authored in Colang flows wrapped in YAML, extendable with custom Python actions — not plain English, SQL, or Rust.
• Guardrails deploys as a PyPI library or a containerized/Kubernetes microservice, with portable configs between the two.
• Knowing where a rail sits in the path is a different fact from knowing how strong the detection logic running at that rail actually is — the second question is what layered safety frameworks address.

Every rail this lesson placed in the request path is, on its own, only as strong as the specific detection methods configured behind it — an input rail with a single keyword list is a real checkpoint and a weak one at the same time. Next: M9-02 picks up exactly that gap — how content safety, jailbreak protection, and topic control each combine multiple detection methods plus a human escalation path, because one filter behind any single rail is never considered enough.