NCP-AAI Concept Glossary — 58 Terms Defined

Agent architecture stylesNCPA-C1

Reactive (no world model, responds to stimuli), deliberative (plans over a world model first), and hybrid — a speed-versus-foresight trade, not a strict hierarchy.

Introduced in Agent architecture styles: reactive, deliberative, and hybrid systems

Structuring multi-step reasoningNCPA-C2

Logic trees, prompt chains, and adaptable architecture that lets an individual agent or tool be swapped without retraining the whole system.

Introduced in Structuring multi-step reasoning: logic trees, prompt chains, and adaptable architecture

ReAct: interleaving reasoning and actingNCPA-C3

A control loop alternating Thought, Action, and Observation, where the interleaved observation grounds reasoning in real tool output and curbs hallucination — distinct from chain-of-thought, which never acts.

Introduced in ReAct: interleaving reasoning and acting

Multi-agent orchestration topologiesNCPA-C4

Four distinct control structures for coordinating agents — centralized, decentralized, federated, and hierarchical — not one undifferentiated "multi-agent" blob.

Introduced in Multi-agent orchestration topologies: centralized, decentralized, federated, and hierarchical

Memory as an architectural concernNCPA-C5

Short-term memory as a session-scoped context window versus long-term memory that persists via a database, knowledge graph, or vector store.

Introduced in Memory as an architectural concern: short-term vs. long-term

Knowledge graphs for relational reasoningNCPA-C6

Entities and their relationships enabling multi-hop reasoning that flat vector search misses — graphs and vector search are not interchangeable.

Introduced in Knowledge graphs for relational, multi-hop reasoning

The human-agent interface as the oversight surfaceNCPA-C7

UI design placed inside the architecture domain because the interface is where oversight, feedback, and intervention actually happen.

Introduced in Designing the human-agent interface as the oversight surface

Dynamic prompt chains and decision refinementNCPA-C8

Prompt chains that branch on intermediate results at runtime rather than following one fixed script, refined by measuring how the agent chose.

Introduced in Prompt chains, dynamic branching, and refining agent decision-making

Integrating multimodal and generative modelsNCPA-C9

Routing each task to the right model — vision, language, speech — and fusing the results, including multimodal RAG pairing a vector index with a toolkit like LlamaIndex.

Introduced in Integrating multimodal and generative models across text, vision, and audio

Building custom tools, APIs, and functionsNCPA-C10

Tools with clear input/output contracts and, wherever possible, idempotency — the precondition that makes a safe retry possible at all.

Introduced in Building and connecting custom tools, APIs, and functions

The Retry patternNCPA-C11

Canceling non-transient faults, retrying immediately for rare blips, and retrying after backoff for busy or connectivity faults — never retrying a non-idempotent operation without a safeguard.

Introduced in The Retry pattern: transient faults, backoff, and idempotency

Retry vs. Circuit BreakerNCPA-C12

Two resilience-pattern state machines: Retry reattempts a transient fault; the Circuit Breaker's Closed → Open → Half-Open cycle stops calling a persistently failing dependency, with Half-Open avoiding flooding a service that is still recovering.

Introduced in Retry vs. Circuit Breaker: two resilience-pattern state machines and when each applies

Streaming conversation flowsNCPA-C13

Token-level streaming that improves perceived latency and lets a user interrupt mid-response, paired with feedback mechanisms feeding evaluation and oversight.

Introduced in Streaming conversation flows and real-time feedback

Evaluation pipelines and task benchmarksNCPA-C14

A repeatable harness running an agent over a fixed task set and scoring it, so results are comparable run to run.

Introduced in Evaluation pipelines and task benchmarks

Profiling vs. evaluationNCPA-C15

Two different questions: evaluation asks how good the outputs are, profiling asks where the time and tokens went — optimizing one without rechecking the other can silently degrade quality.

Introduced in Profiling vs. evaluation: two different questions

Observability for evaluationNCPA-C16

Step-level tracing tools (Phoenix, Weave, Langfuse, OpenTelemetry) providing the raw material for root-cause analysis and version-to-version comparison.

Introduced in Observability for evaluation: tracing an agent run step by step

The accuracy-vs-latency tradeoffNCPA-C17

A production tuning tradeoff where maximizing accuracy alone is usually the wrong answer once cost and latency are part of the scenario.

Introduced in The accuracy-vs-latency trade-off

Structured feedback and the data flywheelNCPA-C18

Feedback that informs refinement without replacing repeatable benchmarks — you still need an evaluation to know a change helped rather than just shifted behavior.

Introduced in Structured feedback and the data flywheel

Targeted optimization from results analysisNCPA-C19

Changing one high-impact thing and re-evaluating on a fixed set — the same experimental discipline as an A/B test, applied to the agent itself.

Introduced in Analyzing results to guide targeted optimization

NVIDIA NIM as a containerized inference microserviceNCPA-C20

A portable, GPU-accelerated microservice serving one model behind a standard API endpoint, tuned per model-plus-GPU pairing — not a model itself.

Introduced in NVIDIA NIM as an agent's inference endpoint: latency budgets and failure handling

Scaling with containers and KubernetesNCPA-C21

Packaging as a container, orchestrating replicas with Kubernetes, and load-balancing in front — scaling is horizontal replicas, not one bigger VM.

Introduced in Scaling with containers, Kubernetes, and load balancing

Profiling under distributed loadNCPA-C22

Contention, network latency, and tail effects that only appear at scale, which a single-node benchmark cannot predict.

Introduced in Profiling performance and reliability under distributed load

MLOps and governanceNCPA-C23

CI/CD, monitoring, and audit as repeatable, automated delivery plus governance for who can change what — not bolted on after launch.

Introduced in MLOps and governance: CI/CD, monitoring, and audit

Deployment cost vs. high availabilityNCPA-C24

Balancing over-provisioning (wasted spend) against under-provisioning (dropped requests), with load profiling revealing the right-sized headroom for peak plus failover.

Introduced in Balancing deployment cost against high availability

The serving stack: NIM, TensorRT-LLM, Triton, KubernetesNCPA-C25

Distinct jobs in the serving stack: NIM packages the model, TensorRT-LLM optimizes it, Triton serves it, and Kubernetes scales the containers.

Introduced in The serving stack in context: NIM, TensorRT-LLM, Triton, and Kubernetes

Why agents need memory at allNCPA-C26

Every LLM call is stateless — memory is an explicit architectural component an agent designer adds, not a property the model has on its own.

Introduced in Why agents need memory at all

The memory taxonomyNCPA-C27

Five categories, not two: short-term, long-term, episodic (specific past events), semantic (generalized facts), and procedural (learned skills).

Introduced in The memory taxonomy: short-term, long-term, episodic, semantic, and procedural

Reasoning frameworks: CoT and task decompositionNCPA-C28

Task decomposition as the backbone of multi-step planning, with ReAct wrapping chain-of-thought-style reasoning around tool actions.

Introduced in Reasoning frameworks: chain-of-thought and task decomposition

The five planning strategiesNCPA-C29

Task Decomposition, Plan Selection, External Module, Reflection, and Memory — reciting only decomposition as "the" planning method is the standing trap.

Introduced in Planning strategies: the five directions

Stateful orchestrationNCPA-C30

The orchestration layer holding the state that both the planner and the memory subsystem read and write across a multi-step task.

Introduced in Stateful orchestration: where memory and planning meet

Adapting reasoning from feedbackNCPA-C31

An agent logging outcomes and reflecting to adjust its approach over time, closing the loop with evaluation and the data flywheel.

Introduced in Adapting reasoning from prior experience and feedback

RAG fundamentalsNCPA-C32

The canonical retrieval pipeline — ingest, chunk, embed, store, retrieve, augment, generate — that reduces hallucination without changing model weights or fixing bad underlying data.

Introduced in RAG fundamentals: the canonical retrieval pipeline

Vector databases and approximate nearest-neighbor retrievalNCPA-C33

Retrieval requiring query and document embeddings from the same model and vector space, or similarity comparisons are meaningless.

Introduced in Vector databases and approximate nearest-neighbor retrieval

Beyond vector RAGNCPA-C34

GraphRAG, HybridRAG (combining graph and vector retrieval), and agentic RAG, which plans sub-questions and retries instead of a single-shot lookup.

Introduced in Beyond vector RAG: GraphRAG, HybridRAG, and agentic RAG

ETL and data quality for agent knowledgeNCPA-C35

Retrieval quality capped by data quality — ETL and preprocessing are not optional plumbing underneath a RAG agent.

Introduced in ETL and data quality for the knowledge an agent retrieves

Real-time reasoning over structured and unstructured knowledgeNCPA-C36

Production agents combining databases, knowledge graphs, and tables with documents, chat, and images in an increasingly multimodal RAG pipeline.

Introduced in Real-time reasoning over structured and unstructured knowledge

The NeMo Agent ToolkitNCPA-C37

Framework-agnostic orchestration working alongside LangChain, LlamaIndex, CrewAI, and Semantic Kernel with no replatforming, supporting MCP bidirectionally as both client and server.

Introduced in The NeMo Agent Toolkit: framework-agnostic orchestration

NIM microservices for high-performance inferenceNCPA-C38

An agent's model-serving layer, tuned per model-plus-GPU combination and backed by TensorRT-LLM or vLLM.

Introduced in Tuning NIM for GPU throughput: batching, TensorRT-LLM, and vLLM backends

TensorRT-LLM and Triton for latency reductionNCPA-C39

TensorRT-LLM optimizing a model for fast GPU inference, with Triton serving it via dynamic batching and concurrency — distinct, often-conflated jobs.

Introduced in TensorRT-LLM and Triton Inference Server for latency reduction

NeMo Guardrails as a platform componentNCPA-C40

A programmable safety layer sitting between application code and the LLM, a first-class part of the NVIDIA agentic stack rather than an afterthought.

Introduced in NeMo Guardrails as a first-class platform component

Multimodal input pipelines on NVIDIA hardwareNCPA-C41

Routing each modality — text, image, audio, video — to the right optimized model or NIM while keeping the pipeline GPU-efficient.

Introduced in Multimodal input pipelines on NVIDIA hardware

The NVIDIA agentic stack end to endNCPA-C42

How the pieces fit: the toolkit orchestrates, NIM serves, TensorRT-LLM optimizes, Triton hosts, a vector database grounds it, and Guardrails wraps the whole thing for safety.

Introduced in How the pieces fit: one NVIDIA agentic stack end to end

Monitoring dashboards and reliability metricsNCPA-C43

Latency percentiles, throughput, error rate, resource utilization, and uptime — the operational-health vocabulary a dashboard question hangs off.

Introduced in Monitoring dashboards and reliability metrics

Per-step tracing for diagnosisNCPA-C44

Logs, errors, and anomalies traced per step, since a single final output cannot diagnose a multi-agent failure on its own.

Introduced in Logs, errors, anomalies, and per-step tracing

Continuous benchmarking against prior versionsNCPA-C45

Comparing a live agent against earlier versions on an ongoing basis, since evaluation does not stop at launch.

Introduced in Online task-success regression: benchmarking a live agent against prior versions

Automated tuning, retraining, and versioningNCPA-C46

Automated (not manual, one-off) tuning and retraining, with versioning making a production rollback traceable and safe.

Introduced in Automated agent/policy versioning: tool contracts and prompt versions in production

Uptime, transparency, and trust in live deploymentsNCPA-C47

Reliability, visibility into behavior, and the ability to audit a decision — what lets an agent stay in production.

Introduced in Uptime, transparency, and trust in live deployments

NeMo Guardrails' five rail stagesNCPA-C48

Input, dialog, retrieval, execution, and output — five distinct stages, not just input and output.

Introduced in NeMo Guardrails: the five rail stages

Layered safety frameworksNCPA-C49

Content safety, jailbreak protection, and topic control, each combining multiple detection methods plus a human escalation path — one filter is never enough.

Introduced in Layered safety frameworks: filters and escalation

PII, agentic security, and audit trailsNCPA-C50

Isolating authentication and authorization away from the LLM, validating tool calls with execution rails, and keeping an audit trail of what the agent did and why.

Introduced in PII, agentic security, and audit trails

Mitigating bias and toxicity in agent outputsNCPA-C51

A data- and training-level concern distinct from guardrails, which constrain outputs but do not retrain or debias the model.

Introduced in Mitigating bias and toxicity in agent outputs

Licensing and regulatory complianceNCPA-C52

The EU AI Act's Article 14, requiring high-risk systems to be designed for effective human oversight during use.

Introduced in Licensing and regulatory compliance

Human-in-the-loop: benefits and drawbacksNCPA-C53

HITL's added cost and latency as a real, tested drawback — not just a benefits list of accuracy and accountability.

Introduced in Human-in-the-loop: benefits and drawbacks

HITL techniquesNCPA-C54

Three distinct mechanisms for embedding human input: supervised labeling, RLHF training a reward model, and active learning targeting only low-confidence cases.

Introduced in HITL techniques: supervised learning, RLHF, and active learning

Structured feedback loopsNCPA-C55

Captured approvals, corrections, and ratings as a learning signal feeding the same flywheel as evaluation — oversight is not just gating.

Introduced in Structured feedback loops for iterative improvement

Transparency and decision traceabilityNCPA-C56

Explainable reasoning and the ability to trace a decision back through its steps — named objectives, not cosmetic polish.

Introduced in Transparency and decision traceability

Human oversight and intervention pointsNCPA-C57

The EU AI Act's Article 14 requirement of effective human oversight for high-risk systems: review, approve, override, or halt.

Introduced in Human oversight and intervention points

Intuitive UI as the oversight surfaceNCPA-C58

The interface as where oversight, feedback, and intervention actually happen — the same concern as C7, seen from the other side of the loop.

Introduced in Intuitive UI as the oversight surface