Integrating Multimodal and Generative Models Across Text, Vision, and Audio

A vision model trained to describe and reason about images, a speech-to-text model trained to transcribe audio accurately, and a large language model trained to reason over text are each optimized for the data type they were built around. Asking a text-only language model to "understand" an image by only reading a caption someone wrote about it throws away most of the information in the image — anything the caption-writer did not think to mention is simply gone. Asking a vision-capable model to also be the best possible speech transcriber, or vice versa, asks one architecture to be state-of-the-art at jobs it was not specifically optimized for.

The routing-and-fusion pattern avoids this by keeping each modality-specific model doing the one job it is actually good at, and adding a coordination layer on top. Concretely: an image goes to a vision model, which returns a structured description or a direct answer to a visual question; an audio clip goes to a speech model, which returns a transcript or an audio-aware analysis; text goes to a language model, which reasons over both the original text and whatever the vision and speech models returned about the non-text inputs. The language model (or an equivalent reasoning step) is usually the fusion point — it is the piece that takes several modality-specific outputs and turns them into one answer that actually uses all of them together, rather than three separate answers bolted next to each other.

The three modalities objective 2.2 names, and what each is suited to

An agent does not need all three in every task — a purely text-based support ticket never touches the vision or audio path — but an agent architecture that assumes only text will eventually meet a task that does not fit, and the routing pattern is what lets the same agent serve both cases without a rewrite.

Generative models: producing new content in a modality, not just interpreting one

Objective 2.2's phrasing pairs "generative" with "multimodal" for a reason worth separating out explicitly: a modality-specific model is not always used to interpret an input, the way a vision model interprets an uploaded image. Some are used generatively, to produce new content in that modality as the agent's output — a diagram-generation model producing a chart from data the agent computed, a text-to-speech model turning a written response into spoken audio for a voice interface, or an image-generation model producing an illustration for a piece of written content. The routing-and-fusion pattern applies to this direction too, just reversed: instead of routing an incoming input to the right interpreting model, the agent routes an outgoing content need to the right generating model, and fusion in this direction means assembling the generated non-text content (an image, an audio clip) alongside the language model's text into a single multimodal response the user receives, rather than a text-only answer that references content it never actually produced.

Routing solves half the problem; fusion solves the other half, and it is the part that is easy to get wrong even after routing is built correctly. Fusion means taking the vision model's description, the speech model's transcript, and the original text input, and combining them into context a single reasoning step can use to produce one final, coherent answer — not three separate answers concatenated together with no relationship between them.

L1 — Intuition

Imagine three colleagues each looking at one piece of evidence — one reads the customer's typed message, one looks at the screenshot they attached, one listens to the voicemail they left — and then all three report back to a fourth colleague who has to write the actual reply. That fourth colleague's job is fusion: taking three separate, modality-specific observations and weaving them into a single response that treats all three as part of the same story, not three unrelated notes stapled together.

L2 — Mechanism

Mechanically, fusion is usually implemented by having each modality-specific model produce a text-shaped output — a description, a transcript, an extracted set of fields — and then assembling those text outputs into the context window of a final reasoning call to a language model, alongside the original text input. The vision model's "the screenshot shows an error dialog reading 'Payment declined — insufficient funds'" becomes a piece of text the language model can reason over exactly as it would reason over a customer's typed sentence. This is why the language model so often ends up as the fusion point specifically: text is the common representation every modality's output gets converted into, and a language model is the piece built to reason fluently over text.

L3 — The exam-relevant edge case: routing without fusion produces disconnected answers, not a wrong error

A system that routes correctly but never properly fuses the results does not typically crash or throw an obvious error — it produces an answer that quietly ignores part of the input. An agent that routes a screenshot to a vision model, gets back an accurate description, but never actually feeds that description into the step generating the customer-facing reply will produce a plausible-sounding answer that responds only to the typed text and silently drops the screenshot's evidence, even though the vision model did its job correctly. This is the multimodal equivalent of a chain that branches correctly but on a bad signal M2-01: the individual pieces can each work exactly as designed while the system as a whole still fails, because the failure is in how the pieces are combined, not in any one piece.

⭐ THE EARNED INSIGHT Every modality-specific model in this pattern can pass its own test in isolation — the vision model correctly describes the image, the speech model correctly transcribes the audio — and the system as a whole can still fail, because "multimodal integration" was never really a claim about any one model's competence. It is a claim about the coordination layer that never shows up in a single model's benchmark score: whether a genuinely fused context reaches the step that writes the final answer. A wrong multimodal answer is far more often a fusion bug hiding behind two correctly working models than it is a failure of either model itself, which is exactly why debugging one should start by asking what actually reached the final reasoning step's context, not by re-testing the vision or speech model in isolation.

Objective 2.2's reading recommendations pair a vector index like Milvus with an orchestration toolkit like LlamaIndex specifically because plain-text RAG, on its own, cannot ground an agent in evidence that is not text. [GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): "NVIDIA's suggested reading pairs LlamaIndex, NIM, and Milvus for multimodal RAG, reflecting that modern retrieval spans more than plain text." A document collection full of scanned invoices, product photos, or charts embedded in PDFs has real evidentiary content that a text-only chunking-and-embedding pipeline either drops entirely or captures only as whatever alt-text or OCR happens to survive.

Multimodal RAG extends the canonical retrieval pipeline (M6-01 covers the plain-text version end to end) by indexing non-text content alongside text — embedding images into the same or a compatible vector space, storing them in an index such as Milvus, and letting a retrieval query surface an image chunk exactly as it would surface a text chunk, ranked by relevance to the query. The toolkit layer (LlamaIndex, in NVIDIA's pairing) is what coordinates this: it manages the multimodal ingestion pipeline, dispatches retrieved image content to a vision model for interpretation when needed, and assembles the combined result into the context a generation step uses — which is routing-and-fusion again, this time applied specifically to what a retrieval step returns rather than to raw user input.

Multimodal RAG vs. plain-text RAG at a glance

Where NVIDIA NIM fits into this pairing

The vendor pairing NVIDIA's reading points to is not just a vector index and a toolkit — it names NIM as the third piece, and NIM's role here is the same containerized-serving role it plays anywhere else in this cert's material: a portable, GPU-accelerated microservice serving one model behind a standard API endpoint. In a multimodal RAG pipeline, that typically means the vision model interpreting a retrieved image, and the embedding model turning both text and images into vectors in the first place, are each served behind their own NIM endpoint rather than run as ad hoc, hand-wired inference code. This matters for the routing-and-fusion pattern specifically: the orchestration toolkit's job of dispatching a retrieved image to "a vision model" is, in NVIDIA's own stack, dispatching an API call to a NIM endpoint serving that vision model, which means the same tool-contract discipline (M2-03) and resilience patterns (M2-04, and the Retry-vs-Circuit-Breaker pairing in M2-05) that apply to any other tool call apply here too — a NIM-served vision model called mid-retrieval is a tool call that can fail transiently exactly like any other, and the multimodal pipeline needs the same failure-handling discipline wrapped around it as any single-modality tool call would.

Consider an expense-reporting agent that a user asks: "How much did I spend at this restaurant, and does the tip look right for a party of four?" — attaching a photo of a scanned receipt.

Step 1 (routing): Input contains an image attachment -> route to vision model.
  Vision model input: [receipt image]
  Vision model output (text): "Subtotal: $84.50. Tax: $7.18. Tip line: $12.00 (blank,
  handwritten '$12'). Total: $103.68. Party size not printed on receipt."

Step 2 (fusion): Combine vision model's output with the user's original text question
  into one context for the reasoning model.
  Language model input:
    User question: "How much did I spend at this restaurant, and does the tip look
    right for a party of four?"
    Vision model finding: [the extracted text above]

  Language model output: "You spent $103.68 total ($84.50 subtotal + $7.18 tax + $12.00
  tip). For a party of four, a 15% tip on the $84.50 subtotal would be about $12.68, and
  an 18% tip would be about $15.21 — the $12.00 written on the receipt is close to a
  standard 15% tip, on the lower end but within a typical range."

Constructed scenario — the receipt figures are illustrative, not drawn from a real transaction. Notice what each modality-specific piece did and did not do: the vision model extracted numbers from an image but never touched the tip-adequacy question, because that question requires arithmetic reasoning over the extracted numbers, not vision. The language model never saw the image at all — it only ever reasoned over text, some of which originated as a photo. The fusion step is the only place in this trace where the receipt's visual content and the user's typed question exist in the same context together, and it is the step that makes the final answer actually respond to both.

The receipt example touched vision and text. Consider a case that exercises all three modalities objective 2.2 names, to see how routing and fusion scale past two inputs. A user reports a problem through a support channel by typing a short message, attaching a screenshot of an error, and leaving a voice note describing what happened right before the error appeared: "It froze right after I hit submit."

Step 1 (routing, three parallel branches):
  Text branch:  user's typed message -> passed through as-is (no modality-specific
                model needed for plain text)
  Vision branch: screenshot -> vision model
    Vision model output: "Error dialog: 'Session expired. Please log in again.'
    Background shows a partially filled multi-step form, step 3 of 5."
  Audio branch:  voice note -> speech model
    Speech model output (transcript): "It froze right after I hit submit."

Step 2 (fusion): assemble all three findings into one context for the reasoning model.
  Language model input:
    User's typed message: "The form won't save my progress, please help."
    Vision model finding: [error dialog + form-step description above]
    Speech model finding (transcript): "It froze right after I hit submit."

  Language model output: "This looks like a session-timeout issue: your login
  session expired while you were on step 3 of the 5-step form, and submitting
  after the session expired triggered the freeze and the 'Session expired' dialog
  you saw. Your progress on steps 1-2 should be saved; logging back in and
  resuming from step 3 should recover it without losing earlier work."

Constructed scenario — the error text and diagnosis are illustrative, not drawn from a real support case. Notice that no single modality's output, on its own, would have supported this diagnosis. The typed message alone says only "won't save my progress" — vague, and consistent with several different bugs. The screenshot alone shows a session-expired dialog on step 3, which is informative but does not explain why the user perceived it as a "freeze" rather than a normal expiry message. The voice note alone establishes timing ("right after I hit submit") but names no specific error. Only the fused combination — session expired, on step 3, timed to a submit action — supports the specific diagnosis the language model produced. This is the strongest version of the case for routing-and-fusion: a task where the correct answer genuinely requires synthesizing evidence that arrived through three different channels, none of which was sufficient alone.

Agent Development carries 15% of the NCP-AAI blueprint, tied with Agent Architecture for the heaviest weight in the exam, and objective 2.2 sits inside it as the domain's explicit call to integrate generative and multimodal models across text, vision, and audio. [GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): the domain's own framing states that "a production agent often routes a task to the right model — a vision model for an image, an LLM for reasoning, a speech model for audio — and fuses the results," naming routing and fusion as the mechanism directly rather than leaving it implicit. The same source names the specific vendor pairing behind multimodal RAG — LlamaIndex, NIM, and Milvus — as the reading NVIDIA points to for this objective, which makes that pairing a plausible target for a direct recall question rather than only a scenario one.

Expect the scenario-question shape to describe a task that touches more than one modality — an image with a text question, an audio clip needing both transcription and reasoning — and ask which integration approach fits, with a distractor built around treating a single model as sufficient for every modality involved. Expect a second shape built around retrieval specifically: a corpus described as containing scanned documents, charts, or photos, with the correct answer recognizing that a plain-text RAG pipeline would miss that evidence entirely and multimodal retrieval is required to surface it.

A multimodal model is a single model that can accept more than one input type — an image and text together, for instance. Multimodal integration, as this domain's objective uses the term, is the broader system design of routing each modality to the model best suited to it and fusing the results, which remains necessary even when one of the underlying pieces happens to be a multimodal model, because fusing several modality-specific findings into one coherent, evidence-grounded answer is a coordination task the model call itself does not perform on its own.

A caption only captures whatever the caption-writer chose to describe, which is almost always less information than the image itself contains, and a query asking about a visual detail the caption never mentioned will retrieve nothing useful even though the answer is sitting directly in the image. Multimodal RAG indexes the image itself (via an image-capable vector index) so retrieval can match a query against the actual visual content, not against a lossy text summary someone wrote in advance, which is why NVIDIA's own reading pairs a vector index like Milvus with an orchestration toolkit for exactly this purpose rather than treating captions as sufficient.

An agent needs its own audio integration whenever the audio itself carries information a plain transcript would discard — tone, emphasis, background sound, or timing that a transcription step captures but a bare text log of "what was said" does not — or whenever the pipeline needs to process raw audio input directly rather than depending on an external transcription step having already run. Where only the words matter and a reliable transcript already exists, treating that transcript as ordinary text input is sufficient and audio-specific routing is not required.

Work through each item before checking the answer key.

1. An agent receives an image and a text question. What does the routing-and-fusion pattern do first?
   • A. Sends both the image and the text to the same language model call together, unprocessed.
   • B. Routes the image to a vision model and the text to a language model, each handled by the model built for it.
   • C. Discards the image and answers from text alone.
   • D. Converts the text to an image before processing.
2. A vision model correctly describes an attached screenshot, but the agent's final reply never mentions anything from that description. What went wrong?
   • A. The vision model failed.
   • B. Routing worked, but fusion did not — the vision model's finding was never included in the context of the step that wrote the final answer.
   • C. The language model does not support images.
   • D. The user's question was too short.
3. Why does a language model usually end up as the fusion point in a multimodal pipeline?
   • A. It is the fastest model to run.
   • B. Text is the common representation every modality's output gets converted into, and the language model is built to reason fluently over text.
   • C. Vision and speech models cannot produce text output.
   • D. Fusion always requires the largest available model.
4. A document corpus contains scanned invoices with handwritten totals. A plain-text RAG pipeline is built over this corpus. What happens when a query asks about a specific handwritten total?
   • A. Retrieval succeeds normally, since RAG works the same regardless of content type.
   • B. Retrieval likely fails to surface the relevant evidence, because the handwritten total was never indexed as searchable content.
   • C. The vector database automatically applies OCR at query time.
   • D. The query is rejected outright.
5. What does NVIDIA's own reading recommendation pair together for multimodal RAG specifically?
   • A. A vector index like Milvus and an orchestration toolkit like LlamaIndex.
   • B. Two separate language models.
   • C. A single audio-only model.
   • D. A relational database and a spreadsheet tool.
6. An agent's support-diagnosis case requires a typed message, a screenshot, and a voice note to reach the correct answer, where none of the three alone is sufficient. What does this illustrate?
   • A. That fusion is unnecessary when enough modalities are present.
   • B. That routing and fusion together can synthesize evidence no single modality's output supports alone.
   • C. That voice notes should always be converted to screenshots.
   • D. That three modalities always produce three separate valid answers.
7. Why is treating a single "multimodal model" as sufficient, on its own, a common exam trap?
   • A. Multimodal models do not exist.
   • B. Even a model that accepts multiple input types still needs an explicit fusion step to combine several modality-specific findings into one coherent, evidence-grounded answer.
   • C. Multimodal models can only process one modality at a time internally.
   • D. Fusion is only needed for text-only agents.
8. What is the risk of fusing modality-specific outputs with no preserved provenance?
   • A. The agent runs out of context window.
   • B. A user cannot tell whether a claim in the final answer traces back to what they typed, what they attached, or a hallucination.
   • C. The vision model stops working.
   • D. Retrieval speed decreases.

Answers

1. B. Routing sends each modality to the model built for it; combining unprocessed inputs into one call skips the modality-specific strength each specialized model provides.
2. B. The vision model did its job; the failure is that its output was never carried into the final answer's context, which is a fusion failure, not a routing failure.
3. B. Every modality's output gets converted into text as the shared representation, and a language model is the piece built to reason over text fluently, which is why it is the natural fusion point.
4. B. A handwritten total inside a scanned image is not searchable text unless it was explicitly indexed as image content; a plain-text pipeline has no path to surface it.
5. A. This is the exact vendor pairing the source material names for multimodal RAG.
6. B. This is the strongest case for routing-and-fusion: a correct answer that genuinely requires synthesizing evidence from more than one modality.
7. B. Accepting multiple input types in one call does not perform the coordination work of combining several modality-specific findings into one coherent answer; that is a separate, still-necessary step.
8. B. Without provenance, a final answer's claims cannot be traced back to their source, which matters for trust and for diagnosing a wrong answer later.

• A production agent typically routes each modality of input to the model best suited to it — vision, language, speech — rather than relying on one model to natively excel at all three.
• Fusion is the step that combines modality-specific outputs into one coherent answer; routing without fusion produces answers that quietly ignore part of the input while looking plausible.
• Text is usually the common representation every modality's output gets converted into, which is why a language model so often ends up as the fusion point.
• Multimodal RAG extends the canonical retrieval pipeline to non-text evidence, using an image-capable vector index (Milvus) alongside an orchestration toolkit (LlamaIndex) so retrieval can surface photos, charts, and scanned content — not just text chunks.
• On the exam, watch for scenarios that assume a single "multimodal model" removes the need to route and fuse, and for corpora described as containing non-text evidence that a plain-text RAG pipeline would silently fail to retrieve.

Getting a task's raw input to the right model, and its retrieved evidence into the right context, still assumes the agent has a way to reach out and act on the world beyond generating text — looking something up, calling an API, writing to a database.

Next: M2-03 covers building and connecting exactly those custom tools, APIs, and functions, including the input/output contract discipline that later makes it safe to wrap a tool call in the resilience patterns this module builds toward.