NVIDIA · NCA-GENM

The generative pipeline thread

Building one text-to-image system end to end. Opens in M1 with multimodal loss functions (contrastive loss ahead of CLIP), runs through M3's diffusion/DDPM forward-and-reverse process and FID scoring, through M4's CLIP embedding space, and closes in M6 wiring CLIP conditioning into a diffusion U-Net served by TensorRT and Triton.

NCAM-T1 · 34 lessons across 4 modules

  1. M1M1-01Machine learning fundamentals: learning paradigms, feature engineering, and cross-validation
  2. M1M1-02Overfitting, underfitting, and the bias-variance tradeoff
  3. M1M1-03Model comparison metrics: accuracy, precision, recall, F1, ROC-AUC, MAE, MSE, R²
  4. M1M1-04Deep learning frameworks: TensorFlow, PyTorch, and Keras
  5. M1M1-05Neural network basics: neurons, activation functions, and the training loop
  6. M1M1-06Convolutions and the building blocks of vision models
  7. M1M1-07Nonsequential networks and residual connections
  8. M1M1-08Multimodal loss functions: cross-entropy, contrastive, reconstruction, adversarial, and composite
  9. M1M1-09Training stability in multimodal settings: normalization, LR warmup, loss weighting, gradient clipping
  10. M1M1-10Multimodal transfer learning: pretrained encoders, full fine-tuning vs. parameter-efficient adaptation
  11. M1M1-11Model fusion and orchestration: early, intermediate, and late fusion; modality vs. agent orchestration
  12. M1M1-12Prompt engineering fundamentals and emerging multimodal trends
  13. M3M3-01Experiment design: one variable at a time, A/B testing, fixed evaluation sets, reproducibility
  14. M3M3-02Managing and preprocessing multimodal data from multiple sources
  15. M3M3-03Diffusion models: forward noising and reverse denoising (DDPM)
  16. M3M3-04Controlling diffusion output with context embeddings
  17. M3M3-05GANs and the generator-discriminator min-max game
  18. M3M3-06Evaluating generative image quality: Fréchet Inception Distance (FID)
  19. M3M3-07Conversational AI on Riva: ASR, NLP/LLM, and TTS pipelines
  20. M3M3-08Evaluation metrics by task: classification, regression, text generation, image generation, and RAG
  21. M3M3-09Explainability and testing data/model quality and consistency
  22. M4M4-01Data modalities and making them neural-network ready: text, image, audio, time-series, geospatial
  23. M4M4-02Model fusion in depth: early, intermediate, and late fusion tradeoffs
  24. M4M4-03CLIP and shared multimodal embedding spaces
  25. M4M4-04Handling missing or incomplete modalities
  26. M4M4-05Autoencoders and anomaly detection across modalities
  27. M4M4-06Application patterns: RAG, chatbots, and summarizers over multimodal data
  28. M4M4-07Python multimodal tooling: NumPy, spaCy, Keras, and vector databases
  29. M6M6-01U-Net architecture: encoder-decoder structure and skip connections
  30. M6M6-02The U-Net as diffusion denoising backbone and as an autoencoder
  31. M6M6-03CLIP plus diffusion: building a text-to-image pipeline
  32. M6M6-04NVIDIA SDKs: NeMo, Riva, Triton, ACE, cuDNN, and AI Blueprints/VIA
  33. M6M6-05Prompt engineering for generative systems, and software quality practices
  34. M6M6-06Putting it together: a text-to-image service end to end

Part of the throughlines running across the NCA-GENM prep course.