NVIDIA · NCA-GENL

The infrastructure thread

The plumbing underneath the ideas: environment and hardware setup in M0, the document-ingestion pipeline in M06, and deployment infrastructure in M12 — the parts of the course that are systems work, not concepts.

T3 · 35 lessons across 4 modules

  1. M0M0.1Linear algebra for LLMs: dot product, cosine similarity, and NumPy shapes
  2. M0M0.2How GPUs work and what happens during a training run
  3. M0M0.2aSetting up Google Colab and Jupyter notebooks
  4. M0M0.3Probability basics: distributions, variance, and expectation
  5. M0M0.4Units and order-of-magnitude estimation (GB vs GiB)
  6. M0606-01Document parsing for RAG: PDFs, tables, and silent failures
  7. M0606-02Chunking strategies for RAG: fixed, recursive, and semantic
  8. M0606-03Metadata in RAG: what to embed versus what to return
  9. M0606-04Deduplication and corpus cleaning for RAG
  10. M0707-01Sparse retrieval and BM25 keyword search
  11. M0707-02Dense retrieval with embeddings
  12. M0707-03Limits of embedding search: negation, recency, and authority
  13. M0707-04Vector databases and ANN indexes (HNSW, IVF)
  14. M0707-05Access control and permissions in RAG retrieval
  15. M0707-06Hybrid search: combining keyword and vector retrieval
  16. M0707-07Reranking with a cross-encoder
  17. M0707-08Assembling context: chunk order and the lost-in-the-middle problem
  18. M0707-09The complete RAG pipeline, stage by stage
  19. M0707-10Debugging RAG: retrieval failure versus generation failure
  20. M0707-11Grounding, citations, and letting a model say I don't know
  21. M0707-12When RAG is the wrong tool
  22. M1212-01Numeric precision: FP32, TF32, FP16, BF16, and INT8
  23. M1212-02Quantization: PTQ vs QAT and recovering accuracy
  24. M1212-03Reading loss curves to diagnose a training run
  25. M1212-04Distributed training: data parallelism, AllReduce, and NCCL
  26. M1212-05The KV cache and why LLM generation is memory-bound
  27. M1212-06Batching for inference: static, dynamic, and continuous
  28. M1212-07PagedAttention and vLLM: virtual memory for the KV cache
  29. M1212-08ONNX, TensorRT, and TensorRT-LLM: compiling a model for inference
  30. M1212-09LLM cost per million tokens, per request, and per month
  31. M1212-10Latency and throughput: TTFT, tokens per second, and p95
  32. M1212-11Multi-turn chat history and query rewriting
  33. M1212-12Keeping a vector index fresh: re-embedding and migration
  34. M1212-13Deploying with NVIDIA Triton Inference Server and NIM
  35. M1212-14Monitoring an LLM in production and detecting drift

Part of the throughlines running across the NCA-GENL prep course.