Setting up Google Colab and Jupyter notebooks for LLM work

A Jupyter notebook is a file (extension .ipynb) containing an ordered list of cells. A cell is either code, which you execute and which prints its output directly beneath itself, or markdown prose. The file stores the code, the prose, and the last output of each cell, which is why a notebook is simultaneously a program and a document. A background process called the kernel holds the live Python interpreter: your variables live in the kernel, not in the file.

Google Colab (Colaboratory) is a Google-hosted Jupyter environment. You open it in a browser, it gives you a virtual machine with Python and a large set of preinstalled data-science and deep-learning packages, and you can request that the machine have a GPU or other accelerator attached. Notebooks are stored in Google Drive by default. There is a free tier and paid tiers that offer more capable accelerators and longer sessions.

The vocabulary that matters, because the error messages use it:

The single most important sentence in this lesson: a Colab runtime is disposable by design. Treat it as a machine you rent by the hour and lose without warning, and every one of the traps below becomes obvious. Treat it as your computer, and you will lose work.

L1 — Intuition: rented machine, not your machine

When you open a Colab notebook and run a cell, Google allocates you a virtual machine somewhere, starts Python on it, streams your code to it and streams the output back. The notebook file lives in Drive; the machine does not belong to you and will be taken back. Anything you want to keep has to be written to a place that is not the machine.

Because it is rented, three things follow:

1. You get what you ask for, not what you want. The runtime comes with no accelerator unless you select one. A notebook that assumes a GPU will happily run on CPU, just far too slowly, or fail on the first .cuda() call.
2. Idleness ends the rental. Close the tab or leave it untouched and the session eventually ends. Long unattended jobs are the wrong shape for this environment.
3. Availability is not guaranteed. Free-tier accelerator access is allocated on availability and subject to usage limits, and both the limits and which accelerator types are offered change over time. Any number you read about them — here or anywhere — should be treated as subject to change, and confirmed in the product itself.

L2 — Mechanism: attach, verify, and the four-line check

Attaching an accelerator. In the Colab menu, open the runtime settings and change the hardware accelerator from the default to a GPU option, then let the runtime reconnect. This reassigns you a new machine, so anything you had installed or written before the change is gone. The correct order is therefore always: set the runtime type first, then install, then work.

Verifying it. Never assume. Run this before anything else, in every session:

# Cell 1 — the verification cell. Run this first, every session.
import sys, subprocess, torch

print("python  :", sys.version.split()[0])
print("torch   :", torch.__version__)
print("cuda ok :", torch.cuda.is_available())
if torch.cuda.is_available():
    print("device  :", torch.cuda.get_device_name(0))
    total = torch.cuda.get_device_properties(0).total_memory
    print(f"VRAM    : {total / 2**30:.1f} GiB total")
    print(f"in use  : {torch.cuda.memory_allocated() / 2**30:.2f} GiB")
else:
    print("NO GPU ATTACHED — set the runtime type before continuing")

Four things this tells you, all of which you need:

• Is a GPU attached at all (cuda ok). If this is False, stop. Nothing else you do will behave as expected.
• Which GPU you got. Free-tier allocation varies, so the answer differs between sessions, and your memory budget differs with it.
• How much VRAM it has. This is the number from M0.2 that decides what will fit. Note it, and budget against it with headroom — advertised capacity is gross.
• What is already allocated. In a fresh runtime this should be near zero. If it is not, something is still holding memory.

The companion command, which reads the same information from the driver rather than the framework:

!nvidia-smi

This prints the driver version, the CUDA version the driver supports, the GPU name, its memory used and total, its utilisation, and which processes hold memory. When your framework and nvidia-smi disagree about memory, the difference is the framework's caching allocator — the pool described in M0.2. When they disagree about whether a GPU exists at all, you have a driver or environment problem, not a code problem.

L3 — Persistence: what survives what

This is the table that saves the most time, because "it vanished" has three distinct causes and three distinct fixes.

Read the two "lost" columns carefully, because they are the whole lesson:

• Restarting the kernel loses variables but keeps installs. This is the cheap recovery, and it is step 1 of the OOM fix ladder from M0.2. Use it liberally.
• Anything that gives you a new machine loses installs and local files. So a pip install is not a one-time act — it belongs in a cell at the top of the notebook, so that re-running the notebook from a clean runtime reproduces the environment.
• Only Drive (or another external store) genuinely persists. Model checkpoints, evaluation sets, cached embeddings, results — all of it goes to Drive or it does not exist.

Mounting Drive:

from google.colab import drive
drive.mount('/content/drive')

# Then treat this path as the only durable location in the session.
WORKDIR = '/content/drive/MyDrive/nca-genl'
import os; os.makedirs(WORKDIR, exist_ok=True)
print("durable workdir:", WORKDIR)

One nuance that catches people: writing many small files to a mounted Drive is slow, because each write goes over the network. The idiomatic pattern is to work on the fast local disk (/content) and copy finished artefacts to Drive at checkpoints — accepting that anything not yet copied dies with the session.

Dependency installs. Two habits prevent most environment pain. First, put every install in one cell at the top, pinned:

!pip install -q "some-package==1.2.3" "another==4.5.6"

Pinning matters because hosted environments update their preinstalled stack on their own schedule, and an unpinned install can resolve differently next month and break a notebook that worked. Second, be aware that some installs require a kernel restart before the new version is importable — a hosted environment that already imported an older version of the same package in the same process will keep serving the old one. If a freshly installed package reports the wrong version, restart the kernel and re-run the import; do not debug the code.

The environment choice is a real decision with real trade-offs, and being able to justify it is worth more than the setup mechanics.

The decision rule for this course: if you do not own a GPU, use Colab's free tier and structure every notebook so that it can be re-run from scratch in a fresh runtime. If you do own a capable GPU, a local Jupyter install is better in every respect except setup effort, and the setup effort is real — driver, CUDA toolkit and framework versions must be mutually compatible, and getting that wrong produces exactly the "no kernel image is available" class of error named in M0.2.

A second comparison worth having, because the terms are used loosely:

The exam does not test Colab menu paths. What it can reasonably test, under official objective 4.4 — identify system data, hardware, or software components required to meet user needs — is whether you can match a described need to an appropriate environment. The table above is that mapping.

Rather than a memory calculation, the worked example here is the notebook header worth writing once and reusing, because it converts silent misconfiguration into an immediate, readable failure. Every line in it exists because of a specific failure mode named in this lesson.

# ── Cell 1: environment verification. Fails loudly rather than silently. ──────
import os, sys, platform

REQUIRE_GPU = True          # flip to False for CPU-only lessons

import torch
print(f"python   : {sys.version.split()[0]}  on {platform.system()}")
print(f"torch    : {torch.__version__}")
print(f"cuda avail: {torch.cuda.is_available()}")

if torch.cuda.is_available():
    props = torch.cuda.get_device_properties(0)
    total_gib = props.total_memory / 2**30
    print(f"device   : {props.name}")
    print(f"VRAM     : {total_gib:.1f} GiB total")
    # Budget with headroom: advertised capacity is gross, not net (see M0.2).
    print(f"budget   : plan for about {total_gib * 0.85:.1f} GiB usable")
elif REQUIRE_GPU:
    raise RuntimeError(
        "This notebook needs an accelerator. Change the runtime type, "
        "then re-run from Cell 1 — changing it reassigns the machine."
    )

# ── Durable output location. /content dies with the session. ──────────────────
try:
    from google.colab import drive
    drive.mount('/content/drive')
    WORKDIR = '/content/drive/MyDrive/nca-genl'
except Exception:
    WORKDIR = os.path.expanduser('~/nca-genl')     # local Jupyter fallback
os.makedirs(WORKDIR, exist_ok=True)
print("durable  :", WORKDIR)

# ── Pinned installs. Re-runnable from a clean runtime. ────────────────────────
# !pip install -q "package==x.y.z"

# ── Reproducibility. Not perfect on GPU, but far better than nothing. ────────
import random, numpy as np
SEED = 0
random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(SEED)
print("seeded   :", SEED)

Walk through why each block is there:

That last one deserves a caveat rather than a promise. Seeding makes runs far more comparable, but GPU execution involves nondeterministic reduction orders in some kernels, so bitwise-identical results across runs are not guaranteed by seeding alone. The honest claim is: seed everything, and still expect small numerical differences. When you later compare two prompts or two models, that residual noise is precisely why a difference of a fraction of a percent on twenty examples means nothing — a point 01-08 and the experimentation module make properly.

And a related judgement that appears in real work: notebooks are excellent for exploration and poor as the final home for code. The mature pattern is to explore in a notebook, then move anything you will run more than a few times into an importable module with tests, keeping the notebook as the narrative front end. Every serious criticism of notebooks — hidden state, unreproducible execution order, untestable logic, painful diffs — is a criticism of using them as production code, not of using them to think.

The hidden-state problem is worth one concrete illustration because it is the most under-appreciated hazard of the format:

Cell [1]   x = 10
Cell [2]   x = x + 5      # x is now 15
Cell [2]   x = x + 5      # ran again: x is now 20
Cell [3]   print(x)       # prints 20

Nothing in the saved notebook records that cell 2 ran twice. Reopening the file and reading top to bottom, you would predict 15. This is exactly how a "working" notebook fails for the next person, and the defence is the discipline of restarting the kernel and running all cells from a clean state before you believe any result.

Colab menu paths are not exam content. Three things in this lesson genuinely are, and it is worth separating them from the mechanics.

First, environment-to-need matching. Official objective 4.4 asks you to identify system data, hardware, or software components required to meet user needs, and objective 1.1 / 4.1 concerns assisting in deployment and evaluation of scalability, performance and reliability under supervision. A question that describes a workload — long unattended training, sensitive data, a quick exploratory analysis — and asks which environment suits it is squarely inside that scope. Section 3's table is the answer key.

Second, the Python ecosystem's identity. Objectives 1.6, 1.10 and 4.6 name Python packages explicitly — spaCy, NumPy, Keras, vector databases — and the Core ML module devotes a lesson to the identity and purpose of each tool in the ecosystem. A notebook is where that ecosystem is exercised. Knowing that Jupyter is the notebook technology, that a kernel holds state, and that Colab is a hosted flavour of it is baseline vocabulary for every one of those objectives.

Third, reproducibility and monitoring discipline. Official objective 4.5 is monitor functioning of data collection, experiments, and other software processes, and 2.3 / 3.3 concern conducting data analysis under supervision. Hidden notebook state, unpinned dependencies and unseeded runs are the concrete, everyday form of the experimental-hygiene failures that the Experimentation domain — 22% of the blueprint — tests abstractly. When a later lesson tells you a benchmark result is contaminated or an A/B comparison is invalid, the mechanism is very often something as mundane as cells run out of order.

Where it feeds forward:

This is a short lesson by design — a recognition item rather than a mechanism to master. But it is the lesson whose absence costs the most hours, because every hour lost to a vanished install or an unattached GPU is an hour not spent on the 30%-weighted Core ML domain.

The last two are the ones with consequences beyond wasted time. Quota assumptions are a planning risk: build every lab so that losing the session costs you minutes, not the whole run. Data governance is a policy risk, and it connects directly to the Trustworthy AI domain's privacy and consent objectives — moving personal data to a third-party environment to save yourself a setup afternoon is a decision with a compliance dimension, not just a convenience one.

No. The free tier is sufficient for the labs in this course, which are deliberately designed to be short and interruptible, and much of the course's work does not need an accelerator at all — tokenisation, chunking, evaluation scripts and small-corpus retrieval are comfortable CPU tasks. Paid tiers buy longer sessions and more capable accelerators, which matters if you extend the labs into a larger fine-tuning experiment. Treat any specific quota or limit you read as subject to change, including in this lesson: the durable strategy is to checkpoint to Drive and structure notebooks so that losing a session costs you minutes.

Two different causes, and they need different responses. If torch.cuda.is_available() returns False right after you changed the runtime type, the most likely cause is that the runtime has not actually reconnected with the new type, or that you are looking at a stale kernel — re-run from the first cell. If the environment itself refuses to give you an accelerator, that is availability and usage limits: free-tier accelerator access is allocated on availability, is subject to quotas, and both are subject to change without notice. There is no code fix for the second case. The practical response is to build notebooks that detect the situation loudly at cell one, fall back to CPU for the parts that can, and wait or use a paid tier for the parts that cannot.

Restarting the kernel kills the Python process, so every variable is lost — but the machine, its installed packages, and its local files remain. Resetting the runtime, or being reassigned a new one, gives you a different machine: variables, packages and local files are all gone, and only your Drive-mounted files and the notebook document itself survive. This distinction is the reason a kernel restart is a cheap first move when you hit an out-of-memory error (it releases everything pinned by stale references, as described in M0.2), while a factory reset means re-running your install cell.

No, not across runtimes. A pip install lives only as long as the machine you installed it on, so an idle timeout, a runtime reset, or changing the runtime type all take your installs with them. The consequence is a habit rather than a workaround: keep every install in a single pinned cell at the top of the notebook, so re-running from a clean runtime rebuilds the environment exactly. Pinning specifically — not just installing — matters because hosted environments update their preinstalled stack on their own schedule, and an unpinned install can resolve to a different version next session and break code that previously worked.

Both, for different jobs. Notebooks are the right format for exploration, analysis with lots of intermediate output, and anything you will read as a narrative. Scripts and importable modules are the right format for anything you will run repeatedly, test, schedule, or put in version control. The failure mode is using a notebook as the permanent home for logic: hidden state, execution order that the file does not record, untestable functions and unreadable diffs are all real costs. The mature pattern is to think in a notebook and then move the stable parts into modules, leaving the notebook as the front end that calls them.

Two complementary ways, and it is worth running both. From the framework, torch.cuda.get_device_properties(0).total_memory gives total VRAM in bytes, and torch.cuda.memory_allocated() gives what your process currently holds. From the driver, !nvidia-smi prints the GPU name, driver and CUDA versions, memory used and total, utilisation, and which processes hold memory. When these two disagree about memory, the gap is the framework's caching allocator, which holds freed memory in a pool rather than returning it — the mechanism described in M0.2. When they disagree about whether a GPU exists at all, you have an environment problem rather than a code problem.

1. Colab is a hosted Jupyter notebook with optional GPU attachment, which makes it the shortest path to a working LLM environment for anyone without their own hardware.
2. A runtime is disposable by design. Treat it as a rented machine you will lose without warning, and every trap in this lesson becomes predictable.
3. The accelerator is not attached by default. Set the runtime type before you install anything, because changing it reassigns the machine.
4. Verify, never assume. One cell printing Python, framework, cuda_available, device name and total VRAM — raising loudly if there is no GPU — pays for itself the first time it fires.
5. Three different "it vanished" events. Kernel restart loses variables. Runtime reset loses installs and local files. Only Drive or another external store genuinely persists.
6. Pin your installs and keep them in one top cell, so a clean runtime rebuilds the environment and a future reader knows what the environment was.
7. Restart and run all before you believe a result. Notebooks store outputs and hide execution order; a saved output is not evidence that the current code produced it.
8. Seed everything, and still expect small numerical differences — GPU kernels are not all deterministic, and that residual noise is why tiny differences on tiny eval sets mean nothing.
9. Notebooks are for thinking, modules are for running. Move anything you will execute repeatedly into importable, testable code.
10. Treat every published quota as subject to change. Free-tier availability, limits and accelerator types move; design labs to be interruptible and to checkpoint.
11. Hosted notebooks are a data-governance decision, not just a convenience. Your data executes on someone else's infrastructure — which is a Trustworthy AI question as much as a setup one.
12. Environment-to-need matching is the examinable part, under objective 4.4. Menu paths are not; the decision table in section 3 is.

You now have somewhere to run code and a way to know it is really running where you think. What you cannot yet read is the output. A language model does not emit a word; it emits a probability distribution over its entire vocabulary, and every generation control you will ever touch — temperature, top-k, top-p — is a manipulation of that distribution. Every metric that judges a model, from cross-entropy to perplexity, is a statement about it. And every claim that one prompt beat another rests on whether a difference is larger than the variance you would expect by chance.

Next: M0.3 Probability basics: distributions, variance, and expectation — what a probability distribution over a vocabulary actually is, how expectation and variance are computed, why softmax turns arbitrary scores into probabilities, and why a difference of two examples in twenty is not evidence of anything.