AI energy use, carbon, and efficient inference

Energy-consciousness is the practice of treating the compute an AI system consumes as a cost with ethical weight, not merely an operational line item, and designing to reduce compute per unit of delivered value.

Three components, and being able to separate them is the whole of the concept:

The engineering point that follows is worth stating plainly, because it reframes where effort belongs. Training is a one-time cost; inference is a cost per request, forever. For a system serving meaningful traffic over years, the accumulated inference compute becomes the dominant term, and it is the term an associate engineer actually controls. You will probably never decide whether to pretrain a foundation model. You will decide, every week, whether a task needs a 70-billion-parameter model or whether a much smaller one passed the eval — and that decision, multiplied by every request for the service's whole life, is the largest energy lever you personally have.

Two adjacent ideas that belong here and that questions conflate with energy:

Cost is a proxy for energy, and a good one. Cost per thousand tokens tracks compute closely enough that the cost calculation from 12-09 doubles as an energy proxy. Where an exam question offers "measure and reduce cost per request" as a way to address energy-consciousness, that is defensible engineering rather than a dodge — and it has the practical advantage that you can actually measure it.

Efficiency is not only environmental. Every lever in this lesson also reduces latency, increases throughput, or lowers spend. That alignment is unusual in ethics topics: the energy-conscious choice is nearly always also the cheaper and faster one, which is why this pillar produces less friction in practice than the others.

L1 — The intuition: energy is compute, and compute is model size times tokens times requests

A rough mental model, correct in structure even where the constants are uncertain:

energy  ≈  (work per token)  ×  (tokens per request)  ×  (requests)  ÷  (efficiency of the hardware path)

work per token   grows with model size and with numeric precision
tokens/request   = prompt tokens + generated tokens (and prompts are usually the larger half in RAG)
requests         includes retries, redundant calls, and the ones a cache could have answered
efficiency       is how much of the accelerator you actually used per unit of work

Every lever in the next section attacks one of those four terms. That is a complete taxonomy, and it is a better answer to "how do you make AI more energy-efficient" than any list of tips, because it tells you where a proposed optimisation acts.

L2 — The mechanism: the levers, in order of typical leverage

Lever 1 — Right-size the model (attacks: work per token). The largest single lever. If a smaller model passes your evaluation set, using a larger one spends compute on every request forever for no measured gain. This is the practical meaning of energy-consciousness for an application engineer, and it depends entirely on having an evaluation set good enough to tell you when smaller is sufficient — which is why 01-08 and 09-01 are prerequisites for taking this pillar seriously. Without an eval set you cannot make the energy-conscious choice, because you cannot tell whether the small model was good enough. That connection between the measurement thread and the energy pillar is the most useful non-obvious idea in this lesson.

Lever 2 — Lower numeric precision (attacks: work per token). Serving in FP16 or BF16 rather than FP32, and quantizing to INT8 or FP8 where accuracy permits, reduces both memory movement and arithmetic work. 12-01 and 12-02 cover the mechanics and the accuracy trade-off, including the distinction between post-training quantization and quantization-aware training. The relevance here: quantization is an energy lever that usually costs little or no measured quality, verified against your eval set.

Lever 3 — Batch and use the accelerator properly (attacks: efficiency). A GPU serving one request at a time is mostly idle silicon drawing power. Dynamic and continuous batching raise the useful work per joule substantially. 12-06 covers the mechanics; 12-10 covers the latency trade-off, which is real — batching improves throughput and can worsen the tail latency users notice.

Lever 4 — Cache (attacks: requests). A request answered from cache consumes no generation compute at all. Exact-match caching for repeated queries, embedding caching so a document is not re-embedded on every ingestion run, and KV-cache reuse within a session (12-05) all remove work rather than making it cheaper. Removing work beats optimising it.

Lever 5 — Shorten the prompt (attacks: tokens per request). In a RAG system the prompt is frequently much larger than the output — a long system prompt plus several retrieved chunks. Retrieving three well-chosen chunks instead of ten wastes less compute and usually produces a better answer, since the lost-in-the-middle effect from 07-08 means padding context with marginal passages can actively hurt. Reranking (07-07) to select fewer, better chunks is simultaneously a quality lever and an energy lever.

Lever 6 — Generate fewer tokens (attacks: tokens per request). Set sensible maximum output lengths. Ask for the answer rather than the answer plus a restatement of the question plus a summary. Note that chain-of-thought prompting (05-03) buys accuracy on hard reasoning tasks by spending more tokens, which makes it a deliberate accuracy-for-energy trade rather than a free win — apply it where it measurably helps, not by default.

Lever 7 — Retrieve instead of retrain (attacks: training energy). Updating a corpus is nearly free; retraining or re-fine-tuning a model to absorb new knowledge is a training run. When the requirement is fresh knowledge, retrieval is the energy-conscious architecture as well as the cheaper and faster one. This is the same conclusion 11-08 reaches on cost grounds and 13-05 reaches on privacy grounds — three independent arguments converging on one architecture, which is the most robust kind of design decision.

Lever 8 — Use PEFT rather than full fine-tuning when you must adapt (attacks: training energy). LoRA and other parameter-efficient methods (11-05) update a small fraction of parameters and need far less memory and compute than a full fine-tune, and 11-04 makes the memory arithmetic explicit.

Lever 9 — Do not train what you can prompt (attacks: training energy). The customization ladder exists partly for this reason: prompting costs nothing to set up, retrieval costs an index, tuning costs a training run. Climb only as far as the requirement forces.

Lever 10 — Compile and optimise the serving path (attacks: efficiency). TensorRT and TensorRT-LLM (12-08) exist to make the same model produce the same outputs with less work per token, through fusion, precision calibration, and LLM-specific mechanisms. Optimised serving is an efficiency lever available without any quality trade at all.

L3 — Why the public figures are contested, and how to talk about it

You will encounter confident numbers. Here is why they disagree, stated at a level you can reason with rather than as a list of citations:

Boundary problems. Does the figure count the accelerator's draw, the whole server, or the facility including cooling and power-delivery losses? Facility overhead is a multiplier on the compute figure and is often silently included or excluded. Does it count the hardware's manufacturing footprint? Does it include the network and storage the workload used?

Amortization problems. Training energy divided over how many inferences? A model serving billions of requests amortizes its training cost to something negligible per request; the same model serving a thousand requests does not. Reported per-query figures embed an assumption about lifetime volume that is rarely stated.

Attribution problems. Carbon depends on the electricity mix at the time and place of the run, which varies by grid, by hour, and by season. The same compute in two regions has materially different emissions. Averages hide this, and "carbon-free" claims may rest on accounting instruments rather than on the physical power drawn.

Comparability problems. "A query to an LLM" is not one thing. Model size, quantization, output length, batch efficiency, and serving stack each move the number by a large factor. A figure for one configuration is not a figure for the category.

Source problems. Much of the disclosed data comes from the organisations selling the compute or the models, using their own methodologies, with limited independent verification. That does not make it wrong; it makes it non-comparable and not independently checkable — which is exactly what the verifiability property in the same scope statement asks for and does not get.

Velocity problems. Hardware efficiency per unit of work, serving-stack efficiency, and model efficiency at a given quality level are all changing quickly. A figure two years old may be wrong by a large factor in either direction.

So how do you talk about it responsibly? Three moves:

1. Report what you measured, with the boundary stated. "Our service used X GPU-hours on this instance type over this month, serving Y requests" is a claim you can stand behind and someone can check. It is worth more than a borrowed number with more digits.
2. Compare, do not absolutize. "Switching to the smaller model cut GPU-hours per thousand requests by roughly half at equal eval score" is robust to every boundary problem above, because both sides of the comparison share the same boundary. Relative claims survive; absolute claims do not.
3. Say when you do not know. "Published per-query energy figures vary widely by methodology and we have not measured ours" is a better answer in an interview, an audit, or a design review than a confident wrong number. It is also this course's own posture, which is why no figure appears in this lesson.

The row that decides questions: for a production service with real traffic, inference is the term you manage, and the reason is arithmetic rather than ideology — a per-request cost multiplied by a large number of requests eventually exceeds a fixed one-time cost.

A constructed scenario. All numbers below are invented for teaching — they illustrate the shape of the reasoning, not measured results from any real system.

The system. An internal documentation assistant over a company's engineering docs. RAG: retrieve ten chunks, stuff them into a long system prompt, generate an answer with a large general-purpose model at FP16, no cache, one request per GPU pass. It answers about 20,000 questions a month. The team has a 120-item evaluation set built the way 09-01 describes, and the current configuration scores 0.86 on their faithfulness-and-helpfulness rubric.

Baseline, in the only unit that survives every boundary argument: GPU-hours per thousand answered questions.

Constructed illustration — invented numbers

Baseline configuration
  prompt tokens/request      ~ 4,200   (long system prompt + 10 chunks)
  generated tokens/request   ~   320
  model                      large, FP16
  batching                   none (batch size 1)
  cache                      none
  eval score                 0.86
  measured                   9.0 GPU-hours per 1,000 questions   ← the baseline

Now apply the levers one at a time, re-running the eval set after each — which is the discipline that makes this an engineering exercise instead of guesswork.

Step 1 — Rerank and retrieve fewer chunks. Retrieve 20 candidates, rerank with a cross-encoder (07-07), pass the top 3. Prompt tokens fall from ~4,200 to ~1,600.

eval score after:  0.87   (up — fewer marginal chunks, less lost-in-the-middle dilution)
measured:          5.4 GPU-hours / 1,000   (−40%)

The score went up. This is the most common real result and the most important one to expect: a shorter, better-selected context frequently improves quality while cutting compute. Energy-consciousness and quality are not opposed here.

Step 2 — Cap and tighten output. Remove the instruction that asks the model to restate the question, and set a maximum output length matched to what a good answer actually needs. Generated tokens fall from ~320 to ~180.

eval score after:  0.87   (unchanged)
measured:          4.6 GPU-hours / 1,000   (−15% more)

Step 3 — Enable continuous batching. No change to the model or the prompt; the accelerator simply stops idling between requests.

eval score after:  0.87   (unchanged — same model, same outputs)
measured:          2.3 GPU-hours / 1,000   (−50% more)
p95 latency:       slightly worse under load — the documented trade-off

Batching is the largest single win in this constructed example and it costs nothing in output quality, because it changes when work happens rather than what work is done. It does move tail latency, which is exactly the trade 12-10 describes and which you should disclose rather than bury.

Step 4 — Quantize to INT8, verified against the eval set.

eval score after:  0.86   (down 0.01 — within the team's accepted tolerance)
measured:          1.6 GPU-hours / 1,000   (−30% more)

Note the discipline: the quantization decision is made by the eval set, not by a vendor claim about accuracy retention. One point of rubric score is a judgement the team records with a reason.

Step 5 — Try the smaller model. With a good retrieval stage doing the knowledge work, the generator's job is narrower than it was.

eval score after:  0.85   (down 0.01 more)
measured:          0.6 GPU-hours / 1,000   (−60% more)

Step 6 — Cache exact repeats. Roughly a fifth of questions in this constructed corpus are near-duplicates of earlier ones.

eval score after:  0.85
measured:          0.5 GPU-hours / 1,000

Result of the constructed exercise.

Baseline:  9.0 GPU-hours per 1,000 questions at eval 0.86
Final:     0.5 GPU-hours per 1,000 questions at eval 0.85

≈ 18× less compute per answered question, for 0.01 of rubric score.
(Invented figures, illustrating the shape of the result — not a measurement.)

Five things to take from this, none of which depend on the invented numbers.

The levers compound multiplicatively. Each stage acts on a different term in the energy model, so their effects multiply rather than add. That is why a stack of unglamorous optimisations produces a large factor.

The eval set is the enabling instrument. Every step was a decision about whether a compute reduction cost measurable quality, and none of those decisions is available to a team without an eval set. Energy-consciousness is downstream of measurement — the same thread 01-08 opened.

The biggest wins were free. Fewer, better chunks improved quality. Batching changed nothing about the output. Caching removed work entirely. Only quantization and the smaller model cost anything measurable, and together they cost one point of rubric score.

Cost fell in proportion. The same reduction shows up in the cost calculation from 12-09. If your organisation will not fund an energy argument, it will fund the identical spend argument — and it is the same work.

Document the trade-offs you accepted. Two points of the change cost measurable quality and one worsened tail latency. Those belong in the model card's evaluation and limitations sections (13-06), because a quality reduction accepted for efficiency is a design decision someone downstream deserves to know about.

Three reasons this topic is examinable despite having no formal objective of its own.

First, the blueprint's scope statement names it. The Trustworthy AI domain is defined as the creation and assessment of "ethical, energy-conscious, and reliable" AI systems that are "transparent, fair, and verifiable." Energy-consciousness is on that list, and a question asking which properties the domain covers can key on it. It is the item candidates most reliably cannot produce when asked to complete the list, which makes it disproportionately worth the two minutes it takes to memorize.

Second, objective 5.3 — describe how to use NVIDIA and other technologies to improve AI trustworthiness — reaches efficiency technologies, because the NVIDIA stack's optimisation components exist to do the same work with less compute. TensorRT and TensorRT-LLM for compilation, quantization support, Triton's dynamic batching, NIM's pre-optimised engines: all are efficiency technologies, and all are already on your stack map from Module 12.

Third, the topic reappears throughout the exam under other names. Every question about quantization, batching, precision, model sizing, or cost per token is an efficiency question. The energy framing is the ethical clothing on engineering you have already learned.

Question phrasings to expect:

1. "According to the exam's scope for trustworthy AI, which property is required in addition to ethics and reliability?" — energy-conscious.
2. "Which reduces inference energy without changing model outputs?" — batching, or compiling the serving path. Both raise efficiency without altering what is computed.
3. "Which consumes more energy over a widely used model's lifetime — training or inference?" — inference, because it is a per-request cost accumulated over years.
4. "A team needs the model to have current knowledge. Which is the more resource-efficient approach: retraining periodically or retrieval?" — retrieval.
5. "Which technique reduces memory and compute per token at some risk to accuracy?" — quantization.
6. "How should an organisation report the energy footprint of its AI service?" — measure its own consumption with the boundary stated; published per-query figures vary by methodology.

Distractor families:

Training is a large one-time cost; inference is a smaller per-request cost that accumulates without limit. Which one dominates therefore depends entirely on request volume over the deployment's lifetime — and for a widely used production service over years, inference wins, because a fixed number is eventually exceeded by a per-request number multiplied by a large count.

The design consequences are what matter:

• You mostly do not control training. If you use a foundation model someone else pretrained, that energy was spent before your project existed. Your training-side lever is whether to add more training — a fine-tune, a continued pretraining run — and to prefer PEFT or retrieval when you can.
• You control inference almost completely. Model size, precision, batching, caching, prompt length, output length, and serving stack are all yours, and they compound.
• Amortization cuts both ways. A model serving enormous volume spreads its training cost thin per request. A model trained for a project that then serves a thousand requests has an appalling per-request footprint dominated by training — which is a real argument against fine-tuning a model for a low-volume use case where prompting or retrieval would have done.

Resist the temptation to attach numbers to any of that. The structure is what is examinable; the magnitudes are what the field disagrees about.

Because the available figures are not the kind of thing a study guide can responsibly assert as fact, and because getting this wrong has a specific cost: you would answer an exam question, or a real design review, with a number you could not defend.

The specific defects, restated compactly: inconsistent boundaries (chip, server, or facility; with or without embodied hardware), hidden amortization assumptions (training spread over how many inferences), grid-dependent carbon (same compute, different emissions by place and hour), non-comparable configurations ("an LLM query" spans orders of magnitude), vendor-reported provenance with limited independent verification, and rapid change in hardware, serving stacks, and model efficiency.

Note that the domain's own scope statement asks for verifiable systems, and most published AI energy figures are precisely not verifiable by an outside party. Applying the framework's own standard to the framework's own subject matter is not evasion; it is consistency.

What this course does instead: it names the direction (large-scale AI is energy-intensive, and inference at scale accumulates), names the levers (this lesson's list), insists you measure your own system in a stated unit with a stated boundary, and prefers relative before-and-after claims over absolute ones. If you need a concrete number for a document, generate it from your own measurement and label the boundary — as §4 does, where every figure is explicitly invented for teaching.

Every optimisation component in the stack is, viewed from this pillar, an energy technology. The mapping, at the identity depth the exam tests:

Two cautions. Name what each is for; do not invent numbers for it. The exam tests identity and fit — TensorRT-LLM is the LLM-specific optimiser, Triton batches and serves, NIM packages an optimised deployment — and a claimed speedup multiple is exactly the kind of vendor-reported figure this lesson refuses. And the largest lever is still architectural: choosing the smallest model that passes your eval, retrieving rather than retraining, and not generating tokens nobody needed. No compiler recovers the compute wasted by an oversized model answering an over-stuffed prompt.

• Energy-consciousness — treating compute consumption as an ethical design dimension; named in the NCA-GENL Trustworthy AI scope statement.
• Training energy — the one-time compute of pretraining, continued pretraining, fine-tuning, or alignment runs.
• Inference energy — the per-request compute of serving, accumulated over the deployment's life.
• Embodied energy — the footprint of manufacturing the hardware.
• Facility overhead — cooling and power-delivery losses, a multiplier on compute energy; one of the boundary ambiguities in reported figures.
• Amortization (of training energy) — spreading a training run's cost over the number of inferences the model serves; a hidden assumption in per-query figures.
• Right-sizing — selecting the smallest model that passes your evaluation set.
• Utilisation — how much of the accelerator's capacity is doing useful work; what batching improves.
• GPU-hours per thousand requests — a measurable, boundary-explicit efficiency unit suitable for before-and-after comparison.
• Relative reporting — presenting before-and-after comparisons rather than absolute footprint claims, so shared boundary assumptions cancel.
• Boundary (of an energy figure) — exactly what was counted: chip, server, facility, embodied hardware, network, storage.

1. Energy-consciousness is a named property in the domain's scope statement: ethical, energy-conscious, reliable, transparent, fair, verifiable. It is the item most candidates cannot produce.
2. Energy ≈ work per token × tokens per request × requests ÷ efficiency. Every lever attacks one of those four terms.
3. Training is one-time; inference accumulates. For a widely used service, inference dominates — and inference is what you control.
4. The biggest lever is right-sizing the model, and it is only available to a team with an evaluation set. Energy-consciousness is downstream of measurement.
5. Batching and compilation cost nothing in output quality — they change utilisation and operation count, not what is computed.
6. In RAG the prompt is usually the larger token half. Rerank to fewer, better chunks; it often improves quality at the same time.
7. Retrieve rather than retrain for fresh knowledge; use PEFT rather than full fine-tuning when you must adapt.
8. Chain-of-thought is an accuracy-for-tokens trade, not a free improvement. Apply it where measured to help.
9. Do not quote borrowed energy or carbon figures. They differ in boundary, amortization, grid, configuration, and provenance, and they change fast.
10. Measure your own system in a stated unit and report relative improvements. "GPU-hours per thousand requests, halved" is defensible; "0.3 Wh per query" usually is not.
11. Document quality you traded for efficiency in the model card, so nobody downstream is surprised.
12. Efficiency is also cost and latency. The energy-conscious choice is nearly always the cheap and fast one, which is why this pillar meets the least resistance in practice.

You now have the whole domain in pieces: four pillars and three added properties, a guardrail layer that logs, an injection threat model, a bias measurement method, a privacy architecture, a transparency document set, and an efficiency lever list. What you do not yet have is the single deliverable that puts them in one place, in the order you would actually work through them, for a specific service — the thing you could hand to a manager, an auditor, or an interviewer and have them see that you checked.

Next: 13-08 builds a trustworthy AI checklist for your own LLM service — every pillar mapped to the artifact that evidences it, in a form you can fill in for a real system and show an employer.