What is inference in AI?
Inference is what happens when a trained AI model is actually used — every chatbot reply, generated image, or spam score is one inference. Training builds the model by adjusting its weights; inference runs the finished model with weights frozen. Training is a one-time cost, inference is forever, which is why it dominates AI spending.
Why — the first-principles explanation
A neural network is a huge pile of numbers called weights — billions of them. Those numbers encode everything the model knows. There are only two things you can ever do with them: change them, or use them. Changing them is training. Using them is inference. That's the entire distinction, and everything else follows from it.
During training, data flows forward through the network, the model's output is compared against the right answer, and the error is pushed backward to nudge every weight slightly (backpropagation). Repeat trillions of times. This needs the forward pass and a backward pass, plus memory to store gradients and optimizer state for every weight — which is why training is so brutally expensive and needs the biggest possible clusters.
During inference, the weights are frozen. Data flows forward only. No backward pass, no gradients, no learning. The model is a fixed function: input in, output out. This is much cheaper per run — but it runs constantly. That asymmetry is the whole economics of AI. Training a frontier model is a large one-time bill. Inference is a bill that arrives with every single user request, forever, and it scales linearly with success. Industry estimates commonly put inference at roughly 60–80% of AI compute spend in production systems — that ratio varies a lot by company, so treat it as a rough shape, not a constant.
One consequence surprises people: the model does not learn from your conversation. When ChatGPT seems to remember what you said earlier, that isn't learning — the weights haven't budged a hair. Your earlier messages are simply being fed back in as part of the input each turn. It's a goldfish with an excellent notepad. Actual learning only happens if the lab runs a new training job later.
For language models specifically, inference has a distinctive shape: it's autoregressive. The model doesn't produce a sentence, it produces one token — roughly a word-piece — then re-reads everything including its own new token and produces the next. A 500-word answer is hundreds of sequential forward passes. This is why text streams out word by word, and why longer outputs cost proportionally more.
An example that makes it click
Think of a chef learning a recipe versus cooking it.
Learning is training. She makes the dish three thousand times, tastes each one, and adjusts — a little less salt, thirty seconds longer. Every failed attempt changes what's in her head. It takes months, wastes a mountain of ingredients, and happens once. Cooking is inference. A customer orders, she makes the dish, it goes out. She isn't experimenting now; her technique is fixed. One plate is far cheaper than the whole apprenticeship — but she cooks four hundred plates a night, every night, for years. Add it up and the cooking dwarfs the learning. And here's the part people miss: when a customer says "less salt please," she does it for that plate — but she hasn't changed the recipe in her head. Tomorrow's default is exactly the same. That's why an AI model doesn't remember you unless someone retrains it.
Key facts
- Inference is the stage where a trained model produces outputs with weights frozen; training is the stage where weights are updated via backpropagation.
- Inference requires only a forward pass; training requires a forward pass plus a backward pass and memory for gradients and optimizer state, making it far more resource-intensive per example.
- Industry estimates commonly place inference at roughly 60-80% of total AI compute spending in production systems — the exact split varies widely by company and workload.
- Large language model inference is autoregressive: the model generates one token at a time and re-processes the growing sequence for each new token, so a long answer is hundreds of sequential forward passes.
- The IEA reports data centres consumed about 415 TWh of electricity in 2024 — roughly 1.5% of global electricity — and projects about 945 TWh by 2030, with AI the most important driver of that growth.
- Inference does not change a model's weights, so a model cannot learn from your conversation; apparent memory comes from prior messages being re-fed as input.
▶ The 60-second explainer (script)
Inference is what happens when an AI model is actually used. Every chatbot reply, every generated image, every spam filter decision — that's one inference. Here's the cleanest way to understand it. A neural network is a giant pile of numbers called weights. Billions of them. They hold everything the model knows. And there are only two things you can ever do with those numbers: change them, or use them. Changing them is training. Using them is inference. That's it. In training, data flows forward, the model's answer gets compared to the right answer, and the error gets pushed backward to nudge every weight. Do that trillions of times. Expensive, slow, happens once. In inference, the weights are frozen. Data flows forward only. No learning. The model is just a fixed function — input in, output out. Cheap per run. But it runs every time anyone uses it, forever. That asymmetry is the entire economics of AI: training is one big bill, inference is a bill that arrives with every request and grows as you succeed. Estimates commonly put inference at sixty to eighty percent of production compute spend. And one consequence people always miss: the model does not learn from your conversation. When it seems to remember, your earlier messages are just being fed back in as input. The weights haven't moved a hair.
What authoritative sources say
People also ask
Is inference the same as prediction?
Effectively yes, in machine learning usage. Inference means running a trained model to get an output — classification, prediction, or generation. Note it's a different sense from "inference" in logic, which means deriving conclusions from premises.
Why is inference cheaper than training but still costs so much?
Per run it's far cheaper — one forward pass, no gradients. But training happens once and inference happens on every request forever. Volume beats unit cost, which is why inference typically dominates total AI compute spending.
Does the model learn from my chats during inference?
No. The weights are frozen during inference. Apparent memory within a conversation comes from your earlier messages being re-sent as input each turn. A model only changes if the developer runs a new training job.
Why does text appear word by word?
Language model inference is autoregressive — it generates one token at a time, then re-reads the whole sequence including its own output to pick the next one. You're watching hundreds of sequential forward passes happen live.
Can inference run on my own device?
Often, yes. Because inference skips gradients and optimizer memory, small and quantized models run on phones and laptops. Training the same model from scratch would still need a data centre.
The same question, asked other ways
- What is AI inference?1,600/mo