How does Stable Diffusion work?
Stable Diffusion starts with a field of random noise and removes it step by step, guided by your text prompt, until a matching image appears. A text encoder turns your words into numbers, a neural network predicts and subtracts noise over about 20 to 50 steps, and a decoder turns the cleaned-up result into the final picture.
Why — the first-principles explanation
The core idea is learning to undo noise. During training, the model repeatedly took a clean image, added random static, and practiced predicting what static was added so it could remove it. Do this billions of times across a huge dataset, and the model learns what real images look like well enough to reconstruct one from pure noise.
Generating an image reverses that process. You start with a random noise pattern and ask the model, step by step, 'what noise should I remove to get closer to the description?' After roughly 20 to 50 denoising steps, the noise has been sculpted into a coherent image. Each step nudges the picture toward your prompt.
Your text prompt steers everything. A component called a text encoder (CLIP) converts your words into a list of numbers that captures their meaning. At every denoising step, the model consults these numbers so the emerging image matches 'a red bicycle in the rain' rather than something random. A setting called CFG scale controls how strictly it obeys the prompt.
The efficiency secret is latent space. Working on full-resolution pixels would be slow, so Stable Diffusion compresses the image into a small coded form using a component called a VAE, does all the denoising there, then the VAE decodes the result back into a full picture. This compression is why it runs on a normal graphics card instead of a supercomputer.
An example that makes it click
Imagine a photo buried under a thick layer of sand. You know it's a photo of 'a red bicycle' because someone told you. You gently brush away sand a little at a time, checking after each brush whether the shape looks more like a bicycle, and after a few dozen passes the picture emerges.
Stable Diffusion does the same, except the sand is random noise, the 'someone telling you' is your text prompt, and it makes about thirty brushing passes. To go faster, it works on a tiny thumbnail first and only blows it up to full size at the very end.
Key facts
- Stable Diffusion is a latent diffusion model that generates images by iteratively removing noise, typically over 20 to 50 steps.
- A CLIP text encoder converts the prompt into numerical embeddings that guide each denoising step.
- Denoising happens in a compressed latent space, not on full-resolution pixels, for speed and lower memory use.
- A VAE (variational autoencoder) decodes the final latent result into the visible image.
- The CFG (classifier-free guidance) scale controls how closely the output follows the prompt.
The open-source image model you can run on your own hardware.
Affiliate link — we may earn a commission at no cost to you.▶ The 60-second explainer (script)
How does Stable Diffusion work? It builds a picture by cleaning up noise. Here's the step by step. First, your text prompt goes into a text encoder that turns your words into numbers capturing their meaning. Then the model starts with a screen of pure random static. Now the magic: over about twenty to fifty steps, a neural network looks at the static and predicts which noise to remove to get closer to your description, subtracting a little each pass. Slowly, shapes and colors emerge until a clear image matching your prompt appears. To make this fast enough to run on a normal graphics card, it doesn't work on full pixels. It compresses the image into a tiny coded sketch, does all the denoising there, then a decoder expands it back to full resolution at the end. It learned this whole skill by studying millions of images with captions, practicing adding and removing noise until it understood what real pictures look like.
What authoritative sources say
People also ask
Does Stable Diffusion copy images from a database?
No. It generates new pixels by denoising, using patterns learned during training. It does not paste stored images together.
What are 'steps' in Stable Diffusion?
Steps are the number of denoising passes, usually 20 to 50. More steps can add detail but take longer, with diminishing returns.
What does CFG scale do?
It sets how strictly the image follows your prompt. Low values are more creative and loose; high values obey the text more but can look harsh.
Why is it called 'latent' diffusion?
Because the denoising happens in a compressed 'latent' representation of the image rather than on full pixels, which makes it fast and memory-efficient.