How to train a lora for Stable Diffusion?
To train a LoRA for Stable Diffusion, collect 15-30 varied, captioned images of your subject or style, install a trainer like Kohya_ss, pick a base model (SD 1.5 or SDXL), set a trigger word, learning rate, and steps, then train on a GPU with about 12GB VRAM for roughly 20-90 minutes. Load the resulting small file in AUTOMATIC1111 or ComfyUI.
Why — the first-principles explanation
A LoRA (Low-Rank Adaptation) is a clever shortcut. Instead of retraining the whole billion-parameter model to learn one new face or style, LoRA trains a small set of correction values that plug into the model's existing layers. Because it only learns the 'difference' needed, the file is tiny (a few MB to a few hundred MB) and training fits on a consumer GPU.
The technique works because a specific concept is a small adjustment to a model that already knows almost everything. Teaching it 'my dog Rex' or 'flat pastel comic style' doesn't require rebuilding its knowledge of dogs or comics, only steering it. LoRA captures that steering compactly, and you can apply it at adjustable strength or stack several together.
As with all fine-tuning, the dataset is decisive. For a subject, use 15-30 images with the same subject in varied poses, lighting, and backgrounds so the model learns the subject, not the background. Each image gets a caption with a unique trigger word (like 'rexdog') plus descriptions of everything else, so the model ties the concept to that word and treats the rest as variable.
The practical settings that matter most are learning rate, steps/epochs, and network dimension (rank). Too many steps or too high a rate 'overfits', copying training images too literally; too few underfits and the concept barely appears. Trainers like Kohya_ss and OneTrainer expose sensible defaults, and you test the LoRA at several checkpoints to find the sweet spot.
An example that makes it click
The base model is a chef who can cook thousands of dishes. You want them to nail your grandma's exact soup. You don't send the chef to culinary school again, you give them a recipe card, that's the LoRA, listing just the tweaks: a pinch more salt, this special herb, simmer ten minutes longer.
You make the card by cooking the soup with them fifteen to thirty times, adjusting until it's right. The finished card is tiny and slips into their pocket. Any time you want grandma's soup, they pull out the card, apply the tweaks, and the dish comes out perfect, no retraining needed.
How to do it
- Collect 15-30 images of one subject or style, varied in pose, lighting, and background.
- Crop or resize images to your base model's resolution (512px for SD 1.5, 1024px for SDXL).
- Caption each image with a unique trigger word plus descriptions of the changeable elements.
- Install a trainer such as Kohya_ss or OneTrainer and select your base model.
- Set key parameters: learning rate, steps/epochs, and network rank (start from the tool's defaults).
- Train on a GPU with about 12GB VRAM (roughly 20-90 minutes), saving checkpoints along the way.
- Test each checkpoint in AUTOMATIC1111 or ComfyUI, pick the best, and adjust LoRA strength when generating.
Key facts
- A LoRA trains small add-on weights, producing files typically a few MB to a few hundred MB.
- About 15-30 varied, captioned images is a common dataset size for a subject LoRA.
- LoRA training is comfortable on a GPU with roughly 12GB VRAM and takes about 20-90 minutes.
- A unique trigger word in captions ties the learned concept to a specific prompt keyword.
- Kohya_ss and OneTrainer are the most popular LoRA training tools; overtraining causes overfitting to the input images.
The open-source image model you can run on your own hardware.
Affiliate link — we may earn a commission at no cost to you.▶ The 60-second explainer (script)
Here's how to train a LoRA for Stable Diffusion, a tiny file that teaches the model your face, character, or style. First, why it's easy: instead of retraining the whole giant model, a LoRA learns just the small adjustment needed, so the file is tiny and it trains on a normal graphics card with about twelve gigabytes of memory. Step one, and the most important: gather your images. Fifteen to thirty photos of one subject, in different poses, lighting, and backgrounds, so the model learns the subject, not the wallpaper behind it. Step two: caption each image, and include a unique trigger word, like 'rexdog', plus a description of everything else. Step three: install a trainer like Kohya_ss, pick your base model, SD 1.5 or SDXL, and start from the default learning rate and step count. Step four: train, about twenty to ninety minutes, saving checkpoints along the way. Finally, load the LoRA in AUTOMATIC1111 or ComfyUI, type your trigger word, and adjust its strength. Watch out for overtraining, which makes it copy your photos too literally.
What authoritative sources say
People also ask
How many images do I need for a LoRA?
Around 15-30 varied, well-captioned images works for most subject or style LoRAs. Variety in pose and background matters more than count.
What GPU do I need to train a LoRA?
About 12GB of VRAM is comfortable. Lower-VRAM methods exist but are slower and more limited. Cloud GPUs also work.
Why does my LoRA copy the training photos exactly?
That's overfitting, usually from too many steps or too high a learning rate. Use fewer steps and test earlier checkpoints.
What's a trigger word?
A unique keyword you put in captions and later in prompts to activate the learned concept, like 'rexdog' for a specific dog.