How to test AI models?
Test an AI model on your own held-out data, not public benchmarks — benchmark scores leak into training sets and stop measuring anything. Build 50-200 real examples with known right answers, score them automatically, and re-run on every model change. NIST's AI Risk Management Framework (released January 26, 2023) makes Measure one of its four core functions.
Why — the first-principles explanation
Testing an AI model breaks the assumption normal software testing rests on. Regular code is deterministic: same input, same output, so a test either passes or fails forever. A language model is a probability distribution over next tokens. Run the same prompt twice and you can get two different answers, both defensible. So "did it pass?" stops being a yes/no and becomes "what fraction of the time, on what distribution of inputs, by whose judgment?" Everything about AI evaluation follows from that one shift.
That's why the industry's headline numbers deserve suspicion. Public benchmarks — the leaderboards vendors quote in launch posts — are published question sets on the open internet. The next model gets trained on a scrape of that internet. The benchmark ends up inside the training data, and the model is no longer being tested, it's being quizzed on homework it already saw. This is called contamination, and it's the reason a model can post a spectacular score on a famous benchmark and then fall apart on your actual invoices. A benchmark measures what it measures; the moment it becomes a target, it stops being a measurement.
So the real work is building your own held-out set: real inputs from your actual use case, paired with correct outputs you trust, that the model has never seen. Fifty to two hundred well-chosen examples beat ten thousand scraped ones, because each one has to be right — a test set with wrong labels teaches you nothing except to distrust your own numbers. Then you need a scoring function. If there's one correct answer, exact match works. If output is free text, you're choosing between human review (accurate, slow, expensive) and using a second model as a judge (fast, cheap, and quietly biased — LLM judges favor longer answers and their own writing style, so you calibrate them against human labels before trusting them).
Finally, average accuracy is the least interesting number you'll produce. What matters is where it fails and how badly. Red teaming — deliberately hunting for inputs that break the thing — finds the failures your happy-path test set never will: prompt injection, edge-case formats, adversarial users. NIST's AI Risk Management Framework, released January 26, 2023, organizes this into four functions — Govern, Map, Measure, Manage — and its Generative AI Profile (NIST-AI-600-1, published July 26, 2024) covers risks specific to generative systems. The framework is voluntary, but it's a useful checklist against the temptation to ship on vibes.
An example that makes it click
Imagine hiring a cook. You could ask which culinary school he attended and what he scored on the final exam — that's a public benchmark. Impressive, and nearly useless, because everyone studies for that exam and some people have seen the answer key.
So you do the sensible thing instead: you hand him your ten actual recipes, the ones your restaurant serves, and you taste all ten. That's a held-out set. Then you do the thing most people skip — you make it weird on purpose. You give him a recipe with an ingredient missing. You give him one written in messy handwriting. You have someone walk in and tell him to ignore the recipe and make whatever he wants. That's red teaming, and it's how you find out he's brilliant on Tuesday and a disaster the moment anything goes sideways. Nine perfect dishes and one that poisons someone isn't a 90%. It's a fire.
How to do it
- Write down what 'good' means for your specific task before you test anything — accuracy, format compliance, refusal rate, latency, cost per call.
- Collect 50-200 real inputs from your actual use case and label the correct output for each. Label quality matters more than volume.
- Hold this set back. Never paste it into a public tool, a prompt you share, or anything that could end up in a training scrape.
- Pick a scoring method: exact match for closed answers, human review for judgment calls, or an LLM judge calibrated against human labels first.
- Run a baseline on your current model and record the number with the date and model version.
- Red team it: adversarial inputs, prompt injection, missing fields, wrong language, absurd edge cases. Log every break.
- Slice the results — accuracy by input type, by length, by user segment. Averages hide the failures that matter.
- Re-run the whole suite on every prompt edit, model upgrade, or vendor version bump, and treat a score drop as a build failure.
- Monitor in production; real user inputs drift away from your test set within months.
Key facts
- NIST released the AI Risk Management Framework on January 26, 2023, built around four core functions: Govern, Map, Measure, and Manage.
- NIST published the Generative AI Profile (NIST-AI-600-1) on July 26, 2024, covering risks unique to generative AI.
- The NIST AI RMF is voluntary guidance, developed through a consensus-driven, open, and collaborative process — not a regulation.
- Public benchmarks suffer contamination: published test questions get absorbed into later models' training data, inflating scores.
- Language models are non-deterministic — the same prompt can produce different outputs, so evaluation must be statistical rather than pass/fail.
- NIST launched the Trustworthy and Responsible AI Resource Center on March 30, 2023, alongside the AI RMF Playbook.
▶ The 60-second explainer (script)
How do you test an AI model? Start by throwing out the leaderboard. Those public benchmark scores that vendors quote? Those questions are published on the internet. The next model gets trained on the internet. So the test ends up inside the training data, and the model isn't being tested anymore — it's being quizzed on homework it already saw. That's called contamination, and it's why a model can ace a famous benchmark and then choke on your invoices. Here's what actually works. Take fifty to two hundred real examples from your own use case. Label the correct answer for each one — and get those labels right, because a test set with bad labels is worse than no test set. Never publish it. That's your held-out set. Then pick how you'll score it. One right answer? Exact match. Free-form text? Either humans read it, or you use a second model as a judge — but calibrate that judge against real human ratings first, because LLM judges quietly prefer longer answers and their own writing style. Then do the part everyone skips: try to break it. Weird formats, missing fields, prompt injection, hostile users. Average accuracy is the least interesting number you'll get. Where and how it fails is the whole story. Nine good answers and one catastrophe isn't ninety percent — it's an incident. If you want a structure for all this, NIST's AI Risk Management Framework, out since January 2023, organizes it as Govern, Map, Measure, Manage, with a generative AI profile added in July 2024. It's voluntary. Use it anyway.
What authoritative sources say
People also ask
How many test examples do I need?
Fifty to two hundred carefully labeled, representative examples is a solid start and beats thousands of sloppy ones. Add more once you know which slices of input you're weakest on.
Can I trust published benchmark scores?
Treat them as marketing, not measurement. Published questions leak into later training data, and a model tuned to a benchmark tells you little about your workload.
Is using an LLM to grade another LLM legitimate?
Yes, if you calibrate it. Compare the judge's scores against human ratings on a sample first — LLM judges show known biases toward longer responses and their own style.
What is red teaming?
Deliberately attacking your own system with adversarial inputs — prompt injection, malformed data, hostile phrasing — to find failures that normal test cases never surface.
How often should I re-test?
On every prompt change, model version bump, or vendor update, and continuously in production. Vendor models change under you, and real user inputs drift within months.