How accurate is AI?
There's no single accuracy number, and anyone quoting one is misleading you. Accuracy swings from near-perfect to near-useless depending on the task and how common the topic is. The number that matters isn't the error rate — it's that errors arrive in the same confident tone as correct answers, so you can't tell which is which without checking.
Why — the first-principles explanation
A language model doesn't retrieve facts. It predicts the next chunk of text given everything before it. When a fact appeared thousands of times in training, the highest-probability continuation is the true one — accuracy is a side effect of popularity. Ask when World War II ended and you get 1945, reliably, because that string is everywhere. Ask about a small company's 2019 revenue and the model still produces a fluent, specific, plausible number, because producing fluent specific text is the only thing it does. It has no separate circuit that says "I never actually saw this." That's the mechanism behind hallucination, and it explains the shape of the error curve: accuracy is high on well-documented things and degrades smoothly as topics get more obscure — with no warning signal at the point where it stops knowing.
This is why reported accuracy figures range from over 90% to under 50% and all of them can be honest. They're measuring different things. A benchmark of famous facts and a benchmark of obscure ones will produce wildly different numbers from the identical model. Add three more sources of divergence: whether the model could search the web (grounded answers are far more accurate than recalled ones), whether the test questions leaked into training data (contamination, which inflates scores), and who graded the answers (exact match versus human judgment). Before you accept any accuracy claim, ask which of those five knobs it was set with.
The distinction that actually protects you is calibration versus accuracy. Accuracy is how often it's right. Calibration is whether its confidence tracks its correctness. A weather forecaster who says 70% and is right 70% of the time is perfectly calibrated even though he's wrong constantly — and he's useful, because you know when to bring an umbrella. Language models are far better at accuracy than at calibration. They sound identical when certain and when confabulating. That's the real cost: it's not the errors, it's that the errors are undetectable from tone.
Where accuracy is rigorously measured, it's because someone forced the issue. Medical AI is the clearest case — the FDA has authorized more than 1,000 AI-enabled devices, and each one carries measured performance data on a defined task and population. That's what accuracy looks like when it's a real claim rather than a vibe: narrow task, named population, published numbers. Nobody can produce that for a general chatbot, because the task isn't defined.
An example that makes it click
Think of a very well-read friend who has never once said "I don't know." Ask about the French Revolution and he's superb — he's read forty books on it. Ask about the zoning history of your hometown and he answers in exactly the same relaxed, authoritative voice, with dates and names, all invented.
He's not lying. He genuinely can't tell the difference between remembering and confabulating, because to him both feel like sentences arriving. Your only defense is knowing which topics he actually read about — or checking.
How to do it
- Ask yourself how common the topic is. Well-documented, widely-written-about subjects get high accuracy. Obscure, recent, local, or numerically specific ones degrade sharply.
- Prefer grounded modes. An assistant that searches the web and cites sources is substantially more reliable than one recalling from training — and you get links to verify.
- Click the citations. Fabricated or mismatched sources are a known failure mode; a link's existence isn't proof it says what the summary claims.
- Treat all specific numbers, dates, names, quotes, legal citations, and URLs as unverified by default. These are exactly the outputs that look most authoritative and fail most often.
- Ask the model to state its uncertainty explicitly, and re-ask the same question in a fresh conversation. Answers that shift between runs are a signal the model is generating rather than recalling.
- For anything with consequences — medical, legal, financial — use AI to draft and orient, never as the final authority. Verify against a primary source.
Key facts
- There is no single accuracy figure for AI; measured accuracy varies by task, topic obscurity, whether the model can search, benchmark contamination, and grading method.
- Language models generate the highest-probability continuation of text rather than retrieving stored facts, which is why accuracy correlates with how frequently a topic appeared in training data.
- Models are better at accuracy than at calibration — they produce correct and fabricated answers in the same confident register, giving no internal signal of uncertainty.
- Benchmark contamination (test questions appearing in training data) inflates reported scores and is a known unresolved problem in AI evaluation.
- The FDA has authorized more than 1,000 AI-enabled medical devices through established premarket pathways — each with measured performance on a defined task, which is what rigorous accuracy measurement requires.
- NIST's AI Risk Management Framework (released 2023-01-26) separates 'Map' from 'Measure' precisely because accuracy claims are only meaningful within a defined deployment context.
▶ The 60-second explainer (script)
How accurate is AI? There's no single number, and anyone who gives you one is misleading you. Here's why. A language model doesn't look facts up. It predicts the next chunk of text. So when a fact appeared thousands of times in training, the most probable continuation happens to be the true one. Accuracy is a side effect of popularity. Ask when World War Two ended, you get 1945, reliably — that string is everywhere. Ask about a small company's 2019 revenue and the model still gives you a fluent, specific, plausible number. Because producing fluent specific text is the only thing it does. There's no separate circuit that says 'I never actually saw this.' That's hallucination, and it explains the shape of the error curve: high accuracy on well-documented things, degrading smoothly into obscurity, with no warning at the point where it stops knowing. That's why reported accuracy ranges from over ninety percent to under fifty, and all of it can be honest — they're measuring different things. Famous facts versus obscure ones. Web search on or off. Whether the test leaked into training. Who graded it. But here's the distinction that actually protects you: accuracy versus calibration. Accuracy is how often it's right. Calibration is whether its confidence matches. A forecaster who says seventy percent and is right seventy percent of the time is wrong constantly and still useful, because you know when to bring an umbrella. Models are way better at accuracy than calibration. They sound identical when they're certain and when they're making it up. That's the real cost. Not the errors. The undetectability.
What authoritative sources say
People also ask
What percentage of the time is ChatGPT correct?
There's no valid single figure. It depends on the topic's popularity in training data, whether web search is enabled, and how you grade. On common factual questions it's usually right; on obscure specifics it fabricates confidently.
Why does AI make up sources and citations?
Because a citation is just a text pattern — author, year, journal, volume. The model generates plausible-looking patterns whether or not the paper exists. Always click the link.
Is AI more accurate than a Google search?
Different failure modes. Search shows you sources and makes you judge them. AI gives you a synthesized answer that may be right, or may be a fluent blend of things that were never true together. Grounded AI with clickable citations is the middle ground.
Is newer AI more accurate?
Generally yes, and the gains are real. But hallucination hasn't been eliminated — it's a consequence of how the models generate text, not a bug that gets patched. Newer models fail less often but just as confidently.
How can I tell when AI is wrong?
From the output alone, you often can't — that's the core problem. The practical signals: unusually specific claims about obscure topics, and answers that change when you ask the same question in a fresh conversation.