What are the benefits of AI?
AI's measured benefits are real but uneven, and they concentrate on beginners. In a study of 5,179 customer-support agents, AI access raised issues resolved per hour by 14% on average — but 34% for novices and near zero for experts. A 2025 randomized trial found experienced developers were 19% slower with AI while believing they were 20% faster.
Why — the first-principles explanation
Start with what AI actually is, because the benefits follow from the mechanism. A modern AI model is a system that has absorbed an enormous quantity of examples and learned to predict what comes next. That gives it one genuinely new economic property: it makes expertise copyable. Knowledge that used to live in one experienced person's head, unavailable at 2 a.m. and impossible to clone, can now be queried instantly by anyone, at nearly zero marginal cost. Almost every real benefit of AI is a consequence of that single fact.
This predicts exactly who gains. If AI's value is delivering competent-average performance on demand, then its benefit to you depends on the gap between your current skill and competent-average. A novice has a large gap, so AI closes it — a big gain. An expert already performs above competent-average, so AI offers little, and can actively drag if checking its output costs more than doing the work. The evidence matches this shape precisely. Brynjolfsson, Li and Raymond studied 5,179 support agents at a Fortune 500 firm and found a 14% average productivity gain, but 34% for novice and low-skilled workers and minimal impact on the most experienced. The average conceals the whole story: AI didn't lift everyone a little, it lifted beginners a lot and experts barely at all. That same study also found improved customer sentiment and higher employee retention, and suggested AI spread best practices from strong workers to weak ones.
Now the honest counterweight, because this is where most "benefits of AI" pages lie by omission. In July 2025, METR ran a randomized controlled trial with 16 experienced open-source developers across 246 real issues in large, mature codebases they already knew well. The developers predicted AI would make them 24% faster. Afterward they reported feeling 20% faster. They were actually 19% slower. Not slower and aware of it — slower and convinced of the opposite. That perception gap is the most important number in AI productivity research, because it means self-reported benefit is not trustworthy evidence, and "our team says AI helps" is not a measurement.
These two studies do not contradict each other; they define the boundary. Support agents were often novices handling problems where a competent-average answer is the goal. The METR developers were experts on codebases where they held context no model had, working to a standard above competent-average — so AI's suggestions had to be read, judged, and often repaired. Read together: AI's benefit is largest where the task is routine, the standard is average competence, and the user is inexperienced. It shrinks or inverts where the user is expert, the context is idiosyncratic, and errors are expensive to catch. METR itself cautions its result doesn't show AI fails to help most developers, or in other domains — it studied 16 people in one specific setting. Both papers are narrow. Anyone quoting either as the universal verdict on AI is selling something.
An example that makes it click
Think of AI as a very well-read intern who has skimmed everything ever written and never sleeps — but has never once visited your office and will never say "I don't know."
If you're brand new, this intern is a gift. You don't know how to phrase a refund email or start a spreadsheet formula; the intern does it in seconds, at a solid B. Your work jumps from a D to a B. That's the 34%.
If you're the person who's run this department for ten years, the same intern is a mixed bag. Your work is already an A. The intern hands you a B and says it's an A-plus. Now you have to read it, find the two things that are quietly wrong, and fix them — and that can take longer than writing it yourself. Worse, it feels fast, because reading a finished draft feels easier than writing one. That's the trap the developers fell into: they were slower by 19% and would have sworn they were faster by 20%.
Key facts
- In a study of 5,179 customer-support agents at a Fortune 500 software firm, access to a generative AI assistant increased issues resolved per hour by 14% on average (Brynjolfsson, Li & Raymond, NBER w31161, April 2023, revised November 2023).
- The same study found a 34% improvement for novice and low-skilled workers, and minimal impact on experienced and highly skilled workers.
- That study also reported improved customer sentiment, increased employee retention, and evidence AI disseminated best practices from more-skilled colleagues.
- In METR's July 2025 randomized controlled trial, 16 experienced open-source developers took 19% LONGER to complete 246 real issues when allowed AI tools, working in codebases with 22k+ stars and 1M+ lines of code.
- Those developers predicted a 24% speedup beforehand and still believed AI had sped them up 20% afterward — a large gap between perceived and actual benefit.
- METR explicitly cautions its result does not show AI fails to speed up most developers or fails in other domains; the sample was 16 developers in one specific context.
▶ The 60-second explainer (script)
What are the benefits of AI? The honest answer: real, but wildly uneven — and they mostly go to beginners. Here's the best evidence we have. Researchers studied 5,179 customer support agents at a Fortune 500 company. With an AI assistant, they resolved 14% more issues per hour. Sounds modest. But that average hides everything. Novice workers improved 34%. The most experienced workers? Almost nothing. Now the part most articles skip. In July 2025, METR ran a proper randomized trial: 16 experienced developers, 246 real coding tasks, in codebases they knew well. They predicted AI would make them 24% faster. Afterward, they said it made them 20% faster. The stopwatch said they were 19% slower. They were slower — and certain they were faster. Why the split? Because AI's real trick is making expertise copyable. It delivers competent-average work instantly, to anyone. If you're below average, that's a huge lift. If you're an expert, it hands you a B and calls it an A-plus — and you have to find the two things quietly wrong with it. So: AI helps most when the task is routine, the bar is average competence, and you're new. It helps least — or hurts — when you're expert, the context is unusual, and mistakes are expensive. And whatever you do, don't trust the feeling. Measure it.
What authoritative sources say
People also ask
What is the single biggest proven benefit of AI?
Compressing the skill gap. The strongest measured effect is that inexperienced workers improve a lot — 34% in the largest workplace study — because AI delivers competent-average output instantly to people who couldn't produce it themselves.
Does AI always make people more productive?
No. A 2025 randomized trial found experienced developers were 19% slower with AI on codebases they knew well. Benefit depends on the task, the user's skill, and how expensive it is to check the AI's output.
Why do people feel AI helps even when it doesn't?
Reading a finished draft feels easier than writing one, so the work feels faster even when the clock disagrees. METR's developers felt 20% faster while being 19% slower — which is why self-reports are weak evidence.
Are there benefits beyond speed?
Yes. The customer-support study found improved customer sentiment and higher employee retention, and evidence that AI spread experienced workers' best practices to newer staff — a training effect, not just a throughput effect.
How should an organization decide if AI is worth it?
Measure it against a control group rather than surveying users, and measure per skill level rather than in aggregate. Averages hide the real pattern: large gains for novices, little or negative for experts.