The productivity paradox in software engineering
A controlled trial found experienced developers were 19 percent slower with AI tools, while believing they were faster. The lesson is to measure AI's effect rather than trust your instinct about it.
TL;DR
- In a randomized controlled trial, experienced developers were about 19 percent slower when allowed to use AI tools.
- After the study they estimated AI had sped them up by 20 percent.
- The perception gap is the real warning for anyone measuring AI by how it feels.
When the independent lab METR ran a randomized controlled trial on AI and developer productivity in early 2025, it expected to confirm a speed-up. It found the reverse. Sixteen seasoned open-source developers, working on repositories they knew deeply, completed tasks about 19 percent slower when permitted to use AI tools than when they were not.
The 39-point gap between how fast developers felt and how fast they actually were is the finding that should travel.
How the trial was run
The methodology is what makes this hard to dismiss. It was a randomized controlled trial on 246 real tasks from the developers' own projects, using current tools such as Cursor Pro with Claude. The result also contradicted forecasts: economics and machine-learning experts had predicted AI would make these developers around 38 to 39 percent faster.
Two caveats matter. The sample was small and the confidence interval wide, and the developers were experts on mature codebases, which is the setting where AI suggestions are least likely to beat fluent human recall. METR has since said a follow-up experiment in late 2025 gave an unreliable signal, partly because many developers now refuse to work without AI at all.
The perception gap
The slowdown is the attention-grabbing number, but the durable insight is the perception gap. After the study, 69 percent of participants kept using AI. People adopt these tools because they feel faster and more pleasant, even when a measurement says otherwise. That is precisely why self-reported productivity, the basis of most corporate AI dashboards, is unreliable.
How to measure it
- Do not measure AI's value by how productive your teams feel. Measure cycle time, throughput, and defect rates against a baseline.
- Expect context to decide the outcome. AI tends to help most on unfamiliar code, boilerplate, and prototyping, and least on mature systems an expert already knows.
- Treat the validation and clean-up of AI output as real work, and budget for it, because in the study it consumed much of the supposed time saving.
AI does not always make engineers slower; in many settings it helps. What the study shows is that nobody, including the engineers themselves, can tell which without measuring.
Related reading
Source: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR.