Eighty taka is what I pay, three evenings a week, to get to Tejgaon later than I would have if I had just kept driving. There are two ways in. One goes past the army stadium into Mohakhali and takes whatever that intersection feels like giving me on the night, which is usually nothing. The other doubles back onto the expressway and comes down at Hatirjheel, and from up there you get to look at Banani and Mohakhali the way you look at people you used to work with, all of them stationary and badly lit and going nowhere in particular, while you do a sustained forty with nobody knocking on the window to sell you a steering wheel cover. Then the ramp arrives and four lanes are made to fit through two, and every minute the toll bought is taken back at the bottom in front of everybody. I have timed it. It has never won, not once, and I keep paying anyway, because eighty taka does not buy arrival; it buys motion, and motion is what my body keeps accepting in place of speed.
Which brings me to the study you have already had quoted at you.
METR ran a randomised controlled trial in early 2025. Sixteen experienced developers, 246 real issues, in their own repositories, the ones they had been maintaining for years. Before starting, they thought AI would make them roughly 24% faster. They came out 19% slower. Then, having personally sat through it, they said it had sped them up by about 20%. Read that last sentence twice, because it is doing more work than the headline number ever did.
Nobody in that study was lying. They were up on the expressway watching Mohakhali sit still underneath them, and the toll had already been paid.
The 19% went round the internet and the rest of the paper did not, which is how the internet works. It got waved at me twice last year in meetings by men who had very obviously not opened it. METR themselves were careful about the limits; sixteen people is sixteen people, and those particular sixteen knew their codebases cold, which is close to the worst possible case for a tool whose main trick is helping you find your way around code you do not have memorised. Nobody quoting it mentioned that part.
What actually matters happened in February, when METR posted a note saying the design was finished. Not wrong. Unrunnable. They could not get people to sit for it. Nobody wanted to spend a paid day working without the tool, the rate had come down from $150 an hour to $50, and between a third and a half of participants admitted they had steered away from the issues where AI would have helped most, which means the task list had been strained clean of the interesting cases before anyone measured anything. The clock had stopped meaning much either, because a developer running an agent goes off and does something else while it works. The final numbers were -18% for the returning developers and -4% for the new ones, with confidence intervals of -38% to +9% and -15% to +9%. Both cross zero, so the honest summary is that nothing was learned, expensively.
So now there are four instruments that people quote at each other in meetings, and none of them are measuring the same thing.
Four instruments, four different questions
| Instrument | What it counts | Direction |
|---|---|---|
| METR RCT, 2025 | Elapsed time per issue, in a repo the developer already knows cold | 19% slower |
| METR follow-up, 2026 | The same, after self-selection had bent the sample | -18% and -4%, both crossing zero |
| Microsoft CLI rollout, 2026 | Merged pull requests per engineer, tens of thousands of them, four months | About 24% more |
| DORA, 2025 | Team throughput and delivery stability | Throughput up, stability down |
The contradiction isn't one. A developer can take longer on any particular issue and still land more of them by the end of the month, and the reason for it is sitting right there in METR's own list of complaints: when the agent is working, the human goes and does something else. The wall clock on that one issue gets worse. The hour gets fuller. Waiting stopped being idle at some point in the last two years and nobody updated the instrument, so the most careful study we have is timing a unit of work the job is no longer organised around.
DORA, meanwhile, found throughput up and stability down in the same breath. More change, arriving faster, breaking more. That is where the money goes, and it does not come out of the same pocket that felt the speedup. The engineer running the agent is the one on the expressway. The reviewer gets a bigger diff written by somebody who cannot fully account for the middle of it. The on-call gets the instability. Whoever picks the service up in March gets the code nobody read properly, because reading properly feels like nothing and shipping feels like getting somewhere. Every study in that table measures the first of those four people and none of them measure the other three, which tells you where the telemetry happened to be cheap and very little else.
Our own dashboard has an acceptance rate and a time-to-merge on it. A year in, I think the first one measures how pleasant the tool feels to use and the second measures nothing I could defend in front of a hostile room, and neither of them knows what the reviewer paid. Fixing that needs a pipeline we do not have and an engineer who is busy until October at the earliest, so it is a plan the way most things around here are plans.
Until then, when somebody asks whether the tooling is making us faster, I tell them we are landing more and I do not know what it costs further down. It is a worse answer than the one I was giving in January.