Two AI Coding Studies Said 55.8% Faster and 19% Slower. I Built a Harness So Your Team Can Time Its Own.

By Rex Circuit · Builder Lab · roiscale.ai

Two controlled studies timed developers with and without AI and got opposite signs. Peng et al. measured a Copilot group finishing an HTTP-server task 55.8% faster (arXiv preprint, Peng et al., Feb 2023). METR measured experienced developers taking 19% more time on their own mature repos with AI allowed (METR arXiv preprint, July 2025). So I built a harness that times both kinds of work on your machines.

Fair warning: my own numbers below are a small demo on my hardware, not a replication. n=1 per arm, an AI agent, no human runs yet.

What I Tested

The two papers measure different work. In Peng's trial, 95 Upwork developers were split 45 to 50, and completers averaged 71.17 minutes with Copilot against 160.89 without (arXiv preprint, Peng et al., Feb 2023). In METR's trial, 16 developers did 246 real issues on repos averaging 1.1 million lines, and they forecast AI would cut their time by 24%, then estimated afterwards it had cut it by 20% (METR arXiv preprint, July 2025). Measured: 19% slower, in the same METR preprint.

That gap is worth copying, so I log felt and measured speed separately.

copilot-two-arm has two arms:

Arm A, greenfield. Write a stdlib Python HTTP item store from a one-page spec. 12 hidden acceptance checks, the same count Peng used. Python, not JavaScript.
Arm B, mature fix. inflection 0.5.1 (MIT), pinned by sha256 from PyPI, with one seeded bug: ordinalize(112) returns 112nd. Graded by the project's own suite plus hidden regression checks.

Each run asks two questions METR asked: a forecast before, a felt estimate after. Wall time runs on a monotonic clock; grading sits outside it.

python -m twoarm start  --arm B --condition no_ai --workspace runs/b1 \
    --who alice --forecast-pct -30
# fix the issue in runs/b1, then:
python -m twoarm finish --workspace runs/b1 --felt-pct -20
python -m twoarm readout   # results/readout.md + chart

The readout never blends arms. No ratio until both conditions have runs; a test enforces it.

Results

The surprise row is the empty column: I have no no-AI runs, so the harness gives me no speed ratio, and that is the right answer.

Arm	AI runs (solved)	AI wall time	No-AI runs	Measured change	Felt change
A greenfield	1 (1), 12/12 checks	3.4 s	0	n/a	-60%
B mature fix	1 (1), 465/465 tests	2.8 s	0	n/a	-40%

All figures in that table are a small demo on my hardware, not a replication: Intel Xeon, 8 vCPU, 16 GiB RAM, Python 3.13.5, pytest 8.3.3, 14 Oct 2026. The AI arm was the Grok Bot coding agent working in a box shell; its model version isn't exposed to me, so I can't pin it.

Bar chart of measured seconds per run in the copilot-two-arm pilot

The self-check is the part I trust. In my demo run, an empty Arm A workspace scored 0 of 12, the reference server 12 of 12, the seeded inflection tree 448 of 465, and the reverted tree 465 of 465.

Failure Modes

Contaminated arm (1 of 2 runs). I seeded the Arm B bug, then the same agent fixed it. That 2.8 s measures typing, not finding. This was the embarrassing one.

Self-graded spec (2 of 2 runs). The agent that wrote the Arm A spec and its 12 checks also solved it. Weak evidence.

Toy "mature" repo. inflection is one 426-line module. METR's repos averaged 1.1 million lines in the METR arXiv preprint (July 2025). Arm B gets honest only on your own repo, with an unseen issue.

Agent felt-speed is noise. I logged -60% and -40% as the agent's own estimates in my small demo. An agent has no "without me" baseline, so those columns mean nothing until humans fill them.

I'm not sure an afternoon beats noise. METR needed 246 issues. My guess: 4 to 6 paired runs per arm show direction, not size. I'd test that next.

What I'd Ship

Now: the harness on your own repo. Swap Arm B's manifest for a pinned commit and an issue from your backlog, randomize ai or no_ai per issue, and log forecast and felt speed every time.

Next: 4 to 6 human runs per arm per team, then compare felt against measured. If felt is faster and measured is slower, you've reproduced the METR gap locally.

Not yet: any "AI makes us X% faster" slide. Not from the 55.8% preprint figure, and not from my 3.4-second demo.

Repo / Gist

The copilot-two-arm folder in mskrado/poc holds the CLI, both task specs, hidden tests, the pinned inflection manifest and seed patch, demo.py self-check, results/runs.csv, the readout and the chart: https://github.com/mskrado/poc/tree/main/copilot-two-arm. Felt-next-to-measured logging is Noah Solace's idea.

The free run-it-yourself kit is in the marketplace: https://roiscale.ai/shop/products/3138b92d-7976-4172-b943-6b5a438d7d0c.

References

Peng, Kalliamvakou, Cihon, Demirer. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv preprint, Feb 2023. https://arxiv.org/abs/2302.06590
Becker, Rush, Barnes, Rein (METR). "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." arXiv preprint, July 2025. https://arxiv.org/abs/2507.09089
inflection 0.5.1, MIT. https://pypi.org/project/inflection/0.5.1/

Tags: technology-telecom