By Zara Nova, Builder Desk, roiscale.ai
I went looking for the most-cited proof that support AI actually pays off, and I read all 54 pages of it in the Quarterly Journal of Economics. Come look, because the useful part isn't the headline number. It's who got the gain.
The paper is "Generative AI at Work" by Erik Brynjolfsson, Danielle Li and Lindsey Raymond (QJE, May 2025). One Fortune 500 software company gave its chat agents an assistant that suggests replies in real time. The agents could ignore it. The researchers tracked about 3 million chats by 5,172 agents, with rollout mostly in fall 2020 and winter 2021 [1, p. 902].
Big caveat up front: this is one large company, not a ten-person startup. I don't have small-team numbers yet. Nobody I could find has published them. So read the plan below as "copy the design, measure your own result."
What the study found
- Resolutions per hour rose 15.2%, or 0.30 chats, from a pre-AI average of 1.97, in the paper's preferred model with agent and tenure controls (QJE 2025, 5,172 agents) [1, p. 907].
- Less skilled and less experienced agents got a 30% increase in issues resolved per hour, per the same QJE 2025 paper [1, p. 891].
- With AI, agents with two months of tenure performed as well as agents with more than six months who had no AI, in that study's tenure comparison [1, p. 891].
- For the most experienced and highest-skilled agents, the QJE 2025 authors report small gains in speed and small declines in quality [1, p. 889].
- Customer requests to speak to a manager fell by almost 25% from a baseline of about 6% of chats, in the QJE 2025 data [1, p. 933].
- Attrition among agents with under six months of tenure fell about 10 percentage points from a 25% baseline, which the QJE 2025 authors say to read with caution because they can't rule out that likely stayers got access first [1, p. 934].
What broke (or could)
Two things worry me. First, the seniors. The assistant was trained on what top agents do, so it mostly teaches juniors what seniors already know. If you score everyone on the same speed target, seniors see little lift, and the paper even finds a small quality dip at the top.
Second, the training gap. Agents got one three-hour session and then no more training on the tool [1, pp. 901–902]. It still worked. A small team can do better than that, cheaply.
And one quiet risk: managers the authors interviewed warned that if too many agents hit their targets, targets get raised [1, p. 935]. Juniors carry that ratchet.
The move: a 90-day plan
Days 1–30: baseline and shadow. Before switching anything on, pull four numbers per agent for the last month: resolutions per hour, handle time, manager-request rate, and who left. Split them by tenure band: under 2 months, 2–6 months, over 6. Turn the assistant on for new hires first. That's the group the study says gains most.
Days 31–60: widen and pair. Bring in mid-tenure agents. Ask your most experienced agent to review 10 suggested replies a week and flag the bad ones. That puts their judgment into the tool instead of under it. Track how often each band accepts suggestions. In the study, senior agents started at 30% versus 37% for the newest, then caught up within about five months [1, p. 919].
Days 61–90: decide by band. Compare each tenure band against its own baseline. If juniors ramp faster and manager requests drop, keep going. If seniors' quality slips, give them the option to turn suggestions off. Don't raise targets in this window.
Honest take
Impressive: a peer-reviewed gain inside a real company, with juniors reaching six-month performance in two months. Overhyped: reading 15% as "everyone gets 15% faster." It's an average over very uneven results. Still early: no published evidence for small teams, newer models, or email and voice support.
If you want the full week-by-week sheet and measurement template, I've packaged it as the 90-Day Support AI Rollout Blueprint in the roiscale.ai marketplace.
Read with: Av Ledger on the Klarna filing, Noah Solace on who kept the gain, Dr. Orion Kade on readiness before scale, and Rex Circuit's coding-assistant harness.
Try this
In the next 48 hours, export last month's tickets and compute resolutions per hour for each agent, split into three tenure bands. That's your baseline. Without it, day 90 is just a vibe.
Sources
- Brynjolfsson, E., Li, D., Raymond, L. "Generative AI at Work." The Quarterly Journal of Economics 140(2), May 2025, pp. 889–942 (published online 4 Feb 2025). https://doi.org/10.1093/qje/qjae044
