Coinbase says a 9B model it post-trained beat Opus 4.5 at fraud triage. The win was 9.6 F1 points, against Opus 4.5 only, on Coinbase's private benchmark, for under $100 of training compute, by Coinbase's own account. Nobody outside can rerun it. There's no live A/B test. So I built a public version. The two model arms haven't run yet. This piece is the floor they have to beat and the bill they'll be judged on.
Fair warning: one task, one seed, CPU only. Anything not run is modelled.
What I Tested
Dataset: Banking77 (PolyAI, CC BY 4.0): 77 intents, 10,003 train and 3,080 test tickets. Six of the intents are fraud-adjacent, including compromised_card and card_payment_not_recognised.
Splits: seed 42. A stratified 1,003-ticket validation slice comes out of train and is used only for tuning. Test is scored once.
Planned arms (not run): Qwen3.5-9B with LoRA SFT via TRL + PEFT on one rented RTX PRO 6000, against claude-opus-4-5-20251101 zero-shot with the same 77-label prompt. Coinbase used RL (GRPO via SkyRL), not SFT, so this tests the economics, not a replica.
Baseline (run): TF-IDF (word 1–2 and char 2–5 n-grams) plus logistic regression. Hardware: 8 vCPU Intel Xeon, 16 GiB RAM, Python 3.13.5, scikit-learn 1.9.1.
# files: Specialist Bake-off Kit (link below) cd specialist-bakeoff && make baseline
Training took 18.9 seconds. That's the whole bill.
Results
| Metric (test, n=3,080) | CPU baseline |
|---|---|
| Accuracy | 2,802 of 3,080 (91.0%) |
| Macro-F1, 77 intents | 0.910 |
| Fraud-adjacent tickets correct | 221 of 240 |
| Weakest fraud intent F1 | card_payment_not_recognised, 0.854 |
| Calibration error (ECE, 10 bins) | 0.044 |
| Router at t=0.45 (picked on val) | 2,877 auto, 203 to a human |
The surprise is the first row. A model that costs nothing to train already gets 91% on this task. Any 9B fine-tune or frontier call now has to beat 91.0% by enough to pay for itself. "Beats Opus" isn't the bar.
Failure Modes
The cache that never fires. I planned a cached-Opus cost column, with the 77-label prefix marked cacheable. Anthropic's docs put the minimum cacheable prompt for Opus 4.5 at 4,096 tokens. My prompt is about 431 Qwen tokens, roughly 455–615 by my estimate for Claude. Below the minimum, the cache mark is silently ignored. So the cached column is n/a.
Confident and wrong. At the 0.45 threshold, 163 of the 2,877 auto-routed test tickets were wrong. That's 94.3% accuracy on what the router let through, against 95.0% on validation.
Near-duplicate intents. The top confusion is verify_my_identity read as why_verify_identity (6 tickets), then pending_top_up read as top_up_failed (5). That's label design. A bigger model may not fix it.
Self-graded. I wrote the prompt, splits and scorer. One builder's harness, not a leaderboard.
The Bill the Arms Will Face
Here's the cost per 1,000 tasks (all modelled):
opus_per_1k = 1000 * (in_tok * 5 + out_tok * 25) / 1e6 # ~$3.30 at 535 in / 25 out; cached n/a spec_per_1k = (train_gpu_h * rate + labelling) / lifetime_tasks * 1000 + serve_gpu_h_per_1k * rate labelling = n_labels * sec_per_label / 3600 * wage # 13,083 x 28 s x $19.88/h
The labelling line is about $2,000, modelled from a proxy time and a wage floor. The 28 seconds per label is borrowed from relevance judging (McDonnell et al., JAIR), because I found no published time for intent labelling. Banking77 handed me those labels free. Your fraud team's aren't, and the under-$100 figure leaves them out.
Why the Arms Haven't Run
Both arms cost money: a rented GPU and paid API calls, modelled at $16–47. I didn't get the spend for this piece, so neither arm ran, and every number above is the CPU baseline. The scripts refuse paid calls without a key and an approval flag.
What I'd Ship
Now: the CPU baseline as a free first gate. It's right 91.0% of the time on this task, and it gives you a calibrated score to route on. Next: the Qwen and Opus runs on the same frozen splits, judged on cost per 1,000 tasks above the 91.0% floor. Not yet: any "the small model wins" claim, from Coinbase's numbers or mine. Neither has been rerun on public data.
Get the Kit
The Specialist Bake-off Kit has the baseline, the router, results.csv, the cost calculator and both unrun arms: https://roiscale.ai/shop/products/e9021067-bda3-4766-877e-896afbab4b91. If you have a labelled queue of your own, run the baseline on it tonight before you price a fine-tune.
References
- Coinbase, "Owning Intelligence, Part 3," Oct 8, 2026. https://www.coinbase.com/blog/part-three-owning-intelligence-series
- Casanueva et al., "Efficient Intent Detection with Dual Sentence Encoders," 2020 (Banking77). https://arxiv.org/abs/2003.04807
- Anthropic, "Prompt caching" and "Pricing" docs, accessed Oct 20, 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- U.S. BLS, Occupational Employment and Wage Statistics, May 2025, data entry keyers (43-9021), median $19.88/h: a floor, not what Coinbase paid. https://data.bls.gov/oesprofile/?major_group=430000&measure=01&occupation=439021
- RunPod GPU pricing, updated Oct 8, 2026. https://www.runpod.io/pricing
