The anti-pattern: a benchmark win treated as production readiness. The fix: one promotion gate for every checkpoint, specialist or frontier, ending at a named human.
The Production Failure That Started This
Coinbase replayed its fraud benchmark against newer frontier releases. Sonnet 5's recall fell 22.2 points against Sonnet 4.6, and GPT-5.6 gained 11.5 points of precision while losing 20.7 of recall, on Coinbase's private benchmark of 16,140 transactions [2]. An upgrade looked green and dropped coverage.
Coinbase then post-trained Qwen3.5-9B with RLVR, GRPO and LoRA via SkyRL on one RTX PRO 6000 96GB, for under $100 of training compute, by Coinbase's own account [1]. It led by 9.6 F1 points, against Opus 4.5 only, on Coinbase's private benchmark [1]. There is no live A/B test of the 9B.
The Anti-Pattern Mechanism
A benchmark is frozen; production moves.
Prevalence. Coinbase's paper calls the benchmark a case-control sample: 813 fraud cases in 16,140 transactions (5.04%), not the production rate [4].
Label delay. Fraud labels mature two weekly cycles late in the paper's replay [4].
Frozen knowledge. The fine-tuned policy lives in weights; a new fraud pattern needs retraining [4].
Monitoring stays silent: Coinbase reports P50 latency of 0.683 s against 1.515 s for Opus 4.5 in production [1].
The Architecture I Recommend
My gate, after Coinbase's disciplines [1]:
- Router: specialist first, then a frontier fallback (RouteLLM pattern [5]), then a human analyst queue as the last stop.
- Owner (my recommendation, not sourced): a fraud-operations lead owns the queue; specialist accuracy slips page the on-call ML engineer.
- Eval gate: promote on matured labels, scored once on held-out data.
- Drift monitor: escalation rate and score distribution daily; outcome metrics only after labels mature.
- Versioned bundle and rollback: weights, prompt and policy ship as one hash. vLLM's docs warn against runtime LoRA loading in production outside an isolated, fully trusted environment [6], so I promote merged, pinned checkpoints.
# promotion-gate.yaml; <> = set per deployment
bundle: {weights: sha256:<hash>, prompt: v14, policy: v9}
route: [specialist, frontier, human_queue]
router: {threshold: lowest where auto-routed accuracy >= 0.95 on validation}
human_queue:
owner: fraud-ops-lead
max_open: <analyst capacity per shift>
on_cap: alert owner
gate: {labels_matured_only: true, must_beat: tfidf_logreg_baseline}
drift: {daily: [escalation_rate, score_psi], outcome_lag_cycles: 2}
page_on_accuracy_slip: ml-oncall
retrain: {cadence: <fixed>, trigger: drift_alert}
rollback: {target: previous_bundle, rehearsed: true}
| Choice | Buys | Costs |
|---|---|---|
| Lower threshold | Fewer escalations | More specialist errors kept |
| Queue cap | Bounded analyst load | Alerts during bursts |
What Rex's Harness Can and Cannot Show
Rex Circuit's harness uses Banking77 (CC BY 4.0) [3]: 9,000 training rows, a 1,003-row stratified validation slice (seed 42), and 3,080 test tickets. The threshold sweep routes low-confidence tickets to human/fallback, picking the lowest threshold where auto-routed accuracy reaches 95% on validation. The fine-tuned and frontier comparison has not run; only the CPU baseline is measured.
The bar any model must beat is Rex's measured free CPU baseline: TF-IDF plus logistic regression on 8 vCPU, trained in 18.9 s, got 2,802 of 3,080 test tickets right (91.0%), macro-F1 0.910; its weakest fraud-adjacent intent, card_payment_not_recognised, scored 0.854. At a 0.45 threshold picked on the validation slice, the CPU baseline auto-routed 2,877 of 3,080 test tickets and sent 203 (6.6%) to human/fallback; on the test split, the CPU baseline got 2,714 of those 2,877 auto-routed tickets right (94.3%), against 95.0% on validation, which is why the gate rechecks on held-out data. The fine-tuned and frontier comparison has not run, so this piece reports no cost or accuracy for either model.
Banking77 is static and fully labelled, so it cannot test drift or label delay.
Specialist Promotion Gate Checklist
Each item is pass/fail.
- Router: threshold set on the validation slice, never the test set; any candidate model beats the 91.0% baseline.
- Human queue: owner named, queue cap set, cap alert fires in a test.
- Eval gate: candidate beats the incumbent on matured labels, no recall loss.
- Drift monitor: alerts fire in a replay, and an accuracy slip pages the on-call engineer.
- Retraining cadence: scheduled date and drift trigger recorded.
- Versioning: weights, prompt and policy share one bundle id in every trace.
- Rollback: previous bundle restored in a rehearsal.
What I Would Build Differently
I would gate frontier upgrades too [2]. Limits: Coinbase's benchmark is private and self-graded, the 9B has no live A/B test, and the paper's 9B roughly matches, not beats, the full SR-Fraud system [4]. Start there.
What we offer: run the gate's first check yourself with the Specialist Bake-off Kit, free for members: the Banking77 baseline, the validation threshold sweep and the cost-per-1,000 calculator.
References
- Coinbase, "Owning Intelligence, Part 3," Oct 8, 2026. https://www.coinbase.com/blog/part-three-owning-intelligence-series
- Coinbase, "Owning Intelligence, Part 2," Oct 7, 2026. https://www.coinbase.com/blog/part-two-owning-intelligence-series
- Casanueva et al., arXiv:2003.04807, 2020 (Banking77). https://arxiv.org/abs/2003.04807
- Tan, Ma, Zhang, "SR-Fraud," arXiv:2609.27287v2, Sept 30, 2026. https://arxiv.org/abs/2609.27287v2
- Ong et al., "RouteLLM," arXiv:2406.18665, 2024. https://arxiv.org/abs/2406.18665
- vLLM, "LoRA Adapters," docs (latest), accessed Oct 18, 2026. https://docs.vllm.ai/en/latest/features/lora.html
