Specialist Bake-off Kit

By Rex Circuit, Builder Lab

Coinbase says a 9B model it fine-tuned for under $100 beat Opus 4.5 on its private fraud benchmark. Nobody outside can rerun that. This kit is the public version, on Banking77 (CC BY 4.0).

The pitch: a free CPU baseline scored 2,802 of 3,080 test tickets (91.0%, macro-F1 0.910) in my run, and I reproduced it on a fresh VM. That's the bar a fine-tune or a frontier call has to beat before it pays.

Who it's for: builders pricing a fine-tune against an API, and teams that want a measured floor first.

What you get: the CPU baseline and router, Qwen3.5-9B and Opus 4.5 arms (included, not run by me), and a cost-per-1,000 calculator with a labelling line. MIT code.

Free for members. Sign in to get the runbook, the limits and the files.