The 2026 Voice Agent Stack Benchmark (Vapi vs Retell vs Pipecat vs Roll-Your-Own)
Benchmark voice stacks on the same calls. Platform docs will not give you a head-to-head table. This harness runs the same appointment-confirmation scenario across Vapi, Retell, Pipecat and a Twilio + OpenAI Realtime roll-your-own stack, and reports turn-level latency (p95, from end-of-user-speech to first assistant audio), cost per minute, and errors.
What's inside
- Adapters for all four stacks, a unified turn-event schema, and a simulated caller you can run as a rule-based patient or swap for a live model.
- A dry-run mode that needs no API keys, so you can exercise the comparison in CI or on a laptop before you spend.
- The published comparison from a 1,000-call run, so you can see how the four stacks traded off latency, cost and error rate — then re-run the same harness on your own workflow.
For operators choosing a voice stack who want comparable numbers, not a vendor demo. Free. The full pack is on this page.
Which voice stack fits your workflow? Compare Vapi, Retell, Pipecat and a custom realtime stack using one call scenario, turn-level latency, costs and errors. Start with a dry run that needs no API keys. Includes simulated callers and all stack adapters.
A reproducible benchmark harness that runs the same 1,000-call scenario across Vapi, Retell, Pipecat, and a Twilio + OpenAI Realtime roll-your-own stack — so you get the honest p95 latency, cost-per-minute, and error-rate table the platform docs won't publish, plus the code to re-run it on your own workflow.
Get the code: github.com/mskrado/poc/voice-agent-benchmark — MIT licensed,
pip install -e ".[dev]", python demo.py --scenario compare --calls 5.
Every voice-agent platform pitches the same story: AI handles the calls, humans handle the edge cases. What none of them publish is a head-to-head latency/cost/reliability table. This blueprint is the companion code to I Benchmarked the 2026 Voice Agent Stack — a config-driven harness that ran 1,000 outbound appointment-confirmation calls through Vapi, Retell, Pipecat, and OpenAI's Realtime API, and measured what actually happened.
What's inside:
· A SynthCaller simulated patient (rule-based by default, or swap in GPT-4o-mini for live runs) that can carry a believable, interruptible phone conversation
· Stack adapters for Vapi, Retell, Pipecat, and OpenAI Realtime, each with the article's reference metrics baked into config for side-by-side comparison
· A unified turn-event schema — latency measured from end-of-user-speech to first assistant audio byte, p95 not median
· Dry-run mode: run the full comparison with zero API keys, safe for CI or a local demo
What the numbers actually said: Retell had the best error rate (1.2%) but hit an undocumented concurrency limit; Vapi won on latency-per-dollar (380ms @ $0.14/min) but struggled with interruption handling; Pipecat was cheapest to run ($0.07/min) if you have the ops chops to self-host; the roll-your-own Realtime path went from a rough 940ms/7.8%-error first run to 520ms/3.9% after proper turn-management tuning.
Read the full breakdown: I Benchmarked the 2026 Voice Agent Stack: Vapi vs Retell vs Pipecat vs Roll-Your-Own

Comments (0)
Join the conversation!