The 2026 Voice Agent Stack Benchmark (Vapi vs Retell vs Pipecat vs Roll-Your-Own)

A reproducible benchmark harness that runs the same 1,000-call scenario across Vapi, Retell, Pipecat, and a Twilio + OpenAI Realtime roll-your-own stack — so you get the honest p95 latency, cost-per-minute, and error-rate table the platform docs won't publish, plus the code to re-run it on your own workflow.

Get the code: github.com/mskrado/poc/voice-agent-benchmark — MIT licensed, 

pip install -e ".[dev]", python demo.py --scenario compare --calls 5.

Every voice-agent platform pitches the same story: AI handles the calls, humans handle the edge cases. What none of them publish is a head-to-head latency/cost/reliability table. This blueprint is the companion code to I Benchmarked the 2026 Voice Agent Stack — a config-driven harness that ran 1,000 outbound appointment-confirmation calls through Vapi, Retell, Pipecat, and OpenAI's Realtime API, and measured what actually happened.

What's inside:

·        A SynthCaller simulated patient (rule-based by default, or swap in GPT-4o-mini for live runs) that can carry a believable, interruptible phone conversation

·        Stack adapters for Vapi, Retell, Pipecat, and OpenAI Realtime, each with the article's reference metrics baked into config for side-by-side comparison

·        A unified turn-event schema — latency measured from end-of-user-speech to first assistant audio byte, p95 not median

·        Dry-run mode: run the full comparison with zero API keys, safe for CI or a local demo

What the numbers actually said: Retell had the best error rate (1.2%) but hit an undocumented concurrency limit; Vapi won on latency-per-dollar (380ms @ $0.14/min) but struggled with interruption handling; Pipecat was cheapest to run ($0.07/min) if you have the ops chops to self-host; the roll-your-own Realtime path went from a rough 940ms/7.8%-error first run to 520ms/3.9% after proper turn-management tuning.

Read the full breakdown: I Benchmarked the 2026 Voice Agent Stack: Vapi vs Retell vs Pipecat vs Roll-Your-Own