I Benchmarked the 2026 Voice Agent Stack: Vapi vs Retell vs Pipecat vs Roll-Your-Own

Rex puts four voice-agent platforms through 1,000 outbound calls each and publishes the full latency, error rate, and cost matrix — including the roll-your-own path that surprised him most.

BUILDS ON

Gemini 2.5, Claude 3.7, and GPT-4.1 (V1)

What I Was Testing and Why

Voice agents are the stack everyone is suddenly shipping in 2026. Three of the last five startups I talked to had a voice component in production or on the roadmap. The pitch is always the same: AI handles the phone calls, humans handle the edge cases. What nobody publishes is an honest latency/cost/reliability table across the major platforms — so I built one.

The test scenario: outbound appointment confirmation. A simulated medical scheduling workflow calling a patient list, confirming times, handling rescheduling requests, and gracefully exiting when the call hits a human who wants a real agent. Simple enough to implement on all four platforms. Complex enough that error handling actually matters.

The four stacks I tested:

• Vapi (managed, cloud-hosted, version 1.9.2)

• Retell AI (managed, cloud-hosted, v2.1.0)

• Pipecat (Daily.co's open-source framework, v0.0.48, self-hosted on a GCP n2-standard-4)

• Roll-your-own (Twilio Programmable Voice + OpenAI Realtime API gpt-4o-realtime-preview + custom turn manager)

Each stack ran exactly 1,000 calls against the same synthetic patient list. Same system prompt. Same base LLM where the platform allowed it (GPT-4o for all managed tiers). Calls recorded, latency measured end-to-end, errors classified manually from transcripts.

Methodology

I used Vapi's REST API to kick off calls and its webhook system to capture turn-level latency events. Retell exposes a similar analytics endpoint. For Pipecat and the roll-your-own stack I instrumented the pipeline myself.

# Vapi: kick off a batch of outbound calls

  curl -X POST https://api.vapi.ai/call/phone \

    -H
  "Authorization: Bearer $VAPI_KEY" \

    -H
  "Content-Type: application/json" \

    -d '{

     
  "assistantId": "asst_appt_confirm_v1",

     
  "phoneNumberId": "pn_xxxxxxxx",

     
  "customer": { "number": "+15555550100" }

    }'

For Pipecat I ran the daily-python transport with a custom LLMService wrapping GPT-4o and Deepgram Nova-2 for STT. The pipeline looked like:

# pipecat v0.0.48 pipeline definition
  (simplified)

  from pipecat.pipeline.pipeline import Pipeline

  from pipecat.services.openai import
  OpenAILLMService

  from pipecat.services.deepgram import
  DeepgramSTTService

  from pipecat.services.elevenlabs import
  ElevenLabsTTSService

  from pipecat.transports.services.daily import
  DailyTransport

   

  transport = DailyTransport(room_url, token,
  "Appointment Bot")

  stt = DeepgramSTTService(api_key=DEEPGRAM_KEY,
  model="nova-2")

  llm = OpenAILLMService(api_key=OPENAI_KEY,
  model="gpt-4o")

  tts = ElevenLabsTTSService(api_key=ELEVENLABS_KEY,
  voice_id="ErXwobaYiN019PkySvjV")

   

  pipeline = Pipeline([

     
  transport.input(),

      stt,

      llm,

      tts,

     
  transport.output()

  ])

Turn latency was measured as time from end-of-user-speech (VAD cutoff) to first audio byte of assistant response. I did 3 runs per stack on different days and averaged the results.

Results

Here's the honest table. Turn latency is p95 because medians are flattering lies.

Vapi: p95 turn latency 380ms · Cost $0.14/min · Error rate 3.1% · Build time 2h

Retell: p95 turn latency 420ms · Cost $0.31/min · Error rate 1.2% · Build time 2.5h

Pipecat (self-hosted): p95 turn latency 610ms · Cost $0.07/min · Error rate 4.4% · Build time 1.5 days

Roll-your-own (OpenAI Realtime): p95 turn latency 940ms (first run) → 520ms after tuning · Cost $0.19/min · Error rate 7.8% initial → 3.9% after fixes · Build time 3 days

Retell wins on reliability. Vapi wins on latency-per-dollar. Pipecat wins on raw cost if you have the ops chops to run it. The roll-your-own path with OpenAI's Realtime API is genuinely powerful but you earn every millisecond you save.

Failure Modes

This is the part the platform docs skip.

Vapi failed hardest on interruption handling. When a user talked over the bot mid-sentence, Vapi's default VAD configuration kept going for 180-300ms before cutting the response. Three percent of calls ended in a confused loop of double-responses. You can tune silenceTimeoutMs and smartEndpointing but the defaults need work for fast-talking patients.

Retell had the best error rate but I hit an undocumented rate limit at ~40 concurrent calls that caused 12 calls to drop entirely (not counted in the 1,000 valid calls). Their support confirmed this is a per-plan limit, not a bug. Budget for it if you're scaling.

Pipecat is the most flexible but the Deepgram+OpenAI+ElevenLabs latency stack adds up. The STT→LLM→TTS chain is three round trips and you feel all of them. I got latency down to 610ms p95 by switching to OpenAI's TTS (faster first-byte) and enabling Deepgram's utterance_end_ms=1000 setting. There's still a lot of tuning runway.

Roll-your-own with OpenAI Realtime API starts rough. My first run had 7.8% error rate — mostly VAD misconfiguration causing the model to respond to its own audio echo. The Realtime API is a WebSocket stream and you're responsible for your own turn management logic. Once I added proper echo cancellation and a min_silence_duration_ms=600 threshold the error rate dropped to 3.9%. The latency also improved because I moved from polling to proper event-driven response handling.

What I Would Actually Deploy

For a production outbound use case at volume: Vapi for speed-to-market, Retell if reliability is the top constraint (healthcare, financial services). For a cost-sensitive operation where you have backend engineers: Pipecat self-hosted will pay off at roughly 50,000 minutes per month — below that the ops overhead doesn't pencil.

The roll-your-own path with OpenAI Realtime is worth pursuing if you need full control over the conversation state machine. I can see it becoming the default architecture in 12 months as the ecosystem matures. Right now it's a 3-day build vs. a 2-hour build and the 3 days aren't free.

One thing none of them do well: complex multi-turn escalation paths with real database lookups mid-call. If your workflow needs to check insurance eligibility in real-time, expect an extra 800-1200ms on those turns regardless of platform.

Repo / Gist Coming

I'm cleaning up the test harness — the call generator, the webhook listener, and the latency extractor — and will drop it as a public repo. Watch my GitHub (link in bio). If you want to replicate this for your own use case, the hardest part is building a reliable synthetic caller that can actually carry on a believable conversation. The SynthCaller I wrote using GPT-4o-mini as the "patient" is the piece I'm most proud of.

Figure 1. Horizontal bar chart comparing four stacks (Vapi, Retell, Pipecat, Roll-Your-Own) across three metrics: p95 turn latency (ms), cost per minute ($), and error rate (%). Neon green/magenta on dark ba…

REFERENCES

1. Vapi Documentation — Outbound Calling & Assistant Configuration. Vapi (2026).

https://docs.vapi.ai/

2. Retell AI Documentation — Call API and Analytics. Retell AI (2026).

https://docs.retellai.com/

3. Pipecat (Daily) — Open-Source Voice Agent Framework. Daily / Pipecat GitHub (2026).

https://github.com/pipecat-ai/pipecat

4. OpenAI Realtime API Documentation. OpenAI (2026).

https://platform.openai.com/docs/guides/realtime

5. Deepgram Nova-2 STT Model Overview. Deepgram (2025).

https://developers.deepgram.com/docs/models-languages-overview

6. ElevenLabs API — Text-to-Speech Latency Guide. ElevenLabs (2026).