Skip to content

Roark Live Bench · Suite v1.3 · Sweep of 20 November 2026

Voice AI models, ranked on live phone calls.

Every speech-to-text, language model and text-to-speech model, run in every combination on the same real call to the same agent. A model is ranked on every stack it runs in; the stacks are the evidence.

1234 stacks published24,562 scored callsPublished at 20 scored calls

Our picks

The read

  • 505 of 916 cascades complete the task in at least 90% of calls, and the top 10 sit within 1 point of each other. What separates the top stacks is speed and cost.
  • Median response time runs from 1.2 s to 6.2 s on the same agent and the same caller, a spread of 5.0 s that comes entirely from the models behind it.
  • No stack wins everything. The fastest, Speechmatics Linden-1 (own VAD) + Anthropic Claude Haiku 4.5 + AssemblyAI TTS-1 at 1.2 s, ranks 855 of 916 on task completion; the most complete is Speechmatics Linden-1 (own VAD) + OpenAI GPT-6.1 Sol + OpenAI TTS-1 HD at 99%.
  • The language model matters most: swapping it moves task completion by 9 points, against 7 for the transcriber and 5 for the speech engine.

Cascade vs Speech-to-Speech

The first choice to make. A cascade chains three models: one transcribes, one decides what to say, one speaks. A speech-to-speech model does all three at once. Both run the same agent on the same calls here.

  • Cascades lead on all 6 measures compared here.
  • Speech-to-speech costs 3.7 times as much per completed task: $0.24 for speech-to-speech against $0.07 for cascades.
CascadeSpeech-to-speech
Task completionTool useTime to first wordResponse timeBarge-in handlingCost per task255075100
MeasureCascadeSpeech-to-speech
Task completion
92% (leads)85%
Tool use
87% (leads)78%
Time to first word
0.6 s (leads)1.1 s
Response time
2.1 s (leads)2.1 s
Barge-in handling
100% (leads)96%
Cost per task
$0.07 (leads)$0.24

The three strongest of each, averaged. Cascades: Deepgram Flux + OpenAI GPT-6.1 Terra + AssemblyAI TTS-1; Deepgram Flux + Meta Muse Ember 1.0 + AssemblyAI TTS-1; Speechmatics Linden-1 (own VAD) + OpenAI GPT-6.1 Terra + AssemblyAI TTS-1. Speech-to-speech: grok-voice-think-fast-2.0; gpt-live-1 (backend gpt-6-luna); grok-voice-3.0. Each measure is scored 0 to 100 on its own scale, further out is better; the table gives the same figures in their own units.

Which to use

  • Choose speech-to-speech when the pause matters more than the outcome: short transactional calls, callers who interrupt, queues where every half second costs you callers. Expect to pay more, and choose the model carefully (grok-voice-3.0 leads here).
  • Choose a cascade when the call has to complete, when cost matters, or when you need to see which stage to fix. It is 12 points ahead on task completion here.
  • Costs are what each stack’s providers charged for these calls, at the rates in effect when they ran. Telephony and hosting are not included.

Speech-to-Text

The transcriber turns the caller's speech into text. It decides what the agent thinks it heard, and nothing after it can start until it finishes.

  • Soniox v6 and AssemblyAI-STT-3.7 Realtime-Agent are level: neither leads by more than the margin of error. Of the two, Soniox v6 costs least, at $0.008 a call.
  • Deepgram Flux delivers its transcript in 0.38 s, OpenAI Whisper-1 in 1.93 s: 1.54 s added to every turn of the call.
  • Soniox v6 is the most accurate: it changes the meaning of 0.2% of the caller's words, against 2% for Speechmatics Linden-1 (own VAD).
  • In 587 of 1,218 cascades the transcriber is the slowest step of the turn, whichever model runs it.

Roark score, best first

100Soniox v697AssemblyAI-STT-3.7 Realtime-Agent92Universal-3.6 Pro84Nova-382Universal-3.5 Pro73Enhanced (realtime)70Flux69Nova-269Linden-167Scribe v3 Live64Standard (realtime)57Linden-1 (own VAD)55Whisper-151GPT Transcribe
Every transcriber on every figure, sorted by roark score. Column headers sort the table.
#
1Soniox v6Soniox100 / 100. Semantic word error rate 0.2%, 71.4 of 71 points; Time to final transcript 0.85 s, 28.6 of 29 points(among the best)0.2%(among the best)0.85 s(among the best)$0.008(among the best)
2AssemblyAI-STT-3.7 Realtime-AgentAssemblyAI97 / 100. Semantic word error rate 0.6%, 68.1 of 71 points; Time to final transcript 0.86 s, 28.6 of 29 points(among the best)0.6%(among the best)0.86 s$0.030
3Universal-3.6 ProAssemblyAI92 / 100. Semantic word error rate 0.8%, 63.7 of 71 points; Time to final transcript 0.90 s, 28.6 of 29 points(among the best)0.8%(among the best)0.90 s$0.029
4Nova-3Deepgram84 / 100. Semantic word error rate 1.1%, 55.4 of 71 points; Time to final transcript 0.88 s, 28.6 of 29 points1.1%0.88 s$0.031
5Universal-3.5 ProAssemblyAI82 / 100. Semantic word error rate 1.1%, 53.9 of 71 points; Time to final transcript 0.90 s, 28.6 of 29 points1.1%0.90 s$0.029
6Enhanced (realtime)Speechmatics73 / 100. Semantic word error rate 1.4%, 44.4 of 71 points; Time to final transcript 0.98 s, 28.6 of 29 points1.4%0.98 s$0.054
7FluxDeepgram70 / 100. Semantic word error rate 1.6%, 41.4 of 71 points; Time to final transcript 0.38 s, 28.6 of 29 points1.6%0.38 s(among the best)$0.031
8Nova-2Deepgram69 / 100. Semantic word error rate 1.6%, 40.3 of 71 points; Time to final transcript 0.89 s, 28.6 of 29 points1.6%0.89 s$0.023
9Linden-1Speechmatics69 / 100. Semantic word error rate 1.6%, 40.2 of 71 points; Time to final transcript 0.98 s, 28.6 of 29 points1.6%0.98 s$0.020
10Scribe v3 LiveElevenLabs67 / 100. Semantic word error rate 1.7%, 38.4 of 71 points; Time to final transcript 0.88 s, 28.6 of 29 points1.7%0.88 s$0.015(among the best)
11Standard (realtime)Speechmatics64 / 100. Semantic word error rate 1.8%, 35.6 of 71 points; Time to final transcript 0.96 s, 28.6 of 29 points1.8%0.96 s$0.030
12Linden-1 (own VAD)Speechmatics57 / 100. Semantic word error rate 2%, 28.2 of 71 points; Time to final transcript 0.46 s, 28.6 of 29 points2%0.46 s(among the best)$0.020
13Whisper-1OpenAI55 / 100. Semantic word error rate 1.5%, 43.6 of 71 points; Time to final transcript 1.93 s, 10.9 of 29 points1.5%1.93 s$0.025
14GPT TranscribeOpenAI51 / 100. Semantic word error rate 1.9%, 31.0 of 71 points; Time to final transcript 1.47 s, 19.5 of 29 points1.9%1.47 s$0.018(among the best)

Open a model to see where it ranks and the stacks it works best in.

Cost against qualitySemantic word error rate · Cost per call
$0$0.02$0.04$0.06$0.080%1%2%3%Cheap and accurateAccurate, pricierCheap, less accuratePricier and less accurateSoniox v6Scribe v3 LiveGPT TranscribeLinden-1 (own VAD)Linden-1Nova-2Whisper-1Universal-3.6 ProUniversal-3.5 ProAssemblyAI-STT-3.7 Realtime-AgentStandard (realtime)FluxNova-3Enhanced (realtime)Cost per call · ← betterSemantic word error rate · better →
Speed against qualitySemantic word error rate · Time to final transcript
0 s1 s2 s3 s0%1%2%3%Fast and accurateAccurate, slowerFast, less accurateSlower and less accurateFluxLinden-1 (own VAD)Soniox v6AssemblyAI-STT-3.7 Realtime-AgentScribe v3 LiveNova-3Nova-2Universal-3.5 ProUniversal-3.6 ProStandard (realtime)Linden-1Enhanced (realtime)GPT TranscribeWhisper-1Time to final transcript · ← betterSemantic word error rate · better →

Language model

The language model decides what the agent says and does. It has the most influence on whether the task gets done, and usually takes the largest share of each turn.

  • OpenAI GPT-6.2 Sol, OpenAI GPT-6.1 Terra and OpenAI GPT-6.1 Sol are level: none leads by more than the margin of error. Of them, OpenAI GPT-6.1 Terra costs least, at $0.013 a call. Avoid Google Gemini 4 Neon: last on task completion (81%), instruction following (3.8 / 5) and time to first token (2.67 s).
  • Anthropic Claude Haiku 4.5 starts answering in 0.36 s, Google Gemini 4 Neon in 2.67 s: 2.32 s added to every turn of the call.
  • OpenAI GPT-6.2 Sol uses its tools most reliably: it handles its tools without a mistake on 86% of calls, against 42% for OpenAI GPT-5.1.
  • In 378 of 1,218 cascades the language model is the slowest step of the turn, whichever model runs it.

Roark score, best first

66GPT-6.2 Sol65GPT-6.1 Terra63GPT-6.1 Sol57Claude Sonnet 5.557Muse Ember 1.056Claude Haiku 5.549Gemini 3.8 Flash47GPT-4.145Claude Haiku 4.543Gemini 4 Hydrogen43GPT-4o mini42GPT-4o40GPT-5.120Gemini 4 Neon

Not ranked here for missing data: Groq GPT-OSS 120B (no task completion or tool use or instruction following or time to first token).

Every language model on every figure, sorted by roark score. Column headers sort the table.
#
1GPT-6.2 SolOpenAI66 / 100. Task completion 91%, 20.3 of 36 points; Tool use 86%, 19.9 of 27 points; Instruction following 4.3 / 5, 11.8 of 18 points; Time to first token 0.87 s, 13.7 of 18 points(among the best)91%(among the best)4.3 / 5(among the best)86%(among the best)0.87 s67%$0.025
2GPT-6.1 TerraOpenAI65 / 100. Task completion 90%, 19.0 of 36 points; Tool use 86%, 19.8 of 27 points; Instruction following 4.3 / 5, 11.6 of 18 points; Time to first token 0.83 s, 14.2 of 18 points(among the best)90%4.3 / 5(among the best)86%(among the best)0.83 s67%$0.013
3GPT-6.1 SolOpenAI63 / 100. Task completion 92%, 22.0 of 36 points; Tool use 82%, 17.5 of 27 points; Instruction following 4.3 / 5, 11.8 of 18 points; Time to first token 1.00 s, 12.1 of 18 points(among the best)92%(among the best)4.3 / 5(among the best)82%(among the best)1.00 s72%$0.018
4Claude Sonnet 5.5Anthropic57 / 100. Task completion 91%, 19.9 of 36 points; Tool use 75%, 13.6 of 27 points; Instruction following 4.2 / 5, 11.0 of 18 points; Time to first token 0.93 s, 12.9 of 18 points91%(among the best)4.2 / 575%0.93 s–$0.114
5Muse Ember 1.0Meta57 / 100. Task completion 90%, 18.4 of 36 points; Tool use 67%, 9.2 of 27 points; Instruction following 4.3 / 5, 11.8 of 18 points; Time to first token 0.56 s, 17.5 of 18 points90%4.3 / 5(among the best)67%0.56 s85%(among the best)$0.005(among the best)
6Claude Haiku 5.5Anthropic56 / 100. Task completion 90%, 17.3 of 36 points; Tool use 69%, 10.2 of 27 points; Instruction following 4.3 / 5, 12.2 of 18 points; Time to first token 0.63 s, 16.5 of 18 points90%4.3 / 5(among the best)69%0.63 s–$0.007(among the best)
7Gemini 3.8 FlashGoogle49 / 100. Task completion 92%, 21.2 of 36 points; Tool use 69%, 10.4 of 27 points; Instruction following 4.3 / 5, 11.5 of 18 points; Time to first token 1.53 s, 5.7 of 18 points92%(among the best)4.3 / 5(among the best)69%1.53 s–$0.015
8GPT-4.1OpenAI47 / 100. Task completion 89%, 17.1 of 36 points; Tool use 52%, 1.2 of 27 points; Instruction following 4.2 / 5, 11.2 of 18 points; Time to first token 0.52 s, 18.0 of 18 points89%4.2 / 552%0.52 s82%(among the best)$0.016
9Claude Haiku 4.5Anthropic45 / 100. Task completion 88%, 14.3 of 36 points; Tool use 52%, 0.9 of 27 points; Instruction following 4.3 / 5, 11.4 of 18 points; Time to first token 0.36 s, 18.2 of 18 points88%4.3 / 5(among the best)52%0.36 s(among the best)–$0.052
10Gemini 4 HydrogenGoogle43 / 100. Task completion 91%, 20.9 of 36 points; Tool use 70%, 11.0 of 27 points; Instruction following 4.0 / 5, 9.4 of 18 points; Time to first token 1.88 s, 1.4 of 18 points91%(among the best)4.0 / 570%1.88 s–$0.049
11GPT-4o miniOpenAI43 / 100. Task completion 86%, 10.1 of 36 points; Tool use 56%, 3.3 of 27 points; Instruction following 4.2 / 5, 11.1 of 18 points; Time to first token 0.50 s, 18.2 of 18 points86%4.2 / 556%0.50 s(among the best)82%(among the best)$0.001(among the best)
12GPT-4oOpenAI42 / 100. Task completion 86%, 11.0 of 36 points; Tool use 53%, 1.8 of 27 points; Instruction following 4.2 / 5, 11.2 of 18 points; Time to first token 0.45 s, 18.2 of 18 points86%4.2 / 553%0.45 s(among the best)–$0.021
13GPT-5.1OpenAI40 / 100. Task completion 87%, 12.9 of 36 points; Tool use 42%, 0.0 of 27 points; Instruction following 4.1 / 5, 10.4 of 18 points; Time to first token 0.59 s, 17.1 of 18 points87%4.1 / 542%0.59 s74%$0.026
14Gemini 4 NeonGoogle20 / 100. Task completion 81%, 1.2 of 36 points; Tool use 70%, 10.7 of 27 points; Instruction following 3.8 / 5, 7.7 of 18 points; Time to first token 2.67 s, 0.0 of 18 points81%3.8 / 570%2.67 s–$0.078

Open a model to see where it ranks and the stacks it works best in.

Cost against qualityTask completion · Cost per call
$0$0.05$0.1$0.1575%80%85%90%95%Cheap and reliableReliable, pricierCheap, less reliablePricier and less reliableGPT-4o miniMuse Ember 1.0Claude Haiku 5.5GPT-6.1 TerraGemini 3.8 FlashGPT-4.1GPT-6.1 SolGPT-4oGPT-6.2 SolGPT-5.1Gemini 4 HydrogenClaude Haiku 4.5Gemini 4 NeonClaude Sonnet 5.5Cost per call · ← betterTask completion · better →
Speed against qualityTask completion · Time to first token
0 s1 s2 s3 s4 s75%80%85%90%95%Fast and reliableReliable, slowerFast, less reliableSlower and less reliableClaude Haiku 4.5GPT-4oGPT-4o miniGPT-4.1Muse Ember 1.0GPT-5.1Claude Haiku 5.5GPT-6.1 TerraGPT-6.2 SolClaude Sonnet 5.5GPT-6.1 SolGemini 3.8 FlashGemini 4 HydrogenGemini 4 NeonTime to first token · ← betterTask completion · better →

Text-to-Speech

The speech engine is the voice the caller hears, and the last step before they hear anything.

  • AssemblyAI TTS-1, Cartesia Sonic-3.6 and Cartesia Sonic-3.5 are level: none leads by more than the margin of error. Of them, AssemblyAI TTS-1 costs least, at $0.052 a call. Avoid OpenAI TTS-1 HD: last on readback accuracy (81%), word error rate (7.4%) and time to first audio byte (2.09 s).
  • AssemblyAI TTS-1 starts speaking in 0.04 s, OpenAI TTS-1 HD in 2.09 s: 2.04 s added to every turn of the call.
  • AssemblyAI TTS-1 is also the most accurate: it reads codes, amounts and dates back correctly 90% of the time, against 81% for OpenAI TTS-1 HD.
  • Rime Coda sounds the most human: 3.5 / 5, against 3.2 / 5 for ElevenLabs Turbo v2.5.
  • In 119 of 1,218 cascades the speech engine is the slowest step of the turn, most often with OpenAI TTS-1 (65 of them).

Roark score, best first

84TTS-183Flux TTS83Sonic-3.683Sonic-3.583v4 Turbo79Sonic-376Turbo v2.576Flash v2.574Coda58TTS-144TTS-1 HD
Every speech engine on every figure, sorted by roark score. Column headers sort the table.
#
1TTS-1AssemblyAI84 / 100. Readback accuracy 90%, 28.5 of 38 points; Word error rate 4.9%, 17.8 of 25 points; Time to first audio byte 0.04 s, 25.0 of 25 points; Voice consistency 100%, 12.3 of 13 points(among the best)4.9%90%(among the best)3.3 / 5100%(among the best)0.04 s(among the best)$0.052(among the best)
2Flux TTSDeepgram83 / 100. Readback accuracy 90%, 27.7 of 38 points; Word error rate 4.7%, 18.3 of 25 points; Time to first audio byte 0.13 s, 25.0 of 25 points; Voice consistency 100%, 12.3 of 13 points(among the best)4.7%90%(among the best)3.2 / 5100%(among the best)0.13 s$0.116
3Sonic-3.6Cartesia83 / 100. Readback accuracy 90%, 27.7 of 38 points; Word error rate 4.8%, 18.0 of 25 points; Time to first audio byte 0.12 s, 25.0 of 25 points; Voice consistency 100%, 12.3 of 13 points(among the best)4.8%90%(among the best)3.4 / 5(among the best)100%(among the best)0.12 s(among the best)$0.156
4Sonic-3.5Cartesia83 / 100. Readback accuracy 89%, 27.0 of 38 points; Word error rate 4.6%, 18.4 of 25 points; Time to first audio byte 0.12 s, 25.0 of 25 points; Voice consistency 100%, 12.5 of 13 points(among the best)4.6%(among the best)89%3.3 / 5100%(among the best)0.12 s(among the best)$0.153
5v4 TurboElevenLabs83 / 100. Readback accuracy 89%, 27.1 of 38 points; Word error rate 4.4%, 18.9 of 25 points; Time to first audio byte 0.11 s, 25.0 of 25 points; Voice consistency 99%, 11.7 of 13 points(among the best)4.4%(among the best)89%3.4 / 5(among the best)99%0.11 s(among the best)$0.451
6Sonic-3Cartesia79 / 100. Readback accuracy 87%, 25.6 of 38 points; Word error rate 5.4%, 16.6 of 25 points; Time to first audio byte 0.16 s, 25.0 of 25 points; Voice consistency 100%, 12.3 of 13 points5.4%87%3.2 / 5100%(among the best)0.16 s$0.149
7Turbo v2.5ElevenLabs76 / 100. Readback accuracy 82%, 20.6 of 38 points; Word error rate 4.4%, 18.9 of 25 points; Time to first audio byte 0.15 s, 25.0 of 25 points; Voice consistency 99%, 11.7 of 13 points4.4%(among the best)82%3.2 / 599%0.15 s$0.435
8Flash v2.5ElevenLabs76 / 100. Readback accuracy 82%, 20.6 of 38 points; Word error rate 4.7%, 18.2 of 25 points; Time to first audio byte 0.12 s, 25.0 of 25 points; Voice consistency 99%, 12.0 of 13 points4.7%82%3.3 / 599%0.12 s(among the best)$0.399
9CodaRime74 / 100. Readback accuracy 85%, 23.1 of 38 points; Word error rate 6.5%, 13.8 of 25 points; Time to first audio byte 0.16 s, 25.0 of 25 points; Voice consistency 99%, 12.1 of 13 points6.5%85%3.5 / 5(among the best)99%0.16 s$0.107
10TTS-1OpenAI58 / 100. Readback accuracy 83%, 21.7 of 38 points; Word error rate 5.2%, 16.9 of 25 points; Time to first audio byte 1.12 s, 7.3 of 25 points; Voice consistency 100%, 12.5 of 13 points5.2%83%3.4 / 5(among the best)100%(among the best)1.12 s$0.028(among the best)
11TTS-1 HDOpenAI44 / 100. Readback accuracy 81%, 20.0 of 38 points; Word error rate 7.4%, 11.6 of 25 points; Time to first audio byte 2.09 s, 0.0 of 25 points; Voice consistency 100%, 12.5 of 13 points7.4%81%3.4 / 5(among the best)100%(among the best)2.09 s$0.049(among the best)

Open a model to see where it ranks and the stacks it works best in.

Cost against qualityReadback accuracy · Cost per call
$0$0.2$0.4$0.675%80%85%90%95%Cheap and accurateAccurate, pricierCheap, less accuratePricier and less accurateTTS-1TTS-1 HDTTS-1CodaFlux TTSSonic-3Sonic-3.5Sonic-3.6Flash v2.5Turbo v2.5v4 TurboCost per call · ← betterReadback accuracy · better →
Speed against qualityReadback accuracy · Time to first audio byte
0 s1 s2 s3 s75%80%85%90%95%Fast and accurateAccurate, slowerFast, less accurateSlower and less accurateTTS-1v4 TurboSonic-3.6Flash v2.5Sonic-3.5Flux TTSTurbo v2.5CodaSonic-3TTS-1TTS-1 HDTime to first audio byte · ← betterReadback accuracy · better →

Speech-to-Speech models

One model listens, reasons and speaks, so there is nothing to swap and no stages to split. Each is judged on the whole call, with the same calls and the same scoring as every cascade.

  • The model matters as much as the architecture: xAI Grok Voice 3.0 completes 87% of tasks, ElevenLabs Realtime v1 Mini 42%.
  • Phonic 1 answers in 1.5 s against 4.2 s for Google Gemini 2.5 Flash native audio.
  • Price varies as much as quality: $0.06 per completed task for Amazon Nova 2 Sonic, $0.69 for OpenAI GPT Realtime 2.

Roark score, best first

67Grok Voice 3.066GPT Live 1 (Sol)65GPT Live 1 (Luna)65Grok Voice Think Fast 2.064GPT-Live 1.1 (Luna)60Phonic 153GPT Realtime 2.153Gemini 3.8 Live49GPT Realtime 248Realtime v1 Max49Ultravox v0.748Gemini 3.1 Flash Live (preview)46Nova 2 Sonic46Realtime v1 Mini45GPT Realtime 2.1 mini36Gemini 2.5 Flash native audio

Whiskers show the 95% interval. The shaded models tie for first: these calls cannot separate them.

Every speech-to-speech model on every figure, sorted by roark score. Column headers sort the table.
#
1= (tied)Grok Voice 3.0xAI67 / 100±4.7. Task completion 87%, 8.0 of 23 points; Response time 1.9 s, 18.2 of 18 points; Instruction following 4.5 / 5, 10.5 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 66%, 4.4 of 14 points; Time to first word 0.9 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points(among the best)87%(among the best)4.5 / 5(among the best)66%1.9 s(among the best)0.9 s94%3.0 / 5100%(among the best)$0.26
2= (tied)GPT Live 1 (Sol)OpenAI66 / 100±3.8. Task completion 85%, 6.0 of 23 points; Response time 2.3 s, 16.6 of 18 points; Instruction following 4.2 / 5, 8.2 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 87%, 10.2 of 14 points; Time to first word 1.4 s, 3.9 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points(among the best)85%4.2 / 587%(among the best)2.3 s1.4 s77%3.1 / 5100%(among the best)$0.24
3= (tied)GPT Live 1 (Luna)OpenAI65 / 100±3.5. Task completion 85%, 5.7 of 23 points; Response time 2.3 s, 16.6 of 18 points; Instruction following 4.2 / 5, 8.4 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 86%, 9.9 of 14 points; Time to first word 1.5 s, 3.8 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points(among the best)85%4.2 / 586%(among the best)2.3 s1.5 s93%3.1 / 5100%(among the best)$0.21
4= (tied)Grok Voice Think Fast 2.0xAI65 / 100±3.7. Task completion 82%, 2.8 of 23 points; Response time 2.2 s, 17.4 of 18 points; Instruction following 4.6 / 5, 10.7 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 82%, 8.6 of 14 points; Time to first word 0.9 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points(among the best)82%4.6 / 5(among the best)82%(among the best)2.2 s(among the best)0.9 s100%(among the best)3.6 / 5100%(among the best)$0.26
5= (tied)GPT-Live 1.1 (Luna)OpenAI64 / 100±3.9. Task completion 86%, 6.4 of 23 points; Response time 2.3 s, 17.0 of 18 points; Instruction following 4.2 / 5, 7.8 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 81%, 8.5 of 14 points; Time to first word 1.9 s, 3.1 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points86%(among the best)4.2 / 581%2.3 s1.9 s75%3.2 / 5100%(among the best)$0.22
6= (tied)Phonic 1Phonic60 / 100±5.4. Task completion 86%, 6.4 of 23 points; Response time 1.5 s, 18.2 of 18 points; Instruction following 4.4 / 5, 9.2 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 59%, 2.4 of 14 points; Time to first word 2.1 s, 2.9 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points86%(among the best)4.4 / 5(among the best)59%1.5 s(among the best)2.1 s–3.2 / 5100%(among the best)$0.28
7= (tied)GPT Realtime 2.1OpenAI53 / 100±5.7. Task completion 78%, 0.0 of 23 points; Response time 2.6 s, 15.6 of 18 points; Instruction following 3.8 / 5, 5.6 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 75%, 6.8 of 14 points; Time to first word 0.8 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 99%, 4.3 of 5 points78%3.8 / 575%2.6 s0.8 s(among the best)83%3.9 / 5(among the best)99%$0.43
8= (tied)Gemini 3.8 LiveGoogle53 / 100±4.8. Task completion 78%, 0.0 of 23 points; Response time 2.8 s, 14.6 of 18 points; Instruction following 4.3 / 5, 8.7 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 69%, 5.2 of 14 points; Time to first word 1.6 s, 3.6 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points78%4.3 / 569%2.8 s1.6 s100%(among the best)4.1 / 5(among the best)100%(among the best)$0.14(among the best)
9= (tied)GPT Realtime 2OpenAI49 / 100±6.7. Task completion 76%, 0.0 of 23 points; Response time 2.9 s, 14.0 of 18 points; Instruction following 3.8 / 5, 5.4 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 64%, 3.7 of 14 points; Time to first word 0.8 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points76%3.8 / 564%2.9 s0.8 s(among the best)91%3.7 / 5100%(among the best)$0.69
10= (tied)Realtime v1 MaxElevenLabs48 / 100±7.5. Task completion 56%, 0.0 of 23 points; Response time 2.6 s, 15.6 of 18 points; Instruction following 4.0 / 5, 6.7 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 40%, 0.0 of 14 points; Time to first word 0.7 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points56%4.0 / 540%2.6 s0.7 s(among the best)76%3.9 / 5(among the best)100%(among the best)$0.65
11= (tied)Ultravox v0.7Ultravox49 / 100±5.0. Task completion 79%, 0.0 of 23 points; Response time 2.4 s, 16.2 of 18 points; Instruction following 3.7 / 5, 4.9 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 61%, 3.0 of 14 points; Time to first word 1.6 s, 3.6 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points79%3.7 / 561%2.4 s1.6 s94%3.6 / 5100%(among the best)$0.22
12= (tied)Gemini 3.1 Flash Live (preview)Google48 / 100±5.1. Task completion 80%, 0.0 of 23 points; Response time 2.7 s, 15.1 of 18 points; Instruction following 4.2 / 5, 8.0 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 38%, 0.0 of 14 points; Time to first word 1.5 s, 3.8 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points80%4.2 / 538%2.7 s1.5 s100%(among the best)4.0 / 5(among the best)100%(among the best)$0.15
13= (tied)Nova 2 SonicAmazon46 / 100±5.1. Task completion 69%, 0.0 of 23 points; Response time 3.0 s, 13.8 of 18 points; Instruction following 4.2 / 5, 7.9 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 48%, 0.0 of 14 points; Time to first word 1.5 s, 3.7 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points69%4.2 / 548%3.0 s1.5 s100%(among the best)3.5 / 5100%(among the best)$0.06(among the best)
14= (tied)Realtime v1 MiniElevenLabs46 / 100±7.1. Task completion 42%, 0.0 of 23 points; Response time 2.7 s, 15.1 of 18 points; Instruction following 3.8 / 5, 5.1 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 34%, 0.0 of 14 points; Time to first word 0.7 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points42%3.8 / 534%2.7 s0.7 s(among the best)94%3.4 / 5100%(among the best)$0.64
15= (tied)GPT Realtime 2.1 miniOpenAI45 / 100±6.7. Task completion 70%, 0.0 of 23 points; Response time 2.8 s, 14.5 of 18 points; Instruction following 3.6 / 5, 4.4 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 51%, 0.3 of 14 points; Time to first word 1.0 s, 4.5 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points70%3.6 / 551%2.8 s1.0 s84%3.9 / 5(among the best)100%(among the best)$0.14(among the best)
16Gemini 2.5 Flash native audioGoogle36 / 100±5.4. Task completion 77%, 0.0 of 23 points; Response time 4.2 s, 8.2 of 18 points; Instruction following 3.9 / 5, 6.2 of 14 points; Semantic word error rate 1.1% (field median, not measured), 10.3 of 14 points; Tool use 40%, 0.0 of 14 points; Time to first word 3.4 s, 0.9 of 5 points; Readback accuracy 86% (field median, not measured), 3.0 of 5 points; Word error rate 5.2% (field median, not measured), 3.1 of 5 points; Voice consistency 100%, 4.5 of 5 points77%3.9 / 540%4.2 s3.4 s100%(among the best)3.7 / 5100%(among the best)$0.08(among the best)

Open a model for its full report, every figure it was scored on and the calls behind them.

Cost against qualityTask completion · Cost per task
$0$0.5$120%40%60%80%100%Cheap and reliableReliable, pricierCheap, less reliablePricier and less reliableNova 2 SonicGemini 2.5 Flash native audioGPT Realtime 2.1 miniGemini 3.8 LiveGemini 3.1 Flash Live (preview)GPT Live 1 (Luna)GPT-Live 1.1 (Luna)Ultravox v0.7GPT Live 1 (Sol)Grok Voice 3.0Grok Voice Think Fast 2.0Phonic 1GPT Realtime 2.1Realtime v1 MiniRealtime v1 MaxGPT Realtime 2Cost per task · ← betterTask completion · better →
Speed against qualityTask completion · Response time
0 s1 s2 s3 s4 s5 s20%40%60%80%100%Fast and reliableReliable, slowerFast, less reliableSlower and less reliablePhonic 1Grok Voice 3.0Grok Voice Think Fast 2.0GPT-Live 1.1 (Luna)GPT Live 1 (Luna)GPT Live 1 (Sol)Ultravox v0.7Realtime v1 MaxGPT Realtime 2.1Gemini 3.1 Flash Live (preview)Realtime v1 MiniGemini 3.8 LiveGPT Realtime 2.1 miniGPT Realtime 2Nova 2 SonicGemini 2.5 Flash native audioResponse time · ← betterTask completion · better →

Every stack

The evidence behind the model boards: each combination on its own calls, scored on the same recipe. Open a stack for its full report, every figure it was scored on and the calls behind them.

Roark score, top 15

Whiskers show the 95% interval. The shaded stacks tie for first: these calls cannot separate them.

Each score carries its 95% margin of error. A rank is the best place a stack could hold once every margin is allowed for, so stacks the calls cannot tell apart share one and are listed by score; hover a rank for the worst place it could hold. 38 stacks are level with the leader: their calls cannot tell them apart. Among them, 10 OpenAI GPT-6.2 Sol, 9 Meta Muse Ember 1.0, 7 OpenAI GPT-6.1 Sol, 5 OpenAI GPT-6.1 Terra, 3 Anthropic Claude Haiku 5.5, 2 Google Gemini 4 Hydrogen, 1 Anthropic Claude Sonnet 5.5 and 1 Google Gemini 3.8 Flash; 18 Soniox, 16 AssemblyAI and 4 Deepgram; 14 ElevenLabs, 12 Cartesia, 6 AssemblyAI, 4 Deepgram and 2 Rime. A cascade's figures are 25% its own calls and 75% what every other stack shows about its transcriber, language model and voice. Not ranked here for missing data: AssemblyAI Universal-3.6 Pro + Google Gemini 4 Neon + AssemblyAI TTS-1 (no response time or readback accuracy or voice consistency); Soniox v6 + OpenAI GPT-6.1 Sol + Cartesia Sonic-3 (no response time or readback accuracy or voice consistency); Speechmatics Linden-1 + OpenAI GPT-6.1 Sol + OpenAI TTS-1 HD (no response time or tool use or readback accuracy or voice consistency).

Loading every stack

By industry

Each industry ranked on its own calls: the same five callers, its own tasks and tools. A stack has a handful of calls in each, so a cascade is ranked on what its transcriber, language model and voice show across every stack in that industry, and a speech-to-speech model on its own calls there. Pairings show on the main board. Each row reads across: the main board's figure first, then every industry's on the same figure.

No stack matches these filters.

No stack matches

Each industry column is that industry's own board, every row it ranks. A dash is a row the industry does not rank on this figure; the arrow beside an industry opens its full board.

How it works

Every number on this page comes from real phone calls, placed and scored the same way for every stack.

01

Real calls, four industries, five callers

Roark places real phone calls over the public phone network to a reference agent for each of four industries: rescheduling a clinic appointment, booking an urgent home repair, opening an auto insurance claim and fixing an internet outage. Each agent has its own prompt and tools, identical for every stack. The tasks carry the details real calls trip on: a corrected date of birth, a policy number spelled out, a constraint added halfway through.

Every industry is called by the same five simulated callers: a quiet US English line, a Spanish-accented caller on a city street, an Indian English caller in a busy café, a flustered Southern caller with a baby crying, and a fast caller who interrupts over construction noise. They are equally clear about what they want, so a call fails because of the agent, not the caller. Only the models change, so any difference on the board comes from them.

Each caller rings each industry once for a cascade and four times for a speech-to-speech model. A cascade's figures also draw on its parts, heard in dozens of other stacks; a speech-to-speech model has only its own calls, so it takes more of them.

That makes industries comparable over the same callers, and callers over the same tasks, and the board can be re-ranked by either. One caller within one industry is too few calls to judge, so the board never shows it.

932

stacks

20

calls per stack

19,550

calls scored

02

Every model, in every combination

14 transcribers, 14 language models and 11 voices are run in every combination with each other, beside 16 speech-to-speech models. A model's own figures pool every stack it runs in, adjusted for its partners in each one, so every model is compared as if it ran with the same partners. A transcriber's figure rests on hundreds of calls with every language model and every voice, not on one lucky pairing.

That measures a model where it actually works: inside a running agent, on live phone audio, through interruptions, corrections and real turn-taking. A test that feeds a model clean audio files or one prompt at a time never sees any of that.

03

One score, weighted by what matters on a call

Each section ranks on a Roark score. Every figure scores 0 to 100 between two published points: full marks where doing better makes no difference to the caller, zero where the call breaks down. Each then counts at its weight, and the weights follow one order everywhere: whether the caller got what they rang for, then whether the agent got it right, then speed.

A stack, cascade or speech-to-speech, is scored on the outcome and the reply first, then on what each of its components got right on its own calls: the words its transcriber got wrong in a way that changes the meaning, whether its voice read codes and amounts back right and kept one steady voice, and whether its tools were handled. How human a voice sounds is shown on the boards and not scored. A component that leads its section lifts every stack that carries it, by the same recipe for every stack. A figure a stack cannot carry, such as a semantic word error rate for a speech-to-speech model with no transcript, is credited at the field median, so the stack is neither flattered nor punished for it.

On a full sweep each cascade's figure is 25% what its own calls measured and 75% what every other stack shows about its transcriber, language model and voice, fitted across the whole sweep. The split is the same for every stack, so a lucky run does not lead and a stack's score does not move with how noisy its calls were. In an industry a stack has a handful of calls, so a cascade's figure there is what its transcriber, language model and voice show across every stack in that industry's calls, and a speech-to-speech model's is its own calls there. How a particular pairing beats or misses its parts shows on the main board, where each stack has all of its calls. The best overall, best value and best speech-to-speech picks, and the best stack in each industry, are each the one whose score is highest at the low end of its margin of error, so a few lucky calls cannot carry a stack past one whose score rests on hundreds. Stacks that share a rank are listed by score, so the pick can sit below a higher score it cannot be told from, and its reason says how many stacks are level with it.

That order comes from real-world voice agent calls, where getting the conversation right predicts how a call ends more strongly than speed does. The latency points come from the same calls: replying faster than about 2 seconds made no measurable difference, and calls went measurably worse once the agent took more than about 3 seconds to reply.

Stacks and speech-to-speech

FigureWeightFull marksZero
Task completionscenario_adherence_check23%100%80%
An agent that completes fewer than four calls in five is not fit to deploy. On a full sweep, a cascade's figure is 25% what its own calls (20 or more) measured and 75% what every other stack shows about its transcriber, language model and voice, the same split for every stack, so a lucky run does not lead. Where the sweep is too small for that, and for a speech-to-speech model, the figure is the stack's own. In an industry a stack has a handful of calls, so a cascade's figure there is what its transcriber, language model and voice show across every stack in that industry's calls, and a speech-to-speech model's is its own calls there. How a particular pairing beats or misses its parts shows on the main board, where each stack has all of its calls.
Response timeresponse_time18%2.0 s6.0 s
In real calls, replying faster than 2 s made no measurable difference, and calls went worse past 3 s.
Instruction followinginstruction_follow14%5.0 / 53.0 / 5
Every model scores between 3 and 5 on the judge's 1 to 5 scale, so that is the range.
Semantic word error ratestt_semantic_word_error_rate14%0.5%3%
At 0.5% the best transcribers change the meaning of one word in two hundred on phone audio; at 3% one word in thirty sends the agent the wrong way.
Tool usetool_use14%100%50%
An agent that mishandles its tools on half its calls cannot be trusted with the systems it runs on.
Time to first wordtime_to_first_word5%1.0 s4.0 s
A greeting within 1 s sounds natural; 4 s of silence sounds like a dead line.
Readback accuracytts_entity_readback_accuracy5%100%60%
Reading a code, amount or date wrong breaks the task it was for; the weakest voices get two in five wrong.
Word error ratetts_word_error_rate5%2%12%
At 2% the voice says what it was given; at 12% one word in eight comes out wrong.
Voice consistencyvoice_consistency_check5%100%80%
A voice that changes speaker on one call in five is not one a business can put its name to.

Shown, not scored: Turn budget, Barge-in handlingbarge_in_yield_appropriateness, Human-likenessvoice_human_likeness

Counted only with price factored in: Cost per task

Language model

FigureWeightFull marksZero
Task completionscenario_adherence_check36%100%80%
An agent that completes fewer than four calls in five is not fit to deploy.
Tool usetool_use27%100%50%
An agent that mishandles its tools on half its calls cannot be trusted with the systems it runs on.
Instruction followinginstruction_follow18%5.0 / 53.0 / 5
Every model scores between 3 and 5 on the judge's 1 to 5 scale, so that is the range.
Time to first tokenllm_time_to_first_token18%0.50 s2.00 s
Under 0.5 s is not noticeable; at 2 s the first token alone takes two thirds of the 3 s past which real calls go worse.

Shown, not scored: Prompt cache hitsllm_prompt_cache_hit_rate

Counted only with price factored in: Cost per call

Speech-to-Text

FigureWeightFull marksZero
Semantic word error ratestt_semantic_word_error_rate71%0.5%3%
At 0.5% the best transcribers change the meaning of one word in two hundred on phone audio; at 3% one word in thirty sends the agent the wrong way.
Time to final transcriptstt_time_to_final_transcript29%1.00 s2.50 s
The fastest transcribers finish in about 1 s; past 2.5 s the transcript alone uses most of a natural reply time.

Shown, not scored: Word error ratestt_word_error_rate

Counted only with price factored in: Cost per call

Text-to-Speech

FigureWeightFull marksZero
Readback accuracytts_entity_readback_accuracy38%100%60%
Reading a code, amount or date wrong breaks the task it was for; the weakest voices get two in five wrong.
Word error ratetts_word_error_rate25%2%12%
At 2% the voice says what it was given; at 12% one word in eight comes out wrong.
Time to first audio bytetts_time_to_first_byte25%0.20 s1.50 s
Under 0.2 s is not noticeable; 1.5 s is half the point where calls go worse.
Voice consistencyvoice_consistency_check13%100%80%
A voice that changes speaker on one call in five is not one a business can put its name to.

Shown, not scored: Human-likenessvoice_human_likeness

Counted only with price factored in: Cost per call

Factor in price and cost takes 25% of the score; the other weights keep their proportions. Price is scored on a log scale, so each doubling costs the same points.

A figure listed as shown, not scored is on the section’s boards and outside its score. Where a section has not collected a scored figure yet, the others share its weight.

04

Every call recorded and judged

Every call is recorded. Judge models score each transcript for the outcome and the conversation, and the timings come from the recording. Every figure links to the calls behind it.

Roark is on both sides of every call: it plays the caller, so it keeps the exact text its caller said, and it runs the agent under test, so it sees what the agent’s transcriber heard and what its model wrote for its voice. The transcript and voice figures come from lining those up, with no hand-made transcript in between. Each carries a margin from the synthetic voice or Roark’s transcriber in the loop. Every model faces the same ones under the same conditions, so the figures compare models on the same calls through the same voice and the same ear, though those slips need not fall evenly on each model. The rates will not match a vendor’s figure on a curated set; read them against each other.

An example: one call from this sweep

Speechmatics Linden-1 (own VAD) + OpenAI GPT-6.1 Sol + OpenAI TTS-1 HD, 20 November 2026. Highlighted turns are the ones that affected a score. Every stack's report links to its calls.

Transcript, verbatim.

Read the full methodology

See how your own agent compares

Roark runs the same kind of calls against your agent, on your scenarios, with the same scoring.

Test your agent

GPT Transcribe + Claude Sonnet 5.5 + v4 Turbo

Cascade20 scored calls · Nov 15, 2026

Rank on Roark score

#70of 929 stacks

60 / 100

Leader AssemblyAI-STT-3.7 Realtime-Agent + OpenAI GPT-6.2 Sol + ElevenLabs Flash v2.5, 79 / 100

Where it ranks

Roark score with price55 / 100395 of 929LeaderSoniox v6 + Muse Ember 1.0 + TTS-181 / 100
Task completion88%697 of 932LeaderLinden-1 (own VAD) + GPT-6.1 Sol + TTS-1 HD99%
Tool use78%172 of 908LeaderAssemblyAI-STT-3.7 Realtime-Agent + GPT-6.2 Sol + Flash v2.5100%
Time to first word0.6 s57 of 932LeaderUniversal-3.5 Pro + GPT-6.1 Sol + Flash v2.50.5 s
Response time3.6 s538 of 929LeaderLinden-1 (own VAD) + Claude Haiku 4.5 + TTS-11.2 s
Barge-in handling100%97 of 538LeaderUniversal-3.6 Pro + Gemini 4 Neon + TTS-1100%
Turn budget3.6 s494 of 858LeaderLinden-1 (own VAD) + Claude Haiku 4.5 + TTS-11.2 s
Cost per task$0.73884 of 930LeaderSoniox v6 + GPT-6.1 Terra + TTS-1$0.04

Its models

Task completion

Task completion

How much of the caller’s task the agent completed: the share of the scenario’s checks a judge model found met in the transcript, averaged over calls. Before v1.3, the share of calls judged 4 or better out of 5.

20 calls
88%Completed

#812 of 1184 on the board · best 99%

Flow adherence

How closely the agent followed the intended conversation flow, judged per call. 5 = fully followed.

20 calls
4.6/ 5avg

med

5

#1 of 1184 on the board · best 5.0 / 5

Instruction following

How well the agent stuck to the instructions in its own prompt. 5 = followed throughout.

20 calls
4.2/ 5avg

med

4

#94 of 1184 on the board · best 5.0 / 5

Turn-taking

Barge-in handling

When the caller interrupts, how often the agent made the right call: yielding to a real correction, and holding the floor through a filler.

16 barge-ins
100%Handled correctly(16/16 barge-ins)

#1 of 698 on the board · best 100%

Incorrect interruptions

How often the agent interrupted the caller when it should have kept listening.

20 calls
0.00%avg

p50

0.00%

p95

0.00%

#1 of 1184 on the board · best 0%

Overtalk

Share of the conversation where both sides were speaking over each other.

20 calls
2.71%avg

p50

0.23%

p95

8.75%

#605 of 1184 on the board · best 0.0%

Agent self-cutoffs

Times the agent cut its own sentence off mid-word.

20 calls
0.00per callavg

p50

0.00 per call

p95

0.00 per call

#1 of 1184 on the board · best 0.0 per call

Understanding & effort

Caller effort

How hard the caller had to work: repeating themselves, correcting, re-explaining. 1 = effortless.

Not scored by this sweep.

Repeated-info loops

Times per call the agent re-asked for something the caller had already given.

27 calls
0.52per callavg

p50

1.00 per call

p95

1.00 per call

#315 of 825 on the board · best 0.0 per call

Speech-to-Text

Semantic word error rate

The caller's words the agent's transcriber got wrong in a way that changes the meaning, as a judge reads each difference.

20 calls
1.45%avg

p50

0.00%

p95

9.45%

#767 of 1167 on the board · best 0.0%

Word error rate

The caller's words the agent's transcriber missed, changed or invented, against what the caller actually said.

20 calls
2.18%avg

p50

1.23%

p95

12.3%

#323 of 1167 on the board · best 0.3%

Text-to-Speech

Readback accuracy

Share of the codes, amounts, dates and phone numbers the agent wrote that its voice read out correctly. One wrong digit is a miss.

17 calls
95.3%avg

p50

95.4%

p95

100%

#76 of 974 on the board · best 100%

Word error rate

The share of the words the agent wrote that its voice dropped or changed, read back from Roark’s transcription of the recording; the same transcriber hears every voice.

17 calls
1.51%avg

p50

1.44%

p95

3.32%

#16 of 1043 on the board · best 0.8%

Voice

Human-likeness

How human the agent's voice sounds, scored by an audio model on intonation, rhythm and expressiveness. 5 = hard to tell from a person.

20 calls
3.2/ 5avg

med

3.2

#836 of 1169 on the board · best 4.3 / 5

Voice consistency

Share of calls where every line the agent spoke was the same speaker, with no change of speaker, gender or language voice partway. It checks who is speaking, not how they sound.

20 calls

Not scored by this sweep.

#1 of 1165 on the board · best 100%

Latency

Response time

The silence between the caller finishing and the agent starting to speak, measured from the recording on every turn of the call.

152 turns
3.86savg

p50

3.61s

p95

5.97s

#707 of 1169 on the board · best 1.22 s

Time to first word

How long the caller waits before the agent first speaks after pickup.

20 calls
0.73savg

p50

0.65s

p95

1.32s

#69 of 1184 on the board · best 0.45 s

Latency spikes

Turns where the response took far longer than the rest of the conversation.

20 calls
7.05per callavg

p50

7.00 per call

p95

10.1 per call

#998 of 1184 on the board · best 0.0 per call

By industry

IndustryTask completionLatency P50Barge-in handling
Healthcare87%3.45 s—
Home services95%3.59 s—
Insurance85%3.78 s—
Customer support84%3.72 s—

By caller

CallerTask completionLatency P50Barge-in handling
US English, quiet line91%3.78 s—
Spanish-accented English, city street83%3.73 s—
Indian English, busy café92%3.54 s—
Southern US, flustered, crying baby91%4.01 s—
Fast and interrupting, construction outside81%3.39 s100%

The calls behind the numbers

Customer support

AGENTBrightwave Internet

CUSTOMERHey, uh, yeah, hi. This is Wade Carter. I'm calling because my internet's been down for about two hours now. I work from home, and I've got a baby crying here, so I'm a little flustered. Sorry. There's a road crew digging up the street outside, and I'm wondering if that's the reason. I've already rebooted the gateway twice and checked every cable. The internet light is blinking orange. My account number is BW883017—oh, sorry, last digit is one. Could y'all check if the road work is what's causing this? I really need to get back online.

AGENTNo problem, Wade. Let me pull up your account and check for an outage right now. I'll use the corrected account number ending in 1.

AGENTAccount found. Checking for an outage at your address now.

AGENTI didn't see an outage reported for your area, so I'm going to run a line test next to see what's happening between our network and your gateway.

CUSTOMERAll right, could you please check if the road crew might have messed with anything? I'm just a bit worried since the digging's right outside. And just to be clear, I already restarted the gateway twice and checked all the cables, so I'm hoping the line test will tell us more.

AGENTGood news, Wade: the remote reset worked, and your gateway now shows as registered and online. Give it about two minutes for your devices to reconnect. The orange light should change once it's fully up. On the road crew, there's no outage or maintenance reported for your area. The line test also showed a good signal to your gateway, so nothing looks damaged between our network and your home. The problem was the gateway not registering on our network, and the reset fixed that. I can't say for certain what caused it, but I'm not seeing any sign the digging did. Your account number is BW883011, the corrected one ending in 1. Check whether you're back online in a couple of minutes. If it drops again or the light stays orange, let me know and I'll set up a technician

AGENTvisit.

CUSTOMEROkay, thanks for checking all that. The light just went solid white now. So, how long until my devices reconnect? I'm just anxious to get everything working again. And since I lost two hours of work time, could you help me out with a credit for the downtime?

AGENTSolid white means you're connected. Devices usually reconnect within about two minutes of the reset. If a laptop or phone doesn't pick it up, toggle its Wi-Fi off and on. On the credit, that's reasonable for two hours of downtime, so I'll apply it now.

AGENTI've applied

CUSTOMERAll right.

CUSTOMERSorry, did you get cut off there? Were you about to say something about the credit? I just want to make sure nothing gets missed. Could you confirm if it's been applied and how much it is? I really did lose two hours of work, so if there's any way to make it a bit more, I'd appreciate it.

AGENTSorry about that, my message got cut off. The credit went through: $10.00 on account BW883011, and it will show up on your next bill. It's for the roughly two hours of downtime today. I can't raise that amount myself. The credit tool applies a set amount, and I can't adjust it. If you want to ask for more because of the lost work time, you can request a review through billing, but I don't want to promise you an outcome I can't back up. Your gateway is online and the light is solid white. If anything drops again, call back and I'll get a technician scheduled. Is everything reconnecting okay on your

AGENTend?

Healthcare

AGENTRiverbend Health Clinic,

CUSTOMERHey there, this is Wade Carter. I was just calling to see if I could move my dermatology appointment next Wednesday morning. I've had a work shift come up and I'm not going to be able to make it then. I can only do after 3 p.m. Sorry, my baby's fussing a little bit. Could y'all help me reschedule?

AGENTHappy to help, Wade, and no worries about the little one. First I need to verify you. Could you give me your full name as it's on your chart and your date of birth?

CUSTOMERYeah, um, my full name is Wade Carter, and my date of birth is March 17, 1948. Oh, sorry, no, 1984. March 17, 1984. Is that okay?

AGENTThanks, Wade. I'll use the corrected year, 1984.

CUSTOMERYeah, that's right. Sorry for the mix-up. Is there anything else you need from me to pull up my appointment?

AGENTHere's what Dr. Okafor has open after 3 PM: - Tuesday at 3:45 PM

- Thursday at 410 PM

There's nothing after 3 PM on Wednesday itself. Would either of those work for you?

CUSTOMERSorry, Tuesdays don't work after all. I gotta pick up my daughter then. Could I maybe get the Thursday at 4:10 p.m.? Is that still open?

AGENTYes, Thursday at 4:10 PM is still open. I'll move your appointment now.

CUSTOMEROkay, thank you. Is there any kind of fee for changing the appointment? Sorry, just want to make sure.

AGENTLet me check the change policy for you.

AGENTGood

CUSTOMERAll right, thanks. I appreciate it.

AGENTYou're welcome, Wade. Here's the policy: changes made more than 24 hours before the appointment are free, and later changes carry a $40 fee. Since you changed this one well ahead of time, you shouldn't be charged anything. Your appointment is now confirmed: - Wade Carter, date of birth March 17, 1984 - Dermatology with Dr. Okafor - Thursday at 4:10 PM, Main Campus, Building B

- Confirmation codeRVB-3M8P2

Please arrive 15 minutes early with your photo ID and insurance card. Parking is free in Lot C, with the entrance on River Road.

Is there anything else I can help with today?

CUSTOMEROkay, that's a relief about the fee. Thank you. Where exactly is Lot C? Is it easy to find from River Road? Sorry, just want to make sure I don't get lost.

AGENTAll I have is that Lot C is the free patient parking and its entrance is on River Road. I don't have more detail than that, like landmarks or how far it is from Building B, and I don't want to guess and send you the wrong way. When you arrive, the front desk can point you to the right building. If you'd like to be sure beforehand, you can call the clinic and ask for directions from River Road. Is there anything else I can help you with?

CUSTOMERAll right, I'll do that. Thanks for letting me know. One more thing, does Dr. Okafor take Aetna insurance? I just want to double check before I come in.

AGENTI can't confirm that, Wade. I don't have insurance or provider network information. The best way to check is to call the clinic's front desk or call the member services number on the back of your Aetna card and ask whether Dr. Okafor at Riverbend Health Clinic is in network. That way you'll know before your visit on Thursday. Anything else I can help with today?

CUSTOMERNo, I think that covers it. So, just to double check, my appointment's Thursday at 4:10 p.m., confirmation code RVB 3M8P2, right? Sorry, I just want to make sure I've got it all straight.

AGENTYes, that's right, Wade. Your dermatology appointment with Dr. Okafor is Thursday at 4:10 PM at Main Campus, Building B. Your confirmation code is RVB-3M8P2, spelled R-V-B, 3, M, 8, P, 2. Remember to arrive 15 minutes early with your photo ID and insurance card. Take care, and good luck with the little one.

Home services

AGENTCopperline Home

CUSTOMERSorry, could you say that again? I'm not sure what you mean by copper. Is this about the pipe under my sink?

AGENTSorry about that. This is Alex with Copperline Home Services, a plumbing, heating, and electrical company. Copperline is just our company name. If you have a problem with the pipe under your sink, I can help. First, is water leaking right now? If it is, and it's more than a slow drip, turn the shutoff valve under the sink clockwise, or shut off the main water valve for the house. Also, please keep clear of any outlets or electrical items near the water. Can you tell me what you're seeing? Is it dripping, spraying, or flooding? And is anyone in danger or hurt?

CUSTOMEROh, sorry, I'm just a bit frazzled. Yeah, there's water spraying out from the pipe under my kitchen sink, and it's spreading across the floor, getting close to a power strip. Nobody's hurt, but I've got a baby here who's crying, and I'm worried it's going to reach the outlet. I haven't shut the water off yet. Should I just turn that valve under the sink?

AGENTYes, turn that valve now. Here's what to do: 1. Shut off the water. Turn the valve under the sink clockwise as far as it will go. If it's stuck or the spray doesn't stop, go to the main water shutoff for the house, usually near the water heater, by the meter, or where the main line enters the house, and turn it clockwise. 2. Don't touch the power strip or anything plugged into it while the floor is wet. If you can reach the breaker panel without walking through the water, switch off the kitchen circuit. 3. Keep the baby out of the kitchen, away from the wet floor and the outlet. Tell me when the water is off, or if it's still spraying. While you do that, I'll pull up your account.

Can you give me the phone number on the account

CUSTOMERAll right, I'm turning the valve now. Okay, the water stopped. Thank goodness. Sorry, I'm just nervous. Is that alright? I didn't touch anything electric. My phone number is 503-555-0139. No, sorry, 0193. Is that what you need?

AGENTYou did exactly the right thing. I'll look up your account with the corrected number, 503-555-0193.

AGENTI meant to look

CUSTOMEROkay.

CUSTOMERSorry.

AGENTNo need to apologize, you're doing fine. I found your account. Before we go on, can you confirm the service address on file is 1480 Larkspur Avenue, Unit 4B? And is there a gate code or any access instructions the technician will need to get in? Since the water is off, the leak is stopped, but the pipe still needs a repair. The earliest emergency visit is today between 2:00 PM and 4:00 PM. Because you're a Copperline Care member, the emergency callout fee for you is $99. Does that window and fee work for you? Please also leave the kitchen power strip alone until the technician has looked at the area, and keep the baby away from the wet floor.

CUSTOMERActually, sorry, I just want to make sure it's unit 4D now, not 4B. We moved last month. Is that going to be a problem? And yeah, the gate code is pound two seven one nine. About the visit, I'm a little anxious. Is there any way someone could come before two? It's just making me nervous with the water and the power strip. If not, I'll take the earliest you have. And ninety-nine dollars, that's the full fee for the emergency callout, right? I just want to make sure there aren't any other charges I should know about.

AGENTUnit 4D is no problem. I'll use 4D, and 4B won't be on the ticket. The 2:00 to 4:00 PM window is the earliest emergency slot the system shows, so I can't offer anything sooner. As for the fee, $99 is the emergency callout fee for you as a Copperline Care member. That's all the system lists for the callout, and I don't have any other charges to quote. I don't have repair or parts pricing, so the technician will go over that with you on site. Since the water is off and the power strip is out of use, you're in a safe spot until then. I'll open the ticket now.

AGENTYour ticket

CUSTOMERSorry, could you finish what you were saying? I just want to make sure I get all the details right. You about to give me the ticket number?

AGENTYou're booked. Here are the details: - Ticket number: CPL-58210 - Visit: Today, 2:00 PM to 4:00 PM, with a technician named Dana - Service address: 1480 Larkspur Avenue, Unit 4D - Gate code: pound 2719 - Callback number: 503-555-0193

- Fee$99 emergency callout, at the Copperline Care member rate

Dana will text you at 503-555-0193 about 30 minutes before arriving.

Until then, keep the sink valve closed and stay off the wet floor. Leave the power strip alone, and keep the baby out of the kitchen. If the water starts again, shut off the main valve for the house and call us back. If you see sparks, smell burning, or feel unsafe, leave the area and call 911.

Is the address, unit, gate code, and phone number all correct?

CUSTOMERYeah, I think so. Please tell me that's the right one. Unit 4D, gate code pound 2719, and the phone number 503-555-0193. Is that all set on your end, just to be sure? And the ticket number was CPL 58210, right? Sorry, I just want to make sure nothing gets mixed up.

AGENTYes, everything is set and correct on my end: - Ticket number: CPL-58210 - Address: 1480 Larkspur Avenue, Unit 4D

- Gate codepound 2719

- Callback number503-555-0193

- VisitToday, 2:00 PM to 4:00 PM, with Dana

- Fee$99 emergency callout

You didn't mix anything up, and you handled this well. Keep the sink valve closed, stay clear of the wet floor and power strip, and Dana will text you about 30 minutes before arriving. Is there anything else I can help with?

Insurance

AGENTNorthgate Mutual claims, this is Alex. Are you somewhere safe right now?

CUSTOMERYes, I am safe. I am standing next to my car on the sidewalk. Nobody got hurt.

AGENTI'm glad you're safe and that no one was hurt. To pull up your policy, could you give me your policy number and the last name of the policyholder?

CUSTOMERSure, my policy number is N G four four seven one nine zero three, no sorry, last one is two, and the last name is Ramirez.

AGENTThank you. I'll use the corrected number, NG44719002. Let me verify that now.

CUSTOMERYes, that's right, for our heads.

AGENTThanks, Mr. or Ms. Ramirez. I've verified policy NG44719002 on your 2021 Honda CR-V. To make sure I have it right, that's N, G, 4, 4, 7, 1, 9, 0, 0, 2. Now I need some details about the accident: 1. When did it happen (date and approximate time)? 2. Where did it happen (street or intersection, city)? 3. What happened, in your own words?

CUSTOMEROkay, so the accident happened about 20 minutes ago. I was stopped at a red light at the intersection of Southeast Fifth Avenue and Alder Street here in Portland. Another car hit the back of my car. The back bumper is crushed and the rear wheel is rubbing. I cannot drive it. Police did not come.

AGENTThank you, that's clear. I'm sorry this happened, especially while you were stopped at a light. Since the rear wheel is rubbing and the car isn't safe to drive, I can set up a tow for you. I'll also need the other driver's information. Could you give me:

1. Their full name

2. Their insurance company

3. Their license plate number

Also, what's the best phone number for the tow driver to reach

AGENTyou at?

CUSTOMERSure, the other driver's name is Kevin Lund. He is insured with Pacific Shore. The license plate is seven K X like X-ray R two one four, and the best phone number for the tow driver is one nine zero seven two one five three zero three.

AGENTLet me read back what I heard so we get it exactly right. The plate is 7KXR214 (7, K, X as in X-ray, R, 2, 1, 4). The phone number I heard is 1-9-0-7-2-1-5-3-0-3, which is only 10 digits and starts with a 1, so I may have missed something. Could you repeat the phone number, including the area code? I want the tow driver to be able to reach you. In the meantime, I'll look up your coverage so I can tell you what applies.

AGENTHere is what

CUSTOMERअहाँ।

CUSTOMEROkay, I'm listening.

AGENTHere's what your policy shows: - Collision deductible: $500 - Rental: $40 per day for up to 30 days, starting from the day the claim is opened - Towing: Covered up to 25 miles So the tow is covered as long as it's within 25 miles. I still need to confirm your phone number. I heard 1-9-0-7-2-1-5-3-0-3, which is one digit short of a full number with an area code. Could you say it again, slowly, with the area code?

Also, the tow location is the corner of Southeast Fifth Avenue and Alder Street in Portland, where you're standing with your car. Is that right?

CUSTOMERYes that's right about the location Southeast Fifth Avenue and Alder Street in Portland and my phone number is one nine zero seven two one five three zero three one that's the full number with the area code.

AGENTThank you. The full number is 1-9-0-7-2-1-5-3-0-3-1, which is 11 digits. I'll use it as you gave it. I'll arrange the tow and open your claim now.

AGENTThe claim

CUSTOMEROkay. Right, go ahead.

AGENTYour claim is open and the tow is on its way. Claim: Your claim number is NGM-904417 (N, G, M, 9, 0, 4, 4, 1, 7). Your adjuster is Priya Shah, who will contact you within one business day. Tow: Eastside Towing has been dispatched to Southeast Fifth Avenue and Alder Street. They should arrive in about 45 minutes. They'll take your CR-V to our partner repair shop at 9th and Morrison. They have your callback number as 1-9-0-7-2-1-5-3-0-3-1. Details on file: - Policy: NG44719002 - Other driver: Kevin Lund, insured with Pacific Shore, plate 7KXR214 - Police: did not attend, so there is no report number - Injuries: none Coverage reminders: Your collision deductible is $500. Rental is $40 per day for up to 30 days, counting from today, when the claim was opened. Towing is covered up to 25 miles.

Please stay on the sidewalk, away from traffic, while you wait. If you can, take photos of the damage and of Mr. Lund's car and plate.

Is there anything I got wrong, or anything else

AGENTI can help with?

CUSTOMERJust one thing, about the deductible, since he hit me, I should not have to pay that, right? And will my premium go up because of this? Also, how do I get the rental car?

AGENTI can only speak to what's on your policy, and I don't have anything that answers the first two questions. Deductible: Your policy shows a $500 collision deductible. I can't tell you whether you'll end up paying it. That depends on how fault is decided and whether the other driver's insurer, Pacific Shore, covers the damage. Priya Shah, your adjuster, can explain how that works, and she'll contact you within one business day. Premium: I don't have any information about whether your premium could change, and I don't want to guess. Priya or your agent can answer that. Rental: Your policy includes rental reimbursement of $40 per day for up to 30 days, counting from today, when the claim was opened. I don't have a step-by-step process for getting the rental, and I can't book one from here. Ask Priya when she calls. Until then, keep your rental receipts, since the coverage is reimbursement. Is there anything else I can help with while you wait

AGENTfor the tow?

CUSTOMERNo, that's all. Thank you for your help. Goodbye.

AGENTYou're welcome, Mr. or Ms. Ramirez. Your claim number is NGM-904417, and Priya Shah will be in touch within one business day. Eastside Towing should be there in about 45 minutes. Take care, and goodbye.

How these numbers are measured →