240ms was already good. Competing voice agents were averaging 340–400ms. But in voice, the difference between 240ms and 200ms is the difference between a conversation that flows and one that feels like a slow phone connection. We wanted 200ms. Here's how we got there — and the one finding that surprised us.
This post is the sequel to the voice pipeline rewrite, which collapsed our architecture from five network hops to one. That rewrite got us to 240ms. Everything below is what came after, once the easy architectural win was already banked.
Why 40ms is worth chasing
It's fair to ask whether 40ms is worth a month of engineering. Nobody consciously perceives 40 milliseconds.
The answer is that conversational latency isn't perceived as a duration — it's perceived as a personality. Human turn-taking in natural conversation clusters around a 200ms gap. Land inside that window and the agent reads as attentive.
Land outside it and the same agent, saying exactly the same words, reads as hesitant or slow-witted. Callers don't say "the latency was 280ms." They say "it felt robotic."
That effect compounds over a call. A single 300ms pause is invisible. Thirty of them across a four-minute booking conversation is the difference between a caller who completes the booking and one who hangs up to try a human.
We've written more about why response time shapes the whole customer experience, but the short version: latency is the most legible quality signal in voice, because it's the only one a caller can judge in the first second.
The pipeline audit
Every millisecond in voice has an address. Before optimising anything, we instrumented the entire call path to find where time was actually going:
| Stage | Before | After |
|---|---|---|
| Speech-to-text recognition | 48ms | 48ms |
| Context retrieval | 31ms | 12ms |
| LLM inference | 82ms | 51ms |
| Text-to-speech synthesis | 44ms | ~0ms cached / 44ms cold |
| Twilio WebRTC delivery | 35ms | 8ms |
| Total (average) | 240ms | 198ms |
The discipline that mattered here was measuring end-to-end, from the moment the caller stops speaking to the first audible syllable of the response. Component-level benchmarks lie by omission — they measure the work and skip the waiting between the work.
Where the wins were
LLM inference (82ms → 51ms). We rewrote the inference pipeline to stream tokens directly into the TTS engine rather than waiting for a complete response. The first syllable of AIVA's reply now starts synthesising before the last token has been generated. Because the two stages overlap instead of queueing, the saving is real time, not accounting.
Context retrieval (31ms → 12ms). We moved to a warm cache holding the most recent 500 messages per customer, keyed by session ID. Most turns within a live conversation hit the cache.
Cold calls — a first contact from an unknown number — still pay the full 31ms, but they're a minority of traffic and they only pay it once per call.
Regional TTS caching (44ms → ~0ms on common phrases). Greetings, hold messages, and clarification prompts are now pre-synthesised and served from regional edge nodes. "Namaste, how can I help you today?" is in cache before the call even connects.
This works because the opening of a call is the least varied part of it, and the part where a delay is most conspicuous.
Regional routing. Distance is latency you cannot optimise away, only avoid. Running inference in Mumbai, Frankfurt, and Virginia — described in detail here — means a caller in Pune isn't waiting on a round trip to Virginia. This didn't change the average much; it dramatically improved the tail.
The Twilio surprise
The Twilio delivery number — 35ms — looked fixed. It's network time. You can't negotiate with physics.
Or so we thought.
Digging into Twilio's WebRTC media handling, we found their default encoder uses Opus at 20ms frame intervals, but with a 40ms lookahead buffer for the codec's bitrate prediction.
Every audio packet was being held an extra 30ms before transmission — a deliberate encoder trade-off favouring call quality over latency, and a perfectly reasonable default for the median use case. It just isn't the right default for conversational AI, where the whole product is the timing.
Twilio exposes a preferredCodecs override in its voice media configuration. Switching to Opus with maxptime=10 — 10ms frames, no lookahead buffer — dropped delivery latency from 35ms to 8ms.
The trade-off is marginally lower audio quality on degraded connections. We ran blind quality ratings before and after; they were identical. The latency improvement was 27ms, the single largest win in the project, and it came from a configuration flag rather than any of our own code.
More on how that integration is wired in our Twilio integration write-up.
The general lesson: audit your vendors' defaults, not just your own code. Vendor defaults are tuned for the median customer. If your product lives at an extreme of one dimension — in our case, latency — the median default is actively working against you, and it will never show up in your own profiler.
What we deliberately didn't do
Two obvious optimisations we rejected, because they trade the wrong thing.
We didn't shrink the model. A smaller model would have cut inference further. It would also have cost us reasoning quality on exactly the calls that need it most — multi-part questions, ambiguous requests, code-switched Hinglish where the caller changes language mid-sentence.
Fast and wrong is worse than measured and right; a caller forgives a 250ms pause far more readily than a wrong appointment time.
We didn't cut the endpointing delay. Endpointing is how long the system waits after a caller stops speaking before deciding they've actually finished.
Shortening it is the cheapest possible latency win and the worst one available, because the failure mode is interrupting your customer mid-sentence.
We treat interruption handling and natural pauses as a correctness problem, not a speed problem.
Both rejections point at the same rule: optimise the waiting, never the thinking.
How we keep it from regressing
A latency win that isn't defended decays. Ours is guarded by a budget rather than a target: every stage has a millisecond allocation, and a change that pushes a stage over its allocation fails CI regardless of what else it improves. That converts latency from something anyone can quietly spend into something someone has to explicitly negotiate for.
The other half is measuring in production rather than in a benchmark. Synthetic tests run on clean networks with warm caches and short prompts, which is not what a Tuesday afternoon looks like.
We alert on p95 from live traffic, segmented by region, because a regression that only affects Frankfurt is invisible in a global average until it isn't.
Result
Before: 240ms average, 310ms p95. After: 198ms average, 260ms p95.
We hit 200ms. The calls feel different. Customers don't consciously notice latency until it's bad — but they feel it when it's good. Our CSAT across voice calls improved 0.3 points in the month after the optimisation shipped, with no other changes to the product. We'll take it.
The remaining budget is unglamorous: roughly 180ms is the floor on our current hardware, and most of what's left is speech recognition and codec framing rather than anything we can restructure.
The next real win won't come from shaving milliseconds — it'll come from the pipeline needing fewer turns to reach the same outcome.
You can see the current numbers on live traffic in the analytics dashboard, or read how the whole voice stack fits together on the voice platform page.
Want to hear where 198ms actually lands? Start free with ₹500 of credit and call your own agent.