We've written before about shaving milliseconds out of AIVA's inference pipeline — token streaming, warm caches, a Twilio codec setting that bought us 27ms nobody expected. All of that work assumed one thing: that the call and the server answering it are physically close together. If they're not, none of the pipeline optimization in the world gets you out of the one cost you can't engineer around — the time it takes a signal to travel.
The part of latency that isn't software
A round trip from a phone in India to a server in the US eastern seaboard carries real, fixed network time before a single token of inference even starts — the physical distance the signal has to travel, both ways, plus every hop in between. No amount of faster inference changes that number, because it isn't an inference cost. It's a distance cost. We could get our processing time to zero and a caller in Mumbai talking to a server in Virginia would still feel a lag that a caller talking to a server in Mumbai wouldn't.
The physics behind that is straightforward, even if it's easy to lose sight of once you're deep in optimizing code. A signal in fiber travels at roughly two-thirds the speed of light, not the full speed — and real routes aren't a straight line, they follow whatever cable paths and exchange points actually exist between two points on the map. Mumbai and the US east coast are separated by well over ten thousand kilometers as the crow flies, and considerably more once you account for actual routing. Do that math and you land on a round trip that can eat up a meaningful fraction of our entire latency budget — before accounting for any switching delay at the exchanges in between, and long before a single word gets processed. So the highest-leverage latency work we ever did wasn't in the model. It was deciding not to make every call travel further than it has to.
Where we put the servers
AIVA runs regional inference out of three locations: Mumbai for calls in India, and Frankfurt and Virginia for calls in the EU and US respectively. When a call comes in, we route it to whichever region is actually close to the caller, rather than to one central location everyone shares. A caller in Ahmedabad and a caller in Chennai both land in Mumbai. A caller in Berlin lands in Frankfurt. Splitting by region isn't about load — any one region could likely handle more volume than it currently does — it's entirely about not paying distance you don't have to pay.
Routing itself happens early and cheaply: a call's origin is known well before any language understanding needs to happen, so the decision of which region should handle it is made at the network layer, alongside the rest of the call setup Twilio already handles, not somewhere deep inside the inference pipeline where it would cost extra time to figure out. That ordering matters — a routing decision that itself took tens of milliseconds to compute would eat directly into the same latency budget regional routing exists to protect, so it has to be resolved essentially for free, before the actual conversation work begins.
This is a big part of how we hit our current average: about 198ms, under the roughly 300ms gap that shows up in normal human conversation. Regional routing didn't get us there by itself — the pipeline rewrite and the inference optimizations did most of that work — but it's the piece that keeps a caller anywhere in our supported regions from silently paying a distance tax the number doesn't show.
What happens outside the three regions
Three regions cover the geography where the overwhelming majority of AIVA's calls originate, but they don't cover the entire globe, and it's worth being honest about what happens at the edges. A customer traveling abroad who calls a business's AIVA number, or a business serving customers well outside India, the EU, or the US, still gets a working call — routing falls back to whichever of the three regions is nearest, rather than failing or refusing the call. What that caller doesn't get is the same latency profile as someone calling from inside one of the three covered regions; they're paying some of the distance cost the regional split exists to avoid. It's a real trade-off, not a hidden one, and it's a large part of why the three regions we did pick are the ones covering the geography where AIVA actually operates today.
The part that's easy to get wrong
Running three regions instead of one creates an obvious failure mode: it's easy to end up with three slightly different products. A model update ships to Virginia first because that's where someone happened to test it. A language capability lands in Mumbai and takes weeks to reach the others. Left unmanaged, "regional inference" quietly turns into "which region you're unlucky enough to be routed to determines which AIVA you actually get" — and a Hindi caller in Mumbai ending up with a worse experience than an English caller in Virginia would defeat the entire point of the languages work we've done.
We treat deployment across the three regions as a single release, not three independent ones. Whatever ships, ships to all three at the same time, verified against the same test set. The regions differ in exactly one respect — geography — and nothing else. That discipline costs real coordination overhead compared to shipping to a single region and calling it done; a release that would be a single deploy against one location becomes a release that has to be validated against three, each with its own network conditions and its own edge cases in how calls actually arrive. We think that overhead is worth paying, because the alternative — three regions quietly drifting apart — would be a much harder problem to notice and fix after the fact than it is to prevent up front.
Why this over one big fast region
The alternative — one very fast, very well-optimized central region — would still lose to this approach for any caller far enough away from it, no matter how good the optimization got. Distance is the one latency cost that scales with kilometers, not engineering hours. Once we'd pushed the pipeline itself about as far as it could go, moving inference to where the calls actually are was the only remaining lever, and it turned out to be one of the largest ones we pulled.
It's worth being honest that this isn't a decision you get to make once and forget. Adding a fourth region — say, for a growing base of Southeast Asian customers — would mean repeating this entire calculation from scratch: is the distance cost for that geography actually large enough to justify tripling our operational surface again, or is it small enough that routing those calls through an existing region is the better trade. We don't expect that answer to be obvious in advance; it's the same kind of measurement-first decision that got us to three regions rather than one in the first place. It's also the same reasoning behind why native-language models beat a translation layer bolted onto a single English model — in both cases, the fix that actually mattered wasn't a smarter algorithm, it was refusing to make every call travel further, structurally, than it needed to. You can hear the result on the voice platform page, or read how we measure the 300ms line this whole effort is aimed at staying under.