We built AIVA's voice pipeline around a hard number: get from "customer stops talking" to "AIVA starts responding" in under 200ms, wherever the call is happening. For a business in Ahmedabad calling into our Mumbai region, that's achievable — the customer, the business, and our inference hardware are all within a few hundred kilometers of each other. For a business in Austin, Texas, it's a physics problem, not an engineering one, and physics problems don't yield to better code.
The math that forced the decision
Light in fiber travels at roughly two-thirds the speed of light in a vacuum, and real submarine and terrestrial cable routes are never straight lines — a signal from the eastern United States to Mumbai doesn't take the great-circle path, it hops through landing stations and exchange points that add real distance on top of the geography. A round trip on well-routed fiber between the two costs somewhere well north of 150ms before AIVA has processed a single word of what was said. Add that on top of our own pipeline latency — the part we actually control — and a call that should feel instant starts to feel like a bad international phone connection, the kind with the awkward half-second pause every parent who's called relatives abroad recognizes immediately.
We could not optimize our way out of this. You can shave milliseconds off inference, off text-to-speech, off the audio codec — we've done all of that, it's documented in the pipeline rewrite and in the 40ms we clawed back separately, and it's a real story on its own. But you cannot shave milliseconds off the distance between Virginia and Mumbai, because that number isn't a software cost — it's a distance cost, fixed by geography and the speed of light, sitting underneath every other optimization we could make. The only fix that actually touches that number is to not send the call that far in the first place.
What we tried before committing to full regional inference
Before we stood up complete inference stacks in Frankfurt and Virginia, we prototyped a cheaper middle ground: a lightweight regional edge node that would terminate the call locally — handling the audio streaming and the earliest part of the pipeline close to the customer — while still sending the actual language understanding back to a central location in Mumbai. It was an appealing shortcut, because it meant standing up something far smaller than a full mirrored deployment in every region.
It didn't work well enough. Terminating the call locally shaved off some of the round trip, but the part of the pipeline that actually takes the most time — understanding what was said and deciding how to respond — was still making the long trip back to Mumbai and back again. We measured it honestly, and the improvement was real but partial: noticeably better than doing nothing, still nowhere near the under-200ms bar we'd set for every call regardless of where it originated. A half-measure that gets you most of the way to a hard latency budget still fails the calls that land right on the edge of it, and "usually under 200ms" wasn't the commitment we'd made. That result is what pushed us from a cheaper edge-only architecture to the more expensive decision: full inference, same pipeline, same model, actually running in the region the customer is in.
Why Frankfurt and Virginia specifically
Once "put inference closer to the customer" was the answer, the question was where. We picked regions the way most infrastructure decisions get made when you're small: where the reliable cloud capacity already existed, and where our early non-India customers actually were — not where we hoped future customers might someday show up.
Frankfurt covers our EU customers with strong connectivity across Western and Central Europe, and it keeps EU customer data on EU infrastructure by default — which matters to a specific kind of prospect before they've asked a single question about it, even though data residency wasn't the original reason we picked it. Data privacy questions come up early in almost every EU sales conversation, and having an honest, structural answer — the data already lives here, by default, because of a latency decision we made for unrelated reasons — has turned out to be a better answer than any policy document alone could be. Virginia is the default center of gravity for US cloud infrastructure, and it puts us within a comfortable latency radius of both coasts, which a single US region on its own can't quite claim; a customer in Seattle and a customer in Miami both land within our latency budget from the same Virginia deployment, even though neither is especially close to it in absolute terms.
Neither region runs a stripped-down version of AIVA. Same pipeline, same model, same languages where relevant. The only difference is geography — a customer in Berlin should never be able to tell, from latency alone, that they're not our home market.
This is a meaningful part of how we hit our current average of roughly 198ms end to end — under the roughly 300ms pause that shows up naturally in human conversation, which is the number we actually measure ourselves against rather than an arbitrary target. Regional routing didn't get us there alone — the pipeline work did most of the heavy lifting — but it's the piece that keeps a caller anywhere in our supported regions from silently paying a distance tax that number wouldn't otherwise show.
What it actually cost us
I won't pretend this was free. Running the same stack in three regions instead of one roughly triples the operational surface: three sets of infrastructure to patch, three regions to monitor, three places a deploy can go wrong instead of one. Our on-call rotation now has to think in three time zones, not one — a bad deploy at 2am India time might be the middle of a US business day, and someone needs to be awake for both possibilities. We built our regional failover and deployment tooling before we strictly needed it in India, specifically so that adding Frankfurt and Virginia later would be a configuration change, not a rewrite. That decision paid for itself the week we flipped Frankfurt on — what could have been a multi-month infrastructure project became a matter of pointing existing tooling at a new set of servers.
The cost that's easier to miss is the ongoing one: keeping three regions behaviorally identical over time, not just on launch day. Every model update, every pipeline tweak, every new language has to roll out to all three in a way that doesn't leave one region running a subtly different version of AIVA than the other two, even briefly. That discipline is invisible when it works and immediately obvious to a customer when it doesn't, which is exactly the kind of cost that's easy to underinvest in until it bites you once.
What we'd tell another small team
Don't build multi-region infrastructure because it looks impressive on a slide. Build it because a customer's physics problem became your product problem. We didn't stand up Frankfurt until we had EU customers who needed it, and we didn't stand up Virginia until the same was true for the US — the same discipline behind serving customers across 14 countries without overbuilding ahead of where the demand actually was. Regional inference is expensive enough, in money and in operational complexity, that "just in case" isn't a good enough reason to run it, and we tried the cheaper half-measure first specifically to make sure full regional deployment was actually necessary before we committed to it.
It's a good enough reason once someone eight time zones away is waiting on your response, and every one of those time zones is costing them milliseconds you can't optimize back. If you want to see what that latency actually feels like on a real call, try AIVA's voice agent — or read how the call actually gets routed once it lands, region by region, in our companion piece on regional routing.