The original AIVA voice pipeline was a masterpiece of short-term thinking. Four months of moving fast had produced something that worked in demos, held together in production, and was completely unmaintainable.
It was four separate vendor APIs duct-taped together with 3,000 lines of orchestration code that nobody fully understood — least of all the person who'd written most of it, which was me.
This is the story of the rewrite:
- what broke
- what we built
- the one lesson I wish we'd learned earlier
It's the engineering half of the wider AIVA 2.0 rebuild.
What was wrong
The architecture was called "the chain" internally. A voice call arrived at Twilio, got transcribed by API-A, the transcript went to API-B for language understanding, the response went to API-C for synthesis, and the audio routed back through Twilio. Five network hops between a human speaking and AIVA responding.
Each hop had its own error surface. If API-B returned a 503 at 2am, the call failed silently — the customer heard hold music until the Twilio timeout fired. We had no visibility into which hop had failed.
Our on-call routine was to check four vendor dashboards in sequence, which took long enough that the customer had usually hung up by the time we found the problem.
The latency was also stuck. We'd optimised everything we could reach — our own code, our caching layers, our database queries — and p95 was still 380ms. The ceiling was the five-hop architecture itself. You cannot make five network round trips inside 200ms, regardless of how fast each one is.
The arithmetic that settled the argument
Rewrites are usually the wrong answer, and we argued about this for weeks before anyone produced a number worth acting on.
The number was a best-case simulation. We took each vendor's documented p50, assumed every network hop ran at the theoretical minimum for the physical distance involved, and assumed our own orchestration cost zero. That fantasy pipeline landed at roughly 300ms. Our requirement was 200ms.
That's what ended the debate. "The code is ugly" and "we'd design it differently now" are always true and never sufficient — they justify refactoring, not replacement. The case for a rewrite is when the best possible version of what you have still misses the requirement. Ours did, by 100ms, and no amount of profiling was going to find 100ms inside network physics.
How we approached the rewrite
We ran the rewrite in parallel with the production system for six weeks before cutting over.
This sounds obvious. It took us three weeks to decide to do it, because the parallel approach felt slower — we wanted to just build the new thing and switch. Those six weeks found eleven bugs that would have caused production incidents on day one.
Almost all of them were cases where real caller behaviour diverged from our test fixtures: people who start speaking before the greeting finishes, people who pause mid-sentence in ways our endpointing read as "done", people switching language halfway through a request. No unit test suite was going to generate those.
The new architecture is a unified inference service. Speech recognition, language understanding, response generation, and synthesis all run in a single process, on hardware we control, in the same region as the Twilio media server.
One network hop — from Twilio to us and back. The specifics of that integration are in our Twilio write-up.
The latency improvement was immediate and dramatic. On the first day the new pipeline was live, p95 dropped from 380ms to 220ms. Subsequent tuning — described millisecond by millisecond in how we got from 240ms to 198ms — took the average to 198ms. The theoretical minimum on our current hardware is around 180ms.
The most important thing we got right: we didn't rewrite the API contract. External callers — the webhook system, the dashboard, customer integrations — saw no change. The rewrite was internal infrastructure. Nobody outside the engineering team needed to do anything when it shipped, which is why we could cut over on a Tuesday morning instead of coordinating a migration.
What owning the layer unlocked
The latency number is the headline, but it's not the most valuable outcome. Three things became possible that simply weren't before.
Regional deployment. With one process instead of four vendor endpoints, we could deploy the whole pipeline close to the caller.
That's what made Mumbai, Frankfurt, and Virginia practical — the reasoning behind those three is a separate post, but none of it was available while our latency was hostage to wherever four vendors happened to host.
Languages as configuration. In the old chain, adding a language meant negotiating support across three vendors who each covered a different subset.
Owning the models turned it into a config file and a checkpoint — which is the only reason twelve Indian languages was a realistic goal rather than a roadmap item.
Real observability. One process means one trace. Debugging went from checking four dashboards to reading a single timeline, and it's the same instrumentation that now surfaces call quality in the analytics dashboard.
The failure modes we traded for
Owning the layer isn't free, and the honest version of this post has to include what got worse.
We now own uptime. When API-B fell over, it was someone else's 3am. Now every outage is ours, which means real on-call rotation, real runbooks, and real capacity planning.
The compensation is that debugging went from "which of four vendors is lying" to reading one trace — but the total volume of operational work went up, not down.
We now own capacity. Vendor APIs absorb traffic spikes as somebody else's problem. Running our own inference means provisioning for peak, and peak in Indian voice traffic is sharply seasonal:
- festival weeks
- exam results season
- the last three days before a tax deadline
We got that wrong twice in the first quarter before we learned to read our own calendar.
We now own model quality. Previously, when a vendor improved their model, we improved for free. Now improvement is a project we schedule.
That's better in the long run, because we can tune for the specific thing we care about instead of the vendor's median customer — but the free lunch genuinely ended.
None of these outweighed the latency win. All three were invisible to us before the cutover, and they'd be the first things I'd budget for if I were doing it again.
The one lesson
We spent the first year building on vendor APIs because it felt safer. Vendors handle reliability, uptime, billing, and scaling — you don't have to. This is true, and for early-stage companies it's the right call. I'd make it again.
But vendor APIs have a ceiling. When your requirements exceed what the vendor optimised for — and ours did, around month 10 — you hit it. At that point you have two choices: stay on the ceiling and compete with one hand tied behind your back, or do the work of owning the layer.
We stayed on the ceiling for six months longer than we should have, because the rewrite felt scary. Those six months cost us more in competitive disadvantage than the rewrite cost us in engineering time.
The lesson: vendor abstraction is debt, not safety. It defers cost, it doesn't eliminate it. Understand what you're deferring, and roughly when you'll have to pay it.
You can see what the current pipeline does on live traffic on the voice platform page, or start free with ₹500 of credit and put a call through it yourself.