When to Build vs Buy Voice AI: What a 3/10 Client Review Taught Us

All names, numbers, and identifiers in this post are anonymized. The patterns are real.

The review that forced the question

We spent months building what we thought was the safe version of a custom voice agent for a public-sector office. Every caller turn passed through a nine label classifier, then a fifteen rule routing table. Cheap intents short-circuited into one of about twenty canned responses without touching the main LLM. Calls about one specific request type were handed to a dedicated sub-agent with its own prompt, its own state machine, and its own LLM call. The whole thing was heavily tested, deterministic, and auditable.

The client rated it 3 out of 10.

The word that kept coming back was robotic. That review is why this post exists, because the failure was not a bug. It was a wrong answer to the build vs buy question, given by a team that builds custom voice agents for a living. If you are deciding between a platform like Retell or Vapi and a custom build, the useful lesson is not which one we picked. It is what we learned about where each one actually earns its keep.

The architecture in 30 seconds

The agent answers a public office’s phone line: what a notice means, how to pay, how to dispute. It runs on LiveKit, retrieves answers from a Postgres vector store, and speaks through a neural TTS. The layered version worked like this on every turn:

caller -> STT -> classifier (9 labels)
              -> routing table (15 rules)
                   -> canned response (about 20 of them)  -> TTS
                   -> main LLM -> tools                   -> TTS
                        -> dedicated sub-agent (one request type, own prompt, own state machine)

The design intent was textbook. The classifier kept abuse and off-topic turns away from the main model. The routing table made common answers instant and word-for-word repeatable. The sub-agent existed to keep a fiddly multi-step flow out of the main prompt. The deleted router’s own docstring shows how deliberate this all was:

"""Declarative turn router: generic machinery only.

The fast path ladder is an ordered list of Rules. route() walks the
table: the first rule whose matches() returns True runs its handler;
the handler either fully handles the turn or falls through to the
next rule.
"""

Fifteen rules, each with its own match predicate, handler, and dedicated test coverage. Elegantly designed. The elegance was the problem.

Why callers could hear the layers

Three costs stacked on every call.

Cumulative latency. The classifier was fast in isolation, about 100 ms on a small model. In production, sitting in series before everything else, the gate added 1.5 to 3 seconds per turn before the agent even began to respond. On turns inside the sub-agent’s flow, a second LLM call stacked on top. Phone silence is expensive. We had already built thinking fillers to paper over it, a fix we wrote about in masking a voice agent’s thinking silence. We were treating the symptom of a latency budget the architecture itself had spent.

Voice discontinuity. Every canned response was spoken by the same TTS voice but written by a human, so it had a different register from the LLM turns around it. The sub-agent was literally a different model call with a different prompt, and callers could feel the persona shift mid-call. People cannot see your architecture diagram, but they can hear it.

Prompt drift in a smaller room. The sub-agent’s docstring made an honest promise:

"""Request-flow sub-agent: deterministic code-driven flow.

Why: prompt rules drift under conversation history bias. This module
owns one request type's flow end to end so behavior is testable and
deterministic.
"""

What actually happened is that the sub-agent’s own prompt drifted under the same pressure, just in a smaller room. Callers changed their minds mid-flow. They gave dates in formats the state machine did not expect. They asked general questions from inside the sub-agent’s flow. Every real-call edge case became a patch, and every patch grew the sub-agent’s prompt. Our own engineering note from that period says it plainly: the sub-agent prompt grew while patching real-call edge cases, which defeats the module’s own design intent. We were paying the tax of determinism without collecting the benefit.

The honest root cause

We optimized for auditability. The client cares about naturalness. A public office does need factual accuracy about the law and the payment process. It does not need a state machine that guarantees a date gets collected in a fixed order. A good prompt on a good model gets that right on 99 percent of calls. The state machine got it right on 100 percent of calls and sounded like a robot on all of them. The client’s 3 out of 10 was the correct score for that trade.

What we deleted

Midway through 2026 we made the call: one LLM per turn, one prompt file, no pre-classification, no canned short-circuits, no sub-agent. The working tree tells the story. This is an abridged excerpt of the deletion list, about 2,000 lines of Python plus a similar volume of test fixtures:

D  agent/agent/request_flow.py
D  agent/agent/request_subagent.py
D  agent/agent/routing.py
D  agent/tests/fixtures/classifier_baseline.json
D  agent/tests/fixtures/classifier_cases.jsonl
D  agent/tests/test_classifier_labels.py
D  agent/tests/test_fillers.py
D  agent/tests/test_guardrails.py
D  agent/tests/test_routing_replay.py
D  agent/tests/test_routing_rules.py
D  agent/tests/test_tool_compliance.py

The main agent class went from 2,857 lines to a fraction of that in a single commit. The previous architecture stays on its own branch, deployable as the rollback path, because 2,000 lines of tested code is not something you rebuild from memory if the bet goes wrong.

What we kept, and where the audit went

Deleting the layers did not mean deleting the guarantees. It meant moving them to the boundary, off the caller’s clock:

• A post-call eval, LLM as judge. After hangup, an eval reads the full transcript plus the knowledge base results the agent actually saw, and scores every assistant turn for faithfulness and citation coverage. Unfaithful turns get flagged into the payload the client’s dashboard receives. This is where the audit lives now, not in a pre-turn gate the caller has to wait for.
• A deterministic output filter. Every assistant message is scrubbed on the way to TTS. Sensitive identifiers are redacted by code, not by trust in the model.
• Hard rules stay in code. The business-hours gate inside the transfer tool. The daily rate limiter in Postgres. Failures we cannot afford even once do not belong in a prompt.
• The knowledge base, untouched. Same vector store, same similarity threshold, same top-K. A rewrite should change one thing at a time.

One deletion surprised us: the sanitizer that stripped email addresses out of knowledge base results before the model saw them. In the old world, canned responses handled email delivery, so the model never needed the address. In the new world, the speech layer needs the raw address so it can convert it to letter-by-letter SSML at synthesis time. A protective layer in one architecture was an active obstacle in the other. Redaction of true PII stays.

The migration ledger that made it safe

More than 60 business rules lived somewhere in the old architecture: in the prompt, in the classifier labels, in routing rules, in canned text, in the sub-agent. Before writing the new prompt we built a business-rules inventory, one row per rule, each mapped to exactly one destination:

PROMPT  the rule moves into the prompt file
CODE    the rule stays in code (rate limit, business hours)
TOOL    the rule is enforced inside a function tool
DROP    the rule was not worth keeping

The new prompt file is about 150 lines and owns every PROMPT row. The inventory outlives the migration. It is the paper trail the client can read, and the checklist each row gets verified against on real calls.

The verification bar moved too. The old test suite died with the old architecture, so the new one had to be harder where it counts: 158 offline tests plus 9 live golden scenarios that replay full scripted calls against the real model. The goldens cover the exact behaviors from the 3/10 review: a statutory deadline gate, the partial-payment rule, date routing in the sub-agent’s old flow, pushback followed by transfer, refusing legal advice, and a clarifier between two legal processes callers mix up. A full live pass costs about $0.05, cheap enough to run on every prompt change that touches a business rule.

When to buy a platform instead

This experience sharpened our answer to the question clients actually ask: Retell or Vapi or custom, which one?

Buy a platform when the conversation is the product and your requirements are standard. Booking, lead capture, call routing, FAQ answering. The platform’s opinionated pipeline is tuned for exactly these, you get barge-in and turn detection and telephony for free, and you can be live in days. We build starter agents on Retell ourselves and hand them to clients who own them outright. A platform is the right answer more often than a services firm likes to admit.

Build custom when a requirement cannot live inside a platform. In our practice that list is short and specific: data residency, when audio and transcripts must stay in a particular country or tenant. Audit evidence, when procurement wants per-turn faithfulness scoring, not a marketing page. Statutory rules, when a wrong answer about a legal deadline has real consequences. Deep integration, when the agent must sit inside systems no platform connector reaches.

And if you do build custom, do not rebuild a platform’s rigidity by hand. That is the trap in the middle, and it is exactly where our 3/10 came from. The point of custom is owning the requirements. The moment you wrap the model in classifiers and state machines for comfort, you have bought control and spent it on latency.

The rule we use now

Every guardrail must catch a real failure often enough to justify what it costs the caller. Failures with unaffordable consequences go in code at the boundary: payments, business hours, rate limits, PII redaction. Conversational behavior goes in the prompt, where the model can apply judgment in context. And any guardrail that needs weekly patching to keep up with real calls has already failed at its own job. Move it back into the prompt and let the eval watch it.

What this means if you are an IT services firm

If a client scores your carefully layered LLM system at five or below, the system is probably not unsafe. It is probably audible. Ask of every layer: which real failure does this catch, how often, and what does it cost the user experience while it waits to catch it? Keep the boundary guarantees in code, put the behavior in the prompt, and hold it all to a live eval that runs off the caller’s clock. That discipline, not the layer count, is what a serious buyer is paying for. If you are an IT services firm weighing a platform build against a custom one for your own client, this decision framework is the conversation we have every week, and we are happy to have it with you at taritas partners.