Voice Agent Responsiveness: The Latency Callers Feel, and a Fix That Broke Silently

All names, numbers, and identifiers in this post are anonymized. The patterns are real.

The problem: a caller does not experience the model’s speed

A voice agent for a public-sector office felt slow at launch. Every caller turn had roughly a second of dead air between the caller finishing a sentence and the agent starting to reply. A second of silence on a phone call is not a small thing. It is the difference between a caller believing they are talking to a person and a caller wondering if the line dropped.

The instinct is to blame the model. It is almost never the model.

The latency chain a caller actually feels

Responsiveness on a phone call is the sum of a chain of small waits, most of which happen before the language model does anything:

1. Speech-to-text segmentation silence. The transcription engine’s own floor for deciding one spoken segment has ended. On the engine we use, 300 milliseconds is the minimum; anything lower is ignored.
2. Voice-activity-detection silence. How long the audio-level detector waits after speech stops before declaring the caller done. The shipped default was 1.5 seconds. In practice that meant about 1.2 seconds of dead air every single turn before the agent even began to think.
3. Semantic turn-detector patience. A small language model that reads the transcript so far and judges, by meaning, whether the sentence sounds finished. This one is dynamic: it holds back on an incomplete-sounding sentence and lets a complete one through quickly.
4. Language model time-to-first-token. On a small, cost-efficient model, roughly 400 milliseconds typical.
5. Text-to-speech time-to-first-byte. Roughly 200 milliseconds typical after the engine has warmed up.

The caller’s dead air is the sum of the first three stages, the wait before any model work even starts. Vendor latency numbers almost always describe stage four alone. That is not what a caller feels.

We had already added the semantic turn detector in an earlier fix, after discovering that a fixed silence timer alone forces a bad trade: short enough to feel snappy and it fragments sentences and cuts off callers who pause to think, long enough to be safe and it feels slow. We wrote about that fix and the bug it exposed in how a voice agent knows you’re done. With that semantic layer already watching the words, the raw voice-activity silence became a much safer knob to shorten, because the semantic layer is the one actually deciding whether a pause means “thinking” or “finished.”

Tuning the safe knobs, by ear

There is no single correct number for voice-activity silence. The right value depends on the voice, the caller demographic, the domain, and the ambient noise on the line. A legal or procedural question needs more thinking pauses than “what time do you open.” So the fix was not a single magic number, it was making every knob tunable without a code change, then dialing it on real test calls:

def _envf(name: str, default: float) -> float:
    try:
        return float(os.getenv(name, default))
    except ValueError:
        return default

min_silence = _envf("VAD_MIN_SILENCE_DURATION", 0.7)
activation = _envf("VAD_ACTIVATION_THRESHOLD", config.speech_threshold)
turn_unlikely = _envf("TURN_UNLIKELY_THRESHOLD", config.unlikely_threshold)

Lowering the voice-activity silence window from 1.5 seconds to 0.7 seconds cut about 800 milliseconds of felt dead air per turn, and it did not shorten anyone’s actual turn, because the semantic turn detector still holds the line open on a sentence that sounds unfinished. The database default stays conservative at 1.5 seconds on purpose, so a deployment that never sets the environment override is still safe; the override is the per-deployment tuning knob, dialed by ear, not a code change.

The trap: a free latency win that broke silently

The one worth telling a reviewer about is a feature called speculative or preemptive generation. It tells the voice framework to start synthesizing the model’s reply before the speech-to-text stream has fully finished processing the caller’s turn, shaving real time off the response. We turned it on. It looked like free latency.

It broke the text-to-speech pipeline in a way that produced no error and no warning, only a wrong answer. The framework’s per-turn hook, the one that runs after a turn is fully processed and sets state for the next stage, is where we set a flag deciding whether a spoken URL or email address should be rendered letter by letter or spoken naturally. Speculative synthesis starts before that hook runs. So the speech step read the previous turn’s flag instead of the current one. A caller who said “can you repeat that slowly?” got the answer spoken at conversational speed, exactly what they had just asked us not to do. We caught it on a test call and turned the feature back off.

session = AgentSession(
    ...
    # MUST stay False. The per-turn hook sets state the speech step
    # depends on, specifically a flag deciding letter-by-letter versus
    # natural delivery. Speculative synthesis starts before that hook
    # runs, so the speech step would read the previous turn's flag.
    # Observed on a test call: "repeat that slowly" spoken at normal
    # pace, email never spelled. The small time-to-first-token win is
    # not worth racing per-turn state.
    preemptive_generation=False,
)

The lesson generalizes past this one feature. Any optimization that starts work early is implicitly betting the state it depends on will not change in the meantime. That bet is safe when the pipeline is stateless from model output to audio. It is not safe the moment a per-turn hook injects state a later stage reads, and the failure mode is not a crash, it is confidently wrong output.

A second trap with the same shape: silent, not loud

A related failure hit earlier in the same project and cost a full afternoon. Tools have to be registered as decorated methods on the agent class, not assigned as instance attributes from a factory function:

class VoiceAgent(Agent):
    # Correct: tools discovered by inspecting the class
    @function_tool
    async def search_knowledge_base(self, query: str) -> dict:
        ...

class VoiceAgent(Agent):
    def __init__(self):
        # Wrong: the framework never sees this
        self.search_knowledge_base = create_search_tool(self)  # never registers

The framework discovers tools by inspecting the class at initialization time, not by looking at what ends up on an instance later. The factory-function version is a completely natural way to write Python, and it fails with no error, no warning, and no test failure unless a test specifically asserts the tool is present. The model simply never gets that capability and no one is told.

Both traps share a shape worth naming: a framework convention that is documented, reasonable, and easy to miss, whose failure mode is silence rather than a stack trace. The fix for that shape is the same both times: put the constraint in code comments at the exact line, and in the project’s onboarding guidance, not in a document nobody reopens.

Two more knobs that are easy to get wrong

Match the noise-cancellation model to the medium. Phone audio is 8 kHz, narrower than the 16 kHz a general-purpose noise-suppression model expects. The general model over-suppresses on phone audio and can clip the caller’s own voice along with the background noise. A telephony-tuned noise-cancellation model, built for the narrower band, avoids that.

Build the speech-to-text phrase list from real mishears, not a guess. A short list of domain terms biased into the recognizer’s decoder measurably improves accuracy on words a generic model gets wrong: regional place names, procedural vocabulary, acronyms specific to the domain. The way to build the list is to listen to the first hundred real calls and note what actually gets misheard, not to front-load it with vocabulary you assume the domain needs. Most vendors recommend keeping it focused; around 50 entries covered the real mishears on this line.

Hardening

• Prewarm the voice-activity model. Loading it at worker startup instead of on the first call moves about 200 milliseconds of cold-start cost off a real caller’s clock.
• Initialize independent services in parallel. The language model, speech-to-text, and database pool have no dependency on each other, so starting them concurrently instead of sequentially removes needless startup latency. Text-to-speech is the exception: it depends on a per-tenant configuration that comes from the database, so it stays sequential, at a cost of roughly 50 to 100 milliseconds, because per-tenant voice selection is a real product requirement.
• Use a native session event for a silence re-greet, not a polling timer. A framework-level event that fires after a fixed span of caller silence avoids racing partial transcription output the way a custom timer loop can.
• Log when a runtime override changes a database default. A startup log line stating the effective voice-activity silence value, and that it overrides the database default, saves the next person from debugging the wrong number.

Key takeaways

• The latency a phone caller feels is the sum of speech-to-text segmentation, voice-activity silence, and turn-detector patience, not the language model’s response time. Tune those three first.
• A semantic turn detector layered on top of a raw silence timer lets you safely shorten the silence timer, because the semantic layer becomes the real judge of whether a pause means the caller is thinking or finished.
• Any feature that starts work before a turn is fully processed is a bet that the state it depends on will not change in the meantime. Test that bet explicitly before shipping it, because the failure mode is a silently wrong answer, not a crash.
• Match infrastructure choices to the actual medium. Phone audio is not full-bandwidth audio, and a model tuned for the wrong bandwidth degrades quietly.

What this means if you are an IT services firm

Vendor latency claims for voice AI are almost always about the language model’s response time. That is not what a caller on a phone experiences. Ask what the voice-activity silence value actually is, whether a semantic turn detector runs on top of it, whether the noise-cancellation model is tuned for telephony audio specifically, and whether the speech-to-text engine has a domain phrase list. If the answers are “the default,” “no,” “the general model,” and “no list,” the agent will feel slow and mishear callers regardless of which language model is behind it. This kind of latency and reliability tuning is part of what we do behind IT services firms, under their brand.