We Split One Prompt Into Two to Fix a Small LLM. Then We Deleted Both.

All names, numbers, and identifiers in this post are anonymized. The patterns are real.

The question a code reviewer would reasonably ask

A voice agent we operate for a public-sector call center runs a small, cost-optimized LLM at temperature 0. Every caller turn fires two separate calls to it, not one:

1. A classifier call. Input is the caller’s latest message plus a short history window. Output is exactly one line: a category label and a brief reason. Nine categories cover things like an immediate transfer, a transfer request, a routine request, a request that needs clarification, an out-of-scope question, legal advice, small talk, and emotional distress. The classifier prompt is about 290 lines.
2. A responder call. Input is a 244-line system prompt, the running conversation, and a per-turn context block the code injects with the classifier’s label and pre-fetched knowledge base results. Output is the spoken response.

Together that is about 534 lines of instructions split across two calls, and roughly 600 milliseconds of extra latency for the classifier step that one combined call would not have. Any reviewer looking at this codebase would reasonably ask why we pay that latency twice instead of using one prompt with structured output that classifies and responds in the same call.

We tried the combined version. It failed in a specific, repeatable way, and we reverted it.

What we tried instead

The combined-prompt attempt was one LLM call with a single prompt containing both the classifier’s rules and the responder’s rules, asked to return a structured object with two fields: a classification and a response. Same model, same temperature, same generation settings as the split version.

The failure showed up in two complementary ways on different calls:

• Sometimes the classification came out correct but the response broke a specific hard rule from the responder prompt, most often reading something in digit form when the spoken form was mandatory, or using a written phrasing that a phone caller would mishear.
• Sometimes the response was fine but the classification was sloppy, for example labeling a clear payment-action request as a routine request instead of an immediate transfer, which then meant downstream code silently disagreed about what the turn actually was.

Both failure modes matter. A bad response is a bad caller experience. A bad classification breaks the deterministic code that depends on it: an abuse counter, a daily rate limit, transfer routing, distress detection, and the gating that decides whether a canned response is appropriate. The classifier was not just deciding what to say next. It was the gatekeeper for guardrails the rest of the system trusted.

Why the combined prompt degraded

The model’s instruction-following got worse as the combined prompt grew. Somewhere between 200 and 400 lines of instructions in one call, hard rules started to be silently dropped, not on every turn, but often enough that roughly 1 in 30 turns violated a specific format rule.

Asking one call to also produce a machine-readable classification adds a second job the model has to hold in working memory while generating a response. The classification stayed correct surprisingly often, since it is a small, example-heavy decision. The response quality is what degraded. The opposite failure, a good response with a sloppy classification, was worse for us, because we had code that acted on the classification directly.

Splitting the work avoided this because each call had one job and one output format:

• The classifier prompt is dominated by labeled examples: representative messages with their correct category. Small models pattern-match examples reliably even when they drift on generalized rules.
• The responder prompt is dominated by formatting rules and per-category playbooks. It assumes the routing decision is already made, since the classifier’s label arrives as part of the per-turn context, and only has to follow the playbook for that one label.
• Each prompt changes independently. A new classifier example does not risk breaking a responder formatting rule, and a new responder rule does not risk breaking a classifier example.

Classifier output format:

Respond with exactly: CATEGORY: reason
CATEGORY is one of nine fixed labels.
The reason is a brief phrase, under 25 words.
Do not add a greeting before the category.
Do not wrap the response in quotes, JSON, or markdown.

Example: ROUTINE_REQUEST: Caller is asking about payment options

Making the extra latency affordable

Two calls instead of one adds latency, so we ran the classifier and a knowledge base search concurrently instead of in sequence. The classifier takes about 0.6 seconds. The knowledge base search, a vector and keyword lookup, takes about 0.7 seconds. Run one after the other, that is 1.3 seconds before the responder call even starts. Run in parallel, it is the slower of the two, about 0.7 seconds.

On the roughly 80 percent of turns classified as a routine request, the knowledge base result is needed anyway, so the parallel run costs nothing extra beyond the 0.7 seconds either call would have taken alone. On other labels, the knowledge base task is cancelled once the classification is known; the wasted lookup costs a fraction of a cent and adds no latency the caller notices.

The reliability gain was worth the added latency. A wrong response is a bad caller experience. A wrong classification breaks code. An extra half second of thinking time is just thinking time, and it is largely hidden by running two things at once instead of one after another.

What changed a few weeks later

Here is the part worth being honest about. A few weeks after we shipped the two-call design, a client review of this same voice agent came back with a low score. Not because of the classifier specifically, but because the whole pipeline it belonged to, the classifier plus a routing table plus a set of canned responses plus a separate sub-agent for one complex flow, made the agent sound like it was reading from a script instead of holding a conversation.

The fix for that was a much bigger redesign: one LLM call per turn, one prompt file, no pre-classification, no canned short-circuits, no sub-agent. The audit trail that the classifier’s label used to feed moved to an evaluation step that runs after the call ends instead of a decision the caller has to wait on. We wrote about that redesign, what got deleted, what stayed, and how we kept it auditable without a live classifier, in when to build a custom voice agent versus a platform.

Once that redesign moved the audit trail off the caller’s clock, the classifier’s output had nowhere left to go. Nothing downstream needed a machine-readable label anymore. That is a different question than the one this post answers. This post is about whether one LLM call can reliably do two jobs at once; the answer on this small model was no. The later redesign is about whether you need the second job at all; the answer, once the audit trail moved elsewhere, was also no. Both conclusions are correct for the question they answer.

Hardening still worth doing

A few items stayed on the list even after the classifier was retired from this particular system, because the underlying lesson applies to every prompt we write for a small model:

• A rule budget. Every rule added to a prompt competes with every other rule for the model’s attention. The discipline that kept both prompts working was simple: add a rule, remove a rule, or move it into code.
• A drift eval. Picking a handful of hard format rules and replaying them against a sample of real calls on every model update would catch silent drift before a caller does.
• Documenting the rule-budget convention at the top of the prompt file itself, not just in engineering memory, so the next person maintaining it inherits the discipline.

Key takeaways

• A small, cost-optimized LLM asked to do two jobs in one call will degrade on one of them once the combined prompt gets long enough. On this model, the cliff started around 200 to 400 lines.
• Structured output constrains the shape of a response, not its content. It does not fix content-level drift.
• A classifier’s output is often a routing signal that other code depends on, not just metadata for a response. Treat it with the weight of the guardrails it feeds.
• Splitting a prompt and deleting a prompt are different decisions that can both be correct. One is about whether a model can do two jobs at once. The other is about whether you still need the second job.

What this means if you are an IT services firm

Ask your team what job each LLM call in your voice agent is doing, and whether anyone has tested what happens when one call is asked to do two of those jobs at once. If a single prompt is quietly growing past a few hundred lines while also producing a machine-readable output your code depends on, that is worth a deliberate test before it becomes a production surprise. This kind of prompt-level reliability work is part of what we do behind IT services firms, under their brand.