How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen
Summary
- Enterprise voice remains an unsolved systems problem even as speech components become cheaper and more available. Early ChatGPT wrappers simply piped ASR, an LLM, and TTS together, but Wen argues that voice adds time, noise, interruption, and adaptation: it is “one step to the real world,” where intelligence means responding to how someone speaks, not merely reasoning over what they said.
- The winning architecture may be end-to-end for listening and turn-taking while retaining text as the controllable enterprise interface. Wen’s audio-native model streams audio directly, predicts whether the user has finished, then emits an answer or tool call, citations, and finally a transcript. Text output preserves guardrails and auditability without forcing the system to wait for transcription before responding.
- Production conversation data is a defensible advantage because the hard cases barely appear in clean laboratory audio. Their platform draws—with customer agreement—from contact-center deployments across banking, utilities, logistics, restaurants, hotels, retail, and outbound sales, redacting personal data and inserting synthetic identities. Training deliberately includes crosstalk and combinations such as a name spoken over “a baby crying in the background”; Wen says old denoising pipelines can make the new model worse by creating an environment that is “too sanitized.”
- Latency is principally a turn-taking problem, not raw model-inference speed. A fixed endpoint threshold either interrupts slower speakers or leaves impatient users waiting, whereas an audio-native model can use pace and unfinished semantics to recognize a mid-sentence pause. The company combines that with precomputation, self-hosting to reduce P95 network delays, and “latency-budgeted reasoning” that asks whether another second of thought will materially improve the answer.
- Voice identity directly affects performance and trust, but enterprises remain wary of excessive anthropomorphism. Generic voices inherit the baggage of broken IVR systems, while a modest regional character—such as a Newcastle accent in the UK—can signal a familiar human service context. The company once found its most natural English voice was a French voice forced to speak English: “It’s the imperfectness of that voice [that] make it real.”
- Evaluation and control—not model access—are emerging as the enterprise value layer. Wen criticizes voice benchmarks that ignore latency, tool use, citations, name recognition, and task completion; his company instead evaluates on real conversational data and planned to release that benchmark within “the next month or two.” Enterprises may switch among model APIs, but many still want to own the harness that encodes their brand, workflows, guardrails, and audit trail.
- AI abundance shifts the organizational bottleneck from generation to supervision. Wen’s engineers already face more code than they can comfortably review: “The bottleneck has already shifted from producing content to actually validating and auditing the content.” The longer-term market may split between playful consumer voice and regulated enterprise voice, but adoption still depends on privacy norms, cultural context, and whether agents can become invisible without producing unaudited “AI slop.”
Deep dive
1. Voice adds time and adaptation beyond increasingly commoditized speech components
Wen initially shared the widespread post-ChatGPT expectation that voice would soon be solved. Wrapper companies connected ASR, ChatGPT, and TTS, yet real deployments revealed a missing dimension: voice is “one step to the real world,” where timing materially changes what a response means.
Text intelligence benefits from abundant documentation and online papers; machines have far less observational data about embodied or spoken interaction. Wen’s distinction is the episode’s foundation: “The intelligence here is not really about reasoning, it’s really about adaptation.”
Tim Scarfe’s robotics analogy makes the difficulty concrete: making coffee or pouring beer is effortless for a person but difficult to specify for a machine. Conversation has the same hidden complexity—humans generate enormous amounts of it, but “machines just never [were] able to observe” the relevant rhythm, hesitation, and adjustment.
2. Enterprises demand human capability with robotic predictability
The enterprise brief contains an explicit contradiction: customers want branded voices that pronounce company names exactly as intended, and they want an agent “super powerful like a human being” yet “super controllable like a robot.” Expectations often benchmark the system against the company’s best employee, not its average operation.
Wen sees efficiency-focused deployments as the practical adoption wedge. Once an agent succeeds in a bounded task, customers begin to conclude that it is “better than I thought,” creating room for broader use without requiring human-level performance everywhere on day one.
Frontier speech-to-speech systems can produce demos that feel like “singularity already,” but contact centers expose different failure modes. An older caller may pause because she is confused, only for an aggressively tuned agent to keep speaking over her—precisely the experience a polished consumer demonstration does not test.
A cascaded stack separately handles recognition, language, and endpoint detection, forcing engineers to tune brittle parameters or bolt on rules such as detecting an older speaker. Collapsing those functions lets one model perceive slow pace, background noise, crosstalk, and hesitation, then learn the appropriate response from deployed conversations.
3. Audio-native input and text output create a controllable contract
Wen’s model is audio-native but not waveform-to-waveform: an LLM consumes streaming audio representations and emits text. Wen keeps text because enterprises can apply guardrails to it more reliably; unconstrained speech output is “a lot fiddlier to deal with.”
As each audio chunk arrives, the model repeatedly generates a cheap first token representing turn state: the user is starting, continuing, or finished. When it predicts the end, the audio is already encoded in the embedding space, so the LLM can immediately stream an answer or tool call without reprocessing the utterance.
The output order is intentionally reversed from a classic pipeline. The response comes first, followed by citations identifying the knowledge-base facts used, and finally the transcript for debugging and audit—meaning “you don’t have to wait for the transcript” before replying.
Retrieval also becomes a tool call rather than conventional transcription-first RAG: the model generates a search query when it needs outside information. Wen says the real innovation is the “IO contract”—the training pairs, data preparation, and prescribed outputs—while open-source base models can be swapped and retrained as stronger candidates arrive “every month, even every week.”
4. Real noise is training signal, not contamination
Their platform already receives relevant audio from deployments spanning banks, utilities, logistics, restaurants, hotels, retail, and outbound sales. Customer permission remains non-trivial under GDPR and personal-data constraints, so the company redacts real identities and generates synthetic names and addresses; the goal is to learn conversational mechanics, not customer facts.
Audio is represented through mel spectrograms streamed segment by segment, making perception resemble image recognition. Scarfe recalls how trained humans can read speech, harmonic instruments, percussion, or salt-and-pepper noise visually; neural models can likewise learn patterns from that representation.
Noise augmentation is deliberate. Wen’s example combines a synthesized pronunciation of his Mandarin name with realistic production disturbances such as “a baby crying in the background,” multiplying the environments the model can experience without exposing private conversations.
Counterintuitively, legacy denoising and pipeline sanitation can degrade the integrated model because the cleaned signal is unlike deployment audio. For crosstalk, their model currently uses no specialized diarization mechanism; the fused model can instead be prompted to ignore background speakers, follow the main participant, or join a multi-speaker exchange, depending on its training and instructions.
5. Natural latency comes from knowing when not to answer
In cascaded systems, endpointing is governed by a coarse wait-time setting. Set it short and the agent interrupts slower or older callers; set it long and faster speakers assume the system is unresponsive. One global threshold cannot represent every user’s conversational tempo.
The integrated model sees both pace and meaning. If a speaker pauses before completing the semantics of a sentence, it can keep waiting; meanwhile, incremental encoding precomputes work, and hosting the model beside the rest of the company’s platform removes network travel that disproportionately damages P95 latency.
Wen wants to reinvest that saved time in “latency-budgeted reasoning.” The model estimates whether stopping now would still yield a good answer and is trained with constraints such as, “Do you actually need to think that much time? Can you cap it to, like, one second?”
Longer backend calls still require visible feedback. Fillers and typing sounds work until they become repetitive, while hold music buys time for deeper reasoning or slow enterprise APIs. Silence is dangerous: if the AI goes dark for even “3 or 5 seconds,” callers may panic and wonder whether it is working, particularly after years of unreliable automation.
6. Familiar imperfection often earns more trust than generic polish
Most enterprises do not appear to want an assistant pushed far toward indistinguishability from a person; they want it to solve the problem while carrying a restrained brand identity. A South American casino explored a strongly localized opening voice but ultimately declined to launch it because it “feels a bit too much,” illustrating how enthusiasm recedes near production.
Generic voices often perform worse because callers associate them with rigid IVR prompts that repeatedly failed to recognize “payment” or “transfer.” Regional cues can reverse that expectation: UK users may trust a Newcastle voice because many contact-center employees are from there, signaling that they can state their problem and have the agent solve it.
The company’s early English voices sounded less natural than a French voice made to speak English. Its slight accent supplied the useful imperfection. Wen therefore treats voice selection as brand- and region-specific, not as a universal race toward maximum smoothness or a single supposedly neutral voice.
7. A credible voice benchmark must measure the whole interaction
Wen argues that standalone ASR and LLM benchmarks miss the multidimensional nature of an agent: response latency, transcription, factual quality, reasoning difficulty, personal-name recognition, instruction following, citations, and tool calls all matter. “A voice agent benchmark without latency consideration is not a real one”; a caller cannot wait 60 seconds.
Public datasets are often synthetic because real voice is private, and existing tests do not sufficiently represent production. The company built an internal benchmark from consumer conversations and Wen says its model is currently the best on that benchmark; his stated plan was to release it to researchers and the market within “the next month or two.”
Subjective experience remains harder than factual scoring because accent and tone vary by culture and task. At production grade, Wen prefers behavioral evidence: did users engage, complete the task, and obtain a satisfactory resolution? Those outcomes evaluate the full model-plus-harness system rather than the model alone.
8. Enterprises will own boundaries and workflows before they own weights
Enterprises increasingly view AI as technology they must own, but model ownership requires tuning expertise, GPUs, and budgets many lack. They therefore accept switching among model APIs while treating the harness—the prompts, tools, brand behavior, and process logic—as the attainable form of proprietary intelligence.
Wen’s proposed boundary is a fast voice agent as the customer-facing “front door,” delegating difficult work to the enterprise’s own “mastermind behind the scenes.” He expects companies to discover that they need not own every layer, though some will retain custom harnesses when the task is sufficiently critical.
B2B conversation is unusually amenable to optimization because its specification is clearer than social chat: was the problem solved, did the customer feel good, and how quickly? Wen contrasts this with human chitchat, where success may alter a relationship and no comparable gold standard exists.
Autonomy still collides with governance. One large utility customer said future PRs would need an auditable review cycle lasting two weeks; automatic merging would not be accepted. The company therefore retains flow visualizations, bounded tool sets, and per-response citations—no-code-style interfaces now serving as audit views rather than the primary construction surface.
9. The next bottleneck is human attention, context, and adoption
Scarfe describes “cognitive debt”: agents can work while he walks, yet returning to the computer may reveal duplicated agents, wrong delegation, or reasoning nobody supervised. Wen sees the same organizationally as AI floods Slack and code review: “Don’t send people the AI slop that you have not read yet.” Humans must understand and endorse what they transmit.
Voice limits that problem but introduces another. Their production replies average roughly 50 output tokens, avoiding walls of prose, yet speech is a low-bandwidth, “very ping-pong-y” channel. Scarfe’s breadth-first workflow can coordinate email, editing, and coding agents, but Wen doubts voice alone can communicate the full picture without a harness that surfaces concise high-level bullets.
Making agents invisible requires richer organizational context, not just more text. Written values omit why those values arose, so a model may overemphasize one of six principles. Even “slop” is recipient-dependent: a message optimized for Wen’s context may feel irrelevant to a colleague, creating the circular prospect of agents generating rich output for other agents to personalize.
Wen expects end-to-end turn-taking to become foundational, then split into two branches: playful consumer assistants and professional, regulated, auditable enterprise voice. Meeting coordination, internal productivity, and a “second wave of IoT” may follow, but he is explicit about uncertainty: the technology could mature within ten years while social discomfort around office speech, privacy, smart glasses, and surrendering control still slows adoption.