fix(voice): don't start reply playout while the user is speaking

[ajayarora1235]

The bug

A reply can start playing into the user's speech when the user resumes talking inside the reply's generation window — after their turn commits, before the first TTS frame. Sequence (from a production call, all four occurrences of this class reproduced offline):

1. User pauses mid-thought → turn commits, reply generation starts.
2. User resumes 100-300ms later. on_start_of_speech's pre-playout hold fires and pauses the output (or no-ops, if the reply's SpeechHandle wasn't scheduled yet at onset).
3. Reply's TTS starts streaming → _audio_forwarding_task's first act is an unconditional audio_output.resume() — releasing the hold while the user is still mid-sentence.
4. The reply is audible over the user until an interruption path cuts it (the user hears the agent barge into their sentence and get chopped off).

Both existing defenses are edge-triggered, so neither can catch this: the playout-authorization _user_silence_event gather is sampled once at authorization (≈ the commit, before the generation window), and the on_start_of_speech hold both requires the handle to already exist and gets undone by the forwarding resume.

The fix

Make the check level-triggered at the true launch moment — the start of audio forwarding:

• perform_audio_forwarding / _audio_forwarding_task accept an optional hold_playout callback; when it returns True the initial resume() (which exists to clear a stale pause from an earlier speech) is skipped.
• AgentActivity._hold_playout_if_user_speaking(speech_handle) implements the check: user currently speaking + agent not audibly speaking + pause enabled + speech interruptible → apply the same hold as on_start_of_speech (_update_paused_speech(timeout=0) + pause()).
• Wired at the two pipeline call sites in _tts_task_impl. The realtime/shared-core path is untouched (hold_playout=None → previous behavior).

No new release machinery. Once held, resolution is the existing paths in both directions: user stops without a turn commit → their end-of-speech arms the false-interruption timer (timeout 0 → immediate resume); user's continuation commits a turn → the held speech is interrupted and the superseding reply (generated from the full transcript) plays instead — the stale reply is never audible.

Verification

• 7 unit tests (tests/test_playout_launch_hold.py) covering the level-check conditions and the forwarding head honoring the hold.
• End-to-end against a live STT session (our downstream harness, synthesized caller audio, scripted LLM at incident latency):
  • before: hold placed at user onset → forwarding resume 0.18s later → 0.56s of reply audible over the user → cut.
  • after: first TTS frame captured while paused; a continuation supersedes the held reply with 0.00s played; a backchannel releases at end-of-speech with the reply then delivered in full.
• tests/test_false_interruption_resume.py still passes; ruff check/format clean.

Trade-off

A short utterance landing exactly in the generation window now delays the reply until the user's end-of-speech (typically a few hundred ms) instead of playing over them — silence instead of overlap.

Related: #6714 (same family — pause/resume vs. late transcripts — different window).

🤖 Generated with Claude Code

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 2 additional findings in Devin Review.

[devin-ai-integration]

🔴 Agent can fall silent when a spoken message is held back during a realtime-model session

The reply's audio is put on hold (_hold_playout_if_user_speaking at livekit-agents/livekit/agents/voice/agent_activity.py:4286-4321) whenever the user is marked as talking, but in sessions where the talking/not-talking signal comes from the speech model's own server-side detection nothing ever releases that hold, so the message stays stuck and unheard.
Impact: In realtime-model sessions the agent can go silent indefinitely — the queued message never plays and later replies stay queued behind it — until the user happens to start talking again.

Why the release machinery never fires for realtime server-side turn detection

The hold's only release paths are (a) AgentActivity.on_end_of_speech arming the false-interruption timer (livekit-agents/livekit/agents/voice/agent_activity.py:2117-2118) and (b) on_final_transcript → _cancel_speech_pause (livekit-agents/livekit/agents/voice/agent_activity.py:2244-2246). Both hooks are driven exclusively by the local VAD/STT recognition loop (livekit-agents/livekit/agents/voice/audio_recognition.py:1321,1421,1860).

With a RealtimeModel using server-side turn detection and no user VAD, user_state is instead driven by _on_input_speech_started/_on_input_speech_stopped (livekit-agents/livekit/agents/voice/agent_activity.py:1892-1919), which call self._session._update_user_state(...) directly and never invoke on_end_of_speech, and realtime transcripts go through _on_input_audio_transcription_completed, not on_final_transcript.

say() in such a session still routes through _tts_task_impl whenever a TTS is configured (livekit-agents/livekit/agents/voice/agent_activity.py:1466-1468), which is exactly where hold_playout is now wired. So: user starts talking (server VAD) → say() starts forwarding → hold placed, audio_output.pause() → _tts_task_impl blocks on audio_output.wait_for_playout() (the room output's playout wait blocks on _playback_enabled, livekit-agents/livekit/agents/voice/room_io/_output.py:148-158). The scheduler is itself blocked in speech._wait_for_generation() (livekit-agents/livekit/agents/voice/agent_activity.py:1717), so its resume safety net at livekit-agents/livekit/agents/voice/agent_activity.py:1720-1725 cannot run either. Recovery only happens if the user speaks again, since _on_input_speech_started calls self.interrupt().

Prompt for agents

The new pre-playout hold in AgentActivity._hold_playout_if_user_speaking pauses the audio output and records _paused_speech, relying entirely on the existing false-interruption machinery for release. That machinery is only armed by AgentActivity.on_end_of_speech and on_final_transcript, which are driven by the local VAD/STT recognition loop. When a RealtimeModel drives turn detection server-side (no user VAD), user_state transitions come from _on_input_speech_started/_on_input_speech_stopped, which never call those hooks — yet say() still routes through _tts_task_impl (where hold_playout is wired) when a TTS is configured. The result is a hold that is never released: _tts_task_impl blocks in wait_for_playout, the scheduling task blocks in _wait_for_generation, and the scheduler's own resume safety net cannot run. Consider either (a) not applying the hold when turn detection is server-side/realtime (e.g. gate on self._rt_turn_detection_enabled or on the recognition hooks being the source of user_state), or (b) giving the hold its own guaranteed release (arm a timer at hold time, and/or release on the realtime input-speech-stopped path).

---

Was this helpful? React with 👍 or 👎 to provide feedback.