fix(voice): defer false-interruption resume while a transcript is pending - #6714
fix(voice): defer false-interruption resume while a transcript is pending#6714ajayarora1235 wants to merge 2 commits into
Conversation
A pause backed by live speech evidence (SOS + word-bearing interims, final
still in flight) resumes on a bare END_OF_SPEECH: on_end_of_speech arms the
false-interruption timer with the pause's own timeout (0 for a pre-playout
pause), and _on_timeout only defers on an OPEN _end_of_turn_task — between a
speech window's EOS and its pending final there is no open decision, so the
stale reply resumes into the caller's breath. Observed in production as held
replies leaking one word ('I'm' ... 'I') into each pause of a caller's
narration before being killed by the commit.
…ding With STT-based turn detection, a speech window's END_OF_SPEECH can arrive before its FINAL_TRANSCRIPT. In that gap no _end_of_turn_task is open, so the deferral from livekit#6662 has nothing to hook: the resume timer fires — with timeout=0 for a pre-playout pause taken by on_start_of_speech — and stale held audio resumes into the caller's breath, emits a word, and is killed by the commit moments later. Defer the resume while word-bearing interims are in flight, mirroring the existing eot-task deferral: the coming final will start (or refresh) the turn decision that owns the pause. The deferral is bounded (_PENDING_TRANSCRIPT_MAX_DEFERRAL, 2s) so a dead STT stream with pending interims degrades to the plain timeout behavior instead of holding the pause forever.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| # a speech window closed but its transcript is still in flight: the | ||
| # coming final will start (or refresh) the turn decision that owns | ||
| # this pause — re-check instead of resuming stale audio into the gap | ||
| if self._audio_recognition and getattr( | ||
| self._audio_recognition, "_audio_interim_transcript", "" | ||
| ): | ||
| if transcript_wait_deadline is None: | ||
| transcript_wait_deadline = time.monotonic() + _PENDING_TRANSCRIPT_MAX_DEFERRAL | ||
| if time.monotonic() < transcript_wait_deadline: | ||
| self._false_interruption_timer = self._session._loop.call_later( | ||
| _PENDING_TRANSCRIPT_RECHECK, _on_timeout | ||
| ) | ||
| return | ||
| # the transcript never materialized (e.g. the STT stream died): | ||
| # fall through and let the timeout rule, as before this guard |
There was a problem hiding this comment.
🟡 Agent playback can stay silent for two extra seconds after a leftover partial transcript
The resume of paused agent audio is postponed whenever any leftover partial transcript text is present (getattr(..., "_audio_interim_transcript", "") at livekit-agents/livekit/agents/voice/agent_activity.py:4369-4371), even when that text is stale from an earlier turn, so the agent can stay silent for up to two seconds longer than configured.
Impact: After a brief non-interrupting noise, the caller may hear an unexpected multi-second gap before the agent continues speaking.
Stale interim text is never scoped to the current speech window
_audio_interim_transcript is only cleared when a non-empty final transcript arrives (livekit-agents/livekit/agents/voice/audio_recognition.py:1235), in the manual-commit path (livekit-agents/livekit/agents/voice/audio_recognition.py:1104) or in _clear_user_turn (livekit-agents/livekit/agents/voice/audio_recognition.py:1009). It is NOT cleared when a user turn commits through _run_eou_detection (livekit-agents/livekit/agents/voice/audio_recognition.py:1744-1747 clears only _audio_transcript), and an empty final returns early before the clear (livekit-agents/livekit/agents/voice/audio_recognition.py:1212-1213).
So if a trailing interim never receives its final (dropped/empty final from the provider), the field stays non-empty for the rest of the session. Every subsequent false-interruption pause then hits the new guard, re-arms at 100 ms intervals and only resumes after _PENDING_TRANSCRIPT_MAX_DEFERRAL (2 s) on top of the configured false_interruption_timeout.
A scoping condition (e.g. only defer when the interim text changed after the pause/EOS, or track a per-window generation counter) would keep the fix targeted at genuinely in-flight transcripts.
Prompt for agents
The new pending-transcript deferral in AgentActivity._start_false_interruption_timer (livekit-agents/livekit/agents/voice/agent_activity.py) treats any non-empty AudioRecognition._audio_interim_transcript as evidence that a transcript is in flight. That field is not scoped to the current speech window: it is cleared only on a non-empty FINAL_TRANSCRIPT (audio_recognition.py:1235), in the manual commit path (audio_recognition.py:1104) and in _clear_user_turn (audio_recognition.py:1009). An empty final returns early before the clear (audio_recognition.py:1212-1213) and a turn commit via _run_eou_detection clears only _audio_transcript (audio_recognition.py:1744-1747). Consequently a stale interim left over from an earlier turn will delay every later false-interruption resume by the full 2s cap. Consider scoping the guard, e.g. only defer when the interim was updated after the pause started / after the last END_OF_SPEECH (record a timestamp or monotonically increasing counter on AudioRecognition when _audio_interim_transcript is assigned), or clear _audio_interim_transcript when a user turn commits.
Was this helpful? React with 👍 or 👎 to provide feedback.
| self._false_interruption_timer = self._session._loop.call_later( | ||
| _PENDING_TRANSCRIPT_RECHECK, _on_timeout | ||
| ) | ||
| return |
There was a problem hiding this comment.
If the deadline expires while the interim transcript is still non-empty, we fall through and resume the paused speech.
This creates vendor-dependent behavior: ghost interims unnecessarily extend the pause, while finals delayed beyond the window can still reproduce the stale-resume issue.
Should the maximum deferral be configurable, preferably as a duration rather than a boolean, so integrations can control how long interim transcripts are trusted?
2 participants
Problem
With
resume_false_interruptionenabled, stale held agent audio resumes into the caller's pause. Observed in production (STT turn detection, Cartesia Ink-2): an agent reply held pre-playout behind a long caller narration resumed at each of the caller's breaths, emitted one word ("I'm" … then, on the next held reply, "I"), and was killed by the turn commit moments later — heard by the caller as the agent repeatedly butting in at every pause they took. (Details anonymized; the repro below is fully synthetic.)Mechanism
The EOS-to-final gap. With STT-based turn detection, a speech window's
END_OF_SPEECHcan arrive before itsFINAL_TRANSCRIPT(a long utterance produces several SOS/EOS windows; the committing final lags). In that gap:on_end_of_speecharms the false-interruption timer with the pause's own timeout — which is 0 for a pre-playout pause taken byon_start_of_speech("resume immediately when user stops speaking") — so the resume is instantaneous at the caller's first breath;_on_timeout's deferral (from fix(voice): avoid dropping turns and resuming early while a speech is paused #6662) only hooks an open_end_of_turn_task, and no bounce is open until the final arrives — so nothing defers the resume even though word-bearing interims prove live speech and a turn decision is imminent.Repro
Two deterministic tests (
tests/test_stale_resume_stt_windows.py), built on thetest_false_interruption_resume.pyidiom — red onmainat the first commit (check out the test-only commit to see them fail):test_resume_is_immediate_when_no_turn_decision_is_open: there the EOS is a lone noise glitch; here a real speech window with pending interims precedes it.)Fix
Defer the resume while word-bearing interims are in flight, mirroring the existing eot-task deferral: the coming final will start (or refresh) the turn decision that owns the pause (and the final path already interrupts it via
_cancel_speech_pause). The deferral re-checks at 100ms and is bounded (_PENDING_TRANSCRIPT_MAX_DEFERRAL, 2s): if the transcript never materializes (e.g. the STT stream died), behavior degrades to the plain timeout — the pre-fix fallback. A third test pins the bound.Design question for maintainers: would you prefer hooking the transcription-timeout event instead of the bounded re-check? The re-check is self-contained in
_start_false_interruption_timer; the event hook would avoid polling but couples the timer to the recognition lifecycle. Happy to rework.All 8 existing tests in
tests/test_false_interruption_resume.pystill pass (plustest_agent_session.py,test_agent_task_close_race.py— 91 total across the four files). Extends #6662.🤖 Generated with Claude Code