fix(deepgram): keep streaming audio while an utterance is in progress - #2243
Open
CampbellMBXJ wants to merge 1 commit into
Open
fix(deepgram): keep streaming audio while an utterance is in progress#2243CampbellMBXJ wants to merge 1 commit into
CampbellMBXJ wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 3f04b5b The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
AudioEnergyFilter stops forwarding frames once its cooldown elapses with no audio above the RMS threshold. Deepgram only fires endpointing on silence it actually receives, so when the gate closed after Deepgram had reported speech but before it had endpointed the utterance, speech_final never arrived. The pending final was only flushed when the speaker started talking again, and until then audio_recognition had no transcript to commit the turn with. Bypass the gate between start of speech and endpoint, and clear that state when a websocket session starts so a reconnect cannot inherit an unfinished utterance from the previous socket. The sarvam plugin already resets its equivalent flag per session. pushFrame is still evaluated on every frame so the filter's cooldown stays accurate once the utterance ends.
CampbellMBXJ
force-pushed
the
fix/deepgram-audio-gating-during-utterance
branch
from
August 7, 2026 19:43
47f6a72 to
3f04b5b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #2242.
AudioEnergyFilterstops forwarding frames once its cooldown elapses with no audio above the RMS threshold. Deepgram only firesendpointingon silence it actually receives, so when the gate closed after Deepgram had reported speech but before it had endpointed the utterance,speech_finalnever arrived. The pending final was only flushed when the speaker started talking again, and until thenaudio_recognitionhad no transcript to commit the turn with -runEOUDetectionskips on an emptyaudioTranscript, so the agent sat silent untiluserAwayTimeoutfired 15s later.Changes Made
plugins/deepgram/src/stt.ts- bypass the energy gate while an utterance is in flight, using the existing#speakingflag (set onSpeechStartedor the first non-empty result, cleared onspeech_final/UtteranceEnd).plugins/deepgram/src/stt.ts- reset#speakingat the start of#runWS. This is a pre-existing leak (updateOptions()resolves#resetWSand forces a reconnect, e.g. via deferred keyterm updates), but the bypass above raises its cost from a stale flag to the gate being bypassed for the rest of the session.plugins/sarvam/src/stt.tsalready resets its equivalent flag per session; this matches it.plugins/deepgram/src/stt.test.ts- three regression tests on the existing local-websocket harness.Pre-Review Checklist
build,lintand the deepgram test file pass.Testing
restaurant_agent.tsandrealtime_agent.tswork properly (for major changes)Three tests added to
plugins/deepgram/src/stt.test.ts, using the existingWebSocketServerharness.stops sending low-energy audio once the cooldown elapses- pins the existing gating behaviour, so the fix can't silently disable the filter.keeps sending low-energy audio while an utterance is in progress- the fix itself.does not carry an unfinished utterance into a reconnected websocket- the#runWSreset. Asserts_speakingis false after a reconnect triggered mid-utterance byupdateOptions.Additional Notes
Test 3 asserts on
_speaking, which is@internal. It's the most direct deterministic signal for the reconnect case. I first tried asserting on forwarded-frame counts, but reconnection drops frames on its own, so the signal was 4 vs 7 frames rather than 4 vs 15 — too timing-dependent to assert on without flake. Happy to change the approach if you'd rather not have a test reach for that getter.Billing. More audio is forwarded between gate-close and endpoint - normally a few hundred milliseconds per utterance. The duration collector only counts sent frames, so reported recognition usage reflects it. The
#runWSreset is what bounds this: without it a stuck#speakingwould forward audio indefinitely.I marked the changeset
patch. It is a bugfix, but it does change audio-forwarding behaviour for every Deepgram streaming user, so say the word if you'd rather it were a minor.