fix(voice): playback-aware muting when \--speak\ and \--voice\ run to… - #349
fix(voice): playback-aware muting when \--speak\ and \--voice\ run to…#349Adityakk9031 wants to merge 2 commits into
Conversation
…gether Introduce a thread-safe registry (_active_speakers guarded by _speakers_lock) in _capture.py to track active audio synthesis / playback sinks. Modify SpeakerSink._worker in _speaker.py to register itself to _active_speakers during active playback chunk writes. Update MicrophoneCapture._callback in _capture.py to zero-fill captured blocks if a speaker is active, preventing TTS feedback echo. Bump plugin version to 0.5.2 and add unit tests. Closes robocurve#332
|
@jeqcho have a look |
jeqcho
left a comment
jeqcho
left a comment
There was a problem hiding this comment.
Thanks @Adityakk9031 for taking on #332 — the mechanism you built is the right shape (a small shared signal between SpeakerSink and MicrophoneCapture, exactly what the issue asked for), the try/finally bracketing is careful, and the tests for registration and muting are clear. Two behavioral issues need addressing before this fixes the bug end to end, though:
1. Zero-filling collapses the adaptive noise floor (blocking)
The muted blocks still flow into EnergyGate.push, and a zero block has RMS 0. While the gate is closed, _push_closed in plugins/inspect-robots-voice/src/inspect_robots_voice/_segmenter.py runs
self._noise_floor = (1.0 - self.ema_alpha) * self._noise_floor + self.ema_alpha * rmsso each 0.1s zero block multiplies the noise floor by 0.95. A 10-second narration is ~100 blocks, shrinking the floor ~170x. When the mute lifts, ordinary room noise is far above the near-zero threshold, so the gate opens immediately — and since the floor only re-adapts while the gate is closed, it can stay open until the 30s max_utterance_s cap and ship a long noise "utterance" to the transcriber. That's the very hallucination path this PR is meant to close, so the description's claim that zeroing "keeps the adaptive energy gate statistics stable" is unfortunately inverted. Options, roughly in order of preference:
- Drop muted blocks in
MicrophoneCapture._callback(_capture.py~line 138) instead of enqueueing zeros — this matches the issue's "discard captured segments" wording and freezes the gate state during playback; or - keep zero-filling but have the segmenter skip noise adaptation for muted blocks; or
reset()the gate when playback ends.
2. Echo tail leaks after unmute (blocking, smaller)
_active_speakers.discard(self) in _speaker.py fires as soon as the last blocking playback.write returns, but output-stream latency plus room acoustics plus the 0.1s input blocksize mean the tail of the narration is still arriving at the mic for a few hundred milliseconds afterward — and there are brief unmuted gaps between queued utterances (register/discard is per queue item). A short mute hangover would cover both: e.g. record last_playback_end = time.monotonic() on discard and treat capture as muted while a speaker is active or within ~0.3–0.5s of the last one. That would also let you add the test the issue asks for (a segment overlapping a playback window is dropped, including its tail).
Smaller points
_speaker.py(~line 312): thefrom inspect_robots_voice._capture import ...sits inside the worker loop and re-executes every iteration. There's no import cycle (_capturedoesn't import_speaker), so please hoist it to the module top.uv.lock: ~100 lines of environment-marker churn unrelated to the version bump — looks like it was regenerated with a differentuvversion. Could you re-lock so only theinspect-robots-voiceversion entry changes?tests/test_tts.py: there.escapefix is sensible (Windows paths, I'd guess) but unrelated to this PR — worth a mention in the description or splitting out so the changelog story stays clean.- Taking a
threading.Lockinside the PortAudio callback is fine here given both critical sections are tiny, so no change needed — just noting it was considered.
Really appreciate the thorough write-up and the passing CI matrix — with the drop-instead-of-zero change and a small unmute hangover this will land nicely.
2 participants
close #332
Description
With both
--speakand--voiceenabled, the microphone can capture the TTS playback and feed it back into the operator-message channel. Since the energy gate and Whisper hallucination filters are tuned to accept clean speech, they will not filter out synthesized narration.This PR implements a lightweight, thread-safe synchronization mechanism inside the
inspect-robots-voiceplugin to mute input audio chunks captured during active TTS playback.Changes
inspect-robots-voice(version 0.5.2):_active_speakersset guarded by_speakers_lock) in_capture.pyto track active audio synthesis / playback sinks.SpeakerSink._workerin_speaker.pyto register itself to_active_speakersimmediately before chunked audio output (playback.write) and reliably discard/unregister itself when done via atry...finallyblock.MicrophoneCapture._callbackin_capture.pyto check_active_speakers. If playback is active, the block is zero-filled (muted) before entering the queue, preventing speaker echo while keeping the adaptive energy gate statistics stable.0.5.2and updated related factory test constraints.Verification & Testing
test_playback_aware_muting_callbackintests/test_capture.pyto verify that input blocks are zero-filled when a speaker registry entry is active and preserved when empty.test_speaker_active_playback_registrationintests/test_speaker.pyto verify that the sink registration correctly brackets chunked writes.CHANGELOG.mdunder### Fixed.