Harden NVIDIA Speech STT and TTS support - #6706
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| start_history: int | None = None | ||
| start_threshold: float | None = None | ||
| stop_history: int | None = None | ||
| stop_threshold: float | None = None | ||
| stop_history_eou: int | None = None | ||
| stop_threshold_eou: float | None = None | ||
|
|
||
|
|
||
| @dataclass |
There was a problem hiding this comment.
🟡 New public configuration class ships without any documentation
The newly exported configuration object is added (EndpointingConfig at livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/stt.py:49-57) with no explanation of what any of its settings do, so users of the published API docs get an undocumented knob.
Impact: People adopting the NVIDIA plugin cannot tell from the generated documentation what the new tuning options mean or what units they use.
Repository rule requiring documentation for new public classes
CONTRIBUTING.md states: "If writing new methods/enums/classes, document them. This project uses pdoc3 for automatic API documentation generation, and every new addition has to be properly documented."
EndpointingConfig is exported from the package (livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/__init__.py:44-56, added to __all__), so pdoc will render it, but neither the class nor any of its seven fields carry a docstring. The same rule applies to the other new public additions in this PR — ChunkedStream (livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/tts.py:214) and the many new STT/TTS constructor keyword arguments (inference_mode, endpointing, options, boosted_lm_words, quality, audio_prompt_file, ...), which are also undocumented (no Google-style docstrings as required by AGENTS.md).
Was this helpful? React with 👍 or 👎 to provide feedback.
|
|
| status_codes = { | ||
| grpc.StatusCode.INVALID_ARGUMENT: 400, | ||
| grpc.StatusCode.NOT_FOUND: 404, | ||
| grpc.StatusCode.ALREADY_EXISTS: 409, | ||
| grpc.StatusCode.PERMISSION_DENIED: 403, | ||
| grpc.StatusCode.RESOURCE_EXHAUSTED: 429, | ||
| grpc.StatusCode.FAILED_PRECONDITION: 400, | ||
| grpc.StatusCode.ABORTED: 409, | ||
| grpc.StatusCode.OUT_OF_RANGE: 400, | ||
| grpc.StatusCode.UNIMPLEMENTED: 501, | ||
| grpc.StatusCode.INTERNAL: 500, | ||
| grpc.StatusCode.UNAVAILABLE: 503, | ||
| grpc.StatusCode.DATA_LOSS: 500, | ||
| grpc.StatusCode.UNAUTHENTICATED: 401, | ||
| } | ||
| return APIStatusError( | ||
| f"{operation} failed: {details}", | ||
| status_code=status_codes.get(code, -1), | ||
| retryable=code | ||
| in { | ||
| grpc.StatusCode.UNKNOWN, | ||
| grpc.StatusCode.RESOURCE_EXHAUSTED, | ||
| grpc.StatusCode.ABORTED, | ||
| grpc.StatusCode.INTERNAL, | ||
| grpc.StatusCode.UNAVAILABLE, | ||
| }, | ||
| ) | ||
|
|
||
| return stt.SpeechData( | ||
| language=LanguageCode(self._language), | ||
| start_time=start_time, | ||
| end_time=end_time, | ||
| confidence=confidence, | ||
| text=transcript, | ||
| speaker_id=speaker_id, | ||
| words=[ | ||
| TimedString( | ||
| text=getattr(word, "word", ""), | ||
| start_time=getattr(word, "start_time", 0) + self.start_time_offset, | ||
| end_time=getattr(word, "end_time", 0) + self.start_time_offset, | ||
| ) | ||
| for word in words | ||
| ] | ||
| if words | ||
| else None, | ||
| if isinstance(error, (TypeError, ValueError)): |
There was a problem hiding this comment.
🟡 Temporary NVIDIA speech-recognition conflicts are never retried despite being marked as retryable
Transient recognition failures are labelled as a conflict (status_code=409 at livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/stt.py:585) even though the same code explicitly lists them as worth retrying, so the retry is silently discarded and the request fails on the first attempt.
Impact: A recoverable, momentary NVIDIA speech recognition hiccup aborts the transcription instead of being retried, dropping the user's utterance.
How the 409 mapping cancels the intended retryable flag
_to_stt_api_error builds an APIStatusError with status_code=status_codes.get(code, -1) and retryable=code in {UNKNOWN, RESOURCE_EXHAUSTED, ABORTED, INTERNAL, UNAVAILABLE} (livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/stt.py:593-604).
For grpc.StatusCode.ABORTED the map yields 409. But APIStatusError.__init__ (livekit-agents/livekit/agents/_exceptions.py:72-79) hard-overrides the caller's value:
if 400 <= status_code < 500 and status_code not in (408, 429, 499):
retryable = FalseSo ABORTED always comes back with retryable=False. This matters more now that this PR made both retry paths honour e.retryable:
STT.recognizeatlivekit-agents/livekit/agents/stt/stt.py:241(if conn_options.max_retry == 0 or not e.retryable)RecognizeStream._main_taskatlivekit-agents/livekit/agents/stt/stt.py:471- the plugin's own segment retry loop at
livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/stt.py:407(if not e.retryable: raise)
All three now bail out immediately on ABORTED, which contradicts the explicit intent expressed in the retryable set. RESOURCE_EXHAUSTED (429) and UNKNOWN/INTERNAL/UNAVAILABLE (-1/500/503) are unaffected; ABORTED is the only entry whose flag is silently inverted.
Prompt for agents
In livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/stt.py, the helper _to_stt_api_error maps grpc.StatusCode.ABORTED to HTTP status 409 while simultaneously including ABORTED in the set of retryable gRPC codes. However, APIStatusError.__init__ in livekit-agents/livekit/agents/_exceptions.py unconditionally forces retryable=False for any 4xx status code other than 408, 429 and 499. The result is that ABORTED errors are always non-retryable, silently contradicting the explicit retryable set, and now that this PR made STT.recognize, RecognizeStream._main_task and the plugin's own _finish_recognition_segment loop all honour e.retryable, such transient aborts terminate recognition on the first attempt.
Decide which behaviour is correct. If ABORTED should be retried, avoid mapping it to a 4xx status that the base class treats as terminal (for example map it to 503/500, or use a status the base class permits such as 409 -> not applicable, or construct the error via a type that does not apply the 4xx override). If ABORTED should not be retried, remove it from the retryable set so the code reads consistently. Apply the same reasoning check to the other entries in status_codes to confirm none of them have their retryable flag silently inverted.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Extend the NVIDIA plugin with production-ready streaming and offline speech support, configurable recognition behavior, improved error handling, and compatibility across supported NVIDIA Riva client versions.
Changes:
nvidia-riva-client2.16 and 2.26.Testing
uv run --locked pytest tests/test_plugin_nvidia.py tests/test_stt_base.py --unit -quv run --locked ruff check --output-format=github .uv run --locked ruff format --check .nvidia-riva-client2.16 and 2.26.