Skip to content

fix(elevenlabs): send keyterms on the realtime STT WebSocket - #2283

Merged
chenghao-mou merged 1 commit into
mainfrom
port/elevenlabs-realtime-keyterms
Aug 13, 2026
Merged

fix(elevenlabs): send keyterms on the realtime STT WebSocket#2283
chenghao-mou merged 1 commit into
mainfrom
port/elevenlabs-realtime-keyterms

Conversation

@rosetta-livekit-bot

@rosetta-livekit-bot rosetta-livekit-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Ports livekit/agents#6746 to the ElevenLabs JavaScript plugin.

Realtime Scribe v2 connections now send repeated, percent-encoded keyterms query parameters. Updating keyterms on either the STT instance or an active speech stream triggers a reconnect so the new biasing takes effect.

Adds a patch changeset for @livekit/agents-plugin-elevenlabs.

Verification

  • pnpm test plugins/elevenlabs (24 passed, 2 credential-gated integration tests skipped)
  • pnpm build (40/40 packages passed)
  • pnpm lint (40/40 packages passed; existing warnings only)
  • pnpm exec prettier --check plugins/elevenlabs/src/stt.ts plugins/elevenlabs/src/stt.test.ts .changeset/elevenlabs-realtime-keyterms.md
Source diff coverage

Authoritative source: livekit/agents#6746

  • Adapted: livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py -> plugins/elevenlabs/src/stt.ts. Ported constructor option documentation, repeated percent-encoded realtime WebSocket keyterms, STT-to-stream option forwarding, and reconnect-on-keyterm-update behavior. JavaScript encodeURIComponent replaces Python urllib.parse.quote.
  • Adapted: tests/test_plugin_elevenlabs_stt.py -> plugins/elevenlabs/src/stt.test.ts. Ported all five source tests for inclusion, query-delimiter escaping and URL round trips, omission when unspecified, STT-level forwarding, and stream-level reconnects. The target tests observe URLs through its local WebSocket test server because TypeScript private fields and methods are inaccessible.
  • Not applicable: none. Every source diff file has a target counterpart and is covered.

Ported from livekit/agents#6746

Original PR description

Problem

keyterms is accepted by elevenlabs.STT(...) but only ever sent on the batch
request. It is dropped on the realtime WebSocket path, so scribe_v2_realtime
sessions get no keyterm biasing even when the caller passes them.

ElevenLabs does support it there, as repeated keyterms query params on the connect URL:

  • Realtime STT API reference lists keyterms"List of keyterms the model is biased towards."
  • Keyterm prompting guide"also available for the realtime Speech to Text WebSocket API." Batch allows 1000 keyterms × 50 chars, realtime 50 × 20.

The docstring claims the opposite. It was accurate when written, before ElevenLabs
shipped realtime keyterms — no_verbatim, released alongside them, was wired into
_connect_ws, but keyterms was missed and the docstring never updated.

Changes

  • _connect_ws appends repeated keyterms= query params, percent-encoded.
  • SpeechStream.update_options accepts keyterms and triggers a reconnect, mirroring
    no_verbatim. STT.update_options forwards them to active streams.
  • Docstring corrected with the real per-mode limits.
  • 5 tests, following the existing enable_logging / no_verbatim patterns.

Keyterms are free-form text, unlike the other params in that URL (model ids, enums,
numbers, language codes), so they need encoding: verified against yarl.URL — the parser
aiohttp.ws_connect uses — an unescaped & splits Smith & Sons into keyterms='Smith '
plus a bogus ' Sons' param, and # truncates C# to C.
test_connect_ws_escapes_query_delimiters_in_keyterms pins this via
URL(...).query.getall("keyterms").

Verification

make check clean (ruff, mypy over 629 files). pytest tests/test_plugin_elevenlabs_stt.py — 25 passed.

Those mock the socket, so they only prove the URL is built correctly. Checked separately
against the live endpoint: one TTS-synthesized clip per term, transcribed twice over
scribe_v2_realtimeidentical audio bytes in both arms, so any difference isolates
the keyterm.

term without keyterms with keyterms
Sonic3000 Sonic 3000 Sonic3000
GPT4o GPT-4o GPT4o
K2Turbo K2 Turbo K2Turbo
nova-3 Nova 3 nova-3
scribe-v2 Scribe V2 scribe-v2

@rosetta-livekit-bot
rosetta-livekit-bot Bot requested a review from a team as a code owner August 13, 2026 08:12
@changeset-bot

changeset-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 99217fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 39 packages
Name Type
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-anthropic Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-azure Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-did Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-krisp Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-perplexity Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-protoface Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugin-soniox Patch
@livekit/agents-plugin-tavus Patch
@livekit/agents-plugin-trugen Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugins-test Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@rosetta-livekit-bot
rosetta-livekit-bot Bot requested a review from longcw August 13, 2026 08:12

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

@chenghao-mou
chenghao-mou merged commit a317f16 into main Aug 13, 2026
5 checks passed
@chenghao-mou
chenghao-mou deleted the port/elevenlabs-realtime-keyterms branch August 13, 2026 11:30
@github-actions github-actions Bot mentioned this pull request Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant