examples: switch homepage agent to Fish Audio with expressive mode - #2278
examples: switch homepage agent to Fish Audio with expressive mode#2278rosetta-livekit-bot[bot] wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 210acaa The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| export function normalizeMarkup(provider: string, text: string): string { | ||
| let out = text; | ||
| if (provider in PROVIDER_MARKUP) { | ||
| out = out.replace(EXPR_UNCLOSED_RE, '$1/>'); | ||
| } | ||
| const tags = SELF_CLOSING_TAGS[provider]; | ||
| if (!tags) { | ||
| return out; | ||
| } | ||
| const pattern = new RegExp(`<(${tags.map(escapeRegExp).join('|')})\\b([^>]*[^/])\\s*>`, 'g'); | ||
| return out.replace(pattern, '<$1$2/>'); | ||
| } |
There was a problem hiding this comment.
🟡 A leftover closing tag can be spoken aloud when the model writes a wrapping delivery tag
When the model writes a delivery tag that wraps words instead of standing alone, only its opening half is repaired (normalizeMarkup at agents/src/tts/provider_format.ts:1449-1460) and the unmatched closing half is never removed before the sentence is sent to the voice, so listeners can hear it read out.
Impact: The agent may audibly say something like "slash expression" in the middle of a sentence.
How the repair leaves an orphan closing tag on the audio path
The code explicitly anticipates models writing the wrapping form (see the comment on ATTRIBUTE_MARKUP_TAGS, agents/src/tts/provider_format.ts:1180-1192: "models do write <expression value="warm">words</expression> ... which is exactly why normalizeMarkup repairs that shape").
For input <expression value="warm">Hello there</expression> with provider fishaudio/inworld/cartesia:
normalizeMarkupmatches<(expression)\b([^>]*[^/])\s*>and rewrites the opening tag to<expression value="warm"/>. The trailing</expression>is not matched by any pattern here.convertMarkup(agents/src/tts/provider_format.ts:1463-1489) then lowers the now self-closing tag (FISHAUDIO_EXPRESSION_RE/convertExpressionTags), but neither of those patterns — norconvertExpr, which only drops stray</expr>— removes a stray native</expression>.- The result
[very warm]Hello there</expression>is whatinference/tts.tssends asinput_transcript(agents/src/inference/tts.ts:697-705, which callsconvert(normalize(...))).
Note the un-normalized path is also lossy: without step 1, FISHAUDIO_EXPRESSION_RE's >(?:.*?)</expression> alternative would swallow the wrapped words entirely. A final sweep that drops any remaining unmatched native closing tag (mirroring what convertExpr already does for </expr>) would fix both.
Prompt for agents
In agents/src/tts/provider_format.ts, normalizeMarkup() repairs a model-written wrapping tag such as `<expression value="warm">Hello there</expression>` by rewriting only the opening tag to self-closing form. The matching `</expression>` is left in the text, and nothing downstream removes it: convertExpr only drops stray `</expr>` delimiters, and the per-provider conversions (FISHAUDIO_EXPRESSION_RE, convertExpressionTags, XAI/FISHAUDIO break and emphasis passes) only match opening/self-closing forms. The orphan closing tag therefore reaches the TTS as literal text on the audio path (see inference/tts.ts, which calls markup.convert(markup.normalize(token)) before sending input_transcript) and can be spoken. Consider having normalizeMarkup also drop the now-orphaned closing tag when it converts an opening tag to self-closing, or add a final sweep in convertMarkup that removes any remaining unmatched native closing tags for the provider's tag list — the same defensive stance convertExpr already takes for `</expr>`.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Ports livekit/agents#6822 to the TypeScript homepage example.
Summary
Source diff coverage
examples/homepage/README.md->examples/src/homepage/README.md. Updated the provider and expressive-mode description; retained Krisp as the target counterpart to Python's ai-coustics isolation.examples/homepage/agent.py->examples/src/homepage/agent.ts. Ported the Fish Audio model, Marley voice ID and label,expressive: true, frontend label publication, and removal of the Inworld pronunciation hook into the target's TypeScript APIs.examples/homepage/tests/unit/test_agent_config.py->examples/src/homepage/tests/unit/agent_config.test.ts. Ported the source assertions to the existing Vitest config test.Validation
pnpm buildpassed (all 40 workspace packages)pnpm lintpassedpnpm format:checkpassedpnpm test agentspassed: 114 files, 1640 tests, 5 skippedpnpm exec vitest run examples/src/homepage/tests/unitpassed: 5 files, 9 testscue-clivoice-mode validation passed, asserting Fish Audio S2.1 Pro usage and expressive<expr>markers in framework eventspnpm --filter livekit-agents-examples testwas run twice. The isolated rerun had 105 passing tests, 2 skipped, and one failure in the pre-existingexamples/src/testing/agent_task.test.tsassertion that expects a function-call output after an agent handoff; unrelated asynchronous errors were also reported bysurvey_agent.test.tsandrun_result.test.ts.Ported from livekit/agents#6822
Original PR description
No description.