Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ SUPPORTED STT MODELS:
AssemblyAI offers Universal-3 Pro Streaming with 6 languages, Universal-Streaming for English only, and Universal-Streaming-Multilingual with 6 languages. Cartesia offers Ink Whisper supporting 100 languages. Deepgram offers Flux for English only, Nova-3 for 9 languages, Nova-3 Medical for English, Nova-2 for 33 languages, Nova-2 Medical for English, Nova-2 Conversational AI for English, and Nova-2 Phonecall for English. ElevenLabs offers Scribe V2 Realtime supporting 41 languages.

SUPPORTED TTS MODELS:
Cartesia offers sonic-3 with over 40 languages, sonic-2 and sonic-turbo with 15 languages each, and the original sonic model. Deepgram offers aura-2 for English and Spanish. ElevenLabs offers eleven_flash_v2 for English, eleven_flash_v2_5 for over 30 languages, eleven_turbo_v2 for English, eleven_turbo_v2_5 for over 30 languages, and eleven_multilingual_v2 for 28 languages. Inworld offers inworld-tts-1.5-max and inworld-tts-1.5-mini for 13 languages, and inworld-tts-1-max and inworld-tts-1 for 12 languages. Rime offers arcana for 9 languages and mistv2 for 4 languages.
Cartesia offers sonic-3 with over 40 languages, sonic-2 and sonic-turbo with 15 languages each, and the original sonic model. Deepgram offers aura-2 for English and Spanish. ElevenLabs offers eleven_flash_v2 for English, eleven_flash_v2_5 for over 30 languages, eleven_turbo_v2 for English, eleven_turbo_v2_5 for over 30 languages, and eleven_multilingual_v2 for 28 languages. Inworld offers inworld-tts-1.5-max and inworld-tts-1.5-mini for 13 languages, and inworld-tts-1-max and inworld-tts-1 for 12 languages. Rime offers coda for 8 languages and mistv2 for 4 languages.

COMPARISON TO PLUGINS:
The alternative to LiveKit Inference is using open source plugins. Plugins connect directly to each model provider's API. You need your own account and API key for each provider. Plugins are installed as optional dependencies, for example uv add livekit-agents with openai in Python or pnpm add @livekit/agents-plugin-openai in Node.js. Plugins give you access to a wider range of providers and some provider-specific features not yet available through Inference, such as the OpenAI Realtime API for speech-to-speech models. You can mix and match LiveKit Inference and plugins in the same agent. For example, you could use inference.STT and inference.TTS for speech models while using a plugin for your LLM, or vice versa.
Expand Down
1 change: 0 additions & 1 deletion examples/inference/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ async def entrypoint(ctx: JobContext) -> None:
llm=inference.LLM(model=DEFAULT_LLM),
tts=inference.TTS(
model=DEFAULT_TTS,
voice="Sarah",
extra_kwargs={"delivery_mode": "CREATIVE"},
),
# Flip user_state to "away" after 10s of mutual silence so we can
Expand Down
2 changes: 1 addition & 1 deletion examples/playground.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ examples:
- { value: inworld/inworld-tts-1.5-mini, label: Inworld TTS 1.5 Mini }
- { value: inworld/inworld-tts-1.5-max, label: Inworld TTS 1.5 Max }
- { value: inworld/inworld-tts-2, label: Inworld TTS 2 }
- { value: rime/arcana, label: Rime Arcana }
- { value: rime/coda, label: Rime Coda }
- { value: rime/mist, label: Rime Mist }
- { value: rime/mistv2, label: Rime Mist v2 }
- { value: rime/mistv3, label: Rime Mist v3 }
Expand Down
2 changes: 1 addition & 1 deletion examples/telephony/bank-ivr/ivr_navigator_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ async def dtmf_session(ctx: JobContext) -> None:
session: AgentSession = AgentSession(
llm=inference.LLM("openai/gpt-4.1"),
stt=inference.STT("deepgram/nova-3"),
tts=inference.TTS("rime/arcana"),
tts=inference.TTS("rime/coda", voice="astra"),
# This flag does two things:
# 1. Helps agent avoid getting stuck listening to repeating IVR loops by actively responding when a loop is detected.
# 2. Automatically gives the agent the `send_dtmf_events` tool to allow it to dial DTMF digits.
Expand Down
2 changes: 1 addition & 1 deletion examples/voice_agents/otel_trace.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def __init__(self) -> None:
tts=FallbackTTSAdapter(
tts=[
inference.TTS("cartesia"),
inference.TTS("rime/arcana"),
inference.TTS("rime/coda", voice="astra"),
]
),
tools=[lookup_weather],
Expand Down
5 changes: 2 additions & 3 deletions livekit-agents/livekit/agents/inference/tts.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@
]
RimeModels = Literal[
"rime",
"rime/arcana",
"rime/coda",
"rime/mistv2",
"rime/mistv3",
Expand Down Expand Up @@ -116,7 +115,7 @@ class FallbackModel(TypedDict):
"""

model: str
"""Model name (e.g. "cartesia/sonic", "elevenlabs/eleven_flash_v2", "rime/arcana")."""
"""Model name (e.g. "cartesia/sonic", "elevenlabs/eleven_flash_v2", "rime/coda")."""

voice: str
"""Voice to use for the model."""
Expand Down Expand Up @@ -187,7 +186,7 @@ class ElevenlabsOptions(TypedDict, total=False):


class RimeOptions(TypedDict, total=False):
"""Mistv2-specific parameters. Arcana has no extra WS JSON query params.
"""Rime-specific WebSocket JSON query parameters.
See: https://docs.rime.ai/api-reference/endpoint/websockets-json
"""

Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
from typing import Literal

TTSModels = Literal["mistv2", "mistv3", "arcana", "coda"]

# https://docs.rime.ai/api-reference/voices
ArcanaVoices = Literal[
"luna", "celeste", "orion", "ursa", "astra", "esther", "estelle", "andromeda"
]
TTSModels = Literal["mistv2", "mistv3", "coda"]

DefaultMistVoice = "cove"
DefaultCodaVoice = "lyra"
127 changes: 48 additions & 79 deletions livekit-plugins/livekit-plugins-rime/livekit/plugins/rime/tts.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,10 +44,9 @@

from .langs import TTSLangs
from .log import logger
from .models import ArcanaVoices, DefaultCodaVoice, DefaultMistVoice, TTSModels
from .models import DefaultCodaVoice, DefaultMistVoice, TTSModels

# arcana can take as long as 80% of the total duration of the audio it's synthesizing.
ARCANA_MODEL_TIMEOUT = 60 * 4
CODA_MODEL_TIMEOUT = 60 * 4
MIST_MODEL_TIMEOUT = 30
RIME_BASE_URL = "https://users.rime.ai/v1/rime-tts"
RIME_WS_BASE_URL = "wss://users-ws.rime.ai"
Expand All @@ -58,13 +57,12 @@
class _TTSOptions:
model: TTSModels | str
speaker: str
arcana_options: _ArcanaOptions | None = None
coda_options: _CodaOptions | None = None
mist_options: _MistOptions | None = None


@dataclass
class _ArcanaOptions:
class _CodaOptions:
repetition_penalty: NotGivenOr[float] = NOT_GIVEN
temperature: NotGivenOr[float] = NOT_GIVEN
top_p: NotGivenOr[float] = NOT_GIVEN
Expand All @@ -75,15 +73,6 @@ class _ArcanaOptions:
time_scale_factor: NotGivenOr[float] = NOT_GIVEN


@dataclass
class _CodaOptions:
max_tokens: NotGivenOr[int] = NOT_GIVEN
lang: NotGivenOr[TTSLangs | str] = NOT_GIVEN
sample_rate: NotGivenOr[int] = NOT_GIVEN
speed_alpha: NotGivenOr[float] = NOT_GIVEN
time_scale_factor: NotGivenOr[float] = NOT_GIVEN


@dataclass
class _MistOptions:
lang: NotGivenOr[TTSLangs | str] = NOT_GIVEN
Expand All @@ -99,35 +88,30 @@ def _is_mist_model(model: TTSModels | str) -> bool:
return "mist" in model


def _warn_if_arcana(model: NotGivenOr[TTSModels | str]) -> None:
if is_given(model) and model == "arcana":
logger.warning('Rime Arcana is no longer supported. Use model="coda" instead.')


def _timeout_for_model(model: TTSModels | str) -> int:
if model == "arcana" or model == "coda":
return ARCANA_MODEL_TIMEOUT
if model == "coda":
return CODA_MODEL_TIMEOUT
return MIST_MODEL_TIMEOUT
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.


def _model_params(opts: _TTSOptions) -> dict[str, object]:
"""Per-model option fields shared between the HTTP body and the WS query string."""
params: dict[str, object] = {}
if opts.model == "arcana" and opts.arcana_options is not None:
ao = opts.arcana_options
if is_given(ao.lang):
params["lang"] = ao.lang
if is_given(ao.repetition_penalty):
params["repetition_penalty"] = ao.repetition_penalty
if is_given(ao.temperature):
params["temperature"] = ao.temperature
if is_given(ao.top_p):
params["top_p"] = ao.top_p
if is_given(ao.max_tokens):
params["max_tokens"] = ao.max_tokens
if is_given(ao.speed_alpha):
params["speedAlpha"] = ao.speed_alpha
if is_given(ao.time_scale_factor):
params["timeScaleFactor"] = ao.time_scale_factor
elif opts.model == "coda" and opts.coda_options is not None:
if opts.model == "coda" and opts.coda_options is not None:
co = opts.coda_options
if is_given(co.lang):
params["lang"] = co.lang
if is_given(co.repetition_penalty):
params["repetition_penalty"] = co.repetition_penalty
if is_given(co.temperature):
params["temperature"] = co.temperature
if is_given(co.top_p):
params["top_p"] = co.top_p
if is_given(co.max_tokens):
params["max_tokens"] = co.max_tokens
if is_given(co.speed_alpha):
Expand Down Expand Up @@ -155,7 +139,7 @@ def _check_time_scale_factor_supported(
) -> None:
if is_given(time_scale_factor) and model == "mistv2":
raise ValueError(
"time_scale_factor is not supported by the mistv2 model; use arcana, mistv3, or coda."
"time_scale_factor is not supported by the mistv2 model; use mistv3 or coda."
)


Expand All @@ -164,15 +148,15 @@ def __init__(
self,
*,
base_url: NotGivenOr[str] = NOT_GIVEN,
model: TTSModels | str = "arcana",
speaker: NotGivenOr[ArcanaVoices | str] = NOT_GIVEN,
model: NotGivenOr[TTSModels | str] = NOT_GIVEN,
speaker: NotGivenOr[str] = NOT_GIVEN,
lang: TTSLangs | str = "eng",
# Arcana options
# Coda options
repetition_penalty: NotGivenOr[float] = NOT_GIVEN,
temperature: NotGivenOr[float] = NOT_GIVEN,
top_p: NotGivenOr[float] = NOT_GIVEN,
max_tokens: NotGivenOr[int] = NOT_GIVEN,
# Shared by arcana, mistv3, and coda (HTTP only; use speed_alpha on WebSocket)
# Shared by mistv3 and coda (HTTP only; use speed_alpha on WebSocket)
time_scale_factor: NotGivenOr[float] = NOT_GIVEN,
# Supported by all models; the only speed param that works over WebSocket
speed_alpha: NotGivenOr[float] = NOT_GIVEN,
Expand Down Expand Up @@ -208,22 +192,32 @@ def __init__(
"Rime API key is required, either as argument or set RIME_API_KEY environmental variable" # noqa: E501
)

_check_time_scale_factor_supported(model, time_scale_factor)
_warn_if_arcana(model)
if is_given(model):
resolved_model = model
model_is_explicit = True
else:
resolved_model = "coda"
model_is_explicit = False

_check_time_scale_factor_supported(resolved_model, time_scale_factor)

if not is_given(speaker):
if _is_mist_model(model):
if not model_is_explicit:
speaker = "astra"
elif _is_mist_model(resolved_model):
speaker = DefaultMistVoice
elif model == "coda":
elif resolved_model == "coda":
speaker = DefaultCodaVoice
else:
speaker = "astra"
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

self._opts = _TTSOptions(
model=model,
model=resolved_model,
speaker=speaker,
)
if model == "arcana":
self._opts.arcana_options = _ArcanaOptions(
if resolved_model == "coda":
self._opts.coda_options = _CodaOptions(
repetition_penalty=repetition_penalty,
temperature=temperature,
top_p=top_p,
Expand All @@ -233,15 +227,7 @@ def __init__(
speed_alpha=speed_alpha,
time_scale_factor=time_scale_factor,
)
elif model == "coda":
self._opts.coda_options = _CodaOptions(
max_tokens=max_tokens,
lang=lang,
sample_rate=sample_rate,
speed_alpha=speed_alpha,
time_scale_factor=time_scale_factor,
)
elif _is_mist_model(model):
elif _is_mist_model(resolved_model):
self._opts.mist_options = _MistOptions(
lang=lang,
sample_rate=sample_rate,
Expand All @@ -256,7 +242,7 @@ def __init__(
self._use_websocket = use_websocket
self._segment = segment if is_given(segment) else "bySentence"

self._total_timeout = _timeout_for_model(model)
self._total_timeout = _timeout_for_model(resolved_model)

self._streams: weakref.WeakSet[SynthesizeStream] = weakref.WeakSet()
self._sentence_tokenizer = (
Expand Down Expand Up @@ -354,13 +340,12 @@ def update_options(
model: NotGivenOr[TTSModels | str] = NOT_GIVEN,
speaker: NotGivenOr[str] = NOT_GIVEN,
lang: NotGivenOr[TTSLangs | str] = NOT_GIVEN,
# Arcana parameters
# Coda parameters
repetition_penalty: NotGivenOr[float] = NOT_GIVEN,
temperature: NotGivenOr[float] = NOT_GIVEN,
top_p: NotGivenOr[float] = NOT_GIVEN,
max_tokens: NotGivenOr[int] = NOT_GIVEN,
sample_rate: NotGivenOr[int] = NOT_GIVEN,
# Coda parameters
time_scale_factor: NotGivenOr[float] = NOT_GIVEN,
# Mistv2 parameters
speed_alpha: NotGivenOr[float] = NOT_GIVEN,
Expand All @@ -369,6 +354,7 @@ def update_options(
phonemize_between_brackets: NotGivenOr[bool] = NOT_GIVEN,
base_url: NotGivenOr[str] = NOT_GIVEN,
) -> None:
_warn_if_arcana(model)
effective_model = model if is_given(model) else self._opts.model
_check_time_scale_factor_supported(effective_model, time_scale_factor)

Expand All @@ -380,35 +366,21 @@ def update_options(
self._opts.model = model
self._total_timeout = _timeout_for_model(model)

if model == "arcana" and self._opts.arcana_options is None:
self._opts.arcana_options = _ArcanaOptions()
elif model == "coda" and self._opts.coda_options is None:
if model == "coda" and self._opts.coda_options is None:
self._opts.coda_options = _CodaOptions()
elif _is_mist_model(model) and self._opts.mist_options is None:
self._opts.mist_options = _MistOptions()

if is_given(speaker):
self._opts.speaker = speaker

if self._opts.model == "arcana" and self._opts.arcana_options is not None:
if self._opts.model == "coda" and self._opts.coda_options is not None:
if is_given(repetition_penalty):
self._opts.arcana_options.repetition_penalty = repetition_penalty
self._opts.coda_options.repetition_penalty = repetition_penalty
if is_given(temperature):
self._opts.arcana_options.temperature = temperature
self._opts.coda_options.temperature = temperature
if is_given(top_p):
self._opts.arcana_options.top_p = top_p
if is_given(max_tokens):
self._opts.arcana_options.max_tokens = max_tokens
if is_given(lang):
self._opts.arcana_options.lang = lang
if is_given(sample_rate):
self._opts.arcana_options.sample_rate = sample_rate
if is_given(speed_alpha):
self._opts.arcana_options.speed_alpha = speed_alpha
if is_given(time_scale_factor):
self._opts.arcana_options.time_scale_factor = time_scale_factor

elif self._opts.model == "coda" and self._opts.coda_options is not None:
self._opts.coda_options.top_p = top_p
if is_given(max_tokens):
self._opts.coda_options.max_tokens = max_tokens
if is_given(lang):
Expand Down Expand Up @@ -456,10 +428,7 @@ async def _run(self, output_emitter: tts.AudioEmitter) -> None:
**_model_params(self._opts),
}
format = "audio/pcm"
if self._opts.model == "arcana" and self._opts.arcana_options is not None:
if is_given(self._opts.arcana_options.sample_rate):
payload["samplingRate"] = self._opts.arcana_options.sample_rate
elif self._opts.model == "coda" and self._opts.coda_options is not None:
if self._opts.model == "coda" and self._opts.coda_options is not None:
if is_given(self._opts.coda_options.sample_rate):
payload["samplingRate"] = self._opts.coda_options.sample_rate
elif _is_mist_model(self._opts.model) and self._opts.mist_options is not None:
Expand Down
Loading