Skip to content

Commit 1e8e69c

Browse files
author
Théo Monnom
committed
examples: add expressive_agent
1 parent d636be1 commit 1e8e69c

9 files changed

Lines changed: 357 additions & 1 deletion

File tree

‎examples/expressive_agent/.dockerignore‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Python bytecode and artifacts
2+
**/__pycache__/
3+
**/*.py[cod]
4+
**/*.pyo
5+
**/*.pyd
6+
**/*.egg-info/
7+
**/dist/
8+
**/build/
9+
10+
# Virtual environments
11+
**/.venv/
12+
**/venv/
13+
14+
# Caches and test output
15+
**/.cache/
16+
**/.pytest_cache/
17+
**/.ruff_cache/
18+
**/coverage/
19+
20+
# Logs and temp files
21+
**/*.log
22+
**/*.gz
23+
**/*.tgz
24+
**/.tmp
25+
**/.cache
26+
27+
# Environment variables
28+
**/.env
29+
**/.env.*
30+
31+
# VCS, editor, OS
32+
.git
33+
.gitignore
34+
.gitattributes
35+
.github/
36+
.idea/
37+
.vscode/
38+
.DS_Store
39+
40+
# Project docs and misc
41+
README.md
42+
LICENSE
43+
44+
# Project tests
45+
test/
46+
tests/
47+
eval/
48+
evals/

‎examples/expressive_agent/Dockerfile‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# syntax=docker/dockerfile:1
2+
#
3+
# Shared Dockerfile for every example under examples/. Byte-identical
4+
# across the tree — each example's entry script is named `agent.py`,
5+
# so there's no per-example variation left.
6+
ARG PYTHON_VERSION=3.13
7+
FROM python:${PYTHON_VERSION}-slim AS base
8+
9+
ENV PYTHONUNBUFFERED=1
10+
11+
COPY --from=ghcr.io/astral-sh/uv:0.11.6 /uv /usr/local/bin/uv
12+
13+
ARG UID=10001
14+
RUN adduser \
15+
--disabled-password \
16+
--gecos "" \
17+
--home "/app" \
18+
--shell "/sbin/nologin" \
19+
--uid "${UID}" \
20+
appuser
21+
22+
RUN apt-get update && apt-get install -y \
23+
git \
24+
git-lfs \
25+
gcc \
26+
g++ \
27+
python3-dev \
28+
&& rm -rf /var/lib/apt/lists/*
29+
30+
# Enable git-lfs so git dependencies smudge LFS-tracked binaries
31+
# (e.g. silero's bundled VAD onnx) instead of leaving pointer files.
32+
# --system so the unprivileged appuser below inherits the filters.
33+
RUN git lfs install --system
34+
35+
WORKDIR /app
36+
USER appuser
37+
38+
# The example directory is the whole build context, so pyproject.toml has to
39+
# resolve on its own here — no workspace, no lockfile. A deploy from this repo
40+
# repoints the livekit-* dependencies at the ref being deployed first:
41+
# python scripts/pin_example_to_ref.py examples/<name> --ref <git-ref>
42+
COPY pyproject.toml ./
43+
RUN uv sync --no-cache
44+
ENV PATH="/app/.venv/bin:${PATH}"
45+
46+
# Pre-download model weights plugins ship (silero VAD, turn-detector, …)
47+
# so the container is ready to take traffic without a cold-download stall.
48+
RUN python -m livekit.agents download-files
49+
50+
COPY . .
51+
52+
CMD ["python", "agent.py", "start"]

‎examples/expressive_agent/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Expressive agent
2+
3+
A free-form voice agent that demonstrates [Expressive Mode](https://docs.livekit.io/agents/build/expressive/).
4+
There is no task and no tool: you talk to it like a friend, and it matches your
5+
register. Tell it good news and it gets excited; tell it something went wrong
6+
and it drops the energy.
7+
8+
Expressive Mode is the single `expressive=True` flag on `AgentSession`. With it
9+
enabled the framework injects the TTS provider's markup guide into the LLM
10+
prompt, so the model emits inline delivery tags (emotion, pacing, non-verbal
11+
sounds) that the TTS renders and the transcript never shows.
12+
13+
## Architecture
14+
15+
- `agent.py` is the composition root: session setup and the server entrypoint.
16+
- `prompt.md` holds the persona only. It steers *what* the agent says, and
17+
expressive mode owns *how* it sounds, so the two never restate each other.
18+
- `voices.py` lists the expressive-capable LiveKit Inference voices the demo can
19+
switch between.
20+
21+
The pipeline uses LiveKit Inference with Gemma 4 31B, Deepgram Nova-3, Fish
22+
Audio S2.1 Pro, and the LiveKit turn detector.
23+
24+
## Run locally
25+
26+
Provide LiveKit Cloud credentials in `../.env` or the environment, then:
27+
28+
```bash
29+
uv sync --all-extras --dev # from the repository root
30+
uv run agent.py console
31+
```
32+
33+
Use `uv run agent.py dev` to connect the agent to LiveKit Cloud for a frontend
34+
session.
35+
36+
## Configuration
37+
38+
The agent reads its dispatch metadata, so a frontend can pick the pipeline at
39+
connect time without a redeploy:
40+
41+
```json
42+
{ "expressive": true, "tts": "fishaudio" }
43+
```
44+
45+
- `expressive` (default `true`) toggles Expressive Mode.
46+
- `tts` selects a voice from `voices.py`: `fishaudio`, `inworld`, or `cartesia`.
47+
48+
Both values are echoed back as participant attributes (`expressive`,
49+
`tts_provider`, `tts_label`) so the frontend can display the active pipeline.
50+
51+
Every voice in `voices.py` publishes the `lk.expression` attribute, so switching
52+
provider never leaves a frontend mood indicator dark. See that file for what a
53+
provider needs to qualify.
54+
55+
## Trying it with and without expressive
56+
57+
The comparison is the point of the demo. Run it once with `expressive=True` and
58+
once with `expressive=False`, and say the same thing to each. The words come out
59+
much the same; the delivery does not.
60+
61+
Expressive Mode requires a `livekit.agents.inference.TTS` model that declares a
62+
markup dialect. Fish Audio, Inworld TTS 2, and Cartesia Sonic 3 qualify;
63+
providers without a dialect synthesize normally and the flag stays inert.

‎examples/expressive_agent/agent.py‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
import json
2+
import logging
3+
from pathlib import Path
4+
5+
import voices
6+
from dotenv import load_dotenv
7+
8+
from livekit.agents import (
9+
Agent,
10+
AgentServer,
11+
AgentSession,
12+
JobContext,
13+
TurnHandlingOptions,
14+
cli,
15+
inference,
16+
)
17+
18+
logger = logging.getLogger("expressive-agent")
19+
20+
load_dotenv()
21+
22+
AGENT_NAME = "expressive_agent"
23+
INSTRUCTIONS = (Path(__file__).parent / "prompt.md").read_text()
24+
25+
GREETING = (
26+
"Open the call the way you'd answer the phone to someone you know well. "
27+
"Short and warm, and leave them room to say what's going on."
28+
)
29+
30+
31+
class Friend(Agent):
32+
def __init__(self) -> None:
33+
super().__init__(instructions=INSTRUCTIONS)
34+
35+
async def on_enter(self) -> None:
36+
await self.session.generate_reply(instructions=GREETING)
37+
38+
39+
server = AgentServer()
40+
41+
42+
@server.rtc_session(agent_name=AGENT_NAME)
43+
async def expressive_agent(ctx: JobContext) -> None:
44+
ctx.log_context_fields = {"room": ctx.room.name}
45+
46+
meta = json.loads(ctx.job.metadata) if ctx.job.metadata else {}
47+
expressive = bool(meta.get("expressive", True))
48+
voice = voices.resolve(meta.get("tts"))
49+
logger.info("starting session", extra={"expressive": expressive, "voice": voice.label})
50+
51+
session = AgentSession(
52+
stt=inference.STT("deepgram/nova-3", language="multi"),
53+
llm=inference.LLM("google/gemma-4-31b-it"),
54+
tts=inference.TTS(voice.model, voice=voice.voice),
55+
turn_handling=TurnHandlingOptions(turn_detection=inference.TurnDetector()),
56+
expressive=expressive,
57+
preemptive_generation=True,
58+
)
59+
60+
await session.start(agent=Friend(), room=ctx.room)
61+
await ctx.connect()
62+
63+
# the frontend renders the active config, which only the agent knows once
64+
# the dispatch metadata has been resolved
65+
await ctx.room.local_participant.set_attributes(
66+
{
67+
"expressive": "true" if expressive else "false",
68+
"tts_provider": voice.provider,
69+
"tts_label": voice.label,
70+
}
71+
)
72+
73+
74+
if __name__ == "__main__":
75+
cli.run_app(server)

‎examples/expressive_agent/prompt.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
You are the user's closest friend, catching up over a call. There is no task
2+
here, no ticket to close, no form to fill. You are just talking.
3+
4+
Expressive Mode injects the delivery guide separately, so this prompt covers
5+
only who you are and what you say. Tone and pacing rules don't belong here.
6+
7+
# Output rules
8+
9+
- One or two sentences. Three is already too many.
10+
- Plain prose. No markdown, lists, bullets, headers, or emojis.
11+
- Spell out numbers, money, and dates as you would say them out loud.
12+
- Never use input vocabulary like "enter" or "fill in". They are speaking, not typing.
13+
14+
# How you talk
15+
16+
- Speak naturally, not from a customer-service script. You are not assisting
17+
anyone, you are talking with them.
18+
- Don't open two consecutive turns with the same word.
19+
- React before you respond. If they tell you something big, the reaction comes
20+
first and the follow-up question comes second.
21+
- Ask about one thing at a time, the way a friend would, not the way a survey does.
22+
- Trust their memory. They heard what you said five seconds ago, so don't restate it.
23+
- If they interrupted you, don't restart the sentence. What they said is the
24+
new subject.
25+
- When they are venting, stay on their side. Advice they didn't ask for is
26+
worth less than agreeing that something sucks.
27+
- Don't reach for a silver lining they didn't ask for, and don't rush to fix
28+
what they only wanted to say out loud.
29+
- Never explain or narrate your own tone.
30+
31+
# Guardrails
32+
33+
- You have no name unless they give you one, and you never introduce yourself
34+
by one.
35+
- You are a friend, not a therapist or a doctor. If they raise something that
36+
needs real help, say plainly that you are worried and that this is worth
37+
talking to someone about. Don't lecture, and don't pretend to be qualified.

‎examples/expressive_agent/pyproject.toml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[project]
2+
name = "livekit-example-expressive-agent"
3+
version = "0"
4+
requires-python = ">=3.10"
5+
dependencies = [
6+
"livekit-agents>=1.6",
7+
"python-dotenv>=1.0.0",
8+
]
9+
10+
[tool.uv]
11+
# an example is a script collection, not an installable distribution
12+
package = false
13+
# No [tool.uv.sources] here: the workspace root repoints the livekit-* deps at
14+
# the in-repo copies for members, and staying free of workspace-only references
15+
# is what lets a copy of this directory resolve on its own, outside the repo.

‎examples/expressive_agent/voices.py‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
"""The expressive-capable TTS voices this demo can switch between.
2+
3+
Every voice here publishes ``lk.expression``, so a frontend always has a mood to
4+
render. Two things are required for that, and both are why this list is short:
5+
6+
- a LiveKit Inference TTS whose model declares a markup dialect. deepgram and
7+
rime have none, so they synthesize fine but render no tags.
8+
- an expression or emotion tag in that dialect, which is what becomes the
9+
attribute. xai steers delivery through prosody and sound tags only, so its
10+
speech is expressive but nothing reaches the frontend.
11+
"""
12+
13+
from dataclasses import dataclass
14+
15+
16+
@dataclass(frozen=True, slots=True)
17+
class Voice:
18+
provider: str
19+
model: str
20+
voice: str
21+
label: str
22+
23+
24+
VOICES: dict[str, Voice] = {
25+
"fishaudio": Voice(
26+
provider="fishaudio",
27+
model="fishaudio/s2.1-pro",
28+
voice="9a9cf47702da476aa4629e2506d4a857",
29+
label="Fish Audio S2.1 Pro (Hannah)",
30+
),
31+
"inworld": Voice(
32+
provider="inworld",
33+
model="inworld/inworld-tts-2",
34+
voice="Ashley",
35+
label="Inworld TTS 2 (Ashley)",
36+
),
37+
"cartesia": Voice(
38+
provider="cartesia",
39+
model="cartesia/sonic-3",
40+
voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
41+
label="Cartesia Sonic 3 (Jacqueline)",
42+
),
43+
}
44+
45+
DEFAULT_VOICE = "fishaudio"
46+
47+
48+
def resolve(provider: str | None) -> Voice:
49+
"""Pick a voice by provider name, falling back to the default."""
50+
return VOICES.get(provider or "", VOICES[DEFAULT_VOICE])

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ livekit-plugins-phonic = { workspace = true }
7777
livekit-plugins-did = { workspace = true }
7878

7979
[tool.uv.workspace]
80-
members = ["livekit-plugins/*", "livekit-agents", "examples/avatar", "examples/drive-thru", "examples/frontdesk", "examples/healthcare", "examples/homepage", "examples/hotel_receptionist", "examples/inference", "examples/survey", "examples/voice_agents"]
80+
members = ["livekit-plugins/*", "livekit-agents", "examples/avatar", "examples/drive-thru", "examples/expressive_agent", "examples/frontdesk", "examples/healthcare", "examples/homepage", "examples/hotel_receptionist", "examples/inference", "examples/survey", "examples/voice_agents"]
8181
exclude = [
8282
"livekit-plugins/community",
8383
"livekit-plugins/livekit-blockguard",

‎uv.lock‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)