Skip to content

docs: close gaps found in Kapa source analytics audit #951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
markbackman merged 3 commits into from
Jun 26, 2026
Merged

docs: close gaps found in Kapa source analytics audit #951

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d989d8c
docs: close gaps found in Kapa source analytics audit
jamsea Jun 26, 2026
5156c72
Potential fix for pull request finding
jamsea Jun 26, 2026
84b6aeb
Review feedback
markbackman Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions api-reference/server/services/stt/deepgram.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ Runtime-configurable settings passed via the `settings` constructor argument usi
### Usage

```python
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

stt = DeepgramFluxSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
Expand All @@ -329,7 +329,7 @@ stt = DeepgramFluxSTTService(
#### With EagerEndOfTurn

```python
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

stt = DeepgramFluxSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
Expand All @@ -344,7 +344,7 @@ stt = DeepgramFluxSTTService(
#### Multilingual Support

```python
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService
from pipecat.transcriptions.language import Language

# Use flux-general-multi with language hints
Expand All @@ -363,7 +363,7 @@ The `keyterm`, `eot_threshold`, `eager_eot_threshold`, `eot_timeout_ms`, and `la

```python
from pipecat.frames.frames import STTUpdateSettingsFrame
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService
from pipecat.transcriptions.language import Language

# During pipeline execution, update settings without reconnecting
Expand Down
17 changes: 17 additions & 0 deletions api-reference/server/utilities/turn-detection/smart-turn-overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,23 @@ The `SmartTurnParams` class configures turn detection behavior:
of turn
</ParamField>

<Note>
**This is not the same `stop_secs` as your VAD's.** There are two:

- **VAD `stop_secs`** (default `0.2`, on `VADParams`): how long silence must
last before VAD declares the user stopped speaking. This is what
_triggers_ a Smart Turn analysis.
- **`SmartTurnParams.stop_secs`** (default `3.0`, here): a silence-based
_fallback_. If the model keeps classifying the turn as incomplete, this is
how long to wait before force-completing the turn anyway.

Lowering VAD `stop_secs` makes turn-taking more responsive; changing
`SmartTurnParams.stop_secs` only affects how long an unresolved turn waits
before the fallback fires, which is why tuning it often shows little effect
in normal conversation.

</Note>

<ParamField path="pre_speech_ms" type="float" default="0.0">
Amount of audio (in milliseconds) to include before speech is detected
</ParamField>
Expand Down
138 changes: 128 additions & 10 deletions api-reference/server/utilities/turn-management/user-turn-strategies.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,49 @@ By default, Pipecat uses a combination of VAD (Voice Activity Detection) and AI-

You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models.

<Note>
The user turn strategies API is new in Pipecat 1.x. If you're coming from
0.0.x (where turn detection was configured via `turn_analyzer` on the
transport params), see the [migration guide](/pipecat/migration/migration-1.0)
for where each setting moved.
</Note>

## How It Works

1. **Turn Start Detection**: When any start strategy triggers, the user aggregator:

- Marks the start of a user turn
- Optionally emits `UserStartedSpeakingFrame`
- Optionally emits an interruption frame (if the bot is speaking)

2. **During User Turn**: The aggregator collects transcriptions and audio frames.

3. **Turn Stop Detection**: When a stop strategy triggers, the user aggregator:

- Marks the end of the user turn
- Emits `UserStoppedSpeakingFrame`
- Pushes the aggregated user message to the LLM context

4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended. This timeout is configurable via `LLMUserAggregatorParams` (see [Configuration](#configuration) below). When the timeout fires, the [`on_user_turn_stop_timeout`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_stop_timeout) event is emitted.

### Turn timing timeline

Several timers combine to decide when a user turn ends. They fire in this order after the user stops talking:

1. **VAD `stop_secs`**: The VAD waits this long (default `0.2s`) after audio goes quiet before it reports the user stopped speaking. Configured on your `VADParams`, not here.
2. **`user_speech_timeout`** (on `SpeechTimeoutUserTurnStopStrategy`, default `0.6s`): The policy floor. After VAD silence, this is the window in which the user may resume speaking. It always runs to completion.
3. **STT finalization / `ttfs_p99_latency`** _or_ **smart-turn analysis**: A safety net for STT latency. With `SpeechTimeoutUserTurnStopStrategy`, the turn waits for the STT service to return a final transcript (short-circuited when the STT service emits a finalized transcript). With `TurnAnalyzerUserTurnStopStrategy`, the smart-turn model decides end-of-turn instead.
4. **Turn finalized**: Once the active stop strategy's conditions are met, the turn ends and the bot responds.

`user_turn_stop_timeout` (default `5.0s`, on `LLMUserAggregatorParams`) sits outside all of this as an outer watchdog: if none of the above finalizes the turn in time, it forces the turn to end.

<Note>
The extra silence some users notice (for example, "the aggregator waits an
extra 800ms") is usually steps 2 and 3 stacking: `stop_secs` plus
`user_speech_timeout` plus the STT wait. Tune each timer independently rather
than expecting one value to control total latency.
</Note>

## Configuration

User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:
Expand Down Expand Up @@ -74,6 +101,14 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
)
```

<ParamField path="vad_analyzer" type="VADAnalyzer" default="None">
The Voice Activity Detection analyzer instance used to produce VAD signals for
turn detection. In Pipecat 1.x, VAD is configured on the aggregator rather
than the transport. This is separate from `VADUserTurnStartStrategy`: the
analyzer _produces_ the VAD signals; the start strategy decides _what to do_
with them.
</ParamField>

## Start Strategies

Start strategies determine when a user's turn begins. Multiple strategies can be provided, and the first one to trigger will signal the start of a user turn.
Expand Down Expand Up @@ -317,11 +352,12 @@ strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6)
<Note>
Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the
recommended default). If you change `stop_secs`, the strategy will log a
warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark)
with your VAD settings and pass the measured TTFS P99 latency to your STT
service constructor via `ttfs_p99_latency`. The strategy will also warn if
`stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s
and may cause delayed turn detection.
warning suggesting you re-run the
[stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD
settings and pass the measured TTFS P99 latency to your STT service
constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs
>= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause
delayed turn detection.
</Note>

### TurnAnalyzerUserTurnStopStrategy
Expand Down Expand Up @@ -359,11 +395,12 @@ strategy = TurnAnalyzerUserTurnStopStrategy(
<Note>
Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the
recommended default). If you change `stop_secs`, the strategy will log a
warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark)
with your VAD settings and pass the measured TTFS P99 latency to your STT
service constructor via `ttfs_p99_latency`. The strategy will also warn if
`stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s
and may cause delayed turn detection.
warning suggesting you re-run the
[stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD
settings and pass the measured TTFS P99 latency to your STT service
constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs
>= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause
delayed turn detection.
</Note>

### ExternalUserTurnStopStrategy
Expand Down Expand Up @@ -454,6 +491,87 @@ stop = [
]
```

### Custom stop strategy

Pipecat has no built-in strategy for capping how long a single user turn can run (there is no "max speech" or "max turn duration" strategy). If you need to end a turn after a fixed amount of time, for example to stop a customer from monologuing, write your own stop strategy by subclassing `BaseUserTurnStopStrategy`.

A stop strategy inspects incoming frames in `process_frame` and calls `trigger_user_turn_stopped()` when it decides the turn is over. The example below starts a timer when the user begins speaking and forces the turn to end once the timer fires:

```python
import asyncio

from pipecat.frames.frames import Frame, VADUserStartedSpeakingFrame
from pipecat.turns.types import ProcessFrameResult
from pipecat.turns.user_stop import BaseUserTurnStopStrategy


class MaxTurnDurationStopStrategy(BaseUserTurnStopStrategy):
"""End the user turn after a fixed number of seconds.

Starts a timer when the user starts speaking. If the user is still
holding the turn when the timer fires, the turn is finalized so the
bot can respond.
"""

def __init__(self, *, max_turn_seconds: float = 10.0, **kwargs):
super().__init__(**kwargs)
self._max_turn_seconds = max_turn_seconds
self._timer_task: asyncio.Task | None = None

async def process_frame(self, frame: Frame) -> ProcessFrameResult:
if isinstance(frame, VADUserStartedSpeakingFrame) and self._timer_task is None:
self._timer_task = self.task_manager.create_task(
self._max_turn_handler(),
f"{self}::_max_turn_handler",
)
return ProcessFrameResult.CONTINUE

async def _max_turn_handler(self):
try:
await asyncio.sleep(self._max_turn_seconds)
except asyncio.CancelledError:
return
finally:
self._timer_task = None
# Time is up: end the turn so the bot responds.
await self.trigger_user_turn_stopped()

async def reset(self):
await super().reset()
await self._cancel_timer()

async def cleanup(self):
await super().cleanup()
await self._cancel_timer()

async def _cancel_timer(self):
if self._timer_task:
await self.task_manager.cancel_task(self._timer_task)
self._timer_task = None
```

Pair it with a normal detector so regular turn-end still works. The first strategy to trigger ends the turn, so the timer acts as a backstop on top of smart-turn detection:

```python
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

strategies = UserTurnStrategies(
stop=[
TurnAnalyzerUserTurnStopStrategy(turn_analyzer=LocalSmartTurnAnalyzerV3()),
MaxTurnDurationStopStrategy(max_turn_seconds=10.0),
],
)
```

<Note>
Ending the turn lets the bot respond, but it does not stop the user's audio
from being processed if they keep talking. To also ignore further user input
for a window, combine this with [User Input
Muting](/pipecat/fundamentals/user-input-muting).
</Note>

## Helper Functions

Pipecat provides helper functions to compose custom strategy lists that extend the defaults.
Expand Down
68 changes: 53 additions & 15 deletions client/concepts/choosing-a-transport.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,39 @@ The choice of transport has one important constraint: **the client transport and
## Available transports

<CardGroup cols={2}>
<Card title="SmallWebRTC" icon="link" href="/api-reference/client/js/transports/small-webrtc">
<Card
title="SmallWebRTC"
icon="link"
href="/api-reference/client/js/transports/small-webrtc"
>
Serverless peer-to-peer WebRTC. No third-party account needed.
</Card>
<Card title="Daily" icon="tower-broadcast" href="/api-reference/client/js/transports/daily">
<Card
title="Daily"
icon="tower-broadcast"
href="/api-reference/client/js/transports/daily"
>
WebRTC via Daily's global infrastructure. Recommended for production.
</Card>
<Card title="WebSocket" icon="plug" href="/api-reference/client/js/transports/websocket">
<Card
title="WebSocket"
icon="plug"
href="/api-reference/client/js/transports/websocket"
>
Direct WebSocket connection. For server-to-server setups only.
</Card>
<Card title="Gemini Live" icon="google" href="/api-reference/client/js/transports/gemini">
<Card
title="Gemini Live"
icon="google"
href="/api-reference/client/js/transports/gemini"
>
Direct connection to Google's Gemini Live API. No Pipecat server needed.
</Card>
<Card title="OpenAI WebRTC" icon="robot" href="/api-reference/client/js/transports/openai-webrtc">
<Card
title="OpenAI WebRTC"
icon="robot"
href="/api-reference/client/js/transports/openai-webrtc"
>
Direct connection to OpenAI's Realtime API. No Pipecat server needed.
</Card>
</CardGroup>
Expand Down Expand Up @@ -50,7 +70,9 @@ Once you've settled on WebRTC, the next question is how you route it. There are
A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's simpler than running your own WebRTC infrastructure and avoids a third-party dependency. If you want managed infrastructure that handles global routing, audio processing, and scaling for you, use Daily.

<Note>
For a deeper look at this tradeoff, see the Daily blog post [You don't need a WebRTC server for your voice agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/).
For a deeper look at this tradeoff, see the Daily blog post [You don't need a
WebRTC server for your voice
agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/).
</Note>

---
Expand All @@ -62,11 +84,13 @@ A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's
SmallWebRTC is the default in all Pipecat quickstart templates. No account needed, no third-party services — just a direct peer-to-peer WebRTC connection between your client and your bot.

**Use it when:**

- Developing or testing locally
- Running a self-hosted deployment
- Building embedded or edge deployments where simplicity matters

**Avoid it when:**

- Your users are geographically distributed
- You need built-in echo cancellation, noise reduction, or network resilience at scale
- You're scaling beyond a handful of concurrent sessions
Expand All @@ -89,6 +113,7 @@ await client.connect({ webrtcUrl: "http://localhost:7860/api/offer" });
Daily provides a global WebRTC network with mesh routing, built-in audio processing (echo cancellation, noise suppression, automatic gain control), and resilience to network changes. It's the recommended choice for anything user-facing in production.

**Use it when:**

- Shipping a production app to real users
- Your users are on a variety of networks, devices, or locations
- You want managed infrastructure without operating it yourself
Expand All @@ -114,11 +139,13 @@ await client.startBotAndConnect({ endpoint: "/api/start" });
The WebSocket transport connects to a WebSocket server rather than using WebRTC. It's appropriate for controlled, server-to-server scenarios where both sides are on stable networks, or for text-only bots with no audio requirements.

**Use it when:**

- Integrating telephony media streams (Twilio, Telnyx). This is the most common production WebSocket use case: the phone provider streams audio to your server over a WebSocket, which is a provider-to-server connection, not a browser client.
- Building text-only interactions (no audio)
- Connecting two servers (not a browser client)
- Network constraints make WebRTC impractical in a specific environment

**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above.
**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above. The telephony case is different: the audio comes from a phone network over the provider's media-stream WebSocket, not from a browser, so the WebRTC advantages for browser audio don't apply.

---

Expand All @@ -127,11 +154,22 @@ The WebSocket transport connects to a WebSocket server rather than using WebRTC.
These transports connect your client directly to Google's Gemini Live API or OpenAI's Realtime API, bypassing a Pipecat server entirely. They're useful for prototyping or demos when you want speech-to-speech interactions without running a backend.

<Warning>
Direct API transports expose your API key in the client. They are appropriate for development and demos, but **not for production** — use a server-side Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API keys secure.
Direct API transports expose your API key in the client. They are appropriate
for development and demos, but **not for production** — use a server-side
Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API
keys secure.
</Warning>

---

### What about LiveKit?

Pipecat ships a [LiveKit transport for the **server**](/api-reference/server/services/transport/livekit), so your bot can join a LiveKit room. There is **no** official Pipecat **client** transport for LiveKit (JS/React, React Native, or otherwise).

@jamsea jamsea Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markbackman this is a direct response to people asking the AI chatbots about Livekit. We do have an open PR here to make a transport: pipecat-ai/pipecat-client-web-transports#86

Might be worth merging in the PR and deleting this. But not sure how much of a lift that is for engineering. While we do get people asking the AI chatbot we don't have anyone on paid support asking about this so it might be lower in priority.


Because the client and server transports must be a matching pair, you cannot pair a LiveKit server transport with a Pipecat client SDK today. If you want a self-hostable, production-grade WebRTC transport that you control end to end with Pipecat clients, use `SmallWebRTCTransport`; if you want managed WebRTC infrastructure, use Daily.

---

## Swapping transports

Transports are interchangeable — the rest of your application code stays the same. The only thing that changes is the import and constructor:
Expand All @@ -153,10 +191,10 @@ The Pipecat CLI scaffolds a `config.ts` that selects the transport based on an e

## Summary

| Transport | Best for | Requires |
|---|---|---|
| SmallWebRTC | Local dev, self-hosted | Nothing |
| Daily | Production apps, global users | Daily account |
| WebSocket | Text-only, server-to-server | Custom server |
| Gemini Live | Gemini prototypes | Gemini API key |
| OpenAI WebRTC | OpenAI prototypes | OpenAI API key |
| Transport | Best for | Requires |
| ------------- | ----------------------------- | -------------- |
| SmallWebRTC | Local dev, self-hosted | Nothing |
| Daily | Production apps, global users | Daily account |
| WebSocket | Text-only, server-to-server | Custom server |
| Gemini Live | Gemini prototypes | Gemini API key |
| OpenAI WebRTC | OpenAI prototypes | OpenAI API key |
Loading
Loading