Skip to content

docs: close gaps found in Kapa source analytics audit #951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
markbackman merged 3 commits into from
Jun 26, 2026
Merged

docs: close gaps found in Kapa source analytics audit #951

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d989d8c
docs: close gaps found in Kapa source analytics audit
jamsea Jun 26, 2026
5156c72
Potential fix for pull request finding
jamsea Jun 26, 2026
84b6aeb
Review feedback
markbackman Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions api-reference/server/services/stt/deepgram.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ Runtime-configurable settings passed via the `settings` constructor argument usi
### Usage

```python
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

stt = DeepgramFluxSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
Expand All @@ -329,7 +329,7 @@ stt = DeepgramFluxSTTService(
#### With EagerEndOfTurn

```python
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

stt = DeepgramFluxSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
Expand All @@ -344,7 +344,7 @@ stt = DeepgramFluxSTTService(
#### Multilingual Support

```python
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService
from pipecat.transcriptions.language import Language

# Use flux-general-multi with language hints
Expand All @@ -363,7 +363,7 @@ The `keyterm`, `eot_threshold`, `eager_eot_threshold`, `eot_timeout_ms`, and `la

```python
from pipecat.frames.frames import STTUpdateSettingsFrame
from pipecat.services.deepgram.flux import DeepgramFluxSTTService
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService
from pipecat.transcriptions.language import Language

# During pipeline execution, update settings without reconnecting
Expand Down
9 changes: 7 additions & 2 deletions api-reference/server/utilities/turn-detection/smart-turn-overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,13 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
The `SmartTurnParams` class configures turn detection behavior:

<ParamField path="stop_secs" type="float" default="3.0">
Duration of silence in seconds required before triggering a silence-based end
of turn
Duration of silence in seconds required before triggering a silence-based end
of turn.

Note: This value is different from the VAD's `stop_secs` parameter.
Set the `SmartTurnParams.stop_secs` to specify how long the Turn Analyzer should
wait before classifying the turn as complete.

</ParamField>

<ParamField path="pre_speech_ms" type="float" default="0.0">
Expand Down
80 changes: 70 additions & 10 deletions api-reference/server/utilities/turn-management/user-turn-strategies.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ By default, Pipecat uses a combination of VAD (Voice Activity Detection) and AI-

You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models.

<Note>
In Pipecat >=1.0.0, the `turn_analyzer` must be specified in the
`TurnAnalyzerUserTurnStopStrategy`. See the [migration
guide](/pipecat/migration/migration-1.0#3-vad-&-turn-analyzer-configuration)
for more information.
</Note>

## How It Works

1. **Turn Start Detection**: When any start strategy triggers, the user aggregator:
Expand All @@ -30,6 +37,49 @@ You can customize this behavior by providing your own strategies for more sophis

4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended. This timeout is configurable via `LLMUserAggregatorParams` (see [Configuration](#configuration) below). When the timeout fires, the [`on_user_turn_stop_timeout`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_stop_timeout) event is emitted.

### When a user turn ends

A **user turn stop strategy** decides when the user is done talking and the bot should respond. You choose the strategy; the rest of the timing follows from it. **If you want to adjust how long the bot waits before responding, do it in the stop strategy** — that's the knob built for it.

Whatever the strategy, closing a turn takes two ingredients:

1. **A transcript.** Every stop strategy waits for the STT service to transcribe what the user said (`wait_for_transcript`, on by default). This is usually the largest and most variable part of the delay, and it's dominated by your STT provider's latency — see the [STT benchmark](https://github.com/pipecat-ai/stt-benchmark) to compare services.
2. **The strategy's own end-of-turn criteria.** Once the transcript is in, the strategy applies its logic to decide whether the turn is actually complete.

The turn closes only when both are satisfied.

#### Choosing a strategy

| Strategy | How it decides the turn is complete |
| ------------------------------------ | --------------------------------------------------------------------------------------------- |
| `SpeechTimeoutUserTurnStopStrategy` | A fixed silence window (`user_speech_timeout`, default `0.6s`) elapses after the user pauses. |
| `TurnAnalyzerUserTurnStopStrategy` | A turn-detection ("smart turn") model predicts end-of-turn from the audio and transcript. |
| `FilterIncompleteUserTurnStrategies` | An LLM assesses whether the user's utterance is semantically complete. |
| Custom | Your own logic implementing the stop-strategy interface. |

This is also where you tune responsiveness. For example, raise or lower `user_speech_timeout` on `SpeechTimeoutUserTurnStopStrategy` to give users more or less time to resume before the bot replies.

#### Where the latency comes from

Before any strategy can act, two things have to happen:

- **VAD silence (`stop_secs`):** The VAD waits a short, fixed interval (default `0.2s`) after audio goes quiet before reporting that the user stopped speaking. This is a low-level detection threshold — leave it alone. To change wait time, use the stop strategy, not `stop_secs`.
- **Transcription:** STT returns a transcript. Latency here is mostly your STT provider's; strategies use the provider's reported p99 latency as a fallback timer and short-circuit it the moment a finalized transcript arrives.

Then the strategy's criteria run (the silence window, the turn model, or the LLM check) to finalize the turn.

<Note>
Extra silence users notice usually isn't one tunable value — it's VAD silence
plus transcription latency plus the strategy's own window stacking up. If your
STT service is slow to return transcripts, that delay shows up in every turn
regardless of which strategy you use. Adjust responsiveness in the stop
strategy, and check the [STT
benchmark](https://github.com/pipecat-ai/stt-benchmark) if transcription is
the bottleneck.
</Note>

`user_turn_stop_timeout` (default `5.0s`, on `LLMUserAggregatorParams`) is a backstop, not part of normal timing: if a turn somehow never finalizes, it forces the turn to end so the bot isn't stuck waiting.

## Configuration

User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:
Expand Down Expand Up @@ -74,6 +124,14 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
)
```

<ParamField path="vad_analyzer" type="VADAnalyzer" default="None">
The Voice Activity Detection analyzer instance used to produce VAD signals for
turn detection. In Pipecat 1.x, VAD is configured on the aggregator rather
than the transport. This is separate from `VADUserTurnStartStrategy`: the
analyzer _produces_ the VAD signals; the start strategy decides _what to do_
with them.
</ParamField>

## Start Strategies

Start strategies determine when a user's turn begins. Multiple strategies can be provided, and the first one to trigger will signal the start of a user turn.
Expand Down Expand Up @@ -317,11 +375,12 @@ strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6)
<Note>
Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the
recommended default). If you change `stop_secs`, the strategy will log a
warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark)
with your VAD settings and pass the measured TTFS P99 latency to your STT
service constructor via `ttfs_p99_latency`. The strategy will also warn if
`stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s
and may cause delayed turn detection.
warning suggesting you re-run the
[stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD
settings and pass the measured TTFS P99 latency to your STT service
constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs
>= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause
delayed turn detection.
</Note>

### TurnAnalyzerUserTurnStopStrategy
Expand Down Expand Up @@ -359,11 +418,12 @@ strategy = TurnAnalyzerUserTurnStopStrategy(
<Note>
Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the
recommended default). If you change `stop_secs`, the strategy will log a
warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark)
with your VAD settings and pass the measured TTFS P99 latency to your STT
service constructor via `ttfs_p99_latency`. The strategy will also warn if
`stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s
and may cause delayed turn detection.
warning suggesting you re-run the
[stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD
settings and pass the measured TTFS P99 latency to your STT service
constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs
>= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause
delayed turn detection.
</Note>

### ExternalUserTurnStopStrategy
Expand Down
68 changes: 53 additions & 15 deletions client/concepts/choosing-a-transport.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,39 @@ The choice of transport has one important constraint: **the client transport and
## Available transports

<CardGroup cols={2}>
<Card title="SmallWebRTC" icon="link" href="/api-reference/client/js/transports/small-webrtc">
<Card
title="SmallWebRTC"
icon="link"
href="/api-reference/client/js/transports/small-webrtc"
>
Serverless peer-to-peer WebRTC. No third-party account needed.
</Card>
<Card title="Daily" icon="tower-broadcast" href="/api-reference/client/js/transports/daily">
<Card
title="Daily"
icon="tower-broadcast"
href="/api-reference/client/js/transports/daily"
>
WebRTC via Daily's global infrastructure. Recommended for production.
</Card>
<Card title="WebSocket" icon="plug" href="/api-reference/client/js/transports/websocket">
<Card
title="WebSocket"
icon="plug"
href="/api-reference/client/js/transports/websocket"
>
Direct WebSocket connection. For server-to-server setups only.
</Card>
<Card title="Gemini Live" icon="google" href="/api-reference/client/js/transports/gemini">
<Card
title="Gemini Live"
icon="google"
href="/api-reference/client/js/transports/gemini"
>
Direct connection to Google's Gemini Live API. No Pipecat server needed.
</Card>
<Card title="OpenAI WebRTC" icon="robot" href="/api-reference/client/js/transports/openai-webrtc">
<Card
title="OpenAI WebRTC"
icon="robot"
href="/api-reference/client/js/transports/openai-webrtc"
>
Direct connection to OpenAI's Realtime API. No Pipecat server needed.
</Card>
</CardGroup>
Expand Down Expand Up @@ -50,7 +70,9 @@ Once you've settled on WebRTC, the next question is how you route it. There are
A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's simpler than running your own WebRTC infrastructure and avoids a third-party dependency. If you want managed infrastructure that handles global routing, audio processing, and scaling for you, use Daily.

<Note>
For a deeper look at this tradeoff, see the Daily blog post [You don't need a WebRTC server for your voice agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/).
For a deeper look at this tradeoff, see the Daily blog post [You don't need a
WebRTC server for your voice
agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/).
</Note>

---
Expand All @@ -62,11 +84,13 @@ A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's
SmallWebRTC is the default in all Pipecat quickstart templates. No account needed, no third-party services — just a direct peer-to-peer WebRTC connection between your client and your bot.

**Use it when:**

- Developing or testing locally
- Running a self-hosted deployment
- Building embedded or edge deployments where simplicity matters

**Avoid it when:**

- Your users are geographically distributed
- You need built-in echo cancellation, noise reduction, or network resilience at scale
- You're scaling beyond a handful of concurrent sessions
Expand All @@ -89,6 +113,7 @@ await client.connect({ webrtcUrl: "http://localhost:7860/api/offer" });
Daily provides a global WebRTC network with mesh routing, built-in audio processing (echo cancellation, noise suppression, automatic gain control), and resilience to network changes. It's the recommended choice for anything user-facing in production.

**Use it when:**

- Shipping a production app to real users
- Your users are on a variety of networks, devices, or locations
- You want managed infrastructure without operating it yourself
Expand All @@ -114,11 +139,13 @@ await client.startBotAndConnect({ endpoint: "/api/start" });
The WebSocket transport connects to a WebSocket server rather than using WebRTC. It's appropriate for controlled, server-to-server scenarios where both sides are on stable networks, or for text-only bots with no audio requirements.

**Use it when:**

- Integrating telephony media streams (Twilio, Telnyx). This is the most common production WebSocket use case: the phone provider streams audio to your server over a WebSocket, which is a provider-to-server connection, not a browser client.
- Building text-only interactions (no audio)
- Connecting two servers (not a browser client)
- Network constraints make WebRTC impractical in a specific environment

**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above.
**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above. The telephony case is different: the audio comes from a phone network over the provider's media-stream WebSocket, not from a browser, so the WebRTC advantages for browser audio don't apply.

---

Expand All @@ -127,11 +154,22 @@ The WebSocket transport connects to a WebSocket server rather than using WebRTC.
These transports connect your client directly to Google's Gemini Live API or OpenAI's Realtime API, bypassing a Pipecat server entirely. They're useful for prototyping or demos when you want speech-to-speech interactions without running a backend.

<Warning>
Direct API transports expose your API key in the client. They are appropriate for development and demos, but **not for production** — use a server-side Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API keys secure.
Direct API transports expose your API key in the client. They are appropriate
for development and demos, but **not for production** — use a server-side
Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API
keys secure.
</Warning>

---

### What about LiveKit?

Pipecat ships a [LiveKit transport for the **server**](/api-reference/server/services/transport/livekit), so your bot can join a LiveKit room. There is **no** official Pipecat **client** transport for LiveKit (JS/React, React Native, or otherwise).

@jamsea jamsea Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markbackman this is a direct response to people asking the AI chatbots about Livekit. We do have an open PR here to make a transport: pipecat-ai/pipecat-client-web-transports#86

Might be worth merging in the PR and deleting this. But not sure how much of a lift that is for engineering. While we do get people asking the AI chatbot we don't have anyone on paid support asking about this so it might be lower in priority.


Because the client and server transports must be a matching pair, you cannot pair a LiveKit server transport with a Pipecat client SDK today. If you want a self-hostable, production-grade WebRTC transport that you control end to end with Pipecat clients, use `SmallWebRTCTransport`; if you want managed WebRTC infrastructure, use Daily.

---

## Swapping transports

Transports are interchangeable — the rest of your application code stays the same. The only thing that changes is the import and constructor:
Expand All @@ -153,10 +191,10 @@ The Pipecat CLI scaffolds a `config.ts` that selects the transport based on an e

## Summary

| Transport | Best for | Requires |
|---|---|---|
| SmallWebRTC | Local dev, self-hosted | Nothing |
| Daily | Production apps, global users | Daily account |
| WebSocket | Text-only, server-to-server | Custom server |
| Gemini Live | Gemini prototypes | Gemini API key |
| OpenAI WebRTC | OpenAI prototypes | OpenAI API key |
| Transport | Best for | Requires |
| ------------- | ----------------------------- | -------------- |
| SmallWebRTC | Local dev, self-hosted | Nothing |
| Daily | Production apps, global users | Daily account |
| WebSocket | Text-only, server-to-server | Custom server |
| Gemini Live | Gemini prototypes | Gemini API key |
| OpenAI WebRTC | OpenAI prototypes | OpenAI API key |
36 changes: 36 additions & 0 deletions pipecat/learn/context-management.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ You can programmatically add new messages to the context by pushing or queueing

- **`LLMMessagesAppendFrame`**: Appends a new message to the existing context
- **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages
- **`LLMMessagesTransformFrame`**: Edits the existing context in place using a transform function

```python
# Add a new user message to context and trigger a response
Expand All @@ -235,6 +236,41 @@ await worker.queue_frames([
])
```

#### Adding a message silently

All three frames take a `run_llm` argument that controls whether the change also prompts a bot response. Pass `run_llm=True` to respond; the default (`None`, which behaves like `False`) updates the context silently. This is useful when you collect information in the background and don't want the bot to react every time:

```python
# Add a message to context without triggering a bot response
note = {"role": "user", "content": "Caller's name is Maria. Account verified."}
await worker.queue_frames([
LLMMessagesAppendFrame([note], run_llm=False), # Silent: no response
])
```

#### Editing or removing specific messages

To surgically edit or remove individual messages without rebuilding the whole list, push an `LLMMessagesTransformFrame`. It takes a function that receives the current list of messages and returns a modified list. Use it to drop stale instructions, remove an offensive turn, or rewrite content:

```python
from pipecat.frames.frames import LLMMessagesTransformFrame

def remove_language_instructions(messages):
# Drop any message that contains a per-turn language instruction
return [m for m in messages if "LANGUAGE INSTRUCTION" not in str(m.get("content", ""))]

await worker.queue_frames([
LLMMessagesTransformFrame(transform=remove_language_instructions, run_llm=False),
])
```

<Note>
To make the bot **speak** specific text and have it recorded in context (for
example, a greeting the LLM did not generate), use [`TTSSpeakFrame` with
`append_to_context=True`](/pipecat/learn/text-to-speech). That path is for
driving speech output; the frames above are for editing the context directly.
</Note>

### Retrieving Current Context

The context aggregator provides a `context` property for getting the current context:
Expand Down
Loading
Loading