From d989d8c53e5bc8329dc62cccd07bd944da9b845d Mon Sep 17 00:00:00 2001 From: James Hush Date: Fri, 26 Jun 2026 16:35:41 +0800 Subject: [PATCH 1/3] docs: close gaps found in Kapa source analytics audit Audited the top 10 most-cited docs.pipecat.ai pages against the user questions that cited them, and fixed the gaps that made Kapa give weak, stale, or contradictory answers. Each change was verified against the pipecat source. - stt/deepgram: fix broken Flux import path (was an ImportError) - user-turn-strategies: add custom max-turn stop strategy, turn-timing timeline, vad_analyzer param, and 1.x migration note - speech-input: explain why VAD is on the aggregator, link noise filters - function-calling: document run_in_parallel / group_parallel_tools for duplicate responses with parallel tool calls - context-management: silent add (run_llm=False), LLMMessagesTransformFrame, TTSSpeakFrame append_to_context link - text-to-speech: state append_to_context default (True, v1.4.0) - choosing-a-transport: LiveKit is server-only, telephony WebSocket use case - migration-1.0: warn that removed params are silently ignored, add a concrete interruptions mapping - smart-turn-overview: disambiguate the two stop_secs values - pipeline-termination: frame table, complete the end-call example, on_pipeline_finished cleanup, max call duration, dangling-tasks entry --- .../server/services/stt/deepgram.mdx | 8 +- .../turn-detection/smart-turn-overview.mdx | 17 +++ .../turn-management/user-turn-strategies.mdx | 138 ++++++++++++++++-- client/concepts/choosing-a-transport.mdx | 68 +++++++-- pipecat/learn/context-management.mdx | 36 +++++ pipecat/learn/function-calling.mdx | 29 ++++ pipecat/learn/pipeline-termination.mdx | 72 +++++++++ pipecat/learn/speech-input.mdx | 23 +++ pipecat/learn/text-to-speech.mdx | 12 +- pipecat/migration/migration-1.0.mdx | 72 ++++++--- 10 files changed, 423 insertions(+), 52 deletions(-) diff --git a/api-reference/server/services/stt/deepgram.mdx b/api-reference/server/services/stt/deepgram.mdx index b94e2e721..80958d1b4 100644 --- a/api-reference/server/services/stt/deepgram.mdx +++ b/api-reference/server/services/stt/deepgram.mdx @@ -319,7 +319,7 @@ Runtime-configurable settings passed via the `settings` constructor argument usi ### Usage ```python -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService stt = DeepgramFluxSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), @@ -329,7 +329,7 @@ stt = DeepgramFluxSTTService( #### With EagerEndOfTurn ```python -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService stt = DeepgramFluxSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), @@ -344,7 +344,7 @@ stt = DeepgramFluxSTTService( #### Multilingual Support ```python -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService from pipecat.transcriptions.language import Language # Use flux-general-multi with language hints @@ -363,7 +363,7 @@ The `keyterm`, `eot_threshold`, `eager_eot_threshold`, `eot_timeout_ms`, and `la ```python from pipecat.frames.frames import STTUpdateSettingsFrame -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService from pipecat.transcriptions.language import Language # During pipeline execution, update settings without reconnecting diff --git a/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx b/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx index b71babceb..b5c275a1c 100644 --- a/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx +++ b/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx @@ -102,6 +102,23 @@ The `SmartTurnParams` class configures turn detection behavior: of turn + + **This is not the same `stop_secs` as your VAD's.** There are two: + +- **VAD `stop_secs`** (default `0.2`, on `VADParams`): how long silence must + last before VAD declares the user stopped speaking. This is what + _triggers_ a Smart Turn analysis. +- **`SmartTurnParams.stop_secs`** (default `3.0`, here): a silence-based + _fallback_. If the model keeps classifying the turn as incomplete, this is + how long to wait before force-completing the turn anyway. + +Lowering VAD `stop_secs` makes turn-taking more responsive; changing +`SmartTurnParams.stop_secs` only affects how long an unresolved turn waits +before the fallback fires, which is why tuning it often shows little effect +in normal conversation. + + + Amount of audio (in milliseconds) to include before speech is detected diff --git a/api-reference/server/utilities/turn-management/user-turn-strategies.mdx b/api-reference/server/utilities/turn-management/user-turn-strategies.mdx index 2536edde3..fd262be0b 100644 --- a/api-reference/server/utilities/turn-management/user-turn-strategies.mdx +++ b/api-reference/server/utilities/turn-management/user-turn-strategies.mdx @@ -14,9 +14,17 @@ By default, Pipecat uses a combination of VAD (Voice Activity Detection) and AI- You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models. + + The user turn strategies API is new in Pipecat 1.x. If you're coming from + 0.0.x (where turn detection was configured via `turn_analyzer` on the + transport params), see the [migration guide](/pipecat/migration/migration-1.0) + for where each setting moved. + + ## How It Works 1. **Turn Start Detection**: When any start strategy triggers, the user aggregator: + - Marks the start of a user turn - Optionally emits `UserStartedSpeakingFrame` - Optionally emits an interruption frame (if the bot is speaking) @@ -24,12 +32,31 @@ You can customize this behavior by providing your own strategies for more sophis 2. **During User Turn**: The aggregator collects transcriptions and audio frames. 3. **Turn Stop Detection**: When a stop strategy triggers, the user aggregator: + - Marks the end of the user turn - Emits `UserStoppedSpeakingFrame` - Pushes the aggregated user message to the LLM context 4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended. This timeout is configurable via `LLMUserAggregatorParams` (see [Configuration](#configuration) below). When the timeout fires, the [`on_user_turn_stop_timeout`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_stop_timeout) event is emitted. +### Turn timing timeline + +Several timers combine to decide when a user turn ends. They fire in this order after the user stops talking: + +1. **VAD `stop_secs`**: The VAD waits this long (default `0.2s`) after audio goes quiet before it reports the user stopped speaking. Configured on your `VADParams`, not here. +2. **`user_speech_timeout`** (on `SpeechTimeoutUserTurnStopStrategy`, default `0.6s`): The policy floor. After VAD silence, this is the window in which the user may resume speaking. It always runs to completion. +3. **STT finalization / `ttfs_p99_latency`** _or_ **smart-turn analysis**: A safety net for STT latency. With `SpeechTimeoutUserTurnStopStrategy`, the turn waits for the STT service to return a final transcript (short-circuited when the STT service emits a finalized transcript). With `TurnAnalyzerUserTurnStopStrategy`, the smart-turn model decides end-of-turn instead. +4. **Turn finalized**: Once the active stop strategy's conditions are met, the turn ends and the bot responds. + +`user_turn_stop_timeout` (default `5.0s`, on `LLMUserAggregatorParams`) sits outside all of this as an outer watchdog: if none of the above finalizes the turn in time, it forces the turn to end. + + + The extra silence some users notice (for example, "the aggregator waits an + extra 800ms") is usually steps 2 and 3 stacking: `stop_secs` plus + `user_speech_timeout` plus the STT wait. Tune each timer independently rather + than expecting one value to control total latency. + + ## Configuration User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`: @@ -74,6 +101,14 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( ) ``` + + The Voice Activity Detection analyzer instance used to produce VAD signals for + turn detection. In Pipecat 1.x, VAD is configured on the aggregator rather + than the transport. This is separate from `VADUserTurnStartStrategy`: the + analyzer _produces_ the VAD signals; the start strategy decides _what to do_ + with them. + + ## Start Strategies Start strategies determine when a user's turn begins. Multiple strategies can be provided, and the first one to trigger will signal the start of a user turn. @@ -317,11 +352,12 @@ strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6) Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the recommended default). If you change `stop_secs`, the strategy will log a - warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) - with your VAD settings and pass the measured TTFS P99 latency to your STT - service constructor via `ttfs_p99_latency`. The strategy will also warn if - `stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s - and may cause delayed turn detection. + warning suggesting you re-run the + [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD + settings and pass the measured TTFS P99 latency to your STT service + constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs + >= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause + delayed turn detection. ### TurnAnalyzerUserTurnStopStrategy @@ -359,11 +395,12 @@ strategy = TurnAnalyzerUserTurnStopStrategy( Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the recommended default). If you change `stop_secs`, the strategy will log a - warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) - with your VAD settings and pass the measured TTFS P99 latency to your STT - service constructor via `ttfs_p99_latency`. The strategy will also warn if - `stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s - and may cause delayed turn detection. + warning suggesting you re-run the + [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD + settings and pass the measured TTFS P99 latency to your STT service + constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs + >= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause + delayed turn detection. ### ExternalUserTurnStopStrategy @@ -454,6 +491,87 @@ stop = [ ] ``` +### Custom stop strategy + +Pipecat has no built-in strategy for capping how long a single user turn can run (there is no "max speech" or "max turn duration" strategy). If you need to end a turn after a fixed amount of time, for example to stop a customer from monologuing, write your own stop strategy by subclassing `BaseUserTurnStopStrategy`. + +A stop strategy inspects incoming frames in `process_frame` and calls `trigger_user_turn_stopped()` when it decides the turn is over. The example below starts a timer when the user begins speaking and forces the turn to end once the timer fires: + +```python +import asyncio + +from pipecat.frames.frames import Frame, VADUserStartedSpeakingFrame +from pipecat.turns.types import ProcessFrameResult +from pipecat.turns.user_stop import BaseUserTurnStopStrategy + + +class MaxTurnDurationStopStrategy(BaseUserTurnStopStrategy): + """End the user turn after a fixed number of seconds. + + Starts a timer when the user starts speaking. If the user is still + holding the turn when the timer fires, the turn is finalized so the + bot can respond. + """ + + def __init__(self, *, max_turn_seconds: float = 10.0, **kwargs): + super().__init__(**kwargs) + self._max_turn_seconds = max_turn_seconds + self._timer_task: asyncio.Task | None = None + + async def process_frame(self, frame: Frame) -> ProcessFrameResult: + if isinstance(frame, VADUserStartedSpeakingFrame) and self._timer_task is None: + self._timer_task = self.task_manager.create_task( + self._max_turn_handler(), + f"{self}::_max_turn_handler", + ) + return ProcessFrameResult.CONTINUE + + async def _max_turn_handler(self): + try: + await asyncio.sleep(self._max_turn_seconds) + except asyncio.CancelledError: + return + finally: + self._timer_task = None + # Time is up: end the turn so the bot responds. + await self.trigger_user_turn_stopped() + + async def reset(self): + await super().reset() + await self._cancel_timer() + + async def cleanup(self): + await super().cleanup() + await self._cancel_timer() + + async def _cancel_timer(self): + if self._timer_task: + await self.task_manager.cancel_task(self._timer_task) + self._timer_task = None +``` + +Pair it with a normal detector so regular turn-end still works. The first strategy to trigger ends the turn, so the timer acts as a backstop on top of smart-turn detection: + +```python +from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3 +from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy +from pipecat.turns.user_turn_strategies import UserTurnStrategies + +strategies = UserTurnStrategies( + stop=[ + TurnAnalyzerUserTurnStopStrategy(turn_analyzer=LocalSmartTurnAnalyzerV3()), + MaxTurnDurationStopStrategy(max_turn_seconds=10.0), + ], +) +``` + + + Ending the turn lets the bot respond, but it does not stop the user's audio + from being processed if they keep talking. To also ignore further user input + for a window, combine this with [User Input + Muting](/pipecat/fundamentals/user-input-muting). + + ## Helper Functions Pipecat provides helper functions to compose custom strategy lists that extend the defaults. diff --git a/client/concepts/choosing-a-transport.mdx b/client/concepts/choosing-a-transport.mdx index d3e18fd0b..cb44f6717 100644 --- a/client/concepts/choosing-a-transport.mdx +++ b/client/concepts/choosing-a-transport.mdx @@ -10,19 +10,39 @@ The choice of transport has one important constraint: **the client transport and ## Available transports - + Serverless peer-to-peer WebRTC. No third-party account needed. - + WebRTC via Daily's global infrastructure. Recommended for production. - + Direct WebSocket connection. For server-to-server setups only. - + Direct connection to Google's Gemini Live API. No Pipecat server needed. - + Direct connection to OpenAI's Realtime API. No Pipecat server needed. @@ -50,7 +70,9 @@ Once you've settled on WebRTC, the next question is how you route it. There are A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's simpler than running your own WebRTC infrastructure and avoids a third-party dependency. If you want managed infrastructure that handles global routing, audio processing, and scaling for you, use Daily. - For a deeper look at this tradeoff, see the Daily blog post [You don't need a WebRTC server for your voice agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/). + For a deeper look at this tradeoff, see the Daily blog post [You don't need a + WebRTC server for your voice + agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/). --- @@ -62,11 +84,13 @@ A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's SmallWebRTC is the default in all Pipecat quickstart templates. No account needed, no third-party services — just a direct peer-to-peer WebRTC connection between your client and your bot. **Use it when:** + - Developing or testing locally - Running a self-hosted deployment - Building embedded or edge deployments where simplicity matters **Avoid it when:** + - Your users are geographically distributed - You need built-in echo cancellation, noise reduction, or network resilience at scale - You're scaling beyond a handful of concurrent sessions @@ -89,6 +113,7 @@ await client.connect({ webrtcUrl: "http://localhost:7860/api/offer" }); Daily provides a global WebRTC network with mesh routing, built-in audio processing (echo cancellation, noise suppression, automatic gain control), and resilience to network changes. It's the recommended choice for anything user-facing in production. **Use it when:** + - Shipping a production app to real users - Your users are on a variety of networks, devices, or locations - You want managed infrastructure without operating it yourself @@ -114,11 +139,13 @@ await client.startBotAndConnect({ endpoint: "/api/start" }); The WebSocket transport connects to a WebSocket server rather than using WebRTC. It's appropriate for controlled, server-to-server scenarios where both sides are on stable networks, or for text-only bots with no audio requirements. **Use it when:** + +- Integrating telephony media streams (Twilio, Telnyx). This is the most common production WebSocket use case: the phone provider streams audio to your server over a WebSocket, which is a provider-to-server connection, not a browser client. - Building text-only interactions (no audio) - Connecting two servers (not a browser client) - Network constraints make WebRTC impractical in a specific environment -**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above. +**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above. The telephony case is different: the audio comes from a phone network over the provider's media-stream WebSocket, not from a browser, so the WebRTC advantages for browser audio don't apply. --- @@ -127,11 +154,22 @@ The WebSocket transport connects to a WebSocket server rather than using WebRTC. These transports connect your client directly to Google's Gemini Live API or OpenAI's Realtime API, bypassing a Pipecat server entirely. They're useful for prototyping or demos when you want speech-to-speech interactions without running a backend. - Direct API transports expose your API key in the client. They are appropriate for development and demos, but **not for production** — use a server-side Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API keys secure. + Direct API transports expose your API key in the client. They are appropriate + for development and demos, but **not for production** — use a server-side + Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API + keys secure. --- +### What about LiveKit? + +Pipecat ships a [LiveKit transport for the **server**](/api-reference/server/services/transport/livekit), so your bot can join a LiveKit room. There is **no** official Pipecat **client** transport for LiveKit (JS/React, React Native, or otherwise). + +Because the client and server transports must be a matching pair, you cannot pair a LiveKit server transport with a Pipecat client SDK today. If you want a self-hostable, production-grade WebRTC transport that you control end to end with Pipecat clients, use `SmallWebRTCTransport`; if you want managed WebRTC infrastructure, use Daily. + +--- + ## Swapping transports Transports are interchangeable — the rest of your application code stays the same. The only thing that changes is the import and constructor: @@ -153,10 +191,10 @@ The Pipecat CLI scaffolds a `config.ts` that selects the transport based on an e ## Summary -| Transport | Best for | Requires | -|---|---|---| -| SmallWebRTC | Local dev, self-hosted | Nothing | -| Daily | Production apps, global users | Daily account | -| WebSocket | Text-only, server-to-server | Custom server | -| Gemini Live | Gemini prototypes | Gemini API key | -| OpenAI WebRTC | OpenAI prototypes | OpenAI API key | +| Transport | Best for | Requires | +| ------------- | ----------------------------- | -------------- | +| SmallWebRTC | Local dev, self-hosted | Nothing | +| Daily | Production apps, global users | Daily account | +| WebSocket | Text-only, server-to-server | Custom server | +| Gemini Live | Gemini prototypes | Gemini API key | +| OpenAI WebRTC | OpenAI prototypes | OpenAI API key | diff --git a/pipecat/learn/context-management.mdx b/pipecat/learn/context-management.mdx index b5c65e43f..77ead2a50 100644 --- a/pipecat/learn/context-management.mdx +++ b/pipecat/learn/context-management.mdx @@ -226,6 +226,7 @@ You can programmatically add new messages to the context by pushing or queueing - **`LLMMessagesAppendFrame`**: Appends a new message to the existing context - **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages +- **`LLMMessagesTransformFrame`**: Edits the existing context in place using a transform function ```python # Add a new user message to context and trigger a response @@ -235,6 +236,41 @@ await worker.queue_frames([ ]) ``` +#### Adding a message silently + +All three frames take a `run_llm` argument. Set `run_llm=False` (or leave it unset) to add or change the context **without** prompting the bot to respond. This is useful when you collect information in the background and don't want the bot to react every time: + +```python +# Add a message to context without triggering a bot response +note = {"role": "user", "content": "Caller's name is Maria. Account verified."} +await worker.queue_frames([ + LLMMessagesAppendFrame([note], run_llm=False), # Silent: no response +]) +``` + +#### Editing or removing specific messages + +To surgically edit or remove individual messages without rebuilding the whole list, push an `LLMMessagesTransformFrame`. It takes a function that receives the current list of messages and returns a modified list. Use it to drop stale instructions, remove an offensive turn, or rewrite content: + +```python +from pipecat.frames.frames import LLMMessagesTransformFrame + +def remove_language_instructions(messages): + # Drop any message that contains a per-turn language instruction + return [m for m in messages if "LANGUAGE INSTRUCTION" not in str(m.get("content", ""))] + +await worker.queue_frames([ + LLMMessagesTransformFrame(transform=remove_language_instructions, run_llm=False), +]) +``` + + + To make the bot **speak** specific text and have it recorded in context (for + example, a greeting the LLM did not generate), use [`TTSSpeakFrame` with + `append_to_context=True`](/pipecat/learn/text-to-speech). That path is for + driving speech output; the frames above are for editing the context directly. + + ### Retrieving Current Context The context aggregator provides a `context` property for getting the current context: diff --git a/pipecat/learn/function-calling.mdx b/pipecat/learn/function-calling.mdx index 3a7066c7e..524da4e1a 100644 --- a/pipecat/learn/function-calling.mdx +++ b/pipecat/learn/function-calling.mdx @@ -173,6 +173,35 @@ llm = OpenAILLMService( When `enable_async_tool_cancellation=True` and at least one async function is available, Pipecat automatically adds the built-in `cancel_async_tool_call` tool and supporting system instructions. The LLM can call that tool to cancel a stale in-progress async function call — for example, when the user changes their request before a long-running lookup completes. +## Parallel and Multiple Tool Calls + +When the LLM calls more than one tool in a single turn, two `LLMService` constructor options control how those calls run and when the LLM responds: + +```python +llm = OpenAILLMService( + api_key="your-api-key", + run_in_parallel=True, # default + group_parallel_tools=True, # default +) +``` + + + Whether multiple tool calls in one turn run in parallel or one after another. + + + + When `True`, all tool calls in a batch are grouped so the LLM is triggered + **exactly once** after every call in the batch completes. When `False`, each + function call result triggers the LLM independently as it arrives. + + + + If multiple tools firing in one turn produce **duplicate or repeated + responses** (the bot answers once per tool instead of once total), check that + `group_parallel_tools` is `True`. With it disabled, each result re-triggers + the LLM, so the model responds once for every tool call in the batch. + + ## Changing Tools Mid-Conversation To change the set of tools the LLM can use during a session, push an `LLMSetToolsFrame`. Its `tools` field takes the same things as `LLMContext(tools=[...])` — a list of direct functions and/or `FunctionSchema` objects. Whatever you pass becomes the LLM's new tool set. diff --git a/pipecat/learn/pipeline-termination.mdx b/pipecat/learn/pipeline-termination.mdx index 5131509e2..9f8b22568 100644 --- a/pipecat/learn/pipeline-termination.mdx +++ b/pipecat/learn/pipeline-termination.mdx @@ -29,6 +29,23 @@ pipeline = Pipeline([ Both frames flow downstream through the pipeline, allowing each processor to clean up resources appropriately. +### Termination frames at a glance + +| Frame | Job | Push from / direction | +| ------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| `EndFrame` | Graceful shutdown (drains pending frames) | From outside the pipeline via `worker.queue_frame(EndFrame())` | +| `CancelFrame` | Immediate shutdown (discards pending frames) | Via `worker.cancel()` | +| `EndWorkerFrame` | Graceful shutdown signal from inside the pipeline | `push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to an `EndFrame` | +| `CancelWorkerFrame` | Immediate shutdown signal from inside the pipeline | `push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to a `CancelFrame` | + + + If you see `EndTaskFrame`, `CancelTaskFrame`, `PipelineTask`, or + `task.cancel()` in older code or examples, those are **deprecated aliases** + (since 1.3.0/1.4.0) of `EndWorkerFrame`, `CancelWorkerFrame`, + `PipelineWorker`, and `worker.cancel()`. They still work but will be removed + in 2.0.0. Use the `Worker` names in new code. + + ## Termination Methods Pipecat provides two primary approaches for pipeline termination, each designed for different scenarios: @@ -64,10 +81,21 @@ from pipecat.processors.frame_processor import FrameDirection async def end_conversation(params: FunctionCallParams): await params.llm.push_frame(TTSSpeakFrame("Have a nice day!")) + # Resolve the function call so the LLM call doesn't hang + await params.result_callback({"status": "ended"}) + # Signal that the worker should end after processing this frame await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM) + +# Register the handler so the LLM can call it as a tool +llm.register_function("end_conversation", end_conversation) ``` + + Always call `params.result_callback(...)` in your handler before pushing the + end frame. Skipping it can leave the LLM function call unresolved. + + **How graceful termination works:** 1. `EndFrame` is queued and processes after any pending frames (like goodbye messages) @@ -153,6 +181,31 @@ You can further configure the idle detection behavior. To learn more, refer to t being wasted on inactive conversations. +### Maximum Call Duration + +Idle detection ends a call that goes quiet, but it does not cap the total length of an active call. To enforce a maximum call duration, run an `asyncio` timer that speaks a goodbye and then queues an `EndFrame` so the bot can sign off gracefully before shutdown: + +```python +import asyncio +from pipecat.frames.frames import EndFrame, TTSSpeakFrame + +async def end_after(worker, timeout_secs: float): + await asyncio.sleep(timeout_secs) + await worker.queue_frame(TTSSpeakFrame("We've reached our time limit. Goodbye!")) + await worker.queue_frame(EndFrame()) # Graceful: plays the goodbye, then shuts down + +@transport.event_handler("on_client_connected") +async def on_client_connected(transport, client): + asyncio.create_task(end_after(worker, timeout_secs=300)) # 5-minute cap +``` + + + On Pipecat Cloud, there is also a platform-level hard cap via + [`maxSessionDuration`](/pipecat-cloud/fundamentals/active-sessions#session-duration-limits) + (default 7200s). That cap is a forced cut with no goodbye, so use the + bot-level timer above when you want the bot to speak before the call ends. + + ## Implementation Patterns ### Event-Driven Termination @@ -205,6 +258,19 @@ except Exception as e: await worker.cancel() ``` +### Running Cleanup Code on Shutdown + +To run cleanup or persist data when a call ends, use the `on_pipeline_finished` event handler. It fires after the pipeline reaches any terminal state, so it runs for **both** graceful (`EndFrame`) and cancelled (`CancelFrame`) shutdowns. This makes it the single write point for end-of-call work like saving a transcript or recording: + +```python +@worker.event_handler("on_pipeline_finished") +async def on_pipeline_finished(worker, frame): + # Runs for both graceful and cancelled shutdown + await save_transcript_to_db() +``` + +`on_client_disconnected`, by contrast, fires only when the client disconnects. Use it to _tag the reason_ for the shutdown (for example, "user hung up"), and do the actual persistence in `on_pipeline_finished` so you write data exactly once regardless of how the call ended. See the [`PipelineWorker` events](/api-reference/server/pipeline/pipeline-worker) reference for the full event signature. + ## Troubleshooting If your pipeline isn't shutting down properly, check these common issues: @@ -245,6 +311,12 @@ await self.push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM) termination handling throughout the pipeline. +### "dangling tasks detected" Warning + +**Problem:** On shutdown you see a log warning like `PipelineWorker#0 dangling tasks detected: [...]`. + +**Solution:** This means `asyncio` tasks created during the session were never cancelled or awaited before the pipeline shut down. The usual causes are the two above: a custom processor that doesn't propagate termination frames, or a background task started inside a processor (for example, a timer or long-running coroutine) that isn't cleaned up. Create background tasks through the pipeline task manager so they are tracked and cancelled on shutdown, and make sure your processors push `EndFrame`/`CancelFrame` downstream. + ## Key Takeaways - **Frame-based termination** - shutdown uses the same frame system as processing diff --git a/pipecat/learn/speech-input.mdx b/pipecat/learn/speech-input.mdx index 52d70c3b8..571aa97e1 100644 --- a/pipecat/learn/speech-input.mdx +++ b/pipecat/learn/speech-input.mdx @@ -63,6 +63,19 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( In the vast number of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it. + + **Why is VAD configured on the aggregator?** VAD lives in + `LLMUserAggregatorParams` because its speech start/stop signals feed the [user + turn + strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) + that decide when a turn begins and ends. VAD runs on the raw input audio, so + where the aggregator sits in your pipeline relative to STT does not change + what VAD sees. For advanced setups (for example, VAD-only pipelines or sharing + one analyzer across multiple aggregators) a standalone `VADProcessor` is also + available, but configuring `vad_analyzer` on the aggregator is the recommended + default. + + ### Key Parameters **`start_secs` (default: 0.2)** @@ -88,6 +101,16 @@ In the vast number of cases, the default values will work well. Only adjust thes optimal performance across different audio environments and use cases. + + `confidence` and `min_volume` are the in-VAD levers for noisy audio, but they + only raise the bar for what counts as speech. If your bot keeps reacting to + background voices or your STT keeps transcribing noise, remove the noise + upstream with an input audio filter such as [Krisp + VIVA](/pipecat/features/krisp-viva) or + [RNNoise](/api-reference/server/utilities/audio/rnnoise-filter) before the + audio reaches VAD and STT. + + ## User Turn Detection While VAD detects speech vs. silence, it can't understand linguistic context. A pause doesn't mean the user is done. User turn strategies interpret VAD signals and transcriptions to determine actual turn boundaries. diff --git a/pipecat/learn/text-to-speech.mdx b/pipecat/learn/text-to-speech.mdx index 48dc9e634..1eb580c1c 100644 --- a/pipecat/learn/text-to-speech.mdx +++ b/pipecat/learn/text-to-speech.mdx @@ -32,6 +32,7 @@ pipeline = Pipeline([ **TTS generates speech through two primary mechanisms:** 1. **Streamed LLM tokens** via `LLMTextFrame`s: + - By default, TTS aggregates streaming tokens into complete sentences before synthesis (`TextAggregationMode.SENTENCE`) - Set `text_aggregation_mode=TextAggregationMode.TOKEN` to stream tokens directly for lower latency - Audio bytes stream back and play immediately @@ -314,10 +315,10 @@ Use `TTSSpeakFrame` for immediate speech synthesis: ```python from pipecat.frames.frames import TTSSpeakFrame -# Make bot speak directly +# Make bot speak directly (added to context by default) await tts.queue_frame(TTSSpeakFrame("Hello, how can I help you?")) -# Append spoken text to conversation context +# Explicitly append spoken text to conversation context await tts.queue_frame( TTSSpeakFrame("Welcome! Let's begin.", append_to_context=True) ) @@ -330,6 +331,13 @@ await tts.queue_frame( The `append_to_context` parameter controls whether the spoken text is added to the conversation history. When `append_to_context=True`, the text is automatically committed to the context after being spoken, making it useful for bot greetings and injected speech that should be part of the conversation flow. + + As of Pipecat v1.4.0, `append_to_context` defaults to `True`. A plain + `TTSSpeakFrame("...")` **is** added to the conversation context after it is + spoken; pass `append_to_context=False` to speak without recording it. (`None` + was the previous default and is no longer supported.) + + ### Dynamic Settings Updates Update TTS settings during conversation using typed settings objects: diff --git a/pipecat/migration/migration-1.0.mdx b/pipecat/migration/migration-1.0.mdx index 6e434604a..9a6d3605f 100644 --- a/pipecat/migration/migration-1.0.mdx +++ b/pipecat/migration/migration-1.0.mdx @@ -12,6 +12,17 @@ Before upgrading, search your codebase for the deprecated imports and patterns l been dropped. Python 3.11 through 3.14 are supported. + + **Removed parameters are silently ignored, not errored.** `TransportParams` + and `PipelineParams` are Pydantic models that drop unknown fields, so leftover + 0.0.x params like `vad_analyzer`, `turn_analyzer`, and `allow_interruptions` + are accepted with no error or warning. Your app can start cleanly and look + upgraded while turn detection is actually broken (missed user speech, no bot + responses). Don't rely on the app starting: delete these params from your + transport/pipeline config and move them to `LLMUserAggregatorParams` as shown + below. + + ## 1. Universal LLMContext `LLMContext` is a universal context that works with all LLM providers. It allows you to dynamically switch `LLMService` at runtime, passing a compatible context between services. Pipecat maintains adapters for each LLM service, which are applied automatically at the time of inference. @@ -133,22 +144,31 @@ from pipecat.processors.aggregators.llm_response_universal import ( LLMUserAggregatorParams, ) from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3 +from pipecat.turns.user_start import ( + TranscriptionUserTurnStartStrategy, + VADUserTurnStartStrategy, +) +from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy from pipecat.turns.user_turn_strategies import UserTurnStrategies # Everything configured on the aggregator user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( - user_turn_strategies=[ - UserTurnStrategies( - start=[ - # Start strategies - ], - stop=[ - # Stop strategies - ], - ), - ], + user_turn_strategies=UserTurnStrategies( + start=[ + # Interruptions are ON by default. To disable, pass + # enable_interruptions=False to the start strategy. + VADUserTurnStartStrategy(), + TranscriptionUserTurnStartStrategy(), + ], + stop=[ + TurnAnalyzerUserTurnStopStrategy( + turn_analyzer=LocalSmartTurnAnalyzerV3() + ), + ], + ), user_mute_strategies=[ # Mute strategies ], @@ -159,17 +179,27 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( ) ``` + + **Where did `allow_interruptions` go?** Interruptions are now controlled per + start strategy via `enable_interruptions` (enabled by default). To turn + interruptions off, replace `allow_interruptions=False` with + `VADUserTurnStartStrategy(enable_interruptions=False)` in your `start` + strategies. See the [User Turn Strategies + reference](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies) + for all start-strategy options. + + ### What was removed -| Removed | Replacement | -| ---------------------------------------- | ------------------------------------------------------------ | -| `PipelineParams.allow_interruptions` | `user_turn_strategies` on `LLMUserAggregatorParams` | -| `PipelineParams.interruption_strategies` | `user_turn_strategies` on `LLMUserAggregatorParams` | -| `UserResponseAggregator` | `LLMUserAggregator` (created via `LLMContextAggregatorPair`) | -| `UserIdleProcessor` | `user_idle_timeout` on `LLMUserAggregatorParams` | -| `STTMuteFilter` | `user_mute_strategies` on `LLMUserAggregatorParams` | -| `MinWordsInterruptionStrategy` | `MinWordsUserTurnStartStrategy` | -| `TranscriptionUserTurnStopStrategy` | `SpeechTimeoutUserTurnStopStrategy` | +| Removed | Replacement | +| ---------------------------------------- | -------------------------------------------------------------------- | +| `PipelineParams.allow_interruptions` | `enable_interruptions` on a start strategy in `user_turn_strategies` | +| `PipelineParams.interruption_strategies` | `user_turn_strategies` on `LLMUserAggregatorParams` | +| `UserResponseAggregator` | `LLMUserAggregator` (created via `LLMContextAggregatorPair`) | +| `UserIdleProcessor` | `user_idle_timeout` on `LLMUserAggregatorParams` | +| `STTMuteFilter` | `user_mute_strategies` on `LLMUserAggregatorParams` | +| `MinWordsInterruptionStrategy` | `MinWordsUserTurnStartStrategy` | +| `TranscriptionUserTurnStopStrategy` | `SpeechTimeoutUserTurnStopStrategy` | ## 3. VAD & Turn Analyzer Configuration @@ -296,7 +326,7 @@ Several frame classes were renamed or removed. Update any direct references in y | `InputTransportMessageUrgentFrame` | `InputTransportMessageFrame` | | `KeypadEntryFrame` | `DTMFFrame` | | `StartInterruptionFrame` | `InterruptionFrame` | -| `BotInterruptionFrame` | `InterruptionWorkerFrame` | +| `BotInterruptionFrame` | `InterruptionWorkerFrame` | | `TranscriptionMessage` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `TranscriptionUpdateFrame` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `DailyTransportMessageFrame` | `DailyOutputTransportMessageFrame` | @@ -354,7 +384,7 @@ Smaller breaking changes across the rest of the API. | `PipelineTask.on_pipeline_ended` | `on_pipeline_finished` | | `PipelineTask.on_pipeline_cancelled` | `on_pipeline_finished` | | `PipelineTask.on_pipeline_stopped` | `on_pipeline_finished` | -| `DailyRunner.configure_with_args()` | Use `WorkerRunner` with `RunnerArguments` | +| `DailyRunner.configure_with_args()` | Use `WorkerRunner` with `RunnerArguments` | ### TTS service From 5156c72d267aa77efdeb0f98697dc4538e2c5ab3 Mon Sep 17 00:00:00 2001 From: James Hush Date: Fri, 26 Jun 2026 16:47:20 +0800 Subject: [PATCH 2/3] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- pipecat/learn/context-management.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pipecat/learn/context-management.mdx b/pipecat/learn/context-management.mdx index 77ead2a50..7b670722b 100644 --- a/pipecat/learn/context-management.mdx +++ b/pipecat/learn/context-management.mdx @@ -238,8 +238,8 @@ await worker.queue_frames([ #### Adding a message silently -All three frames take a `run_llm` argument. Set `run_llm=False` (or leave it unset) to add or change the context **without** prompting the bot to respond. This is useful when you collect information in the background and don't want the bot to react every time: - +All three frames take a `run_llm` argument. Set `run_llm=False` to add or change the context **without** prompting the bot to respond. +(Leaving `run_llm` as `None` uses the context aggregator's default behavior.) This is useful when you collect information in the background and don't want the bot to react every time: ```python # Add a message to context without triggering a bot response note = {"role": "user", "content": "Caller's name is Maria. Account verified."} From 84b6aebf60e2268d86a9914147048f3b11b54e53 Mon Sep 17 00:00:00 2001 From: Mark Backman Date: Fri, 26 Jun 2026 09:50:19 -0400 Subject: [PATCH 3/3] Review feedback --- .../turn-detection/smart-turn-overview.mdx | 24 +-- .../turn-management/user-turn-strategies.mdx | 138 +++++------------- pipecat/learn/context-management.mdx | 4 +- pipecat/learn/pipeline-termination.mdx | 24 ++- pipecat/learn/speech-input.mdx | 29 +--- pipecat/migration/migration-1.0.mdx | 33 ++--- 6 files changed, 87 insertions(+), 165 deletions(-) diff --git a/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx b/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx index b5c275a1c..55c9def67 100644 --- a/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx +++ b/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx @@ -98,26 +98,14 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( The `SmartTurnParams` class configures turn detection behavior: - Duration of silence in seconds required before triggering a silence-based end - of turn - - - - **This is not the same `stop_secs` as your VAD's.** There are two: +Duration of silence in seconds required before triggering a silence-based end +of turn. -- **VAD `stop_secs`** (default `0.2`, on `VADParams`): how long silence must - last before VAD declares the user stopped speaking. This is what - _triggers_ a Smart Turn analysis. -- **`SmartTurnParams.stop_secs`** (default `3.0`, here): a silence-based - _fallback_. If the model keeps classifying the turn as incomplete, this is - how long to wait before force-completing the turn anyway. +Note: This value is different from the VAD's `stop_secs` parameter. +Set the `SmartTurnParams.stop_secs` to specify how long the Turn Analyzer should +wait before classifying the turn as complete. -Lowering VAD `stop_secs` makes turn-taking more responsive; changing -`SmartTurnParams.stop_secs` only affects how long an unresolved turn waits -before the fallback fires, which is why tuning it often shows little effect -in normal conversation. - - + Amount of audio (in milliseconds) to include before speech is detected diff --git a/api-reference/server/utilities/turn-management/user-turn-strategies.mdx b/api-reference/server/utilities/turn-management/user-turn-strategies.mdx index fd262be0b..3c48652dc 100644 --- a/api-reference/server/utilities/turn-management/user-turn-strategies.mdx +++ b/api-reference/server/utilities/turn-management/user-turn-strategies.mdx @@ -15,16 +15,15 @@ By default, Pipecat uses a combination of VAD (Voice Activity Detection) and AI- You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models. - The user turn strategies API is new in Pipecat 1.x. If you're coming from - 0.0.x (where turn detection was configured via `turn_analyzer` on the - transport params), see the [migration guide](/pipecat/migration/migration-1.0) - for where each setting moved. + In Pipecat >=1.0.0, the `turn_analyzer` must be specified in the + `TurnAnalyzerUserTurnStopStrategy`. See the [migration + guide](/pipecat/migration/migration-1.0#3-vad-&-turn-analyzer-configuration) + for more information. ## How It Works 1. **Turn Start Detection**: When any start strategy triggers, the user aggregator: - - Marks the start of a user turn - Optionally emits `UserStartedSpeakingFrame` - Optionally emits an interruption frame (if the bot is speaking) @@ -32,31 +31,55 @@ You can customize this behavior by providing your own strategies for more sophis 2. **During User Turn**: The aggregator collects transcriptions and audio frames. 3. **Turn Stop Detection**: When a stop strategy triggers, the user aggregator: - - Marks the end of the user turn - Emits `UserStoppedSpeakingFrame` - Pushes the aggregated user message to the LLM context 4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended. This timeout is configurable via `LLMUserAggregatorParams` (see [Configuration](#configuration) below). When the timeout fires, the [`on_user_turn_stop_timeout`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_stop_timeout) event is emitted. -### Turn timing timeline +### When a user turn ends + +A **user turn stop strategy** decides when the user is done talking and the bot should respond. You choose the strategy; the rest of the timing follows from it. **If you want to adjust how long the bot waits before responding, do it in the stop strategy** — that's the knob built for it. + +Whatever the strategy, closing a turn takes two ingredients: + +1. **A transcript.** Every stop strategy waits for the STT service to transcribe what the user said (`wait_for_transcript`, on by default). This is usually the largest and most variable part of the delay, and it's dominated by your STT provider's latency — see the [STT benchmark](https://github.com/pipecat-ai/stt-benchmark) to compare services. +2. **The strategy's own end-of-turn criteria.** Once the transcript is in, the strategy applies its logic to decide whether the turn is actually complete. + +The turn closes only when both are satisfied. + +#### Choosing a strategy + +| Strategy | How it decides the turn is complete | +| ------------------------------------ | --------------------------------------------------------------------------------------------- | +| `SpeechTimeoutUserTurnStopStrategy` | A fixed silence window (`user_speech_timeout`, default `0.6s`) elapses after the user pauses. | +| `TurnAnalyzerUserTurnStopStrategy` | A turn-detection ("smart turn") model predicts end-of-turn from the audio and transcript. | +| `FilterIncompleteUserTurnStrategies` | An LLM assesses whether the user's utterance is semantically complete. | +| Custom | Your own logic implementing the stop-strategy interface. | -Several timers combine to decide when a user turn ends. They fire in this order after the user stops talking: +This is also where you tune responsiveness. For example, raise or lower `user_speech_timeout` on `SpeechTimeoutUserTurnStopStrategy` to give users more or less time to resume before the bot replies. -1. **VAD `stop_secs`**: The VAD waits this long (default `0.2s`) after audio goes quiet before it reports the user stopped speaking. Configured on your `VADParams`, not here. -2. **`user_speech_timeout`** (on `SpeechTimeoutUserTurnStopStrategy`, default `0.6s`): The policy floor. After VAD silence, this is the window in which the user may resume speaking. It always runs to completion. -3. **STT finalization / `ttfs_p99_latency`** _or_ **smart-turn analysis**: A safety net for STT latency. With `SpeechTimeoutUserTurnStopStrategy`, the turn waits for the STT service to return a final transcript (short-circuited when the STT service emits a finalized transcript). With `TurnAnalyzerUserTurnStopStrategy`, the smart-turn model decides end-of-turn instead. -4. **Turn finalized**: Once the active stop strategy's conditions are met, the turn ends and the bot responds. +#### Where the latency comes from -`user_turn_stop_timeout` (default `5.0s`, on `LLMUserAggregatorParams`) sits outside all of this as an outer watchdog: if none of the above finalizes the turn in time, it forces the turn to end. +Before any strategy can act, two things have to happen: + +- **VAD silence (`stop_secs`):** The VAD waits a short, fixed interval (default `0.2s`) after audio goes quiet before reporting that the user stopped speaking. This is a low-level detection threshold — leave it alone. To change wait time, use the stop strategy, not `stop_secs`. +- **Transcription:** STT returns a transcript. Latency here is mostly your STT provider's; strategies use the provider's reported p99 latency as a fallback timer and short-circuit it the moment a finalized transcript arrives. + +Then the strategy's criteria run (the silence window, the turn model, or the LLM check) to finalize the turn. - The extra silence some users notice (for example, "the aggregator waits an - extra 800ms") is usually steps 2 and 3 stacking: `stop_secs` plus - `user_speech_timeout` plus the STT wait. Tune each timer independently rather - than expecting one value to control total latency. + Extra silence users notice usually isn't one tunable value — it's VAD silence + plus transcription latency plus the strategy's own window stacking up. If your + STT service is slow to return transcripts, that delay shows up in every turn + regardless of which strategy you use. Adjust responsiveness in the stop + strategy, and check the [STT + benchmark](https://github.com/pipecat-ai/stt-benchmark) if transcription is + the bottleneck. +`user_turn_stop_timeout` (default `5.0s`, on `LLMUserAggregatorParams`) is a backstop, not part of normal timing: if a turn somehow never finalizes, it forces the turn to end so the bot isn't stuck waiting. + ## Configuration User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`: @@ -491,87 +514,6 @@ stop = [ ] ``` -### Custom stop strategy - -Pipecat has no built-in strategy for capping how long a single user turn can run (there is no "max speech" or "max turn duration" strategy). If you need to end a turn after a fixed amount of time, for example to stop a customer from monologuing, write your own stop strategy by subclassing `BaseUserTurnStopStrategy`. - -A stop strategy inspects incoming frames in `process_frame` and calls `trigger_user_turn_stopped()` when it decides the turn is over. The example below starts a timer when the user begins speaking and forces the turn to end once the timer fires: - -```python -import asyncio - -from pipecat.frames.frames import Frame, VADUserStartedSpeakingFrame -from pipecat.turns.types import ProcessFrameResult -from pipecat.turns.user_stop import BaseUserTurnStopStrategy - - -class MaxTurnDurationStopStrategy(BaseUserTurnStopStrategy): - """End the user turn after a fixed number of seconds. - - Starts a timer when the user starts speaking. If the user is still - holding the turn when the timer fires, the turn is finalized so the - bot can respond. - """ - - def __init__(self, *, max_turn_seconds: float = 10.0, **kwargs): - super().__init__(**kwargs) - self._max_turn_seconds = max_turn_seconds - self._timer_task: asyncio.Task | None = None - - async def process_frame(self, frame: Frame) -> ProcessFrameResult: - if isinstance(frame, VADUserStartedSpeakingFrame) and self._timer_task is None: - self._timer_task = self.task_manager.create_task( - self._max_turn_handler(), - f"{self}::_max_turn_handler", - ) - return ProcessFrameResult.CONTINUE - - async def _max_turn_handler(self): - try: - await asyncio.sleep(self._max_turn_seconds) - except asyncio.CancelledError: - return - finally: - self._timer_task = None - # Time is up: end the turn so the bot responds. - await self.trigger_user_turn_stopped() - - async def reset(self): - await super().reset() - await self._cancel_timer() - - async def cleanup(self): - await super().cleanup() - await self._cancel_timer() - - async def _cancel_timer(self): - if self._timer_task: - await self.task_manager.cancel_task(self._timer_task) - self._timer_task = None -``` - -Pair it with a normal detector so regular turn-end still works. The first strategy to trigger ends the turn, so the timer acts as a backstop on top of smart-turn detection: - -```python -from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3 -from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy -from pipecat.turns.user_turn_strategies import UserTurnStrategies - -strategies = UserTurnStrategies( - stop=[ - TurnAnalyzerUserTurnStopStrategy(turn_analyzer=LocalSmartTurnAnalyzerV3()), - MaxTurnDurationStopStrategy(max_turn_seconds=10.0), - ], -) -``` - - - Ending the turn lets the bot respond, but it does not stop the user's audio - from being processed if they keep talking. To also ignore further user input - for a window, combine this with [User Input - Muting](/pipecat/fundamentals/user-input-muting). - - ## Helper Functions Pipecat provides helper functions to compose custom strategy lists that extend the defaults. diff --git a/pipecat/learn/context-management.mdx b/pipecat/learn/context-management.mdx index 7b670722b..04bf3e846 100644 --- a/pipecat/learn/context-management.mdx +++ b/pipecat/learn/context-management.mdx @@ -238,8 +238,8 @@ await worker.queue_frames([ #### Adding a message silently -All three frames take a `run_llm` argument. Set `run_llm=False` to add or change the context **without** prompting the bot to respond. -(Leaving `run_llm` as `None` uses the context aggregator's default behavior.) This is useful when you collect information in the background and don't want the bot to react every time: +All three frames take a `run_llm` argument that controls whether the change also prompts a bot response. Pass `run_llm=True` to respond; the default (`None`, which behaves like `False`) updates the context silently. This is useful when you collect information in the background and don't want the bot to react every time: + ```python # Add a message to context without triggering a bot response note = {"role": "user", "content": "Caller's name is Maria. Account verified."} diff --git a/pipecat/learn/pipeline-termination.mdx b/pipecat/learn/pipeline-termination.mdx index 9f8b22568..8e756507d 100644 --- a/pipecat/learn/pipeline-termination.mdx +++ b/pipecat/learn/pipeline-termination.mdx @@ -66,19 +66,25 @@ Push an `EndFrame` from outside your pipeline: ```python # From outside the pipeline -from pipecat.frames.frames import EndFrame, TTSSpeakFrame +from pipecat.frames.frames import EndFrame await worker.queue_frame(EndFrame()) ``` -Push an `EndWorkerFrame` downstream from inside your pipeline: +Push an `EndWorkerFrame` downstream from inside your pipeline. Here `end_conversation` is a **direct function** — an `async` function whose first parameter is `params: FunctionCallParams`, with a Google-style docstring that becomes the tool description. Register it by passing the function itself in your tools list: ```python -# From inside a function call from pipecat.frames.frames import EndWorkerFrame, TTSSpeakFrame +from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.frame_processor import FrameDirection +from pipecat.services.llm_service import FunctionCallParams + async def end_conversation(params: FunctionCallParams): + """End the conversation and shut down the bot. + + Call this when the user says goodbye or the task is complete. + """ await params.llm.push_frame(TTSSpeakFrame("Have a nice day!")) # Resolve the function call so the LLM call doesn't hang @@ -87,13 +93,15 @@ async def end_conversation(params: FunctionCallParams): # Signal that the worker should end after processing this frame await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM) -# Register the handler so the LLM can call it as a tool -llm.register_function("end_conversation", end_conversation) + +# Pass the function directly in the tools list; it's registered automatically +context = LLMContext(tools=[end_conversation]) ``` Always call `params.result_callback(...)` in your handler before pushing the - end frame. Skipping it can leave the LLM function call unresolved. + end frame. Skipping it can leave the LLM function call unresolved. If you + don't want the LLM to respond, you can provide `None` as the result. **How graceful termination works:** @@ -161,8 +169,8 @@ worker = PipelineWorker( pipeline, # Configure idle detection timeout cancel_on_idle_timeout=True, # Default is True - idle_timeout_seconds=600, # Default is 300 seconds - idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, LLMFullResponseEndFrame) + idle_timeout_secs=600, # Default is 300 seconds + idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, UserSpeakingFrame) ) ``` diff --git a/pipecat/learn/speech-input.mdx b/pipecat/learn/speech-input.mdx index 571aa97e1..eee04be6d 100644 --- a/pipecat/learn/speech-input.mdx +++ b/pipecat/learn/speech-input.mdx @@ -57,24 +57,11 @@ vad_analyzer = SileroVADAnalyzer( user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, - user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer)), + user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer), ) ``` -In the vast number of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it. - - - **Why is VAD configured on the aggregator?** VAD lives in - `LLMUserAggregatorParams` because its speech start/stop signals feed the [user - turn - strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) - that decide when a turn begins and ends. VAD runs on the raw input audio, so - where the aggregator sits in your pipeline relative to STT does not change - what VAD sees. For advanced setups (for example, VAD-only pipelines or sharing - one analyzer across multiple aggregators) a standalone `VADProcessor` is also - available, but configuring `vad_analyzer` on the aggregator is the recommended - default. - +VAD is configured on the user aggregator because its speech start/stop signals feed the [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) that decide when a turn begins and ends. In the vast majority of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it. ### Key Parameters @@ -102,12 +89,12 @@ In the vast number of cases, the default values will work well. Only adjust thes - `confidence` and `min_volume` are the in-VAD levers for noisy audio, but they - only raise the bar for what counts as speech. If your bot keeps reacting to - background voices or your STT keeps transcribing noise, remove the noise - upstream with an input audio filter such as [Krisp - VIVA](/pipecat/features/krisp-viva) or - [RNNoise](/api-reference/server/utilities/audio/rnnoise-filter) before the + `confidence` and `min_volume` only raise the bar for what counts as speech — + blunt instruments for noisy audio. If your bot reacts to background voices or + your STT transcribes noise, it's usually better to remove the noise upstream + with an input audio filter — [Krisp VIVA](/pipecat/features/krisp-viva), + [ai-coustic](/api-reference/server/utilities/audio/aic-filter), or + [RNNoise](/api-reference/server/utilities/audio/rnnoise-filter) — before the audio reaches VAD and STT. diff --git a/pipecat/migration/migration-1.0.mdx b/pipecat/migration/migration-1.0.mdx index 9a6d3605f..93187dbca 100644 --- a/pipecat/migration/migration-1.0.mdx +++ b/pipecat/migration/migration-1.0.mdx @@ -158,8 +158,6 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( user_params=LLMUserAggregatorParams( user_turn_strategies=UserTurnStrategies( start=[ - # Interruptions are ON by default. To disable, pass - # enable_interruptions=False to the start strategy. VADUserTurnStartStrategy(), TranscriptionUserTurnStartStrategy(), ], @@ -180,26 +178,25 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( ``` - **Where did `allow_interruptions` go?** Interruptions are now controlled per - start strategy via `enable_interruptions` (enabled by default). To turn - interruptions off, replace `allow_interruptions=False` with - `VADUserTurnStartStrategy(enable_interruptions=False)` in your `start` - strategies. See the [User Turn Strategies - reference](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies) - for all start-strategy options. + **Where did `allow_interruptions` go?** The `allow_interruptions` parameter + was a legacy parameter used for non-voice AI applications. If you're looking + to prevent an interruption from occurring, consider using a user mute strategy + instead. See the [User Mute Strategies + reference](/api-reference/server/utilities/turn-management/user-mute-strategies) + for more information. ### What was removed -| Removed | Replacement | -| ---------------------------------------- | -------------------------------------------------------------------- | -| `PipelineParams.allow_interruptions` | `enable_interruptions` on a start strategy in `user_turn_strategies` | -| `PipelineParams.interruption_strategies` | `user_turn_strategies` on `LLMUserAggregatorParams` | -| `UserResponseAggregator` | `LLMUserAggregator` (created via `LLMContextAggregatorPair`) | -| `UserIdleProcessor` | `user_idle_timeout` on `LLMUserAggregatorParams` | -| `STTMuteFilter` | `user_mute_strategies` on `LLMUserAggregatorParams` | -| `MinWordsInterruptionStrategy` | `MinWordsUserTurnStartStrategy` | -| `TranscriptionUserTurnStopStrategy` | `SpeechTimeoutUserTurnStopStrategy` | +| Removed | Replacement | +| ---------------------------------------- | ------------------------------------------------------------ | +| `PipelineParams.allow_interruptions` | `user_turn_strategies` on `LLMUserAggregatorParams` | +| `PipelineParams.interruption_strategies` | `user_turn_strategies` on `LLMUserAggregatorParams` | +| `UserResponseAggregator` | `LLMUserAggregator` (created via `LLMContextAggregatorPair`) | +| `UserIdleProcessor` | `user_idle_timeout` on `LLMUserAggregatorParams` | +| `STTMuteFilter` | `user_mute_strategies` on `LLMUserAggregatorParams` | +| `MinWordsInterruptionStrategy` | `MinWordsUserTurnStartStrategy` | +| `TranscriptionUserTurnStopStrategy` | `SpeechTimeoutUserTurnStopStrategy` | ## 3. VAD & Turn Analyzer Configuration