diff --git a/api-reference/server/services/stt/deepgram.mdx b/api-reference/server/services/stt/deepgram.mdx index b94e2e721..80958d1b4 100644 --- a/api-reference/server/services/stt/deepgram.mdx +++ b/api-reference/server/services/stt/deepgram.mdx @@ -319,7 +319,7 @@ Runtime-configurable settings passed via the `settings` constructor argument usi ### Usage ```python -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService stt = DeepgramFluxSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), @@ -329,7 +329,7 @@ stt = DeepgramFluxSTTService( #### With EagerEndOfTurn ```python -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService stt = DeepgramFluxSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), @@ -344,7 +344,7 @@ stt = DeepgramFluxSTTService( #### Multilingual Support ```python -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService from pipecat.transcriptions.language import Language # Use flux-general-multi with language hints @@ -363,7 +363,7 @@ The `keyterm`, `eot_threshold`, `eager_eot_threshold`, `eot_timeout_ms`, and `la ```python from pipecat.frames.frames import STTUpdateSettingsFrame -from pipecat.services.deepgram.flux import DeepgramFluxSTTService +from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService from pipecat.transcriptions.language import Language # During pipeline execution, update settings without reconnecting diff --git a/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx b/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx index b71babceb..55c9def67 100644 --- a/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx +++ b/api-reference/server/utilities/turn-detection/smart-turn-overview.mdx @@ -98,8 +98,13 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( The `SmartTurnParams` class configures turn detection behavior: - Duration of silence in seconds required before triggering a silence-based end - of turn +Duration of silence in seconds required before triggering a silence-based end +of turn. + +Note: This value is different from the VAD's `stop_secs` parameter. +Set the `SmartTurnParams.stop_secs` to specify how long the Turn Analyzer should +wait before classifying the turn as complete. + diff --git a/api-reference/server/utilities/turn-management/user-turn-strategies.mdx b/api-reference/server/utilities/turn-management/user-turn-strategies.mdx index 2536edde3..3c48652dc 100644 --- a/api-reference/server/utilities/turn-management/user-turn-strategies.mdx +++ b/api-reference/server/utilities/turn-management/user-turn-strategies.mdx @@ -14,6 +14,13 @@ By default, Pipecat uses a combination of VAD (Voice Activity Detection) and AI- You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models. + + In Pipecat >=1.0.0, the `turn_analyzer` must be specified in the + `TurnAnalyzerUserTurnStopStrategy`. See the [migration + guide](/pipecat/migration/migration-1.0#3-vad-&-turn-analyzer-configuration) + for more information. + + ## How It Works 1. **Turn Start Detection**: When any start strategy triggers, the user aggregator: @@ -30,6 +37,49 @@ You can customize this behavior by providing your own strategies for more sophis 4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended. This timeout is configurable via `LLMUserAggregatorParams` (see [Configuration](#configuration) below). When the timeout fires, the [`on_user_turn_stop_timeout`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_stop_timeout) event is emitted. +### When a user turn ends + +A **user turn stop strategy** decides when the user is done talking and the bot should respond. You choose the strategy; the rest of the timing follows from it. **If you want to adjust how long the bot waits before responding, do it in the stop strategy** — that's the knob built for it. + +Whatever the strategy, closing a turn takes two ingredients: + +1. **A transcript.** Every stop strategy waits for the STT service to transcribe what the user said (`wait_for_transcript`, on by default). This is usually the largest and most variable part of the delay, and it's dominated by your STT provider's latency — see the [STT benchmark](https://github.com/pipecat-ai/stt-benchmark) to compare services. +2. **The strategy's own end-of-turn criteria.** Once the transcript is in, the strategy applies its logic to decide whether the turn is actually complete. + +The turn closes only when both are satisfied. + +#### Choosing a strategy + +| Strategy | How it decides the turn is complete | +| ------------------------------------ | --------------------------------------------------------------------------------------------- | +| `SpeechTimeoutUserTurnStopStrategy` | A fixed silence window (`user_speech_timeout`, default `0.6s`) elapses after the user pauses. | +| `TurnAnalyzerUserTurnStopStrategy` | A turn-detection ("smart turn") model predicts end-of-turn from the audio and transcript. | +| `FilterIncompleteUserTurnStrategies` | An LLM assesses whether the user's utterance is semantically complete. | +| Custom | Your own logic implementing the stop-strategy interface. | + +This is also where you tune responsiveness. For example, raise or lower `user_speech_timeout` on `SpeechTimeoutUserTurnStopStrategy` to give users more or less time to resume before the bot replies. + +#### Where the latency comes from + +Before any strategy can act, two things have to happen: + +- **VAD silence (`stop_secs`):** The VAD waits a short, fixed interval (default `0.2s`) after audio goes quiet before reporting that the user stopped speaking. This is a low-level detection threshold — leave it alone. To change wait time, use the stop strategy, not `stop_secs`. +- **Transcription:** STT returns a transcript. Latency here is mostly your STT provider's; strategies use the provider's reported p99 latency as a fallback timer and short-circuit it the moment a finalized transcript arrives. + +Then the strategy's criteria run (the silence window, the turn model, or the LLM check) to finalize the turn. + + + Extra silence users notice usually isn't one tunable value — it's VAD silence + plus transcription latency plus the strategy's own window stacking up. If your + STT service is slow to return transcripts, that delay shows up in every turn + regardless of which strategy you use. Adjust responsiveness in the stop + strategy, and check the [STT + benchmark](https://github.com/pipecat-ai/stt-benchmark) if transcription is + the bottleneck. + + +`user_turn_stop_timeout` (default `5.0s`, on `LLMUserAggregatorParams`) is a backstop, not part of normal timing: if a turn somehow never finalizes, it forces the turn to end so the bot isn't stuck waiting. + ## Configuration User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`: @@ -74,6 +124,14 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( ) ``` + + The Voice Activity Detection analyzer instance used to produce VAD signals for + turn detection. In Pipecat 1.x, VAD is configured on the aggregator rather + than the transport. This is separate from `VADUserTurnStartStrategy`: the + analyzer _produces_ the VAD signals; the start strategy decides _what to do_ + with them. + + ## Start Strategies Start strategies determine when a user's turn begins. Multiple strategies can be provided, and the first one to trigger will signal the start of a user turn. @@ -317,11 +375,12 @@ strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6) Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the recommended default). If you change `stop_secs`, the strategy will log a - warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) - with your VAD settings and pass the measured TTFS P99 latency to your STT - service constructor via `ttfs_p99_latency`. The strategy will also warn if - `stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s - and may cause delayed turn detection. + warning suggesting you re-run the + [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD + settings and pass the measured TTFS P99 latency to your STT service + constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs + >= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause + delayed turn detection. ### TurnAnalyzerUserTurnStopStrategy @@ -359,11 +418,12 @@ strategy = TurnAnalyzerUserTurnStopStrategy( Built-in STT P99 latency values assume `VADParams.stop_secs=0.2` (the recommended default). If you change `stop_secs`, the strategy will log a - warning suggesting you re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) - with your VAD settings and pass the measured TTFS P99 latency to your STT - service constructor via `ttfs_p99_latency`. The strategy will also warn if - `stop_secs >= STT p99 latency`, which collapses the STT wait timeout to 0s - and may cause delayed turn detection. + warning suggesting you re-run the + [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your VAD + settings and pass the measured TTFS P99 latency to your STT service + constructor via `ttfs_p99_latency`. The strategy will also warn if `stop_secs + >= STT p99 latency`, which collapses the STT wait timeout to 0s and may cause + delayed turn detection. ### ExternalUserTurnStopStrategy diff --git a/client/concepts/choosing-a-transport.mdx b/client/concepts/choosing-a-transport.mdx index d3e18fd0b..cb44f6717 100644 --- a/client/concepts/choosing-a-transport.mdx +++ b/client/concepts/choosing-a-transport.mdx @@ -10,19 +10,39 @@ The choice of transport has one important constraint: **the client transport and ## Available transports - + Serverless peer-to-peer WebRTC. No third-party account needed. - + WebRTC via Daily's global infrastructure. Recommended for production. - + Direct WebSocket connection. For server-to-server setups only. - + Direct connection to Google's Gemini Live API. No Pipecat server needed. - + Direct connection to OpenAI's Realtime API. No Pipecat server needed. @@ -50,7 +70,9 @@ Once you've settled on WebRTC, the next question is how you route it. There are A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's simpler than running your own WebRTC infrastructure and avoids a third-party dependency. If you want managed infrastructure that handles global routing, audio processing, and scaling for you, use Daily. - For a deeper look at this tradeoff, see the Daily blog post [You don't need a WebRTC server for your voice agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/). + For a deeper look at this tradeoff, see the Daily blog post [You don't need a + WebRTC server for your voice + agents](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/). --- @@ -62,11 +84,13 @@ A useful way to think about it: if you're self-hosting, use SmallWebRTC — it's SmallWebRTC is the default in all Pipecat quickstart templates. No account needed, no third-party services — just a direct peer-to-peer WebRTC connection between your client and your bot. **Use it when:** + - Developing or testing locally - Running a self-hosted deployment - Building embedded or edge deployments where simplicity matters **Avoid it when:** + - Your users are geographically distributed - You need built-in echo cancellation, noise reduction, or network resilience at scale - You're scaling beyond a handful of concurrent sessions @@ -89,6 +113,7 @@ await client.connect({ webrtcUrl: "http://localhost:7860/api/offer" }); Daily provides a global WebRTC network with mesh routing, built-in audio processing (echo cancellation, noise suppression, automatic gain control), and resilience to network changes. It's the recommended choice for anything user-facing in production. **Use it when:** + - Shipping a production app to real users - Your users are on a variety of networks, devices, or locations - You want managed infrastructure without operating it yourself @@ -114,11 +139,13 @@ await client.startBotAndConnect({ endpoint: "/api/start" }); The WebSocket transport connects to a WebSocket server rather than using WebRTC. It's appropriate for controlled, server-to-server scenarios where both sides are on stable networks, or for text-only bots with no audio requirements. **Use it when:** + +- Integrating telephony media streams (Twilio, Telnyx). This is the most common production WebSocket use case: the phone provider streams audio to your server over a WebSocket, which is a provider-to-server connection, not a browser client. - Building text-only interactions (no audio) - Connecting two servers (not a browser client) - Network constraints make WebRTC impractical in a specific environment -**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above. +**Do not use it for browser-to-server voice interactions.** See [WebRTC vs WebSocket](#webrtc-vs-websocket) above. The telephony case is different: the audio comes from a phone network over the provider's media-stream WebSocket, not from a browser, so the WebRTC advantages for browser audio don't apply. --- @@ -127,11 +154,22 @@ The WebSocket transport connects to a WebSocket server rather than using WebRTC. These transports connect your client directly to Google's Gemini Live API or OpenAI's Realtime API, bypassing a Pipecat server entirely. They're useful for prototyping or demos when you want speech-to-speech interactions without running a backend. - Direct API transports expose your API key in the client. They are appropriate for development and demos, but **not for production** — use a server-side Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API keys secure. + Direct API transports expose your API key in the client. They are appropriate + for development and demos, but **not for production** — use a server-side + Pipecat pipeline with `DailyTransport` or `SmallWebRTCTransport` to keep API + keys secure. --- +### What about LiveKit? + +Pipecat ships a [LiveKit transport for the **server**](/api-reference/server/services/transport/livekit), so your bot can join a LiveKit room. There is **no** official Pipecat **client** transport for LiveKit (JS/React, React Native, or otherwise). + +Because the client and server transports must be a matching pair, you cannot pair a LiveKit server transport with a Pipecat client SDK today. If you want a self-hostable, production-grade WebRTC transport that you control end to end with Pipecat clients, use `SmallWebRTCTransport`; if you want managed WebRTC infrastructure, use Daily. + +--- + ## Swapping transports Transports are interchangeable — the rest of your application code stays the same. The only thing that changes is the import and constructor: @@ -153,10 +191,10 @@ The Pipecat CLI scaffolds a `config.ts` that selects the transport based on an e ## Summary -| Transport | Best for | Requires | -|---|---|---| -| SmallWebRTC | Local dev, self-hosted | Nothing | -| Daily | Production apps, global users | Daily account | -| WebSocket | Text-only, server-to-server | Custom server | -| Gemini Live | Gemini prototypes | Gemini API key | -| OpenAI WebRTC | OpenAI prototypes | OpenAI API key | +| Transport | Best for | Requires | +| ------------- | ----------------------------- | -------------- | +| SmallWebRTC | Local dev, self-hosted | Nothing | +| Daily | Production apps, global users | Daily account | +| WebSocket | Text-only, server-to-server | Custom server | +| Gemini Live | Gemini prototypes | Gemini API key | +| OpenAI WebRTC | OpenAI prototypes | OpenAI API key | diff --git a/pipecat/learn/context-management.mdx b/pipecat/learn/context-management.mdx index b5c65e43f..04bf3e846 100644 --- a/pipecat/learn/context-management.mdx +++ b/pipecat/learn/context-management.mdx @@ -226,6 +226,7 @@ You can programmatically add new messages to the context by pushing or queueing - **`LLMMessagesAppendFrame`**: Appends a new message to the existing context - **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages +- **`LLMMessagesTransformFrame`**: Edits the existing context in place using a transform function ```python # Add a new user message to context and trigger a response @@ -235,6 +236,41 @@ await worker.queue_frames([ ]) ``` +#### Adding a message silently + +All three frames take a `run_llm` argument that controls whether the change also prompts a bot response. Pass `run_llm=True` to respond; the default (`None`, which behaves like `False`) updates the context silently. This is useful when you collect information in the background and don't want the bot to react every time: + +```python +# Add a message to context without triggering a bot response +note = {"role": "user", "content": "Caller's name is Maria. Account verified."} +await worker.queue_frames([ + LLMMessagesAppendFrame([note], run_llm=False), # Silent: no response +]) +``` + +#### Editing or removing specific messages + +To surgically edit or remove individual messages without rebuilding the whole list, push an `LLMMessagesTransformFrame`. It takes a function that receives the current list of messages and returns a modified list. Use it to drop stale instructions, remove an offensive turn, or rewrite content: + +```python +from pipecat.frames.frames import LLMMessagesTransformFrame + +def remove_language_instructions(messages): + # Drop any message that contains a per-turn language instruction + return [m for m in messages if "LANGUAGE INSTRUCTION" not in str(m.get("content", ""))] + +await worker.queue_frames([ + LLMMessagesTransformFrame(transform=remove_language_instructions, run_llm=False), +]) +``` + + + To make the bot **speak** specific text and have it recorded in context (for + example, a greeting the LLM did not generate), use [`TTSSpeakFrame` with + `append_to_context=True`](/pipecat/learn/text-to-speech). That path is for + driving speech output; the frames above are for editing the context directly. + + ### Retrieving Current Context The context aggregator provides a `context` property for getting the current context: diff --git a/pipecat/learn/function-calling.mdx b/pipecat/learn/function-calling.mdx index 3a7066c7e..524da4e1a 100644 --- a/pipecat/learn/function-calling.mdx +++ b/pipecat/learn/function-calling.mdx @@ -173,6 +173,35 @@ llm = OpenAILLMService( When `enable_async_tool_cancellation=True` and at least one async function is available, Pipecat automatically adds the built-in `cancel_async_tool_call` tool and supporting system instructions. The LLM can call that tool to cancel a stale in-progress async function call — for example, when the user changes their request before a long-running lookup completes. +## Parallel and Multiple Tool Calls + +When the LLM calls more than one tool in a single turn, two `LLMService` constructor options control how those calls run and when the LLM responds: + +```python +llm = OpenAILLMService( + api_key="your-api-key", + run_in_parallel=True, # default + group_parallel_tools=True, # default +) +``` + + + Whether multiple tool calls in one turn run in parallel or one after another. + + + + When `True`, all tool calls in a batch are grouped so the LLM is triggered + **exactly once** after every call in the batch completes. When `False`, each + function call result triggers the LLM independently as it arrives. + + + + If multiple tools firing in one turn produce **duplicate or repeated + responses** (the bot answers once per tool instead of once total), check that + `group_parallel_tools` is `True`. With it disabled, each result re-triggers + the LLM, so the model responds once for every tool call in the batch. + + ## Changing Tools Mid-Conversation To change the set of tools the LLM can use during a session, push an `LLMSetToolsFrame`. Its `tools` field takes the same things as `LLMContext(tools=[...])` — a list of direct functions and/or `FunctionSchema` objects. Whatever you pass becomes the LLM's new tool set. diff --git a/pipecat/learn/pipeline-termination.mdx b/pipecat/learn/pipeline-termination.mdx index 5131509e2..8e756507d 100644 --- a/pipecat/learn/pipeline-termination.mdx +++ b/pipecat/learn/pipeline-termination.mdx @@ -29,6 +29,23 @@ pipeline = Pipeline([ Both frames flow downstream through the pipeline, allowing each processor to clean up resources appropriately. +### Termination frames at a glance + +| Frame | Job | Push from / direction | +| ------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| `EndFrame` | Graceful shutdown (drains pending frames) | From outside the pipeline via `worker.queue_frame(EndFrame())` | +| `CancelFrame` | Immediate shutdown (discards pending frames) | Via `worker.cancel()` | +| `EndWorkerFrame` | Graceful shutdown signal from inside the pipeline | `push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to an `EndFrame` | +| `CancelWorkerFrame` | Immediate shutdown signal from inside the pipeline | `push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to a `CancelFrame` | + + + If you see `EndTaskFrame`, `CancelTaskFrame`, `PipelineTask`, or + `task.cancel()` in older code or examples, those are **deprecated aliases** + (since 1.3.0/1.4.0) of `EndWorkerFrame`, `CancelWorkerFrame`, + `PipelineWorker`, and `worker.cancel()`. They still work but will be removed + in 2.0.0. Use the `Worker` names in new code. + + ## Termination Methods Pipecat provides two primary approaches for pipeline termination, each designed for different scenarios: @@ -49,25 +66,44 @@ Push an `EndFrame` from outside your pipeline: ```python # From outside the pipeline -from pipecat.frames.frames import EndFrame, TTSSpeakFrame +from pipecat.frames.frames import EndFrame await worker.queue_frame(EndFrame()) ``` -Push an `EndWorkerFrame` downstream from inside your pipeline: +Push an `EndWorkerFrame` downstream from inside your pipeline. Here `end_conversation` is a **direct function** — an `async` function whose first parameter is `params: FunctionCallParams`, with a Google-style docstring that becomes the tool description. Register it by passing the function itself in your tools list: ```python -# From inside a function call from pipecat.frames.frames import EndWorkerFrame, TTSSpeakFrame +from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.frame_processor import FrameDirection +from pipecat.services.llm_service import FunctionCallParams + async def end_conversation(params: FunctionCallParams): + """End the conversation and shut down the bot. + + Call this when the user says goodbye or the task is complete. + """ await params.llm.push_frame(TTSSpeakFrame("Have a nice day!")) + # Resolve the function call so the LLM call doesn't hang + await params.result_callback({"status": "ended"}) + # Signal that the worker should end after processing this frame await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM) + + +# Pass the function directly in the tools list; it's registered automatically +context = LLMContext(tools=[end_conversation]) ``` + + Always call `params.result_callback(...)` in your handler before pushing the + end frame. Skipping it can leave the LLM function call unresolved. If you + don't want the LLM to respond, you can provide `None` as the result. + + **How graceful termination works:** 1. `EndFrame` is queued and processes after any pending frames (like goodbye messages) @@ -133,8 +169,8 @@ worker = PipelineWorker( pipeline, # Configure idle detection timeout cancel_on_idle_timeout=True, # Default is True - idle_timeout_seconds=600, # Default is 300 seconds - idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, LLMFullResponseEndFrame) + idle_timeout_secs=600, # Default is 300 seconds + idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, UserSpeakingFrame) ) ``` @@ -153,6 +189,31 @@ You can further configure the idle detection behavior. To learn more, refer to t being wasted on inactive conversations. +### Maximum Call Duration + +Idle detection ends a call that goes quiet, but it does not cap the total length of an active call. To enforce a maximum call duration, run an `asyncio` timer that speaks a goodbye and then queues an `EndFrame` so the bot can sign off gracefully before shutdown: + +```python +import asyncio +from pipecat.frames.frames import EndFrame, TTSSpeakFrame + +async def end_after(worker, timeout_secs: float): + await asyncio.sleep(timeout_secs) + await worker.queue_frame(TTSSpeakFrame("We've reached our time limit. Goodbye!")) + await worker.queue_frame(EndFrame()) # Graceful: plays the goodbye, then shuts down + +@transport.event_handler("on_client_connected") +async def on_client_connected(transport, client): + asyncio.create_task(end_after(worker, timeout_secs=300)) # 5-minute cap +``` + + + On Pipecat Cloud, there is also a platform-level hard cap via + [`maxSessionDuration`](/pipecat-cloud/fundamentals/active-sessions#session-duration-limits) + (default 7200s). That cap is a forced cut with no goodbye, so use the + bot-level timer above when you want the bot to speak before the call ends. + + ## Implementation Patterns ### Event-Driven Termination @@ -205,6 +266,19 @@ except Exception as e: await worker.cancel() ``` +### Running Cleanup Code on Shutdown + +To run cleanup or persist data when a call ends, use the `on_pipeline_finished` event handler. It fires after the pipeline reaches any terminal state, so it runs for **both** graceful (`EndFrame`) and cancelled (`CancelFrame`) shutdowns. This makes it the single write point for end-of-call work like saving a transcript or recording: + +```python +@worker.event_handler("on_pipeline_finished") +async def on_pipeline_finished(worker, frame): + # Runs for both graceful and cancelled shutdown + await save_transcript_to_db() +``` + +`on_client_disconnected`, by contrast, fires only when the client disconnects. Use it to _tag the reason_ for the shutdown (for example, "user hung up"), and do the actual persistence in `on_pipeline_finished` so you write data exactly once regardless of how the call ended. See the [`PipelineWorker` events](/api-reference/server/pipeline/pipeline-worker) reference for the full event signature. + ## Troubleshooting If your pipeline isn't shutting down properly, check these common issues: @@ -245,6 +319,12 @@ await self.push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM) termination handling throughout the pipeline. +### "dangling tasks detected" Warning + +**Problem:** On shutdown you see a log warning like `PipelineWorker#0 dangling tasks detected: [...]`. + +**Solution:** This means `asyncio` tasks created during the session were never cancelled or awaited before the pipeline shut down. The usual causes are the two above: a custom processor that doesn't propagate termination frames, or a background task started inside a processor (for example, a timer or long-running coroutine) that isn't cleaned up. Create background tasks through the pipeline task manager so they are tracked and cancelled on shutdown, and make sure your processors push `EndFrame`/`CancelFrame` downstream. + ## Key Takeaways - **Frame-based termination** - shutdown uses the same frame system as processing diff --git a/pipecat/learn/speech-input.mdx b/pipecat/learn/speech-input.mdx index 52d70c3b8..eee04be6d 100644 --- a/pipecat/learn/speech-input.mdx +++ b/pipecat/learn/speech-input.mdx @@ -57,11 +57,11 @@ vad_analyzer = SileroVADAnalyzer( user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, - user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer)), + user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer), ) ``` -In the vast number of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it. +VAD is configured on the user aggregator because its speech start/stop signals feed the [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) that decide when a turn begins and ends. In the vast majority of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it. ### Key Parameters @@ -88,6 +88,16 @@ In the vast number of cases, the default values will work well. Only adjust thes optimal performance across different audio environments and use cases. + + `confidence` and `min_volume` only raise the bar for what counts as speech — + blunt instruments for noisy audio. If your bot reacts to background voices or + your STT transcribes noise, it's usually better to remove the noise upstream + with an input audio filter — [Krisp VIVA](/pipecat/features/krisp-viva), + [ai-coustic](/api-reference/server/utilities/audio/aic-filter), or + [RNNoise](/api-reference/server/utilities/audio/rnnoise-filter) — before the + audio reaches VAD and STT. + + ## User Turn Detection While VAD detects speech vs. silence, it can't understand linguistic context. A pause doesn't mean the user is done. User turn strategies interpret VAD signals and transcriptions to determine actual turn boundaries. diff --git a/pipecat/learn/text-to-speech.mdx b/pipecat/learn/text-to-speech.mdx index 48dc9e634..1eb580c1c 100644 --- a/pipecat/learn/text-to-speech.mdx +++ b/pipecat/learn/text-to-speech.mdx @@ -32,6 +32,7 @@ pipeline = Pipeline([ **TTS generates speech through two primary mechanisms:** 1. **Streamed LLM tokens** via `LLMTextFrame`s: + - By default, TTS aggregates streaming tokens into complete sentences before synthesis (`TextAggregationMode.SENTENCE`) - Set `text_aggregation_mode=TextAggregationMode.TOKEN` to stream tokens directly for lower latency - Audio bytes stream back and play immediately @@ -314,10 +315,10 @@ Use `TTSSpeakFrame` for immediate speech synthesis: ```python from pipecat.frames.frames import TTSSpeakFrame -# Make bot speak directly +# Make bot speak directly (added to context by default) await tts.queue_frame(TTSSpeakFrame("Hello, how can I help you?")) -# Append spoken text to conversation context +# Explicitly append spoken text to conversation context await tts.queue_frame( TTSSpeakFrame("Welcome! Let's begin.", append_to_context=True) ) @@ -330,6 +331,13 @@ await tts.queue_frame( The `append_to_context` parameter controls whether the spoken text is added to the conversation history. When `append_to_context=True`, the text is automatically committed to the context after being spoken, making it useful for bot greetings and injected speech that should be part of the conversation flow. + + As of Pipecat v1.4.0, `append_to_context` defaults to `True`. A plain + `TTSSpeakFrame("...")` **is** added to the conversation context after it is + spoken; pass `append_to_context=False` to speak without recording it. (`None` + was the previous default and is no longer supported.) + + ### Dynamic Settings Updates Update TTS settings during conversation using typed settings objects: diff --git a/pipecat/migration/migration-1.0.mdx b/pipecat/migration/migration-1.0.mdx index 6e434604a..93187dbca 100644 --- a/pipecat/migration/migration-1.0.mdx +++ b/pipecat/migration/migration-1.0.mdx @@ -12,6 +12,17 @@ Before upgrading, search your codebase for the deprecated imports and patterns l been dropped. Python 3.11 through 3.14 are supported. + + **Removed parameters are silently ignored, not errored.** `TransportParams` + and `PipelineParams` are Pydantic models that drop unknown fields, so leftover + 0.0.x params like `vad_analyzer`, `turn_analyzer`, and `allow_interruptions` + are accepted with no error or warning. Your app can start cleanly and look + upgraded while turn detection is actually broken (missed user speech, no bot + responses). Don't rely on the app starting: delete these params from your + transport/pipeline config and move them to `LLMUserAggregatorParams` as shown + below. + + ## 1. Universal LLMContext `LLMContext` is a universal context that works with all LLM providers. It allows you to dynamically switch `LLMService` at runtime, passing a compatible context between services. Pipecat maintains adapters for each LLM service, which are applied automatically at the time of inference. @@ -133,22 +144,29 @@ from pipecat.processors.aggregators.llm_response_universal import ( LLMUserAggregatorParams, ) from pipecat.audio.vad.silero import SileroVADAnalyzer +from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3 +from pipecat.turns.user_start import ( + TranscriptionUserTurnStartStrategy, + VADUserTurnStartStrategy, +) +from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy from pipecat.turns.user_turn_strategies import UserTurnStrategies # Everything configured on the aggregator user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( - user_turn_strategies=[ - UserTurnStrategies( - start=[ - # Start strategies - ], - stop=[ - # Stop strategies - ], - ), - ], + user_turn_strategies=UserTurnStrategies( + start=[ + VADUserTurnStartStrategy(), + TranscriptionUserTurnStartStrategy(), + ], + stop=[ + TurnAnalyzerUserTurnStopStrategy( + turn_analyzer=LocalSmartTurnAnalyzerV3() + ), + ], + ), user_mute_strategies=[ # Mute strategies ], @@ -159,6 +177,15 @@ user_aggregator, assistant_aggregator = LLMContextAggregatorPair( ) ``` + + **Where did `allow_interruptions` go?** The `allow_interruptions` parameter + was a legacy parameter used for non-voice AI applications. If you're looking + to prevent an interruption from occurring, consider using a user mute strategy + instead. See the [User Mute Strategies + reference](/api-reference/server/utilities/turn-management/user-mute-strategies) + for more information. + + ### What was removed | Removed | Replacement | @@ -296,7 +323,7 @@ Several frame classes were renamed or removed. Update any direct references in y | `InputTransportMessageUrgentFrame` | `InputTransportMessageFrame` | | `KeypadEntryFrame` | `DTMFFrame` | | `StartInterruptionFrame` | `InterruptionFrame` | -| `BotInterruptionFrame` | `InterruptionWorkerFrame` | +| `BotInterruptionFrame` | `InterruptionWorkerFrame` | | `TranscriptionMessage` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `TranscriptionUpdateFrame` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `DailyTransportMessageFrame` | `DailyOutputTransportMessageFrame` | @@ -354,7 +381,7 @@ Smaller breaking changes across the rest of the API. | `PipelineTask.on_pipeline_ended` | `on_pipeline_finished` | | `PipelineTask.on_pipeline_cancelled` | `on_pipeline_finished` | | `PipelineTask.on_pipeline_stopped` | `on_pipeline_finished` | -| `DailyRunner.configure_with_args()` | Use `WorkerRunner` with `RunnerArguments` | +| `DailyRunner.configure_with_args()` | Use `WorkerRunner` with `RunnerArguments` | ### TTS service