Skip to content

fix(voice): answer every tool call #2289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rosetta-livekit-bot wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(voice): answer every tool call #2289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/quiet-tools-always-answer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@livekit/agents': patch
'@livekit/agents-plugin-google': patch
'@livekit/agents-plugin-phonic': patch
---

Answer every completed tool call, including interrupted, silent, and invalid calls, and keep realtime providers quiet when supported.
4 changes: 4 additions & 0 deletions agents/src/llm/__snapshots__/chat_context.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ exports[`ChatContext.toJSON > should match snapshot for edge cases > edge-cases-
"isError": false,
"name": "",
"output": "OK",
"replyRequired": true,
"type": "function_call_output",
},
{
Expand Down Expand Up @@ -109,6 +110,7 @@ exports[`ChatContext.toJSON > should match snapshot for edge cases > edge-cases-
"isError": false,
"name": "",
"output": "OK",
"replyRequired": true,
"type": "function_call_output",
},
{
Expand Down Expand Up @@ -158,6 +160,7 @@ exports[`ChatContext.toJSON > should match snapshot for function calls > convers
"isError": false,
"name": "get_weather",
"output": "{"temperature": 22, "condition": "partly cloudy", "humidity": 65}",
"replyRequired": true,
"type": "function_call_output",
},
{
Expand Down Expand Up @@ -201,6 +204,7 @@ exports[`ChatContext.toJSON > should match snapshot for function calls > convers
"isError": false,
"name": "get_weather",
"output": "{"temperature": 22, "condition": "partly cloudy", "humidity": 65}",
"replyRequired": true,
"type": "function_call_output",
},
{
Expand Down
8 changes: 8 additions & 0 deletions agents/src/llm/chat_context.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -557,6 +557,9 @@ export class FunctionCallOutput {

isError: boolean;

/** Whether a realtime model should answer after receiving this output. */
replyRequired: boolean;

createdAt: number;

constructor(params: {
Expand All @@ -566,6 +569,7 @@ export class FunctionCallOutput {
id?: string;
createdAt?: number;
name?: string;
replyRequired?: boolean;
}) {
const {
callId,
Expand All @@ -574,13 +578,15 @@ export class FunctionCallOutput {
id = shortuuid('item_'),
createdAt = Date.now(),
name = '',
replyRequired = true,
} = params;
this.id = id;
this.callId = callId;
this.output = output;
this.isError = isError;
this.name = name;
this.createdAt = createdAt;
this.replyRequired = replyRequired;
}

static create(params: {
Expand All @@ -590,6 +596,7 @@ export class FunctionCallOutput {
id?: string;
createdAt?: number;
name?: string;
replyRequired?: boolean;
}) {
return new FunctionCallOutput(params);
}
Expand All @@ -602,6 +609,7 @@ export class FunctionCallOutput {
callId: this.callId,
output: this.output,
isError: this.isError,
replyRequired: this.replyRequired,
};

if (!excludeTimestamp) {
Expand Down
1 change: 1 addition & 0 deletions agents/src/voice/__snapshots__/report.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ exports[`sessionReportToJSON > serializes the full chat history to the Python sn
"is_error": false,
"name": "get_weather",
"output": "{"temperature": 22, "condition": "partly cloudy"}",
"reply_required": true,
"type": "function_call_output",
},
{
Expand Down
36 changes: 19 additions & 17 deletions agents/src/voice/agent_activity.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1328,7 +1328,6 @@ describe('AgentActivity - interrupted tool completion', () => {
toolCall: call,
toolCallOutput: output,
rawOutput: 'charged',
replyRequired: true,
}),
],
firstToolStartedFuture: new Future<void>(),
Expand All @@ -1345,12 +1344,13 @@ describe('AgentActivity - interrupted tool completion', () => {

expect(chatCtx.items).toContain(output);
expect(output.createdAt).toBe(300);
expect(output.replyRequired).toBe(false);
expect(chatCtx.items).toEqual([call, output]);
expect(toolItemsAdded).toHaveBeenCalledWith([output]);
expect(generateReply).not.toHaveBeenCalled();
});

it('does not persist an interrupted handoff as completed', () => {
it('records an interrupted handoff as a silent error', () => {
const emit = vi.fn();
const toolItemsAdded = vi.fn();
const activity = Object.create(AgentActivity.prototype) as AgentActivity;
Expand Down Expand Up @@ -1384,7 +1384,6 @@ describe('AgentActivity - interrupted tool completion', () => {
toolCall: call,
toolCallOutput: output,
rawOutput: 'transferred',
replyRequired: false,
agentTask: new AgentTask({ instructions: 'Handle the specialist request.' }),
}),
],
Expand All @@ -1400,11 +1399,16 @@ describe('AgentActivity - interrupted tool completion', () => {
}
)._commitInterruptedToolOutputs(toolOutput, SpeechHandle.create());

expect(chatCtx.items).not.toContain(call);
expect(chatCtx.items).not.toContain(output);
expect(sessionHistory.items).not.toContain(call);
expect(emit).not.toHaveBeenCalled();
expect(toolItemsAdded).not.toHaveBeenCalled();
expect(chatCtx.items).toEqual([call, output]);
expect(sessionHistory.items).toEqual([call]);
expect(output.isError).toBe(true);
expect(output.output).toContain('handoff was interrupted');
expect(output.replyRequired).toBe(false);
expect(emit).toHaveBeenCalledWith(
AgentSessionEventTypes.FunctionToolsExecuted,
expect.objectContaining({ functionCalls: [call], functionCallOutputs: [output] }),
);
expect(toolItemsAdded).toHaveBeenCalledWith([output]);
});

it('commits and emits only completed regular tools from a mixed interrupted batch', () => {
Expand Down Expand Up @@ -1448,13 +1452,11 @@ describe('AgentActivity - interrupted tool completion', () => {
toolCall: completedCall,
toolCallOutput: completedOutput,
rawOutput: 'saved',
replyRequired: true,
}),
ToolExecutionOutput.create({
toolCall: handoffCall,
toolCallOutput: handoffOutput,
rawOutput: 'transferred',
replyRequired: false,
agentTask: new AgentTask({ instructions: 'Handle the specialist request.' }),
}),
],
Expand All @@ -1470,15 +1472,16 @@ describe('AgentActivity - interrupted tool completion', () => {
}
)._commitInterruptedToolOutputs(toolOutput, SpeechHandle.create());

expect(chatCtx.items).toHaveLength(2);
expect(chatCtx.items).toEqual(expect.arrayContaining([completedCall, completedOutput]));
expect(sessionHistory.items).toEqual([completedCall]);
expect(toolItemsAdded).toHaveBeenCalledWith([completedOutput]);
expect(chatCtx.items).toEqual([completedCall, handoffCall, completedOutput, handoffOutput]);
expect(sessionHistory.items).toEqual([completedCall, handoffCall]);
expect(completedOutput.replyRequired).toBe(false);
expect(handoffOutput).toMatchObject({ isError: true, replyRequired: false });
expect(toolItemsAdded).toHaveBeenCalledWith([completedOutput, handoffOutput]);
expect(emit).toHaveBeenCalledWith(
AgentSessionEventTypes.FunctionToolsExecuted,
expect.objectContaining({
functionCalls: [completedCall],
functionCallOutputs: [completedOutput],
functionCalls: [completedCall, handoffCall],
functionCallOutputs: [completedOutput, handoffOutput],
}),
);
});
Expand Down Expand Up @@ -1603,7 +1606,6 @@ describe('AgentActivity - interruption while waiting for tools', () => {
toolCall: call,
toolCallOutput: output,
rawOutput: 'saved',
replyRequired: true,
}),
],
firstToolStartedFuture: new Future<void>(),
Expand Down
67 changes: 35 additions & 32 deletions agents/src/voice/agent_activity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ import {
applyInstructionsModality,
forwardedTextFor,
hasExpressiveInstructions,
interruptedToolOutput,
performAudioForwarding,
performLLMInference,
performTTSInference,
Expand Down Expand Up @@ -3519,7 +3520,7 @@ export class AgentActivity implements RecognitionHooks {
);
}

const { functionToolsExecutedEvent, shouldGenerateToolReply, newAgentTask, ignoreTaskSwitch } =
const { functionToolsExecutedEvent, newAgentTask, ignoreTaskSwitch } =
this.summarizeToolExecutionOutput(toolOutput, speechHandle);

this.agentSession.emit(
Expand All @@ -3546,7 +3547,7 @@ export class AgentActivity implements RecognitionHooks {
this.agentSession._toolItemsAdded(toolCallOutputs);
}

if (shouldGenerateToolReply) {
if (functionToolsExecutedEvent.hasToolReply) {
_stripRunningToolCalls(chatCtx);
chatCtx.insert(toolMessages);

Expand Down Expand Up @@ -4050,6 +4051,20 @@ export class AgentActivity implements RecognitionHooks {
speechHandle._markGenerationDone();
await this.cancelToolExecutions(executeToolsTask, speechHandle, toolOutput);

const interruptedOutputs = this._commitInterruptedToolOutputs(toolOutput, speechHandle);
if (interruptedOutputs.length > 0) {
const chatCtx = realtimeSession.chatCtx.copy();
chatCtx.items.push(...interruptedOutputs);
try {
await realtimeSession.updateChatCtx(chatCtx);
} catch (error) {
this.logger.warn(
{ error },
'failed to sync the tool results of an interrupted generation',
);
}
}

// TODO(brian): close tees
return;
}
Expand Down Expand Up @@ -4090,7 +4105,7 @@ export class AgentActivity implements RecognitionHooks {
return;
}

const { functionToolsExecutedEvent, shouldGenerateToolReply, newAgentTask, ignoreTaskSwitch } =
const { functionToolsExecutedEvent, newAgentTask, ignoreTaskSwitch } =
this.summarizeToolExecutionOutput(toolOutput, speechHandle);

this.agentSession.emit(
Expand Down Expand Up @@ -4136,7 +4151,7 @@ export class AgentActivity implements RecognitionHooks {
let fut: Future<void, never> | undefined;
if (
realtimeModel.capabilities.autoToolReplyGeneration &&
shouldGenerateToolReply &&
functionToolsExecutedEvent.hasToolReply &&
this.pendingAutoToolReplyFut === undefined
) {
const runState = this.agentSession._globalRunState;
Expand Down Expand Up @@ -4187,7 +4202,10 @@ export class AgentActivity implements RecognitionHooks {
}

// skip realtime reply if not required or auto-generated
if (!shouldGenerateToolReply || realtimeModel.capabilities.autoToolReplyGeneration) {
if (
!functionToolsExecutedEvent.hasToolReply ||
realtimeModel.capabilities.autoToolReplyGeneration
) {
return;
}

Expand Down Expand Up @@ -4283,22 +4301,14 @@ export class AgentActivity implements RecognitionHooks {
}

/** @internal */
_commitInterruptedToolOutputs(toolOutput: ToolOutput, speechHandle: SpeechHandle): void {
const interruptedHandoffCallIds = toolOutput.output
.filter((output) => output.agentTask !== undefined)
.map((output) => output.toolCall.callId);
if (interruptedHandoffCallIds.length > 0) {
const interruptedHandoffCallIdSet = new Set(interruptedHandoffCallIds);
for (const chatCtx of [this.agent._chatCtx, this.agentSession.history]) {
chatCtx.items = chatCtx.items.filter(
(item) => item.type !== 'function_call' || !interruptedHandoffCallIdSet.has(item.callId),
);
}
}
const completedOutputs = toolOutput.output.filter((output) => output.agentTask === undefined);
if (completedOutputs.length === 0) return;
_commitInterruptedToolOutputs(
toolOutput: ToolOutput,
speechHandle: SpeechHandle,
): FunctionCallOutput[] {
if (toolOutput.output.length === 0) return [];
for (const output of toolOutput.output) interruptedToolOutput(output);
const { functionToolsExecutedEvent } = this.summarizeToolExecutionOutput(
{ ...toolOutput, output: completedOutputs },
toolOutput,
speechHandle,
);
this.agentSession.emit(
Expand All @@ -4310,6 +4320,7 @@ export class AgentActivity implements RecognitionHooks {
this.agent._chatCtx.insert(outputs);
this.agentSession._toolItemsAdded(outputs);
}
return outputs;
}

private summarizeToolExecutionOutput(toolOutput: ToolOutput, speechHandle: SpeechHandle) {
Expand All @@ -4318,19 +4329,12 @@ export class AgentActivity implements RecognitionHooks {
functionCallOutputs: [],
});

let shouldGenerateToolReply = false;
let newAgentTask: Agent | null = null;
let ignoreTaskSwitch = false;

for (const sanitizedOut of toolOutput.output) {
if (sanitizedOut.toolCallOutput !== undefined) {
// Keep event payload symmetric for pipeline + realtime paths.
functionToolsExecutedEvent.functionCalls.push(sanitizedOut.toolCall);
functionToolsExecutedEvent.functionCallOutputs.push(sanitizedOut.toolCallOutput);
if (sanitizedOut.replyRequired) {
shouldGenerateToolReply = true;
}
}
functionToolsExecutedEvent.functionCalls.push(sanitizedOut.toolCall);
functionToolsExecutedEvent.functionCallOutputs.push(sanitizedOut.toolCallOutput);

if (newAgentTask !== null && sanitizedOut.agentTask !== undefined) {
this.logger.error('expected to receive only one agent task from the tool executions');
Expand All @@ -4344,16 +4348,15 @@ export class AgentActivity implements RecognitionHooks {
speechId: speechHandle.id,
name: sanitizedOut.toolCall?.name,
args: sanitizedOut.toolCall.args,
output: sanitizedOut.toolCallOutput?.output,
isError: sanitizedOut.toolCallOutput?.isError,
output: sanitizedOut.toolCallOutput.output,
isError: sanitizedOut.toolCallOutput.isError,
},
'Tool call execution finished',
);
}

return {
functionToolsExecutedEvent,
shouldGenerateToolReply,
newAgentTask,
ignoreTaskSwitch,
};
Expand Down
8 changes: 8 additions & 0 deletions agents/src/voice/events.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ export type FunctionToolsExecutedEvent = {
functionCalls: FunctionCall[];
functionCallOutputs: FunctionCallOutput[];
createdAt: number;
readonly hasToolReply: boolean;
cancelToolReply(): void;
Comment on lines +191 to +192

@devin-ai-integration devin-ai-integration Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Newly added public event members and helper function are undocumented

The two new members added to the tool-execution event and the new interrupted-output helper carry no documentation comments (hasToolReply/cancelToolReply at agents/src/voice/events.ts:191-192), which the repository's contribution rules require for every new method or interface member.
Impact: The public API documentation generated for the framework is missing entries for the newly exposed members.

Repository rule

CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented."

Undocumented additions in this PR:

  • FunctionToolsExecutedEvent.hasToolReply and FunctionToolsExecutedEvent.cancelToolReply() (agents/src/voice/events.ts:191-192)
  • the exported interruptedToolOutput() helper (agents/src/voice/generation.ts:495)

FunctionCallOutput.replyRequired (agents/src/llm/chat_context.ts:560-561) does have a doc comment and is fine.

Suggested change
readonly hasToolReply: boolean;
cancelToolReply(): void;
/** Whether any completed tool result still expects the model to reply. */
readonly hasToolReply: boolean;
/** Suppress the model reply for every tool result in this batch. */
cancelToolReply(): void;
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

};

export const createFunctionToolsExecutedEvent = ({
Expand All @@ -204,6 +206,12 @@ export const createFunctionToolsExecutedEvent = ({
functionCalls,
functionCallOutputs,
createdAt,
get hasToolReply() {
return functionCallOutputs.some((output) => output.replyRequired);
},
cancelToolReply() {
for (const output of functionCallOutputs) output.replyRequired = false;
},
Comment on lines +209 to +214

@devin-ai-integration devin-ai-integration Bot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Session report now contains an extra field and a non-serializable callback for tool-execution events

The tool-execution event is copied field-by-field into the uploaded report ({...event} at agents/src/voice/report.ts:160), which now also picks up the two newly added members, so the report carries a field that does not exist in the reference wire format plus a function value.
Impact: Consumers of the session report see an unexpected extra field for tool-execution events, and the report object holds a value that cannot be serialized.

New getter/method on FunctionToolsExecutedEvent leak into the wire format

createFunctionToolsExecutedEvent (agents/src/voice/events.ts:202-215) now returns an object with an own enumerable getter hasToolReply and an own method cancelToolReply. eventToJSON spreads the event (agents/src/voice/report.ts:160) and then runs toSnakeCaseDeep (agents/src/voice/report.ts:109-127), which copies every own enumerable entry. The result gains has_tool_reply: boolean and cancel_tool_reply: <function> for every function_tools_executed event pushed at agents/src/voice/report.ts:250. The Python model exposes these as a property/method, so they are absent from its model_dump().

Defining the two members as non-enumerable, or excluding them in eventToJSON's switch (like speechHandle is deleted for speech_created), would keep the wire shape aligned.

Prompt for agents 
FunctionToolsExecutedEvent gained an own enumerable getter (hasToolReply) and an own method (cancelToolReply) in agents/src/voice/events.ts. agents/src/voice/report.ts eventToJSON spreads every event object and toSnakeCaseDeep copies all own enumerable entries, so serialized session reports now contain has_tool_reply and a cancel_tool_reply function value for function_tools_executed events — neither exists in the Python wire format. Consider making these members non-enumerable (Object.defineProperty) or explicitly stripping them in eventToJSON's per-type switch, similar to how speechHandle is deleted for speech_created.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

};
};

Expand Down
Loading
Loading