Skip to content

fix(livekit): deliver inbound app messages to RTVI instead of back to the client - #5224

Closed
XinZhou0417 wants to merge 2 commits into
pipecat-ai:mainfrom
XinZhou0417:fix/livekit-inbound-rtvi
Closed

fix(livekit): deliver inbound app messages to RTVI instead of back to the client#5224
XinZhou0417 wants to merge 2 commits into
pipecat-ai:mainfrom
XinZhou0417:fix/livekit-inbound-rtvi

Conversation

@XinZhou0417

Copy link
Copy Markdown

Summary

Over LiveKitTransport, a message sent from a client to the bot never reaches RTVIProcessor. client-ready never completes, so a session never becomes ready; send-text, output-mode switching and every other client-to-server RTVI message are inoperative. Bot-to-client messaging is unaffected, so a session looks healthy right up until the client tries to say something.

SmallWebRTCTransport handles this correctly. This brings LiveKit in line with it.

Fixes #5218.

Cause

Three defects compound, the first masking the second.

1 — push_app_message emits an output frame.

frame = LiveKitOutputTransportMessageUrgentFrame(message=message, participant_id=sender)
await self.push_frame(frame)

RTVIProcessor dispatches on InputTransportMessageFrame. The two are siblings under SystemFrame and neither extends the other, so the isinstance check never matches. The frame rides past RTVI and is then matched by BaseOutputTransport:

elif isinstance(frame, OutputTransportMessageUrgentFrame):
    await self.send_message(frame)

— which does its job and sends the client's own message back to the client.

2 — the payload is forwarded undecoded. _on_data_received passes data.decode(), a str, where RTVIProcessor._handle_transport_message needs a mapping for transport_message.get("label"). Unreachable while defect 1 stands, and immediate once it is fixed.

3 — the sender is reported as a SID. _async_on_data_received passes data.participant.sid, which is carried through to publish_data(destination_identities=[...]). LiveKit resolves that against identity, so the misrouted reply above is addressed to a participant that does not exist and is dropped without error. This one is independent of RTVI — it makes any directed send_message on this transport silently undeliverable.

Fix

Decode the payload, broadcast an InputTransportMessageFrame — what SmallWebRTCInputTransport already does — and report the sender by identity. A non-JSON packet is logged and ignored rather than raising, since the data channel is shared.

No changes to constructor arguments, defaults, or public API.

Verification

Four regression tests in tests/test_livekit_transport.py. All four fail on main and pass with the fix:

FAILED  test_inbound_message_is_broadcast_as_an_input_frame
FAILED  test_payload_is_decoded_before_it_reaches_the_pipeline
FAILED  test_a_non_json_packet_is_ignored_rather_than_raising
FAILED  test_sender_is_reported_by_identity_not_sid

The full file passes with the fix applied (14 tests, including the 10 already there).

Found and confirmed against a live LiveKit server before writing the patch — a real pipeline ([transport.input(), rtvi, transport.output()]) plus a bare rtc.Room client publishing one client-ready packet. The runnable reproduction is in #5218. Measured there:

check result
RTVIProcessor fired on_client_ready False
client's own message re-sent by the output transport True
directed send delivered, addressed by SID False
directed send delivered, addressed by identity True

Note

Defect 3 is separable from the other two if you would prefer it in its own PR — say the word and I will split it out.

… the client

Over LiveKitTransport, a message sent from a client to the bot never reached
RTVIProcessor. Three defects compound, the first masking the second:

push_app_message wrapped the message in a LiveKitOutputTransportMessageUrgentFrame
— an output frame. RTVIProcessor dispatches on InputTransportMessageFrame, and the
two are siblings under SystemFrame with neither extending the other, so the
isinstance check never matched. The frame rode past RTVI and was then matched by
BaseOutputTransport, which did its job and sent it back to the client.

_on_data_received forwarded the payload as a str, where
RTVIProcessor._handle_transport_message needs a mapping for
transport_message.get("label"). Unreachable while the frame class was wrong.

_async_on_data_received reported the sender as data.participant.sid, which is
carried through to publish_data(destination_identities=[...]). LiveKit resolves
that against identity, so the misrouted reply above was addressed to a
participant that does not exist and was dropped without error — and any directed
send_message on this transport has the same problem, independent of RTVI.

The effect was that client-ready never completed, so a session never became
ready, and send-text, output-mode switching and every other client-to-server
RTVI message were inoperative. Bot-to-client messaging was unaffected, so a
session looked healthy until the client tried to say something.

SmallWebRTCTransport does all three correctly; this brings LiveKit in line with
it. Fixes pipecat-ai#5218.

The four regression tests fail on the unpatched transport and pass with the fix.

The identity change is separable from the other two if you would prefer it in its
own PR.
@XinZhou0417

Copy link
Copy Markdown
Author

Parallel fix #5297 merged, hence closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant