feat(telemetry): tag content-bearing trace and log keys with lk.pii. for redaction AGT-3074 by chenghao-mou · Pull Request #6356 · livekit/agents

chenghao-mou · 2026-07-08T20:25:20Z

Make Cloud redaction reliable by tagging content-bearing OTEL trace and log attributes with a dot-delimited pii segment, including nested STT keyterms, and by removing sensitive content from unredactable log bodies.

Exceptions now omit messages and stack traces when job redaction is enabled. A guard test requires every telemetry key to be classified as safe or PII-tagged.

Downstream: dashboards querying renamed keys must adopt lk.pii.*; older SDKs remain unredacted.

Fixes AGT-3074

…for redaction The LiveKit Cloud collector strips OTLP attribute keys carrying a dot-delimited pii segment for PII-enabled projects; none of the SDK's content-bearing keys matched, so transcripts, chat context, tool payloads, and TTS text survived redaction. - rename 10 content span attribute keys to lk.pii.* (constants unchanged) - rename content log extras in core and all plugins to lk.pii.* - move content out of unredactable log message bodies into lk.pii extras (DTMF digits, AWS Sonic event/chat dumps, realtime event dumps, raw provider WS frames) - add a guard test forcing every new telemetry key to be classified Evaluations-scope records are exempt by design. Identifier PII (participant identity, room name) is deferred to AGT-3074. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

- Mark participant identity and room name as PII trace attributes - Add an Ink-2 voice agent example with local VAD start-of-speech handling

…on-support # Conflicts: # livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py

devin-ai-integration

Devin Review found 2 new potential issues.

View 5 additional findings in Devin Review.

devin-ai-integration · 2026-08-12T19:32:16Z

+def record_exception(
+    span: trace.Span, exception: Exception, *, redacted: NotGivenOr[bool] = NOT_GIVEN
+) -> None:
+    if redacted is NOT_GIVEN:
+        redacted = _redaction_enabled()
+
+    if redacted:
+        attrs = {
+            trace_types.ATTR_EXCEPTION_TYPE: exception.__class__.__name__,
+            trace_types.ATTR_EXCEPTION_MESSAGE: REDACTED_EXCEPTION_MESSAGE,
+        }
+        span.add_event("exception", attrs)
+        span.set_status(trace.Status(trace.StatusCode.ERROR, REDACTED_EXCEPTION_MESSAGE))
+        span.set_attributes(attrs)
+        return


🟡 Error details can still reach telemetry uncensored when redaction is turned on

Error text and stack traces are only suppressed for spans opened through the framework's own span helper (start_as_current_span guard at livekit-agents/livekit/agents/telemetry/traces.py:102-107), so error details recorded on spans opened the other way still get uploaded, meaning content the customer asked to be hidden can leave the process.
Impact: Sessions running with redaction enabled can still ship exception messages and stack traces (which may quote user speech or tool payloads) to the observability backend.

Only _DynamicTracer.start_as_current_span is guarded; trace.use_span call sites keep OTel defaults

The new guard rewrites kwargs only inside _DynamicTracer.start_as_current_span. Several code paths instead activate an existing span with opentelemetry.trace.use_span(...), which defaults to record_exception=True and set_status_on_exception=True: livekit-agents/livekit/agents/voice/audio_recognition.py:1324, :1366, :1385, :1424, :1527, :1863, and livekit-agents/livekit/agents/voice/agent_activity.py:812. Any exception escaping those blocks (e.g. a user hook raising with transcript text in the message) adds an exception event carrying exception.message and exception.stacktrace. Those keys are explicitly safe-listed in tests/test_trace_types_pii.py, so the LiveKit Cloud collector will not strip them either — exactly the leak telemetry_utils.record_exception's redacted branch was added to prevent.

A fix would be to centralize the redaction decision (e.g. a wrapper around use_span that passes record_exception=False/set_status_on_exception=False when telemetry_utils._redaction_enabled()), and use it at every trace.use_span site.

Was this helpful? React with 👍 or 👎 to provide feedback.

…on-support # Conflicts: # tests/test_recording.py

…on-support

devin-ai-integration

Devin Review found 2 new potential issues.

View 8 additional findings in Devin Review.

devin-ai-integration · 2026-08-13T17:08:16Z

            logger.exception(
                f"exception executing AI function `{tool_call.name}`",
-                extra={"call_id": tool_call.call_id, "arguments": tool_call.arguments},
+                extra={"call_id": tool_call.call_id, "lk.pii.arguments": tool_call.arguments},


🟡 Some logs still send tool arguments, transcripts and chat history in a form that cannot be scrubbed

Several log entries that carry conversational data are still recorded under untagged names (e.g. alongside the tagged call at livekit-agents/livekit/agents/llm/utils.py:1035) even though sibling entries were renamed, so this data is not removed for projects that request scrubbing.
Impact: Tool call arguments, user transcripts and chat history keep reaching LiveKit Cloud unredacted for redaction-enabled projects, defeating the purpose of the change.

Untagged content-bearing log fields left behind by the rename

The PR tags content keys with an lk.pii. prefix so the Cloud collector can strip them, and the new docstring in livekit-agents/livekit/agents/telemetry/trace_types.py:10-17 states that the segment is the only marker honored. However these core-package log records still use untagged keys carrying the same kinds of data:

livekit-agents/livekit/agents/voice/generation.py:816 and livekit-agents/livekit/agents/voice/generation.py:857 — "arguments": fnc_call.arguments (the exact field renamed to lk.pii.arguments in llm/utils.py).

livekit-agents/livekit/agents/voice/amd/detector.py:293 and livekit-agents/livekit/agents/voice/amd/detector.py:528 — "transcript": ... (renamed to lk.pii.user_transcript in voice/audio_recognition.py).

livekit-agents/livekit/agents/inference/llm.py:436 — "chat_ctx": chat_ctx (the chat context is tagged as PII in trace_types.py).

Because the collector matches only on the pii dot-segment, these attributes survive redaction.

Prompt for agents

The PR renames content-bearing log `extra` keys to carry a dot-delimited `pii` segment (e.g. `lk.pii.arguments`) so the LiveKit Cloud collector can strip them for redaction-enabled projects. The transformation was applied inconsistently: several core-package log records still pass conversational content under untagged keys and therefore remain unredactable. Known remaining sites: `livekit-agents/livekit/agents/voice/generation.py` (two `"arguments": fnc_call.arguments` entries in the tool-execution logs), `livekit-agents/livekit/agents/voice/amd/detector.py` (two `"transcript": ...` entries), and `livekit-agents/livekit/agents/inference/llm.py` (`"chat_ctx": chat_ctx` in the debug log). Audit all `extra={...}` log fields in `livekit-agents/` (and the touched plugins) for keys holding transcripts, tool arguments/outputs, chat contexts, instructions, or provider payloads, and rename them consistently. Consider adding a test or lint-style guard analogous to `tests/test_trace_types_pii.py` that covers log `extra` keys, since trace_types.py constants alone do not catch these.

Was this helpful? React with 👍 or 👎 to provide feedback.

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

chenghao-mou changed the title ~~feat(telemetry): tag content-bearing trace and log keys with lk.pii. for redaction~~ feat(telemetry): tag content-bearing trace and log keys with lk.pii. for redaction AGT-3074 Jul 8, 2026

cursor Bot mentioned this pull request Jul 17, 2026

docs: daily engineering digest 2026-07-17 Seventhen/agents#32

Draft

chenghao-mou mentioned this pull request Jul 23, 2026

feat(telemetry): tag content-bearing trace and log keys with lk.pii. for redaction AGT-3074 livekit/agents-js#2100

Open

chenghao-mou and others added 4 commits July 27, 2026 12:44

fix: redact exception and provider log details

1b8bf72

chore: use protocol redaction flag directly

00c0dfc

fix(telemetry): mark session keyterms as PII

4f142cd

chenghao-mou force-pushed the chenghao/feat/redaction-support branch 2 times, most recently from 5838cdb to 4f142cd Compare July 27, 2026 11:50

chenghao-mou marked this pull request as ready for review July 27, 2026 13:43

chenghao-mou requested a review from a team as a code owner July 27, 2026 13:43

devin-ai-integration Bot reviewed Jul 27, 2026

View reviewed changes

Merge branch 'main' into chenghao/feat/redaction-support

b438a78

This comment was marked as resolved.

Sign in to view

feat: add trace PII markers and local SOS example

dacdcd4

- Mark participant identity and room name as PII trace attributes - Add an Ink-2 voice agent example with local VAD start-of-speech handling

This comment was marked as resolved.

Sign in to view

Merge remote-tracking branch 'origin/main' into chenghao/feat/redacti…

355a42f

…on-support # Conflicts: # livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py

This comment was marked as resolved.

Sign in to view

chenghao-mou added 2 commits August 11, 2026 14:03

fix(telemetry): address redaction review feedback

139dd3f

fix: require transcripts for redacted audio uploads

9294350

devin-ai-integration Bot reviewed Aug 12, 2026

View reviewed changes

chenghao-mou added 2 commits August 13, 2026 10:05

Merge remote-tracking branch 'origin/main' into chenghao/feat/redacti…

60a590a

…on-support # Conflicts: # tests/test_recording.py

fix: suppress automatic exception details under redaction

2456461

This comment was marked as resolved.

Sign in to view

Merge remote-tracking branch 'origin/main' into chenghao/feat/redacti…

a5a5aec

…on-support

devin-ai-integration Bot reviewed Aug 13, 2026

View reviewed changes

livekit-agents@1.6.11.rc1 (#6847)

6421a84

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Navigation Menu

Uh oh!

feat(telemetry): tag content-bearing trace and log keys with lk.pii. for redaction AGT-3074 - #6356

feat(telemetry): tag content-bearing trace and log keys with lk.pii. for redaction AGT-3074#6356
chenghao-mou wants to merge 13 commits into
mainfrom
chenghao/feat/redaction-support

chenghao-mou commented Jul 8, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Aug 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

chenghao-mou commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

chenghao-mou commented Jul 8, 2026 •

edited

Loading

devin-ai-integration Bot Aug 12, 2026 •

edited

Loading