feat(copilot): add Langfuse tracing to baseline LLM path - #12281
Conversation
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds Langfuse tracing: replaces OpenAI AsyncOpenAI with Langfuse AsyncOpenAI, wraps baseline streaming flow in Langfuse trace propagation/enter/teardown, and patches the OpenTelemetry TracerProvider Resource to include Langfuse environment. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant CopilotService
participant Langfuse
participant OpenAI
participant StreamConsumer
Client->>CopilotService: Request baseline stream (user_id, session_id)
CopilotService->>Langfuse: propagate_attributes -> enter trace ("copilot-baseline", tag="baseline")
CopilotService->>OpenAI: Stream request via LangfuseAsyncOpenAI
OpenAI-->>CopilotService: Streaming chunks
CopilotService->>StreamConsumer: yield StreamStart then stream chunks
StreamConsumer-->>CopilotService: acknowledge/consume
CopilotService->>Langfuse: exit / teardown trace (on complete/error)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
201f8d0 to
c9d55ae
Compare
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 0 conflict(s), 0 medium risk, 3 low risk (out of 3 PRs with file overlap) Auto-generated on push. Ignores: |
c9d55ae to
9d593f1
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
autogpt_platform/backend/backend/copilot/baseline/service.py (1)
202-216:⚠️ Potential issue | 🟡 MinorConsider moving the
tryblock to start immediately after entering the trace context.The trace context is entered at line 210, but the
tryblock doesn't begin until line 216. If an exception were to occur in lines 212-215 (unlikely but possible), the trace context would remain open since thefinallyblock wouldn't execute.🛠️ Suggested restructuring
# Propagate user/session context to Langfuse so all LLM calls within # this request are grouped under a single trace with proper attribution. _trace_ctx = propagate_attributes( user_id=user_id, session_id=session_id, trace_name="copilot-baseline", tags=["baseline"], ) _trace_ctx.__enter__() - - assistant_text = "" - text_block_id = str(uuid.uuid4()) - text_started = False - step_open = False try: + assistant_text = "" + text_block_id = str(uuid.uuid4()) + text_started = False + step_open = False for _round in range(_MAX_TOOL_ROUNDS):🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/baseline/service.py` around lines 202 - 216, Move the try block to begin immediately after entering the Langfuse trace context so the trace is always exited in the finally handler; specifically, after calling _trace_ctx = propagate_attributes(...) and _trace_ctx.__enter__(), start the try that currently begins later so any exception raised between propagate_attributes/_trace_ctx.__enter__() and the original try's start still flows through the existing finally which should call _trace_ctx.__exit__ (or close the trace). Update the placement around the variables assistant_text, text_block_id, text_started, and step_open so they remain initialized inside the new try scope without changing their names.
🧹 Nitpick comments (1)
autogpt_platform/backend/backend/copilot/baseline/service.py (1)
398-403: Consider passing exception info to__exit__for richer trace metadata.Currently
__exit__(None, None, None)is always called, even when an exception occurred. Passing the actual exception info would allow Langfuse to mark the trace as errored, improving observability.♻️ Proposed enhancement
+ except Exception as e: + error_msg = str(e) or type(e).__name__ + logger.error("[Baseline] Streaming error: %s", error_msg, exc_info=True) + # Close any open text/step before emitting error + if text_started: + yield StreamTextEnd(id=text_block_id) + if step_open: + yield StreamFinishStep() + yield StreamError(errorText=error_msg, code="baseline_error") + # Store exception info for trace context + _exc_info = (type(e), e, e.__traceback__) + # Still persist whatever we got finally: # Close Langfuse trace context try: - _trace_ctx.__exit__(None, None, None) + _trace_ctx.__exit__(*_exc_info if '_exc_info' in dir() else (None, None, None)) except Exception: logger.warning("[Baseline] Langfuse trace context teardown failed")Alternatively, use
sys.exc_info()within the finally block if still in exception context.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/baseline/service.py` around lines 398 - 403, The finally block always calls _trace_ctx.__exit__(None, None, None) which hides real exception information; change it to pass the actual exception info to __exit__ (e.g., use sys.exc_info() or capture the exception tuple) so Langfuse can mark traces as errored, and keep the existing try/except around _trace_ctx.__exit__ to catch teardown failures and still call logger.warning if teardown fails.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@autogpt_platform/backend/backend/copilot/baseline/service.py`:
- Around line 202-216: Move the try block to begin immediately after entering
the Langfuse trace context so the trace is always exited in the finally handler;
specifically, after calling _trace_ctx = propagate_attributes(...) and
_trace_ctx.__enter__(), start the try that currently begins later so any
exception raised between propagate_attributes/_trace_ctx.__enter__() and the
original try's start still flows through the existing finally which should call
_trace_ctx.__exit__ (or close the trace). Update the placement around the
variables assistant_text, text_block_id, text_started, and step_open so they
remain initialized inside the new try scope without changing their names.
---
Nitpick comments:
In `@autogpt_platform/backend/backend/copilot/baseline/service.py`:
- Around line 398-403: The finally block always calls _trace_ctx.__exit__(None,
None, None) which hides real exception information; change it to pass the actual
exception info to __exit__ (e.g., use sys.exc_info() or capture the exception
tuple) so Langfuse can mark traces as errored, and keep the existing try/except
around _trace_ctx.__exit__ to catch teardown failures and still call
logger.warning if teardown fails.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 575f6749-e93d-41e4-baa8-ac90c13e269d
📒 Files selected for processing (2)
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
- GitHub Check: Seer Code Review
- GitHub Check: types
- GitHub Check: test (3.12)
- GitHub Check: test (3.11)
- GitHub Check: test (3.13)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (4)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
🧠 Learnings (2)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
📚 Learning: 2026-03-04T08:04:35.881Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12273
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:216-220
Timestamp: 2026-03-04T08:04:35.881Z
Learning: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path.
Applied to files:
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
🔇 Additional comments (2)
autogpt_platform/backend/backend/copilot/service.py (1)
15-15: LGTM! Langfuse-wrapped AsyncOpenAI integration looks correct.The
langfuse.openai.AsyncOpenAIwrapper is a drop-in replacement that auto-instruments OpenAI API calls when Langfuse credentials are configured, falling back to standard behavior otherwise. The# type: ignore[attr-defined]comment is appropriate since the module doesn't explicitly export this in__all__.Also applies to: 29-29
autogpt_platform/backend/backend/copilot/baseline/service.py (1)
16-16: LGTM!Import for
propagate_attributesis appropriate for establishing Langfuse trace context.
9d593f1 to
b3fd6e3
Compare
- Swap shared OpenAI client to langfuse.openai.AsyncOpenAI which auto-captures all LLM calls as Langfuse generations (token usage, latency, model, prompts) when Langfuse is configured - Add propagate_attributes() context in baseline streaming for user_id/session_id attribution, matching the SDK path's tracing
b3fd6e3 to
c349149
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
autogpt_platform/backend/backend/copilot/baseline/service.py (1)
202-214: Track whether__enter__succeeded to avoid calling__exit__on an un-entered context.If
propagate_attributes()succeeds but__enter__()raises,_trace_ctxwill be non-None. The finally block will then call__exit__()on a context manager that was never entered, violating the context manager protocol.While the try/except in finally handles this gracefully, it's cleaner to track entry success explicitly.
♻️ Suggested fix
# Propagate user/session context to Langfuse so all LLM calls within # this request are grouped under a single trace with proper attribution. _trace_ctx: Any = None + _trace_entered = False try: _trace_ctx = propagate_attributes( user_id=user_id, session_id=session_id, trace_name="copilot-baseline", tags=["baseline"], ) _trace_ctx.__enter__() + _trace_entered = True except Exception: logger.warning("[Baseline] Langfuse trace context setup failed")Then in the finally block (lines 403-408):
# Close Langfuse trace context - if _trace_ctx is not None: + if _trace_ctx is not None and _trace_entered: try: _trace_ctx.__exit__(None, None, None) except Exception: logger.warning("[Baseline] Langfuse trace context teardown failed")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/baseline/service.py` around lines 202 - 214, propagate_attributes() may return a context manager but __enter__ can fail, leaving _trace_ctx non-None and causing __exit__ to be called on an un-entered context; modify the block that calls propagate_attributes() (the _trace_ctx setup in service.py) to track entry success with a boolean (e.g., entered = False), set entered = True only after _trace_ctx.__enter__ completes, and in the finally block only call _trace_ctx.__exit__ if _trace_ctx is not None and entered is True (wrap the __exit__ call in a try/except as before for safety).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@autogpt_platform/backend/backend/copilot/baseline/service.py`:
- Line 16: The current manual context management of Langfuse via
propagate_attributes and direct __enter__/__exit__ calls can call __exit__ on an
un-entered context; modify the trace setup in the function that creates
_trace_ctx (using propagate_attributes) to track enter success with a boolean
(e.g., _trace_ctx_entered = False), set it to True only after
_trace_ctx.__enter__() completes, wrap the __enter__() call in try/except to log
via logger.warning on failure, and in the finally/teardown block only call
_trace_ctx.__exit__(...) if _trace_ctx_entered is True, wrapping that call in
its own try/except to log teardown failures; reference the existing _trace_ctx,
propagate_attributes and logger symbols when making these changes.
---
Nitpick comments:
In `@autogpt_platform/backend/backend/copilot/baseline/service.py`:
- Around line 202-214: propagate_attributes() may return a context manager but
__enter__ can fail, leaving _trace_ctx non-None and causing __exit__ to be
called on an un-entered context; modify the block that calls
propagate_attributes() (the _trace_ctx setup in service.py) to track entry
success with a boolean (e.g., entered = False), set entered = True only after
_trace_ctx.__enter__ completes, and in the finally block only call
_trace_ctx.__exit__ if _trace_ctx is not None and entered is True (wrap the
__exit__ call in a try/except as before for safety).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ddd7101c-eb1b-4fa8-a078-87bcb6b2863b
📒 Files selected for processing (2)
autogpt_platform/backend/backend/copilot/baseline/service.pyautogpt_platform/backend/backend/copilot/service.py
🚧 Files skipped from review as they are similar to previous changes (1)
- autogpt_platform/backend/backend/copilot/service.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: Seer Code Review
- GitHub Check: types
- GitHub Check: test (3.11)
- GitHub Check: test (3.13)
- GitHub Check: test (3.12)
- GitHub Check: Analyze (python)
- GitHub Check: Analyze (typescript)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (4)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/baseline/service.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/baseline/service.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/baseline/service.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/baseline/service.py
🧠 Learnings (2)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/baseline/service.py
📚 Learning: 2026-03-04T08:04:35.881Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12273
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:216-220
Timestamp: 2026-03-04T08:04:35.881Z
Learning: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path.
Applied to files:
autogpt_platform/backend/backend/copilot/baseline/service.py
🔇 Additional comments (1)
autogpt_platform/backend/backend/copilot/baseline/service.py (1)
402-408: LGTM on the teardown pattern.The finally block correctly ensures trace context cleanup, with proper null check and defensive exception handling to avoid blocking the main flow.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@autogpt_platform/backend/backend/copilot/sdk/service.py`:
- Around line 82-108: The current _patch_tracer_provider_resource function
mutates private attributes (_proxy, _resource) on the global provider and runs
at import time which is fragile; instead, detect when the global provider is an
SDKTracerProvider via get_tracer_provider()/SDKTracerProvider, build a new
Resource by merging existing resource + {"langfuse.environment": environment}
using opentelemetry.sdk.resources.Resource, create a new SDK TracerProvider (or
clone via public constructor) with that merged resource, and install it with
opentelemetry.trace.set_tracer_provider() at a safe time (e.g., defer until
first use or hook into app lifecycle in _setup_langfuse_otel) so you do not
access private attributes or mutate provider internals. Ensure you no longer
reference _proxy or _resource and perform replacement only when you know the
framework has finished initializing the global TracerProvider.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 33b1b9ff-ca0e-4384-8979-0d7b6461f739
📒 Files selected for processing (1)
autogpt_platform/backend/backend/copilot/sdk/service.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: types
- GitHub Check: Seer Code Review
- GitHub Check: test (3.12)
- GitHub Check: test (3.13)
- GitHub Check: test (3.11)
- GitHub Check: Analyze (python)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (4)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/sdk/service.py
🧠 Learnings (2)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/service.py
📚 Learning: 2026-03-04T08:04:35.881Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12273
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:216-220
Timestamp: 2026-03-04T08:04:35.881Z
Learning: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/service.py
🔇 Additional comments (1)
autogpt_platform/backend/backend/copilot/sdk/service.py (1)
143-147: Cleaner approach than env-var patching; timing caveat applies.Calling
_patch_tracer_provider_resourceafterconfigure_claude_agent_sdk()is logically correct—LangSmith's helper should have installed its TracerProvider by then. The inline comment explains the rationale well.The caveat from the previous comment (framework OTEL init timing) still applies. If manual verification confirms the TracerProvider is stable at this point, this is good to merge.
a38ed94 to
c349149
Compare
Summary
Depends on #12276 (baseline code).
langfuse.openai.AsyncOpenAI— auto-captures all LLM calls (token usage, latency, model, prompts) as Langfuse generations when configuredpropagate_attributes()context in baseline streaming foruser_id/session_idattribution, matching the SDK path's OTEL tracinglangfuse.openai.AsyncOpenAIfalls back to standardopenai.AsyncOpenAIbehaviorObservability parity
configure_claude_agent_sdk()langfuse.openai.AsyncOpenAIauto-instrumentationpropagate_attributes()propagate_attributes()_build_system_prompt()_build_system_prompt()Test plan
poetry run formatpasses (pyright, ruff, black, isort)CHAT_USE_CLAUDE_AGENT_SDK=false