feat(backend/copilot): attach uploaded images and PDFs as multimodal vision blocks - #12273
Conversation
…content blocks - Bump MAX_INLINE_SIZE_BYTES from 32KB to 20MB (Claude vision limit) so read_workspace_file returns base64 for images and documents - Replace _extract_image_block with generic _extract_content_block supporting images (png/jpeg/gif/webp/svg) and documents (pdf) - Add _MULTIMODAL_TYPES mapping for easy extension of supported types - Add _INLINEABLE_MIME_TYPES superset in workspace_files.py - Thread file_ids from processor to SDK service layer - Add file attachment hint so Claude reads uploaded files via tool - Accept file_ids in non-SDK service for backward compatibility Closes OPEN-3022
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughPropagates Changes
Sequence DiagramsequenceDiagram
participant Client
participant Processor
participant Executor
participant StreamFn as StreamFn (stream_chat_completion_sdk)
participant Claude
participant WorkspaceTool as Workspace File Tool
Client->>Processor: request (may include file_ids)
Processor->>Executor: _execute_async(..., file_ids=entry.file_ids)
Executor->>StreamFn: stream_fn(..., file_ids=entry.file_ids)
StreamFn->>StreamFn: _build_file_attachment_hint(file_ids)
StreamFn->>Claude: send query + file hint
Claude->>WorkspaceTool: read_workspace_file(file_id)
WorkspaceTool->>WorkspaceTool: _extract_content_block(multimodal)
WorkspaceTool-->>Claude: return file content (image/document or text)
Claude-->>Client: streamed response (with file context)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🔴 Merge Conflicts DetectedThe following PRs have been tested and will have merge conflicts if merged after this PR. Consider coordinating with the authors.
🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 1 conflict(s), 0 medium risk, 5 low risk (out of 6 PRs with file overlap) Auto-generated on push. Ignores: |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
- Remove noqa comment from non-SDK service file_ids param - Replace if/elif branching with _BLOCK_BUILDERS dispatch table - Add bmp, tiff, svg to supported image types - Add Callable import for builder type hints - Text files already supported via existing is_text path - Videos not supported (Claude API has no native video content blocks)
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
autogpt_platform/backend/backend/copilot/sdk/service.py (1)
574-578: Docstring overstates what this function does with attachments.Current text says files are fetched/base64-attached here, but this code path only appends a hint and relies on tool calls (
read_workspace_file) for retrieval. Tightening this wording will reduce future confusion.✏️ Suggested docstring wording
- file_ids: Optional workspace file IDs attached to the user's message. - When provided, files are fetched, base64-encoded, and attached as - multimodal content blocks (images, PDFs, etc.) to the query. + file_ids: Optional workspace file IDs attached to the user's message. + When provided, a hint is appended so Claude can fetch each file + via `read_workspace_file` and process supported multimodal content.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/service.py` around lines 574 - 578, The docstring incorrectly claims that provided file_ids are fetched and base64-attached; instead update the docstring for the function that accepts the file_ids parameter to state that when file_ids are provided the function only appends hints/metadata about attachments to the query and relies on tool calls (e.g., read_workspace_file) or downstream handlers to actually fetch and decode file contents. Mention file_ids and read_workspace_file by name so readers know where responsibility for retrieval lives.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py`:
- Around line 268-273: The code currently duplicates multimodal base64 payloads
by leaving them in the plain text block and also emitting them as a separate
content_block (see _extract_content_block, content_block, content_blocks, and
the variable text), which causes large attachments to be truncated/corrupted;
update the logic so that when _extract_content_block(text) returns a
content_block you remove/strip the embedded content_base64 (or other multimodal
marker) from the text before appending it to content_blocks (and do the same fix
for the other occurrence around the 295-299 logic), ensuring only the separate
image/document content_block carries the base64 payload while the text block
contains the descriptive metadata only.
In `@autogpt_platform/backend/backend/copilot/tools/workspace_files.py`:
- Line 441: Replace the single MAX_INLINE_SIZE_BYTES constant with two
caps—e.g., MAX_INLINE_SIZE_BYTES_MULTIMODAL = 20 * 1024 * 1024 and a much
smaller MAX_INLINE_SIZE_BYTES_TEXT (e.g., 100 * 1024) —and update all
inline-decision logic that currently references MAX_INLINE_SIZE_BYTES to choose
the right cap based on file type (multimodal like images/PDFs vs text/plain).
Locate uses of MAX_INLINE_SIZE_BYTES in the inline check code paths (the
function(s) that decide whether to base64-inline a file) and switch them to use
the multimodal constant when mime/type indicates images/PDFs and the text
constant for general text files so large text files are not inlined.
- Around line 216-220: The _INLINEABLE_MIME_TYPES set is missing
"image/svg+xml", so SVG files aren't returned inline as base64; update the
definition of _INLINEABLE_MIME_TYPES (which currently unions _IMAGE_MIME_TYPES
with {"application/pdf"}) to include "image/svg+xml" (or add it into
_IMAGE_MIME_TYPES if more appropriate) so SVG uploads are treated as inlineable
multimodal content; ensure the symbol name _INLINEABLE_MIME_TYPES is modified
and that any tests or callers expecting inline SVGs now receive base64 data.
---
Nitpick comments:
In `@autogpt_platform/backend/backend/copilot/sdk/service.py`:
- Around line 574-578: The docstring incorrectly claims that provided file_ids
are fetched and base64-attached; instead update the docstring for the function
that accepts the file_ids parameter to state that when file_ids are provided the
function only appends hints/metadata about attachments to the query and relies
on tool calls (e.g., read_workspace_file) or downstream handlers to actually
fetch and decode file contents. Mention file_ids and read_workspace_file by name
so readers know where responsibility for retrieval lives.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (5)
autogpt_platform/backend/backend/copilot/executor/processor.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: types
- GitHub Check: Seer Code Review
- GitHub Check: test (3.12)
- GitHub Check: test (3.11)
- GitHub Check: test (3.13)
- GitHub Check: Analyze (python)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (4)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/executor/processor.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/executor/processor.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/executor/processor.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/executor/processor.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/service.py
🧠 Learnings (1)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/executor/processor.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/service.py
🧬 Code graph analysis (1)
autogpt_platform/backend/backend/copilot/sdk/service.py (2)
autogpt_platform/backend/backend/copilot/response_model.py (1)
StreamBaseResponse(48-56)autogpt_platform/backend/backend/util/logging.py (1)
info(41-43)
🔇 Additional comments (3)
autogpt_platform/backend/backend/copilot/service.py (1)
326-326: Compatibility signature update is solid.Accepting
file_idshere prevents stream function signature drift when the processor routes to the non-SDK path.autogpt_platform/backend/backend/copilot/executor/processor.py (1)
232-232:file_idspropagation is wired correctly here.This is the right place to thread attachment context into both streaming backends.
autogpt_platform/backend/backend/copilot/sdk/service.py (1)
852-857: Attachment hint injection + telemetry update look good.Appending the per-turn file hint and logging
attached_filesgives clear traceability for attachment-aware prompts.Also applies to: 859-865
There was a problem hiding this comment.
♻️ Duplicate comments (2)
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (1)
263-273:⚠️ Potential issue | 🟠 MajorAvoid duplicating
content_base64in both text and multimodal blocks.At Line 267–273, the raw JSON text block still carries
content_base64while a multimodal block is also appended. This duplicates large payloads and can trigger truncation/corruption under response-size limits.🔧 Suggested fix
- content_blocks: list[dict[str, str]] = [{"type": "text", "text": text}] + text_block_text = text + content_blocks: list[dict[str, Any]] = [] @@ content_block = _extract_content_block(text) if content_block: + try: + payload = json.loads(text) + if isinstance(payload, dict) and "content_base64" in payload: + payload["content_base64"] = "[omitted: delivered via multimodal block]" + text_block_text = json.dumps(payload) + except (json.JSONDecodeError, TypeError): + pass content_blocks.append(content_block) + content_blocks.insert(0, {"type": "text", "text": text_block_text})🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py` around lines 263 - 273, The text block currently includes the full raw JSON (variable text) even when _extract_content_block returns a multimodal content_block, causing duplicated content_base64; change the logic in the area that builds content_blocks (using the text variable and _extract_content_block) to strip or remove any content_base64 fields from the JSON/text used for the "text" content block when a multimodal content_block is present (e.g., parse result.output into a dict, delete content_base64 entries or replace them with a placeholder, then json.dumps that cleaned payload into text) so only the multimodal content_block carries the large base64 payload.autogpt_platform/backend/backend/copilot/tools/workspace_files.py (1)
444-445:⚠️ Potential issue | 🟠 MajorSplit text vs multimodal inline size caps.
Line 444 and Line 549 currently allow text files up to 20MB to be inlined, which can bloat MCP payloads and context unnecessarily. Keep the large cap for multimodal only, and retain a small text cap.
🔧 Suggested fix
class ReadWorkspaceFileTool(BaseTool): """Tool for reading file content from workspace.""" - MAX_INLINE_SIZE_BYTES = 20 * 1024 * 1024 # 20MB (Claude vision/document limit) + MAX_INLINE_TEXT_SIZE_BYTES = 32 * 1024 + MAX_INLINE_MULTIMODAL_SIZE_BYTES = 20 * 1024 * 1024 # 20MB PREVIEW_SIZE = 500 @@ - is_small = file_info.size_bytes <= self.MAX_INLINE_SIZE_BYTES is_text = _is_text_mime(file_info.mime_type) is_inlineable = file_info.mime_type in _INLINEABLE_MIME_TYPES + is_small_text = is_text and ( + file_info.size_bytes <= self.MAX_INLINE_TEXT_SIZE_BYTES + ) + is_small_multimodal = is_inlineable and ( + file_info.size_bytes <= self.MAX_INLINE_MULTIMODAL_SIZE_BYTES + ) @@ - if is_small and (is_text or is_inlineable) and not force_download_url: + if (is_small_text or is_small_multimodal) and not force_download_url:Also applies to: 544-549
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/tools/workspace_files.py` around lines 444 - 445, Replace the single MAX_INLINE_SIZE_BYTES with two caps: a small TEXT_MAX_INLINE_SIZE_BYTES (e.g., 500KB) and a large MULTIMODAL_MAX_INLINE_SIZE_BYTES (20MB) and leave PREVIEW_SIZE as-is; then update all places that compare a file's size against MAX_INLINE_SIZE_BYTES (look for functions/methods like can_inline_file, _should_inline, should_inline, or any size checks in workspace_files.py that use MAX_INLINE_SIZE_BYTES) to choose the appropriate cap based on the file type (use existing file metadata/mimetype or file.is_multimodal/is_text helpers) so text files use TEXT_MAX_INLINE_SIZE_BYTES and images/docs/multimodal files use MULTIMODAL_MAX_INLINE_SIZE_BYTES.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py`:
- Around line 263-273: The text block currently includes the full raw JSON
(variable text) even when _extract_content_block returns a multimodal
content_block, causing duplicated content_base64; change the logic in the area
that builds content_blocks (using the text variable and _extract_content_block)
to strip or remove any content_base64 fields from the JSON/text used for the
"text" content block when a multimodal content_block is present (e.g., parse
result.output into a dict, delete content_base64 entries or replace them with a
placeholder, then json.dumps that cleaned payload into text) so only the
multimodal content_block carries the large base64 payload.
In `@autogpt_platform/backend/backend/copilot/tools/workspace_files.py`:
- Around line 444-445: Replace the single MAX_INLINE_SIZE_BYTES with two caps: a
small TEXT_MAX_INLINE_SIZE_BYTES (e.g., 500KB) and a large
MULTIMODAL_MAX_INLINE_SIZE_BYTES (20MB) and leave PREVIEW_SIZE as-is; then
update all places that compare a file's size against MAX_INLINE_SIZE_BYTES (look
for functions/methods like can_inline_file, _should_inline, should_inline, or
any size checks in workspace_files.py that use MAX_INLINE_SIZE_BYTES) to choose
the appropriate cap based on the file type (use existing file metadata/mimetype
or file.is_multimodal/is_text helpers) so text files use
TEXT_MAX_INLINE_SIZE_BYTES and images/docs/multimodal files use
MULTIMODAL_MAX_INLINE_SIZE_BYTES.
ℹ️ Review info
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (3)
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.py
🚧 Files skipped from review as they are similar to previous changes (1)
- autogpt_platform/backend/backend/copilot/service.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
- GitHub Check: types
- GitHub Check: test (3.13)
- GitHub Check: test (3.11)
- GitHub Check: test (3.12)
- GitHub Check: Seer Code Review
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (4)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
🧠 Learnings (2)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
📚 Learning: 2026-02-27T10:45:55.700Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12213
File: autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx:23-24
Timestamp: 2026-02-27T10:45:55.700Z
Learning: As of PR `#12213`, MCP tool response types (MCPToolsDiscoveredResponse, MCPToolOutputResponse) are defined in openapi.json and frontend code in autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx uses the generated types from `@/app/api/__generated__/`. Other tools like RunBlock still use inline TypeScript interfaces (e.g., BlockDetailsResponse) for SSE stream payloads that are not included in openapi.json schemas. The pattern is tool-specific: use generated types when available in openapi.json, use inline types only when the payload schema is truly SSE-stream-only and not exposed via OpenAPI.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
🔇 Additional comments (2)
autogpt_platform/backend/backend/copilot/tools/workspace_files.py (1)
216-223: Good MIME coverage for inline multimodal types.Line 216–223 aligns the inlineable set with the new multimodal scope (SVG/BMP/TIFF/PDF), which keeps read behavior consistent with downstream block extraction.
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (1)
281-305: Nice extensible multimodal dispatch design.The MIME mapping + builder table keeps Line 281–357 easy to extend and avoids branching growth.
Also applies to: 307-357
- Remove unsupported image types (svg, bmp, tiff) from multimodal types (Claude only supports png, jpeg, gif, webp) - Split inline size thresholds: 32KB for text, 20MB for images/PDFs (prevents bloating context with large text files) - Flatten nested if for file_ids hint (walrus operator)
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
autogpt_platform/backend/backend/copilot/tools/workspace_files.py (1)
542-567:⚠️ Potential issue | 🟠 MajorNormalize MIME type before inline checks and response emission.
file_info.mime_typeis used raw here, butsdk/tool_adapter.pydoes exact MIME matching in_MULTIMODAL_TYPES. If MIME includes parameters, multimodal extraction can silently fail even when content is inlined.🔧 Proposed fix
- is_text = _is_text_mime(file_info.mime_type) - is_inlineable = file_info.mime_type in _INLINEABLE_MIME_TYPES + normalized_mime = (file_info.mime_type or "").split(";", 1)[0].strip().lower() + is_text = _is_text_mime(normalized_mime) + is_inlineable = normalized_mime in _INLINEABLE_MIME_TYPES @@ return WorkspaceFileContentResponse( file_id=file_info.id, name=file_info.name, path=file_info.path, - mime_type=file_info.mime_type, + mime_type=normalized_mime, content_base64=base64.b64encode(content).decode("utf-8"), message=msg, session_id=session_id, )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/tools/workspace_files.py` around lines 542 - 567, Normalize file_info.mime_type by stripping any parameters before doing inline checks and returning it in the response: compute a local mime_type = file_info.mime_type.split(';', 1)[0].strip() (or equivalent) and use mime_type when calling _is_text_mime, checking membership in _INLINEABLE_MIME_TYPES, computing is_inlineable/is_text, and when populating WorkspaceFileContentResponse.mime_type; leave manager.read_file_by_id, size checks (MAX_INLINE_MULTIMODAL_SIZE_BYTES / MAX_INLINE_TEXT_SIZE_BYTES) and content encoding unchanged. This ensures exact MIME matching (matching _MULTIMODAL_TYPES behavior) and that the emitted response contains the normalized MIME string.
♻️ Duplicate comments (1)
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (1)
269-273:⚠️ Potential issue | 🟠 MajorRemove duplicated multimodal base64 from the text block.
At Line 269-273, multimodal payload is still sent twice: once inside the JSON text block and again as the multimodal block. This reintroduces truncation/corruption risk for large attachments in the SDK path.
Proposed fix
- text = ( - result.output if isinstance(result.output, str) else json.dumps(result.output) - ) - - content_blocks: list[dict[str, str]] = [{"type": "text", "text": text}] + text = ( + result.output if isinstance(result.output, str) else json.dumps(result.output) + ) + text_block_text = text + content_blocks: list[dict[str, Any]] = [] # If the tool result contains inline multimodal data, add a content block # so Claude can "see" images or read documents (e.g. read_workspace_file). content_block = _extract_content_block(text) if content_block: + try: + payload = json.loads(text) + if isinstance(payload, dict) and "content_base64" in payload: + payload["content_base64"] = "[omitted: delivered via multimodal block]" + text_block_text = json.dumps(payload) + except (json.JSONDecodeError, TypeError): + pass content_blocks.append(content_block) + content_blocks.insert(0, {"type": "text", "text": text_block_text})🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py` around lines 269 - 273, The text block currently contains inline multimodal base64 and is also extracted into a separate multimodal content block (via _extract_content_block), causing duplicated payloads; modify the flow so that after calling _extract_content_block(text) you remove or replace the embedded multimodal data from the original text before appending it (or change _extract_content_block to return both the extracted content block and a cleaned_text); ensure only the cleaned text is added to the message payload while the multimodal data lives solely in content_blocks (refer to the _extract_content_block function name, the text variable, and the content_blocks list for where to make the change).
🧹 Nitpick comments (2)
autogpt_platform/backend/backend/copilot/sdk/service.py (1)
568-577: Minor docstring clarification suggested.The docstring states files are "fetched, base64-encoded, and attached as multimodal content blocks" in this function, but the actual implementation appends a text hint instructing Claude to use
read_workspace_file. The fetching and encoding occur when Claude invokes that tool. Consider clarifying:📝 Suggested docstring update
Args: file_ids: Optional workspace file IDs attached to the user's message. - When provided, files are fetched, base64-encoded, and attached as - multimodal content blocks (images, PDFs, etc.) to the query. + When provided, a hint is appended to the query instructing Claude + to use read_workspace_file to access the content (which returns + base64-encoded multimodal content blocks for images, PDFs, etc.).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/service.py` around lines 568 - 577, The docstring for the async generator function (returning AsyncGenerator[StreamBaseResponse, None]) incorrectly says workspace files are "fetched, base64-encoded, and attached"; update the docstring to state that when file_ids are provided the function appends text hints instructing the Claude agent to call the read_workspace_file tool, and that actual fetching/encoding is performed by the agent/tool at runtime rather than by this function (reference the function signature and the file_ids parameter to locate the docstring to edit).autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (1)
336-347: Harden parsed payload type checks before size validation.
data.get(...)may return non-string values. Adding strictstrchecks avoids malformed block construction and makes this parser safer.Proposed hardening
- mime_type: str = data.get("mime_type", "") - base64_content: str = data.get("content_base64", "") - if not mime_type or not base64_content: + mime_type = data.get("mime_type", "") + base64_content = data.get("content_base64", "") + if ( + not isinstance(mime_type, str) + or not isinstance(base64_content, str) + or not mime_type + or not base64_content + ): return None🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py` around lines 336 - 347, The parser currently assumes data.get("mime_type") and data.get("content_base64") are strings before using them and before checking size; add explicit type checks/casts so non-string values don't slip through. In the block that reads mime_type and base64_content, ensure you validate types (e.g., isinstance(..., str)) or coerce safely to str only after confirming non-None, return None if either is not a string, then continue to lookup _MULTIMODAL_TYPES and compare len(base64_content) against max_b64 (from entry) knowing base64_content is a real string; reference mime_type, base64_content, _MULTIMODAL_TYPES, and the block returning None on size check.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@autogpt_platform/backend/backend/copilot/tools/workspace_files.py`:
- Around line 542-567: Normalize file_info.mime_type by stripping any parameters
before doing inline checks and returning it in the response: compute a local
mime_type = file_info.mime_type.split(';', 1)[0].strip() (or equivalent) and use
mime_type when calling _is_text_mime, checking membership in
_INLINEABLE_MIME_TYPES, computing is_inlineable/is_text, and when populating
WorkspaceFileContentResponse.mime_type; leave manager.read_file_by_id, size
checks (MAX_INLINE_MULTIMODAL_SIZE_BYTES / MAX_INLINE_TEXT_SIZE_BYTES) and
content encoding unchanged. This ensures exact MIME matching (matching
_MULTIMODAL_TYPES behavior) and that the emitted response contains the
normalized MIME string.
---
Duplicate comments:
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py`:
- Around line 269-273: The text block currently contains inline multimodal
base64 and is also extracted into a separate multimodal content block (via
_extract_content_block), causing duplicated payloads; modify the flow so that
after calling _extract_content_block(text) you remove or replace the embedded
multimodal data from the original text before appending it (or change
_extract_content_block to return both the extracted content block and a
cleaned_text); ensure only the cleaned text is added to the message payload
while the multimodal data lives solely in content_blocks (refer to the
_extract_content_block function name, the text variable, and the content_blocks
list for where to make the change).
---
Nitpick comments:
In `@autogpt_platform/backend/backend/copilot/sdk/service.py`:
- Around line 568-577: The docstring for the async generator function (returning
AsyncGenerator[StreamBaseResponse, None]) incorrectly says workspace files are
"fetched, base64-encoded, and attached"; update the docstring to state that when
file_ids are provided the function appends text hints instructing the Claude
agent to call the read_workspace_file tool, and that actual fetching/encoding is
performed by the agent/tool at runtime rather than by this function (reference
the function signature and the file_ids parameter to locate the docstring to
edit).
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py`:
- Around line 336-347: The parser currently assumes data.get("mime_type") and
data.get("content_base64") are strings before using them and before checking
size; add explicit type checks/casts so non-string values don't slip through. In
the block that reads mime_type and base64_content, ensure you validate types
(e.g., isinstance(..., str)) or coerce safely to str only after confirming
non-None, return None if either is not a string, then continue to lookup
_MULTIMODAL_TYPES and compare len(base64_content) against max_b64 (from entry)
knowing base64_content is a real string; reference mime_type, base64_content,
_MULTIMODAL_TYPES, and the block returning None on size check.
ℹ️ Review info
Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 5a5a4517-b192-468f-81ad-d901e499f321
📒 Files selected for processing (3)
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: types
- GitHub Check: Seer Code Review
- GitHub Check: test (3.12)
- GitHub Check: test (3.13)
- GitHub Check: test (3.11)
- GitHub Check: Analyze (python)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (4)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
🧠 Learnings (2)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
📚 Learning: 2026-02-27T10:45:55.700Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12213
File: autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx:23-24
Timestamp: 2026-02-27T10:45:55.700Z
Learning: As of PR `#12213`, MCP tool response types (MCPToolsDiscoveredResponse, MCPToolOutputResponse) are defined in openapi.json and frontend code in autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx uses the generated types from `@/app/api/__generated__/`. Other tools like RunBlock still use inline TypeScript interfaces (e.g., BlockDetailsResponse) for SSE stream payloads that are not included in openapi.json schemas. The pattern is tool-specific: use generated types when available in openapi.json, use inline types only when the payload schema is truly SSE-stream-only and not exposed via OpenAPI.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
🔇 Additional comments (7)
autogpt_platform/backend/backend/copilot/sdk/service.py (3)
539-556: LGTM!The helper function is well-designed with clear docstring explaining the purpose. The early return for empty
file_idsis appropriately defensive, and the message format gives Claude clear instructions on how to access the attached files.
851-854: LGTM!Clean use of the walrus operator to conditionally build and append the file hint. The double newline separator provides clear visual separation between the user's query and the file attachment instructions.
856-863: LGTM!Good observability enhancement. The
attached_filescount in the log message will help with debugging file attachment flows. The null-safe ternary correctly handles bothNoneand empty list cases.autogpt_platform/backend/backend/copilot/tools/workspace_files.py (2)
216-220: Good multimodal inline type expansion.Adding
application/pdfto_INLINEABLE_MIME_TYPESkeeps this tool aligned with multimodal handling goals and the current supported set in the adapter.
441-442: Nice threshold split for text vs multimodal content.Keeping text at 32KB while allowing 20MB for multimodal files addresses context bloat without blocking image/PDF attachment flows.
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (2)
14-14: Nice import modernization.Using
Callablefromcollections.abcis the correct modern typing import for Python 3.11.
291-315: MIME-to-block dispatch structure looks good.The
_MULTIMODAL_TYPES+_BLOCK_BUILDERSsplit is clean and extensible, and the supported MIME set aligns with the inline workspace types.
…ents - Protect multimodal content blocks from truncation corruption - Strip duplicated base64 from text blocks when multimodal block exists - Normalize MIME type before inline checks (strip parameters like charset) - Harden type checks in _extract_content_block for non-string values - Fix docstring accuracy (hint-based, not direct attachment) - Warn when file_ids passed to non-SDK path - Add comprehensive tests for multimodal extraction, truncation, and helpers
majdyz
changed the title
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
autogpt_platform/backend/backend/copilot/sdk/service.py (1)
539-556: Consider bounding attachment hint size to avoid prompt bloat.If many files are attached, this helper can add a large token payload. A small cap (count + ID length) keeps prompts predictable.
🔧 Proposed hardening
def _build_file_attachment_hint(file_ids: list[str]) -> str: @@ - ids_list = ", ".join(f"`{fid}`" for fid in file_ids) - noun = "file" if len(file_ids) == 1 else "files" + MAX_IDS_IN_HINT = 20 + clipped = [str(fid)[:64] for fid in file_ids[:MAX_IDS_IN_HINT]] + ids_list = ", ".join(f"`{fid}`" for fid in clipped) + noun = "file" if len(file_ids) == 1 else "files" + suffix = ( + f" (+{len(file_ids) - MAX_IDS_IN_HINT} more not listed)" + if len(file_ids) > MAX_IDS_IN_HINT + else "" + ) return ( f"[The user attached {len(file_ids)} {noun} to this message. " - f"File IDs: {ids_list}. " + f"File IDs: {ids_list}{suffix}. " f"Use the read_workspace_file tool with the file_id to view " f"the content of each attached file.]" )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/service.py` around lines 539 - 556, The helper _build_file_attachment_hint can produce very large hints when file_ids is long; modify it to enforce a hard cap by limiting the number of IDs included and total ID characters: compute a max_files_to_show (e.g. 10) and/or max_total_id_chars (e.g. 256), build ids_list from the first N file_ids truncating individual IDs with ellipses if they exceed per-id limit, and if any IDs were omitted include a short suffix like "`... and X more files`" or "`(Y more IDs omitted)`"; update references to file_ids, ids_list, noun and the returned string accordingly so the hint stays bounded and you still report the total count via len(file_ids).autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py (1)
157-239: Add a regression test for MIME values with parameters.Given the extraction path depends on MIME matching, add a case like
application/pdf; charset=binaryto lock behavior and prevent future regressions.✅ Suggested test addition
class TestExtractContentBlock: @@ def test_pdf_returns_document_block(self): @@ assert block["source"]["data"] == "JVBERi0=" + + def test_pdf_mime_with_parameters_is_supported(self): + payload = json.dumps( + { + "content_base64": "JVBERi0=", + "mime_type": "application/pdf; charset=binary", + } + ) + block = _extract_content_block(payload) + assert block is not None + assert block["type"] == "document" + assert block["source"]["media_type"] == "application/pdf"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py` around lines 157 - 239, Add a regression test inside TestExtractContentBlock that verifies MIME values with parameters are accepted: create a JSON payload with "content_base64": "JVBERi0=" and "mime_type": "application/pdf; charset=binary", call _extract_content_block(payload), assert the result is not None, assert block["type"] == "document" and that block["source"]["media_type"] starts with "application/pdf" (or equals "application/pdf" if implementation strips parameters). Place this new test near test_pdf_returns_document_block so future changes to _extract_content_block will be covered.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py`:
- Around line 340-349: The MIME type string from data should be normalized
before lookup so parameterized values like "application/pdf; charset=binary"
match keys in _MULTIMODAL_TYPES: trim whitespace, lowercase, and split on ';'
taking the media type only (update the local mime_type variable used in the
function in tool_adapter.py), then use that normalized value for the lookup at
entry = _MULTIMODAL_TYPES.get(mime_type) and the second lookup around line 359;
ensure you apply the same normalization wherever mime_type is read from data
(e.g., content_base64 handling) to prevent missed multimodal extraction.
- Around line 295-296: The base64 ceiling constants are too low: update
_IMAGE_MAX_B64 and _DOCUMENT_MAX_B64 to account for base64 expansion (4/3) of
binary sizes so a 20 MiB image and 32 MiB document do not get dropped; set
_IMAGE_MAX_B64 to at least 27,962,028 (ceil(20*1024*1024*4/3)) and
_DOCUMENT_MAX_B64 to at least 44,739,243 (ceil(32*1024*1024*4/3)), and adjust
any checks that compare base64 length (the code paths that drop multimodal block
assets) to use these revised constants (_IMAGE_MAX_B64, _DOCUMENT_MAX_B64).
---
Nitpick comments:
In `@autogpt_platform/backend/backend/copilot/sdk/service.py`:
- Around line 539-556: The helper _build_file_attachment_hint can produce very
large hints when file_ids is long; modify it to enforce a hard cap by limiting
the number of IDs included and total ID characters: compute a max_files_to_show
(e.g. 10) and/or max_total_id_chars (e.g. 256), build ids_list from the first N
file_ids truncating individual IDs with ellipses if they exceed per-id limit,
and if any IDs were omitted include a short suffix like "`... and X more files`"
or "`(Y more IDs omitted)`"; update references to file_ids, ids_list, noun and
the returned string accordingly so the hint stays bounded and you still report
the total count via len(file_ids).
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py`:
- Around line 157-239: Add a regression test inside TestExtractContentBlock that
verifies MIME values with parameters are accepted: create a JSON payload with
"content_base64": "JVBERi0=" and "mime_type": "application/pdf; charset=binary",
call _extract_content_block(payload), assert the result is not None, assert
block["type"] == "document" and that block["source"]["media_type"] starts with
"application/pdf" (or equals "application/pdf" if implementation strips
parameters). Place this new test near test_pdf_returns_document_block so future
changes to _extract_content_block will be covered.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 2629883c-2462-4edc-9349-086c5170af02
📒 Files selected for processing (6)
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/tools/workspace_files.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: Seer Code Review
- GitHub Check: types
- GitHub Check: Analyze (python)
- GitHub Check: Check PR Status
- GitHub Check: test (3.11)
- GitHub Check: test (3.12)
- GitHub Check: test (3.13)
🧰 Additional context used
📓 Path-based instructions (6)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/service.py
autogpt_platform/backend/**/*_test.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
autogpt_platform/backend/**/*_test.py: Always review snapshot changes withgit diffbefore committing when updating snapshots withpoetry run pytest --snapshot-update
Use pytest with snapshot testing for API responses in test files
Colocate test files with source files using the*_test.pynaming convention
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
autogpt_platform/backend/**/*test*.py
📄 CodeRabbit inference engine (AGENTS.md)
Run
poetry run testfor backend testing (runs pytest with docker based postgres + prisma)
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
🧠 Learnings (5)
📚 Learning: 2026-03-04T08:04:35.881Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12273
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:216-220
Timestamp: 2026-03-04T08:04:35.881Z
Learning: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path.
Applied to files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/service.py
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/tools/workspace_files.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.pyautogpt_platform/backend/backend/copilot/sdk/service.py
📚 Learning: 2026-02-04T16:49:42.490Z
Learnt from: CR
Repo: Significant-Gravitas/AutoGPT PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2026-02-04T16:49:42.490Z
Learning: Applies to autogpt_platform/backend/backend/blocks/**/*.py : Write tests alongside block implementation when adding new blocks in backend
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py
📚 Learning: 2026-02-27T10:45:55.700Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12213
File: autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx:23-24
Timestamp: 2026-02-27T10:45:55.700Z
Learning: As of PR `#12213`, MCP tool response types (MCPToolsDiscoveredResponse, MCPToolOutputResponse) are defined in openapi.json and frontend code in autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx uses the generated types from `@/app/api/__generated__/`. Other tools like RunBlock still use inline TypeScript interfaces (e.g., BlockDetailsResponse) for SSE stream payloads that are not included in openapi.json schemas. The pattern is tool-specific: use generated types when available in openapi.json, use inline types only when the payload schema is truly SSE-stream-only and not exposed via OpenAPI.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
📚 Learning: 2026-02-04T16:50:20.508Z
Learnt from: CR
Repo: Significant-Gravitas/AutoGPT PR: 0
File: autogpt_platform/backend/CLAUDE.md:0-0
Timestamp: 2026-02-04T16:50:20.508Z
Learning: Applies to autogpt_platform/backend/backend/blocks/*.py : Never hardcode workspace checks when using `store_media_file()` - let `for_block_output` handle context adaptation automatically
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
🔇 Additional comments (5)
autogpt_platform/backend/backend/copilot/service.py (1)
326-374: Nice compatibility guard forfile_idsin non-SDK flow.The signature expansion plus explicit warning path on Line 326 and Lines 368-374 is clean and prevents silent misuse.
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (1)
267-277: Great handling of multimodal truncation safety and base64 de-duplication.This keeps non-text blocks intact while truncating only text, and avoids corrupting multimodal payloads.
Also applies to: 517-543
autogpt_platform/backend/backend/copilot/tools/workspace_files.py (1)
216-220: Looks good: MIME normalization + split inline thresholds are correctly implemented.The text vs multimodal size split and normalized MIME checks are consistent with the intended behavior and avoid the previous large-text inlining problem.
Based on learnings: "In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path."
Also applies to: 441-557
autogpt_platform/backend/backend/copilot/sdk/service_test.py (1)
6-26: Good helper coverage for attachment hint generation.The empty/singular/plural assertions cover the core contract of
_build_file_attachment_hintwell.autogpt_platform/backend/backend/copilot/sdk/service.py (1)
568-568:file_idspropagation and query-level hint injection look correct.This wires attachment context into the SDK path without changing the stream contract.
Also applies to: 852-864
- Increase _IMAGE_MAX_B64 from 27M to 28M so 20 MiB images are not dropped (ceil(20*1024*1024 * 4/3) ≈ 27,962,028) - Normalize MIME type in _extract_content_block (strip parameters like "application/pdf; charset=binary") to match workspace_files.py - Add tests for MIME normalization in multimodal extraction
Extract the content-block splitting logic into a reusable _split_content_blocks() function used by both _text_from_mcp_result and the _truncating wrapper.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py (1)
199-207: Add an explicit SVG exclusion regression testPlease add a dedicated
image/svg+xmlcase in the unsupported MIME tests so this behavior is locked and doesn’t regress.Based on learnings: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding
image/svg+xmlfromINLINEABLE_MIME_TYPESandMULTIMODAL_TYPESintool_adapter.py.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py` around lines 199 - 207, Add a regression test case asserting that payloads with mime_type "image/svg+xml" are treated as unsupported: update the test function (or add a new test) to call _extract_content_block with a JSON payload where "mime_type": "image/svg+xml" and assert the result is None. Also ensure the implementation in tool_adapter.py excludes "image/svg+xml" from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES so SVGs are not classified as vision/inlineable image types (adjust those constants or their construction accordingly).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py`:
- Around line 223-230: The test builds an oversized base64 string using the
wrong constant (_MCP_MAX_CHARS) so it may not exceed the image ceiling checked
by _extract_content_block; change the test to construct huge using
_IMAGE_MAX_B64 (e.g. huge = "A" * (_IMAGE_MAX_B64 + 1_000_000)) so the payload's
content_base64 truly exceeds the image limit and the assert
_extract_content_block(payload) is None reliably validates the oversize-image
branch.
---
Nitpick comments:
In `@autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py`:
- Around line 199-207: Add a regression test case asserting that payloads with
mime_type "image/svg+xml" are treated as unsupported: update the test function
(or add a new test) to call _extract_content_block with a JSON payload where
"mime_type": "image/svg+xml" and assert the result is None. Also ensure the
implementation in tool_adapter.py excludes "image/svg+xml" from
INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES so SVGs are not classified as
vision/inlineable image types (adjust those constants or their construction
accordingly).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e1dffd99-c5dc-4532-bd03-d631bcbdcb3a
📒 Files selected for processing (2)
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
- GitHub Check: Seer Code Review
- GitHub Check: types
- GitHub Check: test (3.12)
- GitHub Check: Analyze (python)
- GitHub Check: test (3.13)
- GitHub Check: test (3.11)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (6)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/**/*.{py,txt}
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use
poetry runprefix for all Python commands, including testing, linting, formatting, and migrations
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/**/*_test.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
autogpt_platform/backend/**/*_test.py: Always review snapshot changes withgit diffbefore committing when updating snapshots withpoetry run pytest --snapshot-update
Use pytest with snapshot testing for API responses in test files
Colocate test files with source files using the*_test.pynaming convention
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py
autogpt_platform/backend/backend/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/CLAUDE.md)
Use Prisma ORM for database operations in PostgreSQL with pgvector for embeddings
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
autogpt_platform/backend/**/*test*.py
📄 CodeRabbit inference engine (AGENTS.md)
Run
poetry run testfor backend testing (runs pytest with docker based postgres + prisma)
Files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.py
🧠 Learnings (3)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
📚 Learning: 2026-03-04T08:04:35.881Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12273
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:216-220
Timestamp: 2026-03-04T08:04:35.881Z
Learning: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter_test.pyautogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
📚 Learning: 2026-02-27T10:45:55.700Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12213
File: autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx:23-24
Timestamp: 2026-02-27T10:45:55.700Z
Learning: As of PR `#12213`, MCP tool response types (MCPToolsDiscoveredResponse, MCPToolOutputResponse) are defined in openapi.json and frontend code in autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx uses the generated types from `@/app/api/__generated__/`. Other tools like RunBlock still use inline TypeScript interfaces (e.g., BlockDetailsResponse) for SSE stream payloads that are not included in openapi.json schemas. The pattern is tool-specific: use generated types when available in openapi.json, use inline types only when the payload schema is truly SSE-stream-only and not exposed via OpenAPI.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py
🔇 Additional comments (2)
autogpt_platform/backend/backend/copilot/sdk/tool_adapter.py (2)
267-277: Good multimodal/text dedup handlingThis correctly avoids duplicating
content_base64in both text and multimodal blocks, which keeps payloads smaller and cleaner for downstream handling.
519-545: Truncation hardening looks solidSeparating non-text blocks before
truncate()and reattaching them intact is the right protection against base64/data corruption.
Dev refactored stream_chat_completion into baseline/service.py and changed SDK function signature to **_kwargs. Kept file_ids as explicit param in SDK path, added _on_stop and CompactionTracker from dev's restructured code, and re-applied vision content block logic.
github-actions
Bot
removed
the
conflicts
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
github-project-automation
Bot
moved this from 🆕 Needs initial review
to 🚧 Needs work
in AutoGPT development kanban
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
github-actions
Bot
added
the
conflicts
…eModel - Resolve workspace_files.py conflict: take dev's ranged read and image inlining, restore _IMAGE_MIME_TYPES, use MAX_INLINE_SIZE_BYTES - Convert PreparedAttachments from @DataClass to Pydantic BaseModel (per reviewer feedback)
github-actions
Bot
removed
the
conflicts
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
github-project-automation
Bot
moved this from 🚧 Needs work
to ✅ Done
in AutoGPT development kanban
Issues attributed to commits in this pull requestThis pull request was merged and Sentry observed the following issues:
|
4 participants
Requested by @majdyz
When users upload images or PDFs to CoPilot, the AI couldn't see the content because the CLI's Zod validator rejects large base64 in MCP tool results and even small images were misidentified (the CLI silently drops or corrupts image content blocks in tool results).
Approach
Embed uploaded images directly as vision content blocks in the user message via
client._transport.write(). The SDK'sclient.query()only accepts string content, so we bypass it for multimodal messages — writing a properly structured user message with[...image_blocks, {"type": "text", "text": query}]directly to the transport. This ensures the CLI binary receives images as native vision blocks, matching how the Anthropic API handles multimodal input.For binary files accessed via workspace tools at runtime, we save them to the SDK's ephemeral working directory (
sdk_cwd) and return a file path for the CLI's built-inReadtool to handle natively.Changes
Vision content blocks for attached files —
service.py_prepare_file_attachmentsdownloads workspace files before the query, converts images to base64 vision blocks ({"type": "image", "source": {"type": "base64", ...}})client._transportinstead of usingclient.query()sdk_cwdwith a hint to use the Read toolFile-path based access for workspace tools —
workspace_files.pyread_workspace_filesaves binary files tosdk_cwdinstead of returning base64, returning a path for the Read toolSDK context for ephemeral directory —
tool_adapter.pysdk_cwdcontext variable so workspace tools can access the ephemeral directory_extract_content_block,_strip_base64_from_text,_BLOCK_BUILDERS, etc.)Frontend — rendering improvements
MessageAttachments.tsx— usesOutputRendererssystem (globalRegistry+OutputItem) for image/video preview rendering instead of custom componentsGenericTool.tsx— usesOutputRendererssystem for inline image rendering of base64 contentroutes.py— returns 409 for duplicate workspace filenamesTests
tool_adapter_test.py— removed multimodal extraction/stripping tests, addedget_sdk_cwdtestsservice_test.py— rewritten for_prepare_file_attachmentswith file-on-disk assertionsCloses OPEN-3022