perf(backend): enable cross-user prompt caching via SystemPromptPreset - #12758
Conversation
Use SystemPromptPreset with exclude_dynamic_sections=True in the SDK path so the Claude Code default prompt serves as a cacheable prefix shared across all users. Our custom prompt is appended after it, and dynamic sections (working dir, git status, auto-memory) are excluded from the prefix -- giving cross-user cache hits that reduce input token cost by ~90%. Add claude_agent_exclude_dynamic_sections config field (default True) to make this configurable, with fallback to raw string when disabled.
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdded Changes
Sequence Diagram(s)sequenceDiagram
participant Service as CopilotService
participant Config as ChatConfig
participant SDK as ClaudeSDK
Service->>Config: read claude_agent_cross_user_prompt_cache & system_prompt
Service->>Service: _build_system_prompt_value(system_prompt, flag)
alt flag = true
Service-->>SDK: system_prompt = { type: "preset", preset: "claude_code", append: orig, exclude_dynamic_sections: true }
else flag = false
Service-->>SDK: system_prompt = "original prompt string"
end
Service->>SDK: construct ClaudeAgentOptions(..., system_prompt)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🔴 Merge Conflicts DetectedThe following PRs have been tested and will have merge conflicts if merged after this PR. Consider coordinating with the authors.
🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 2 conflict(s), 0 medium risk, 3 low risk (out of 5 PRs with file overlap) Auto-generated on push. Ignores: |
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c6cd6883-7a7f-406f-8a61-8f99fc90165f
📒 Files selected for processing (3)
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: check API types
- GitHub Check: Seer Code Review
- GitHub Check: type-check (3.11)
- GitHub Check: test (3.13)
- GitHub Check: test (3.11)
- GitHub Check: type-check (3.12)
- GitHub Check: type-check (3.13)
- GitHub Check: test (3.12)
- GitHub Check: end-to-end tests
- GitHub Check: Check PR Status
- GitHub Check: Analyze (python)
🧰 Additional context used
📓 Path-based instructions (3)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
autogpt_platform/backend/**/*.py: Usepoetry run ...command for executing Python package dependencies
Use top-level imports only — avoid local/inner imports except for lazy imports of heavy optional dependencies likeopenpyxl
Use absolute imports withfrom backend.module import ...for cross-package imports; single-dot relative imports are acceptable for sibling modules within the same package; avoid double-dot relative imports
Do not use duck typing — avoidhasattr/getattr/isinstancefor type dispatch; use typed interfaces/unions/protocols instead
Use Pydantic models over dataclass/namedtuple/dict for structured data
Do not use linter suppressors — no# type: ignore,# noqa,# pyright: ignore; fix the type/code instead
Prefer list comprehensions over manual loop-and-append patterns
Use early return with guard clauses first to avoid deep nesting
Use%sfor deferred interpolation indebuglog statements for efficiency; use f-strings elsewhere for readability (e.g.,logger.debug("Processing %s items", count)vslogger.info(f"Processing {count} items"))
Sanitize error paths by usingos.path.basename()in error messages to avoid leaking directory structure
Be aware of TOCTOU (Time-Of-Check-Time-Of-Use) issues — avoid check-then-act patterns for file access and credit charging
Usetransaction=Truefor Redis pipelines to ensure atomicity on multi-step operations
Usemax(0, value)guards for computed values that should never be negative
Keep files under ~300 lines; if a file grows beyond this, split by responsibility (extract helpers, models, or a sub-module into a new file)
Keep functions under ~40 lines; extract named helpers when a function grows longer
...
Files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
autogpt_platform/{backend,autogpt_libs}/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
autogpt_platform/backend/**/*_test.py
📄 CodeRabbit inference engine (autogpt_platform/backend/AGENTS.md)
autogpt_platform/backend/**/*_test.py: Use pytest with snapshot testing for API responses
Colocate test files with source files using*_test.pynaming convention
Mock at boundaries — mock where the symbol is used, not where it's defined; after refactoring, update mock targets to match new module paths
UseAsyncMockfromunittest.mockfor async functions in tests
When writing tests, use Test-Driven Development (TDD): write failing tests marked with@pytest.mark.xfailbefore implementation, then remove the marker once the implementation is complete
When creating snapshots in tests, usepoetry run pytest path/to/test.py --snapshot-update; always review snapshot changes withgit diffbefore committing
Files:
autogpt_platform/backend/backend/copilot/sdk/service_test.py
🧠 Learnings (8)
📚 Learning: 2026-03-27T08:39:45.696Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12592
File: autogpt_platform/frontend/AGENTS.md:1-3
Timestamp: 2026-03-27T08:39:45.696Z
Learning: In Significant-Gravitas/AutoGPT, Claude is the primary coding agent. AGENTS.md files intentionally retain Claude-specific wording (e.g., "CLAUDE.md - Frontend", "This file provides guidance to Claude Code") even though AGENTS.md is the canonical cross-agent instruction source. Do not flag Claude-specific titles or phrasing in AGENTS.md files as issues.
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.py
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📚 Learning: 2026-03-04T08:04:35.881Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12273
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:216-220
Timestamp: 2026-03-04T08:04:35.881Z
Learning: In the AutoGPT Copilot backend, ensure that SVG images are not treated as vision image types by excluding 'image/svg+xml' from INLINEABLE_MIME_TYPES and MULTIMODAL_TYPES in tool_adapter.py; the Claude API supports PNG, JPEG, GIF, and WebP for vision. SVGs (XML text) should be handled via the text path instead, not the vision path.
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📚 Learning: 2026-04-01T04:17:41.600Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12632
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:0-0
Timestamp: 2026-04-01T04:17:41.600Z
Learning: When reviewing AutoGPT Copilot tool implementations, accept that `readOnlyHint=True` (provided via `ToolAnnotations`) may be applied unconditionally to *all* tools—even tools that have side effects (e.g., `bash_exec`, `write_workspace_file`, or other write/save operations). Do **not** flag these tools for having `readOnlyHint=True`; this is intentional to enable fully-parallel dispatch by the Anthropic SDK/CLI and has been E2E validated. Only flag `readOnlyHint` issues if they conflict with the established `ToolAnnotations` behavior (e.g., missing/incorrect propagation relative to the intended annotation mechanism).
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📚 Learning: 2026-03-05T15:42:08.207Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12297
File: .claude/skills/backend-check/SKILL.md:14-16
Timestamp: 2026-03-05T15:42:08.207Z
Learning: In Python files under autogpt_platform/backend (recursively), rely on poetry run format to perform formatting (Black + isort) and linting (ruff). Do not run poetry run lint as a separate step after poetry run format, since format already includes linting checks.
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📚 Learning: 2026-03-16T16:35:40.236Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12440
File: autogpt_platform/backend/backend/api/features/workflow_import.py:54-63
Timestamp: 2026-03-16T16:35:40.236Z
Learning: Avoid using the word 'competitor' in public-facing identifiers and text. Use neutral naming for API paths, model names, function names, and UI text. Examples: rename 'CompetitorFormat' to 'SourcePlatform', 'convert_competitor_workflow' to 'convert_workflow', '/competitor-workflow' to '/workflow'. Apply this guideline to files under autogpt_platform/backend and autogpt_platform/frontend.
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📚 Learning: 2026-03-31T15:37:38.626Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12623
File: autogpt_platform/backend/backend/copilot/tools/agent_generator/fixer.py:37-47
Timestamp: 2026-03-31T15:37:38.626Z
Learning: When validating/constructing Anthropic API model IDs in Significant-Gravitas/AutoGPT, allow the hyphen-separated Claude Opus 4.6 model ID `claude-opus-4-6` (it corresponds to `LlmModel.CLAUDE_4_6_OPUS` in `autogpt_platform/backend/backend/blocks/llm.py`). Do NOT require the dot-separated form in Anthropic contexts. Only OpenRouter routing variants should use the dot separator (e.g., `anthropic/claude-opus-4.6`); `claude-opus-4-6` should be treated as correct when passed to Anthropic, and flagged only if it’s used in the OpenRouter path where the dot form is expected.
Applied to files:
autogpt_platform/backend/backend/copilot/config.pyautogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
📚 Learning: 2026-03-17T10:57:12.953Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12440
File: autogpt_platform/backend/backend/copilot/workflow_import/converter.py:0-0
Timestamp: 2026-03-17T10:57:12.953Z
Learning: In Significant-Gravitas/AutoGPT PR `#12440`, `autogpt_platform/backend/backend/copilot/workflow_import/converter.py` was fully rewritten (commit 732960e2d) to no longer make direct LLM/OpenAI API calls. The converter now builds a structured text prompt for AutoPilot/CoPilot instead. There is no `response.choices` access or any direct LLM client usage in this file. Do not flag `response.choices` access or LLM client initialization patterns as issues in this file.
Applied to files:
autogpt_platform/backend/backend/copilot/sdk/service.pyautogpt_platform/backend/backend/copilot/sdk/service_test.py
🔇 Additional comments (3)
autogpt_platform/backend/backend/copilot/config.py (1)
175-182: Config toggle is clear and safely defaulted.The new
claude_agent_exclude_dynamic_sectionsflag is well-scoped, and defaulting toTruewith an explicit fallback path is a good rollout pattern.autogpt_platform/backend/backend/copilot/sdk/service.py (1)
2223-2241: Preset/raw system prompt branching is implemented cleanly.Line 2229-Line 2237 keeps behavior explicitly configurable and preserves the raw-string path when disabled.
autogpt_platform/backend/backend/copilot/sdk/service_test.py (1)
237-239: Assertion formatting update looks good.No behavioral change here; readability is fine.
…ict, rename config field - Extract _build_system_prompt_value() helper so tests exercise production code instead of reconstructing the dict locally. - Add _SystemPromptPreset TypedDict for proper type annotation (replaces str | dict[str, Any]). - Rename claude_agent_exclude_dynamic_sections → claude_agent_cross_user_prompt_cache for clarity.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #12758 +/- ##
==========================================
+ Coverage 63.74% 63.76% +0.02%
==========================================
Files 1815 1815
Lines 132669 132720 +51
Branches 14369 14369
==========================================
+ Hits 84567 84627 +60
+ Misses 45485 45476 -9
Partials 2617 2617
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
test_default_config_is_enabled uses _clean_config_env to ensure env vars don't pollute the ChatConfig constructor test. The new claude_agent_cross_user_prompt_cache field reads from CHAT_CLAUDE_AGENT_CROSS_USER_PROMPT_CACHE, but that var was missing from the list — leaving the test non-deterministic if that env var is set in CI.
majdyz
left a comment
majdyz
left a comment
There was a problem hiding this comment.
Review of latest fixes (54f507b → cd8079d)
Verified the fixes address all previous blockers correctly:
Fixed
_SystemPromptPresetlocal TypedDict added — code now type-checks on older SDK pins (0.1.45) while remaining compatible with 0.1.58+ once #12747 lands- Return type narrowed to
str | _SystemPromptPreset— no moredict[str, Any]bypass - Tests are no longer tautological —
test_preset_dict_structure_when_enabledandtest_raw_string_when_disablednow import and call the production_build_system_prompt_value()helper directly _make_confighelper removed; all tests constructChatConfigdirectly- Config field renamed to
claude_agent_cross_user_prompt_cache(intent-oriented) - Graphiti warm context verified to land in
append(per-user), not the cacheable prefix
New finding fixed in cd8079d
CHAT_CLAUDE_AGENT_CROSS_USER_PROMPT_CACHEwas absent from_CONFIG_ENV_VARS— the_clean_config_envfixture would not have cleared it, makingtest_default_config_is_enablednon-deterministic if that env var is set in CI. Added incd8079d.
Remaining notes (informational, not blockers)
_SystemPromptPreset.appendis typed asstr(required) vsNotRequired[str]in the SDK's definition — acceptable since we always pass it, worth aligning when #12747 lands and the local TypedDict is removedexclude_dynamic_sectionsrequires SDK >=0.1.58 — correctly documented as depending on #12747; no action needed beyond merge ordering
Overall: LGTM once #12747 is merged first.
Resolves E402 import ordering issue flagged by coderabbitai: the TypedDict was defined before module-level third-party imports.
🧪 E2E Test Result — preview/all-active-prsResult: PASS
Screenshots
|
…pat + cost controls (#12747) ## Why We've been pinned at `claude-agent-sdk==0.1.45` (bundled CLI 2.1.63) since PR #12294 because newer versions had two OpenRouter incompatibilities: 1. **`tool_reference` content blocks** (CLI 2.1.69+) — OpenRouter's Zod validation rejects them 2. **`context-management-2025-06-27` beta header** (CLI 2.1.91+) — OpenRouter returns 400 Both are now resolved: - **`tool_reference`: Fixed by CLI's built-in proxy detection.** CLI 2.1.70+ detects `ANTHROPIC_BASE_URL` pointing to a non-Anthropic endpoint and disables `tool_reference` blocks automatically. Verified working in CLI 2.1.97 — the bare CLI test only XFAILs on the beta header, NOT on tool_reference. - **`context-management` beta: Fixed by `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1` env var.** Injected via `build_sdk_env()` for all SDK subprocess calls. Verified in CI. ## What - Upgrades `claude-agent-sdk` from **0.1.45 → 0.1.58** (bundled CLI 2.1.63 → 2.1.97) - Injects `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1` in `build_sdk_env()` (all modes) - Adds `claude_agent_cli_path` config override with executable validation - Adds `claude_agent_max_thinking_tokens=8192` (was unlimited — 54% of $14K/5-day spend was thinking tokens at $75/M) - Lowers `max_budget_usd` from $100 → $15 and `max_turns` from 1000 → 50 ### Features unlocked by the upgrade | Feature | SDK | Impact | |---|---|---| | `exclude_dynamic_sections` | 0.1.57 | Cross-user prompt cache hits (see #12758) | | `AssistantMessage.usage` per-turn | 0.1.49 | Cost attribution per LLM call | | `task_budget` | 0.1.51 | Per-task cost ceiling at SDK level | | `get_context_usage()` | 0.1.52 | Live context-window monitoring | | MCP large-tool-result fix | 0.1.55 | No more silent truncation >50K chars | | MCP HTTP/SSE buffer leak fix | CLI 2.1.97 | Production memory creep ~50 MB/hr | | 429 retry exponential backoff | CLI 2.1.97 | Rate-limit recovery (was burning all retries in ~13s) | | `--resume` cache miss fix | CLI 2.1.90 | Prompt cache works after resume | | SDK session quadratic-write fix | CLI 2.1.90 | No more slowdown on long sessions | | `max_thinking_tokens` | 0.1.57 | Cap extended thinking cost | ## How - `build_sdk_env()` in `env.py` injects the env var unconditionally (all 3 auth modes) - `service.py` passes `max_thinking_tokens` to `ClaudeAgentOptions` - `config.py` adds 3 new fields with env var overrides - Regression tests verify both OpenRouter compat issues are handled ## Test plan - [x] CI green on all test matrices (3.11, 3.12, 3.13) - [x] `test_disable_experimental_betas_env_var_strips_headers` passes — verifies env var strips both patterns - [x] `test_bare_cli_*` correctly XFAILs — documents the CLI regression exists - [x] `test_sdk_exposes_max_thinking_tokens_option` guards the new param - [x] Config validation tests use real temp executables
github-actions
Bot
added
the
conflicts
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
github-actions
Bot
removed
the
conflicts
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
…me turns CLI 2.1.97 (claude-agent-sdk 0.1.58) exits with code 1 when excludeDynamicSections=True is sent in the SDK initialize request AND --resume is active. This caused every second message in a copilot session to fail immediately. Workaround: disable the SystemPromptPreset (fall back to a plain system prompt string) on resumed turns so excludeDynamicSections is never sent when --resume is active. Turn 1 still gets the preset for cross-user prompt caching; turns 2+ use a plain string.
… import SDK is now pinned to >=0.1.58 which exports SystemPromptPreset from claude_agent_sdk.types. Remove the local TypedDict workaround (TODO #12747) and import the type directly from the SDK.
…e changes On context-reduction retry, ctx.use_resume can flip from False to True (compaction creates a CLI session file). Previously the retry block copied system_prompt_value from the initial options unchanged, so a SystemPromptPreset (with exclude_dynamic_sections=True) could be combined with --resume — which crashes CLI 2.1.97 (as documented in the T1 workaround comment). Recompute system_prompt_value for the retry using the updated ctx.use_resume so the preset is always disabled when --resume is active. Fixes Sentry prediction: r3079924820
github-actions
Bot
added
the
conflicts
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
…ion_id + PR #12758 system_prompt recompute)
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
github-actions
Bot
removed
the
conflicts
github-project-automation
Bot
moved this from 🆕 Needs initial review
to ✅ Done
in AutoGPT development kanban
1 participant
Summary
SystemPromptPresetwithexclude_dynamic_sections=Truein the SDK path so the Claude Code default prompt serves as a cacheable prefix shared across all users, reducing input token cost by ~90%claude_agent_cross_user_prompt_cacheconfig field (defaultTrue) to make this configurable, with fallback to raw string when disabled_build_system_prompt_value()helper for testability, with_SystemPromptPresetTypedDict for proper type annotationChanges
config.py: Newclaude_agent_cross_user_prompt_cache: bool = Truefield onChatConfigsdk/service.py:_SystemPromptPresetTypedDict for type safety;_build_system_prompt_value()helper that constructs the preset dict or returns the raw string; call site uses the helpersdk/service_test.py: Tests exercise the production_build_system_prompt_value()helper directly — verifying preset dict structure (enabled), raw string fallback (disabled), and default config valueHow it works
The Claude Code CLI supports
SystemPromptPresetwhich uses the built-in Claude Code default prompt as a static prefix. By settingexclude_dynamic_sections=True, per-user dynamic sections (working dir, git status, auto-memory) are stripped from that prefix so it stays identical across users and benefits from Anthropic's prompt caching. Our custom prompt (tool notes, supplements, graphiti context) is appended after the cacheable prefix.Test plan
_build_system_prompt_value()returns correct preset dict when enabledCHAT_CLAUDE_AGENT_CROSS_USER_PROMPT_CACHE=false