ci(frontend): add Playwright PR smoke suite with seeded QA accounts - #12682
Merged
Abhi1992002 merged 93 commits intoApr 14, 2026
Merged
Conversation
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughDeterministic E2E overhaul: seeded auth/accounts and storage-state reuse; Playwright CI scripting and caching; Playwright config made environment-driven; many legacy Playwright suites removed and replaced by focused “happy path” specs; Vitest+RTL+MSW tests added; test page-objects and helper utilities refactored for robust synchronization. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor CI
participant GS as GlobalSetup
participant BE as BackendSeeder
participant BR as PlaywrightBrowser
participant FS as Filesystem
CI->>GS: run globalSetup
GS->>BE: ensure deterministic backend data
note right of BE: create users, creator, deterministic submission
GS->>BR: launch browser to seed auth states
BR->>GS: perform seeded logins
BR->>FS: write storageState -> .auth/states/{account}.json
GS->>GS: resolve baseURL / reuse existing states
alt missing or invalid states
GS->>BR: repeat per-account seeding
BR->>FS: persist each storageState
end
CI->>BR: run Playwright specs using persisted storageState
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
documentation
Contributor
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🔴 Merge Conflicts DetectedThe following PRs have been tested and will have merge conflicts if merged after this PR. Consider coordinating with the authors.
🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 2 conflict(s), 0 medium risk, 5 low risk (out of 7 PRs with file overlap) Auto-generated on push. Ignores: |
Abhi1992002
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #12682 +/- ##
==========================================
+ Coverage 63.82% 64.26% +0.43%
==========================================
Files 1815 1815
Lines 132920 132919 -1
Branches 14379 14414 +35
==========================================
+ Hits 84839 85423 +584
+ Misses 45467 44823 -644
- Partials 2614 2673 +59
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
4 tasks
github-actions
Bot
added
the
conflicts
Contributor
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
…12623) ### Why / What / How Users need a way to choose between fast, cheap responses (Sonnet) and deep reasoning (Opus) in the copilot. Previously only the SDK/Opus path existed, and the baseline path was a degraded fallback with no tool calling, no file attachments, no E2B sandbox, and no permission enforcement. This PR adds a copilot mode toggle and brings the baseline (fast) path to full feature parity with the SDK (extended thinking) path. ### Changes 🏗️ #### 1. Mode toggle (UI → full stack) - Add Fast / Thinking mode toggle to ChatInput footer (Phosphor `Brain`/`Zap` icons via lucide-react) - Thread `mode: "fast" | "extended_thinking" | null` from `StreamChatRequest` → RabbitMQ queue → executor → service selection - Fast → baseline service (Sonnet 4 via OpenRouter), Thinking → SDK service (Opus 4.6) - Toggle gated behind `CHAT_MODE_OPTION` feature flag with server-side enforcement - Mode persists in localStorage with SSR-safe init #### 2. Baseline service full tool parity - **Tool call persistence**: Store structured `ChatMessage` entries (assistant + tool results) instead of flat concatenated text — enables frontend to render tool call details and maintain context across turns - **E2B sandbox**: Wire up `get_or_create_sandbox()` so `bash_exec` routes to E2B (image download, Python/PIL compression, filesystem access) - **File attachments**: Accept `file_ids`, download workspace files, embed images as OpenAI vision blocks, save non-images to working dir - **Permissions**: Filter tool list via `CopilotPermissions` (whitelist/blacklist) - **URL context**: Pass `context` dict to user message for URL-shared content - **Execution context**: Pass `sandbox`, `sdk_cwd`, `permissions` to `set_execution_context()` - **Model**: Changed `fast_model` from `google/gemini-2.5-flash` to `anthropic/claude-sonnet-4` for reliable function calling - **Temp dir cleanup**: Lazy `mkdtemp` (only when files attached) + `shutil.rmtree` in finally #### 3. Transcript support for Fast mode - Baseline service now downloads / validates / loads / appends / uploads transcripts (parity with SDK) - Enables seamless mode switching mid-conversation via shared transcript - Upload shielded from cancellation, bounded at 5s timeout #### 4. Feature-flag infrastructure fixes - `FORCE_FLAG_*` env-var overrides on both backend and frontend for local dev / E2E - LaunchDarkly context parity (frontend mirrors backend user context) - `CHAT_MODE_OPTION` default flipped to `false` to match backend #### 5. Other hardening - Double-submit ref guard in `useChatInput` + reconnect dedup in `useCopilotStream` - `copilotModeRef` pattern to read latest mode without recreating transport - Shared `CopilotMode` type across frontend files - File name collision handling with numeric suffix - Path sanitization in file description hints (`os.path.basename`) ### Test plan - [x] 30 new unit tests: `_env_flag_override` (12), `envFlagOverride` (8), `_filter_tools_by_permissions` (4), `_prepare_baseline_attachments` (6) - [x] E2E tested on dev: fast mode creates E2B sandbox, calls 7-10 tools, generates and renders images - [x] Mode switching mid-session works (shared transcript + session messages) - [x] Server-side flag gate enforced (crafted `mode=fast` stripped when flag off) - [x] All 37 CI checks green - [x] Verified via agent-browser: workspace images render correctly in all message positions 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Zamil Majdy <majdy.zamil@gmail.com>
…g-pipeline-for-pr-releases-2
github-actions
Bot
removed
the
conflicts
Contributor
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
…g-pipeline-for-pr-releases-2
…g-pipeline-for-pr-releases-2
majdyz
reviewed
Apr 14, 2026
majdyz
reviewed
Apr 14, 2026
majdyz
previously requested changes
Apr 14, 2026
majdyz
left a comment
majdyz
left a comment
Contributor
There was a problem hiding this comment.
🤖 Review: PR #12682 — ci(frontend): add Playwright PR smoke suite with seeded QA accounts
Good consolidation work. The single-suite approach is cleaner than the old PR-vs-full split, parallel account isolation (one account per spec) is the right pattern, and the Vitest migration for page-level tests follows the testing strategy correctly. 28 passing locally with 0 retries is a good signal.
Issues found:
🟠 Should Fix — SEEDED_TEST_PASSWORD = "testpassword123" is hardcoded in accounts.ts (committed to source). Even though these are test accounts, the password should come from an env var with a local fallback, not be committed in the clear. See inline comment on accounts.ts line 1.
🟠 Should Fix — Graph seeding is not idempotent: create_graph is called unconditionally with reassign_graph_id=True on every seed run, creating a new orphaned graph each time while the submission idempotency check reuses the old submission pointing to the old graph. See inline comment on e2e_test_data.py line 652.
🟡 Nice to Have — Tests are all happy-path smoke tests, which is the stated intent. However there are no tests for the error/sad path on any flow (e.g. bad credentials at login, 404 on builder, marketplace with no results). At least one negative-path test per critical flow would improve the signal quality.
🟡 Nice to Have — The PR-local CI step still hasn't gone green (pnpm test:e2e:no-build CI step — checklist item unchecked). The review should wait for CI green before merge.
🔵 Nit — The workers count of 8 in playwright.config.ts is set via a comment ("Higher worker count") with no explanation of the stability/speed tradeoff. Consider adding a comment or env var to make it tunable without a code change if parallelism issues arise in CI.
github-project-automation
Bot
moved this from 🆕 Needs initial review
to 🚧 Needs work
in AutoGPT development kanban
…g-pipeline-for-pr-releases-2
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
0ubbe
previously approved these changes
Apr 14, 2026
…edback - Check for existing deterministic marketplace graph before creating a new one, preventing orphaned graphs on repeated seed runs - Fix hasNoAgentsMessage to use isVisible() instead of null check on Locator - Move SEEDED_TEST_PASSWORD to env var with local fallback
…-for-pr-releases-2' of https://github.com/Significant-Gravitas/AutoGPT into abhimanyuyadav/secrt-2184-automated-qa-testing-pipeline-for-pr-releases-2
0ubbe
previously approved these changes
Apr 14, 2026
The codecov/project/Platform Backend check was blocking PRs that only touch frontend files due to tiny coverage fluctuations from upstream merges. Mark it informational (same as Platform Frontend) so it reports but does not block.
This reverts commit 514967a.
…g-pipeline-for-pr-releases-2
…g-pipeline-for-pr-releases-2
0ubbe
approved these changes
Apr 14, 2026
github-project-automation
Bot
moved this from 🚧 Needs work
to 👍🏼 Mergeable
in AutoGPT development kanban
Abhi1992002
deleted the
abhimanyuyadav/secrt-2184-automated-qa-testing-pipeline-for-pr-releases-2
branch
April 14, 2026 16:10
github-project-automation
Bot
moved this from 👍🏼 Mergeable
to ✅ Done
in AutoGPT development kanban
12 tasks
15 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why / What / How
This PR simplifies frontend PR validation to one Playwright E2E suite, moves redundant page-level browser coverage into Vitest integration tests, and switches Playwright auth to deterministic seeded QA accounts. It also folds in the follow-up fixes that came out of review and CI: lint cleanup, CodeQL feedback, PR-local type regressions, and the flaky Library run helper.
The approach is:
Changes 🏗️
pnpm test:e2e:no-buildauth-happy-path.spec.tssettings-happy-path.spec.tsapi-keys-happy-path.spec.tsbuilder-happy-path.spec.tslibrary-happy-path.spec.tsmarketplace-happy-path.spec.tspublish-happy-path.spec.tscopilot-happy-path.spec.tsRun nowfrom Library6to8backend/test/e2e_test_data.pyand reused auth states fromfrontend/src/tests/credentials/Setup your task,New task,Rerun task, andRun nowChecklist 📋
For code changes:
pnpm lintpnpm typespnpm test:unitpnpm exec playwright test --listpnpm test:e2e:no-buildlocallyFor configuration changes:
.env.defaultis updated or already compatible with my changesdocker-compose.ymlis updated or already compatible with my changesNotes:
28 passed,0 flaky,0 retries,3m 25s.63.14% -> 63.61%(+0.47%), with frontend coverage up+2.32%and frontend E2E coverage up+2.10%.backend/test/e2e_test_data.py.