Skip to content

dispatch-e2e validation + agent time discipline rules #10024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
azooz2003-bit wants to merge 7 commits into
base: main
Choose a base branch
from
+473 −5
Open

dispatch-e2e validation + agent time discipline rules #10024

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c1b52cd
Add dispatch-e2e.sh validation wrapper and agent time discipline rules
azooz2003-bit Aug 12, 2026
96181f6
Document advisory-only PR checks and no-red-loop E2E dispatch rule
azooz2003-bit Aug 13, 2026
8778eac
CLAUDE.md: drop stale PR-CI language (red/green commits, notify titles)
azooz2003-bit Aug 13, 2026
03e3a26
docs: address CodeRabbit quick wins (checkout-matches-ref note, per-r…
azooz2003-bit Aug 13, 2026
b7caca1
dispatch-e2e: class-scoped method validation; fail --watch on unresol…
azooz2003-bit Aug 13, 2026
8d93900
dispatch-e2e: only accept parameterless test* methods as filters
azooz2003-bit Aug 13, 2026
b91b7ee
dispatch-e2e: exact run resolution via dispatch_id nonce; validate on…
azooz2003-bit Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/test-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: E2E test with video recording
run-name: ${{ inputs.test_filter }} on ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }} @ ${{ inputs.ref || github.ref_name }}
run-name: ${{ inputs.test_filter }} on ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }} @ ${{ inputs.ref || github.ref_name }}${{ inputs.dispatch_id != '' && format(' [{0}]', inputs.dispatch_id) || '' }}

on:
workflow_dispatch:
Expand All @@ -11,6 +11,10 @@ on:
test_filter:
description: "Test class or class/method; optionally target-qualified as cmuxUITests/Class or cmuxTests/Class"
required: true
dispatch_id:
description: "Optional caller-generated id echoed into the run name so dispatchers can resolve their exact run"
required: false
default: ""
test_timeout:
description: "Per-test timeout in seconds"
required: false
Expand Down
32 changes: 29 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,17 +58,43 @@ If the iPhone is unreachable at build time, the reload still completes: the sign

## Regression test commits

Two commits, so CI proves the test catches the bug: commit 1 adds the failing test only (CI red), commit 2 adds the fix (CI green). This is visible in the PR Commits tab.
Two commits, so the history proves the test catches the bug: commit 1 adds the failing test only, commit 2 adds the fix. When red/green proof is wanted, dispatch the hosted run once against commit 1 and once against the head.

## Hosted E2E dispatch

Dispatch hosted UI/E2E runs through the validated wrapper. It checks every filter item against the local test sources and refuses to dispatch a filter that matches zero tests, which otherwise wastes a full dispatch+watch cycle before the run fails:

```bash
./scripts/dispatch-e2e.sh --ref <branch-or-sha> --filter "<Class or Class/method>[,more]" --watch
```

`--dry-run` validates and prints the `gh` command without dispatching; `--runner`, `--record-video`, `--timeout` (per-test seconds), and `--job-timeout` (minutes) pass through. Validation reads the current checkout, so run it from a checkout of the same ref you dispatch. Raw fallback:

```bash
gh workflow run test-e2e.yml --repo manaflow-ai/cmux -f ref=<branch-or-sha> -f test_filter="<Class or Class/method>"
Comment on lines +65 to +74

@coderabbitai coderabbitai Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

State that the checkout must match the dispatched ref.

The wrapper validates sources from the current checkout, not from --ref. If these differ, validation can reject a selector that exists in the dispatched ref or approve one that does not.

  • CLAUDE.md#L65-L74: State that the command must run from a worktree checked out at <branch-or-sha>.
  • skills/cmux-testing/references/local-vs-ci-validation.md#L20-L20: Add the same checkout/ref precondition beside the wrapper command.
📍 Affects 2 files
  • CLAUDE.md#L65-L74 (this comment)
  • skills/cmux-testing/references/local-vs-ci-validation.md#L20-L20
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CLAUDE.md` around lines 65 - 74, Update the wrapper guidance in CLAUDE.md
lines 65-74 and beside the wrapper command in
skills/cmux-testing/references/local-vs-ci-validation.md line 20 to state that
the command must run from a worktree checked out at the same branch or SHA
passed via --ref; make no other changes.

```

At most one dispatch per dogfood round; each comma-separated filter item becomes its own hosted run, so a multi-item filter spends the round's budget in one command. A red run means root-cause locally or on a fleet simulator first; never loop dispatch-fix-redispatch. A new XCUITest needs one green hosted run before the task is done; nothing else does.

## Agent time discipline

Every poll slice pays a full model reasoning pass, so waiting is where agent hours disappear.

- **Park on one blocking command per wait**: `gh run watch <id> --exit-status`, `./scripts/dispatch-e2e.sh --watch`, or `gh pr checks --watch`. Never 30-60s poll slices, sleep-and-recheck chains, or PTY heartbeat polls.
- **One build dispatch per need.** The cloud reload waits internally for a builder slot; never cycle `RELOAD_CLOUD_BUILDER` or re-dispatch a build that is already queued.
- **Batch fixes per rebuild.** Accumulate a dogfood round's fixes, preflight with focused tests, then do one tagged rebuild for the round. Never rebuild per one-line fix.
- **Locked or offline iPhone: probe once, then queue.** One reachability probe, enqueue in the install queue, notify, and end the turn with "queued; unlock to receive". Never write unlock/ready watcher scripts or repeat devicectl probes.
- **CI is advisory.** Never block a handoff on bot checks, and never re-dispatch a failing hosted run without a local root cause.

## First pass, then dogfood

A first pass ends when the change is implemented, the tagged build succeeded on the pushed HEAD, focused tests ran, and the PR is open (for `web/` PRs, also the live Vercel preview URL). Then hand off to the user. Do not sit in the main conversation watching CI or running speculative review passes after that point.

Do not launch a background review agent (`$autoreview`, `codex review`, `claude review`, or a judge loop) by default. Second-model review is explicit user opt-in in the current conversation; an implementation request, open PR, CI failure, closeout, or handoff is not that opt-in. Let required GitHub checks and the automatic review bots run asynchronously, then return to address only concrete check failures and actionable findings before merge.
Do not launch a background review agent (`$autoreview`, `codex review`, `claude review`, or a judge loop) by default. Second-model review is explicit user opt-in in the current conversation; an implementation request, open PR, CI failure, closeout, or handoff is not that opt-in. PR checks are advisory review bots only; there are no required status checks. Let them run asynchronously and return only for actionable findings. Merge validation happens at merge time via the merge gate, not by watching PR checks.

The main agent owns dogfood, approval, mergeability, and every pushed fix. Merging app/runtime/UI changes requires the user's explicit approval after dogfood; if a fix changes runtime behavior mid-dogfood, rebuild the tag and re-notify, since the earlier verdict covers only the build the user tested.

Notify through `cmux notify` so the user can leave and return. Handoff: `--title "Dogfood ready: <short task>" --subtitle "<branch> · <tag>" --body "Was: <prior bad behavior>. Now: <expected behavior>. <concrete check>. PR: <pr-url>"`. Later closeout notifications use `"CI green: <branch>"` or `"CI blocked: <branch>"` with a one-line cause and the next decision. Titles carry outcome and branch, bodies carry the single next action. Skip notify if there is no cmux socket.
Notify through `cmux notify` so the user can leave and return. Handoff: `--title "Dogfood ready: <short task>" --subtitle "<branch> · <tag>" --body "Was: <prior bad behavior>. Now: <expected behavior>. <concrete check>. PR: <pr-url>"`. Later closeout notifications use `"Merge gate green: <branch>"` or `"Merge gate blocked: <branch>"` with a one-line cause and the next decision. Titles carry outcome and branch, bodies carry the single next action. Skip notify if there is no cmux socket.

## Pitfalls

Expand Down
Loading