Skip to content

fix(copilot): honor reasoning effort and deliver pinned review diffs - #567

Closed
jcarosi wants to merge 6 commits into
gmickel:mainfrom
jcarosi:fix/copilot-high-effort-upstream-20261018
Closed

jcarosi wants to merge 6 commits into
gmickel:mainfrom
jcarosi:fix/copilot-high-effort-upstream-20261018

Conversation

@jcarosi

@jcarosi jcarosi commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

Honor the selected Copilot review effort for Claude and OpenAI models, and give shell-less reviewers the exact pinned patch.

What changes for a user or operator

Claude review selections such as copilot:claude-opus-5.5:high now send --reasoning-effort high. Implementation, plan and completion reviews, including each implementation fan-out draw, receive the full pinned diff at a readable path.

Scope

14 files changed; +555/-97 lines; 3 generated, 2 mechanical.

1. Make High effort and patch coverage real

The adapter previously dropped Claude-family effort. Shell-less review sessions also lacked a readable pinned patch.

2. Forward effort and materialize pinned diffs before dispatch

Send the resolved effort on non-floor Copilot runs. Reuse the existing guarded diff writer before managed or CLI execution, with per-receipt/range/axis identity and failure before dispatch.

1 generated file

3. Verify argv, fallback receipts and diff transport offline

Tests assert the CLI option/value, precedence, floor behavior, byte-equal pinned diffs, fan-out identity and failure before managed/CLI execution. Both affected diff-boundary tests now run POSIX argv and Windows stdin subtests on every host; the implementation dispatch test also initializes its hooks independently.

4. Document the operator contract and retain review provenance

Explain readable patch delivery under existing tool denials. Keep historical attempts at their original ranges and record the fresh completion verdict.

Rest of diff: 2 mechanical files; 2 generated files

Coverage: R1 → groups 1, 2, 3, 4; R2 → groups 1, 2, 3, 4; R3 → groups 1, 2, 3, 4; R4 → groups 1, 2, 3, 4

Blast radius

Copilot review argv, effort attribution, diff transport and their tests/docs. Existing low/medium/high/xhigh values, configuration precedence, session handling and read-only restrictions remain supported. The auto floor omits the effort option; it does not attribute an effort that was not selected.

Verification

  • Focused regression suites: 345 tests passed: backend spec 109, model resolution 64, review convergence 172.
  • Full offline suite: Before the Windows test correction: 4061 tests, 0 failures/errors, 7 skipped across 218 files; hosted CI verifies the final head.
  • Static/generated checks: Ruff 0.16.0, manifest, Codex mirror, documentation anchors, spec validation and git diff --check pass.
  • Implementation review: Historical complete fan-out 5388114c: SHIP on 627efd0..97f0cd4a; the port retains identical runtime source hunks.
  • Fresh completion review: Official Copilot SHIP on f360d3a..ae80c9c; R1-R4 and G1/G2/G4 covered; 0 introduced findings.
  • Effective reviewer provenance: Native session.start: claude-opus-5.5, reasoningEffort high, Copilot CLI 1.0.98; pinned code/test diff read in chunks.
  • Real-world use: Earlier 8.8.0 fix supported Opus 5.5 High reviews during XPosed delivery; its PR feat: add Beads integration to flow skills #2 is merged with green CI.
  • Windows test correction: 958ea8b reproduces and fixes the initial Windows StopIteration errors; 64 focused tests pass with both transport channels.
  • Scoped correction reviews: Official SHIP on 02467aa..958ea8b and 958ea8b..88a06d0; introduced P3 corrected; no remaining introduced findings.
  • Hosted CI correction: The initial run exposed two Windows StopIteration errors; both are fixed and the final full Windows suite passes.
  • fail: Legacy Flow validation: 23 pre-existing fn-16/fn-24 heading errors, identical to upstream; contribution spec validates individually.
  • Final hosted CI: 13/13 checks pass at 88a06d0: https://github.com/gmickel/flow-next/actions/runs/39080275454
  • Hosted full-suite results: 4061 tests per OS, 0 failures/errors; skipped: Linux 8, macOS 9, Windows 200. All three unit jobs pass.

Tradeoffs

Ported onto upstream 9.0.0 (f360d3a) without changing runtime source hunks from historical fan-out 5388114c (627efd0..97f0cd4a). The older diff-less SHIP is retained as history and excluded from accepted evidence. A fresh completion SHIP covers f360d3a..ae80c9c. The later test-only commits have scoped SHIP reviews on 02467aa..958ea8b and 958ea8b..88a06d0, including independent diff creation per transport. There is no UI QA surface.

This is independent of #546. Both touch flowctl.py, test_model_resolution.py and runtime docs, but no changed function overlaps. A combined merge preserves both behaviors and merges code/tests/docs automatically; only CHANGELOG and the generated manifest need mechanical reconciliation if both are integrated. Neither contribution includes the other's fix.

The earlier corrected 8.8.0 implementation was used for Opus 5.5 High reviews during the completed XPosed delivery (https://github.com/jcarosi/xposed/pull/2). Current 9.0.0 compatibility is verified here with deterministic tests and a live official completion review; XPosed was not changed or retested.

Open items

The pre-existing optional Copilot triage judge still has its older effort handling (P3); it is outside the review dispatch fixed here. CLI compatibility was exercised with Copilot 1.0.98. Maintainer-owned website follow-up is separate. Global Flow validation has the same 23 legacy heading errors as upstream; this spec validates cleanly. Gordon retains the upstream merge decision.

Port the preserved b5d103c1/97f0cd4a/d216d39c contribution onto upstream 9.0.0 f360d3a. Python and test hunks match the previously reviewed source exactly; retain historical Flow review attempts at their original ranges and add the Unreleased entry. This contribution does not include or depend on PR gmickel#546. Validation: 4061 tests, 0 failures, 0 errors, 7 skips; 345 focused tests; Ruff, manifest, mirror, anchors and scoped Git checks pass.
Official Copilot completion review returned SHIP on f360d3a..ae80c9c. Native session 5bb1c8da-f36a-45ee-82de-10cb68c32974 records claude-opus-5.5, high, and CLI 1.0.98; the reviewer read the complete code/test diff in chunks and verified R1-R4 plus G1/G2/G4. Historical implementation SHIP remains bound to 627efd0..97f0cd4a; source/test hunks are unchanged by the upstream port. The pre-existing optional triage-judge P3 remains out of scope.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two new tests assume POSIX prompt delivery and raise StopIteration on Windows.

1 open issue
What changed in this PR

Updates Flow-Next’s Copilot review backend to honor selected reasoning effort and provide shell-less reviewers with pinned diffs.

Changes:

  • Forward reasoning effort across model families while preserving fallback-floor behavior.
  • Materialize pinned diffs before primary and fan-out review dispatch.
  • Update regression tests, documentation, and review records.
File Description
plugins/​flow-next/​tests/​test_review_convergence_cap.py Updates effort-attribution assertions.
plugins/​flow-next/​tests/​test_model_resolution.py Tests diff delivery, fan-out identity, and dispatch failures.
plugins/​flow-next/​tests/​test_backend_spec.py Tests reasoning-effort arguments and precedence.
plugins/​flow-next/​skills/​flow-next-impl-review/​workflow-cli.md Documents Copilot diff access.
plugins/​flow-next/​scripts/​flowctl.py Implements effort forwarding and pinned-diff delivery.
plugins/​flow-next/​scripts/​flowctl_tracker/​MANIFEST.json Refreshes the source checksum.
plugins/​flow-next/​docs/​flowctl.md Documents the diff-delivery contract.
plugins/​flow-next/​codex/​skills/​flow-next-impl-review/​workflow-cli.md Synchronizes generated workflow guidance.
plugins/​flow-next/​codex/​docs/​flow-next/​flowctl.md Synchronizes generated runtime documentation.
CHANGELOG.md Records the user-facing correction.
.flow/​tasks/​fn-304-honor-copilot-reasoning-effort-for.1.md Preserves implementation and validation evidence.
.flow/​tasks/​fn-304-honor-copilot-reasoning-effort-for.1.json Records completed task state.
.flow/​specs/​fn-304-honor-copilot-reasoning-effort-for.md Defines effort and diff-transport requirements.
.flow/​specs/​fn-304-honor-copilot-reasoning-effort-for.json Preserves review history and completion status.

Review effort: Balanced

Fix the two hosted Windows StopIteration errors by explicitly exercising POSIX argv and Windows stdin transport. Preserve byte-equal pinned diff, range, receipt identity and tool-denial assertions; initialize hooks so the implementation dispatch test also runs independently. No runtime behavior changes. Validation: all 64 model-resolution tests and Ruff pass; native implementer provenance verified.
Address correctness:1 from official review e85b332f: remove the prior pinned diff before every POSIX and Windows subtest so the exists-before-CLI assertion proves fresh materialization independently. Only two setup lines change; all 64 model-resolution tests pass and native implementer provenance is verified.
Merge upstream without rewriting the published branch. Use backend-neutral names for the existing shared guarded diff writer as requested in gmickel#529, and verify that an optional Copilot pass resumes with the primary pinned artifact intact.
@jcarosi

jcarosi commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

I published a40a62cc to the PR branch after integrating Flow-Next 9.2.0 (f70767b9) with a non-force merge (72866d25).

  • Renamed the shared diff-delivery helpers to backend-neutral names, as requested in Copilot reviewer can't read the diff it's reviewing #529, while preserving the existing on-disk path.
  • Strengthened the existing optional-pass resume test: a resumed Copilot pass confirms the primary pinned diff remains readable and byte-identical. Delivery is unconditional; this does not add the historical review.copilotDiffDelivery=file option or claim that Copilot reviewer can't read the diff it's reviewing #529 is fully resolved. No Cursor behavior change is included.
  • Corrected the smoke-test rationale and the task handoff's recorded completion-review status. The latter reflects the authenticated historical SHIP receipt; it is not a new completion review.

Validation: focused suites passed (64 model-resolution tests and 26 managed-review tests); the full suite passed before the final comment/task-note-only edits (4,073 tests, 0 failures, 0 errors, 7 skipped). Ruff, Codex mirror, schema/manifest checks, anchor check, shell syntax, and git diff --check passed. The existing Windows prompt-transport finding remains resolved; POSIX argv and Windows stdin paths are covered. The separate opt-in triage-judge effort path remains outside this change.

GitHub CI passed on the final PR head: the aggregate, unit tests and functional smokes on all three OSes, Python compatibility, and the Windows interpreter stub. Main CI run · Docs CI run.

Independent implementation review used copilot:claude-opus-5.5:high; native audit evidence confirms the reviewer model and effort. The three-axis review returned SHIP at 72866d25; the scoped re-review returned SHIP at a40a62cc with zero introduced findings after the factual corrections. The separate opt-in triage-judge effort path remains a pre-existing out-of-scope item.

@gmickel

gmickel commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Thanks for this. The diff delivery is in #589, which fixes several backlog issues for 9.5.1. Copilot primary reviews and fan-out draws now get the reviewed diff by path through one shared helper, as Claude reviews do, and Cursor gets it too, since ask mode blocks shell where there's no OS sandbox. You're credited in the changelog and as co-author.

I didn't take the reasoning-effort change. It changes what Copilot sends for Claude-family models, and I want to see that behaviour on its own, with evidence that the flag is accepted, before it ships. If you'd like to pursue it, a small PR with just that change would be easier to judge.

Closing this in favour of #589.

@gmickel gmickel closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants