fix(go): gate resolver permission-mode to claude, truthful push reporting, test-before-push prompt - #118
Open
AbirAbbas wants to merge 3 commits into
Open
fix(go): gate resolver permission-mode to claude, truthful push reporting, test-before-push prompt#118AbirAbbas wants to merge 3 commits into
AbirAbbas wants to merge 3 commits into
Conversation
AbirAbbas
force-pushed
the
fix/agent-reliability
branch
from
July 24, 2026 19:58
e040e53 to
d88c69a
Compare
…untime run_pr_resolver was handed cfg.PermissionMode verbatim, which defaults to "". Under the claude harness an empty permission mode omits --permission-mode, so `claude --print` falls back to its interactive "prompting" default; with nobody to answer the prompt every write is denied and the resolver finishes having produced no commits. Default to "auto" (→ bypassPermissions) so the resolver can write to the throwaway clone it owns — but only when the resolved provider is claude. The other two harnesses must not get "auto": - codex already yields `--sandbox workspace-write` for an empty mode, so the workspace is writable; "auto" would escalate to --dangerously-bypass-approvals-and-sandbox and drop the sandbox around the whole machine for no benefit. - opencode never reads PermissionMode, so a value there is inert and only implies a guarantee we do not make. An explicitly configured permission mode is still passed through unchanged for every provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sable When the harness returns nothing parseable, run_pr_resolver hands back a deterministic all-false PRResolveResult. The orchestrator took that at face value, so a run where the agent committed and pushed but failed to emit a final structured answer was reported as "nothing happened" — and the CI gate was skipped because pushed was false. Compare the remote head branch before and after the agent runs (via ls-remote, since the agent pushes from its own process and leaves this workspace's tracking refs stale) and record only what the comparison can actually prove: - report_invalid is set whenever the sentinel report is seen, so callers can never mistake a reconstructed result for a first-hand one. - The remote tip moved and equals this workspace's HEAD → pushed=true. fixed stays false: a landed commit is not evidence that CI passes or that review comments were addressed, so overall success (fixed && pushed) correctly stays false. - The remote tip moved to some other commit → remote_advanced is recorded and nothing is attributed. A third party's push is not ours to claim, and their commits are not ours to describe. - The remote did not move → nothing is claimed at all. Commit and file lists are reconstructed only after `git fetch origin <branch>` makes the post-push objects resolvable locally. If that fetch fails the lists stay empty and verification_partial marks them as unknown rather than as "nothing changed". The sentinel string itself is now the exported ci.InvalidResolverReport const, referenced from both the fallback that emits it and the orchestrator that matches on it, so the two copies cannot drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The resolver's instructions told it to re-run the *failing* tests, which says nothing about the tests its own edits break. Add an explicit test-and-lint-before-commit requirement in both halves of the prompt: - the task prompt gets it as a real numbered step (incrementing the step counter, so it does not render as a bare paragraph wedged between two numbered steps); - the system prompt gets it as item 6 of "You are NOT done until". Goldens under testdata/ regenerated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AbirAbbas
force-pushed
the
fix/agent-reliability
branch
from
August 10, 2026 13:27
d88c69a to
434e4a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilt on top of current
mainafter a deep review of the original branch. Three fixes land here; a fourth was dropped (see below).1. Gate the resolver's writable permission mode to the claude runtime
run_pr_resolverwas handedcfg.PermissionModeverbatim, which defaults to"". Under the claude harness an empty permission mode omits--permission-modeentirely, soclaude --printfalls back to its interactive prompting default — with nobody to answer the prompt, every write is denied and the resolver finishes having produced no commits.The default is now
"auto"(→bypassPermissions), but only when the resolved provider is claude. Checked againstsdk/go/harness/:"auto"-pbypassPermissions--sandbox workspace-write— already writable--dangerously-bypass-approvals-and-sandbox"auto"would be harmful: drops the sandbox around the whole machine for no benefitPermissionModeis never readAn explicitly configured permission mode is still passed through unchanged for every provider.
2. Truthful push reporting
When the harness returns nothing parseable,
run_pr_resolverhands back a deterministic all-falsePRResolveResult. The orchestrator took that at face value — so a run where the agent committed and pushed but failed to emit a final structured answer was reported as "nothing happened", and the CI gate was skipped becausepushedwas false.The head branch is now sampled on the remote (
git ls-remote, since the agent pushes from its own process and leaves this workspace's tracking refs stale) before and after the agent runs, and only what that comparison can actually prove is recorded:report_invalid: trueis set whenever the sentinel report is seen, independently of the remote, so callers can never mistake a reconstructed result for a first-hand one.pushed=true.fixedstays false: a landed commit is not evidence that CI passes or that review comments were addressed. Overallsuccessisfixed && pushed, so it correctly stays false.remote_advanced: trueand nothing is attributed. Their push is not ours to claim and their commits are not ours to describe.Commit/file lists are reconstructed only after
git fetch origin <branch>makes the post-push objects resolvable locally. If that fetch fails, the lists stay empty andverification_partial: truemarks them as unknown rather than as "nothing changed".The sentinel string is now the exported
ci.InvalidResolverReportconst, referenced from both the fallback that emits it and the orchestrator that matches on it, so the two copies cannot drift.3. Make the resolver test before it pushes
The resolver's instructions told it to re-run the failing tests, which says nothing about the tests its own edits break. Added an explicit test-and-lint-before-commit requirement to both halves of the prompt — as a properly numbered task step (incrementing the step counter, so it does not render as a bare paragraph wedged between two numbered steps) and as item 6 of the system prompt's "You are NOT done until". Goldens regenerated.
Dropped: the existing-branch-builds fix
All changes to
go/internal/fast/build.go,go/internal/prompts/gitops/git_init.goandgo/internal/roles/gitops/workspace.gofrom the original branch are removed. That fix targetedfast.Build, which is not the deployed path; it also contradicted the unmodified git-init system prompt and would double-open PRs. It will be redone separately againstorch/build.goas a follow-up.Test coverage
New tests in
go/internal/orch/resolve_test.go:TestResolverPermissionModeGatedToClaude— drives realBuildConfigs forclaude_code/codex/open_codeand assertsauto/""/"".TestResolverPermissionModeRespectsExplicitConfig— an explicit mode is never overridden.TestResolveSendsGatedPermissionMode— end-to-end: thepermission_modekwarg the reasoner actually receives is the gated value.TestClassifyRemoteAdvance— table over unchanged / unknown-before / unknown-after / ours / third-party / unknown-local-HEAD.TestResolverReportInvalidMatchesCISentinel— both fields, plus a real report is not flagged.TestResolveInvalidReportWithOurPushReportsPushedNotFixed—pushed=true,fixed=false,report_invalid=true, commit/file lists populated, overallsuccess=false.TestResolveInvalidReportDegradesWhenFetchFails—pushed=true,verification_partial=true, no invented detail.TestResolveInvalidReportDoesNotAttributeThirdPartyPush—pushedstays false,remote_advanced=true, no commits described.TestResolveInvalidReportRemoteUnchangedClaimsNothing— nothing claimed.Gates run locally (CI's
gojob, replicated verbatim on Go 1.23.0,GOWORK=off, with the pinned SDK sparse clone as a sibling)gofmt -l .clean ·go build ./...OK ·go vet ./...OK ·go test -race -count=1 ./...all 27 packages ok.No Python files are touched, so the
testjob is unaffected.🤖 Generated with Claude Code