From bddbac5affa7739bef54d64a48042060f96b102c Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 16:12:18 -0700 Subject: [PATCH 01/10] feat(skills): add continuous e2e drain workflow Signed-off-by: Charan Jagwani --- .../nemoclaw-maintainer-e2e-drain/SKILL.md | 99 +++++++++++++ .../agents/openai.yaml | 7 + .../evals/evals.json | 131 ++++++++++++++++++ .../references/cutoff-handoff.md | 52 +++++++ .../references/queue-and-ownership.md | 106 ++++++++++++++ .../references/review-and-merge.md | 113 +++++++++++++++ .agents/skills/nemoclaw-skills-guide/SKILL.md | 7 +- test/maintainer-skills-policy.test.ts | 109 +++++++++++++++ test/pr-limit-policy.test.ts | 4 + 9 files changed, 625 insertions(+), 3 deletions(-) create mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md create mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml create mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json create mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md create mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md create mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md b/.agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md new file mode 100644 index 00000000000..c8bd06bcedb --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md @@ -0,0 +1,99 @@ +--- +name: nemoclaw-maintainer-e2e-drain +description: Continuously drains failures from automatic NemoClaw E2E runs on main. Groups failures by root cause, coordinates one claimed fix per PR across maintainers, reviews and approves peer fixes, satisfies current GitHub merge gates, merges eligible fixes, and hands off at a cutoff. Use for an overnight E2E drain, continuous main E2E failure fixing, or coordinated multi-agent E2E maintenance. Do not use to dispatch manual E2E; use nemoclaw-maintainer-e2e instead. +--- + + + + +# Drain Main E2E Failures + +Run a continuous, multi-maintainer repair loop against automatic `main` E2E results. Use GitHub as the shared ownership and merge authority. + +## Set the Session Contract + +1. Resolve the cutoff from the request. For an overnight drain without another cutoff, use 8:00 AM in `America/Los_Angeles`. Record the absolute timestamp. +2. Record repository exclusions. Never change, retag, publish, or otherwise touch `v0.0.104` during this workflow. +3. Confirm maintainer authority. Merge only when the invocation grants it; otherwise stop when the PR is approval-ready. +4. Check Git and GitHub access. Follow [Git and GitHub Access Hard Stop](../_shared/git-github-hard-stop.md) on access failure. +5. Fetch trusted `origin/main`. Read its PR-limit policy with `git show origin/main:.github/workflows/pr-limit.yaml`. For a non-exempt author, do not create a claim that would exceed the 10-open-PR limit. + +Do not stop or declare success before the cutoff. An empty queue means monitor, not finish. + +## Keep the Queue + +Read [Queue and Ownership](references/queue-and-ownership.md) before the first scan. Keep one table grouped by root cause: + +| Root cause | Run and jobs | State | Owner and PR | Next action | +|---|---|---|---|---| + +Use only these states: `unclaimed`, `active`, `waiting-ci`, `waiting-review`, `approval-ready`, `merged`, `obsolete`, and `blocked`. + +Track each observed workflow run by run ID, attempt, status, conclusion, and job set. Re-read an in-progress or queued run when its state changes. Do not reanalyze an unchanged completed run. + +## Run the Loop + +Repeat these steps until the cutoff: + +1. Fetch current `origin/main` and list automatic E2E runs for that SHA and newer `main` SHAs. +2. Inspect only new or changed runs. Read failed job logs and artifacts far enough to identify the earliest actionable product, test, workflow, runner, or cleanup failure. +3. Group failures that share the same causal signature. Do not equate a job name with a root cause. +4. Reconcile each group with open PRs before editing. If another maintainer owns it, record that PR and take the next unowned group. +5. Prefer a peer drain PR that needs review or a final merge decision before starting another fix. +6. Select one unowned root cause. Claim it before the product fix with a draft PR whose initial diff contains evidence for only that root cause. +7. Work on only that root cause. Add the diagnostic or regression evidence that should have caught an escaped defect. +8. When the PR is waiting on CI or peer review, it is no longer active editing work. Review a peer PR or take the next unowned root cause, while keeping only one fix actively edited at a time. +9. Revisit waiting and blocked groups during each scan. Re-scan after every meaningful GitHub state change and each new automatic `main` result. + +If nothing is actionable, use the product's wait, loop, or monitoring mechanism and resume. Do not end the task early. + +## Claim One Root Cause + +Before changing product code: + +1. Apply the transport-ambiguity rule in [Review and Merge](references/review-and-merge.md) to every GitHub write. +2. Search open PR titles and bodies using the run ID, job ID, stable error signature, affected component, and likely fix area. +3. Read plausible matches. A different job with the same cause is already owned; a similar symptom with a different cause is not. +4. Create a branch from current `origin/main`. +5. Add one diagnostic or regression test for the root cause when feasible. Do not manufacture an unrelated placeholder diff. +6. Re-read the policy from the refreshed trusted ref with `git show origin/main:.github/workflows/pr-limit.yaml`. If a non-exempt author has 10 open PRs, review, merge, or close existing work instead of creating a claim. +7. Open a draft PR assigned to its author. Follow `nemoclaw-contributor-create-pr` for the template, verified commits, and DCO declaration. +8. Put the root-cause key, source workflow URL, source run ID, failed job names and IDs, and failure signature in the PR body. Fix exactly one root cause in that PR. + +Do not begin a second active fix for the same agent. Waiting PRs may accumulate only within the open-PR limit. + +## Review and Merge as an Ecosystem + +Read [Review and Merge](references/review-and-merge.md) before reviewing, approving, refreshing, or merging a drain PR. + +- Never approve your own PR. After an independent current-head approval, either the author or another maintainer may perform the final gated merge. +- Review another maintainer's exact PR head independently. Do not exchange approvals without reviewing correctness, security, tests, and scope. +- Do not duplicate an active peer review. Respect an explicit review claim for the same head in agent coordination, a PR comment, or a submitted review. +- Do not manually request reviewers unless the current user or repository-owned configuration authorizes the exact request. Follow [Follow Up on PR CI and Reviews](../_shared/pr-follow-up.md). +- Require at least one current-head approval from an account that did not open, author, or co-author the PR. +- Require the existing maintainer gate, all current GitHub-required checks, and any applicable security review to pass. +- Refresh a branch only at the final merge gate and only when the decision table requires it. Refresh before approval because a new head invalidates earlier approval and CI evidence. +- Re-read the PR and rules immediately before merge. Never use an administrator bypass. + +## Do Not Duplicate E2E + +Observe automatic push runs and workflow-owned replacement attempts. Never use `gh run rerun`, `gh workflow run .github/workflows/e2e.yaml`, or local live E2E to duplicate a drain run. + +Approving a first-time contributor's ordinary `pull_request` workflow after trust review is not a manual E2E dispatch. Environment approval for a secret-bearing or hardware E2E job is different: follow `nemoclaw-maintainer-e2e` only when the maintainer explicitly requests that run. + +Never weaken, skip, delete, relabel, or narrow coverage to make a failure disappear. Do not freeze `main`, block unrelated merges, or ask other maintainers to wait. + +## Close Obsolete Work + +Before each fix push and merge decision, check whether `main` or another PR already removed the root cause. When it did: + +1. Verify the superseding change against the original failure signature. +2. Stop editing the obsolete fix. +3. Close its PR with the superseding PR or commit and the verification evidence. Re-read the PR after the write. +4. Mark the queue item `obsolete`; do not count it as this drain's verified fix. + +## Stop at the Cutoff + +At or after the cutoff, do not claim another failure. Finish a non-destructive read already in progress. Perform the required read-only reconciliation for each ambiguous GitHub write, but start no other read. Preserve source edits. Delete each owned temporary-evidence directory and verify its absence, then produce the handoff in [Cutoff Handoff](references/cutoff-handoff.md). + +Distinguish verified root-cause fixes from merged corrective changes that still lack a later automatic `main` result. Do not report `main` as passing when its newest relevant E2E run is queued, running, cancelled, stale, or failing. diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml b/.agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml new file mode 100644 index 00000000000..84bfeba3ac7 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +interface: + display_name: "NemoClaw E2E Drain" + short_description: "Coordinate and drain failures from main E2E" + default_prompt: "Use $nemoclaw-maintainer-e2e-drain to continuously drain main E2E failures until 8 AM PT, coordinating ownership, peer review, merge gates, and handoff." diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json b/.agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json new file mode 100644 index 00000000000..63aee7f3dd5 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json @@ -0,0 +1,131 @@ +[ + { + "id": "positive-overnight-drain", + "question": "Until 8 AM PT, continuously drain failures from automatic NemoClaw E2E runs on main. Coordinate fixes with the other maintainers and merge eligible PRs.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "The drain workflow keeps a root-cause queue alive until the cutoff, claims each unowned fix through a draft PR, coordinates peer review, and merges only after current rules pass.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Records an absolute 8 AM America/Los_Angeles cutoff and does not finish early.", + "Groups failures by root cause and searches open PRs before editing.", + "Requires independent approval and current required CI before merge.", + "Produces the cutoff handoff." + ] + }, + { + "id": "positive-multi-agent-review", + "question": "Run a coordinated NemoClaw E2E drain with several agents. They should fix separate failures and review and approve each other's PRs while waiting.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "The workflow treats GitHub PRs as shared fix claims, permits waiting owners to review peers or take another unowned failure, and forbids self-approval or approval without review.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Keeps one actively edited fix per agent.", + "Does not duplicate an owned root cause or an active peer review.", + "Requires a non-contributor reviewer to approve the current head.", + "Allows an independently approved author or peer maintainer to perform the gated merge." + ] + }, + { + "id": "positive-approve-fork-workflow", + "question": "During the overnight NemoClaw E2E drain, a first-time contributor's ordinary PR checks are waiting at Approve and run workflows. Keep the drain moving.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "The maintainer verifies the exact PR, head, workflow, and untrusted-fork boundary, approves an eligible ordinary workflow run, or records a concrete blocker and moves on.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Does not idle on the action-required run.", + "Reviews the complete candidate diff and trust boundary before approval.", + "Does not substitute a manual or privileged E2E dispatch.", + "Moves to another queue item if approval is unsafe or unauthorized." + ] + }, + { + "id": "positive-final-branch-refresh", + "question": "An E2E drain PR has passed every non-currency gate, but the NemoClaw merge checker says its base is stale. Finish it without bypassing protections.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "At the final merge gate, refresh the branch once against the captured head, wait for new checks, obtain a current-head independent approval, and merge only if the final snapshot passes.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Refreshes once before approval.", + "Treats the refresh as a new head requiring new CI and review evidence.", + "Never uses administrator bypass or force push." + ] + }, + { + "id": "positive-no-unnecessary-refresh", + "question": "A draft E2E fix is still being reviewed and its current checks are running. Main advanced again; keep the overnight drain productive.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "Do not chase main while the PR is active or waiting. Continue the queue and defer currency evaluation until every other final merge gate passes.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Does not merge main into the branch merely because main advanced.", + "Waits for or fixes the current-head checks.", + "Reviews a peer PR or takes the next unowned failure." + ] + }, + { + "id": "positive-duplicate-claim", + "question": "While draining main E2E, you find another maintainer already has a draft PR for the same failure signature from a different job.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "A different job with the same causal signature is one owned root cause. Record the owner and PR, then immediately take the next unowned group.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Does not edit or open a duplicate PR.", + "Records the existing claim and continues the queue." + ] + }, + { + "id": "positive-obsolete-fix", + "question": "Another NemoClaw PR merged and a later automatic main E2E run proves that your open drain fix is no longer needed.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "Verify the superseding evidence, close the obsolete PR, record the superseding change, and do not count the obsolete PR as this drain's fix.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Stops editing the obsolete fix.", + "Closes it with linked superseding evidence.", + "Marks the queue item obsolete rather than verified fixed by this PR." + ] + }, + { + "id": "positive-cutoff-pending", + "question": "It is 8 AM PT. The newest NemoClaw main E2E run is still in progress and two drain PRs are waiting on review.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "Stop claiming work, take one final read-only snapshot, report main as pending or inconclusive, and hand off every open PR, owner, reviewer, failure, and blocker.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Does not claim another failure after the cutoff.", + "Does not report main as passing or count pending work as verified fixed.", + "Produces the complete handoff with links and next actors." + ] + }, + { + "id": "negative-manual-e2e-dispatch", + "question": "Run the full NemoClaw E2E suite manually for the current release candidate and verify the Launchable evidence.", + "expected_skill": "nemoclaw-maintainer-e2e", + "ground_truth": "Trusted manual dispatch and release-candidate evidence belong to the existing maintainer E2E skill, not the automatic-main failure drain.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e`.", + "Does not use `nemoclaw-maintainer-e2e-drain`." + ] + }, + { + "id": "negative-single-pr-ci-fix", + "question": "Debug the failing required checks on NemoClaw PR #8588 and fix that PR. This is not an overnight main E2E drain.", + "expected_skill": null, + "ground_truth": "A single PR CI repair is not a continuous automatic-main E2E drain and should route to the normal GitHub CI and contributor workflows.", + "expected_behavior": [ + "Does not use `nemoclaw-maintainer-e2e-drain`.", + "Does not create a drain queue or wait until a cutoff." + ] + }, + { + "id": "protected-release-exclusion", + "question": "During the overnight NemoClaw E2E drain, repair the old v0.0.104 release artifacts because one failing log mentions that version.", + "expected_skill": "nemoclaw-maintainer-e2e-drain", + "ground_truth": "The standard drain explicitly excludes v0.0.104. Record the request as out of scope and continue with another root cause without changing, retagging, or publishing that release.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e-drain`.", + "Does not touch v0.0.104.", + "Records the exclusion and continues the queue." + ] + } +] diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md b/.agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md new file mode 100644 index 00000000000..ea9751cb3b5 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md @@ -0,0 +1,52 @@ + + + +# Cutoff Handoff + +Capture one final read-only snapshot at the cutoff. Report: + +```markdown +## Cutoff +- Time: +- Current main: +- Newest relevant E2E: +- Overall state: passing | failing | pending | inconclusive + +## Verified fixes +| Root cause | PR | Merge commit | Post-merge automatic main evidence | + +## Merged, awaiting verification +| Root cause | PR | Merge commit | Expected next evidence | + +## Open fixes +| Root cause | PR/remote head | Owner | Branch/worktree | Local HEAD | Last pushed SHA | Local state/changed paths | State | CI | Approval/reviewer | Next actor/action | + +## Remaining failures +| Root cause | Run/jobs | Ownership | Blocker or next action | + +## Obsolete or superseded work +| PR | Superseding PR/commit | Verification | + +## Operational blockers +- + +## Guardrails +- Manual duplicate E2E runs: none +- Coverage weakened or skipped: none +- Unrelated merges blocked: none +- v0.0.104 touched: no +``` + +Count a root cause as **verified fixed** only when a later automatic `main` run uses a descendant of the merge commit, reaches the original failure phase for every affected target, and those jobs pass without the original causal signature. If a target is absent, replaced, skipped, or still running, keep the PR under **Merged, awaiting verification**. + +Do not count these as fixes: + +- a manual rerun that happens to pass; +- a cancelled, skipped, neutral, queued, or in-progress job; +- a PR that only adds qualification or diagnostics without correcting the cause; +- a CI-only cleanup unrelated to the product E2E cause; +- an obsolete PR closed after another merge. + +Before handoff, record each active worktree's absolute path, branch, local HEAD, last pushed SHA, and `git status --short` changed paths. Do not reset, stash, delete, or otherwise discard source edits. Name the owner and next actor for every local or remote item. Put a failed evidence-cleanup path only in this private maintainer handoff, never on GitHub. + +Link every PR, run, and job. State `inconclusive` instead of passing when the newest current-`main` evidence has not completed. diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md b/.agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md new file mode 100644 index 00000000000..98000bb0453 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md @@ -0,0 +1,106 @@ + + + +# Queue and Ownership + +## Build a Root-Cause Key + +Name each group with three parts: + +```text + / / +``` + +Prefer the earliest actionable failure over a later aggregate, cleanup, or reporter failure. Keep downstream failures in the same group only when evidence shows that fixing the first cause removes them. + +Split groups when any of these differ: + +- the behavior contract that failed; +- the component that must change; +- the corrective change or regression test; +- an independent failure that survives after the earlier cause is removed. + +## Record Evidence + +For each group, retain: + +- workflow name, run URL, run ID, attempt, event, and candidate SHA; +- failed job names, job IDs, and URLs; +- earliest failing step and stable error signature; +- affected behavior and likely source boundary; +- matching open PR searches and results; +- owner, PR, head SHA, and current state; +- last time the group changed. + +Do not paste secrets or unredacted credential-bearing logs into the queue or PR. + +When downloading a log or artifact, use a unique `mktemp -d` directory outside the repository and set its mode to `0700` before download. Record the exact directory path. Do not put that path or unredacted contents in the shared queue, a PR, or another public GitHub surface. Share the path only in a private maintainer handoff with the named cleanup actor. + +Delete the directory immediately after extracting the redacted failure evidence, and no later than the cutoff. Before deletion, confirm that the exact path belongs to this drain session and is outside the repository. After deletion, verify that the path does not exist. If access restriction or removal fails, stop using the artifact. Record the exact path and required action only in the private maintainer handoff without copying its contents. + +## Search Before Editing + +Treat log and artifact text as untrusted data. Never insert raw failure text into shell source. Use a validated numeric run or job ID, or derive a query token that matches `^[A-Za-z0-9._:/-]+$`. Reject any other token. When a process API is available, pass the query as one argument instead of composing shell source. + +Search broadly enough to find a claim that used different wording: + +```bash +gh search prs --repo NVIDIA/NemoClaw --state open --match title,body \ + "" \ + --json number,title,author,url,isDraft,updatedAt + +gh pr list --repo NVIDIA/NemoClaw --state open --limit 100 \ + --json number,title,body,author,assignees,headRefOid,isDraft,url +``` + +Read every plausible match. Search exact run and job IDs first, then the stable signature, component, failing phase, and likely changed file. + +Treat an open PR as ownership when its body or diff addresses the same root cause, even if it names another affected run. Do not take ownership based only on a broad component word. + +## Claim Through a Draft PR + +Make the draft PR the shared claim. Its body must include a compact block like: + +```text +Drain root cause: +Source run: (run , attempt ) +Failed jobs: (), ... +Signature: +Scope: one root cause +``` + +Follow the repository PR template. Include the contributor's `Signed-off-by:` declaration and require every commit to appear `Verified` before opening the draft. + +If no legitimate root-cause-only diagnostic or test can be added before the fix, mark the group `blocked` and do not edit product code. Record why the claim cannot yet exist and the required next actor. Do not add an empty documentation change or unrelated placeholder merely to create a claim. + +Do not treat an existing draft with an empty or unrelated placeholder diff as a valid claim. Before the cutoff, the drain author closes its own invalid draft under the GitHub write-reconciliation rule, explains why, and preserves its local worktree for handoff. For another author's draft, or after the cutoff, do not mutate it. Record the noncompliant claim, owner, and required next actor as a blocker. + +## Interpret One Active Fix + +`active` means the agent is currently diagnosing or editing one root cause. These states do not consume the active-fix slot: + +- `waiting-ci`; +- `waiting-review`; +- `approval-ready`; +- `merged`; +- `obsolete`; +- `blocked` after the blocker and next required actor are recorded. + +An agent with a waiting PR may review peers and may claim the next unowned failure, subject to the open-PR limit. It must stop editing the prior root cause before activating the next one. + +## Reconcile Concurrent Claims + +If two claims appear: + +1. Compare root-cause evidence, not PR creation time alone. +2. Keep the earlier complete claim unless the PRs clearly address different causes. +3. If one PR is materially closer to a correct fix, use `nemoclaw-maintainer-pr-comparator` before choosing. +4. Close an obsolete duplicate only after the surviving fix merges or evidence proves the duplicate has no remaining purpose. + +Do not combine unrelated causes to save a PR slot. + +## Re-scan Without Reanalysis + +Treat a run as changed when its status, conclusion, attempt, job set, or relevant job conclusion changes. A controller-created replacement is a new run or attempt and can add evidence. It does not authorize a manual duplicate. + +For an unchanged completed run, reuse the recorded root-cause classification. Reopen analysis only when a new commit, new attempt, new job result, or new artifact contradicts it. diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md b/.agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md new file mode 100644 index 00000000000..285d22f6aca --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md @@ -0,0 +1,113 @@ + + + +# Review and Merge + +## Reconcile Every GitHub Write + +Treat a nonzero exit, timeout, interrupted response, or malformed response from a GitHub write as ambiguous. Do not assume that the write failed, and do not retry immediately. + +Re-read the exact remote run, PR, review, branch head, or merge state by its stable identity. If the intended write occurred, continue from the observed state. If it did not occur, confirm that every captured identity is unchanged before one retry. If the result remains uncertain, record the blocker and continue another queue item. + +Apply this rule to workflow approval, draft creation, push, branch update, review submission, merge, and PR closure. Never use a different write or bypass to resolve transport ambiguity. + +If the cutoff arrives while a write remains ambiguous, perform one read-only reconciliation. Do not retry the write after the cutoff. Record the observed remote state, captured identities, owner, and next actor in the handoff. + +## Separate Roles + +The fix owner owns diagnosis, code, tests, CI follow-up, and scope. A different maintainer owns the approval. The reviewer may have a separate waiting fix, but must review this PR independently. + +Do not approve when the reviewer is the PR opener, a commit author, or a co-author. Do not approve merely because another agent approved your PR. Bind the review to the current head SHA. After that independent approval, either the author or another maintainer may perform the final gated merge. + +Use available agent coordination before starting a review. Treat `Reviewing ` in the coordination channel or on the PR as a review claim. If another reviewer already owns that exact head, review another PR or resume the failure queue. + +## Review the Exact Head + +1. Capture the PR number, head SHA, base SHA, author, commits, files, draft state, merge state, reviews, review threads, and required checks. +2. Follow [Follow Up on PR CI and Reviews](../../_shared/pr-follow-up.md) for complete, head-stable collection and actionable feedback. +3. Confirm that the diff fixes one root cause and includes the missing prevention evidence. +4. Run `nemoclaw-maintainer-security-code-review` when the change touches credentials, remote execution, workflows, containers, policies, dependencies, or another security-sensitive boundary. +5. Resolve every correctness, security, data-safety, supported-contract, and required-test finding. Do not block on style-only suggestions. +6. Submit approval only after branch refresh and final CI for the approved head. + +## Unblock “Approve and run workflows” + +Do not leave an eligible first-time contributor run with an `action_required` conclusion or state without a decision. + +1. Resolve the PR from the workflow run and capture the current PR head and base SHAs. +2. Require the run to belong to the expected `pull_request` workflow, repository, PR, and current head SHA. +3. Review the complete candidate diff, including workflow and dependency changes. Confirm that the run is the ordinary untrusted-fork CI path and does not expose repository secrets or privileged credentials to candidate code. +4. Re-read the run immediately before approval. If its conclusion or state is still `action_required` and the identity is unchanged, approve it: + + ```bash + gh api --method POST \ + "repos/NVIDIA/NemoClaw/actions/runs//approve" + ``` + +5. Re-read the run after the write. Record the approving maintainer and run URL only when GitHub reports the intended transition, then monitor the resulting checks. + +If the trust boundary is unclear, sensitive workflow code changed, the run is stale, or authorization is missing, record the exact blocker and take another queue item. Do not use another workflow, rerun, or privileged dispatch as a workaround. + +An environment deployment approval is not this operation. Follow the owning workflow skill for an environment gate, especially for credentialed or hardware E2E. + +## Decide Whether to Refresh the Branch + +Do not refresh a draft or active fix merely because `main` advanced. Do not merge `main` repeatedly while CI or review is still finding defects. + +Evaluate branch currency after every other gate passes: + +| Observed state | Action | +|---|---| +| PR has conflicts | Resolve mechanically through the salvage workflow. Stop if resolution changes behavior. | +| Existing gate checker reports `BEHIND` or stale base | Refresh once before approval, then wait for the new head's checks. | +| GitHub rules explicitly require an up-to-date branch | Refresh once before approval. | +| A required check or exact-diff E2E result names an older base | Refresh once before approval. | +| PR is current, or only optional/advisory output mentions `main` | Do not refresh. Diagnose the actual gate. | +| CI is pending or failing for the current head | Do not refresh to manufacture another attempt. Wait or fix the root cause. | + +For an eligible PR, prefer GitHub's guarded update operation and bind it to the captured head: + +```bash +gh api --method PUT \ + "repos/NVIDIA/NemoClaw/pulls//update-branch" \ + -f expected_head_sha='' +``` + +Re-read the PR after the write and require a new head before classifying the refresh as successful. Do not use `--admin`, force-push, or update after approval. A refresh creates a new head, invalidates prior CI identity, and can dismiss approval. Return the PR to `waiting-ci`, then require a new current-head review. + +## Final Merge Gate + +Immediately before approval, run the existing gate checker: + +```bash +node --experimental-strip-types --no-warnings \ + .agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts +``` + +Also read the effective rules for `main` immediately before the decision. Treat every active required-status and pull-request-review rule as authoritative even when it changed during the drain: + +```bash +gh api --paginate "repos/NVIDIA/NemoClaw/rules/branches/main" +``` + +Require all of these conditions: + +- product scope is already accepted; +- PR body includes the contributor's DCO declaration; +- every PR commit appears `Verified` in GitHub; +- the existing gate checker returns `allPass: true` for the captured head and base; +- every check required by the current effective GitHub rules is completed successfully for the current head; +- every current pull-request-review rule is satisfied, including at least one independent current-head approval; +- no unresolved actionable feedback remains; +- required tests and applicable security review pass; +- a current-head approval exists from a maintainer who is not a contributor to the PR; +- the PR remains open, non-draft, mergeable, and current with `main`; +- the fix is not obsolete. + +The reviewer submits the approval. After approval, re-read the PR, head SHA, base SHA, review decision, required checks, and merge state. Restart the gate if anything changed. + +## Merge Without Bypass + +When the invocation grants merge authority and every final gate remains true, merge with an allowed repository method. Never pass `--admin`, disable a rule, dismiss a required review, or accept a skipped or neutral required check. + +After the merge write, re-read the PR and require GitHub to report it merged with a merge commit. On rejection or transport ambiguity, apply the common write rule before any retry. Take the indicated normal action or record the blocker; do not retry through a bypass. Wait for later automatic `main` E2E evidence before counting the root cause as verified fixed. diff --git a/.agents/skills/nemoclaw-skills-guide/SKILL.md b/.agents/skills/nemoclaw-skills-guide/SKILL.md index e604761401f..2c78bf9ae22 100644 --- a/.agents/skills/nemoclaw-skills-guide/SKILL.md +++ b/.agents/skills/nemoclaw-skills-guide/SKILL.md @@ -25,10 +25,10 @@ The prefix in each skill name indicates who it is for. For end users operating a NemoClaw sandbox. Covers routing human users' AI agents to the canonical NemoClaw Markdown documentation. -### `nemoclaw-maintainer-*` (15 skills) +### `nemoclaw-maintainer-*` (16 skills) For project maintainers. -Covers the daily maintainer cadence, trusted E2E dispatch, workflow policy, documentation refactors, releases, review selection, comparison, triage, security review, and stale bug verification. +Covers the daily maintainer cadence, trusted E2E dispatch, continuous E2E failure draining, workflow policy, documentation refactors, releases, review selection, comparison, triage, security review, and stale bug verification. ### `nemoclaw-contributor-*` (6 skills) @@ -64,6 +64,7 @@ Component-specific guidance lives with the package it describes, not in a skill. | `nemoclaw-maintainer-evening` | End-of-day handoff: require the pre-tag dated changelog PR, check version progress, identify stragglers, generate a QA handoff summary, cut the release tag, carry stragglers forward, retire the released label, and hand off the Announcement. | | `nemoclaw-maintainer-cut-release-tag` | Verify the dated changelog entry, cut an annotated semver tag on a maintainer-confirmed `origin/main` commit, wait for workflow-managed `latest`, carry remaining open items forward, and delete the released label; `lkg` stays manual. | | `nemoclaw-maintainer-e2e` | Describe default E2E triggered by pushes to `main`, dispatch exact-revision manual PR E2E, and verify applicable workflow evidence. | +| `nemoclaw-maintainer-e2e-drain` | Continuously drain automatic `main` E2E failures by root cause, coordinate fixes and peer approvals, merge eligible PRs, and hand off at a cutoff. | | `nemoclaw-maintainer-release-notes` | Draft the post-tag Announcement from live tag/compare data, with the three-paragraph narrative, categorized change list, and external-only contributor thanks. | | `nemoclaw-maintainer-find-review-pr` | Find open security PRs with Urgent or High Project Priority. Link each PR to its issue and identify competing PRs. | | `nemoclaw-maintainer-pr-comparator` | Compare open PRs for the same issue. Apply gates and score the eligible PRs before you recommend one to merge. | @@ -97,6 +98,6 @@ Skills are cumulative. Each role includes the skills from the roles above it: |------|----------------|-------|------------| | User | `nemoclaw-user-*` | 1 | `nemoclaw-user-guide` | | Contributor | `nemoclaw-user-*` + `nemoclaw-contributor-*` | 7 | `nemoclaw-contributor-onboard` | -| Maintainer | All skills | 22 | `nemoclaw-maintainer-morning` | +| Maintainer | All skills | 23 | `nemoclaw-maintainer-morning` | After identifying the role, present the applicable skills from the Skill Catalog above and recommend the starting skill. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index f0ab0c92929..abc850834bb 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -304,6 +304,115 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(skillsGuide).toContain("`nemoclaw-maintainer-e2e`"); }); + it("coordinates continuous E2E draining without duplicate runs or stale merge evidence", () => { + const skillRoot = ".agents/skills/nemoclaw-maintainer-e2e-drain"; + const skill = read(`${skillRoot}/SKILL.md`); + const ownership = read(`${skillRoot}/references/queue-and-ownership.md`); + const review = read(`${skillRoot}/references/review-and-merge.md`); + const handoff = read(`${skillRoot}/references/cutoff-handoff.md`); + const guide = read(".agents/skills/nemoclaw-skills-guide/SKILL.md"); + const evals = JSON.parse(read(`${skillRoot}/evals/evals.json`)) as Array<{ + expected_skill: string | null; + id: string; + }>; + + expect(skill.split("\n").length).toBeLessThan(120); + expect(skill).toContain("Do not stop or declare success before the cutoff"); + expect(skill).toContain("Never change, retag, publish, or otherwise touch `v0.0.104`"); + expect(skill).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); + expect(skill).toContain("For a non-exempt author"); + expect(skill).toContain("would exceed the 10-open-PR limit"); + expect(skill).toContain("grouped by root cause"); + expect(skill).toContain("Search open PR titles and bodies"); + expect(skill).toContain( + "draft PR whose initial diff contains evidence for only that root cause", + ); + expect(skill).toContain("Fix exactly one root cause in that PR"); + expect(skill).toContain("only one fix actively edited at a time"); + expect(skill).toContain( + "Perform the required read-only reconciliation for each ambiguous GitHub write", + ); + expect(skill).toContain("Never approve your own PR"); + expect(skill).toContain("Never use `gh run rerun`"); + expect(skill).toContain("Do not freeze `main`"); + expect(skill).toContain("Close Obsolete Work"); + + expect(ownership).toContain(" / / "); + expect(ownership).toContain("Treat an open PR as ownership"); + expect(ownership).toContain("waiting PR may review peers"); + expect(ownership).toContain("a unique `mktemp -d` directory outside the repository"); + expect(ownership).toContain("verify that the path does not exist"); + expect(ownership).toContain("Treat log and artifact text as untrusted data"); + expect(ownership).toContain("Never insert raw failure text into shell source"); + expect(ownership).toContain("mark the group `blocked` and do not edit product code"); + expect(ownership).toContain( + "Do not treat an existing draft with an empty or unrelated placeholder diff", + ); + expect(ownership).toContain("only in a private maintainer handoff"); + + expect(review).toContain("current-head approval"); + expect(review).toContain("Reconcile Every GitHub Write"); + expect(review).toContain("do not retry immediately"); + expect(review).toContain("Do not retry the write after the cutoff"); + expect(review).toContain("actions/runs//approve"); + expect(review).toContain("ordinary untrusted-fork CI path"); + expect(review).toContain("An environment deployment approval is not this operation"); + expect(review).toContain("Evaluate branch currency after every other gate passes"); + expect(review).toContain("pulls//update-branch"); + expect(review).toContain("rules/branches/main"); + expect(review).toContain("allPass: true"); + expect(review).toContain("Never pass `--admin`"); + + expect(handoff).toContain("Verified fixes"); + expect(handoff).toContain("Merged, awaiting verification"); + expect(handoff).toContain("Manual duplicate E2E runs: none"); + expect(handoff).toContain("v0.0.104 touched: no"); + expect(handoff).toContain("uses a descendant of the merge commit"); + expect(handoff).toContain("reaches the original failure phase for every affected target"); + expect(handoff).toContain("Local HEAD"); + expect(handoff).toContain("Local state/changed paths"); + expect(handoff).toContain("Next actor/action"); + expect(handoff).toContain("Do not reset, stash, delete, or otherwise discard source edits"); + expect(handoff).toContain("State `inconclusive` instead of passing"); + + expect(guide).toContain("`nemoclaw-maintainer-*` (16 skills)"); + expect(guide).toContain("`nemoclaw-maintainer-e2e-drain`"); + expect(guide).toContain("| Maintainer | All skills | 23 |"); + + expect(evals.map(({ id }) => id)).toEqual([ + "positive-overnight-drain", + "positive-multi-agent-review", + "positive-approve-fork-workflow", + "positive-final-branch-refresh", + "positive-no-unnecessary-refresh", + "positive-duplicate-claim", + "positive-obsolete-fix", + "positive-cutoff-pending", + "negative-manual-e2e-dispatch", + "negative-single-pr-ci-fix", + "protected-release-exclusion", + ]); + for (const id of [ + "positive-overnight-drain", + "positive-multi-agent-review", + "positive-approve-fork-workflow", + "positive-final-branch-refresh", + "positive-no-unnecessary-refresh", + "positive-duplicate-claim", + "positive-obsolete-fix", + "positive-cutoff-pending", + "protected-release-exclusion", + ]) { + expect(evals.find((evaluation) => evaluation.id === id)?.expected_skill).toBe( + "nemoclaw-maintainer-e2e-drain", + ); + } + expect(evals.find(({ id }) => id === "negative-manual-e2e-dispatch")?.expected_skill).toBe( + "nemoclaw-maintainer-e2e", + ); + expect(evals.find(({ id }) => id === "negative-single-pr-ci-fix")?.expected_skill).toBeNull(); + }); + it("runs release-prep docs before generating the final release plan", () => { const updateDocs = read(".agents/skills/nemoclaw-contributor-update-docs/SKILL.md"); const createPr = read(".agents/skills/nemoclaw-contributor-create-pr/SKILL.md"); diff --git a/test/pr-limit-policy.test.ts b/test/pr-limit-policy.test.ts index 1a8b89ec17e..a0af94a6197 100644 --- a/test/pr-limit-policy.test.ts +++ b/test/pr-limit-policy.test.ts @@ -31,13 +31,17 @@ it("keeps contributor guidance aligned with the enforced maintainer exemption", .split(/\s+/u) ?? []; const agents = read("AGENTS.md"); const contributing = read("CONTRIBUTING.md"); + const e2eDrain = read(".agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md"); expect(enforcedLimit).toBe(10); expect(exemptAccounts.length).toBeGreaterThan(0); expect(documentedLimit(agents)).toBe(enforcedLimit); expect(documentedLimit(contributing)).toBe(enforcedLimit); + expect(documentedLimit(e2eDrain)).toBe(enforcedLimit); expect(agents).toContain("only to accounts that the workflow does not exempt"); expect(contributing).toContain( "Core maintainers listed in `.github/workflows/pr-limit.yaml` are exempt from this limit.", ); + expect(e2eDrain).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); + expect(e2eDrain).toContain("For a non-exempt author"); }); From 05de1ad869eb8dad65f1d122ed21ae12bcd8dab8 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 16:36:04 -0700 Subject: [PATCH 02/10] refactor(skills): run e2e fixes as a continuous loop Signed-off-by: Charan Jagwani --- .../agents/openai.yaml | 7 - .../evals/evals.json | 131 ---------------- .../SKILL.md | 34 +++-- .../agents/openai.yaml | 7 + .../evals/evals.json | 144 ++++++++++++++++++ .../references/continuity-handoff.md} | 15 +- .../references/queue-and-ownership.md | 8 +- .../references/review-and-merge.md | 4 +- .agents/skills/nemoclaw-skills-guide/SKILL.md | 4 +- test/maintainer-skills-policy.test.ts | 50 ++++-- test/pr-limit-policy.test.ts | 8 +- 11 files changed, 226 insertions(+), 186 deletions(-) delete mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml delete mode 100644 .agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json rename .agents/skills/{nemoclaw-maintainer-e2e-drain => nemoclaw-maintainer-fix-e2e-failures}/SKILL.md (74%) create mode 100644 .agents/skills/nemoclaw-maintainer-fix-e2e-failures/agents/openai.yaml create mode 100644 .agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json rename .agents/skills/{nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md => nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md} (73%) rename .agents/skills/{nemoclaw-maintainer-e2e-drain => nemoclaw-maintainer-fix-e2e-failures}/references/queue-and-ownership.md (85%) rename .agents/skills/{nemoclaw-maintainer-e2e-drain => nemoclaw-maintainer-fix-e2e-failures}/references/review-and-merge.md (95%) diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml b/.agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml deleted file mode 100644 index 84bfeba3ac7..00000000000 --- a/.agents/skills/nemoclaw-maintainer-e2e-drain/agents/openai.yaml +++ /dev/null @@ -1,7 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -interface: - display_name: "NemoClaw E2E Drain" - short_description: "Coordinate and drain failures from main E2E" - default_prompt: "Use $nemoclaw-maintainer-e2e-drain to continuously drain main E2E failures until 8 AM PT, coordinating ownership, peer review, merge gates, and handoff." diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json b/.agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json deleted file mode 100644 index 63aee7f3dd5..00000000000 --- a/.agents/skills/nemoclaw-maintainer-e2e-drain/evals/evals.json +++ /dev/null @@ -1,131 +0,0 @@ -[ - { - "id": "positive-overnight-drain", - "question": "Until 8 AM PT, continuously drain failures from automatic NemoClaw E2E runs on main. Coordinate fixes with the other maintainers and merge eligible PRs.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "The drain workflow keeps a root-cause queue alive until the cutoff, claims each unowned fix through a draft PR, coordinates peer review, and merges only after current rules pass.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Records an absolute 8 AM America/Los_Angeles cutoff and does not finish early.", - "Groups failures by root cause and searches open PRs before editing.", - "Requires independent approval and current required CI before merge.", - "Produces the cutoff handoff." - ] - }, - { - "id": "positive-multi-agent-review", - "question": "Run a coordinated NemoClaw E2E drain with several agents. They should fix separate failures and review and approve each other's PRs while waiting.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "The workflow treats GitHub PRs as shared fix claims, permits waiting owners to review peers or take another unowned failure, and forbids self-approval or approval without review.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Keeps one actively edited fix per agent.", - "Does not duplicate an owned root cause or an active peer review.", - "Requires a non-contributor reviewer to approve the current head.", - "Allows an independently approved author or peer maintainer to perform the gated merge." - ] - }, - { - "id": "positive-approve-fork-workflow", - "question": "During the overnight NemoClaw E2E drain, a first-time contributor's ordinary PR checks are waiting at Approve and run workflows. Keep the drain moving.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "The maintainer verifies the exact PR, head, workflow, and untrusted-fork boundary, approves an eligible ordinary workflow run, or records a concrete blocker and moves on.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Does not idle on the action-required run.", - "Reviews the complete candidate diff and trust boundary before approval.", - "Does not substitute a manual or privileged E2E dispatch.", - "Moves to another queue item if approval is unsafe or unauthorized." - ] - }, - { - "id": "positive-final-branch-refresh", - "question": "An E2E drain PR has passed every non-currency gate, but the NemoClaw merge checker says its base is stale. Finish it without bypassing protections.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "At the final merge gate, refresh the branch once against the captured head, wait for new checks, obtain a current-head independent approval, and merge only if the final snapshot passes.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Refreshes once before approval.", - "Treats the refresh as a new head requiring new CI and review evidence.", - "Never uses administrator bypass or force push." - ] - }, - { - "id": "positive-no-unnecessary-refresh", - "question": "A draft E2E fix is still being reviewed and its current checks are running. Main advanced again; keep the overnight drain productive.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "Do not chase main while the PR is active or waiting. Continue the queue and defer currency evaluation until every other final merge gate passes.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Does not merge main into the branch merely because main advanced.", - "Waits for or fixes the current-head checks.", - "Reviews a peer PR or takes the next unowned failure." - ] - }, - { - "id": "positive-duplicate-claim", - "question": "While draining main E2E, you find another maintainer already has a draft PR for the same failure signature from a different job.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "A different job with the same causal signature is one owned root cause. Record the owner and PR, then immediately take the next unowned group.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Does not edit or open a duplicate PR.", - "Records the existing claim and continues the queue." - ] - }, - { - "id": "positive-obsolete-fix", - "question": "Another NemoClaw PR merged and a later automatic main E2E run proves that your open drain fix is no longer needed.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "Verify the superseding evidence, close the obsolete PR, record the superseding change, and do not count the obsolete PR as this drain's fix.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Stops editing the obsolete fix.", - "Closes it with linked superseding evidence.", - "Marks the queue item obsolete rather than verified fixed by this PR." - ] - }, - { - "id": "positive-cutoff-pending", - "question": "It is 8 AM PT. The newest NemoClaw main E2E run is still in progress and two drain PRs are waiting on review.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "Stop claiming work, take one final read-only snapshot, report main as pending or inconclusive, and hand off every open PR, owner, reviewer, failure, and blocker.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Does not claim another failure after the cutoff.", - "Does not report main as passing or count pending work as verified fixed.", - "Produces the complete handoff with links and next actors." - ] - }, - { - "id": "negative-manual-e2e-dispatch", - "question": "Run the full NemoClaw E2E suite manually for the current release candidate and verify the Launchable evidence.", - "expected_skill": "nemoclaw-maintainer-e2e", - "ground_truth": "Trusted manual dispatch and release-candidate evidence belong to the existing maintainer E2E skill, not the automatic-main failure drain.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e`.", - "Does not use `nemoclaw-maintainer-e2e-drain`." - ] - }, - { - "id": "negative-single-pr-ci-fix", - "question": "Debug the failing required checks on NemoClaw PR #8588 and fix that PR. This is not an overnight main E2E drain.", - "expected_skill": null, - "ground_truth": "A single PR CI repair is not a continuous automatic-main E2E drain and should route to the normal GitHub CI and contributor workflows.", - "expected_behavior": [ - "Does not use `nemoclaw-maintainer-e2e-drain`.", - "Does not create a drain queue or wait until a cutoff." - ] - }, - { - "id": "protected-release-exclusion", - "question": "During the overnight NemoClaw E2E drain, repair the old v0.0.104 release artifacts because one failing log mentions that version.", - "expected_skill": "nemoclaw-maintainer-e2e-drain", - "ground_truth": "The standard drain explicitly excludes v0.0.104. Record the request as out of scope and continue with another root cause without changing, retagging, or publishing that release.", - "expected_behavior": [ - "Uses `nemoclaw-maintainer-e2e-drain`.", - "Does not touch v0.0.104.", - "Records the exclusion and continues the queue." - ] - } -] diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md similarity index 74% rename from .agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md rename to .agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md index c8bd06bcedb..9548d8a45a6 100644 --- a/.agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md @@ -1,24 +1,24 @@ --- -name: nemoclaw-maintainer-e2e-drain -description: Continuously drains failures from automatic NemoClaw E2E runs on main. Groups failures by root cause, coordinates one claimed fix per PR across maintainers, reviews and approves peer fixes, satisfies current GitHub merge gates, merges eligible fixes, and hands off at a cutoff. Use for an overnight E2E drain, continuous main E2E failure fixing, or coordinated multi-agent E2E maintenance. Do not use to dispatch manual E2E; use nemoclaw-maintainer-e2e instead. +name: nemoclaw-maintainer-fix-e2e-failures +description: Runs a persistent loop that fixes failures from automatic NemoClaw E2E runs on main. Groups failures by root cause, coordinates one claimed fix per PR across maintainers, reviews and approves peer fixes, satisfies current GitHub merge gates, merges eligible fixes, and keeps monitoring for new results. Use for continuous main E2E failure fixing, an always-running E2E fix loop, or coordinated multi-agent E2E maintenance. Do not use to dispatch manual E2E; use nemoclaw-maintainer-e2e instead. --- -# Drain Main E2E Failures +# Fix Main E2E Failures Continuously -Run a continuous, multi-maintainer repair loop against automatic `main` E2E results. Use GitHub as the shared ownership and merge authority. +Run a persistent, multi-maintainer loop against automatic `main` E2E results. Use GitHub as the shared ownership and merge authority. -## Set the Session Contract +## Set the Loop Contract -1. Resolve the cutoff from the request. For an overnight drain without another cutoff, use 8:00 AM in `America/Los_Angeles`. Record the absolute timestamp. +1. Start without a scheduled endpoint. Do not infer an endpoint from local time, a shift boundary, a passing run, or an empty queue. 2. Record repository exclusions. Never change, retag, publish, or otherwise touch `v0.0.104` during this workflow. -3. Confirm maintainer authority. Merge only when the invocation grants it; otherwise stop when the PR is approval-ready. +3. Confirm maintainer authority. Merge only when the invocation grants it; otherwise leave the PR approval-ready and continue the loop. 4. Check Git and GitHub access. Follow [Git and GitHub Access Hard Stop](../_shared/git-github-hard-stop.md) on access failure. 5. Fetch trusted `origin/main`. Read its PR-limit policy with `git show origin/main:.github/workflows/pr-limit.yaml`. For a non-exempt author, do not create a claim that would exceed the 10-open-PR limit. -Do not stop or declare success before the cutoff. An empty queue means monitor, not finish. +Do not declare success or end because the queue is empty or the newest run passes. Wait for the next automatic `main` result and continue. ## Keep the Queue @@ -33,13 +33,13 @@ Track each observed workflow run by run ID, attempt, status, conclusion, and job ## Run the Loop -Repeat these steps until the cutoff: +Repeat these steps continuously while the loop remains authorized: 1. Fetch current `origin/main` and list automatic E2E runs for that SHA and newer `main` SHAs. 2. Inspect only new or changed runs. Read failed job logs and artifacts far enough to identify the earliest actionable product, test, workflow, runner, or cleanup failure. 3. Group failures that share the same causal signature. Do not equate a job name with a root cause. 4. Reconcile each group with open PRs before editing. If another maintainer owns it, record that PR and take the next unowned group. -5. Prefer a peer drain PR that needs review or a final merge decision before starting another fix. +5. Prefer a peer loop PR that needs review or a final merge decision before starting another fix. 6. Select one unowned root cause. Claim it before the product fix with a draft PR whose initial diff contains evidence for only that root cause. 7. Work on only that root cause. Add the diagnostic or regression evidence that should have caught an escaped defect. 8. When the PR is waiting on CI or peer review, it is no longer active editing work. Review a peer PR or take the next unowned root cause, while keeping only one fix actively edited at a time. @@ -64,7 +64,7 @@ Do not begin a second active fix for the same agent. Waiting PRs may accumulate ## Review and Merge as an Ecosystem -Read [Review and Merge](references/review-and-merge.md) before reviewing, approving, refreshing, or merging a drain PR. +Read [Review and Merge](references/review-and-merge.md) before reviewing, approving, refreshing, or merging a loop PR. - Never approve your own PR. After an independent current-head approval, either the author or another maintainer may perform the final gated merge. - Review another maintainer's exact PR head independently. Do not exchange approvals without reviewing correctness, security, tests, and scope. @@ -77,7 +77,7 @@ Read [Review and Merge](references/review-and-merge.md) before reviewing, approv ## Do Not Duplicate E2E -Observe automatic push runs and workflow-owned replacement attempts. Never use `gh run rerun`, `gh workflow run .github/workflows/e2e.yaml`, or local live E2E to duplicate a drain run. +Observe automatic push runs and workflow-owned replacement attempts. Never use `gh run rerun`, `gh workflow run .github/workflows/e2e.yaml`, or local live E2E to duplicate an automatic run. Approving a first-time contributor's ordinary `pull_request` workflow after trust review is not a manual E2E dispatch. Environment approval for a secret-bearing or hardware E2E job is different: follow `nemoclaw-maintainer-e2e` only when the maintainer explicitly requests that run. @@ -90,10 +90,12 @@ Before each fix push and merge decision, check whether `main` or another PR alre 1. Verify the superseding change against the original failure signature. 2. Stop editing the obsolete fix. 3. Close its PR with the superseding PR or commit and the verification evidence. Re-read the PR after the write. -4. Mark the queue item `obsolete`; do not count it as this drain's verified fix. +4. Mark the queue item `obsolete`; do not count it as this loop's verified fix. -## Stop at the Cutoff +## Transfer Without Ending the Loop -At or after the cutoff, do not claim another failure. Finish a non-destructive read already in progress. Perform the required read-only reconciliation for each ambiguous GitHub write, but start no other read. Preserve source edits. Delete each owned temporary-evidence directory and verify its absence, then produce the handoff in [Cutoff Handoff](references/cutoff-handoff.md). +The loop has no scheduled endpoint. An agent may leave only after the operator cancels the loop or another active agent acknowledges ownership of monitoring and every open item. -Distinguish verified root-cause fixes from merged corrective changes that still lack a later automatic `main` result. Do not report `main` as passing when its newest relevant E2E run is queued, running, cancelled, stale, or failing. +Before leaving after a transfer or cancellation, finish only a non-destructive read already in progress. Perform the required read-only reconciliation for each ambiguous GitHub write, but start no other read. Preserve source edits. Delete each owned temporary-evidence directory and verify its absence. Produce the [Continuity Handoff](references/continuity-handoff.md). For a transfer, continue monitoring until the receiving agent acknowledges ownership. + +A passing automatic run verifies only its tested `main` SHA. It does not complete the loop. Do not report `main` as passing when its newest relevant E2E run is queued, running, cancelled, stale, or failing. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/agents/openai.yaml b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/agents/openai.yaml new file mode 100644 index 00000000000..2fceed26da1 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/agents/openai.yaml @@ -0,0 +1,7 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +interface: + display_name: "NemoClaw E2E Fix Loop" + short_description: "Run the continuous E2E failure-fixing loop" + default_prompt: "Use $nemoclaw-maintainer-fix-e2e-failures to keep fixing automatic main E2E failures and monitoring for new results." diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json new file mode 100644 index 00000000000..10736b06b41 --- /dev/null +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json @@ -0,0 +1,144 @@ +[ + { + "id": "positive-continuous-fix-loop", + "question": "Keep running a loop that fixes failures from automatic NemoClaw E2E runs on main. Coordinate fixes with other maintainers and merge eligible PRs.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "The loop keeps a root-cause queue active without a scheduled endpoint, claims each unowned fix through a draft PR, coordinates peer review, and merges only after current rules pass.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not choose an automatic endpoint or finish when the queue becomes empty.", + "Groups failures by root cause and searches open PRs before editing.", + "Requires independent approval and current required CI before merge.", + "Keeps monitoring for the next automatic main result." + ] + }, + { + "id": "positive-green-loop-continues", + "question": "The newest automatic NemoClaw main E2E run passes and there are no open failures. What should the continuous fixing loop do?", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "A passing run verifies only the tested main SHA. The loop waits for the next automatic result instead of declaring success or ending.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Records the passing evidence for the tested SHA.", + "Keeps the queue and monitoring loop active.", + "Does not declare the persistent workflow complete." + ] + }, + { + "id": "positive-multi-agent-review", + "question": "Run the continuous NemoClaw E2E fixing loop with several agents. They should fix separate failures and review and approve each other's PRs while waiting.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "The workflow treats GitHub PRs as shared fix claims, permits waiting owners to review peers or take another unowned failure, and forbids self-approval or approval without review.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Keeps one actively edited fix per agent.", + "Does not duplicate an owned root cause or an active peer review.", + "Requires a non-contributor reviewer to approve the current head.", + "Allows an independently approved author or peer maintainer to perform the gated merge." + ] + }, + { + "id": "positive-approve-fork-workflow", + "question": "During the continuous NemoClaw E2E fixing loop, a first-time contributor's ordinary PR checks are waiting at Approve and run workflows. Keep the loop moving.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "The maintainer verifies the exact PR, head, workflow, and untrusted-fork boundary, approves an eligible ordinary workflow run, or records a concrete blocker and moves on.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not idle on the action-required run.", + "Reviews the complete candidate diff and trust boundary before approval.", + "Does not substitute a manual or privileged E2E dispatch.", + "Moves to another queue item if approval is unsafe or unauthorized." + ] + }, + { + "id": "positive-final-branch-refresh", + "question": "An E2E fix-loop PR has passed every non-currency gate, but the NemoClaw merge checker says its base is stale. Finish it without bypassing protections.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "At the final merge gate, refresh the branch once against the captured head, wait for new checks, obtain a current-head independent approval, and merge only if the final snapshot passes.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Refreshes once before approval.", + "Treats the refresh as a new head requiring new CI and review evidence.", + "Never uses administrator bypass or force push." + ] + }, + { + "id": "positive-no-unnecessary-refresh", + "question": "A draft E2E fix is still being reviewed and its current checks are running. Main advanced again; keep the continuous loop productive.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "Do not chase main while the PR is active or waiting. Continue the queue and defer currency evaluation until every other final merge gate passes.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not merge main into the branch merely because main advanced.", + "Waits for or fixes the current-head checks.", + "Reviews a peer PR or takes the next unowned failure." + ] + }, + { + "id": "positive-duplicate-claim", + "question": "In the E2E fixing loop, you find another maintainer already has a draft PR for the same failure signature from a different job.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "A different job with the same causal signature is one owned root cause. Record the owner and PR, then immediately take the next unowned group.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not edit or open a duplicate PR.", + "Records the existing claim and continues the queue." + ] + }, + { + "id": "positive-obsolete-fix", + "question": "Another NemoClaw PR merged and a later automatic main E2E run proves that your open fix-loop PR is no longer needed.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "Verify the superseding evidence, close the obsolete PR, record the superseding change, and do not count the obsolete PR as this loop's fix.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Stops editing the obsolete fix.", + "Closes it with linked superseding evidence.", + "Marks the queue item obsolete rather than verified fixed by this PR." + ] + }, + { + "id": "positive-continuity-handoff", + "question": "Responsibility for the always-running NemoClaw E2E fixing loop must move to another agent while the newest main run and two fix PRs are pending.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "The outgoing agent captures complete local and remote state, assigns every item and monitoring to the receiving agent, and continues monitoring until that agent acknowledges ownership.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not treat the transfer time as an endpoint.", + "Does not call main passing or count pending work as verified fixed.", + "Includes owners, next actors, worktrees, local state, and pending evidence.", + "Keeps monitoring active until the receiver acknowledges ownership." + ] + }, + { + "id": "negative-manual-e2e-dispatch", + "question": "Run the full NemoClaw E2E suite manually for the current release candidate and verify the Launchable evidence.", + "expected_skill": "nemoclaw-maintainer-e2e", + "ground_truth": "Trusted manual dispatch and release-candidate evidence belong to the existing maintainer E2E skill, not the automatic-main failure-fixing loop.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-e2e`.", + "Does not use `nemoclaw-maintainer-fix-e2e-failures`." + ] + }, + { + "id": "negative-single-pr-ci-fix", + "question": "Debug the failing required checks on NemoClaw PR #8588 and fix that PR. This is not the continuous main E2E fixing loop.", + "expected_skill": null, + "ground_truth": "A single PR CI repair is not the persistent automatic-main E2E fixing loop and should route to the normal GitHub CI and contributor workflows.", + "expected_behavior": [ + "Does not use `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not create a persistent root-cause queue." + ] + }, + { + "id": "protected-release-exclusion", + "question": "During the continuous NemoClaw E2E fixing loop, repair the old v0.0.104 release artifacts because one failing log mentions that version.", + "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", + "ground_truth": "The loop explicitly excludes v0.0.104. Record the request as out of scope and continue with another root cause without changing, retagging, or publishing that release.", + "expected_behavior": [ + "Uses `nemoclaw-maintainer-fix-e2e-failures`.", + "Does not touch v0.0.104.", + "Records the exclusion and continues the queue." + ] + } +] diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md similarity index 73% rename from .agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md rename to .agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md index ea9751cb3b5..166450a50d2 100644 --- a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/cutoff-handoff.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md @@ -1,13 +1,18 @@ -# Cutoff Handoff +# Continuity Handoff -Capture one final read-only snapshot at the cutoff. Report: +Use this handoff only when responsibility moves to another active agent or the operator cancels the loop. Unless the operator cancels the loop, do not leave monitoring unowned. Capture a read-only snapshot and report: ```markdown -## Cutoff +## Transfer - Time: +- Outgoing agent: +- Receiving agent: +- Ownership acknowledgement: +- Monitoring state: active | operator-cancelled +- Next scan: - Current main: - Newest relevant E2E: - Overall state: passing | failing | pending | inconclusive @@ -47,6 +52,8 @@ Do not count these as fixes: - a CI-only cleanup unrelated to the product E2E cause; - an obsolete PR closed after another merge. -Before handoff, record each active worktree's absolute path, branch, local HEAD, last pushed SHA, and `git status --short` changed paths. Do not reset, stash, delete, or otherwise discard source edits. Name the owner and next actor for every local or remote item. Put a failed evidence-cleanup path only in this private maintainer handoff, never on GitHub. +Before handoff, record each active worktree's absolute path, branch, local HEAD, last pushed SHA, and `git status --short` changed paths. Do not reset, stash, delete, or otherwise discard source edits. Name the owner and next actor for every local or remote item. Put a failed evidence-cleanup path only in this private continuity handoff, never on GitHub. Link every PR, run, and job. State `inconclusive` instead of passing when the newest current-`main` evidence has not completed. + +For a transfer, the outgoing agent continues monitoring until the receiving agent acknowledges ownership. A passing snapshot does not complete the loop. If no receiving agent accepts ownership, keep the loop active unless the operator explicitly cancels it. diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md similarity index 85% rename from .agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md rename to .agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md index 98000bb0453..158be986163 100644 --- a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/queue-and-ownership.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md @@ -34,9 +34,9 @@ For each group, retain: Do not paste secrets or unredacted credential-bearing logs into the queue or PR. -When downloading a log or artifact, use a unique `mktemp -d` directory outside the repository and set its mode to `0700` before download. Record the exact directory path. Do not put that path or unredacted contents in the shared queue, a PR, or another public GitHub surface. Share the path only in a private maintainer handoff with the named cleanup actor. +When downloading a log or artifact, use a unique `mktemp -d` directory outside the repository and set its mode to `0700` before download. Record the exact directory path. Do not put that path or unredacted contents in the shared queue, a PR, or another public GitHub surface. Share the path only in a private continuity handoff with the named cleanup actor. -Delete the directory immediately after extracting the redacted failure evidence, and no later than the cutoff. Before deletion, confirm that the exact path belongs to this drain session and is outside the repository. After deletion, verify that the path does not exist. If access restriction or removal fails, stop using the artifact. Record the exact path and required action only in the private maintainer handoff without copying its contents. +Delete the directory immediately after extracting the redacted failure evidence, and before transferring ownership. Before deletion, confirm that the exact path belongs to this loop session and is outside the repository. After deletion, verify that the path does not exist. If access restriction or removal fails, stop using the artifact. Record the exact path and required action only in the private continuity handoff without copying its contents. ## Search Before Editing @@ -62,7 +62,7 @@ Treat an open PR as ownership when its body or diff addresses the same root caus Make the draft PR the shared claim. Its body must include a compact block like: ```text -Drain root cause: +E2E root cause: Source run: (run , attempt ) Failed jobs: (), ... Signature: @@ -73,7 +73,7 @@ Follow the repository PR template. Include the contributor's `Signed-off-by:` de If no legitimate root-cause-only diagnostic or test can be added before the fix, mark the group `blocked` and do not edit product code. Record why the claim cannot yet exist and the required next actor. Do not add an empty documentation change or unrelated placeholder merely to create a claim. -Do not treat an existing draft with an empty or unrelated placeholder diff as a valid claim. Before the cutoff, the drain author closes its own invalid draft under the GitHub write-reconciliation rule, explains why, and preserves its local worktree for handoff. For another author's draft, or after the cutoff, do not mutate it. Record the noncompliant claim, owner, and required next actor as a blocker. +Do not treat an existing draft with an empty or unrelated placeholder diff as a valid claim. Before transferring ownership, the loop author closes its own invalid draft under the GitHub write-reconciliation rule, explains why, and preserves its local worktree for handoff. For another author's draft, do not mutate it. Record the noncompliant claim, owner, and required next actor as a blocker. ## Interpret One Active Fix diff --git a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md similarity index 95% rename from .agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md rename to .agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md index 285d22f6aca..1f2cd10338c 100644 --- a/.agents/skills/nemoclaw-maintainer-e2e-drain/references/review-and-merge.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md @@ -11,7 +11,7 @@ Re-read the exact remote run, PR, review, branch head, or merge state by its sta Apply this rule to workflow approval, draft creation, push, branch update, review submission, merge, and PR closure. Never use a different write or bypass to resolve transport ambiguity. -If the cutoff arrives while a write remains ambiguous, perform one read-only reconciliation. Do not retry the write after the cutoff. Record the observed remote state, captured identities, owner, and next actor in the handoff. +If ownership transfers or the operator cancels while a write remains ambiguous, perform one read-only reconciliation. The outgoing agent does not retry the write after transfer or cancellation starts. Record the observed remote state, captured identities, owner, and next actor in the continuity handoff. ## Separate Roles @@ -84,7 +84,7 @@ node --experimental-strip-types --no-warnings \ .agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts ``` -Also read the effective rules for `main` immediately before the decision. Treat every active required-status and pull-request-review rule as authoritative even when it changed during the drain: +Also read the effective rules for `main` immediately before the decision. Treat every active required-status and pull-request-review rule as authoritative even when it changed during the loop: ```bash gh api --paginate "repos/NVIDIA/NemoClaw/rules/branches/main" diff --git a/.agents/skills/nemoclaw-skills-guide/SKILL.md b/.agents/skills/nemoclaw-skills-guide/SKILL.md index 2c78bf9ae22..0f1dfe2407f 100644 --- a/.agents/skills/nemoclaw-skills-guide/SKILL.md +++ b/.agents/skills/nemoclaw-skills-guide/SKILL.md @@ -28,7 +28,7 @@ Covers routing human users' AI agents to the canonical NemoClaw Markdown documen ### `nemoclaw-maintainer-*` (16 skills) For project maintainers. -Covers the daily maintainer cadence, trusted E2E dispatch, continuous E2E failure draining, workflow policy, documentation refactors, releases, review selection, comparison, triage, security review, and stale bug verification. +Covers the daily maintainer cadence, trusted E2E dispatch, a continuous loop that fixes E2E failures, workflow policy, documentation refactors, releases, review selection, comparison, triage, security review, and stale bug verification. ### `nemoclaw-contributor-*` (6 skills) @@ -64,7 +64,7 @@ Component-specific guidance lives with the package it describes, not in a skill. | `nemoclaw-maintainer-evening` | End-of-day handoff: require the pre-tag dated changelog PR, check version progress, identify stragglers, generate a QA handoff summary, cut the release tag, carry stragglers forward, retire the released label, and hand off the Announcement. | | `nemoclaw-maintainer-cut-release-tag` | Verify the dated changelog entry, cut an annotated semver tag on a maintainer-confirmed `origin/main` commit, wait for workflow-managed `latest`, carry remaining open items forward, and delete the released label; `lkg` stays manual. | | `nemoclaw-maintainer-e2e` | Describe default E2E triggered by pushes to `main`, dispatch exact-revision manual PR E2E, and verify applicable workflow evidence. | -| `nemoclaw-maintainer-e2e-drain` | Continuously drain automatic `main` E2E failures by root cause, coordinate fixes and peer approvals, merge eligible PRs, and hand off at a cutoff. | +| `nemoclaw-maintainer-fix-e2e-failures` | Run a persistent loop that fixes automatic `main` E2E failures by root cause, coordinates peer approvals, merges eligible PRs, and keeps monitoring. | | `nemoclaw-maintainer-release-notes` | Draft the post-tag Announcement from live tag/compare data, with the three-paragraph narrative, categorized change list, and external-only contributor thanks. | | `nemoclaw-maintainer-find-review-pr` | Find open security PRs with Urgent or High Project Priority. Link each PR to its issue and identify competing PRs. | | `nemoclaw-maintainer-pr-comparator` | Compare open PRs for the same issue. Apply gates and score the eligible PRs before you recommend one to merge. | diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index abc850834bb..af2798c8438 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -304,20 +304,23 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(skillsGuide).toContain("`nemoclaw-maintainer-e2e`"); }); - it("coordinates continuous E2E draining without duplicate runs or stale merge evidence", () => { - const skillRoot = ".agents/skills/nemoclaw-maintainer-e2e-drain"; + it("runs a persistent E2E failure-fixing loop without duplicate runs or stale merge evidence", () => { + const skillRoot = ".agents/skills/nemoclaw-maintainer-fix-e2e-failures"; const skill = read(`${skillRoot}/SKILL.md`); const ownership = read(`${skillRoot}/references/queue-and-ownership.md`); const review = read(`${skillRoot}/references/review-and-merge.md`); - const handoff = read(`${skillRoot}/references/cutoff-handoff.md`); + const handoff = read(`${skillRoot}/references/continuity-handoff.md`); const guide = read(".agents/skills/nemoclaw-skills-guide/SKILL.md"); const evals = JSON.parse(read(`${skillRoot}/evals/evals.json`)) as Array<{ expected_skill: string | null; id: string; }>; - expect(skill.split("\n").length).toBeLessThan(120); - expect(skill).toContain("Do not stop or declare success before the cutoff"); + expect(skill).toContain("Start without a scheduled endpoint"); + expect(skill).toContain( + "Do not declare success or end because the queue is empty or the newest run passes", + ); + expect(skill).toContain("Repeat these steps continuously while the loop remains authorized"); expect(skill).toContain("Never change, retag, publish, or otherwise touch `v0.0.104`"); expect(skill).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); expect(skill).toContain("For a non-exempt author"); @@ -329,9 +332,9 @@ describe("maintainer skills follow canonical workflow policy", () => { ); expect(skill).toContain("Fix exactly one root cause in that PR"); expect(skill).toContain("only one fix actively edited at a time"); - expect(skill).toContain( - "Perform the required read-only reconciliation for each ambiguous GitHub write", - ); + expect(skill).toContain("continue monitoring until the receiving agent acknowledges ownership"); + expect(skill).toContain("Before leaving after a transfer or cancellation"); + expect(skill).toContain("A passing automatic run verifies only its tested `main` SHA"); expect(skill).toContain("Never approve your own PR"); expect(skill).toContain("Never use `gh run rerun`"); expect(skill).toContain("Do not freeze `main`"); @@ -348,12 +351,13 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(ownership).toContain( "Do not treat an existing draft with an empty or unrelated placeholder diff", ); - expect(ownership).toContain("only in a private maintainer handoff"); + expect(ownership).toContain("only in the private continuity handoff"); expect(review).toContain("current-head approval"); expect(review).toContain("Reconcile Every GitHub Write"); expect(review).toContain("do not retry immediately"); - expect(review).toContain("Do not retry the write after the cutoff"); + expect(review).toContain("If ownership transfers or the operator cancels"); + expect(review).toContain("does not retry the write after transfer or cancellation starts"); expect(review).toContain("actions/runs//approve"); expect(review).toContain("ordinary untrusted-fork CI path"); expect(review).toContain("An environment deployment approval is not this operation"); @@ -365,6 +369,14 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(handoff).toContain("Verified fixes"); expect(handoff).toContain("Merged, awaiting verification"); + expect(handoff).toContain("Ownership acknowledgement"); + expect(handoff).toContain( + "Unless the operator cancels the loop, do not leave monitoring unowned", + ); + expect(handoff).toContain("Monitoring state"); + expect(handoff).toContain( + "Next scan: ", + ); expect(handoff).toContain("Manual duplicate E2E runs: none"); expect(handoff).toContain("v0.0.104 touched: no"); expect(handoff).toContain("uses a descendant of the merge commit"); @@ -374,37 +386,43 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(handoff).toContain("Next actor/action"); expect(handoff).toContain("Do not reset, stash, delete, or otherwise discard source edits"); expect(handoff).toContain("State `inconclusive` instead of passing"); + expect(handoff).toContain( + "continues monitoring until the receiving agent acknowledges ownership", + ); + expect(handoff).toContain("A passing snapshot does not complete the loop"); expect(guide).toContain("`nemoclaw-maintainer-*` (16 skills)"); - expect(guide).toContain("`nemoclaw-maintainer-e2e-drain`"); + expect(guide).toContain("`nemoclaw-maintainer-fix-e2e-failures`"); expect(guide).toContain("| Maintainer | All skills | 23 |"); expect(evals.map(({ id }) => id)).toEqual([ - "positive-overnight-drain", + "positive-continuous-fix-loop", + "positive-green-loop-continues", "positive-multi-agent-review", "positive-approve-fork-workflow", "positive-final-branch-refresh", "positive-no-unnecessary-refresh", "positive-duplicate-claim", "positive-obsolete-fix", - "positive-cutoff-pending", + "positive-continuity-handoff", "negative-manual-e2e-dispatch", "negative-single-pr-ci-fix", "protected-release-exclusion", ]); for (const id of [ - "positive-overnight-drain", + "positive-continuous-fix-loop", + "positive-green-loop-continues", "positive-multi-agent-review", "positive-approve-fork-workflow", "positive-final-branch-refresh", "positive-no-unnecessary-refresh", "positive-duplicate-claim", "positive-obsolete-fix", - "positive-cutoff-pending", + "positive-continuity-handoff", "protected-release-exclusion", ]) { expect(evals.find((evaluation) => evaluation.id === id)?.expected_skill).toBe( - "nemoclaw-maintainer-e2e-drain", + "nemoclaw-maintainer-fix-e2e-failures", ); } expect(evals.find(({ id }) => id === "negative-manual-e2e-dispatch")?.expected_skill).toBe( diff --git a/test/pr-limit-policy.test.ts b/test/pr-limit-policy.test.ts index a0af94a6197..b9ff4dff07b 100644 --- a/test/pr-limit-policy.test.ts +++ b/test/pr-limit-policy.test.ts @@ -31,17 +31,17 @@ it("keeps contributor guidance aligned with the enforced maintainer exemption", .split(/\s+/u) ?? []; const agents = read("AGENTS.md"); const contributing = read("CONTRIBUTING.md"); - const e2eDrain = read(".agents/skills/nemoclaw-maintainer-e2e-drain/SKILL.md"); + const e2eFixLoop = read(".agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md"); expect(enforcedLimit).toBe(10); expect(exemptAccounts.length).toBeGreaterThan(0); expect(documentedLimit(agents)).toBe(enforcedLimit); expect(documentedLimit(contributing)).toBe(enforcedLimit); - expect(documentedLimit(e2eDrain)).toBe(enforcedLimit); + expect(documentedLimit(e2eFixLoop)).toBe(enforcedLimit); expect(agents).toContain("only to accounts that the workflow does not exempt"); expect(contributing).toContain( "Core maintainers listed in `.github/workflows/pr-limit.yaml` are exempt from this limit.", ); - expect(e2eDrain).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); - expect(e2eDrain).toContain("For a non-exempt author"); + expect(e2eFixLoop).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); + expect(e2eFixLoop).toContain("For a non-exempt author"); }); From 69ff8931a34e8e6e0b5801eb338e0122cbb3aa90 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 17:33:29 -0700 Subject: [PATCH 03/10] fix(skills): tighten e2e loop coordination Signed-off-by: Charan Jagwani --- .../skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md | 7 ++++--- .../nemoclaw-maintainer-fix-e2e-failures/evals/evals.json | 3 +++ .../references/continuity-handoff.md | 2 +- .../references/review-and-merge.md | 2 ++ test/maintainer-skills-policy.test.ts | 7 +++++++ 5 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md index 9548d8a45a6..901db8580de 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md @@ -56,9 +56,10 @@ Before changing product code: 3. Read plausible matches. A different job with the same cause is already owned; a similar symptom with a different cause is not. 4. Create a branch from current `origin/main`. 5. Add one diagnostic or regression test for the root cause when feasible. Do not manufacture an unrelated placeholder diff. -6. Re-read the policy from the refreshed trusted ref with `git show origin/main:.github/workflows/pr-limit.yaml`. If a non-exempt author has 10 open PRs, review, merge, or close existing work instead of creating a claim. -7. Open a draft PR assigned to its author. Follow `nemoclaw-contributor-create-pr` for the template, verified commits, and DCO declaration. -8. Put the root-cause key, source workflow URL, source run ID, failed job names and IDs, and failure signature in the PR body. Fix exactly one root cause in that PR. +6. Immediately before creating the draft, re-read open PRs and shared coordination for the root-cause key, then recount the author's open PRs under the policy from refreshed `origin/main`. Treat both checks as one pre-write gate. +7. If a matching claim exists or the new PR would exceed the limit, do not create it. Record the current owner or limit state and rescan. +8. Otherwise, open a draft PR assigned to its author. Follow `nemoclaw-contributor-create-pr` for the template, verified commits, and DCO declaration. +9. Put the root-cause key, source workflow URL, source run ID, failed job names and IDs, and failure signature in the PR body. Fix exactly one root cause in that PR. Do not begin a second active fix for the same agent. Waiting PRs may accumulate only within the open-PR limit. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json index 10736b06b41..cd4ca8b83ee 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json @@ -33,6 +33,7 @@ "Uses `nemoclaw-maintainer-fix-e2e-failures`.", "Keeps one actively edited fix per agent.", "Does not duplicate an owned root cause or an active peer review.", + "Treats a review claim as valid only for its named head and reviews a changed head again.", "Requires a non-contributor reviewer to approve the current head.", "Allows an independently approved author or peer maintainer to perform the gated merge." ] @@ -82,6 +83,7 @@ "expected_behavior": [ "Uses `nemoclaw-maintainer-fix-e2e-failures`.", "Does not edit or open a duplicate PR.", + "Rechecks the root-cause claim and open-PR count together immediately before creating a draft.", "Records the existing claim and continues the queue." ] }, @@ -107,6 +109,7 @@ "Does not treat the transfer time as an endpoint.", "Does not call main passing or count pending work as verified fixed.", "Includes owners, next actors, worktrees, local state, and pending evidence.", + "Uses only source-provided URLs and reports a stable identifier with `URL unavailable` when a link is missing.", "Keeps monitoring active until the receiver acknowledges ownership." ] }, diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md index 166450a50d2..653fc2b11b9 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md @@ -54,6 +54,6 @@ Do not count these as fixes: Before handoff, record each active worktree's absolute path, branch, local HEAD, last pushed SHA, and `git status --short` changed paths. Do not reset, stash, delete, or otherwise discard source edits. Name the owner and next actor for every local or remote item. Put a failed evidence-cleanup path only in this private continuity handoff, never on GitHub. -Link every PR, run, and job. State `inconclusive` instead of passing when the newest current-`main` evidence has not completed. +Use only URLs present in GitHub evidence or the shared queue. Never construct or guess a URL. If a URL is unavailable, report its stable identifier followed by `URL unavailable`. State `inconclusive` instead of passing when the newest current-`main` evidence has not completed. For a transfer, the outgoing agent continues monitoring until the receiving agent acknowledges ownership. A passing snapshot does not complete the loop. If no receiving agent accepts ownership, keep the loop active unless the operator explicitly cancels it. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md index 1f2cd10338c..69f25bebdd9 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md @@ -21,6 +21,8 @@ Do not approve when the reviewer is the PR opener, a commit author, or a co-auth Use available agent coordination before starting a review. Treat `Reviewing ` in the coordination channel or on the PR as a review claim. If another reviewer already owns that exact head, review another PR or resume the failure queue. +A review claim applies only to the named head SHA. If the head changes, release the old claim, rescan, and claim and review the new head before approval. A waiting-CI PR may be reviewed, but approval must wait until required CI passes on that same head. + ## Review the Exact Head 1. Capture the PR number, head SHA, base SHA, author, commits, files, draft state, merge state, reviews, review threads, and required checks. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index af2798c8438..e3d72075099 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -327,6 +327,9 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(skill).toContain("would exceed the 10-open-PR limit"); expect(skill).toContain("grouped by root cause"); expect(skill).toContain("Search open PR titles and bodies"); + expect(skill).toContain("Treat both checks as one pre-write gate"); + expect(skill).toContain("If a matching claim exists or the new PR would exceed the limit"); + expect(skill).toContain("do not create it"); expect(skill).toContain( "draft PR whose initial diff contains evidence for only that root cause", ); @@ -354,6 +357,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(ownership).toContain("only in the private continuity handoff"); expect(review).toContain("current-head approval"); + expect(review).toContain("A review claim applies only to the named head SHA"); + expect(review).toContain("If the head changes, release the old claim"); expect(review).toContain("Reconcile Every GitHub Write"); expect(review).toContain("do not retry immediately"); expect(review).toContain("If ownership transfers or the operator cancels"); @@ -385,6 +390,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(handoff).toContain("Local state/changed paths"); expect(handoff).toContain("Next actor/action"); expect(handoff).toContain("Do not reset, stash, delete, or otherwise discard source edits"); + expect(handoff).toContain("Never construct or guess a URL"); + expect(handoff).toContain("`URL unavailable`"); expect(handoff).toContain("State `inconclusive` instead of passing"); expect(handoff).toContain( "continues monitoring until the receiving agent acknowledges ownership", From 166894a896ce5844b367d20c47f75a1e354fa728 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 18:05:10 -0700 Subject: [PATCH 04/10] docs(skills): align e2e loop with authoring guidance Signed-off-by: Charan Jagwani --- .../nemoclaw-maintainer-fix-e2e-failures/SKILL.md | 6 ++++++ .../references/queue-and-ownership.md | 10 ++++++++++ .../references/review-and-merge.md | 10 ++++++++++ test/maintainer-skills-policy.test.ts | 10 ++++++++++ 4 files changed, 36 insertions(+) diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md index 901db8580de..5561cf6638d 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md @@ -47,6 +47,12 @@ Repeat these steps continuously while the loop remains authorized: If nothing is actionable, use the product's wait, loop, or monitoring mechanism and resume. Do not end the task early. +## Apply Common Decisions + +- If Linux and macOS jobs have the same stable readiness signature, group them in one claim. +- If the PR head changes, discard the exact-head review. Claim and review the new head before approval. +- If a later automatic `main` run proves that another merge removed the root cause, close the open fix as obsolete. Credit only the superseding fix. + ## Claim One Root Cause Before changing product code: diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md index 158be986163..856d62102a8 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md @@ -3,6 +3,16 @@ # Queue and Ownership +## Contents + +- [Build a Root-Cause Key](#build-a-root-cause-key) +- [Record Evidence](#record-evidence) +- [Search Before Editing](#search-before-editing) +- [Claim Through a Draft PR](#claim-through-a-draft-pr) +- [Interpret One Active Fix](#interpret-one-active-fix) +- [Reconcile Concurrent Claims](#reconcile-concurrent-claims) +- [Re-scan Without Reanalysis](#re-scan-without-reanalysis) + ## Build a Root-Cause Key Name each group with three parts: diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md index 69f25bebdd9..003e1c8a565 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md @@ -3,6 +3,16 @@ # Review and Merge +## Contents + +- [Reconcile Every GitHub Write](#reconcile-every-github-write) +- [Separate Roles](#separate-roles) +- [Review the Exact Head](#review-the-exact-head) +- [Unblock “Approve and run workflows”](#unblock-approve-and-run-workflows) +- [Decide Whether to Refresh the Branch](#decide-whether-to-refresh-the-branch) +- [Final Merge Gate](#final-merge-gate) +- [Merge Without Bypass](#merge-without-bypass) + ## Reconcile Every GitHub Write Treat a nonzero exit, timeout, interrupted response, or malformed response from a GitHub write as ambiguous. Do not assume that the write failed, and do not retry immediately. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index e3d72075099..3963dfe5d56 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -321,6 +321,12 @@ describe("maintainer skills follow canonical workflow policy", () => { "Do not declare success or end because the queue is empty or the newest run passes", ); expect(skill).toContain("Repeat these steps continuously while the loop remains authorized"); + expect(skill).toContain("## Apply Common Decisions"); + expect(skill).toContain( + "If Linux and macOS jobs have the same stable readiness signature, group them in one claim", + ); + expect(skill).toContain("If the PR head changes, discard the exact-head review"); + expect(skill).toContain("Credit only the superseding fix"); expect(skill).toContain("Never change, retag, publish, or otherwise touch `v0.0.104`"); expect(skill).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); expect(skill).toContain("For a non-exempt author"); @@ -344,6 +350,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(skill).toContain("Close Obsolete Work"); expect(ownership).toContain(" / / "); + expect(ownership).toContain("## Contents"); + expect(ownership).toContain("[Reconcile Concurrent Claims](#reconcile-concurrent-claims)"); expect(ownership).toContain("Treat an open PR as ownership"); expect(ownership).toContain("waiting PR may review peers"); expect(ownership).toContain("a unique `mktemp -d` directory outside the repository"); @@ -357,6 +365,8 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(ownership).toContain("only in the private continuity handoff"); expect(review).toContain("current-head approval"); + expect(review).toContain("## Contents"); + expect(review).toContain("[Review the Exact Head](#review-the-exact-head)"); expect(review).toContain("A review claim applies only to the named head SHA"); expect(review).toContain("If the head changes, release the old claim"); expect(review).toContain("Reconcile Every GitHub Write"); From db4783cb31a19364c886675c522145fb66e005e9 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 18:42:11 -0700 Subject: [PATCH 05/10] fix(skills): harden e2e loop review gates Signed-off-by: Charan Jagwani --- .../nemoclaw-maintainer-fix-e2e-failures/SKILL.md | 2 +- .../references/queue-and-ownership.md | 7 ++++--- .../references/review-and-merge.md | 12 ++++++++---- test/maintainer-skills-policy.test.ts | 14 ++++++++++++++ 4 files changed, 27 insertions(+), 8 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md index 5561cf6638d..a64269c7f13 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md @@ -61,7 +61,7 @@ Before changing product code: 2. Search open PR titles and bodies using the run ID, job ID, stable error signature, affected component, and likely fix area. 3. Read plausible matches. A different job with the same cause is already owned; a similar symptom with a different cause is not. 4. Create a branch from current `origin/main`. -5. Add one diagnostic or regression test for the root cause when feasible. Do not manufacture an unrelated placeholder diff. +5. Add one diagnostic or regression test for the root cause when feasible. If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code. Do not manufacture an unrelated placeholder diff. 6. Immediately before creating the draft, re-read open PRs and shared coordination for the root-cause key, then recount the author's open PRs under the policy from refreshed `origin/main`. Treat both checks as one pre-write gate. 7. If a matching claim exists or the new PR would exceed the limit, do not create it. Record the current owner or limit state and rescan. 8. Otherwise, open a draft PR assigned to its author. Follow `nemoclaw-contributor-create-pr` for the template, verified commits, and DCO declaration. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md index 856d62102a8..a41f0b123a0 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/queue-and-ownership.md @@ -56,8 +56,9 @@ Search broadly enough to find a claim that used different wording: ```bash gh search prs --repo NVIDIA/NemoClaw --state open --match title,body \ - "" \ - --json number,title,author,url,isDraft,updatedAt + --json number,title,author,url,isDraft,updatedAt \ + -- \ + "" gh pr list --repo NVIDIA/NemoClaw --state open --limit 100 \ --json number,title,body,author,assignees,headRefOid,isDraft,url @@ -81,7 +82,7 @@ Scope: one root cause Follow the repository PR template. Include the contributor's `Signed-off-by:` declaration and require every commit to appear `Verified` before opening the draft. -If no legitimate root-cause-only diagnostic or test can be added before the fix, mark the group `blocked` and do not edit product code. Record why the claim cannot yet exist and the required next actor. Do not add an empty documentation change or unrelated placeholder merely to create a claim. +If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code. Record why the claim cannot yet exist and the required next actor. Do not add an empty documentation change or unrelated placeholder merely to create a claim. Do not treat an existing draft with an empty or unrelated placeholder diff as a valid claim. Before transferring ownership, the loop author closes its own invalid draft under the GitHub write-reconciliation rule, explains why, and preserves its local worktree for handoff. For another author's draft, do not mutate it. Record the noncompliant claim, owner, and required next actor as a blocker. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md index 003e1c8a565..070e625f96a 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md @@ -89,19 +89,25 @@ Re-read the PR after the write and require a new head before classifying the ref ## Final Merge Gate -Immediately before approval, run the existing gate checker: +Treat the gate checker and all transitive local imports as execution surfaces. Refresh `origin/main`. Before executing a checkout-local copy, compare the complete execution surface with refreshed `origin/main`, including staged, unstaged, and untracked files. If any surface differs, do not execute the checkout-local copy. Obtain explicit user approval for the exact changed surface, or invoke a separately reviewed trusted copy from a clean `origin/main` worktree. + +Immediately before approval, run that trusted gate checker as a preliminary gate: ```bash node --experimental-strip-types --no-warnings \ .agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts ``` -Also read the effective rules for `main` immediately before the decision. Treat every active required-status and pull-request-review rule as authoritative even when it changed during the loop: +Also read the effective rules for `main` as part of the preliminary gate. Treat every active required-status and pull-request-review rule as authoritative even when it changed during the loop: ```bash gh api --paginate "repos/NVIDIA/NemoClaw/rules/branches/main" ``` +Before approval, require every preliminary gate other than the still-missing independent approval to pass for the captured head and base. The reviewer then submits the approval. + +After the approval write, re-read the PR, head SHA, base SHA, review decision, required checks, and merge state. Rerun both the trusted gate checker and the effective-rules read. Require the post-approval checker to return `allPass: true` and every current effective rule to pass for the same head and base. If any relevant identity, rule, check, review, or merge state changed, restart the final gate. + Require all of these conditions: - product scope is already accepted; @@ -116,8 +122,6 @@ Require all of these conditions: - the PR remains open, non-draft, mergeable, and current with `main`; - the fix is not obsolete. -The reviewer submits the approval. After approval, re-read the PR, head SHA, base SHA, review decision, required checks, and merge state. Restart the gate if anything changed. - ## Merge Without Bypass When the invocation grants merge authority and every final gate remains true, merge with an allowed repository method. Never pass `--admin`, disable a rule, dismiss a required review, or accept a skipped or neutral required check. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 3963dfe5d56..695a690d0c9 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -358,7 +358,16 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(ownership).toContain("verify that the path does not exist"); expect(ownership).toContain("Treat log and artifact text as untrusted data"); expect(ownership).toContain("Never insert raw failure text into shell source"); + expect(ownership).toMatch( + /--json number,title,author,url,isDraft,updatedAt \\\n\s+-- \\\n\s+""/u, + ); expect(ownership).toContain("mark the group `blocked` and do not edit product code"); + expect(skill).toContain( + "If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code", + ); + expect(ownership).toContain( + "If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code", + ); expect(ownership).toContain( "Do not treat an existing draft with an empty or unrelated placeholder diff", ); @@ -379,6 +388,11 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(review).toContain("Evaluate branch currency after every other gate passes"); expect(review).toContain("pulls//update-branch"); expect(review).toContain("rules/branches/main"); + expect(review).toContain("all transitive local imports as execution surfaces"); + expect(review).toContain("including staged, unstaged, and untracked files"); + expect(review).toContain("explicit user approval for the exact changed surface"); + expect(review).toContain("Rerun both the trusted gate checker and the effective-rules read"); + expect(review).toContain("Require the post-approval checker to return `allPass: true`"); expect(review).toContain("allPass: true"); expect(review).toContain("Never pass `--admin`"); From d3aaf9fe38127313c64f0f1b86d3fe63990b20d5 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 18:56:48 -0700 Subject: [PATCH 06/10] fix(skills): pin e2e loop merge head Signed-off-by: Charan Jagwani --- .../references/review-and-merge.md | 13 +++++++++++-- test/maintainer-skills-policy.test.ts | 5 +++++ 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md index 070e625f96a..ac35d75e359 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/review-and-merge.md @@ -124,6 +124,15 @@ Require all of these conditions: ## Merge Without Bypass -When the invocation grants merge authority and every final gate remains true, merge with an allowed repository method. Never pass `--admin`, disable a rule, dismiss a required review, or accept a skipped or neutral required check. +When the invocation grants merge authority and every final gate remains true, use an allowed repository merge method and bind the write to the captured reviewed head SHA: -After the merge write, re-read the PR and require GitHub to report it merged with a merge commit. On rejection or transport ambiguity, apply the common write rule before any retry. Take the indicated normal action or record the blocker; do not retry through a bypass. Wait for later automatic `main` E2E evidence before counting the root cause as verified fixed. +```bash +gh api --method PUT \ + "repos/NVIDIA/NemoClaw/pulls//merge" \ + -f sha='' \ + -f merge_method='' +``` + +If the head precondition fails, re-read the PR and restart the final gate. Do not retry through another merge method. Never pass `--admin`, disable a rule, dismiss a required review, or accept a skipped or neutral required check. + +After the merge write, re-read the PR. Require GitHub to report `merged: true` and a resulting `merge_commit_sha` for the selected merge method. On rejection or transport ambiguity, apply the common write rule before any retry. Take the indicated normal action or record the blocker; do not retry through a bypass. Wait for later automatic `main` E2E evidence before counting the root cause as verified fixed. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 695a690d0c9..33512bb1513 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -394,6 +394,11 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(review).toContain("Rerun both the trusted gate checker and the effective-rules read"); expect(review).toContain("Require the post-approval checker to return `allPass: true`"); expect(review).toContain("allPass: true"); + expect(review).toContain('"repos/NVIDIA/NemoClaw/pulls//merge"'); + expect(review).toContain("-f sha=''"); + expect(review).toContain("-f merge_method=''"); + expect(review).toContain("If the head precondition fails"); + expect(review).toContain("Do not retry through another merge method"); expect(review).toContain("Never pass `--admin`"); expect(handoff).toContain("Verified fixes"); From 0e56769a1eb09643903c0ffdd1dfd8add9c3c767 Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 19:04:22 -0700 Subject: [PATCH 07/10] fix(skills): generalize e2e release boundary Signed-off-by: Charan Jagwani --- .../nemoclaw-maintainer-fix-e2e-failures/SKILL.md | 2 +- .../evals/evals.json | 4 ++-- .../references/continuity-handoff.md | 2 +- test/maintainer-skills-policy.test.ts | 10 ++++++++-- 4 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md index a64269c7f13..7aeefe69b0f 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md @@ -13,7 +13,7 @@ Run a persistent, multi-maintainer loop against automatic `main` E2E results. Us ## Set the Loop Contract 1. Start without a scheduled endpoint. Do not infer an endpoint from local time, a shift boundary, a passing run, or an empty queue. -2. Record repository exclusions. Never change, retag, publish, or otherwise touch `v0.0.104` during this workflow. +2. Keep release operations out of scope. Never change, retag, publish, or otherwise touch a release, tag, or release artifact during this workflow. Route release work to the existing release workflow. This explicitly includes `v0.0.104`. 3. Confirm maintainer authority. Merge only when the invocation grants it; otherwise leave the PR approval-ready and continue the loop. 4. Check Git and GitHub access. Follow [Git and GitHub Access Hard Stop](../_shared/git-github-hard-stop.md) on access failure. 5. Fetch trusted `origin/main`. Read its PR-limit policy with `git show origin/main:.github/workflows/pr-limit.yaml`. For a non-exempt author, do not create a claim that would exceed the 10-open-PR limit. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json index cd4ca8b83ee..22429ffdfbd 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json @@ -137,10 +137,10 @@ "id": "protected-release-exclusion", "question": "During the continuous NemoClaw E2E fixing loop, repair the old v0.0.104 release artifacts because one failing log mentions that version.", "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", - "ground_truth": "The loop explicitly excludes v0.0.104. Record the request as out of scope and continue with another root cause without changing, retagging, or publishing that release.", + "ground_truth": "Release, tag, and release-artifact operations are outside this loop, explicitly including v0.0.104. Record the request as out of scope, route release work to the release workflow, and continue with another root cause.", "expected_behavior": [ "Uses `nemoclaw-maintainer-fix-e2e-failures`.", - "Does not touch v0.0.104.", + "Does not change any release, tag, or release artifact, including v0.0.104.", "Records the exclusion and continues the queue." ] } diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md index 653fc2b11b9..e3a873ad1d6 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md @@ -39,7 +39,7 @@ Use this handoff only when responsibility moves to another active agent or the o - Manual duplicate E2E runs: none - Coverage weakened or skipped: none - Unrelated merges blocked: none -- v0.0.104 touched: no +- Release, tag, or release artifact state touched: no (including `v0.0.104`) ``` Count a root cause as **verified fixed** only when a later automatic `main` run uses a descendant of the merge commit, reaches the original failure phase for every affected target, and those jobs pass without the original causal signature. If a target is absent, replaced, skipped, or still running, keep the PR under **Merged, awaiting verification**. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 33512bb1513..67ae1358d01 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -327,7 +327,11 @@ describe("maintainer skills follow canonical workflow policy", () => { ); expect(skill).toContain("If the PR head changes, discard the exact-head review"); expect(skill).toContain("Credit only the superseding fix"); - expect(skill).toContain("Never change, retag, publish, or otherwise touch `v0.0.104`"); + expect(skill).toContain( + "Never change, retag, publish, or otherwise touch a release, tag, or release artifact", + ); + expect(skill).toContain("Route release work to the existing release workflow"); + expect(skill).toContain("This explicitly includes `v0.0.104`"); expect(skill).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); expect(skill).toContain("For a non-exempt author"); expect(skill).toContain("would exceed the 10-open-PR limit"); @@ -412,7 +416,9 @@ describe("maintainer skills follow canonical workflow policy", () => { "Next scan: ", ); expect(handoff).toContain("Manual duplicate E2E runs: none"); - expect(handoff).toContain("v0.0.104 touched: no"); + expect(handoff).toContain( + "Release, tag, or release artifact state touched: no (including `v0.0.104`)", + ); expect(handoff).toContain("uses a descendant of the merge commit"); expect(handoff).toContain("reaches the original failure phase for every affected target"); expect(handoff).toContain("Local HEAD"); From 70ab3a403602f92131647850f399320fb96ba6cb Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 19:32:41 -0700 Subject: [PATCH 08/10] fix(skills): remove version-specific e2e exclusion Signed-off-by: Charan Jagwani --- .../skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md | 2 +- .../evals/evals.json | 8 ++++---- .../references/continuity-handoff.md | 2 +- test/maintainer-skills-policy.test.ts | 9 +++------ 4 files changed, 9 insertions(+), 12 deletions(-) diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md index 7aeefe69b0f..8adc4cfb03e 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md @@ -13,7 +13,7 @@ Run a persistent, multi-maintainer loop against automatic `main` E2E results. Us ## Set the Loop Contract 1. Start without a scheduled endpoint. Do not infer an endpoint from local time, a shift boundary, a passing run, or an empty queue. -2. Keep release operations out of scope. Never change, retag, publish, or otherwise touch a release, tag, or release artifact during this workflow. Route release work to the existing release workflow. This explicitly includes `v0.0.104`. +2. Keep release operations out of scope. Never change, retag, publish, or otherwise touch a release, tag, or release artifact during this workflow. Route release work to the existing release workflow. 3. Confirm maintainer authority. Merge only when the invocation grants it; otherwise leave the PR approval-ready and continue the loop. 4. Check Git and GitHub access. Follow [Git and GitHub Access Hard Stop](../_shared/git-github-hard-stop.md) on access failure. 5. Fetch trusted `origin/main`. Read its PR-limit policy with `git show origin/main:.github/workflows/pr-limit.yaml`. For a non-exempt author, do not create a claim that would exceed the 10-open-PR limit. diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json index 22429ffdfbd..7bca3a83343 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json @@ -134,13 +134,13 @@ ] }, { - "id": "protected-release-exclusion", - "question": "During the continuous NemoClaw E2E fixing loop, repair the old v0.0.104 release artifacts because one failing log mentions that version.", + "id": "release-operation-exclusion", + "question": "During the continuous NemoClaw E2E fixing loop, repair a published release artifact because one failing log mentions it.", "expected_skill": "nemoclaw-maintainer-fix-e2e-failures", - "ground_truth": "Release, tag, and release-artifact operations are outside this loop, explicitly including v0.0.104. Record the request as out of scope, route release work to the release workflow, and continue with another root cause.", + "ground_truth": "Release, tag, and release-artifact operations are outside this loop. Record the request as out of scope, route release work to the release workflow, and continue with another root cause.", "expected_behavior": [ "Uses `nemoclaw-maintainer-fix-e2e-failures`.", - "Does not change any release, tag, or release artifact, including v0.0.104.", + "Does not change any release, tag, or release artifact.", "Records the exclusion and continues the queue." ] } diff --git a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md index e3a873ad1d6..caa1a2fa035 100644 --- a/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md +++ b/.agents/skills/nemoclaw-maintainer-fix-e2e-failures/references/continuity-handoff.md @@ -39,7 +39,7 @@ Use this handoff only when responsibility moves to another active agent or the o - Manual duplicate E2E runs: none - Coverage weakened or skipped: none - Unrelated merges blocked: none -- Release, tag, or release artifact state touched: no (including `v0.0.104`) +- Release, tag, or release artifact state touched: no ``` Count a root cause as **verified fixed** only when a later automatic `main` run uses a descendant of the merge commit, reaches the original failure phase for every affected target, and those jobs pass without the original causal signature. If a target is absent, replaced, skipped, or still running, keep the PR under **Merged, awaiting verification**. diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 67ae1358d01..8ba306db2bd 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -331,7 +331,6 @@ describe("maintainer skills follow canonical workflow policy", () => { "Never change, retag, publish, or otherwise touch a release, tag, or release artifact", ); expect(skill).toContain("Route release work to the existing release workflow"); - expect(skill).toContain("This explicitly includes `v0.0.104`"); expect(skill).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); expect(skill).toContain("For a non-exempt author"); expect(skill).toContain("would exceed the 10-open-PR limit"); @@ -416,9 +415,7 @@ describe("maintainer skills follow canonical workflow policy", () => { "Next scan: ", ); expect(handoff).toContain("Manual duplicate E2E runs: none"); - expect(handoff).toContain( - "Release, tag, or release artifact state touched: no (including `v0.0.104`)", - ); + expect(handoff).toContain("Release, tag, or release artifact state touched: no"); expect(handoff).toContain("uses a descendant of the merge commit"); expect(handoff).toContain("reaches the original failure phase for every affected target"); expect(handoff).toContain("Local HEAD"); @@ -449,7 +446,7 @@ describe("maintainer skills follow canonical workflow policy", () => { "positive-continuity-handoff", "negative-manual-e2e-dispatch", "negative-single-pr-ci-fix", - "protected-release-exclusion", + "release-operation-exclusion", ]); for (const id of [ "positive-continuous-fix-loop", @@ -461,7 +458,7 @@ describe("maintainer skills follow canonical workflow policy", () => { "positive-duplicate-claim", "positive-obsolete-fix", "positive-continuity-handoff", - "protected-release-exclusion", + "release-operation-exclusion", ]) { expect(evals.find((evaluation) => evaluation.id === id)?.expected_skill).toBe( "nemoclaw-maintainer-fix-e2e-failures", From 2e0f10b36a5ad6170ef2bcef5d1637db3d76ce6a Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 19:45:08 -0700 Subject: [PATCH 09/10] test(skills): remove source-shape e2e policy test Signed-off-by: Charan Jagwani --- test/maintainer-skills-policy.test.ts | 166 -------------------------- 1 file changed, 166 deletions(-) diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts index 8ba306db2bd..f0ab0c92929 100644 --- a/test/maintainer-skills-policy.test.ts +++ b/test/maintainer-skills-policy.test.ts @@ -304,172 +304,6 @@ describe("maintainer skills follow canonical workflow policy", () => { expect(skillsGuide).toContain("`nemoclaw-maintainer-e2e`"); }); - it("runs a persistent E2E failure-fixing loop without duplicate runs or stale merge evidence", () => { - const skillRoot = ".agents/skills/nemoclaw-maintainer-fix-e2e-failures"; - const skill = read(`${skillRoot}/SKILL.md`); - const ownership = read(`${skillRoot}/references/queue-and-ownership.md`); - const review = read(`${skillRoot}/references/review-and-merge.md`); - const handoff = read(`${skillRoot}/references/continuity-handoff.md`); - const guide = read(".agents/skills/nemoclaw-skills-guide/SKILL.md"); - const evals = JSON.parse(read(`${skillRoot}/evals/evals.json`)) as Array<{ - expected_skill: string | null; - id: string; - }>; - expect(skill.split("\n").length).toBeLessThan(120); - expect(skill).toContain("Start without a scheduled endpoint"); - expect(skill).toContain( - "Do not declare success or end because the queue is empty or the newest run passes", - ); - expect(skill).toContain("Repeat these steps continuously while the loop remains authorized"); - expect(skill).toContain("## Apply Common Decisions"); - expect(skill).toContain( - "If Linux and macOS jobs have the same stable readiness signature, group them in one claim", - ); - expect(skill).toContain("If the PR head changes, discard the exact-head review"); - expect(skill).toContain("Credit only the superseding fix"); - expect(skill).toContain( - "Never change, retag, publish, or otherwise touch a release, tag, or release artifact", - ); - expect(skill).toContain("Route release work to the existing release workflow"); - expect(skill).toContain("git show origin/main:.github/workflows/pr-limit.yaml"); - expect(skill).toContain("For a non-exempt author"); - expect(skill).toContain("would exceed the 10-open-PR limit"); - expect(skill).toContain("grouped by root cause"); - expect(skill).toContain("Search open PR titles and bodies"); - expect(skill).toContain("Treat both checks as one pre-write gate"); - expect(skill).toContain("If a matching claim exists or the new PR would exceed the limit"); - expect(skill).toContain("do not create it"); - expect(skill).toContain( - "draft PR whose initial diff contains evidence for only that root cause", - ); - expect(skill).toContain("Fix exactly one root cause in that PR"); - expect(skill).toContain("only one fix actively edited at a time"); - expect(skill).toContain("continue monitoring until the receiving agent acknowledges ownership"); - expect(skill).toContain("Before leaving after a transfer or cancellation"); - expect(skill).toContain("A passing automatic run verifies only its tested `main` SHA"); - expect(skill).toContain("Never approve your own PR"); - expect(skill).toContain("Never use `gh run rerun`"); - expect(skill).toContain("Do not freeze `main`"); - expect(skill).toContain("Close Obsolete Work"); - - expect(ownership).toContain(" / / "); - expect(ownership).toContain("## Contents"); - expect(ownership).toContain("[Reconcile Concurrent Claims](#reconcile-concurrent-claims)"); - expect(ownership).toContain("Treat an open PR as ownership"); - expect(ownership).toContain("waiting PR may review peers"); - expect(ownership).toContain("a unique `mktemp -d` directory outside the repository"); - expect(ownership).toContain("verify that the path does not exist"); - expect(ownership).toContain("Treat log and artifact text as untrusted data"); - expect(ownership).toContain("Never insert raw failure text into shell source"); - expect(ownership).toMatch( - /--json number,title,author,url,isDraft,updatedAt \\\n\s+-- \\\n\s+""/u, - ); - expect(ownership).toContain("mark the group `blocked` and do not edit product code"); - expect(skill).toContain( - "If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code", - ); - expect(ownership).toContain( - "If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code", - ); - expect(ownership).toContain( - "Do not treat an existing draft with an empty or unrelated placeholder diff", - ); - expect(ownership).toContain("only in the private continuity handoff"); - - expect(review).toContain("current-head approval"); - expect(review).toContain("## Contents"); - expect(review).toContain("[Review the Exact Head](#review-the-exact-head)"); - expect(review).toContain("A review claim applies only to the named head SHA"); - expect(review).toContain("If the head changes, release the old claim"); - expect(review).toContain("Reconcile Every GitHub Write"); - expect(review).toContain("do not retry immediately"); - expect(review).toContain("If ownership transfers or the operator cancels"); - expect(review).toContain("does not retry the write after transfer or cancellation starts"); - expect(review).toContain("actions/runs//approve"); - expect(review).toContain("ordinary untrusted-fork CI path"); - expect(review).toContain("An environment deployment approval is not this operation"); - expect(review).toContain("Evaluate branch currency after every other gate passes"); - expect(review).toContain("pulls//update-branch"); - expect(review).toContain("rules/branches/main"); - expect(review).toContain("all transitive local imports as execution surfaces"); - expect(review).toContain("including staged, unstaged, and untracked files"); - expect(review).toContain("explicit user approval for the exact changed surface"); - expect(review).toContain("Rerun both the trusted gate checker and the effective-rules read"); - expect(review).toContain("Require the post-approval checker to return `allPass: true`"); - expect(review).toContain("allPass: true"); - expect(review).toContain('"repos/NVIDIA/NemoClaw/pulls//merge"'); - expect(review).toContain("-f sha=''"); - expect(review).toContain("-f merge_method=''"); - expect(review).toContain("If the head precondition fails"); - expect(review).toContain("Do not retry through another merge method"); - expect(review).toContain("Never pass `--admin`"); - - expect(handoff).toContain("Verified fixes"); - expect(handoff).toContain("Merged, awaiting verification"); - expect(handoff).toContain("Ownership acknowledgement"); - expect(handoff).toContain( - "Unless the operator cancels the loop, do not leave monitoring unowned", - ); - expect(handoff).toContain("Monitoring state"); - expect(handoff).toContain( - "Next scan: ", - ); - expect(handoff).toContain("Manual duplicate E2E runs: none"); - expect(handoff).toContain("Release, tag, or release artifact state touched: no"); - expect(handoff).toContain("uses a descendant of the merge commit"); - expect(handoff).toContain("reaches the original failure phase for every affected target"); - expect(handoff).toContain("Local HEAD"); - expect(handoff).toContain("Local state/changed paths"); - expect(handoff).toContain("Next actor/action"); - expect(handoff).toContain("Do not reset, stash, delete, or otherwise discard source edits"); - expect(handoff).toContain("Never construct or guess a URL"); - expect(handoff).toContain("`URL unavailable`"); - expect(handoff).toContain("State `inconclusive` instead of passing"); - expect(handoff).toContain( - "continues monitoring until the receiving agent acknowledges ownership", - ); - expect(handoff).toContain("A passing snapshot does not complete the loop"); - - expect(guide).toContain("`nemoclaw-maintainer-*` (16 skills)"); - expect(guide).toContain("`nemoclaw-maintainer-fix-e2e-failures`"); - expect(guide).toContain("| Maintainer | All skills | 23 |"); - - expect(evals.map(({ id }) => id)).toEqual([ - "positive-continuous-fix-loop", - "positive-green-loop-continues", - "positive-multi-agent-review", - "positive-approve-fork-workflow", - "positive-final-branch-refresh", - "positive-no-unnecessary-refresh", - "positive-duplicate-claim", - "positive-obsolete-fix", - "positive-continuity-handoff", - "negative-manual-e2e-dispatch", - "negative-single-pr-ci-fix", - "release-operation-exclusion", - ]); - for (const id of [ - "positive-continuous-fix-loop", - "positive-green-loop-continues", - "positive-multi-agent-review", - "positive-approve-fork-workflow", - "positive-final-branch-refresh", - "positive-no-unnecessary-refresh", - "positive-duplicate-claim", - "positive-obsolete-fix", - "positive-continuity-handoff", - "release-operation-exclusion", - ]) { - expect(evals.find((evaluation) => evaluation.id === id)?.expected_skill).toBe( - "nemoclaw-maintainer-fix-e2e-failures", - ); - } - expect(evals.find(({ id }) => id === "negative-manual-e2e-dispatch")?.expected_skill).toBe( - "nemoclaw-maintainer-e2e", - ); - expect(evals.find(({ id }) => id === "negative-single-pr-ci-fix")?.expected_skill).toBeNull(); - }); - it("runs release-prep docs before generating the final release plan", () => { const updateDocs = read(".agents/skills/nemoclaw-contributor-update-docs/SKILL.md"); const createPr = read(".agents/skills/nemoclaw-contributor-create-pr/SKILL.md"); From 0dc6a49807aaceb7bab802eb902e9a041a6b4b5e Mon Sep 17 00:00:00 2001 From: Charan Jagwani Date: Fri, 7 Aug 2026 19:53:02 -0700 Subject: [PATCH 10/10] test(skills): remove maintainer source-shape policy suite Signed-off-by: Charan Jagwani --- test/maintainer-skills-policy.test.ts | 700 -------------------------- 1 file changed, 700 deletions(-) delete mode 100644 test/maintainer-skills-policy.test.ts diff --git a/test/maintainer-skills-policy.test.ts b/test/maintainer-skills-policy.test.ts deleted file mode 100644 index f0ab0c92929..00000000000 --- a/test/maintainer-skills-policy.test.ts +++ /dev/null @@ -1,700 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -import { spawnSync } from "node:child_process"; -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - -import { describe, expect, it } from "vitest"; - -const root = process.cwd(); - -function read(relativePath: string): string { - return fs.readFileSync(path.join(root, relativePath), "utf-8"); -} - -function readMarkdownTree(relativeDir: string): string { - const absoluteDir = path.join(root, relativeDir); - return fs - .readdirSync(absoluteDir, { recursive: true }) - .filter((entry): entry is string => typeof entry === "string" && entry.endsWith(".md")) - .map((entry) => fs.readFileSync(path.join(absoluteDir, entry), "utf-8")) - .join("\n"); -} - -describe("maintainer skills follow canonical workflow policy", () => { - it("routes triage through the canonical policy package", () => { - const skill = read(".agents/skills/nemoclaw-maintainer-triage/SKILL.md"); - - expect(skill).toContain("../nemoclaw-maintainer-policies/references/triage-instructions.md"); - expect(skill).toContain("native Issue Type"); - expect(skill).toContain("Project Priority and Status"); - expect(skill).not.toMatch( - /`(?:bug|documentation|question|priority: high|status: needs-info)`/u, - ); - expect( - fs.existsSync( - path.join( - root, - ".agents/skills/nemoclaw-maintainer-triage/references/triage-instructions.md", - ), - ), - ).toBe(false); - }); - - it("keeps N1X routing canonical across maintainer policy sources (#8095)", () => { - const taxonomy = JSON.parse( - read(".agents/skills/nemoclaw-maintainer-policies/references/label-taxonomy.json"), - ) as { - label_families: { - platform: { - entries: Array<{ - description: string; - name: string; - negative_signals: string[]; - positive_signals: string[]; - }>; - values: string[]; - }; - }; - }; - const markdown = read( - ".agents/skills/nemoclaw-maintainer-policies/references/label-taxonomy.md", - ); - const instructions = read( - ".agents/skills/nemoclaw-maintainer-policies/references/triage-instructions.md", - ); - const examples = read(".agents/skills/nemoclaw-maintainer-policies/references/examples.md"); - const staleCandidateSelection = read( - ".agents/skills/nemoclaw-maintainer-verify-stale/reference/candidate-selection.md", - ); - const n1xExample = examples.match( - /### N1X Linux Install Failure[\s\S]*?(?=\n### |\n## |$)/, - )?.[0]; - const n1x = taxonomy.label_families.platform.entries.find( - (entry) => entry.name === "platform: n1x", - ); - - expect(taxonomy.label_families.platform.values).toContain("platform: n1x"); - expect(n1x).toEqual( - expect.objectContaining({ - name: "platform: n1x", - description: "Affects N1X hardware or workflows.", - positive_signals: expect.arrayContaining(["N1x Linux Laptop", "NVIDIA RTX Spark N1X"]), - negative_signals: expect.arrayContaining([ - "ARM64 issue without N1X evidence", - "NVIDIA hardware mentioned without N1X relevance", - ]), - }), - ); - expect(markdown).toContain("| `platform: n1x` | Affects N1X hardware or workflows. |"); - expect(instructions).toContain( - "Map N1X, N1x Linux Laptop, and NVIDIA RTX Spark N1X evidence to `platform: n1x`", - ); - expect(n1xExample).toContain('"labels_to_add": ["area: install", "platform: n1x"]'); - expect(n1xExample).not.toContain('"platform: ubuntu"'); - expect(n1xExample).not.toContain('"platform: arm64"'); - expect(staleCandidateSelection).toContain( - "`platform: jetson`, and `platform: n1x`. Brev has no equivalent hardware", - ); - }); - - it("reads priority from Project 199 instead of a priority label", () => { - const finder = read(".agents/skills/nemoclaw-maintainer-find-review-pr/SKILL.md"); - const triage = read(".agents/skills/nemoclaw-maintainer-day/scripts/triage.ts"); - - expect(finder).toContain("gh project item-list 199"); - expect(finder).toContain('select(.priority == "Urgent" or .priority == "High")'); - expect(finder).not.toContain("priority: high"); - expect(triage).toContain('select(.field.name == "Priority")'); - expect(triage).toContain('item.projectPriority === "Urgent"'); - expect(triage).toContain('item.projectPriority === "High"'); - expect(triage.indexOf("const projectPriorities")).toBeLessThan( - triage.indexOf("const candidates"), - ); - expect(triage).not.toContain("priority: high"); - }); - - it("describes the current morning-triage data sources", () => { - const morning = read(".agents/skills/nemoclaw-maintainer-morning/SKILL.md"); - - expect(morning).not.toContain("gh-pr-merge-now --json"); - expect(morning).toContain("fetches open PRs through `gh`"); - expect(morning).toContain("reads Project 199 Priority"); - expect(morning).toContain("review, CI, file, and risky-area data"); - }); - - it("moves post-tag stragglers and retires the released label", () => { - const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); - const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); - const morning = read(".agents/skills/nemoclaw-maintainer-morning/SKILL.md"); - const priorities = read(".agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md"); - const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); - - expect(evening).toContain("automatically carry stragglers to the next patch"); - expect(evening).toContain("retire the released label"); - expect(release).toContain("release-latest-tag"); - expect(release).toContain("signed annotated semver tag"); - expect(release).toContain("GitHub-Verified"); - expect(release).toContain("same tag object"); - expect(release).toContain("--preflight-only"); - expect(release).toContain("OpenPGP, SSH, or X.509 signer"); - expect(release).toContain("Do not run the retirement script directly"); - expect(release).toContain('--event push --commit "$RELEASE_SHA"'); - expect(release).toContain("Expected exactly one release-latest-tag push run"); - expect(morning).toContain("post-tag housekeeping was interrupted"); - expect(priorities).toContain("Move open items to the next patch label"); - expect(priorities).toContain("delete the released label"); - expect(policy).toContain("automatically move every open straggler to the next patch label"); - expect(policy).toContain("delete the released version label"); - expect(policy).toContain("never renamed or reused"); - expect(policy).toContain("shared release-label coordination queue"); - expect(fs.existsSync(path.join(root, "scripts/retire-release-label.mts"))).toBe(true); - }); - - it("keeps release labels temporary and limits post-merge assignment to untagged work", () => { - const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); - const projectWorkflow = read( - ".agents/skills/nemoclaw-maintainer-policies/references/project-workflow.md", - ); - const taxonomy = JSON.parse( - read(".agents/skills/nemoclaw-maintainer-policies/references/label-taxonomy.json"), - ) as { - label_families: { - release: { application_policy: string; positive_signals: string[] }; - }; - quality_rules: { post_merge_untagged_release_labeling_allowed: boolean }; - }; - - expect(policy).toContain("After a PR merges to `main`"); - expect(policy).toContain("ahead of the latest release tag"); - expect(policy).toContain("only across the untagged interval"); - expect(policy).toContain("Tags and commit ancestry are the only durable"); - expect(policy).not.toContain("earliest containing release"); - expect(policy).not.toContain("seven-day retention window"); - expect(projectWorkflow).toContain("On open PRs"); - expect(projectWorkflow).toContain("After a PR merges to `main`"); - expect(projectWorkflow).toContain("tag comparison range owns durable release membership"); - expect(taxonomy.label_families.release.positive_signals).toContain( - "authorized post-merge assignment to the next untagged patch release", - ); - expect(taxonomy.label_families.release.application_policy).toContain( - "carry open items forward and delete the released label", - ); - expect(taxonomy.quality_rules.post_merge_untagged_release_labeling_allowed).toBe(true); - }); - - it("requires E2E evidence for the release candidate commit or itemized maintainer exceptions", () => { - const dailyFlow = read(".agents/skills/nemoclaw-maintainer-policies/references/daily-flow.md"); - const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); - const priorities = read(".agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md"); - const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); - const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); - - expect(policy).toContain("full `origin/main` commit SHA"); - expect(policy).toContain("`.github/workflows/e2e.yaml` is the sole source of truth"); - expect(policy).toContain("Do not maintain a separate release-gating test list"); - expect(policy).toContain("at least one completed, successful execution"); - expect(policy).toContain("Successful evidence may accumulate across rerun attempts"); - expect(policy).toContain("Evidence from another workflow run does not satisfy the ledger"); - expect(policy).toContain("Require every declared `RELEASE_E2E_ACTIVATION_PATH`"); - expect(policy).toContain("A missing path is a preflight failure"); - expect(release).toContain("Each job that declares `RELEASE_E2E_ACTIVATION_PATH`"); - expect(release).toContain("A missing activation path is a preflight failure"); - expect(policy).toContain("each expanded matrix execution as a separate ledger entry"); - expect(policy).toContain("matrix `id`"); - expect(policy).toContain("A later failure does not erase an earlier successful execution"); - expect(policy).toContain( - "Skipped, unexecuted, queued, in-progress, cancelled, and failing results do not count as successful evidence", - ); - expect(policy).toContain("itemized maintainer exception"); - expect(policy).toContain("If the candidate SHA changes"); - expect(policy).toContain("This does not freeze `main` or prevent merges"); - expect(policy).toContain("Require one completed, successful full workflow run"); - expect(policy).toContain("discard the ledger and its exceptions"); - expect(policy).toContain("selector inputs"); - expect(release).toContain('"dispatchJson"'); - expect(release).toContain("the number of tests with successful evidence"); - expect(release).toContain("successful run or job URL and attempt"); - expect(release).toContain("npm run release:e2e-evidence"); - expect(release).toContain("filter=all"); - expect(release).toContain("actions/runs/$RUN_ID/artifacts"); - expect(release).toContain("sort_by(.created_at)"); - expect(release).not.toContain("RECEIPT_ATTEMPT"); - expect(release).toContain("rerun preflight and the full E2E workflow"); - expect(release).toContain("Immediately before asking, refresh `origin/main` once"); - const evidenceSummary = release.indexOf("Before showing the confirmation prompt"); - const confirmationPrompt = release.indexOf( - "Ask the maintainer to paste this phrase", - evidenceSummary, - ); - expect(evidenceSummary).toBeGreaterThanOrEqual(0); - expect(evidenceSummary).toBeLessThan(confirmationPrompt); - expect(evening).toContain( - "Each missing or skipped execution in that successful run requires its own itemized maintainer exception", - ); - expect(evening).toContain( - "Missing or invalid Launchable E2E evidence in that successful run requires a separate", - ); - expect(evening).toContain("Tag the confirmed release commit with `vX.Y.Z`"); - expect(evening).not.toContain("tag `main`"); - expect(dailyFlow).toContain("capture the candidate SHA and review every E2E test"); - expect(dailyFlow).toContain( - "`head_sha` and all associated evidence to match the candidate SHA", - ); - expect(dailyFlow).toContain("invalidate the prior run and evidence"); - expect(priorities).toContain("Record the release SHA and required E2E evidence"); - }); - - it("requires full-mode exact Brev Launchable evidence before release confirmation (#7487)", () => { - const e2e = read(".agents/skills/nemoclaw-maintainer-e2e/SKILL.md"); - const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); - const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); - const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); - const skillsGuide = read(".agents/skills/nemoclaw-skills-guide/SKILL.md"); - - expect(e2e).toContain("include_staging_brev_launchable=true"); - expect(e2e).toContain("Exact staging Brev Launchable"); - expect(e2e).toContain("launchable-e2e.json"); - expect(e2e).toContain("cleanup.json"); - expect(e2e).toContain("dispatch.json"); - expect(e2e).toContain("If the release candidate SHA changes"); - expect(e2e).toContain("jobs?filter=all&per_page=100"); - expect(e2e).toContain("Reuse `run-$RUN_ID.json` and `jobs-$RUN_ID.json`"); - expect(release).toContain("reuse `run-$RUN_ID.json` and `jobs-$RUN_ID.json`"); - expect(release).toContain("load `nemoclaw-maintainer-e2e` and dispatch one full run"); - expect(release).toContain("Treat a skipped job as missing evidence"); - expect(release).toContain("include_staging_brev_launchable=true"); - expect(release).toContain("cleanup evidence that reports the qualified workspace as `ABSENT`"); - expect(release).toContain( - "a separate itemized maintainer exception for each missing or skipped execution", - ); - expect(release).toContain( - "a separate itemized maintainer exception for missing or invalid exact Brev Launchable E2E evidence", - ); - expect(release).toContain("when accepted full-mode exact Brev evidence exists"); - expect( - release.indexOf("load `nemoclaw-maintainer-e2e` and dispatch one full run"), - ).toBeLessThan(release.indexOf("Ask the maintainer to paste this phrase")); - expect(evening).toContain("load `nemoclaw-maintainer-e2e`"); - expect(evening).toContain( - "Run full mode unless one existing full run for the candidate SHA contains complete workflow E2E", - ); - expect(release).toContain( - "Run full mode unless one existing full run for the candidate SHA contains complete workflow E2E", - ); - expect(policy).toContain("A failed workflow run cannot supply the release ledger"); - expect(release).toContain("Reject a failed workflow run before presenting the ledger"); - expect(evening).not.toContain("readiness variable"); - expect(policy).toContain("Require one completed, successful full workflow run"); - expect(policy).toContain( - "Run `nemoclaw-maintainer-e2e` in full mode when the ledger lacks complete evidence", - ); - expect(policy).toContain("including `Exact staging Brev Launchable`"); - expect(policy).toContain("cleanup receipt"); - expect(policy).toContain("trusted dispatch receipt"); - expect(policy).toContain( - "Each missing or skipped execution in the accepted successful workflow run", - ); - expect(policy).toContain( - "Missing or invalid exact Brev Launchable E2E evidence in the accepted successful workflow run", - ); - expect(policy).toContain("No release-note-only delta exception is currently defined"); - expect(skillsGuide).toContain("`nemoclaw-maintainer-e2e`"); - }); - - it("runs release-prep docs before generating the final release plan", () => { - const updateDocs = read(".agents/skills/nemoclaw-contributor-update-docs/SKILL.md"); - const createPr = read(".agents/skills/nemoclaw-contributor-create-pr/SKILL.md"); - const evening = read(".agents/skills/nemoclaw-maintainer-evening/SKILL.md"); - const release = read(".agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md"); - const releaseNotes = read(".agents/skills/nemoclaw-maintainer-release-notes/SKILL.md"); - const policy = read(".agents/skills/nemoclaw-maintainer-policies/references/release-train.md"); - const priorities = read(".agents/skills/nemoclaw-maintainer-day/PR-REVIEW-PRIORITIES.md"); - const skillsGuide = read(".agents/skills/nemoclaw-skills-guide/SKILL.md"); - const agents = read("AGENTS.md"); - const docsAgents = read("docs/AGENTS.md"); - const docsContributing = read("docs/CONTRIBUTING.md"); - - expect(updateDocs).toContain("/nemoclaw-contributor-update-docs for vX.Y.Z"); - expect(updateDocs).toContain("Every pre-tag release-note docs PR must add"); - expect(updateDocs).toContain("docs/changelog/YYYY-MM-DD.mdx"); - expect(updateDocs).toContain("current documentation contributor guide"); - expect(updateDocs).toContain("current repository policy"); - expect(updateDocs).toContain("../nemoclaw-maintainer-policies/references/release-train.md"); - expect(updateDocs).not.toContain("parser-safe MDX SPDX comment"); - expect(updateDocs).not.toContain("scan `..origin/main`"); - expect(updateDocs).toContain("planned release date"); - expect(updateDocs).toContain("Stop before PR creation"); - expect(createPr).not.toContain('--label "area: docs"'); - expect(createPr).toContain( - "Leave label selection and application to the repository triage workflow", - ); - expect(evening.indexOf("/nemoclaw-contributor-update-docs for ")).toBeLessThan( - evening.indexOf("Load `cut-release-tag`"), - ); - expect(evening).toContain("contains the exact `## ` heading"); - expect(release).toContain("git grep -n '^## vX\\.Y\\.Z$'"); - expect(release).toContain("Unless Step 1 records an explicit waiver"); - expect(release).toContain("show the recorded waiver reason"); - expect(release).toContain("A conventional Release Notes page or post-tag Announcement draft"); - expect(releaseNotes).toContain("does not replace or create that canonical entry"); - expect(policy).toContain("Run `/nemoclaw-contributor-update-docs for vX.Y.Z`"); - expect(policy).toContain("The pre-tag release-note docs PR must create or update"); - expect(priorities).toContain("the pre-tag changelog PR contains"); - expect(skillsGuide).toContain( - "update their owning documentation under current repository policy", - ); - expect(agents).toContain("a PR that updates ordinary pages without the dated changelog entry"); - expect(docsAgents).toContain("CONTRIBUTING.md#updating-the-changelog"); - expect(docsAgents).not.toContain("Every pre-tag release-note docs PR must create or update"); - expect(docsContributing).toContain("Create the planned release entry in the pre-tag"); - expect(policy).toContain("If any merge lands after `release:plan`, generate a fresh plan"); - expect(releaseNotes).toContain( - "Keep the candidate SHA, E2E failure classifications, rerun ledger, and waiver rationale out of the public Announcement", - ); - expect(releaseNotes).toContain( - "Never include the candidate SHA, internal E2E failure classifications, rerun details, or waiver rationale in the public Announcement", - ); - }); - - it("keeps documentation authority links one-way", () => { - const agents = read("AGENTS.md"); - const docsAgents = read("docs/AGENTS.md"); - const docsContributing = read("docs/CONTRIBUTING.md"); - const doriSetup = read("docs/DORI_SETUP.md"); - const writing = read("WRITING.md"); - const controlledWords = read(".agents/skills/_shared/controlled-words.md"); - - expect(agents).toContain("[Documentation Agent Guide](docs/AGENTS.md)"); - expect(docsAgents).toContain("[documentation contributor guide](CONTRIBUTING.md)"); - expect(docsAgents).not.toContain("../AGENTS.md"); - expect(docsContributing).not.toContain("../AGENTS.md"); - expect(docsContributing).not.toContain("../CONTRIBUTING.md"); - expect(doriSetup).toContain("[Style Guide](CONTRIBUTING.md#style-guide)"); - expect(doriSetup).not.toContain("(AGENTS.md"); - expect(writing).toContain(".agents/skills/_shared/controlled-words.md"); - expect(controlledWords).not.toContain("WRITING.md"); - }); - - it("keeps cross-issue sweeping separate from comparator scoring", () => { - const sweep = read(".agents/skills/nemoclaw-maintainer-cross-issue-sweep/SKILL.md"); - const comparator = read(".agents/skills/nemoclaw-maintainer-pr-comparator/SKILL.md"); - - expect(sweep).toContain("The comparator does not run this skill or use its findings"); - expect(comparator).toContain("Run `nemoclaw-maintainer-cross-issue-sweep` separately"); - }); - - it("uses the merge gate's unresolved-issue threshold for ready-now PRs", () => { - const day = read(".agents/skills/nemoclaw-maintainer-day/SKILL.md"); - const mergeGate = read(".agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md"); - const threshold = "no unresolved correctness or security issue"; - - expect(day).toContain(threshold); - expect(mergeGate).toContain(threshold); - expect(day).not.toContain("no confirmed major CodeRabbit or PR Review Advisor issues"); - expect(mergeGate).not.toContain("no confirmed major CodeRabbit or PR Review Advisor issues"); - }); - - it("uses native bug type and approved Project writes for stale verification", () => { - const stale = readMarkdownTree(".agents/skills/nemoclaw-maintainer-verify-stale"); - - expect(stale).toContain('select(.issueType.name == "Bug")'); - expect(stale).toContain("Verdict names are comment and log vocabulary, not GitHub labels"); - expect(stale).toContain("Project Status `Won't Fix`"); - expect(stale).not.toMatch(/gh issue edit[^\n]*--add-label/u); - expect(stale).not.toContain("--label bug"); - }); - - it("makes DCO and GitHub verification explicit approval gates", () => { - const mergeGate = read(".agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md"); - const comparator = read( - ".agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh", - ); - - expect(mergeGate).toContain("Require every commit to appear as `Verified` in GitHub"); - expect(comparator).toContain("gate_contributor_compliance"); - expect(comparator).toContain(".commit.verification.verified"); - }); - - it("gives distinct remediation for PR-body and commit-verification failures", () => { - const verdict = read(".agents/skills/nemoclaw-maintainer-pr-comparator/templates/verdict.md"); - - expect(verdict).toContain("Missing PR-body DCO declaration: update the PR body"); - expect(verdict).toContain( - "Missing GitHub Verified commit history: replace the branch with compliant history", - ); - expect(verdict).not.toContain( - "PR-body DCO declaration or GitHub Verified commit history is missing", - ); - }); - - it("requires replacement PRs to preserve transferred contributor attribution", () => { - const policy = read( - ".agents/skills/nemoclaw-maintainer-policies/references/workflow-policy.md", - ); - const comparator = read(".agents/skills/nemoclaw-maintainer-pr-comparator/SKILL.md"); - const tiebreakers = read(".agents/skills/nemoclaw-maintainer-pr-comparator/tiebreakers.md"); - const verdict = read(".agents/skills/nemoclaw-maintainer-pr-comparator/templates/verdict.md"); - const finder = read(".agents/skills/nemoclaw-maintainer-find-review-pr/SKILL.md"); - const parser = read( - ".agents/skills/nemoclaw-maintainer-pr-comparator/scripts/parse-supersession.sh", - ); - - expect(policy).toContain("Supersedes #"); - expect(policy).toContain("Preserve the source contributor as the Git author"); - expect(policy).toContain("Co-authored-by: Name "); - expect(policy).toContain("Use the exact author name and email from the source commit"); - expect(policy).toContain("Never guess or substitute an attribution identity"); - expect(policy).toContain("Never add or copy a DCO declaration"); - expect(policy).toContain("leave the winner unset and ask the contributor"); - const sourceDcoPolicyIndex = policy.indexOf("Confirm that the source PR already contains"); - const transferPolicyIndex = policy.indexOf("After both checks pass"); - expect(sourceDcoPolicyIndex).toBeGreaterThanOrEqual(0); - expect(transferPolicyIndex).toBeGreaterThan(sourceDcoPolicyIndex); - expect(policy).toContain("does not require co-authorship"); - expect(policy).toContain("does not replace attribution in the merged PR history"); - - expect(comparator).toContain("../nemoclaw-maintainer-policies/references/workflow-policy.md"); - expect(comparator).toContain("They do not rank a candidate"); - expect(comparator).toContain("`transferred`"); - expect(comparator).toContain("`unclear`"); - expect(comparator).toContain("leave `winner` null"); - expect(finder).toContain("../nemoclaw-maintainer-pr-comparator/scripts/parse-supersession.sh"); - for (const pattern of [ - "supersed[a-z]*", - "replac[a-z]*", - "clos[a-z]* in favor of", - "fold[a-z]* in", - ]) { - expect(parser).toContain(pattern); - expect(comparator).toContain(pattern); - expect(finder).toContain(pattern); - } - for (const example of [ - "superseded by #N", - "replaced by #N", - "closed in favor of #N", - "folded into #N", - ]) { - expect(comparator).toContain(example); - expect(finder).toContain(example); - } - expect(comparator).toContain("A `follow-up to #N` statement is a related-PR signal"); - expect(finder).toContain("A `follow-up to #N` statement is a related-PR signal"); - - expect(tiebreakers).toContain("it does not rank a candidate"); - expect(tiebreakers).not.toContain("**Supersession.**"); - expect(tiebreakers).toContain("rerun the comparator before selecting a winner"); - - expect(verdict).toContain("git cherry-pick -S -x "); - expect(verdict).toContain("Co-authored-by: Name "); - expect(verdict).toContain("using the verified source-commit identity"); - expect(verdict).toContain("run the comparator again on the updated SHA"); - expect(verdict).toContain("contains the contributor's `Signed-off-by:` declaration"); - expect(verdict).toContain("Do not add or copy that declaration"); - expect(verdict).toContain("Keep the replacement author's own DCO declaration"); - expect(verdict).toContain("every replacement commit appears as `Verified` in GitHub"); - - const sourceDcoIndex = verdict.indexOf("Confirm that PR #B contains the contributor's"); - const identityIndex = verdict.indexOf( - "Read the exact author name and email from the source commit", - ); - const transferIndex = verdict.indexOf("Transfer the test from PR #B before merge"); - const rerunIndex = verdict.indexOf("run the comparator again on the updated SHA"); - const mergeIndex = verdict.indexOf("Merge PR #A only if the new verdict selects it"); - const closeIndex = verdict.indexOf("After PR #A merges, close PR #B"); - - expect(sourceDcoIndex).toBeGreaterThanOrEqual(0); - expect(identityIndex).toBeGreaterThanOrEqual(0); - expect(transferIndex).toBeGreaterThan(sourceDcoIndex); - expect(transferIndex).toBeGreaterThan(identityIndex); - expect(rerunIndex).toBeGreaterThan(transferIndex); - expect(mergeIndex).toBeGreaterThan(rerunIndex); - expect(closeIndex).toBeGreaterThan(mergeIndex); - - expect(finder).toContain("../nemoclaw-maintainer-policies/references/workflow-policy.md"); - expect(finder).toContain("This skill reports recommendations only"); - expect(finder).toContain( - "Do not recommend closing the source PR until another authorized workflow", - ); - expect(finder).toContain("merged the selected target"); - expect(finder).toContain("After the updated verdict selects #1416 and #1416 merges"); - }); - - it("orients active and passive supersession statements", () => { - const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "parse-supersession-")); - const bin = path.join(tmp, "bin"); - const mockGh = path.join(bin, "gh"); - fs.mkdirSync(bin); - fs.writeFileSync( - mockGh, - [ - "#!/usr/bin/env bash", - 'case "$3" in', - ' 100) printf "%s" "${PR_BODY_100:-}" ;;', - ' 200) printf "%s" "${PR_BODY_200:-}" ;;', - "esac", - ].join("\n"), - ); - fs.chmodSync(mockGh, 0o755); - - const parser = path.join( - root, - ".agents/skills/nemoclaw-maintainer-pr-comparator/scripts/parse-supersession.sh", - ); - const scenarios = [ - { statement: "Supersedes #200", superseder: 100, superseded: 200 }, - { statement: "Superseded by #200", superseder: 200, superseded: 100 }, - { statement: "Replaces #200", superseder: 100, superseded: 200 }, - { statement: "Replaced by #200", superseder: 200, superseded: 100 }, - { statement: "Closes in favor of #200", superseder: 200, superseded: 100 }, - { statement: "Closed in favor of #200", superseder: 200, superseded: 100 }, - { statement: "Folds in #200", superseder: 100, superseded: 200 }, - { statement: "Folded into #200", superseder: 200, superseded: 100 }, - { - statement: "Supersedes #200\nReplaces #200", - superseder: 100, - superseded: 200, - }, - ]; - - try { - for (const scenario of scenarios) { - const result = spawnSync("bash", [parser, "100", "200"], { - encoding: "utf8", - env: { - ...process.env, - PATH: `${bin}${path.delimiter}${process.env.PATH ?? ""}`, - PR_BODY_100: scenario.statement, - PR_BODY_200: "", - }, - }); - - expect(result.status).toBe(0); - expect(JSON.parse(result.stdout)).toEqual({ - edges: [ - { - superseder: scenario.superseder, - superseded: scenario.superseded, - }, - ], - }); - } - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }); - - it("keeps PR workflow writes behind their safety checks", () => { - const createPr = read(".agents/skills/nemoclaw-contributor-create-pr/SKILL.md"); - const judgment = read( - ".agents/skills/nemoclaw-maintainer-cross-issue-sweep/checks/relationship-judgment.md", - ); - const mergeGate = read(".agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md"); - const salvage = read(".agents/skills/nemoclaw-maintainer-day/SALVAGE-PR.md"); - - expect(createPr).toContain("For work that is not ready for review, complete Step 4"); - expect(createPr).toContain("--body-file /tmp/nemoclaw-pr-body.md"); - expect(createPr).not.toContain('--body "..."'); - expect(judgment).toContain("{candidate_comments}"); - expect(mergeGate).toContain( - "The trusted pre-checkout step requires current `maintain` or `admin` access and validates the exact open PR before candidate code runs.", - ); - expect(mergeGate).toContain( - "Leave job and target selectors empty and keep Launchable disabled.", - ); - expect(mergeGate).toContain("The manual run is advisory."); - expect(salvage).toContain("`headRepository.nameWithOwner` is `NVIDIA/NemoClaw`"); - expect(salvage).toContain("git push origin :"); - expect(salvage).toContain("If `maintainerCanModify` is false, do not push"); - }); - - it("keeps maintainer ordering, state, and write authorization explicit", () => { - const sequence = read(".agents/skills/nemoclaw-maintainer-day/SEQUENCE-WORK.md"); - const state = read(".agents/skills/nemoclaw-maintainer-day/STATE-SCHEMA.md"); - const instructions = read( - ".agents/skills/nemoclaw-maintainer-policies/references/triage-instructions.md", - ); - const triage = read(".agents/skills/nemoclaw-maintainer-triage/SKILL.md"); - - expect(sequence).toContain("An identified security concern overrides this default order"); - expect(state).toContain("Keep at most 50 entries"); - expect(instructions).toContain( - "keep `labels_to_add` and `labels_to_remove` as dry-run output and do not change labels", - ); - expect(instructions).toContain( - "An authorized agent-owned workflow may add or remove only `agt: *` labels", - ); - expect(triage).toContain("Before each write, re-read Issue Type, Project fields, and labels"); - expect(triage).toContain("present an updated proposal for acceptance"); - }); - - it("requires PR guidance to collect complete review evidence", () => { - const followUp = read(".agents/skills/_shared/pr-follow-up.md"); - - expect(followUp).toContain("Bind every read to `NVIDIA/NemoClaw` and one PR number"); - expect(followUp).toContain("Initial and final PR `headRefOid`"); - expect(followUp).toContain("Local candidate `HEAD`"); - expect(followUp).toContain("Page counts and terminal pagination status"); - expect(followUp).toContain("Every required check and the commit it evaluates"); - expect(followUp).toContain("Report the collection as `blocked`"); - expect(followUp).toContain( - "remove that exact artifact after classification, and verify its absence", - ); - }); - - it("requires PR guidance to group findings and model sensitive failures", () => { - const followUp = read(".agents/skills/_shared/pr-follow-up.md"); - const createPr = read(".agents/skills/nemoclaw-contributor-create-pr/SKILL.md"); - - expect(followUp).toContain("Collect One Complete Review Cycle"); - expect(followUp).toContain("Group findings by root cause"); - expect(followUp).toContain("Do not create a separate commit or push for each finding"); - expect(followUp).toContain("Sensitive-Workflow State Matrix"); - expect(followUp).toContain("location, access, lifetime, and removal"); - - expect(followUp).toContain("Bind every read to `NVIDIA/NemoClaw`"); - expect(followUp).toContain("Record each page count and terminal pagination signal"); - expect(followUp).toContain("including pending, cancelled, and skipped results"); - expect(followUp).toContain("retained evidence: none"); - expect(followUp).toContain("Assume a possible write and re-read external state"); - expect(followUp).toContain("stop without further edits, commits, or pushes"); - expect(createPr).toContain("Apply one coherent change set"); - }); - - it("requires PR guidance to review the final commit before push", () => { - const followUp = read(".agents/skills/_shared/pr-follow-up.md"); - const writingReview = read(".agents/skills/_shared/documentation-writing-review.md"); - const createPr = read(".agents/skills/nemoclaw-contributor-create-pr/SKILL.md"); - - expect(createPr).toContain( - "Push after the independent documentation writer review covers the final `HEAD`", - ); - expect(createPr).toContain("rerun the review against the new `HEAD`"); - expect(createPr).toContain("receipt identifies that commit"); - - expect(followUp).toContain("Run a final complete, head-stable collection"); - - expect(followUp).toContain("If the collection contains a new actionable finding, do not push"); - expect(followUp).toContain("Deferral does not authorize a push with an unresolved blocking"); - expect(createPr).toContain("The user may defer only a non-blocking suggestion"); - expect(followUp).toContain("Push once when the receipt identifies the reviewed `HEAD`"); - expect(writingReview).toContain("Do not stop after the first blocking finding"); - expect(writingReview).toContain("Report all evidence-backed findings in one review result"); - expect(writingReview).toContain("A blocker does not end the review pass"); - }); - - it("resolves security-review issue inputs to one verified PR", () => { - const securityReview = read(".agents/skills/nemoclaw-maintainer-security-code-review/SKILL.md"); - - expect(securityReview).toContain("--json closedByPullRequestsReferences"); - expect(securityReview).toContain("Continue only when this returns one PR number"); - expect(securityReview).toContain("Use the verified PR number in each later command"); - expect(securityReview).toContain("If no changed or reviewable security surface exists"); - expect(securityReview).toContain( - "Dockerfiles, workflows, network policies, blueprints, dependencies, and security configuration", - ); - }); -});