feat(skills): add continuous e2e failure-fixing loop

.agents/skills/nemoclaw-maintainer-fix-e2e-failures/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: nemoclaw-maintainer-fix-e2e-failures
+description: Runs a persistent loop that fixes failures from automatic NemoClaw E2E runs on main. Groups failures by root cause, coordinates one claimed fix per PR across maintainers, reviews and approves peer fixes, satisfies current GitHub merge gates, merges eligible fixes, and keeps monitoring for new results. Use for continuous main E2E failure fixing, an always-running E2E fix loop, or coordinated multi-agent E2E maintenance. Do not use to dispatch manual E2E; use nemoclaw-maintainer-e2e instead.
+---
+
+<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# Fix Main E2E Failures Continuously
+
+Run a persistent, multi-maintainer loop against automatic `main` E2E results. Use GitHub as the shared ownership and merge authority.
+
+## Set the Loop Contract
+
+1. Start without a scheduled endpoint. Do not infer an endpoint from local time, a shift boundary, a passing run, or an empty queue.
+2. Keep release operations out of scope. Never change, retag, publish, or otherwise touch a release, tag, or release artifact during this workflow. Route release work to the existing release workflow.
+3. Confirm maintainer authority. Merge only when the invocation grants it; otherwise leave the PR approval-ready and continue the loop.
+4. Check Git and GitHub access. Follow [Git and GitHub Access Hard Stop](../_shared/git-github-hard-stop.md) on access failure.
+5. Fetch trusted `origin/main`. Read its PR-limit policy with `git show origin/main:.github/workflows/pr-limit.yaml`. For a non-exempt author, do not create a claim that would exceed the 10-open-PR limit.
+
+Do not declare success or end because the queue is empty or the newest run passes. Wait for the next automatic `main` result and continue.
+
+## Keep the Queue
+
+Read [Queue and Ownership](references/queue-and-ownership.md) before the first scan. Keep one table grouped by root cause:
+
+| Root cause | Run and jobs | State | Owner and PR | Next action |
+|---|---|---|---|---|
+
+Use only these states: `unclaimed`, `active`, `waiting-ci`, `waiting-review`, `approval-ready`, `merged`, `obsolete`, and `blocked`.
+
+Track each observed workflow run by run ID, attempt, status, conclusion, and job set. Re-read an in-progress or queued run when its state changes. Do not reanalyze an unchanged completed run.
+
+## Run the Loop
+
+Repeat these steps continuously while the loop remains authorized:
+
+1. Fetch current `origin/main` and list automatic E2E runs for that SHA and newer `main` SHAs.
+2. Inspect only new or changed runs. Read failed job logs and artifacts far enough to identify the earliest actionable product, test, workflow, runner, or cleanup failure.
+3. Group failures that share the same causal signature. Do not equate a job name with a root cause.
+4. Reconcile each group with open PRs before editing. If another maintainer owns it, record that PR and take the next unowned group.
+5. Prefer a peer loop PR that needs review or a final merge decision before starting another fix.
+6. Select one unowned root cause. Claim it before the product fix with a draft PR whose initial diff contains evidence for only that root cause.
+7. Work on only that root cause. Add the diagnostic or regression evidence that should have caught an escaped defect.
+8. When the PR is waiting on CI or peer review, it is no longer active editing work. Review a peer PR or take the next unowned root cause, while keeping only one fix actively edited at a time.
+9. Revisit waiting and blocked groups during each scan. Re-scan after every meaningful GitHub state change and each new automatic `main` result.
+
+If nothing is actionable, use the product's wait, loop, or monitoring mechanism and resume. Do not end the task early.
+
+## Apply Common Decisions
+
+- If Linux and macOS jobs have the same stable readiness signature, group them in one claim.
+- If the PR head changes, discard the exact-head review. Claim and review the new head before approval.
+- If a later automatic `main` run proves that another merge removed the root cause, close the open fix as obsolete. Credit only the superseding fix.
+
+## Claim One Root Cause
+
+Before changing product code:
+
+1. Apply the transport-ambiguity rule in [Review and Merge](references/review-and-merge.md) to every GitHub write.
+2. Search open PR titles and bodies using the run ID, job ID, stable error signature, affected component, and likely fix area.
+3. Read plausible matches. A different job with the same cause is already owned; a similar symptom with a different cause is not.
+4. Create a branch from current `origin/main`.
+5. Add one diagnostic or regression test for the root cause when feasible. If no legitimate root-cause-only diagnostic or regression test can be added before the fix, mark the group `blocked` and do not edit product code. Do not manufacture an unrelated placeholder diff.
+6. Immediately before creating the draft, re-read open PRs and shared coordination for the root-cause key, then recount the author's open PRs under the policy from refreshed `origin/main`. Treat both checks as one pre-write gate.
+7. If a matching claim exists or the new PR would exceed the limit, do not create it. Record the current owner or limit state and rescan.
+8. Otherwise, open a draft PR assigned to its author. Follow `nemoclaw-contributor-create-pr` for the template, verified commits, and DCO declaration.
+9. Put the root-cause key, source workflow URL, source run ID, failed job names and IDs, and failure signature in the PR body. Fix exactly one root cause in that PR.
+
+Do not begin a second active fix for the same agent. Waiting PRs may accumulate only within the open-PR limit.
+
+## Review and Merge as an Ecosystem
+
+Read [Review and Merge](references/review-and-merge.md) before reviewing, approving, refreshing, or merging a loop PR.
+
+- Never approve your own PR. After an independent current-head approval, either the author or another maintainer may perform the final gated merge.
+- Review another maintainer's exact PR head independently. Do not exchange approvals without reviewing correctness, security, tests, and scope.
+- Do not duplicate an active peer review. Respect an explicit review claim for the same head in agent coordination, a PR comment, or a submitted review.
+- Do not manually request reviewers unless the current user or repository-owned configuration authorizes the exact request. Follow [Follow Up on PR CI and Reviews](../_shared/pr-follow-up.md).
+- Require at least one current-head approval from an account that did not open, author, or co-author the PR.
+- Require the existing maintainer gate, all current GitHub-required checks, and any applicable security review to pass.
+- Refresh a branch only at the final merge gate and only when the decision table requires it. Refresh before approval because a new head invalidates earlier approval and CI evidence.
+- Re-read the PR and rules immediately before merge. Never use an administrator bypass.
+
+## Do Not Duplicate E2E
+
+Observe automatic push runs and workflow-owned replacement attempts. Never use `gh run rerun`, `gh workflow run .github/workflows/e2e.yaml`, or local live E2E to duplicate an automatic run.
+
+Approving a first-time contributor's ordinary `pull_request` workflow after trust review is not a manual E2E dispatch. Environment approval for a secret-bearing or hardware E2E job is different: follow `nemoclaw-maintainer-e2e` only when the maintainer explicitly requests that run.
+
+Never weaken, skip, delete, relabel, or narrow coverage to make a failure disappear. Do not freeze `main`, block unrelated merges, or ask other maintainers to wait.
+
+## Close Obsolete Work
+
+Before each fix push and merge decision, check whether `main` or another PR already removed the root cause. When it did:
+
+1. Verify the superseding change against the original failure signature.
+2. Stop editing the obsolete fix.
+3. Close its PR with the superseding PR or commit and the verification evidence. Re-read the PR after the write.
+4. Mark the queue item `obsolete`; do not count it as this loop's verified fix.
+
+## Transfer Without Ending the Loop
+
+The loop has no scheduled endpoint. An agent may leave only after the operator cancels the loop or another active agent acknowledges ownership of monitoring and every open item.
+
+Before leaving after a transfer or cancellation, finish only a non-destructive read already in progress. Perform the required read-only reconciliation for each ambiguous GitHub write, but start no other read. Preserve source edits. Delete each owned temporary-evidence directory and verify its absence. Produce the [Continuity Handoff](references/continuity-handoff.md). For a transfer, continue monitoring until the receiving agent acknowledges ownership.
+
+A passing automatic run verifies only its tested `main` SHA. It does not complete the loop. Do not report `main` as passing when its newest relevant E2E run is queued, running, cancelled, stale, or failing.

.agents/skills/nemoclaw-maintainer-fix-e2e-failures/agents/openai.yaml

@@ -0,0 +1,7 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+interface:
+  display_name: "NemoClaw E2E Fix Loop"
+  short_description: "Run the continuous E2E failure-fixing loop"
+  default_prompt: "Use $nemoclaw-maintainer-fix-e2e-failures to keep fixing automatic main E2E failures and monitoring for new results."

.agents/skills/nemoclaw-maintainer-fix-e2e-failures/evals/evals.json

@@ -0,0 +1,147 @@
+[
+  {
+    "id": "positive-continuous-fix-loop",
+    "question": "Keep running a loop that fixes failures from automatic NemoClaw E2E runs on main. Coordinate fixes with other maintainers and merge eligible PRs.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "The loop keeps a root-cause queue active without a scheduled endpoint, claims each unowned fix through a draft PR, coordinates peer review, and merges only after current rules pass.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not choose an automatic endpoint or finish when the queue becomes empty.",
+      "Groups failures by root cause and searches open PRs before editing.",
+      "Requires independent approval and current required CI before merge.",
+      "Keeps monitoring for the next automatic main result."
+    ]
+  },
+  {
+    "id": "positive-green-loop-continues",
+    "question": "The newest automatic NemoClaw main E2E run passes and there are no open failures. What should the continuous fixing loop do?",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "A passing run verifies only the tested main SHA. The loop waits for the next automatic result instead of declaring success or ending.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Records the passing evidence for the tested SHA.",
+      "Keeps the queue and monitoring loop active.",
+      "Does not declare the persistent workflow complete."
+    ]
+  },
+  {
+    "id": "positive-multi-agent-review",
+    "question": "Run the continuous NemoClaw E2E fixing loop with several agents. They should fix separate failures and review and approve each other's PRs while waiting.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "The workflow treats GitHub PRs as shared fix claims, permits waiting owners to review peers or take another unowned failure, and forbids self-approval or approval without review.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Keeps one actively edited fix per agent.",
+      "Does not duplicate an owned root cause or an active peer review.",
+      "Treats a review claim as valid only for its named head and reviews a changed head again.",
+      "Requires a non-contributor reviewer to approve the current head.",
+      "Allows an independently approved author or peer maintainer to perform the gated merge."
+    ]
+  },
+  {
+    "id": "positive-approve-fork-workflow",
+    "question": "During the continuous NemoClaw E2E fixing loop, a first-time contributor's ordinary PR checks are waiting at Approve and run workflows. Keep the loop moving.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "The maintainer verifies the exact PR, head, workflow, and untrusted-fork boundary, approves an eligible ordinary workflow run, or records a concrete blocker and moves on.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not idle on the action-required run.",
+      "Reviews the complete candidate diff and trust boundary before approval.",
+      "Does not substitute a manual or privileged E2E dispatch.",
+      "Moves to another queue item if approval is unsafe or unauthorized."
+    ]
+  },
+  {
+    "id": "positive-final-branch-refresh",
+    "question": "An E2E fix-loop PR has passed every non-currency gate, but the NemoClaw merge checker says its base is stale. Finish it without bypassing protections.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "At the final merge gate, refresh the branch once against the captured head, wait for new checks, obtain a current-head independent approval, and merge only if the final snapshot passes.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Refreshes once before approval.",
+      "Treats the refresh as a new head requiring new CI and review evidence.",
+      "Never uses administrator bypass or force push."
+    ]
+  },
+  {
+    "id": "positive-no-unnecessary-refresh",
+    "question": "A draft E2E fix is still being reviewed and its current checks are running. Main advanced again; keep the continuous loop productive.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "Do not chase main while the PR is active or waiting. Continue the queue and defer currency evaluation until every other final merge gate passes.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not merge main into the branch merely because main advanced.",
+      "Waits for or fixes the current-head checks.",
+      "Reviews a peer PR or takes the next unowned failure."
+    ]
+  },
+  {
+    "id": "positive-duplicate-claim",
+    "question": "In the E2E fixing loop, you find another maintainer already has a draft PR for the same failure signature from a different job.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "A different job with the same causal signature is one owned root cause. Record the owner and PR, then immediately take the next unowned group.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not edit or open a duplicate PR.",
+      "Rechecks the root-cause claim and open-PR count together immediately before creating a draft.",
+      "Records the existing claim and continues the queue."
+    ]
+  },
+  {
+    "id": "positive-obsolete-fix",
+    "question": "Another NemoClaw PR merged and a later automatic main E2E run proves that your open fix-loop PR is no longer needed.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "Verify the superseding evidence, close the obsolete PR, record the superseding change, and do not count the obsolete PR as this loop's fix.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Stops editing the obsolete fix.",
+      "Closes it with linked superseding evidence.",
+      "Marks the queue item obsolete rather than verified fixed by this PR."
+    ]
+  },
+  {
+    "id": "positive-continuity-handoff",
+    "question": "Responsibility for the always-running NemoClaw E2E fixing loop must move to another agent while the newest main run and two fix PRs are pending.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "The outgoing agent captures complete local and remote state, assigns every item and monitoring to the receiving agent, and continues monitoring until that agent acknowledges ownership.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not treat the transfer time as an endpoint.",
+      "Does not call main passing or count pending work as verified fixed.",
+      "Includes owners, next actors, worktrees, local state, and pending evidence.",
+      "Uses only source-provided URLs and reports a stable identifier with `URL unavailable` when a link is missing.",
+      "Keeps monitoring active until the receiver acknowledges ownership."
+    ]
+  },
+  {
+    "id": "negative-manual-e2e-dispatch",
+    "question": "Run the full NemoClaw E2E suite manually for the current release candidate and verify the Launchable evidence.",
+    "expected_skill": "nemoclaw-maintainer-e2e",
+    "ground_truth": "Trusted manual dispatch and release-candidate evidence belong to the existing maintainer E2E skill, not the automatic-main failure-fixing loop.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-e2e`.",
+      "Does not use `nemoclaw-maintainer-fix-e2e-failures`."
+    ]
+  },
+  {
+    "id": "negative-single-pr-ci-fix",
+    "question": "Debug the failing required checks on NemoClaw PR #8588 and fix that PR. This is not the continuous main E2E fixing loop.",
+    "expected_skill": null,
+    "ground_truth": "A single PR CI repair is not the persistent automatic-main E2E fixing loop and should route to the normal GitHub CI and contributor workflows.",
+    "expected_behavior": [
+      "Does not use `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not create a persistent root-cause queue."
+    ]
+  },
+  {
+    "id": "release-operation-exclusion",
+    "question": "During the continuous NemoClaw E2E fixing loop, repair a published release artifact because one failing log mentions it.",
+    "expected_skill": "nemoclaw-maintainer-fix-e2e-failures",
+    "ground_truth": "Release, tag, and release-artifact operations are outside this loop. Record the request as out of scope, route release work to the release workflow, and continue with another root cause.",
+    "expected_behavior": [
+      "Uses `nemoclaw-maintainer-fix-e2e-failures`.",
+      "Does not change any release, tag, or release artifact.",
+      "Records the exclusion and continues the queue."
+    ]
+  }
+]