Onboard build failure analysis workflow

[YuliiaKovalova]

Summary

• add automatic analysis for failed Azure Pipelines msbuild-pr checks and an /analyze-build-failure maintainer command
• select only build-log artifacts for failed or canceled jobs, while preserving Source Build retries and analyzing every binlog in each artifact
• diagnose bootstrap/stage-1 and stage-2 failures through a digest-pinned binlog-mcp container, with stale-revision checks and PR-bound safe outputs
• enforce compressed, expanded, and cumulative archive limits for untrusted PR artifacts

The implementation is based on the build-failure analysis workflow from dotnet/arcade, adapted to MSBuild's pipeline, artifact names, and repository conventions.

Example

The example of the output given by the workflow from the repos that already use it:
dotnet/arcade#17348 (comment)
microsoft/testfx#10637 (comment)

[github-actions]

🔍 Skill Validator Results

✅ All checks passed

Scope	Checked
Skills	0
Agents	1
Total	1
Severity	Count
---	---:
❌ Errors	0
⚠️ Warnings	0
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	Found 1 agent(s)
ℹ️	Validated 1 agent(s)
ℹ️	✅ All checks passed (1 agent(s))

Full validator output

```text Found 1 agent(s) Validated 1 agent(s) ✅ All checks passed (1 agent(s)) ```

[Copilot]

Pull request overview

Adds automated and maintainer-triggered analysis of failed Azure Pipelines msbuild-pr builds using binlogs and a digest-pinned binlog-mcp container.

Changes:

• Adds automatic and /analyze-build-failure command workflows.
• Selects relevant artifacts with revision and archive safeguards.
• Adds shared guidance, analyst playbooks, generated locks, and pinned dependencies.

The workflow extraction paths must reject symlink and hardlink archive entries before extraction to prevent writes outside the extraction directory.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Summary	Review status
.github/workflows/shared/build-failure-analysis-shared.md	Shared analysis instructions	No final comments
.github/workflows/build-failure-analysis.agent.md	Automatic failure-analysis workflow	Critical archive-link extraction issue
.github/workflows/build-failure-analysis.agent.lock.yml	Generated automatic workflow lock	No final comments
.github/workflows/build-failure-analysis-command.agent.md	Maintainer command workflow	Critical archive-link extraction issue
.github/aw/actions-lock.json	Pinned actions and container dependencies	No final comments
.github/agents/build-failure-analyst.agent.md	Detailed analysis playbook	No final comments

Suppressed comments (10)

.github/agents/build-failure-analyst.agent.md:50

• This fallback is unreachable in both callers: each workflow's top-level if requires needs.fetch-binlog.outputs.binlog-found == 'true', while this path is specifically for GH_AW_BINLOG_LIST being empty. Consequently a failed build with no usable binlog is silently skipped rather than receiving the documented diagnostic. Either remove this dead path and its schema, or allow the agent to run for the false fetch result.

3. If the value is `failure` but `GH_AW_BINLOG_LIST` is empty, post a single comment via `add_comment` with the body:

   > 🔍 **Build Failure Analysis** — the build failed but no binary log was produced. See the originating [Azure DevOps build](${GH_AW_BINLOG_HOST_PATH}) for the authoritative build logs (this workflow reuses that build's binlogs and does not build locally). The [GitHub Actions run](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}) has the fetch-step diagnostics.

   Attach the structured data object
   `{"workflow_artifact":"build-failure-analysis","artifact_kind":"no-binlog"}`

.github/agents/build-failure-analyst.agent.md:115

• This recheck repeats the same gap: it rejects only differing non-empty merge SHAs, so a PR can become conflicted or otherwise lose merge_commit_sha while analysis runs and still receive suggestions from the old merge. Require both the expected and current merge SHA to be present and equal before the first safe-output call.

When there is a build failure, first re-verify the target revision: read PR `GH_AW_PR_NUMBER` with the GitHub `pull_requests` read tool exposed by the github MCP server (the pull-request "get"/read operation) and take `head.sha` and `merge_commit_sha`. If `head.sha` cannot be read or no longer equals `GH_AW_PR_HEAD_SHA` — or `GH_AW_PR_MERGE_SHA` is non-empty and `merge_commit_sha` is non-empty but differs from it (the base branch advanced) — the PR moved while you were downloading/analyzing, so `noop` with a short reason and stop: your inline suggestions carry no `commit_id` and would land on the wrong lines of the new diff/merge. Otherwise post **exactly one** summary comment via `add_comment` with structured data `{"workflow_artifact":"build-failure-analysis","artifact_kind":"analysis"}`. The workflow binds this output to `GH_AW_PR_NUMBER`, and the gh-aw `add-comment` config has `hide-older-comments: true`, which collapses prior runs from the same workflow.

.github/workflows/build-failure-analysis-command.agent.md:353

• Because this condition requires both values to be non-empty, a PR whose current merge_commit_sha is unavailable (for example, after a conflict) passes even though the selected build may be based on an obsolete merge. Fail closed when either merge SHA cannot be resolved; matching BUILD_PR_SHA is insufficient to validate the base merge.

          if [ -n "${BUILD_MERGE_SHA}" ] && [ -n "${CURRENT_MERGE}" ] && [ "${BUILD_MERGE_SHA}" != "${CURRENT_MERGE}" ]; then

.github/workflows/build-failure-analysis-command.agent.md:564

• The final command-workflow check has the same gap: if LATEST_MERGE becomes unavailable while artifacts are downloading, the guarded comparison silently skips the stale-merge check. Treat an unavailable merge revision as unresolved and call emit_none so a base update or conflict cannot lead to suggestions from an obsolete binlog.

          if [ -n "${BUILD_MERGE_SHA}" ] && [ -n "${LATEST_MERGE}" ] && [ "${LATEST_MERGE}" != "${BUILD_MERGE_SHA}" ]; then

.github/workflows/build-failure-analysis-command.agent.md:674

• The shared prompt requires exactly one summary, but max: 5 permits up to five add_comment safe-output calls in a single run. Set this to 1 so the workflow enforces the one-comment contract; hide-older-comments does not collapse multiple comments from the current run.

    max: 5

.github/workflows/build-failure-analysis-command.agent.md:61

• The concurrency test uses a raw startsWith, but the generated command-position gate accepts only /analyze-build-failure followed by whitespace/end-of-input. Consequently /analyze-build-failure-now (which the fetch step explicitly rejects) still gets the PR-scoped cancel-in-progress: false group; a write-access user can queue lookalike comments and delay a real analysis. Use the same delimiter-aware command match here so invalid lookalikes receive run-unique groups.

  group: ${{ startsWith(github.event.comment.body, '/analyze-build-failure') && contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association) && format('build-failure-analysis-cmd-{0}', github.event.issue.number) || format('build-failure-analysis-cmd-run-{0}', github.run_id) }}

.github/workflows/build-failure-analysis.agent.md:290

• Because this condition requires both values to be non-empty, a PR whose current merge_commit_sha is unavailable (for example, after a conflict) passes even though the build's merge revision may be stale. BUILD_PR_SHA only detects head changes; fail closed when either merge SHA cannot be resolved so suggestions are never based on an unverified base merge.

          if [ -n "${BUILD_MERGE_SHA}" ] && [ -n "${CURRENT_MERGE}" ] && [ "${BUILD_MERGE_SHA}" != "${CURRENT_MERGE}" ]; then

.github/workflows/build-failure-analysis.agent.md:503

• The same missing-value gap exists in the final check: if LATEST_MERGE becomes unavailable during artifact download, this condition skips the staleness check. Treat an unavailable build/current merge revision as stale or unresolved and call emit_none; otherwise a base update or conflict during download can still produce analysis from an obsolete binlog.

          if [ -n "${BUILD_MERGE_SHA}" ] && [ -n "${LATEST_MERGE}" ] && [ "${LATEST_MERGE}" != "${BUILD_MERGE_SHA}" ]; then

.github/workflows/build-failure-analysis.agent.md:614

• The shared prompt requires exactly one summary, but max: 5 permits up to five add_comment safe-output calls in a single run. Set this to 1 so the workflow enforces the one-comment contract; hide-older-comments does not collapse multiple comments from the current run.

    max: 5

.github/workflows/shared/build-failure-analysis-shared.md:56

• This clean-compile branch can also be reached when a binlog-mcp query failed: the absence of errors/evidence is indistinguishable from an unavailable leg. The analyst playbook explicitly says to post a partial result when an MCP call fails, so track query success and take this noop path only after all required per-leg queries completed successfully.

     shell. If no leg shows errors **and**
     no failed-target/process evidence, the build compiled cleanly — the
     pipeline failure is then a **non-build** (test/packaging/publishing) failure,
     which is **out of scope**. This workflow analyses build failures only, so
     **post nothing**: call `noop` with a short reason and stop. Do **not**

[YuliiaKovalova]

Fork E2E validation

The hardened workflow was rerun successfully against the real Source Build failure from Azure DevOps build 1555236 / source PR #14370:

• Successful workflow run at fork-only workflow commit 167d1a6015721b6e35d8bb31befbe33eeddd8188
• Selected the exact failed Source Build artifact (1 of 7) and securely extracted its single binlog (1/1 selected artifacts)
• Diagnosed the expected IDE0051 failure in Terminal.NormalizeDimension
• Posted exactly one root-cause summary and one revision-pinned inline suggestion; no placeholder output was emitted
• The final Revalidate PR revision before applying queued outputs gate and safe-output processing both passed

The earlier multi-binlog/test-only run also selected all three failed Windows artifacts, extracted all five binlogs, and correctly produced noop because the failures were test-only and outside build-analysis scope.