Skip to content

ci(governance): rearm the OpenAPI workflow's route/SDK exit integrity [Tier 4] - #9719

Merged
scarmani merged 1 commit into
mainfrom
structex/cdg-openapi-rearm
Aug 6, 2026
Merged

ci(governance): rearm the OpenAPI workflow's route/SDK exit integrity [Tier 4]#9719
scarmani merged 1 commit into
mainfrom
structex/cdg-openapi-rearm

Conversation

@scarmani

@scarmani scarmani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

VAL-CDG-012: rearm the OpenAPI workflow's route/SDK exit integrity [Tier 4]

Milestone cdg-bootstrap-route-truth, feature cdg-openapi-rearm-impl. Sequenced after merged #9672 (fail-closed aggregators, base of this branch) and merged #9717 (method-aware route-core plane, 9148ba2934). Prepared under the operator ledger line-388 narrow overlap lift: this PR's openapi.yml hunks are disjoint from open #9690's single setup-node hunk (L169–176) and do not touch the Set up Node.js step.

Workflow rearm (.github/workflows/openapi.yml)

  1. Method-aware route authority bound to the exact execution SHA. Validate handler route coverage now first runs scripts/validate_openapi_routes.py --ref "$(git rev-parse HEAD)" (the VAL-CDG-011 operation plane, exact-40-hex enforced by _require_exact_ref) and refuses to continue unless the plane echoes the identical SHA back; the baseline-gated --fail-on-missing coverage check then runs unchanged as the drift ratchet. Verified locally: plane binds and echoes 622fc3df8f…, 652 served operations, step exit 0; a failing authority propagates through the armed pipefail.
  2. Contract-matrix pytest is now authoritative. The || true that discarded its exit code is removed (live main runs showed 1 passed under the guard — the guard was masking, not carrying, the verdict).
  3. operationId audit pipe armed. set -o pipefail before the producer|consumer pipe: the count stays advisory, but a crashed producer now fails the step.
  4. Summaries cannot present stale verdicts. Both if: always() drift-summary steps require parseable current-run inputs (/tmp/route-coverage.json + generated spec; then /tmp/contract-drift-summary.json) before invoking their generators. Missing or truncated inputs fail the step (verified in both directions).
  5. continue-on-error census — remove-or-justify resolved as justify-in-place (2 seams), both now annotated in the workflow:
    • Set up Python 3.11 (L124): runner-resilience seam only; the immediately following Ensure python toolchain is available step re-verifies the interpreter fail-closed (set -euo pipefail, exit 1), so a masked setup failure cannot reach any validation step.
    • Generate Live API types (best-effort on PR) (L329): PR-lane frontend-types twin, not a route/SDK authority; the strict twin runs fail-closed on every push/schedule/dispatch execution.
    • (Comment on PR and sync-job Commit updated spec are notification/manual-dispatch-publish paths, outside the validation plane; censused by the tests.)
  6. Non-authoritative recovery || true guards kept (checkout self-repair, ensurepip recovery re-verified fail-closed, npm cache ownership, sync publish path) — censused and bounded by the contract tests so no authoritative step can ever join that set.

SDK-authority question — resolved: current extraction already satisfies method-awareness

VAL-CDG-012 requires the workflow to invoke "method-aware route and SDK authorities". For the SDK plane, no new ref-bound mode is needed: scripts/verify_sdk_contracts.py extracts method-bearing (method, path) operation pairs from both SDKs via (?P<method>…) named-group regexes and compares them against the spec's per-method operations — the comparison is method-aware by construction, and --strict --baseline makes it fail-closed against its pinned baseline. Same for check_sdk_namespace_parity.py --strict --baseline. Exact-SHA binding for the execution is provided once by the route plane's --ref echo in the same job on the same checkout; duplicating a ref-bound mode inside the SDK scripts would add surface without adding evidence. The contract test pins the method-bearing extraction ((?P<method> present) so silent regression to path-only comparison fails CI.

The 12 VAL-CDG-012 contract tests (tests/scripts/test_openapi_workflow_contract.py)

Census (exactly one active OpenAPI workflow, live workflow ID 226588000; single emitter of the live Generate & Validate check name), trigger topology, method-aware authorities, reference selection rules mirrored from the CDG helpers (unfiltered per_page=100 pagination newest-by (run_started_at, run_id, run_attempt); attempt-specific jobs/check URLs; run-level SHA-bound artifacts; movement-in-(main, run, attempt, conclusion) restart), behavioral failure-propagation simulations of the real run blocks under bash -e with command stubs (route authority masked-by-tee, contract-matrix exit, SDK verify), pipefail-before-first-pipe census, summary-input preconditions (missing/truncated inputs fail), exhaustive 12-case gate aggregator table, and conclusion-rewrite exclusion (gate reads only needs.*; summaries/uploads/comment cannot write checks, statuses, or job outputs).

Verification

Gate Result
pytest tests/scripts/test_openapi_workflow_contract.py 12 passed
pytest tests/scripts/test_validate_openapi_routes.py tests/ci/test_required_aggregator_fail_closed.py tests/server/openapi/test_contract_matrix.py + contract file 181 passed
Real Validate handler route coverage run block (bash -e) exit 0; plane echoes exact SHA, 652 served ops, coverage 98.1%, new-missing 0
Real summary + backlog run blocks exit 0 with current-run inputs; exit 1 without
make lint, actionlint openapi.yml, pre-commit hooks pass

Red-first: the contract test file did not exist at base (pytest exit 4 captured before implementation).

Tier 4

Draft + operator-review-required. Touches a required-check workflow (Generate & Validate); merge only via exact-head operator settlement (scripts/settle_tier4_pr.py). Base 622fc3df8f87b23dc4d499c79b5638842b0fbd65 (merged #9672).

… [Tier 4]

VAL-CDG-012: the OpenAPI workflow's authoritative validation steps must
propagate failure, and their verdicts must be method-aware and bound to
the exact execution SHA.

Workflow rearm (.github/workflows/openapi.yml):
- Bind the route authority to the execution's exact resolved commit SHA:
  run scripts/validate_openapi_routes.py with --ref set to the resolved
  HEAD (method-aware VAL-CDG-011 operation plane) and require the plane
  to echo the same binding back, before the baseline-gated coverage check.
- Remove the unconditional-success guard from the contract-matrix pytest
  step: its exit code is now the verdict.
- Arm pipefail in the operationId audit pipe so a crashed producer fails
  the step instead of vanishing into the pipe.
- Make both drift-summary steps require parseable CURRENT-run inputs
  (route-coverage.json, openapi_generated.json, contract-drift-summary.json)
  before presenting any verdict.
- Justify the two remaining continue-on-error seams in place: the
  setup-python runner-resilience seam (closed fail-closed by the next
  step) and the PR-lane best-effort Live types twin (strict twin covers
  push/schedule/dispatch).

Contract tests (tests/scripts/test_openapi_workflow_contract.py): the 12
VAL-CDG-012 named tests: census (single active workflow, single live
check-name emitter), triggers, method-aware route/SDK authorities,
reference selection rules (unfiltered pagination newest-by
(run_started_at, run_id, run_attempt), attempt-specific jobs/checks,
run-level artifacts, movement-restart), failure propagation (behavioral
bash -e simulation of the real run blocks), pipefail census,
summary-input preconditions, gate aggregator exhaustive case table, and
conclusion-rewrite exclusion.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@scarmani scarmani added the operator-review-required Tier-4 parked draft awaiting operator label Aug 6, 2026
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

OpenAPI Spec Update

The OpenAPI specification has changed. Please review the generated spec in the workflow artifacts.

@scarmani

scarmani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

Claude independent model review

Reviewer: claude (anthropic) — independent adversarial model review via the Aragora Claude reviewer, grounded on the exact PR head.
Head: d773453 (d773453), committed 2026-08-06T07:51:14Z.
PR: #9719.
Model family: claude

Verdict: PASS

Verification performed (all at head d773453 in this worktree):

  • Validator CLI contract: scripts/validate_openapi_routes.py has --ref, --internal-prefixes, --json, _require_exact_ref (line 1414), validate_method_aware_plane (line 2338), and emits both "ref" (line 2422) and served_operations_count in the plane payload — every key the new workflow step depends on exists.
  • Inputs exist: scripts/baselines/internal_route_prefixes.json and scripts/baselines/contract_drift_inventory.json are present; real-inventory load_operation_projection() tests pass.
  • Advisory step stays advisory: audit_openapi_docs.py with only --spec --json always returns exit 0 (min-coverage defaults to 0, no fail flags) — the added pipefail in "Check for missing operationIds" surfaces only producer crashes, verified by reading main() (lines 386–409).
  • Unmasked pytest is safe: --timeout=60 requires pytest-timeout, which scripts/ci_install_project.sh --extras dev installs (line 46) in the job's "Install dependencies" step; tests/server/openapi/test_contract_matrix.py passes locally (1 test, 0.43s), so removing || true doesn't turn the required check red.
  • Test suites: new tests/scripts/test_openapi_workflow_contract.py — 12/12 pass; tests/scripts/test_validate_openapi_routes.py — 90/90 pass.
  • Live exposure: .github/actions/pr-scope-classifier/action.yml line 97 puts .github/workflows/openapi.yml in openapi scope, so this PR's own required check executes the new plane step before merge — a plane failure at head cannot silently reach main. (I could not execute the plane script directly; this permission mode denies non-test script runs.)
  • Gate integrity: the generate aggregator is untouched and fail-closed; new preconditions/heredocs run under bash -e, the heredoc-in-YAML pattern is already proven in this workflow (toolchain check, lines 152–155), and EXEC_SHA is 40-hex from git rev-parse (no injection surface).

Findings:

  • [P3] tests/scripts/test_openapi_workflow_contract.py:157-320 — the "reference live-selection rules" block (~200 lines: _paginate_openapi_runs, _plan_date_shards, _validate_run_artifact, _selection_is_stable, and their three tests) tests only functions defined in the same file. It can never catch a regression in the workflow or any shipped code — only an edit to the test file itself can fail it. assert OPENAPI_WORKFLOW_ID == 226588000 (line 341) is likewise a tautology comparing the constant to its own literal. Follows the test_contract_drift_workflow.py precedent, but it is self-referential ballast, not coverage.
  • [P3] .github/workflows/openapi.yml:391-401 — the ref-binding check can essentially never fire: validate_method_aware_plane copies the argv ref into the payload after format validation (it analyzes the checked-out working tree, not the tree at the given SHA), and in CI the working tree is by construction git rev-parse HEAD. The check is ceremonial and fails safe, but the comment's "bind the route authority to this execution's exact resolved commit SHA" overstates what is enforced. Verified by reading validate_method_aware_plane (lines 2338–2422): ref is used only as a stamp on built operations and the payload.
  • [P3] Unverified: I could not execute the plane invocation end-to-end locally (script execution denied by the session's permission mode, and no repo test runs validate_method_aware_plane against the real spec/registry). Whether it exits 0 at head will be verified by this PR's own Generate & Validate required check, which the scope classifier guarantees runs for this diff.

dogfood: yes

@scarmani

scarmani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

OpenAI independent model review

Reviewer: openai (openai) — independent adversarial model review via Codex CLI OpenAI harness, grounded on the exact PR head.
Head: d773453 (d773453), committed 2026-08-06T07:51:14Z.
PR: #9719.
Model family: openai

Verdict: PASS

  • [P3] tests/scripts/test_openapi_workflow_contract.py:329 hard-codes the live workflow ID and comments that it was verified externally, but the test itself only asserts the constant equals itself. This is not a blocking workflow regression, but it can give false confidence if the GitHub workflow ID is ever recreated; verifying via GitHub API would make it real coverage.

dogfood: yes

@scarmani
scarmani marked this pull request as ready for review August 6, 2026 20:31
@scarmani
scarmani requested a review from an0mium as a code owner August 6, 2026 20:31
@scarmani

scarmani commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

Tier-4 Human Settlement Authorization

PR: #9719
Exact head: d773453
Authorized action: admin_squash_merge and branch_protection_reconcile, only if #9719 is non-draft and live exact-head checks/merge-packet remain otherwise green.

Human-risk settlement: I accept the Tier 4 risk for this PR.

@scarmani
scarmani merged commit 56af53b into main Aug 6, 2026
74 of 75 checks passed
@scarmani
scarmani deleted the structex/cdg-openapi-rearm branch August 6, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

operator-review-required Tier-4 parked draft awaiting operator

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant