Skip to content

⏱️ Give the runtime suites deadlines that clear them by a margin (#538) - #539

Merged
taras merged 1 commit into
mainfrom
ci-main-red-538
Aug 22, 2026
Merged

⏱️ Give the runtime suites deadlines that clear them by a margin (#538)#539
taras merged 1 commit into
mainfrom
ci-main-red-538

Conversation

@taras

@taras taras commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Why

main is red (#538). On 5542c92 the composability battery killed the complete Deno suite at its thirty-minute ceiling; the suite's own last line was ok | 677 passed (4667 steps) | 0 failed (29m30s). That is #482 one ceiling higher — a deadline firing on a healthy command, which costs the signal it exists to give and supplies a false one.

The thirty minutes #485 gave the suite tracked the duration observed then, and the suite has grown into it. Standalone (test-deno job) over the last four pushes to main: 1102s → 1150s → 1314s → 1327s. Under the battery, which stretches it by about a third: 1427s → 1534s → 1697s → killed at 1800s. Bun's suite in the same run finished healthy at 1,031s against the shared 1,200s ceiling (Node's at 856s), so it is the next one to fire for the same reason.

Run: https://github.com/taras/executable.md/actions/runs/32584003585

What changes

Before:

test carried 30m; every other command, the Node and Bun suites included, carried the 20m default. The battery announced 20m deadline each except test 30m.

After:

test carries 45m; test:node and test:bun carry 30m; everything else keeps 20m. The battery announces 20m deadline each except test 45m, test:node 30m, test:bun 30m, and a settled command is still reported with the deadline that settled it.

What must stay true

  • One place answers a deadline — enforced by commandTimeout() in scripts/lib/verify.ts, read by the race and both report lines; checked by commandTimeout and deadline reporting in scripts/tests/verify-coordinator.test.ts.
  • The opening line names every command whose deadline differs — checked by "announces exactly the deadlines the battery will apply".
  • An explicit coordinator timeout still overrides every command's own — checked by "lets an explicit coordinator timeout override every command's own".

How to verify it

  • deno task test scripts/tests/verify-coordinator.test.ts — 5 passed (32 steps). The commandTimeout cases fail if any suite's deadline is not the one stated here or another command picked one up; the deadline reporting cases fail if the announcement names a deadline no command runs under.
  • deno task lint and pnpm exec tsc --project tsconfig.node.json --noEmit pass.
  • The proof this PR is for is composability running here under the ci-main-red-fix label: the same battery that killed the suite on main, with the suite now clearing its ceiling.

Scope

Included

  • The three runtime-suite deadlines in BATTERY, their rationale, and the tests that pin them.

Intentionally unchanged

  • The shared 20m ceiling for the other eight commands — each still clears it by a wide margin (longest: check at 83s).
  • No retries, no sharding, no change to which commands run or when composability runs.
  • The growth itself. The standalone Deno suite gained ~200s between d1b9ccb and dda464c, mostly in packages/workflow/tests/git-*.test.ts and pull-request*.test.ts (each roughly 1.5–2× slower); that is worth its own look, but it is a speed question, not the defect that reddened main.

Risks and limitations

  • A real wedge in the Deno suite now takes 45 minutes to surface in the battery instead of 30. The battery's purpose for the ceiling is to settle a wedge eventually rather than hang; the three suites' own jobs (test-deno, test-node, test-bun) are where a hang would be seen first.
  • This is a ceiling that still tracks a growing suite, only from further away. If the suite keeps growing at this rate it will need revisiting again; a stall-based deadline (no output for N minutes) would catch a wedge without tracking total duration, and would be a separate design change.

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

On `main` at 5542c92 the composability battery killed the complete Deno suite
at its thirty-minute ceiling, and the suite's own last line was
`ok | 677 passed (4667 steps) | 0 failed (29m30s)`. That is #482 again one
ceiling higher: a deadline firing on a healthy command, costing the signal it
exists to give and supplying a false one.

The thirty minutes #485 gave the suite tracked the duration observed then. The
suite has grown since — on a runner of its own it went 1102s → 1150s → 1314s →
1327s over the last four pushes to `main`, and the battery stretches it by
about a third (1427s → 1534s → 1697s → killed at 1800s). A ceiling that tracks
a growing suite closely fires as soon as the suite grows into it, so the Deno
suite now carries forty-five minutes: wide of the observed duration rather than
just past it.

The Node and Bun suites run under the same load, and Bun's is stretched to more
than twice its standalone time: in the same run it finished healthy at 1,031s
against the shared 1,200s ceiling, with Node's at 856s. Both now carry thirty
minutes, so the next growth does not open the next `ci-main-red` issue for the
same reason.

Everything else keeps the twenty-minute default. `commandTimeout()` is still
the one place a deadline is answered, the opening line still names every
command whose deadline differs, and an explicit coordinator timeout still
overrides them all.
@taras taras added the ci-main-red-fix Maintainer-controlled: lands one PR while main is red, only with the full clean-checkout proof label Aug 22, 2026
@github-actions

Copy link
Copy Markdown

PR #539: ⏱️ Give the runtime suites deadlines that clear them by a margin (#538)

2 files, +57 / -24

Scope

✅ PR scope looks good.

Structural

Oxlint structural signals:

  • no-unnecessary-type-assertion ×4: scripts/lib/verify.ts

Slop

✅ Slop indicators look low.

Static Analysis

Oxlint: 4 diagnostics across 1 file (1 rule)
Density: 0.070 violations/added-line

no-unnecessary-type-assertion (4): scripts/lib/verify.ts

Correctness

No extraneous code patterns detected.

@taras
taras enabled auto-merge (squash) August 22, 2026 17:09
@taras
taras merged commit d8e136e into main Aug 22, 2026
32 checks passed
@taras
taras deleted the ci-main-red-538 branch August 22, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-main-red-fix Maintainer-controlled: lands one PR while main is red, only with the full clean-checkout proof

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant