Skip to content

Bound Included Files recovery cleanup - #829

Merged
Infiland merged 1 commit into
mainfrom
dev/bound-included-files-recovery-cleanup
Aug 10, 2026
Merged

Bound Included Files recovery cleanup#829
Infiland merged 1 commit into
mainfrom
dev/bound-included-files-recovery-cleanup

Conversation

@Infiland

@Infiland Infiland commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Closes #826

What changed

  • Treat a recorded directory that is identity-verified absent during the top-down cleanup pass as an absent subtree, and skip its recorded file descendants.
  • Keep the existing bottom-up directory and root checks authoritative. A directory that reappears, changes identity, or becomes otherwise unknown is preserved with a warning; cleanup does not traverse or delete the replacement.
  • Add a deterministic 16-versus-256-entry operation-count regression requiring zero descendant file-state probes and bounded ancestor work independent of descendant count.
  • Add an adversarial late-reappearance regression that preserves the external replacement bytes and retains the stage root.
  • Bump the release surfaces to 0.7.54 (2026-08-10) and document the recovery invariant.
  • Rebase onto merged Shard Windows Included Files scale gate #831 while preserving its v0.7.53 history and fail-closed Windows scale-gate contract.

Why

The native Windows failure documented in #826 occurred after the moved stage's included_files directory had already been proven absent. Cleanup nevertheless repeated descendant file-state and fallback-ancestor work for every recorded file: about 20,000 unnecessary file-state probes and about 240,000 redundant lstat calls at 10,000 entries.

#831 separated the broad Windows transaction suite from the exact 10,000-entry scale gate without increasing either 20-minute timeout. That makes the gate reliable and independently observable, but it does not remove this production cleanup cost. This PR fixes the measured absent-subtree path. The distinct present-tree repeated-parent optimization remains scoped to #828 and is not a prerequisite for this fix.

Windows CI contract

Fresh checks at the exact rebased head must satisfy both #831 jobs:

  • windows-artifact-transactions runs the broad Windows transaction coverage on windows-2025 / CPython 3.12.10 with GM2GODOT_SKIP_WINDOWS_INCLUDED_FILES_SCALE_GATE=1; only the exact 10,000-entry case may skip there.
  • windows-included-files-scale runs python -m scripts.run_windows_included_files_scale_gate with GM2GODOT_REQUIRE_WINDOWS_INCLUDED_FILES_SCALE_GATE=1. Its fail-closed runner must collect and execute exactly the pinned 10,000-entry test, with no skip, failure, error, expected failure, unexpected success, or fixture exit.

Both jobs retain the unchanged 20-minute timeout. The broad job must pass, and the dedicated scale job must pass with substantial timing margin.

Local validation at rebased head 4b6716a

  • Included Files module: 178 tests passed, 13 skipped.
  • Exact fail-closed 10,000-entry runner: 1 test passed, 0 skipped, in 21.117s.
  • Workflow-policy and runner coverage: 80 tests passed.
  • Version and documentation health: 12 tests passed.
  • Exact Godot 4.7.1 discovery: 83 tests passed against 4.7.1.stable.official.a13da4feb.
  • Godot validation, golden conversion, and project-settings coverage: 60 tests passed.
  • Ruff passed repository-wide; Pyright reported 0 errors and 0 warnings.
  • git diff --check passed; the diff is the intended 15-file scope.
  • Three independent read-only reviews found no correctness, safety, reappearance, mount/link, or asymptotic blocker. Synthetic cleanup retained constant filesystem work through 10,000 absent descendants and near-linear in-memory iteration through 100,000 records.
  • The full local run executed 2,428 tests and exposed three already-tracked load-sensitive failures outside this diff: the buffered Godot reader race (Drain buffered Godot diagnostics before stopping the timeout reader #827) and two 3-second Linux GUI verifier races (Make Linux GUI artifact verifier tests deterministic under host load #832). Each exact case passed when rerun in its own fresh process. Fresh exact-head CI remains authoritative.

#831's workflow, fail-closed runner, and workflow-policy files are byte-identical to main.

Exact-head acceptance

All 20 expected checks at 4b6716ae6b30424ed5318e65a579fbfeaa7d4f0d completed with 17 successes and the 3 expected PR-only release skips; there were no failures, cancellations, duplicates, missing records, or unexpected checks.

  • Windows broad transactions: 618 tests in 454.861s, both Bound 10,000-entry Included Files recovery cleanup on native Windows #826 regressions explicit ok, and only the exact 10,000-entry case delegated to the dedicated job. Whole job: 8m08s, leaving 11m52s.
  • Dedicated Windows scale: exactly 1 pinned test in 606.388s, OK, zero skips. Whole job: 10m43s, leaving 9m17s.
  • Windows crash recovery: 19 tests in 913.451s, OK (skipped=2). Whole job: 15m48s.
  • Linux full suite: 2,428 tests in 692.951s, OK (skipped=151), 81.38% coverage; native bind-mount gate 1/1.
  • Exact Godot smoke verified 4.7.1.stable.official.a13da4feb and passed 83 + 60 tests.
  • Ruff, Pyright, dependency locks, macOS transactions, TCC/LTS conversions, and all platform builds passed.

The PR is ready for a normal merge.

@Infiland

Infiland commented Aug 3, 2026

Copy link
Copy Markdown
Owner Author

Exact-head Windows evidence is not sufficient to merge yet. Run 30832195303 executed all 614 tests successfully, including both new regressions and the 10,000-entry case, but the job was still marked CANCELLED at its 20-minute cap during normal teardown. The test step took 1162.026 seconds; the 10,000-entry test occupied approximately 10m46s from the preceding test timestamp.

This patch removes the measured absent-subtree descendant probes, but the separate present-tree repeated-parent scaling tracked by #828 still dominates the same 10,000-entry scenario. Keeping this PR draft and unmerged. I will land #828 independently first, then rebase this PR and require a fresh exact-head Windows run with substantial margin.

@Infiland
Infiland force-pushed the dev/bound-included-files-recovery-cleanup branch from 2db0c86 to 4b6716a Compare August 10, 2026 07:52
@Infiland

Copy link
Copy Markdown
Owner Author

Superseding my 2026-08-03 status comment: #828 is no longer a prerequisite for this PR.

#831 has since merged and shipped as v0.7.53. It preserves the unchanged 20-minute limit while separating the native Windows contract into:

  • windows-artifact-transactions, which runs the broad suite and may skip only the pinned 10,000-entry case under GM2GODOT_SKIP_WINDOWS_INCLUDED_FILES_SCALE_GATE=1; and
  • windows-included-files-scale, whose fail-closed runner requires that exact case to be the sole collected and successfully executed test under GM2GODOT_REQUIRE_WINDOWS_INCLUDED_FILES_SCALE_GATE=1.

#829 is now rebased onto that mainline at 4b6716ae6b30424ed5318e65a579fbfeaa7d4f0d and bumped to v0.7.54. Its behavioral patch is unchanged: it removes the measured absent-subtree descendant probes while retaining the later bottom-up checks that preserve any external reappearance. #828 continues to track the separate present-tree repeated-parent cost.

The previous canceled combined run remains historical evidence of the original bottleneck, but it no longer defines the dependency order. This PR stays draft until fresh checks at the exact rebased head show both Windows jobs green with substantial margin and every other required check green. No exact-head CI success is claimed yet.

@Infiland

Copy link
Copy Markdown
Owner Author

Exact-head acceptance is complete for 4b6716ae6b30424ed5318e65a579fbfeaa7d4f0d.

All 20 expected checks resolved as 17 successes plus the 3 normal PR-only release skips, with no failure or cancellation. Native Windows evidence:

Linux passed 2,428 tests with 81.38% coverage and its native bind-mount gate. Exact Godot 4.7.1.stable.official.a13da4feb, Ruff, Pyright, all locks/builds, macOS, LTS 2026, and TCC conversions also passed. Three independent final audits found no code, safety, scope, versioning, or release blocker. Marking this PR ready for normal merge.

@Infiland
Infiland marked this pull request as ready for review August 10, 2026 08:14
@Infiland
Infiland merged commit 827e24b into main Aug 10, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound 10,000-entry Included Files recovery cleanup on native Windows

1 participant