test(cage,fdtables): add CONC-001..005 concurrency and isolation coverage - #1349
Open
Shounak-Ghosh wants to merge 1 commit into
Open
test(cage,fdtables): add CONC-001..005 concurrency and isolation coverage#1349Shounak-Ghosh wants to merge 1 commit into
Shounak-Ghosh wants to merge 1 commit into
Conversation
Shounak-Ghosh
force-pushed
the
test-conc
branch
from
August 12, 2026 20:00
aae991b to
076c3ef
Compare
Contributor
End-to-End Test ReportTest Previewgrate harnessGrate Test Report
Cases
static harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-math harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-filesystem harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-memory harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
wasm-process harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-signals harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
wasm-networking harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-dynamic-linking harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Yaxuan-w
reviewed
Aug 12, 2026
Yaxuan-w
left a comment
Member
There was a problem hiding this comment.
Thanks for doing all the work here. These are valuable! I think some changes will need discussion (especially for the glibc and vmmap bug fixes). That would be great to:
- Only include test in this PR (put test into skip list if they cannot be passed on current version)
- Open up issues for each bug you found
- Submit PR per bug after the discussion with the team
Shounak-Ghosh
force-pushed
the
test-conc
branch
from
August 20, 2026 01:17
076c3ef to
7ced602
Compare
Implements the CONC-001..005 oracles from #1304. Tests that cannot pass on current main are marked #[ignore] or listed in skip_test_cases.txt, each naming the issue that unblocks it. The 11 bug fixes these tests surfaced are split into separate PRs per review feedback; the full original work is preserved on branch test-conc-bug-fixes. Refs #1304
Shounak-Ghosh
force-pushed
the
test-conc
branch
from
August 20, 2026 01:28
7ced602 to
a0ef35b
Compare
Contributor
End-to-End Test ReportTest Previewgrate harnessGrate Test Report
Cases
static harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-math harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-filesystem harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-memory harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
wasm-process harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-signals harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
wasm-networking harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
wasm-dynamic-linking harnessTest ReportDeterministic TestsSummary
Test Results by Category
Fail TestsSummary
Test Results by Category
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full CONC-001..005 test matrix from #1304, which asks us to verify
that cage creation/destruction, fd-table operations, cage-table operations,
resource accounting, and concurrent syscall execution preserve runtime invariants.
This PR contains tests only. An earlier revision also carried the 11 bug fixes
these tests surfaced; per review feedback those are now split out into one issue
and one PR per bug.
Also adds a
rust-unit-testsCI job for thecageandfdtablescrates, whoseunit tests have never run in CI. The
cagesuite does not currently compile: twotests in
vmmap.rsbind aVmmapwithoutmutand then mutate it. Both areone-word corrections inside
#[cfg(test)] mod tests, so they touch test coderather than runtime code, and they are here as the job cannot run
without them.
Related issues
Closes #1304.
Mapping to the issue's matrix (the issue lists CONC-005 as one row covering three
resources; it is implemented here as three separate tests, one per resource):
src/cage/src/cage.rs:conc_001_cage_spawn_destroy_stress,conc_001_finalize_waits_for_grate_inflight(matches the issue's "Cage lib inner test suite")conc_002_cage_fd_fs_stress.c+ 4 in-crateconc_002_*testsconc_003_cage_fd_refcounts.c+ 4 in-crateconc_003_*testsconc_004_dup_close_fork_refcounts.c+ 2 in-crateconc_004_*testsconc_005_fd_exhaustion_isolation.c,conc_005_memory_pressure_isolation.c,conc_005_syscall_flood_isolation.c+ 3 in-crateconc_005_*testsThe C and Rust suites are deliberately complementary, not redundant: the C side
is a black-box differential test (the same binary is run natively and under lind,
and stdout is diffed), while the in-crate Rust side asserts exact refcount
equalities that the public C/POSIX interface cannot observe at all. Where the C
test can only watch a pipe reach EOF, the Rust mirror reads the count straight
out of a registered close handler.
Bugs found
Writing these tests surfaced 11 defects, plus one missing capability and one
behavioural inconsistency that needs a team decision. Per review feedback none of
the fixes are in this PR. Each is filed as its own issue and will be submitted as
its own PR after discussion. The full set of fixes is preserved verbatim on branch
test-conc-bug-fixes, where each is paired with the test that found it.Three of the 11 are guest-triggerable, including one that lets any cage terminate
the entire runtime; those are called out in their respective issues.
Issue list: (to be filled in once filed)
Currently ignored / skipped
Eight tests cannot pass on current
mainand are marked#[ignore]or listed inskip_test_cases.txt. The CI job is green as submitted. Each is annotated inplace with the condition that unblocks it, so the fix PRs can find them by grep.
Rust:
#[ignore](5)maincross_cage_fd_isolationmain, not from this PRget_specific_virtual_fd_rejects_fd_at_maxconc_002_shared_underfd_refcount_raceconc_002_copy_fdtable_vs_concurrent_churnconc_003_dup2_overwrite_refcount_conservationThe three deadlocks are not a judgement call: the refcount corruption panics one
worker thread, and the surviving workers then block forever on the round barrier.
Unfixed, these hang the test binary rather than reporting a failure, so
#[ignore]is required rather than merely tidy.
C:
skip_test_cases.txt(3)conc_004_dup_close_fork_refcounts.cdup2(BADFD, BADFD)); the rest of the file passes todayconc_005_fd_exhaustion_isolation.cconc_005_memory_pressure_isolation.csetrlimit()reaches alind_debug_panic!and takes the whole runtime down; unfixed it does not fail, it kills the harnessEach skipped file's header comment explains what blocks it and what to remove when
the fix lands.
Testing
Results on this branch:
cage: 38/38 pass, including bothconc_001_*tests.fdtables: 45 pass, 0 fail, 6 ignored (the 5 above plus one pre-existing#[ignore]), and a second binary at 24/24.fdtables) and 15× (cage) consecutively with zero failures orhangs, since several of these tests are probabilistic by construction and a single
green run proves little.
cargo fmt --checkclean on both crates.Every C test is written to be harness-safe: exactly one line of stdout, and no
pids, clocks, addresses, fd numbers, sizes, or errno values are ever printed or
compared, since those legitimately differ between the native and lind runs.
Diagnostics go to fd 2, which the harness surfaces only on a nonzero exit.
CI
Adds a
rust-unit-testsjob tolint.ymlrunning both crates' tests.--test-threads=1is mandatory: both crates' tests mutate process-global state(
CAGE_MAP;FDTABLE/FDCOUNT/CLOSEHANDLERTABLE) and serialize on hand-rolledmutexes rather than real test isolation.
One thing worth knowing about the existing suite:
refresh()clearsFDTABLEandCLOSEHANDLERTABLEbut notFDCOUNT, so refcount state leaks between tests inone binary. At least one pre-existing test (
test_dup_close, a#[should_panic])depends on that leakage and fails when run in isolation. Every test added here uses
a dedicated
fdkindand a disjointunderfdwindow specifically so it neveraliases another test's leftovers, but the ordering fragility in the older tests is
worth a follow-up.
Compatibility
No production code changes, so no guest-visible behaviour changes. The only
non-test files touched are
.github/workflows/lint.yml(new job) andskip_test_cases.txt.Notes for reviewers
Suggested reading order:
.github/workflows/lint.yml, and the ignore/skip set above.conc_*.cfiles and the in-cratemod testsadditions.Each ignored or skipped test states, at its definition, the condition that unblocks
it. Those are worth reading alongside the corresponding issue.
Checklist