Repository navigation
test: re-instate doc-count gate (#752, lost to overwrite) + add SAFETY.md ref guard #756
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| /** | ||
| * Cold-gate enforcement for the Move-test-count docs. | ||
| * | ||
| * scripts/check-doc-counts.sh is a fast, no-compile guard: it asserts every | ||
| * judge-facing doc states the SAME Move-test count AND that the count matches the | ||
| * actual `#[test]` total in move/ (an exact `sui move test` proxy — see #738). It | ||
| * monitors all the count forms a doc can use: "N/N tests", bolded "**N/N** Move | ||
| * tests" (#750), the badge URL + ALT text (#751), "N Move tests", "Total tests: N", | ||
| * "N/N passing" (#749), and the "# N/N" command comment. 9/9 forms covered. | ||
| * | ||
| * Without this test the guard only ran when someone remembered to. Wiring it into | ||
| * the cold gate means a Move test added WITHOUT `npm run sync:count` fails `npm | ||
| * test`, with a one-command fix — instead of a judge running `sui move test` and | ||
| * seeing a number that mismatches the README. Mirrors test:doc-links (the doc-link | ||
| * consistency check already in the gate). | ||
| * | ||
| * Run: npx tsx --test scripts/doc-counts.test.ts | ||
| */ | ||
| import { test } from "node:test"; | ||
| import { strict as assert } from "node:assert"; | ||
| import { execFileSync } from "node:child_process"; | ||
| import { fileURLToPath } from "node:url"; | ||
| import { dirname, resolve } from "node:path"; | ||
|
|
||
| const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); | ||
|
|
||
| test("docs state ONE Move-test count that matches the source (check:doc-counts)", () => { | ||
| let out = ""; | ||
| try { | ||
| // The guard cd's to the repo root itself; run from there for good measure. | ||
| out = execFileSync("bash", ["scripts/check-doc-counts.sh"], { | ||
| cwd: repoRoot, | ||
| encoding: "utf8", | ||
| }); | ||
| } catch (err) { | ||
| const e = err as { stdout?: string; stderr?: string; message?: string }; | ||
| assert.fail( | ||
| `check:doc-counts FAILED — a doc states a Move-test count that disagrees with the\n` + | ||
| `others or with the move/ source. The one-command fix is: npm run sync:count\n\n` + | ||
| `${e.stdout ?? e.stderr ?? e.message ?? "(no output)"}`, | ||
| ); | ||
| } | ||
| // Belt-and-suspenders: a 0 exit must be the genuine all-consistent message, not | ||
| // some other early return. | ||
| assert.match( | ||
| out, | ||
| /consistent across all docs AND matches move\/ source/, | ||
| `check:doc-counts exited 0 but without the all-consistent confirmation:\n${out}`, | ||
| ); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| /** | ||
| * Guard: every test named in move/SAFETY.md exists as a Move fun. | ||
| * | ||
| * SAFETY.md is the judge-facing "safety property → test" traceability map; its | ||
| * intro says: "Run one property: `sui move test <test_name>` (substring match on | ||
| * the function)." If a test is renamed or removed but its SAFETY.md reference | ||
| * isn't updated, a judge running that command finds 0 tests — a broken honesty | ||
| * claim ("every fund-safety property maps to a named test you can run"). | ||
| * | ||
| * The fleet edits SAFETY.md frequently (mapping new properties — #739/#743/#745), | ||
| * so a stale/typo'd reference is plausible. This catches it in the cold gate. | ||
| * Mirrors test:doc-links (link refs) + test:doc-counts (count refs). | ||
| * | ||
| * Run: npx tsx --test scripts/safety-test-refs.test.ts | ||
| */ | ||
| import { test } from "node:test"; | ||
| import { strict as assert } from "node:assert"; | ||
| import { readFileSync, readdirSync } from "node:fs"; | ||
| import { fileURLToPath } from "node:url"; | ||
| import { dirname, resolve, join } from "node:path"; | ||
|
|
||
| const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); | ||
|
|
||
| /** Every `fun NAME` declared in the Move package (tests + sources). */ | ||
| function moveFunNames(): Set<string> { | ||
| const names = new Set<string>(); | ||
| for (const d of ["move/sources", "move/tests"]) { | ||
| const dir = join(repoRoot, d); | ||
| for (const f of readdirSync(dir)) { | ||
| if (!f.endsWith(".move")) continue; | ||
| const src = readFileSync(join(dir, f), "utf8"); | ||
| for (const m of src.matchAll(/\bfun\s+([a-z_][a-z0-9_]*)/g)) names.add(m[1]); | ||
| } | ||
| } | ||
| return names; | ||
| } | ||
|
|
||
| /** Backtick-quoted test names in SAFETY.md's middle table column: | prop | TESTS | file |. */ | ||
| function safetyTestRefs(): string[] { | ||
| const md = readFileSync(join(repoRoot, "move", "SAFETY.md"), "utf8"); | ||
| const refs = new Set<string>(); | ||
| for (const line of md.split("\n")) { | ||
| if (!line.startsWith("|") || line.includes("---")) continue; // data rows only | ||
| const testsCol = line.split("|")[2] ?? ""; // index 2 = the middle (tests) column | ||
| for (const m of testsCol.matchAll(/`([a-z_][a-z0-9_]{6,})`/g)) { | ||
| const n = m[1]; | ||
| if (n.includes("_tests") || n.endsWith("_move")) continue; // file refs, not test names | ||
| refs.add(n); | ||
| } | ||
| } | ||
| return [...refs]; | ||
| } | ||
|
|
||
| test("every test named in move/SAFETY.md exists as a Move fun (judges run `sui move test <name>`)", () => { | ||
| const funs = moveFunNames(); | ||
| const refs = safetyTestRefs(); | ||
| // Sanity: the parser must actually find the table, not silently match nothing. | ||
| assert.ok( | ||
| refs.length > 100, | ||
| `expected 100+ test refs parsed from SAFETY.md, got ${refs.length} — did the table format change?`, | ||
| ); | ||
| const missing = refs.filter((r) => !funs.has(r)); | ||
| assert.deepEqual( | ||
| missing, | ||
| [], | ||
| `move/SAFETY.md references test name(s) that no longer exist as a Move fun — a judge running ` + | ||
| `'sui move test <name>' would find 0 tests. Update SAFETY.md (or restore the test). Stale: ${missing.join(", ")}`, | ||
| ); | ||
| }); | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This treats the second column of every Markdown table as test references, but
move/SAFETY.mdalso has tables such asProperty | Proof | Where; the current parser already extracts proof prose identifiers likerug_chance_bpsandmultiplier_bpsand validates them as if they were runnable tests. When a proof row legitimately mentions a field, script, or other non-funidentifier, the cold gate will fail even though no SAFETY test reference is stale, so the parser should track the table header and only scanProperty | Test | Filerows.Useful? React with 👍 / 👎.