Add check-claims.mjs / claim-briefs.mjs: mechanize the claim-verification pre-pass - #11
Merged
Merged
Conversation
…tion pre-pass check-facts.mjs already verifies the curated .docs-assist/reference.yml registry deterministically. This extends the same approach to the whole doc corpus, cold: check-claims.mjs extracts identifier-shaped claims (CLI commands/flags, config/env keys, function/class names, file paths, version requirements) from every doc and resolves each against the code with git grep/git ls-files, no agent needed. What's left (described behavior, numeric assertions) goes to claims-needs-judgment.json; claim-briefs.mjs turns that into one self-contained brief per doc for the doc-auditor fan-out claim-verification.md already describes. Wired into /docs-assist:audit and claim-verification.md as the recommended first pass, and into /docs-assist:setup-hooks as an opt-in CI check (non-strict by default, since "missing" can also mean gitignored-but-real rather than actually gone). Built and validated against two real repos: 453 claims from a 14-doc corpus resolved in seconds, 192 judgment-only claims fanned out to 14 parallel agents, 6 real drifted/inconsistent claims found. Fixed two git-grep argument-parsing bugs (a leading "--" in a flag pattern being read as a git-grep option; ** glob pathspecs silently matching nothing without :(glob) magic) during that validation run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI's Vale check caught one in claim-verification.md (a real error, not
a warning); swept the rest of this PR's changed files for the same
pattern in code comments and generated report text too, for consistency
with the plugin's own no_em_dashes convention even where Vale doesn't
lint (.mjs/.yml aren't in its [*.{md,mdx}] scope).
Verified with the same check CI runs: vale --config=assets/lint/vale/.vale.ini
--minAlertLevel=error . -> 0 errors, 0 warnings, 0 suggestions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per the maintainer's explicit call, same policy as 0.9.6: the check-claims.mjs / claim-briefs.mjs claim-verification tooling is a real, notable addition worth a version bump past the 0.9.5 ceiling agent-driven work otherwise caps at. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check-facts.mjsalready verifies the curated.docs-assist/reference.ymlregistry deterministically, but everything else in a doc set's claims (CLI flags, config keys, function/class names, file paths, version requirements) still relied on an agent re-deriving the same grep-and-compare work by hand.assets/ci/check-claims.mjsextracts those identifier-shaped claims from the whole doc set and resolves each against the code withgit grep/git ls-files, no agent involved. What it can't settle (described behavior, numeric assertions) goes toclaims-needs-judgment.json, grouped by doc.assets/ci/claim-briefs.mjsturns that judgment queue into one self-contained brief per doc, ready to hand to adoc-auditorfan-out perclaim-verification.md's existing method./docs-assist:auditandclaim-verification.mdas the recommended first pass, and into/docs-assist:setup-hooksas an opt-in, non-strict-by-default CI check (assets/ci/github/check-claims.yml).Why
Built and validated this end-to-end against a real 14-doc corpus (a Python weather-data repo, unrelated to this plugin): extracted and mechanically resolved 453 candidate claims in seconds, fanned the 192 that needed judgment out to 14 parallel agents, and found 6 real drifted or inconsistent claims — a stale numeric default, an inverted behavior claim, a fix presented as a to-do that was already applied, an internal count inconsistency, and a couple of wording mismatches. A lint pass alone would have caught none of them.
Two real bugs surfaced and got fixed during that validation run:
--(any CLI flag) was misparsed as agit grepoption itself, dumping usage text instead of searching. Fixed by passing the pattern via-e.git ls-files -- '**/basename'silently matches nothing without:(glob)pathspec magic — switched to filtering the full tracked-file listing in JS instead.Test plan
node scripts/validate.mjs— 2038 checks passednpx markdownlint-cli2— 0 issuesnpx cspell "**/*.md" ".docs-assist/**"— 0 issuescheck-claims.mjs+claim-briefs.mjsagainst this repo's own docs (caught a real gitignored-vendored-file edge case, documented rather than "fixed" — the tool correctly scopes to tracked content)🤖 Generated with Claude Code