feat(doc-index): infer real section granularity, hash sections for staleness

.github/workflows/ci.yml

@@ -34,6 +34,36 @@ jobs:
       - name: Run tests
         run: make test
 
+  # Surfaces the enforcement corpus as its own check. The `test` job already
+  # runs these assertions, but a green suite of ~4,700 tests is not evidence
+  # that a gate can fail -- and a gate that never fails is worthless. This job
+  # makes the red direction visible in the pipeline: known-bad fixtures must be
+  # rejected, known-good fixtures must stay clean.
+  gates:
+    name: Enforcement Gates (seeded violations must fail)
+    runs-on: ubuntu-latest
+    permissions:
+      # This job runs pull-request code (`make install`, `make test-gates`), so
+      # it gets the least privilege that still lets it check out and read.
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          # Keep the job token out of .git/config so the test corpus this job
+          # executes cannot read it back. The initial fetch is unaffected.
+          persist-credentials: false
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install test dependencies
+        run: make install
+
+      - name: Known-bad must be red, known-good must stay green
+        run: make test-gates
+
   coverage:
     name: Coverage (≥90%)
     runs-on: ubuntu-latest

.gitignore

@@ -569,6 +569,10 @@ coverage.xml
 **/.cache/decisions.jsonl.1
 **/.cache/decisions.jsonl.lock
 
+# doc-index cache (doc_index.py) — per-project structural index for JIT
+# retrieval, read-once-per-file. Rebuilds automatically on content change.
+**/.cache/doc-index/
+
 # Superpowers brainstorming/planning specs (local-only)
 docs/superpowers/
 

Makefile

@@ -40,6 +40,7 @@ help:
 	@echo ""
 	@echo "Available targets:"
 	@echo "  make test                          - Run all tests in parallel (-n 6)"
+	@echo "  make test-gates                    - Run the enforcement gate corpus (seeded violations must fail)"
 	@echo "  make test-verbose                  - Run tests with verbose output"
 	@echo "  make test-quick                    - Run fast tests only (skip slow integration tests)"
 	@echo "  make test-coverage                 - Run tests with coverage report"
@@ -149,6 +150,14 @@ test: check-pytest
 	@echo "Running Constructor Studio tests with pipx..."
 	$(PYTEST_PIPX) tests/ -n 6 -v --tb=short
 
+# Runs the enforcement corpus on its own so the pipeline shows, as a named
+# check, that these gates fail on a seeded violation. The assertions already
+# live in the suite; a green `test` job does not evidence the red direction
+# unless you know the tests exist and go and read them.
+test-gates: check-pytest
+	@echo "Running enforcement gate corpus (known-bad must be red, known-good must be green)..."
+	$(PYTEST_PIPX) tests/test_enforcement_empty_codebase.py tests/test_enforcement_empty_scan.py -v --tb=short
+
 # Run tests with verbose output
 test-verbose: check-pytest
 	@echo "Running Constructor Studio tests (verbose) with pipx..."

architecture/features/eval-harness.md

@@ -13,6 +13,7 @@
   - [Run Eval Suite](#run-eval-suite)
   - [Score Structural Compliance](#score-structural-compliance)
   - [Score Rule Compliance (Advisory Judge)](#score-rule-compliance-advisory-judge)
+  - [Assess Semantic Coverage](#assess-semantic-coverage)
 - [4. States (CDSL)](#4-states-cdsl)
   - [Eval Report Lifecycle](#eval-report-lifecycle)
 - [5. Definitions of Done](#5-definitions-of-done)
@@ -209,6 +210,40 @@ or unreadable gold file is *excluded* from calibration rather than counted as a
 - [x] - `p1` - A deterministic reference-stub `JudgeFn` for tests and calibration wiring (not a model) - `inst-judge-stub`
 - [x] - `p1` - The `Calibration` result model (accuracy, consistency, coverage) - `inst-judge-calibrate`
 
+### Assess Semantic Coverage
+
+- [x] `p1` - **ID**: `cpt-studio-algo-eval-semantic`
+
+The semantic-coverage engine: it asks whether a marked block *implements the requirement it
+cites*, which marker density cannot. A deterministic stdlib token-overlap pre-filter scores
+every block against its requirement text and surfaces the weak links; only those go to a
+pluggable `SemanticJudgeFn` (the model call lives out-of-tree). It is advisory throughout —
+no verdict here touches an exit code — and honest: a block with no retrievable requirement, or
+too little text to compare, is reported `unjudgeable`, never a silent "covered". Files a human
+declared `excluded` in the coverage report are skipped; files flagged `whole_file_claims` are
+prioritised. Every model verdict carries an evidence check: a quote absent from the code is not
+trusted.
+
+**Steps**:
+1. [x] - `p1` - Tokenise code and requirement (camelCase + Unicode word-boundary split, stopwords dropped) and score their overlap against the requirement set, or `None` below the token floor - `inst-semantic-tokenize`
+2. [x] - `p1` - Rank every pairing weakest-first, marking below-threshold blocks as weak links and below-floor blocks as unjudgeable, and marking every block in a whole-file-claim file a weak link regardless of overlap (always judged) - `inst-semantic-prefilter`
+3. [x] - `p1` - Judge only the weak links through the injected `SemanticJudgeFn`, degrading a raising or malformed reply to unjudgeable and evidence-checking every quote - `inst-semantic-finding`
+4. [x] - `p1` - Assess pairings end-to-end — scope, rank, judge weak links, and report unjudgeable coverage gaps — advisory throughout - `inst-semantic-report`
+
+**Supporting**:
+- [x] - `p1` - Module imports, verdict constants, and the provisional calibration constants + stopword set - `inst-semantic-imports`
+- [x] - `p1` - The pairing, ranked (with a whole-file-claim `forced` flag), request, reply data model and the `SemanticJudgeFn` seam — the seam is called synchronously and unbounded, so a judge that can hang must enforce its own timeout - `inst-semantic-datamodel`
+- [x] - `p1` - Blank character spans in place (spaces, newlines preserved) so a verbatim quote of the untouched text still matches - `inst-semantic-blank`
+- [x] - `p1` - Derive two code views in one grammar-aware `tokenize` pass — an overlap view (comments and all strings blanked, so prose cannot inflate the score) and an evidence view (comments and docstrings blanked, inline strings kept) — falling back to a plain comment strip for a non-tokenisable fragment - `inst-semantic-strip`
+- [x] - `p1` - Read the frozen coverage-report contract (`excluded[]` / `whole_file_claims[]`, path-separator normalised, `excluded` taking precedence), degrading to an empty scope - `inst-semantic-scope`
+- [x] - `p1` - Build the deterministic judge prompt (bounded fields) and parse a reply to a verdict - `inst-semantic-prompt`
+- [x] - `p1` - The evidence-present hallucination guard: a quote must occur in the evidence view (comments/docstrings stripped), so a quote matching only a comment is not evidence - `inst-semantic-evidence`
+- [x] - `p1` - The `SemanticGap` coverage-gap record (block_id, path, start_line, reason), located like a finding so a report consumer can point at every gap - `inst-semantic-gap`
+- [x] - `p1` - The reference stub judge (deterministic overlap buckets) and its best-evidence-line quote - `inst-semantic-stub`
+- [x] - `p1` - The requirement resolver adapter over `get_content_scoped` - `inst-semantic-resolve`
+- [x] - `p1` - Read a `[gold]` verdict label for calibration, or `None` when absent/malformed - `inst-semantic-gold`
+- [x] - `p1` - Calibrate the judge over gold-backed pairings: report accuracy, consistency, and effective sample size per case, excluding unscoreable / crashed / strict-tie cases — accuracy over cases with a majority, consistency over cases with ≥2 surviving runs — `None` when nothing is measurable - `inst-semantic-calibrate`
+
 ## 4. States (CDSL)
 
 ### Eval Report Lifecycle
@@ -253,6 +288,7 @@ them (subscripting when the flag is absent raises `KeyError`):
 | Eval Harness | `skills/studio/scripts/studio/utils/eval_harness.py` | Scenario/run loading, scorer seam, runner, report, regression diff |
 | Structural Scorer | `skills/studio/scripts/studio/utils/eval_structural.py` | Deterministic structural checks over a run's manifest + phase frontmatter |
 | Advisory Judge | `skills/studio/scripts/studio/utils/eval_judge.py` | Advisory rule-compliance judge (pluggable model seam) + gold-set calibration |
+| Semantic Coverage | `skills/studio/scripts/studio/utils/eval_semantic.py` | Token-overlap pre-filter + advisory judge seam over marked-block vs requirement, with honesty + evidence guards |
 
 ## 7. Acceptance Criteria
 
@@ -267,3 +303,7 @@ them (subscripting when the flag is absent raises `KeyError`):
 - [x] `p1` - Judge calibration reports accuracy against a human gold set and run-to-run consistency, kept separate from structural compliance; judge coverage is derived from which scenarios carry a gold set
 - [x] `p1` - The run evidence is hard-capped: the total (headers, truncation markers, separators and the omission line included) never exceeds the evidence budget. A trimmed phase is still judged; when whole phases are omitted to fit, the request is marked incomplete and the judge returns `UNKNOWN` without a model call — a verdict is never certified from evidence with entire phases unseen
 - [x] `p1` - In default (non-JSON) mode the human summary prints the scorer coverage line, explains that advisory-only UNKNOWNs come from an unwired judge (never counted against the gate), and prints the calibration metrics under `--calibrate`; the JSON payload is unchanged
+- [x] `p1` - Semantic coverage ranks marked blocks by deterministic token-overlap against their requirement and judges only the surfaced weak links — a strong-overlap block triggers no model call, except a block in a `whole_file_claims` file, which is always judged regardless of overlap (its structural coverage rests on a whole-file scope marker, so a high lexical score there must not buy a free pass)
+- [x] `p1` - A block with no retrievable requirement, or with too little text to compare, is reported `unjudgeable`, never a silent `covered`; with no `SemanticJudgeFn` wired every weak link is `unjudgeable` and nothing gates
+- [x] `p1` - A model `evidence_quote` absent from the cited code sets `evidence_ok=false`; a raising or malformed judge reply degrades that one finding to `unjudgeable` without sinking the assessment
+- [x] `p1` - Files a human declared in the coverage report's `excluded[]` are skipped; a report lacking the `excluded[]` / `whole_file_claims[]` fields yields an empty scope (skip nothing, prioritise nothing), never an error

architecture/features/traceability-validation.md

@@ -21,6 +21,7 @@
   - [List ID Kinds](#list-id-kinds)
   - [Validate TOC](#validate-toc)
   - [TOC Utilities](#toc-utilities)
+  - [Document Index](#document-index)
   - [Markdown Parsing Utilities](#markdown-parsing-utilities)
   - [Fixing Prompt Enrichment](#fixing-prompt-enrichment)
   - [Headings Contract Validation](#headings-contract-validation)
@@ -398,11 +399,12 @@ Catches structural and traceability issues that AI agents miss or hallucinate 
    2. [x] - `p1` - Generate expected TOC from headings - `inst-toc-generate-expected`
    3. [x] - `p1` - Compare existing vs expected: check anchor validity, heading coverage, staleness - `inst-toc-compare`
    4. [x] - `p1` - **IF** mismatch, record error with diff details - `inst-toc-if-mismatch`
-4. [x] - `p1` - **RETURN** JSON: `{status, files_checked, errors}` - `inst-toc-return`
+4. [x] - `p1` - **RETURN** JSON: `{status, files_validated, error_count, warning_count, results}`, each `results[]` entry `{file, status, error_count, warning_count}` plus `errors`/`warnings` arrays when `--verbose` or non-empty - `inst-toc-return`
 
 **Supporting**:
 - [x] - `p1` - Imports and module setup for validate-toc command - `inst-toc-imports`
-- [x] - `p1` - Human-friendly formatter for validate-toc output - `inst-toc-format`
+- [x] - `p1` - Validate a single file, never raising: a missing file or a read failure (permission denied, binary/non-UTF-8 content, a TOCTOU race) is reported as its own ERROR result rather than aborting the whole batch and discarding results already collected for earlier files - `inst-toc-validate-one`
+- [x] - `p1` - Human-friendly formatter for validate-toc output: a WARN-only file prints its warnings the same way a FAIL file prints its errors, not just the bare status - `inst-toc-format`
 
 ### TOC Utilities
 
@@ -416,6 +418,9 @@ Catches structural and traceability issues that AI agents miss or hallucinate 
 4. [x] - `p1` - Insert/update TOC using heading-based insertion (`## Table of Contents`) for kit file generator - `inst-toc-util-insert-heading`
 5. [x] - `p1` - Process file: strip manual TOC, insert marker-based TOC, write if changed - `inst-toc-util-process-file`
 6. [x] - `p1` - Validate TOC: check existence, anchor validity, completeness, staleness - `inst-toc-util-validate`
+7. [x] - `p1` - Parse headings with line numbers, fence-aware and front-matter-aware (skips a leading YAML block so a `#`-prefixed front-matter line is never mistaken for a heading); shared by doc-index and the JIT-retrieval readiness checks, which need section boundaries the plain heading list doesn't carry -- `parse_headings` itself now delegates here, stripping the line number, so both share one fence/heading-match implementation - `inst-toc-util-parse-headings-lines`
+8. [x] - `p1` - Collect JIT-retrieval readiness warnings for a document: gathers all four signals below over *every* heading level, independent of whatever level cap the caller configured for TOC-completeness checking - `inst-toc-jit-readiness-collect`
+9. [x] - `p1` - Compute the four JIT-retrieval readiness signals -- duplicate heading titles (compared case-insensitively, with internal whitespace collapsed and Unicode-normalized, though the original text is still shown in the warning), heading depth jumps, oversized sections (`--max-section-lines`, default 300, validated against non-finite/non-positive input independent of the CLI's own argparse guard), and a missing top-of-file description/frontmatter block -- all warning-only, never errors (see constructorfabric/studio#104) - `inst-toc-jit-readiness`
 
 **Supporting**:
 - [x] - `p1` - Imports, constants, fence tracking, GitHub anchor slug generation - `inst-toc-util-datamodel`
@@ -437,6 +442,43 @@ Catches structural and traceability issues that AI agents miss or hallucinate 
 - [x] - `p1` - Heading-based TOC new-insert branch: inject `## Table of Contents` before first heading when absent - `inst-toc-util-insert-heading-new`
 - [x] - `p1` - TOC validate init: build heading list and expected TOC string before comparison checks - `inst-toc-util-validate-init`
 
+### Document Index
+
+- [x] `p1` - **ID**: `cpt-studio-algo-traceability-validation-doc-index`
+
+**Input**: Markdown file path
+
+**Output**: `cfs doc-index`'s JSON is `{file, cache_hit, total_lines, section_count, sections, section_level, retrieval_section_count, retrieval_sections}` -- every heading's own line range in `sections[]` (`{level, heading, line_start, line_end, summary}`), plus a coarser "one chunk per real section" grouping in `retrieval_sections[]` at an inferred heading level, each with a content hash and a summary slot. The underlying cached index dict additionally carries `schema_version`, `path`, and `etag`.
+
+A cached, read-once-per-file structural index for Markdown JIT retrieval (see
+constructorfabric/studio#104): parsing a file's headings/section boundaries
+happens once, not once per query, until its stat fingerprint (`mtime_ns` +
+size) changes.
+The cache-validity fingerprint is deliberately metadata-only (`mtime` + file
+size via `Path.stat()`), never a content hash — the point of the cache is to
+avoid reading the file at all on a hit, and a content hash would defeat that
+by requiring the read it's meant to save. A build reads the content and
+takes that fingerprint bracketed by a stat snapshot on each side, so the
+fingerprint saved is provably the one that matches what was actually parsed
+even if a write lands in the narrow window during the read.
+
+1. [x] - `p1` - Build a fresh structural index: parse headings + line ranges from current content, compute the stat-based fingerprint, stamp the current schema version - `inst-doc-index-build`
+2. [x] - `p1` - Load a cached index for a file, validated against current stat metadata (no content read on a hit) and against the required-field shape at the current schema version; returns `None` if missing, stale, corrupt, or an incomplete/outdated shape - `inst-doc-index-load`
+3. [x] - `p1` - Persist an index to its cache location atomically (temp file + `os.replace`, so a concurrent reader never observes a torn write); no-ops silently outside a Studio-adapted project - `inst-doc-index-save`
+4. [x] - `p1` - Return the cached index or build-and-cache a fresh one; reports cache hit/miss for benchmarking - `inst-doc-index-get-or-build`
+5. [x] - `p1` - Attach a one-line, LLM-authored summary to a cached section by its `line_start`, for a future per-section-summary caller - `inst-doc-index-annotate`
+6. [x] - `p1` - Infer which heading level represents one retrievable section: the most-recurring level wins over a level that appears only once (however shallow), since PDF-conversion heading levels don't reliably encode true nesting depth — a fixed level assumption silently produces a degenerate mega-section on such documents - `inst-doc-index-infer-level`
+7. [x] - `p1` - Group headings at exactly the inferred level into retrieval sections (off-level headings stay inside whichever section they fall under, never split one apart); hash each section's own text for section-granularity staleness detection - `inst-doc-index-retrieval-sections`
+8. [x] - `p1` - Diff the current file against its last cached build at section granularity: which retrieval sections are unchanged vs. changed, or whether the section count itself changed (a structural change, matched by position not heading text, since duplicate titles are real) - `inst-doc-index-diff-stale`
+
+**Supporting**:
+- [x] - `p1` - Stat-based cache-validity fingerprint (`mtime_ns` + size); resolved from the file's own path, never a content hash - `inst-doc-index-etag`
+- [x] - `p1` - Resolve the cache file location within the Studio directory owning the indexed file, resolved from the file's own path (not the process's working directory) - `inst-doc-index-cache-path`
+- [x] - `p1` - `cfs doc-index` CLI wrapper: parse arguments, build the JSON output payload, reporting a clean error for a missing or unreadable file - `inst-doc-index-cmd`
+- [x] - `p1` - Human-friendly formatter for `cfs doc-index` output - `inst-doc-index-cmd-format`
+- [x] - `p1` - Read a file's content bracketed by an etag snapshot on each side, retrying on mismatch: closes the window where a write between the read and the fingerprint could save stale headings under a fresh-looking etag - `inst-doc-index-stable-read`
+- [x] - `p1` - Re-parse a file's current content into retrieval sections for staleness comparison, and build the `(heading, line_start)` identity pair that disambiguates a duplicate heading title in a diff result - `inst-doc-index-diff-stale-helpers`
+
 ### Markdown Parsing Utilities
 
 - [x] `p1` - **ID**: `cpt-studio-algo-traceability-validation-parsing-utils`