Monte Carlo coverage studies for bootstrap_ci and simulate_hhi_ci

[soodoku]

src/covered/metrics.py ships two confidence intervals. Neither had ever been
checked against a known truth. This adds tests/test_metrics_coverage.py: real
Monte Carlo coverage studies, every gate taken from
simcheck so its tolerance is derived
from the replicate count rather than chosen by hand.

The cautionary example: three tests that cannot fail

These are the tests that made the intervals look covered.

tests/test_metrics.py:24 test_bootstrap_ci_brackets_mean

lo, hi = metrics.bootstrap_ci([0.4] * 100, n_boot=200, seed=1)
assert lo == pytest.approx(0.4, abs=1e-9)
assert hi == pytest.approx(0.4, abs=1e-9)

Every resample of a constant vector is that constant, so both endpoints are 0.4
by construction. A wrong quantile, a wrong alpha, an off-by-one resample, and
return (lo, lo) all pass it. It is a test of np.mean.

tests/test_metrics.py:30 test_simulate_hhi_no_error_is_deterministic

res = metrics.simulate_hhi_ci(counts, recall=1.0, precision=1.0, n_draws=50, seed=7)

recall=1.0, precision=1.0 is the one branch in which kept = obs and
add_factor = 0, so no simulation runs at all. The three asserted quantiles
are equal because all fifty draws are the same number. It is a test of hhi.

tests/test_metrics.py:38 test_simulate_hhi_with_error_brackets_and_orders

assert res["lo"] <= res["median"] <= res["hi"]

True of any three sorted quantiles of any array whatsoever, including an all-zero
one.

I renamed all three to say what they actually check, added a docstring to each
explaining why the original name over-claimed, and added two real assertions:
test_bootstrap_ci_widens_with_the_confidence_level (same sample, same
resampling seed, so the 95% and 80% intervals must nest strictly — the constant
sample cannot see alpha at all), and a strict lo < median < hi in place of
the non-strict one. Nothing was deleted.

Results

Replicate counts below are from the development runs quoted in the docstrings;
the gates themselves run at simcheck's tier (100 fast / 400 deep), or at a fixed
600 where a pinned failure needs the power. Both tiers were run and both pass.

bootstrap_ci

claim	verdict	measured	nominal	reps
Bernoulli(0.4), n=60	HOLDS	0.961	0.95	2000
Bernoulli(0.4), n=200	HOLDS	0.946	0.95	2000
Poisson(3), n=40	HOLDS	0.946	0.95	2000
alpha=0.20, Bernoulli n=100	HOLDS	0.823	0.80	2000
alpha=0.20, Bernoulli n=200	HOLDS	0.810	0.80	2000
lognormal(0, 1.25), n=25	FAILS	0.819	0.95	2000

The heavy-tailed failure is the headline. The percentile bootstrap is not
second-order accurate for the mean of a skewed distribution — the interval is
centred on the sample mean and inherits its skew instead of correcting for it,
which is the standard motivation for BCa and the bootstrap-t. It is a known
property rather than a defect introduced here, but per-source quote counts in
the CNN corpus are exactly that shape, and a 95% label on an interval that
covers 82% of the time is a claim the paper should not make silently. The deficit
closes only slowly with sample size:

shape                    n=20   n=25   n=40   n=60   n=100  n=200
Bernoulli(0.4)              -      -      -  0.961  0.952  0.946
Normal(0.4, 0.15)           -  0.928  0.934      -  0.931      -
Poisson(3)              0.933      -  0.946      -      -      -
lognormal(0, 1.00)      0.842  0.856  0.885      -      -      -
lognormal(0, 1.25)      0.805  0.819  0.854      -      -      -
lognormal(0, 1.50)      0.755  0.776  0.808      -      -      -

It is pinned the way geoinference pins its known-bad cases: pytest.raises(AssertionError)
around assert_coverage, plus a directional check against binomial_band(0.95, reps)[0].
Also recorded but not gated: the Normal row's uniform 1.7-point deficit, which is
the percentile bootstrap applying no t correction.

simulate_hhi_ci

claim	verdict	measured	nominal	reps
well-covered sources, none can vanish (recall 0.9)	HOLDS	0.945	0.95	1000
long tail, 10 of 100 one-off sources vanish (recall 0.9)	FAILS	0.921	0.95	600
long tail, 30 vanish (recall 0.7)	FAILS	0.892	0.95	600
long tail, 50 vanish (recall 0.5)	FAILS	0.873	0.95	600
tail-biased recall (0.9 headline / 0.6 one-off)	FAILS	0.057	0.95	600

Finding 1: the point correction is arithmetically a no-op. simulate_hhi_ci
keeps each observed event with probability precision and then adds back
(1 - recall) / recall times what it kept, so in expectation it multiplies every
source's count by the same factor precision / recall. HHI is a function of
shares and invariant to a common scale. The corrected median is therefore
hhi(observed), whatever recall and precision are set to — measured on a fixed
five-source market with raw HHI 0.603340:

recall  precision   median      median - raw
0.60    0.80        0.603394    +0.000055
0.80    0.95        0.603773    +0.000434
0.95    0.70        0.603175    -0.000165

The module docstring promises "a corrected series with credible intervals next to
the raw one". The interval is real; the correction is not. This is pinned by
test_the_point_correction_is_a_no_op, which anything that genuinely moved the
point estimate would break — and that is the intended way for it to break.

Finding 2: the case the docstring actually claims is the case it cannot
handle. covered/metrics.py opens by saying missed sources "drop mostly
tail/one-off voices and bias HHI upward". That is a statement about recall
being lower for the tail than for the headline sources — the realistic case, an
extractor that reliably attributes a quote to the President but misses a one-time
caller. simulate_hhi_ci takes a single scalar recall and applies it to every
source alike, and by Finding 1 that rescaling cancels out of HHI. The
correction is structurally incapable of removing the bias its own module
docstring is written to motivate. With headline recall 0.9 and one-off recall
0.6, coverage is 0.057 over 600 replicates, with a mean signed error of +0.0148
against a mean interval width of 0.0156 — the bias is almost the whole width of
the interval. Forty of the hundred one-off sources vanish outright. Pinned.

Finding 3: the weakness the docstring admits is real, monotone, and much
smaller than the docstring implies. "Missed events are conservatively attributed
to already-observed sources, so the correction is a lower bound on tail
diversity." A source missed entirely is not in the input, so nothing in the
simulation can restore it; the imputed market has fewer sources than the real one
and reads more concentrated. Coverage falls monotonically with the number of
sources lost — 0.921 / 0.892 / 0.873 as recall drops through 0.9 / 0.7 / 0.5 and
10 / 30 / 50 of a hundred one-off sources vanish — with the signed error always
positive, exactly as predicted.

But the effect is small, and that is the part worth recording, because it is
not what the module docstring implies. Under a uniform recall the tail is thinned
in the same proportion as the head, so the surviving shares stay nearly unbiased
even though whole sources are gone; HHI reads shares, is dominated by the head,
and a one-off source contributes (1/N)**2 to it. Losing half the one-off
sources moves the point estimate by 0.0004 on an HHI of 0.20. The admitted
weakness bites hard on the diversity measures in covered.hhi (n_distinct,
normalized_entropy), which have no interval at all, and only barely on HHI.

Only the recall-0.5 rung is far enough below nominal for a gate this file can
afford to resolve — 0.873 against a 600-replicate floor of 0.923 is 3.7 standard
errors, while pinning 0.921 would need about 2400 replicates and eighteen minutes
— so that is the rung gated, with the other two recorded in the docstring.

One methodological trap worth knowing about. Coverage of this interval depends
on n_draws. It is a pair of empirical quantiles of n_draws simulated HHIs, and
np.nanquantile's linear interpolation places the nominal 2.5% endpoint at about
the 0.025 + 0.975/n_draws quantile, so the interval spans about
0.95 - 1.95/n_draws of the simulated distribution — 0.948 at the default
n_draws=1000, 0.940 at 199. Verified directly against a standard normal over
40000 trials: 0.9490 and 0.9413. My first pass ran the studies at 199 draws to
save time and measured a systematic ~2-point deficit across fifteen
(recall, precision) settings; at the library default it disappears. The studies
now run at the default, and the module docstring says so, so that nobody reruns
them cheaply and concludes the estimator is broken.

Negative controls

Every claim has a case that must fail.

1. test_a_shrunk_interval_is_caught — the 95% interval shrunk by
   z(0.90)/z(0.975) = 0.654, the derived factor that turns a Gaussian 95%
   interval into an 80% one. Measured 0.790 over 400 replicates. It must fail a
   95% gate and pass an 80% one, and both halves are asserted. Proves the
   coverage gate rejects a too-narrow interval, and — the half that matters — that
   it is not a blanket rejector.
2. test_the_two_nominal_levels_are_distinguishable — the alpha=0.20 study
   judged at 95% must raise, and the alpha=0.05 study judged at 80% must raise.
   Measured 0.810 and 0.946 over 400 replicates each. Proves the two alpha
   tests are not both passing on a band wide enough to admit either answer, so an
   implementation that ignored alpha entirely cannot satisfy both.
3. test_assuming_perfect_extraction_is_caught — simulate_hhi_ci(recall=1.0, precision=1.0), the exact call the vacuous existing test makes, pointed at
   genuinely corrupted counts. It returns a degenerate interval at hhi(observed).
   Measured 0.000. Proves assert_coverage is measuring coverage in the HHI
   studies; if it ever stops failing, every other HHI gate is certifying nothing.
4. test_tail_biased_recall_is_caught_under_covering and
   test_a_vanishing_tail_under_covers — Findings 2 and 3 above, also serving
   as controls. Prove the HHI fixture family is capable of producing a biased
   interval at all, so test_well_covered_sources_cover holding is not an
   artefact of a fixture too easy to fail.
5. test_a_skewed_mean_at_small_n_under_covers — the bootstrap failure,
   likewise. Proves the bootstrap fixture family can produce under-coverage.

Verification

$ uv run pytest
137 passed in 70.30s

$ uv run pytest tests/test_metrics_coverage.py
15 passed in 55.11s          # the new file; 0 failed, 0 skipped

$ uv run pytest tests/test_metrics.py
6 passed in 0.36s            # was 5

$ SIMCHECK_DEEP=1 uv run pytest tests/test_metrics_coverage.py
15 passed                    # 400 replicates instead of 100, ~3 min

$ uv run ruff check src tests     # and `ruff check .`, which CI runs
All checks passed!

$ uv run ruff format --check src tests
38 files already formatted

$ uv run pyright src
0 errors, 0 warnings, 0 informations

$ uv lock --check
Resolved 122 packages

$ uvx zizmor --min-severity high .github/workflows/
No findings to report.

The whole suite was 121 passing before this branch; it is 137 now — 15 from the
new file and 1 new assertion in tests/test_metrics.py. Nothing skips.

pyproject.toml gains simcheck in the test dependency group as a git reference,
matching geoinference's pattern — PyPI rejects direct URL references in published
metadata, so it must not go in a published extra. uv.lock is committed;
regenerating it also picked up pandera 0.31.1 -> 0.32.1 and typer 0.26.7 -> 0.27.1, which pyproject.toml on main already required (pandera>=0.32.1,
typer>=0.27.0) but the lock had not been refreshed for — so uv sync --frozen,
which CI runs, was already inconsistent on main.

pydoclint is not run: it fails to import in this venv (ImportError: cannot import name 'DocstringYields' from 'docstring_parser.common'), the pre-existing
docstring_parser conflict that .github/workflows/ci.yml documents and disables
with run-pydoclint: false.

🤖 Generated with Claude Code

[soodoku]

Note on the one red check: `ci / dependency-review` fails with

Dependency review is not supported on this repository. Please ensure that Dependency graph is enabled

That is a repository setting (Dependency graph is off for `notnews/covered`), not anything this branch introduced — PRs #11, #12 and #13 all fail the same check in 4-5 seconds with the same error. Every other check passes: lint, pyright, the built-wheel smoke test, workflow-security, docs, and the test matrix on Python 3.11/3.12/3.13.