Bound the cycle, not just the index, in pg_index_bloat (#2617)

[erikdarlingdata]

Found by dogfooding v99 on the Aurora target. This collector had never returned a single row since it shipped — rows_ever = 0, failing every cycle with Exception while reading from stream.

I bounded the wrong dimension

pgstatindex reads every page of the index it is pointed at — that is what makes it a measurement rather than an estimate, and why #2561 chose it over an estimator that returns nothing under pg_monitor.

MeasureCeilingBytes bounds one index. Nothing bounded the statement:

indexes_tracked = 1517
total_bytes     = 461 GB

One query, 461 GB of index pages, default command timeout. It never finished and the connection dropped mid-read — which is also why it surfaced as an unclassified stream failure rather than a named miss.

The fix

A cycle work budget: measure the 200 largest indexes per run.

A count, not a byte cap. The cost is per page and index sizes are wildly uneven, so a byte cap would measure three indexes on one server and four hundred on another with no way for an operator to predict which. A count is legible, and largest-first means the 200 measured are where bloat is actually worth reclaiming.

Everything past the budget is still returned, carrying skipped_reason:

not measured this cycle (work budget): pgstatindex reads every page, so only the largest 200 indexes are measured per run. This one is recorded at its size so it is never mistaken for healthy.

An index missing from the result reads as one that does not exist. An index present with a stated reason cannot be mistaken for healthy — the same argument that put skipped_reason on the size ceiling.

The budget gates the LATERAL, not just the label. Gating only the reason would name the rows correctly while still reading every page — which would have fixed nothing.

CommandTimeoutSecondsOverride = 300 so a slow single index yields a classified timeout instead of a dropped connection; index_object_stats took the same override for the same reason (#1135).

Verified

Against 212 real btree indexes:

measured (pgstatindex ran):  200
returned-but-not-measured:    12

Worth saying plainly

The rig that verified #2561 had two indexes on one table. The measurement logic was correct and the question how much work is this in total never arose, because the answer was none. That is the class of defect a container structurally cannot show you, and it is the argument for this dogfood pass.

[claude]

The work budget is deterministic and largest-first with no rotation across cycles: size_rank is recomputed from index_bytes DESC every run, so an index that lands at rank 201+ today will land there again tomorrow (index sizes don't reorder that fast). For any database with more candidate btree indexes than the budget, everything past rank 200 is not merely "not measured this cycle" as the message says — it is never measured, permanently, until enough of the top 200 shrink or get dropped.

That compounds with the reader/UI, which this PR doesn't touch but whose assumptions this collector now breaks:

• DarlingPgIndexBloatReader.PgIndexBloatSql sorts (skipped_reason IS NOT NULL) DESC first — written when skip was the rare over-20GB-ceiling case, so surfacing it first made sense ("skipped indexes sort to the top, not out of sight").
• ViewerServerTab.Postgres.cs calls it with PgGridRowLimit = 200 — the exact same number as BudgetLiteral.

So on any database with more than ~400 qualifying btree indexes, every skipped row (i.e. every index below rank 200, always the same ones) sorts ahead of every measured row, and the 200-row grid limit is entirely consumed by "not measured this cycle" placeholders. The actually-measured bloat data for the top 200 largest indexes — the whole point of this collector — never appears in the UI at all for those databases, and the smaller/mid-sized indexes where bloat is easiest to miss are permanently invisible too.

Worth considering a rotating or sampled budget (e.g. offset by collection cycle, or a random sample from beyond the top N) so coverage eventually reaches every index, and/or fixing the skipped_reason wording so it doesn't promise something ("this cycle") the current implementation can't deliver.

[claude]

Reviewed. The fix itself is sound: bounding the statement (not just the per-index ceiling) is the right diagnosis for the 461 GB / 1,517-index Aurora failure, LEFT JOIN LATERAL gating on size_rank <= 200 correctly keeps pgstatindex from being invoked on skipped rows (matches the existing ceiling pattern, and the PR's own 212-index verification confirms it empirically), the CommandTimeoutSecondsOverride = 300 mirrors the established index_object_stats (#1135) precedent and is honored uniformly by both DarlingCollectorRunner and Lite's RemoteCollectorService.DefinitionRunner, and the new tests pin the behavior that would have caught this originally.

Lite/Darling parity: no drift. PgIndexBloatCollector lives in the shared PerformanceMonitor.Collectors project and both apps run it through the same code path; Lite's DuckDB schema is generated from PayloadColumns (unchanged shape — skipped_reason already existed), so nothing needed updating on that side. Lite has no UI surface for pg_index_bloat at all currently, so this is a Darling-only concern.

One substantive design issue, left as an inline comment on PgIndexBloatCollector.cs: the budget is deterministic and largest-first with no rotation between cycles, so any index ranked below 200 in a database is not "not measured this cycle" as the message states — it's never measured, permanently, absent size churn. Combined with DarlingPgIndexBloatReader's "skipped sorts first" ordering (written when skip was the rare over-ceiling case) and ViewerServerTab.Postgres.cs's PgGridRowLimit = 200 — the same number as the budget — any database with more than ~400 qualifying btree indexes would have its entire 200-row grid consumed by skip placeholders, with the real measured bloat data (the point of the collector) never surfacing in the UI. Worth a rotation/sampling strategy or at least fixing the wording before this ships to a fleet with databases that large.

No SQL injection, secrets, or missing-index-DMV concerns; T-SQL style guide doesn't apply here (Postgres collector), and the embedded-SQL conventions used match the rest of the file.