Volume API and index stats over-report after log deletion; log queries stay correct

[V2arK]

Describe the bug

After the compactor processes a delete request (filter-and-delete), /loki/api/v1/index/volume and /loki/api/v1/index/stats over-report the affected streams for hours, while log/metric queries (bytes_over_time etc.) remain exact throughout. A second delete against the same stream selector roughly doubles the namespace's index references, inflating even query windows that are disjoint from every deleted range. The inflation survives pod restarts and does not converge within hours.

This matters to anyone composing the Volume API with the delete API for per-tenant/per-namespace usage accounting: after the first delete, volume readings for that stream set can no longer be trusted.

To Reproduce

Loki 3.7.4, single binary, filesystem object store, TSDB v13 schema, auth_enabled: false. Compactor settings tightened only to shorten the wait: compaction_interval: 2m, delete_request_cancel_period: 1m, retention_delete_delay: 5m.

1. Push ~30 MB of synthetic logs (3 streams, one namespace label value, timestamps spread over the past 6 h), then POST /flush.
2. Baseline agrees: instant volume query reports 29.95 MB vs 30.18 MB ground truth from sum(bytes_over_time(...[7h])) (−0.8%).
3. POST /loki/api/v1/delete for the oldest 3 h of that namespace; wait until the request lists as processed.
4. bytes_over_time immediately reports the expected 14.84 MB (exact). The volume query for the same selector/window reports 68.1 MB, decays stepwise (39.9 → 6.6 MB over ~10 min), then sticks at ~1.9× truth. Restarting the pod does not clear it (it read 10.6 MB after a restart). index/stats at that point returns 17 streams / 16,201 entries where the actual remaining data is 3 streams / 6,946 lines.
5. Issue a second delete for part of the remaining range. After it processes, index references for the whole namespace roughly double (2,359 chunks / 11.16 M entries vs a clean-baseline 1,176 / 5.99 M), and a volume query restricted to a window disjoint from both deleted ranges reports ~2.03× its bytes_over_time truth, flat for 14+ minutes.

A larger second dataset (3.018 GB, 6 streams) reproduced the same shape: pre-delete volume/truth ratio 1.017; after one delete a window past the deleted range stayed byte-stable; after a second delete the whole namespace's refs doubled.

Expected behavior

Volume/stats either apply delete-request filtering the way log queries do, or the docs state that these endpoints are not deletion-aware.

Root cause (from reading v3.7.4 source; code paths unchanged on main as of 2026-08-05)

1. SelectLogs/SelectSamples inject pending (and recently completed) delete requests as query-time filters (pkg/querier/querier.go), which is why log queries stay exact through the index-cleanup transient. IndexStats and Volume in the same file take no delete parameters at all — the transient is exposed raw.
2. The querier's volume also sums the ingester's in-memory contribution (pkg/ingester/instance.go, unflushed chunks), which is delete-blind and WAL-durable — hence restart-proof inflation that decays as chunks flush.
3. In single-binary, the index view merges the compactor-rewritten object-store index with ingester-uploaded head TSDBs (indexshipper iterates the uploads manager too), which the compactor's delete processing never rewrites and which are retained for IndexCacheValidity + 3×ResyncInterval; the volume aggregation (ChunkStats summing ChunkMeta.KB, with pre-aggregated page markers) has no chunk-identity dedup across the merged sources, so rewritten chunks and their stale predecessors are counted together.

Environment

• Infrastructure: Kubernetes (k3d), grafana-community helm chart 18.5.2
• Deployment tool: helm
• Loki 3.7.4 (also inspected current main), single binary, filesystem object store, TSDB v13

I'm not asking for a commitment to fix — filing so the behavior is documented and searchable, since we could find no existing issue covering it. Happy to provide the full reproduction scripts.

[mehrdadbn9]

Opened #23866 for the documentation half of this — a caveat on index/stats and index/volume saying they are not deletion-aware, with a pointer to bytes_over_time for accounting that has to be exact.

I stopped there deliberately. Your root-cause read matches what I see on main: SelectLogs/SelectSamples pass deletes through deletion.DeletesForUserQuery, IndexStats/Volume take no delete parameters at all, and Volume fans out to the ingesters on top of that. But the other half of the ask — filtering these endpoints by delete request, or deduplicating chunk identity across the compactor-rewritten index and the retained ingester-uploaded head TSDBs — is a behaviour change I would rather not guess the shape of. Leaving that to a maintainer.

Thanks for the reproduction detail, it made the code paths easy to confirm.

[yashraj4]

The root cause is clear from the code paths you identified - SelectLogs/SelectSamples get delete filtering via deletion.DeletesForUserQuery, but IndexStats/Volume don't.

For the fix side, plumb the same delete parameters into IndexStats and Volume so they apply the same filters the log queries already do. The ingester-side inflation is trickier since it's in-memory, but at least the index-side overcount should be addressable.

@V2arK are you planning to work on this, or would you want someone to pick it up? @mehrdadbn9 since you already have the docs PR going, curious if you've looked at the code side too.

[V2arK]

The root cause is clear from the code paths you identified - SelectLogs/SelectSamples get delete filtering via deletion.DeletesForUserQuery, but IndexStats/Volume don't.

For the fix side, plumb the same delete parameters into IndexStats and Volume so they apply the same filters the log queries already do. The ingester-side inflation is trickier since it's in-memory, but at least the index-side overcount should be addressable.

@V2arK are you planning to work on this, or would you want someone to pick it up? @mehrdadbn9 since you already have the docs PR going, curious if you've looked at the code side too.

I can pick up

[V2arK]

I picked this up as promised. Two updates: a PR for the delete-awareness half, and results from a deeper verification pass that corrects part of my original root-cause analysis.

PR for the delete-awareness defect

#24556 plumbs delete requests into IndexStats and Volume the same way SelectLogs/SelectSamples already receive them (deletion.DeletesForUserQuery in the querier, a new deletes field on the request protos, applied down at the TSDB index and ingester levels). Whole-stream deletes are applied exactly (full-covered chunks dropped, partial overlap prorated with the same GetFactorOfTime estimation these endpoints already use); deletes with line filters cannot be evaluated from index metadata and are skipped. E2E on a single binary: with a pending (unprocessed) whole-stream delete of the first 2h of a 6h/30 MB dataset, volume reports 20.56 MB vs 20.40 MB ground truth (same 1.008 ratio as the no-delete baseline), where unpatched 3.7.4 keeps reporting the full amount.

Verification of the other mechanisms (3.7.4, code identical on main at b645ac3)

The remaining over-reporting turned out to have three independent causes. Points 1 and 3 below confirm and sharpen my original analysis; point 2 corrects it.

1. Cross-source duplicate counting — confirmed, with exact arithmetic. MultiIndex.Stats/Volume (multi_file_index.go:373,379) fan every sub-index into one shared accumulator with no chunk-identity dedup (contrast Series, which dedups by fingerprint). The same chunks are routinely present in three sources at once: the in-memory head kept serving after rotation, the locally retained uploaded TSDB (indexShipper.ForEach iterates the uploads manager as well as downloads), and the downloaded compacted file. Measured on a fresh instance: baseline 30.841 MB → 61.682 MB (exactly 2×) once the built TSDB and the compacted file coexist → 92.522 MB (exactly 3×) during the prev-head window. Dumping the index files shows byte-identical chunk checksum sets (19/19) across the copies. After a delete is processed the duplication multiplies against the shrunken truth, which is where ratios like 6–12× come from.

2. The "restart-proof smear" is the results cache, not the ingester — correcting my original point 2/3. The post-processing behavior where fully-deleted windows report nonzero volume and disjoint windows stay inflated is produced by the volume/stats results caches, which are on by default (cache_volume_results / cache_index_stats_results, embedded in-process cache, 1h default validity). Their Extract implementations prorate a cached extent linearly by time ("Extract favors the ability to cache over exactness of results. It assumes a constant distribution..."), so a stale pre-delete extent answers any sub-window with total × window/extent: deleted windows get fabricated bytes, windows holding all remaining data under-report, and the distribution smears uniformly over the original span. Cache-busted probes over identical windows are exactly correct (deleted windows 0, full window at 1.008 ratio) while plain probes smear — and a restart "fixes" totals only because it clears the in-process cache. My original attribution of this to unflushed ingester chunks / WAL durability was wrong.

3. New finding: a restart can permanently resurrect deleted index entries. If the process restarts while the pre-delete built TSDB is still in tsdb-shipper-active/multitenant/ (it is retained ~10 min after upload), the uploads manager re-uploads it and the compactor merges it back into the compacted index — and since the delete request is already processed, it is never re-applied. Verified end to end: after the restart, the compacted index again contains chunk entries from inside the processed delete range while the chunk objects themselves are gone. The result is a permanent overcount for stats/volume plus dangling chunk references (we also observed bytes_over_time under-reporting in this state, consistent with queries hitting index entries whose chunks no longer exist). This one affects deletion correctness beyond the volume API — happy to split it into its own issue if that's preferred.

Repro details for all three are scripted (synthetic 3-stream dataset, cache-busting probes, TSDB index dumps reconciling live readings against per-file chunk sets); I can attach them if useful.