fix(cubestore): stop reporting remote files nothing references

[waralexrom]

Summary

Two paths log an ERROR for a file that was removed legitimately. The startup warmup takes one metastore snapshot and then walks it one download at a time, so on a node with many partitions the pass runs long enough that whatever compaction replaces meanwhile is already gone from remote storage by the time the pass asks for it. And GCTask::RemoveRemoteFile logs an error whenever the object is already gone, which on GCS is a 404 where S3 answers 204. Neither says anything is wrong, and both drown out the errors that do.

The warmup carried a TODO for exactly this — "propagate 'not found' and log in debug mode. Compaction might remove files, so they are not errors most of the time."

Changes

• CubeErrorCauseType::FileNotFound for an absent remote object, so a caller can tell it from a remote it could not reach. It still counts as corrupt data for is_corrupt_data(), and it keeps the name and index of CorruptData everywhere it leaves the process: the wire, which a node of any version has to be able to read, and Display, whose output scheduler matches on to deactivate a table whose import job failed. A listing stays the only classifier, since object stores answer 404 both for a missing key and for a missing bucket, and mistaking the second for the first would deactivate every table. A listing that fails leaves the download error to speak for both rather than replacing it.
• GCS reports a delete of an already absent object as done, the way S3's 204 does, and still drops the local copy. A missing bucket carries the same reason code and the client crate keeps the detail private, so the two cannot be told apart there; a bucket that is not there fails every upload and download loudly anyway.
• The warmup reports what it can actually tell: a file that is not there is a file it did not warm, at debug, with cs.warmup.missing to say how far behind its snapshot the pass ran. Every other failure stays an error, now naming the file. Whether such a file is merely replaced or really lost is not something the pass can judge — the query that needs it reports it anyway, and by then there is no snapshot to be behind.

Testing

• cargo test -p cubestore --lib — 319 pass.
• New tests: three pin the wire and Display compatibility of the new cause, since both have consumers that would fail quietly; two pin the GCS detection against the payload a bucket really answers with, because the reason mapping lives in the client crate and a change there would bring the flood back silently.

Heads up for CI: metastore::tests::delete_old_snapshots and sql::tests::decimal_partition_pruning flake on repeated parallel runs without this branch too.

[claude]

Claude finished @waralexrom's task in 5m 35s —— View job

---

Review: no blocking issues — 0 high, 1 medium, 3 low (4 inline comments)

The compatibility story checks out: I verified scheduler/mod.rs:903 matches on the "CorruptData" substring of to_string(), which external_repr() preserves, and get_partitions_out_of_queue / get_chunks_out_of_queue (metastore/mod.rs:3915-3943) really do skip ids they no longer hold — so the deleted-row-vs-unreachable-metastore distinction the batching relies on is sound.

Full review — findings, verification notes, and checklist

Checklist

• Gather context (diff, PR body, repo instructions)
• Verify the wire/Display compatibility claim against actual consumers
• Verify the stale-vs-active classification against the metastore row semantics
• Review the GCS delete change
• Review test coverage
• Post inline comments and summary

Verification notes

The stale/active classification is correct, including a subtlety worth recording. Recheck-by-id is only valid because a row's file name is stable: suffix is assigned once in Partition::new/new_child/Chunk::new and never mutated (only active, main_table_row_count, file_size, warmed_up, last_used change). If suffix were mutable, an active row could legitimately name a different file than the one the pass failed to fetch, and every such row would be reported as missing data.

has_main_table_file() is also the right predicate rather than is_active(), and not by coincidence — it's active && main_table_row_count != 0, exactly what get_full_name() gates the download on. So the check asks "does the row still name this file", not "is the row alive", which is what the doc comment claims.

stale = absent.len() - active cannot underflow: active_* are filtered subsets of the same vectors.

GCS. Treating an absent object as a completed delete matches S3's 204 and still drops the local copy. The missing-bucket conflation is real but correctly argued away in the PR body — a wrong bucket fails every upload and download loudly.

Tests. Good coverage of the parts that are easy to get wrong later: byte-equality of the flexbuffer against corrupt_data, the Display rendering the scheduler greps, and a fixture of the payload GCS really returns so a reason-mapping change in the client crate breaks a test instead of the scheduler. Driving the classification through real CompactionService::compact rather than hand-deactivating rows is the right call.

Gaps: report_absent_warmup_files itself is untested (the per-batch logging, the metric split, and the ABSENT_BATCH flush at partition boundaries all go unexercised), as is the queue.rs file_not_found path. Not blocking, but the flush is the one bit of arithmetic in the change. Minor: chunk_id + 1000 as a stand-in for a nonexistent id is fragile — u64::MAX would be unambiguous.

Findings

Sev	Where	Issue
Medium	cluster/mod.rs:2193-2200	A failed recheck discards the batch and warns without naming any ids — and this runs at startup, when the RPC to the router is most likely to blip
Low	remotefs/queue.rs:294	? on the listing probe masks the original download error; now load-bearing for the classification
Low	cluster/mod.rs:2161	The trailing flush is skipped by the two is_cancelled() returns; the comment only reads correctly if you know that
Low	cluster/mod.rs:2173	The surviving error line doesn't name the file; warmup_file_is_absent reads as a pure predicate but logs

Not verified

I did not compile or run the test suite — cubestore has no prebuilt target/ in this checkout and a cold cargo check on this dependency tree exceeds the time available here. The reasoning above is from reading the code and its consumers; treat the author's reported cargo test -p cubestore --lib result as the build signal.

Security: nothing relevant — no new inputs, no auth paths, no user-controlled data. Performance: two extra out-of-queue metastore reads per 256 absent files, against one download each, so negligible; memory is bounded by the batch plus one partition's chunks.

· branch `cubestore-fix-warmup-error-log`

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.37%. Comparing base (528d755) to head (6e3a78b).
⚠️ Report is 55 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (528d755) and HEAD (6e3a78b). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (528d755)	HEAD (6e3a78b)
cubesql	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #11601       +/-   ##
===========================================
- Coverage   79.58%   59.37%   -20.22%     
===========================================
  Files         480      229      -251     
  Lines       99544    18247    -81297     
  Branches     3636     3682       +46     
===========================================
- Hits        79224    10834    -68390     
+ Misses      19801     6861    -12940     
- Partials      519      552       +33     

Flag	Coverage Δ
cube-backend	59.37% <ø> (+0.09%)	⬆️
cubesql	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude finished @waralexrom's task in 3m 41s —— View job

---

Re-review: no blocking issues — 0 high, 1 medium, 2 low (1 inline comment; all 4 prior threads resolved as addressed/obsolete)

The change is much smaller than when I last looked: the batched metastore recheck is gone (6e3a78b), and warmup now just downgrades a FileNotFound to debug + a counter. That removes every prior finding.

Full review — checklist, verification notes, findings

Checklist

• Gather context (diff vs 528d755, prior review threads)
• Re-check the four prior findings against the current diff
• Review the simplified warmup path
• Review the CubeErrorCauseType::FileNotFound compatibility shim
• Review the GCS delete change and the queue classification
• Resolve stale threads, post inline comment and summary

Prior threads — all resolved

Prior finding	Status
Failed recheck discards a batch without naming ids	Obsolete — report_absent_warmup_files/AbsentWarmupFiles/ABSENT_BATCH all removed
? on the list() probe masks the download error	Fixed — queue.rs:291-306 now matches, warns on a failed probe, and returns the original err
Error line doesn't name the file; predicate that logs	Fixed — cluster/mod.rs:2160 logs "Warmup of {} failed: {:?}"; report_warmup_download is now honestly named
Flush comment vs. the two is_cancelled() returns	Obsolete — no flush left

Verification notes

Compatibility shim (lib.rs:76-99) checks out. I re-verified the two consumers: scheduler/mod.rs:903 greps to_string() for "CorruptData", and metastore/mod.rs:5178 branches on is_corrupt_data(). external_repr() keeps index 2 / name "CorruptData" for both CorruptData and FileNotFound, Display now renders through it instead of {:?}, and is_corrupt_data() covers both — so both consumers see exactly what they saw before. grep confirms no code anywhere matches on CubeErrorCauseType by value, so the derived PartialEq treating the two as distinct can't affect a live branch. The hand-written Serialize also correctly preserves the old indices for WrongConnection (3) and Panic (4) despite FileNotFound being declared last.

Worth stating explicitly since the tests pin it: the cause does not survive the wire — file_not_found_travels_as_corrupt_data asserts the round-trip lands on CorruptData, so is_file_not_found() is only meaningful in the process that produced the error. Fine here (warmup is local), but it constrains any future remote use.

The FileNotFound really does reach the warmup. config/mod.rs:2814 registers dyn RemoteFs → QueueRemoteFs, and ClusterImpl takes Arc<dyn RemoteFs> via get_service_typed(), so warmup_select_worker's downloads go through the one path that classifies. Not obvious from the diff alone.

GCS. is_absent_object is used only in delete_file, which still drops the local copy on the absent branch — matching S3's 204. The missing-bucket conflation is correctly argued away in the code comment. I could not verify cloud_storage::Error::Google / errors_has_reason / Reason::NotFound against the crate source (0.7.0 isn't vendored in this checkout), so the two new tests rest on the author's local run.

Findings

Medium — the new metric can't distinguish the case worth alerting on (app_metrics.rs:151). Its doc says the count "says how far behind its snapshot a pass ran rather than that anything is wrong", but cs.warmup.missing increments identically for a file compaction legitimately replaced and for a file an operator or a lifecycle rule really deleted. The earlier revision split this into .stale/.active precisely so the second was alertable. Dropping the split is a defensible simplification — a genuinely missing file still surfaces when a query touches it and deactivates the table — but the metric is then a rate-of-churn gauge, not a health signal, and the doc comment reads as if it were the latter. Worth either saying so in the comment or adding a stale/active tag.

Low — the classification is not pinned by a test. Posted inline on queue.rs:297: the two tests that reach the new constructor still assert only is_corrupt_data(), which was already true before this PR, so nothing fails if it regresses to corrupt_data(...).

Low — the PR description no longer matches the code. It still describes the batched 256-file metastore recheck, cs.warmup.missing.stale / cs.warmup.missing.active, and "a test that drives a table through compaction" — none of which are in the final diff (there is one untagged cs.warmup.missing, and no cluster test). Since squash-merge takes the body as the commit message, it's worth trimming to what shipped.

Pre-existing, not introduced here: list(file) is a prefix listing, so a sibling key sharing the requested name as a prefix would read as "the file exists" and suppress the FileNotFound. Unchanged behaviour, and chunk/partition names make a collision unlikely.

Security: nothing relevant — no new inputs, no auth paths. Performance: two extra String clones per warmup download, against a network fetch each; the removed metastore round-trips make this strictly cheaper than the previous revision.

Not verified

I did not compile or run the suite — no prebuilt target/ here and a cold cargo check on this dependency tree exceeds the time budget. Treat the author's reported cargo test -p cubestore --lib as the build signal; note the new gcs.rs tests are the ones most likely to be version-sensitive.

· branch `cubestore-fix-warmup-error-log`

[claude]

This is the only place in the tree that ever constructs a FileNotFound, so the whole warmup change hangs off it — but the two tests that exercise it (queue_download_missing_file at L620, queue_download_wrong_file_size at L660) both still assert only is_corrupt_data(), which was already true before this PR. Nothing would fail if a later refactor swapped this back to corrupt_data(...); the warmup would just quietly return to logging an ERROR per compacted file.

Tightening the two existing asserts pins it at zero cost:

// queue_download_missing_file
Err(e) => assert!(e.is_file_not_found()),

// queue_download_wrong_file_size — a truncated file is present, not absent
Err(e) => assert!(e.is_corrupt_data() && !e.is_file_not_found()),

Fix this →