Problem
In the arrow row filter (PredicateConverter, crates/iceberg/src/arrow/reader/predicate_visitor.rs), when a predicate references a column that is not present in the parquet file (schema evolution: the column was added after the file was written), the comparison arms fall back to inconsistent constants:
less_than / less_than_or_eq → build_always_true() (~L416, ~L436 at 2768ccf8)
greater_than / greater_than_or_eq / eq / not_eq → build_always_false()
All carry the same comment "A missing column, treating it as null" — but under null semantics every comparison against a missing (all-null) column is non-matching, so the correct constant is always_false for all six.
References
- Spec, Column Projection: a field id not present in the data file reads as null (or its initial-default).
- iceberg-java
ParquetMetricsRowGroupFilter#lt (and every other comparison): valueCount == null → "the column is not present and is all nulls" → ROWS_CANNOT_MATCH.
Impact
always_true only over-keeps rows, so engines that re-apply the predicate (RisingWave, DataFusion) still return correct results and merely lose pruning. A consumer relying on the parquet-level row filter being exact gets wrong results for < / <= on evolved columns: every row of every old file passes.
Suggested fix
Change the less_than / less_than_or_eq fallbacks to build_always_false(), and document whether the row filter is exact or advisory for unresolvable references (see the variant handling, where present-but-unresolvable columns are conservatively kept — matching java's ParquetMetricsRowGroupFilter#notNull variant carve-out "Leave these type filters to be evaluated post scan").
Problem
In the arrow row filter (
PredicateConverter,crates/iceberg/src/arrow/reader/predicate_visitor.rs), when a predicate references a column that is not present in the parquet file (schema evolution: the column was added after the file was written), the comparison arms fall back to inconsistent constants:less_than/less_than_or_eq→build_always_true()(~L416, ~L436 at2768ccf8)greater_than/greater_than_or_eq/eq/not_eq→build_always_false()All carry the same comment "A missing column, treating it as null" — but under null semantics every comparison against a missing (all-null) column is non-matching, so the correct constant is
always_falsefor all six.References
ParquetMetricsRowGroupFilter#lt(and every other comparison):valueCount == null→ "the column is not present and is all nulls" →ROWS_CANNOT_MATCH.Impact
always_trueonly over-keeps rows, so engines that re-apply the predicate (RisingWave, DataFusion) still return correct results and merely lose pruning. A consumer relying on the parquet-level row filter being exact gets wrong results for</<=on evolved columns: every row of every old file passes.Suggested fix
Change the
less_than/less_than_or_eqfallbacks tobuild_always_false(), and document whether the row filter is exact or advisory for unresolvable references (see the variant handling, where present-but-unresolvable columns are conservatively kept — matching java'sParquetMetricsRowGroupFilter#notNullvariant carve-out "Leave these type filters to be evaluated post scan").