Skip to content

Row filter: </<= on a column missing from the file keeps all rows instead of matching none #221

Description

@yuhao-su

Problem

In the arrow row filter (PredicateConverter, crates/iceberg/src/arrow/reader/predicate_visitor.rs), when a predicate references a column that is not present in the parquet file (schema evolution: the column was added after the file was written), the comparison arms fall back to inconsistent constants:

  • less_than / less_than_or_eqbuild_always_true() (~L416, ~L436 at 2768ccf8)
  • greater_than / greater_than_or_eq / eq / not_eqbuild_always_false()

All carry the same comment "A missing column, treating it as null" — but under null semantics every comparison against a missing (all-null) column is non-matching, so the correct constant is always_false for all six.

References

  • Spec, Column Projection: a field id not present in the data file reads as null (or its initial-default).
  • iceberg-java ParquetMetricsRowGroupFilter#lt (and every other comparison): valueCount == null → "the column is not present and is all nulls" → ROWS_CANNOT_MATCH.

Impact

always_true only over-keeps rows, so engines that re-apply the predicate (RisingWave, DataFusion) still return correct results and merely lose pruning. A consumer relying on the parquet-level row filter being exact gets wrong results for < / <= on evolved columns: every row of every old file passes.

Suggested fix

Change the less_than / less_than_or_eq fallbacks to build_always_false(), and document whether the row filter is exact or advisory for unresolvable references (see the variant handling, where present-but-unresolvable columns are conservatively kept — matching java's ParquetMetricsRowGroupFilter#notNull variant carve-out "Leave these type filters to be evaluated post scan").

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions