Skip to content

Row filter casts literals down to the file's physical type; out-of-range literals on promoted columns silently drop rows #231

Description

@yuhao-su

The arrow row filter compares in the file's physical type, not the table type: try_cast_literal (crates/iceberg/src/arrow/reader/predicate_visitor.rs, fn try_cast_literal) casts the bound literal down to left.data_type() with arrow's default safe cast, so a literal outside the file type's range becomes a null scalar, the comparison yields null for every row, and the row filter drops them all.

Reproducer shape (type promotion, spec "Schema Evolution"): a column written as int and later promoted to long; an old file still stores int32.

predicate on the promoted long column expected on the old file actual
d < 3000000000 every row (all int32 values satisfy it) none
d <= 3000000000 every row none
d != 3000000000 every row none
d > 3000000000, d = 3000000000, d IN (3000000000) none none (correct by accident)

Bind already folds literals that exceed the table type into AboveMax / BelowMin (Predicate::bind, PrimitiveLiteral::AboveMax), so the gap is only the file type being narrower than the table type: int → long, float → double, decimal precision widening. Java evaluates residuals on the record after promotion, i.e. in the table type (BaseParquetReaders promotes on read; Evaluator compares Literal values of the bound type).

Fix options:

  • Compare in the table type: when the projected column's type differs from the field's table type (the Promote column source), cast the column up before applying the kernel, and never cast the literal down. Cheap (the filter batch is small) and consistent with projection.
  • Or keep casting the literal down but with safe: false and fold an out-of-range literal into the constant answer the operator has for it (the AboveMax / BelowMin logic from bind, applied per file). More code, same result.

Pre-existing; noticed while restructuring missing-column evaluation (#221 / #230) and deferred from that work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions