The arrow row filter compares in the file's physical type, not the table type: try_cast_literal (crates/iceberg/src/arrow/reader/predicate_visitor.rs, fn try_cast_literal) casts the bound literal down to left.data_type() with arrow's default safe cast, so a literal outside the file type's range becomes a null scalar, the comparison yields null for every row, and the row filter drops them all.
Reproducer shape (type promotion, spec "Schema Evolution"): a column written as int and later promoted to long; an old file still stores int32.
predicate on the promoted long column |
expected on the old file |
actual |
d < 3000000000 |
every row (all int32 values satisfy it) |
none |
d <= 3000000000 |
every row |
none |
d != 3000000000 |
every row |
none |
d > 3000000000, d = 3000000000, d IN (3000000000) |
none |
none (correct by accident) |
Bind already folds literals that exceed the table type into AboveMax / BelowMin (Predicate::bind, PrimitiveLiteral::AboveMax), so the gap is only the file type being narrower than the table type: int → long, float → double, decimal precision widening. Java evaluates residuals on the record after promotion, i.e. in the table type (BaseParquetReaders promotes on read; Evaluator compares Literal values of the bound type).
Fix options:
- Compare in the table type: when the projected column's type differs from the field's table type (the
Promote column source), cast the column up before applying the kernel, and never cast the literal down. Cheap (the filter batch is small) and consistent with projection.
- Or keep casting the literal down but with
safe: false and fold an out-of-range literal into the constant answer the operator has for it (the AboveMax / BelowMin logic from bind, applied per file). More code, same result.
Pre-existing; noticed while restructuring missing-column evaluation (#221 / #230) and deferred from that work.
The arrow row filter compares in the file's physical type, not the table type:
try_cast_literal(crates/iceberg/src/arrow/reader/predicate_visitor.rs,fn try_cast_literal) casts the bound literal down toleft.data_type()with arrow's default safecast, so a literal outside the file type's range becomes a null scalar, the comparison yields null for every row, and the row filter drops them all.Reproducer shape (type promotion, spec "Schema Evolution"): a column written as
intand later promoted tolong; an old file still storesint32.longcolumnd < 3000000000d <= 3000000000d != 3000000000d > 3000000000,d = 3000000000,d IN (3000000000)Bind already folds literals that exceed the table type into
AboveMax/BelowMin(Predicate::bind,PrimitiveLiteral::AboveMax), so the gap is only the file type being narrower than the table type:int → long,float → double, decimal precision widening. Java evaluates residuals on the record after promotion, i.e. in the table type (BaseParquetReaderspromotes on read;EvaluatorcomparesLiteralvalues of the bound type).Fix options:
Promotecolumn source), cast the column up before applying the kernel, and never cast the literal down. Cheap (the filter batch is small) and consistent with projection.safe: falseand fold an out-of-range literal into the constant answer the operator has for it (theAboveMax/BelowMinlogic from bind, applied per file). More code, same result.Pre-existing; noticed while restructuring missing-column evaluation (#221 / #230) and deferred from that work.