Search before asking
Paimon version
master, 9c7deebbd (2.1-SNAPSHOT)
Compute Engine
Flink (sort_compact with order_strategy = zorder) and Spark (sys.compact with order_strategy => 'zorder', and clustered writes). Each engine has its own copy of the encoding: ZIndexer in paimon-common, SparkZOrderUDF in paimon-spark-common.
Minimal reproduce step
Z-order encodes each order column into eight bytes and then interleaves the bits. ZOrderByteUtils.NULL_BYTES is eight zero bytes and stands for null. The boolean encoder writes only the first byte of its buffer:
// ZIndexer, BOOLEAN visitor
ZOrderByteUtils.reuse(reuse, PRIMITIVE_BUFFER_SIZE);
reuse.put(0, (byte) (row.getBoolean(fieldIndex) ? -127 : 0));
return reuse.array();
The remaining seven bytes of that per-column buffer are never written, so they stay zero for the life of the indexer. FALSE therefore encodes to eight zero bytes, byte for byte identical to NULL_BYTES, and a FALSE row and a NULL row get the same z-order key:
CREATE TABLE T (a BOOLEAN, b INT) TBLPROPERTIES ('bucket' = '-1');
INSERT INTO T VALUES (true, 1), (false, 2), (null, 3);
CALL paimon.sys.compact(table => 'T', order_strategy => 'zorder', order_by => 'a,b');The compaction succeeds, and the rows with a = false and a = null are clustered as if that column held the same value.
What doesn't meet your expectations?
A boolean column has three states and each should get its own encoding, as it does for every other type in that class: TRUE is 0x81, NULL is all zeros, and FALSE needs to be something else. Sharing the null encoding means the clustering silently does less than it claims: files that could be skipped for a = false are read anyway, and nothing reports a problem.
Anything else?
SparkZOrderUDF.booleanToOrderedBytesUDF has the same literal, so both engines are affected, and they have to stay in step: a column clustered by Spark and later compacted by Flink must land in the same order.
While looking at this I also noticed that NULL_BYTES equals the encoding of Long.MIN_VALUE for a BIGINT or TIMESTAMP column, since tinyintToOrderedBytes and friends flip the sign bit. That one needs a different remedy and I am not proposing to change it here.
Are you willing to submit a PR?
Search before asking
Paimon version
master,
9c7deebbd(2.1-SNAPSHOT)Compute Engine
Flink (
sort_compactwithorder_strategy = zorder) and Spark (sys.compactwithorder_strategy => 'zorder', and clustered writes). Each engine has its own copy of the encoding:ZIndexerin paimon-common,SparkZOrderUDFin paimon-spark-common.Minimal reproduce step
Z-order encodes each order column into eight bytes and then interleaves the bits.
ZOrderByteUtils.NULL_BYTESis eight zero bytes and stands for null. The boolean encoder writes only the first byte of its buffer:The remaining seven bytes of that per-column buffer are never written, so they stay zero for the life of the indexer. FALSE therefore encodes to eight zero bytes, byte for byte identical to
NULL_BYTES, and a FALSE row and a NULL row get the same z-order key:The compaction succeeds, and the rows with
a = falseanda = nullare clustered as if that column held the same value.What doesn't meet your expectations?
A boolean column has three states and each should get its own encoding, as it does for every other type in that class: TRUE is
0x81, NULL is all zeros, and FALSE needs to be something else. Sharing the null encoding means the clustering silently does less than it claims: files that could be skipped fora = falseare read anyway, and nothing reports a problem.Anything else?
SparkZOrderUDF.booleanToOrderedBytesUDFhas the same literal, so both engines are affected, and they have to stay in step: a column clustered by Spark and later compacted by Flink must land in the same order.While looking at this I also noticed that
NULL_BYTESequals the encoding ofLong.MIN_VALUEfor a BIGINT or TIMESTAMP column, sincetinyintToOrderedBytesand friends flip the sign bit. That one needs a different remedy and I am not proposing to change it here.Are you willing to submit a PR?