[common][spark] Fix z-order boolean FALSE colliding with the null sentinel - #9527
Open
LuciferYang wants to merge 1 commit into
Open
[common][spark] Fix z-order boolean FALSE colliding with the null sentinel#9527LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
…tinel The boolean encoder writes only the first byte of its eight-byte buffer, and the rest stay zero for the life of the indexer, so FALSE encoded to eight zero bytes, byte for byte identical to ZOrderByteUtils.NULL_BYTES. A FALSE row and a NULL row got the same z-order key and were clustered as if the column held the same value. FALSE now writes 0x01, which keeps the unsigned order NULL < FALSE < TRUE and leaves TRUE at 0x81. Both engines carry their own copy of the encoding, so ZIndexer and SparkZOrderUDF are changed together. Assisted-by: GLM-5.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9526
Z-order encodes each order column into eight bytes and then interleaves the bits, and
ZOrderByteUtils.NULL_BYTESis eight zero bytes standing for null. The boolean encoder writes only the first byte of its per-column buffer, and the remaining seven are never written, so they stay zero for the life of the indexer. FALSE therefore encoded to eight zero bytes, byte for byte identical toNULL_BYTES: a FALSE row and a NULL row got the same z-order key and were clustered as if the column held the same value, with nothing reporting a problem. FALSE now writes0x01, which keeps the unsigned order NULL, FALSE, TRUE and leaves TRUE at0x81.SparkZOrderUDF.booleanToOrderedBytesUDFcarries its own copy of the encoding, so both are changed together. They have to agree: a column clustered by Spark and later compacted by Flink must land in the same order.Tests
TestZOrderByteUtil.testBooleanDistinctFromNullSentinelbuilds a two-boolean-columnZIndexerand asserts the three states are pairwise distinct and that the unsigned order really is NULL, then FALSE, then TRUE, using theUnsignedBytescomparator the file already uses. Interleaving two identical inputs is monotone in the input, so comparing the interleaved output compares the encodings. Pinning the order matters as much as the distinctness:0x82for FALSE would also be distinct while sorting FALSE above TRUE.SparkZOrderUDFTest.testBooleanColumnKeepsFalseOffTheNullSentinel(new file) runs a localSparkSessionover a nullable BOOLEAN column with true, false and null rows throughsortedLexicographically(col, BooleanType)and asserts the three encodings are8100000000000000,0100000000000000and0000000000000000. It compares hex rather than the raw arrays on purpose: the UDF returns a per-column buffer it reuses for every row, so three collectedbyte[]values all alias one another and carry the last row's contents. The conversion happens inside the same projection, before the buffer is overwritten. Production is unaffected, becauseZorderSorterfeeds the result tointerleaveBitswithin the same row's evaluation, which copies the bits into its own output buffer.Both fail against the pre-fix code: the indexer test on the FALSE-versus-NULL comparison, and the Spark test with FALSE coming back as
0000000000000000.Note on running them:
TestZOrderByteUtilis named with aTestprefix, and the root pom'stest.unit.patternis**/*Test.*, so that class runs in the integration-test execution rather than inmvn test. CI runsmvn clean installand reaches it; running it directly needs-Dtest=TestZOrderByteUtil, which gives 14 tests, 0 failures.mvn -pl paimon-common teston JDK 8: 12465 tests, 0 failures, 0 errors.mvn -pl paimon-spark/paimon-spark-common -Dtest='SparkZOrderUDFTest,SortedIndexTopoBuilderTest' test: 5 tests, 0 failures. checkstyle, spotless, enforcer and rat run clean on both modules.