Add Variant objects with supplementary-character field names - #126
Open
peterxcli wants to merge 1 commit into
Open
Add Variant objects with supplementary-character field names#126peterxcli wants to merge 1 commit into
peterxcli wants to merge 1 commit into
Conversation
The Variant spec orders object fields by unsigned lexicographic UTF-8 bytes, which differs from UTF-16 code-unit order for supplementary characters: UTF-16 sorts U+10000 (surrogate pair D800 DC00) before U+E000 and U+FFFF, while UTF-8 byte order sorts it after both. This is an easy mistake in languages whose native string comparison operates on UTF-16 code units (e.g. Java, JavaScript, C#), and an implementation that sorts or binary-searches object fields that way will silently mishandle such objects. Add two examples encoding the same object, whose field names straddle the surrogate range ($ 0 A a ~ U+00A2 U+20AC U+E000 U+FFFF U+10000 U+1F600 U+10FFFF), so implementations can verify they order and look up object fields consistently: * object_unicode_keys_sorted: sorted metadata dictionary (sorted_strings = 1), field ids in dictionary order * object_unicode_keys_unsorted: deliberately scrambled dictionary, so field ids are non-monotonic and readers must compare key bytes Each field value is a short string spelling the field name's code point, making a lookup that lands on the wrong field self-evident. The files are generated directly from the spec by the new standalone regen_unicode_keys.py script. Also fix a pre-existing trailing comma that made data_dictionary.json invalid JSON.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
The Variant spec requires object fields to be ordered by unsigned lexicographic UTF-8 bytes of their names. For supplementary characters this differs from UTF-16 code-unit order: UTF-16 sorts U+10000 (surrogate pair
D800 DC00) before U+E000 and U+FFFF, while UTF-8 byte order sorts it after both.This is an easy mistake in languages whose native string comparison operates on UTF-16 code units (Java's
String.compareTo, JavaScript, C#), and an implementation that sorts or binary-searches object fields that way will silently miss fields in canonical data (see SPARK-58949 for an instance of this). None of the existing variant examples contain field names outside the BMP, so such implementations currently pass this test suite.What's added
Two examples encoding the same 12-field object whose field names straddle the surrogate range (
$ 0 A a ~ U+00A2 U+20AC U+E000 U+FFFF U+10000 U+1F600 U+10FFFF):object_unicode_keys_sorted— sorted metadata dictionary (sorted_strings = 1), field ids in dictionary order; exercises the sorted-dictionary binary-search fast path.object_unicode_keys_unsorted— deliberately scrambled dictionary (sorted_strings = 0), so object field ids are non-monotonic and readers must compare the referenced field name bytes rather than assume id order.Each field's value is a short string spelling its field name's code point (e.g.
"U+FFFF"), so a lookup that lands on the wrong field is self-evident.The files are generated directly from the spec by the new standalone
regen_unicode_keys.pyscript (notregen.py, which relies on Spark), and were verified to round-trip with an independent decoder.data_dictionary.jsonand the README are updated accordingly; this also fixes a pre-existing trailing comma that madedata_dictionary.jsoninvalid JSON.