Skip to content

Add Variant objects with supplementary-character field names - #126

Open
peterxcli wants to merge 1 commit into
apache:masterfrom
peterxcli:variant-unicode-object-keys
Open

Add Variant objects with supplementary-character field names#126
peterxcli wants to merge 1 commit into
apache:masterfrom
peterxcli:variant-unicode-object-keys

Conversation

@peterxcli

Copy link
Copy Markdown
Member

Rationale

The Variant spec requires object fields to be ordered by unsigned lexicographic UTF-8 bytes of their names. For supplementary characters this differs from UTF-16 code-unit order: UTF-16 sorts U+10000 (surrogate pair D800 DC00) before U+E000 and U+FFFF, while UTF-8 byte order sorts it after both.

This is an easy mistake in languages whose native string comparison operates on UTF-16 code units (Java's String.compareTo, JavaScript, C#), and an implementation that sorts or binary-searches object fields that way will silently miss fields in canonical data (see SPARK-58949 for an instance of this). None of the existing variant examples contain field names outside the BMP, so such implementations currently pass this test suite.

What's added

Two examples encoding the same 12-field object whose field names straddle the surrogate range ($ 0 A a ~ U+00A2 U+20AC U+E000 U+FFFF U+10000 U+1F600 U+10FFFF):

  • object_unicode_keys_sorted — sorted metadata dictionary (sorted_strings = 1), field ids in dictionary order; exercises the sorted-dictionary binary-search fast path.
  • object_unicode_keys_unsorted — deliberately scrambled dictionary (sorted_strings = 0), so object field ids are non-monotonic and readers must compare the referenced field name bytes rather than assume id order.

Each field's value is a short string spelling its field name's code point (e.g. "U+FFFF"), so a lookup that lands on the wrong field is self-evident.

The files are generated directly from the spec by the new standalone regen_unicode_keys.py script (not regen.py, which relies on Spark), and were verified to round-trip with an independent decoder. data_dictionary.json and the README are updated accordingly; this also fixes a pre-existing trailing comma that made data_dictionary.json invalid JSON.

The Variant spec orders object fields by unsigned lexicographic UTF-8
bytes, which differs from UTF-16 code-unit order for supplementary
characters: UTF-16 sorts U+10000 (surrogate pair D800 DC00) before
U+E000 and U+FFFF, while UTF-8 byte order sorts it after both. This is
an easy mistake in languages whose native string comparison operates on
UTF-16 code units (e.g. Java, JavaScript, C#), and an implementation
that sorts or binary-searches object fields that way will silently
mishandle such objects.

Add two examples encoding the same object, whose field names straddle
the surrogate range ($ 0 A a ~ U+00A2 U+20AC U+E000 U+FFFF U+10000
U+1F600 U+10FFFF), so implementations can verify they order and look up
object fields consistently:

* object_unicode_keys_sorted: sorted metadata dictionary
  (sorted_strings = 1), field ids in dictionary order
* object_unicode_keys_unsorted: deliberately scrambled dictionary, so
  field ids are non-monotonic and readers must compare key bytes

Each field value is a short string spelling the field name's code point,
making a lookup that lands on the wrong field self-evident. The files
are generated directly from the spec by the new standalone
regen_unicode_keys.py script.

Also fix a pre-existing trailing comma that made data_dictionary.json
invalid JSON.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant