Unicode 18 UCD tables

[Charlie Lin (clin1234)]

Regenerate UCD tables for <format> and Grapheme Cluster breaks

Getting closer to 18's final release on the 16th: https://www.unicode.org/versions/beta-18.0.0.html

Claude was only used to discover that the Chisoi block was unassigned; all subsequent effort consisted of manually pulling updated UCD, regenerating relevant files, and manually-inspecting them for correctness.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The updated grapheme expectations conflict with the unchanged GB9c implementation.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates Unicode-dependent formatting tables and grapheme-cluster test data to Unicode 18.0.0.

Changes:

• Updates Unicode source references.
• Regenerates formatting UCD tables.
• Regenerates grapheme-break test vectors.

File summaries

File	Description
tools/unicode_properties_parse/unicode_properties_data_gen.py	Updates Unicode 18 UCD sources.
tools/unicode_properties_parse/grapheme_break_test_data_gen.py	Updates the Unicode 18 grapheme-test source.
tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp	Adds regenerated Unicode 18 grapheme data; GB9c expectations require implementation updates, and generated markers should be preserved.
stl/inc/__msvc_format_ucd_tables.hpp	Contains regenerated Unicode 18 formatting tables.

Review details

Suppressed comments (1)

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:16

• This regeneration removes the // Generator: and // Beginning of generated data markers from the generated section, while the matching // End of generated data marker remains at line 1175. Please preserve the paired delimiters (and update cpp_template if these are meant to be regenerated) so the checked-in file does not advertise an unmatched generated region.

// GraphemeBreakTest-18.0.0.txt
// Date: 2026-06-12, 00:44:16 GMT

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The Unicode 18 corpus exposes unsupported GB9c behavior, and the test regeneration loses provenance markers and adds concatenated status output.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:17

• The regenerated corpus drops the existing generator and “Beginning of generated data” markers while retaining the closing marker at line 1176. That removes the provenance and regeneration warning for this generated block; please emit these markers from the generator and regenerate the file so future updates preserve them.

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:436

• These expected breaks changed from {0, 2, 3} to {0, 3}, but the current _GB9c_regex cannot produce that result: U+0061/U+003F have InCB=No_value, so after U+094D (Linker) the regex resets instead of matching GB9c. The iterator therefore returns a break at index 2 and run_unicode_test_data_utf32() asserts; please update the grapheme algorithm for the Unicode 18 rule (and its tests), or defer this corpus update until the implementation supports it.

    {{U'\x61', U'\x94d', U'\x924'}, {0, 3}}, {{U'\x3f', U'\x94d', U'\x924'}, {0, 3}},

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:1359

• These new success messages have no \n, so a passing run emits all six statuses concatenated on one line before the test runner's output. This is the only Passed ... output in the standard test suite; please remove this debugging output or terminate each message with a newline if it is intentional.

    std::printf("Passed run_unicode_test_data_utf32()");
    run_unicode_test_data_utf32_utf8_comparison();
    std::printf("Passed run_unicode_test_data_utf32_utf8_comparison()");

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🟡 Changes recommended

The GB9c matcher has a correctness issue and grapheme iteration performs a redundant property lookup.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🟡 Changes recommended

Fix the two GB9c matching issues and remove the unconditional diagnostic output.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

stl/inc/format:374

• _Match is called with the current left code point before the right-hand property is tested, so Linker × Consonant must be accepted when this transition is taken. Returning false here makes every direct GB9c pair (for example the generated 0x094D × 0x0915 case at test.cpp:102) break instead of matching the Unicode 18 rule.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Linker_value) {
                _State = _Linker;
            }
            return false;

stl/inc/format:380

• The accepting suffix is Linker Extend*: an Extend here must keep the match true, while any other property must reset the DFA rather than return true and leave _State as _Linker. As written, Linker Extend* × Consonant fails, and a later unrelated character can be treated as a GB9c prefix.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Extend_value) {
                return false;
            }
            return true;

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🔵 Needs a closer look

Two moderate issues remain, along with one generated-data marker nit.

Review details

Suppressed comments (3)

stl/inc/format:374

• The transition into _Linker is the accepting state shown in the DFA, but this branch still returns false. Because _Match is called with the current left code point before _Right_InCB is tested, a Linker x Consonant boundary never matches GB9c; the new U+1CF5 U+0995 case at test.cpp:339-340 therefore fails. Return true when consuming the linker.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Linker_value) {
                _State = _Linker;
            }
            return false;

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:1189

• This unconditional diagnostic prints one line for every emitted grapheme cluster on every successful run, flooding CI logs and adding needless formatting overhead to this regression test. Please remove the debug printf (and the now-unused <cstdio> include), or make the output failure-only.

            printf("Iter: %lX, rhs: %lX\n",
                (unsigned long)*iter,
                (unsigned long)test_case.code_points[test_case.breaks[i]]
            );

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:17

• The generated block's opening provenance markers were removed while the closing // End of generated data marker remains. This leaves the generated-data delimiters unbalanced and removes the direct script reference that helps future UCD refreshes; please retain these markers (or have the generator emit them).

// GraphemeBreakTest-18.0.0.txt
// Date: 2026-06-12, 00:44:16 GMT

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🟡 Changes recommended

Unresolved grapheme-clustering correctness issues and a test compilation error block approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

stl/inc/format:374

• Because _Match is invoked on _Left_InCB before evaluating the current pair (line 477), the transition that consumes a Linker must be accepting. Returning false here makes Linker × Consonant a break; the new U+1CF5 U+0995 conformance case at test.cpp:449 expects {0,2} and will fail. Return the accepting result when entering _Linker.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Linker_value) {
                _State = _Linker;
            }
            return false;

stl/inc/format:380

• In _Linker, Extend is part of the accepted Linker Extend* prefix, but this branch returns false for it; every other value returns true and leaves _State as _Linker. That both misses Linker Extend* × Consonant and lets a prior linker cause unrelated ... × Consonant pairs to join across a break. Keep the accepting state for Extend (and a new Linker), but reset and return false for other properties.

        case _Linker:
            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Extend_value) {
                return false;
            }
            return true;

stl/inc/format:473

• Adding _Right_InCB here introduces a second property-table lookup for each code point: the loop's increment expression still initializes _Left_InCB by looking up the same *_WrappedIter, while this line already computed that value as _Right_InCB. Reuse _Right_InCB when advancing the loop, as is already done for _Right_gbp and _Right_ExtPic, to avoid an extra binary search through the 483-range table per code point.

            _Right_InCB   = _Indic_Conjunct_Break_property_data._Get_property_for_codepoint(*_WrappedIter);

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🟡 Changes recommended

The GB9c transition and test diagnostic issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

stl/inc/format:374

• _GB9c_Match is evaluated for the boundary after the current code point, so a Linker itself is an accepting suffix of Linker Extend* (zero Extends are allowed). Returning false here makes the new rule fail for the simplest InCB=Linker followed by InCB=Consonant case, including the generated 0x094D, 0x0915 case at tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:102. Return true when transitioning to _Linker.

            return false;

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🔵 Needs a closer look

GB9c state handling can incorrectly suppress a required grapheme break.

Review details

Suppressed comments (1)

stl/inc/format:380

• This branch treats every non-Extend code point as a successful continuation and never leaves _Linker. That lets a Linker followed by an InCB=No character (for example U+1CF5 × U+200C ÷ U+0995, covered by the generated case at test.cpp:450) incorrectly suppress the later break. Accept only Extend (or a new Linker start), otherwise reset to _Start and return false.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Extend_value) {
                return false;
            }
            return true;

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🔵 Needs a closer look

GB9c state handling is incorrect, and the test does not verify all expected break positions.

Review details

Suppressed comments (3)

stl/inc/format:374

• _Match returns false when it first sees a Linker, and also for every Extend after it, so _GB9c_Match can never be true for the new Linker Extend* × Consonant rule. For example, the regenerated U'\x94d', U'\x915' case will still break at the linker instead of producing the expected single cluster; the non-Extend path also needs to reset the DFA so it cannot carry a stale match across unrelated characters.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Linker_value) {
                _State = _Linker;
            }
            return false;

stl/inc/format:380

• While in _Linker, this branch returns true for every non-Extend property and never resets the state. After a Linker followed by an InCB=No character such as U+200C, the next consonant is therefore incorrectly accepted by GB9c; the new test at test.cpp:450 expects a break there. Keep the accepting transition only for Extend/Linker and reset to _Start otherwise.

        case _Linker:
            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Extend_value) {
                return false;
            }
            return true;

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:1190

• This loop only validates the emitted prefix and never checks that all expected break positions were emitted. If the iterator reaches the sentinel early (for example, by missing a required break in the new U+1CF5 U+200C U+0995 case), the test exits successfully without detecting it. Assert that the number of emitted cluster starts is one less than breaks.size() after the loop.

            if (*iter != test_case.code_points[test_case.breaks[i]]) {
                fprintf(stderr, "Iter: %lX, rhs: %lX\n", (unsigned long) *iter,
                    (unsigned long) test_case.code_points[test_case.breaks[i]]);
                assert(false);
            }

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🟡 Changes recommended

The GB9c state machine has unresolved correctness issues, plus a diagnostic-label nit.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

stl/inc/format:374

• This transition enters the accepting _Linker state but returns false; the _Linker branch likewise returns false for Extend. As a result, the new Linker Extend* × Consonant rule is not recognized for sequences such as U+094D U+0924 in the regenerated test data. Make the Linker/Extend transitions accept the current left code point, and reset to _Start when a non-Extend/non-Linker value ends the pattern so the state cannot leak into later boundaries.

            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Linker_value) {
                _State = _Linker;
            }
            return false;

stl/inc/format:380

• In _Linker, the DFA must continue accepting Linker Extend* and reset when the current property is unrelated. Returning false for Extend shifts the match, while returning true for every other property leaves the state stale; for example, the new U+1CF5 U+200C U+0995 case at test.cpp:450 is incorrectly joined instead of breaking before the consonant. Accept Extend/Linker here and reset to _Start otherwise.

        case _Linker:
            if (_Left_InCB == _Indic_Conjunct_Break_property_values::_Extend_value) {
                return false;
            }
            return true;

stl/inc/format:380

• For any non-Extend value other than a new Linker, this unconditional true leaves _State as _Linker after a grapheme boundary. Consequently, after Linker Consonant, the following ordinary Consonant can be treated as GB9c context and incorrectly suppress its break; reset to _Start when the current value is not Linker.

            return true;

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[A. Jiang (frederick-vs-ja)]

Do we want to refer to a fixed version instead of latest? Ditto below.

[Charlie Lin (clin1234)]

I think it's more preferable to point to the specific version corresponding to the generated files

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:16

• The regenerated block now starts directly with the Unicode file metadata, removing the Generator: and Beginning of generated data sentinels while the matching End of generated data sentinel remains at line 1175. This leaves the generated-data markers inconsistent and loses the documented regeneration source; please restore the two sentinels (or remove the end sentinel too).

stl/inc/format:477

• This unconditional _Get_property_for_codepoint call adds a binary-search table lookup for every code point, even though _Right_InCB is only consumed when _GB9c_Match is true below. This iterator is used by the string-width hot path, so defer the lookup or inline it in the guarded GB9c condition to avoid charging ordinary text for the new property lookup.

            _Right_InCB   = _Indic_Conjunct_Break_property_data._Get_property_for_codepoint(*_WrappedIter);

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Copilot]

🔵 Needs a closer look

Resolve the GB9c documentation/behavior mismatch and restore the generated-data markers.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

stl/inc/format:543

• The DFA keeps accepting after another InCB=Linker, but the adjacent regex documents only Linker Extend*. This mismatch is observable for the Unicode 18 test case U+0915 U+094D U+094D U+0924, which expects no break and relies on the Linker loop. Please document the implemented repetition (or change the DFA if the narrower rule is intended).

tests/std/tests/P0645R10_text_formatting_grapheme_clusterization/test.cpp:16

• The regeneration dropped the existing Generator: and Beginning of generated data markers while leaving the End of generated data marker at line 1175. This removes the documented path for reproducing the large fixture and makes the generated region ambiguous; please retain these markers (or emit them from cpp_template) when refreshing the data.

// GraphemeBreakTest-18.0.0.txt
// Date: 2026-06-12, 00:44:16 GMT

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

[Charlie Lin (clin1234)]

Stephan T. Lavavej (@StephanTLavavej) Unicode 18 is officially out: https://blog.unicode.org/2026/09/announcing-unicode-standard-version-180.html