[Bug] kuromoji_build_dict fails to link on arm64: B/BL out of range (__TEXT exceeds 128MB)

[morningman]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master (ddbaaab1388), macOS 26.5 arm64, Homebrew clang 20.1.8, DISABLE_BUILD_UI=ON sh build.sh --fe --be

What's Wrong?

Linking bin/kuromoji_build_dict fails on arm64 because the binary's __TEXT segment exceeds the AArch64 ±128 MB branch range:

FAILED: [code=1] bin/kuromoji_build_dict
ld: fixup error (kind=arm64_b26) at '__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC
    from libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o), B/BL out of range
    (displacement=-135691464, max is +/-128MB), from 0x10816EF18 to 0x100007250 ('___clang_call_terminate')
final section layout:
    __TEXT               addr=0x100000000, size=0x008310000   <-- 137 MB
        __text           ... size=0x0059254bc
        google_malloc    addr=0x10816da40, ...
        malloc_hook      addr=0x10816ef3c, ...

The linker normally inserts branch islands, but it can only do that inside __text. gperftools puts its allocator in the custom sections google_malloc and malloc_hook, which the linker places after __text, so a branch from there back to ___clang_call_terminate at the start of __TEXT cannot be relaxed.

doris_be itself links fine — ninja doris_be succeeds. Only kuromoji_build_dict is over the limit.

The failure also blocks the install step, which hard-fails when the generated dictionary is absent:

CMake Error at cmake_install.cmake:96 (message):
  kuromoji dictionary file .../be/dict/kuromoji/system.bin is missing; build the
  'kuromoji_dict' target (stage mecab-ipadic via thirdparty) before packaging.

so output/ is never produced.

What You Expected?

sh build.sh --be completes on arm64.

How to Reproduce?

On an Apple Silicon host:

DISABLE_BUILD_UI=ON sh build.sh --fe --be

Anything Else?

kuromoji_build_dict is defined under if (NOT MAKE_TEST) in be/CMakeLists.txt and kuromoji_dict is an ALL target, so a normal --be build links it. be-ut-mac.yml runs ./build.sh --be on runs-on: macos-15 (arm64) with -DMAKE_TEST=OFF, so that CI job should build this target too — worth checking whether it is currently green, since my margin was only ~1.4 MB over the limit and could easily differ with another linker version or flag set.

Ideas, roughly in order of appeal:

• the tool only needs the dictionary builder, not the whole BE; trimming its target_link_libraries(${DORIS_LINK_LIBS}) down to what kuromoji_build_dict.cpp actually uses would take it far under the limit;
• link the tool against jemalloc (or no custom allocator) so gperftools' custom sections are not in play;
• ship the generated dictionary as a thirdparty artifact instead of building it during the BE build.

Found while building BE for #67366; worked around locally by dropping ALL from kuromoji_dict and skipping the install-time check.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_f68ed80a2dc5

Initial assessment

Triage: confirmed master build regression / macOS arm64 build blocker. The issue is open with no labels or assignee. Suggested existing labels are kind/fix and area/aarch64, with routing to the BE build owner.

Verified facts

• The reported commit wires kuromoji_build_dict to the complete ${DORIS_LINK_LIBS} closure, and kuromoji_dict is an ALL target in normal (MAKE_TEST=OFF) builds (CMake target definition). The install failure is therefore downstream: the link failure prevents generation of the four required dictionary files, and the install guard then correctly rejects an incomplete package.
• On Darwin, build.sh defaults USE_JEMALLOC=OFF (selection logic), so a Release build selects and links tcmalloc. Doris already has a macOS-arm64 workaround for the same arm64_b26 failure, but it is limited to MAKE_TEST or BUILD_BENCHMARK (existing allocator exclusion); this new tool is created only when NOT MAKE_TEST, so that protection cannot apply.
• The exact ddbaaab1388 workflow run was canceled by concurrency, but the immediately following master run #153446 failed in Build BE at bin/kuromoji_build_dict with the same tcmalloc fixup: displacement -135769416, __TEXT size 0x08324000. The later master run #153448 repeated the identical failure and layout. This independently reproduces the report on the repository's macos-15 arm64 CI image.
• The CI verbose link command confirms that this small offline converter is presented with all BE component archives, the full third-party closure, and libtcmalloc.a. Its source directly needs the Kuromoji builder/parser plus Doris Status; it does not conceptually need the BE service/runtime or a production allocator. The target wiring was introduced by #64667.

Root-cause judgment

The evidence supports the issue's mechanism: the new offline tool reuses the whole BE link closure in a normal macOS build, while that build selects gperftools. The resulting Mach-O __TEXT places tcmalloc's custom google_malloc/malloc_hook sections beyond the arm64 direct-branch range from ___clang_call_terminate; the linker reports that exact out-of-range branch. This is not a missing-dictionary root cause, and it is no longer only a reporter-specific/toolchain hypothesis.

The same failure family and allocator workaround already exist in #66615, which is a useful project-local reference.

Recommended next step

1. Preferred durable fix: give the dictionary converter a dedicated, small dependency target containing the builder/parser implementation and only the support libraries it actually needs; do not link it with ${DORIS_LINK_LIBS} or a custom allocator. Audit the actual closure rather than assuming that Storage/Common alone is small, because those targets use unity objects and can pull additional unresolved dependencies at archive granularity.
2. Contained unblocker if the dependency split is not immediately practical: omit tcmalloc only for kuromoji_build_dict on macOS arm64 and provide the required no-op gperftools symbols using the established [fix](build) Fix the BE unit-test and benchmark build on macOS arm64 #66615 pattern. Keep doris_be allocator behavior unchanged. This removes the failing custom sections but leaves an unnecessarily large build tool, so it is less maintainable than dependency isolation.
3. Keep kuromoji_dict in ALL and keep the install-time file validation. Dropping either only hides the failure and can produce a package without data required by the runtime feature.

Evidence required on the fixing PR

• A clean macOS arm64 ninja -v kuromoji_build_dict showing the reduced link line, no libtcmalloc.a, and a comfortably sub-128-MB __TEXT (record size/otool output before and after).
• A clean ninja kuromoji_dict followed by validation that system.bin, matrix.bin, chardef.bin, and unkdict.bin are generated and non-empty.
• A full DISABLE_BUILD_UI=ON sh build.sh --be on macOS arm64, including installation into output/, plus the relevant Kuromoji BE unit tests on a supported test host.
• A Linux BE build to catch any dependency-list portability regression.

No additional reporter logs or reproduction details are required for initial triage. The remaining unknown is patch-specific: the exact minimal support-library closure must be demonstrated by the candidate link command and clean build, not guessed.