recipes: pyarrow 24.0.0 (+ flet-libarrow) and tornado 6.5.7 #95
Merged
Conversation
Pure-Python async networking lib with one optional C extension (tornado.speedups, abi3/websocket_mask). Requested in flet#6632 to unblock the jupyter-client dependency leg. build.script_env TORNADO_EXTENSION=1 forces the extension: setup.py marks it optional, so without this a compile failure would silently ship a pure-Python wheel with no .so. No patches; no runtime deps. Full matrix builds green locally (android arm64-v8a/x86_64/armeabi-v7a + iOS device/arm64-sim/x86_64-sim).
Foundation recipe for pyarrow (flet#3275 / #6632): cross-compiles Apache Arrow C++ 24.0.0 into a shared libarrow + libarrow_compute via CMake, driving the NDK/iOS toolchain (Arrow's own CMake has no mobile awareness). Ported from the prior 'pyarrow' branch, which had Android verified on-device. Key cross fixes (see build.sh): ARROW_CPU_FLAG to bypass Arrow's 'Unknown system processor' FATAL_ERROR; SIMD_LEVEL=NONE; SYSTEM zlib (BUNDLED builds host-arch libz that breaks the x86_64-sim slice); iOS -framework CoreFoundation (vendored date/ios.mm); Android date/locale patches (nl_langinfo API24, BUILD_TZ_LIB); iOS dylib de-versioning so flet's per-slice framework relocation works. Component floor = COMPUTE+CSV+JSON+FILESYSTEM+IPC (pyarrow's irreducible set); heavy deps (parquet/dataset/acero/flight/re2/utf8proc/cloud fs) OFF. Improvement over baseline: excluded_arches [armeabi-v7a, x86] — Arrow's CMake 32/64-bit gate makes 32-bit libarrow unconsumable by pyarrow, so don't build it. Built green locally: android arm64-v8a + x86_64, iOS device + arm64-sim + x86_64-sim. libarrow/libarrow_compute .so 16KB-aligned; iOS dylibs unversioned with @rpath install-ids.
Consumes flet-libarrow (libarrow + libarrow_compute) and builds pyarrow's Cython ext modules + arrow_python via scikit-build-core/CMake, cross-compiled for mobile. Unblocks the google-colab-cli / jupyter chain (flet#6632) and the long-standing pyarrow request (flet#3275). Ported from the prior 'pyarrow' branch (Android was on-device verified 5/5). Component set = compute-only floor (lib/_compute/_csv/_feather/_fs/_json); all PYARROW_WITH_* heavy components (parquet/dataset/acero/flight/orc/cuda/cloud-fs) hard-locked OFF via script_env. excluded_arches [armeabi-v7a, x86] (Arrow's CMake 32/64-bit gate). iOS uses the Unix Makefiles generator (ninja pip pkg crashes importing under the iOS crossenv python). Two patches: - arrow-python-optlib.patch: install libarrow_python into the wheel's opt/lib (unversioned) so it rides the same routing as libarrow/libarrow_compute — APK jniLibs on Android, per-slice *.fwork frameworks on iOS. - mobile.patch: pyarrow/__init__ preload shim — on iOS, RTLD_GLOBAL-load the arrow dylibs from their relocated *.fwork frameworks before importing pyarrow.lib so the ext modules' @rpath load-commands bind by install-name. No-op on Android/desktop. Build validated locally, full matrix: android arm64-v8a + x86_64, iOS device + arm64-sim + x86_64-sim. Android linkage confirmed (lib.so DT_NEEDED resolves libarrow/_compute/_python/libc++_shared by basename from jniLibs). iOS on-device runtime (the .fwork preload path) still to be verified — next.
The prior iOS shim assumed flet relocates the opt/lib arrow *.dylib files into *.framework bundles (leaving <name>.fwork markers). flet 0.85.2 does NOT: it relocates the pyarrow *.so ext modules into frameworks but leaves the arrow C++ dylibs as plain files in site-packages/opt/lib, with no markers. So the old shim found no markers and did nothing, and importing pyarrow.lib died with 'Library not loaded: @rpath/libarrow_python.dylib'. New shim CDLLs the three arrow dylibs directly from opt/lib (RTLD_GLOBAL, dependency order) before importing pyarrow.lib; dyld then binds the ext modules' @rpath refs by install-id. The .fwork-marker path is kept as a forward-compat fallback. Verified: with the correct iOS-simulator arrow dylibs present, all 5 on-device pyarrow tests PASS on the arm64 iOS simulator (import, array/table, schema, IPC roundtrip, compute kernels).
Each patch now carries its own description header at the top of the file (patch(1) ignores everything before the first '---' hunk header), and meta.yaml's patches: list is just the filenames. Keeps the recipe metadata lean and the rationale next to the diff it explains.
The python-build manifest now ships only three Android ABIs (arm64-v8a, x86_64, armeabi-v7a) and the support tree has no x86 python — so x86 can't be built at all, and listing it in excluded_arches was dead config. - pyarrow / flet-libarrow: excluded_arches [armeabi-v7a, x86] -> [armeabi-v7a] (armeabi-v7a is the only shipped 32-bit ABI; that's what the Arrow 32/64 gate needs excluded). - meta-schema: drop x86 from the excluded_arches enum so the schema rejects it; refresh the description (also removed a stale note claiming 32-bit arches only build on Python < 3.13 — the manifest lists armeabi-v7a for 3.12/3.13/3.14). Note: src/forge/cross.py still carries legacy x86 triplet entries — harmless dead lookups (no x86 support tree to build against); left for a separate sweep.
The 'x86' in arch test is a family match (hits the shipped x86_64 slice, not the now-only 32-bit arch armeabi-v7a). Logic unchanged — x86_64 -> 'x86', arm64/arm64-v8a -> 'aarch64' still covers every shipped slice.
Every other flet-lib* foundation recipe builds for all available arches; none use excluded_arches. flet-libarrow was the lone exception. The 32-bit exclusion belongs on the consumer (pyarrow keeps it — Arrow's ArrowConfigVersion.cmake 32/64-bit gate rejects a 32-bit find_package). Verified Arrow C++ builds green for armeabi-v7a. On py3.12 this publishes an unconsumed armeabi-v7a libarrow; harmless, and consistent with the flet-lib* convention (armeabi-v7a isn't a target on py3.13+ anyway, so it's a no-op there).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds three recipes that close the native-dependency gaps for the Jupyter / Google Colab stack on mobile:
tornado6.5.7 — async networking library (pure-Python + one optional abi3 C extension)flet-libarrow24.0.0 — Apache Arrow C++ cross-compiled to a sharedlibarrow+libarrow_computepyarrow24.0.0 — the Arrow Python bindings, linked againstflet-libarrowTogether these unblock the
google-colab-cli→jupyter-kernel-client→jupyter-clientchain from flet-dev/flet#6632, and land the long-standing pyarrow request flet-dev/flet#3275.What's in each recipe
tornado
tornado.speedups, an abi3websocket_mask); stable ABI (cp39-abi3), so a single wheel serves every Python. No third-party libs, no patches, zero runtime deps.build.script_env: TORNADO_EXTENSION=1forces the extension.setup.pymarks itoptional=(TORNADO_EXTENSION != "1"), so without this a compile failure would silently ship a pure-Python wheel with no.so. Forcing it makes failures loud and lets the on-device test assert the.soactually loaded.flet-libarrow + pyarrow
lib/_compute/_csv/_feather/_fs/_json); all heavyPYARROW_WITH_*(parquet / dataset / acero / flight / orc / cuda / cloud-fs) are hard-locked off.flet-libarrow/build.sh):ARROW_CPU_FLAGto bypass Arrow's "Unknown system processor" FATAL_ERROR;SIMD_LEVEL=NONE;ZLIB_SOURCE=SYSTEM(BUNDLED builds a host-arch libz that breaks the x86_64-sim slice); iOS-framework CoreFoundation(vendoreddate/ios.mm) plus dylib de-versioning to a single@rpath/lib*.dylib; Android vendored date/locale patches (nl_langinfoAPI-24,BUILD_TZ_LIB).excluded_arches: [armeabi-v7a]on pyarrow: Arrow'sArrowConfigVersion.cmake32/64-bit gate makes a 32-bitfind_package(Arrow)failarrow-python-optlib.patch— installlibarrow_pythoninto the wheel'sopt/lib(unversioned) so it rides the same routing aslibarrow/libarrow_compute: APK jniLibs on Android, per-slice*.fworkframeworks on iOS.mobile.patch— apyarrow/__init__preload shim thatRTLD_GLOBAL-loads the Arrow dylibs before importingpyarrow.lib, so the ext modules'@rpathrefs bind by install-id. No-op on Android/desktop.Testing
pyarrow (+ flet-libarrow)
CI Workflow
lib.soresolves libarrow / _compute / _python / libc++_shared byDT_NEEDEDbasename from jniLibs.serious_python >= 4.2.1(fix(darwin): framework-ize ctypes.dyliblibs so they load on the iOS simulator serious-python#223, which framework-izes ctypes.dyliblibs on the simulator). Verified locally by patching the darwin scripts to fix(darwin): framework-ize ctypes.dyliblibs so they load on the iOS simulator serious-python#223. The iOS CI leg stays red untilflet-clibumps itsserious_pythondependency — no recipe change is needed when it does.