workflows: add build-pyarrow.yml for riscv64 manywheel builds by luhenry · Pull Request #294 · riseproject-dev/python-wheels

luhenry · 2026-08-20T18:44:19Z

What

Adds .github/workflows/build-pyarrow.yml to build riscv64 pyarrow wheels (25.0.1) and publish them to pypi.riseproject.dev.

pyarrow is Cython bindings over the Apache Arrow C++ libraries, so this is a heavy build: the Arrow C++ libs are built once from source with bundled third-party deps (ARROW_DEPENDENCY_SOURCE=BUNDLED — the manylinux riscv64 image has no vcpkg binary cache, unlike upstream's x86/arm images), then a wheel is built per Python against that shared C++ build. This mirrors apache/arrow's own ci/scripts/python_wheel_xlinux_build.sh (minus vcpkg) and tests like ci/scripts/python_wheel_unix_test.sh.

Ref: #85

Shape

Host-driven docker run (not cibuildwheel): pyarrow's C++ sources are a sibling of the Python package, which cibuildwheel's copy-package-dir model can't handle, and the manylinux riscv64 image ships no Node (so a container: job can't run JS actions). Checkout/upload run on the riscv host; the heavy build runs in docker run inside quay.io/pypa/manylinux_2_39_riscv64 on the native riscv64 runner.
Build C++ once, then loop cp312 / cp313 / cp314 / cp314t building wheels (scikit-build-core, PYARROW_BUNDLE_ARROW_CPP=ON), then auditwheel repair --strip.
Test like upstream: import smoke test for enabled modules + pytest -r s --pyargs pyarrow, feature-gated via PYARROW_TEST_*.

Feature set (workflow `env`, single source of truth)

ON: Parquet, Dataset, Acero, Compute, CSV, JSON, Filesystem, HDFS, ORC, Substrait, Flight, Parquet-encryption + all compression (snappy/zlib/zstd/lz4/brotli/bz2), mimalloc.

OFF (enabled incrementally later): S3, GCS, Azure, Gandiva, jemalloc, OpenTelemetry, TensorFlow. The network-storage stacks (aws-sdk / google-cloud-cpp / azure-sdk) and LLVM-based Gandiva have dep trees unverified on riscv64; enabling them is a one-line flip in the env block per feature.

Notes for review

OpenSSL is the one dep Arrow can't bundle (comes from the OS). This image (Rocky 10) ships only shared libssl/libcrypto, so -DARROW_OPENSSL_USE_SHARED=ON is pinned — without it, ARROW_DEPENDENCY_USE_SHARED=OFF cascades to a static-OpenSSL lookup that fails and breaks the bundled gRPC/parquet link. (Found and fixed via a riscv64 qemu configure check.)
The parallel upstream effort (apache/arrow PR #49556, same RISE initiative) targets the identical image + runner but via vcpkg, whose full dep tree its own notes flag as unverified on riscv64. This PR sidesteps that with BUNDLED.
numpy/libcst/pandas are pulled from pypi.riseproject.dev (no public-PyPI riscv64 wheels); pandas is optional in the test step and gates the pandas-dependent tests (matters for cp314t, which may lack a free-threaded pandas wheel).

🤖 Generated with Claude Code

pyarrow is Cython bindings over the Apache Arrow C++ libraries. Build the Arrow C++ libs once from source with bundled third-party deps (ARROW_DEPENDENCY_SOURCE=BUNDLED — the manylinux riscv64 image has no vcpkg binary cache, unlike upstream's x86/arm images), then build a wheel per Python (cp312/313/314/314t) against that shared C++ build. Mirrors apache/arrow's ci/scripts/python_wheel_xlinux_build.sh, minus vcpkg, and tests like ci/scripts/python_wheel_unix_test.sh (pytest --pyargs pyarrow, feature-gated via PYARROW_TEST_*). Feature set (workflow env, single source of truth): Parquet, Dataset, Acero, Compute, CSV, JSON, Filesystem, HDFS, ORC, Substrait, Flight, Parquet-encryption + all compression. S3/GCS/Azure/Gandiva off for now — their dep trees (aws-sdk/google-cloud-cpp/azure-sdk/LLVM) are unverified on riscv64 and will be enabled incrementally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

luhenry · 2026-08-20T19:02:29Z

@threexc @justeph that's ready, PTAL, thank you! :)

With the OpenSSL fix in, the build got 1h37m deep and then failed compiling bundled c-ares: "redefinition of hash_func/bucket_key/bucket_free/key_eq", all under c-ares.dir/Unity/unity_*.c. CMAKE_UNITY_BUILD=ON (copied from upstream) merges each target's .c files into a single translation unit; c-ares reuses the same file-local `static` symbol names across ares_htable_vpvp.c / ares_htable_vpstr.c / etc. — fine per-file, a redefinition once unity-merged. Upstream never hits this because their c-ares comes prebuilt from the vcpkg cache; our BUNDLED build compiles it from source, and Arrow adds a unity-OFF guard for re2/grpc/protobuf but not for c-ares. Unity build is only a compile-speed optimization, so disable it globally — robust across the whole bundled dep tree, which will keep including deps upstream never had to make unity-clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The `: "${VAR:=default}"` idiom traces under `set -x` as a bare `+ : ON`, with no hint which flag `ON` belongs to — so the build log's feature-flag dump was an unreadable column of `+ : ON` / `+ : OFF`. Switch to `VAR="${VAR:-default}"` (and self-assignment for the `:?` required-var guards): identical semantics — default applied when unset, hard fail when a required var is missing — but the trace now reads `+ ARROW_DATASET=ON`, naming each variable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Replace the single build job that looped all four interpreters (one Arrow C++ build shared across them) with a matrix over python-tag: one job per cp312/cp313/cp314/cp314t. Each job rebuilds Arrow C++ from source — deliberately duplicated — so the four run in parallel in roughly the wall-clock of one, and a single interpreter failing (fail-fast: false) no longer blocks the rest. The ccache mount means parallel jobs sharing a runner still hit warm cache. PYTHON_TAGS now comes from matrix.python-tag per job (the build/test scripts loop over it, so a single value just iterates once — no script change). Each job uploads its own artifact (name carries the python-tag so parallel uploads don't collide); publish-wheels' artifact-pattern + merge-multiple recombine them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Now that each matrix job builds exactly one interpreter, the `for pytag in ${PYTHON_TAGS}` loops (a leftover from the pre-matrix single job) iterated once and only obscured the logic. Rename PYTHON_TAGS -> PYTHON_TAG throughout and inline both loop bodies as straight-line code. Also drop the per-iteration `rm -rf python/build dist _skbuild` — a fresh job has nothing to clean. No behavior change; the build/test scripts just read the one tag directly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Rename the matrix key python-tag -> python and bake ${{ matrix.python }} straight into the build/test script text instead of routing it through a PYTHON_TAG env var and a docker -e passthrough. GitHub expands ${{ }} in the whole run: block before the heredoc is written, so each job's script gets the literal interpreter tag (e.g. /opt/python/cp312-cp312/bin/python); the tag has no shell metacharacters so direct embedding is safe. Drops the now-unused PYTHON_TAG plumbing and the required-var guard for it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Instead of `cat >`-writing pyarrow_build.sh / pyarrow_test.sh to the workspace, bind-mounting them, and running `bash /pyarrow_*.sh`, feed each script straight into the container on stdin: `docker run -i ... bash -s <<'EOF'`. This drops the two separate "Write ... script" steps and the script bind-mounts, folding each build/test into a single step. `-i` is required so docker forwards stdin to the container; the quoted heredoc keeps the runner shell from expanding `$...` while GitHub still substitutes ${{ matrix.python }} in the run: block beforehand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

These workflows are mined as copy-paste reference, so verbose per-step narration makes it look like we customized more than we did (PR #308 review). Cut the multi-line paragraphs and step narration down to single "why" lines, keeping only the non-obvious: BUNDLED-not-vcpkg, the OpenSSL-shared and UNITY_BUILD-off riscv deviations, registry-only deps, and the pandas test fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

luhenry mentioned this pull request Aug 20, 2026

workflows: add build-pyarrow.yml for riscv64 manywheel builds #292

Closed

luhenry changed the title ~~pyarrow: build wheels for riscv64~~ workflows: add build-pyarrow.yml for riscv64 manywheel builds Aug 20, 2026

luhenry force-pushed the pyarrow branch from 453d75a to 6f1d1db Compare August 20, 2026 18:58

luhenry force-pushed the pyarrow branch from 6f1d1db to c554841 Compare August 20, 2026 19:00

luhenry requested review from justeph and threexc August 20, 2026 19:02

luhenry and others added 2 commits August 21, 2026 08:31

luhenry marked this pull request as draft August 21, 2026 14:15

luhenry and others added 4 commits August 24, 2026 11:55

threexc marked this pull request as ready for review August 24, 2026 13:48

luhenry linked an issue Aug 24, 2026 that may be closed by this pull request

pyarrow riscv64 support #85

Open

Navigation Menu

Uh oh!

workflows: add build-pyarrow.yml for riscv64 manywheel builds - #294

workflows: add build-pyarrow.yml for riscv64 manywheel builds#294
luhenry wants to merge 8 commits into
mainfrom
pyarrow

luhenry commented Aug 20, 2026 •

edited

Loading

Uh oh!

luhenry commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

luhenry commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Shape

Feature set (workflow env, single source of truth)

Notes for review

Uh oh!

luhenry commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

luhenry commented Aug 20, 2026 •

edited

Loading

Feature set (workflow `env`, single source of truth)