Skip to content

av: add build-av.yml for riscv64 wheels - #461

Draft
luhenry wants to merge 5 commits into
mainfrom
av
Draft

av: add build-av.yml for riscv64 wheels#461
luhenry wants to merge 5 commits into
mainfrom
av

Conversation

@luhenry

@luhenry luhenry commented Aug 26, 2026

Copy link
Copy Markdown
Member

Cython bindings over FFmpeg. Upstream publishes no riscv64 wheel.

Mirrors upstream's tests.yml package-wheel job, including its CIBW_BUILD selection and its vendored-FFmpeg fetch step.

Differs from upstream

  • PIP_EXTRA_INDEX_URL added — numpy resolves from our registry.
  • Licence texts for the bundled FFmpeg stack are packaged, and its sources published.

Matrix: cp311-abi3 and cp314-cp314t, as upstream builds — PyAV's extensions are limited-API.

Testing

  • same as upstream

License: Wheel bundles libx264 and libx265 (GPL-2.0-or-later) plus FFmpeg, GnuTLS, Nettle, GMP, libunistring, alsa-lib and LAME (LGPL) and several permissive libraries; upstream ships no licence text for any of them, so the build adds it and publishes their sources as a release asset.

Built on cp311 and cp314t, manylinux and musllinux; 490 passed, 38 skipped.

luhenry and others added 2 commits August 26, 2026 20:12
PyAV compiles ~49 Cython extension modules against FFmpeg. Upstream's
tests.yml fetches a prebuilt FFmpeg tarball from PyAV-Org/pyav-ffmpeg in
CIBW_BEFORE_BUILD; that project already publishes
ffmpeg-manylinux-riscv64.tar.gz, so the recipe carries over unchanged and
auditwheel bundles the FFmpeg libraries into the wheel.

setup.py turns on Py_LIMITED_API (cp311) for CPython 3.11-3.13, so the
build on our cp312 floor emits the same cp311-abi3 wheel upstream ships;
free-threaded builds get their own cp314-cp314t wheel. Upstream builds on
cp311, which is below this repo's floor.

Validated on aarch64 in quay.io/pypa/manylinux_2_39_aarch64: the wheel
builds, auditwheel repairs it to 34MB with 32 bundled libraries, and
upstream's test command passes 490 tests against the installed wheel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
setup.py tags the wheel cp311-abi3, so the stable-ABI guarantee only holds
if it is compiled against 3.11 headers; building it on cp312 would ship a
wheel pip installs on 3.11 that may fail at runtime there. Upstream builds
on cp311 for the same reason. numpy resolves to 2.4.3 on 3.11, which our
registry ships for riscv64.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@luhenry

luhenry commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

@threexc what should we do here for the licensing? I'm not sure I understand the situation exactly

@luhenry
luhenry marked this pull request as draft August 26, 2026 19:24
@luhenry
luhenry marked this pull request as ready for review August 27, 2026 00:18
@luhenry

luhenry commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

We will need to release the sources of the GPL binaries as well. I see that https://github.com/PyAV-Org/PyAV#installing-from-source outlines how to build from source. Maybe we should be using that on our CI?

From looking at https://github.com/PyAV-Org/PyAV/blob/main/.github/workflows/tests.yml, we must be doing the same of gathering the sources and shipping them: https://github.com/PyAV-Org/PyAV/blob/main/.github/workflows/tests.yml#L7-L23

@luhenry
luhenry marked this pull request as draft August 27, 2026 00:35
luhenry and others added 3 commits August 27, 2026 02:38
Upstream builds both wheels in one cibuildwheel invocation with
CIBW_BUILD: "cp311* cp314t*" and sets CIBW_ENVIRONMENT_LINUX; this was a
two-entry matrix with per-entry only:, and CIBW_ENVIRONMENT. Nothing about
riscv64 required either change, so drop them.
Collapsing the matrix to one job renamed the artifact to
av-<version>-manylinux_riscv64, but artifact-pattern still expected a
segment between the version and the platform, so it matched nothing.
The wheels bundle 33 prebuilt shared libraries fetched from a pyav-ffmpeg
release, and shipped only PyAV's own BSD-3-Clause LICENSE.txt.

Two of those libraries are GPL. pyav-ffmpeg patches FFmpeg's configure to
move libx264 and libx265 out of EXTERNAL_LIBRARY_GPL_LIST into
EXTERNAL_LIBRARY_VERSION3_LIST, so FFmpeg builds without --enable-gpl and
reports "LGPL version 3 or later" while libavcodec still has DT_NEEDED on
libx264.so.165 and libx265.so.216, both GPL-2.0-or-later. The rest of the
stack is LGPL (FFmpeg, GnuTLS, Nettle, GMP, libunistring, alsa-lib, LAME)
or permissive.

A new vendor_sources job reads the URL and SHA-256 pins out of
pyav-ffmpeg's scripts/pkg.py, downloads every source tarball plus the
pyav-ffmpeg tree that carries the build recipe and its FFmpeg/GMP/LAME/vpx
patches, and extracts each project's licence texts. The sources are
published as a GitHub Release asset; the licence texts are staged into the
checkout so setuptools' default glob ships them in dist-info/licenses, and
the post-build check asserts the whole set is present in every wheel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@luhenry

luhenry commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

Here is what is actually in the wheel, and what the latest push does about it.

The situation. av.libs/ holds 33 prebuilt shared libraries fetched from pyav-ffmpeg 8.1.2-1. Two of them are GPL, and it is easy to miss why: pyav-ffmpeg patches FFmpeg's configure to move libx264 and libx265 out of EXTERNAL_LIBRARY_GPL_LIST into EXTERNAL_LIBRARY_VERSION3_LIST. FFmpeg therefore builds without --enable-gpl and every libav*.so reports license: LGPL version 3 or later — but libavcodec still has DT_NEEDED on libx264.so.165 and libx265.so.216, and both ship in the wheel. x264 and x265 are GPL-2.0-or-later. So the wheel as we would distribute it is a GPL work, whatever FFmpeg's own version string says. The rest is LGPL (FFmpeg, GnuTLS, Nettle, GMP, libunistring, alsa-lib, LAME) or permissive.

Before this push the wheel shipped only PyAV's own BSD-3-Clause LICENSE.txt, for any of that.

What the push does. A vendor_sources job reads the URL and SHA-256 pins out of pyav-ffmpeg's scripts/pkg.py, downloads all 17 source tarballs plus the pyav-ffmpeg tree itself (it carries the build recipe and the FFmpeg/GMP/LAME/libvpx patches, so it is part of the corresponding source), and pulls each project's licence texts out of the same tarballs. The sources go to a GitHub Release asset via publish-wheels; the licence texts end up in dist-info/licenses/, and the post-build check fails if any is missing.

On building FFmpeg ourselves. tests.yml L7-23 is upstream's package-source job — it builds PyAV's own sdist, not FFmpeg's sources, so it does not help here. We could run pyav-ffmpeg's build-ffmpeg.py on our runners instead, but it compiles 17 packages and would be a large divergence for the same compliance outcome. Fetching the pinned sources gets us there without it.

Three things left for you to decide.

  1. METADATA still says License-Expression: BSD-3-Clause, which is upstream's on every platform and now understates what the wheel contains. Overriding it is a divergence, so I left it alone.
  2. auditwheel also vendors libxcb, libXau and libdrm from the Rocky 10 build image (per the wheel's own sboms/auditwheel.cdx.json). All MIT, so notice only, no source obligation — currently not shipped. Easy to add if you want it.
  3. If shipping GPL at all is not acceptable, the only way out is building FFmpeg without --enable-libx264/--enable-libx265, which means dropping the pyav-ffmpeg tarball entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant