Stable ABI support

[rgommers]

Stable ABI support is pretty actively being worked on; there are a lot of moving parts so this issue aims to link to those.

There are two competing PEPs (a lot of the design is identical, but they have different naming, wheel tags and compatibility/stability rules):

• PEP 803 – Stable ABI for Free-Threaded Builds
  • DPO discussion thread
  • DPO discussion thread (packaging)
  • C API Workgroup issue on PEP 803 & opaque PyObject
• PEP 809 – Stable ABI for the Future
  • DPO discussion thread
• And both rely on PEP 793 – PyModExport: A new entry point for C extension modules (accepted)

Support in binding generators:

• Cython: [INFO] Limited API and freethreading Python 3.15 cython/cython#7399
• PyO3:
  • Implement support for PEP 793 PyO3/pyo3#5644
  • Add support for the opaque PyObject ABI PyO3/pyo3#5786
  • Opaque PyObject ABI support PyO3/pyo3#5807
• Nanobind:
  • Opinions wanted for PEP 793 – PyModExport: A new entry point for C extension modules wjakob/nanobind#1187

Build & packaging tools:

• Setuptools:
  • Remove use of /EXPORT for PyInit_* on Windows: Don't automatically add PyInit_<name> to export symbols in Python 3.15+ pypa/distutils#395
  • Add initial support for abi3.abi3t tag from PEP 803 pypa/setuptools#5193
• meson-python:
  • Test PyModExport usage: TST: use PyModExport hook in one test package for Python >=3.15 mesonbuild/meson-python#833
• scikit-build-core: Explore support for abi3t ABI tag scikit-build/scikit-build-core#1252
• Maturin:
  • Explore support for abi3.abi3t wheels (PEP 803) PyO3/maturin#3064
• Meson:
• CMake:
• pypa/packaging:
  • Add support for abi3.abi3t tag from PEP 803 pypa/packaging#1099
• Pip:
• uv:

Other:

• NumPy C API: Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809) numpy/numpy#30704
• Test suite with test packages covering a matrix of "build backend x binding generator": https://github.com/Quansight-Labs/stable-abi-testing/

[kumaraditya303]

nanobind stable ABI support

nanobind uses PyUnstable_TryIncRef¹ to implement a weakmap ² to map C++ to the corresponding Python object. That API is unstable and will not make it into the stable ABI so for stable ABI it would need a new implementation which would ideally work on both ft and non-ft builds at the cost of lesser performance. Also, we cannot just change the weakmap to a strong map because doing so would hold Python object until the interpreter is finalized so would consume a lot of memory.

I think the new implementation would have to use Python's weakrefs to handle borrowed references correctly.

Slightly related, asyncio has a very similar similar weakmap implementation now which was previously using weakrefs in Python < 3.14 so it might make sense to use that as basis for implementing it.

Footnotes

1. https://github.com/wjakob/nanobind/blob/301596fe1060c741162316072697d570516d51eb/src/nb_ft.h#L32 ↩

2. https://github.com/wjakob/nanobind/blob/301596fe1060c741162316072697d570516d51eb/src/nb_type.cpp#L93-L100 ↩

[ngoldbaum]

During today's meeting we identified working on adding the PyMutex and critical section APIs to the stable API as a medium-term priority.

It would help to find concrete packages that need these APIs we can point to in our argument to the C API workgroup. Off the top of my head:

• CFFI uses PyMutex
• PyO3 (and any package written using PyO3) uses the critical section APIs in several places.
• Cython has wrappers for the critical_section API that are being used in the ecosystem.

[da-woods]

Cython has wrappers for the critical_section API that are being used in the ecosystem.

Cython also uses them internally. E.g. to guard PyDict_Next where needed, to prevent our CyFunction class getting corrupted (that's a bit of an edge case though...), some locking around "do once" imports.

I don't think this should be controversial with the C API workgroup though - PEP 803 says that these types will be exposed so I think it's waiting on either someone to do it, or a PEP to be approved.

I'd start from a position of "you said you'd do it - when is it planned?" rather than spending time trying to build a case for it.

[ngoldbaum]

I opened a tracking issue for CFFI: python-cffi/cffi#226

[ngoldbaum]

I also opened a discussion for nanobind: wjakob/nanobind#1284

[ngoldbaum]

I opened a tracking issue for PyO3: PyO3/pyo3#5786

[ngoldbaum]

I just edited the issue description to add links to work I've done in the past week or two.