gh-143120: pixi builds for free-threading and TSAN #142872

crusaderky · 2025-12-17T12:08:15Z

Follow-up to #142469

Add pixi infrastructure for
- freethreading: compiles with --disable-gil
- tsan-freethreading: compiles with --disable-gil --with-thread-sanitizer
Refactor recipes:
- Make recipe.yaml and pixi.toml all identical symlinks
- Remove clang-19 since we are not building the experimental jit
- Remove ld_impl because we are not hacking the makefiles
- Add site_packages_path
Add support for MacOS Intel (specifically macos-15-intel github runners)
Signal support for Linux ARM (it already worked but was insofar untested)
Remove ASAN/TSAN DLLs hack on MacOS

Tested in CI on https://github.com/crusaderky/hdf5-pixi

Issue: Add pixi builds for TSan + free-threading to aid downstream development. #143120

crusaderky · 2025-12-17T12:08:51Z

Tools/pixi-packages/free-threading/recipe.yaml

+  script:
+    file: ../build.sh
+    env:
+      PYTHON_VARIANT: "free-threading"


copy-paste of asan/recipe.yaml, except this one line

crusaderky · 2025-12-17T12:09:04Z

Tools/pixi-packages/tsan-free-threading/pixi.toml

copy-paste of asan/pixi.toml

crusaderky · 2025-12-17T12:09:21Z

Tools/pixi-packages/tsan/recipe.yaml

+  script:
+    file: ../build.sh
+    env:
+      PYTHON_VARIANT: "tsan"


copy-paste of asan/recipe.yaml, except this one line

lucascolley · 2025-12-17T12:10:08Z

@kumaraditya303 @lucascolley any idea what's wrong here?

no clue sorry I haven't seen this before

crusaderky · 2025-12-18T17:23:27Z

The tsan crash is due to the default mmap_rnd_bits on Ubuntu 24.04; it goes away after lowering it:

$ sudo sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 32  # too high
$ sudo sysctl vm.mmap_rnd_bits=28  # reduce it
vm.mmap_rnd_bits = 28

ubuntu-latest github actions workers seems to be unaffected.

hugovk · 2025-12-23T21:56:05Z

There's a lot of copy/paste here, which can make maintenance harder. Does pixi have some concept of code reuse or parametrisation? Maybe not the best example, but something like GitHub Actions' reusable workflows?

ngoldbaum · 2025-12-23T21:58:36Z

@lucascolley is there anything we can do right now to reduce the duplication here? Or does that require fixes in pixi?

lucascolley · 2025-12-23T21:59:35Z

As mentioned in the README already, this is blocked on prefix-dev/pixi#4599.

ngoldbaum

I opened an issue which is needed to make this mergeable. I'm also playing with a NumPy recipe based on this.

ngoldbaum · 2025-12-23T20:34:55Z

Tools/pixi-packages/README.md

- `asan`: ASan-instrumented build with `PYTHON_ASAN=1`
+- `free-threading`
+- `asan`: ASan-instrumented build
+- `tsan`: free-threading, TSan-instrumented build


Should probably be called tsan-free-threading. There should probably also be an asan-free-threading. @lucascolley is there a way to avoid the combinatorial explosion of variants here? Ideally you'd be able to somehow combine these but I think that's probably not tenable at the moment.

I think the answer to this is no because of prefix-dev/pixi#4599
Barring the above, we could write a template + generation script and then commit the output to git? Very ugly if you ask me.

Yes, I would vote for keeping things simple but verbose at the minute until there is a proper solution upstream

Renamed to tsan-free-threading

Tools/pixi-packages/build.sh

crusaderky · 2026-01-02T16:16:43Z

Ready for final review

Tools/pixi-packages/build.sh

isuruf · 2026-01-15T19:57:51Z

@isuruf at the end of the numpy build I get
│ │ ⚠ warning Overdepending against python_abi
but I can't figure out if it's actually a problem or what it is about; could you investigate?

You can ignore it. Harmless warning.

crusaderky · 2026-01-20T13:59:59Z

This is out of scope of this PR and can be addressed in a follow-up.
Compilation on macos-15-intel gh action runners fails with

│ │ $SRC_DIR/Objects/mimalloc/prim/osx/../unix/prim.c:559:17: error: '_CLOCK_MONOTONIC' is only available on macOS 10.12 or newer [-Werror,-Wunguarded-availabi
│ │ lity]

As the image is macOS 15, this feels like it should be fixable with an env variable?
Not entirely sure what VM images conda-forge uses to build mac intel.
@isuruf did you hit this before?

Tools/pixi-packages/build.sh

Tools/pixi-packages/README.md

Co-authored-by: Lucas Colley <lucas.colley8@gmail.com>

crusaderky · 2026-01-22T12:13:23Z

@isuruf the libasan fix works in my hdf5 recipe, but not in the python recipe:
https://github.com/crusaderky/hdf5-pixi/actions/runs/21245433717/job/61133422587?pr=8
This is as of fd6ca6d.

Reverting for now.

This reverts commit ad6c740.

This reverts commit 2649286.

isuruf · 2026-01-22T15:11:18Z

Your hdf5 recipe is still not using fd6ca6d. Can we please stop the reverting back and forth and debug why the correct commit from cpython is not being used?

crusaderky · 2026-01-22T16:34:20Z

For the sake of test velocity and thoroughness, I've created https://github.com/crusaderky/cpython-pixi
Latest Python and Python+NumPy builds can be seen here.

@isuruf the latest macos-15-intel changes let the python build complete; thank you.
Now however I'm seeing similar failures when building numpy ASAN/TSAN on the same VMs (not blocking this PR).

Pending

(blocking) Confirmation that symlinks in the repository are OK
(not blocking) Investigate why the latest libcompiler-rt fixes macos ASAN/TSAN builds for my hdf5-pixi project, but not for python. I don't have a mac so my velocity here is very limited. Note that the changes to this PR have been reverted in the last two commits.

lucascolley · 2026-01-22T16:41:16Z

(blocking) Confirmation that symlinks in the repository are OK

@hugovk would you be able to advise?

crusaderky · 2026-01-22T16:50:05Z

Your hdf5 recipe is still not using fd6ca6d. Can we please stop the reverting back and forth and debug why the correct commit from cpython is not being used?

@isuruf I appreciate your frustration. I'm frustrated too by this issue. I'm fairly sure that the last CI run I pointed to shows the effect of the libcompiler-rt changes - because I checked - but I'm fallible, so I'll do it again this time making extra sure not to muddle any evidence afterwards.

For the sake of velocity and clarity I've moved my CI to https://github.com/crusaderky/cpython-pixi
It takes several hours for CI to finish. I'll report back.

hugovk · 2026-01-22T19:36:25Z

(blocking) Confirmation that symlinks in the repository are OK

@hugovk would you be able to advise?

Let's not add symlinks to the repo, they're harder to use on Windows and make cross-platform support generally more complex.

bedevere-app bot added the awaiting review label Dec 17, 2025

crusaderky commented Dec 17, 2025

View reviewed changes

Tools/pixi-packages/tsan-free-threading/pixi.toml Outdated

Copy link

Contributor Author

crusaderky Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy-paste of asan/pixi.toml

crusaderky commented Dec 17, 2025

View reviewed changes

crusaderky marked this pull request as draft December 17, 2025 12:10

bedevere-app bot removed the awaiting review label Dec 17, 2025

crusaderky changed the title ~~WIP pixi builds for free-threading and tsan~~ WIP pixi builds for free-threading and TSAN Dec 17, 2025

This was referenced Dec 23, 2025

Add pixi builds for TSan + free-threading to aid downstream development. #143120

Open

Add note to reduce ASLR randomness for TSan Quansight-Labs/free-threaded-compatibility#304

Merged

hugovk changed the title ~~WIP pixi builds for free-threading and TSAN~~ gh-143120: WIP pixi builds for free-threading and TSAN Dec 23, 2025

hugovk added the skip news label Dec 23, 2025

hugovk added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Dec 23, 2025

ngoldbaum reviewed Dec 23, 2025

View reviewed changes

ngoldbaum mentioned this pull request Dec 23, 2025

ENH: TSan pixi package definitions numpy/numpy#30510

Draft

crusaderky force-pushed the tsan branch from eb0149f to 242081e Compare January 2, 2026 15:53

crusaderky commented Jan 2, 2026

View reviewed changes

Tools/pixi-packages/build.sh Show resolved Hide resolved

crusaderky added 4 commits January 2, 2026 16:03

pixi builds for free-threading and tsan

d930e12

Rename to tsan-free-threading

57d45b9

Add libclang_rt hack

5ff29b4

Add troubleshooting for tsan

9a2a896

crusaderky force-pushed the tsan branch from b528931 to 9a2a896 Compare January 2, 2026 16:03

crusaderky marked this pull request as ready for review January 2, 2026 16:05

bedevere-app bot added the awaiting review label Jan 2, 2026

crusaderky changed the title ~~gh-143120: WIP pixi builds for free-threading and TSAN~~ gh-143120: pixi builds for free-threading and TSAN Jan 2, 2026

crusaderky marked this pull request as ready for review January 15, 2026 13:04

bedevere-app bot added the awaiting review label Jan 15, 2026

crusaderky added 2 commits January 15, 2026 13:09

README

cd4cd26

Merge remote-tracking branch 'upstream/main' into tsan

f028ded

crusaderky commented Jan 15, 2026

View reviewed changes

Tools/pixi-packages/build.sh Show resolved Hide resolved

Use libcompiler-rt and remove hack

a86d345

ngoldbaum reviewed Jan 20, 2026

View reviewed changes

Tools/pixi-packages/build.sh Show resolved Hide resolved

isuruf and others added 4 commits January 20, 2026 09:43

use at least macos 10.12

0aa5872

Merge remote-tracking branch 'upstream/main' into tsan

62b7d1c

Use libcompiler-rt and remove hack

ad6c740

note

fd6ca6d

lucascolley reviewed Jan 21, 2026

View reviewed changes

Tools/pixi-packages/README.md Show resolved Hide resolved

Fix CPPFLAGS too

abae5a5

crusaderky force-pushed the tsan branch from a5b2d1c to fd6ca6d Compare January 22, 2026 10:41

crusaderky and others added 6 commits January 22, 2026 10:41

Merge remote-tracking branch 'upstream/main' into tsan

a3f59cc

Merge remote-tracking branch 'upstream/main' into tsan

fd87e84

Update Tools/pixi-packages/README.md

2649286

Co-authored-by: Lucas Colley <lucas.colley8@gmail.com>

Update platforms

0e98a4a

Merge remote-tracking branch 'isuruf/libcompiler-rt' into tsan

4f49228

Merge branch 'main' into tsan

7b8f9d3

crusaderky added 2 commits January 22, 2026 12:21

Revert "Use libcompiler-rt and remove hack"

1ae77a5

This reverts commit ad6c740.

Revert "Update Tools/pixi-packages/README.md"

cddc933

This reverts commit 2649286.

Uh oh!

gh-143120: pixi builds for free-threading and TSAN #142872

Are you sure you want to change the base?

gh-143120: pixi builds for free-threading and TSAN #142872

Conversation

crusaderky commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crusaderky Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

crusaderky Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

crusaderky Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

lucascolley commented Dec 17, 2025

Uh oh!

crusaderky commented Dec 18, 2025

Uh oh!

hugovk commented Dec 23, 2025

Uh oh!

ngoldbaum commented Dec 23, 2025

Uh oh!

lucascolley commented Dec 23, 2025

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

ngoldbaum Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

crusaderky Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

lucascolley Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

crusaderky Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

crusaderky commented Jan 2, 2026

Uh oh!

Uh oh!

isuruf commented Jan 15, 2026

Uh oh!

crusaderky commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

crusaderky commented Jan 22, 2026

Uh oh!

isuruf commented Jan 22, 2026

Uh oh!

crusaderky commented Jan 22, 2026

Pending

Uh oh!

lucascolley commented Jan 22, 2026

Uh oh!

crusaderky commented Jan 22, 2026

Uh oh!

hugovk commented Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

crusaderky commented Dec 17, 2025 •

edited

Loading

crusaderky commented Jan 20, 2026 •

edited

Loading