Skip to content

Thread limit introspection, part 2: Empirical observation in CI - #224

Merged
ogrisel merged 13 commits into
joblib:masterfrom
itamarst:214-limit-semantics-introspection-api
Sep 4, 2026
Merged

Thread limit introspection, part 2: Empirical observation in CI#224
ogrisel merged 13 commits into
joblib:masterfrom
itamarst:214-limit-semantics-introspection-api

Conversation

@itamarst

Copy link
Copy Markdown
Contributor

Fixes #214

In theory one could, say, distribute another package in PyPI/conda-forge of compiled OpenMP code so users can run this locally. Or I guess ... pick some part of e.g. scikit-learn that is unlikely to change but that seems brittle too. So this could be better, and maybe sometime it's worth doing that.

But at least it should run in CI and give us useful info, so it's a good first step.

@itamarst

itamarst commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Looks like occasionally the output is erroneously "not sure", but mostly gets everything right.

@itamarst
itamarst marked this pull request as ready for review July 13, 2026 18:53
@ogrisel

ogrisel commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Is this PR still needed if we merge #228?

@itamarst

itamarst commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

I think so - the technique I use for testing in #228 is Linux-specific (strace). There's apparently dtrace on macOS but that's restricted by System Integrity Protection, not sure what you'd do on Windows, etc.. So this PR is a nice OS-agnostic approach that is still informative.

@ogrisel

ogrisel commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Ok, I'll add it to my review backlog.

For Windows, feel free to take over or inspiration from this vibecoded proof of concept: #221.

@ogrisel ogrisel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some improvement suggestions but otherwise LGTM.

Comment thread tests/empirical_scope_observation.py
Comment thread tests/empirical_scope_observation.py Outdated
Comment thread tests/empirical_scope_observation.py Outdated
Comment thread tests/empirical_scope_observation.py Outdated
Comment thread tests/empirical_scope_observation.py Outdated
@ogrisel

ogrisel commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Note: I noticed that threadpoolctl.threadpool_info fails to detect opennblas (or any other BLAS) on some Windows builds. The conda package list still reports openblas as installed and therefore probably used by numpy. This is not a problem for that PR but this is probably a new bug.

EDIT: I opened #231 to track that issue.

@itamarst

itamarst commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

OK I've addressed all review comments.

@ogrisel
ogrisel merged commit 46b5ca8 into joblib:master Sep 4, 2026
20 of 22 checks passed
@ogrisel

ogrisel commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Merged! Thanks again @itamarst.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API to introspect semantic meaning of setting a thread pool limit

2 participants