Document using limiting APIs for Python thread pools - #234
Conversation
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
|
I think once this merged maybe this could be released? |
There was a problem hiding this comment.
A first pass of feedback. I think it would be helpful if someone else who did not review all your past PRs could have a fresh look at this. Maybe @cakedev0, @lesteve and/or @jeremiedbb?
| ### Switching Back And Forth Between Main Thread and Python Threads | ||
|
|
||
| Unfortunately not all controlled libraries providing limiting APIs that are | ||
| thread-specific. Limiting some libraries' thread pool sizes can therefore impact |
There was a problem hiding this comment.
I think it would be helpful to maintain a separate markdown file with a reference assesment of the known semantics of common libraries that we consolidated by running the helpers from #213 and #214 on our CI.
Here is my current understanding:
- OpenMP (including BLAS runtimes configured with an OpenMP threading layer):
- libgomp (GCC) and libomp (LLVM): thread-local limits and thread-pools as per the OpenMP 3.0+ spec
- msvc: process-wide limits and shared thread-pool (as per the OpenMP 2.0 spec)
- OpenBLAS with the pthreads threading layer: process-wide limits and shared thread-pool
- OpenBLAS with the windows threading layer: process-wide limits and shared thread-pool
- MKL (all threading layers?): thread-local limits and thread-pools
Alternatively, we could have a short README section that explains how to use the CLI and the Python script that allows the user to dynamically inspect the effective semantics as we do on our CI.
There was a problem hiding this comment.
I went with the second approach, let me know what you think.
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
2 participants
Fixes #215