Document using limiting APIs for Python thread pools

README.md

@@ -8,18 +8,9 @@ Fine control of the underlying thread-pool size can be useful in
 workloads that involve nested parallelism so as to mitigate
 oversubscription issues.
 
-> **Important:** In its current state, `threadpoolctl` is only designed for
-> situations where BLAS and OpenMP are only called from the main Python thread.
-> Or, to be more accurate, `threadpoolctl` and BLAS/OpenMP APIs should only ever
-> called from the same, single Python thread. For example:
->
-> * When you're using it to configure a worker in a process pool, which then calls BLAS or OpenMP APIs directly in the main thread.
-> * A Jupyter notebook, where the BLAS or OpenMP APIs are being called from code running in the cell's main thread.
->
-> However, once you start calling BLAS or OpenMP APIs and `threadpoolctl` from
-> multiple different Python threads, the impact of the `threadpoolctl` limiting
-> APIs will be very inconsistent. For more details and a plan to fix this, see
-> https://github.com/joblib/threadpoolctl/issues/208
+Note that "limiting the number of threads" in practice involves a variety of
+potential semantics depending on the underlying third-party library; see the
+section on [semantics](#semantics) below.
 
 ## Installation
 
@@ -43,7 +34,7 @@ oversubscription issues.
   pytest
   ```
 
-## Usage
+## Usage: Introspection and debugging
 
 ### Command Line Interface
 
@@ -152,6 +143,22 @@ The state of these libraries is also accessible through the object oriented API:
 True
 ```
 
+## Usage when not using Python threads: Restricting Controlled Library Thread Pool Sizes
+
+There are two scenarios in which you might want to use `threadpoolctl`; each
+requires you to use different APIs.
+
+1. You do not expect to use any Python threads, so all the work will be started
+   directly from the main thread in the process. This is a simple case
+   where we can globally set thread limits.
+2. You will be parallelizing work using a Python thread pool, and your goal is
+   therefore to limit controlled libraries' thread pool sizes when
+   concurrently called from Python threads. This case is a bit more
+   complex to handle properly and requires a bit more verbose code.
+
+This section will cover the former case, and the latter is covered in the next
+usage section.
+
 ### Setting the Maximum Size of Thread-Pools
 
 Control the number of threads used by the underlying runtime libraries
@@ -184,11 +191,11 @@ however not act on libraries loaded after the instantiation of the
 ...     a_squared = a @ a
 ```
 
-### Restricting the limits to the scope of a function
+### Restricting the Limits to the Scope of a Function
 
 `threadpool_limits` and `ThreadpoolController` can also be used as decorators to set
 the maximum number of threads used by the supported libraries at a function level. The
-decorators are accessible through their `wrap` method:
+decorators are accessible through their `wrap` method.
 
 ```python
 >>> from threadpoolctl import ThreadpoolController, threadpool_limits
@@ -205,6 +212,123 @@ decorators are accessible through their `wrap` method:
 ...
 ```
 
+## Usage for Python threads: Restricting Controlled Library Thread Pool Sizes
+
+This section covers APIs to use when you will be using Python thread pools to parallelize work.
+The usage is more complicated than one might expect because the underlying APIs have a variety of different semantics, as [explained later in the docs](#semantics).
+
+### Setting the Maximum Size of Thread-Pools, When Python Thread Pools Are Used
+
+Limiting thread pool size in controlled libraries requires a two-step process.
+**Importantly, each Python worker thread must also call a method to limit
+controlled libraries in that thread.** With Python's
+`concurrent.futures.ThreadPoolExecutor`, you can do so by passing in an
+initializer function that will get called on thread startup.
+
+```python
+from threadpoolctl import threadpool_limits
+from concurrent.futures import ThreadPoolExecutor
+
+# This top-level limiter doesn't actually change the limits initially; it is
+# there to ensure the limits are reset _after_ the Python thread pool is done.
+# This is necessary because some underlying limiting APIs operate on a
+# process-wide basis.
+with threadpool_limits():
+    # Make sure each Python worker thread also calls threadpool_limits(). If
+    # you're using another thread pool class, you will need to do so some other
+    # way.
+    with ThreadPoolExecutor(4, initializer=lambda: threadpool_limits(limits=1)) as pool:
+        # ... run some BLAS-using code in the thread pool ...
+        pool.map(somefunc, someargs)
+```
+
+Whenever `threadpool_limits` is called, it needs to do some work (inspecting and getting access to third-party shared libraries) that can take some time.
+To prevent the performance cost of doing this work every time, you can reuse a
+`ThreadpoolController` object:
+
+```python
+from threadpoolctl import ThreadpoolController
+
+# This won't have any side-effects:
+CONTROLLER = ThreadpoolController()
+
+with (
+    CONTROLLER.limit(),
+    ThreadPoolExecutor(4, initializer=lambda: CONTROLLER.limit(limits=1)) as pool,
+):
+    # ... run some BLAS-using code in the thread pool ...
+    pool.map(somefunc, someargs)
+
+# Later...
+with (
+    CONTROLLER.limit(),
+    ThreadPoolExecutor(4, initializer=lambda: CONTROLLER.limit(limits=2)) as pool,
+):
+    # ... run some BLAS-using code in the thread pool ...
+    pool.map(somefunc, someargs)
+```
+
+You can also operate without a context manager:
+
+```python
+from threadpoolctl import ThreadpoolController
+
+CONTROLLER = ThreadpoolController()
+try:
+    limiter = CONTROLLER.limit()
+    with ThreadPoolExecutor(
+            4, initializer=lambda: CONTROLLER.limit(limits=1)) as pool:
+        # ... run some BLAS-using code in the thread pool ...
+        pool.map(somefunc, someargs)
+finally:
+    limiter.restore_original_limits()
+
+```
+
+### Switching Back And Forth Between Main Thread and Python Threads
+
+Unfortunately not all controlled libraries providing limiting APIs that are
+thread-specific, as detailed in the section on [semantics](#semantics) below.
+Limiting some libraries' thread pool sizes can therefore impact the whole
+process. This makes switching back and forth between running code that
+uses these libraries in Python threads and running it in the main thread a bit
+more complex: you need to set the limits each time you switch back and forth.
+
+Let's say your computer has 4 cores, and you're using some OpenMP API.
+
+```python
+POOL = ThreadPoolExecutor(4)
+CONTROLLER = ThreadpoolController()
+
+# 1. Run some work in a Python thread pool (OpenMP effectively disabled).
+with CONTROLLER.limit(limits=1):
+
+    def limit_then_do_work(*args, **kwargs):
+        # Set a limit on OpenMP in the current thread:
+        CONTROLLER.limit(limits=1)
+        # Do the actual work:
+        return do_real_work_with_openmp(*args, **kwargs)
+
+    results = POOL.map(limit_then_do_work, args)
+
+
+# 2. Run some work with 4-threads OpenMP parallelism:
+with CONTROLLER.limit(limits=4):
+    results2 = do_more_work_with_openmp(results)
+
+
+# 3. Nest some OpenMP parallelism under Python-level parallelism:
+with CONTROLLER.limit(limits=2):
+
+    def limit_then_do_work2(*args, **kwargs):
+        CONTROLLER.limit(limits=2)
+        return do_even_more_real_work_with_openmp(*args, **kwargs)
+
+    results3 = POOL.map(limit_then_do_work2, results2)
+```
+
+## Usage: Additional APIs and details
+
 ### Switching the FlexiBLAS backend
 
 `FlexiBLAS` is a BLAS wrapper for which the BLAS backend can be switched at runtime.
@@ -291,6 +415,7 @@ that this part of the API is experimental and subject to change without deprecat
 You can observe that the previously linked OpenBLAS shared object stays loaded by
 the Python program indefinitely, but FlexiBLAS itself no longer delegates BLAS calls
 to OpenBLAS as indicated by the `current_backend` attribute.
+
 ### Writing a custom library controller
 
 Currently, `threadpoolctl` has support for `OpenMP` and the main `BLAS` libraries.
@@ -346,6 +471,101 @@ https://github.com/xianyi/OpenBLAS/issues/2985).
   on Windows, the setting is process-wide and impacts the size of a process-wide
   thread pool shared across all threads in the process.
 
+## <div id="semantics"> Semantics of thread limiting </div>
+
+Setting the number of threads may seem like a simple operation, but in practice
+it can do quite different things depending on the underlying third-party library
+being limited.
+
+### Kind of thread pool
+
+* In some libraries, there is a shared process-wide thread pool. Setting the
+  number of threads changes the size of this shared pool.
+* In other libraries, there is a thread pool per calling thread. So each Python
+  thread gets its own personal thread pool from the third-party library.
+
+To give a concrete example: if you have 10 cores, and you're using OpenBLAS with
+the `pthreads` backend on Linux, by default there is a single 10-worker pool of
+threads created by OpenBLAS. If you start 10 Python threads, each calling BLAS
+routines, all those Python threads will feed into that single 10-thread pool.
+In total, you will have 20 threads running (10 Python, 10 OpenBLAS).
+
+On the other hand, if you use MKL, each Python thread gets its own individual
+pool of worker threads from MKL, so if each Python threads calls a BLAS routine
+in MKL, you will get 10×10 = 100 MKL threads!
+
+### Scope of limiting
+
+If you have a third-party library that creates a thread pool per calling thread,
+there is another question about what limiting the number of threads means: which
+threads are affected by the limiting API?
+
+* **All threads in the process:** In this case, setting the limits changes it
+  for all threads in the process. So if you limit from the main Python thread,
+  all other Python threads will have the same limit.
+* **Current thread only:** Only the current thread's thread pool size will be
+  limited.
+
+MKL for example has two APIs, covering both cases,
+[`mkl_set_num_threads()`](https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2026-0/mkl-set-num-threads.html)
+and [`mkl_set_num_threads_local()`](https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2026-0/mkl-set-num-threads-local.html) respectively.
+
+### `threadpoolctl`'s policy for thread limiting
+
+When there is a choice between different APIs, `threadpoolctl` will prefer APIs
+that:
+
+1. Have a per-thread worker pool (so each Python thread gets its own pool from
+   the underlying third-party library)
+2. Only affect the current thread.
+
+Current libraries where `threadpoolctl` explicitly makes this choice are:
+
+* MKL, for all threading backends.
+* OpenBLAS (v0.3.34 or later) when using the OpenMP backend, on Linux and macOS.
+
+When using OpenMP, this is also the default on Linux and macOS. On Windows
+OpenMP has a per-thread worker pool but the limiting API affects all threads in
+the process, not just the current one.
+
+### Checking the behavior of your installed libraries
+
+When you run `python -m threadpoolctl` it will include the scope of the API
+limit in the `"thread_limit_scope"` field, with the value of `"process"`
+indicating the API affects the whole process and `"current_thread"` indicating a
+per-thread thread pool. Information about the kind of limiting in the former
+case (shared process-wide pool, or per-thread pool) is not included.
+
+Here is example output for OpenBLAS with pthreads:
+
+```shell-session
+$ python -m threadpoolctl -i numpy
+[
+  {
+    "user_api": "blas",
+    "internal_api": "openblas",
+    "num_threads": 12,
+    "prefix": "libscipy_openblas",
+    "version": "0.3.33.112.0",
+    "threading_layer": "pthreads",
+    "architecture": "Haswell",
+    "thread_limit_scope": "process"
+  }
+]
+```
+
+You can also use another script to empirically determine both the kind and scope of the API. From a checkout of the [`threadpoolctl` GitHub repo](https://github.com/joblib/threadpoolctl):
+
+```shell-session
+$ python -m tests.empirical_scope_observation blas
+== Observed behavior: blas ==
+10x12 Python threads each running a parallel workload with a 2 thread limit.
+Maximum number of observed threads (includes Python and blas threads): 53
+Presumed API scope: process-wide shared thread pool
+```
+
+You can also call this script with an `openmp` argument instead of `blas`.
+
 ## Maintainers
 
 To make a release:

conftest.py

@@ -1 +1,5 @@
-collect_ignore = ["tests/_openmp_test_helper", "tests/_limit_blas"]
+collect_ignore = [
+    "tests/empirical_scope_observation.py",
+    "tests/_openmp_test_helper.py",
+    "tests/_limit_blas.py",
+]

pyproject.toml

@@ -27,3 +27,6 @@ homepage = "https://github.com/joblib/threadpoolctl"
 line-length = 88
 target_version = ['py39', 'py310', 'py311', 'py312', 'py313']
 preview = true
+
+[tool.ruff]
+line-length = 88