Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 39 additions & 2 deletions docs/trouble_shooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,45 @@ The `coverage combine` command merges the data from the main process and subproc

## Python Version
Executorlib supports all current Python version ranging from 3.9 to 3.13. Still some of the dependencies and especially
the [flux](http://flux-framework.org) job scheduler are currently limited to Python 3.12 and below. Consequently for high
performance computing installations Python 3.12 is the recommended Python verion.
the [flux](http://flux-framework.org) job scheduler are currently limited to Python 3.13 and below. Consequently for high
performance computing installations Python 3.13 is the recommended Python verion.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix spelling error.

"verion" should be "version".

📝 Proposed fix
-performance computing installations Python 3.13 is the recommended Python verion. 
+performance computing installations Python 3.13 is the recommended Python version.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
performance computing installations Python 3.13 is the recommended Python verion.
performance computing installations Python 3.13 is the recommended Python version.
🧰 Tools
🪛 LanguageTool

[grammar] ~56-~56: Ensure spelling is correct
Context: ...s Python 3.13 is the recommended Python verion. ## Cores, Threads per Core and Maximum Work...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/trouble_shooting.md` at line 56, Fix the spelling mistake in the
sentence "performance computing installations Python 3.13 is the recommended
Python verion." by changing "verion" to "version" so it reads "...recommended
Python version."; update that line in the troubleshooting document where that
exact sentence appears.


## Cores, Threads per Core and Maximum Workers
A common point of confusion is the difference between the `cores`, `threads_per_core` and `max_workers` (or `max_cores`)
parameters, as they all control how many compute resources executorlib uses, but on different levels:

* `max_workers` / `max_cores` are arguments of the `Executor` itself. They define the *total* number of compute cores
the executor is allowed to use in parallel across all submitted function calls - essentially the size of the resource
pool or allocation that all tasks share. `max_workers` exists for backwards compatibility with the
[Executor interface](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor) of the
Python standard library, while `max_cores` is the recommended way to express the same limit, as it makes clear that the
limit refers to the number of compute cores. Setting either is optional - when neither is provided executorlib uses the
number of cores available on the machine.
Comment on lines +67 to +68
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Clarify defaulting behavior when neither max_cores nor max_workers is set.

This sentence is too absolute. The code path in src/executorlib/standalone/inputcheck.py shows fallback to CPU count is conditional (set_local_cores=True); otherwise it raises ValueError. Please scope this statement to the relevant executors/modes.

Suggested wording
-  limit refers to the number of compute cores. Setting either is optional - when neither is provided executorlib uses the
-  number of cores available on the machine.
+  limit refers to the number of compute cores. Setting either is optional in executors that support local-core fallback;
+  in that case, when neither is provided, executorlib uses the number of cores available on the machine.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
limit refers to the number of compute cores. Setting either is optional - when neither is provided executorlib uses the
number of cores available on the machine.
limit refers to the number of compute cores. Setting either is optional in executors that support local-core fallback;
in that case, when neither is provided, executorlib uses the number of cores available on the machine.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/trouble_shooting.md` around lines 67 - 68, The doc line claiming "when
neither is provided executorlib uses the number of cores available on the
machine" is too absolute; update the wording to scope the fallback to CPU count
to executors/modes that enable local core fallback (see
src/executorlib/standalone/inputcheck.py and the set_local_cores=True path).
Mention that when set_local_cores is False the library raises a ValueError
instead, and clarify which executors or modes use the automatic CPU-count
fallback versus those that require explicit max_cores/max_workers.

* `cores` is an entry of the `resource_dict` and is defined *per function call*. It specifies how many Python processes
executorlib starts for a single task. These processes are connected via
[mpi4py](https://mpi4py.readthedocs.io) and together form one MPI application. Consequently, `cores` is primarily
intended for functions implemented with [mpi4py](https://mpi4py.readthedocs.io), where the same Python function is
executed once per MPI rank. For a typical serial Python function, increasing `cores` does **not** provide additional
parallelism. Instead, executorlib launches multiple copies of the function, which usually wastes resources and can
lead to incorrect behavior. Unless you are using MPI through [mpi4py](https://mpi4py.readthedocs.io), `cores`
should generally be left at its default value of `1`.
* `threads_per_core` is also an entry of the `resource_dict` and defined *per function call*. In contrast to `cores`,
executorlib starts only a single Python process for the task and reserves the requested resources for that process.
The number of reserved cores is communicated through environment variables such as `OMP_NUM_THREADS`. This parameter
should be used whenever the Python function itself is executed only once, but internally uses multiple cores. Common
examples include thread-parallel libraries such as NumPy, BLAS, MKL or OpenMP-enabled code, as well as Python
functions which launch external applications. In the latter case, executorlib starts a single Python process, which
then launches the external application. Whether that external application internally uses OpenMP, MPI or a hybrid
MPI/OpenMP parallelization strategy is transparent to executorlib. This functionality is demonstrated in the Quantum
ESPRESSO application example.

A useful rule of thumb is:

* Use `cores` when executorlib should start multiple Python processes which together form an MPI application via
`mpi4py`.
* Use `threads_per_core` when executorlib should start the Python function only once and reserve multiple cores for it
or for an external application launched by it.
* Use `max_cores` to limit how many resources all submitted tasks may consume collectively.

## Resource Dictionary
The resource dictionary parameter `resource_dict` can contain one or more of the following options:
Expand Down
Loading