add way to stop running calculation

[ltalirz]

add MCP endpoint that allows to stop a running calculation

[ltalirz]

@jan-janssen what approach would you suggest for this?

[jan-janssen]

Currently we do it by getting the future, then from the Future get the HDF5 file f._file_name, which contains the queue ID. You can read it using: executorlib.standalone.hdf.get_output(). With the queue ID we delete the job using https://pysqa.readthedocs.io which executorlib is using internally and then also delete the HDF5 file.

Alternatively you can use the cache of all HDF5 files, for example when you have the hash name or any other unique identifier:
https://executorlib.readthedocs.io/en/latest/1-single-node.html#cache

In practice some users even prefer to use the command line, as they anyway want to watch the queuing system to see the jobs are utilizing the available resources.

Still there might be a potential to do this in a more programmatic way, for example by using f.cancel() but that is currently not compatible to the wait=False parameter which detaches the executor from the submitted task. So I would have to think about this a bit more.

[ltalirz]

Ok. And the future we would again get by submitting the same calculation again?
In principle we could then even do it in a wait=True context manager...

I do think it makes sense for executorlib to help users who want to cancel a given task

As for the semantics and wait=False - I chatted a bit with Claude and my understanding is:

• in the standard library, leaving the context manager waits for all futures to complete
• but: you can use the executor without the context manager and pass wait=False

e.g.

executor = ProcessPoolExecutor(max_workers=4)
fut = executor.submit(some_task)
# ... do other stuff, executor stays alive ...
executor.shutdown(wait=False)  # don't block, let things finish in background

In python 3.9+ you can even pass cancel_futures=True if you want to cancel the futures as well.

This seems very reasonable to me: people who want this behavior should simply not use the context manager but shutdown the executor explicitly.

I see no reason why this pattern should not be adopted by executorlib as well.

[jan-janssen]

So far our focus was on the context but I am going to take a look at supporting the other cases for the next release. The current wait=False during the initialization is a bit confusing.

[jan-janssen]

The challenge is that the Python API currently does not support future objects which are already running:

from concurrent.futures import Future

f1 = Future()
f1.set_running_or_notify_cancel()
f1.cancel()  # False
f1.done() # False

f2 = Future()
f2.cancel()  # True
f2.done() # True

In executorlib we currently have the option to kill future object on shutdown but we would have to think about supporting cancelling future objects which are running.

[ltalirz]

Maybe don't use that API then but something like executor.kill(future) which sets it to a FINISHED state with a custom exception

[ltalirz]

Is it actually possible to kill individual tasks in the executorlib implementation?
For the SLURM and flux versions I guess yes. How does it work for the single node - does it use a dedicated process per task?

[jan-janssen]

Is it actually possible to kill individual tasks in the executorlib implementation? For the SLURM and flux versions I guess yes.

Yes, I just added the functionality that cancelling the future actually cancels the job on the queue:
pyiron/executorlib#903
As we cannot cancel running jobs, using f.cancel() I like the idea of adding a exe.kill() function. Still this is not yet implemented.

How does it work for the single node - does it use a dedicated process per task?

By default yes, but we can also keep the process running and use it for multiple tasks, this called block_allocation:
https://executorlib.readthedocs.io/en/latest/1-single-node.html#block-allocation

[jan-janssen]

I looked again at the SingleNodeExecutor and wait already works as expected:

from executorlib import SingleNodeExecutor
from time import sleep

def echo_with_delay(i):
    sleep(10)
    return i

exe = SingleNodeExecutor(max_workers=1, hostname_localhost=True)
f1 = exe.submit(echo_with_delay, 1)
f2 = exe.submit(echo_with_delay, 2)
exe.shutdown(wait=False, cancel_futures=False)

f1.done(), f2.done(), f1.cancelled(), f2.cancelled()  # (False, False, False, False)
sleep(12)
f1.done(), f2.done(), f1.cancelled(), f2.cancelled()  # (True, False, False, False)

[ltalirz]

Thanks @jan-janssen , this looks like great progress!

I looked again at the SingleNodeExecutor and wait already works as expected:

Cool! With "already" you mean with 1.8.0 or the next release?

Anyhow, in this case, I will probably switch back to the SingleNodeExecutor for CI to keep things simple.
The problem in the API PR is somewhere else, I just need a bit of time to sit down and pin it down.

[jan-janssen]

Cool! With "already" you mean with 1.8.0 or the next release?

In 1.8.0 already. I created a mybinder environment to compare both the ProcessPoolExecutor and the SingleNodeExecutor. https://github.com/pyiron-dev/executor-without-context

[ltalirz]

The ability to stop running jobs was introduced in 10cfaea