Skip to content

Rewrite worker with concurrent.futures instead of multiprocessing

Ningú requested to merge topic/default/worker-silent-death-3 into branch/default

Unlike multiprocessing pools, the concurrent library pools will raise an exception if child processes have died and can't accept further tasks. This should cause the main worker to stop instead of stalling in an inifinite wait.

Introduce a dummy noop execution in the pool to probe if it is still operative to mitigate the volume of lost tasks if the next submited execution would raise a BrokenExecutor exception. This isn't 100% bulletproof because the pool might still break between the probe and a new task being pulled.

The maxtasksperchild CLI option will only work for Python 3.11 onwards because former concurrent.futures pools do not support this option.

Closes #11104 (closed)

Edited by Ningú

Merge request reports