Heptapod PAAS Runner: State persistence
The main goal is to make it possible to restart heptapod-paas-runner
without losing sight of running jobs, so that:
- the maximal concurrency limit is not impaired by the restart
- resources can be decommissioned
Closes #23 (closed)
Also makes #22 unnecessary for the time being: a simple restart done with SIGTERM
can be used to change configuration. This works in particular with systemd and its defaults, or any process manager that uses SIGTERM
and has a reasonable timeout before resorting to SIGKILL
.
It goes in three broad steps:
- Introduction of a graceful shutdown feature, tied to signal handling. Previously, some threads would not even have any termination logic. This pushes the multiple reporting threads system over its limits in terms of maintainability.
- Change to a single thread to process events. The intent is to give it write monopoly to the job tracking collections, but that is reached only at the end of the stack.
- State saving at shutdown and loading at startup. It is of course crucial for this to be sure that all threads have stopped, not only so that state mutation cannot occur any more, but also to prevent untracked actions to occur. At this stage, a single JSON file is enough for our purposes. Doing better wouldn't be too hard.
The docstring of PaasDispatcher
at the end is probably interesting to read independently.
There are many possible improvements, especially in the resuming of pending jobs that occurs at startup: for now, launcher threads are not supposed to be interrupted, and that can become a problem at scale. It would be better to make interruption possible after each call to PAAS systems. In the future, we may want to also implement resue of already provisioned resources, and that would be the occasion to also take care of shutdown between provisioning and actual launching.