Problem with oar stop signal
Created originally on Bitbucket by Mosuin (Clément Savaro)
When preprocessing a large set of image on OAR, the checkpoint signal seem to be ignored : with delay_signal_walltime=600, The error log contain :
## OAR [2018-10-03 21:17:31] Job 541850 CHECKPOINTED with signal 12 ##
Same signal every 10 sec then :
## OAR [2018-10-03 21:27:22] Job 541850 CHECKPOINTED with signal 12 ##
## OAR [2018-10-03 21:27:33] Job 541850 KILLED ##
The process was saving more than 150 image per minutes so 600 sec delay should be more than enough.
The code is basically the same as the example "submit_job_legi" calling the script "preproc_with_topology" but a large amount of images is recquired to reproduce this bug so that the computation goes for more than a few second.