Skip to content
  • Georges Racinet's avatar
    PAAS Dispatcher: tracking jobs progress to limit concurrency · 3e2f4fa66069
    Georges Racinet authored
    To limit concurrency, we have to track jobs progress once they've
    been launched.
    
    This is done via a permanent thread that calls the coordinator
    API endpoint to retrieve job info by job token. As it turns out,
    the endpoint returns a 401 code unless the job is ongoing.
    This feels weird, but given that we are using the token that
    the coordinator itself sent us as part of the job payload, this
    is good enough for now, but it makes debugging of any other
    problem leading to 401 responses really hard.
    
    Later versions of the PAAS dispatcher may have a endpoint on
    the provisioned ressources in order to follow progress directly
    there, which would be less hacky and alleviate the load on
    coordinator a bit (of course it's otherwise hammered, but full
    job details by token are expected to be heavy to generate).
    
    The current design of the `PaasDispatcher` class makes it so
    that progress is fully interpreted only at next polling cycle,
    because there's a reporting thread reading the queue for each
    cycle. We should later make a permanent reporting thread.
    3e2f4fa66069