PAAS dispatching: end user logs

Currently, when there is a provisioning problem (typically because there aren't enough resources left), we have ordinary server-side logs, but nothing is accesible to the end user. This would be very interesting if we implement a retry logic, flavor capping (interesting with the weighted quota system, to reduce the margin that is enforced to prevent overflowing).

The obvious thing to do would be to append to the job logs, as visible in the interface (job trace, in GitLab terminology). There is an API endpoint for this, unsurprisingly. Here's a successful example (pdb session within heptapod-tests):

(Pdb) job_url =  test_project.heptapod.api_url + '/jobs/715'
(Pdb) trace_url = job_url + '/trace'
(Pdb) range = '0-4'                                                                                                    
(Pdb) resp = requests.patch(trace_url, params=dict(token=job['token']), headers={'content-range': range}, data='foo\n')
(Pdb) resp.status_code
202

Unfortunately, if we put anything to the job trace before handing the job over to the PAAS Resource, it won't work: the trick is that the range has to be tailored exactly so that the first value is the offset of the append (hence exactly the previous length of the job trace), and the second value has to be the resulting full trace size. Hence if we write anything before starting heptapod-runner in the PAAS resource, it will break the job traces. Example:

(Pdb) range = '0-8'   
(Pdb) resp = requests.patch(trace_url, params=dict(token=job['token']), headers={'content-range': range}, data='aaaanew\n')    
(Pdb) resp.text
'{"error":"416 Range Not Satisfiable"}'

A solution could be to introduce the suitable offset in the job payload for gitlab-runner exec-fetched-job and have the counter start at the proper value in newJobTrace. Perhaps we can use sentTrace, perhaps we need a separate offset. In all cases, this will impact all Runner operation, hence it must be treated with care and proper testing.

Perhaps also a more recent version of GitLab Runner provides the kind of facility we need.