heptapod-runner on kubernetes restarting frequently and loosing control of existing jobs
This might be related to #23 (closed)
When our kubernetes cluster is under heavy load we can sometimes see the heptapod-runner restarting and there are some side effects to that :
- jobs that were running don't send back their logs when they continue
- pods keep running and hog resources (since we use requests on CPU and memory)
- we added a runner-terminator job to clean those up, see code https://forge.extranet.logilab.fr/emartinet/runner-terminator
This was happening too often so we're exploring the error log using sentry by activating sentry_dns
in the runner configuration.
With that in place we get the following traceback that might be related :
*errors.errorString: K8S stream error
File "/home/gracinet/heptapod-runner/helpers/sentry/log_hook.go", line 45, in Fire
File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/hooks.go", line 28, in Fire
File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 246, in fireHooks
File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 224, in log
File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 268, in Log
File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 297, in Error
File "/home/gracinet/heptapod-runner/executors/kubernetes/exec.go", line 213, in func1
File "/home/gracinet/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/runtime/runtime.go", line 112, in HandleError
File "/home/gracinet/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191004102930-01520b8320fc+incompatible/tools/remotecommand/v2.go", line 147, in func1
Is this of any use for debugging or understanding the problem ?
We are using octobus/heptapod-runner:0.4.0