Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • H heptapod-runner
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 21
    • Issues 21
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • heptapod
  • heptapod-runner
  • Issues
  • #26
Closed
Open
Created Sep 02, 2021 by Arthur Lutz@arthur.lutzMaintainer

heptapod-runner on kubernetes restarting frequently and loosing control of existing jobs

This might be related to #23 (closed)

When our kubernetes cluster is under heavy load we can sometimes see the heptapod-runner restarting and there are some side effects to that :

  • jobs that were running don't send back their logs when they continue
  • pods keep running and hog resources (since we use requests on CPU and memory)
  • we added a runner-terminator job to clean those up, see code https://forge.extranet.logilab.fr/emartinet/runner-terminator

This was happening too often so we're exploring the error log using sentry by activating sentry_dns in the runner configuration.

With that in place we get the following traceback that might be related :

*errors.errorString: K8S stream error
  File "/home/gracinet/heptapod-runner/helpers/sentry/log_hook.go", line 45, in Fire
  File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/hooks.go", line 28, in Fire
  File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 246, in fireHooks
  File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 224, in log
  File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 268, in Log
  File "/home/gracinet/go/pkg/mod/github.com/sirupsen/logrus@v1.4.2/entry.go", line 297, in Error
  File "/home/gracinet/heptapod-runner/executors/kubernetes/exec.go", line 213, in func1
  File "/home/gracinet/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/runtime/runtime.go", line 112, in HandleError
  File "/home/gracinet/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191004102930-01520b8320fc+incompatible/tools/remotecommand/v2.go", line 147, in func1

Is this of any use for debugging or understanding the problem ?

We are using octobus/heptapod-runner:0.4.0

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking