Error correlated to sidekiq getting stuck
The correlation is just temporal, so nothing guarantees it's really significant:
2020-02-20T15:48:50.278Z 437 TID-gtli2jj9c WARN: /home/hg/heptapod/lib/gitlab/git/repository.rb:1646:in `rescue in gitaly_migrate'
/home/hg/heptapod/lib/gitlab/git/repository.rb:1639:in `gitaly_migrate'
/home/hg/heptapod/lib/gitlab/git/commit.rb:235:in `batch_by_oid'
/home/hg/heptapod/app/models/repository.rb:137:in `commits_by'
/home/hg/heptapod/app/models/commit.rb:97:in `block (2 levels) in lazy'
/home/hg/heptapod/app/models/commit.rb:94:in `each'
/home/hg/heptapod/app/models/commit.rb:94:in `block in lazy'
/home/hg/heptapod/vendor/bundle/ruby/2.4/gems/batch-loader-1.2.1/lib/batch_loader.rb:90:in `__ensure_batched'
/home/hg/heptapod/vendor/bundle/ruby/2.4/gems/batch-loader-1.2.1/lib/batch_loader.rb:51:in `__sync'
/home/hg/heptapod/vendor/bundle/ruby/2.4/gems/batch-loader-1.2.1/lib/batch_loader.rb:75:in `__sync!'
/home/hg/heptapod/vendor/bundle/ruby/2.4/gems/batch-loader-1.2.1/lib/batch_loader.rb:71:in `method_missing'
/home/hg/heptapod/app/workers/expire_pipeline_cache_worker.rb:16:in `perform'
There's no more detail in the logs, we don't have the error message.
After a quick source read, it really seems that this was treated by Gitaly, which barfed on the oids it's been passed.
How that could lead to the sidekiq workers getting stuck is not clear.