HGitaly (?) causes Internal server error
I have a bunch of repositories hosted by Heptapod. Some of them are in Git format, but mostly Mercurial. The biggest pain so far are constant "500" errors after anything being pushed to a server. In a nutshell, this is what I see:
- Everything works fine until someone pushes an update to any of the Mercurial repositories.
- The web interface becomes inaccessible for all Mercurial repos (but seems to be OK for git).
- Any repo can be pulled using a Mercurial client, but an attempt to push fails with the "API is not acessible" error.
- Usually the server remains inaccessible for quite a while (up to few hours), but then comes back.
- Sometimes it does not come back, so I have to restart Heptapod.
I tried reading the server's "production" log, and all I see there is
GRPC::DeadlineExceeded (4:Deadline Exceeded. debug_error_string:{"created":"@1691078759.000722367","description":"Error received from peer unix:/var/opt/gitlab/hgitaly/hgitaly.socket","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"Deadline Exceeded","grpc_status":4}):
lib/gitlab/gitaly_client.rb:192:in `execute'
...
The machine serving Heptapod isn't stellar (A hi-end Synology NAS), but when it works, it works fine. Some of the repos are very large and have large binary files, so I guess the internal git conversion might take time for them. HGitaly doesn't seem to write its own logs, so I have no idea where else to look and what can be done. So I would appreciate any help (Heptapod 0.32.1)
UPD: I checked mercurial.log
, and the only errors I see are something like
repo:/var/opt/gitlab/git-data/repositories/@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.hg [hg] Got code 1 while sending GitLab 'GitLab 'pre-receive' hook' hook details=API is not accessible
So I guess it is git mirroring after all, but it blocks everything, and it seems that even small changes might trigger a long process.