Workhorse and lack of Side Channel in HGitaly
To avoid gRPC itself being the bottleneck when serving repeatedly binary data from Gitaly, upstream introduced multiplexing of the Gitaly transport, using the Yamux library and protocol.
References:
- https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/463
- https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1051
- Yamux protocol
As of GitLab 15.0, this becomes unconditional, and Workhorse initializes all its Gitaly server connections with side channels, hence as a Yamux session in which gRPC content is but one stream. The other streams are created on demand to carry over binary data response to UploadPack
and similar Git data that can be cached on the Gitaly server side.
Because HGitaly does not support such Yamux encapsulation, HGitaly servers appear to be down from the point of view of Workhorse. The end consequence for Heptapod is that Workhorse does not even try requests that don't use side channels on HGitaly servers, of which GetArchive
is the prime example. From a user perspective, this breaks all Heptapod archive downloads (spotted by heptapod/heptapod-tests>).
Unfortunately, it is not obvious how Yamux multiplexing could be introduced in HGitaly, for two reasons
- lack of language support: there is a very immature yamux Python package and a much more ready yamux Rust crate
- it is not clear if a Yamux stream could be given to python-grpc instead of a plain socket
Hence we will have to introduce a work-around, fallbacking to gRPC over plain sockets when side channels aren't necessary. It seems reasonable to think that side channels won't be used any time soon for data that is not cached on the Gitaly server side, because the gRPC protocol isn't the bottleneck in that case. For instance, archive data is cached on the Workhorse side.