Tests: a new layer for direct Gitaly comparison

One of the challenges of HGitaly development is to make sure that our interpretation of the protocol is correct.

It's one thing to know that there is an expected return attribute called branch and its type is bytes, it's another to determine whether it should be filled with, e.g., branch/default or refs/heads/branch/default. There are of course more complicated cases where exact behaviour can be arbitrary (usually depending on what an underlying call to Git or libgit2 makes of it).

For a while, we've been using integration tests on the Rails app end to make sure that HGitaly really behaves as a replacement for Gitaly, but it has many drawbacks:

• RSpec tests have a heavy entangled setup. On developer systems, a fully working HDK is pretty much required, and the setup actually recompiles Gitaly for the separate use-case of running the tests, using the configuration from gitlab.yml.example. In CI, this results in a nightmarish preinstall going through the same code, with conditions that avoid redoing the heavy work…
• RSpec tests have a high latency: I see routinely 30s before the first example is run, and that is three times more than the whole pytest suite of HGitaly with coverage and all.
• There's been lots of effort with respect to maintaining fixtures and obtaining insulation both in general Rails world (factories, fixtures). The end result in GitLab isn't convincing: way too much start from scratch (hence slowness) except when we would need it (hence the numerous after cleanups)
• Too much proficiency is required currently to work on HGitaly. Taking Ruby, Rails and the GitLab Rails app out of the loop would allow to onboard developers requiring "only" Python, gRPC, Mercurial, Git and the ability to read enough Golang to understand what Gitaly is doing.
• In terms of calendar time, it takes us a week to close the whole loop when the HGitaly part iself needs one day.

So here's the idea: we could have a second layer of integration tests in the HGitaly project, for direct comparison with Gitaly. It would assume by design

• that a correct version of Gitaly is available, and maybe already running.
• that a minimal Gitaly configuration is already available
• to avoid mismatchs, a small test could just check the Gitaly version.

On developer machines, the HDK would take care of all of that, but it would be nice to have a lighter setup option as well. After all, compiling Gitaly is not that hard.

The CI jobs would probably run on a Docker image with the proper precompiled Gitaly version and would just need to start it on the proper configuration.