GitLab switching to the 'reftable' Git refs storage

A loud whistle was blown by the first pipeline of !867 (merged): Gitaly now implements the reftable storage format, but Dulwich (the Git backend behind hg-git) does not support it, raising dulwich.repo.UnsupportedExtension: (b'refstorage', b'reftable').

The problem was raised because RSpec tests always run with all Gitaly feature flags activated. In this case, the feature flag is about creating new repositories with the reftable format, defaults to false and the milestone for rollout is GitLab 17.8, hence the problem is not immediate, but the same upstream commit also mentions an upcoming automatic migration of existing Git repositories, we definitely need to keep both eyes on that.

At first sight, it seems that more recent versions of Dulwich do not support reftable either (needs to be confirmed, will reach out to see if it is planned for the near future).

It is possible that we would need to force the migration of all Mercurial projects to native for the 17.8 time frame, and find a way to make existing and still needed conversions out of the grasp of Gitaly (perhaps a rename could be in order). Typical use-case for these is to mirror to Git-centric services, e.g., for cross-platform CI/CD.

In the immediate future, it looks like simply stopping to run the RSpec tests with Gitaly feature flags activated is good enough.

Edit 2024-12-01: the rollout strategy was included in GitLab team handbook through this MR. The plan is that repositories get transparently migrated upon access, controlled by feature flags, including new_repo_reftable_backend, whose state we are watching in Gitaly sources (we hope it will also control the migration). A final background job would take care of dormant repositories.

Task list for Heptapod 17.517.6

• watch the file defining the Gitaly feature flag in the upstream merge process
• engage with Dulwich to see if support is being considered and how far in time it would be
• remove all usage of HgGitRepository in HGitaly or more generally native Mercurial RSpec tests. These are remnants of early times of the HGitaly project, when it was the only way to compare results with those of the more mature HgGitRepository. Nowadays we have a much better way of doing that, in HGitaly's own integration tests.
• finish making sure that native Mercurial projects without legacy hg-git conversion are fully functional (activate the feature flag in a dev environment and have the functional tests pass in native mode, including backups etc.)
• expose native Mercurial migration in the UI (can already be done through the API and a Rake task)
• #1866 (closed): consider a group-level API endpoint and UI for the native Mercurial migration
• #1848: move the remaining converted Git repositories (due to hg_legacy_git_conversion) out of Gitaly reach.