GitLab switching to the 'reftable' Git refs storage
A loud whistle was blown by the first pipeline of !867: Gitaly now implements the `reftable` storage format, but Dulwich (the Git backend behind `hg-git`) does not support it, raising `dulwich.repo.UnsupportedExtension: (b'refstorage', b'reftable')`.
The problem was raised because RSpec tests always run with all Gitaly feature flags activated. In this case, the feature flag is about creating new repositories with the `reftable` format, defaults to `false` and the milestone for rollout is GitLab 17.8, hence the problem is not immediate, but [the same upstream commit](https://gitlab.com/gitlab-org/gitaly/-/commit/23a6e749f8afc38cbbada2684304c4363d32ba45) also mentions an upcoming automatic migration of existing Git repositories, we definitely need to keep both eyes on that.
At first sight, it seems that more recent versions of Dulwich do not support `reftable` either (needs to be confirmed, will reach out to see if it is planned for the near future).
It is possible that we would need to force the migration of all Mercurial projects to native for the 17.8 time frame, and find a way to make existing and still needed conversions out of the grasp of Gitaly (perhaps a rename could be in order). Typical use-case for these is to mirror to Git-centric services, e.g., for cross-platform CI/CD.
In the immediate future, it looks like simply stopping to run the RSpec tests with Gitaly feature flags activated is good enough.
Edit 2024-12-01: the rollout strategy was included in [GitLab team handbook](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/gitaly_reftable_rollout/) through [this MR](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/8008). The plan is that repositories get transparently migrated upon access, controlled by feature flags, including `new_repo_reftable_backend`, whose state we are watching in Gitaly sources (we hope it will also control the migration). A final background job would take care of dormant repositories.
### Task list for Heptapod ~~17.5~~17.6
- [x] watch the file defining the Gitaly feature flag in the upstream merge process
- [x] engage with Dulwich to see if support is being considered and how far in time it would be
- [x] remove all usage of `HgGitRepository` in HGitaly or more generally native Mercurial RSpec tests. These are remnants of early times of the HGitaly project, when it was the only way to compare results with those of the more mature `HgGitRepository`. Nowadays we have a much better way of doing that, in HGitaly's own integration tests.
- [ ] finish making sure that native Mercurial projects without legacy hg-git conversion are fully functional (activate the feature flag in a dev environment and have the functional tests pass in native mode, including backups etc.)
- [x] expose native Mercurial migration in the UI (can already be done through the API and a Rake task)
- [x] heptapod#1866: consider a group-level API endpoint and UI for the native Mercurial migration
- [x] heptapod#1848: move the remaining converted Git repositories (due to `hg_legacy_git_conversion`) out of Gitaly reach.
### Task list for Heptapod 17.7
- [ ] heptapod#1961: implement everything that is needed on the auxiliary repositories, without handing them over to Gitaly.
- [ ] heptapod#1963: cleanup the code base by putting Git mirroring on its own specific calls
- [ ] heptapod#1964: force native migration for all `hg_git` Mercurial projects
issue