State file for GitLab branches
We have again a convergence between different questions with the mapping between Mercurial metadata (branches, topics…) and the GitLab branches (Git-style pointer to a unique revision).
The computation is expensive, notably because it involves the branchmap curation for obsolete heads provided by evolve (which could be improved to some extent, but that's a different topic). This is an obvious culprit in the current performance problems.
With HGitaly, the same computation happens at read time, for almost any request involving GitLab branches (a lot), making it unusable on medium sized repos. We've discussed caching strategies at various levels of the stack with @marmoute. Some will benefit all topics users, but Heptapod will need more than that anyway.
For %HGitaly2: fully native, we won't convert to Git any more, but we will still need to send appropriate notifications to the Rails application: a pair of commit IDs (before/after) for each branch (each ref, actually). Today, the Git repository plays the role of the previous state record. We could perhaps reconstruct the previous state by looking at transaction data, but the simplest here will be to record the resulting GitLab branches, so that each transaction can compare with the previous state. Once that is done, we don't need a cache on the HGitaly side.
General Mercurial caches work on harder constraints that one can trust that previous transactions have updated the cache. They could have been done with a different client, or just command-line options that disable the cache. Within Heptapod, we are in a special case: it is guaranteed that all transactions go through the gitlab-mirror
hook.
Also, the branch mapping system has painfully evolved to recover from inconsistencies, so I don't worry too much about the robustness of this state system. Still, it will be useful to keep the existing logic on the HGitaly side, in order to rebuild the state file if needed. This could even be made automatic later on once we have a transaction id file to check for staleness.
If we compare with what hg-git does, a simple flat file mounted as a dict
would be good enough for a first version. In any case, a precomputed mapping would be a more appropriate support for branch querying than what we currently have. The day we want an append-only structure, we may have to go the append-only radix tree way (again!). For instance hgitaly#9 (closed) needs prefix matching. Perhaps more useful methods also do.
We'll certainly need a version header, but that's not a problem.
Finally, .hg/requires
is usually the place to state that the repository needs a Mercurial that understands some data format to operate properly. But in that case, it's probably inappropriate, because the repository will be perfectly viable in itself, as the data is only useful for interaction with the Rails app and HGitaly.