heptapod issueshttps://foss.heptapod.net/heptapod/heptapod/-/issues2021-02-09T12:17:16Zhttps://foss.heptapod.net/heptapod/heptapod/-/issues/373State file for GitLab branches2021-02-09T12:17:16ZGeorges RacinetState file for GitLab branchesWe have again a convergence between different questions with the mapping between Mercurial metadata (branches, topics…) and the GitLab branches (Git-style pointer to a unique revision).
The computation is expensive, notably because it i...We have again a convergence between different questions with the mapping between Mercurial metadata (branches, topics…) and the GitLab branches (Git-style pointer to a unique revision).
The computation is expensive, notably because it involves the branchmap curation for obsolete heads provided by evolve (which could be improved to some extent, but that's a different topic). This is an obvious culprit in the current [performance problems](heptapod#351).
With HGitaly, the same computation happens at read time, for almost any request involving GitLab branches (a lot), making it unusable on medium sized repos. We've discussed caching strategies at various levels of the stack with @marmoute. Some will benefit all topics users, but Heptapod will need more than that anyway.
For %"HGitaly2: fully native", we won't convert to Git any more, but we will still need to send appropriate notifications to the Rails application: a pair of commit IDs (before/after) for each branch (each ref, actually). Today, the Git repository plays the role of the previous state record. We could perhaps reconstruct the previous state by looking at transaction data, but the simplest here will be to record the resulting GitLab branches, so that each transaction can compare with the previous state. Once that is done, we don't need a cache on the HGitaly side.
General Mercurial caches work on harder constraints that one can trust that previous transactions have updated the cache. They could have been done with a different client, or just command-line options that disable the cache. Within Heptapod, we are in a special case: it is guaranteed that all transactions go through the `gitlab-mirror` hook.
Also, the branch mapping system has painfully evolved to recover from inconsistencies, so I don't worry too much about the robustness of this state system. Still, it will be useful to keep the existing logic on the HGitaly side, in order to rebuild the state file if needed. This could even be made automatic later on once we have a transaction id file to check for staleness.
If we compare with what hg-git does, a simple flat file mounted as a `dict` would be good enough for a first version. In any case, a precomputed mapping would be a more appropriate support for branch querying than what we currently have. The day we want an append-only structure, we may have to go the append-only radix tree way (again!). For instance hgitaly#9 needs prefix matching. Perhaps more useful methods also do.
We'll certainly need a version header, but that's not a problem.
Finally, `.hg/requires` is usually the place to state that the repository needs a Mercurial that understands some data format to operate properly. But in that case, it's probably inappropriate, because the repository will be perfectly viable in itself, as the data is only useful for interaction with the Rails app and HGitaly.Heptapod 0.18.1https://foss.heptapod.net/heptapod/heptapod/-/issues/193Use zstd for revlog compression2020-02-12T14:40:04ZPierre-Yves DavidUse zstd for revlog compressionMercurial is able to use zstd to compress its revlog. This means less CPU being used to access the content of repositories.
If is available in core, but not enabled by default. We need to add this to the global config:
```
[format]
rev...Mercurial is able to use zstd to compress its revlog. This means less CPU being used to access the content of repositories.
If is available in core, but not enabled by default. We need to add this to the global config:
```
[format]
revlog-compression=zstd
```
We should also consider upgrading the existing repository using `hg debugupgraderepo --run` once the config is set.https://foss.heptapod.net/heptapod/heptapod/-/issues/192optimize repository after import2020-07-29T14:08:32ZPierre-Yves Davidoptimize repository after importMercurial had various improvement to the way it store delta. They make the repository smaller and less CPU hungry to look at.
It is worth running the following commands aftern the initial import to make sure we are getting the best poss...Mercurial had various improvement to the way it store delta. They make the repository smaller and less CPU hungry to look at.
It is worth running the following commands aftern the initial import to make sure we are getting the best possible repository format
`hg debugupgraderepo -o re-delta-all --run --no-backup`Heptapod 0.12.0https://foss.heptapod.net/heptapod/heptapod/-/issues/189Automatic clonebundles2023-07-25T06:01:40ZPierre-Yves DavidAutomatic clonebundlesThe `clonebundles` features improve clone performance for client while reducing the load generated by clones on the server.
We should have an easy option to automatically generate and use these with heptapod. I see the following questio...The `clonebundles` features improve clone performance for client while reducing the load generated by clones on the server.
We should have an easy option to automatically generate and use these with heptapod. I see the following questions:
* [ ] When are we triggering new `bundle` generation
* [ ] How do we deal with the bundle generation task
* [ ] How do we serve the bundles
* [ ] How do we deal with bundle for private repositories (where user need to be identified to pull a bundle)
* [ ] How do we store/garbage collect the bundles.
(This task is here to track discussion and progress on this issue, but does not comes with any actual plan to implement it at a given date)Pierre-Yves DavidPierre-Yves Davidhttps://foss.heptapod.net/heptapod/heptapod/-/issues/120Fork and adapt GitLab Workhorse for Mercurial2020-04-24T13:37:52ZGeorges RacinetFork and adapt GitLab Workhorse for MercurialGitlab Workhorse is a reverse proxy sitting between Nginx and the Rails application.
Among its purposes is to deflect any Git HTTP requests that are potentially long (pull, push) directly to Git without going through the Rails applicati...Gitlab Workhorse is a reverse proxy sitting between Nginx and the Rails application.
Among its purposes is to deflect any Git HTTP requests that are potentially long (pull, push) directly to Git without going through the Rails application. Among its obvious performance and robustness benefits, this helps keeping the unicorn (Rails server) to a minimum, while Workhorse itself is designed to handle long timeouts gracefully. I suppose it's also able to stream requests and responses completely, something I doubt the Rails app would do easily.
As far as I know, it works by first calling an internal API on the Rails application for authentication and authorization, calling the inner Git serving process depending on the result.
We would need to have the same for Mercurial. Actually going through the Rails app has already been a problem in the past, because of weird handling of `Content-Encoding: chunked` when we switched from `hg serve` to `gunicorn`. Obviously, it would be much faster and easier on the RAM than what we're currently doing (reading the whole from Mercurial, I think that actually de-chunks IIRC).
That means probably to fork Workhorse. Perhaps we'd have to wait until we get back to the standard Omnibus build/deploy system. Or maybe we'll want that so much at that point that we'll be happy to build from `heptapod-docker`. We'll see.Heptapod 0.12.1