- Apr 15, 2021
-
-
Georges Racinet authored
-
Georges Racinet authored
-
Georges Racinet authored
-
Georges Racinet authored
-
Georges Racinet authored
-
Georges Racinet authored
The forking that the `hgitaly-serve` command is doing by default is harmful when we want to debug with pdb: request handling always end in an error. The new `--mono-process` option solves that by serving directly from the main process. This is probably a matter of forwarding all the right file descriptors for the tty, but it's not worth the investigation time when there is such a simple solution. Could have done it in the command itself, but this is probably easier to maintain in the `server` module and certainly easier to test. No strong assertions that it doesn't fork in the test, but coverage proves that the early return got executed; that should be enough.
-
Georges Racinet authored
Previous fixes have been developed and measured in a Python 3.9 virtualenv (HDK). Replaying the measurements in a Python 3.8 HDK (the only supported Python version in all current Heptapod branches) immediately showed that this wasn't enough. After reconstruction of the virtualenv with Python 3.8, it became obvious that repository instances (and classes) where doubled, and the reason was the systematic instantiation of a sharedrepo source. Applying the same treatment for the latter got us back to the improvement previously obtained. It seems frankly dubious that the Python version is really implicated, but it could happen: we're after all dealing with stuff that the GC didn't clear by itself. Still, it could simply have been a matter of library version (not Mercurial, it's the same one).
-
Georges Racinet authored
For something similar with hgweb, look at changeset ff2370a70fe8 in Mercurial. Surprisingly, this does not change the memory footprint in our current investigations (which is just using examples/client.py with more calls). This means that the regular garbage collector was able to reap benefits of the parent changeset (calling `gc.collect()` is supposed to be more aggressive on the oldest generation). Still, keeping this as a safety net because our example certainly wouldn't cover all possible situations With rates of 100 and 500, the new log lines show us the GC working, with more to collect at 500 than at 100. But at 1000, we get back to the same amounts of collected objects and time spent than with 100. This suggests that this is close to the natural rate, hence we aren't really harming performance. In any case, this is a constant, and we could make it more configurable if needed.
-
Georges Racinet authored
This brings down the leaking over 1000 requests from 55MB to about 5MB. We are not yet forcing garbage collection (as `hgwebdir` does). The `_filteredrepotypes` keeps a correspondence between unfiltered repository classes and the filtered version. The problem is that each new repository is an instance of the single-usage class generated in `hgext3rd.topic.reposetup`, so these accumulate in `_filteredrepotypes`. Ironically the mapping seems to be there to avoid some other case of leaking: ``` # Python <3.4 easily leaks types via __mro__. See # https://bugs.python.org/issue17950. We cache dynamically created types # so they won't be leaked on every invocation of repo.filtered(). _filteredrepotypes = weakref.WeakKeyDictionary() ``` One could imagine that garbage collection of the repo would also just clear its reference from the `WeakKeyDictionary`, but it doesn't happen. A final suprise is that this unlocks the collection of `ui` instances. According to investigation made with `guppy3` the class objects were actually the ones eating the most memory, but getting rid of `ui` instances is also good news.
-
Georges Racinet authored
Found out while investigating heptapod#466 that actually the machinery behind `hg.repository` actually performs a copy on its own. Our copy was adding pressure on the Python memory system. This does not cure the memory leak, but it saves about 2MB of leak over 1000 requests, even though it doesn't change the fact that none of these ui instances are actually collected (seen through instrumentation not part of this changeset). From `localrepo.py` in Mercurial 5.6.1: ``` def makelocalrepository(baseui, path, intents=None): """Long docstring not cited here.""" ui = baseui.copy() # Prevent copying repo configuration. ui.copy = baseui.copy (...) ``` Note that this last line actually prevents `baseui` ( would be the copy we were creating) to be freed immediately. makelocalrepository in turn is called via this backtrace: ``` venv/lib64/python3.9/site-packages/mercurial/hg.py(221)repository() -> peer = _peerorrepo( venv/lib64/python3.9/site-packages/mercurial/hg.py(188)_peerorrepo() -> obj = _peerlookup(path).instance( venv/lib64/python3.9/site-packages/mercurial/localrepo.py(3214)instance() -> return makelocalrepository(ui, localpath, intents=intents) > venv/lib64/python3.9/site-packages/mercurial/localrepo.py(517)makelocalrepository() -> ui = baseui.copy() ```
-
Georges Racinet authored
This will make instrumentation for memory investigation easier.
-
Georges Racinet authored
Blob and Tree related service methods Closes #16, #43, and #44 See merge request !57
-
- Apr 06, 2021
-
-
Georges Racinet authored
This will in particular unskip related RSpec tests in dev setups where HGitaly is at this revision, and of course also from the upcoming release.
-
- Apr 07, 2021
-
-
Georges Racinet authored
This relies on the previously listing methods introduced in `ManifestMiner` Closes: #16
-
Georges Racinet authored
This method will be needed for the non recursive case of `CommitService.GetTreeEntries`. The complication is the "flat path" computation, starting with its actual definition. We keep on using a fake changeset class in the tests (with a single integration case to tie it up), and we include for reference a direct Python translation of the Gitaly algorithm for reference. This algorithm is of course meant for Git, where the directory structure is explicit and naturally arborescent, but in Mercurial (with the current manifests), it would be O(n^2), where n is the total length of the manifest, and that is not acceptable.
-
Georges Racinet authored
This new method will be needed for the `recursive` request option of `CommitService.GetTreeEntries`. This direct implementation can well be later on superseded by better usage of the manifest API. In the tests, to easily multiply the cases encountered, we use a pseudo manifest whose sole purpose is to be an iterator of paths. After all, that's about all the method needs. This is much lighter that creating actual Mercurial content for all the cases. A single integration test case is then enough to validate that it really works.
-
- Mar 05, 2021
-
-
Georges Racinet authored
Closes #43
-
- Feb 11, 2021
-
-
Georges Racinet authored
The `iter_blob_chunks` now does duplicate the `stream.split_batches()` that was introduced actually later, but is interesting on its own: - it could have better performance profile - it returns the `is_first` boolean, that allows not to repeat oid and size, which callers of `stream.split_batches()` IIRC also need in a different case. Closes #44
-
- Feb 10, 2021
-
-
Georges Racinet authored
-
- Apr 06, 2021
-
-
Georges Racinet authored
This method allows to retrieve a specific entry in a Git tree, hence itself a tree or a blob, while GetTreeEntries lists the entries in a tree. For instance, on the Rails side, it is currently the method called behind `Blob.find`. The size computation is a bit expensive in case the entry is itself a tree for how useful it is supposed to be. Closes #16
-
Georges Racinet authored
yet another error helper modelled on the existing ones
-
- Apr 01, 2021
-
-
Georges Racinet authored
The `file_context.git_perms` function will be useful in Blob related service methods, is easily testable outside of gRPC, and was the occasion to start a new Python module dealing with `filecontext` instances.
-
- Mar 20, 2021
-
-
Sushil Khanchi authored
-
Sushil Khanchi authored
-
- Mar 16, 2021
-
-
Sushil Khanchi authored
-
Sushil Khanchi authored
-
- Apr 04, 2021
-
-
Georges Racinet authored
We just provide the HGitaly version. All other information is mostly prematurate. Maybe we'll end up filling in the Mercurial version as the `git_version` field for the HGitaly3 milestone, but for now (approaching HGitaly2 milestone), HGitaly runs on the same system as the Rails, which obtains that information already by a direct `hg` call.
-
Georges Racinet authored
This makes the version number importable from Python code as `hgitaly.__version__`, with the single source of truth being the `hgitaly/VERSION` file.
-
Georges Racinet authored
-
- Mar 04, 2021
-
-
Sushil Khanchi authored
-
Sushil Khanchi authored
-
- Mar 30, 2021
-
-
Sushil Khanchi authored
-
Sushil Khanchi authored
It was recorded as "copy" instead of "rename" as we didn't pass the 'bar' to wrapper.commit() It didn't have any impact on response because response attrs won't really have any difference b/w rename and copy. But this change is to make sure that we have correct repo setup as explained by comments.
-
- Feb 28, 2021
-
-
Sushil Khanchi authored
-
Sushil Khanchi authored
This will be used by upcoming patches.
-
Sushil Khanchi authored
-
Sushil Khanchi authored
-
Sushil Khanchi authored
-
- Apr 01, 2021
-
-
Georges Racinet authored
-
Georges Racinet authored
-