1. 15 Apr, 2021 14 commits
    • Georges Racinet's avatar
      New dev cycle · 4d4be16dc5c7
      Georges Racinet authored
      4d4be16dc5c7
    • Georges Racinet's avatar
      Added tag 0.12.0 for changeset df53b71c17a4 · e87cb7e11064
      Georges Racinet authored
      e87cb7e11064
    • Georges Racinet's avatar
      Setting version for release · df53b71c17a4
      Georges Racinet authored
      df53b71c17a4
    • Georges Racinet's avatar
      Merged stable branch into default · fb6208acd24f
      Georges Racinet authored
      fb6208acd24f
    • Georges Racinet's avatar
      New stable dev cycle · 065e782d5751
      Georges Racinet authored
      --HG--
      branch : stable
      065e782d5751
    • Georges Racinet's avatar
      Added tag 0.11.1 for changeset 39a6480f5501 · 8a8b7f110f21
      Georges Racinet authored
      --HG--
      branch : stable
      8a8b7f110f21
    • Georges Racinet's avatar
      Setting version for release · 39a6480f5501
      Georges Racinet authored
      --HG--
      branch : stable
      39a6480f5501
    • Georges Racinet's avatar
      server: mono-process mode · bdd1f208778c
      Georges Racinet authored
      The forking that the `hgitaly-serve` command is doing by
      default is harmful when we want to debug with pdb: request
      handling always end in an error. The new `--mono-process`
      option solves that by serving directly from the main process.
      This is probably a matter of forwarding all the
      right file descriptors for the tty, but it's not worth
      the investigation time when there is such a simple solution.
      
      Could have done it in the command itself, but this is probably
      easier to maintain in the `server` module and certainly easier
      to test.
      
      No strong assertions that it doesn't fork in the test, but
      coverage proves that the early return got executed; that should
      be enough.
      
      --HG--
      branch : stable
      bdd1f208778c
    • Georges Racinet's avatar
      load_repo: fix the class leak for sharedrepo source as well · 58e290c6dbb1
      Georges Racinet authored
      Previous fixes have been developed and measured in a Python 3.9
      virtualenv (HDK). Replaying the measurements in a Python 3.8 HDK
      (the only supported Python version in all current Heptapod branches)
      immediately showed that this wasn't enough.
      
      After reconstruction of the virtualenv with Python 3.8, it became
      obvious that repository instances (and classes) where doubled, and
      the reason was the systematic instantiation of a sharedrepo source.
      Applying the same treatment for the latter got us back to the
      improvement previously obtained.
      
      It seems frankly dubious that the Python version is really
      implicated, but it could happen: we're after all dealing with stuff
      that the GC didn't clear by itself. Still, it could simply have been
      a matter of library version (not Mercurial, it's the same one).
      
      --HG--
      branch : stable
      58e290c6dbb1
    • Georges Racinet's avatar
      load_repo: forcing garbage collection every 1000 calls. · 56bcfbb515df
      Georges Racinet authored
      For something similar with hgweb, look at changeset ff2370a70fe8
      in Mercurial.
      
      Surprisingly, this does not change the memory footprint in our
      current investigations (which is just using examples/client.py
      with more calls). This means that the regular garbage collector
      was able to reap benefits of the parent changeset (calling
      `gc.collect()` is supposed to be more aggressive on the oldest
      generation).
      
      Still, keeping this as a safety net because our example certainly
      wouldn't cover all possible situations
      
      With rates of 100 and 500, the new log lines show us the GC
      working, with more to collect at 500 than at 100. But at 1000,
      we get back to the same amounts of collected objects and time
      spent than with 100. This suggests that this is close to the
      natural rate, hence we aren't really harming performance.
      
      In any case, this is a constant, and we could make it more
      configurable if needed.
      
      --HG--
      branch : stable
      56bcfbb515df
    • Georges Racinet's avatar
      load_repo: main cure for memory leak · 73abd75e439b
      Georges Racinet authored
      This brings down the leaking over 1000 requests from
      55MB to about 5MB. We are not yet forcing garbage collection
      (as `hgwebdir` does).
      
      The `_filteredrepotypes` keeps a correspondence between
      unfiltered repository classes and the filtered version.
      
      The problem is that each new repository is an instance of
      the single-usage class generated in `hgext3rd.topic.reposetup`,
      so these accumulate in `_filteredrepotypes`.
      
      Ironically the mapping seems to be there to avoid some other
      case of leaking:
      
      ```
      # Python <3.4 easily leaks types via __mro__. See
      # https://bugs.python.org/issue17950. We cache dynamically created types
      # so they won't be leaked on every invocation of repo.filtered().
      _filteredrepotypes = weakref.WeakKeyDictionary()
      ```
      
      One could imagine that garbage collection of the repo would also just
      clear its reference from the `WeakKeyDictionary`, but it doesn't
      happen.
      
      A final suprise is that this unlocks the collection of `ui` instances.
      According to investigation made with `guppy3` the class objects were
      actually the ones eating the most memory, but getting rid of `ui`
      instances is also good news.
      
      --HG--
      branch : stable
      73abd75e439b
    • Georges Racinet's avatar
      load_repo: avoid useless ui object copy · e15ae3992048
      Georges Racinet authored
      Found out while investigating heptapod#466 that actually
      the machinery behind `hg.repository` actually performs a
      copy on its own. Our copy was adding pressure on the
      Python memory system. This does not cure the memory leak,
      but it saves about 2MB of leak over 1000 requests, even though
      it doesn't change the fact that none of these ui instances
      are actually collected (seen through instrumentation not
      part of this changeset).
      
      From `localrepo.py` in Mercurial 5.6.1:
      
      ```
      def makelocalrepository(baseui, path, intents=None):
          """Long docstring not cited here."""
          ui = baseui.copy()
          # Prevent copying repo configuration.
          ui.copy = baseui.copy
          (...)
      ```
      
      Note that this last line actually prevents `baseui` (
      would be the copy we were creating) to be freed immediately.
      
      makelocalrepository in turn is called via this backtrace:
      
      ```
        venv/lib64/python3.9/site-packages/mercurial/hg.py(221)repository()
      -> peer = _peerorrepo(
        venv/lib64/python3.9/site-packages/mercurial/hg.py(188)_peerorrepo()
      -> obj = _peerlookup(path).instance(
        venv/lib64/python3.9/site-packages/mercurial/localrepo.py(3214)instance()
      -> return makelocalrepository(ui, localpath, intents=intents)
      > venv/lib64/python3.9/site-packages/mercurial/localrepo.py(517)makelocalrepository()
      -> ui = baseui.copy()
      
      ```
      
      --HG--
      branch : stable
      e15ae3992048
    • Georges Racinet's avatar
      load_repo: using a local variable before returning · 09d6901ca5d7
      Georges Racinet authored
      This will make instrumentation for memory investigation
      easier.
      
      --HG--
      branch : stable
      09d6901ca5d7
    • Georges Racinet's avatar
      Merge branch 'topic/default/blob-tree' into 'branch/default' · e509cf7b74c4
      Georges Racinet authored
      Blob and Tree related service methods
      
      Closes #16, #43, and #44
      
      See merge request !57
      e509cf7b74c4
  2. 06 Apr, 2021 1 commit
  3. 07 Apr, 2021 3 commits
    • Georges Racinet's avatar
      CommitService: implementation of GetTreeEntries · 230354fec952
      Georges Racinet authored
      This relies on the previously listing methods introduced
      in `ManifestMiner`
      
      Closes: #16
      230354fec952
    • Georges Racinet's avatar
      ManifestMiner.iter_dir_with_flat_paths() · c11dcc9cdf5d
      Georges Racinet authored
      This method will be needed for the non recursive case
      of `CommitService.GetTreeEntries`.
      
      The complication is the "flat path" computation, starting
      with its actual definition.
      
      We keep on using a fake changeset class in the tests (with
      a single integration case to tie it up), and we include for
      reference a direct Python translation of the Gitaly algorithm for
      reference.
      
      This algorithm is of course meant for Git, where
      the directory structure is explicit and naturally arborescent,
      but in Mercurial (with the current manifests), it would be O(n^2),
      where n is the total length of the manifest, and that is not
      acceptable.
      c11dcc9cdf5d
    • Georges Racinet's avatar
      ManifestMiner.iter_dir_recursive() · 1409a3e0bcf8
      Georges Racinet authored
      This new method will be needed for the `recursive` request option
      of `CommitService.GetTreeEntries`.
      
      This direct implementation can well be later on superseded by
      better usage of the manifest API.
      
      In the tests, to easily multiply the cases encountered, we use
      a pseudo manifest whose sole purpose is to be an iterator of
      paths. After all, that's about all the method needs.
      This is much lighter that creating actual Mercurial content
      for all the cases. A single integration test case is then enough
      to validate that it really works.
      1409a3e0bcf8
  4. 01 Apr, 2021 3 commits
  5. 20 Mar, 2021 2 commits
  6. 16 Mar, 2021 2 commits
  7. 04 Apr, 2021 3 commits
  8. 14 Mar, 2021 4 commits
  9. 19 Mar, 2021 2 commits
  10. 12 Mar, 2021 3 commits
    • Georges Racinet's avatar
      Setting version for release · 79d8f8a8df49
      Georges Racinet authored
      79d8f8a8df49
    • Georges Racinet's avatar
      RepositoryService.WriteRef: special ref target can be any revision · 28d8fc467b7a
      Georges Racinet authored
      Found out about this while testing with the Rails app. Of course,
      it is natural for symbolic refs, and doesn't sound very logical
      with normal refs, a result of under-specification ultimately due
      to the fact that Gitaly simply uses `git update-ref`.
      
      We keep refusing ref target revisions for keep-arounds because
      we are much surer that the Rails app will only use full SHAs as
      target revisions. Also, it swallows exceptions, avoiding failures
      if `WriteRef` refuses to create the keep-around
      (see `Gitlab::Git::KeepAround`).
      
      We should have a Gitaly comparison test for this, but we already
      have RSpec tests in the Rails app, so this can be in a follow-up.
      28d8fc467b7a
    • Georges Racinet's avatar
      Bumping future version · 5cc50c9c21ca
      Georges Racinet authored
      The work to support special refs is well worth a minor version
      update.
      5cc50c9c21ca
  11. 11 Mar, 2021 3 commits