Skip to content
Snippets Groups Projects
  1. Apr 26, 2024
  2. May 06, 2024
  3. Apr 22, 2024
    • Georges Racinet's avatar
      rust: blanket implementation of Graph for Graph references · b08c5fbe
      Georges Racinet authored
      The need comes from the fact that `AncestorsIterator` and many
      Graph-related algorithms take ownership of the `Graph` they work with.
      This, in turn is due to them needing to accept the `Index` instances
      that are provided by the Python layers (that neither rhg nor `RHGitaly`
      use, of course): the fact that nowadays the Python layer holds an object
      that is itself implemented in Rust does not change the core problem that
      they cannot be tracked by the borrow checker.
      
      Even though it looks like cloning `Changelog` would be cheap, it seems
      hard to guarantee that on the long run. The object is already too rich
      for us to be comfortable with it, when using references is the most
      natural and guaranteed way of proceeding.
      
      The added test seems a bit superfleous, but it will act as a reminder
      that this feature is really useful until something in the Mercurial code
      base actually uses it.
      b08c5fbe
  4. May 06, 2024
  5. May 02, 2024
    • Pierre-Yves David's avatar
      re2: make errors quiet · 6c39edd1
      Pierre-Yves David authored
      By default, the re2 library will output error on its own instead of keeping the
      error in an exception. This make re2 printing spurious error before fallback to
      the stdlib remodule that may accept the pattern or also fails to parse it and
      raise a proper error that will be handled by Mercurial.
      
      So we also pass an Option object that changes this default.
      6c39edd1
    • Pierre-Yves David's avatar
      fold-or-prune-me: update proposal · fc317bd5
      Pierre-Yves David authored
      This does the same things but with a narrower wrapping.
      fc317bd5
  6. Mar 26, 2024
  7. Apr 16, 2024
  8. Mar 31, 2024
    • Felipe Resende's avatar
      subrepo: propagate non-default path on outgoing · 3e0f86f0
      Felipe Resende authored
      There was already a fix made in 5dbff89cf107 for pull and push commands. I did
      the same for the outgoing command.
      
      The problem I identified is that when the parent repository has multiple paths,
      the outgoing command was not respecting the parent path used and was always
      using the default path for subrepositories.
      3e0f86f0
  9. Apr 16, 2024
  10. Apr 05, 2024
  11. Apr 09, 2024
  12. Apr 17, 2024
  13. Mar 13, 2024
    • Pierre-Yves David's avatar
      tags-cache: directly perform a monimal walk for hgtagsfnodescache warming · b0aaffcb
      Pierre-Yves David authored
      We do something narrower than the path retrieving data. So lets use dedicated
      code instead.
      
      This provides further useful speedup:
      
      ### data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
        # benchmark.name                = hg.debug.debug-update-cache
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.pre-state  = warm
      before-this-series:  19.947581
      skip-fnode-filter:   18.916804   (-5.17%, -1.03)
      use-rev-num:         17.493725  (-12.30%, -2.45)
      this-changesets:     15.919466  (-20.19%, -4.03)
      b0aaffcb
    • Pierre-Yves David's avatar
      tags-cache: directly operate on rev-num warming hgtagsfnodescache · 2664cacd
      Pierre-Yves David authored
      Not having to goes through nodeid speed up things notably.
      
      ### data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
        # benchmark.name                = hg.debug.debug-update-cache
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.pre-state  = warm
      before-this-series:  19.947581
      before-this-changes: 18.916804   (-5.17%, -1.03)
      this-changesets:     17.493725  (-12.30%, -2.45)
      2664cacd
    • Pierre-Yves David's avatar
      tags-cache: skip the filternode step if we are not going to use it · b4a6a1ff
      Pierre-Yves David authored
      When warming the hgtagsfnodescache, we don't need the actual result, so we can
      simply skip the part that "filter" fnode we read from the cache. So provide a
      quite visible speed up to the top level `hg debugupdatecache` function.
      
      ### data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
        # benchmark.name                = hg.debug.debug-update-cache
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.pre-state  = warm
      before: 19.947581
      after:  18.916804   (-5.17%, -1.03)
      b4a6a1ff
    • Pierre-Yves David's avatar
      tags-cache: add a dedicated warm cache function to hgtagsfnodescache · 6378d575
      Pierre-Yves David authored
      Having a dedicated API point will help to optimize that specific usage. Right
      doing a full phases weam takes a long time, even when the cache is already
      filled.
      6378d575
  14. Apr 09, 2024
    • Pierre-Yves David's avatar
      outgoing: add a simple fastpath when there is no common · 3a6fae3b
      Pierre-Yves David authored
      This further speed up case like `hg bundle --all` for larger repository.
      
      ### data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
        # benchmark.name                = hg.command.bundle
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.revs       = all
        # benchmark.variants.type       = none-streamv2
      before: 316.749699
      after:  311.165461   (-1.76%, -5.58)
      
      
      There is further work to be done in this area like not doing any outgoing
      computation in the stream case for example. however the recent changes already
      gives use a large win for a small amount of local work.
      
      ### benchmark.name                = hg.command.bundle
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.revs       = all
        # benchmark.variants.type       = none-streamv2
       ## data-env-vars.name            = mercurial-public-2024-03-22-zstd-sparse-revlog
      pre-%ln-change: 1.263859
      the-%ln-change: 0.700229  (-44.60%, -0.56)
      prev-changeset: 0.496050  (-60.75%, -0.77)
      this-changeset: 0.495243  (-60.81%, -0.77)
       ## data-env-vars.name            = tryton-public-2024-03-22-zstd-sparse-revlog
      pre-%ln-change: 2.975765
      the-%ln-change: 1.870798  (-37.13%, -1.10)
      prev-changeset: 1.461583  (-50.88%, -1.51)
      this-changeset: 1.469185  (-50.63%, -1.51)
       ## data-env-vars.name            = pypy-2024-03-22-zstd-sparse-revlog
      pre-%ln-change: 4.540080
      the-%ln-change: 3.401700  (-25.07%, -1.14)
      prev-changeset: 2.915810  (-35.78%, -1.62)
      this-changeset: 2.911643  (-35.87%, -1.63)
       ## data-env-vars.name            = heptapod-public-2024-03-25-zstd-sparse-revlog
      pre-%ln-change: 10.138396
      the-%ln-change: 7.750458  (-23.55%, -2.39)
      prev-changeset: 6.665565  (-34.25%, -3.47)
      this-changeset: 6.672078  (-34.19%, -3.47)
       ## data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
      pre-%ln-change: 399.484481
      the-%ln-change: 346.508952 (-13.26%, -52.98)
      prev-changeset: 316.749699 (-20.71%, -82.73)
      this-changeset: 311.165461 (-22.11%, -88.32)
      3a6fae3b
    • Pierre-Yves David's avatar
      outgoing: rework the handling of the `missingroots` case to be faster · e3a5ec2d
      Pierre-Yves David authored
      The previous implementation was slow, to the point it was taking a significant
      amount of `hg bundle --type none-streamv2` call. We rework the code to compute
      the same value much faster, making the operation disappear from the `hg bundle
      --type none-streamv2` profile. Someone would remark that producing a streamclone
      does not requires an `outgoing` object. However that is a matter for another
      day. There is other user of `missingroots` (non stream `hg bundle` call for
      example), and they will also benefit from this rework.
      
      We implement an old TODO in the process, directly computing the missing and
      common attribute as we have most element at hand already.
      
      ### benchmark.name                = hg.command.bundle
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.revs       = all
        # benchmark.variants.type       = none-streamv2
       ## data-env-vars.name            = heptapod-public-2024-03-25-zstd-sparse-revlog
      before: 7.750458
      after:  6.665565  (-14.00%, -1.08)
       ## data-env-vars.name            = mercurial-public-2024-03-22-zstd-sparse-revlog
      before: 0.700229
      after:  0.496050  (-29.16%, -0.20)
       ## data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
      before: 346.508952
      after:  316.749699  (-8.59%, -29.76)
       ## data-env-vars.name            = pypy-2024-03-22-zstd-sparse-revlog
      before: 3.401700
      after:  2.915810  (-14.28%, -0.49)
       ## data-env-vars.name            = tryton-public-2024-03-22-zstd-sparse-revlog
      before: 1.870798
      after:  1.461583  (-21.87%, -0.41)
      
      
      
      note: this whole `missingroots` of outgoing has a limited number of callers and
      could likely be replace by something simpler (like taking an explicit
      "missing_revs" set for example). However this is a wider change and we focus on
      a small impact, quick rework that does not change the API for now.
      e3a5ec2d
  15. Apr 14, 2024
    • Pierre-Yves David's avatar
      proxy-vfs: also proxy the `audit` attribute · b5500857
      Pierre-Yves David authored
      In the previous changeset, we had to do a little dance to access the useful
      `audit` attribute. We now provide a proper accessors to it.
      
      We don't update the code in `perf.py` because it has to remain compatible with
      older version of Mercurial. This will just be nicer in the future.
      b5500857
  16. Apr 13, 2024
    • Pierre-Yves David's avatar
      perf: clear vfs audit_cache before each run · 24844407
      Pierre-Yves David authored
      When generating a stream clone, we spend a large amount of time auditing path.
      Before this changes, the first run was warming the vfs cache for the other
      runs, leading to a large runtime difference and a "faulty" reported timing for
      the operation.
      
      We now clear this important cache between run to get a more realistic timing.
      
      Below are some example of median time change when clearing these cases. The
      maximum time for a run did not changed significantly.
      
      ### data-env-vars.name             = mozilla-central-2018-08-01-zstd-sparse-revlog
        # benchmark.name                = hg.perf.exchange.stream.generate
        # bin-env-vars.hg.flavor         = default
        # bin-env-vars.hg.py-re2-module  = default
        # benchmark.variants.version    = latest
      no-clearing:    17.289905
      cache-clearing: 21.587965  (+24.86%, +4.30)
       ## data-env-vars.name             = mozilla-central-2024-03-22-zstd-sparse-revlog
      no-clearing:    32.670748
      cache-clearing: 40.467095  (+23.86%, +7.80)
       ## data-env-vars.name             = mozilla-try-2019-02-18-zstd-sparse-revlog
      no-clearing:    37.838858
      cache-clearing: 46.072749  (+21.76%, +8.23)
       ## data-env-vars.name             = mozilla-unified-2024-03-22-zstd-sparse-revlog
      no-clearing:    32.969395
      cache-clearing: 39.646209  (+20.25%, +6.68)
      
      In addition, this significantly reduce the timing difference between the
      performance command, from the perf extensions and a `real `hg bundle` call
      producing a stream bundle. Some significant differences remain especially on
      the "mozilla-try" repositories, but they are now smaller.
      
      Note that some of that difference will actually not be
      attributable to the stream generation (like maybe phases or branch map
      computation).
      
      Below are some benchmarks done on a currently draft changeset fixing some
      unrelated slowness in `hg bundle` (34a78972af40)
      
      ### data-env-vars.name             = mozilla-central-2018-08-01-zstd-sparse-revlog
        # bin-env-vars.hg.flavor         = default
        # bin-env-vars.hg.py-re2-module  = default
      hg.perf.exchange.stream.generate:  21.587965
      hg.command.bundle:                 24.301799  (+12.57%, +2.71)
       ## data-env-vars.name             = mozilla-central-2024-03-22-zstd-sparse-revlog
      hg.perf.exchange.stream.generate:  40.467095
      hg.command.bundle:                 44.831317  (+10.78%, +4.36)
       ## data-env-vars.name             = mozilla-unified-2024-03-22-zstd-sparse-revlog
      hg.perf.exchange.stream.generate:  39.646209
      hg.command.bundle:                 45.395258  (+14.50%, +5.75)
       ## data-env-vars.name             = mozilla-try-2019-02-18-zstd-sparse-revlog
      hg.perf.exchange.stream.generate:  46.072749
      hg.command.bundle:                 55.882608  (+21.29%, +9.81)
       ## data-env-vars.name             = mozilla-try-2023-03-22-zlib-general-delta
      hg.perf.exchange.stream.generate: 334.716708
      hg.command.bundle:                377.856767  (+12.89%, +43.14)
       ## data-env-vars.name             = mozilla-try-2023-03-22-zstd-sparse-revlog
      hg.perf.exchange.stream.generate: 302.972301
      hg.command.bundle:                326.098755  (+7.63%, +23.13)
      24844407
  17. Apr 14, 2024
    • Pierre-Yves David's avatar
      perf: start recording total time after warming · ecf4b959
      Pierre-Yves David authored
      The warming might be costly and this should not affect the "time profile" of the
      actual collection.
      ecf4b959
    • Pierre-Yves David's avatar
      perf: run the gc before each run · a6bdd2e6
      Pierre-Yves David authored
      The python garbage collector is a large source of performance troubles, we run
      it right before the timed section to reduce the change for the gc to add noise
      to the benchmark.
      a6bdd2e6
    • Pierre-Yves David's avatar
      perf: allow profiling of more than one run · 90ef3e04
      Pierre-Yves David authored
      By default, we still profile the first run only. However profiling more run help
      to understand side effect from one run to the other. So we add an option to be
      able to do so.
      90ef3e04
    • Pierre-Yves David's avatar
      profiler: flush after writing the profiler output · 1574718f
      Pierre-Yves David authored
      Otherwise, the profiler output might only partially appears until the next flush
      of the buffer. Since profiling often happens for long operation, the next flush
      can be a long time away.
      1574718f
    • Pierre-Yves David's avatar
      stream-clone: disable gc for the entry listing section for the v2 format · 8cd317c0
      Pierre-Yves David authored
      This is similar to the change we did for the v3 format in 6e4c8366c5ce.
      
      The benchmark bellow show this gives us a notable gains, especially on larger
      repositories.
      
      ### benchmark.name                = hg.perf.stream-locked-section
        # benchmark.name                = hg.perf.stream-locked-section
        # bin-env-vars.hg.flavor        = default
        # bin-env-vars.hg.py-re2-module = default
        # benchmark.variants.version    = v2
       ## data-env-vars.name            = pypy-2018-08-01-zstd-sparse-revlog
      5e931bf8707c: 0.503820  ~~~~~
      1106d1bf695e: 0.470078   (-6.70%, -0.03)
       ## data-env-vars.name            = pypy-2024-03-22-zstd-sparse-revlog
      5e931bf8707c: 0.535756  ~~~~~
      1106d1bf695e: 0.490249   (-8.49%, -0.05)
       ## data-env-vars.name            = heptapod-public-2024-03-25-zstd-sparse-revlog
      5e931bf8707c: 1.327041  ~~~~~
      1106d1bf695e: 1.174636  (-11.48%, -0.15)
       ## data-env-vars.name            = netbeans-2018-08-01-zstd-sparse-revlog
      5e931bf8707c: 2.439158  ~~~~~
      1106d1bf695e: 2.220515   (-8.96%, -0.22)
       ## data-env-vars.name            = netbeans-2019-11-07-zstd-sparse-revlog
      5e931bf8707c: 2.630794  ~~~~~
      1106d1bf695e: 2.261473  (-14.04%, -0.37)
       ## data-env-vars.name            = mozilla-central-2018-08-01-zstd-sparse-revlog
      5e931bf8707c: 5.769002  ~~~~~
      1106d1bf695e: 5.062000  (-12.26%, -0.71)
       ## data-env-vars.name            = mozilla-try-2019-02-18-zstd-sparse-revlog
      5e931bf8707c: 13.351750  ~~~~~
      1106d1bf695e: 12.346655   (-7.53%, -1.01)
       ## data-env-vars.name            = mozilla-central-2024-03-22-zstd-sparse-revlog
      5e931bf8707c: 10.772939  ~~~~~
      1106d1bf695e: 9.495407  (-11.86%, -1.28)
       ## data-env-vars.name            = mozilla-unified-2024-03-22-zstd-sparse-revlog
      5e931bf8707c: 10.864297  ~~~~~
      1106d1bf695e: 9.475597  (-12.78%, -1.39)
       ## data-env-vars.name            = mozilla-try-2023-03-22-zstd-sparse-revlog
      5e931bf8707c: 17.448335  ~~~~~
      1106d1bf695e: 16.027474   (-8.14%, -1.42)
      8cd317c0
  18. Apr 09, 2024
    • Pierre-Yves David's avatar
      phases: rework the logic of _pushdiscoveryphase to bound complexity · 1cef1412
      Pierre-Yves David authored
      This rework the various graph traversal in _pushdiscoveryphase to keep the
      complexity in check.
      
      This is done though a couple of things:
      
      - first, limiting the space we have to explore, for example, if we are not in
        publishing push, we don't need to consider remote draft roots that are also
        draft locally, as there is nothing to be moved there.
      
      - avoid unbounded descendant computation, and use the faster "rev between"
        computation.
      
      This provide a massive boost to performance when exchanging with repository with
      a massive amount of draft, like mozilla-try:
      
      ### data-env-vars.name                             = mozilla-try-2023-03-22-zstd-sparse-revlog
        # benchmark.name                                 = hg.command.push
        # bin-env-vars.hg.flavor                         = default
        # bin-env-vars.hg.py-re2-module                  = default
        # benchmark.variants.explicit-rev                = all-out-heads
        # benchmark.variants.issue6528                   = disabled
        # benchmark.variants.protocol                    = ssh
        # benchmark.variants.reuse-external-delta-parent = default
      
       ## benchmark.variants.revs                        = any-1-extra-rev
      before: 20.346590 seconds
      after:  11.232059 seconds (-38.15%, -7.48 seconds)
       ## benchmark.variants.revs                        = any-100-extra-rev
      before: 24.752051 seconds
      after:  15.367412 seconds (-37.91%, -9.38 seconds)
      
      
      After this changes, the push operation is still quite too slow. Some of this
      can be attributed to general phases slowness (reading all the roots from disk
      for example) and other know slowness (not using persistent-nodemap, branchmap,
      tags, etc. We are also working on them, but with this series, phase discovery
      during push no longer showing up in profile and this is a pretty nice and bit
      low-hanging fruit out of the way.
      
      ### (same case as the above)
        # benchmark.variants.revs                        = any-1-extra-rev
      pre-%ln-change: 44.235070
      this-changeset: 11.232059 seconds (-74.61%, -33.00 seconds)
        # benchmark.variants.revs                        = any-100-extra-rev
      pre-%ln-change: 49.234697
      this-changeset: 15.367412 seconds (-68.79%, -33.87 seconds)
      
      
      Note that with this change, the `hg push` performance is now much closer to the
      `hg pull` performance, even it still lagging behind a bit. (and the overall
      performance are still too slow).
      
      ### data-env-vars.name                           = mozilla-try-2023-03-22-ds2-pnm
        # benchmark.variants.explicit-rev              = all-out-heads
        # benchmark.variants.issue6528                 = disabled
        # benchmark.variants.protocol                  = ssh
        # benchmark.variants.pulled-delta-reuse-policy = default
        # bin-env-vars.hg.flavor                       = rust
       ## benchmark.variants.revs                      = any-1-extra-rev
      hg.command.pull: 6.517450
      hg.command.push: 11.219888
       ## benchmark.variants.revs                      = any-100-extra-rev
      hg.command.pull: 10.160991
      hg.command.push: 14.251107
      
      ### data-env-vars.name                           = mozilla-try-2023-03-22-zstd-sparse-revlog
        # bin-env-vars.hg.py-re2-module                = default
        # benchmark.variants.explicit-rev              = all-out-heads
        # benchmark.variants.issue6528                 = disabled
        # benchmark.variants.protocol                  = ssh
        # benchmark.variants.pulled-delta-reuse-policy = default
       ## bin-env-vars.hg.flavor                       = default
       ## benchmark.variants.revs                      = any-1-extra-rev
      hg.command.pull: 8.577772
      hg.command.push: 11.232059
       ## bin-env-vars.hg.flavor                       = default
       ## benchmark.variants.revs                      = any-100-extra-rev
      hg.command.pull: 13.152976
      hg.command.push: 15.367412
       ## bin-env-vars.hg.flavor                       = rust
       ## benchmark.variants.revs                      = any-1-extra-rev
      hg.command.pull: 8.731982
      hg.command.push: 11.178751
       ## bin-env-vars.hg.flavor                       = rust
       ## benchmark.variants.revs                      = any-100-extra-rev
      hg.command.pull: 13.184236
      hg.command.push: 15.620843
      1cef1412
Loading