Skip to content
Snippets Groups Projects
  1. Feb 22, 2025
  2. Feb 18, 2025
  3. Jan 27, 2025
    • Pierre-Yves David's avatar
      stream-clone-v2: introduce a way to limit memory usage of the threaded version · aee193b1c784
      Pierre-Yves David authored
      This mechanism limit the read-aheadto avoid uncontrolled memory usage. Note that
      the memory target is not a hard limit and that mercurial will use more memory
      that specified in some cases, but "not too much more".
      
      The current implementation focus on "simplicity" and could be more efficient.
      However this provide a "safe" baseline for the feature. Now that the overall
      shape of the feature is here we can start making it faster.
      
      Here is more benchmark of how everything slows down. The next series will focus
      on optimizing this code path to actually speeds things up.
      
      ### benchmark.name                         = hg.perf.exchange.stream.consume
        # bin-env-vars.hg.flavor                 = default
        # bin-env-vars.hg.py-re2-module          = default
        # benchmark.variants.memory-target       = default
        # benchmark.variants.num-writer          = default
        # benchmark.variants.parallel-processing = yes
        # benchmark.variants.progress            = no
        # benchmark.variants.read-from-memory    = yes
        # benchmark.variants.version             = v2
       ## data-env-vars.name                     = heptapod-public-2024-03-25-zstd-sparse-revlog
      no-thread:    7.244015  ~~~~~
      write-thread: 9.128669  (+26.02%, +1.88)
      read-thread:  9.387256  (+29.59%, +2.14)
      mem-target:   9.901032  (+36.68%, +2.66)
       ## data-env-vars.name                     = mercurial-public-2024-03-22-zstd-sparse-revlog
      no-thread:    0.249693  ~~~~~
      write-thread: 0.275081  (+10.17%, +0.03)
      read-thread:  0.292601  (+17.18%, +0.04)
      mem-target:   0.305973  (+22.54%, +0.06)
       ## data-env-vars.name                     = netbeans-2019-11-07-zstd-sparse-revlog
      no-thread:    13.136674  ~~~~~
      write-thread: 16.374306  (+24.65%, +3.24)
      read-thread:  17.008865  (+29.48%, +3.87)
      mem-target:   18.467590  (+40.58%, +5.33)
       ## data-env-vars.name                     = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog
      no-thread:    5.317709  ~~~~~
      write-thread: 6.783031  (+27.56%, +1.47)
      read-thread:  7.107141  (+33.65%, +1.79)
      mem-target:   7.338505  (+38.00%, +2.02)
       ## data-env-vars.name                     = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog
      no-thread:    5.398368  ~~~~~
      write-thread: 6.737864  (+24.81%, +1.34)
      read-thread:  7.163505  (+32.70%, +1.77)
      mem-target:   7.333354  (+35.84%, +1.93)
       ## data-env-vars.name                     = pypy-2024-03-22-zstd-sparse-revlog
      no-thread:    3.acbb55  ~~~~~
      write-thread: 3.758324  (+22.48%, +0.69)
      read-thread:  3.907693  (+27.34%, +0.84)
      mem-target:   4.238172  (+38.11%, +1.17)
      
       ## data-env-vars.name                     = mozilla-central-2024-03-22-zstd-sparse-revlog
      no-thread:    51.934795  ~~~~~
      write-thread: 66.561340 (+28.16%, +14.63)
      read-thread:  66.902619 (+28.82%, +14.97)
      mem-target:   78.194540 (+50.56%, +26.26)
       ## data-env-vars.name                     = mozilla-unified-2024-03-22-zstd-sparse-revlog
        # benchmark.variants.read-from-memory    = yes
      no-thread:    52.253858  ~~~~~
      write-thread: 66.955037 (+28.13%, +14.70)
      read-thread:  66.397609 (+27.07%, +14.14)
      mem-target:   77.492938 (+48.30%, +25.24)
       ## data-env-vars.name                     = mozilla-try-2024-03-26-zstd-sparse-revlog
        # benchmark.variants.read-from-memory    = no
      no-thread:    130.584329  ~~~~~
      write-thread: 138.770454   (+6.27%, +8.19)
      read-thread:  145.137477 (+11.14%, +14.55)
      mem-target:   164.366925 (+25.87%, +33.78)
      aee193b1c784
  4. Jan 21, 2025
    • Pierre-Yves David's avatar
      stream-clone-v2: use the pass through for threaded stream clone · 9abf173a958b
      Pierre-Yves David authored
      The reader thread just want to iterate over chunk, so let it do just that. This avoid some silly layer of filechunkiter and chunk buffer.
      
      Before this change, we had the following layers:
      
          bundle2part (iterator of chunks)
          → chunkbuffer (iterator of chunks to file-like)
          → filechunkiter (file-like to chunkgs)
          → _DataQueue (thread safe iterator of chunks)
          → chunkbuffer (iterator of chunks to file-like).
      
      We now have removed the first layer of chunkbuffer + filechunkiter:
      
          bundle2part (iterator of chunks)
          → _DataQueue (thread safe iterator of chunks)
          → chunkbuffer (iterator of chunks to file-like).
      
      This fix the output order change that was previously introduced.
      9abf173a958b
  5. Jan 20, 2025
    • Pierre-Yves David's avatar
      stream-clone-v2: also use a thread to read the streamed data · d5ae681834e8
      Pierre-Yves David authored
      We could do useful thing while waiting for data, so we introduce another thread
      dedicated to reading the bundle data and pass them to the thread parsing the
      bundle.
      
      This has a small impact on test output because the bundle part is consumed at a
      different time. However this will be smoothed out very soon so lets ignore that
      for now.
      
      As for the previous patch, the speed up in not there yet. We need to adjust
      various part of the implementation to see some benefit. However, this put the
      generic architecture in place so that we can focus on performance later.
      
      Below are some benchmark result to highlight the slowdown.
      
      ### benchmark.name                         = hg.perf.exchange.stream.consume
        # bin-env-vars.hg.flavor                 = default
        # bin-env-vars.hg.py-re2-module          = default
        # benchmark.variants.memory-target       = default
        # benchmark.variants.num-writer          = default
        # benchmark.variants.parallel-processing = yes
        # benchmark.variants.progress            = no
        # benchmark.variants.read-from-memory    = yes
        # benchmark.variants.version             = v2
       ## data-env-vars.name                     = heptapod-public-2024-03-25-zstd-sparse-revlog
      no-thread:    7.244015  ~~~~~
      write-thread: 9.128669  (+26.02%, +1.88)
      read-thread:  9.387256  (+29.59%, +2.14)
       ## data-env-vars.name                     = mercurial-public-2024-03-22-zstd-sparse-revlog
      no-thread:    0.249693  ~~~~~
      write-thread: 0.275081  (+10.17%, +0.03)
      read-thread:  0.292601  (+17.18%, +0.04)
       ## data-env-vars.name                     = netbeans-2019-11-07-zstd-sparse-revlog
      no-thread:    13.136674  ~~~~~
      write-thread: 16.374306  (+24.65%, +3.24)
      read-thread:  17.008865  (+29.48%, +3.87)
       ## data-env-vars.name                     = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog
      no-thread:    5.317709  ~~~~~
      write-thread: 6.783031  (+27.56%, +1.47)
      read-thread:  7.107141  (+33.65%, +1.79)
       ## data-env-vars.name                     = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog
      no-thread:    5.398368  ~~~~~
      write-thread: 6.737864  (+24.81%, +1.34)
      read-thread:  7.163505  (+32.70%, +1.77)
       ## data-env-vars.name                     = pypy-2024-03-22-zstd-sparse-revlog
      no-thread:    3.acbb55  ~~~~~
      write-thread: 3.758324  (+22.48%, +0.69)
      read-thread:  3.907693  (+27.34%, +0.84)
       ## data-env-vars.name                     = mozilla-central-2024-03-22-zstd-sparse-revlog
      no-thread:    51.934795  ~~~~~
      write-thread: 66.561340 (+28.16%, +14.63)
      read-thread:  66.902619 (+28.82%, +14.97)
       ## data-env-vars.name                     = mozilla-unified-2024-03-22-zstd-sparse-revlog
      no-thread:    52.253858  ~~~~~
      write-thread: 66.955037 (+28.13%, +14.70)
      read-thread:  66.397609 (+27.07%, +14.14)
       ## data-env-vars.name                     = mozilla-try-2024-03-26-zstd-sparse-revlog
        # benchmark.variants.read-from-memory    = no
      no-thread:    130.584329  ~~~~~
      write-thread: 138.770454   (+6.27%, +8.19)
      read-thread:  145.137477 (+11.14%, +14.55)
      d5ae681834e8
    • Pierre-Yves David's avatar
      stream-clone-v2: use dedicated threads to write the data on disk · 7f848cfc4286
      Pierre-Yves David authored
      This could provide a significant performance boost, but various implementation
      details means it is currently slower. More update will help make this boost
      real, but here is the basic idea. The implementation in this patch is unbounded
      in memory which could be a problem in some situation. We will deal with that
      soon.
      
      There is the benchmark result showing a slower run
      
      ### benchmark.name                         = hg.perf.exchange.stream.consume
        # bin-env-vars.hg.flavor                 = default
        # bin-env-vars.hg.py-re2-module          = default
        # benchmark.variants.parallel-processing = yes
        # benchmark.variants.progress            = no
        # benchmark.variants.read-from-memory    = yes
        # benchmark.variants.version             = v2
       ## data-env-vars.name                     = mercurial-public-2024-03-22-zstd-sparse-revlog
      before: 0.249693  ~~~~~
      after:  0.275081  (+10.17%, +0.03)
       ## data-env-vars.name                     = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog
      before: 5.317709  ~~~~~
      after:  6.783031  (+27.56%, +1.47)
       ## data-env-vars.name                     = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog
      before: 5.398368  ~~~~~
      after:  6.737864  (+24.81%, +1.34)
       ## data-env-vars.name                     = pypy-2024-03-22-zstd-sparse-revlog
      before: 3.acbb55  ~~~~~
      after:  3.758324  (+22.48%, +0.69)
       ## data-env-vars.name                     = heptapod-public-2024-03-25-zstd-sparse-revlog
      before: 7.244015  ~~~~~
      after:  9.128669  (+26.02%, +1.88)
       ## data-env-vars.name                     = netbeans-2019-11-07-zstd-sparse-revlog
      before: 13.136674  ~~~~~
      after:  16.374306  (+24.65%, +3.24)
       ## data-env-vars.name                     = mozilla-unified-2024-03-22-zstd-sparse-revlog
      before: 52.253858  ~~~~~
      after:  66.955037 (+28.13%, +14.70)
       ## data-env-vars.name                     = mozilla-central-2024-03-22-zstd-sparse-revlog
      before: 51.934795  ~~~~~
      after:  66.561340 (+28.16%, +14.63)
       ## data-env-vars.name                     = mozilla-try-2024-03-26-zstd-sparse-revlog
        # benchmark.variants.read-from-memory    = no
      before: 130.584329  ~~~~~
      after:  138.770454   (+6.27%, +8.19)
      7f848cfc4286
  6. Nov 28, 2024
  7. Oct 01, 2024
    • Raphaël Gomès's avatar
      update: add a Rust fast-path when updating from null (and clean) · 8b7123c8947b
      Raphaël Gomès authored
      This case is easy to detect and we have all we need to generate a valid
      working copy and dirstate entirely in Rust, which speeds things up
      considerably:
      
      On my machine updating a repo of ~300k files goes from 10.00s down to 4.2s,
      all while consuming 50% less system time, with all caches hot.
      Something to note is that further improvements will probably happen
      with the upcoming `InnerRevlog` series that does smarter
      mmap hanlding, especially for filelogs.
      
      Here are benchmark numbers on a machine with only 4 cores (and no SMT enabled)
      
      ```
      ### data-env-vars.name               = heptapod-public-2024-03-25-ds2-pnm
        # benchmark.name                   = hg.command.update
        # bin-env-vars.hg.py-re2-module    = default
        # bin-env-vars.hg.changeset.node   = <this change>
        # benchmark.variants.atomic-update = no
        # benchmark.variants.scenario      = null-to-tip
        # benchmark.variants.worker        = default
      default: 5.328762  ~~~~~
      rust: 1.308654  (-75.44%, -4.02)
      ### data-env-vars.name               = mercurial-devel-2024-03-22-ds2-pnm
        # benchmark.name                   = hg.command.update
        # bin-env-vars.hg.py-re2-module    = default
        # bin-env-vars.hg.changeset.node   = <this change>
        # benchmark.variants.atomic-update = no
        # benchmark.variants.scenario      = null-to-tip
        # benchmark.variants.worker        = default
      default: 1.693271  ~~~~~
      rust: 1.151053  (-32.02%, -0.54)
      ### data-env-vars.name               = mozilla-unified-2024-03-22-ds2-pnm
        # benchmark.name                   = hg.command.update
        # bin-env-vars.hg.py-re2-module    = default
        # bin-env-vars.hg.changeset.node   = <this change>
        # benchmark.variants.atomic-update = no
        # benchmark.variants.scenario      = null-to-tip
        # benchmark.variants.worker        = default
      default: 38.901613  ~~~~~
      rust: 11.637880 (-70.08%, -27.26)
      ### data-env-vars.name               = netbsd-xsrc-public-2024-09-19-ds2-pnm
        # benchmark.name                   = hg.command.update
        # bin-env-vars.hg.py-re2-module    = default
        # bin-env-vars.hg.changeset.node   = <this change>
        # benchmark.variants.atomic-update = no
        # benchmark.variants.scenario      = null-to-tip
        # benchmark.variants.worker        = default
      default: 4.793727  ~~~~~
      rust: 1.505905  (-68.59%, -3.29)
      ```
      8b7123c8947b
  8. Sep 24, 2024
  9. Apr 09, 2024
  10. Dec 11, 2023
    • Pierre-Yves David's avatar
      changelog: never inline changelog · dcaa2df1f688
      Pierre-Yves David authored
      The test suite mostly use small repositories, that implies that most changelog in the
      tests are inlined. As a result, non-inlined changelog are quite poorly tested.
      Since non-inline changelog are most common case for serious repositories, this
      lack of testing is a significant problem that results in high profile issue like
      the one recently fixed by 66417f55ea33 and 849745d7da89.
      
      Inlining the changelog does not bring much to the table, the number of total
      file saved is negligible, and the changelog will be read by most operation
      anyway.
      
      So this changeset is make it so we never inline the changelog, and de-inline the
      one that are still inlined whenever we touch them.
      
      By doing that, we remove the "dual code path" situation for writing new entry to
      the changelog and move to a "single code path" situation. Having a single
      code path simplify the code and make sure it is covered by test (if test cover
      that situation obviously)
      
      
      This impact all tests that care about the number of file and the exchange size,
      but there is nothing too complicated in them just a lot of churn.
      
      The churn is made "worse" by the fact rust will use the persistent nodemap on
      any changelog now. Which is overall a win as it means testing the persistent
      nodemap more and having less special cases.
      
      In short, having inline changelog is mostly useless and an endless source of
      pain. We get rid of it.
      dcaa2df1f688
  11. May 23, 2023
  12. Jun 01, 2023
    • Arseniy Alekseyev's avatar
      stream-clone: add a v3 version of the protocol · 0452af304808
      Arseniy Alekseyev authored
      This new version is less rigid regarding the extract number of files and number
      of bytes to be actually transfered, it also lays the groundwork for other
      improvements.
      
      The format stays experimental, but this is an interesting base to build upon.
      0452af304808
  13. Jun 19, 2023
  14. May 22, 2023
  15. May 21, 2023
  16. Jan 27, 2022
    • Pierre-Yves David's avatar
      stream-clone: allow to change persistent-nodemap format during stream clone · de3ac3d2c60b
      Pierre-Yves David authored
      Persistent nodemap affect the store format. However it is fairly isolated and
      fast to generate locally. So not making it a fixed part of the stream clone is
      useful.
      
      This allow clients without persistent-nodemap support (default for client
      without Rust enabled, or simply older client). So it make it possible to enable
      persistent nodemap on client, where it can provide a massive boost. without too
      much consequence.
      
      To do so, we stop using it in the advertisement requirements for streaming and
      let the client add/remove the necessary file depending of its configuration.
      
      We still send the files as it seems like a small save to not regenerate them.
      In addition, the way we match them will overlap with the changelog-v2/revlog-v2
      so we can't simply skip the associated patterns.
      
      Differential Revision: https://phab.mercurial-scm.org/D12096
      de3ac3d2c60b
  17. Feb 02, 2022
  18. Feb 01, 2022
  19. Jan 17, 2022
  20. Oct 19, 2021
    • Pierre-Yves David's avatar
      dirstate-v2: freeze the on-disk format · bf11ff22a9af
      Pierre-Yves David authored
      It seems the format as reached a good balance. With a core of new capabilities
      that motivated it initially and enough new feature and room for future
      improvement to be a clear progress we can set a milestone for.
      
      Having the format frozen will help the feature to get real life testing, outside
      of the test suite.
      
      The feature itself stay experimental but the config gains a new name to avoid
      people enable non-frozen version by default.
      
      If too many bugs are reported during the RC we might move the format back to
      experimental and drop its support in future version (in favor of a new one)
      
      Differential Revision: https://phab.mercurial-scm.org/D11709
      bf11ff22a9af
  21. Aug 03, 2021
    • sliquister's avatar
      tests: rely on dummyssh being the default · 9c4204b7f3e4
      sliquister authored
      This commit is exactly the result of running this command:
      
      sed -i -e 's! *\(-e \|--ssh \|--config ui.ssh=\)[ \"]*$PYTHON[ \"]*$\(RUN\|\)TESTDIR/dummyssh[\"]* *! !g' -e '/^[ >]*ssh *=[ "]*$PYTHON[ "]*$\(RUN\|\)TESTDIR\/dummyssh[ "]*$/d' -e 's/^\(  [$] .*[^ ]\) *$/\1/' *.t *.sh
      
      Sometimes the tests can be simplified further, but I think it's
      preferable to do the simplification separately.
      
      Differential Revision: https://phab.mercurial-scm.org/D11245
      9c4204b7f3e4
  22. Aug 25, 2021
  23. May 19, 2021
  24. Jan 14, 2021
  25. Apr 06, 2021
  26. Jan 15, 2021
    • Pierre-Yves David's avatar
      clone: make sure we warm the cache after a clone · 95a615dd77bf
      Pierre-Yves David authored
      This work around any deviciency/limitation of the clone process. In our case
      this ensure the persistent nodemap exist with valid content.
      
      Ideally, the cloning process would also do "the right thing". However since
      older server will never be able to do "the right thing". The local workaround
      will be necessary anyway.
      
      I am not worried by the performance impact of this as `hg clone` is non-instant
      on large repositories where is could matters. Warming the cache if they are
      already correct is very fast. And if they are not already warm, this seems like
      a good time to do so.
      
      This impact various test as more cache are now warmed sooner, all the change
      should be harmless.
      
      Differential Revision: https://phab.mercurial-scm.org/D9789
      95a615dd77bf
  27. Sep 11, 2019
  28. Nov 12, 2018
  29. Oct 01, 2018
    • Gregory Szorc's avatar
      httppeer: report http statistics · 393e44324037
      Gregory Szorc authored
      Now that keepalive.py records HTTP request count and the
      number of bytes sent and received as part of performing those
      requests, we can easily print a report on the activity when
      closing a peer instance!
      
      Exact byte counts are globbed in tests because they are influenced
      by non-deterministic things, such as hostnames and port numbers.
      Plus, the exact byte count isn't too important anyway.
      
      I feel obliged to note that printing the byte count could have
      security implications. e.g. if sending a password via HTTP basic
      auth, the length of that password will influence the byte count
      and the reporting of the byte count could be a side-channel leak
      of the password length. I /think/ this is beyond our threshold
      for concern. But if we think it poses a problem, we can teach the
      byte count logging code to e.g. ignore sensitive HTTP request
      headers. We could also consider not reporting the byte count of
      request headers altogether. But since the wire protocol uses HTTP
      headers for sending command arguments, it is kind of important to
      report their size.
      
      Differential Revision: https://phab.mercurial-scm.org/D4858
      393e44324037
  30. Aug 25, 2018
    • Yuya Nishihara's avatar
      transaction: remember original len(repo) instead of tracking added revs (API) · 5763216ba311
      Yuya Nishihara authored
      It's silly to keep updating xrange(len(changelog), len(changelog) + 1) for
      each added revision. Instead, let's simply remember the first revision to
      be added.
      
      The test output slightly changed as the branch cache is also warmed up by
      stream clone, which seems more consistent.
      
      .. api::
      
         ``tr.changes['revs']`` is replaced by ``tr.changes['origrepolen']`` which
         is the first revision number to be added.
      5763216ba311
  31. May 08, 2018
  32. Apr 03, 2018
  33. Mar 31, 2018
  34. Jan 31, 2018
Loading