Skip to content
Snippets Groups Projects
Commit aee193b1c784 authored by Pierre-Yves David's avatar Pierre-Yves David :octopus:
Browse files

stream-clone-v2: introduce a way to limit memory usage of the threaded version

This mechanism limit the read-aheadto avoid uncontrolled memory usage. Note that
the memory target is not a hard limit and that mercurial will use more memory
that specified in some cases, but "not too much more".

The current implementation focus on "simplicity" and could be more efficient.
However this provide a "safe" baseline for the feature. Now that the overall
shape of the feature is here we can start making it faster.

Here is more benchmark of how everything slows down. The next series will focus
on optimizing this code path to actually speeds things up.

### benchmark.name                         = hg.perf.exchange.stream.consume
  # bin-env-vars.hg.flavor                 = default
  # bin-env-vars.hg.py-re2-module          = default
  # benchmark.variants.memory-target       = default
  # benchmark.variants.num-writer          = default
  # benchmark.variants.parallel-processing = yes
  # benchmark.variants.progress            = no
  # benchmark.variants.read-from-memory    = yes
  # benchmark.variants.version             = v2
 ## data-env-vars.name                     = heptapod-public-2024-03-25-zstd-sparse-revlog
no-thread:    7.244015  ~~~~~
write-thread: 9.128669  (+26.02%, +1.88)
read-thread:  9.387256  (+29.59%, +2.14)
mem-target:   9.901032  (+36.68%, +2.66)
 ## data-env-vars.name                     = mercurial-public-2024-03-22-zstd-sparse-revlog
no-thread:    0.249693  ~~~~~
write-thread: 0.275081  (+10.17%, +0.03)
read-thread:  0.292601  (+17.18%, +0.04)
mem-target:   0.305973  (+22.54%, +0.06)
 ## data-env-vars.name                     = netbeans-2019-11-07-zstd-sparse-revlog
no-thread:    13.136674  ~~~~~
write-thread: 16.374306  (+24.65%, +3.24)
read-thread:  17.008865  (+29.48%, +3.87)
mem-target:   18.467590  (+40.58%, +5.33)
 ## data-env-vars.name                     = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog
no-thread:    5.317709  ~~~~~
write-thread: 6.783031  (+27.56%, +1.47)
read-thread:  7.107141  (+33.65%, +1.79)
mem-target:   7.338505  (+38.00%, +2.02)
 ## data-env-vars.name                     = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog
no-thread:    5.398368  ~~~~~
write-thread: 6.737864  (+24.81%, +1.34)
read-thread:  7.163505  (+32.70%, +1.77)
mem-target:   7.333354  (+35.84%, +1.93)
 ## data-env-vars.name                     = pypy-2024-03-22-zstd-sparse-revlog
no-thread:    3.acbb55  ~~~~~
write-thread: 3.758324  (+22.48%, +0.69)
read-thread:  3.907693  (+27.34%, +0.84)
mem-target:   4.238172  (+38.11%, +1.17)

 ## data-env-vars.name                     = mozilla-central-2024-03-22-zstd-sparse-revlog
no-thread:    51.934795  ~~~~~
write-thread: 66.561340 (+28.16%, +14.63)
read-thread:  66.902619 (+28.82%, +14.97)
mem-target:   78.194540 (+50.56%, +26.26)
 ## data-env-vars.name                     = mozilla-unified-2024-03-22-zstd-sparse-revlog
  # benchmark.variants.read-from-memory    = yes
no-thread:    52.253858  ~~~~~
write-thread: 66.955037 (+28.13%, +14.70)
read-thread:  66.397609 (+27.07%, +14.14)
mem-target:   77.492938 (+48.30%, +25.24)
 ## data-env-vars.name                     = mozilla-try-2024-03-26-zstd-sparse-revlog
  # benchmark.variants.read-from-memory    = no
no-thread:    130.584329  ~~~~~
write-thread: 138.770454   (+6.27%, +8.19)
read-thread:  145.137477 (+11.14%, +14.55)
mem-target:   164.366925 (+25.87%, +33.78)
parent 2787db338b15
No related branches found
No related tags found
2 merge requests!1292Draft: 7.0rc preparation,!1217add the ability to use multiple thread to apply stream bundles
Pipeline #99644 passed
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment