stream-clone-v2: introduce a way to limit memory usage of the threaded version
This mechanism limit the read-aheadto avoid uncontrolled memory usage. Note that the memory target is not a hard limit and that mercurial will use more memory that specified in some cases, but "not too much more". The current implementation focus on "simplicity" and could be more efficient. However this provide a "safe" baseline for the feature. Now that the overall shape of the feature is here we can start making it faster. Here is more benchmark of how everything slows down. The next series will focus on optimizing this code path to actually speeds things up. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ write-thread: 9.128669 (+26.02%, +1.88) read-thread: 9.387256 (+29.59%, +2.14) mem-target: 9.901032 (+36.68%, +2.66) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ write-thread: 0.275081 (+10.17%, +0.03) read-thread: 0.292601 (+17.18%, +0.04) mem-target: 0.305973 (+22.54%, +0.06) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ write-thread: 16.374306 (+24.65%, +3.24) read-thread: 17.008865 (+29.48%, +3.87) mem-target: 18.467590 (+40.58%, +5.33) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ write-thread: 6.783031 (+27.56%, +1.47) read-thread: 7.107141 (+33.65%, +1.79) mem-target: 7.338505 (+38.00%, +2.02) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ write-thread: 6.737864 (+24.81%, +1.34) read-thread: 7.163505 (+32.70%, +1.77) mem-target: 7.333354 (+35.84%, +1.93) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ write-thread: 3.758324 (+22.48%, +0.69) read-thread: 3.907693 (+27.34%, +0.84) mem-target: 4.238172 (+38.11%, +1.17) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ write-thread: 66.561340 (+28.16%, +14.63) read-thread: 66.902619 (+28.82%, +14.97) mem-target: 78.194540 (+50.56%, +26.26) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # benchmark.variants.read-from-memory = yes no-thread: 52.253858 ~~~~~ write-thread: 66.955037 (+28.13%, +14.70) read-thread: 66.397609 (+27.07%, +14.14) mem-target: 77.492938 (+48.30%, +25.24) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ write-thread: 138.770454 (+6.27%, +8.19) read-thread: 145.137477 (+11.14%, +14.55) mem-target: 164.366925 (+25.87%, +33.78)
parent
2787db338b15
No related branches found
No related tags found
Pipeline #99644 passed
Stage: build
Stage: checks
Stage: tests
Stage: platform-compat
Stage: py-version-compat
Loading
Please register or sign in to comment