- Feb 17, 2025
-
-
Raphaël Gomès authored
This was missed in 0bd91b0a1a93. It still works because we haven't removed the `hg-cpython` bridge yet. I'm honestly not sure if we should continue advertising for fsmonitor at all with the Rust extensions, but that's another discussion.
-
- Feb 18, 2025
-
-
Pierre Augier authored
The Makefile is now usable by Mercurial developers and for downstream packaging. `make help` explains how. I had to introduce more options (PIP_OPTIONS_INSTALL, PIP_PREFIX) to be able to change the behavior of the install targets (usable for local installation and downstream packaging) and to deal with an issue on Linux with the --prefix pip option (see https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240).
-
Pierre Augier authored
It is useless now and its presence could be misunderstood. People using it will get an error, run `make help` and adapt their code.
-
Pierre Augier authored
-
Pierre Augier authored
-
Pierre Augier authored
-
- Feb 04, 2025
-
-
Pierre Augier authored
-
- Feb 17, 2025
-
-
Pierre-Yves David authored
This is in line with the current project goal. (do we need a `dist-notest` target at all?)
-
Pierre-Yves David authored
This is in line with the current project goal. (do we need a `wheel` target at all?)
-
Pierre Augier authored
-
Pierre Augier authored
-
- Jan 30, 2025
-
-
Pierre Augier authored
-
- Jan 29, 2025
-
-
Pierre-Yves David authored
Sadly this is still significantly faster. I am not sure what the python file object does internally, but bypassing it provide a good speedup. Probably because we spend less time with the GIL. This is the end of this first round of optimization. Overall, the threaded version is almost twice faster as the baseline on our benchmark machine (and three time faster than the naive implementation) (see first batch of benchmark below). In reality, various improvement we did also help the non-threaded case. Our benchmark show about 30% improvement after this series. (second benchmark batch below) However, the threaded version remains faster in our benchmark (third benchmark). Further improvements can probably be achieved by speeding up the directories creation process and but moving to a compiled language (both faster in general and without a GIL). ########### Comparing evolution of the threaded code ########################## ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog baseline: 0.249693 ~~~~~ naive-thread: 0.305973 (+22.54%, +0.06) prev-change: 0.154647 (-38.07%, -0.10) this-change: 0.143486 (-42.54%, -0.11) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog baseline: 13.136674 ~~~~~ naive-thread: 18.467590 (+40.58%, +5.33) prev-change: 7.843554 (-40.29%, -5.29) this-change: 7.423071 (-43.49%, -5.71) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog baseline: 5.317709 ~~~~~ naive-thread: 7.338505 (+38.00%, +2.02) prev-change: 3.040664 (-42.82%, -2.28) this-change: 2.776906 (-47.78%, -2.54) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog baseline: 5.398368 ~~~~~ naive-thread: 7.333354 (+35.84%, +1.93) prev-change: 3.070976 (-43.11%, -2.33) this-change: 2.795593 (-48.21%, -2.60) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog baseline: 3.acbb55 ~~~~~ naive-thread: 4.238172 (+38.11%, +1.17) prev-change: 1.832118 (-40.30%, -1.24) this-change: 1.640016 (-46.56%, -1.43) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog baseline: 7.244015 ~~~~~ naive-thread: 9.901032 (+36.68%, +2.66) prev-change: 4.478754 (-38.17%, -2.77) this-change: 3.946716 (-45.52%, -3.30) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog baseline: 51.934795 ~~~~~ naive-thread: 78.194540 (+50.56%, +26.26) prev-change: 30.349379 (-41.56%, -21.59) this-change: 27.228980 (-47.57%, -24.71) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog baseline: 52.253858 ~~~~~ naive-thread: 77.492938 (+48.30%, +25.24) prev-change: 31.179906 (-40.33%, -21.07) this-change: 28.110867 (-46.20%, -24.14) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no baseline: 130.584329 ~~~~~ naive-thread: 164.366925 (+25.87%, +33.78) prev-change: 91.454508 (-29.97%, -39.13) this-change: 91.002420 (-30.31%, -39.58) ########### Comparing evolution of the non-threaded code ###################### ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = no # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog base-line: 0.247757 ~~~~~ this-change: 0.168695 (-31.91%, -0.08) ### data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog base-line: 13.195582 ~~~~~ this-change: 9.240878 (-29.97%, -3.95) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog base-line: 5.369459 ~~~~~ this-change: 3.637088 (-32.26%, -1.73) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog base-line: 5.345086 ~~~~~ this-change: 3.587604 (-32.88%, -1.76) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog base-line: 3.110852 ~~~~~ this-change: 2.069172 (-33.49%, -1.04) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog base-line: 7.293048 ~~~~~ this-change: 4.931636 (-32.38%, -2.36) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog base-line: 52.067856 ~~~~~ this-change: 35.360483 (-32.09%, -16.71) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog base-line: 52.347182 ~~~~~ this-change: 35.465007 (-32.25%, -16.88) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no base-line: 143.083922 ~~~~~ this-change: 107.609183 (-24.79%, -35.47) ########### Comparing non-threaded vs threaded ################################ ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = fec3492130e0 # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog not-threaded: 0.168695 ~~~~~ threaded: 0.143486 (-14.94%, -0.03) ### data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog not-threaded: 9.240878 ~~~~~ threaded: 7.423071 (-19.67%, -1.82) ### data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog not-threaded: 3.637088 ~~~~~ threaded: 2.776906 (-23.65%, -0.86) ### data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog not-threaded: 3.587604 ~~~~~ threaded: 2.795593 (-22.08%, -0.79) ### data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog not-threaded: 2.069172 ~~~~~ threaded: 1.640016 (-20.74%, -0.43) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog not-threaded: 4.931636 ~~~~~ threaded: 3.946716 (-19.97%, -0.98) ### data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog not-threaded: 35.360483 ~~~~~ threaded: 27.228980 (-23.00%, -8.13) ### data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog not-threaded: 35.465007 ~~~~~ threaded: 28.110867 (-20.74%, -7.35) ### data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no not-threaded: 107.609183 ~~~~~ threaded: 91.002420 (-15.43%, -16.61)
-
Pierre-Yves David authored
Now that all preparation is done in the main thread, we no longer need the vfs to be involved at the writing end. This very significantly speeds things up. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog prev-change: 0.217293 ~~~~~ this-change: 0.154647 (-28.83%, -0.06) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog prev-change: 11.222771 ~~~~~ this-change: 7.843554 (-30.11%, -3.38) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog prev-change: 4.465113 ~~~~~ this-change: 3.040664 (-31.90%, -1.42) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog prev-change: 4.667360 ~~~~~ this-change: 3.070976 (-34.20%, -1.60) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog prev-change: 2.559670 ~~~~~ this-change: 1.832118 (-28.42%, -0.73) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog prev-change: 6.123469 ~~~~~ this-change: 4.478754 (-26.86%, -1.64) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog prev-change: 44.781498 ~~~~~ this-change: 30.349379 (-32.23%, -14.43) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog prev-change: 44.396959 ~~~~~ this-change: 31.179906 (-29.77%, -13.22) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no prev-change: 108.552706 ~~~~~ this-change: 91.454508 (-15.75%, -17.10)
-
Pierre-Yves David authored
The vfs will do some operation to prepare the creation of the streamed file. It creates the necessary directories and it adds the file to the fncache. The directory creation is much more expensive than it should, and the same directory are being checked over and over. And the other part, like "adding the file to the fncache" increase the amount of GIL holding time. So we add a function to the vfs to do such preparation, and we centralise it in the "parsing" thread. This is done so that each directory are only "created" once. This does not yield speed up yet, because the writer thread still go through the same slow logic. This will be dealt with in the next changesets.
-
- Jan 20, 2025
-
-
Pierre-Yves David authored
Not going through python's buffer while writing chunk is not only faster, but it reduce the amount of operation that are needed under the GIL, increasing the benefit from the threads. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog prev-change: 0.233780 ~~~~~ this-change: 0.217293 (-7.05%, -0.02) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog prev-change: 11.754377 ~~~~~ this-change: 11.222771 (-4.52%, -0.53) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog prev-change: 4.735520 ~~~~~ this-change: 4.465113 (-5.71%, -0.27) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog prev-change: 4.728870 ~~~~~ this-change: 4.667360 (-1.30%, -0.06) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog prev-change: 2.783987 ~~~~~ this-change: 2.559670 (-8.06%, -0.22) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog prev-change: 6.532561 ~~~~~ this-change: 6.123469 (-6.26%, -0.41) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog prev-change: 47.154728 ~~~~~ this-change: 44.781498 (-5.03%, -2.37) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog prev-change: 47.166569 ~~~~~ this-change: 44.396959 (-5.87%, -2.77) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no prev-change: 113.696529 ~~~~~ this-change: 108.552706 (-4.52%, -5.14)
-
- Nov 30, 2024
-
-
Pierre-Yves David authored
This option let you control python buffering behavior on open file. We will use it to speed up the writing of data on disk during stream clone.
-
- Nov 29, 2024
-
-
Pierre-Yves David authored
As often with code that creates a lot of temporary small container (e.g. tuple), disable the gc yield a visible speedup. This this code does not create reference cycle this should not result in any increased memory usage. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog prev-change: 0.234764 ~~~~~ this-change: 0.233780 ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog prev-change: 12.037305 ~~~~~ this-change: 11.754377 (-2.35%, -0.28) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog prev-change: 4.763710 ~~~~~ this-change: 4.735520 ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog prev-change: 4.816293 ~~~~~ this-change: 4.728870 (-1.82%, -0.09) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog prev-change: 2.801525 ~~~~~ this-change: 2.783987 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog prev-change: 6.611609 ~~~~~ this-change: 6.532561 (-1.20%, -0.08) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog prev-change: 47.733498 ~~~~~ this-change: 47.154728 (-1.21%, -0.58) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog prev-change: 48.270778 ~~~~~ this-change: 47.166569 (-2.29%, -1.10) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no prev-change: 115.240169 ~~~~~ this-change: 113.696529 (-1.34%, -1.54)
-
- Feb 05, 2025
-
-
Pierre-Yves David authored
We don't need to a use a thread safe queue for file that come in one block. In such case we can significantly simplify the code to avoid using a Queue and locks for each small files. This yield a very significant speedup. The threaded version is finally faster than our baseline. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog baseline: 0.249693 ~~~~~ prev-change: 0.259730 (+4.02%, +0.01) this-change: 0.234764 (-5.98%, -0.01) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog baseline: 13.136674 ~~~~~ prev-change: 13.768783 (+4.81%, +0.63) this-change: 12.037305 (-8.37%, -1.10) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog baseline: 5.317709 ~~~~~ prev-change: 5.679052 (+6.80%, +0.36) this-change: 4.763710 (-10.42%, -0.55) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog baseline: 5.398368 ~~~~~ prev-change: 5.616571 (+4.04%, +0.22) this-change: 4.816293 (-10.78%, -0.58) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog baseline: 3.acbb55 ~~~~~ prev-change: 3.164637 (+3.13%, +0.10) this-change: 2.801525 (-8.70%, -0.27) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog baseline: 7.244015 ~~~~~ prev-change: 7.729637 (+6.70%, +0.49) this-change: 6.611609 (-8.73%, -0.63) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog baseline: 51.934795 ~~~~~ prev-change: 56.567493 (+8.92%, +4.63) this-change: 47.733498 (-8.09%, -4.20) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog baseline: 52.253858 ~~~~~ prev-change: 56.093516 (+7.35%, +3.84) this-change: 48.270778 (-7.62%, -3.98) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no baseline: 130.584329 ~~~~~ prev-change: 125.985212 (-3.52%, -4.60) this-change: 115.240169 (-11.75%, -15.34)
-
Pierre-Yves David authored
This will help us to decide when to use it or not in the next changeset.
-
- Feb 03, 2025
-
-
Pierre-Yves David authored
Letting the thread waiting for something it can use seems more reasonable.
-
- Feb 05, 2025
-
-
Pierre-Yves David authored
The `threading.Condition` object from the stdlib suffer for excessive creation/destruction of the underlying locks. This destroy its performance to a quite noticeable point. Recreating the same logic with persistent lock yields an absurdly big performance gain. In practice, most of the slowdown introduced by threading is recovered. This benchmark compare the non-threaded baseline, the naive threaded implementation with memory limitation, the previous changeset and this one. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ naive-thread: 0.305973 (+22.54%, +0.06) prev-change: 0.305442 (+22.33%, +0.06) this-change: 0.258450 (+3.51%, +0.01) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ naive-thread: 18.467590 (+40.58%, +5.33) prev-change: 17.788205 (+35.41%, +4.65) this-change: 13.715187 (+4.40%, +0.58) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ naive-thread: 7.338505 (+38.00%, +2.02) prev-change: 7.122798 (+33.94%, +1.81) this-change: 5.687069 (+6.95%, +0.37) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ naive-thread: 7.333354 (+35.84%, +1.93) prev-change: 7.123533 (+31.96%, +1.73) this-change: 5.749082 (+6.50%, +0.35) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ naive-thread: 4.238172 (+38.11%, +1.17) prev-change: 4.095587 (+33.47%, +1.03) this-change: 3.179653 (+3.62%, +0.11) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ naive-thread: 9.901032 (+36.68%, +2.66) prev-change: 9.728038 (+34.29%, +2.48) this-change: 7.638277 (+5.44%, +0.39) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ naive-thread: 78.194540 (+50.56%, +26.26) prev-change: 72.574373 (+39.74%, +20.64) this-change: 56.267824 (+8.34%, +4.33) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-thread: 52.253858 ~~~~~ naive-thread: 77.492938 (+48.30%, +25.24) prev-change: 72.845963 (+39.41%, +20.59) this-change: 55.368316 (+5.96%, +3.11) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ naive-thread: 164.366925 (+25.87%, +33.78) prev-change: 154.873570 (+18.60%, +24.29) this-change: 132.137611 (+1.19%, +1.55)
-
- Feb 03, 2025
-
-
Pierre-Yves David authored
The `threading.Condition` object from the stdlib suffer for excessive creation/destruction of the underlying locks. This destroy its performance to a quite noticeable point. Recreating the same logic with persistent lock yield an absurd performance gain (or recovery from the slow down). Note that this problem also affect the stdlib `queue.Queue` object. This will be dealt with in the next changesets. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ threaded: 0.292601 (+17.18%, +0.04) mem-target: 0.305973 (+22.54%, +0.06) this-change: 0.305442 (+22.33%, +0.06) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ threaded: 9.387256 (+29.59%, +2.14) mem-target: 9.901032 (+36.68%, +2.66) this-change: 9.728038 (+34.29%, +2.48) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ threaded: 17.008865 (+29.48%, +3.87) mem-target: 18.467590 (+40.58%, +5.33) this-change: 17.788205 (+35.41%, +4.65) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ threaded: 7.107141 (+33.65%, +1.79) mem-target: 7.338505 (+38.00%, +2.02) this-change: 7.122798 (+33.94%, +1.81) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ threaded: 7.163505 (+32.70%, +1.77) mem-target: 7.333354 (+35.84%, +1.93) this-change: 7.123533 (+31.96%, +1.73) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ threaded: 3.907693 (+27.34%, +0.84) mem-target: 4.238172 (+38.11%, +1.17) this-change: 4.095587 (+33.47%, +1.03) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ threaded: 66.902619 (+28.82%, +14.97) mem-target: 78.194540 (+50.56%, +26.26) this-change: 72.574373 (+39.74%, +20.64) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-thread: 52.253858 ~~~~~ threaded: 66.397609 (+27.07%, +14.14) mem-target: 77.492938 (+48.30%, +25.24) this-change: 72.845963 (+39.41%, +20.59) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ threaded: 145.137477 (+11.14%, +14.55) mem-target: 164.366925 (+25.87%, +33.78) this-change: 154.873570 (+18.60%, +24.29)
-
- Jan 27, 2025
-
-
Pierre-Yves David authored
This mechanism limit the read-aheadto avoid uncontrolled memory usage. Note that the memory target is not a hard limit and that mercurial will use more memory that specified in some cases, but "not too much more". The current implementation focus on "simplicity" and could be more efficient. However this provide a "safe" baseline for the feature. Now that the overall shape of the feature is here we can start making it faster. Here is more benchmark of how everything slows down. The next series will focus on optimizing this code path to actually speeds things up. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ write-thread: 9.128669 (+26.02%, +1.88) read-thread: 9.387256 (+29.59%, +2.14) mem-target: 9.901032 (+36.68%, +2.66) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ write-thread: 0.275081 (+10.17%, +0.03) read-thread: 0.292601 (+17.18%, +0.04) mem-target: 0.305973 (+22.54%, +0.06) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ write-thread: 16.374306 (+24.65%, +3.24) read-thread: 17.008865 (+29.48%, +3.87) mem-target: 18.467590 (+40.58%, +5.33) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ write-thread: 6.783031 (+27.56%, +1.47) read-thread: 7.107141 (+33.65%, +1.79) mem-target: 7.338505 (+38.00%, +2.02) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ write-thread: 6.737864 (+24.81%, +1.34) read-thread: 7.163505 (+32.70%, +1.77) mem-target: 7.333354 (+35.84%, +1.93) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ write-thread: 3.758324 (+22.48%, +0.69) read-thread: 3.907693 (+27.34%, +0.84) mem-target: 4.238172 (+38.11%, +1.17) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ write-thread: 66.561340 (+28.16%, +14.63) read-thread: 66.902619 (+28.82%, +14.97) mem-target: 78.194540 (+50.56%, +26.26) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # benchmark.variants.read-from-memory = yes no-thread: 52.253858 ~~~~~ write-thread: 66.955037 (+28.13%, +14.70) read-thread: 66.397609 (+27.07%, +14.14) mem-target: 77.492938 (+48.30%, +25.24) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ write-thread: 138.770454 (+6.27%, +8.19) read-thread: 145.137477 (+11.14%, +14.55) mem-target: 164.366925 (+25.87%, +33.78)
-
- Jan 29, 2025
-
-
Pierre-Yves David authored
This make it usable and will make pytype happy.
-
- Jan 27, 2025
-
-
Pierre-Yves David authored
That method indicate the amount of data read so far. It will be used to limit the memory usage of the threaded stream-clone application.
-
- Jan 21, 2025
-
-
Pierre-Yves David authored
The reader thread just want to iterate over chunk, so let it do just that. This avoid some silly layer of filechunkiter and chunk buffer. Before this change, we had the following layers: bundle2part (iterator of chunks) → chunkbuffer (iterator of chunks to file-like) → filechunkiter (file-like to chunkgs) → _DataQueue (thread safe iterator of chunks) → chunkbuffer (iterator of chunks to file-like). We now have removed the first layer of chunkbuffer + filechunkiter: bundle2part (iterator of chunks) → _DataQueue (thread safe iterator of chunks) → chunkbuffer (iterator of chunks to file-like). This fix the output order change that was previously introduced.
-
- Jan 27, 2025
-
-
Pierre-Yves David authored
Since the bundle2 already come in fixed size chunk, there is few need to go have the chunkbuffer slice and recompose them, we can just yield the chunk that the format provides.
-
Pierre-Yves David authored
If the consumer just want to iterate over chunk of arbitrary size, the chunk buffer is just an overhead. So we provide a way to bypass it in such cases. It is put to use in the next changesets.
-
- Jan 20, 2025
-
-
Pierre-Yves David authored
We could do useful thing while waiting for data, so we introduce another thread dedicated to reading the bundle data and pass them to the thread parsing the bundle. This has a small impact on test output because the bundle part is consumed at a different time. However this will be smoothed out very soon so lets ignore that for now. As for the previous patch, the speed up in not there yet. We need to adjust various part of the implementation to see some benefit. However, this put the generic architecture in place so that we can focus on performance later. Below are some benchmark result to highlight the slowdown. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ write-thread: 9.128669 (+26.02%, +1.88) read-thread: 9.387256 (+29.59%, +2.14) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ write-thread: 0.275081 (+10.17%, +0.03) read-thread: 0.292601 (+17.18%, +0.04) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ write-thread: 16.374306 (+24.65%, +3.24) read-thread: 17.008865 (+29.48%, +3.87) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ write-thread: 6.783031 (+27.56%, +1.47) read-thread: 7.107141 (+33.65%, +1.79) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ write-thread: 6.737864 (+24.81%, +1.34) read-thread: 7.163505 (+32.70%, +1.77) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ write-thread: 3.758324 (+22.48%, +0.69) read-thread: 3.907693 (+27.34%, +0.84) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ write-thread: 66.561340 (+28.16%, +14.63) read-thread: 66.902619 (+28.82%, +14.97) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-thread: 52.253858 ~~~~~ write-thread: 66.955037 (+28.13%, +14.70) read-thread: 66.397609 (+27.07%, +14.14) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ write-thread: 138.770454 (+6.27%, +8.19) read-thread: 145.137477 (+11.14%, +14.55)
-
- Jan 27, 2025
-
-
Pierre-Yves David authored
The current value are a bit arbitrary. I don't believe they will be the final value, but this changesets make sure some value exists, and we can adjust the actual value before graduating the feature from experimental.
-
- Jan 20, 2025
-
-
Pierre-Yves David authored
This could provide a significant performance boost, but various implementation details means it is currently slower. More update will help make this boost real, but here is the basic idea. The implementation in this patch is unbounded in memory which could be a problem in some situation. We will deal with that soon. There is the benchmark result showing a slower run ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 0.249693 ~~~~~ after: 0.275081 (+10.17%, +0.03) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog before: 5.317709 ~~~~~ after: 6.783031 (+27.56%, +1.47) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog before: 5.398368 ~~~~~ after: 6.737864 (+24.81%, +1.34) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 3.acbb55 ~~~~~ after: 3.758324 (+22.48%, +0.69) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 7.244015 ~~~~~ after: 9.128669 (+26.02%, +1.88) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog before: 13.136674 ~~~~~ after: 16.374306 (+24.65%, +3.24) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog before: 52.253858 ~~~~~ after: 66.955037 (+28.13%, +14.70) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog before: 51.934795 ~~~~~ after: 66.561340 (+28.16%, +14.63) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no before: 130.584329 ~~~~~ after: 138.770454 (+6.27%, +8.19)
-
Pierre-Yves David authored
We did this in two step: extracting the code to a function, then turning the function into a class for clarity. The introduction of the class adds a significant amount of boiler plate and complexity and make the previous move hard to follow.
-
Pierre-Yves David authored
Same rational as for extracting the parsing code.
-
Pierre-Yves David authored
This will be useful to spread the work across different threads. However, before being able to introduce threading, we need to finish refactoring the code.
-
- Feb 17, 2025
-
-
Matt Harbison authored
-
Matt Harbison authored
This makes it easier to notice when the declaration is missing. (And yes, some are missing.)
-
Matt Harbison authored
I'm pretty sure these were missing- when I created a file with the corresponding name, it ground through and generated some files, and then said there was nothing to do. Without the file, it fails after generating the files, saying I need `make` 4.0 or later.
-
Matt Harbison authored
This makes it easier to notice when the declaration is missing.
-
Matt Harbison authored
This is more conventional.
-