- Feb 22, 2025
-
-
Pierre-Yves David authored
Same rational as the parent changeset.
-
- Feb 18, 2025
-
-
Pierre-Yves David authored
-
Julien Cristau authored
stream bundle specs specify requirements, but we were ignoring this when selecting which bundle to apply, causing spurious clone failures.
-
- Jan 27, 2025
-
-
Pierre-Yves David authored
This mechanism limit the read-aheadto avoid uncontrolled memory usage. Note that the memory target is not a hard limit and that mercurial will use more memory that specified in some cases, but "not too much more". The current implementation focus on "simplicity" and could be more efficient. However this provide a "safe" baseline for the feature. Now that the overall shape of the feature is here we can start making it faster. Here is more benchmark of how everything slows down. The next series will focus on optimizing this code path to actually speeds things up. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ write-thread: 9.128669 (+26.02%, +1.88) read-thread: 9.387256 (+29.59%, +2.14) mem-target: 9.901032 (+36.68%, +2.66) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ write-thread: 0.275081 (+10.17%, +0.03) read-thread: 0.292601 (+17.18%, +0.04) mem-target: 0.305973 (+22.54%, +0.06) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ write-thread: 16.374306 (+24.65%, +3.24) read-thread: 17.008865 (+29.48%, +3.87) mem-target: 18.467590 (+40.58%, +5.33) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ write-thread: 6.783031 (+27.56%, +1.47) read-thread: 7.107141 (+33.65%, +1.79) mem-target: 7.338505 (+38.00%, +2.02) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ write-thread: 6.737864 (+24.81%, +1.34) read-thread: 7.163505 (+32.70%, +1.77) mem-target: 7.333354 (+35.84%, +1.93) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ write-thread: 3.758324 (+22.48%, +0.69) read-thread: 3.907693 (+27.34%, +0.84) mem-target: 4.238172 (+38.11%, +1.17) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ write-thread: 66.561340 (+28.16%, +14.63) read-thread: 66.902619 (+28.82%, +14.97) mem-target: 78.194540 (+50.56%, +26.26) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog # benchmark.variants.read-from-memory = yes no-thread: 52.253858 ~~~~~ write-thread: 66.955037 (+28.13%, +14.70) read-thread: 66.397609 (+27.07%, +14.14) mem-target: 77.492938 (+48.30%, +25.24) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ write-thread: 138.770454 (+6.27%, +8.19) read-thread: 145.137477 (+11.14%, +14.55) mem-target: 164.366925 (+25.87%, +33.78)
-
- Jan 21, 2025
-
-
Pierre-Yves David authored
The reader thread just want to iterate over chunk, so let it do just that. This avoid some silly layer of filechunkiter and chunk buffer. Before this change, we had the following layers: bundle2part (iterator of chunks) → chunkbuffer (iterator of chunks to file-like) → filechunkiter (file-like to chunkgs) → _DataQueue (thread safe iterator of chunks) → chunkbuffer (iterator of chunks to file-like). We now have removed the first layer of chunkbuffer + filechunkiter: bundle2part (iterator of chunks) → _DataQueue (thread safe iterator of chunks) → chunkbuffer (iterator of chunks to file-like). This fix the output order change that was previously introduced.
-
- Jan 20, 2025
-
-
Pierre-Yves David authored
We could do useful thing while waiting for data, so we introduce another thread dedicated to reading the bundle data and pass them to the thread parsing the bundle. This has a small impact on test output because the bundle part is consumed at a different time. However this will be smoothed out very soon so lets ignore that for now. As for the previous patch, the speed up in not there yet. We need to adjust various part of the implementation to see some benefit. However, this put the generic architecture in place so that we can focus on performance later. Below are some benchmark result to highlight the slowdown. ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.memory-target = default # benchmark.variants.num-writer = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog no-thread: 7.244015 ~~~~~ write-thread: 9.128669 (+26.02%, +1.88) read-thread: 9.387256 (+29.59%, +2.14) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog no-thread: 0.249693 ~~~~~ write-thread: 0.275081 (+10.17%, +0.03) read-thread: 0.292601 (+17.18%, +0.04) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog no-thread: 13.136674 ~~~~~ write-thread: 16.374306 (+24.65%, +3.24) read-thread: 17.008865 (+29.48%, +3.87) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog no-thread: 5.317709 ~~~~~ write-thread: 6.783031 (+27.56%, +1.47) read-thread: 7.107141 (+33.65%, +1.79) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog no-thread: 5.398368 ~~~~~ write-thread: 6.737864 (+24.81%, +1.34) read-thread: 7.163505 (+32.70%, +1.77) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog no-thread: 3.acbb55 ~~~~~ write-thread: 3.758324 (+22.48%, +0.69) read-thread: 3.907693 (+27.34%, +0.84) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-thread: 51.934795 ~~~~~ write-thread: 66.561340 (+28.16%, +14.63) read-thread: 66.902619 (+28.82%, +14.97) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-thread: 52.253858 ~~~~~ write-thread: 66.955037 (+28.13%, +14.70) read-thread: 66.397609 (+27.07%, +14.14) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no no-thread: 130.584329 ~~~~~ write-thread: 138.770454 (+6.27%, +8.19) read-thread: 145.137477 (+11.14%, +14.55)
-
Pierre-Yves David authored
This could provide a significant performance boost, but various implementation details means it is currently slower. More update will help make this boost real, but here is the basic idea. The implementation in this patch is unbounded in memory which could be a problem in some situation. We will deal with that soon. There is the benchmark result showing a slower run ### benchmark.name = hg.perf.exchange.stream.consume # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.parallel-processing = yes # benchmark.variants.progress = no # benchmark.variants.read-from-memory = yes # benchmark.variants.version = v2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 0.249693 ~~~~~ after: 0.275081 (+10.17%, +0.03) ## data-env-vars.name = netbsd-xsrc-all-2024-09-19-zstd-sparse-revlog before: 5.317709 ~~~~~ after: 6.783031 (+27.56%, +1.47) ## data-env-vars.name = netbsd-xsrc-draft-2024-09-19-zstd-sparse-revlog before: 5.398368 ~~~~~ after: 6.737864 (+24.81%, +1.34) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 3.acbb55 ~~~~~ after: 3.758324 (+22.48%, +0.69) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 7.244015 ~~~~~ after: 9.128669 (+26.02%, +1.88) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog before: 13.136674 ~~~~~ after: 16.374306 (+24.65%, +3.24) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog before: 52.253858 ~~~~~ after: 66.955037 (+28.13%, +14.70) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog before: 51.934795 ~~~~~ after: 66.561340 (+28.16%, +14.63) ## data-env-vars.name = mozilla-try-2024-03-26-zstd-sparse-revlog # benchmark.variants.read-from-memory = no before: 130.584329 ~~~~~ after: 138.770454 (+6.27%, +8.19)
-
- Nov 28, 2024
-
-
Pierre-Yves David authored
See inline documentation and next commit for details.
-
Pierre-Yves David authored
Strictly speaking we could also do that for v1, but that is old enough that I don't want to bother.
-
- Oct 01, 2024
-
-
Raphaël Gomès authored
This case is easy to detect and we have all we need to generate a valid working copy and dirstate entirely in Rust, which speeds things up considerably: On my machine updating a repo of ~300k files goes from 10.00s down to 4.2s, all while consuming 50% less system time, with all caches hot. Something to note is that further improvements will probably happen with the upcoming `InnerRevlog` series that does smarter mmap hanlding, especially for filelogs. Here are benchmark numbers on a machine with only 4 cores (and no SMT enabled) ``` ### data-env-vars.name = heptapod-public-2024-03-25-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 5.328762 ~~~~~ rust: 1.308654 (-75.44%, -4.02) ### data-env-vars.name = mercurial-devel-2024-03-22-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 1.693271 ~~~~~ rust: 1.151053 (-32.02%, -0.54) ### data-env-vars.name = mozilla-unified-2024-03-22-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 38.901613 ~~~~~ rust: 11.637880 (-70.08%, -27.26) ### data-env-vars.name = netbsd-xsrc-public-2024-09-19-ds2-pnm # benchmark.name = hg.command.update # bin-env-vars.hg.py-re2-module = default # bin-env-vars.hg.changeset.node = <this change> # benchmark.variants.atomic-update = no # benchmark.variants.scenario = null-to-tip # benchmark.variants.worker = default default: 4.793727 ~~~~~ rust: 1.505905 (-68.59%, -3.29) ```
-
- Sep 24, 2024
-
-
Pierre-Yves David authored
We want to ensure no older clients will truncate the file under us. So we need to change their name. We don't change the rest of the format (unfortunaly).
-
- Apr 09, 2024
-
-
Pierre-Yves David authored
In c4aab3661f25, we broken this feature by adding unicode instead of bytes to the dictionary. On the other hand, this feature was never tested, so augment the tests to tests this.
-
- Dec 11, 2023
-
-
Pierre-Yves David authored
The test suite mostly use small repositories, that implies that most changelog in the tests are inlined. As a result, non-inlined changelog are quite poorly tested. Since non-inline changelog are most common case for serious repositories, this lack of testing is a significant problem that results in high profile issue like the one recently fixed by 66417f55ea33 and 849745d7da89. Inlining the changelog does not bring much to the table, the number of total file saved is negligible, and the changelog will be read by most operation anyway. So this changeset is make it so we never inline the changelog, and de-inline the one that are still inlined whenever we touch them. By doing that, we remove the "dual code path" situation for writing new entry to the changelog and move to a "single code path" situation. Having a single code path simplify the code and make sure it is covered by test (if test cover that situation obviously) This impact all tests that care about the number of file and the exchange size, but there is nothing too complicated in them just a lot of churn. The churn is made "worse" by the fact rust will use the persistent nodemap on any changelog now. Which is overall a win as it means testing the persistent nodemap more and having less special cases. In short, having inline changelog is mostly useless and an endless source of pain. We get rid of it.
-
- May 23, 2023
-
-
Pierre-Yves David authored
Altering the format may introduce significant changes in output and in order. It is going to be simpler to deal with explicitly different blocks for the output.
-
- Jun 01, 2023
-
-
Arseniy Alekseyev authored
This new version is less rigid regarding the extract number of files and number of bytes to be actually transfered, it also lays the groundwork for other improvements. The format stays experimental, but this is an interesting base to build upon.
-
- Jun 19, 2023
-
-
Pierre-Yves David authored
The old wire command does not take argument so we need a new command. We make that new command take argument to prepare it for the future (like narrow support).
-
- May 22, 2023
-
-
Arseniy Alekseyev authored
We add support and test for this. The support is still experimental, so the various name and identifier will eventually need to be renamed when stream-v3 gets out of experimental.
-
- May 21, 2023
-
-
Pierre-Yves David authored
Creating the `changelog.i` file make the repository usable, so dealing with phase earlier seems better.
-
- Jan 27, 2022
-
-
Pierre-Yves David authored
Persistent nodemap affect the store format. However it is fairly isolated and fast to generate locally. So not making it a fixed part of the stream clone is useful. This allow clients without persistent-nodemap support (default for client without Rust enabled, or simply older client). So it make it possible to enable persistent nodemap on client, where it can provide a massive boost. without too much consequence. To do so, we stop using it in the advertisement requirements for streaming and let the client add/remove the necessary file depending of its configuration. We still send the files as it seems like a small save to not regenerate them. In addition, the way we match them will overlap with the changelog-v2/revlog-v2 so we can't simply skip the associated patterns. Differential Revision: https://phab.mercurial-scm.org/D12096
-
- Feb 02, 2022
-
-
Raphaël Gomès authored
-
- Feb 01, 2022
-
-
Pierre-Yves David authored
The rename of the old experimental name was overlooked before the 6.0 release. We rename everything to use the new name (and keep the released name as an alias for compatibility). Differential Revision: https://phab.mercurial-scm.org/D12129
-
- Jan 17, 2022
-
-
Pierre-Yves David authored
We were wrongly putting irrelevant requirements in the bundle and the receiving side was getting confused, treating them as being missing while still putting them in the `requires` file. Leading do corrupted repositories. This changes fix stream-clone behavior regarding format when bundle-2 is involved, so we now also test this cases. Behavior with older version of Mercurial will be fine as they filter the requirements they get from the bundle on their side anyway. Differential Revision: https://phab.mercurial-scm.org/D12084
-
Pierre-Yves David authored
These line refer to a "dirstate-v2" variant that is no longer in use. The associated line drifted out of sync with reality. Differential Revision: https://phab.mercurial-scm.org/D12026
-
- Oct 19, 2021
-
-
Pierre-Yves David authored
It seems the format as reached a good balance. With a core of new capabilities that motivated it initially and enough new feature and room for future improvement to be a clear progress we can set a milestone for. Having the format frozen will help the feature to get real life testing, outside of the test suite. The feature itself stay experimental but the config gains a new name to avoid people enable non-frozen version by default. If too many bugs are reported during the RC we might move the format back to experimental and drop its support in future version (in favor of a new one) Differential Revision: https://phab.mercurial-scm.org/D11709
-
- Aug 03, 2021
-
-
sliquister authored
This commit is exactly the result of running this command: sed -i -e 's! *\(-e \|--ssh \|--config ui.ssh=\)[ \"]*$PYTHON[ \"]*$\(RUN\|\)TESTDIR/dummyssh[\"]* *! !g' -e '/^[ >]*ssh *=[ "]*$PYTHON[ "]*$\(RUN\|\)TESTDIR\/dummyssh[ "]*$/d' -e 's/^\( [$] .*[^ ]\) *$/\1/' *.t *.sh Sometimes the tests can be simplified further, but I think it's preferable to do the simplification separately. Differential Revision: https://phab.mercurial-scm.org/D11245
-
- Aug 25, 2021
-
-
Pierre-Yves David authored
Touching `test-clone-stream.t` is very painful otherwise. Differential Revision: https://phab.mercurial-scm.org/D11337
-
- May 19, 2021
-
-
Simon Sapin authored
Fix most test failures (except in test-narrow-debugrebuilddirstate.t and test-upgrade-repo.t) caused by the new entry in config or in .hg/requires when running `run-tests.py --extra-config-opt format.exp-dirstate-v2=1` There is no CI so far for this configuration. Differential Revision: https://phab.mercurial-scm.org/D10718
-
- Jan 14, 2021
-
-
Pierre-Yves David authored
As discussed at the 5.6 sprint, we can make it enabled by default, but only for Rust installation. Differential Revision: https://phab.mercurial-scm.org/D9765
-
- Apr 06, 2021
-
-
Pierre-Yves David authored
As see in changeset bb271ec2fbfb, zstd is 20% to 50% faster for reading and writing. Use take advantage of the new config behavior to try zstd by default, falling back to zlib is zstd is not available on that plateform. Differential Revision: https://phab.mercurial-scm.org/D10326
-
- Jan 15, 2021
-
-
Pierre-Yves David authored
This work around any deviciency/limitation of the clone process. In our case this ensure the persistent nodemap exist with valid content. Ideally, the cloning process would also do "the right thing". However since older server will never be able to do "the right thing". The local workaround will be necessary anyway. I am not worried by the performance impact of this as `hg clone` is non-instant on large repositories where is could matters. Warming the cache if they are already correct is very fast. And if they are not already warm, this seems like a good time to do so. This impact various test as more cache are now warmed sooner, all the change should be harmless. Differential Revision: https://phab.mercurial-scm.org/D9789
-
- Sep 11, 2019
-
-
Martin von Zweigbergk authored
Differential Revision: https://phab.mercurial-scm.org/D6850
-
- Nov 12, 2018
-
-
Boris Feld authored
Now that sparse-revlog is enabled by default, we no longer needs it. Differential Revision: https://phab.mercurial-scm.org/D5346
-
Boris Feld authored
We are about to enable sparse-revlog globally. To help with reviewing the tests change, we isolate them in individual changesets. Differential Revision: https://phab.mercurial-scm.org/D5339
-
- Oct 01, 2018
-
-
Gregory Szorc authored
Now that keepalive.py records HTTP request count and the number of bytes sent and received as part of performing those requests, we can easily print a report on the activity when closing a peer instance! Exact byte counts are globbed in tests because they are influenced by non-deterministic things, such as hostnames and port numbers. Plus, the exact byte count isn't too important anyway. I feel obliged to note that printing the byte count could have security implications. e.g. if sending a password via HTTP basic auth, the length of that password will influence the byte count and the reporting of the byte count could be a side-channel leak of the password length. I /think/ this is beyond our threshold for concern. But if we think it poses a problem, we can teach the byte count logging code to e.g. ignore sensitive HTTP request headers. We could also consider not reporting the byte count of request headers altogether. But since the wire protocol uses HTTP headers for sending command arguments, it is kind of important to report their size. Differential Revision: https://phab.mercurial-scm.org/D4858
-
- Aug 25, 2018
-
-
Yuya Nishihara authored
It's silly to keep updating xrange(len(changelog), len(changelog) + 1) for each added revision. Instead, let's simply remember the first revision to be added. The test output slightly changed as the branch cache is also warmed up by stream clone, which seems more consistent. .. api:: ``tr.changes['revs']`` is replaced by ``tr.changes['origrepolen']`` which is the first revision number to be added.
-
- May 08, 2018
-
-
Boris Feld authored
Most parts are mandatory but when introducing new parts, they should be advisory if included by default or old clients won't be able to process it. Differential Revision: https://phab.mercurial-scm.org/D3480
-
- Apr 03, 2018
-
-
Gregory Szorc authored
There are a handful of test failures in the simple store with regards to stream clones. Fixing them will require a lot of conditional output. Disabling the tests wholesale is easier at this juncture. Differential Revision: https://phab.mercurial-scm.org/D3064
-
- Mar 31, 2018
-
-
Matt Harbison authored
-
- Jan 31, 2018
-
-
Boris Feld authored
When asked specifically, the code do a sanity check on the clone bundle to ensure it's a stream bundle. As we introduced a new stream bundle spec, update the logic to support it. With this final changeset, we can now announce safely a stream v2 clone bundle and old clients would not crash trying to process it. This changeset address the last comment from Gregory Szorc on the stream v2 bundle series. Differential Revision: https://phab.mercurial-scm.org/D1957
-
Boris Feld authored
Differential Revision: https://phab.mercurial-scm.org/D1956
-