- Jan 06, 2025
-
-
Matt Harbison authored
These were different classes in py2, but now a handful of error classes are just an alias of `OSError`, like `IOError`, `EnvironmentError`, `WindowsError`, etc. This is the result of running a hacked version of `pyupgrade` 3.19.1[1] $ hg files -0 'relglob:**.py' | xargs -0 \ pyupgrade --py38-plus --keep-percent-format --keep-mock --keep-runtime-typing The hack is because it doesn't have command line switches to disable most changes, so it makes tons of unrelated changes all at once. The hack is to 1) patch `pyupgrade._main._fix_tokens()` to immediately return its content arg 2) change `pyupgrade._data.register_decorator()` to only register the function if it's from the fixer we're interested in: if func.__module__ in ( "pyupgrade._plugins.exceptions", ): FUNCS[tp].append(func) return func [1] https://github.com/asottile/pyupgrade
-
Matt Harbison authored
Strings are unicode on Python 3. These were rewritten by `pyupgrade`. It's arguably better to fix the `contrib` stuff upstream and then re-vendor it, but I don't feel like waiting for that, and then all of the regression testing involved to get a minor improvement in the codebase. It was last vendored 5 years ago, and will likely be a large change anyway to drop py2 support. Also, we've already made minor formatting changes to it locally.
-
Matt Harbison authored
These were rewritten by `pyupgrade`.
-
Matt Harbison authored
The regex is complex enough, so I'm not bothering with figuring out how to detect and fail the older style of wrapping the constant string in `()`.
-
Matt Harbison authored
These aliases were introduced back in 5209fc94b982, because `black` was going to strip away the extra parentheses, but they're needed to subvert `test-check-code.t`. That obviously changed at some point, but `pyupgrade`[1] also strips these out. While that tool is very useful in adapting code to modern standards, it lacks the ability to turn off most conversions, so constantly reverting these is a pain. Even without that, the code is more understandable with an explicit declaration. It also would have been an easy typo to miss the leading `_` in the i18n method `_()` that the checker is looking for, and fail to detect the problem. The `contrib/perf.py` code just uses a local alias to the original methods because (IIUC), this tries to be compatible with old versions of hg. But practically, these noi18n aliases were added before useful py3 support, and at some point, it won't be feasible to do py2 benchmarking anymore, and maybe this module can be cleaned up some. [1] https://github.com/asottile/pyupgrade
-
- Jan 07, 2025
-
-
Matt Harbison authored
This covers `test-contrib-check-code.t` and `test-contrib-check-commit.t`. I overlooked these recently when hacking on `contrib/check-code.py`, and ran `test-check-*` locally. I was initially inclined to rename these `test-check-contrib-*.t`, but we already have `test-check-code.t` and `test-check-commit.t`, so this would break tab completion. Since these test rarely modified tools, let's leave them as-is. This change simply tests them once in CI, instead of running them for each flavor of py + modulepolicy that we test.
-
Matt Harbison authored
Also, `ValueError` wants str anyway.
-
- Jan 04, 2025
-
-
Jeff Sipek authored
This change makes `hg heads` work. Prior to this change, the error was: $ hg heads ** unknown exception encountered, please report by visiting ** https://mercurial-scm.org/wiki/BugTracker ** Python 3.11.10 (main, Oct 31 2024, 01:10:40) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15 ** Mercurial Distributed SCM (version 6.9.post1.dev337+hg.72af9fa34832) ** Extensions loaded: absorb, churn, git (pygit2 1.14.1), githelp, gpg, hgk, histedit, patchbomb, rebase Traceback (most recent call last): File "/home/jeffpc/src/oss/hg-gitext-test/../hg-gitext/hg", line 61, in <module> dispatch.run() ... File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/commands.py", line 3462, in heads heads += repo.branchheads(branch, start, opts.get('closed')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 3469, in branchheads if not branches.hasbranch(branch): ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/branchmap.py", line 701, in hasbranch self._verifybranch(label) File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/branchmap.py", line 670, in _verifybranch _unknownnode(n) File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/branchmap.py", line 186, in _unknownnode raise ValueError('node %s does not exist' % node.hex()) ValueError: node f2f80ac809875855ac843f9e5e7480604b5cbff5 does not exist
-
Jeff Sipek authored
For whatever reason, LIKE is way slower than GLOB. The slowdown is a function of the number of commits in the repository. The following data has been collected on a repository with approximately 1.2M commits. The numbers are in milliseconds and they represent the latency of the changelog.shortest function as measured by time.time(). The shortest function was invoked via `hg log -l50 -T '{node|shortest}\n'`. Min. 1st Qu. Median Mean 3rd Qu. Max. 279.0 284.3 419.8 381.7 421.3 426.2 (LIKE) 0.08535 0.10437 0.12231 0.11919 0.12779 0.21291 (GLOB)
-
- Jan 09, 2025
-
-
Jeff Sipek authored
$ hg log -l1 ... Traceback (most recent call last): File "/home/jeffpc/src/oss/hg-gitext-test/../hg-gitext/hg", line 61, in <module> dispatch.run() ... File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/dispatch.py", line 1253, in _dispatch repo.close() File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 1607, in close self._writecaches() File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 1611, in _writecaches self._revbranchcache.write() File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/branching/rev_cache.py", line 362, in write wlock.release() File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/lock.py", line 392, in release self.releasefn() File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 3162, in unlock if self.dirstate.is_changing_any: ^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 225, in __get__ return super(unfilteredpropertycache, self).__get__(unfi) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/util.py", line 1822, in __get__ result = self.func(obj) ^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 1804, in dirstate self._dirstate.refresh() ^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'gitdirstate' object has no attribute 'refresh'
-
- Jan 04, 2025
-
-
Jeff Sipek authored
It wasn't clear what was happening to the refs and commits. Adding 'indexing' to the message makes it immediately obvious.
-
Jeff Sipek authored
This includes 3 one-line fixes: 1. use the "no node" message instead of "no such node" for consistency 2. drop superfluous (and wrong & unused) %d format string token 3. correct one message to "no rev" since it indicates a rev that wasn't found
-
- Oct 04, 2024
-
-
Jeff Sipek authored
Instead of iterating over the whole changelog table every time we want to know how many commits there are, we can cache the number between mercurial invocations. Unsurprisingly, this speeds up certain operations on repos with large histories. The following measurements are all in seconds and they represent the runtime of `hg log -T ' ' -l1 > /dev/null`. In other words, this includes python startup overhead, etc. On small and medium repos, there is no observable difference in runtime (because of the relatively large overhead of python runtime startup, and the rest of mercurial doing useful work), but on large repos the user-visible execution time drops by a factor of 10x or more. small repo (~600 commits): Min. 1st Qu. Median Mean 3rd Qu. Max. 0.1052 0.1076 0.1096 0.1102 0.1110 0.1210 (before) 0.1049 0.1087 0.1106 0.1120 0.1127 0.1302 (after) medium repo (12k commits): Min. 1st Qu. Median Mean 3rd Qu. Max. 0.1063 0.1095 0.1116 0.1129 0.1153 0.1349 (before) 0.1044 0.1092 0.1108 0.1115 0.1130 0.1326 (after) large repo (1.4M commits): Min. 1st Qu. Median Mean 3rd Qu. Max. 1.973 2.105 2.256 2.243 2.406 2.443 (before) 0.144 0.147 0.148 0.150 0.151 0.176 (after)
-
Jeff Sipek authored
Instead of indexing the changed files for every commit immediately, we can index... 1. heads' changed files immediately 2. other commits' changed files on-demand This helps a lot on repositories with large histories since the initial mercurial invocation doesn't have to wait for the complete repo history to be indexed.
-
Jeff Sipek authored
Since git and mercurial commit hashes are a function of their contents, we can skip indexing the changed files of a commit if we have already indexed it as it will never change. To accomplish this, we can add a bool to the changelog table to track whether or not we have indexed the files of each commit.
-
Jeff Sipek authored
-
- Mar 10, 2024
-
-
Jeff Sipek authored
Octopus merges in git are merge commits with more than 2 parents. To make them fit into mercurial core's assumption about commits having 0-2 parents, the git indexing code creates "sythetic" commits to represent the octopus commit as a sequence of regular 2-parent commits. The synthetic commit hashes are just an incrementing commit number (which is the same as the generated rev number). The last commit in the sequence of commits uses the actual git commit hash. As a result, `hg checkout -r <commit>` produces the same working directory as `git checkout <commit>` for all git commit hashes. The synthetic commit hashes are stored in the changelog table as any other commit - with the two parents - but they also contain the commit hash of the octopus merge commit. For example, given the git DAG (manually pruned `git log --graph`): *-. commit 23480d86e2689703b33f693907c40fbe6e1620e4 Merge branches... |\ \ | | | | | * commit 2eda9984b06c75448598ec6c0a9028e49dacf616 C | | | | * | commit 5e634a12f12fedaf7b8ef0f0fcdbb07222871953 B | |/ | | * | commit 8883a1296c5ae323a1b18d1f6410398ce43ebd3a D |/ | * commit 95f241588fded9554cae91be0fefd576f61ebfc6 A Where M is the octopus merge commit with 3 parents, the corresponding mercurial DAG is: $ hg log -G -T '{node} {desc}' @ 23480d86e2689703b33f693907c40fbe6e1620e4 Merge branches 'abc' and 'def' |\ | o 0000000000000000000000000000000000000004 Merge branches 'abc' and 'def' | |\ | | o 8883a1296c5ae323a1b18d1f6410398ce43ebd3a D | | | o---+ 2eda9984b06c75448598ec6c0a9028e49dacf616 C / / o / 5e634a12f12fedaf7b8ef0f0fcdbb07222871953 B |/ o 95f241588fded9554cae91be0fefd576f61ebfc6 A
-
- Oct 03, 2024
-
-
Jeff Sipek authored
This reduces the length of the ever growing _index_repo function.
-
- Oct 02, 2024
-
-
Jeff Sipek authored
Benchmarking of 50 iterations of indexing (see below) shows that there is essentially no difference for small repos (<1k commits), similarly medium repos (~12k commits) see some benefit but other overheads completely overwhelm it, but for large repos (~122k commits) the 80-100x speedup is clearly visible to the user. All of the numbers are in seconds and were measured with time.time() calls placed in _index_repo(). The times exclude the time taken by changedfiles processing. Small repo (guilt, 553 commits, 1 head): Min. 1st Qu. Median Mean 3rd Qu. Max. 0.0008781 0.0009274 0.0009800 0.0012285 0.0014637 0.0024107 (before) 0.0003092 0.0003281 0.0003519 0.0003777 0.0003927 0.0006843 (after) Medium repo (hamlib, 12k commits, 53 heads): Min. 1st Qu. Median Mean 3rd Qu. Max. 0.04881 0.05135 0.07632 0.06672 0.08042 0.09415 (before) 0.004249 0.004420 0.004799 0.004809 0.005051 0.006416 (after) Large repo (qemu, 122k commits, 50 heads): Min. 1st Qu. Median Mean 3rd Qu. Max. 4.274 4.595 4.832 6.578 8.397 9.721 (before) 0.05180 0.05643 0.05865 0.06130 0.06712 0.06872 (after)
-
Jeff Sipek authored
-
- Jan 02, 2025
-
-
Jeff Sipek authored
-
Jeff Sipek authored
$ hg log ** unknown exception encountered, please report by visiting ** https://mercurial-scm.org/wiki/BugTracker ** Python 3.11.10 (main, Oct 31 2024, 01:10:40) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15 ** Mercurial Distributed SCM (version 6.9.post1.dev337+hg.72af9fa34832) ** Extensions loaded: absorb, churn, git (pygit2 1.14.1), githelp, gpg, hgk, histedit, patchbomb, rebase Traceback (most recent call last): File "/home/jeffpc/src/oss/hg-gitext-test/../hg-gitext/hg", line 61, in <module> dispatch.run() ... File "/usr/home/jeffpc/src/oss/hg-gitext/hgext/git/gitlog.py", line 564, in get return gitmanifest.gittreemanifestctx(self.gitrepo, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: Can't instantiate abstract class gittreemanifestctx with abstract methods node, read_any_fast_delta, read_delta_new_entries, read_delta_parents, readdelta
-
Jeff Sipek authored
-
Jeff Sipek authored
$ hg log ** unknown exception encountered, please report by visiting ** https://mercurial-scm.org/wiki/BugTracker ** Python 3.11.10 (main, Oct 31 2024, 01:10:40) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15 ** Mercurial Distributed SCM (version 6.9.post1.dev337+hg.72af9fa34832) ** Extensions loaded: absorb, churn, git (pygit2 1.14.1), githelp, gpg, hgk, histedit, patchbomb, rebase Traceback (most recent call last): File "/home/jeffpc/src/oss/hg-gitext-test/../hg-gitext/hg", line 61, in <module> dispatch.run() ... File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/tags.py", line 507, in _readtagcache if not len(repo.file(b'.hgtags')): ^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/hgext/git/__init__.py", line 138, in file return gitlog.filelog(self.store.git, self.store._db, path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: Can't instantiate abstract class filelog with abstract methods addgroup, addrevision, censorrevision, children, commonancestorsheads, descendants, emitrevisions, files, getstrippoint, heads, iscensored, parentrevs, rawdata, revision, revs, size, storageinfo, strip, verifyintegrity
-
Jeff Sipek authored
$ hg log ** unknown exception encountered, please report by visiting ** https://mercurial-scm.org/wiki/BugTracker ** Python 3.11.10 (main, Oct 31 2024, 01:10:40) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15 ** Mercurial Distributed SCM (version 6.9.post1.dev337+hg.72af9fa34832) ** Extensions loaded: absorb, churn, git (pygit2 1.14.1), githelp, gpg, hgk, histedit, patchbomb, rebase Traceback (most recent call last): File "/home/jeffpc/src/oss/hg-gitext-test/../hg-gitext/hg", line 61, in <module> dispatch.run() ... File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 1789, in changelog repo.dirstate.prefetch_parents() ^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 225, in __get__ return super(unfilteredpropertycache, self).__get__(unfi) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/util.py", line 1822, in __get__ result = self.func(obj) ^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/mercurial/localrepo.py", line 1802, in dirstate self._dirstate = self._makedirstate() ^^^^^^^^^^^^^^^^^^^^ File "/usr/home/jeffpc/src/oss/hg-gitext/hgext/git/__init__.py", line 311, in _makedirstate return dirstate.gitdirstate( ^^^^^^^^^^^^^^^^^^^^^ TypeError: Can't instantiate abstract class gitdirstate with abstract methods _checkexec, _ignorefileandline, changing_files, clear, copy, hasdir, invalidate, is_changing_files, rebuild, verify
-
- Dec 16, 2024
-
-
Georges Racinet authored
We take the existing doc-comment from rust-cpython and make it compile (validated by `cargo test`). With the added explanations, the ordinary comment was no longer useful, we could therefore remove it. The new explanation stresses that "not leaking the internal faked reference" is definitely not enough, because the problem is about *all references* that can be derived from it. We ended up duplicating the explanation, because that is a way to ensure that people do not miss it. Also, it was a bit misleading that the previous example was for `try_borrow_mut()`, so we made a similar, simpler one for `try_borrow()`.
-
Georges Racinet authored
We are actually about to lift it, and it will be more convincing if this compiling version of the old example could be refused by the compiler. Note: the `no_run` attribute on the example block checks that it compiles but does not run the example.
-
- Dec 15, 2024
-
-
Georges Racinet authored
Now that the renamings are done in main code, we give local variables and helper functions in the integration tests matching names.
-
Georges Racinet authored
The pattern to borrow (with owner) for the sole purpose of calling `share_immutable`, and then `map()` is pretty common, being exactly what is needed in the constructors of derived Python objects. we therefore introduce new helpers doing exactly that. We keep the intermediate `share()` and `try_share()` that are used in integration tests, but is is possible that consumers of the API never need it. A nice feature of the new `share_map()` helper is that it collapses all the unsafety in constructors of derived object to one call. It makes sense, since the reason for unsafety is the same all along the call stack: the `owner` object, that cannot be guessed from `PyShareable` itself, must be the right one. It is remarkable that it is only at this point that the compiler insists that `T` should be `'static`, which is actually very reasonable, as it encompasses owned data. We could have set it since the beginning, but the added value is low, as PyO3 would not let us use some `PyShareable<&'a T>` as a Python object data field (with `'a` not outliving `'static` of course)
-
Georges Racinet authored
Rationale: - the object itself is not unsafe, only most of its methods are. This is similar to raw pointers: obtaining them is not unsafe, using them is. That being said, we are not ready yet to declare, e.g, `borrow_with_owner()` safe because that gives an early warning to users. - it was not really a leak, as the data is actually owned by a Python object on which we keep a (Python) reference with proper increment of the refcount. Hence the terminology was too much of a deterrent. We hope that the new terminology conveys both that the data contains a shared reference and that it was generously lended by a Python object. The `SharedByPyObjectRef` name is arguably heavy, but it is merely an implementation detail that users will not be much exposed to, thanks to the `Deref` implementation. As previously, we keep the changes in tests to a minimum, to make obvious in the diff that the tests have not changed. We will take care of renaming their local variables and functions in a later move.
-
Georges Racinet authored
There were two problems with the naming: - the PyO3 version is not based on `RefCell`. Rather than calling this `SomethingRwLock` we decided it was best to consider this an implementation detail. We mentioned it in the doc earlier on, but that is merely just bringing some reassuring context to the reader, not intended to be a needed structural explanation. - the data is not shared: it is ready to be shared To keep the changeset readable, we only make the very minimal changes to the tests to make them pass, i.e., not renaming local variables (this will be done in a later move).
-
Georges Racinet authored
The "borrow" word had way too many uses in this context. Originally, it came from the rust-cpython version, which is based on `RefCell`. Before this change, we had untracktable stacks of borrows (first the `Bound`, then the `PySharedRefCell`). In any case, the change to `RwLock` is far from being neutral, and we may need in a future without GIL to also expose blocking methods, to account for in-Rust possible concurrency. Using `read()` and `write()` right now matches our PyO3 habits anyway, where all non-Sync objects have to be wrapped in a RwLock.
-
Georges Racinet authored
We had previously duplicated the `new` associated function on `PySharedRef` with a method on `PySharedRefCell`: in `rust-cpython`, the former was hidden by the accessor defined by the `py_class!` macro, which we did not port yet. On `PySharedRefCell` itself, replacing the `new` associated function by the `From` trait carries all the needed semantics, and has the advantage of less repetititons of the type name, which will help with further refactorings and renamings.
-
- Dec 14, 2024
-
-
Georges Racinet authored
This should bring full confidence on the conversion to PyO3. It highlights also the difference between the two bindings systems: in PyO3, the struct defined by the user is the inner Rust one. In rust-cpython, it is the wrapped one exposed to CPythob We enclose some of the boilerplate in helper functions. Perhaps we should first import the rust-cpython integration test, rework it with the same helpers, then only change the helpers.
-
- Dec 15, 2024
-
-
Georges Racinet authored
At this point, we have a full translation, with some tests. The tests are high level, demonstrating the case of iterating over `HashSet` as a doctest. The assertions are written as Python statements, in order not to be obscure and to resist later refactorings. The rust-cpython integration tests will be ported in a subsequent changeset. We find the example of iterating over `HashSet` to be more convincing than the iterating over `Vec`, the example provided in rust-cpython, because in the case of `Vec`, it would be simple enough to use `Arc<Vec>` and an index. This would of course be reimplementing the iterator, but is simple enough that it leads the reader to believe that having a reference is the problem, whereas the problem is having a reference that is itself enclosed in a type with lifetime that can be arbitrary complex. It took us some time to remember that subtlety, and hence we reworded the documentation to stress that point. We decided to put this work a separate crate, which makes running `cargo test` work for the crate, but not for the entire workspace: the `extension-module` feature gets in the way. That is why we reexpose the feature on `hg-pyo3` as default and run the tests with `--no-default-feature` in Makefile, hence in CI. An important difference with rust-cpython is that everything has to be `Sync`, hence we replace `RefCell` with `RwLock` and only speak of "interior mutability" in the documentation. Since everything happens in `hg-pyo3` behind the GIL, there is at this point no reason to use `read()` and `write()` instead of respectively `try_read()` and `try_write()`. But PyO3 is aiming to support free-threaded Python (without GIL), and `PySharedRef` should therefore allow waiting for its inner locks.
-
- Dec 17, 2024
-
-
Matt Harbison authored
-
Matt Harbison authored
I think the typing around whether `open()` returns `IO[bytes]` or `IO[str]` hinges on the content of the mode string. Converting from bytes instead of using a literal can suppress that (though PyCharm currently complains about this). Instead, we can mandate the use of a (vastly reduced) set of mode options. For now, none of the 3 callers provide this argument, so it's not a big deal. Ideally, this would always enforce binary mode. There's a little extra typing required to pull this off. The `_unclosablefile` class can't subclass `typing.BinaryIO`, because there were a bunch of test failures around writing to stdout. Strangely, pytype didn't complain that the abstract methods on `typing.BinaryIO` weren't overridden in this case. Whatever was going on, it's a simple proxy class, so we can just cast to the expected type in the one place it is used.
-
- Jan 01, 2025
-
-
Matt Harbison authored
This mirrors the function override that largefiles has, and validating input near the top of the function makes more sense.
-
- Dec 05, 2024
-
-
Matt Harbison authored
A little less complicated, and guarded by type checking.
-
- Jan 01, 2025
-
-
Matt Harbison authored
This is intertwined with the `mode`, which is also used with `open()`, so the goal is to get that to str so that the `pycompat.open()` call can be dropped. Start simple because I had issues where the generated archive in some tests had 1 different byte for some reason when making the full change, even though the content was binary equivalent when doing a Folder Compare of the archives with BeyondCompare4. The key for the `archivers` map is left alone for now. The classes that are stored are only ever called with 2 args, so this should be safe.
-