I am not able to replicate this on Fedora 39, tortoisehg 6.5.1, hg 6.3 and latest hg-git 1.0.3+ (mercurial/hg-git@00a2910821c5).
I tried to quickly move back to hg-git 0.10.0 but (perhaps not surprisingly) hg-git blew up on the latest hg/thg versions.
[unrelated: @mharbison72: is thg 6.5rc0 shipping hg-git 0.10.x on windows? I can vouch for hg-git 1.0.x being very stable for some time now]
I tried to reproduce this bug on my desktop (Fedora 38, hg 6.3.3, thg 6.4.2), with no success. Is it maybe something that only happens on Windows?
I've created a repo full of UTF-8 extended characters (which on linux can be done typing while keeping pressed AltGr), and then exported the tip commit via export|copy patch
. I pasted the clipboard on Kate (KDE's text editor), and then created a diff via command line with hg diff --change tip > patch-via-cmd-line
.
Apart from the headers, they are bit-for-bit equal. They can be compared with Meld (or any similar Windows program to compare files):
Patch via thg|export|copy patch
+ save via text editor:
# HG changeset patch
# User muxator <a.mux@inwind.it>
# Date 1686165976 -7200
# Wed Jun 07 21:26:16 2023 +0200
# Node ID 52785d757dc6b6ffb487c7a0010b01c0ef38a37e
# Parent 4077c494b8df26b349be8a3974c4136042986706
Patch ² OR 2?
diff -r 4077c494b8df -r 52785d757dc6 ↓ßŧ-æ-đ→ł€
--- a/↓ßŧ-æ-đ→ł€ Wed Jun 07 21:25:03 2023 +0200
+++ b/↓ßŧ-æ-đ→ł€ Wed Jun 07 21:26:16 2023 +0200
@@ -1,4 +1,4 @@
ßð¶
-ŋßðđŋł@ßðđŋßð ĸŋđ@ßđŋĸ ß@ĸŋß@đðł¼½←¶€ħðŧ#ł@ĸ
+ŋßðđŋł@ßðđŋßð ¢ÆÑG€Ð TĦIẞ ß@ĸŋß@đðł¼½←¶€ħðŧ#ł@ĸ
Æ Ł→Ñ€ ØĐ ↓Ñ¢ĦÆÑŊ€Ð ¢ØÑŦ€»Ŧ.
Patch via hg diff --change tip > patch-via-cmd-line
:
diff --git a/↓ßŧ-æ-đ→ł€ b/↓ßŧ-æ-đ→ł€
--- a/↓ßŧ-æ-đ→ł€
+++ b/↓ßŧ-æ-đ→ł€
@@ -1,4 +1,4 @@
ßð¶
-ŋßðđŋł@ßðđŋßð ĸŋđ@ßđŋĸ ß@ĸŋß@đðł¼½←¶€ħðŧ#ł@ĸ
+ŋßðđŋł@ßðđŋßð ¢ÆÑG€Ð TĦIẞ ß@ĸŋß@đðł¼½←¶€ħðŧ#ł@ĸ
Æ Ł→Ñ€ ØĐ ↓Ñ¢ĦÆÑŊ€Ð ¢ØÑŦ€»Ŧ.
Bundle of the repository I used: thg-bug-5900 (with sha256sum bbba72926efa04ff7408be9a148368d69262e5c68926be6de33ac5a7df1fb042).
I did not try Matt's advice of using "Export patch" because "copy patch" worked well.
Confirmed.
Hi,
cloning a big git repository via hg-git has always been not really fast. For example, on my machine cloning https://github.com/bitcoin/bitcoin takes ~30 minutes.
I really hope this could be made faster; to start this effort, I thought that doing some profiling work could help. The bitcoin repository is too big, so I choose another project (https://github.com/kdave/btrfs-progs: 6500 commits, 148 tags, 16 MB of contents in its head revision, a .git folder of 19 MB).
First of all, the baselines:
command | time | %baseline |
---|---|---|
git clone https://github.com/kdave/btrfs-progs |
3.2 s | 100% |
hg clone git+https://github.com/kdave/btrfs-progs |
52.9 s | 1653% (16 times slower!) |
cprofile on hg clone git+https://github.com/kdave/btrfs-progs
|
84.3 s | speed not meaningful due to the measuring overhead |
A cloning that takes ~3 seconds with git takes ~53 seconds with mercurial/hg-git: 16 times as much!
These tests were made in this environment:
Hardware:
I have used cProfile to profile the cloning. The profiling overhead is significant (we go from from 53s to 84s), hopefully this will decrease when we can use perf on python3.12.
time python -m cProfile -o ~/btrfs-progs-clone.cprof /usr/bin/hg clone git+https://github.com/kdave/btrfs-progs
destination directory: btrfs-progs
importing 6699 git commits
new changesets 3fe272fcfd57:8a107059ab38 (6699 drafts)
updating to bookmark master
791 files updated, 0 files merged, 0 files removed, 0 files unresolved
real 1m24,293s
user 1m18,146s
sys 0m2,480s
The result of the profiling is here: btrfs-progs-clone.cprof. It can be visualized, for example, with SnakeViz (pip install snakeviz
):
snakeviz btrfs-progs-clone.cprof
As said, in this thest the total time for the cloning was ~84 seconds. This is a zoom on the first meaningful function that accounts for ~100% of that time. A breakdown of the deeper functions is in the image, while the list of the previous call stack elements is on the right, for completeness:
I've also tried using strace to profile the cloning. The overhead is still higher, however.
time strace --summary -o ~/btrfs-progs-strace.log /usr/bin/hg clone git+https://github.com/kdave/btrfs-progs
destination directory: btrfs-progs
importing 6699 git commits
new changesets 3fe272fcfd57:8a107059ab38 (6699 drafts)
updating to bookmark master
791 files updated, 0 files merged, 0 files removed, 0 files unresolved
real 3m52,715s
user 2m5,755s
sys 1m36,003s
I believe the very high sys time is strace's overhead. In the plain hg-git clone the system time was just 4% of the total time (less than 3 seconds). This aligns nicely with the absolute times that strace produces in its summary:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ------------------
26,91 0,889219 1 548434 3 lseek
24,04 0,794304 2 350410 read
12,80 0,422899 4 101525 17551 openat
12,53 0,414054 2 150241 2878 newfstatat
6,02 0,198903 6027 33 3 wait4
5,62 0,185728 2 83999 close
4,96 0,163822 1 87659 83051 ioctl
4,12 0,136191 3 41115 write
1,02 0,033586 5 6502 utimensat
I'd say that not too much time is spent in the system calls. Maybe there is a tad too much disk activity but nothing stands out.
I suspect the
str
is from ~line 232 where['*'+ext]
and['**'+ext]
are added to thefilter
variable
Yeah, seems right. It would be good to have a way of passing through lines 231-232. It is only a matter of decyphering what happens before them :)
I tried this on current thg (afb171909efb, 6.4+), patching these lines with:
diff --git a/tortoisehg/hgqt/hgignore.py b/tortoisehg/hgqt/hgignore.py
--- a/tortoisehg/hgqt/hgignore.py
+++ b/tortoisehg/hgqt/hgignore.py
@@ -245,8 +245,16 @@ class HgignoreDialog(QDialog):
def insertFilters(
self, pats: Optional[Iterable[bytes]] = None, isregexp: bool = False
):
+ # TRYING TO PRINT self.sender()._patterns here triggers an
+ # AttributeError: 'QPushButton' object has no attribute '_patterns'
if not pats:
+ # I WAS NOT ABLE TO TAKE THIS BRANCH WHEN CLICKING ON "ADD" IN THE
+ # IGNORE DIALOG
pats = self.sender()._patterns
+ else:
+ print(f"{pats=}") # prints: [b'*.o']
+ print(f"{type(pats)=}") # prints: <class 'list'>
h = isregexp and b'syntax: regexp' or b'syntax: glob'
if h in self.ignorelines:
l = self.ignorelines.index(h)
When I add a "*.o" via the GUI in an .hgignore
containing the single line "syntax: glob", it seems that the if not pats
branch is not taken, because pats
is already a list of bytes:
pats=[b'*.o']
type(pats)=<class 'list'>
I was not able to enter the if not pats
branch juggling the GUI.
I was not able to print the value of _patterns
(see the comments in the patch). :(
I am on Fedora 37, using the system's mercurial (currently 6.2.3, just like in the bug report), but I run thg from a code checkout, so I tried to reproduce the bug with both thg 6.2.3 (a6dd9acb5ebe) and 6.4 (d78a9df71cf3).
I was able to create an .hgignore and populate it with "*.o" via the GUI in both cases:
The versions of the other components were the same for both tests, and are:
The error seems to be the "usual" bytes vs string kind of error, but I do not recall ever encountering it in this dialog.
A permalink to the place triggering the error in v6.2.3 is https://foss.heptapod.net/mercurial/tortoisehg/thg/-/blob/6.2.3/tortoisehg/hgqt/hgignore.py#L228, no recent changes from a quick skim.
So I think what is happening here is you're looking at the overall size of the
.hg
directory, which holds the strip-backup of the commit prior to the amend.
Yeah I think this is indeed what is happening to @mhutter. The .hg/strip-backup
is populated with a bundle (something like c3ee55a63c9a-133aa380-amend.hg
) containing the commit being overwritten. Since that commit added a 20 MB file, that bundle is ~20 MB in size too.
@mhutter, just for reference: in case of emergency, the old commit can be reimported via an unbundle command (see https://www.mercurial-scm.org/doc/hg.1.html#unbundle):
$ hg unbundle .hg/strip-backup/c3ee55a63c9a-133aa380-amend.hg
adding changesets
adding manifests
adding file changes
added 1 changesets with 0 changes to 1 files (+1 heads)
new changesets c3ee55a63c9a (1 drafts)
(run 'hg heads' to see heads, 'hg merge' to merge)
After that you will see the old commit reappearing (rev 2 in this simple screenshot):
I can confirm I observe the behavior described here: when pushing a new branch a "confirm new branch" dialog appears.
When pushing a new head of an already existing branch, only an error message and no dialog box appears.
However, judging from its contents, I suspect this dialog was never meant to cover the "new head of existing branch" case.
Hi,
I cannot reproduce this on my setups:
I also never noticed this on earlier versions when I was using Ubuntu; I think the latest version I used was 22.04
A shot in the dark: it might be something related to the compositor and/or your graphic driver. Does your problem persist if you run under X11 with no compositor?
I guess @dan is well aware of this problem, since he already opened https://github.com/jelmer/dulwich/issues/1139.
I just came here to report this exact same error ("AttributeError: 'DiskObjectStore' object has no attribute 'move_in_pack'
" when cloning).
The problem is due to a breaking change in dulwich, I suspect. The reason is that on my usual machine I am able to clone as usual, whereas on a VM I created today for other reasons I encounter this error.
In both cases the hg-git version I am using is the current latest from branch 1.0.x
: c114cb732135 ("NEWS: fix typos").
Where it works:
Where it does not work:
Downgrading dulwich to v0.20.50 (via pip install --user dulwich==0.20.50
) temporarily fixed the problem for me.
Indeed, I can confirm the bug is fixed!
Many thanks, @dan, this is really appreciated!
Thank you @dan for pointing me to import_git_objects()
, that seems indeed the right direction!
The incriminated commit appears among the arguments passed to import_git_objects(). The names of the ref appear to be related to the closed PRs.
$ hg pull
pulling from git+https://github.com/muxator/hg-git-issue-386.git
remote_names:
[b'default']
refs:
{b'HEAD': b'd691e329d0875e1e10a4e8256980a0111fdc1847',
b'refs/heads/main': b'd691e329d0875e1e10a4e8256980a0111fdc1847',
[... old,closed PRs... ]
b'refs/pull/17/head': b'3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa', <-- THIS IS THE INCRIMINATED COMMIT!
[... old,closed PRs... ]}
heads:
None
importing 1 git commits
new changesets 7bc660faaa27 (1 drafts)
(run 'hg heads' to see heads, 'hg merge' to merge)
Maybe there is a way of fixing it, I'll investigate. Thanks again!
(stale) progress update, just for my future memory.
I have tried to remove every mention of the zombie commit (7bc660faaa27) from the example hg repo I've linked in the previous comment.
$ cd <extracted-tar-gz>/.hg/git
$ git update-ref -d refs/heads/PR
$ cd ..
$ rm -rf cache/
$ hg strip -r 7bc660faaa27
$ hg git-cleanup
git commit map cleaned
$ rg "3e1c0c43ee26|7bc660faaa27"
[no results, everything seems to be cleaned up]
After this command, every reference to the zombie commit seem to have disappeared from the local clone (this is not true, read later). However:
hg pull
still makes the commit reappearHi @dan.
As time passes, it becomes important to migrate to a recent hg-git, and this problem becomes more impactful.
I'd really like any insight, but I also realize I might be the only one hitting this problem, and the needs of the many outweigh the needs of the few
I've tried to gather some information.
Let's refer to the github repo https://github.com/muxator/hg-git-issue-386. The repo there has this structure:
@ changeset: 1:576b5fcad248
| git-commit: d691e329d087
| bookmark: main
| tag: default/main
| tag: tip
| user: muxator <a.mux@inwind.it>
| date: Tue Nov 01 21:21:59 2022 +0100
| summary: a: added a line
|
o changeset: 0:d8938645aab2
git-commit: af38c75a55c6
user: muxator <a.mux@inwind.it>
date: Fri Oct 21 23:33:51 2022 +0200
summary: first commit
Rev 1 is the result of the procedure described at the start of this issue.
I've attacched a tar.gz of the "corrupted" clone (hg-git-issue-386.tar.gz
).
If, starting from that clone, one tries to pull, a new commit will be pulled in:
o changeset: 2:7bc660faaa27 <== MONSTER FROM THE PAST!
| git-commit: 3e1c0c43ee26 <== MONSTER FROM THE PAST!
| tag: tip <== MONSTER FROM THE PAST!
| parent: 0:d8938645aab2 <== MONSTER FROM THE PAST!
| user: muxator <a.mux@inwind.it> <== MONSTER FROM THE PAST!
| date: Tue Nov 01 21:21:59 2022 +0100 <== MONSTER FROM THE PAST!
| summary: a: added a line <== MONSTER FROM THE PAST!
|
| @ changeset: 1:576b5fcad248
|/ git-commit: d691e329d087
| bookmark: main
| tag: default/main
| user: muxator <a.mux@inwind.it>
| date: Tue Nov 01 21:21:59 2022 +0100
| summary: a: added a line
|
o changeset: 0:d8938645aab2
git-commit: af38c75a55c6
user: muxator <a.mux@inwind.it>
date: Fri Oct 21 23:33:51 2022 +0200
summary: first commit
The stray revision "2" is the one that was originally part of the PR
branch.
I tried to grep around its hashes: (both the mercurial one and the git one) with the following command:
rg --hidden "7bc660faaa27|3e1c0c43ee26"
These are the results:
This is the status of the repo before the pull described at the start of the issue and whose tar.gz I've attacched.
.hg/git/refs/heads/PR
1:3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa
.hg/cache/branch2-visible
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
.hg/cache/branch2-served.hidden
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
I think this is already an indication of corruption: the hash of the commit is already there, even if the commit is not in the mercurial repo.
Please note that, starting from this status, an hg pull
with hg-git v0.10.x would print no changes found. And I would be happy with this.
.hg/git-mapfile <== NEW
3:3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa 7bc660faaa27c2ad9b641c161e5cb187f310ad6d
.hg/git/refs/heads/PR
1:3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa
.hg/cache/branch2-served <== NEW
1:7bc660faaa27c2ad9b641c161e5cb187f310ad6d 2
3:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
.hg/cache/tags2-visible <== NEW
1:2 7bc660faaa27c2ad9b641c161e5cb187f310ad6d
.hg/cache/branch2-visible
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
.hg/cache/branch2-served.hidden
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
This is the situation represented in the figure where I wrote "MONSTER FROM THE PAST".
At this point I am normally forced to strip the zombie commit:
hg strip -r 'gitnode(3e1c0c43ee26)'
rg --hidden "7bc660faaa27|3e1c0c43ee26"
.hg/git-mapfile
3:3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa 7bc660faaa27c2ad9b641c161e5cb187f310ad6d
.hg/git/refs/heads/PR
1:3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa
.hg/cache/branch2-visible
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
.hg/cache/branch2-served.hidden
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
And then clean the mapfile:
hg gitcleanup
rg --hidden "7bc660faaa27|3e1c0c43ee26"
.hg/git/refs/heads/PR
1:3e1c0c43ee26b6a511fd8a71c94ddf8e027f7efa
.hg/cache/branch2-visible
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
.hg/cache/branch2-served.hidden
2:7bc660faaa27c2ad9b641c161e5cb187f310ad6d o default
This brings me back to the starting point: any new pull would bring back 3e1c0c43ee26.
Where could I start to look? Would giving you write access to the repo help in any way (I suppose you are danchr
there)?
Thanks, thanks, thanks
p.s.: as minimal security precaution: the md5 of the attached file is: cd279a051df1bce15689f93da579f5c8 hg-git-issue-386.tar.gz
Could you please check if this also happens with the
default
branch?
Yep, checked just now: it also happens on default
(currently b963d11cc1c8). Also, mercurial version does not really matter.
Unless there’s something special about using the [you meant GitHub, here, right?] API
The "canonical" manifestation of the problem is when working via the GitHub web UI. The (ugly) script I've written at https://github.com/muxator/hg-git-issue-386 uses the GitHub API because that's the only way I could come up with for making this somehow programmatically repeatable.
as I rely on deleting local bookmarks myself on a daily basis.
Me too. The only problem seems to be with how GitHub handles the "rebase-PR". Personally I am heavily affected because that's my main way of interacting with GitHub (not that I like it: I am forced to). Also, please note that in this case the remote bookmark is deleted first (by GitHub itself), and then the local bookmark is deleted, either manually (hg-git 0.10.x
) or automatically (1.0.x
& default
).
By the way I hate how GitHub subtly changes the meaning of words: a "rebase" PR is more akin to a graft/cherry-pick I'd say.
One thing: are you using the Evolve extension, or did you enable obsolete in the repository? I don’t think that should affect it, but that’s the setup I use, and it really does improve the Mercurial+Git experience
🙂
Nope, I suspect I am missing a lot here in terms of UX; my problem is that I've found a sweet spot from where I am able to (destructively) rewrite history in every conceivable way, and old habits are hard to change.
These are my current extensions:
absorb
amend
churn
extdiff
hggit (6.2!)
histedit
mq
patchbomb
rebase
show
strip
uncommit
One final note: the error checking in the script in https://github.com/muxator/hg-git-issue-386 is not good enough: the script dies right after the PR has been merged. It is enough to cause the problem, but the next operations would have to be done manually. If you intend to try it, I'll need not only to give you the access token, but also to improve it a bit.
From a quick check comparing a fresh clone with a "corrupted" one, the .hg/git/refs/heads
of the "corrupted" repo contains a file pointing to the remotely-deleted branch.
Once the repository is in this state:
.hg/git/refs/heads/XXX
contains itnote: here, for "corrupted" I mean a clone which underwent the PR / rebase-merge / delete remote branch cycle and uses hg-git 1.0.x.
Hi, I have re-tested the latest 1.0.x
hg-git revision (currently 906acb309cc8: "ci: adjust dependencies") and the problem is still present.
I still have no easy way to reproduce the problem in a completely local way. I am working on a little github repo to try to show how to create a "corrupted" local clone: https://github.com/muxator/hg-git-issue-386.
It uses the github API and an access token to create, open, rebase-merge (sic) a PR and delete its corresponding branch.
If the operation is done with hg-git 1.0.x
, the deleted commit "reappears" on pulls.
@dan: if you are interested/have time I can privately send you an access token via email.
Otherwise, I could also attach a tar.gz with the "corrupted" repo generated with hg-git 1.0.x and a good one with hg-git 0.10.x.