I am currently trying to use hggit to clone from a git repo(local directory) and I noticed that hg-git spent a lot of time in memory even before anything was written on disk.
What that means is that with a VERY BIG repo, this approach would not scale as ram is often limited.
Reproduce:
# Try to clone 800k commits and monitor the progresshg clone git@github.com:torvalds/linux.git
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Child items
0
Show closed items
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Adding my own half-remembered example from earlier this year. If details would be helpful, I can try the operation again to fill them in.
I tried cloning https://gitweb.dragonflybsd.org/dports.git using hg-git. (Not sure it was with the https protocol.) This is an automatically generated repo, and may have an unnaturally large number of commits because of that.
The progress bar estimated it would take several hours. Like, 24h or 14h or something like that.
I let the process run for several hours, but it died because the machine ran out of memory+swap. At that point, the process was using on the order of 10s of gigabytes of memory.
Could you perhaps retry using the tip of the default branch? I removed some caching that might help, but I didn't manage to locate any obvious “leaks”.[1] I haven't done much memory profiling of Python — or cloning of large repositories, for that matter — since then, so I'm not sure. But avoiding unnecessary caching is a good starting point.
I'd appreciate it if someone could provide some memory profiles, or similar, to help find out what's actually taking up all that space. One possible cause is that Dulwich caches pack objects internally, and we never tell it to close them — or run the equivalent of git gc for that matter.
So, the situation should be better now, but I'm not convinced I've located the root cause, yet.
[1] Technically speaking, you cannot leak memory from pure Python code; hence the quotation marks.
OK, I've started hg clone git://mirror-master.dragonflybsd.org/dports.git. About an hour (oops, more like three hours; I miscalculated) later it's made progress:
importing [=======> ] 85775/485324 15h56m
I don't think it's using as much memory as last time. I see a python process with about 380MiB resident, and a git process with about 216MiB resident. If that scales with the progress bar it should be fine.
If course it would be nice if it were faster :-)
I don't have much Python experience but can try providing a memory profile if you have instructions.