Mitigations for memory leaks
There are several problems with freeing memory in HGitaly.
Mercurial repository objects often don't get garbage collected, even with our periodic full (generation 2) collection (
gc.collect()). This is the main blocker for #57 (the cache becomes useless once it's full), but it doesn't result in a very fast growth.
The periodic full collection still has a positive impact on memory, but it is often skipped in production because it is based on a thread-local counter (inherited from
hgwebdir) and the gRPC server sometimes starts over in a new thread. Statistically, that occurs before the count is reached. HGitaly uses a pre-fork strategy with only one servicer thread, but that is only a maximum number of threads and does not mean it is always the same thread.
Finally we should have a server process watcher system to gracefully terminate them when their resident size or total memory goes above a configurable threshold : when the OOMKiller kicks in, other services are already suffering from memory starvation, and it's not exactly graceful.