Mitigations for memory leaks
There are several problems with freeing memory in HGitaly.
Mercurial repository objects often don't get garbage collected, even with our periodic full (generation 2) collection (gc.collect()
). This is the main blocker for #57 (closed) (the cache becomes useless once it's full), but it doesn't result in a very fast growth.
The periodic full collection still has a positive impact on memory, but it is often skipped in production because it is based on a thread-local counter (inherited from hgwebdir
) and the gRPC server sometimes starts over in a new thread. Statistically, that occurs before the count is reached. HGitaly uses a pre-fork strategy with only one servicer thread, but that is only a maximum number of threads and does not mean it is always the same thread.
Finally we should have a server process watcher system to gracefully terminate them when their resident size or total memory goes above a configurable threshold : when the OOMKiller kicks in, other services are already suffering from memory starvation, and it's not exactly graceful.