Multiprocessing leaking file descriptors
Created originally on Bitbucket by Callation (Andrew Robinson)
Have migrated a multiprocessing project from PyPy 6.0 (2.7) to PyPy 7.3 (3.6) and have identified an issue with multiprocessing leaking descriptors. I believe this may also relate to the issue reported as #3060.
I have identified this issue as repeatable on both Ubuntu 18.04.2 and CentOS 7.0.
Simple code to reproduce issue (Taken from #3111 which is raised a different issue, but current issue is reproduced with the same code sample):
import multiprocessing def process_chunk(chunk_index): return (,,) def main(): i = 0 with multiprocessing.Pool(5, maxtasksperchild=1) as p: for sub_headers, sub_results, sub_scores in p.imap_unordered(process_chunk, range(5000)): print(i) i += 1 if __name__ == '__main__': main()
If saving the above as “test.py” the descriptor leak can be verified by running “/path/to/pypy3 test.py” then taking the PID from ps and doing an “ls -l /proc/PID/fd” and comparing to the same when running native Python Ie. Running “/path/to/python3 test.py” taking PID and “ls -l /proc/PID/fd”.
Under native Python3 the descriptor list remains constant at 9 pipe entries.
Under PyPy the file descriptor list keeps growing until ulimit is reached and it will throw an exception as “OSError: [Errno 24] Too many open files“. In my case ulimit is set to 1024.
Exception in thread Thread-1: Traceback (most recent call last): File "/opt/callation/pypy3/lib-python/3/threading.py", line 916, in _bootstrap_inner self.run() File "/opt/callation/pypy3/lib-python/3/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 405, in _handle_workers pool._maintain_pool() File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 246, in _maintain_pool self._repopulate_pool() File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 239, in _repopulate_pool w.start() File "/opt/callation/pypy3/lib-python/3/multiprocessing/process.py", line 105, in start self._popen = self._Popen(self) File "/opt/callation/pypy3/lib-python/3/multiprocessing/context.py", line 277, in _Popen return Popen(process_obj) File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 65, in _launch parent_r, child_w = os.pipe() OSError: [Errno 24] Too many open files