Multiprocessing leaking file descriptors
Created originally on Bitbucket by Callation (Andrew Robinson)
Have migrated a multiprocessing project from PyPy 6.0 (2.7) to PyPy 7.3 (3.6) and have identified an issue with multiprocessing leaking descriptors. I believe this may also relate to the issue reported as #3060.
I have identified this issue as repeatable on both Ubuntu 18.04.2 and CentOS 7.0.
Simple code to reproduce issue (Taken from #3111 which is raised a different issue, but current issue is reproduced with the same code sample):
import multiprocessing
def process_chunk(chunk_index):
return ([],[],[])
def main():
i = 0
with multiprocessing.Pool(5, maxtasksperchild=1) as p:
for sub_headers, sub_results, sub_scores in p.imap_unordered(process_chunk, range(5000)):
print(i)
i += 1
if __name__ == '__main__':
main()
If saving the above as “test.py” the descriptor leak can be verified by running “/path/to/pypy3 test.py” then taking the PID from ps and doing an “ls -l /proc/PID/fd” and comparing to the same when running native Python Ie. Running “/path/to/python3 test.py” taking PID and “ls -l /proc/PID/fd”.
Under native Python3 the descriptor list remains constant at 9 pipe entries.
Under PyPy the file descriptor list keeps growing until ulimit is reached and it will throw an exception as “OSError: [Errno 24] Too many open files“. In my case ulimit is set to 1024.
Traceback below:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/callation/pypy3/lib-python/3/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/callation/pypy3/lib-python/3/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 405, in _handle_workers
pool._maintain_pool()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 246, in _maintain_pool
self._repopulate_pool()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 239, in _repopulate_pool
w.start()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 65, in _launch
parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files