Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
pypy
pypy
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 594
    • Issues 594
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 10
    • Merge Requests 10
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar

This instance will be upgraded to the latest Heptapod 0.20rc on 2021-02-26 at 11:00 UTC+1

  • PyPy
  • pypypypy
  • Issues
  • #3154

Closed
Open
Opened Jan 16, 2020 by Bitbucket Importer@bitbucket_importerMaintainer

Multiprocessing leaking file descriptors

Created originally on Bitbucket by Callation (Andrew Robinson)

Have migrated a multiprocessing project from PyPy 6.0 (2.7) to PyPy 7.3 (3.6) and have identified an issue with multiprocessing leaking descriptors. I believe this may also relate to the issue reported as #3060.

I have identified this issue as repeatable on both Ubuntu 18.04.2 and CentOS 7.0.

Simple code to reproduce issue (Taken from #3111 which is raised a different issue, but current issue is reproduced with the same code sample):

import multiprocessing

def process_chunk(chunk_index):
    return ([],[],[])

def main():
    i = 0
    with multiprocessing.Pool(5, maxtasksperchild=1) as p:
        for sub_headers, sub_results, sub_scores in p.imap_unordered(process_chunk, range(5000)):
            print(i)
            i += 1

if __name__ == '__main__':
    main()

If saving the above as “test.py” the descriptor leak can be verified by running “/path/to/pypy3 test.py” then taking the PID from ps and doing an “ls -l /proc/PID/fd” and comparing to the same when running native Python Ie. Running “/path/to/python3 test.py” taking PID and “ls -l /proc/PID/fd”.

Under native Python3 the descriptor list remains constant at 9 pipe entries.

Under PyPy the file descriptor list keeps growing until ulimit is reached and it will throw an exception as “OSError: [Errno 24] Too many open files“. In my case ulimit is set to 1024.

Traceback below:

‌

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/callation/pypy3/lib-python/3/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/callation/pypy3/lib-python/3/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 405, in _handle_workers
    pool._maintain_pool()
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 246, in _maintain_pool
    self._repopulate_pool()
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 239, in _repopulate_pool
    w.start()
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 65, in _launch
    parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files

‌

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: pypy/pypy#3154