Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
pypy
pypy
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 582
    • Issues 582
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 12
    • Merge Requests 12
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards

This instance will be upgraded to Heptapod 0.19.0rc2 on 2021-01-18 at 11:00 UTC+1 (a few minutes of downtime)

  • PyPy
  • pypypypy
  • Issues
  • #3289

Closed
Open
Opened Aug 27, 2020 by Chris Thomson@cthomson

increase in memory usage with reloading of module when JIT is enabled

Hi,

We have a long running application where there is a set of Python files that are periodically reloaded, the Python scripts in turn load JSON files of mappings and lists that are fairly large in size. We have noticed that under CPython and PyPy with the JIT disabled there is a relatively small amount of memory that leaks in a 10 reload cycle scenario (a few MB). However when we run the same scenario using PyPy and the JIT enabled the memory leak is much larger (~1.5 GB).

Details:

Python 2.7.13 (ab0b9caf307d, Apr 24 2018, 18:04:42)
[PyPy 6.0.0 with GCC 6.2.0 20160901]

I tried running with -X track-resources but there was no additional output provided for the scenario.

Running with export PYPYLOG=jit-summary:- and the JIT enabled shows:

[2d0d1712fe3ec] {jit-summary
Tracing:      	157	0.357549
Backend:      	157	0.124614
TOTAL:      		125.859928
ops:             	224984
recorded ops:    	51314
  calls:         	9079
guards:          	12195
opt ops:         	25488
opt guards:      	4908
opt guards shared:	3086
forcings:        	0
abort: trace too long:	0
abort: compiling:	0
abort: vable escape:	0
abort: bad loop: 	0
abort: force quasi-immut:	0
nvirtuals:       	13285
nvholes:         	5995
nvreused:        	4602
vecopt tried:    	0
vecopt success:  	0
Total # of loops:	86
Total # of bridges:	78
Freed # of loops:	27
Freed # of bridges:	25
[2d0d174586d78] jit-summary}

Added a call to gc.dump_rpy_heap and captured three dumps

  • jit_7_iteration.txt - JIT enabled, run with 7 reload cycles
  • jit_8_iteration.txt - JIT enabled, run with 8 reload cycles
  • nojit_8_iteration.txt - JIT disabled, run with 8 reload cycles

Any hints as to why enabling the JIT would result in this additional memory usage or code patterns that we should be on the lookout for that may trigger this problem would be appreciated.

jit_7_iteration.txt

jit_8_iteration.txt

nojit_8_iteration.txt

Edited Aug 27, 2020 by Chris Thomson
To upload designs, you'll need to enable LFS and have admin enable hashed storage. More information
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: pypy/pypy#3289