increase in memory usage with reloading of module when JIT is enabled
Hi,
We have a long running application where there is a set of Python files that are periodically reloaded, the Python scripts in turn load JSON files of mappings and lists that are fairly large in size. We have noticed that under CPython and PyPy with the JIT disabled there is a relatively small amount of memory that leaks in a 10 reload cycle scenario (a few MB). However when we run the same scenario using PyPy and the JIT enabled the memory leak is much larger (~1.5 GB).
Details:
Python 2.7.13 (ab0b9caf307d, Apr 24 2018, 18:04:42)
[PyPy 6.0.0 with GCC 6.2.0 20160901]
I tried running with -X track-resources
but there was no additional output provided for the scenario.
Running with export PYPYLOG=jit-summary:-
and the JIT enabled shows:
[2d0d1712fe3ec] {jit-summary
Tracing: 157 0.357549
Backend: 157 0.124614
TOTAL: 125.859928
ops: 224984
recorded ops: 51314
calls: 9079
guards: 12195
opt ops: 25488
opt guards: 4908
opt guards shared: 3086
forcings: 0
abort: trace too long: 0
abort: compiling: 0
abort: vable escape: 0
abort: bad loop: 0
abort: force quasi-immut: 0
nvirtuals: 13285
nvholes: 5995
nvreused: 4602
vecopt tried: 0
vecopt success: 0
Total # of loops: 86
Total # of bridges: 78
Freed # of loops: 27
Freed # of bridges: 25
[2d0d174586d78] jit-summary}
Added a call to gc.dump_rpy_heap
and captured three dumps
- jit_7_iteration.txt - JIT enabled, run with 7 reload cycles
- jit_8_iteration.txt - JIT enabled, run with 8 reload cycles
- nojit_8_iteration.txt - JIT disabled, run with 8 reload cycles
Any hints as to why enabling the JIT would result in this additional memory usage or code patterns that we should be on the lookout for that may trigger this problem would be appreciated.