Performance issue with the exec statement
My program, written in Python 2, needs to dynamically execute Python code from user input via the exec statement, and I tried to use PyPy to speed it up. However, I found that the program runs much slower under PyPy than under CPython. After investigating, I have found several PyPy performance issue with the exec statement.
Environment information and PoC are as below. Any help or explanation will be greatly appreciated.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.11 (stretch)
Release: 9.11
Codename: stretch
$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3
$ pypy --version
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
[PyPy 5.6.0 with GCC 6.2.0 20161109]
Functions with exec statements
File: function-with-exec.py
import sys
import time
def call1():
return
exec '' # Unreachable.
def call2():
if True:
return
exec '' # Still unreachable in fact, but `call2` is much slower than `call1` under PyPy.
def main():
if sys.argv[1] == '1':
s = time.time()
for _ in range(10000000):
call1()
e = time.time()
print(e-s)
elif sys.argv[1] == '2':
s = time.time()
for _ in range(10000000):
call2()
e = time.time()
print(e-s)
if __name__ == '__main__':
main()
We run this script in different ways:
$ pypy function-with-exec.py 1
0.0103108882904
$ pypy function-with-exec.py 2
18.2597911358
$ pypy --jit off function-with-exec.py 1
3.22720384598
$ pypy --jit off function-with-exec.py 2
3.86836504936
$ python function-with-exec.py 1
1.49009108543
$ python function-with-exec.py 2
1.53774189949
Dynamically execute multiple pieces of code in a short time
File: exec-multiple-codes.py
from __future__ import print_function
import math
import platform
import sys
import time
def exec_(code, scope):
# A known workaround for the issue mentioned above
exec (code, scope)
def main():
# Some parameters
exec_cnt_btwn_time_point = 1000
code_cnt = int(sys.argv[1])
exec_cnt_per_code = 100000
exec_cnt_total = exec_cnt_per_code * code_cnt
round_cnt = int(math.ceil(float(
exec_cnt_total)/exec_cnt_btwn_time_point))
# We compile some pieces of code here.
codes = [compile('sum(range(100))', '<string>',
'exec') for _ in range(code_cnt)]
exec_cnt_current = 0
elapsed = []
# We repeatedly iterate through `codes`, and `exec` its elements.
# We time every 1000 exec statements.
for _ in range(round_cnt): # Make sure that every piece of code is executed at least 100000 times
start = time.time()
for _ in range(exec_cnt_btwn_time_point):
exec_(codes[exec_cnt_current%code_cnt], {})
exec_cnt_current += 1
end = time.time()
elapsed.append((end-start)*1e6/exec_cnt_btwn_time_point) # 1 s == 1e6 us
# Store `elapsed` in a file. Needed when drawing graphs.
with open('results/{}-{}'.format(code_cnt,
platform.python_implementation()), 'w') as f:
print(*elapsed, sep='\n', file=f)
if __name__ == '__main__':
main()
File: draw-results.py (skip reading this if you only want to reproduce results)
from pathlib import Path
import matplotlib.pyplot as plt
def sort_key(s: Path) -> int:
s = s.name
k = int(s.split('-')[0])
if s.endswith('-CPython'):
k += 1000000
return k
def main() -> None:
for p in sorted(Path('results').iterdir(), key=sort_key):
ys = [float(x) for x in p.read_text().split()]
if len(ys) >= 400:
ys = ys[::-len(ys)//400][::-1]
xs = [(x+1)/len(ys)*100000 for x in range(len(ys))]
plt.plot(xs, ys, linewidth=1, label=p.name)
plt.legend()
plt.savefig('fig.png', dpi=625)
if __name__ == '__main__':
main()
We write a shell script and run it:
set -e
mkdir results
pypy exec-multiple-codes.py 1
pypy exec-multiple-codes.py 16
pypy exec-multiple-codes.py 32
pypy exec-multiple-codes.py 64
pypy exec-multiple-codes.py 128
pypy exec-multiple-codes.py 256
pypy exec-multiple-codes.py 512
python exec-multiple-codes.py 16
python exec-multiple-codes.py 512
python3 draw-results.py
mv results/ results-jit-on/
mv fig.png fig-jit-on.png
mkdir results
pypy --jit off exec-multiple-codes.py 1
pypy --jit off exec-multiple-codes.py 16
pypy --jit off exec-multiple-codes.py 32
pypy --jit off exec-multiple-codes.py 64
python3 draw-results.py
mv results/ results-jit-off/
mv fig.png fig-jit-off.png
Then we get two graphs:
X axis means currently how many times each piece of code is executed, and Y axis means the average elapsed time for the last 1000 exec statements. We can see that when the number of pieces of code increases, PyPy will get slower.