Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • pypy pypy
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 677
    • Issues 677
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 13
    • Merge requests 13
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • PyPyPyPy
  • pypypypy
  • Issues
  • #3294
Closed
Open
Issue created Sep 03, 2020 by Chitanda@chitanda-ydteb

Performance issue with the exec statement

My program, written in Python 2, needs to dynamically execute Python code from user input via the exec statement, and I tried to use PyPy to speed it up. However, I found that the program runs much slower under PyPy than under CPython. After investigating, I have found several PyPy performance issue with the exec statement.

Environment information and PoC are as below. Any help or explanation will be greatly appreciated.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.11 (stretch)
Release:	9.11
Codename:	stretch
$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3
$ pypy --version
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
[PyPy 5.6.0 with GCC 6.2.0 20161109]

Functions with exec statements

File: function-with-exec.py

import sys
import time


def call1():
    return
    exec ''  # Unreachable.


def call2():
    if True:
        return
    exec ''  # Still unreachable in fact, but `call2` is much slower than `call1` under PyPy.


def main():
    if sys.argv[1] == '1':
        s = time.time()
        for _ in range(10000000):
            call1()
        e = time.time()
        print(e-s)
    elif sys.argv[1] == '2':
        s = time.time()
        for _ in range(10000000):
            call2()
        e = time.time()
        print(e-s)


if __name__ == '__main__':
    main()

We run this script in different ways:

$ pypy function-with-exec.py 1
0.0103108882904
$ pypy function-with-exec.py 2
18.2597911358
$ pypy --jit off function-with-exec.py 1
3.22720384598
$ pypy --jit off function-with-exec.py 2
3.86836504936
$ python function-with-exec.py 1
1.49009108543
$ python function-with-exec.py 2
1.53774189949

Dynamically execute multiple pieces of code in a short time

File: exec-multiple-codes.py

from __future__ import print_function

import math
import platform
import sys
import time


def exec_(code, scope):
    # A known workaround for the issue mentioned above
    exec (code, scope)


def main():

    # Some parameters
    exec_cnt_btwn_time_point = 1000
    code_cnt = int(sys.argv[1])
    exec_cnt_per_code = 100000
    exec_cnt_total = exec_cnt_per_code * code_cnt
    round_cnt = int(math.ceil(float(
        exec_cnt_total)/exec_cnt_btwn_time_point))

    # We compile some pieces of code here.
    codes = [compile('sum(range(100))', '<string>',
        'exec') for _ in range(code_cnt)]
    exec_cnt_current = 0
    elapsed = []

    # We repeatedly iterate through `codes`, and `exec` its elements.
    # We time every 1000 exec statements.
    for _ in range(round_cnt):  # Make sure that every piece of code is executed at least 100000 times
        start = time.time()
        for _ in range(exec_cnt_btwn_time_point):
            exec_(codes[exec_cnt_current%code_cnt], {})
            exec_cnt_current += 1
        end = time.time()
        elapsed.append((end-start)*1e6/exec_cnt_btwn_time_point)  # 1 s == 1e6 us

    # Store `elapsed` in a file. Needed when drawing graphs.
    with open('results/{}-{}'.format(code_cnt,
            platform.python_implementation()), 'w') as f:
        print(*elapsed, sep='\n', file=f)


if __name__ == '__main__':
    main()

File: draw-results.py (skip reading this if you only want to reproduce results)

from pathlib import Path

import matplotlib.pyplot as plt


def sort_key(s: Path) -> int:
    s = s.name
    k = int(s.split('-')[0])
    if s.endswith('-CPython'):
        k += 1000000
    return k


def main() -> None:
    for p in sorted(Path('results').iterdir(), key=sort_key):
        ys = [float(x) for x in p.read_text().split()]
        if len(ys) >= 400:
            ys = ys[::-len(ys)//400][::-1]
        xs = [(x+1)/len(ys)*100000 for x in range(len(ys))]
        plt.plot(xs, ys, linewidth=1, label=p.name)
    plt.legend()
    plt.savefig('fig.png', dpi=625)


if __name__ == '__main__':
    main()

We write a shell script and run it:

set -e

mkdir results
pypy exec-multiple-codes.py 1
pypy exec-multiple-codes.py 16
pypy exec-multiple-codes.py 32
pypy exec-multiple-codes.py 64
pypy exec-multiple-codes.py 128
pypy exec-multiple-codes.py 256
pypy exec-multiple-codes.py 512
python exec-multiple-codes.py 16
python exec-multiple-codes.py 512
python3 draw-results.py
mv results/ results-jit-on/
mv fig.png fig-jit-on.png

mkdir results
pypy --jit off exec-multiple-codes.py 1
pypy --jit off exec-multiple-codes.py 16
pypy --jit off exec-multiple-codes.py 32
pypy --jit off exec-multiple-codes.py 64
python3 draw-results.py
mv results/ results-jit-off/
mv fig.png fig-jit-off.png

Then we get two graphs:

fig-jit-on

fig-jit-off

X axis means currently how many times each piece of code is executed, and Y axis means the average elapsed time for the last 1000 exec statements. We can see that when the number of pieces of code increases, PyPy will get slower.

Edited Sep 03, 2020 by Chitanda
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking