Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
pypy
pypy
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 582
    • Issues 582
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 12
    • Merge Requests 12
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards

This instance will be upgraded to Heptapod 0.19.0rc2 on 2021-01-18 at 11:00 UTC+1 (a few minutes of downtime)

  • PyPy
  • pypypypy
  • Issues
  • #3294

Closed
Open
Opened Sep 03, 2020 by Chitanda@chitanda-ydteb

Performance issue with the exec statement

My program, written in Python 2, needs to dynamically execute Python code from user input via the exec statement, and I tried to use PyPy to speed it up. However, I found that the program runs much slower under PyPy than under CPython. After investigating, I have found several PyPy performance issue with the exec statement.

Environment information and PoC are as below. Any help or explanation will be greatly appreciated.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 9.11 (stretch)
Release:	9.11
Codename:	stretch
$ python --version
Python 2.7.13
$ python3 --version
Python 3.5.3
$ pypy --version
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
[PyPy 5.6.0 with GCC 6.2.0 20161109]

Functions with exec statements

File: function-with-exec.py

import sys
import time


def call1():
    return
    exec ''  # Unreachable.


def call2():
    if True:
        return
    exec ''  # Still unreachable in fact, but `call2` is much slower than `call1` under PyPy.


def main():
    if sys.argv[1] == '1':
        s = time.time()
        for _ in range(10000000):
            call1()
        e = time.time()
        print(e-s)
    elif sys.argv[1] == '2':
        s = time.time()
        for _ in range(10000000):
            call2()
        e = time.time()
        print(e-s)


if __name__ == '__main__':
    main()

We run this script in different ways:

$ pypy function-with-exec.py 1
0.0103108882904
$ pypy function-with-exec.py 2
18.2597911358
$ pypy --jit off function-with-exec.py 1
3.22720384598
$ pypy --jit off function-with-exec.py 2
3.86836504936
$ python function-with-exec.py 1
1.49009108543
$ python function-with-exec.py 2
1.53774189949

Dynamically execute multiple pieces of code in a short time

File: exec-multiple-codes.py

from __future__ import print_function

import math
import platform
import sys
import time


def exec_(code, scope):
    # A known workaround for the issue mentioned above
    exec (code, scope)


def main():

    # Some parameters
    exec_cnt_btwn_time_point = 1000
    code_cnt = int(sys.argv[1])
    exec_cnt_per_code = 100000
    exec_cnt_total = exec_cnt_per_code * code_cnt
    round_cnt = int(math.ceil(float(
        exec_cnt_total)/exec_cnt_btwn_time_point))

    # We compile some pieces of code here.
    codes = [compile('sum(range(100))', '<string>',
        'exec') for _ in range(code_cnt)]
    exec_cnt_current = 0
    elapsed = []

    # We repeatedly iterate through `codes`, and `exec` its elements.
    # We time every 1000 exec statements.
    for _ in range(round_cnt):  # Make sure that every piece of code is executed at least 100000 times
        start = time.time()
        for _ in range(exec_cnt_btwn_time_point):
            exec_(codes[exec_cnt_current%code_cnt], {})
            exec_cnt_current += 1
        end = time.time()
        elapsed.append((end-start)*1e6/exec_cnt_btwn_time_point)  # 1 s == 1e6 us

    # Store `elapsed` in a file. Needed when drawing graphs.
    with open('results/{}-{}'.format(code_cnt,
            platform.python_implementation()), 'w') as f:
        print(*elapsed, sep='\n', file=f)


if __name__ == '__main__':
    main()

File: draw-results.py (skip reading this if you only want to reproduce results)

from pathlib import Path

import matplotlib.pyplot as plt


def sort_key(s: Path) -> int:
    s = s.name
    k = int(s.split('-')[0])
    if s.endswith('-CPython'):
        k += 1000000
    return k


def main() -> None:
    for p in sorted(Path('results').iterdir(), key=sort_key):
        ys = [float(x) for x in p.read_text().split()]
        if len(ys) >= 400:
            ys = ys[::-len(ys)//400][::-1]
        xs = [(x+1)/len(ys)*100000 for x in range(len(ys))]
        plt.plot(xs, ys, linewidth=1, label=p.name)
    plt.legend()
    plt.savefig('fig.png', dpi=625)


if __name__ == '__main__':
    main()

We write a shell script and run it:

set -e

mkdir results
pypy exec-multiple-codes.py 1
pypy exec-multiple-codes.py 16
pypy exec-multiple-codes.py 32
pypy exec-multiple-codes.py 64
pypy exec-multiple-codes.py 128
pypy exec-multiple-codes.py 256
pypy exec-multiple-codes.py 512
python exec-multiple-codes.py 16
python exec-multiple-codes.py 512
python3 draw-results.py
mv results/ results-jit-on/
mv fig.png fig-jit-on.png

mkdir results
pypy --jit off exec-multiple-codes.py 1
pypy --jit off exec-multiple-codes.py 16
pypy --jit off exec-multiple-codes.py 32
pypy --jit off exec-multiple-codes.py 64
python3 draw-results.py
mv results/ results-jit-off/
mv fig.png fig-jit-off.png

Then we get two graphs:

fig-jit-on

fig-jit-off

X axis means currently how many times each piece of code is executed, and Y axis means the average elapsed time for the last 1000 exec statements. We can see that when the number of pieces of code increases, PyPy will get slower.

Edited Sep 03, 2020 by Chitanda
To upload designs, you'll need to enable LFS and have admin enable hashed storage. More information
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: pypy/pypy#3294