Pypy3: Slow to run on virtual machines
Overview
I am a developer for the Pyomo python package, and we test our package with multiple distributions of CPython and Pypy3 using GitHub Actions. The average time to completion of the test suite on Linux is 13 minutes for all CPython distributions. For pypy3, however, the average time to completion is 60-80 minutes.
After tracking down perceived performance issues in our code's implementation, we happened to run the test suite on a bare-metal (non-virtual) machine, and the time to completion was extremely fast - half the time of the CPython distributions.
We tested further on several other virtual and non-virtual machines and received consistent results - pypy3 is extremely slow to run on virtual machines but blazingly fast on non-virtual machines.
Expected Behavior
We expect that virtual machines will always be slower, no matter the Python distribution. Our expected behavior for Pypy is that it maintains the same speed gains, though, regardless of virtual vs. bare metal machines. Running a standard Pyomo example, Pypy is moderately slower than CPython 3.7.9 on VMs; however, Pypy is faster by a factor of 3 on the bare-metal, non-virtual machine.
Please see detailed timing differences below:
Virtual Machine Runtimes
With CPython 3.7.9:
% python
Python 3.7.9 (default, Jan 13 2021, 10:38:36)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
% time pyomo convert pmedian.py pmedian.test8.dat --output tmp.lp --report-timing
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.00] Creating model
0.00 seconds required to construct instance
0 seconds to construct Param N; 1 index total
0 seconds to construct RangeSet Locations; 1 index total
0 seconds to construct RangeSet P_domain; 1 index total
0 seconds to construct Param P; 1 index total
0 seconds to construct Param M; 1 index total
0 seconds to construct RangeSet Customers; 1 index total
0 seconds to construct Set d_index; 1 index total
0 seconds to construct Set d_index; 1 index total
2.48 seconds to construct Param d; 409600 indices total
0 seconds to construct Set x_index; 1 index total
0 seconds to construct Set x_index; 1 index total
1.78 seconds to construct Var x; 409600 indices total
0 seconds to construct Var y; 640 indices total
4.36 seconds to construct Objective obj; 1 index total
2.86 seconds to construct Constraint single_x; 640 indices total
0 seconds to construct Set bound_y_index; 1 index total
0 seconds to construct Set bound_y_index; 1 index total
7.33 seconds to construct Constraint bound_y; 409600 indices total
0 seconds to construct Constraint num_facilities; 1 index total
19.67 seconds required to construct instance
0.00 seconds required for problem transformations
Model written to file 'tmp.lp'
47.23 seconds required to write file
[ 66.90] Pyomo Finished
70.242u 1.367s 1:11.56 100.0% 0+0k 0+80784io 1pf+0w
With Pypy3.7:
% python
Python 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34)
[PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
% time pyomo convert pmedian.py pmedian.test8.dat --output tmp.lp --report-timing
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.00] Creating model
0.00 seconds required to construct instance
0 seconds to construct Param N; 1 index total
0 seconds to construct RangeSet Locations; 1 index total
0 seconds to construct RangeSet P_domain; 1 index total
0 seconds to construct Param P; 1 index total
0 seconds to construct Param M; 1 index total
0 seconds to construct RangeSet Customers; 1 index total
0 seconds to construct Set d_index; 1 index total
0 seconds to construct Set d_index; 1 index total
2.55 seconds to construct Param d; 409600 indices total
0 seconds to construct Set x_index; 1 index total
0 seconds to construct Set x_index; 1 index total
1.88 seconds to construct Var x; 409600 indices total
0 seconds to construct Var y; 640 indices total
5.05 seconds to construct Objective obj; 1 index total
3.83 seconds to construct Constraint single_x; 640 indices total
0 seconds to construct Set bound_y_index; 1 index total
0 seconds to construct Set bound_y_index; 1 index total
9.28 seconds to construct Constraint bound_y; 409600 indices total
0 seconds to construct Constraint num_facilities; 1 index total
23.54 seconds required to construct instance
0.00 seconds required for problem transformations
Model written to file 'tmp.lp'
47.39 seconds required to write file
[ 70.93] Pyomo Finished
74.736u 1.302s 1:15.98 100.0% 0+0k 0+80744io 1pf+0w
Non-virtual Machine Runtimes
With CPython 3.7.9:
% python
Python 3.7.9 (default, Jan 13 2021, 10:38:36)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
% time pyomo convert pmedian.py pmedian.test8.dat --output tmp.lp --report-timing
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.00] Creating model
0.00 seconds required to construct instance
0 seconds to construct Param N; 1 index total
0 seconds to construct RangeSet Locations; 1 index total
0 seconds to construct RangeSet P_domain; 1 index total
0 seconds to construct Param P; 1 index total
0 seconds to construct Param M; 1 index total
0 seconds to construct RangeSet Customers; 1 index total
0 seconds to construct Set d_index; 1 index total
0 seconds to construct Set d_index; 1 index total
0.66 seconds to construct Param d; 409600 indices total
0 seconds to construct Set x_index; 1 index total
0 seconds to construct Set x_index; 1 index total
0.49 seconds to construct Var x; 409600 indices total
0 seconds to construct Var y; 640 indices total
1.27 seconds to construct Objective obj; 1 index total
0.79 seconds to construct Constraint single_x; 640 indices total
0 seconds to construct Set bound_y_index; 1 index total
0 seconds to construct Set bound_y_index; 1 index total
2.08 seconds to construct Constraint bound_y; 409600 indices total
0 seconds to construct Constraint num_facilities; 1 index total
5.58 seconds required to construct instance
0.00 seconds required for problem transformations
Model written to file 'tmp.lp'
12.27 seconds required to write file
[ 17.86] Pyomo Finished
21.169u 1.261s 0:21.57 103.9% 0+0k 416+81176io 0pf+0w
With Pypy3:
% python
Python 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34)
[PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
% time pyomo convert pmedian.py pmedian.test8.dat --output tmp.lp --report-timing
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.00] Creating model
0.00 seconds required to construct instance
0 seconds to construct Param N; 1 index total
0 seconds to construct RangeSet Locations; 1 index total
0 seconds to construct RangeSet P_domain; 1 index total
0 seconds to construct Param P; 1 index total
0 seconds to construct Param M; 1 index total
0 seconds to construct RangeSet Customers; 1 index total
0 seconds to construct Set d_index; 1 index total
0 seconds to construct Set d_index; 1 index total
0.12 seconds to construct Param d; 409600 indices total
0 seconds to construct Set x_index; 1 index total
0 seconds to construct Set x_index; 1 index total
0.17 seconds to construct Var x; 409600 indices total
0 seconds to construct Var y; 640 indices total
0.27 seconds to construct Objective obj; 1 index total
0.16 seconds to construct Constraint single_x; 640 indices total
0 seconds to construct Set bound_y_index; 1 index total
0 seconds to construct Set bound_y_index; 1 index total
0.33 seconds to construct Constraint bound_y; 409600 indices total
0.01 seconds to construct Constraint num_facilities; 1 index total
1.10 seconds required to construct instance
0.00 seconds required for problem transformations
Model written to file 'tmp.lp'
5.58 seconds required to write file
[ 6.68] Pyomo Finished
7.677u 0.785s 0:09.15 92.3% 0+0k 280+81248io 0pf+0w