Why we ended up discontinuing PyPy usage
I first want to say I'm very grateful for the PyPy project. I think it shows how valuable a JIT compiler can be for Python code, and I hope it continues to grow.
I just wanted to leave a message about why my team has decided to stop using/supporting PyPy3 at this time, in hopes that the feedback is useful for PyPy developers. Again, thank you for working on this project.
For us, ultimately it came down to too many small maintenance paper cuts to justify the extra speed we received from PyPy. Our project is a Django/DRF application where we saw JSON serialization speed ups of 2-3x using PyPy3 for deeply nested JSON serialization.
The maintenance papercuts that added up are:
-
pyscogp2cffi
is not well maintained. We use PostgreSQL and this was worrying to our team for a long term maintenance. It wasn't clear if we could usepsycopg-binary
and if there would be a performance loss.-
pyscogp2cffi
usessetup_requires
, which means it pulls packages in a way that cannot be controlled viapip
configuration
-
- No Pillow wheels for PyPy. Installing from
sdist
added extra complications of needing system libraries installed on Linux and macOS.- We also never got PyPy working on Windows for our project because of this and other sdist related issues.
- PyPy is not compatible with
mypy
, meaning we had to have a separate env that only ran under CPython to runmypy
- Latest PyPy macOS binary not working. We have developers on macOS, Linux, and Windows.
- PyPy Docker image sometimes out of date
- A strange (never could figure this issue out) issue arose recently where we could no longer debug pytest/Django unit tests via PyCharm using the PyPy interpreter. We couldn't determine if this was due to a Django/pytest/pytest-django update, a PyCharm update, or something else but we got
frozendict
/update
exceptions any time we tried to debug our unit tests. - Less of an issue, but just generally wanting to use newer Python 3.7/3.8 language features such as
__future__ annotations
anddataclasses
.- If we
pip-tools compile
on CPython 3.7, it would erase thedataclasses; python_version < '3.7'
requirement for when PyPy3.6 installed.
- If we
Again, I'm writing this in the hopes that it helps shed light on the kinds of papercuts that can arise. A single one of these would not have been a show-stopper, but it's more the aggregate of them.
I'm excited for HPy as that would help solve some of the packaging issues we ran into. Best of luck, and please feel free to close this issue.