Installation from PyPI on PyPy is a no-op
Created originally on Bitbucket by John Vandenberg
As described in the docs, PyPy includes this package. I didnt see any information in the docs regarding how/why/etc. It appears that _cffi_backend
is a modified codebase built into the pypy binary, but cffi
is an unmodified copy of the real cffi
package which is provided by pypy build process, because PyPy then uses cffi to provide stdlib modules such as ssl & curses.
However setup.py
intentionally does nothing, with this comment and guard:
# On PyPy, cffi is preinstalled and it is not possible, at least for now,
# to install a different version. We work around it by making the setup()
# arguments mostly empty in this case.
cpython = ('_cffi_backend' not in sys.builtin_module_names)
This needs to be improved upon, and I couldn't see an open issue for it. https://bitbucket.org/cffi/cffi/issues/371/cffi-should-specify-entry-points-for-pypy improved the situation a bit, but more is needed.
Two main reasons for improving it:
- If another package specifies
cffi >= x.y.z
, it expects a certain version, and pip will fetch it and attempt to install it, and expect that it is updated - Hacking cffi code as part of PyPy is not easy, as the PyPy build process is huge. Doing
pypy3 setup.py build
in cffi codebase is much easier.
The second one is easy to workaround, by manually skipping the guard above. Adding an envvar to do this would be nice.
But more fundamentally, while _cffi_backend
probably cant be replaced on PyPy (yet), cffi
can be, and it should be built and distributed by cffi
. There is already another guard to check that the API version in _cffi_backend
is the same as in cffi
.
PyPy's copy does include a cffi.egg-info
, at least in v7.1.1 , and the PKG-INFO says it includes cffi version 1.12.3 , currently the latest. I see very little differences between their cffi
code and the tip
here. The existence of pre-installed cffi.egg-info
means that the setup.py
could inspect it to make informed decisions about how to do 'the right thing' on PyPy, whatever that should be.
I think the first step is the following change, and fixing any side-effects.
diff -r 8976fd752614 setup.py
--- a/setup.py Fri Sep 20 18:14:40 2019 +0200
+++ b/setup.py Thu Sep 26 09:40:19 2019 +0700
@@ -199,10 +199,10 @@
`Mailing list <https://groups.google.com/forum/#!forum/python-cffi>`_
""",
version='1.13.0',
- packages=['cffi'] if cpython else [],
+ packages=['cffi'],
package_data={'cffi': ['_cffi_include.h', 'parse_c_type.h',
'_embedding.h', '_cffi_errors.h']}
- if cpython else {},
+ ,
zip_safe=False,
url='http://cffi.readthedocs.org',
Then only _cffi_backend
is not built.
With the above, and a quick hack to enable the tests (inc. disabling the _backend_cffi api version check), most tests are passing when running pypy3 -m pytest testing
diff -r 8976fd752614 cffi/api.py
--- a/cffi/api.py Fri Sep 20 18:14:40 2019 +0200
+++ b/cffi/api.py Thu Sep 26 11:07:58 2019 +0700
@@ -47,7 +47,7 @@
# _cffi_backend.so compiled.
import _cffi_backend as backend
from . import __version__
- if backend.__version__ != __version__:
+ if False: #backend.__version__ != __version__:
# bad version! Try to be as explicit as possible.
if hasattr(backend, '__file__'):
# CPython
diff -r 8976fd752614 testing/cffi1/test_parse_c_type.py
--- a/testing/cffi1/test_parse_c_type.py Fri Sep 20 18:14:40 2019 +0200
+++ b/testing/cffi1/test_parse_c_type.py Thu Sep 26 10:20:06 2019 +0700
@@ -2,7 +2,7 @@
import cffi
from cffi import cffi_opcode
-if '__pypy__' in sys.builtin_module_names:
+if False: # '__pypy__' in sys.builtin_module_names:
try:
# pytest >= 4.0
py.test.skip("not available on pypy", allow_module_level=True)
I have 12 failures, but most appear to be because i am using a distro pypy3 rather than installer provided by PyPy - two tests failures look real
___________________________________________________________________________________ test_from_buffer_struct ____________________________________________________________________________________
def test_from_buffer_struct():
ffi = FFI()
ffi.cdef("""struct foo_s { int a, b; };""")
lib = verify(ffi, "test_from_buffer_struct_p", """
struct foo_s { int a, b; };
""")
p = ffi.new("struct foo_s *", [-219239, 58974983])
q = ffi.from_buffer("struct foo_s[]", ffi.buffer(p))
assert ffi.typeof(q) == ffi.typeof("struct foo_s[]")
assert len(q) == 1
assert q[0].a == p.a
assert q[0].b == p.b
assert q == p
> q = ffi.from_buffer("struct foo_s *", ffi.buffer(p))
E TypeError: expected an array ctype, got 'struct foo_s *'
testing/cffi1/test_recompiler.py:2355: TypeError
------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------
generating /tmp/ffi-4/_CFFI_test_from_buffer_struct_p.cpp
setting the current directory to '/tmp/ffi-4'
running build_ext
building '_CFFI_test_from_buffer_struct_p' extension
gcc -pthread -DNDEBUG -O2 -fPIC -UNDEBUG -I/usr/lib64/pypy-7.1.1/include -c _CFFI_test_from_buffer_struct_p.cpp -o ./_CFFI_test_from_buffer_struct_p.o
g++ -pthread -shared ./_CFFI_test_from_buffer_struct_p.o -o ./_CFFI_test_from_buffer_struct_p.pypy3-71-x86_64-linux-gnu.so
___________________________________________________________________________________ test_unnamed_bitfield_2 ____________________________________________________________________________________
def test_unnamed_bitfield_2():
ffi = FFI()
ffi.cdef("""struct A {
short c : 1; short : 1; short d : 1; short : 1; };""")
lib = verify(ffi, "test_unnamed_bitfield_2", """
struct A {
short c : 1; short : 1; short d : 1; short : 1;
};
""")
> p = ffi.new("struct A *")
E KeyError: "duplicate field name ''"
testing/cffi1/test_recompiler.py:2384: KeyError
------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------
generating /tmp/ffi-4/_CFFI_test_unnamed_bitfield_2.cpp
setting the current directory to '/tmp/ffi-4'
running build_ext
building '_CFFI_test_unnamed_bitfield_2' extension
gcc -pthread -DNDEBUG -O2 -fPIC -UNDEBUG -I/usr/lib64/pypy-7.1.1/include -c _CFFI_test_unnamed_bitfield_2.cpp -o ./_CFFI_test_unnamed_bitfield_2.o
g++ -pthread -shared ./_CFFI_test_unnamed_bitfield_2.o -o ./_CFFI_test_unnamed_bitfield_2.pypy3-71-x86_64-linux-gnu.so
___________________________________________________________________________________ test_unnamed_bitfield_3 ____________________________________________________________________________________
def test_unnamed_bitfield_3():
ffi = FFI()
ffi.cdef("""struct A { struct { char : 1; char : 1; } b; };""")
lib = verify(ffi, "test_unnamed_bitfield_3", """
struct A { struct { char : 1; char : 1; } b; };
""")
> p = ffi.new("struct A *")
E KeyError: "duplicate field name ''"
testing/cffi1/test_recompiler.py:2393: KeyError
------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------
generating /tmp/ffi-4/_CFFI_test_unnamed_bitfield_3.cpp
setting the current directory to '/tmp/ffi-4'
running build_ext
building '_CFFI_test_unnamed_bitfield_3' extension
gcc -pthread -DNDEBUG -O2 -fPIC -UNDEBUG -I/usr/lib64/pypy-7.1.1/include -c _CFFI_test_unnamed_bitfield_3.cpp -o ./_CFFI_test_unnamed_bitfield_3.o
g++ -pthread -shared ./_CFFI_test_unnamed_bitfield_3.o -o ./_CFFI_test_unnamed_bitfield_3.pypy3-71-x86_64-linux-gnu.so
___________________________________________________________________________________ test_unnamed_bitfield_4 ____________________________________________________________________________________
def test_unnamed_bitfield_4():
ffi = FFI()
ffi.cdef("""struct A { struct {
unsigned c : 1; unsigned : 1; unsigned d : 1; unsigned : 1; } a;
};
struct B { struct A a; };""")
lib = verify(ffi, "test_unnamed_bitfield_4", """
struct A { struct {
unsigned c : 1; unsigned : 1; unsigned d : 1; unsigned : 1; } a;
};
struct B { struct A a; };
""")
> b = ffi.new("struct B *")
E KeyError: "duplicate field name ''"
testing/cffi1/test_recompiler.py:2411: KeyError
------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------
generating /tmp/ffi-4/_CFFI_test_unnamed_bitfield_4.cpp
setting the current directory to '/tmp/ffi-4'
running build_ext
building '_CFFI_test_unnamed_bitfield_4' extension
gcc -pthread -DNDEBUG -O2 -fPIC -UNDEBUG -I/usr/lib64/pypy-7.1.1/include -c _CFFI_test_unnamed_bitfield_4.cpp -o ./_CFFI_test_unnamed_bitfield_4.o
g++ -pthread -shared ./_CFFI_test_unnamed_bitfield_4.o -o ./_CFFI_test_unnamed_bitfield_4.pypy3-71-x86_64-linux-gnu.so
_____________________________________________________________________________________ TestBasic.test_basic _____________________________________________________________________________________
self = <testing.embedding.test_basic.TestBasic object at 0x00007fb57c6d53d0>
def test_basic(self):
add1_cffi = self.prepare_module('add1')
self.compile('add1-test', [add1_cffi])
output = self.execute('add1-test')
> assert output == ("preparing...\n"
"adding 40 and 2\n"
"adding 100 and -5\n"
"got: 42 95\n")
E AssertionError: assert 'got: 0 0\n' == 'preparing......ngot: 42 95\n'
E - got: 0 0
E + preparing...
E + adding 40 and 2
E + adding 100 and -5
E + got: 42 95
testing/embedding/test_basic.py:188: AssertionError
------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------
* setting env var 'LD_LIBRARY_PATH' to '/usr/bin:/tmp/ffi-4/embedding/test_basic'
RUNNING: ['/usr/bin/pypy3', '/tmp/ffi-4/embedding/test_basic/add1.py'] {'cwd': '/tmp/ffi-4/embedding/test_basic', 'stdout': -1, 'universal_newlines': True}
generating ./_add1_cffi.c
the current directory is '/tmp/ffi-4/embedding/test_basic'
running build_ext
building '_add1_cffi' extension
gcc -pthread -DNDEBUG -O2 -fPIC -I/usr/lib64/pypy-7.1.1/include -c _add1_cffi.c -o ./_add1_cffi.o
gcc -pthread -shared ./_add1_cffi.o -L/usr/lib/pypy-7.1.1/bin -L/usr/lib/pypy-7.1.1/pypy/goal -lpypy3-c -o ./_add1_cffi.so
FILENAME: /tmp/ffi-4/embedding/test_basic/_add1_cffi.so
compiling add1-test with ['/tmp/ffi-4/embedding/test_basic/_add1_cffi.so']
running 'add1-test' in '/tmp/ffi-4/embedding/test_basic'
RUNNING: ['./add1-test'] {'cwd': '/tmp/ffi-4/embedding/test_basic', 'stdout': -1, 'universal_newlines': True}
It shouldn't be difficult to get those passing, or skip them temporarily in order to get the CI in place.
The only hack I think we need is a way to bypass the API check, because the PyPy backend will often be x.y.z
when the development version of this package is x.y+1.0.dev0
. If we could add .dev0
to the actual version string whilst in development, then a specific new rule can be added for "previous minor release" only when in development.