pypy binary is (now) linked to libbz2, libffi, and libtinfo, which could break manylinux wheels
Created originally on Bitbucket by njs (Nathaniel Smith)
With pypy 5.8, ldd pypy
shows that it's linked to libbz2
, libcrypto
, libffi
, libncurses
, ... Likewise for pypy3 5.8 and ldd pypy3.5
.
You would not think so, but it turns out that this is a big problem for distributing wheels.
The issue is that the way ELF works, any libraries that show up in ldd $TOPLEVELBINARY
effectively get LD_PRELOADed into any extension modules that you load later. So, for example, if some wheel distributes its own version of openssl, then any symbols that show up in both their copy of openssl and pypy's copy of openssl will get shadowed and hello segfaults.
The cryptography project recently ran into this with uwsgi: https://github.com/pyca/cryptography/issues/3804#issuecomment-317401627
Fortunately this has not been a big deal so far because, uh... nobody distributes pypy wheels. But in the future maybe this is something that should be supported :-). And while in theory it would be nice if this could be fixed on the wheel side, this is not trivial.
The obvious solution would be to switch things around so that the top-level pypy executable does dlopen("libpypy-c.so", RTLD_LOCAL)
to start the interpreter, instead of linking against it with -lpypy-c
. Then the symbols from libpypy-c.so
and everything it links to would be confined to an ELF local namespace, and would stop polluting the namespace of random extension modules.
However... there is a problem, which is that cpyext extension modules need some way to get at the C API symbols, and I assume cffi extension modules need access to some pypy symbols as well.
This is... tricky, given how rpython wants to mush everything together into one giant .so, and ELF makes it difficult to only expose some symbols from a binary like this. Some options:
- when using libcrypto or whatever from rpython, use
dlopen("libcrypto", RTLD_LOCAL)
instead of-lcrypto
. I guess this could be done systematically in rffi? - provide a special
libcpyext
that usesdlopen
to fetch the symbols fromlibpypy-c.so
and then manually re-exports them?