mechanism for global library initialization in the presence of subinterpreters
Created originally on Bitbucket by Glyph
Cryptography has been consistently plagued by the problem of subinterpreters.
In CPython, you can create a "subinterpreter", which is an entirely new namespace for modules to be imported. However, critically, this new namespace does not include a new namespace for extension modules; all extension modules and all their shared libraries are assumed to be global.
Cryptography needs to initialize its backends at import time. Speaking generally and hypothetically, this could be any number of shared libraries; speaking specifically, it's OpenSSL, because of course it's OpenSSL. OpenSSL needs to have some global state, which points at Python callbacks, filled out at library initialization time.
It's presently (maybe?) possible to hack this together by using imp.acquire_lock
, but this API is deprecated, and not all global state in libraries is necessarily initialized or manipulated exclusively at import time. Relying implicitly on the interpreter lock in python 2.7 has not thus far been robust, since bug reports continue to flow in.
Can CFFI expose a static-library-initialization lock that can be used for these circumstances so that it is unambiguously shared between all subinterpreters? Is there already a mechanism for doing this? Would perhaps a decorator saying "don't release the GIL when I call this" be a possible way forward?