issue with Py_LIMITED_API on Windows
Created originally on Bitbucket by anthrotype (Cosimo Lupo)
When building a cffi extension module on Windows in API mode (i.e. set_source), instead of linking with the PYTHON3.DLL associated with the stable Py_LIMITED_API, the cffi extension module gets linked with the version-specific PYTHON35.DLL or PYTHON36.DLL.
Therefore, the module fails to be imported on a different CPython 3 minor version, even if the file name has a generic *.pyd suffix instead of the full *.cp36-win_amd64.pyd suffix. The generic filename is because cffi forces the use of setuptools' Extension py_limited_api=True option. However it is incorrect, because if one inspects the *.pyd file with Dependency Walker, one can see that the versioned PYTHON36.DLL is being linked.
The reason for this is that the name of the python DLL import library that is used when linking an extension module compiled with MSVC is not passed by distutils to the linker command line, but it is implicitly defined inside the pyconfig.h header via a #pragma directive, and this is based on the current value of Py_LIMITED_API:
The _cffi_include.h header attempts to #define Py_LIMITED_API but only after including the pyconfig.h header. By the time it includes the Python.h, it is too late, since the pyconfig.h has already been included once and it is guarded by a #ifndef Py_CONFIG_H at the top, so the second time it's included by Python.h nothing happens (and the wrong python DLL gets linked).
The only workaround I could find to force the cffi extension module to be linked with PYTHON3.DLL instead of PYTHON36.DLL is to explicitly pass a define_macros=[("Py_LIMITED_API", None)] to the ffibuilder.set_source call, which will then forward the preprocessor definition to setuptools/distutils when the latter build up the linker command line.
When doing that, the _cffi_include.h's #if branch is skipped because Py_LIMITED_API was already defined.
It's a pity that one has to set that with define_macros in order to make it work on Windows. On Linux or macOS it doesn't seem necessary to do that.