cypext: behavior of metaclasses created in the C API
Dear PyPy team (& @mattip),
I am making steady progress to get nanobind to work with PyPy. As far as I can tell, there is only one major discrepancy left after #3844 (closed) and #3845 (closed), and that is something more subtle: the behavior of metaclasses created using the C API.
Nanobind uses a metaclass to create its own types. The type objects created by this metaclass are larger than a normal heap types. Something like this:
typedef struct {
PyHeapTypeObject ht;
uint8_t payload[2048];
} ExtendedType;
The payload is much smaller in practice, I am just copying the example used in the reproducer (https://github.com/wjakob/pypy_issues/blob/master/pypy_issues.c#L114).
Background ("why would you do that??"): nanobind stashes some information in this payload area to translate between the C++ and Python world. pybind11 uses a less intrusive approach: it maintains a separate hash table to map from PyTypeObject* to similar payload area. But this means that it very often needs to perform lookups in that hash table, and this contribute to its relative inefficiency of pybind11 compared to nanobind. Co-locating that data directly with the Python type object makes everything simpler + faster.
In the case of PyPy, I think that I am entering a kind of exotic corner of cpyext and running into some limitations. For starters, if you run
from pypy_issues import metaclass as meta
class MyClass(object, metaclass=meta):
pass
there is an obscure exception that I don't understand:
$ pypy issue_4.py
Traceback (most recent call last):
File "/home/wjakob/pypy_issues/issue_4.py", line 3, in <module>
class MyClass(object, metaclass=meta):
TypeError: metaclass found to be 'metaclass', but calling <class 'pypy_issues.metaclass'> with args ('MyClass', (<class 'object'>,), ...) raised "'metaclass' has no length"
If this refers to __len__
(?), then it should in principle be inherited automatically when the metaclass is created via PyType_Spec
. In any case, this test does the following in CPython:
$ python issue_4.py
Got to metaclass_init.
Assuming this length inheritance issue can be fixed, I believe that this reproducer will turn into another reproducer of an unrelated issue, which is that a memset
in the implementation of metaclass_init
(see below)...
int metaclass_init(PyObject *self, PyObject *args, PyObject *kwds) {
int rv = PyType_Type.tp_init(self, args, kwds);
printf("Got to metaclass_init.\n");
if (rv == 0)
memset(((ExtendedType *) self)->extra, 0, 2048);
return rv;
}
.. overwrites some PyPy-internal data structures because the actually allocated type object too small (likely it has sizeof(PyHeapTypeObject)
and not the requested sizeof(ExtendedType)
).
In nanobind, I am constructing instances of heap types directly from C using PyType_FromMetaclass
on Python 3.12 or a workaround for PyPy and older Python versions (https://github.com/wjakob/nanobind/blob/pypy/src/nb_type.cpp#L456), which allowed me to sidestep the "'metaclass' has no length" issue and create type instances directly.
However, things crash shortly afterwards when writing to the payload area triggers undefined behavior. I haven't added tests for this issue to the pypy_issues
repository because it involves quite a bit of (frankly ugly) code, but that's just because I believe that issue_4.py
will turn into a reproducer for that problem as well once PyPy can be convinced to call the metaclass_init
function.
Thanks, Wenzel