[Bug] The implementation of `functools.partial` in PyPy is different than CPython
The implementation of functools.partial
(actually _functools.partial
) in PyPy is different than CPython. Such as:
- The PyPy
_functools.partial
uses private attributes and properties to manageself.func
,self.args
,self.keywords
. This makes those attributes read-only, while CPython's implementation allows users to modify them. - The PyPy
_functools.partial
uses__init__
method to initialize the new instance, while CPython's implementation using the__new__
method. This breaks subclassingfunctools.partial
withsuper().__new__(...)
.
These breaks some libraries that rely on the implementation details of functools.partial
. And prohibits them add PyPy support. For example, JAX's pytree utilities subclass the functools.partial
:
class Partial(functools.partial):
def __new__(klass, func, *args, **kw):
# In Python 3.10+, if func is itself a functools.partial instance,
# functools.partial.__new__ would merge the arguments of this Partial
# instance with the arguments of the func. We box func in a class that does
# not (yet) have a `func` attribute to defeat this optimization, since we
# care exactly which arguments are considered part of the pytree.
if isinstance(func, functools.partial):
original_func = func
func = _HashableCallableShim(original_func)
out = super().__new__(klass, func, *args, **kw)
func.func = original_func.func
func.args = original_func.args
func.keywords = original_func.keywords
return out
else:
return super().__new__(klass, func, *args, **kw)
In CPython, the functools.partial
(actually _functools.partial
) works fine. In addition, the pure-Python implementation also works.
The pure-Python implementation in CPython standard library:
################################################################################
### partial() argument application
################################################################################
# Purely functional, no descriptor behaviour
class partial:
"""New function with partial application of the given arguments
and keywords.
"""
__slots__ = "func", "args", "keywords", "__dict__", "__weakref__"
def __new__(cls, func, /, *args, **keywords):
if not callable(func):
raise TypeError("the first argument must be callable")
if hasattr(func, "func"):
args = func.args + args
keywords = {**func.keywords, **keywords}
func = func.func
self = super(partial, cls).__new__(cls)
self.func = func
self.args = args
self.keywords = keywords
return self
def __call__(self, /, *args, **keywords):
keywords = {**self.keywords, **keywords}
return self.func(*self.args, *args, **keywords)
...
try:
from _functools import partial
except ImportError:
pass
PyPy copies the functools.py
from the CPython standard library but implements its own _functools.py
. The implementations of class parial
in _functools.py
and functools.py
are quite different.
Copied pure-Python implementation in lib-python/3/functools.py
:
The PyPy implementation of class partial
in lib_pypy/_functools.py
:
class partial(object):
"""
partial(func, *args, **keywords) - new function with partial application
of the given arguments and keywords.
"""
__slots__ = ('_func', '_args', '_keywords', '__dict__')
__module__ = 'functools' # instead of '_functools'
def __init__(*args, **keywords):
if len(args) < 2:
raise TypeError('__init__() takes at least 2 arguments (%d given)'
% len(args))
self, func, args = args[0], args[1], args[2:]
if not callable(func):
raise TypeError("the first argument must be callable")
if isinstance(func, partial):
args = func._args + args
tmpkw = func._keywords.copy()
tmpkw.update(keywords)
keywords = tmpkw
del tmpkw
func = func._func
self._func = func
self._args = args
self._keywords = keywords
def __delattr__(self, key):
if key == '__dict__':
raise TypeError("a partial object's dictionary may not be deleted")
object.__delattr__(self, key)
@property
def func(self):
return self._func
@property
def args(self):
return self._args
@property
def keywords(self):
return self._keywords
def __call__(self, *fargs, **fkeywords):
if self._keywords:
fkeywords = dict(self._keywords, **fkeywords)
return self._func(*(self._args + fargs), **fkeywords)
...
They are different in how to manage attributes (different slots and properties), and different in how to initialize a new instance (__new__
vs. __init__
).