pypy3: macos: threading is preimported early preventing gpython and gevent to work correctly
Hello up there. In the process of fixing virtualenv support for gpython and gevent it was discovered that pypy3 (and only pypy3, not pypy2 nor cpython) on macos (and only on macos, not on linux/windows) imports threading
module very early - before any user code is run.
Gevent-based applications need to be able to import threading, socket, ssl, etc... modules very early to be able to monkey-patch them to use lightweight greenlets instead of OS threads. This patching has to be done as early as possible in the lifetime of the process
http://www.gevent.org/intro.html#monkey-patching
However pypy3/darwin preimports threading during startup via the following chain of imports:
site -> sysconfig -> _sysconfigdata -> platform -> subprocess -> threading.
(details)
Could you please fix this?
Without the fix gpython fails to start with error like below:
(py38.virtualenv) kirr@deco:~/tmp/trashme$ gpython
Traceback (most recent call last):
File "/home/kirr/tmp/trashme/py38.virtualenv/bin/gpython", line 8, in <module>
sys.exit(main())
File "/home/kirr/tmp/trashme/py38.virtualenv/lib/python3.8/site-packages/gpython/__init__.py", line 151, in main
raise RuntimeError('gpython: internal error: the following modules are pre-imported, but must be not:'
RuntimeError: gpython: internal error: the following modules are pre-imported, but must be not:
['threading']
sys.modules:
['__future__', '__main__', '_abc', '_bootlocale', '_codecs', '_collections', '_collections_abc', '_frozen_importlib', '_frozen_importlib_external', '_functools', '_heapq', '_imp', '_io', '_locale', '_operator', '_signal', '_sitebuiltins', '_sre', '_stat', '_thread', '_virtualenv', '_warnings', '_weakref', '_weakrefset', 'abc', 'builtins', 'codecs', 'collections', 'contextlib', 'copyreg', 'encodings', 'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'enum', 'functools', 'genericpath', 'gpython', 'heapq', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'importlib.abc', 'importlib.machinery', 'importlib.util', 'io', 'itertools', 'keyword', 'marshal', 'operator', 'os', 'os.path', 'posix', 'posixpath', 're', 'reprlib', 'site', 'sitecustomize', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'sys', 'threading', 'time', 'types', 'warnings', 'zipimport', 'zope']
Please see https://github.com/pypa/virtualenv/issues/1895 and https://github.com/pypa/virtualenv/pull/1897 to get more context about this issue.
I see several potential ways for the fix:
- import
subprocess
in platform only as needed:
--- a/lib-python/3/platform.py
+++ b/lib-python/3/platform.py
@@ -113,7 +113,7 @@
__version__ = '1.0.8'
import collections
-import sys, os, re, subprocess
+import sys, os, re
import warnings
@@ -804,6 +804,8 @@ def _syscmd_file(target, default=''):
default in case the command should fail.
"""
+ import subprocess
+
if sys.platform in ('dos', 'win32', 'win16'):
# XXX Others too ?
return default
This is already done this way on pypy2 https://foss.heptapod.net/pypy/pypy/-/blob/13582da4d60b6ce78dffd0916c11d2f3aa8cc880/lib-python/2.7/platform.py#L984-1009
However if I understand correctly, lib-python is copied as-is from upstream python, so it could be undesirable to modify it.
- patch
_sysconfigdata.py
to avoid importing platform:
--- a/lib_pypy/_sysconfigdata.py
+++ b/lib_pypy/_sysconfigdata.py
@@ -34,15 +34,18 @@
build_time_vars["CXX"] = "g++ -pthread"
if sys.platform[:6] == "darwin":
- import platform
- if platform.machine() == 'i386':
+ # don't import platform - it imports subprocess -> threading
+ # gevent-based projects need to be first to import threading and
+ # monkey-patch as early as possible in the lifetime of their process.
+ _, _, _, _, machine = os.uname()
+ if machine == 'i386':
if platform.architecture()[0] == '32bit':
arch = 'i386'
else:
arch = 'x86_64'
else:
# just a guess
- arch = platform.machine()
+ arch = machine
build_time_vars['LDSHARED'] += ' -undefined dynamic_lookup'
build_time_vars['CC'] += ' -arch %s' % (arch,)
if "CXX" in build_time_vars:
I had not tested the patches as I don't have mac and my mac VM somehow stopped working.
Thanks beforehand,
Kirill