Skip to content

Problems with CFFI and Python 3.9 (segfault)

Hello.

We are building CFFI from source as an RPM package in Fedora and the build has recently started to fail. I'm trying to debug the problem and would appreciate your help because my knowledge here is limited.

The build itself finishes without any issues, the problem appears when you try to run tests:

cffi-1.14.4/c/test_c.py::test_array_of_unknown_length_instance Fatal Python error: Aborted

Current thread 0x00007fcb99e7b740 (most recent call first):
  File "/src/cffi-1.14.4/c/test_c.py", line 573 in test_array_of_unknown_length_instance
  File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 180 in pytest_pyfunc_call
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 1570 in runtest
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 153 in pytest_runtest_call
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 247 in <lambda>
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 294 in from_call
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 246 in call_runtest_hook
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 207 in call_and_report
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 117 in runtestprotocol
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 100 in pytest_runtest_protocol
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 321 in pytest_runtestloop
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 296 in _main
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 240 in wrap_session
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 289 in pytest_cmdline_main
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/config/__init__.py", line 157 in main
  File "/usr/lib/python3.9/site-packages/_pytest/config/__init__.py", line 180 in console_main
  File "/usr/lib/python3.9/site-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib64/python3.9/runpy.py", line 87 in _run_code
  File "/usr/lib64/python3.9/runpy.py", line 197 in _run_module_as_main
Aborted (core dumped)

cffi-1.14.4/testing/cffi0/test_ffi_backend.py::TestFFI::test_new_array_of_pointer_1 Fatal Python error: Aborted

Current thread 0x00007ff347710740 (most recent call first):
  File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 180 in pytest_pyfunc_call
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 1570 in runtest
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 153 in pytest_runtest_call
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 247 in <lambda>
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 294 in from_call
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 246 in call_runtest_hook
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 207 in call_and_report
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 117 in runtestprotocol
  File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 100 in pytest_runtest_protocol
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 321 in pytest_runtestloop
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 296 in _main
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 240 in wrap_session
  File "/usr/lib/python3.9/site-packages/_pytest/main.py", line 289 in pytest_cmdline_main
  File "/usr/lib/python3.9/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/lib/python3.9/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/lib/python3.9/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/lib/python3.9/site-packages/_pytest/config/__init__.py", line 157 in main
  File "/usr/lib/python3.9/site-packages/_pytest/config/__init__.py", line 180 in console_main
  File "/usr/lib/python3.9/site-packages/pytest/__main__.py", line 7 in <module>
  File "/usr/lib64/python3.9/runpy.py", line 87 in _run_code
  File "/usr/lib64/python3.9/runpy.py", line 197 in _run_module_as_main
Aborted (core dumped)

There are many more tests causing segfaults and some tests just fail in a standard way. I've tried to reproduce the issue in a simple reproducer basically copying the content of test_array_of_unknown_length_instance to a Python script without pytest.

from _cffi_backend import *

p = new_primitive_type("int")
p1 = new_array_type(new_pointer_type(p), None)
try:
    newp(p1, None)
except TypeError:
    print("OK")
try:
    newp(p1, -42)
except ValueError:
    print("OK")
a = newp(p1, 42)
assert len(a) == 42
for i in range(42):
    a[i] -= i
for i in range(42):
    print(a[i])
    assert a[i] == -i
try:
    a[42]
except IndexError:
    print("OK")
try:
    a[-1]
except IndexError:
    print("OK")
try:
    a[42] = 123
except IndexError:
    print("OK")
try:
    a[-1] = 456
except IndexError:
    print("OK")

The output is:

OK
OK
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
407

It seems to me that the array is not properly initialized to zeros.

I'm running on Fedora rawhide with these important packages installed:

gcc-11.0.0-0.14.fc34.x86_64
gcc-c++-11.0.0-0.14.fc34.x86_64
python3-3.9.1-1.fc34.x86_64
python3-pycparser-2.20-2.fc33.noarch
libffi-3.1-27.fc34.x86_64

Another reproducer created from test_assign_string:

from _cffi_backend import *

BChar = new_primitive_type("char")
BArray1 = new_array_type(new_pointer_type(BChar), 5)
BArray2 = new_array_type(new_pointer_type(BArray1), 5)
a = newp(BArray2, [b"abc", b"de", b"ghij"])
assert string(a[1]) == b"de"
assert string(a[2]) == b"ghij"
a[2] = b"."
assert string(a[2]) == b"."
a[2] = b"12345"
assert string(a[2]) == b"12345"
try:
    a[2] = b"123456"
except IndexError as e:
    assert 'char[5]' in str(e)
    assert 'got 6 characters' in str(e)

This script works without any issue but if I run the test the standard way I get:

/usr/lib/python3.9/site-packages/_pytest/compat.py:340: PytestDeprecationWarning: The TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.
See https://docs.pytest.org/en/stable/deprecations.html#terminalreporter-writer for more information.
  return getattr(object, name, default)
==================================== test session starts =====================================
platform linux -- Python 3.9.1, pytest-6.0.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/fedora/lbalhar
collected 2040 items / 2039 deselected / 1 selected                                          

cffi-v1.14.3/c/test_c.py .                                                             [100%]

====================================== warnings summary ======================================
cffi-v1.14.3/testing/cffi0/test_zdistutils.py:1
  /home/fedora/lbalhar/cffi-v1.14.3/testing/cffi0/test_zdistutils.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import sys, os, imp, math, shutil

cffi-v1.14.3/testing/cffi1/test_parse_c_type.py:18
  /home/fedora/lbalhar/cffi-v1.14.3/testing/cffi1/test_parse_c_type.py:18: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/fedora/lbalhar/cffi-v1.14.3/cffi/parse_c_type.h' mode='r' encoding='UTF-8'>
    header = open(os.path.join(cffi_dir, 'parse_c_type.h')).read()

cffi-v1.14.3/testing/cffi1/test_parse_c_type.py:27
  /home/fedora/lbalhar/cffi-v1.14.3/testing/cffi1/test_parse_c_type.py:27: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/fedora/lbalhar/cffi-v1.14.3/cffi/../c/parse_c_type.c' mode='r' encoding='UTF-8'>
    open(os.path.join(cffi_dir, '..', 'c', 'parse_c_type.c')).read() + """

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================= 1 passed, 2039 deselected, 3 warnings in 5.81s =======================
corrupted size vs. prev_size
Fatal Python error: Aborted

Current thread 0x00007fc8f0e1e740 (most recent call first):
<no Python frame>
Aborted (core dumped)

When I tried to investigate the dumped core in gdb I've found out that there might be a problem in some compiler optimizations because if I use python3 the problem appears as described above but if I use python3-debug or if I disable optimizations via CFLAGS="-O0 -g" CXXFLAGS="-O0 -g", all tests are green.

So, in the end, it seems that the issue is probably outside cffi and might lay somewhere in the compilators but it might indicate some incompatibility.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information