Draft: Improve os.scandir() handling of symlinks on Windows

Merged Caleb Burns requested to merge branch/win-scandir-symlinks into branch/py3.9

Here is my proof of concept for fixing the stat/symlink support for the Windows implementation of os.scandir(). Pytest with PyPy 2.7 succeeds, but PyPy 3.9 is failing with both the -A and -D tests. I could use some guidance with fixing these tests.

pytest with PyPy 2.7

Running "test_scandir.py" with PyPy 2.7 succeeds as expected. The command is:

C:\pypy2.7-v7.3.12-win64\pypy.exe pytest.py pypy\module\posix\test\test_scandir.py

See pytest_test_scandir.log.

pytest -A with PyPy 3.9

Running test_scandir.py after translation using the -A flag fails all tests. The tests are failing because my attempt to use pypy.module.posix.interp_posix.symlink() as a replacement for os.symlink() only on Windows for PyPy 2.7. Why is interp_posix.symlink() being called instead of os.symlink()?

The command is:

C:\pypy2.7-v7.3.12-win64\pypy.exe pytest.py -A --python=pypy\goal\pypy3.9-c.exe pypy\module\posix\test\test_scandir.py

And relevant traceback is:

space = <pypy.tool.pytest.objspace.TinyObjSpace object at 0x000001a9f0ada9f8>
source = 'c:\\users\\admin\\appdata\\local\\temp\\usession-py3.9-7\\some_file'
link_name = 'c:\\users\\admin\\appdata\\local\\temp\\usession-py3.9-7\\dir3\\sfile3'

    def _symlink(space, source, link_name):
        if os_symlink is not None:
            os_symlink(source, link_name)
        else:
            w_source = space.newtext(source)
            w_link_name = space.newtext(link_name)
>           interp_posix_symlink(space, w_source, w_link_name)

pypy\module\posix\test\test_scandir.py:20: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

space = <pypy.tool.pytest.objspace.TinyObjSpace object at 0x000001a9f0ada9f8>
w_src = u'c:\\users\\admin\\appdata\\local\\temp\\usession-py3.9-7\\some_file'
w_dst = u'c:\\users\\admin\\appdata\\local\\temp\\usession-py3.9-7\\dir3\\sfile3'
target_is_directory = 0, __kwonly__ = None, dir_fd = -100

    @unwrap_spec(target_is_directory=int, dir_fd=DirFD(rposix.HAVE_SYMLINKAT))
    def symlink(space, w_src, w_dst, target_is_directory=0,
                __kwonly__=None, dir_fd=DEFAULT_DIR_FD):
        """symlink(src, dst, target_is_directory=0, *, dir_fd=None)
    
    Create a symbolic link pointing to src named dst.
    
    target_is_directory is required on Windows if the target is to be
      interpreted as a directory.  (On Windows, symlink requires
      Windows 6.0 or greater, and raises a NotImplementedError otherwise.)
      target_is_directory is ignored on non-Windows platforms.
    
    If dir_fd is not None, it should be a file descriptor open to a directory,
      and path should be relative; path will then be relative to that directory.
    dir_fd may not be implemented on your platform.
      If it is unavailable, using it will raise a NotImplementedError."""
        if _WIN32:
>           src_utf8 = space.fsencode_w(w_src)
E           AttributeError: 'TinyObjSpace' object has no attribute 'fsencode_w'

pypy\module\posix\interp_posix.py:1456: AttributeError
_____________ ERROR at setup of AppTestScandir.test_scandir_files _

See pytest_a_test_scandir.log.

pytest -D with PyPy 3.9

I attempted to write an apptest_scandir.py version of test_scandir.py as recommended by basing it off of apptest_posix.py. Using a class was more straight forward for the directory setup logic than figuring out pytest's fixtures. These tests are failing at setup because a Python 2.7 source file with a print statement is being imported.

The command is:

pypy\goal\pypy3.9-c.exe pytest.py -D pypy\module\posix\test\apptest_scandir.py

And relevant traceback is:

______________ ERROR at setup of TestScandir.test_scandir_empty _______________

item = <Function 'test_scandir_empty'>

    @pytest.hookimpl(tryfirst=True)
    def pytest_runtest_setup(item):
        if isinstance(item, py.test.collect.Function):
            config = item.config
            if item.get_marker(name='pypy_only'):
                if config.applevel is not None and not config.applevel.is_pypy:
                    pytest.skip('PyPy-specific test')
            appclass = item.getparent(py.test.Class)
            if appclass is not None:
>               from pypy.tool.pytest.objspace import gettestobjspace

pypy\conftest.py:221: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pypy\tool\pytest\objspace.py:4: in <module>
    from pypy.tool.option import make_config, make_objspace
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   from pypy.config.pypyoption import get_pypy_config
E     File "C:\Users\Admin\Downloads\pypy-branch-py3.9\pypy\config\pypyoption.py", line 307
E       print config.getpaths()
E       ^
E   SyntaxError: Missing parentheses in call to 'print'. Did you mean 'print(...)'?

pypy\tool\option.py:3: SyntaxError

Running -D pypy\module\math\test\apptest_math.py succeeds, so I'm not sure why apptest_scandir.py fails with this SyntaxError.

See pytest_d_apptest_scandir.log

--HG-- branch : win-scandir-symlinks

Merge request reports