transonic issueshttps://foss.heptapod.net/fluiddyn/transonic/-/issues2020-02-12T22:35:17Zhttps://foss.heptapod.net/fluiddyn/transonic/-/issues/17Bad formating for Pythran error2020-02-12T22:35:17ZBitbucket ImporterBad formating for Pythran error*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
It seems that Pythran errors appear badly formatted
```
b"\x1b[31mCRITICAL\x1b[0m \x1b[34mI am in tro...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
It seems that Pythran errors appear badly formatted
```
b"\x1b[31mCRITICAL\x1b[0m \x1b[34mI am in trouble. Your input file does not seem to match Pythran's constraints...\nE: Too many overloads for function '__for_method__bedload__advection_schemes', probably due to automatic generation of C-style and Fortran-style memory layout. Please force a layout using `order(C)` or `order(F)` in the array signature\x1b[0m\n"many overloads for function '__for_method__bedload__advection_schemes', probably due to automatic generation of C-style and Fortran-style memory layout. Please force a layout using `order(C)` or `order(F)` in the array signature\x1b[0m\n"
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/16--yes / -y option for command line2020-02-12T22:35:17ZBitbucket Importer--yes / -y option for command line*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
It should be possible to do:
`transonic -cc bar.py -y`
to cleanup cached file without asking for con...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
It should be possible to do:
`transonic -cc bar.py -y`
to cleanup cached file without asking for confirmation.https://foss.heptapod.net/fluiddyn/transonic/-/issues/15Wrong code to recompute annotations2020-02-12T22:35:16ZBitbucket ImporterWrong code to recompute annotations*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
```python
from functools import partial
from transonic import boost
@boost
def compute(a: int):
...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
```python
from functools import partial
from transonic import boost
@boost
def compute(a: int):
return a
# if True:
main = partial(lambda x: x, lambda x: x)
```
leads to
```
transonic -np tmp_bug_transonic.py -vv
INFO Namespace(clear_cache=None, force=False, no_pythran=True, path=['tmp_bug_transonic.py'], pythran_flags='', verbose=2, version=False)
DEBUG extast.parse
DEBUG compute ancestors and chains
DEBUG filter_code_typevars
DEBUG
from transonic import boost
main = partial((lambda x: x), (lambda x: x))
DEBUG find boosted objects
DEBUG {'classes': {},
'functions': {'compute': <gast.gast.FunctionDef object at 0x7f3de0afca90>},
'methods': {}}
DEBUG compute the annotations
Traceback (most recent call last):
File "/home/pierre/.pyenv/versions/3.7.2/bin/transonic", line 11, in <module>
load_entry_point('transonic', 'console_scripts', 'transonic')()
File "/home/pierre/Dev/transonic/transonic/run.py", line 84, in run
make_backend_files(paths, force=args.force)
File "/home/pierre/Dev/transonic/transonic/transpiler.py", line 284, in make_backend_files
path_out = make_pythran_file(path, force=force)
File "/home/pierre/Dev/transonic/transonic/transpiler.py", line 236, in make_pythran_file
code_pythran = make_pythran_code(path_py)
File "/home/pierre/Dev/transonic/transonic/backends/pythran.py", line 204, in make_pythran_code
boosted_dicts, code_dependance, annotations, blocks = analyse_aot(code_module)
File "/home/pierre/Dev/transonic/transonic/analyses/__init__.py", line 126, in analyse_aot
exec(code_dependance_annotations, namespace)
File "<string>", line 3, in <module>
NameError: name 'partial' is not defined
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/14Support source in multiple files2020-02-12T22:35:16ZBitbucket ImporterSupport source in multiple files*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
This should work:
```
# util.py
import h5py
const = 1
foo = 1
def func():
return const
...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
This should work:
```
# util.py
import h5py
const = 1
foo = 1
def func():
return const
def use_h5py():
...
```
Another file:
```
from util import func, foo
from transonic import jit
const = 2
@jit
def main():
print(foo, func(const))
```
Note the possible name clash if everything is written on one Pythran file (and there is no namespace keyword in Python!).
I think Pythran supports source in more that one file, so we could use that.https://foss.heptapod.net/fluiddyn/transonic/-/issues/13Support decorator used as `jit(func)`2020-02-12T22:35:15ZBitbucket ImporterSupport decorator used as `jit(func)`*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
This should work:
```
from transonic import jit
def func(x):
return x**2
func_jitted = jit(...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
This should work:
```
from transonic import jit
def func(x):
return x**2
func_jitted = jit(func)
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/12Refactoring frontend / backends as extensions2020-02-12T22:35:15ZBitbucket ImporterRefactoring frontend / backends as extensions*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
Refactor the code to split the analysis ("frontend") and the production and compilations of Pythran files ...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
Refactor the code to split the analysis ("frontend") and the production and compilations of Pythran files (a Pythran backend). This backend should be written as an extension.
The long term goal would be to be able to use different Numpy accelerators (Pythran, Cython, Numba-CuPy, PyTorch) with the same Transonic code (we would have to change the name fluidpythran).
Writing backends as separated extensions would be very nice.
The user API may be adapted. We have to think about what the different backends would need.https://foss.heptapod.net/fluiddyn/transonic/-/issues/8Recursion for @boost method2020-02-12T22:35:12ZBitbucket ImporterRecursion for @boost method*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
I don't think there is any big technical issues to support things like
```
def func(self, arg):
r...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
I don't think there is any big technical issues to support things like
```
def func(self, arg):
return self.attr * arg + self.func(arg - 1)
```
We just need to replace `self.func(arg - 1)` by something like `new_func(self_attr, arg-1)`...
In particular, I guess there is no problem with inheritance...https://foss.heptapod.net/fluiddyn/transonic/-/issues/4Smarter clearing of jit cache2020-02-12T22:35:10ZBitbucket ImporterSmarter clearing of jit cache*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
As seen in issue #3, sometimes the jit cache persists.
Ideas:
* Have a command `transonic --clear-jit`....*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
As seen in issue #3, sometimes the jit cache persists.
Ideas:
* Have a command `transonic --clear-jit`. It would be useful when compilation fails.
* Store a checksum of the linted code and annotation, to know when to clear the jit cache, and recompile?https://foss.heptapod.net/fluiddyn/transonic/-/issues/3Pythran annotation parsing bug "float or int"?2020-02-12T22:35:09ZBitbucket ImporterPythran annotation parsing bug "float or int"?*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Run the following file:
```python
from transonic import jit
import numpy as np
@jit
def linear_eig...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Run the following file:
```python
from transonic import jit
import numpy as np
@jit
def linear_eigenmode_from_values_1k(
ux_fft: np.complex128,
uy_fft: np.complex128,
eta_fft: np.complex128,
kx: float,
ky: float,
f: "float or int",
c2: "float or int",
):
"""Compute q, d, a (fft) for a single wavenumber."""
div_fft = 1j * (kx * ux_fft + ky * uy_fft)
rot_fft = 1j * (kx * uy_fft - ky * ux_fft)
q_fft = rot_fft - f * eta_fft
k2 = kx ** 2 + ky ** 2
ageo_fft = f * rot_fft / c2 + k2 * eta_fft
return q_fft, div_fft, ageo_fft
if __name__ == "__main__":
import time
while True:
time.sleep(1)
print(linear_eigenmode_from_values_1k(1j, 1j, 1j, 2., 3., 4, 5.))
```
It results in
```sh
CRITICAL I am in trouble. Your input file does not seem to match Pythran's constraints...
E: Ambiguous overloads
linear_eigenmode_from_values_1k(complex128, complex128, complex128, float64, float64, float, float)
linear_eigenmode_from_values_1k(complex128, complex128, complex128, float, float, float, float).
Error! File /proj/kthmech/users/x_ashmo/data/.transonic/__jit__/fluidsim/solvers/sw1l/output/linear_eigenmode_from_values_1k_0a237d87741074dd44df6794aaab014b_729e5defb353ff4f34ac3744fde6ded9.cpython-37m-x86_64-linux-gnu.so has not been created by Pythran
```
The input was:
```sh
cat /home/x_ashmo/lustre/data/.transonic/__jit__/fluidsim/solvers/sw1l/output/linear_eigenmode_from_values_1k.pythran
export linear_eigenmode_from_values_1k(complex128, complex128, complex128, float, float, float or int, float or int)
export linear_eigenmode_from_values_1k(complex128, complex128, complex128, float64, float64, float, float)
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/1test_justintime fails in locally (ArchLinux) but not in CI2020-02-12T22:35:05ZBitbucket Importertest_justintime fails in locally (ArchLinux) but not in CI*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Error log
```python
❯❯❯ pytest transonic/test_justintime.py ...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Error log
```python
❯❯❯ pytest transonic/test_justintime.py (gfdyn)
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.7.2, pytest-4.1.1, py-1.7.0, pluggy-0.7.1
rootdir: /home/avmo/src/gfdyn/transonic, inifile:
plugins: xdist-1.26.0, forked-0.2, cov-2.6.1
collected 4 items
transonic/test_justintime.py ...F [100%]
======================================================================================= FAILURES ========================================================================================
____________________________________________________________________________________ test_jit_method ____________________________________________________________________________________
def test_jit_method():
from .for_test_justintime import MyClass
obj = MyClass()
obj.check()
if not pythran:
return
obj = MyClass()
obj.check()
wait_for_all_extensions()
> obj.check()
transonic/test_justintime.py:133:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transonic/for_test_justintime.py:55: in check
assert self.myfunc(1) == 3
../../../.transonic/__jit_classes__/transonic/for_test_justintime/MyClass.py:12: in __new_method__MyClass__myfunc
return __for_method__MyClass__myfunc(self.attr0, self.attr1, arg)
transonic/justintime.py:403: in type_collector
self.path_extension, name_mod
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = PosixPath('/home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc_1a561a32e66dbed370c2ca73ad4a8e16_cc85ed47cd216c9460058222831d443a.cpython-37m-x86_64-linux-gnu.so')
def import_from_path(path: Path, module_name: str):
"""Import a .py file or an extension from its path
"""
if not path.exists():
raise ImportError(
> f"File {path} does not exist. "
f"[path.name for path in path.parent.glob('*')]:\n{[path.name for path in path.parent.glob('*')]}\n"
)
E ImportError: File /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc_1a561a32e66dbed370c2ca73ad4a8e16_cc85ed47cd216c9460058222831d443a.cpython-37m-x86_64-linux-gnu.so does not exist. [path.name for path in path.parent.glob('*')]:
E ['__for_method__MyClass__myfunc.py', '__for_method__MyClass__myfunc.pythran']
transonic/util.py:240: ImportError
--------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
Pythranizing /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc.py
--------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------
INFO Schedule pythranization of file /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc.py
INFO sys file exists: /scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/pythran.cfg
INFO platform file exists: /scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/pythran-linux.cfg
INFO user file exists: /home/avmo/.pythranrc
INFO pythranrc section [pythran] is valid and options are correct
INFO pythranrc section [typing] is valid and options are correct
INFO pythranrc section [compiler] is valid and options are correct
Traceback (most recent call last):
File "/scratch/avmo/opt/gfdyn/bin/pythran", line 11, in <module>
sys.exit(run())
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/run.py", line 166, in run
**compile_flags(args))
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 469, in compile_pythranfile
**kwargs)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 394, in compile_pythrancode
module_dir)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 141, in generate_cxx
optimizations, module_dir)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 109, in front_middle_end
refine(pm, ir, optimizations)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/middlend.py", line 22, in refine
pm.apply(ExpandBuiltins, node)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 218, in apply
res = a.apply(node, ctx)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 179, in apply
new_node = self.run(node, ctx)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 172, in run
n = super(Transformation, self).run(node, ctx)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 115, in run
return self.visit(node)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 81, in visit
return super(ContextManager, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/usr/lib64/python3.7/ast.py", line 317, in generic_visit
value = self.visit(value)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 81, in visit
return super(ContextManager, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/usr/lib64/python3.7/ast.py", line 317, in generic_visit
value = self.visit(value)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 81, in visit
return super(ContextManager, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/transformations/expand_builtins.py", line 39, in visit_Name
s not in self.locals[node] and
KeyError: <gast.gast.Name object at 0x7fcfe28d0c50>
Error! File /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc_1a561a32e66dbed370c2ca73ad4a8e16_cc85ed47cd216c9460058222831d443a.cpython-37m-x86_64-linux-gnu.so has not been created by Pythran
----------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------
pythranizer.py 156 INFO Schedule pythranization of file /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc.py
```