transonic issueshttps://foss.heptapod.net/fluiddyn/transonic/-/issues2020-02-12T22:35:15Zhttps://foss.heptapod.net/fluiddyn/transonic/-/issues/13Support decorator used as `jit(func)`2020-02-12T22:35:15ZBitbucket ImporterSupport decorator used as `jit(func)`*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
This should work:
```
from transonic import jit
def func(x):
return x**2
func_jitted = jit(...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
This should work:
```
from transonic import jit
def func(x):
return x**2
func_jitted = jit(func)
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/12Refactoring frontend / backends as extensions2020-02-12T22:35:15ZBitbucket ImporterRefactoring frontend / backends as extensions*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
Refactor the code to split the analysis ("frontend") and the production and compilations of Pythran files ...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
Refactor the code to split the analysis ("frontend") and the production and compilations of Pythran files (a Pythran backend). This backend should be written as an extension.
The long term goal would be to be able to use different Numpy accelerators (Pythran, Cython, Numba-CuPy, PyTorch) with the same Transonic code (we would have to change the name fluidpythran).
Writing backends as separated extensions would be very nice.
The user API may be adapted. We have to think about what the different backends would need.https://foss.heptapod.net/fluiddyn/transonic/-/issues/11Implement keyword `or` in block signatures2020-02-12T22:35:14ZBitbucket ImporterImplement keyword `or` in block signatures*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
```
if ts.is_transpiled:
result = ts.use_block("myblock")
else:
# transonic block (int or floa...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
```
if ts.is_transpiled:
result = ts.use_block("myblock")
else:
# transonic block (int or float a, b)
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/10Be able to define PyCapsule2020-02-12T22:35:14ZBitbucket ImporterBe able to define PyCapsule*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
See
- https://serge-sans-paille.github.io/pythran-stories/the-capsule-corporation.html
- https://nu...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
See
- https://serge-sans-paille.github.io/pythran-stories/the-capsule-corporation.html
- https://numba.pydata.org/numba-doc/dev/user/cfunc.html
And also: http://www.stochasticlifestyle.com/why-numba-and-cython-are-not-substitutes-for-julia/https://foss.heptapod.net/fluiddyn/transonic/-/issues/9Type hints: ShapeVar2020-02-12T22:35:13ZBitbucket ImporterType hints: ShapeVar*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
```
from transonic import boost, Type, NDim, Shape, Array
T = Type(int, float)
# here the shape of ...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
```
from transonic import boost, Type, NDim, Shape, Array
T = Type(int, float)
# here the shape of the array is only defined with the ShapeVar
A = Array[T, Shape("[3, :]", "[3, :, :]", "[::, ::]", "[::, ::, ::]")]
@boost
def compute(a: A, b: A, c: T):
return a + b
# if there is a NDimVar, we can use the ellipsis
A1 = Array[T, NDim(1, 3), Shape("[3, ...]", "[::, ...]")]
@boost
def compute1(a: A1, b: A1, c: T):
return c * (a + b)
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/8Recursion for @boost method2020-02-12T22:35:12ZBitbucket ImporterRecursion for @boost method*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
I don't think there is any big technical issues to support things like
```
def func(self, arg):
r...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
I don't think there is any big technical issues to support things like
```
def func(self, arg):
return self.attr * arg + self.func(arg - 1)
```
We just need to replace `self.func(arg - 1)` by something like `new_func(self_attr, arg-1)`...
In particular, I guess there is no problem with inheritance...https://foss.heptapod.net/fluiddyn/transonic/-/issues/7Detect unsupported assignment self.attr = 12020-02-12T22:35:12ZBitbucket ImporterDetect unsupported assignment self.attr = 1*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
https://foss.heptapod.net/fluiddyn/transonic/-/issues/6Alternative syntax for blocks with decorator and context manager2020-06-13T19:10:14ZBitbucket ImporterAlternative syntax for blocks with decorator and context manager*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
I'd like to support such things:
### for functions:
```python
from transonic import with_blocks, block
...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
I'd like to support such things:
### for functions:
```python
from transonic import with_blocks, block
@with_blocks
def func(arg: int):
a = non_pythranizable(arg)
with block():
a: float
tmp = a + arg
return non_pythranizable(tmp)
```
I think supporting `return` in a block is going to be difficult.
### for classes:
```python
from transonic import boost, with_blocks, block
@boost
class MyClass:
attr0: int
@with_blocks
def func(self, arg: int):
a = non_pythranizable(arg)
with block():
a: float
tmp = a + arg + self.attr0
return non_pythranizable(tmp)
```
- We could also support block signatures `# transonic block (int: a, b; A: c, d, e)` in the block.https://foss.heptapod.net/fluiddyn/transonic/-/issues/5Transonic jit compilation+mpi blocks simulations2020-02-12T22:35:11ZBitbucket ImporterTransonic jit compilation+mpi blocks simulations*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
## Issue
Simulations are frozen if attempted to launch with MPI from a fresh installation of `fluidsim`.
...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
## Issue
Simulations are frozen if attempted to launch with MPI from a fresh installation of `fluidsim`.
## Reproduce
* make a fresh virtualenv with the following requirements:
```
Cython==0.29.4
decorator==4.3.2
gast==0.2.2
mpi4py==3.0.0
networkx==2.2
numpy==1.16.1
ply==3.11
pythran==0.9.1.post0
six==1.12.0
```
```pip install -r requirements.txt```
* Install `fluidfft` and `fluiddyn` using `make develop` from the respective repos.
* Clear cache:
```sh
TRANSONIC_DIR=$FLUIDSIM_PATH/.transonic transonic -cc fluidsim.solvers.sw1l
```
* Run a set of tests with mpi and jit compilation
```sh
TRANSONIC_COMPILE_JIT=1 mpirun -n 2 fluidsim-test -m fluidsim.solvers.sw1l
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/4Smarter clearing of jit cache2020-02-12T22:35:10ZBitbucket ImporterSmarter clearing of jit cache*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
As seen in issue #3, sometimes the jit cache persists.
Ideas:
* Have a command `transonic --clear-jit`....*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
As seen in issue #3, sometimes the jit cache persists.
Ideas:
* Have a command `transonic --clear-jit`. It would be useful when compilation fails.
* Store a checksum of the linted code and annotation, to know when to clear the jit cache, and recompile?https://foss.heptapod.net/fluiddyn/transonic/-/issues/3Pythran annotation parsing bug "float or int"?2020-02-12T22:35:09ZBitbucket ImporterPythran annotation parsing bug "float or int"?*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Run the following file:
```python
from transonic import jit
import numpy as np
@jit
def linear_eig...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Run the following file:
```python
from transonic import jit
import numpy as np
@jit
def linear_eigenmode_from_values_1k(
ux_fft: np.complex128,
uy_fft: np.complex128,
eta_fft: np.complex128,
kx: float,
ky: float,
f: "float or int",
c2: "float or int",
):
"""Compute q, d, a (fft) for a single wavenumber."""
div_fft = 1j * (kx * ux_fft + ky * uy_fft)
rot_fft = 1j * (kx * uy_fft - ky * ux_fft)
q_fft = rot_fft - f * eta_fft
k2 = kx ** 2 + ky ** 2
ageo_fft = f * rot_fft / c2 + k2 * eta_fft
return q_fft, div_fft, ageo_fft
if __name__ == "__main__":
import time
while True:
time.sleep(1)
print(linear_eigenmode_from_values_1k(1j, 1j, 1j, 2., 3., 4, 5.))
```
It results in
```sh
CRITICAL I am in trouble. Your input file does not seem to match Pythran's constraints...
E: Ambiguous overloads
linear_eigenmode_from_values_1k(complex128, complex128, complex128, float64, float64, float, float)
linear_eigenmode_from_values_1k(complex128, complex128, complex128, float, float, float, float).
Error! File /proj/kthmech/users/x_ashmo/data/.transonic/__jit__/fluidsim/solvers/sw1l/output/linear_eigenmode_from_values_1k_0a237d87741074dd44df6794aaab014b_729e5defb353ff4f34ac3744fde6ded9.cpython-37m-x86_64-linux-gnu.so has not been created by Pythran
```
The input was:
```sh
cat /home/x_ashmo/lustre/data/.transonic/__jit__/fluidsim/solvers/sw1l/output/linear_eigenmode_from_values_1k.pythran
export linear_eigenmode_from_values_1k(complex128, complex128, complex128, float, float, float or int, float or int)
export linear_eigenmode_from_values_1k(complex128, complex128, complex128, float64, float64, float, float)
```https://foss.heptapod.net/fluiddyn/transonic/-/issues/2HPC related qustions on fresh install and JIT compilation on jobs2020-02-12T22:35:08ZBitbucket ImporterHPC related qustions on fresh install and JIT compilation on jobs*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Some of my jobs were frozen during JIT compilation, and it was never released, even after 24 hrs. It was prob...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Some of my jobs were frozen during JIT compilation, and it was never released, even after 24 hrs. It was probably because I had the environment variable `OMP_NUM_THREADS=1` on. Nevertheless, would it be possible to check if the threads are released after JIT compilation is done.https://foss.heptapod.net/fluiddyn/transonic/-/issues/1test_justintime fails in locally (ArchLinux) but not in CI2020-02-12T22:35:05ZBitbucket Importertest_justintime fails in locally (ArchLinux) but not in CI*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Error log
```python
❯❯❯ pytest transonic/test_justintime.py ...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Error log
```python
❯❯❯ pytest transonic/test_justintime.py (gfdyn)
================================================================================== test session starts ==================================================================================
platform linux -- Python 3.7.2, pytest-4.1.1, py-1.7.0, pluggy-0.7.1
rootdir: /home/avmo/src/gfdyn/transonic, inifile:
plugins: xdist-1.26.0, forked-0.2, cov-2.6.1
collected 4 items
transonic/test_justintime.py ...F [100%]
======================================================================================= FAILURES ========================================================================================
____________________________________________________________________________________ test_jit_method ____________________________________________________________________________________
def test_jit_method():
from .for_test_justintime import MyClass
obj = MyClass()
obj.check()
if not pythran:
return
obj = MyClass()
obj.check()
wait_for_all_extensions()
> obj.check()
transonic/test_justintime.py:133:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
transonic/for_test_justintime.py:55: in check
assert self.myfunc(1) == 3
../../../.transonic/__jit_classes__/transonic/for_test_justintime/MyClass.py:12: in __new_method__MyClass__myfunc
return __for_method__MyClass__myfunc(self.attr0, self.attr1, arg)
transonic/justintime.py:403: in type_collector
self.path_extension, name_mod
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = PosixPath('/home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc_1a561a32e66dbed370c2ca73ad4a8e16_cc85ed47cd216c9460058222831d443a.cpython-37m-x86_64-linux-gnu.so')
def import_from_path(path: Path, module_name: str):
"""Import a .py file or an extension from its path
"""
if not path.exists():
raise ImportError(
> f"File {path} does not exist. "
f"[path.name for path in path.parent.glob('*')]:\n{[path.name for path in path.parent.glob('*')]}\n"
)
E ImportError: File /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc_1a561a32e66dbed370c2ca73ad4a8e16_cc85ed47cd216c9460058222831d443a.cpython-37m-x86_64-linux-gnu.so does not exist. [path.name for path in path.parent.glob('*')]:
E ['__for_method__MyClass__myfunc.py', '__for_method__MyClass__myfunc.pythran']
transonic/util.py:240: ImportError
--------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------
Pythranizing /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc.py
--------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------
INFO Schedule pythranization of file /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc.py
INFO sys file exists: /scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/pythran.cfg
INFO platform file exists: /scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/pythran-linux.cfg
INFO user file exists: /home/avmo/.pythranrc
INFO pythranrc section [pythran] is valid and options are correct
INFO pythranrc section [typing] is valid and options are correct
INFO pythranrc section [compiler] is valid and options are correct
Traceback (most recent call last):
File "/scratch/avmo/opt/gfdyn/bin/pythran", line 11, in <module>
sys.exit(run())
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/run.py", line 166, in run
**compile_flags(args))
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 469, in compile_pythranfile
**kwargs)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 394, in compile_pythrancode
module_dir)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 141, in generate_cxx
optimizations, module_dir)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/toolchain.py", line 109, in front_middle_end
refine(pm, ir, optimizations)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/middlend.py", line 22, in refine
pm.apply(ExpandBuiltins, node)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 218, in apply
res = a.apply(node, ctx)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 179, in apply
new_node = self.run(node, ctx)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 172, in run
n = super(Transformation, self).run(node, ctx)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 115, in run
return self.visit(node)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 81, in visit
return super(ContextManager, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/usr/lib64/python3.7/ast.py", line 317, in generic_visit
value = self.visit(value)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 81, in visit
return super(ContextManager, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/usr/lib64/python3.7/ast.py", line 317, in generic_visit
value = self.visit(value)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/passmanager.py", line 81, in visit
return super(ContextManager, self).visit(node)
File "/usr/lib64/python3.7/ast.py", line 262, in visit
return visitor(node)
File "/scratch/avmo/opt/gfdyn/lib/python3.7/site-packages/pythran/transformations/expand_builtins.py", line 39, in visit_Name
s not in self.locals[node] and
KeyError: <gast.gast.Name object at 0x7fcfe28d0c50>
Error! File /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc_1a561a32e66dbed370c2ca73ad4a8e16_cc85ed47cd216c9460058222831d443a.cpython-37m-x86_64-linux-gnu.so has not been created by Pythran
----------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------
pythranizer.py 156 INFO Schedule pythranization of file /home/avmo/.transonic/__jit__/__jit_classes__/transonic/for_test_justintime/MyClass/__for_method__MyClass__myfunc.py
```