Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
fluiddyn
transonic
Commits
3c238f68fe5a
Commit
95449c41
authored
Sep 13, 2019
by
Pierre Augier
Browse files
List some Cython issues
parent
d75a2c0a057d
Changes
15
Hide whitespace changes
Inline
Side-by-side
.hgignore
View file @
3c238f68
...
...
@@ -31,4 +31,6 @@ doc/ipynb/*.rst
**/__pythran__/*
**/__cython__/*
**/__numba__/*
**/__python__/*
\ No newline at end of file
**/__python__/*
doc/for_dev/**/*.c
ROADMAP.rst
View file @
3c238f68
...
...
@@ -44,6 +44,12 @@ Less bugs and more Cython features...
- "exterior" functions + understand the tests
- void type
- nogil for function definition
- nogil context manager
setup.py & more than one backend at runtime
-------------------------------------------
...
...
doc/backends/cython.md
View file @
3c238f68
...
...
@@ -7,36 +7,33 @@ However, a descent set of Cython can be supported. We need to find Python
syntaxes for the most useful Cython special syntaxes.
Note that some Cython features are useless in Pythran (for example cdef of
local
s
variable, or
`nogil`
).
local variable
s
, or
`nogil`
).
Note that ideally, we want to write Cython code that can be executed without
the Cython package and without Cython compilation. It is possible with a subset
of the "pure Python mode" of Cython for which the Python file does not depend
on Cython, i.e. all the supplementary information needed by Cython has to be
written in the .pxd file.
the Cython package and without Cython compilation. It is possible with the
[
"pure Python mode" of
Cython
](
https://cython.readthedocs.io/en/latest/src/tutorial/pure.html
)
.
Therefore, we first need examples of Cython code written in this mode.
## Cython syntaxes already supported
Note however, that this mode is currently still experimental and that we hit
simple Cython bugs which limit a lot what can be done in practice with the
Cython backend. For example:
### Simple fused types, memory views
-
Pure-Python mode and fused types
<https://github.com/cython/cython/issues/3142>
-
`cython.locals(arr=np.ndarray[...])`
https://github.com/cython/cython/issues/3129
We already have fused types in Transonic. With Transonic, we can already do (no
notion of C order):
More generally, there are many known bugs in Cython which do not help! For example:
```
python
import
numpy
as
np
from
transonic
import
Transonic
,
Type
,
NDim
,
Array
-
`ctypedef`
and buffer
<https://github.com/cython/cython/issues/754>
-
<https://stackoverflow.com/questions/57887972/defining-a-fused-type-using-a-fused-type>
np_floats
=
Type
(
np
.
float32
,
np
.
float64
)
N
=
NDim
(
2
,
3
,
4
)
A
=
Array
[
np_floats
,
N
]
A1
=
Array
[
np_floats
,
N
+
1
]
# or simply
A3d
=
Array
[
np_floats
,
"3d"
]
```
I think at least some of these bugs have to be solved upstream...
## Cython syntaxes already supported
### `cpdef` signature with simple (basic and array) types for arguments
### cdef for type declaration of local variables: `cython.locals`
###
`
cdef
`
for type declaration of local variables: `cython.locals`
In "pure Python mode", one can write
...
...
@@ -45,10 +42,9 @@ In "pure Python mode", one can write
cpdef
mysum
(
np
.
float64_t
[:]
arr_input
)
```
With variables annotations (which
have to b
e removed for Pythran / Numba):
With variables annotations (which
ar
e removed for Pythran / Numba):
```
python
from
transonic
import
boost
@
boost
...
...
@@ -59,7 +55,6 @@ def mysum(arr_input: "float[]"):
for
i
in
range
(
n
):
result
+=
arr_input
[
i
]
return
result
```
### Cython decorators
...
...
@@ -71,7 +66,7 @@ def mysum(arr_input: "float[]"):
Currently only for simple functions (no methods).
## Cython syntaxes
ne
arly supported
## Cython syntaxes
p
ar
t
ly supported
### Function definition (cdef, cpdef, inline, nogil, return type)
...
...
@@ -86,45 +81,141 @@ def func(a: "float[]", n: int) -> "void":
which would translate in Cython as something like:
```
cython
cdef
inline
void
func
(
np
.
float_t
[:
]
a
,
cython
.
int
n
)
nogil
c
p
def
inline
void
func
(
np
.
ndarray
[
np
.
float_t
,
ndim
=
1
]
a
,
cython
.
int
n
)
nogil
```
`boost(inline=True)`
is supported for functions, see
[
this
example
](
https://transonic.readthedocs.io/en/latest/examples/inlined/txt.html
)
.
-
all function signatures use
`cpdef`
## Cython syntaxes that can be supported quite easily
-
`boost(inline=True)`
is supported for functions, see
[
this
example
](
https://transonic.readthedocs.io/en/latest/examples/inlined/txt.html
)
.
-
Return type is supported but there is no void type.
### Fused types
We already have fused types in Transonic. With Transonic, we can already do:
```
python
import
numpy
as
np
from
transonic
import
Array
,
Type
,
NDim
np_floats
=
Type
(
np
.
float32
,
np
.
float64
)
N
=
NDim
(
2
,
3
,
4
)
A
=
Array
[
np_floats
,
N
]
A1
=
Array
[
np_floats
,
N
+
1
]
# or simply
A3d
=
Array
[
np_floats
,
"3d"
]
```
### ctypedef, fused types and contiguous arrays
However, Cython Fused types are currently very limited.
Even with something as simple as that
```
python
from
transonic
import
Array
,
Type
A
=
Array
[
Type
(
np
.
float64
,
np
.
complex128
),
"1d"
]
def
mysum
(
arr
:
A
):
result
:
A
.
dtype
=
arr
.
dtype
.
type
(
0.
)
i
:
int
for
i
in
range
(
arr
.
shape
[
0
]):
result
+=
arr
[
i
]
return
result
```
should be translated to this (not supported, see
<https://github.com/cython/cython/issues/754>
) Cython code:
```
cython
cimport
numpy
as
cnp
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
T0
:
np
.
complex128_t
np
.
float64_t
ctypedef
np
.
ndarray
[
T0
,
ndim
=
1
]
A
def
mysum
(
A
arr
):
cdef
T0
ret
=
arr
.
dtype
.
type
(
0.
)
cdef
cython
.
int
i
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
```
Note that it works with a memoryview... (but not in pure-Python mode!)
ctypedef
fused
np_floats
:
cnp
.
float32_t
cnp
.
float64_t
Note that another working alternative is:
cdef
np_floats
[:,
:,
::
1
]
myvar
```
import cython
import numpy as np
cimport numpy as np
ctypedef fused T0:
np.complex128_t
np.float64_t
def mysum(np.ndarray[T0, ndim=1] arr):
cdef T0 ret = arr.dtype.type(0.)
cdef cython.int i
for i in range(arr.shape[0]):
ret += arr[i]
return ret
```
A nice Python syntax has to be found for
`np_floats[:, :, ::1]`
But the corresponding pure-Python version does not work!
## Cython syntaxes that can be supported quite easily
### More array types (contiguous arrays, C or F order, memoryviews)
I
t could just be
I
think we should support:
```
python
A2
=
Array
[
np_floats
,
"[:, :, ::1]"
]
Array
[
int
,
NDim
(
3
),
"order=C"
]
Array
[
int
,
"3d"
,
"order=C"
]
Array
[
"int[:, :, ::1]"
]
Array
[
int
,
"[:, :, ::1]"
]
transonic
.
str2type
(
"int[:, :, ::1]"
)
transonic
.
typeof
(
np
.
empty
((
2
,
2
,
2
)))
```
and maybe also:
```
python
transonic
.
int64
[:,
:,
::
1
]
```
I tend to think that the default (
`"int[:,:]"`
) should correspond to
`"order=C"`
. "Fortran" order and "any" order (contiguous C or F) could be
obtained with
`"order=F"`
and
`"order=any"`
.
Strided arrays could be obtained with
`Array[int, NDim(3), "strided"]`
or
`str2type("int[::, ::, ::]")`
.
Note that for Pythran, we could also support:
```
python
A_fixed_dim
=
Array
[
np_floats
,
"[:, :, 3]"
]
A_fixed_dim
=
Array
[
Type
(
np
.
float32
,
float
),
"[:, :, 3]"
]
```
For Cython, we need to be able to specify if an array is a
`np.ndarray`
or a
`memoryview`
. By default, we will use
`np.ndarray`
and
`memoryview`
could be
obtained with:
```
python
Array
[
int
,
"[:, :, ::1]"
,
"memview"
]
```
### Special C types
For example
`Py_ssize_t`
For example
`Py_ssize_t`
and
`void`
```
python
from
ctypes
import
c_ssize_t
as
Py_ssize_t
...
...
@@ -176,7 +267,7 @@ to suppress the `with nogil()`.
cdef
Py_ssize_t
*
p_indexer
```
### Definition `struc`, `enum`, `class`
### Definition `struc
t
`, `enum`, `class`
```
cython
cdef
struct
Heap
:
...
...
doc/for_dev/cython_bugs/ctypedef_buffer/bug.pyx
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
T0
:
np
.
complex128_t
np
.
float64_t
ctypedef
np
.
ndarray
[
T0
,
ndim
=
1
]
A
def
mysum
(
A
arr
):
cdef
T0
ret
=
arr
.
dtype
.
type
(
0
)
cdef
cython
.
int
i
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
\ No newline at end of file
doc/for_dev/cython_bugs/ctypedef_buffer/ctype_memview.pyx
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
T0
:
np
.
complex128_t
np
.
float64_t
ctypedef
T0
[:]
A
def
mysum
(
A
arr
):
cdef
T0
ret
=
arr
.
dtype
.
type
(
0
)
cdef
cython
.
int
i
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
\ No newline at end of file
doc/for_dev/cython_bugs/ctypedef_buffer/pure.pxd
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
T0
:
np
.
complex128_t
np
.
float64_t
@
cython
.
locals
(
ret
=
T0
,
i
=
cython
.
int
)
cpdef
mysum
(
np
.
ndarray
[
T0
,
ndim
=
1
]
arr
)
doc/for_dev/cython_bugs/ctypedef_buffer/pure.py
0 → 100644
View file @
3c238f68
def
mysum
(
arr
):
ret
=
arr
.
dtype
.
type
(
0
)
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
doc/for_dev/cython_bugs/ctypedef_buffer/pure_ctype_memview.pxd
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
T0
:
np
.
complex128_t
np
.
float64_t
ctypedef
T0
[:]
A
@
cython
.
locals
(
ret
=
A
,
i
=
cython
.
int
)
cpdef
mysum
(
A
arr
)
doc/for_dev/cython_bugs/ctypedef_buffer/pure_ctype_memview.py
0 → 100644
View file @
3c238f68
def
mysum
(
arr
):
ret
=
arr
.
dtype
.
type
(
0
)
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
\ No newline at end of file
doc/for_dev/cython_bugs/ctypedef_buffer/simpler.pyx
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
# not supported, see: https://github.com/cython/cython/issues/754
ctypedef
np
.
ndarray
[
np
.
float64_t
,
ndim
=
1
]
A
def
mysum
(
A
arr
):
cdef
np
.
float64_t
ret
=
arr
.
dtype
.
type
(
0
)
cdef
cython
.
int
i
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
\ No newline at end of file
doc/for_dev/cython_bugs/ctypedef_buffer/works.pyx
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
T0
:
np
.
complex128_t
np
.
float64_t
def
mysum
(
np
.
ndarray
[
T0
,
ndim
=
1
]
arr
):
cdef
T0
ret
=
arr
.
dtype
.
type
(
0
)
cdef
cython
.
int
i
for
i
in
range
(
arr
.
shape
[
0
]):
ret
+=
arr
[
i
]
return
ret
\ No newline at end of file
doc/for_dev/cython_bugs/purepy_fusedtype/bug.pxd
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
A
:
np
.
int_t
[:]
np
.
float_t
[:]
@
cython
.
locals
(
arr
=
A
)
cpdef
func
(
A
arg
)
doc/for_dev/cython_bugs/purepy_fusedtype/bug.py
0 → 100644
View file @
3c238f68
import
numpy
as
np
def
func
(
arg
):
arr
=
np
.
empty_like
(
arg
)
return
arr
doc/for_dev/cython_bugs/purepy_fusedtype/works.pyx
0 → 100644
View file @
3c238f68
import
cython
import
numpy
as
np
cimport
numpy
as
np
ctypedef
fused
A
:
np
.
int_t
[:]
np
.
float_t
[:]
cpdef
func
(
A
arg
):
cdef
A
arr
=
np
.
empty_like
(
arg
)
return
arr
doc/for_dev/typing.md
View file @
3c238f68
...
...
@@ -112,18 +112,31 @@ def f(x, y):
## Transonic
To define multisignatures from fused types, we can do:
To define multi
-
signatures from fused types, we can do:
```
python
from
transonic
import
Array
,
Type
,
NDim
,
boost
A
=
Array
[
Type
(
int
,
float
),
NDim
(
1
,
2
)
,
"memview"
]
A
=
Array
[
Type
(
int
,
float
),
NDim
(
1
,
2
)]
@
boost
def
func
(
a
:
A
,
b
:
A
):
pass
```
Note that this other code should give a different result in term of signatures:
```
python
from
transonic
import
Array
,
Type
,
NDim
,
boost
A0
=
Array
[
Type
(
int
,
float
),
NDim
(
1
,
2
)]
A1
=
Array
[
Type
(
int
,
float
),
NDim
(
1
,
2
)]
@
boost
def
func
(
a
:
A0
,
b
:
A1
):
pass
```
## Issue about fused types
This works with Pythran and it should also work with other backends.
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment