Read about our upcoming Code of Conduct on this issue

Commit c34ade2b authored by Pierre Augier's avatar Pierre Augier
Browse files

pythran_def (v. 0.0.3)

parent 55ccf3f78c26
......@@ -17,6 +17,6 @@ dist*
*.egg-info/*
**/_pythran/_pythran_*
**/_pythran/_*
fluidpythran/_path_data_tests.py
\ No newline at end of file
FluidPythran: Pythran annotations in Python files
=================================================
FluidPythran: use Pythran in non-pythranizable code
===================================================
|release| |coverage|
......@@ -14,22 +14,49 @@ FluidPythran: Pythran annotations in Python files
.. warning ::
This is really just a prototype. See `this post
FluidPythran is a prototype. Remarks and suggestions are very welcome.
FluidPythran just starts to be used in `fluidsim
<https://bitbucket.org/fluiddyn/fluidsim>`_ (for example in `this file
<https://bitbucket.org/fluiddyn/fluidsim/src/c0e170ea7c68f2abc4b0f7749b1c89df79db6573/fluidsim/base/time_stepping/pseudo_spect.py>`_)
See `this post
<http://www.legi.grenoble-inp.fr/people/Pierre.Augier/broadcasting-numpy-abstraction-cython-pythran-fluidpythran.html>`_
(and also `this method
<https://bitbucket.org/fluiddyn/fluidsim/src/c0e170ea7c68f2abc4b0f7749b1c89df79db6573/fluidsim/base/time_stepping/pseudo_spect.py#lines-240>`_)
for an explanation of my motivations.
This pure-Python package will provide few supplementary pythran commands,
namely :code:`pythran block` and :code:`pythran def` (see examples in the doc
folder).
FluidPythran is a pure Python package (requiring Python >= 3.6 or Pypy3) to
help to write Python code that can use Pythran.
**FluidPythran does not depend on Pythran.**
Overview
--------
Python + Numpy + Pythran is a great combo to easily write highly efficient
scientific programs and libraries.
To use Pythran, one needs to isolate the numerical kernels functions in modules
that are compiled by Pythran. The C++ code produced by Pythran never uses the
Python interpretor. It means that only a fraction of what is doable in Python
can be done in Pythran files. This is true in terms of Python features (for
example no classes) and in terms of external packages.
With FluidPythran, we try to overcome these limitations. FluidPythran provides
few supplementary Pythran commands and a tiny Python API to define Pythran
functions without writing the Pythran modules. The code of the numerical
kernels can stay in the modules and in the classes where they were written. The
Pythran files (i.e. the files compiled by Pythran), which are usually written
by the user, are produced automatically by FluidPythran.
**Implementation detail:** for each Python file using FluidPythran, an associated
Pythran file is created in a directory ``_pythran``. For example, for a Python
file ``foo.py``, the associated file would be ``_pythran/_foo.py``.
The code of the numerical kernels can stay in the modules and in the classes
where they were written. The Pythran files (i.e. the files compiled by
Pythran), which are usually written by the user, are produced automatically.
At run time, FluidPythran replaces the Python functions (and blocks) by their
versions in the Pythran files.
Codes using FluidPythran work fine without Pythran.
The code continues to work fine without Pythran, which is used only when
available.
Installation
------------
......@@ -38,6 +65,133 @@ Installation
pip install fluidpythran
Using Pythran in Python files
-----------------------------
Functions: :code:`pythran def`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code :: python
import h5py
import mpi4py
from fluidpythran import pythran_def
# pythran def myfunc(int, float)
@pythran_def
def myfunc(a, b):
return a * b
...
Most of this code looks familiar to Pythran users. The differences:
- One can use (for example) h5py and mpi4py (of course not in the Pythran
functions).
- :code:`# pythran def` instead of :code:`# pythran export` (to stress that it
is not the same command).
- A tiny bit of Python... The decorator :code:`@pythran_def` replaces the
Python function by the pythranized function if FluidPythran has been used to
produced the associated Pythran file.
Blocks: :code:`pythran block`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
One of the most evident application of :code:`# pythran block` is code in
classes:
.. code :: python
from fluidpythran import FluidPythran
fp = FluidPythran()
class MyClass:
...
def func(self, n):
a, b = self.something_that_cannot_be_pythranized()
if fp.is_pythranized:
result = fp.use_pythranized_block("name_block")
else:
# pythran block (
# float a, b;
# int n
# ) -> result
# pythran block (
# complex a, b;
# int n
# ) -> result
result = a**n + b**n
return self.another_func_that_cannot_be_pythranized(result)
For blocks, we need pa little bit more of Python.
- At import time, we have :code:`fp = FluidPythran()`, which detect which
Pythran module should be used and import it. This is done at import time since
we want to be very fast at run time.
- In the function, we define a block with three lines of Python and special
Pythran annotations (:code:`# pythran block`). The 3 lines of Python are used
(i) at run time to choose between the two branches (:code:`is_pythranized` or
not) and (ii) at compiled time to detect the blocks.
.. warning ::
The two branches of the :code:`fp.is_pythranized` are not equivalent, so
the user has to be careful because it is not difficult to write such buggy
code:
.. code ::
c = 0
if fp.is_pythranized:
a, b = fp.use_pythranized_block("name_block")
else:
# pythran block () -> (a, b)
a = b = c = 1
print(c)
.. warning ::
The Pythran keywork :code:`or` cannot yet be used in block annotations.
Make the Pythran files
----------------------
There is a command-line tool ``fluidpythran`` which make the associated Pythran
files from Python files with annotations and fluidpythran code.
There is also a function :code:`make_pythran_files` that can be used in a
setup.py like this:
.. code ::
from pathlib import Path
from fluidpythran.files_maker import make_pythran_files
here = Path(__file__).parent.absolute()
paths = ["fluidsim/base/time_stepping/pseudo_spect.py"]
make_pythran_files([here / path for path in paths])
Note that FluidPythran never uses Pythran. Compiling the associated Pythran
file can be done if wanted (see for example how it is done in `fluidsim's
setup.py <https://bitbucket.org/fluiddyn/fluidsim/src/default/setup.py>`_).
License
-------
......
import numpy as np
from fluidpythran import FluidPythran
fp = FluidPythran()
from fluidpythran import pythran_def
# pythran import numpy as np
......@@ -14,6 +12,6 @@ fp = FluidPythran()
# pythran def func(int[][], float[][])
@fp.pythranize
@pythran_def
def func(a, b):
return (a * np.log(b)).max()
......@@ -3,9 +3,7 @@ import numpy as np
from fluiddyn.util import mpi
from fluidpythran import FluidPythran
fp = FluidPythran()
from fluidpythran import pythran_def
# pythran import numpy as np
......@@ -13,7 +11,7 @@ fp = FluidPythran()
# pythran def func(int[][], float[][])
@fp.pythranize
@pythran_def
def func(a, b):
return (a * np.log(b)).max()
......
import inspect
import os
import importlib.util
from ._version import __version__
......@@ -13,24 +12,44 @@ except ImportError:
__all__ = ["__version__", "FluidPythran", "path_data_tests"]
_modules = {}
def pythran_def(func):
frame = inspect.stack()[1]
module_name = get_module_name(frame)
if module_name in _modules:
fp = _modules[module_name]
else:
fp = FluidPythran(frame=frame)
return fp.pythran_def(func)
def get_module_name(frame):
module_name = inspect.getmodule(frame[0]).__name__
if module_name == "__main__":
module_name = inspect.getmodulename(frame.filename)
return module_name
class FluidPythran:
def __init__(self, use_pythran=True):
def __init__(self, use_pythran=True, frame=None):
if not use_pythran or "FLUIDPYTHRAN_COMPILING" in os.environ:
if not use_pythran:
self.is_pythranized = False
return
frame = inspect.stack()[1]
module_name = inspect.getmodule(frame[0]).__name__
if module_name == "__main__":
module_name = inspect.getmodulename(frame.filename)
if frame is None:
frame = inspect.stack()[1]
module_name = get_module_name(frame)
if "." in module_name:
package, module = module_name.rsplit(".", 1)
module_pythran = package + "._pythran._pythran_" + module
module_pythran = package + "._pythran._" + module
else:
module_pythran = "_pythran._pythran_" + module_name
module_pythran = "_pythran._" + module_name
try:
self.module_pythran = importlib.import_module(module_pythran)
......@@ -43,7 +62,9 @@ class FluidPythran:
self.module_pythran, "arguments_blocks"
)
def pythranize(self, func):
_modules[module_name] = self
def pythran_def(self, func):
"""Decorator used for functions"""
if self.is_pythranized:
return getattr(self.module_pythran, func.__name__)
......
__version__ = "0.0.2"
__version__ = "0.0.3"
try:
from pyfiglet import figlet_format
__about__ = (
figlet_format("fluidpythran", font="big")
)
__about__ = figlet_format("fluidpythran", font="big")
except ImportError:
__about__ = r"""
......
......@@ -147,7 +147,6 @@ def get_code_functions(path_py, func_names):
if in_def and in_def != "def":
if indent == 0 and toknum == DEDENT:
print("codes[in_def]", codes[in_def])
codes[in_def] = untokenize(codes[in_def])
in_def = False
else:
......@@ -158,7 +157,7 @@ def get_code_functions(path_py, func_names):
return codes
def create_pythran_code(path_py):
def make_pythran_code(path_py):
"""Create a pythran code from a Python file"""
blocks, signatures_blocks, code_blocks, functions, signatures_func, imports = parse_py(
......@@ -255,14 +254,14 @@ def has_to_build(output_file, input_file):
return False
def create_pythran_file(path_py, force=False, log_level=None):
def make_pythran_file(path_py, force=False, log_level=None):
"""Create a Python file from a Python file (if necessary)"""
if log_level is not None:
set_log_level(log_level)
path_py = Path(path_py)
if path_py.name.startswith("_pythran_"):
if path_py.absolute().parent.name == "_pythran":
logger.debug(f"skip file {path_py}")
return
if not path_py.name.endswith(".py"):
......@@ -271,13 +270,13 @@ def create_pythran_file(path_py, force=False, log_level=None):
)
path_dir = path_py.parent / "_pythran"
path_pythran = path_dir / ("_pythran_" + path_py.name)
path_pythran = path_dir / ("_" + path_py.name)
if not has_to_build(path_pythran, path_py) and not force:
logger.info(f"File {path_pythran} already up-to-date.")
return
code_pythran = create_pythran_code(path_py)
code_pythran = make_pythran_code(path_py)
if not code_pythran:
return
......@@ -302,7 +301,7 @@ def create_pythran_file(path_py, force=False, log_level=None):
return path_pythran
def create_pythran_files(paths, force=False, log_level=None):
def make_pythran_files(paths, force=False, log_level=None):
"""Create Pythran files from a list of Python files"""
if log_level is not None:
......@@ -310,7 +309,7 @@ def create_pythran_files(paths, force=False, log_level=None):
paths_out = []
for path in paths:
path_out = create_pythran_file(path, force=force)
path_out = make_pythran_file(path, force=force)
if path_out:
paths_out.append(path_out)
......
......@@ -2,18 +2,28 @@ import numpy as np
# pythran import numpy as np
from fluidpythran import FluidPythran
from fluidpythran import FluidPythran, pythran_def
fp = FluidPythran()
# pythran def func(int, float)
@fp.pythranize
@pythran_def
def func(a, b):
return a + b
# pythran def func2(int, float)
@pythran_def
def func2(a, b):
return a - b
fp = FluidPythran()
def func1(a, b):
n = 10
......
......@@ -8,7 +8,7 @@ from pathlib import Path
from glob import glob
from . import __version__
from .files_maker import create_pythran_files
from .files_maker import make_pythran_files
from .log import logger, set_log_level
doc = """
......@@ -31,10 +31,7 @@ def parse_args():
)
parser.add_argument(
"-V",
"--version",
help="print version and exit",
action="store_true",
"-V", "--version", help="print version and exit", action="store_true"
)
parser.add_argument("-v", "--verbose", help="verbose mode", action="count")
......@@ -71,4 +68,4 @@ def run():
else:
paths = glob(str(path))
create_pythran_files(paths, force=args.force)
make_pythran_files(paths, force=args.force)
from .for_test_init import func, func1
from . import FluidPythran
def test_not_fluidpythranized():
func(1, 3.14)
func1(1.1, 2.2)
def test_use_pythran_false():
FluidPythran(use_pythran=False)
......@@ -2,7 +2,7 @@ from pathlib import Path
import importlib
import unittest
from .files_maker import create_pythran_file
from .files_maker import make_pythran_file
path_for_test = Path(__file__).parent / "for_test_init.py"
......@@ -10,14 +10,12 @@ assert path_for_test.exists()
print(path_for_test)
path_output = path_for_test.parent / ("_pythran/_pythran_" + path_for_test.name)
path_output = path_for_test.parent / ("_pythran/_" + path_for_test.name)
if path_output.exists():
path_output.unlink()
create_pythran_file(path_for_test)
print("path_output", path_output)
make_pythran_file(path_for_test)
from . import for_test_init
......
......@@ -28,7 +28,7 @@ def test_create_pythran_files():
path_file.touch()
run()
path_file = path_data_tests / "_pythran/_pythran_classic.py"
path_file = path_data_tests / "_pythran/_classic.py"
path_file.unlink()
run()
......
......@@ -19,7 +19,7 @@ source = ./fluidpythran
data_file = .coverage/coverage
omit = */test*.py
*/_try_*.py
fluidpythran/_pythran/_pythran_for_test_init.py
fluidpythran/_pythran/_for_test_init.py
[coverage:report]
show_missing = True
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment