diff --git a/.hgignore b/.hgignore index bd619c09a14d03788cdfa808004dd46a7c4376bf_LmhnaWdub3Jl..75c84a3ae1316eaed4169d63cce2524af922a286_LmhnaWdub3Jl 100644 --- a/.hgignore +++ b/.hgignore @@ -14,6 +14,7 @@ *.pdf *.synctex.gz *.backup +*.pyg */*.pstats diff --git a/fluiddyn/Makefile b/fluiddyn/Makefile index bd619c09a14d03788cdfa808004dd46a7c4376bf_Zmx1aWRkeW4vTWFrZWZpbGU=..75c84a3ae1316eaed4169d63cce2524af922a286_Zmx1aWRkeW4vTWFrZWZpbGU= 100644 --- a/fluiddyn/Makefile +++ b/fluiddyn/Makefile @@ -12,5 +12,5 @@ rm -f $(name).pdf $(name).pdf: $(name).tex - pdflatex $(name).tex - if [ `grep "Package rerunfilecheck Warning: File" $(name).log | wc -l` != 0 ]; then pdflatex $(name).tex; fi + pdflatex -shell-escape $(name).tex + if [ `grep "Package rerunfilecheck Warning: File" $(name).log | wc -l` != 0 ]; then pdflatex -shell-escape $(name).tex; fi diff --git a/fluiddyn/fluiddyn_paper.tex b/fluiddyn/fluiddyn_paper.tex index bd619c09a14d03788cdfa808004dd46a7c4376bf_Zmx1aWRkeW4vZmx1aWRkeW5fcGFwZXIudGV4..75c84a3ae1316eaed4169d63cce2524af922a286_Zmx1aWRkeW4vZmx1aWRkeW5fcGFwZXIudGV4 100644 --- a/fluiddyn/fluiddyn_paper.tex +++ b/fluiddyn/fluiddyn_paper.tex @@ -29,7 +29,7 @@ % software and the area it relates to rather than making claims about the % software, e.g. “Easy-to-useâ€.} -FluidDyn: a Python open-source suite for research and teaching in fluid +FluidDyn: a Python open-source framework for research and teaching in fluid dynamics \section*{Paper Authors} @@ -63,14 +63,29 @@ described: what problem the software addresses, how it was implemented and architected, where it is stored, and its reuse potential.} -FluidDyn is a project to foster open-source with Python in the fluid dynamics -community. A set of packages forms a framework to study fluid dynamics with -different methods, in particular laboratory experiments, simulations and data -processing. We first list the specialized packages of the project and then -focus on the base package called fluiddyn, which contains basic tools used in -the specialized packages. - - +\href{http://fluiddyn.readthedocs.io}{FluidDyn} is a project to foster +open-science and open-source in the fluid dynamics community. A set of Python +packages forms a framework to study fluid dynamics with different methods, in +particular laboratory experiments (package \fluidpack{lab}), simulations +(packages \fluidpack{fft}, \fluidpack{sim} and \fluidpack{foam}) and data +processing (package \fluidpack{image}). In this article, we give an overview +of the specialized packages of the project (\fluidpack{fft} and \fluidpack{sim} +are described in two companion papers) and then focus on the base package +called \fluidpack{dyn}, which contains basic tools used in the specialized +packages. +% +With the projet FluidDyn, we try to demonstrate that specialized scientific +code can be written with methods and good practices of the open-source +community. The Mercurial repositories are available in bitbucket +(\url{https://bitbucket.org/fluiddyn/}). All codes are documented using sphinx +and readthedocs, and tested with continuous integration run on Bitbucket +Pipeline and Travis. +% +To improve the reuse potential, the codes are as modular as possible taking +advantage of the simple object-oriented programming model of Python. +% +All codes are also written to be highly efficient, using C++, Cython and +Pythran for to speedup the performance critical functions. \section*{Keywords} @@ -88,5 +103,5 @@ articles. A short comparison with software which implements similar functionality should be included in this section. } -\subsection{Software, open-source and sciences} +\subsection*{Science, software, open-source and the computer revolution} @@ -92,8 +107,7 @@ -Science is mainly a collective activity. -\href{https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_giants}{"Standing -on the shoulders of giants"}: how to build new knowledge from the work of -others. Need to exchange ideas. +Science is mainly a collective activity. \href{https://en.wikipedia.org/wiki/% +Standing_on_the_shoulders_of_giants}{"Standing on the shoulders of giants"}: +how to build new knowledge from the work of others. Need to exchange ideas. These last decades, we have lived a revolution on how people exchange ideas. @@ -97,5 +111,6 @@ These last decades, we have lived a revolution on how people exchange ideas. - +Computers of all kinds (from smartphones to computer clusters) connected by a +world wide web are very common and used for so many applications. @@ -101,3 +116,8 @@ -Software and programming in science. +A lot of money and work. Approximately 5\% PIB. Many companies, some very +powerfull (Google, Facebook). + +This is changing our world and also the way science is done. + +Software and programming in science take a much bigger place than before. @@ -103,5 +123,8 @@ -Big changes in software engineering, driven primarily by web. A lot of money -and work. Approximately 5\% PIB. Very powerful companies. Open-source, -distributed source management tools and web. +Change the status of software. Before programs were badly considered. A +mathematical demonstration has to be included in a paper but not a code. Now +codes are at the center of research. + +Big changes in software engineering, driven primarily by web. Massive use of +the open-source methods, distributed source management tools and web. @@ -107,3 +130,5 @@ -GPU. Artificial intelligence. Data science. +Increase computer power, in particular with GPU and increase knowledge +algorithms of artificial intelligence. Data science. Big boom. Even more money +in software engineering and open-source software. @@ -109,4 +134,5 @@ -Using the open-source methods and tools for science: ``open-science''. +Using the open-source methods and tools for science and sharing/collaborating +via the web: ``open-science''. One very important tool in recent open-source dynamics and in particular in @@ -111,9 +137,8 @@ One very important tool in recent open-source dynamics and in particular in -science: Python -(see -\href{https://stackoverflow.blog/2017/09/06/incredible-growth-python/}{this -blog article} and -\href{https://spectrum.ieee.org/computing/software/the-2017-top-programming-languages}{this -analysis on the top programming languages 2017}) +science: Python (see \href{https://stackoverflow.blog/2017/09/06/% +incredible-growth-python/}{this blog article} and +\href{https://spectrum.ieee.org/computing/software/% +the-2017-top-programming-languages}{this analysis on the top programming +languages 2017}) @@ -119,4 +144,6 @@ -Fluiddyn: project to foster open-source in fluid mechanics. Based on Python. +This paper presents Fluiddyn: a project to foster open-source in fluid +mechanics. Based on the language Python. The characteristics of Python make +possible the success of such project. Python in fluid mechanics is growing fast but is still only weakly used. It is @@ -121,4 +148,6 @@ Python in fluid mechanics is growing fast but is still only weakly used. It is -necessary to present characteristics of this tool. +thus still necessary to present its characteristics. + +\subsection*{Python} @@ -124,3 +153,5 @@ -\subsection{Python} +Python is a dynamical general purpose programming language designed to improve +communication of technical ideas between humans thanks to its very powerful and +simple syntax. @@ -126,3 +157,5 @@ -Dynamical language designed to help communication between humans. +Dynamical... Variables (names) not attached to an object. Types of objects +inferred from the code and in many situation not written explicitly. Automatic +memory management. @@ -128,4 +161,9 @@ -Simple use: interpreted. A program called the interpreter execute instruction -by instruction the code, with very few optimization. +The most simple way to run a Python code is to ``interpret'' it\footnote{There +are also tools to compile Python code (usually via C or C++ code, for example +Cython and Pythran).}. A program called the interpreter executes instruction +by instruction the code, with very few optimization. In this case, there is no +proper compilation, transforming the code in machine instructions. In contrast +to other languages as Fortran, C or C++ which can not be interpreted and which +have to be compiled. @@ -131,4 +169,6 @@ -There are tools to compile Python code (via C or C++ code, cf Cython and -Pythran). +A disadvantage is that badly written Python code can be very slow. But there +are many strong advantages. Developing with Python is much faster than for many +other languages, typically one order of magnitude faster than C++ and 5x faster +than Java. @@ -134,6 +174,11 @@ -A "glue" language for fast prototyping. Much faster development than for many -other languages (typically 10x faster than C++, 5x faster than Java). +Less bugs because much less lines of code. + +Code easier to read and understand. + +Easy to use code written in other languages, in particular C, C++ and Fortran. + +Then, Python is one of the best "glue" languages for fast prototyping. Very powerful and generalist language (multiparadigm, monkey patching, great standard library and external libraries to do many things). @@ -163,5 +208,5 @@ like in particular Anaconda. One of the main languages for data science with the packages pandas, -statmodels, scikit-learn, ... +statmodels, scikit-learn, TensorFlow... @@ -167,3 +212,3 @@ -\subsubsection{Drawbacks} +\subsubsection*{Drawbacks} @@ -169,5 +214,5 @@ -\subparagraph{No compilation, so relatively slow.} It is first a "glue" language. -Common strategy: write extensions in compiled languages to speedup the -performance critical code. +\subparagraph{Since the standard Python interpreter does not do any proper +compilation,} pure Python code can be is some cases (CPU bounded programs) too +slow. @@ -173,7 +218,9 @@ -Don't need to write the compiled extensions in C or C++: Cython and Pythran -(Ahead Of Time compiling). +Ahead Of Time compiling... Python is first a "glue" language so the most simple +strategy is to use extensions in compiled languages to speedup the performance +critical code. We don't need to write the compiled extensions in C, C++ or +Fortran: Cython and Pythran. JIT (Just In Time) compilation. Characteristics of the language and of the main interpreter CPython make it very difficult to write an interpreter with a JIT compatible with the compiled extensions written for CPython. @@ -176,9 +223,13 @@ JIT (Just In Time) compilation. Characteristics of the language and of the main interpreter CPython make it very difficult to write an interpreter with a JIT compatible with the compiled extensions written for CPython. -Pypy... Numba... Other attempts (Pyston). See -https://faster-cpython.readthedocs.io/ + +Pypy... + +Numba... + +Other attempts (Pyston). See https://faster-cpython.readthedocs.io/ Glue language. Fast Python with a JIT is actually not so necessary. Not as much money put in accelerating Python than for example for JavaScript. @@ -191,7 +242,7 @@ Computational parallelism with the GIL: extensions, multiprocessing and mpi. -\subsubsection{A versatile Swiss Army Knife for the scientists} +\subsubsection*{A versatile Swiss Army Knife for the scientists} Python has become a great versatile tool for scientists. The scientific ecosystem of Python is nowadays really impressive. @@ -205,7 +256,7 @@ and possibilities (type checking, ) -\subsection{Rich landscape of open-source languages useful for science} +\subsection*{Rich landscape of open-source languages useful for science} There are many interesting open-source languages that can be used for scientific applications. All have strengths and weaknesses. @@ -221,8 +272,8 @@ Python in this landscape... -\subsection{Productivity at individual, group and community levels} +\subsection*{Productivity at individual, group and community levels} + discussion on short-term and middle-term productivity... Quick and dirty scripts can be efficient in the short term, but ... @@ -225,8 +276,8 @@ + discussion on short-term and middle-term productivity... Quick and dirty scripts can be efficient in the short term, but ... -\subsection{Programming in the field of fluid mechanics} +\subsection*{Programming in the field of fluid mechanics} Fluid mechanics: experiments and in situ measurements, analytic computation, numerical simulations and data processing. @@ -253,8 +304,7 @@ Nevertheless there are also a lot of code produced by workers in the field. But without the technical knowledge on how to work collectively on code, this production is most of the case of very bad quality (compared to the industry -standard) and it is basically doomed to be put at the trash after its first -use. +standard) and it is basically doomed to be abandon after its use. However, it is natural to try to reuse code or at least to build the next steps on what have been done before. Thousands of hours of highly qualified people @@ -278,8 +328,8 @@ Matlab is a good tool for simple processing with matrices and data plotting. The language is well adapted for these tasks. The development -environment is nice and the interpreter is fast, in particular now that it has -a JIT compiler. +environment is nice and the interpreter is quite fast, in particular now that +it has a JIT compiler. However, the language suffers from serious technical problems which make Matlab a bad tool for doing more than simple processing and data plotting. The @@ -283,7 +333,7 @@ However, the language suffers from serious technical problems which make Matlab a bad tool for doing more than simple processing and data plotting. The -comparison with Python is eloquent. Matlab weaknesses are +comparison with Python is eloquent. Matlab weaknesses are in particular \begin{itemize} \item one file for each function, @@ -287,8 +337,10 @@ \begin{itemize} \item one file for each function, -\item no namespaces and no real organization of the standard library, +\item no real organization of the standard library. All function available in a +huge flat namespace. No import mechanism so that we do not know looking at the +code from where a function is coming. \item no default argument mechanism, \item weak error handling, \item no object oriented programming, \item massive use of ";" and ".*" and nothing like the Python pep8, which makes @@ -291,7 +343,7 @@ \item no default argument mechanism, \item weak error handling, \item no object oriented programming, \item massive use of ";" and ".*" and nothing like the Python pep8, which makes -a huge proportion of the Matlab codes difficult to read and understand. +a large proportion of the Matlab codes difficult to read and understand. \end{itemize} @@ -296,7 +348,6 @@ \end{itemize} - Matlab is a big black box so it is impossible to check the implementation of its functions. Matlab is not free. The price for one license for non-commercial use is not @@ -299,9 +350,13 @@ Matlab is a big black box so it is impossible to check the implementation of its functions. Matlab is not free. The price for one license for non-commercial use is not -very expensive. Still it would be interesting to know what could be done with -open-source solutions with the total amount paid by public research +very expensive for most lab. However, when we start to add the price of the +toolboxes necessary to run most of the codes, it starts to be non negligible +for some institutions. + +It would be interesting to estimate what could be done with open-source +solutions with a part of the total amount paid by public research institutes. How many developers could be paid to improve the alternative open-source solutions? @@ -312,7 +367,7 @@ There are also open-source Matlab interpreters called Octave and Scilab. Unfortunately, they are not serious alternative today. Their user community is too small and they are not very efficient, especially now that -Matlab uses a JIT compiler and that optimized Matlab code realized on it. +Matlab uses a JIT compiler and that optimized Matlab codes rely on it. To conclude on Matlab, if some people can use it efficiently for their work, it starts to be a problem for the collective efficiency of the community. Except @@ -326,4 +381,7 @@ importantly, it forbids the use of source management tools which are so important for open-source. +Matlab and Labview are also problems for students, who learns bad coding habits +and languages that are much less in demand by employers than for example +Python. @@ -329,7 +387,8 @@ +For symbolic computation Mathematica. Industrial Computational Fluid Dynamics (CFD) is still dominated by (for example Fluent). Analyzing of images of fluid: Dantec and Lavision. @@ -330,14 +389,14 @@ Industrial Computational Fluid Dynamics (CFD) is still dominated by (for example Fluent). Analyzing of images of fluid: Dantec and Lavision. -software developed by researcher for example Digiflow -http://www.damtp.cam.ac.uk/user/fdl/digiflow/index.htm. +software developed by researcher for example +\href{http://www.damtp.cam.ac.uk/user/fdl/digiflow/index.htm}{Digiflow}. Another very common model used in fundamental research is to use code shared by few people at the level of one group (for example ns3d, diablo). Share for ``friends'' @@ -338,21 +397,15 @@ Another very common model used in fundamental research is to use code shared by few people at the level of one group (for example ns3d, diablo). Share for ``friends'' -Some open-source codes have also emerged. - -- NEK5000 https://nek5000.mcs.anl.gov/ (Fortran) - -- OpenFOAM (C++) - -- Basilisk http://basilisk.fr (C) - -- Channelflow (http://channelflow.org/, C++) - +Some open-source codes have also emerged, for example +\href{https://nek5000.mcs.anl.gov/}{NEK5000} (Fortran), OpenFOAM (C++), +\href{http://basilisk.fr}{Basilisk} (C) and +\href{http://channelflow.org/}{Channelflow} (C++). People start to use Python, in particular with CFD codes dedalus, spectralDNS and Oasis https://github.com/mikaem/Oasis. @@ -354,7 +407,6 @@ People start to use Python, in particular with CFD codes dedalus, spectralDNS and Oasis https://github.com/mikaem/Oasis. -For symbolic computation Mathematica @@ -360,6 +412,5 @@ - -\subsection{FluidDyn project} +\subsection*{FluidDyn project} Fluiddyn is a project to foster open-science and open-source coding in Python in the field of fluid mechanics. @@ -384,7 +435,7 @@ architecture where relevant. Use of relevant diagrams is appropriate. Please also describe any variants and associated implementation differences.} -\subsection{Organization of the code in packages} +\subsection*{Organization of the code in packages} FluidDyn was originally intended to be a single package to perform experiments and simulations. Since a typical user may not be involved in both experiments @@ -393,7 +444,7 @@ Now, FluidDyn project hosts a number of specialized packages, namely: \begin{itemize} - \item FluidDyn : The base package which provides necessary Application +\item fluiddyn: The base package which provides necessary Application Programming Interfaces (API) and backend for the packages below. It also defines miscellaneous utilities to do repetitive tasks required by a typical fluid dynamics user. @@ -401,7 +452,10 @@ All pure-python library that can be reused in scripts or in specialized fluid... packages + command-line utilities. - \item FluidSim : Numerically oriented package to run sequential and parallel +\item fluidfft: A library which provides C++, CUDA / OpenCL? and Python +classes useful to perform fast Fourier transform (FFT). + +\item fluidsim: Numerically oriented package to run sequential and parallel Computational Fluid Dynamics (CFD) simulations and on-the-fly post-processing for a variety of problems (Navier-Stokes, Shallow Water, F\"oppl von K\'arm\'an equations, to name a few). @@ -405,6 +459,6 @@ Computational Fluid Dynamics (CFD) simulations and on-the-fly post-processing for a variety of problems (Navier-Stokes, Shallow Water, F\"oppl von K\'arm\'an equations, to name a few). - \item FluidLab : Package to handles laboratory experiments. Primarily used to +\item fluidlab: Package to handles laboratory experiments. Primarily used to communicate with various hardware devices such as motors and pumps, to handle I/O between sensors, and to store data. @@ -409,5 +463,5 @@ communicate with various hardware devices such as motors and pumps, to handle I/O between sensors, and to store data. - \item FluidImage : Scalable image processing package which implements various +\item fluidimage: Scalable image processing package which implements various algorithms to calibrate cameras, to preprocess, to do Particle Image Velocimetry (PIV) and to postprocess data. @@ -412,6 +466,5 @@ algorithms to calibrate cameras, to preprocess, to do Particle Image Velocimetry (PIV) and to postprocess data. - \item FluidFFT : A library which provides C++, CUDA / OpenCL? and Python -classes useful to perform fast Fourier transform (FFT). + \end{itemize} @@ -416,8 +469,9 @@ \end{itemize} -A detailed discussion on the above packages can be found in their respective -articles. This article will now focus on the base package FluidDyn. +A detailed presentation on the above packages can be found in their respective +documentations on the we and for fluidfft and fluidsim in the two companion +papers. This article will now focus on the base package FluidDyn. All functions and classes defined in FluidDyn are pure Python elements. The code base was designed to follow Python 2.7 syntax during its genesis. Now, it has been made forward compatible with Python 3 through the use of external @@ -420,6 +474,6 @@ All functions and classes defined in FluidDyn are pure Python elements. The code base was designed to follow Python 2.7 syntax during its genesis. Now, it has been made forward compatible with Python 3 through the use of external -package \emph{future}. +package \codeinline{future}. @@ -425,8 +479,8 @@ -The package FluidDyn is organized into four sub-packages: io, util, clusters -and output. +The package FluidDyn is organized into five sub-packages: io, util, calcul, +clusters and output. \subsection*{The Python library fluiddyn (API)} Application Programming Interfaces @@ -428,11 +482,12 @@ \subsection*{The Python library fluiddyn (API)} Application Programming Interfaces -Sub-packages \emph{io} and \emph{util}, are the largest in terms of lines of -code and provides the APIs to support FluidSim, FluidLab and FluidImage. The -common code base to function for such varied applications is a proof of its -versatility and generality. For the sake of brevity, we shall look at the most -important modules, viz. \emph{paramcontainer}, \emph{serieofarrays} +Sub-packages \codeinline{io}, calcul and \codeinline{util}, are the largest in +terms of lines of code and provides the APIs to support fluidsim, fluidlab and +fluidimage. The common code base to function for such varied applications is a +proof of its versatility and generality. For the sake of brevity, we shall look +at the most important modules, viz. \codeinline{paramcontainer}, +\codeinline{serieofarrays} @@ -438,5 +493,5 @@ -Sub-package \emph{paramcontainer} defines class \emph{ParamContainer} which -is a hierarchical container for any type of parameters. Various strengths of an -object of this class include: +Sub-package \codeinline{paramcontainer} defines class +\codeinline{ParamContainer} which is a hierarchical container for any type of +parameters. Various strengths of an object of this class include: \begin{itemize} @@ -442,6 +497,6 @@ \begin{itemize} - \item Support printing to console and saving as XML - \item Evaluate data types automatically while loading from saved files - \item Easy expansion of default parameters dynamically, through function calls - \item Allows modification of default parameters through simple Python script +\item Support printing to console and saving as XML +\item Evaluate data types automatically while loading from saved files +\item Easy expansion of default parameters dynamically, through function calls +\item Allows modification of default parameters through simple Python script files. @@ -447,5 +502,5 @@ files. - \item Graphical User Interface (GUI) frontend +\item Graphical User Interface (GUI) frontend \end{itemize} Thus, it makes it a much more robust implementation for saving key parameters, diff --git a/jors.cls b/jors.cls index bd619c09a14d03788cdfa808004dd46a7c4376bf_am9ycy5jbHM=..75c84a3ae1316eaed4169d63cce2524af922a286_am9ycy5jbHM= 100644 --- a/jors.cls +++ b/jors.cls @@ -78,6 +78,8 @@ \usepackage{listings} \usepackage{fancyref} +\usepackage{minted} + %% Set source code listings style \lstset{basicstyle=\ttfamily, language=Python} @@ -88,4 +90,11 @@ \rhead{\footnotesize 3} \rhead{\textcolor{gray}{UP JORS software Latex paper template version 0.1}} + +\newcommand{\fluidpack}[1]{\href{http://fluid#1.readthedocs.io}{fluid#1}} + +% \newcommand{\codeinline}[1]{\mintinline{python}{#1}} +\newcommand{\codeinline}[1]{\texttt{#1}} + + \endinput \ No newline at end of file