# HG changeset patch # User paugier <pierre.augier@ens-lyon.org> # Date 1516871263 -3600 # Thu Jan 25 10:07:43 2018 +0100 # Node ID ed42b753876893e8c4b7724b95edb69a5e11f69f # Parent fcb9fa612f13a99a3fc3c3530116fe11eaa7a247 Before Ashwin on fluiddyn... diff --git a/fluiddyn/fluiddyn_paper.tex b/fluiddyn/fluiddyn_paper.tex --- a/fluiddyn/fluiddyn_paper.tex +++ b/fluiddyn/fluiddyn_paper.tex @@ -96,8 +96,8 @@ % Keywords should make it easy to identify who and what the software will be % useful for.} -Fluid dynamics research with Python. Modular, object-oriented, collaborative, -efficiency, tested, documented, free and open-source software. +Fluid dynamics research with Python. Free and open-source software, modular, +object-oriented, collaborative, efficient, tested, documented. \section*{Introduction} @@ -112,13 +112,13 @@ \href{https://en.wikipedia.org/wiki/% Standing_on_the_shoulders_of_giants}{``standing on the shoulders of giants''} (and of a huge number of technicians and scientists). Science is a lot about -how to build new knowledge from the work of others so of course, one important -aspect is the exchange of ideas. +how to build new knowledge from the work of others so of course, the exchange +of ideas is a fundamental aspect. These last decades, we have lived a revolution on how people exchange ideas. % Computers of all kinds (from smartphones to HPC clusters) connected by a world -wide web are used for human communication but also for many other applications. +wide web are used for human communication and also for many other applications. % It has become very easy to reproduce and exchange ideas and data. % @@ -128,22 +128,25 @@ The information technology revolution opens fantastic opportunities for human collaboration and of course for science. -The web related activities represent approximately 5\% of GDP in the USA! A -huge amount of money (and work) is spent for the related technologies. We all -know the most powerful and famous companies involved in this dynamics -(Google, Facebook), but there are several others smaller and less known. Most -of these companies base part of their work on the open-source paradigm and use -and contribute to open-source languages, libraries, software and operating -systems. +The web related activities represent in 2017 approximately 5\% of GDP in the +USA! +% reference? +A huge amount of money (and work) is spent for the related technologies. We +all know the most powerful and famous companies involved in this dynamics +(Google, Facebook, ...), but there are several other companies smaller and less +known. Most of these companies base part of their work on the open-source +paradigm and use and contribute to open-source languages, libraries, software +and operating systems. % This has lead to deep changes in software engineering, with a massive use of open-source methods and tools, for example distributed source management tools and web-based source development tools. The computer performance still increases exponentially, now also with the help -of GPU. This has allowed a big boom in practical uses of data science and -machine learning, which drives a strong research on artificial intelligence. -This dynamics contributes to progresses in open-source software. +of Graphical Processing Units (GPU). This has allowed a big boom in practical +uses of data science and machine learning, which drives a strong research on +artificial intelligence. This dynamics contributes to progresses in +open-source software. These dynamics are changing our world and also the way science is done. % @@ -154,7 +157,7 @@ % https://www.numfocus.org/ % The status of software has changed. Before coding was sometimes badly -considered by some scientists. It was thought that a mathematical +considered by some scientists. It is thought that a mathematical demonstration has to be elegant and that it is normal to included it in a paper. In contrast, it was normal to write crappy code and to just show the results. Now, codes tend to be much more at the center of research. @@ -162,24 +165,25 @@ ``Open-science'' is a new trend taking advantage of these new facts. Pioneers try to do better science, improving reproducibility and collective efficiency, by using the open-source methods and tools for science and sharing and -collaborating via the web. +collaborating via the world wide web. % http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261 This paper presents FluidDyn: a project to foster open-science and open-source in fluid mechanics. This first section ``Overview'' is organized in two large -parts. A long introduction presents the context and describes our motivation +parts. A long introduction presents the context and describes our motivations and the concept of the project FluidDyn. % The second subsection called ``Implementation and architecture'' is devoted to present the organization of the packages of the project and to describe in more -details the base package fluiddyn. +details the base package \fluiddyn\footnote{We use FluidDyn (with capital +letters) to name the project and \fluiddyn for the base package.}. The following of this introduction is organized as follow. \begin{itemize} \item A scientific project based on the collaboration through open-source as FluidDyn is possible only thanks to the recent developments in methods and -tools for Open-source software engineering. We start by presenting these +tools for open-source software engineering. We start by presenting these developments and discuss the implications for the scientists. \item FluidDyn is based on the language Python. Moreover, these are the @@ -188,13 +192,14 @@ Python is a very important tool in recent open-source dynamics and in particular in science (see \href{https://stackoverflow.blog/2017/09/06/% -incredible-growth-python/}{this article of the stackoverflow blog} and +incredible-growth-python/}{this article of the Stackoverflow blog} and \href{https://spectrum.ieee.org/computing/software/% the-2017-top-programming-languages}{this analysis on the top programming languages 2017}). % Python in fluid mechanics is growing fast but is still only weakly used. It is -thus still necessary to present its characteristics. +thus still useful to present its characteristics for readers potentially not +specialist in programming. \item We will discuss about the possible contradiction between productivity of individuals and productivity at the community level. @@ -207,8 +212,7 @@ \end{itemize} -\subsection*{Methods and tools for Open-source software engineering} - +\subsection*{Methods and tools for open-source software engineering} \subparagraph{Distributed source management tools and web-based source development tools.} The collective collaboration on development has been @@ -217,17 +221,30 @@ Github, Bitbucket and more recently Gitlab. % These tools are designed to work collectively on text files so they could also -be very useful to write scientific articles. +be very useful to collectively write scientific documents. % These tools are so useful for collective work that a young scientific of today -should should now master them and be aware of the notions of repository, +should now master them and be aware of the notions of repository, changeset, commit, clone, pull, push, fork and pull request. -Git has become the de facto standard for source management. However, for the +Git has become the \textit{de facto} standard for source management. However, for the FluidDyn project, we prefer to use Mercurial because we consider that it is simpler for the beginners and as powerful for the expert users. The simplicity for the scientists who are not expert in development is especially important for our project. +% +Note that it is now possible to use Mercurial to work with Git repositories +hosted for example in Github and Gitlab\footnote{By using the Mercurial +extension hg-git.}. + +%todo +\subparagraph{Third-party software repositories.} Python Package Index +(Pypi). Started with the Comprehensive Perl Archive Network (CPAN) for the +language Perl... The Comprehensive R Archive Network (CRAN) for R is also +important. + +%todo +Remark Conda... \subparagraph{Continuous integration.} The need of strong reliability of programs of increasing complexity has forced the developers to use continuous @@ -240,12 +257,22 @@ dynamical languages and open-source code since it allows open-source projects to welcome participation from many people while keeping a high reliability. +%todo +Remark codecov... and other similar tools for quality control. + \subparagraph{Websites to share knowledge.} Learning programming and computer science is now greatly helped by web tools like the community driven encyclopedia \href{https://www.wikipedia.org/}{Wikipedia} or the community -driven site \href{http://stackoverflow.com/}{stackoverflow}, where people write +driven site \href{http://stackoverflow.com/}{Stackoverflow}, where people write thousands of questions and answers on programming each week. +%todo For Ashwin! :-) +IRC, Slack, Riot, + +%todo +\subparagraph{Automatic web documentations built on servers.} + +Doxygen, Sphinx, Readthedocs... \subsection*{Python, a programming language adapted for open-science} @@ -263,7 +290,7 @@ in many situations, one does not need to write them explicitly. \item Automatic memory management. The user do not have to handle by hand the memory allocation and deallocation. There are of course ways to control the -memory usage of a program, though not as precise as for low-level languages. +memory usage of a program, though not as precisely as for low-level languages. \item The standard way\footnote{There are also tools to compile Python code (usually via C or C++ code, for example Cython and Pythran).} to run a Python @@ -276,7 +303,7 @@ % We will discuss in the following the disadvantages of this characteristic. Two evident advantages are the shorter development cycle and the possibility of -interactive workflow (usually with Ipython or Jupyter notebooks). +interactive workflow (usually with IPython or Jupyter notebooks). Note that there are many implementations of Python interpreters\footnote{We can site CPython (written in C), Jython (Java), IronPython (C\#), PyPy (Rpython, a @@ -308,13 +335,13 @@ languages. Less bugs are introduced by the developers only because there are much less lines of code. % -Then, Python is one of the best "glue" languages for fast prototyping. +Thus, Python is one of the best "glue" languages for fast prototyping. % Maintaining and modifying existing code is also eased by the very good code readability associated with the language. Another consequence of the apparent simplicity of Python is that the learning -curve has a good shape. Learning Python is very easy at the beginning and the +curve has a nice shape. Learning Python is very easy at the beginning and the complexity gradually increases while the level of the developer increases. % Therefore, Python is good for developers of all levels: very gentle for @@ -324,13 +351,13 @@ evolve. \begin{itemize} \item A hard and controversial decision was taken to clean up Python from its -historical incoherence by having a new Python 3 backwards-incompatible versions +historical incoherences by having a new Python 3 backwards-incompatible versions of the language. Python 3.0 was released in December 2008. After a long and difficult process, the transition Python 2 to Python 3 is now (2018) behind us for nearly all important packages. Especially for scientific applications, we can now work only in Python 3. As a result, \href{https://github.com/% numpy/numpy/blob/master/doc/neps/dropping-python2.7-proposal.rst}{next releases -of numpy will drop compatibility with Python 2}. Finally, the difficult +of \numpy will drop compatibility with Python 2}. Finally, the difficult transition leads to very clean and coherent recent versions of Python. Therefore, one should not used Python 2 for science anymore and instead prefer the newer versions (3.5 or 3.6 in 2018). @@ -372,17 +399,18 @@ \item Science! Python is now widely used in scientific applications. It has now a mature and -powerful scientific ecosystem with well-established based packages (numpy for -N-dimensional homogeneous arrays, scipy for fundamental feature of scientific -computing, matplotlib for plotting and pandas for data structures) and several -more specialized packages (to name a few, h5py, mpi4py, scikit-image, -scikit-learn, ...). There are now great (and often shiny) tools for most of the -applications. For example, the +powerful scientific ecosystem with well-established based packages (\numpy for +N-dimensional homogeneous arrays, \pack{scipy} for fundamental feature of +scientific computing, \pack{matplotlib} for plotting and \pack{pandas} for data +structures) and several more specialized packages (to name a few, \pack{h5py}, +\pack{mpi4py}, \pack{skimage}, \pack{sklearn}, ...). There are now great (and +often shiny) tools for most of the applications. For example, the \href{https://github.com/rougier/python-visualization-landscape}{% landscape of visualization tools usable through Python} is impressive. % Python is one of the main languages for data science with for example the -packages pandas, statmodels, scikit-learn, Keras and TensorFlow. +packages \pack{pandas}, \pack{statmodels}, \pack{sklearn}, \pack{keras} and +\pack{tensorflow}. There are now ready-to-use integrated environments (similar to Matlab) like in particular the @@ -397,36 +425,36 @@ \end{itemize} Python has a very large and supportive community (see for example -the \href{http://stackoverflow.com/tags}{stackoverflow tags}). +the \href{http://stackoverflow.com/tags}{Stackoverflow tags}). % But Python is also supported by companies using Python and the open-source dynamics. Most of these companies that support Python are specialized in -computer technologies. But, recently, companies basing their business on -Python for science have started to become quite powerful in terms of +computer and web technologies. But, recently, companies basing their business +on Python for science have started to become quite powerful in terms of development. For example Anaconda Inc. is strongly implied in the development -of tools like numba or dask. Intel and Nvidia also contribute to the Python -scientific ecosystem. +of tools like \pack{numba} or \pack{dask}. Intel and Nvidia also contribute to +the Python scientific ecosystem. \subsubsection*{Some Python issues} \subparagraph{Since the standard Python interpreter does not do any proper compilation,} Python code can be in some CPU bounded cases too slow. One have -then to use other tools to get optimized machine instructions. +thus to use tools to get optimized machine instructions. \textbf{Ahead Of Time compilation.} Since Python is first a "glue" language, a common solution is to use code written in C, C++ or Fortran to speedup the performance critical code. Libraries can be used directly or ones has to used -extensions, i.e. Python modules written in C or C++ using the CPython API that +extensions, i.e.\ Python modules written in C or C++ using the CPython API that can be imported directly from Python like any Python modules. % This strategy has given the base modules of the scientific Python ecosystem, -namely numpy, scipy and matplotlib. +namely \numpy, \pack{scipy} and \pack{matplotlib}. The Python developers do not actually have to write the compiled extensions in C, C++ or Fortran, since there are tools to write them automatically from Python code (Pythran) or from Cython code, which is basically very similar to -C/C++ but with a syntax similar to Python with the possibility to add type +C/C++ but with a syntax similar to Python but with the possibility to add type declarations. Pythran is a quite recent tool which gives impressive results. It creates @@ -450,7 +478,7 @@ explicit loops. \textbf{Just In Time (JIT) compilation.} Another way to obtain optimized -machine instructions is by using JIT compilation, i.e. by compiling only the +machine instructions is by using JIT compilation, i.e.\ by compiling only the critical code at run time. % This strategy can gives good results with other languages as for example Matlab @@ -468,18 +496,20 @@ so necessary for many purposes. Therefore, not as much money and work has been put in accelerating Python than for example for Java and JavaScript. -Another strategy is to add a JIT to CPython through an external module (numba) -and to only compile the critical functions. Numba is particularly interesting -because it can take advantage of the GPU\footnote{See for example +Another strategy is to add a JIT to CPython through an external package +(\pack{numba}) and to only compile the critical functions. Moreover, +\pack{numba} is particularly interesting because it can take advantage of the +GPU\footnote{See for example \url{https://devblogs.nvidia.com/parallelforall/seven-things-numba/}}. -We also discuss efficiency in Python in the two companion papers -\citep{fluidfft, fluidsim}. +We present in the two companion papers \citep{fluidfft, fluidsim} examples of +highly efficient Python codes. \subparagraph{No type-checking.} In contrast to languages using Ahead Of Time compilation, there is no type-checking in Python (and more generally in many -dynamic languages). Type-checking can avoid simple bugs. However, it is not a -good practice to rely only on type-checking to look for bugs in a code. +dynamical languages). Type-checking is useful since it can avoid simple bugs. +However, it is not a good practice to rely only on type-checking to look for +bugs in a code. % It is known that a good set of unittests is also needed to obtain more reliable codes. @@ -496,11 +526,11 @@ mentioned, Python is a versatile language. For many applications, the standard library is not sufficient and one needs external packages. % -For most of the applications there are many projects usable through Python. A -user/developer has to make choices regarding the tools to use. Especially for -the beginners who are not used to the open-source realm, it can be difficult to -understand a ``landscape of projects'' for a particular application and to make -good technological choices. +For most of the applications there are several projects usable through Python. +A user/developer has to make choices regarding the tools to use. Especially +for the beginners who are not used to the open-source realm, it can be +difficult to understand a ``landscape of projects'' for a particular +application and to make good technological choices. It is important to get a good introduction on how work the open-source dynamics and the scientific Python ecosystem. Is can also be very useful to ask to @@ -514,7 +544,7 @@ time\footnote{See \href{https://opensource.com/article/17/4/grok-gil}{this blog article on ``How to grok the GIL''} and \href{https://faster-cpython.readthedocs.io/cpython37.html}{this technical -presentation by a core developer of CPython}.}. +presentation by a CPython core developer}.}. % This limitation comes from an important detail of the implementation of the CPython interpreter: a Global Interpreter Lock (GIL) is used to prevent race @@ -524,16 +554,20 @@ it is very difficult to remove it while keeping other nice technical properties of CPython\footnote{See \url{https://wiki.python.org/moin/GlobalInterpreterLock}.}. Note that Pypy -also uses a GIL, while two other implementation of the Python language, Jython +also uses a GIL, while two other implementations of the Python language, Jython and IronPython, do not have this limitation. Anyway, threads using the interpreter do not use the CPU at the same time so it is not possible to use threads to do multicore computational parallelism with -pure python code. +pure python code to accelerate CPU bounded tasks. +% +Note however, that threads can be (and are widely) used in Python for +concurrency, i.e.\ to perform io tasks (which does not need CPU) concurrently. % -To do computational parallelism with the GIL, one has to use other strategies: -extensions (fine grain), multiprocessing (coarse grain) and processes -communicated with MPI (with mpi4py). +To do proper computational parallelism for CPU bounded tasks with the GIL, one +has to use other strategies: extensions (fine grain parallelism), +multiprocessing (coarse grain parallelism) and processes communicating for +example with MPI (with mpi4py) or zeromq. \subsubsection*{Rich landscape of open-source languages useful for science} @@ -563,20 +597,19 @@ between coexisting tools. So what about Python? As already mentioned, it is today the language with the -strongest dynamics for science and data analysis. Several students learn and +strongest dynamics in science and data analysis. Several students learn and will learn it. Many scientists in the future will therefore be able to appreciate his strengths and will like the pythonic approaches. With such dynamics today, we can guess that the Python world will be able to embrace the -new technological trends and to overcome its actual limitations. +new technological trends and to overcome some of its actual limitations. % We think that it is reasonable to predict that Python will continues to shine (i) as a language particularly good for human communication and to write ideas, -(ii) a language for fast prototyping and (iii) as a glue language able to +(ii) as a language for fast prototyping and (iii) as a glue language able to interact nicely with code written in other languages (see for example the recent projects \href{https://cppyy.readthedocs.io}{cppyy} and \href{https://github.com/pybind/pybind11/}{pybind11}, or -\href{https://github.com/PyO3}{the possibility to interact with Rust -code}). +\href{https://github.com/PyO3}{the possibility to interact with Rust code}). \subsubsection*{Conclusion: Python, a versatile Swiss Army Knife for the scientists} @@ -588,10 +621,6 @@ very strong and quickly growing. Therefore, in most fields, today, if a scientists has to acquire good skills in one language, it is in Python. -We can guess that the Python world (the interpreters and the tools) will -continue to evolve, gaining interesting new features and possibilities (for -example GPU acceleration or type checking). - \subsection*{Productivity at individual, group and community levels} @@ -610,11 +639,11 @@ the same features. Good ideas will be spread in different codes and since the number of users and developers are smaller for each codes, their quality will not improve as fast as it could with an open-source strategy. -% + Finally, a community using open-source methods can be efficient to co-develop its tools. However, some of its members have to spend more energy to bear this -dynamics. -% +dynamics so that they somehow sacrifice their individual efficiency. + We see that short-term efficiency, long-term efficiency and productivity at different levels can often be incompatible and that we have to consider these conflicts when choosing between different technologies. @@ -651,7 +680,7 @@ ready to pay more for it. % Such vicious circle are difficult to break but this can be done with -open-source solution for which the similar circle is instead positive. +open-source solutions for which the similar circle is instead positive. We have seen that it was very common to rely on commercial companies and close-source software for most of the new developments in fluid mechanics. @@ -701,16 +730,16 @@ \begin{itemize} \item One file for each function available outside the file where it is implemented. -\item No real organization of the standard library. All function available in a -huge flat namespace. No import mechanism so that we do not know looking at the -code from where a function comes. +\item No real organization of the standard library. All functions are available +in a huge flat namespace. No import mechanism so that we do not know looking at +the code from where a function comes. \item A standard way to organize multi-file code is to write scripts that modify and define global variables (see for example \href{www.damtp.cam.ac.uk/user/jrt51/files/diablo_mat.tar.gz}{% this Matlab version of the code Diablo}). It is so simple to do this that we can say that the language strongly encourages this practice. Matlab files are -not self consistent, i.e. it is normal to use in a file a global variable +not self consistent, i.e.\ it is normal to use in a file a global variable defined outside of the file. \item Very bad default argument mechanism. @@ -766,25 +795,26 @@ \item The syntax \codeinline{a(100, 100) = 1;} to create and extend matrix is a very good way to hide bugs. The code \codeinline{a = eye(2); a(i0, i1) = 1;} will never raise any error regardless the values of \codeinline{i0} and -\codeinline{i1}! No error is raised is a user misspell the variable -\codeinline{a} and write \codeinline{aa(100, 100) = 1;}. +\codeinline{i1}! No error is raised if a user misspell the variable +\codeinline{a} and write something like \codeinline{aa(100, 100) = 1;}. \end{itemize} By definition of a close-source software, Matlab is a big black box: it is impossible to study the implementation of its functions. -Matlab is not free. The price for one license for non-commercial use is not -very expensive for most lab. However, when we start to add the price of the -toolboxes necessary to run most codes, it starts to be non negligible for some -institutions. Working with Matlab on a cluster is soon very expensive since -one needs on licence per processes! +Using Matlab is not free. The price for one license for non-commercial use is +not very expensive for most research and teaching institutes. However, when we +start to add the price of the toolboxes necessary to run most codes, it starts +to be non negligible for some institutions. Working with Matlab on a cluster +with MPI is soon very expensive since one needs on licence per processes! It would be interesting to estimate what could be done with open-source solutions with a part of the amount paid by public research institutes. How many developers could be paid to improve the alternative open-source solutions? % -This raises the issue of the funding of the scientific open-source software. +This question raises the issue of the funding of the scientific open-source +software. Moreover, it is well known that the lack of license for the personal computers of staff is sometimes solved with cracked versions of Matlab. How can serious @@ -796,11 +826,10 @@ Matlab uses a JIT compiler and that optimized Matlab codes rely on it. The recent open-source and Python dynamics represent a hard punch for Matlab, -which is loosing its dominant position it has in some domains. +which is loosing the dominant position it has in some domains. % -Note that this fact is very recent. The Python scientific ecosystem can -now compete for nearly all domains but this was not the case typically 5 years -ago. +Note that this fact is very recent. The Python scientific ecosystem can now +compete in nearly all domains but this was not the case typically 5 years ago. % There are still a large number of researchers and teachers in sciences who are strongly attached to Matlab mainly because they have used it a lot. @@ -821,16 +850,16 @@ applications because we need ``add-on libraries for performing even basic mathematics'' and that the ``quality, comprehensiveness, and maintenance [of these libraries] varies widely''. These add-on libraries are in particular -Numpy, Scipy and Ipython which are very mature, robust, well documented and -which fully integrate into the scientific Python ecosystem. +\numpy, \pack{scipy} and IPython which are very mature, robust, well documented +and which fully integrated into the scientific Python ecosystem. -\item Matlab teams present the scientific ecosystem as being developed by +\item Matlab team presents the scientific ecosystem as being developed by amateurs and therefore being basically badly documented and buggy. Python for science today is not old-school open-source. First, new open-source methods give impressive results in term of code quality and second, most of the big Python packages are not mainly written by amateurs and are supported by -companies or public research institutes: Numba (Anaconda, Nvidia), Tensorflow -(Google), scikit-learn (INRIA), Mercurial (Facebook). +companies or research institutes: Numba (Anaconda, Nvidia), TensorFlow +(Google), Scikit-learn (INRIA), Mercurial (Facebook). \item Matlab team claims that Matlab ``is the more natural way to express computational mathematics''. We are skeptical. Let's compare code. First the @@ -847,7 +876,7 @@ row2 = row .* (row + 1); result = row2 ./ (row2 + 1); \end{minted} -and then the equivalent code in Python 3.6 (using numpy): +and then the equivalent code in Python 3.6 (using \numpy): \begin{minted}[fontsize=\footnotesize]{python} import numpy as np # creation of a 1D array with shape [1, 3] @@ -861,25 +890,25 @@ row2 = row * (row + 1) result = row2 / (row2 + 1) \end{minted} -We see that Matlab and Numpy syntax are actually very similar. Most Python -codes start with import statements and here with import the package -\codeinline{numpy}. It is of course simple to start an Ipython session with -numpy already imported. +We see that Matlab and \numpy syntax are actually very similar. Most Python +codes start with import statements and here we have to import the package +\numpy. It is of course simple to start an IPython session with +\numpy already imported. % -Numpy is more explicit and longer to create the object -\codeinline{row}. Instead, in Matlab, the user writes a 1d object (here a list -of 3 numbers) and obtain a 2d object. Actually, even a number is a 2d object -in Matlab (\codeinline{size(1)} returns \codeinline{[1, 1]}). +The \numpy syntax for creating the object \codeinline{row} is longer and more +explicit. Instead, in Matlab, the user writes a 1d object (here a list of 3 +numbers) and obtain a 2d object. Actually, even a number is a 2d object in +Matlab (\codeinline{size(1)} returns \codeinline{[1, 1]}). % -Except from these differences, we see that numpy and Matlab codes are actually +Except from these differences, we see that \numpy and Matlab codes are actually as clear and ``natural''. \item Matlab team claims that teaching is mostly done in Matlab and that scientific books mostly use Matlab. Both statements are nowadays wrong. Most programming teaching is now based on Python and there are now several books on scientific applications based on Python. There are also much more videos on -Python than on Matlab: researches on Youtube give in early 2018 3800000 videos -on Python and 900000 on Matlab. +Python than on Matlab: researches on Youtube give in early 2018 3,800,000 +videos on Python and 900,000 on Matlab. \item Matlab team praises their support and claims that no support exists for Python. This is now completely wrong. First the support by the community is @@ -894,12 +923,12 @@ speed-up computations. % To our experience, the translation in Python of a real Matlab code taken in a -laboratory gives faster computations even with only the basic scientific +laboratory gives faster computations even using only the basic scientific packages and without any fancy tools. It is actually mainly because the principle of Matlab is to keep the users with bad habits and a bad level. % -With the tools that we have today, it is very simple to get very efficient code -using Python. +With the tools available today, we can get with little efforts very efficient +code using Python. \end{itemize} @@ -910,7 +939,7 @@ find a good technical reason to continue to use Matlab. The same remarks can be done for Labview. It is a graphical programming -language and the programs are saved as binary files. It is a problem because +language and the programs are saved as binary files. This is a problem because first it is impossible to read a Labview program without Labview and, more importantly, it forbids the use of source management tools which are so important for collective work and open-source. @@ -923,8 +952,8 @@ very expensive proprietary close-source mathematical symbolic computation program. % -There are two complementary open-source alternative, both based on Python, Sage -and Sympy. +Sage and Sympy are two complementary open-source alternative both based on +Python. \subsubsection*{Different models for software development in fluid mechanics} @@ -941,16 +970,16 @@ Close-source or even undistributed software is a very common model in the field. % -The arguments against sharing are quite diverse: +The arguments against sharing are diverse: \begin{itemize} -\item ``We do not share to keep comparative advantage.'' +\item ``We do not share to keep a comparative advantage.'' \item ``We do not share because we do not provide support.'' -\item ``We do not share because people won't be able to correctly use the code -or interpret the results.'' +\item ``We do not share because people would not be able to correctly use the +code or interpret the results.'' -\item ``We do not share we want to control industrial usage.'' +\item ``We do not share because we want to control industrial usage.'' \item ``We do not share because we do not want people to review and criticize our code.'' @@ -960,7 +989,7 @@ \end{itemize} Between close-source and open-source, there is a grey zone gathering different -behaviors. +practices. % It is quite common to share only for ``friends'' to control the dissemination of the software. @@ -968,26 +997,29 @@ This model can be institutionalized with an official group as for example for the code \href{https://www.coria-cfd.fr/index.php/YALES2}{Yales2}. % -The lack of a real open-source license can of course lead to problems and is a -limitation for users/developers with such model. +With such model, the lack of a proper open-source license can of course lead to +problems and is a limitation for users/developers. Another common practice is to share without repository (for example \href{http://www.damtp.cam.ac.uk/user/jrt51/files.html}{Diablo} or \href{http://yakari.polytechnique.fr/people/deloncle/ns3d.html}{ns3d}) and/or -without license... +without license. Finally, we can also mention the open-source code written with proprietary -tools (typically written in Matlab, see for example PIVlab, UVmat, PIVmat). +tools (typically written in Matlab, see for example +\href{http://pivlab.blogspot.de/}{PIVlab}, +\href{http://servforge.legi.grenoble-inp.fr/projects/soft-uvmat/}{UVmat} and +\href{https://de.mathworks.com/matlabcentral/fileexchange/10902-pivmat-4-00}{PIVmat}). % -To use them, you need to pay but you do not pay the authors of the +To use them, we need to pay but we do not pay the authors of the software. Actually the authors of the software also give money to the same company than us. Do they realize that when they work on these programs, they work (often for free) for the company that sells the proprietary tool? Some real open-source codes have also emerged, for example -\href{https://nek5000.mcs.anl.gov/}{NEK5000} (Fortran), OpenFOAM (C++), +\href{https://nek5000.mcs.anl.gov/}{NEK5000} (Fortran), \href{https://www.openfoam.com/}{OpenFOAM} (C++), \href{http://basilisk.fr}{Basilisk} (C) and -\href{http://channelflow.org/}{Channelflow} (C++), +\href{http://channelflow.org/}{Channelflow} (C++) and \href{https://www.code-saturne.org}{Code\_Saturn} (C/Fortran). Big companies have started to use open-source development for fluid mechanics @@ -998,11 +1030,11 @@ % EDF, the main French electric utility company has also a very interesting \href{http://linuxfr.org/news/strat\%C3\%A9gie-open-source-\%C3\%A0-edf-rd}{% -open-source strategy}. It leads to many codes and in particular -\href{https://www.code-saturne.org}{Code\_Saturn} for the fluid simulations. +open-source strategy}. It has produced many programs and in particular the CFD +solver \href{https://www.code-saturne.org}{Code\_Saturn}. % EDF is also involved in the development of -\href{https://www.paraview.org/}{Paraview}. +\href{https://www.paraview.org/}{Paraview}\footnote{\url{https://blog.kitware.com/introducing-some-new-paraview-5-0-features/}}. Finally, scientists start to use Python to study fluid mechanics, in particular with CFD codes (\href{http://dedalus-project.org/}{Dedalus}, @@ -1010,6 +1042,10 @@ \href{https://github.com/mikaem/Oasis}{Oasis}, \href{http://pyfr.org/}{PyFR} and \href{https://fenicsproject.org/}{FEniCS}) or for data analysis (\href{http://www.openptv.net/}{OpenPTV}). +% +The packages of the FluidDyn project are also part of this trend. + + \subsection*{FluidDyn project} @@ -1021,7 +1057,7 @@ The potential of these tools is not yet fully exploited in the field of fluid mechanics. -Fluiddyn is a project to foster open-science and open-source coding in Python +FluidDyn is a project to foster open-science and open-source coding in Python in the field of fluid mechanics. % It is a tentative to provide the technical framework to allows a collaborative @@ -1037,7 +1073,7 @@ (PEP~8). \item Source control management (Mercurial) and forge (Bitbucket) simple for the new comers. -\item Packaging and install procedure. All packages are available from +\item Packaging and installation procedure. All packages are available from \href{https://pypi.org/}{Pypi} and can be install simply with \href{https://pypi.org/project/pip/}{pip}, the standard Python installation tool. @@ -1048,7 +1084,7 @@ \item Documentation produced with the standard and up-to-date tools: Sphinx, Anaconda and Jupyter. Built and hosted online at \href{https://readthedocs.org/}{Readthedocs}. Can also be generated offline. -\item Unittest and continuous integration. +\item Unittest and continuous integration with Bitbucket Pipeline and Travis. \end{itemize} We hope that such clean framework will allow contributions from scientists in the field and that we can build together a nice, user-friendly and efficient @@ -1114,10 +1150,10 @@ A detailed presentation on the above packages can be found in their respective documentations on the web and for fluidfft and fluidsim in the two companion papers \cite[]{fluidfft, fluidsim}. This article will now focus on the base -package fluiddyn. +package \fluiddyn. -All functions and classes defined in fluiddyn are pure Python elements, meaning -that no extensions are implemented in fluiddyn. Thus the package fluiddyn is +All functions and classes defined in \fluiddyn are pure Python elements, meaning +that no extensions are implemented in \fluiddyn. Thus the package \fluiddyn is extremely easy to install with just a \codeinline{pip install} command and no compilation. @@ -1125,9 +1161,9 @@ it has been made forward compatible with Python 3 through the use of external package future. -\subsection*{API of the Python library fluiddyn} +\subsection*{API of the Python library \fluiddyn} -The package fluiddyn is organized into five sub-packages: +The package \fluiddyn is organized into five sub-packages: \begin{itemize} \item \codeinline{fluiddyn.io}: This subpackage provides utilities for input/output to different file formats. @@ -1159,7 +1195,7 @@ \item Evaluate data types automatically while loading from saved files. \item Easy exploration in an interactive console. \item Allows modification of default parameters through simple Python script -files. Errors are raised if the user try to used an invalid parameter. +files. An error is raised if the user try to used an invalid parameter. \item Graphical User Interface (GUI) frontend with PyQt. \end{itemize} @@ -1196,14 +1232,15 @@ \subsubsection*{Module \codeinline{fluiddyn.calcul.easyfft}} -Thin wrapper for an unified API using classes around the packages pyfftw and -scipy.fftpack. It is very easy to perform forward and backward Fast Fourier -Transforms (FFT) in 1d, 2d and 3d. The FFT can be multithreaded if the -environment variable \codeinline{OMP\_NUM\_THREADS} is defined. +Thin wrapper for an unified API using classes around the packages \pack{pyfftw} +and \pack{scipy.fftpack}. It is very easy to perform forward and backward Fast +Fourier Transforms (FFT) in one dimension, two dimensions and three dimensions. +The FFT can be multithreaded if the environment variable +\codeinline{OMP\_NUM\_THREADS} is defined. -\subsection*{The fluiddyn command-line utilities} +\subsection*{The \fluiddyn command-line utilities} -The package fluiddyn also provides few simple command-line utilities to perform +The package \fluiddyn also provides few simple command-line utilities to perform simple tasks useful for a scientist developing with Python and using the FluidDyn packages. \begin{itemize} @@ -1212,17 +1249,23 @@ Displays all important information related to software and hardware. It also includes detailed information such as currently installed FluidDyn packages, -other third-party packages, C compiler, MPI and NumPy configuration. +other third-party packages, C compiler, MPI and \numpy configuration. \item fluiddump Utility to print the hierarchy of hdf5 and netcdf files. Do not depend on the netcdf4 library. -\item fluidnbstriout +\item fluidnbstripout -Very simple layer to stripout notebooks with the tool nbstripout. By default, -fluidnbstripout excludes the notebooks whose name ends with '.nbconvert.ipynb'. +Very simple layer to stripout notebooks of output, which is useful to not +include in a repository the output of notebooks. +% +This tool is based on \href{https://github.com/kynan/nbstripout}{nbstripout} +but, in contrast to nbstripout, +% +by default fluidnbstripout excludes the notebooks whose name ends with +'.nbconvert.ipynb'. \item fluidmat2py @@ -1242,6 +1285,8 @@ % could quickly understand if the software is working (e.g. providing examples % of running the software with sample input and output data). } +%todo For Ashwin! :-) + Describe coverage tests and tox tests Continuous integration with Bitbucket pipelines and Travis @@ -1272,11 +1317,11 @@ % compatibility.} \begin{itemize} -\item {\bf Minimum:} numpy, matplotlib, psutil, future, subprocess32 (for -Python 2.7 only). +\item {\bf Minimum:} \numpy, \pack{matplotlib}, \pack{psutil}, \pack{future}, +\pack{subprocess32} (for Python 2.7 only). -\item {\bf Full functionality:} h5py, h5netcdf, mpi4py, scipy, pyfftw (requires -FFTW library), pillow. +\item {\bf Full functionality:} \pack{h5py}, \pack{h5netcdf}, \pack{mpi4py}, +\pack{scipy}, \pack{pyfftw} (requires FFTW library), \pack{pillow}. \item {\bf Optional:} OpenCV with Python bindings, scikit-image. \end{itemize} @@ -1290,12 +1335,13 @@ majority of the FluidDyn packages, future-proofing with Python 3 compatibility and documentation. -\item Ashwin Vishnu Mohanan (KTH): developer of the packages fluidsim, -fluidfft, fluidimage and fluiddyn; documentation, code coverage and continuous -integration (Docker, Bitbucket pipeline and Travis). +\item Ashwin Vishnu Mohanan (KTH): developer of the packages +\fluidpack{sim}, \fluidpack{fft}, \fluidpack{image} and +\fluiddyn; documentation, code coverage and continuous integration (Docker, +Bitbucket pipeline and Travis). -\item Cyrille Bonamy (LEGI): developer of fluidfft, fluidsim and -fluidfoam. Main maintainer of fluidfoam. +\item Cyrille Bonamy (LEGI): developer of \fluidpack{fft}, \fluidpack{sim} and +\fluidpack{foam}. Main maintainer of \fluidpack{foam}. \end{itemize} \section*{Software location:} @@ -1309,6 +1355,7 @@ \item[Persistent identifier:] https://pypi.org/project/fluiddyn \item[Licence:] CeCILL-B, a BSD compatible French licence. \item[Publisher:] Pierre Augier +%todo \item[Version published:] ? \item[Date published:] ? \end{description} @@ -1320,6 +1367,7 @@ \item[Name:] Bitbucket \item[Persistent identifier:] https://bitbucket.org/fluiddyn/fluiddyn \item[Licence:] CeCILL-B +%todo \item[Date published:] ? \end{description} @@ -1348,17 +1396,19 @@ of what support mechanisms are in place for this software (even if there is no support).} +%todo For Ashwin! :-) + As a library, used in FluidDyn specialized packages. Can be used in other non ``fluid'' packages. The command-lines tools can be useful to all scientists working with Python. -There is no real support mechanisms. Issues on bitbucket. Discussions and -questions on riot. +There is no real support mechanisms. Issues on Bitbucket. Discussions and +questions on \href{https://riot.im/app/#/room/#fluiddyn:matrix.org}{Riot}. \subsection*{Conclusions} -Fluiddyn is an attempt to set off a collaborative dynamics based on open-source +FluidDyn is an attempt to set off a collaborative dynamics based on open-source development in the research on fluid dynamics. % We try with this project to open the possibility of another usage of @@ -1368,11 +1418,12 @@ interesting features are implemented and a framework for packaging, documenting and testing is now well set. % -However, there is not yet any community around the project and we have know to -work on this task since an active community of users and of developers is a -condition for success. +However, the community around the project is currently tiny and we have now to +work on attracting users and developers since an active community is a +condition for success. -Will people use these tools and collaborate with these tools? +Will people use the FluidDyn tools and collaborate through the project +FluidDyn? % There are clearly many challenges and potential barriers: \begin{itemize} @@ -1401,28 +1452,31 @@ collective efficiency related to open-source methods and tools can be a strong booster. - \section*{Acknowledgements} \textcolor{blue}{Please add any relevant acknowledgements to anyone else who supported the project in which the software was created, but did not work directly on the software itself.} +%todo + CNRS CINES/GENCI CIMENT/GRICAD -Fondation Simone et Cino Del Duca - -ERC WATU - \section*{Funding statement} \textcolor{blue}{If the software resulted from funded research please give the funder and grant number.} +%todo + +Fondation Simone et Cino Del Duca + +ERC WATU + \section*{Competing interests} % \textcolor{blue}{If any of the authors have any competing interests then these diff --git a/jors.cls b/jors.cls --- a/jors.cls +++ b/jors.cls @@ -83,6 +83,8 @@ \usepackage{natbib} \usepackage{har2nat} +\usepackage{xspace} + %% Set source code listings style \lstset{basicstyle=\ttfamily, language=Python} @@ -94,10 +96,17 @@ \rhead{\textcolor{gray}{UP JORS software Latex paper template version 0.1}} -\newcommand{\fluidpack}[1]{\href{http://fluid#1.readthedocs.io}{fluid#1}} +\newcommand{\fluidpack}[1]{\href{http://fluid#1.readthedocs.io}{% +\codeinline{fluid#1}}} % \newcommand{\codeinline}[1]{\mintinline{python}{#1}} \newcommand{\codeinline}[1]{\texttt{#1}} +\newcommand{\fluiddyn}{\fluidpack{dyn}\xspace} + +\newcommand{\numpy}{\codeinline{numpy}\xspace} +\newcommand{\scipy}{\codeinline{scipy}\xspace} + +\newcommand{\pack}[1]{\codeinline{#1}} \endinput \ No newline at end of file