Skip to content
Snippets Groups Projects
Commit 93bedad6 authored by Pierre Augier's avatar Pierre Augier
Browse files

Finalize fluiddyn paper.

parent 4fb12db5
No related branches found
No related tags found
No related merge requests found
......@@ -44,8 +44,6 @@
FluidDyn: a framework to study fluid dynamics taking advantage of recent
advances in open-source software engineering
% pa: yes this one is really not great neither...
\section*{Paper Authors}
......@@ -54,7 +52,6 @@
1. AUGIER Pierre$^a$\\
2. MOHANAN Ashwin Vishnu$^b$\\
3. BONAMY Cyrille$^a$\\
\smallskip
......@@ -69,8 +66,6 @@
1. Researcher, LEGI, Universit\'e Grenoble Alpes, CNRS, France\\
2. Ph.D. student, Deparment of Mechanics, KTH Royal Institute of Technology,
Sweden; \\
3. Research Engineer, LEGI, Universit\'e Grenoble Alpes, CNRS, France;
\section*{Abstract}
......@@ -151,7 +146,7 @@
The information technology revolution open doors to fantastic opportunities for
human collaboration, and of course, for science.
Web related activities alone account for atleast 5\% of GDP in the USA and the
Web related activities alone account for at least 5\% of GDP in the USA and the
European Union\footnote{See the report by Internet Association titled
\href{https://internetassociation.org/reports/refreshing-understanding-internet-economy-ia-report/}
{``Refreshing Our Understanding of the Internet Economy''}}.
......@@ -245,12 +240,14 @@
FOSS is a misnomer, as it actually stands for \emph{freedom} (to use, modify
and distribute). The FOSS movement has dramatically decreased the cost of using
computers, and is evident from the widespread use of GNU/Linux systems in
desktops, computing clusters and web-servers in academia and beyond. The
beginnings of FOSS culture can be traced back to the release of free
type-setting standard \TeX\ (a predecessor of \LaTeX) in 1977, which was used in
the effort to replace a commercial word processing system \emph{Scribe} with
\codeinline{texinfo} in 1986. This was an initiative under the GNU project,
started in 1983 and continued under the umbrella of the Free Software Foundation
(FSF) from 1985.
desktops, computing clusters and web-servers in academia and beyond.
%
The beginnings of FOSS culture can be traced back to the success for scientific
publication of the free type-setting standards \TeX\ (1977) and \LaTeX (1885).
%
Another founding moment for the FOSS movement was the launching in 1983 by
\href{https://en.wikipedia.org/wiki/Richard_Stallman}{Richard Stallman} of the
GNU project to create a Unix-like computer operating system composed entirely
of free software.
%
GNU is today known for its compiler collection (GCC) and a multitude of tools
......@@ -256,5 +253,7 @@
GNU is today known for its compiler collection (GCC) and a multitude of tools
which when combined with the Linux kernel forms the GNU/Linux operating systems
that we are familiar today.
which when combined with the Linux kernel (created by
\href{https://en.wikipedia.org/wiki/Linus_Torvalds}{Linus Torvalds} in 1991)
forms the GNU/Linux operating systems that we are familiar today.
%
FOSS movement has had huge successes in many frontiers --- Apache could be
termed as the ``first killer-app of Linux'' and now with an ever-increasing
......@@ -259,9 +258,10 @@
FOSS movement has had huge successes in many frontiers --- Apache could be
termed as the ``first killer-app of Linux'' and now with an ever-increasing
suite of softwares including Firefox, LibreOffice, Gimp as solid alternatives to
proprietary offerings.
Linux has become the most widely used kernel, being deployed on
servers, personal computers, embedded devices, and also smart phones (with Android).
suite of softwares including Firefox, LibreOffice, Gimp as solid alternatives
to proprietary offerings.
%
Linux has become the most widely used kernel, being deployed on servers,
personal computers, embedded devices, and also smart phones (with Android).
Over the years, FOSS development has transitioned from an organic community of
volunteers, towards an organized system with participation from industries,
......@@ -337,7 +337,7 @@
%
Note that, it is now possible to use Mercurial to work with Git repositories,
for example hosted in Github and Gitlab\footnote{By using the Mercurial
extension hg-git.}.
extension \href{http://hg-git.github.io/}{hg-git}.}.
\subparagraph{Package management and third-party software repositories.} Python
as a medium for scientific computing owes its success to a rich ecosystem of
......@@ -353,7 +353,9 @@
delivering and downloading Python packages, assisted by package managers
\pack{pip} and \pack{conda} respectively.
% Almost all major
% languages\footnote{\href{https://stackoverflow.com/questions/1693529/list-of-top-repositories-by-programming-language}{https://stackoverflow.com/questions/1693529/list-of-top-repositories-by-programming-language}}
% languages\footnote{\href{https://stackoverflow.com/questions/1693529/%
% list-of-top-repositories-by-programming-language}{https://stackoverflow.com/%
% questions/1693529/list-of-top-repositories-by-programming-language}}
% have similar repositories, with exceptions such as Fortran and Matlab.
Specialized package managers have also emerged to install scientific
libraries and compilers such as \pack{easybuild} and \pack{spack}.
......@@ -408,13 +410,9 @@
Python community has thrived remarkably in the last few years, evident from the
fact that by the end of 2017, Python became the second most popular language in
terms of pull-requests in
%
Github\footnote{see Github's annual report titled
\href{https://octoverse.github.com/}{ ``The State of the Octoverse 2017''}}
and questions on
%
StackOverflow\footnote{see
terms of pull-requests in Github\footnote{see Github's annual report titled
\href{https://octoverse.github.com/}{ ``The State of the Octoverse 2017''}} and
questions on StackOverflow\footnote{see
\href{https://stackoverflow.blog/2017/09/06/incredible-growth-python/}{ ``The
Incredible Growth of Python''}, a blog post by Stack Overflow}.
%
......@@ -423,12 +421,15 @@
third-party vendors, ranks Python at number four. IEEE Spectrum\footnote{see
the analysis titled \href{https://spectrum.ieee.org/computing/software/%
the-2017-top-programming-languages}{``The 2017 Top Programming Languages''} on
IEEE spectrum.} ranks Python as the most popular language with a metric based
on trends, demand by employers and users on open-source hubs. Python
outclasses languages like Fortran and Matlab according to these statistics, and
bested only by mainstream languages such as C, Java, C++ and perhaps
Javascript. All the rankings do agree that, Python usage has been steadily
growing and is here to stay.
IEEE spectrum.} ranks Python as the most popular language in 2017 with a metric
based on trends, demand by employers and users on open-source hubs. All the
rankings do agree that, Python usage has been steadily growing and is here to
stay. Python largely outclasses languages like Fortran and Matlab according to
these statistics, and is now clearly one of the mainstream languages with C,
Java, C++ and Javascript.
%
Let us summarize the technical characteristics of the Python languages that
have lead to such incredible success.
% todo: DONE
% We need an introductory paragraph on how big is Python today
......@@ -444,8 +445,6 @@
\begin{itemize}
\item \textbf{Aesthetics.} The syntax is nice and elegant. Blocks of code are
defined with the indentation.
Maintaining and modifying existing code is also eased by the very good code
readability associated with the language.
\item \textbf{Code style.} There are explicit guidelines
(\href{https://www.python.org/dev/peps/pep-0008/}{PEP~8}) supporting the
regularity (and thereby readability) of the code and tools
......@@ -501,6 +500,9 @@
Thus, Python is one of the best ``glue'' languages for fast prototyping. Less
bugs are introduced by the developers simply because there are much less lines
of code \cite{nanz_comparative_2015}.
%
Maintaining and modifying existing code is also eased by the very good code
readability associated with the language.
Another consequence of the apparent simplicity of Python is that the learning
curve has a nice shape. Learning Python is very easy at the beginning and the
......@@ -547,7 +549,7 @@
\item System, database and network administration
\item Linux distribution software
\item GNU/Linux distribution software
\item Python is now the main scripting language to add programmability to
applications (for example Paraview, Visit, QGIS, Blender, ...).
......@@ -572,7 +574,7 @@
\href{https://github.com/rougier/python-visualization-landscape}{%
landscape of visualization tools usable through Python} is impressive.
%
Python is one of the main languages for data science, for example, with the
Python is one of the main languages for data science, with for example the
packages \pack{pandas}, \pack{statmodels}, \pack{sklearn}, \pack{keras} and
\pack{tensorflow}.
......@@ -606,7 +608,8 @@
\subsubsection*{Some Python issues}
\subparagraph{Since the standard Python interpreter does not do any proper
compilation,} Python code in some CPU bounded cases can be too slow.
Thus, one has to use special tools to get optimized machine instructions.
compilation,} Python code in some CPU bounded cases can be too slow. Thus, one
has to use special tools to get optimized machine instructions to speedup
performance-critical code.
\textbf{Ahead Of Time compilation.} Since Python is primarily a ``glue''
......@@ -611,7 +614,6 @@
\textbf{Ahead Of Time compilation.} Since Python is primarily a ``glue''
language, a common solution is to use code written in C, C++ or Fortran to
speedup performance-critical code.
language, a common solution is to use code written in C, C++ or Fortran.
%
% todo: DONE
% add a reference to ctypes and cffi to call C with pure Python code
......@@ -691,8 +693,8 @@
\subparagraph{No type-checking.} In contrast to languages using Ahead Of Time
compilation, there is no type-checking in Python (and more generally in many
dynamical languages). Type-checking is useful since it can avoid simple bugs.
However, it is not a good practice to rely only on type-checking to look for
bugs in a code.
However, it is not a good practice even with static languages to rely only on
type-checking to look for bugs in a code.
%
It is known that a rigorous set of unit tests is also needed to obtain more
reliable codes.
......@@ -700,6 +702,6 @@
Dynamical languages rely a lot on unit tests so it is very important to put at
least a little bit of time and energy to write a decent battery of tests.
%
We have to mention the very young and promising project
We have to mention the young and promising project
\href{http://mypy-lang.org/}{Mypy}\footnote{Guido van Rossum, the author and
``Benevolent Dictator For Life'' of Python, is highly involved in the project
......@@ -704,6 +706,7 @@
\href{http://mypy-lang.org/}{Mypy}\footnote{Guido van Rossum, the author and
``Benevolent Dictator For Life'' of Python, is highly involved in the project
Mypy.}, which allows for Optional Static Typing and thus type-checking for Python.
Mypy.}, which allows for Optional Static Typing and thus type-checking for
Python.
\subparagraph{A lively, huge and thus complicated ecosystem.} As already
mentioned, Python is a versatile language. For many applications, the standard
......@@ -716,13 +719,14 @@
difficult to understand a ``landscape of projects'' for a particular
application and to make good technological choices.
It is important to get a good introduction on how work with the open-source
dynamics and the scientific Python ecosystem. Is can also be very useful to ask
to experts and attend Python courses or workshops\footnote{This could be regular
academic courses, MOOCs or workshops like ones organized by
\href{https://software-carpentry.org/}{Software Carpentry}.}. One can also think
of this aspect as a non-issue, since this means that Python is much richer and
diverse than, for example Matlab. This is often an advantage.
It is important to get a good introduction on how to work with the open-source
dynamics and the scientific Python ecosystem. Is can also be very useful to
ask to experts and attend Python courses or workshops\footnote{This could be
regular academic courses, MOOCs or workshops like ones organized by
\href{https://software-carpentry.org/}{Software Carpentry}.}.
%
Of course, such abundance of projects can also be seen as an advantage, since
this means that Python is much richer and diverse than, for example Matlab.
\subparagraph{Absence of multicore computational parallelism using threads.} Threads
behave like light subprograms. In principle, they can use the different cores of
......@@ -854,7 +858,7 @@
community.
%
Even today, it is quite commonplace to start a PhD without any serious training
in Linux and modern programming tools.
in GNU/Linux and modern programming tools.
%
A large majority of scientists and technicians are also unaware of the
challenges and opportunities of open-source.
......@@ -903,9 +907,10 @@
complex programs. It is known that for scientific purposes, one should restrict
shell scripting to extremely simple tasks. It is also well known that Matlab is
not adapted for complex programs. These technical solutions also represent a
problem for code reuse, sharing and collaborative development. This is in
particular true for Matlab, whose massive usage decreases the collective
efficiency of the community.
problem for code reuse, sharing and collaborative development.
%
We are now going to present factual arguments explaining why we think that the
massive usage of Matlab decreases the collective efficiency of the community.
\subsubsection*{Discussion on Matlab, Labview and Mathematica}
......@@ -921,8 +926,10 @@
However, the language suffers from serious technical issues which make Matlab a
bad tool
\citationneeded[opinion]
% \citationneeded[opinion] % pa: I think what we say is fine for an article. We
% are not writing a wikipedia page which has the ambition to be neutral. Here,
% we gives an argued opinion, which is totally normal in a scientific article.
for doing more than simple processing and data plotting. The comparison with
Python often debated upon. We list a few striking Matlab weaknesses:
\begin{itemize}
......@@ -925,12 +932,13 @@
for doing more than simple processing and data plotting. The comparison with
Python often debated upon. We list a few striking Matlab weaknesses:
\begin{itemize}
\item One file for each function available outside the file where it is
implemented.
\item No real organization of the standard library. All functions are available
in a huge flat namespace. No import mechanism and as a result we cannot deduce, simply
by looking at the code, where a function comes from.
\item One file for each function (or class) available outside the file where it
is implemented. No notion of package or module.
\item No real organization of the standard library. All built-in and
user-defined functions are available in a huge flat namespace. No import
mechanism and as a result we cannot deduce, simply by looking at the code,
where a function comes from.
\item A standard way to organize multi-file code is to write scripts that
modify and define global variables (see for example
......@@ -980,11 +988,18 @@
code.
\item Less intuitive syntax for string operations in Matlab, compared to
Python. Take for example for string comparison, \codeinline{strcmp(s1, s2)} in Matlab
versus \codeinline{s1 == s2} in Python or for look up,
\codeinline{contains(s1, pattern)} in Matlab versus \codeinline{pattern in s1} in Python.
\item Weak error handling mechanisms \citationneeded[opinionated].
\item Bad object oriented programming model and syntax \citationneeded[opinionated].
Python. Take for example for string comparison, \codeinline{strcmp(s1, s2)} in
Matlab versus \codeinline{s1 == s2} in Python or for look up,
\codeinline{contains(s1, pattern)} in Matlab versus \codeinline{pattern in s1}
in Python.
% \item Less intuitive and clear syntax for exception
% handling. \citationneeded[opinionated].
% pa: Matlab has now a MException class. The syntax is not nice, but it is
% not so important.
\item Less clear and intuitive model and syntax for object oriented
programming, compared to Python. As for functions, one file per user-defined
class is necessary.
% \citationneeded[opinionated].
\item Parenthesis used for both function calls and indexing, causing ambiguity.
\item Matlab codes are usually full of \codeinline{;}, \codeinline{.*},
\codeinline{./}, \codeinline{\&\&} and \codeinline{||}, which make them quite
......@@ -1095,11 +1110,11 @@
\end{minted}
We see that Matlab and \numpy\ syntaxes are actually very similar. Most Python
codes start with import statements and here we have to import the package
\numpy. It is of course simpler to start an IPython session with
\numpy already imported.
\numpy. It is of course possible to start an IPython session with \numpy
already imported.
%
The \numpy syntax for creating the object \codeinline{row} is longer and more
explicit. Instead, in Matlab, the user writes a 1d object (here a list of 3
numbers) and obtain a 2d object. Actually, even a integer is a 2d object in
Matlab (\codeinline{size(1)} returns \codeinline{[1, 1]}).
%
......@@ -1100,11 +1115,11 @@
%
The \numpy syntax for creating the object \codeinline{row} is longer and more
explicit. Instead, in Matlab, the user writes a 1d object (here a list of 3
numbers) and obtain a 2d object. Actually, even a integer is a 2d object in
Matlab (\codeinline{size(1)} returns \codeinline{[1, 1]}).
%
Except from these differences, we see that \numpy and Matlab codes are actually
as clear and ``natural'' as possible.
Except from these differences, we see that the \numpy code is actually as clear
and ``natural'' as the Matlab code.
\item Matlab team claims that teaching is mostly done in Matlab and that
scientific books mostly use Matlab. Both statements are nowadays wrong. Most
......@@ -1228,8 +1243,8 @@
This can potentially add value for the community through added transparency in
publications, and in return, value for the authors of software in terms of
recognition and citations. We have only looked at the most ideal scenarios and
the question of soundness of the above arguments is left as an open question to the
reader.
the question of soundness of the above arguments is left as an open question to
the reader.
Between closed-source and open-source, is a grey area gathering different
practices.
......@@ -1247,7 +1262,8 @@
Another common practice is to share without repository (for example
\href{http://www.damtp.cam.ac.uk/user/jrt51/files.html}{Diablo} or
\href{http://yakari.polytechnique.fr/people/deloncle/ns3d.html}{ns3d}) and/or
without license.
without license (\href{https://choosealicense.com/no-permission/}{meaning that
the code is actually under exclusive copyright}).
Finally, we can also mention the open-source code written with proprietary tools
(typically written in Matlab, see for example
......@@ -1291,9 +1307,9 @@
\subsection*{FluidDyn project: Breaking the status quo}
We have seen that there is a strong dynamics in play around the use of computers (in
particular with the web) and that this creates very efficient tools and methods
for collective work and software development.
We have seen that there is a strong dynamics in play around the use of
computers (in particular with the web) and that this creates very efficient
tools and methods for collective work and software development.
%
The potential of these tools is not fully exploited yet, in the field of fluid
mechanics.
......@@ -1383,9 +1399,9 @@
\item \fluidpack{foam}: Small package to load OpenFoam data and plot them.
\item \pack{fluidcoriolis}: Small package used to carry out experiments in the
\item \fluidpack{coriolis}: Small package used to carry out experiments in the
Coriolis platform and share the data obtained \cite{ISSF2016}.
\end{itemize}
A detailed presentation on the above packages can be found in their respective
......@@ -1387,11 +1403,11 @@
Coriolis platform and share the data obtained \cite{ISSF2016}.
\end{itemize}
A detailed presentation on the above packages can be found in their respective
documentations on the web and for fluidfft and fluidsim in the two companion
papers \cite[]{fluidfft, fluidsim}. This article will now focus on the base
package \fluiddyn.
documentations on the web and for \fluidpack{fft} and \fluidpack{sim} in the
two companion papers \cite[]{fluidfft, fluidsim}. This article will now focus
on the base package \fluiddyn.
All functions and classes defined in \fluiddyn are pure Python elements, meaning
that no extensions are implemented in \fluiddyn. Thus the package \fluiddyn is
......@@ -1446,15 +1462,16 @@
\subsubsection*{Module \codeinline{fluiddyn.util.serieofarrays}}
This modules provides classes to iterate over files. It is a common task in data
processing to understand how to organized a \textit{serie} of files its from
filenames and formats. Then one has to form smaller set of arrays contained in
the files. For example, we can have such \textit{serie}: \codeinline{im1\_1.png,
im1\_2.png, im1\_3.png, im2\_1.png, im2\_2.png, im2\_3.png} and we could
create subsets like \codeinline{((im1\_1.png, im1\_2.png, im1\_2.png),
(im2\_1.png, im2\_2.png, im2\_3.png))}. From the same \textit{serie} of images, we can
also create other subsets \codeinline{((im1\_1.png, im2\_1.png),
(im1\_2.png, im2\_2.png), (im1\_3.png, im2\_3.png))}.
This modules provides classes to iterate over files. It is a common task in
data processing to understand how to organized a \textit{serie} of files its
from filenames and formats. Then one has to form smaller set of arrays
contained in the files. For example, we can have such \textit{serie}:
\codeinline{im1\_1.png, im1\_2.png, im1\_3.png, im2\_1.png, im2\_2.png,
im2\_3.png} and we could create subsets like \codeinline{((im1\_1.png,
im1\_2.png, im1\_2.png), (im2\_1.png, im2\_2.png, im2\_3.png))}. From the same
\textit{serie} of images, we can also create other subsets
\codeinline{((im1\_1.png, im2\_1.png), (im1\_2.png, im2\_2.png), (im1\_3.png,
im2\_3.png))}.
The classes of this module allows one to do it with a quite simple and general
API as shown in
......@@ -1548,9 +1565,9 @@
% \textcolor{blue}{Please include minimum version compatibility.}
Windows and any POSIX based OS, such as GNU/Linux and Mac OS X
Windows and any POSIX based OS, such as GNU/Linux and Mac OS X.
\section*{Programming language}
% \textcolor{blue}{Please include minimum version compatibility.}
......@@ -1552,9 +1569,9 @@
\section*{Programming language}
% \textcolor{blue}{Please include minimum version compatibility.}
Python 2.7, 3.4 or above
Python 2.7, 3.4 or above.
% \section*{Additional system requirements}
......@@ -1591,6 +1608,16 @@
\item Cyrille Bonamy (LEGI): developer of \fluidpack{fft}, \fluidpack{sim} and
\fluidpack{foam}. Main maintainer of \fluidpack{foam}.
\item Antoine Campagne (LEGI): developer of \fluidpack{image} and
\fluidpack{lab}.
\item Miguel Calpe Linares (LEGI): developer of \fluidpack{sim} and
\fluidpack{lab}.
\item Julien Salort (Laboratoire de physique, ENS de Lyon): developer of
\fluidpack{lab}.
\end{itemize}
\section*{Software location:}
......@@ -1604,9 +1631,8 @@
\item[Persistent identifier:] https://pypi.org/project/fluiddyn
\item[Licence:] CeCILL-B, a BSD compatible French licence.
\item[Publisher:] Pierre Augier
%todo
\item[Version published:] ?
\item[Date published:] ?
\item[Version published:] 0.2.1
\item[Date published:] 14/02/2018
\end{description}
{\bf Code repository}
......@@ -1616,8 +1642,7 @@
\item[Name:] Bitbucket
\item[Persistent identifier:] https://bitbucket.org/fluiddyn/fluiddyn
\item[Licence:] CeCILL-B
%todo
\item[Date published:] ?
\item[Date published:] 2015
\end{description}
% {\bf Emulation environment} \textcolor{blue}{(if appropriate)}
......@@ -1657,7 +1682,7 @@
\item Modules within \codeinline{fluiddyn.util} for miniature tasks,
printing in terminal with colours, getting memory usage
information, detecting if the current Python session is inside
IPython, creating a string with time and date etc.
IPython, creating a string with time and date, etc.
\item Modules within \codeinline{fluiddyn.io} to read
and save images of various formats including TIFF files with
multiple images; easily ask user with yes/no queries; handle
......@@ -1700,7 +1725,7 @@
There are clearly many challenges and potential barriers:
\begin{itemize}
\item The habits in the community.
\item Knowledge and skills in the community
\item Knowledge and skills in the community.
\item Misunderstanding between colleagues and in particular colleagues with
responsibilities and power.
\item Lack of a business model for open-source software in science. A good
......@@ -1731,20 +1756,20 @@
supported the project in which the software was created, but did not work
directly on the software itself.}
%todo
CNRS
CINES/GENCI
CIMENT/GRICAD
Swedish National Infrastructure for Computing (SNIC)
Erik Lindborg
We thank the CNRS to finance the work of Pierre Augier while giving freedom in
terms of scientific project.
%
Similarly, Ashwin Vishnu could not have been as involved in this project
without the kindness of Erik Lindborg.
%
We thank Cyrille Bonamy, Antoine Campagne, Miguel Calpe Linares and Julien
Salort for their implications in the development of some of the FluidDyn
packages.
%
Finally, we thank Gabriel Moreau and Joël Sommeria for interesting discussions.
\section*{Funding statement}
\textcolor{blue}{If the software resulted from funded research please give the
funder and grant number.}
......@@ -1745,16 +1770,17 @@
\section*{Funding statement}
\textcolor{blue}{If the software resulted from funded research please give the
funder and grant number.}
%todo
Fondation Simone et Cino Del Duca
ERC WATU
Swedish Research Council (Vetenskapsr{\aa}det): 2013-5191
This project has indirectly benefit from funding from the foundation Simone et
Cino Del Duca de l'Institut de France, the European Research Council (ERC)
under the European Union's Horizon 2020 research and innovation program (grant
agreement No 647018-WATU and Euhit consortium) and the Swedish Research Council
(Vetenskapsr{\aa}det): 2013-5191.
%
We have also been able to use supercomputers of CIMENT/GRICAD, CINES/GENCI and
the Swedish National Infrastructure for Computing (SNIC).
\section*{Competing interests}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment