Skip to content
Snippets Groups Projects
fluiddyn_paper.tex 86.7 KiB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697
%% Journal of Open Research Software Latex template -- Created By Stephen
%% Bonner and John Brennan, Durham Universtiy, UK.
%% see http://openresearchsoftware.metajnl.com

\documentclass{../jors}

\begin{document}

{\bf Software paper for submission to the Journal of Open Research Software} \\

To complete this template, please replace the blue text with your own. The paper
has three main sections: (1) Overview; (2) Availability; (3) Reuse potential.

Please submit the completed paper to: editor.jors@ubiquitypress.com

\rule{\textwidth}{1pt}

\section*{(1) Overview}

\vspace{0.5cm}

\section*{Title}

% \textcolor{blue}{The title of the software paper should focus on the software,
% e.g. “Text mining software from the X project”. If the software is closely
% linked to a specific research paper, then “Software from Paper Title” is
% appropriate. The title should be factual, relating to the functionality of the
% software and the area it relates to rather than making claims about the
% software, e.g. “Easy-to-use”.}

% old title

% FluidDyn: a Python open-source framework for research and teaching in fluid
% dynamics

% pa: this title may not be really good for this paper...
% - "research and teaching" may not be very important in the title
% - Recent progress in open-source software engineering and consequences on
%   science

% todo: find a better title!

Taking advantage of recent advances in open-source software engineering to study
fluid dynamics: the example of the FluidDyn project


\section*{Paper Authors}

% \textcolor{blue}{1. Last name, first name; (Lead/corresponding author first) \\
% 2. Last name, first name; etc.}

1. AUGIER Pierre$^a$\\
2. MOHANAN Ashwin Vishnu$^b$\\

\smallskip

$^a$ Univ. Grenoble Alpes, CNRS, Grenoble INP\footnote{Institute of Engineering
Univ. Grenoble Alpes}, LEGI, 38000 Grenoble, France.\\
$^b$ Linn\'e Flow Centre, Department of Mechanics, KTH, 10044 Stockholm, Sweden.

\section*{Paper Author Roles and Affiliations}
% \textcolor{blue}{1. First author role and affiliation \\
% 2. Second author role and affiliation etc.}

1. Researcher, LEGI, Universit\'e Grenoble Alpes, CNRS, France\\
2. Ph.D. student, Deparment of Mechanics, KTH Royal Institute of Technology,
Sweden; \\

\section*{Abstract}

% \textcolor{blue}{A short (ca. 100 word) summary of the software being
% described: what problem the software addresses, how it was implemented and
% architected, where it is stored, and its reuse potential.}

\href{http://fluiddyn.readthedocs.io}{FluidDyn} is a project to foster
open-science and open-source in the fluid dynamics community.  It is thought of
as a research project to channel open-source dynamics, methods and tools to do
better science.
%
We propose a set of Python packages forming a framework to study fluid dynamics
with different methods, in particular laboratory experiments (package
\fluidpack{lab}), simulations (packages \fluidpack{fft}, \fluidpack{sim} and
\fluidpack{foam}) and data processing (package \fluidpack{image}).
%
In the present article, we review the recent advancements in web-based software
development tools and Python which facilitated the execution of this project,
and establish a picture of software engineering in fluid mechanics.
%
We give an overview of the specialized packages of the project and then
focus on the base package called \fluidpack{dyn}, which contains common code
used in the specialized packages.  Packages \fluidpack{fft} and \fluidpack{sim}
are described with greater detail in two companion papers, \citet{fluidfft,
fluidsim}.
%
With the project FluidDyn, we demonstrate that specialized scientific code can
be written with methods and good practices of the open-source community. The
Mercurial repositories are available in Bitbucket
(\url{https://bitbucket.org/fluiddyn/}). All codes are documented using Sphinx
and Read the Docs, and tested with continuous integration run on Bitbucket
Pipelines and Travis.
%
To improve the reuse potential, the codes are as modular as possible, leveraging
the simple object-oriented programming model of Python.
%
All codes are also written to be highly efficient, using C++, Cython and
Pythran to speedup the performance of critical functions.

\section*{Keywords}

% \textcolor{blue}{keyword 1; keyword 2; etc. \\
% Keywords should make it easy to identify who and what the software will be
% useful for.}

Fluid dynamics research with Python. Free and open-source software, modular,
object-oriented, collaborative, efficient, tested, documented.

\section*{Introduction}

% \textcolor{blue}{An overview of the software, how it was produced, and the
% research for which it has been used, including references to relevant research
% articles. A short comparison with software which implements similar
% functionality should be included in this section. }

\subsection*{Science, software, open-source and the computer revolution}

Science is mainly a collective activity.  We can go further only by
\href{https://en.wikipedia.org/wiki/%
Standing_on_the_shoulders_of_giants}{``standing on the shoulders of giants''}
(and of a huge number of technicians and scientists).  Science is a lot about
how to build new knowledge from the work of others and hence, the exchange
of ideas is a fundamental aspect.
%
In the last decades, we have lived through a revolution on how people exchange
ideas.
%
Computers of all kinds (from smartphones to HPC clusters) connected by a world
wide web are used for human communication and also for many other applications.
%
It has become nearly effortless to reproduce and exchange ideas and data.
%
The set of intangibles that grows when shared and degrades when hoarded, such as
knowledge and love, has been somehow extended.
% Relevant quote: 
% O Goddess Saraswati, your Treasure is unique in nature.  It increases when it
% is freely spent and gets destroyed if it is hoarded (and not shared with
% others).
%
The information technology revolution open doors to fantastic opportunities for
human collaboration, and of course, for science.

Web related activities alone account for at least 5\% of GDP in the USA and the
European Union\footnote{See the report by Internet Association titled
\href{https://internetassociation.org/reports/refreshing-understanding-internet-economy-ia-report/}
{``Refreshing Our Understanding of the Internet Economy''}}.
% \cite{hooton_christopher_refreshing_2017}.
A huge amount of money (and work) is invested on the related technologies. We
are familiar with the most prominent companies involved in this dynamics
(Google, Facebook, etc.), but there are also several smaller and sometimes
lesser known organizations.  Most of these companies base part of their work on
the open-source paradigm contributing to open-source languages, libraries,
software and operating systems, while also using them --- a win-win situation
for both the corporations and the community.
%
This has lead to deep changes in software engineering, with a massive use of
open-source methods and tools, for example
\href{https://en.wikipedia.org/wiki/Distributed_version_control}{distributed
version control systems (DVCS)} and web-based source development platforms.

The computer performance continues to
\href{https://en.wikipedia.org/wiki/Moore\%27s_law}{increase exponentially},
now also with the help of Graphical Processing Units (GPU). This gave way to a
big boom in practical uses of data science and machine learning, which drives a
strong research on artificial intelligence.  This dynamics contributes to
progresses in open-source software.

These changes in our world also reflect in the way science is done.
%
Software and programming in science occupy a much bigger place than before.
%
% see http://sciencecodemanifesto.org/
% http://lorenabarba.com/gallery/reproducibility-pi-manifesto/
% https://www.numfocus.org/
%
The role of software in science has changed. In the past, coding was sometimes
considered as an inferior activity by some scientists.  The focus was on the
theory and the mathematical demonstration, which had to be elegant as it gets
included in the articles.  In contrast, it was normal to write crude code
% av: crappy = shitty!
% alternatives: shabby, inelegant, amateurish, unpolished, clumsy
% https://en.oxforddictionaries.com/thesaurus/amateurish
and to just show the results. Nowadays, codes tend to be at the heart of
research.

``Open-science'' is a new trend taking advantage of these new facts.  Pioneering
attempts are being made to do better science, improving reproducibility and
collective efficiency, by using the open-source methods and tools for science
and sharing and collaborating via the world wide web.

% http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261

This paper presents FluidDyn, a project to foster open-science and open-source in
fluid mechanics. This first section ``Overview'' is organized into two large
parts.  A long introduction presents the context and describes our motivations and
the concept behind the project FluidDyn.  It is organized as follows:
\begin{itemize}
\item A scientific project based on the collaboration through open-source, such
as FluidDyn, is only possible thanks to the recent developments in methods and
tools for open-source software engineering. We start by presenting these
developments and discuss the implications for the scientists.

\item FluidDyn is based on the language Python. Incidently, it is the defining
characteristics of Python which make the eventual success of such a
project realizable.

Python is one of the most important tool in recent open-source dynamics and in
particular in science.
%
Python in fluid mechanics is growing fast, but is yet to reach widespread
adoption as the language of choice. Therefore, it is useful to present its
characteristics for readers who are unacquainted with its nuances.

\item We will discuss about the possible contradiction between productivity of
individuals and productivity at the community level.

\item We will establish a picture of software engineering in fluid mechanics
and present some of the main tools commonly used.

\item The introduction finally ends with a summary of the motivations for the
project FluidDyn and a presentation of its principles.
\end{itemize}

The section ``Overview'' is completed by a second subsection called
``Implementation and architecture'' devoted to present the organization of the
packages of the project and to describe in more detail the base package
\fluiddyn\footnote{We use FluidDyn (with capital letters) to name the project and
\fluiddyn for the base package.}.

\subsection*{Methods and tools for open-source software engineering}

\subparagraph{Free and Open-Source Software (FOSS).} The term \emph{free} in
FOSS is a misnomer, as it actually stands for \emph{freedom} (to use, modify
and distribute). The FOSS movement has dramatically decreased the cost of using
computers, and is evident from the widespread use of GNU/Linux systems in
desktops, computing clusters and web-servers in academia and beyond.
%
The beginnings of FOSS culture can be traced back to the success for scientific
publication of the free type-setting standards \TeX\ (1977) and \LaTeX\ (1985).
%
Another founding moment for the FOSS movement was the launch of the GNU project
in 1983 by \href{https://en.wikipedia.org/wiki/Richard_Stallman}{Richard
Stallman}, to create a Unix-like computer operating system composed entirely of
free software\footnote{For completeness, see also the work done on
\href{https://www.levenez.com/unix/}{other Unix operating systems,
for example BSD}.}.
%
GNU is today known for its compiler collection (GCC) and a multitude of tools
which when combined with the Linux kernel (created by
\href{https://en.wikipedia.org/wiki/Linus_Torvalds}{Linus Torvalds} in 1991)
forms the GNU/Linux operating systems that we are familiar today.
%
FOSS movement has had huge successes in many frontiers --- Apache could be
termed as the ``first killer-app of Linux'' and now with an ever-increasing
suite of softwares including Firefox, LibreOffice, Gimp as solid alternatives
to proprietary offerings.
%
Linux has become the most widely used kernel, being deployed on servers,
personal computers, embedded devices, and also smart phones (with Android).

% remark Julien Salort: not interesting. Nothing on BSD.
% https://www.levenez.com/unix/

Over the years, FOSS development has transitioned from an organic community of
volunteers, towards an organized system with participation from industries,
non-profit organizations and government institutions. This could be regarded as
the second-age of FOSS movement~\cite{fitzgerald_transformation_2006}.

%todo: DONE
% Here, we have to add a paragraph on free and open-source (libre)
% software:

% The following facts can be mentioned:

% - tex (started in 1977), latex (1985)

% - GNU (started 1983), FSF in 1985, licence and philosophy...
% first complete system (with the Linux kernel) in 1991

% - 90s arrival of internet (soon mass market)

% Some huge open-source successes : Apache ("first killer app of Linux"),
% now Firefox, Open-office
% av: https://www.reddit.com/r/AskReddit/comments/7x639l/what_free_software_is_so_good_you_cant_believe/du6pw11/

% Linux kernel now widely used on servers, personal computers, embedded
% devices, smarth phone (with Android).
% https://en.wikipedia.org/wiki/Linux_kernel

% Now, new period for open-source: 
% see http://www.cepis.org/upgrade/files/full-2005-III.pdf

% - "Libre Software Movement: The Next Evolution of The IT Production
% Organization?"

% - ``the composition of development teams was changing, from all-volunteer
% teams to teams with paid participants from industry, government or
% not-for-profit organizations.''

% By the way, we also have to use the term "libre software" (?)
% av: Libre-software is not so mainstream. Even Linux kernel is not libre with
% binary blobs for drivers.

% We may have to change the order of this list?
% - git and mercurial 2005
% - software repositories ~ 10 years before
% so I would exchange the too subparagraph...
% av:That is probably because we did not mention patches and CVS, subversion
% etc.


\subparagraph{Distributed Version Control Systems (DVCS) and web-based source
development platforms.}  Collaboration was achieved in the early years of FOSS
development through emailing ``patches'', and centralized VCS near the turn of
the 21\textsuperscript{st} century.  The collective collaboration on
development is today streamlined with the emergence of DVCS (especially, Git
and Mercurial) and web-based source development platforms like GitHub,
Bitbucket and more recently GitLab.
%
These tools are designed to work on any kind of text files, not just codes, so
they could even be utilized to write scientific documents with multiple authors.
%
These tools are so useful for work, that using DVCS in science shall reap
benefits in terms of productivity in the long run~\cite{wilson_best_2014}.

Git has become the \textit{de facto} standard for source management. However,
for the FluidDyn project, we prefer to use Mercurial because we consider it to
be simpler for the beginners and just as powerful for expert users\footnote{See
\href{https://www.mercurial-scm.org/wiki/GitConcepts}{``a detailed
comparison''} on Mercurial website.}. Simplicity and lowering the learning
barrier for scientists who are not experts in software development is crucial
for our project.
%
Note that, it is now possible to use Mercurial to work with Git repositories,
for example hosted in GitHub and GitLab\footnote{By using the Mercurial
extension \href{http://hg-git.github.io/}{hg-git}.}.

\subparagraph{Package management and third-party software repositories.} Python
as a medium for scientific computing owes its success to a rich ecosystem of
third-party packages.  Smooth deployment of these packages requires a package
manager and a supported repository. A credible hub to upload source codes as
compressed archives or as pre-compiled binary packages was therefore
indispensable. The use of repositories started with the conception of CPAN and
CRAN repositories for the languages Perl and R in 1993 --- based on the
Comprehensive TeX Archive Network (CTAN) model for \TeX\ packages. 

Today, \href{https://pypi.org}{Python Package Index (PyPI)} and
\href{https://anaconda.org}{Anaconda Cloud} are major repositories for
delivering and downloading Python packages, assisted by package managers
\pack{pip} and \pack{conda} respectively.
% Almost all major 
% languages\footnote{\href{https://stackoverflow.com/questions/1693529/%
% list-of-top-repositories-by-programming-language}{https://stackoverflow.com/%
% questions/1693529/list-of-top-repositories-by-programming-language}}
% have similar repositories, with exceptions such as Fortran and Matlab.
Specialized package managers have also emerged to install scientific
libraries and compilers such as \pack{easybuild} and \pack{spack}.

\subparagraph{Continuous integration (CI).} The need for strong reliability of
programs with increasing complexity has forced developers to use continuous
integration.
%
In CI, firstly the code is \emph{built} against a known, repeatable emulation
environment on a server. Thereafter, a set of \emph{unit tests} are run for each
commit (modification of the code) to check that no bugs have been introduced. It
is also important to analyse what portion of the source code is covered by these
unit tests --- a method referred to as \emph{code coverage}. In FluidDyn project
we stop at this level, but it is possible to further enhance it by adding code
style checks by running linters alongside unit tests; and also to add
\emph{continuous delivery} of documentation and package releases.

Almost all CI platforms are free for open-source projects --- Travis CI,
CircleCI and Appveyor % (sic) av: I have double checked the official names
to name a few. Bitbucket and GitLab have built-in CI
implementations.  Websites such as Codecov and Coveralls help to chart and
analyse the code coverage history, viewable by anyone.

When the code depends on a wide variety of external dependencies, CI is useful
to avoid \href{https://en.wikipedia.org/wiki/Software\_rot}{software rot},
i.e.\ to avoid becoming obsolete with newer versions of the compiler or its
dependencies.  CI is specifically important for dynamical
languages and open-source code since it allows open-source projects to welcome
participation from many people while maintaining a high degree of reliability.

\subparagraph{Websites to share knowledge.} Mutual aid and knowledge sharing
have been important aspects of the web since its inception and these practices
are now of foremost importance for many human activities.
%
In particular, learning programming and computer science is now greatly aided
by websites like the community driven encyclopedia
\href{https://www.wikipedia.org/}{Wikipedia} or the community driven forum
\href{https://stackoverflow.com/}{Stack Overflow}, where people write thousands
of questions and answers on programming each week.
% todo: remark from Julien Salort: it's old.
% av: So are technologies like git and hg, but we are putting together what
% characterizes our workflow, aren't we? We can emphasize that these forums
% often work only when there is a "thriving community" like the Python
% userbase.
IRC protocol has continued to sustain the test of time. Instant messaging
channels such as \codeinline{\#python} on Freenode facilitate interactive
discussions and a form of community-level support system --- a role which used
to be filled by mailing lists alone in the past. Alternatives such as Riot
(Matrix protocol), Gitter, and Slack have gained traction offering different
integrations solutions on top of instant messaging.
%
Through instant messaging a geographically separated team, as is the case with
FluidDyn project, can achieve near real-time collaboration without the hassle
of setting up video conferences or meetings.

\subparagraph{Automatic web documentations built on servers.}  Documentation
generators such as Sphinx (for Python) and Doxygen (for many languages
including C, C++, Fortran and Python) ``automagically'' convert in-line
comments into documentations as PDF, HTML and more. Read the Docs has in the
recent years become the main host for building and publishing documentation on
servers as web pages, thus easing the process for developers and considerably
improving access to information.

% We need transitions and some words at the end of the subsections to help the
% readers to understand why we present what we present and what is the link
% with the FluidDyn project...

We have presented new methods and tools used in software development.  Taken
all together, they greatly improve our efficiency for collective work and open
possibilities of interactions that were unthinkable only few years ago.
%
For example, the success of community-driven software like \pack{astropy} (a
subject-area research library with \href{https://github.com/astropy/astropy}{more
than 240 contributors}) and \pack{scikit-learn} (a machine learning toolkit with
\href{https://github.com/scikit-learn/scikit-learn}{more than 1000 contributors})
would not have been possible without these new software development methods.
%
% pa: done: we need to mention in this paragraph existing community-driven
% subject-area research libraries (see the file
% ./communitydriven_research_soft.md).
%
As a scientists, we can wonder how to fully exploit these new possibilities to do
science.  The project FluidDyn is an attempt to help the fluid dynamics community
to advance in this direction.



\subsection*{Python, a programming language adapted for open-science}

We now turn to the presentation of the
\href{https://www.python.org/}{programming language Python}, another
fundamental tool for the FluidDyn project.  Note that, this is not a detailed
technical presentation: we focus on the ideas necessary to understand why the
FluidDyn project is based on this language and on facts interesting for
potential FluidDyn users and developers.

Python community has thrived remarkably in the last few years, evident from the
fact that by the end of 2017, Python became the second most popular language in
terms of pull-requests in GitHub\footnote{See GitHub's annual report titled
\href{https://octoverse.github.com/}{ ``The State of the Octoverse 2017''}.} and
questions on StackOverflow\footnote{See
\href{https://stackoverflow.blog/2017/09/06/incredible-growth-python/}{ ``The
Incredible Growth of Python''}, a blog post by Stack Overflow.}.
%
TIOBE-index\footnote{\url{https://www.tiobe.com/tiobe-index}}, which assess the
popularity of a language based on number of skilled users, courses and
third-party vendors, ranks Python at number four. IEEE Spectrum\footnote{See
the analysis titled \href{https://spectrum.ieee.org/computing/software/%
the-2017-top-programming-languages}{``The 2017 Top Programming Languages''} on
IEEE spectrum.} ranks Python as the most popular language in 2017 with a metric
based on trends, demand by employers and users on open-source hubs.  All the
rankings do agree that Python usage has been steadily growing and is here to
stay.  Python largely outclasses languages like Fortran and Matlab according to
these statistics, and is now clearly one of the mainstream languages with C,
Java, C++ and Javascript.
%
Let us summarize some characteristics of the Python languages that have lead to
such incredible success.

% todo: DONE
% We need an introductory paragraph on how big is Python today
% + the fact that several scientists miss this fact.

% See the file ./python-one-main-languages.rst

The Python language has been designed to boost the communication of technical
ideas between humans. The resulting code is as simple as possible due to the
inherent emphasis on readability. Humans can focus on the ideas. Python differs
from other mainstream languages for a variety of reasons:

\begin{itemize}
\item \textbf{Aesthetics.} The syntax is nice and elegant. Blocks of code are
defined with the indentation.
\item \textbf{Code style.} There are explicit guidelines
(\href{https://www.python.org/dev/peps/pep-0008/}{PEP~8}) supporting the
regularity (and thereby readability) of the code and tools
(linters) to evaluate adherence to the style.
\item \textbf{Dynamic typing.} The variables (the ``names'' in the code) are
not attached for life to an object in memory\footnote{Opposite behavior
compared to static languages as Fortran, C and C++.}. The types of the objects
are inferred at run time from the code and the context, and in many situations,
one does not need to declare them explicitly.
%
% Being a strongly typed language\footnote{In contrast to weakly typed languages,
% such as JavaScript}, types of the objects are respected and implicit type
% conversions are rarely performed.
% av: "rarely / never"?
% eg: 'hello' * 2 -> 'hellohello'; 1/3 -> 0.3333333
%
% pa: I think we can not cover such complicate notion in this document... It is
% anyway not so important for our demonstration.
\item \textbf{Automatic memory management.}  The user need not handle the
memory allocation and deallocation by hand.  There are, of course, ways to
control the memory usage of a program, though not as precisely as for low-level
languages.
\item \textbf{Interpreted.} The standard way\footnote{There are also tools to
compile Python code (for example Cython and Pythran).} to run a Python code is
to ``interpret'' it.  A program called the interpreter executes the code
nearly\footnote{Actually, the execution of Python code is less simple and
involves an Abstract Syntax Tree.}  instruction-by-instruction, with very few
optimizations.
%
In contrast to other compiled languages such as Fortran, C or C++, there is no
proper compilation step: the code is not translated to optimized machine
instructions, but only to Python bytecode.
%
We will discuss shortly the disadvantages of this characteristic. Two evident
advantages are rapid development cycle and the possibility of interactive
workflow (usually with IPython or Jupyter notebooks).

Note that there are many implementations of Python interpreters\footnote{We can
cite CPython (written in C), Jython (Java), IronPython (C\#), PyPy (Rpython, a
subset of Python) and MicroPython/CircuitPython (C, targeted to
micro-controllers).}. The default and most widely used implementation is written
in C and is called CPython.

\item \textbf{Simple, yet powerful.}

\begin{itemize}
\item Python supports multiple programming paradigms, including imperative,
object-oriented and functional.

\item It is easy to interface with code written in other languages (in particular
C, C++ and Fortran).

\item Python comes with a large and high quality \href{https://docs.python.org%
/3/library/index.html}{standard library}.

\item Python can be deployed on many different machines with different operating
systems (Linux, Windows, macOS, Android) and architectures (from a microcontroller
--- with MicroPython or CircuitPython --- to a Blue Gene supercomputer).

\end{itemize}
\end{itemize}

Thus, Python is one of the best ``glue'' languages for fast prototyping.
Developing with Python is much faster and easier than with many other
languages.  Less bugs are introduced by the developers simply because there are
much less lines of code \cite{nanz_comparative_2015}.
%
Maintaining and modifying existing code is also eased by the very good code
readability associated with the language.

Another consequence of the apparent simplicity of Python is that the learning
curve has a nice shape. Learning Python is very easy at the beginning and the
complexity gradually increases as the level of the developer increases.
%
Therefore, Python is good for developers of all levels: very gentle for
beginners and very powerful for advanced users.

Python is an old language (first implementation in 1991!) but it continues to
evolve.
\begin{itemize}
\item A hard and controversial decision was taken to clean up Python from its
historical incoherences by having a new series of backwards-incompatible
versions of the language, Python 3.  Python 3.0 was released in December 2008.
After a long and difficult process, the transition from Python 2 to Python 3 is
now (2018) behind us for nearly all important packages\footnote{Close to
\href{https://python3wos.appspot.com/}{95\% of the most downloaded packages}
are Python 3 compatible now.}.  Especially for scientific applications, we can
now work only in Python 3. For instance, \href{https://github.com/%
numpy/numpy/blob/master/doc/neps/dropping-python2.7-proposal.rst}{future
releases of \Numpy} and of \href{http://www.python3statement.org/}{the main
scientific packages} will soon drop compatibility with Python 2. These changes
open doors to very clean, coherent and potentially faster Python
experience. Therefore, one should not use Python 2 for science anymore and
instead, adopt the newer versions ($>=3.5$ in 2018).

\item New versions of Python have nice new features to adapt to new usages and to
recent trends in computer science.  For example, three new features were
introduced in Python 3.5 (first released on September 2015): the \codeinline{@}
operator for matrix multiplication, the new \codeinline{async} and
\codeinline{await} keywords for concurrency and type hinting\footnote{See also our
paragraph on the issue of type checking in the following.} (with the module
\codeinline{typing} and an associated syntax presented in the
\href{https://www.python.org/dev/peps/pep-0484/}{PEP~484}).
\end{itemize}

\subsubsection*{An incredible success}

Being a very versatile language, Python is widely used for many different
applications:
\begin{itemize}
\item Simple scripting.

\item System, database and network administration.

\item GNU/Linux distribution software.

\item Python is now one of the main scripting languages to add programmability
to applications (for example Paraview, Visit, QGIS, Blender, ...).

\item Web servers.

\item Web scraping and data analysis.

\item Animation movies, game development and gaming.

\item Education. Python has recently become one of the most popular language
for teaching programming and computer science.

\item Science!
Python is now widely used in scientific applications. It has now a mature and
powerful scientific ecosystem with well-established based packages (\Numpy for
N-dimensional homogeneous arrays, \Scipy as the fundamental toolkit for
scientific computing, \pack{Matplotlib} for plotting and \pack{Pandas} for data
structures) and several more specialized packages (to name a few, \pack{h5py},
\pack{mpi4py}, \pack{skimage}, \pack{sklearn}, ...). There are now great (and
often shiny) tools for most of the applications. For example, the
\href{https://github.com/rougier/python-visualization-landscape}{%
landscape of visualization tools usable through Python} is impressive.
%
Python is one of the main languages for data science with packages such as
\pack{Pandas}, \pack{statmodels}, \pack{sklearn}, \pack{Keras}, \pack{TensorFlow}
and \pack{PyTorch}.

There are now ready-to-use ``batteries-included'' Python installers
(similar to Matlab), like in particular the
\href{https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)}{%
freemium open-source distribution Anaconda}, so it is very easy to start using
Python for scientific purposes.

There are now nice Python integrated development environments (IDE) adapted for
scientists, for example \href{https://github.com/spyder-ide/spyder}{Spyder},
\href{http://www.pyzo.org/}{Pyzo} and
\href{https://jupyterlab.readthedocs.io}{Jupyterlab}.

\end{itemize}

Python has a very large and supportive community (see for example
the \href{http://stackoverflow.com/tags}{Stack Overflow tags}).
%
But Python is also supported by companies using Python and the open-source
dynamics.  Most of these companies that support Python are specialized in
web technologies and data-driven research.  But, recently, companies basing
their business
% av: earlier "computer and web technologies". Computer is quite obvious, isn't
% it?
on Python for science have started to become quite powerful in terms of
development. For example Anaconda Inc.\ is strongly implied in the development of
tools like \pack{Numba} and \pack{dask}. Hardware giants such as Intel and Nvidia
also contribute to the Python scientific ecosystem.


\subsubsection*{Some Python issues}

\subparagraph{Since the standard Python interpreter does not do any proper
compilation,} Python code in some CPU bounded cases can be too slow.  Thus, one
has to use special tools to get optimized machine instructions to speedup
performance-critical code.

\textbf{Ahead Of Time compilation.}  Since Python is primarily a ``glue''
language, a common solution is to use code written in C, C++ or Fortran.
%
% todo: DONE
% add a reference to ctypes and cffi to call C with pure Python code
%
It is possible, without a compilation step, to call functions in C libraries
through pure python code using \pack{ctypes} standard module or the \pack{cffi}
package.
%
Libraries with Python bindings can also be used directly (for e.g., wrapped
using SWIG or Boost.Python) or one has to write extensions, i.e.\ Python
modules written in C or C++ using the CPython API that can be imported directly
from Python like any Python module.
%
However, to rely on such techniques to call C libraries would inhibit
development of extending functionalities, as they have a steeper learning
curve compared to Python.
%
Nevertheless, this strategy has given rise to the base modules of the scientific
Python stack, namely \Numpy, \Scipy and \pack{Matplotlib}.

Python developers do not actually have to write the compiled extensions in C,
C++ or Fortran, since there are tools to generate them automatically from
Python code (Pythran) or from Cython code. Cython \citep{behnel_cython2011} is a
smooth blend of two languages with possibility to add type declarations like in
C/C++, but with a syntax similar to Python.

Pythran \citep{guelton2015pythran, guelton2018pythran} is a quite recent Python
compiler which gives impressive results. It creates compiled extensions from pure
Python code with simple type annotations written as comments.
%
The resulting extensions are usually as fast as Fortran or C++ written by
non-specialists \cite[see, for example, benchmarks for the packages
\fluidpack{fft} and \fluidpack{sim},][]{fluidfft, fluidsim}.
%
They are created with a very interesting two-step compilations: first the code
is optimized at the Python level and then an automatically produced C++ code is
properly compiled.
%
This two-step compilation provides room for very clever optimizations as
demonstrated, for example, in
\href{http://serge-sans-paille.github.io/pythran-stories/%
being-more-than-a-translator.html}{%
this blog post by the main developer of Pythran}.
%
Pythran supports OpenMP pragma \citep{guelton2013compiling} and can use modern
vectorization with SIMD instructions \citep{guelton2014exploring}. Pythran
understands both Matlab-like vectorized code and C-like code with explicit loops
\citep{guelton2018pythran}.

We present in the two companion papers \citep{fluidfft, fluidsim} examples of
highly efficient Python codes, optimized using a combination of Cython and
Pythran extensions.

\textbf{Just In Time (JIT) compilation.} Another way to obtain optimized
machine instructions is by using JIT compilation, i.e.\ by compiling only the
critical code at run time.
%
This strategy can yield good results with other languages as for example Matlab
or Julia.
%
Pypy, an alternative interpreter written in Python, has a JIT compiler. However,
Pypy is not widely used for scientific applications mainly because of
compatibility problems with the extensions written for CPython.
%
However this could change since a recent version of Pypy (6.0) supports the main
packages of the Python scientific stack (\Numpy, \Scipy, \pack{Matplotlib},
\pack{Pandas}, etc.).

Adding a JIT to the interpreter CPython has been notoriously difficult (see
\href{https://faster-cpython.readthedocs.io/}{faster-cpython}). One of the
problem seems to be the C API provided in CPython and used by many extensions.
%
Python being first a glue language, a faster CPython with a JIT is actually not
so necessary for many purposes. Therefore, not as much money and work has been
put in accelerating Python than for example for Java and JavaScript.

Another strategy is to add a JIT to CPython through an external package
(\pack{Numba}) and to only compile the critical functions.  Moreover,
\pack{Numba} is particularly interesting because it can take advantage of the
GPU\footnote{See for example
\url{https://devblogs.nvidia.com/parallelforall/seven-things-numba/}}.


\subparagraph{No type-checking.} In contrast to languages using Ahead Of Time
compilation, there is no type-checking in Python (and more generally in many
dynamical languages).  Type-checking is useful since it can avoid simple bugs.
However, it is not a good practice even with static languages to rely only on
type-checking to look for bugs in a code.
%
It is known that a rigorous set of unit tests is also needed to obtain more
reliable codes.
%
Dynamical languages rely a lot on unit tests so it is very important to put at
least a little bit of time and energy to write a decent battery of tests.
%
We have to mention the young and promising project
\href{http://mypy-lang.org/}{Mypy}\footnote{Guido van Rossum, the author and
``Benevolent Dictator For Life'' of Python, is involved in the project Mypy.},
which allows for Optional Static Typing and thus type-checking for
Python\footnote{We plan to add type hitting in the most important FluidDyn modules
to investigate how we can take advantage of type-checking with Mypy.}.

\subparagraph{A lively, huge and thus complicated ecosystem.} As already
mentioned, Python is a versatile language. For many applications, the standard
library is insufficient and one needs external packages.
%
% For most applications there are several projects usable through Python.
Most of the time there are several Python packages for a certain application.
A user/developer has to make choices regarding the tools to use.  Especially
for the beginners who are not used to the open-source realm, it can be
difficult to understand a ``landscape of projects'' for a particular
application and to make good technological choices.

It is important to get a good introduction on how to work with the open-source
dynamics and the scientific Python ecosystem.  It can also be very useful to ask
to more advanced users and attend Python courses or workshops\footnote{This could
be regular academic courses, MOOCs or workshops like ones organized by
\href{https://software-carpentry.org/}{Software Carpentry}.}.
%
Of course, such abundance of projects can also be seen as an advantage.

\subparagraph{Absence of multicore computational parallelism using threads.}
Threads behave like light subprograms. In principle, they can use the different
cores of the CPU at the same time but the CPython interpreter forbids that they
interpret python code at the same time\footnote{See
\href{https://opensource.com/article/17/4/grok-gil}{this blog article on ``How
to grok the GIL''} and
\href{https://faster-cpython.readthedocs.io/cpython37.html}{this technical
presentation by a CPython core developer}.}.
%
This limitation comes from an important detail in the implementation of the
CPython interpreter: a Global Interpreter Lock (GIL) to prevent race conditions
that could corrupt data.
%
It seems that the GIL greatly simplies the implementation of CPython and that it
is very difficult to remove it while keeping other nice technical properties of
CPython intact\footnote{See \url{https://wiki.python.org/moin/GlobalInterpreterLock}.}.
Note that Pypy also uses a GIL, while two other implementations of the Python
language, Jython and IronPython, do not have this limitation.

At any rate, threads using the interpreter do not use the CPU at the same time
so it is not possible to use threads to do multicore computational parallelism
with pure python code to accelerate CPU bounded tasks.
%
Note however, that threads can be (and are widely) used in Python for
concurrency, i.e.\ to perform I/O tasks (which does not need CPU) concurrently.
%
To do proper computational parallelism for CPU bounded tasks with the GIL, one has
to use other strategies: extensions (fine grain parallelism), multiprocessing
(coarse grain parallelism) and inter-process communication, for example, with MPI
(with mpi4py) or ZeroMQ.


\subsubsection*{Rich landscape of open-source languages useful for science}

So far, we have discussed in detail about Python. However, there are many
interesting open-source languages that can be used for scientific applications.
All have strengths and weaknesses.

Fortran remains widely used for computing codes, at least in fluid mechanics.
C++ is used in many recent developments. Its recent improvements (C++11, C++14)
make it a modern and very useful language.
%
JavaScript, Java, Scala, Smalltalk, Haskell, R, Julia, Perl and Lua are used for
some scientific codes. Go and Rust are quite young languages and are not (yet)
widely used for scientific applications.

It is of course very difficult to predict which technologies will be widely
used in even the near future (10 or 20 years for example).
%
From what we see, it seems that the idea of ``one language to do everything for
science'' will not succeed, at least soon.  This was maybe the project of Julia
creators, but despite the qualities of this language, it also has
limitations\footnote{For example C-like include instead of Python-like
import. From our point of view, a Pythonic flavor of Julia would be very
interesting.}.
%
It seems that a very important aspect in the future will be interoperability
between coexisting tools\footnote{See for example the cross-language development
platform \href{https://arrow.apache.org/}{Apache Arrow}.}.

So what about Python?  As already mentioned, it is today the language with the
strongest dynamics in science, data analysis and much more.  Several students
learn and will learn it. Many scientists in the future would, hopefully, be
able to appreciate these strengths and favour the Pythonic approaches.  With
such dynamics today, we can anticipate that the Python world will be able to
embrace new technological trends and to overcome some of its current
limitations.
%
It is reasonable to predict that Python will continue to shine as a language
(i) particularly good for human communication and to write ideas, (ii) for fast
prototyping and (iii) for interoperability with code written in other languages
(see for example the recent projects \href{https://cppyy.readthedocs.io}{cppyy}
and \href{https://github.com/pybind/pybind11/}{pybind11}, or
\href{https://github.com/PyO3}{the possibility to interact with Rust code}).

\subsubsection*{Summary: Python, a versatile Swiss Army Knife for the
scientists}

Python has become a great versatile tool for scientists.  The scientific
ecosystem of Python is nowadays really impressive.  The scientific Python
community is strong and quickly growing.
%
A scientist with good skills in Python can do most of what she/he needs to do in
programming (which does not mean that Python is the best language to do
anything!).
%
In contrast, no skill in Python is a real disadvantage for many tasks and for
employability.
%
Therefore, in most fields today, if a scientist has to acquire good skills in
one language, it is in Python.

Considering what we have presented on Python, we think that it is reasonable to
bet that it will become one of the \textit{lingua franca} in science and in
fluid dynamics.  This explains why we have chosen it to be the main language
for the project FluidDyn.


\subsection*{Productivity at individual, group and community levels}

% remark Julien Salort "It is well known that": bad
% pa: common sense says ?
% It is well known 
It is not surprising
% av: "It is not surprising"
%
that short-term efficiency and long-term efficiency are
sometimes incompatible.
%
Quick and dirty scripts can be efficient in the short term, but such a workflow
has a negative impact on a longer time scale.

Similarly, we can also differentiate productivity at different scales of
collaboration.
%
An individual can be very efficient with a particular tool, but the same tool
can be very inefficient to collaborate even with co-workers.
%
A group can be very efficient with a code which is closed-source.  If this
strategy can be efficient for the group (at least in the short term), it leads
to waste for the community. Other groups will need to develop codes with the
same features.  Good ideas will be spread in different codes and since the
number of users and developers are smaller for each of these codes, the code
quality will not improve as fast as it would with an open-source strategy. In a
nutshell, as this proverb says, \textit{``If you want to go fast, go alone. But
if you want to go far, go together.''}

Thus, a community using open-source methods can be efficient to co-develop its
tools. However, some of its members have to spend more energy to bear this
dynamics, which implies that they somehow sacrifice their individual efficiency.
We see that short-term efficiency, long-term efficiency and productivity at
different levels can often be incompatible and that we have to consider these
conflicts when choosing between different technologies.

\subsection*{Programming in the field of fluid mechanics}

We study fluid mechanics via laboratory experiments, in situ measurements,
analytic computation, numerical simulations and data processing.
%
Nowadays, programming is involved in all methods, and thus it is difficult to do
fluid mechanics without software development.
%
However, the level in software engineering is on average very low in the
community.
%
Even today, it is quite commonplace to start a PhD without any serious training in
GNU/Linux and modern programming tools.
%
A large majority of scientists and technicians are also unaware of the new
challenges and opportunities of open-source.

Since fluid dynamics is an engineering science, closed-source commercial software
have historically been dominant in the field.
%
Few people are aware of the vicious circle of the closed-source model for people
and groups relying on it.
%
This mechanism is nevertheless very damaging. In the closed-source model, a group
pays for a license or a new development. The group does not learn how to develop
what has been paid for. The company, on contrary, stands to gain from profits and
the feedback generated from users, through which it improves the product sold.
The group produces codes, books, courses using the closed-source product or
acquire knowledge on how to use the closed-source product, so that the group is
increasingly dependent on the product and is ready to pay more for it.
%
Such vicious circles are difficult to break, but not impossible. This can be
done by introducing alternative open-source solutions, bringing in positivity to
this circle.

We have seen that it was very common to rely on commercial companies and
closed-source software for most of the new developments in fluid mechanics.
%
Nevertheless there are also a lot of codes produced by scientists in the
field. But without the technical knowledge on how to work collectively on code,
this production is often of bad quality (compared to the information technology
industry standard) and it is basically doomed to be abandoned after its use.
%
However, it is natural to try to reuse code or at least to build the next steps
on what has been done before.  Thousands of hours of highly qualified people
are spent in trying to understand and reuse codes badly written with
inappropriate languages!

On a technical point of view, the majority of the coding in the field involve a
mix of Fortran/C or C++, shell languages (as Bash) and Matlab. For experiments,
the graphical programming environment
\href{http://www.ni.com/en-us/shop/labview.html}{Labview} is dominant for
control of physical objects and acquisition and Matlab is used for data
processing.
%
Often, languages are used for things for which they are not adapted.
%
Fortran, C or C++ have not been designed for fast prototyping of complex
programs.
%
It is known that for scientific purposes, one should restrict shell scripting
to extremely simple tasks.
%
Similarly, using Matlab for developing complex programs is technically a
questionable choice.
%
These technical solutions also represent a problem for code reuse, sharing and
collaborative development.
%
We are now going to present factual arguments explaining why we think that the