Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
%% Journal of Open Research Software Latex template -- Created By Stephen
%% Bonner and John Brennan, Durham Universtiy, UK.
%% see http://openresearchsoftware.metajnl.com
\documentclass{../jors}
\begin{document}
{\bf Software paper for submission to the Journal of Open Research Software} \\
To complete this template, please replace the blue text with your own. The paper
has three main sections: (1) Overview; (2) Availability; (3) Reuse potential.
Please submit the completed paper to: editor.jors@ubiquitypress.com
\rule{\textwidth}{1pt}
\section*{(1) Overview}
\vspace{0.5cm}
\section*{Title}
% \textcolor{blue}{The title of the software paper should focus on the software,
% e.g. “Text mining software from the X project”. If the software is closely
% linked to a specific research paper, then “Software from Paper Title” is
% appropriate. The title should be factual, relating to the functionality of the
% software and the area it relates to rather than making claims about the
% software, e.g. “Easy-to-use”.}
% old title
% FluidDyn: a Python open-source framework for research and teaching in fluid
% dynamics
% pa: this title may not be really good for this paper...
% - "research and teaching" may not be very important in the title
% - Recent progress in open-source software engineering and consequences on
% science
% todo: find a better title!
Taking advantage of recent advances in open-source software engineering to study
fluid dynamics: the example of the FluidDyn project
\section*{Paper Authors}
% \textcolor{blue}{1. Last name, first name; (Lead/corresponding author first) \\
% 2. Last name, first name; etc.}
1. AUGIER Pierre$^a$\\
2. MOHANAN Ashwin Vishnu$^b$\\
\smallskip
$^a$ Univ. Grenoble Alpes, CNRS, Grenoble INP\footnote{Institute of Engineering
Univ. Grenoble Alpes}, LEGI, 38000 Grenoble, France.\\
$^b$ Linn\'e Flow Centre, Department of Mechanics, KTH, 10044 Stockholm, Sweden.
\section*{Paper Author Roles and Affiliations}
% \textcolor{blue}{1. First author role and affiliation \\
% 2. Second author role and affiliation etc.}
1. Researcher, LEGI, Universit\'e Grenoble Alpes, CNRS, France\\
2. Ph.D. student, Deparment of Mechanics, KTH Royal Institute of Technology,
Sweden; \\
\section*{Abstract}
% \textcolor{blue}{A short (ca. 100 word) summary of the software being
% described: what problem the software addresses, how it was implemented and
% architected, where it is stored, and its reuse potential.}
\href{http://fluiddyn.readthedocs.io}{FluidDyn} is a project to foster
open-science and open-source in the fluid dynamics community. It is thought of
as a research project to channel open-source dynamics, methods and tools to do
better science.
%
We propose a set of Python packages forming a framework to study fluid dynamics
with different methods, in particular laboratory experiments (package
\fluidpack{lab}), simulations (packages \fluidpack{fft}, \fluidpack{sim} and
\fluidpack{foam}) and data processing (package \fluidpack{image}).
%
In the present article, we review the recent advancements in web-based software
development tools and Python which facilitated the execution of this project,
and establish a picture of software engineering in fluid mechanics.
%
We give an overview of the specialized packages of the project and then
focus on the base package called \fluidpack{dyn}, which contains common code
used in the specialized packages. Packages \fluidpack{fft} and \fluidpack{sim}
are described with greater detail in two companion papers, \citet{fluidfft,
fluidsim}.
%
With the project FluidDyn, we demonstrate that specialized scientific code can
be written with methods and good practices of the open-source community. The
Mercurial repositories are available in Bitbucket
(\url{https://bitbucket.org/fluiddyn/}). All codes are documented using Sphinx
and Read the Docs, and tested with continuous integration run on Bitbucket
Pipelines and Travis.
%
To improve the reuse potential, the codes are as modular as possible, leveraging
the simple object-oriented programming model of Python.
%
All codes are also written to be highly efficient, using C++, Cython and
Pythran to speedup the performance of critical functions.
\section*{Keywords}
% \textcolor{blue}{keyword 1; keyword 2; etc. \\
% Keywords should make it easy to identify who and what the software will be
% useful for.}
Fluid dynamics research with Python. Free and open-source software, modular,
object-oriented, collaborative, efficient, tested, documented.
\section*{Introduction}
% \textcolor{blue}{An overview of the software, how it was produced, and the
% research for which it has been used, including references to relevant research
% articles. A short comparison with software which implements similar
% functionality should be included in this section. }
\subsection*{Science, software, open-source and the computer revolution}
Science is mainly a collective activity. We can go further only by
\href{https://en.wikipedia.org/wiki/%
Standing_on_the_shoulders_of_giants}{``standing on the shoulders of giants''}
(and of a huge number of technicians and scientists). Science is a lot about
how to build new knowledge from the work of others and hence, the exchange
of ideas is a fundamental aspect.
%
In the last decades, we have lived through a revolution on how people exchange
ideas.
%
Computers of all kinds (from smartphones to HPC clusters) connected by a world
wide web are used for human communication and also for many other applications.
%
It has become nearly effortless to reproduce and exchange ideas and data.
%
The set of intangibles that grows when shared and degrades when hoarded, such as
knowledge and love, has been somehow extended.
% Relevant quote:
% O Goddess Saraswati, your Treasure is unique in nature. It increases when it
% is freely spent and gets destroyed if it is hoarded (and not shared with
% others).
%
The information technology revolution open doors to fantastic opportunities for
human collaboration, and of course, for science.
Web related activities alone account for at least 5\% of GDP in the USA and the
European Union\footnote{See the report by Internet Association titled
\href{https://internetassociation.org/reports/refreshing-understanding-internet-economy-ia-report/}
{``Refreshing Our Understanding of the Internet Economy''}}.
% \cite{hooton_christopher_refreshing_2017}.
A huge amount of money (and work) is invested on the related technologies. We
are familiar with the most prominent companies involved in this dynamics
(Google, Facebook, etc.), but there are also several smaller and sometimes
lesser known organizations. Most of these companies base part of their work on
the open-source paradigm contributing to open-source languages, libraries,
software and operating systems, while also using them --- a win-win situation
for both the corporations and the community.
%
This has lead to deep changes in software engineering, with a massive use of
open-source methods and tools, for example
\href{https://en.wikipedia.org/wiki/Distributed_version_control}{distributed
version control systems (DVCS)} and web-based source development platforms.
The computer performance continues to
\href{https://en.wikipedia.org/wiki/Moore\%27s_law}{increase exponentially},
now also with the help of Graphical Processing Units (GPU). This gave way to a
big boom in practical uses of data science and machine learning, which drives a
strong research on artificial intelligence. This dynamics contributes to
progresses in open-source software.
These changes in our world also reflect in the way science is done.
%
Software and programming in science occupy a much bigger place than before.
%
% see http://sciencecodemanifesto.org/
% http://lorenabarba.com/gallery/reproducibility-pi-manifesto/
% https://www.numfocus.org/
%
The role of software in science has changed. In the past, coding was sometimes
considered as an inferior activity by some scientists. The focus was on the
theory and the mathematical demonstration, which had to be elegant as it gets
included in the articles. In contrast, it was normal to write crude code
% av: crappy = shitty!
% alternatives: shabby, inelegant, amateurish, unpolished, clumsy
% https://en.oxforddictionaries.com/thesaurus/amateurish
and to just show the results. Nowadays, codes tend to be at the heart of
research.
``Open-science'' is a new trend taking advantage of these new facts. Pioneering
attempts are being made to do better science, improving reproducibility and
collective efficiency, by using the open-source methods and tools for science
and sharing and collaborating via the world wide web.
% http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261
This paper presents FluidDyn, a project to foster open-science and open-source in
fluid mechanics. This first section ``Overview'' is organized into two large
parts. A long introduction presents the context and describes our motivations and
the concept behind the project FluidDyn. It is organized as follows:
\begin{itemize}
\item A scientific project based on the collaboration through open-source, such
as FluidDyn, is only possible thanks to the recent developments in methods and
tools for open-source software engineering. We start by presenting these
developments and discuss the implications for the scientists.
\item FluidDyn is based on the language Python. Incidently, it is the defining
characteristics of Python which make the eventual success of such a
project realizable.
Python is one of the most important tool in recent open-source dynamics and in
particular in science.
%
Python in fluid mechanics is growing fast, but is yet to reach widespread
adoption as the language of choice. Therefore, it is useful to present its
characteristics for readers who are unacquainted with its nuances.
\item We will discuss about the possible contradiction between productivity of
individuals and productivity at the community level.
\item We will establish a picture of software engineering in fluid mechanics
and present some of the main tools commonly used.
\item The introduction finally ends with a summary of the motivations for the
project FluidDyn and a presentation of its principles.
\end{itemize}
The section ``Overview'' is completed by a second subsection called
``Implementation and architecture'' devoted to present the organization of the
packages of the project and to describe in more detail the base package
\fluiddyn\footnote{We use FluidDyn (with capital letters) to name the project and
\fluiddyn for the base package.}.
\subsection*{Methods and tools for open-source software engineering}
\subparagraph{Free and Open-Source Software (FOSS).} The term \emph{free} in
FOSS is a misnomer, as it actually stands for \emph{freedom} (to use, modify
and distribute). The FOSS movement has dramatically decreased the cost of using
computers, and is evident from the widespread use of GNU/Linux systems in
desktops, computing clusters and web-servers in academia and beyond.
%
The beginnings of FOSS culture can be traced back to the success for scientific
publication of the free type-setting standards \TeX\ (1977) and \LaTeX\ (1985).
%
Another founding moment for the FOSS movement was the launch of the GNU project
in 1983 by \href{https://en.wikipedia.org/wiki/Richard_Stallman}{Richard
Stallman}, to create a Unix-like computer operating system composed entirely of
free software\footnote{For completeness, see also the work done on
\href{https://www.levenez.com/unix/}{other Unix operating systems,
for example BSD}.}.
%
GNU is today known for its compiler collection (GCC) and a multitude of tools
which when combined with the Linux kernel (created by
\href{https://en.wikipedia.org/wiki/Linus_Torvalds}{Linus Torvalds} in 1991)
forms the GNU/Linux operating systems that we are familiar today.
%
FOSS movement has had huge successes in many frontiers --- Apache could be
termed as the ``first killer-app of Linux'' and now with an ever-increasing
suite of softwares including Firefox, LibreOffice, Gimp as solid alternatives
to proprietary offerings.
%
Linux has become the most widely used kernel, being deployed on servers,
personal computers, embedded devices, and also smart phones (with Android).
% remark Julien Salort: not interesting. Nothing on BSD.
% https://www.levenez.com/unix/
Over the years, FOSS development has transitioned from an organic community of
volunteers, towards an organized system with participation from industries,
non-profit organizations and government institutions. This could be regarded as
the second-age of FOSS movement~\cite{fitzgerald_transformation_2006}.
%todo: DONE
% Here, we have to add a paragraph on free and open-source (libre)
% software:
% The following facts can be mentioned:
% - tex (started in 1977), latex (1985)
% - GNU (started 1983), FSF in 1985, licence and philosophy...
% first complete system (with the Linux kernel) in 1991
% - 90s arrival of internet (soon mass market)
% Some huge open-source successes : Apache ("first killer app of Linux"),
% now Firefox, Open-office
% av: https://www.reddit.com/r/AskReddit/comments/7x639l/what_free_software_is_so_good_you_cant_believe/du6pw11/
% Linux kernel now widely used on servers, personal computers, embedded
% devices, smarth phone (with Android).
% https://en.wikipedia.org/wiki/Linux_kernel
% Now, new period for open-source:
% see http://www.cepis.org/upgrade/files/full-2005-III.pdf
% - "Libre Software Movement: The Next Evolution of The IT Production
% Organization?"
% - ``the composition of development teams was changing, from all-volunteer
% teams to teams with paid participants from industry, government or
% not-for-profit organizations.''
% By the way, we also have to use the term "libre software" (?)
% av: Libre-software is not so mainstream. Even Linux kernel is not libre with
% binary blobs for drivers.
% We may have to change the order of this list?
% - git and mercurial 2005
% - software repositories ~ 10 years before
% so I would exchange the too subparagraph...
% av:That is probably because we did not mention patches and CVS, subversion
% etc.
\subparagraph{Distributed Version Control Systems (DVCS) and web-based source
development platforms.} Collaboration was achieved in the early years of FOSS
development through emailing ``patches'', and centralized VCS near the turn of
the 21\textsuperscript{st} century. The collective collaboration on
development is today streamlined with the emergence of DVCS (especially, Git
and Mercurial) and web-based source development platforms like GitHub,
Bitbucket and more recently GitLab.
%
These tools are designed to work on any kind of text files, not just codes, so
they could even be utilized to write scientific documents with multiple authors.
%
These tools are so useful for work, that using DVCS in science shall reap
benefits in terms of productivity in the long run~\cite{wilson_best_2014}.
Git has become the \textit{de facto} standard for source management. However,
for the FluidDyn project, we prefer to use Mercurial because we consider it to
be simpler for the beginners and just as powerful for expert users\footnote{See
\href{https://www.mercurial-scm.org/wiki/GitConcepts}{``a detailed
comparison''} on Mercurial website.}. Simplicity and lowering the learning
barrier for scientists who are not experts in software development is crucial
for our project.
%
Note that, it is now possible to use Mercurial to work with Git repositories,
for example hosted in GitHub and GitLab\footnote{By using the Mercurial
extension \href{http://hg-git.github.io/}{hg-git}.}.
\subparagraph{Package management and third-party software repositories.} Python
as a medium for scientific computing owes its success to a rich ecosystem of
third-party packages. Smooth deployment of these packages requires a package
manager and a supported repository. A credible hub to upload source codes as
compressed archives or as pre-compiled binary packages was therefore
indispensable. The use of repositories started with the conception of CPAN and
CRAN repositories for the languages Perl and R in 1993 --- based on the
Comprehensive TeX Archive Network (CTAN) model for \TeX\ packages.
Today, \href{https://pypi.org}{Python Package Index (PyPI)} and
\href{https://anaconda.org}{Anaconda Cloud} are major repositories for
delivering and downloading Python packages, assisted by package managers
\pack{pip} and \pack{conda} respectively.
% Almost all major
% languages\footnote{\href{https://stackoverflow.com/questions/1693529/%
% list-of-top-repositories-by-programming-language}{https://stackoverflow.com/%
% questions/1693529/list-of-top-repositories-by-programming-language}}
% have similar repositories, with exceptions such as Fortran and Matlab.
Specialized package managers have also emerged to install scientific
libraries and compilers such as \pack{easybuild} and \pack{spack}.
\subparagraph{Continuous integration (CI).} The need for strong reliability of
programs with increasing complexity has forced developers to use continuous
integration.
%
In CI, firstly the code is \emph{built} against a known, repeatable emulation
environment on a server. Thereafter, a set of \emph{unit tests} are run for each
commit (modification of the code) to check that no bugs have been introduced. It
is also important to analyse what portion of the source code is covered by these
unit tests --- a method referred to as \emph{code coverage}. In FluidDyn project
we stop at this level, but it is possible to further enhance it by adding code
style checks by running linters alongside unit tests; and also to add
\emph{continuous delivery} of documentation and package releases.
Almost all CI platforms are free for open-source projects --- Travis CI,
CircleCI and Appveyor % (sic) av: I have double checked the official names
to name a few. Bitbucket and GitLab have built-in CI
implementations. Websites such as Codecov and Coveralls help to chart and
analyse the code coverage history, viewable by anyone.
When the code depends on a wide variety of external dependencies, CI is useful
to avoid \href{https://en.wikipedia.org/wiki/Software\_rot}{software rot},
i.e.\ to avoid becoming obsolete with newer versions of the compiler or its
dependencies. CI is specifically important for dynamical
languages and open-source code since it allows open-source projects to welcome
participation from many people while maintaining a high degree of reliability.
\subparagraph{Websites to share knowledge.} Mutual aid and knowledge sharing
have been important aspects of the web since its inception and these practices
are now of foremost importance for many human activities.
%
In particular, learning programming and computer science is now greatly aided
by websites like the community driven encyclopedia
\href{https://www.wikipedia.org/}{Wikipedia} or the community driven forum
\href{https://stackoverflow.com/}{Stack Overflow}, where people write thousands
of questions and answers on programming each week.
% todo: remark from Julien Salort: it's old.
% av: So are technologies like git and hg, but we are putting together what
% characterizes our workflow, aren't we? We can emphasize that these forums
% often work only when there is a "thriving community" like the Python
% userbase.
IRC protocol has continued to sustain the test of time. Instant messaging
channels such as \codeinline{\#python} on Freenode facilitate interactive
discussions and a form of community-level support system --- a role which used
to be filled by mailing lists alone in the past. Alternatives such as Riot
(Matrix protocol), Gitter, and Slack have gained traction offering different
integrations solutions on top of instant messaging.
%
Through instant messaging a geographically separated team, as is the case with
FluidDyn project, can achieve near real-time collaboration without the hassle
of setting up video conferences or meetings.
\subparagraph{Automatic web documentations built on servers.} Documentation
generators such as Sphinx (for Python) and Doxygen (for many languages
including C, C++, Fortran and Python) ``automagically'' convert in-line
comments into documentations as PDF, HTML and more. Read the Docs has in the
recent years become the main host for building and publishing documentation on
servers as web pages, thus easing the process for developers and considerably
improving access to information.
% We need transitions and some words at the end of the subsections to help the
% readers to understand why we present what we present and what is the link
% with the FluidDyn project...
We have presented new methods and tools used in software development. Taken
all together, they greatly improve our efficiency for collective work and open
possibilities of interactions that were unthinkable only few years ago.
%
For example, the success of community-driven software like \pack{astropy} (a
subject-area research library with \href{https://github.com/astropy/astropy}{more
than 240 contributors}) and \pack{scikit-learn} (a machine learning toolkit with
\href{https://github.com/scikit-learn/scikit-learn}{more than 1000 contributors})
would not have been possible without these new software development methods.
%
% pa: done: we need to mention in this paragraph existing community-driven
% subject-area research libraries (see the file
% ./communitydriven_research_soft.md).
%
As a scientists, we can wonder how to fully exploit these new possibilities to do
science. The project FluidDyn is an attempt to help the fluid dynamics community
to advance in this direction.
\subsection*{Python, a programming language adapted for open-science}
We now turn to the presentation of the
\href{https://www.python.org/}{programming language Python}, another
fundamental tool for the FluidDyn project. Note that, this is not a detailed
technical presentation: we focus on the ideas necessary to understand why the
FluidDyn project is based on this language and on facts interesting for
potential FluidDyn users and developers.
Python community has thrived remarkably in the last few years, evident from the
fact that by the end of 2017, Python became the second most popular language in
terms of pull-requests in GitHub\footnote{See GitHub's annual report titled
\href{https://octoverse.github.com/}{ ``The State of the Octoverse 2017''}.} and
questions on StackOverflow\footnote{See
\href{https://stackoverflow.blog/2017/09/06/incredible-growth-python/}{ ``The
Incredible Growth of Python''}, a blog post by Stack Overflow.}.
%
TIOBE-index\footnote{\url{https://www.tiobe.com/tiobe-index}}, which assess the
popularity of a language based on number of skilled users, courses and
third-party vendors, ranks Python at number four. IEEE Spectrum\footnote{See
the analysis titled \href{https://spectrum.ieee.org/computing/software/%
the-2017-top-programming-languages}{``The 2017 Top Programming Languages''} on
IEEE spectrum.} ranks Python as the most popular language in 2017 with a metric
based on trends, demand by employers and users on open-source hubs. All the
rankings do agree that Python usage has been steadily growing and is here to
stay. Python largely outclasses languages like Fortran and Matlab according to
these statistics, and is now clearly one of the mainstream languages with C,
Java, C++ and Javascript.
%
Let us summarize some characteristics of the Python languages that have lead to
such incredible success.
% todo: DONE
% We need an introductory paragraph on how big is Python today
% + the fact that several scientists miss this fact.
% See the file ./python-one-main-languages.rst
The Python language has been designed to boost the communication of technical
ideas between humans. The resulting code is as simple as possible due to the
inherent emphasis on readability. Humans can focus on the ideas. Python differs
from other mainstream languages for a variety of reasons:
\begin{itemize}
\item \textbf{Aesthetics.} The syntax is nice and elegant. Blocks of code are
defined with the indentation.
\item \textbf{Code style.} There are explicit guidelines
(\href{https://www.python.org/dev/peps/pep-0008/}{PEP~8}) supporting the
regularity (and thereby readability) of the code and tools
(linters) to evaluate adherence to the style.
\item \textbf{Dynamic typing.} The variables (the ``names'' in the code) are
not attached for life to an object in memory\footnote{Opposite behavior
compared to static languages as Fortran, C and C++.}. The types of the objects
are inferred at run time from the code and the context, and in many situations,
one does not need to declare them explicitly.
%
% Being a strongly typed language\footnote{In contrast to weakly typed languages,
% such as JavaScript}, types of the objects are respected and implicit type
% conversions are rarely performed.
% av: "rarely / never"?
% eg: 'hello' * 2 -> 'hellohello'; 1/3 -> 0.3333333
%
% pa: I think we can not cover such complicate notion in this document... It is
% anyway not so important for our demonstration.
\item \textbf{Automatic memory management.} The user need not handle the
memory allocation and deallocation by hand. There are, of course, ways to
control the memory usage of a program, though not as precisely as for low-level
languages.
\item \textbf{Interpreted.} The standard way\footnote{There are also tools to
compile Python code (for example Cython and Pythran).} to run a Python code is
to ``interpret'' it. A program called the interpreter executes the code
nearly\footnote{Actually, the execution of Python code is less simple and
involves an Abstract Syntax Tree.} instruction-by-instruction, with very few
optimizations.
%
In contrast to other compiled languages such as Fortran, C or C++, there is no
proper compilation step: the code is not translated to optimized machine
instructions, but only to Python bytecode.
%
We will discuss shortly the disadvantages of this characteristic. Two evident
advantages are rapid development cycle and the possibility of interactive
workflow (usually with IPython or Jupyter notebooks).
Note that there are many implementations of Python interpreters\footnote{We can
cite CPython (written in C), Jython (Java), IronPython (C\#), PyPy (Rpython, a
subset of Python) and MicroPython/CircuitPython (C, targeted to
micro-controllers).}. The default and most widely used implementation is written
in C and is called CPython.
\item \textbf{Simple, yet powerful.}
\begin{itemize}
\item Python supports multiple programming paradigms, including imperative,
object-oriented and functional.
\item It is easy to interface with code written in other languages (in particular
C, C++ and Fortran).
\item Python comes with a large and high quality \href{https://docs.python.org%
/3/library/index.html}{standard library}.
\item Python can be deployed on many different machines with different operating
systems (Linux, Windows, macOS, Android) and architectures (from a microcontroller
--- with MicroPython or CircuitPython --- to a Blue Gene supercomputer).
\end{itemize}
\end{itemize}
Thus, Python is one of the best ``glue'' languages for fast prototyping.
Developing with Python is much faster and easier than with many other
languages. Less bugs are introduced by the developers simply because there are
much less lines of code \cite{nanz_comparative_2015}.
%
Maintaining and modifying existing code is also eased by the very good code
readability associated with the language.
Another consequence of the apparent simplicity of Python is that the learning
curve has a nice shape. Learning Python is very easy at the beginning and the
complexity gradually increases as the level of the developer increases.
%
Therefore, Python is good for developers of all levels: very gentle for
beginners and very powerful for advanced users.
Python is an old language (first implementation in 1991!) but it continues to
evolve.
\begin{itemize}
\item A hard and controversial decision was taken to clean up Python from its
historical incoherences by having a new series of backwards-incompatible
versions of the language, Python 3. Python 3.0 was released in December 2008.
After a long and difficult process, the transition from Python 2 to Python 3 is
now (2018) behind us for nearly all important packages\footnote{Close to
\href{https://python3wos.appspot.com/}{95\% of the most downloaded packages}
are Python 3 compatible now.}. Especially for scientific applications, we can
now work only in Python 3. For instance, \href{https://github.com/%
numpy/numpy/blob/master/doc/neps/dropping-python2.7-proposal.rst}{future
releases of \Numpy} and of \href{http://www.python3statement.org/}{the main
scientific packages} will soon drop compatibility with Python 2. These changes
open doors to very clean, coherent and potentially faster Python
experience. Therefore, one should not use Python 2 for science anymore and
instead, adopt the newer versions ($>=3.5$ in 2018).
\item New versions of Python have nice new features to adapt to new usages and to
recent trends in computer science. For example, three new features were
introduced in Python 3.5 (first released on September 2015): the \codeinline{@}
operator for matrix multiplication, the new \codeinline{async} and
\codeinline{await} keywords for concurrency and type hinting\footnote{See also our
paragraph on the issue of type checking in the following.} (with the module
\codeinline{typing} and an associated syntax presented in the
\href{https://www.python.org/dev/peps/pep-0484/}{PEP~484}).
\end{itemize}
\subsubsection*{An incredible success}
Being a very versatile language, Python is widely used for many different
applications:
\begin{itemize}
\item Simple scripting.
\item System, database and network administration.
\item GNU/Linux distribution software.
\item Python is now one of the main scripting languages to add programmability
to applications (for example Paraview, Visit, QGIS, Blender, ...).
\item Web servers.
\item Web scraping and data analysis.
\item Animation movies, game development and gaming.
\item Education. Python has recently become one of the most popular language
for teaching programming and computer science.
\item Science!
Python is now widely used in scientific applications. It has now a mature and
powerful scientific ecosystem with well-established based packages (\Numpy for
N-dimensional homogeneous arrays, \Scipy as the fundamental toolkit for
scientific computing, \pack{Matplotlib} for plotting and \pack{Pandas} for data
structures) and several more specialized packages (to name a few, \pack{h5py},
\pack{mpi4py}, \pack{skimage}, \pack{sklearn}, ...). There are now great (and
often shiny) tools for most of the applications. For example, the
\href{https://github.com/rougier/python-visualization-landscape}{%
landscape of visualization tools usable through Python} is impressive.
%
Python is one of the main languages for data science with packages such as
\pack{Pandas}, \pack{statmodels}, \pack{sklearn}, \pack{Keras}, \pack{TensorFlow}
and \pack{PyTorch}.
There are now ready-to-use ``batteries-included'' Python installers
(similar to Matlab), like in particular the
\href{https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)}{%
freemium open-source distribution Anaconda}, so it is very easy to start using
Python for scientific purposes.
There are now nice Python integrated development environments (IDE) adapted for
scientists, for example \href{https://github.com/spyder-ide/spyder}{Spyder},
\href{http://www.pyzo.org/}{Pyzo} and
\href{https://jupyterlab.readthedocs.io}{Jupyterlab}.
\end{itemize}
Python has a very large and supportive community (see for example
the \href{http://stackoverflow.com/tags}{Stack Overflow tags}).
%
But Python is also supported by companies using Python and the open-source
dynamics. Most of these companies that support Python are specialized in
web technologies and data-driven research. But, recently, companies basing
their business
% av: earlier "computer and web technologies". Computer is quite obvious, isn't
% it?
on Python for science have started to become quite powerful in terms of
development. For example Anaconda Inc.\ is strongly implied in the development of
tools like \pack{Numba} and \pack{dask}. Hardware giants such as Intel and Nvidia
also contribute to the Python scientific ecosystem.
\subsubsection*{Some Python issues}
\subparagraph{Since the standard Python interpreter does not do any proper
compilation,} Python code in some CPU bounded cases can be too slow. Thus, one
has to use special tools to get optimized machine instructions to speedup
performance-critical code.
\textbf{Ahead Of Time compilation.} Since Python is primarily a ``glue''
language, a common solution is to use code written in C, C++ or Fortran.
%
% todo: DONE
% add a reference to ctypes and cffi to call C with pure Python code
%
It is possible, without a compilation step, to call functions in C libraries
through pure python code using \pack{ctypes} standard module or the \pack{cffi}
package.
%
Libraries with Python bindings can also be used directly (for e.g., wrapped
using SWIG or Boost.Python) or one has to write extensions, i.e.\ Python
modules written in C or C++ using the CPython API that can be imported directly
from Python like any Python module.
%
However, to rely on such techniques to call C libraries would inhibit
development of extending functionalities, as they have a steeper learning
curve compared to Python.
%
Nevertheless, this strategy has given rise to the base modules of the scientific
Python stack, namely \Numpy, \Scipy and \pack{Matplotlib}.
Python developers do not actually have to write the compiled extensions in C,
C++ or Fortran, since there are tools to generate them automatically from
Python code (Pythran) or from Cython code. Cython \citep{behnel_cython2011} is a
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
smooth blend of two languages with possibility to add type declarations like in
C/C++, but with a syntax similar to Python.
Pythran \citep{guelton2015pythran, guelton2018pythran} is a quite recent Python
compiler which gives impressive results. It creates compiled extensions from pure
Python code with simple type annotations written as comments.
%
The resulting extensions are usually as fast as Fortran or C++ written by
non-specialists \cite[see, for example, benchmarks for the packages
\fluidpack{fft} and \fluidpack{sim},][]{fluidfft, fluidsim}.
%
They are created with a very interesting two-step compilations: first the code
is optimized at the Python level and then an automatically produced C++ code is
properly compiled.
%
This two-step compilation provides room for very clever optimizations as
demonstrated, for example, in
\href{http://serge-sans-paille.github.io/pythran-stories/%
being-more-than-a-translator.html}{%
this blog post by the main developer of Pythran}.
%
Pythran supports OpenMP pragma \citep{guelton2013compiling} and can use modern
vectorization with SIMD instructions \citep{guelton2014exploring}. Pythran
understands both Matlab-like vectorized code and C-like code with explicit loops
\citep{guelton2018pythran}.
We present in the two companion papers \citep{fluidfft, fluidsim} examples of
highly efficient Python codes, optimized using a combination of Cython and
Pythran extensions.
\textbf{Just In Time (JIT) compilation.} Another way to obtain optimized
machine instructions is by using JIT compilation, i.e.\ by compiling only the
critical code at run time.
%
This strategy can yield good results with other languages as for example Matlab
or Julia.
%
Pypy, an alternative interpreter written in Python, has a JIT compiler. However,
Pypy is not widely used for scientific applications mainly because of
compatibility problems with the extensions written for CPython.
%
However this could change since a recent version of Pypy (6.0) supports the main
packages of the Python scientific stack (\Numpy, \Scipy, \pack{Matplotlib},
\pack{Pandas}, etc.).
Adding a JIT to the interpreter CPython has been notoriously difficult (see
\href{https://faster-cpython.readthedocs.io/}{faster-cpython}). One of the
problem seems to be the C API provided in CPython and used by many extensions.
%
Python being first a glue language, a faster CPython with a JIT is actually not
so necessary for many purposes. Therefore, not as much money and work has been
put in accelerating Python than for example for Java and JavaScript.
Another strategy is to add a JIT to CPython through an external package
(\pack{Numba}) and to only compile the critical functions. Moreover,
\pack{Numba} is particularly interesting because it can take advantage of the
GPU\footnote{See for example
\url{https://devblogs.nvidia.com/parallelforall/seven-things-numba/}}.
\subparagraph{No type-checking.} In contrast to languages using Ahead Of Time
compilation, there is no type-checking in Python (and more generally in many
dynamical languages). Type-checking is useful since it can avoid simple bugs.
However, it is not a good practice even with static languages to rely only on
type-checking to look for bugs in a code.
%
It is known that a rigorous set of unit tests is also needed to obtain more
reliable codes.
%
Dynamical languages rely a lot on unit tests so it is very important to put at
least a little bit of time and energy to write a decent battery of tests.
%
We have to mention the young and promising project
\href{http://mypy-lang.org/}{Mypy}\footnote{Guido van Rossum, the author and
``Benevolent Dictator For Life'' of Python, is involved in the project Mypy.},
which allows for Optional Static Typing and thus type-checking for
Python\footnote{We plan to add type hitting in the most important FluidDyn modules
to investigate how we can take advantage of type-checking with Mypy.}.
\subparagraph{A lively, huge and thus complicated ecosystem.} As already
mentioned, Python is a versatile language. For many applications, the standard
library is insufficient and one needs external packages.
%
% For most applications there are several projects usable through Python.
Most of the time there are several Python packages for a certain application.
A user/developer has to make choices regarding the tools to use. Especially
for the beginners who are not used to the open-source realm, it can be
difficult to understand a ``landscape of projects'' for a particular
application and to make good technological choices.
It is important to get a good introduction on how to work with the open-source
dynamics and the scientific Python ecosystem. It can also be very useful to ask
to more advanced users and attend Python courses or workshops\footnote{This could
be regular academic courses, MOOCs or workshops like ones organized by
\href{https://software-carpentry.org/}{Software Carpentry}.}.
%
Of course, such abundance of projects can also be seen as an advantage.
\subparagraph{Absence of multicore computational parallelism using threads.}
Threads behave like light subprograms. In principle, they can use the different
cores of the CPU at the same time but the CPython interpreter forbids that they
interpret python code at the same time\footnote{See
\href{https://opensource.com/article/17/4/grok-gil}{this blog article on ``How
to grok the GIL''} and
\href{https://faster-cpython.readthedocs.io/cpython37.html}{this technical
presentation by a CPython core developer}.}.
%
This limitation comes from an important detail in the implementation of the
CPython interpreter: a Global Interpreter Lock (GIL) to prevent race conditions
that could corrupt data.
%
It seems that the GIL greatly simplies the implementation of CPython and that it
is very difficult to remove it while keeping other nice technical properties of
CPython intact\footnote{See \url{https://wiki.python.org/moin/GlobalInterpreterLock}.}.
Note that Pypy also uses a GIL, while two other implementations of the Python
language, Jython and IronPython, do not have this limitation.
At any rate, threads using the interpreter do not use the CPU at the same time
so it is not possible to use threads to do multicore computational parallelism
with pure python code to accelerate CPU bounded tasks.
%
Note however, that threads can be (and are widely) used in Python for
concurrency, i.e.\ to perform I/O tasks (which does not need CPU) concurrently.
%
To do proper computational parallelism for CPU bounded tasks with the GIL, one has
to use other strategies: extensions (fine grain parallelism), multiprocessing
(coarse grain parallelism) and inter-process communication, for example, with MPI
(with mpi4py) or ZeroMQ.
\subsubsection*{Rich landscape of open-source languages useful for science}
So far, we have discussed in detail about Python. However, there are many
interesting open-source languages that can be used for scientific applications.
All have strengths and weaknesses.
Fortran remains widely used for computing codes, at least in fluid mechanics.
C++ is used in many recent developments. Its recent improvements (C++11, C++14)
make it a modern and very useful language.
%
JavaScript, Java, Scala, Smalltalk, Haskell, R, Julia, Perl and Lua are used for
some scientific codes. Go and Rust are quite young languages and are not (yet)
widely used for scientific applications.
It is of course very difficult to predict which technologies will be widely
used in even the near future (10 or 20 years for example).
%
From what we see, it seems that the idea of ``one language to do everything for
science'' will not succeed, at least soon. This was maybe the project of Julia
creators, but despite the qualities of this language, it also has
limitations\footnote{For example C-like include instead of Python-like
import. From our point of view, a Pythonic flavor of Julia would be very
interesting.}.
%
It seems that a very important aspect in the future will be interoperability
between coexisting tools\footnote{See for example the cross-language development
platform \href{https://arrow.apache.org/}{Apache Arrow}.}.
So what about Python? As already mentioned, it is today the language with the
strongest dynamics in science, data analysis and much more. Several students
learn and will learn it. Many scientists in the future would, hopefully, be
able to appreciate these strengths and favour the Pythonic approaches. With
such dynamics today, we can anticipate that the Python world will be able to
embrace new technological trends and to overcome some of its current
limitations.
%
It is reasonable to predict that Python will continue to shine as a language
(i) particularly good for human communication and to write ideas, (ii) for fast
prototyping and (iii) for interoperability with code written in other languages
(see for example the recent projects \href{https://cppyy.readthedocs.io}{cppyy}
and \href{https://github.com/pybind/pybind11/}{pybind11}, or
\href{https://github.com/PyO3}{the possibility to interact with Rust code}).
\subsubsection*{Summary: Python, a versatile Swiss Army Knife for the
scientists}
Python has become a great versatile tool for scientists. The scientific
ecosystem of Python is nowadays really impressive. The scientific Python
community is strong and quickly growing.
%
A scientist with good skills in Python can do most of what she/he needs to do in
programming (which does not mean that Python is the best language to do
anything!).
%
In contrast, no skill in Python is a real disadvantage for many tasks and for
employability.
%
Therefore, in most fields today, if a scientist has to acquire good skills in
one language, it is in Python.
Considering what we have presented on Python, we think that it is reasonable to
bet that it will become one of the \textit{lingua franca} in science and in
fluid dynamics. This explains why we have chosen it to be the main language
for the project FluidDyn.
\subsection*{Productivity at individual, group and community levels}
% remark Julien Salort "It is well known that": bad
% pa: common sense says ?
% It is well known
It is not surprising
% av: "It is not surprising"
%
that short-term efficiency and long-term efficiency are
sometimes incompatible.
%
Quick and dirty scripts can be efficient in the short term, but such a workflow
has a negative impact on a longer time scale.
Similarly, we can also differentiate productivity at different scales of
collaboration.
%
An individual can be very efficient with a particular tool, but the same tool
can be very inefficient to collaborate even with co-workers.
%
A group can be very efficient with a code which is closed-source. If this
strategy can be efficient for the group (at least in the short term), it leads
to waste for the community. Other groups will need to develop codes with the
same features. Good ideas will be spread in different codes and since the
number of users and developers are smaller for each of these codes, the code
quality will not improve as fast as it would with an open-source strategy. In a
nutshell, as this proverb says, \textit{``If you want to go fast, go alone. But
if you want to go far, go together.''}
Thus, a community using open-source methods can be efficient to co-develop its
tools. However, some of its members have to spend more energy to bear this
dynamics, which implies that they somehow sacrifice their individual efficiency.
We see that short-term efficiency, long-term efficiency and productivity at
different levels can often be incompatible and that we have to consider these
conflicts when choosing between different technologies.
\subsection*{Programming in the field of fluid mechanics}
We study fluid mechanics via laboratory experiments, in situ measurements,
analytic computation, numerical simulations and data processing.
%
Nowadays, programming is involved in all methods, and thus it is difficult to do
fluid mechanics without software development.
%
However, the level in software engineering is on average very low in the
community.
%
Even today, it is quite commonplace to start a PhD without any serious training in
GNU/Linux and modern programming tools.
%
A large majority of scientists and technicians are also unaware of the new
challenges and opportunities of open-source.
Since fluid dynamics is an engineering science, closed-source commercial software
have historically been dominant in the field.
%
Few people are aware of the vicious circle of the closed-source model for people
and groups relying on it.
%
This mechanism is nevertheless very damaging. In the closed-source model, a group
pays for a license or a new development. The group does not learn how to develop
what has been paid for. The company, on contrary, stands to gain from profits and
the feedback generated from users, through which it improves the product sold.
The group produces codes, books, courses using the closed-source product or
acquire knowledge on how to use the closed-source product, so that the group is
increasingly dependent on the product and is ready to pay more for it.
%
Such vicious circles are difficult to break, but not impossible. This can be
done by introducing alternative open-source solutions, bringing in positivity to
this circle.
We have seen that it was very common to rely on commercial companies and
closed-source software for most of the new developments in fluid mechanics.
%
Nevertheless there are also a lot of codes produced by scientists in the
field. But without the technical knowledge on how to work collectively on code,
this production is often of bad quality (compared to the information technology
industry standard) and it is basically doomed to be abandoned after its use.
%
However, it is natural to try to reuse code or at least to build the next steps
on what has been done before. Thousands of hours of highly qualified people
are spent in trying to understand and reuse codes badly written with
inappropriate languages!
On a technical point of view, the majority of the coding in the field involve a
mix of Fortran/C or C++, shell languages (as Bash) and Matlab. For experiments,
the graphical programming environment
\href{http://www.ni.com/en-us/shop/labview.html}{Labview} is dominant for
control of physical objects and acquisition and Matlab is used for data
processing.
%
Often, languages are used for things for which they are not adapted.
%
Fortran, C or C++ have not been designed for fast prototyping of complex
programs.
%
It is known that for scientific purposes, one should restrict shell scripting
to extremely simple tasks.
%
Similarly, using Matlab for developing complex programs is technically a
questionable choice.
%
These technical solutions also represent a problem for code reuse, sharing and
collaborative development.
%
We are now going to present factual arguments explaining why we think that the