fluidfft issueshttps://foss.heptapod.net/fluiddyn/fluidfft/-/issues2024-02-11T21:04:44Zhttps://foss.heptapod.net/fluiddyn/fluidfft/-/issues/41Release plugins on PyPI (manual) + conda-forge when needed2024-02-11T21:04:44ZPierre AugierRelease plugins on PyPI (manual) + conda-forge when neededFluidfft 0.4.0https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/31Adding discrete cosine and sine transforms2021-02-10T08:10:58ZAshwin V. MohananAdding discrete cosine and sine transformsThere are possible applications of adding pseudospectral solvers with boundary conditions and to use it for postprocessing non-periodic fields. We could start by using mpi4py-fft's API which spectralDNS uses. Thoughts?There are possible applications of adding pseudospectral solvers with boundary conditions and to use it for postprocessing non-periodic fields. We could start by using mpi4py-fft's API which spectralDNS uses. Thoughts?https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/30Differentiate ifft and ifft_as_destroy and use of const2020-12-22T09:40:09ZAshwin V. MohananDifferentiate ifft and ifft_as_destroy and use of constTwo possibilities:
- [x] Better docstrings. Right now the docstrings are the same for both methods.
- [ ] In C++ level, perhaps we could use `const mycomplex *fieldK` for the first argument of `void FFT3DWithFFTW3D::ifft(mycomplex *fiel...Two possibilities:
- [x] Better docstrings. Right now the docstrings are the same for both methods.
- [ ] In C++ level, perhaps we could use `const mycomplex *fieldK` for the first argument of `void FFT3DWithFFTW3D::ifft(mycomplex *fieldK, myreal *fieldX)` to guarantee that the array of the first argument is not modified. Also add `const` elsewhere if the array is meant to be read-only?Ashwin V. MohananAshwin V. Mohananhttps://foss.heptapod.net/fluiddyn/fluidfft/-/issues/28Fix behavior with FLUIDFFT_TRANSONIC_BACKEND + allow things like "pythran+num...2024-01-29T08:28:19ZPierre AugierFix behavior with FLUIDFFT_TRANSONIC_BACKEND + allow things like "pythran+numba+cython"We now have `FLUIDFFT_TRANSONIC_BACKEND` for build time. But I don't think it really works :-)We now have `FLUIDFFT_TRANSONIC_BACKEND` for build time. But I don't think it really works :-)Pierre AugierPierre Augierhttps://foss.heptapod.net/fluiddyn/fluidfft/-/issues/27Environment variables to control -march=native and -DUSE_XSIMD options2020-10-10T14:22:58ZPierre AugierEnvironment variables to control -march=native and -DUSE_XSIMD optionsIt should be used in `setup_make.py` (near the end) but defined in `setup_configure.py` and mentioned in `doc/install.rst`.
It is important to be able to distribute wheels.It should be used in `setup_make.py` (near the end) but defined in `setup_configure.py` and mentioned in `doc/install.rst`.
It is important to be able to distribute wheels.https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/25Compilation errors with single precision enabled2020-02-15T21:20:41ZBitbucket ImporterCompilation errors with single precision enabled*Created originally on Bitbucket by [ofmla (oscar mojica)](https://bitbucket.org/%7B3db9ab34-466a-4ff3-8bb4-dc830cdd96b6%7D/)*
I am getting errors like this one
```
fft3d_with_fftw3d.cpp:49:10: error: cannot convert ‘float ()[2...*Created originally on Bitbucket by [ofmla (oscar mojica)](https://bitbucket.org/%7B3db9ab34-466a-4ff3-8bb4-dc830cdd96b6%7D/)*
I am getting errors like this one
```
fft3d_with_fftw3d.cpp:49:10: error: cannot convert ‘float ()[2]’ to ‘mycomplex {aka std::complex<float>*}’ in assignmentarrayK = fftwf_alloc_complex(nK0 * nK1 * nK2);
```
when compile the test\_bench\_cpp code using SINGLE\_PREC:=yes in the Makefile. You also should use reintepret\_cast for single precision case
https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/23Multiple dispatch to use Dask (dask.array and dask.fft) and CUDA (cupy arrays...2021-04-14T08:27:34ZBitbucket ImporterMultiple dispatch to use Dask (dask.array and dask.fft) and CUDA (cupy arrays and cuFFT).*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
For postprocessing one generally loads data sequentially on smaller machines, and in such scenarios big simul...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
For postprocessing one generally loads data sequentially on smaller machines, and in such scenarios big simulations would not fit in the memory. It would be nice to have the Operator class to support out-of-core fft using dask arrays for KX, KY etc and `dask.array.fft` either directly or via pyfftw's interface:
* http://docs.dask.org/en/latest/array-api.html?highlight=fft#fast-fourier-transforms
* https://pyfftw.readthedocs.io/en/latest/source/pyfftw/interfaces/dask_fft.htmlhttps://foss.heptapod.net/fluiddyn/fluidfft/-/issues/22Optimization taking into account dealiasing2020-02-12T23:00:03ZBitbucket ImporterOptimization taking into account dealiasing*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
As pointed out by a referee, for some classes (in particular fft3d.mpi_with_fftw1d and fft2d.mpi_with_fftw...*Created originally on Bitbucket by [paugier (Pierre Augier)](https://bitbucket.org/%7Beb327c54-cadd-4fce-9edd-f8fe047a9b93%7D/)*
As pointed out by a referee, for some classes (in particular fft3d.mpi_with_fftw1d and fft2d.mpi_with_fftw1d), it is possible to avoid a significant part of the computation by taking into account dealiasing.
I think these classes have to be sensible to the environment variable `FLUIDFFT_DEALIASING_COEF` (the value has to be checked during the instantiation). The content of this variable can be a simple expression like `2/3` or a number between 0 and 1. For `dealiasing_coef > 1`, we retrieve the standard behavior.
All modes such as `k_i >= dealiasing_coef * max(k_i)` have to be zeros (so the corresponding FFT don't need to be computed and no need for MPI messages).
I think this problem is a good challenge for @CyrilleBonamy ! Can you please work on this @CyrilleBonamy? I think Patrick should also know such kinds of optimization...https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/21Dealiasing optimization to have zero-padded spectral arrays2020-02-12T23:00:02ZBitbucket ImporterDealiasing optimization to have zero-padded spectral arrays*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
To quote a reviewer's suggestion:
> De-aliasing requires to take into account more degrees of freedom in...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
To quote a reviewer's suggestion:
> De-aliasing requires to take into account more degrees of freedom in the spatial fields than in the spectral ones.
> In particular, for 3D fields, de-aliasing implies that many zeros are present in the fields, and actually do not need to be transformed, potentially saving significant computation time.
> As far as I know, FFTW always transforms full fields.
> Does fluidFFT allows for this optimization even if the underlying library does not? Is it relevant?https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/17Some corrections to allow cuFFT backend to atleast install.2020-02-12T22:59:57ZBitbucket ImporterSome corrections to allow cuFFT backend to atleast install.*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
With the old implementation we relied on FFTW, which had an issue of https://github.com/FFTW/fftw3/issues/18....*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
With the old implementation we relied on FFTW, which had an issue of https://github.com/FFTW/fftw3/issues/18. This was circumvented by using `nvcc -ccbin=mpiicpc` such that nvcc does not "pretend" to be a gcc compiler.
It installs like that, but the installed extension cannot be imported.
The second modification was to use the libcufftw.so dynamic library instead of libfftw3.so.
At the moment this is a hack, nothing clean.
EDIT: Simpler diff
## Changes in `src_cpp`
```diff
diff -r bcda25348f32 setup.py
--- a/setup.py Fri Nov 23 15:10:14 2018 +0100
+++ b/setup.py Tue Nov 27 18:00:45 2018 +0100
@@ -142,7 +142,8 @@
elif "p3dfft" in base_name:
libraries.append("p3dfft")
elif "cufft" in base_name:
- libraries.extend(["cufft", "mpi_cxx"])
+ libraries.remove("fftw3")
+ libraries.extend(["cufft", "cufftw", "mpicxx"])
return Extension(
name="fluidfft.fft" + dim + "." + base_name,
diff -r bcda25348f32 src_cpp/2d/fft2d_with_cufft.h
--- a/src_cpp/2d/fft2d_with_cufft.h Fri Nov 23 15:10:14 2018 +0100
+++ b/src_cpp/2d/fft2d_with_cufft.h Tue Nov 27 18:00:45 2018 +0100
@@ -2,6 +2,7 @@
/* #include <stdio.h> */
#include <cuda_runtime.h>
#include <cufft.h>
+#include <cufftw.h>
#include <base_fft2d.h>
// #include <helper_functions.h>
// #include <helper_cuda.h>
diff -r bcda25348f32 src_cpp/3d/fft3d_with_cufft.h
--- a/src_cpp/3d/fft3d_with_cufft.h Fri Nov 23 15:10:14 2018 +0100
+++ b/src_cpp/3d/fft3d_with_cufft.h Tue Nov 27 18:00:45 2018 +0100
@@ -2,6 +2,7 @@
#include <stdio.h>
#include <cuda_runtime.h>
#include <cufft.h>
+#include <cufftw.h>
#include <base_fft3d.h>
/* #include <helper_functions.h> */
/* #include <helper_cuda.h> */
diff -r bcda25348f32 src_cpp/base/base_fft.h
--- a/src_cpp/base/base_fft.h Fri Nov 23 15:10:14 2018 +0100
+++ b/src_cpp/base/base_fft.h Tue Nov 27 18:00:45 2018 +0100
@@ -16,9 +16,8 @@
#include <complex>
using std::complex;
-#include <fftw3.h>
-
-
+// #include <fftw3.h>
+#include <cufftw.h>
using namespace std;
```
and a `site.cfg` like
```ini
[cufft]
use = True
dir =
[environ]
## To modify compiler used to build Cython extensions:
MPICXX = mpiicpc
NVCC = nvcc -ccbin=mpiicc
```
It installs and runs, but also fails energy conservation tests.
```
$ make tests
...
--------
N0 = 24 ; N1 = 48
Initialization (FFT2DWithCUFFT) done in 0.711113 s
tests (FFT2DWithCUFFT)...
Warning: tests not implemented for this class FFT2DWithCUFFT
fail: (energy_X_before - energy_K_before)/energy_X_before = inf > EPS
energy_X_before = 1.693239e-01
fail: (mean_X_before - mean_K_before)/mean_X_before = 1.000000e+00 > EPS
F...
--------
N0 = 16 ; N1 = 24
Initialization (FFT2DWithCUFFT) done in 0.000007 s
.
--------
N0 = 13 ; N1 = 11
Initialization (FFT2DWithCUFFT) done in 0.000004 s
.......
--------
N0 = 24 ; N1 = 24
Initialization (FFT2DWithCUFFT) done in 0.000007 s
tests (FFT2DWithCUFFT)...
Warning: tests not implemented for this class FFT2DWithCUFFT
fail: (energy_X_before - energy_K_before)/energy_X_before = 9.999998e-01 > EPS
energy_X_before = 1.635966e-01
fail: (mean_X_before - mean_K_before)/mean_X_before = 1.000000e+00 > EPS
bench from cpp...
time fft (FFT2DWithCUFFT): 0.000000 s
time ifft (FFT2DWithCUFFT): 0.000001 s
bench from cpp...
time fft (FFT2DWithCUFFT): 0.000000 s
time ifft (FFT2DWithCUFFT): 0.000000 s
....
======================================================================
FAIL: test_fft2d.with_cufft (fluidfft.fft2d.tests.Tests2D)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta-cuda/fluidfft/fluidfft/fft2d/tests.py", line 53, in test
self.assertAlmostEqual(EX, EK)
AssertionError: inf != 0.0 within 7 places (inf difference)
----------------------------------------------------------------------
Ran 16 tests in 1.070s
```
What do you think?https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/16build fail for the last version of fluidfft on SGI/Intel compiler/MPT mpi2020-02-12T22:59:53ZBitbucket Importerbuild fail for the last version of fluidfft on SGI/Intel compiler/MPT mpi*Created originally on Bitbucket by [slgentil (Sylvie.Le.Gentil@ifremer.fr)](https://bitbucket.org/%7B20458af5-59d3-4f03-8f6d-92b2b35ae3fb%7D/)*
problem for compiling last version of fluidfft on our SGI HPC cluster. Intel compiler used ...*Created originally on Bitbucket by [slgentil (Sylvie.Le.Gentil@ifremer.fr)](https://bitbucket.org/%7B20458af5-59d3-4f03-8f6d-92b2b35ae3fb%7D/)*
problem for compiling last version of fluidfft on our SGI HPC cluster. Intel compiler used and MPT for mpi. (see install.txt)
When compiling fluidfft:
```
make develop
python setup.py develop
���������CMPT ERROR: mpiexec_mpt must be used to launch all MPI applications
Makefile:5: recipe for target 'develop' failed
make: *** [develop] Error 255
```https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/9Operators should handle zero-sized arrays2020-02-12T22:59:45ZBitbucket ImporterOperators should handle zero-sized arrays*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Certailn 2D and 3D FFT libraries are capable of operating with zero-sized arrays. Related operators functions...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Certailn 2D and 3D FFT libraries are capable of operating with zero-sized arrays. Related operators functions assume a non-zero size as of now.
Short term
-----------------
Raise NotImplementedError to avoid strange tracebacks.
Long term
----------------
Algorithm to work with zero sized arrays for various functions.
Examples of errors thrown:
1024^3 array with 1024 procs
```py
Traceback (most recent call last):
File "/var//opt/cray/alps/spool/13580068/fluidsim", line 11, in <module>
load_entry_point('fluidsim', 'console_scripts', 'fluidsim')()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/__main__.py", line 61, in run
args.func(args)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/profile.py", line 212, in run
it_end=args.it_end,
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/profile.py", line 160, in profile
_profile(type_fft)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/profile.py", line 124, in _profile
sim = Simul(params)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/base/solvers/base.py", line 115, in __init__
self.oper = Operators(params=params)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/operators/operators3d.py", line 105, in __init__
coef_dealiasing=params.oper.coef_dealiasing,
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidfft/fluidfft/fft3d/operators.py", line 226, in __init__
CONDKY = abs(self.Ky) >= self.coef_dealiasing * self.k1.max()
File "/cfs/klemming/nobackup/a/avmo/my-envs/fluidmeta/lib/python3.6/site-packages/numpy/core/_methods.py", line 26, in _amax
return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity
```
1024^2 array with 64 procs
```py
Traceback (most recent call last):
File "/var//opt/cray/alps/spool/13579943/fluidsim", line 11, in <module>
load_entry_point('fluidsim', 'console_scripts', 'fluidsim')()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/__main__.py", line 61, in run
args.func(args)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/profile.py", line 212, in run
it_end=args.it_end,
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/profile.py", line 160, in profile
_profile(type_fft)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/util/console/profile.py", line 124, in _profile
sim = Simul(params)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/base/solvers/base.py", line 156, in __init__
self.output.init_with_oper_and_state()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/base/output/base.py", line 480, in init_with_oper_and_state
super(OutputBasePseudoSpectral, self).init_with_oper_and_state()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/base/output/base.py", line 303, in init_with_oper_and_state
self.init_with_initialized_state()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/base/output/base.py", line 361, in init_with_initialized_state
self.print_stdout.complete_init_with_state()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/base/output/print_stdout.py", line 51, in complete_init_with_state
self.energy0 = self.output.compute_energy()
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidsim/fluidsim/solvers/ns2d/output/__init__.py", line 105, in compute_energy
return self.sum_wavenumbers(energy_fft)
File "/cfs/klemming/nobackup/a/avmo/src/fluidmeta/fluidfft/fluidfft/fft2d/operators.py", line 287, in sum_wavenumbers
return self.opfft.sum_wavenumbers(np.ascontiguousarray(field_fft))
File "fluidfft/fft2d/fft2dmpi_with_fftwmpi2d.pyx", line 213, in fluidfft.fft2d.mpi_with_fftwmpi2d.FFT2DMPIWithFFTWMPI2D.sum_wavenumbers
IndexError: Out of bounds on buffer access (axis 0)
```https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/5C-API implementation to use cython functions in pythran extensions2020-02-12T22:59:42ZBitbucket ImporterC-API implementation to use cython functions in pythran extensions*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Turns out, cython only produces the `__pyx_capi__` dictionary with `api` modifier only when simple functions ...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Turns out, cython only produces the `__pyx_capi__` dictionary with `api` modifier only when simple functions are defined, for eg.
```cython
cdef api int foo(int f):
return f
```
Even if I use the `api` modifier in class methods, no C API is exported. Then one would need to write wrapper functions which take an instance of the class as an argument. Like this (a modified version of the example in http://docs.cython.org/en/latest/src/userguide/wrapping_CPlusPlus.html)
```cython
cdef api class PyRectangle [object cPyRectangle, type cPyRectangle_t]:
cdef Rectangle c_rect # hold a C++ instance which we're wrapping
def __cinit__(self, int x0, int y0, int x1, int y1):
self.c_rect = Rectangle(x0, y0, x1, y1)
cpdef api int get_area(self):
return self.c_rect.getArea()
def get_size(self):
cdef int width, height
self.c_rect.getSize(&width, &height)
return width, height
def move(self, dx, dy):
self.c_rect.move(dx, dy)
cdef api int getArea(Rectangle c_rect):
return c_rect.getArea()
cdef api int get_area_cy(PyRectangle py_rect):
return py_rect.get_area()
```
The problem with this method is that Pythran won't accept a class as a function parameter.https://foss.heptapod.net/fluiddyn/fluidfft/-/issues/4fft2d.mpi_with_fftwmpi2d sometimes allocates zero-sized arrays2020-02-15T09:47:58ZBitbucket Importerfft2d.mpi_with_fftwmpi2d sometimes allocates zero-sized arrays*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Take this case for instance: nh = (1024, 1024); Lh = (8, 8); np=32
This happened in both Beskow and Trioli...*Created originally on Bitbucket by [avmo (Ashwin Vishnu)](https://bitbucket.org/%7Bbd72905e-cf99-4847-84c2-998104370f6e%7D/)*
Take this case for instance: nh = (1024, 1024); Lh = (8, 8); np=32
This happened in both Beskow and Triolith.
`mpirun -np 32 fluidsim bench -s ns2d 1024 1024 -t fft2d.mpi_with_fftwmpi2d`
On the other hand fft2d.mpi_with_fftw1d has no problem with decomposing the domain. Is this FFTW related or FluidFFT related?
```python
Traceback (most recent call last):
File "/home/x_ashmo/opt/my-envs/fluidmeta3/bin/fluidsim", line 11, in <module>
load_entry_point('fluidsim', 'console_scripts', 'fluidsim')()
File "/home/x_ashmo/src/fluidmeta3/fluidsim/fluidsim/util/console/__main__.py", line 62, in run
args.func(args)
File "/home/x_ashmo/src/fluidmeta3/fluidsim/fluidsim/util/console/bench.py", line 207, in run
type_fft=args.type_fft)
File "/home/x_ashmo/src/fluidmeta3/fluidsim/fluidsim/util/console/bench.py", line 65, in bench
_bench(type_fft)
File "/home/x_ashmo/src/fluidmeta3/fluidsim/fluidsim/util/console/bench.py", line 43, in _bench
sim = Simul(params)
File "/home/x_ashmo/src/fluidmeta3/fluidsim/fluidsim/base/solvers/base.py", line 109, in __init__
self.oper = Operators(params=params)
File "/home/x_ashmo/src/fluidmeta3/fluidsim/fluidsim/operators/operators2d.py", line 64, in __init__
coef_dealiasing=params.oper.coef_dealiasing)
File "/home/x_ashmo/src/fluidmeta3/fluidfft/fluidfft/fft2d/operators.py", line 170, in __init__
khmax = min(self.kx.max(), self.ky.max())
File "/home/x_ashmo/opt/my-envs/fluidmeta3/lib/python3.6/site-packages/numpy/core/_methods.py", line 26, in _amax
return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity
```