I confirm that there is a problem. It is related to the fact that for some shapes and some numbers of MPI processes, fftw3_mpi does not use all the processes. One process has no data and does nothing... Of course this is not very efficient so we should avoid these cases.
I solve some related bugs in fluidfft but still many functions of the operator do not work and I do not know how to solve this.
I added a clear warning.
Remark: there was a bug in the order of the dimensions given with these commands so the shapes in your last message are wrong... It is solve now and it gives:
mpirun -np 4 fluidsim bench 24 16 -t fft2d.mpi_with_fftwmpi2dnh = (24, 16); Lh = (12.0, 8)/home/pierre/Dev/fluidfft/fluidfft/fft2d/operators.py:107: UserWarning: The shape for processor 3 is (0, 24). This means that it has no data to treat. It is not efficient and many of the functions of this operators won't work. Unless you know what you do, change the fft class, the resolution or the number of MPI processes. 'This means that it has no data to treat. 'Traceback (most recent call last): File "/home/pierre/Dev/fluidsim/fluidsim/base/forcing/specific.py", line 382, in compute a_fft, self.shapeK_loc_coarse) File "/home/pierre/Dev/fluidsim/fluidsim/operators/operators2d.py", line 238, in coarse_seq_from_fft_loc f1d_temp[iKyc] = f_fft[iKxloc, iKy]IndexError: index 6 is out of bounds for axis 0 with size 0During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/home/pierre/opt/anaconda3/bin/fluidsim", line 11, in <module> load_entry_point('fluidsim', 'console_scripts', 'fluidsim')() File "/home/pierre/Dev/fluidsim/fluidsim/util/console/__main__.py", line 62, in run args.func(args) File "/home/pierre/Dev/fluidsim/fluidsim/util/console/bench.py", line 209, in run type_fft=args.type_fft) File "/home/pierre/Dev/fluidsim/fluidsim/util/console/bench.py", line 67, in bench _bench(type_fft) File "/home/pierre/Dev/fluidsim/fluidsim/util/console/bench.py", line 44, in _bench sim = Simul(params) File "/home/pierre/Dev/fluidsim/fluidsim/base/solvers/base.py", line 144, in __init__ self.forcing.compute() File "/home/pierre/Dev/fluidsim/fluidsim/base/forcing/base.py", line 108, in compute self.forcing_maker.compute() File "/home/pierre/Dev/fluidsim/fluidsim/base/forcing/specific.py", line 387, in compute self.oper.shapeK_loc))ValueError: rank=3, shapeK_loc(coarse)=(16, 9), shapeK_loc=(0, 24)--------------------------------------------------------------------------MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD with errorcode 0.NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.You may or may not see output from other processes, depending onexactly when Open MPI kills them.--------------------------------------------------------------------------
$ mpirun -np 32 fluidsim bench -s ns2d 1024 1024 -t fft2d.mpi_with_fftwmpi2dnh = (1024, 1024); Lh = (8, 8)/home/pierre/Dev/fluidfft/fluidfft/fft2d/operators.py:107: UserWarning: The shape for processor 31 is (0, 1024). This means that it has no data to treat. It is not efficient and many of the functions of this operators won't work. Unless you know what you do, change the fft class, the resolution or the number of MPI processes. 'This means that it has no data to treat. 'Traceback (most recent call last): File "/home/pierre/opt/anaconda3/bin/fluidsim", line 11, in <module> load_entry_point('fluidsim', 'console_scripts', 'fluidsim')() File "/home/pierre/Dev/fluidsim/fluidsim/util/console/__main__.py", line 62, in run args.func(args) File "/home/pierre/Dev/fluidsim/fluidsim/util/console/bench.py", line 209, in run type_fft=args.type_fft) File "/home/pierre/Dev/fluidsim/fluidsim/util/console/bench.py", line 67, in bench _bench(type_fft) File "/home/pierre/Dev/fluidsim/fluidsim/util/console/bench.py", line 44, in _bench sim = Simul(params) File "/home/pierre/Dev/fluidsim/fluidsim/base/solvers/base.py", line 147, in __init__ self.output.init_with_oper_and_state() File "/home/pierre/Dev/fluidsim/fluidsim/base/output/base.py", line 406, in init_with_oper_and_state super(OutputBasePseudoSpectral, self).init_with_oper_and_state() File "/home/pierre/Dev/fluidsim/fluidsim/base/output/base.py", line 254, in init_with_oper_and_state self.init_with_initialized_state() File "/home/pierre/Dev/fluidsim/fluidsim/base/output/base.py", line 290, in init_with_initialized_state self.print_stdout.complete_init_with_state() File "/home/pierre/Dev/fluidsim/fluidsim/base/output/print_stdout.py", line 50, in complete_init_with_state self.energy0 = self.output.compute_energy() File "/home/pierre/Dev/fluidsim/fluidsim/solvers/ns2d/output/__init__.py", line 87, in compute_energy return self.sum_wavenumbers(energy_fft) File "/home/pierre/Dev/fluidfft/fluidfft/fft2d/operators.py", line 218, in sum_wavenumbers return self.opfft.sum_wavenumbers(np.ascontiguousarray(field_fft)) File "fluidfft/fft2d/fft2dmpi_with_fftwmpi2d.pyx", line 168, in fluidfft.fft2d.mpi_with_fftwmpi2d.FFT2DMPIWithFFTWMPI2D.sum_wavenumbers (fluidfft/fft2d/fft2dmpi_with_fftwmpi2d.cpp:6606)IndexError: Out of bounds on buffer access (axis 0)--------------------------------------------------------------------------MPI_ABORT was invoked on rank 31 in communicator MPI_COMM_WORLD with errorcode 0.NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.You may or may not see output from other processes, depending onexactly when Open MPI kills them.--------------------------------------------------------------------------