https://gitlab.mpcdf.mpg.de/TurTLE/turtle/badges/master/pipeline.svg

1 Turbulence Tools: Lagrangian and Eulerian

1.1 Introduction

TurTLE implements a number of standard functionality of Fourier-based pseudo-spectral numerical simulations, as well as the corresponding numerical particle-tracking functionality. The package also contains a Navier-Stokes solver, as well as a small number of post-processing tools. The solver is production-ready, having already been used for a number of publications.

TurTLE is written in C++ and it uses a hybrid MPI/OpenMP programming paradigm, relying on FFTW3 for an efficient Fourier transform implementation. HDF5 is used for I/O, including for parameter input and output of statistics. CMake is used for compilation and installation.

A Python 3 wrapper is used to prepare parameters and initial conditions, as well as to generate job submission scripts for clusters and to perform basic post-processing of simulation results.

The primary aim of TurTLE is to provide optimal performance, while reducing the time spent on setting up and supervising ensembles of DNS, with the added benefit of a unified launch-postprocess approach through the use of the Python wrapper.

1.2 Background and documentation

Please find up-to-date documentation at http://TurTLE.pages.mpcdf.de/turtle/index.html.

1.2.1 Authors

TurTLE is developed and maintained by the Wilczek group at the Max Planck Institute for Dynamics and Self-Organization and the University of Bayreuth in collaboration with the Application Support Group of the Max Planck Computing and Data Facility.

TurTLE contains contributions from:

  • Cristian C Lalescu <Cristian.Lalescu@mpcdf.mpg.de>

  • Dimitar Vlaykov

  • Berenger Bramas

  • Debarghya Banerjee

  • Jose Agustin Arguedas Leiva

  • Markus Rampp

  • Tobias Baetge

  • Lukas Bentkamp

  • Michael Wilczek

1.3 Installation for postprocessing only

The Python 3 package may be installed directly if only post-processing of existing data is desired. Simply clone the repository and install by executing

bash configure_python.sh
python setup.py install

(add –user or sudo as appropriate). setup.py uses the setuptools package for dependency resolution.

1.4 Full installation

TurTLE can be used on various machines, with laptops routinely being used for development and testing, but large production runs using tens of thousands of CPU cores on large computing clusters.

The C++ library requires a number of dependencies, that CMake will search for before compilation and installation. In brief, an MPI compiler is required, as well as the HDF5 C library compiled with parallel support and FFTW >= 3.3.4. We provide instructions for local compilation of FFTW and HDF5, because default versions packaged with Linux variants are typically inadequately configured.

These installation steps assume that you have a working C/C++ MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it), as well as an installation of Python 3. The list is a work in progress, please contact us (Cristian.Lalescu@mpcdf.mpg.de) if the procedure fails at any step of the process. We recommend to first read the instructions in full, and only afterwards starting to execute the individual steps.

Creating a virtual environment

We recommend creating a virtual environment for TurTLE. To do this, choose an installation location <INSTALL_LOC> on a local fast partition (under unix systems, this could be e.g. ~/.local) and a name for the environment <VENV_NAME> (e.g. turtle-production). To create the virtual environment, execute:

python -m venv <INSTALL_LOC>/<VENV_NAME>

In the following, we refer to the path <INSTALL_LOC> / <VENV_NAME> as <TURTLE_DIR>. For more information on virtual environments, please see https://docs.python-guide.org/dev/virtualenvs/.

Installation of requirements

TurTLE has the following requirements:

  • C/C++ compiler of your choice

  • MPI

  • Python 3

  • FFTW version >= 3.3.4

  • HDF5 version >= 1.12. Important: must be compiled with the option --enable-parallel

  • cmake version >= 3.23

  • (optional) GNU Scientific Library (GSL)

We will assume you already have a working MPI compiler and an installation of Python 3. For the other requirements, we provide installation instructions. We recommend installing them on a local fast partition. In the following, we refer to their locations as <FFTW_DIR>, <HDF5_DIR> and <CMAKE_DIR>. You may choose to install the requirements into your virtual environment. In that case, all of these placeholders are equal to <TURTLE_DIR>.

  • FFTW

    Download latest version from http://www.fftw.org/. To compile and install it in the custom location <FFTW_DIR>, execute the following commands in order:

    ./configure \
    --prefix=<FFTW_DIR> \
    --enable-float \
    --enable-mpi \
    --enable-openmp \
    --enable-threads \
    --enable-shared
make
make install
./configure \
    --prefix=<FFTW_DIR> \
    --enable-mpi \
    --enable-openmp \
    --enable-threads \
    --enable-shared
make
make install

    Optimization: If you are installing on a production machine, please see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html. For a typical x86 machine, one would add --enable-sse --enable-avx for the single precision configure step (i.e. first configure step), and separately --enable-sse2 --enable-avx2 for the double precision configure step (most Intel CPUs will also benefit from --enable-avx512).

    TurTLE will try to find FFTW using PkgConfig. If the package is installed in a non-standard location, make sure the corresponding environment variables are properly exported (see step 3 of installation of TurTLE).

  • HDF5

    Download HDF5 from https://www.hdfgroup.org/downloads/hdf5/. To compile and install it in custom location <HDF5_DIR>, execute the following commands in order:

    ./configure --prefix=<HDF5_DIR> --enable-parallel
make
make install

    The --enable-parallel flag is required because TurTLE uses parallel I/O.

    TurTLE will try to find HDF5 using the regular FindHDF5, which searches system folders, or HDF5_ROOT.

  • cmake

    Check if cmake (version >= 3.23) is available from your default package manager. If not, then download cmake at https://cmake.org/cmake/resources/software.html. To compile and install it in custom location <CMAKE_DIR>, execute the following commands in order:

    ./bootstrap --prefix=<CMAKE_DIR>
make
make install

    The directory <CMAKE_DIR> is only relevant to later executing the cmake binary (which can be found under <CMAKE_DIR>/bin after installation).

Installation of TurTLE

  1. Choose a location for the source code, enter it and clone turtle repository by

    git clone git@gitlab.mpcdf.mpg.de:TurTLE/turtle.git

    Alternatively, you may visit the website https://gitlab.mpcdf.mpg.de/TurTLE/turtle and download the source manually.

  2. Execute

    cd turtle
mkdir build
cd build

  3. Copy the file bash_setup_template.sh into the build folder:

    cp ../bash_setup_template.sh ./bash_setup_for_TurTLE.sh

    This file will set all required environment variables. Please replace all placeholders by their corresponding values.

    Note: In principle it is possible to add this information to your .bashrc, but we recommend against it.

  4. Copy the file pc_host_info.py into the build folder:

    cp ../pc_host_info.py ./host_info.py

    This file contains information about the machine on which TurTLE will run. On desktop machines, no further steps are required. On clusters, please edit the host_info.py file according to the instructions in the file.

  5. Finally, we are ready to install TurTLE. Within the build folder, execute

    source bash_setup_for_TurTLE.sh
cmake .. -DCMAKE_INSTALL_PREFIX=<TURTLE_DIR>

    Then compile TurTLE with <N> cores using

    make -j<N>
make install

Congratulations, you have installed TurTLE! Feel free to checkout the overview.

Using TurTLE as a library.

When requiring functionality not provided by TurTLE, we recommend that users use TurTLE as a library from “external projects”. We are yet to write an explicit tutorial of the procedure, but TurTLE already contains examples of the approach:

  • TurTLE/test/test_Heun_p2p.py and related C++ files

  • TurTLE/test/test_particle_deleter.py and related C++ files

In order for the procedure to work, please note the 3 files that TurTLE installs alongside the C++ headers and library:

TurTLEConfig.cmake
TurTLE_EXPORT.cmake
TurTLE_EXPORT-noconfig.cmake

These files are installed under <TURTLE_DIR>/lib, and they are required for such external projects to work (the Python wrapper calls cmake behind the scenes, and cmake will need these files). In case you encounter compilation errors even though TurTLE itself works without problems, it is probably necessary to update the cmake input config file: turtle/cmake/TurTLEConfig.cmake.in — you are welcome to contact us with the details.

Uninstall

If you installed TurTLE in a virtual environment, you may simply remove the virtual environment.

If you installed TurTLE in a default Python location, then you should navigate to the corresponding site-packages folder, and manually remove all folders/files containing “TurTLE” in their name. On linux systems Python will typically use something like /usr/lib/python3/dist-packages or ~/.local/lib/python3.x/site-packages (you should be able to find all relevant locations in the sys.path list). This also applies if you used a virtual environment, but you’d like to clean it for any reason.

Documentation

A local build of the documentation is possible where necessary. Doxygen is used for the C++ source code, and the Sphinx and breathe Python packages are used to merge the Python wrapper documentation with the C++ documentation. The optional CMake targets doc_doxygen, doc_html and doc_latex generate the required documents in the build directory. As long as the full cmake installation is possible (see below), one would proceed as follows to generate the documentation locally:

mkdir build-doc
cd build-doc
cmake ..
make doc_doxygen
make doc_html
cd sphinx_latex
make

After these steps, the (HTML) manual is available under build-doc/sphinx_html/index.html, and a PDF version of the manual can be found at build-doc/sphinx_latex/TurTLE.pdf. The stand-alone doxygen-generated documentation is present at build-doc/html/index.html.

1.5 Reference publication

Please see CPC v278 p108406 (https://doi.org/10.1016/j.cpc.2022.108406) for a description of TurTLE, as well as a detailed discussion of the novel particle tracking approach. This is also the publication to be cited by works that made use of TurTLE.

Alternatively, you may consult the preprint at https://doi.org/10.1016/j.cpc.2022.108406.

1.6 Contact

If you have any questions, comments or suggestions, please contact us via the gitlab interface at https://gitlab.mpcdf.mpg.de/TurTLE/turtle, or write directly to Dr. Cristian C. Lalescu (Cristian.Lalescu@mpcdf.mpg.de).

1.7 Comments

  • particles: initialization of multistep solvers is done with lower order methods, so direct convergence tests will fail (see the “convergence test” tutorial for more information).

  • code is used mainly with Python 3.5 and later, and it is not tested at all with Python 2.x