.. image:: https://gitlab.mpcdf.mpg.de/TurTLE/turtle/badges/master/pipeline.svg :target: https://gitlab.mpcdf.mpg.de/TurTLE/turtle/commits/master .. sectnum:: .. contents:: ========================================= Turbulence Tools: Lagrangian and Eulerian ========================================= .. _sec-introduction: ------------ Introduction ------------ TurTLE implements a number of standard functionality of Fourier-based pseudo-spectral numerical simulations, as well as the corresponding numerical particle-tracking functionality. The package also contains a Navier-Stokes solver, as well as a small number of post-processing tools. The solver is production-ready, having already been used for a number of publications. TurTLE is written in C++ and it uses a hybrid MPI/OpenMP programming paradigm, relying on FFTW3 for an efficient Fourier transform implementation. HDF5 is used for I/O, including for parameter input and output of statistics. CMake is used for compilation and installation. A Python 3 wrapper is used to prepare parameters and initial conditions, as well as to generate job submission scripts for clusters and to perform basic post-processing of simulation results. The primary aim of TurTLE is to provide optimal performance, while reducing the time spent on setting up and supervising ensembles of DNS, with the added benefit of a unified launch-postprocess approach through the use of the Python wrapper. ---------------------------- Background and documentation ---------------------------- Please find up-to-date documentation at http://TurTLE.pages.mpcdf.de/turtle/index.html. Authors ------- TurTLE is developed and maintained by the Wilczek group at the Max Planck Institute for Dynamics and Self-Organization and the University of Bayreuth in collaboration with the Application Support Group of the Max Planck Computing and Data Facility. TurTLE contains contributions from: .. include:: AUTHORS .. _sec-installation: --------------------------------------------------- Installation for postprocessing only --------------------------------------------------- The Python 3 package may be installed directly if only post-processing of existing data is desired. Simply clone the repository and install by executing .. code:: bash bash configure_python.sh python setup.py install (add `--user` or `sudo` as appropriate). `setup.py` uses the `setuptools` package for dependency resolution. ---------------------- Full installation ---------------------- TurTLE can be used on various machines, with laptops routinely being used for development and testing, but large production runs using tens of thousands of CPU cores on large computing clusters. The C++ library requires a number of dependencies, that `CMake` will search for before compilation and installation. In brief, an MPI compiler is required, as well as the HDF5 C library compiled with parallel support and FFTW >= 3.3.4. We provide instructions for local compilation of FFTW and HDF5, because default versions packaged with Linux variants are typically inadequately configured. These installation steps assume that you have a working C/C++ MPI compiler, properly configured on your system (i.e. the various configure scripts are able to find it), as well as an installation of Python 3. The list is a work in progress, please contact us (Cristian.Lalescu@mpcdf.mpg.de) if the procedure fails at any step of the process. We recommend to first read the instructions in full, and only afterwards starting to execute the individual steps. **Creating a virtual environment** We recommend creating a virtual environment for TurTLE. To do this, choose an installation location :code:`` on a local fast partition (under unix systems, this could be e.g. `~/.local`) and a name for the environment :code:`` (e.g. `turtle-production`). To create the virtual environment, execute: .. code:: bash python -m venv / In the following, we refer to the path :code:`` / :code:`` as :code:``. For more information on virtual environments, please see https://docs.python-guide.org/dev/virtualenvs/. **Installation of requirements** TurTLE has the following requirements: - C/C++ compiler of your choice - MPI - Python 3 - FFTW version >= 3.3.4 - HDF5 version >= 1.12. **Important**: must be compiled with the option :code:`--enable-parallel` - cmake version >= 3.23 - (optional) GNU Scientific Library (GSL) We will assume you already have a working MPI compiler and an installation of Python 3. For the other requirements, we provide installation instructions. We recommend installing them on a local fast partition. In the following, we refer to their locations as :code:``, :code:`` and :code:``. You may choose to install the requirements into your virtual environment. In that case, all of these placeholders are equal to :code:``. - **FFTW** Download latest version from http://www.fftw.org/. To compile and install it in the custom location :code:``, execute the following commands in order: .. code:: bash ./configure \ --prefix= \ --enable-float \ --enable-mpi \ --enable-openmp \ --enable-threads \ --enable-shared make make install ./configure \ --prefix= \ --enable-mpi \ --enable-openmp \ --enable-threads \ --enable-shared make make install **Optimization**: If you are installing on a production machine, please see http://www.fftw.org/fftw3_doc/Installation-on-Unix.html. For a typical x86 machine, one would add :code:`--enable-sse --enable-avx` for the single precision configure step (i.e. first configure step), and separately :code:`--enable-sse2 --enable-avx2` for the double precision configure step (most Intel CPUs will also benefit from :code:`--enable-avx512`). TurTLE will try to find FFTW using PkgConfig. If the package is installed in a non-standard location, make sure the corresponding environment variables are properly exported (see step 3 of installation of TurTLE). - **HDF5** Download HDF5 from https://www.hdfgroup.org/downloads/hdf5/. To compile and install it in custom location :code:``, execute the following commands in order: .. code:: bash ./configure --prefix= --enable-parallel make make install The :code:`--enable-parallel` flag is required because TurTLE uses parallel I/O. TurTLE will try to find HDF5 using the regular FindHDF5, which searches system folders, or `HDF5_ROOT`. - **cmake** Check if cmake (version >= 3.23) is available from your default package manager. If not, then download cmake at https://cmake.org/cmake/resources/software.html. To compile and install it in custom location :code:``, execute the following commands in order: .. code:: bash ./bootstrap --prefix= make make install The directory :code:`` is only relevant to later executing the :code:`cmake` binary (which can be found under :code:`/bin` after installation). **Installation of TurTLE** 1. Choose a location for the source code, enter it and clone turtle repository by .. code:: bash git clone git@gitlab.mpcdf.mpg.de:TurTLE/turtle.git Alternatively, you may visit the website https://gitlab.mpcdf.mpg.de/TurTLE/turtle and download the source manually. 2. Execute .. code:: bash cd turtle mkdir build cd build 3. Copy the file `bash_setup_template.sh` into the build folder: .. code:: bash cp ../bash_setup_template.sh ./bash_setup_for_TurTLE.sh This file will set all required environment variables. Please replace all placeholders by their corresponding values. *Note*: In principle it is possible to add this information to your `.bashrc`, but we recommend against it. 4. Copy the file `pc_host_info.py` into the build folder: .. code:: bash cp ../pc_host_info.py ./host_info.py This file contains information about the machine on which TurTLE will run. On desktop machines, no further steps are required. On clusters, please edit the `host_info.py` file according to the instructions in the file. 5. Finally, we are ready to install TurTLE. Within the build folder, execute .. code:: bash source bash_setup_for_TurTLE.sh cmake .. -DCMAKE_INSTALL_PREFIX= Then compile TurTLE with :code:`` cores using .. code:: bash make -j make install Congratulations, you have installed TurTLE! Feel free to checkout the `overview `_. **Using TurTLE as a library**. When requiring functionality not provided by TurTLE, we recommend that users use TurTLE as a library from "external projects". We are yet to write an explicit tutorial of the procedure, but TurTLE already contains examples of the approach: * `TurTLE/test/test_Heun_p2p.py` and related C++ files * `TurTLE/test/test_particle_deleter.py` and related C++ files In order for the procedure to work, please note the 3 files that TurTLE installs alongside the C++ headers and library: .. code:: bash TurTLEConfig.cmake TurTLE_EXPORT.cmake TurTLE_EXPORT-noconfig.cmake These files are installed under :code:`/lib`, and they are required for such external projects to work (the Python wrapper calls `cmake` behind the scenes, and `cmake` will need these files). In case you encounter compilation errors even though TurTLE itself works without problems, it is probably necessary to update the cmake input config file: `turtle/cmake/TurTLEConfig.cmake.in` --- you are welcome to contact us with the details. **Uninstall** If you installed TurTLE in a virtual environment, you may simply remove the virtual environment. If you installed TurTLE in a default Python location, then you should navigate to the corresponding `site-packages` folder, and manually remove all folders/files containing "TurTLE" in their name. On linux systems Python will typically use something like `/usr/lib/python3/dist-packages` or `~/.local/lib/python3.x/site-packages` (you should be able to find all relevant locations in the `sys.path` list). This also applies if you used a virtual environment, but you'd like to clean it for any reason. **Documentation** A local build of the documentation is possible where necessary. Doxygen is used for the C++ source code, and the `Sphinx` and `breathe` Python packages are used to merge the Python wrapper documentation with the C++ documentation. The optional `CMake` targets `doc_doxygen`, `doc_html` and `doc_latex` generate the required documents in the build directory. As long as the full cmake installation is possible (see below), one would proceed as follows to generate the documentation locally: .. code:: bash mkdir build-doc cd build-doc cmake .. make doc_doxygen make doc_html cd sphinx_latex make After these steps, the (HTML) manual is available under `build-doc/sphinx_html/index.html`, and a PDF version of the manual can be found at `build-doc/sphinx_latex/TurTLE.pdf`. The stand-alone `doxygen`-generated documentation is present at `build-doc/html/index.html`. --------------------- Reference publication --------------------- Please see CPC v278 p108406 (https://doi.org/10.1016/j.cpc.2022.108406) for a description of TurTLE, as well as a detailed discussion of the novel particle tracking approach. This is also the publication to be cited by works that made use of TurTLE. Alternatively, you may consult the preprint at https://doi.org/10.1016/j.cpc.2022.108406. ------- Contact ------- If you have any questions, comments or suggestions, please contact us via the gitlab interface at https://gitlab.mpcdf.mpg.de/TurTLE/turtle, or write directly to Dr. Cristian C. Lalescu (Cristian.Lalescu@mpcdf.mpg.de). -------- Comments -------- * particles: initialization of multistep solvers is done with lower order methods, so direct convergence tests will fail (see the "convergence test" tutorial for more information). * code is used mainly with Python 3.5 and later, and it is not tested at all with Python 2.x