Introduction

The Message Passing Interface (MPI) library allows processes in your parallel application to communicate with one another by sending and receiving messages. There is no default MPI library in your environment when you log in to the cluster. You need to choose the desired MPI implementation for your applications. This is done by loading an appropriate MPI module. Currently the available MPI implementations on our cluster are OpenMPI and Mpich. For both implementations the MPI libraries are compiled and built with either the Intel compiler suite or the GNU compiler suite. These are organized in software modules.

Installation

MPI has many forms, we’ll list a few here, and also look at User_Codes/Parallel_Computing/MPI.

mpi4py with Python

We recommend the Mambaforge Python distribution. The latest version is available with the python/3.10.9-fasrc01 software module. Since mpi4py is not available with the default module you need to install it in your user environment.

The most straightforward way to install mpi4py in your user space is to create a new conda environment with the mpi4py package. For instance, you can do something like the below:

module load python/3.10.12-fasrc01 
mamba create -n python3_env1 python numpy pip wheel mpi4py
source activate python3_env1

This will create a conda environment named python3_env1 with the mpi4py package and activate it. It will also install a MPI library required by mpi4py. By default, the above commands will install MPICH.

For most of the cases the above installation procedure should work well. However, if your workflow requires a specific flavor and/or version of MPI, you could use pip to install mpi4py in your custom conda environment as detailed below:

Load compiler and MPI software modules:

module load gcc/12.2.0-fasrc01
module load openmpi/4.1.5-fasrc03

This will load OpenMPI in your user environment. You can also look at our user documentation to learn more about software modules on the FAS cluster.

Load a Python module:

module load python/3.10.12-fasrc01

Create a conda environment:

mamba create -n python3_env2 python numpy pip wheel

Install mpi4py with pip:

pip install mpi4py

Activate the new environment:

source activate python3_env2

For example code, see Parallel_Computing/Python/mpi4py

OpenMPI with GNU Compiler

If you want to use OpenMPI compiled with the GNU compiler you need to load appropriate compiler and MPI modules. Below are some possible combinations, check module spider MODULENAME to get a full listing of possibilities.

# GCC + OpenMPI, e.g.,
module load gcc/13.2.0-fasrc01 openmpi/5.0.2-fasrc01

# GCC + Mpich, e.g.,
module load gcc/13.2.0-fasrc01 mpich/4.2.0-fasrc01

# Intel + OpenMPI, e.g.,
module load intel/24.0.1-fasrc01 openmpi/5.0.2-fasrc01

# Intel + Mpich, e.g.,
module load intel/24.0.1-fasrc01 mpich/4.2.0-fasrc01

# Intel + IntelMPI (IntelMPI runs mpich underneath), e.g.
module load intel/24.0.1-fasrc01 intelmpi/2021.11-fasrc01

For reproducibility and consistency it is recommended to use the complete module name with the module load command, as illustrated above. Modules on the cluster get updated often so check if there are more recent ones. The modules are set up so that you can only have one MPI module loaded at a time. If you try loading a second one it will automatically unload the first. This is done to avoid dependencies collisions.

There are four ways you can set up your MPI on the cluster:

• Put the module load command in your startup files.
  Most users will find this option most convenient. You will likely only want to use a single version of MPI for all your work. This method also works with all MPI modules currently available on the cluster.

• Load the module in your current shell.
  For the current MPI versions you do not need to have the module load command in your startup files. If you submit a job the remote processes will inherit the submission shell environment and use the proper MPI library. Note this method does not work with older versions of MPI.

• Load the module in your job script.
  If you will be using different versions of MPI for different jobs, then you can put the module load command in your script. You need to ensure your script can execute the module load command properly.

• Do not use modules and set environment variables yourself.
  You obviously do not need to use modules but can hard code paths. However, these locations may change without warning so you should set them in one location only and not scatter them throughout your scripts. This option could be useful if you have a customized local build of MPI you would like to use with your applications.

Parallel HDF5

Parallel HDF5 (PHDF5) is the parallel version of the HDF5 library. It utilizes MPI to perform parallel HDF5 operations. For example, when an HDF5 file is opened with an MPI communicator, all the processes within the communicator can perform various operations on the file. PHDF5 supports file operations such as file create, open and close, as well as dataset operations such as object creation, modification and querying, all in parallel using MPI-IO. User Codes has examples are intended to illustrate the use of PHDF5 on the Cannon cluster. The specific examples are implemented in Fortran, but the could be easily translated to C or C++.

Video Training

Examples

Resources