Parallel invocation

See Running zCFD § Running in parallel for the Hybrid MPI/OpenMP, Full MPI and GPU parallel execution concepts. This page covers running across multiple machines and the lower-level flags and environment variables that control MPI transport selection and thread affinity.

Running across multiple machines with a cluster scheduler

There are many cluster scheduling systems but commonly used ones are Slurm, PBS Pro or GridEngine. Please refer to either the cluster scheduling software or your local HPC systems' documentation for how to submit jobs to the appropriate nodes on your system as configuration varies between machines.

Intel MPI has tight integration to most commonly used scheduling systems and so will typically autodetect the nodes to run on.

Running across multiple machines without a cluster scheduler

If your cluster does not have a scheduling system then I_MPI_HYDRA_HOST_FILE can be set in the environment to point at a file containing the list of hostnames that zCFD will launch on. For example:

export I_MPI_HYDRA_HOST_FILE=~/hosts.file

An example contents of a hosts file is shown below:

compute01:2
compute02:2
compute03:1

This file specifies three hosts, compute01, compute02, compute03 where 2 ranks will be launched on each of compute01 and compute02 and 1 rank will be launched on compute03.

Alternatively, the hosts can be listed without rank counts, in which case processes are launched round robin through the list.

compute01
compute02
compute03

Libfabric provider selection

zCFD autodetects the provider to use for MPI communication. It tries the following providers in order, and uses the first that fi_info lists and that a one-rank MPI test accepts:

efa
mlx
verbs
tcp
udp
sockets

When FI_PROVIDER is set, autodetection is skipped.

Using cluster provided libfabric

By default zCFD will use the version of libfabric that ships with Intel MPI. For some advanced use cases you may wish to make use of preinstalled libfabric. This may be if you have a custom network driver or a specific configuration on your cluster. To enable this, set the I_MPI_OFI_LIBRARY_INTERNAL environment variable to 0 before activating the zCFD environment. The cluster's fi_info must be on the path.

export I_MPI_OFI_LIBRARY_INTERNAL=0

Pre launch script

On some systems the defaults setup by zcfd may not be optimal and so a custom script to modify the environment can be injected by setting the environment variable ZCFD_PRE_LAUNCH to the path to a script. This script will be sourced from within a bash environment just before the solver is launched.

An example use case for this is to bind a GPU to a specific network interface on a multi-homed network system.

OpenMP affinity

By default zCFD binds its OpenMP threads to cores, setting OMP_PROC_BIND to true and OMP_PLACES to cores. When a node runs more MPI ranks than it has NUMA nodes, OMP_PLACES is set to sockets instead.

Other MPI implementations

Where Intel MPI does not work on a platform, zCFD can be built against a specific implementation of MPI. This is a custom deployment for your facility; contact Zenotech if you need it.

Running at large scale

As zCFD is partly written in python, the install consists of a large quantity of small files, which on a large scale cluster can have a negative impact on the cluster filesystem during startup. To mitigate this, the code can be staged into either local storage or memory on the compute nodes using MPI to broadcast the data.