Running zCFD ============ zCFD is designed to exploit parallelism in the target computing hardware, making use of multiple processes (MPI), multiple threads per CPU process (OpenMP) and / or many threads per GPU. zCFD will handle most parallel functions automatically, so the user just needs to specify how much, and what type, of resource to use for a given run. .. contents:: Table of Contents :depth: 1 :local: :backlinks: none .. _`zcfd-command`: Command Line Execution ---------------------- zCFD includes an in-built command-line environment, which should be activated either when run directly or via a cluster management and job scheduling system, such as Slurm. .. code-block:: bash >source /INSTALL_LOCATION/zCFD-version/bin/activate This sets up the environment to enable execution of the specific version. When within the command-line environment, the command prompt will show a zCFD prefix: .. code-block:: bash (zCFD) > To deactivate the command line environment, returning the environment to the previous state use: .. code-block:: bash (zCFD) >zdeactivate Your command prompt should return to its normal appearance. To run zCFD on the command-line: .. code-block:: bash (zCFD) > run_zcfd -n -d -p -c where: .. list-table:: :widths: 20 20 70 :header-rows: 1 * - Parameter - Value - Description * - **num_tasks** - Integer - The number of partitions (one per device socket - CPU or GPU) * - **device_type** - CPU or GPU - Specifies whether or not to use GPU(s) if present * - **mesh_name** - String - The name of the mesh file (.h5) * - **case_name** - String - The name of the control dictionary (.py) For example, to run zCFD from the command-line environment on a desktop computer with a single 12-core CPU processor and no GPU: .. code-block:: bash (zCFD) >run_zcfd -n 1 -d cpu -p -c By default, zCFD will use all 12 cores via OpenMP unless a specific number is required. To run the above case using only 6 cores (there will be one thread per CPU core): .. code-block:: bash (zCFD) >export OMP_NUM_THREADS=6; run_zcfd -n 1 -d cpu -p -c To run zCFD on a desktop computer with a single CPU and two GPUs: .. code-block:: bash (zCFD) >run_zcfd -n 2 -d gpu -p -c .. _`input-validation`: Input Validation ---------------- zCFD provides an input validation script which can be used to check the solver control dictionary before run time reducing the likelihood of a job spending a long time queuing only to fail at start up. The script can be executed from the zCFD environment as follows: .. code-block:: bash (zCFD) >validate_input [-m ] Validating parameters dictionary... Parameters dictionary is valid Checking BC_, FR_, FZ_, IC_ and TR_ numbering... IC_ numbering is correct FR_ numbering is correct BC_ numbering is correct zCFD input file is valid If the -m option is given with a mesh file the script will check whether any zones specified as lists to boundary conditions, transforms or reports in the input exist in the mesh. Overriding Mesh and Case Names --------------------------------- For more advanced simulations with overset meshes (see :ref:`overset`) an override dictionary can be supplied as an additional argument to the command-line, in interactive mode or within a batch submission script, to specify multiple meshes and associated control dictionaries: .. code-block:: bash (zCFD) >run_zcfd -n 2 -d gpu -f An example override.py file for a background domain with a single rotating turbine would be: .. code-block:: python override = {'problem_case_pair': [('background.h5', 'background.py'), ('turbine.h5', 'turbine.py')]} Batch Queue Submission ---------------------- To run zCFD on a compute cluster with a queuing system (such as Slurm), the command-line environment activation is included within the submission script: Example Slurm submission script “run_zcfd.sub”: .. code-block:: bash #!/bin/bash #SBATCH -J account_name #SBATCH --output zcfd.out #SBATCH --nodes 2 #SBATCH --ntasks 4 #SBATCH --exclusive #SBATCH --no-requeue #SBATCH --time=10:00:00 #SBATCH --gres=gpu:2 #SBATCH --cpus-per-task=16 source /INSTALL_LOCATION/zCFD-version/bin/activate run_zcfd -n 4 -d gpu -p -c In this system we have 2 nodes, each with 2 GPUs. We allocate one task per GPU, giving a total of 4 tasks. Note that num_tasks, in this case 4, on the last line should match Slurm “ntasks” on line 5. The case is, optionally, run in “exclusive” mode which means that zCFD has uncontested use of the devices. In this case we have a 64-core CPU, giving 16 CPU cores for each of the 4 tasks. The “gres” line tells zCFD that there are 2 GPUs available on each node. The submission script would be submitted from the command-line: .. code-block:: bash sbatch run_zcfd.sub The job can then be managed using the standard Slurm commands. .. note:: Users unfamiliar with the batch submission parameters in a specific system should consult the system administrator. EPIC ---- For larger jobs or where resource availability is limited, `EPIC `_ allows users to submit zCFD jobs to a range of on-demand computing resources in an intuitive and secure way. EPIC also provides access to interactive post-processing environments without the need to download large data files. .. image:: images/epic.png :width: 600 :alt: EPIC .. _epic_desktops: EPIC Desktops ------------- .. contents:: Table of Contents :depth: 3 :local: :backlinks: none What are EPIC Desktops? ''''''''''''''''''''''' `EPIC `_ provides a way of starting an interactive Linux desktop in the Cloud, these are known as EPIC Desktops. The Desktops come with a range of preinstalled software, including zCFD. Data can automatically be synchronised between EPIC and the Desktops you launch. Launching a desktop ''''''''''''''''''' To launch an EPIC Desktop you need an account on `EPIC `_, you can sign up directly on the website for free. Once you have an account, navigate to Desktops on the right hand menu to launch a new Desktop. The desktops come in a variety of sizes and all have GPUs which can be used for running zCFD in GPU mode as well as for accelerated visualisation. If you organisation already uses EPIC then speak to your account manager for an invite to join the correct EPIC team. Managing Data ''''''''''''' The data you select when launching the desktop will be made available to you in the */home/epic/data* directory on the desktop. If you selected the "Online" option when launching the desktop then any changes or new files you make in that directory will be synchronised with EPIC. Each node also has a */scratch/* directory, this is a local high performance drive that only exists while the node is online. It is recommended to use this directory for running any zCFD jobs on the node and then copy the results back to */home/epic/data* when you are finished. **Note** */scratch/* will be deleted when the node terminates. Modules ''''''' The desktops use Linux *modules* to load applications. To see what applications are available launch a new terminal and run: .. code-block:: bash module avail You can then load the application using the *module load* command, for example to load the latest zCFD: .. code-block:: bash module load zcfd/latest This will add zCFD to your path and enable you to start using it in that terminal. Loading zCFD '''''''''''' The best way to load zCFD is to load the module and then run the *activate* script, this will activate the full zCFD environment. For example: .. code-block:: bash module load zcfd/latest . activate You can then start using zCFD: .. code-block:: bash run_zcfd zCFD Licence '''''''''''' After loading zCFD you need to specify which licence to use, you can do this by either copying your *zcfd.lic* file into your working directory, alongside your mesh and control dictionary, or by setting the *RLM_LICENSE_PASSWORD* environment variable. The password can be found inside your zcfd licence file. For example: .. code-block:: bash module load zcfd/latest . activate export RLM_LICENSE_PASSWORD=mypassword Running Paraview '''''''''''''''' Paraview is also available on the desktops and can be loaded using *modules* and then started from the terminal: .. code-block:: bash module load paraview paraview Using pvserver '''''''''''''' To load a zCFD .h5 file you can use the Paraview server that is bundled with zCFD and then connect to it using Paraview on the desktop. To start the server run the following in the terminal: .. code-block:: bash module load zcfd/latest . activate pvserver You can then launch Paraview from another terminal by running: .. code-block:: bash module load paraview paraview By default *pvserver* will be listening for a connection on port 11111. To connect to the zCFD pvserver do the following in Paraview: 1. From the *File* menu select *Connect*. 2. Click "Add server". 3. Change the server Name to "zcfd" but leave all the other settings as the defaults, click "Configure". On the next screen just click "Save". 4. Click "Connect". 5. The connection should be made and you will see a "Client connected" message output by pvserver. You can then load your zCFD .h5 files via *pvserver* **Note** The version of Paraview used to connect must match the version used in zCFD, if you get a client connection error please check the version numbers. Using Jupyter ------------- We also recommend the use of `Jupyter Notebooks `_ for post-processing zCFD runs. This is one of the easiest ways to use Python, via an interactive notebook on your local computer. The notebook is a locally hosted server that you can access via a web-browser. Jupyter is included in the zCFD distribution and notebooks for visualising the residual data will automatically be created when you run a zCFD case. To automatically start up notebook server on your computer within the zCFD environment run: .. code-block:: bash (zCFD) >start_lab .. _environment_variables: zCFD Environment Variables -------------------------- The following table gives some of the environment variables that can be set before running zCFD. These are generally for advanced users can can impact the way zCFD runs on you system. .. list-table:: :widths: 20 20 70 :header-rows: 1 * - Variable - Default - Description * - ZCFD_GPUDIRECT - unset - Set if you are using NVIDIA GPU direct (custom builds only) with a CUDA aware mpi implementation. * - ZCFD_PRE_LAUNCH - unset - This option allows a script to be run just before the code is launched so that the user can override any environment that has been automatically setup. * - CUDA_VISIBLE_DEVICES - unset - This option restricts the list of GPU devices that zCFD will be able to use. Typically this can be used to remove an incompatible GPU or the display GPU from the list of candidate GPUs. * - ZCFD_DISABLE_PINNED - unset - When set this causes the GPU memory allocated to be page-able and not pinned memory. This should always be used on Windows in WSL as CUDA inside WSL has a fixed low threshold for the maximum amount of allowed pinned memory. * - ZCFD_NCCL - unset - Set to 1 to make use of the NVIDIA NCCL library during parallel exchanges. * - PMPI_CUDA - unset - Set if you are running with platform MPI (custom builds only) to enable GPU aware MPI memory allocations * - MPICH_GPU_SUPPORT_ENABLED - unset - Set to 1 if you are running with Cray MPI (custom builds only) to enable GPU aware MPI memory allocations * - ZCFD_DISPATCH_SYNC - unset - Makes all kernel launches synchronous.