ALIs

kommt noch

Intel MPI

Intel's MPI implementation allows to build an MPI application once and run it on various interconnects. Good performance can be achieved also over proprietary interconnects if the vendor provides a DAPL implementation which Intel MPI can make use of.

Table of contents

Setting up for the use of Intel MPI

On all HPC systems at LRZ, environment modules are used to set up software packages. Since Intel MPI is not the default MPI flavour, you may need to unload the default MPI environment before loading the Intel MPI module:

module unload  mpi.altix mpi.mpt mpi.ibm mpi.parastation mpi.mpich2  

module load mpi.intel

This environment module makes available all tools needed to compile and execute MPI programs as described in the main MPI document. Since Intel MPI is (probably) not binary compatible to any other MPI flavour, you should completely re-compile and re-link your application under the new environment.

Compiling and linking

The following table lists a number of options which can be used with the compiler wrappers in addition to the usual switches for optimization etc.
Option Meaning Remarks
-mt-mpi link against thread-safe MPI thread-safeness up to MPI_THREAD_MULTIPLE is provided
-static-mpi use static instead of dynamic MPI libraries default is dynamic
-t[=log] compile with MPI tracing. The module mpi_tracing must be loaded after the mpi.intel module
-ilp64 link against MPI interface with 8 byte integers you may need to also specify -i8 for compiling your code.

Executing Intel MPI programs

Execution on the Linux Cluster (SGE), on HLRB-II (PBS), or on SuperMUC (LoadLeveler)

Before executing the binary, the mpi.intel module must again be loaded in the executing shell; this will, where necessary, also set up the so-called MPD ring by automatically running the mpdboot command. For both interactive and batch execution, the mpiexec command should be used to start up the MPI program:

mpiexec -n 12 ./myprog.exe

Execution on the Linux Cluster (SLURM)

Please use the srun_ps utility when starting up Intel MPI binaries under control of the SLURM scheduler. For simply running the executalbe program, it is not necessary to load the mpi.intel module.

Hybrid program execution

Newer versions of Intel MPI can use process pinning to achieve good performance for mixed MPI and OpenMP programs: For example, the command sequence

export OMP_NUM_THREADS=4

export I_MPI_PIN_DOMAIN=omp:compact

mpiexec -n 12 ./myprog.exe

will start 12 MPI tasks with 4 threads each, keeping threads as near to their master tasks by spreading out the MPI tasks. This is probably the most efficient way to proceed in the majority of cases.

Handling environment variables

The mpiexec command takes a number of options to control how environment variables are transmitted to the started MPI tasks. A typical command line might look like

mpiexec -genv MY_VAR_1 value1 -genv MY_VAR_2 value2 -n 12 ./myprog.exe

Please consult the documentation linked below for further details and options.

Environment variables controlling the execution

Here some Altix-specific environment variables controlling the execution of Intel MPI are described. Please consult the documentation for Intel MPI for the very large set of I_MPI_* variables which allow to extensively configure and optimize at compile as well as run time.

 

Name

Function

Remarks

SGI Altix xpmem DAPL - Altix specific settings

DAPL_XPMEM_MEMMAP_VERBOSE

This variable can be set (to any value) to issue additional diagnostic messages, especially if your code crashes with DAPL related error messages.

May improve performance if your application generates many small messages. Default is 512.

DAPL_XPMEM_MAPPED_HEAP_SIZE

This variable must be set to the size of the statically defined heap memory area (in Bytes) if your code has many statically defined large arrays. Not setting this may lead to crashes in the DAPL setup phase ("Assertion `remote_ep' failed").

the command size myprog.exe will give you an estimate for the upper limit to set.

DAPL_XPMEM_MAPPED_STACK_SIZE

This variable must be set to the size of the statically defined stack memory area (in Bytes) if your code has many automatic arrays which turn out to be large.

It may be a good idea to use the -heap-arrays compilation switch to circumvent usage of this variable.

Documentation

General Information on MPI

Please refer to the MPI page at LRZ for the API documentation and information about the different flavors of MPI available.

Intel MPI documentation

After the mpi.intel module is loaded, the $MPI_DOC environment variable points at a directory containing PDF format reference manuals and other documents.

For the most up-to-date release, the documentation can also be found on Intel's web site.