ALIs

kommt noch

OpenMPI: A high performance message passing library

The Open MPI Project is an open source MPI-2 implementation that is developed and maintained by a consortium of academic, research, and industry partners. Open MPI is therefore able to combine the expertise, technologies, and resources from all across the High Performance Computing community in order to build the best MPI library available. Open MPI offers advantages for system and software vendors, application developers and computer science researchers.

OpenMPI installations on LRZ systems

OpenMPI is not a regularly supported MPI at LRZ, but is used for research and experimental purposes, such as

This means that while LRZ will do its best to help you with problems, we do not have any commercial level support and hence cannot give any functional or reliability guarantees for using this software. Also, OpenMPI will not scale as well as the proprietary MPI implementations on the high-end systems.

The following table gives an overview over the available OpenMPI installations on the LRZ HPC systems:

Platform Environment module Purpose
Altix (HLRB-II) mpi.ompi/1.2.5 (or 1.2.6) Eclipse PTP development
Altix (HLRB-II) mpi.ompi/1.3/intel newer MPI-2 features, based on 10.1 Intel compilers, with PBS support
Altix (HLRB-II) mpi.ompi/1.3/gcc newer MPI-2 features, based on 4.1 GCC, with PBS support
Altix (HLRB-II, Cluster) mpi.ompi/1.3/heterogeneous MPMD on heterogeneous systems
x86_64 Systems mpi.ompi/1.3/heterogeneous MPMD on heterogeneous systems
ICE and other infiniband clusters mpi.ompi/1.4/infiniband  (requires a suitable gcc or Intel compiler module) newer MPI-2 features

In order to access the OpenMPI installation, the appropriate Environment module must be loaded after unloading the default MPI environment. For example, on HLRB-II the commands

module unload mpi.altix

module load mpi.ompi/1.3

might be used, selecting one of the OpenMPI installations from the table above.

Usage of OpenMPI is typical for the LRZ MPI installations with respect to compiler wrappers and startup commands; special variants are described below. On Itanium-based systems, the Intel compilers are used to compile the code. On x86_64 systems this is also the case but other compilers may be made available upon request.

 

Specific OpenMPI usage variants

Heterogeneous MPMD execution

Heterogeneous execution might involve the following machine combinations:

  • HLRB-II + RVS1
  • Cluster Altix + GVS1 or GVS2

Since this OpenMPI variant does not support a batch system on the IA64 side, only use of a single partition job is supported. Please set up your PBS script appropriately if you work from HLRB-II. Heterogeneous jobs should be started on the Altix to make sure that the resources provided by the batch system are used. For HLRB-II, using the following command sequence in a PBS script is recommended (assuming you wish to start 64 MPI tasks on HLRB-II, and one MPI task on RVS1):

slots_altix=64
slots_vis=1
cat > ./myhosts <<EOF
$(hostname)  slots=$slots_altix
rvs1  slots=$slots_vis max_slots=$slots_vis
EOF
mpiexec -hostfile ./myhosts -n $slots_altix ./myprog_altix.exe : -n $slots_vis ./myprog_vis.exe

For using the Altix + GVS1/2, the same command sequence might be used in an SGE script, where "rvs1" must be replaced by "gvs1" or "gvs2". Note that the binaries myprog_altix.exe and myprog_vis.exe must be built under the OpenMPI environment mpi.ompi/<version>/heterogeneous on the Altix and the visualization system, respectively.

 

Since we normally do not tolerate long-running compute jobs on the visualization systems, we must ask you to contact LRZ HPC support before embarking on production runs with the above.

 

Infiniband Clusters

For using OpenMPI on Infiniband clusters, please set up the correct environment

module unload <loaded MPI version, if any>

module load gcc/<gccversion> # only if a GCC based MPI is desired, see below for required version

module load mpi.ompi/1.4/infiniband

on one of the following systems:

  • sgi ICE - please use gccversion=4.4
  • accelerated IB cluster (not generally available) - please use gccversion=4.6

If no gcc module is loaded, an Intel 11.1 compiler based MPI build will be configured. The executable can then be executed via the command

mpiexec --x LD_LIBRARY_PATH -hostfile ./myhosts -n <number-of-processes> ./myprog.exe

provided the host file "myhosts" contains entries of the form

host1  slots=8

host2  slots=8

host3  slots=8

The number of slots for a host will typically be equal to the number of cores in that host (if your program runs in MPP mode); if you want to run OpenMPI jobs on the ICE - please do not simply go ahead, but contact the LRZ service desk.

 

Documentation

A Frequently asked Questions page is available on the OpenMPI web site.