ALIs

kommt noch

Directory services/software/parallel/mpi/parastation

Table of contents

Setting up for the use of Parastation MPI

On all cluster systems which support this MPI variant, an environment module mpi.parastation is automatically loaded at login. This environment module makes available all tools needed to compile and execute MPI programs as described in the main MPI document. By default, the module makes use of the Intel compilers, but it is also possible to switch to the GCC or the PGI compilers.

Executing Parastation programs

Execution under SGE

On those cluster segments using the Sun Grid Engine (SGE) for batch processing, the mpiexec command is used to start up MPI programs, as described in the main MPI document.

Remark: It is intended to phase out SGE execution of parallel jobs in the near future.

Execution under SLURM

On those cluster segments using SLURM for batch processing, the srun_ps command should be used to start up MPI programs:

srun_ps -n 48 ./myprog.exe

starts up 48 MPI tasks. This will however only work inside a script submitted to SLURM, or inside a salloc allocation.

For a hybrid program, the setting of OMP_NUM_THREADS is accounted for, but can be overridden by using the -t option. For example,

srun_ps -n 6 -t 8 ./myhybridprog.exe

will start 6 MPI tasks with 8 Threads per MPI task. The mpiexec command will work, but should be used only for very short tests (<5 min) and small process counts (up to 4).

The srun_ps command also takes a "-x" option in order to propagate the values of all environment variables to other MPI tasks.

An MPMD startup method is not yet supported.

Hybrid programs

To start up programs using OpenMP threads, it is normally only necessary to set the OMP_NUM_THREADS variable before execution of the startup command. Parastation will perform process and thread pinning such that OpenMP threads are kept as near as possible to the MPI task which generated the threads.

However, the default Parastation MPI library only supports multi-threading on the level MPI_THREAD_SERIALIZED (i.e., no MPI calls within threaded regions). If you need a higher level of thread-safeness, please contact HPC support via the LRZ Service Desk.

Controlling Parastation execution

The mpi.parastation module will provide reasonable default settings for executing parallel jobs; however there are some circumstances where additional configuration settings may be useful:

  1. Round robin process distribution: Normally, nodes are filled up with MPI processes progressively. However, this may not be suitable, especially if you wish to run a hybrid MPI+OpenMP executable. To enable round robin distribution, please set
    export PSI_LOOP_NODES_FIRST=yes
    In general, it is not recommended to use this setting, since newer releases of the Parastation support hybrid execution by taking account of the OMP_NUM_THREADS setting.

  2. Overbooking: Normally, the Parastation control daemon will not allow you to start more MPI processes than available CPUs. To lift this restriction (not recommended), please set
    export PSI_OVERBOOK=yes

  3. Handling Environment Variables: Any environment variables which are required by all MPI processes must be exported as well as listed in the PSI_EXPORTS environment variable as a comma separated list:
    export PSI_EXPORTS="${PSI_EXPORTS},MY_IMPORTANT_VARIABLE"
    Do not overwrite the pre-existing values stored in PSI_EXPORTS; instead, please increment them as shown above, otherwise starting your executable might not work.

  4. Inserting additional commands: Setting the environment variables PSI_RARG_PRE_0, PSI_RARG_PRE_1, ... will execute the corresponding sequence of commands with the argument to mpirun. As a first example, consider how to renice all MPI Tasks of an interactive job:


         export PSI_RARG_PRE_0=/usr/bin/nice
        export PSI_RARG_PRE_1=-n
        export PSI_RARG_PRE_2=19
        nohup mpiexec -n 4 ./myprog.exe &

    As a second example, we wish to execute the MPI built binary myprog.exe from a script myscript started for each MPI task (which usually will contain a last line of the form exec $@):

          export PSI_RARG_PRE_0=$HOME/bin/myscript
      mpiexec -n 4 ./myprog.exe
Further information on Parastation settings is available via the man pages ps_environment (7) and process_placement (7).

Documentation

is available via Par-Tec's documentation page: