SLURM example job scripts

Table of contents

Introductory remarks

The job scripts for SLURM partitions are provided as templates which you can adapt for your own settings. In particular, you should account for the following points:

  • Some entries are placeholders, which you must replace with correct, user-specific settings. In particular, path specifications and e-Mail addresses must be adapted. Always specify the appropriate directories instead of the names with the three periods in the following examples!

  • For recommendations on how to do large-scale I/O please refer to the description of the file systems available on the cluster. It is recommended to keep executables within your HOME file system, in particular for parallel jobs. The example jobs reflect this, assuming that files are opened with relative path names from within the executed program.

  • Because you usually have to work with the environment modules package in your batch script, sourcing the file /etc/profile.d/modules.sh is included in the example scripts.

  • For CooLMUC3 job please note that you need to submit from lxlogin8.

Pfeil nach oben


Shared Memory jobs

This job type uses a single shared memory node of the designated SLURM partition. Parallelization can be achieved either via (POSIX) thread programming or directive-based OpenMP programming.

Here are example scripts for starting an OpenMP program:

CooLMUC2 clusterCooLMUC3 cluster

 Myrinet cluster
(obsolete)

#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J jobname
 #SBATCH --get-user-env
#SBATCH --clusters=mpp2
#SBATCH --nodes=1-1
#SBATCH --cpus-per-task=28
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh cd mydir export OMP_NUM_THREADS=28 # 28 is the maximum reasonable value for CooLMUC2 ./myprog.exe
#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J jobname
 #SBATCH --get-user-env
#SBATCH --clusters=mpp3
#SBATCH --nodes=1-1
#SBATCH --cpus-per-task=64
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh cd mydir export OMP_NUM_THREADS=64 # 256 is the maximum reasonable value for CooLMUC3 ./myprog.exe
#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J jobname
#SBATCH --get-user-env
#SBATCH --clusters=myri

#SBATCH --nodes=1-1
#SBATCH --cpus-per-task=32
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh cd mydir export OMP_NUM_THREADS=32
# up to 32 threads can be configured on the Myrinet nodes (see below)
# The used value should be consistent
# with --cpus-per-task above
./myprog.exe

For each job, the maximum reasonable value of threads is set inside the script. On the UV, please also specify the value via --cpus-per-task.

Pfeil nach oben


MPI jobs

For MPI documentation please consult the MPI page on the LRZ web server. On both the CooLMUC2 and the MPP Cluster, Intel MPI is used as the default environment. The following examples illustrate jobs that use 4 computational nodes.

On the CooLMUC2 cluster

CooLMUC2 MPP-style jobCooLMUC2 hybrid job
#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname
#SBATCH --get-user-env
#SBATCH --clusters=mpp2

#SBATCH --ntasks=112
#
multiples of 28 for mpp2
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh cd $SCRATCH/mydata mpiexec $HOME/exedir/myprog.exe
#!/bin/bash 
#SBATCH -o /home/hpc/..../.../myjob.%j.%N.out
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname
#SBATCH --get-user-env
#SBATCH --clusters=mpp2

#SBATCH --ntasks=16
#SBATCH --cpus-per-task=7
#
the above is a good match for the
#
CooLMUC2 architecture.
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh cd $SCRATCH/mydata
export OMP_NUM_THREADS=7
mpiexec -n 16 --perhost 4 $HOME/exedir/myprog.exe
# will start 16 MPI tasks with 7 threads each. Note that
# each node has 28 cores, so 4 tasks must be started
# per host.

A setup as for the hybrid job can also serve to provide more memory per MPI task without using OpenMP (e.g., by setting OMP_NUM_THREADS=1). Note that this will leave cores unused!

On the CoolMUC3 cluster

TODO: resources for the various KNL modes

Note: starting more than 64 MPI tasks per KNL node is likely to cause startup failures.

CoolMUC3 MPP-style job

use physical cores only

CoolMUC3 hybrid job

use physical cores only

CooLMUC3 hybrid job

use hyperthreads

#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname
#SBATCH --get-user-env
#SBATCH --clusters=mpp3

#SBATCH --nodes=8

#SBATCH --ntasks-per-node=64
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh cd $SCRATCH/mydata mpiexec $HOME/exedir/myprog.exe
# will start 512 MPI tasks
#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname
#SBATCH --get-user-env
#SBATCH --clusters=mpp3

#SBATCH --nodes=8

#SBATCH --ntasks-per-node=16
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh
cd $SCRATCH/mydata
export OMP_NUM_THREADS=4
mpiexec $HOME/exedir/myprog.exe
# will start 128 MPI tasks with 4 threads each

#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname
#SBATCH --get-user-env
#SBATCH --clusters=mpp3

#SBATCH --nodes=8

#SBATCH --ntasks-per-node=16
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh
cd $SCRATCH/mydata
export OMP_NUM_THREADS=16
mpiexec $HOME/exedir/myprog.exe
# will start 128 MPI tasks with 16 threads each
# 4 hyperthreads per core are used

On the Myrinet cluster

Myrinet 10 GE 32-way systems
#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname #SBATCH --get-user-env
#SBATCH --clusters=myri
#SBATCH --partition=myri_large
# MA-TUM users can also specify matum_p or matum_u above
#SBATCH --ntasks=32
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=36:00:00
source /etc/profile.d/modules.sh cd $OPT_TMP/mydata mpiexec $HOME/exedir/myprog.exe # At most 32 cores can be used in myri_large.
# More may be possible in the matum_? queues

Please note:

For some software packages, it is also possible to use SLURM's own srun command; this will however not work well for programs compiled against Intel MPI.

It is also possible to use the --nodes keyword in combination with --tasks-per-node (instead of --ntasks) to configure parallel jobs.

Pfeil nach oben


Special job configurations

Job Farming (starting multiple serial jobs on a shared memory system)

Please use this with care! If the serial jobs are imbalanced with respect to run time, this usage pattern can waste CPU resources. At LRZ's discretion, unbalanced jobs may be removed forcibly. The example job script illustrates how to start up multiple serial MATLAB jobs within a shared memory parallel SLURM script. Note that the various subdirectories subdir_1, ..., subdir_8 must exist and contain the needed input data.

Multi-Serial Example
#!/bin/bash
#SBATCH -o /home/hpc/.../.../myjob.%j.%N.out 
#SBATCH -D /home/hpc/.../.../mydir
#SBATCH -J Jobname
#SBATCH --get-user-env
#SBATCH --clusters=myri
#SBATCH --partition=myri_std
#SBATCH --nodes=1-1
#SBATCH --mail-type=end
#SBATCH --mail-user=xyz@xyz.de
#SBATCH --export=NONE
#SBATCH --time=08:00:00
source /etc/profile.d/modules.sh module load matlab # Prevent matlab-internal multithreading export OMP_NUM_THREADS=1 # Start as many background serial jobs as there are cores available on the node: for i in $(seq 1 8) ; do cd subdir_${i} matlab -nodesktop input.m > output.res & cd .. done wait # for completion of background tasks

 

Pfeil nach oben