Software

Tools and libraries for better and easier use of SuperMUC

Application Software

For a complete overview of software packages that are available on LRZ systems, see: http://www.lrz.de/services/software/.

Access to software packages

LRZ uses the environment module approach to manage the user environment for different software, library or compiler versions. The distinct advantage of the modules approach is that the user is no longer required to explicitly specify paths for different executable versions, and try to keep the MANPATH and related environment variables coordinated. With the modules approach, users simply "load" and "unload" modules to control their environment. Type

  • module avail

to list all the modules which are available to be loaded. Notice that most of them indicate associated version numbers. Modules make it easy to switch between versions of a package. Specifying a module name without a version number will select the default production version.

Many modules.contain the URL of the LRZ documentation for the specific software package. Type

  • module show <modulename>

and look for the variable which ends with _WWW.

LRZ Tools

To get access to the LRZ Tools insert the following command

module load lrztools

After that the following commands are available

ComandPurpose, details and usage

Information

budget_and_quota Displays the CPU time budget and the file system quotas
memory_usage Displays the actual memory usage (high water mark) of a job or all my jobs which are running. Typically this command is inserted as the last command in a batchjob.
Usage:
memory_usage [$LOAD_STEP_ID | myjobs]

Placement of processes and threads

cpuset_query Returns information about the topology, number of core and CPU on which a process can run
get_cpuset_mask
get_cpuset_mask_2
Returns a string of 0's and 1's indicating on which (logical) CPUs a process can run
Returns the host, CPU-Id and the mask
where_do_i_run Returns the CPU-ID on which the command was run
placementtest-mpi.ibm
placementtest-mpi.intel
placementtest-omp
Returns information how processes and threads are placed on nodes and CPUs.
Example:
MP_POCS=5
MP_NODES=2
OMP_NUM_THREADS=10
MP_TASK_AFFINITY=core:$OMP_NUM_THREADS
mpiexec -n 5 placement-test.ibm

Batchjobs

 
lljobs List job accounting data.
Usage : lljobs [-h] [-y year] [-m month] [-d day] [-u user] [-j|-J jobid1,jobid2,...]
default: print jobs of user of current and past month
-h: print help
-y: year of the accounting records to search through (default current year)
-m: month of the accounting records to search through (current and past month)
-d: day of the the accounting records to search through (all days of the current and past month)
-u: userid or all
-j: print details for jobsIDs (without nodelist)
-J: print details for jobIDs (full records)
llu, llg Shortcut to show only all my jobs or all the jobs of my group/project
llx A better and more comprehensive overview of jobs in the queues, including various sort options.
For details see: llx -h
lltime Graphical display of the position of a job in the batch queue
Usage: lltime jobdID
For details see: lltime -h
llrun Submit or interactively run a parallel executable.
Usage : llrun [<options>] <exe> [<user_or_poe_args>]
<options>:
-N - number of nodes (Default: 1)
-n - total number of processes, same as -p (Default: 1)
-P - Task per node
-t - number of threads per process (Default: 1)
-j - jobname
-w,-W - submit batch job with wallclock limit (Default: 02:00:00)
-A - No automatic adjustment of number of nodes (Default: automatic adj.)
-M - MPI Environment (intel|ibm)
-E - copy current environment (COPYALL)
-f - Pin processes and Threads to Fixed physical cores
-b - submit batch job
-c,-C - submit to class (default: test)
-m - email address 
-i - include content of file before parallel execution 
-I - include content of file after parallel execution 
-o - do not run/submit job but save to file 
-h - help (this message)
-v - verbose (Default)
-V - NO verbose
Examples:

llrun -b -n 320 ./myexe
(runs 320 tasks on 8 Nodes because of automatic adjustment)
llrun -b -N 10 -n 320 ./myexe
(runs 320 tasks on 8 Nodes because of automatic adjustment)
llrun -b -N 10 -P 32 -A ./myexe
(runs 320 tasks on 10 Nodes, no automatic adjustment)
llrun -b -t 2 -n 320 ./myexe
(runs 320 tasks each which 2 threads on 16 Nodes)
llrun -b -n 320 -M intel ./myexe.intel
(runs 320 tasks with an Intel-MPI executable)
llrun -b -n 40 -c test ./myexe
(runs 40 tasks in the test queue)
llrun -b -n 40 -c general ./myexe
(runs 40 tasks in the general production queue)
llrun -n 40 ./myexe
(runs directly 40 tasks using terminal for stdout and sterr)
autonomous Embarrassingly parallel commands or jobs can be run as one LoadLeveler job
Usage: autonomous [<options>] [-f] file
<options>: 
-N:    number of nodes (Default: 1)
-n,-p: total number of processes, same as -p (Default: 1)
-f:    file with command
-j:    jobname
-A:    no automatic ajustmend of bnumber of nodes
-w,-W: submit batch job with wallclock limit (Default: 02:00:00)
-o:    do not submit but write to file
-E:    copy current environment (COPYALL)
-b:    submit batch job
-c,-C: submit to class (default: test)
-s:    separate output for each command, otherwiese stdout/stderr
-S:    separate output for each slave, but copyied to stdout
-m:    email adress for notification
-h:    help (this message)
Example: copy 32 files in parallel to one directory
cat >cmdfile <<EOD
cp file1 DIR
cp file2 DIR
...
cp file32 DIR
EOD
autonomous cmdfile
autonomous -n 32 -s -v cmdfile
autonomous -A -C general -N 4 -n 8 -S -v cmdfile
myjobs Display all job owned by user in the queues. Can be used as input to build other commands. e.g., llcancel `myjobs` will cancel all job owned by user.
get_cmd_out Sort output produced with option MP_LABELIO

Programming Environment

I_MPI Displays sorted list of the current settings of the Intel MPI Environment
MP Displays sorted list of the current settings of the IBM Parallel Environment
SN Returns the listing of available snapshot files.
Example: SN myfile

LRZ Library

module load lrztools

Contains useful subroutines and functions. Compile with:

mpicc/mpif90 ... $LIBLRZ_INC ... $LIBLRZ_LIB

Function/Subroutine
C and FORTRAN
Purpose, details and usage
int getpid (void)
INTER GETPID
Returns ithe process ID
int gettid (void)
INTER GETTID
Returns ithe thread ID
int where_do_i_run(void);
INTEGER WHERE_DO_I_RUN()
Returns the physical CPU ID where the task/thread is running
double dwalltime()
Double dcputime()
REAL(KIND=8) DWALLTIME()
REAL(KIND=8) DCPUTIME()
Returns the wallclock time/cputime spent between first and current call to the routine

MEMUSAGE(avail, used, free, buffers, cached)
integer(kind=8):: avail, used, free, buffers, cached

Returns in kb
  • Total available Memory
  • Used Memory
  • Free Memory
  • Memory used for buffers (raw blocks of the disks)
  • Memory used for file caching of the file systems
void place_task_(int cpu[],int *n);
INTEGER CPU(N)
PLACE_TASK(CPU,N)
Sets the mask, that the current task will run on the physical CPUs contained in the array CPU.
void(place_all_tasks(int *debug)LOGICAL DEBUG
PLACE_ALL_TASKS(DEBUG)
Places the tasks and thread on particular CPUs, whether by default algorithm or by using the environment variable CPU_MAPPING. Example:
CPU_MAPPING=0,2,4,8,10,12
OMP_NUM_THREADS=3
MP_NODES=8
mpiexec -n 16 ./a.out
If DEBUG is TRUE or 1 then information about the placement is output.
void place_task_(int cpu[],int *n);
INTEGER CPU(N)
PLACE_TASK(CPU,N)
Sets the mask, that the current task will run on the physical CPUs contained in the array CPU.
void placementinfo()
PLACEMENTINFO()
Outputs information about the placement of tasks and threads.