Overview of the Cluster Configuration

This document gives a short overview of the resources available on the LRZ Linux Cluster.

LRZ Linux Cluster Overview

The LRZ Linux Cluster consists of several segments with different types of interconnect and different sizes of shared memory. All systems have a (virtual) 64 bit address space.

  • Intel Xeon based 4-way nodes for serial processing
  • Intel Westmere-EX based sgi UltraViolet systems with infiniband-based NUMAlink shared memory interconnect
  • MPP Cluster with 16-way AMD-based nodes and QDR Infiniband interconnect
  • CooLMUC2 Cluster with 28-way Haswell-based nodes and FDR14 Infiniband interconnect

Characteristics

Based on the various node types the LRZ Linux cluster offers a wide span of capabilities:

  • mixed shared and distributed memory
  • large software portfolio
  • flexible usage due to various available memory sizes
  • parallelization by message passing (MPI)
  • shared memory parallelization with OpenMP or pthreads
  • mixed (hybrid) programming with MPI and OpenMP

Hardware

Summary of the capacities of Cluster segments

These include the generally available parts only.

Segment

Number of  nodes

Number of processor 
cores

aggregate
peak performance
(TFlop/s)

aggregate
memory
(TByte)

scratch storage
(TByte)

project storage
(TByte)

x86_64/EM64T

all

576

15920

495

36.5

~1000 (NAS)

~1400 (GPFS, late summer 2015)

NAS, group quota

Hugemem 12

240

5

3.1

CoolMUC2 384 10752 447 24.6
MPP

178

2848

22.7

2.8

UV 2 2080 20.0 6.0

Details

Node 
Name

High Performance Network 

CPU Type

Memory
per Node
(GByte)

local
temp. Disk
 ($TMPDIR)
at each node
(GByte)

Login Nodes

ice1-login (used as development front end for UV) Infiniband 8-core Xeon (not Nehalem!)

ssh from front end node

8 0 (diskless)
lxlogin1, lxlogin2, lxlogin4 Infiniband QDR dual socket octo-core AMD Opteron Magny Cours (2.0 GHz) 256 0 (diskless)
lxlogin5, lxlogin6 Infiniband FDR14 dual socket 12-core Haswell EP 128 n/a

Intel Xeon / AMD Opteron  Nodes

Hugemem

Gigabit Ethernet

dual socket 10-way Westmere

240 (one  node with 480)

0 (system disk only)

lx64a133-169, 179

10G Myrinet

8-way Opteron (Dual-Core, 2.6 GHz)

32

>200

CooLMUC2 Infiniband FDR dual socket 14 core Haswell EP 64 0 (diskless)
lxa2-lxa177 Infiniband QDR dual socket octo Core AMD Magny Cours 16 0 (diskless)
uv2, uv3 NUMAlink sgi Ultraviolet using Intel Westmere-EX (2.4 GHz) 3000 >200

The several available types of high performance networks are:

  • Infiniband,  a switched fabric architecture with low latency and high bandwidth
  • Shared memory may be used for MPI programs running within a node
  • NUMALink: high-bandwidth SGI NUMAlink interconnect fabric

Storage Arrays and File Servers for the Linux Cluster

Pseudo-temporary disk space is provided using NAS technology.

Technical Data:

 

$SCRATCH

$PROJECT (== $WORK)

aggregate I/O Bandwidth:

~2 GBytes/sec (NAS)

~25 GBytes/sec (GPFS)

~2 GBytes/sec

For details on the usage of the background storage, please consult the appropriate LRZ document.