ALIs

kommt noch

Overview of the Cluster Configuration

This document gives a short overview of the resources available on the LRZ Linux Cluster.

LRZ Linux Cluster Overview

The LRZ Linux Cluster consists of several segments with different types of interconnect and different sizes of shared memory. All systems have a (virtual) 64 bit address space.

  • AMD Opteron based 2-way and (mostly) 4-way nodes for serial processing
  • AMD Opteron based 8-way and 32-way nodes with 10GE Myrinet for parallel processing
  • Intel Xeon based 4-way nodes for serial processing
  • Intel Nocona based 2-way nodes with Infiniband interconnect

  • Intel Nehalem-EP based 8-way nodes SGI ICE with Infiniband interconnect

  • Intel Nehalem-EX based 256-way sgi UltraViolet with infiniband-based NUMAlink shared memory interconnect

  • MPP Cluster with 16-way AMD-based nodes and Infiniband interconnect (available August 17, 2011, and will use SLURM as batch system)

Characteristics

Based on the various node types the LRZ Linux cluster offers a wide span of capabilities:

  • mixed shared and distributed memory
  • large software portfolio
  • flexible usage due to various available memory sizes
  • parallelization by message passing (MPI)
  • shared memory parallelization with OpenMP or pthreads

  • mixed (hybrid) programming with MPI and OpenMP

Hardware

Summary of the capacities of Cluster segments

  

Segment

Number of  nodes

Number of processor 
cores

aggregate
peak performance
(TFlop/s)

aggregate
memory
(TByte)

aggregate temporary disks
 (TByte)

pseudo-temporary storage
(TByte)

project storage
(TByte)
x86_64/EM64T until 2009

all

510

753

2030

5646

13.2

43.1

4.7

9.5

106

106

>100

>50
ICE 64 512 5.2 1.5 0
MPP

178

2848

22.7

2.8
UV 1 256 2.0 0.5

Details

Node 
Name

High Performance Network 

CPU Type

Memory
per Node
(GByte)

local
temp. Disk
 ($TMPDIR)
at each node
(GByte)

Login and Interactive Nodes

lx64ia2 

shared Memory (ccNUMA)

8-way AMD Opteron (2.6 GHz)

32

>200

lx64ia3

shared memory (SMP)

2-way Nocona 3600MHz

2

130
ice1-login Infiniband 8-core Xeon (not Nehalem!)

ssh from front end node

 8 0 (diskless)
uv1 NUMALink CPUset containing 16 cores

qrsh from front end node needed for access

 32 of 512 do not use

EM64T / Opteron  Nodes

lx64a1

(GE)

2-way Opteron (2.0 GHz)

5

>200

lx64a3-11 

(GE)

2-way Opteron (2.4 GHz)

6

>200

lx64a12-31 

(GE)

4-way Opteron (Dual-Core, 2.6 GHz)

8

>200

lx64a256-487

(GE)

4-way Opteron (Dual-Core, 2.6 GHz)

8

>200

lx64a133-169, 179

10G Myrinet

8-way Opteron (Dual-Core, 2.6 GHz)

32

>200

lx64e2-5

Infiniband 

2-way Nocona 3200MHz

2

130
r1i[0-3]n[0-15] Infiniband dual socket quad core Nehalem 24 0 (diskless)
lxa2-lxa177 Infiniband QDR dual socket octo Core AMD Magny Cours 16 0 (diskless)

The several available types of high performance networks are:

  • 10G Myrinet network with > 500 MB/s bandwidth (presently only connected via standard 10GE switch)
  • Infiniband,  a switched fabric architecture with low latency and high bandwidth
  • Shared memory may be used for MPI programs running within a node

  • NUMALink: high-bandwidth SGI NUMAlink interconnect fabric

Storage Arrays and File Servers for the Linux Cluster

Pseudo-temporary disk space is provided using NAS technology.

Technical Data:

 

$OPT_TMP

$PROJECT

aggregate I/O Bandwidth:

~2 GBytes/sec

~2 GBytes/sec

Net Total
Capacity:

100 TBytes

50 TBytes

For details on the usage of the background storage, please consult the appropriate LRZ document.