if myscript contains:
./my_serial_prog <in.$MP.CHILD >out.$MP_CHILD
poe myscript -procs 40

would start 40 times my_serial_prog with different input and output

Only useful for interactive jobs; otherwise it is set be LoadLeveler.

Only useful for interactive jobs; otherwise it is set be LoadLeveler.

It may be used alone or in conjunction with MP_TASKS_PER_NODE and/or MP_PROCS,

Only useful for interactive jobs; otherwise it is set be LoadLeveler.

It may be used alone or in conjunction with MP_NODES.

• MPI: Indicates that the application makes only MPI calls.
• LAPI: Indicates that the application makes only LAPI calls.
• MPI_LAPI: Indicates that calls to both message passing APIs are used in the application, and the same set of communication resources (windows, IP addresses) is to be shared between them.
• MPI,LAPI: Indicates that calls to both message passing APIs are used in the application, with dedicated resources assigned to each of them.
• LAPI,MPI: Has a meaning identical to MPI,LAPI.

• task geometry
• blocking factor
• machine order
• consumable resources
• memory requirements
• disk space requirements
• machine architecture
• run jobs from more than 1 pool

For more information on the LoadLeveler functionality you can exploit, refer to For more information, see LoadLeveler pages.

Run parallel jobs without specifying a host file or pool, thereby causing LoadLeveler to select nodes for the parallel job from any in its cluster.

It may be saved for debugging and/or for further execution (together with MP_LLFILE or -llfile) without specifing the arguments for poe.

• all: All tasks receive the same input data from STDIN.
• none: No tasks receive input data from STDIN; STDIN will be used by the home node only.
• a task id: STDIN is only sent to the task identified.

• unordered: All tasks write output data to STDOUT asynchronously.
• ordered: Output data from each parallel task is written to its own buffer. Later, all buffers are flushed, in task order, to STDOUT.
• a task id: Only the task indicated writes output data to STDOUT.

• 0: Error
• 1: Warning and error
• 2: Informational, warning, and error
• 3: Informational, warning, and error. Also reports high-level diagnostic messages for use by the IBM Support Center.
• 4, 5, 6: Informational, warning, and error. Also reports high- and low-level diagnostic messages for use by the IBM Support Center

0 - 6

When a variable's value is the same for all tasks, it is printed only once. If it is different for some tasks, an asterisk (*) appears in the report after the word "Task".

• no: Do not produce a report of MPI environment variable settings.
• yes: Produce a report of MPI environment variable settings.
• script_name: Produce the report (same as yes), then run the script specified here.

• yes: Setting this to yes enables some intertask parameter checking which may help uncover certain problems, but slows execution.
• no or nor: Normal mode does only relatively inexpensive, local parameter checking. Setting this variable to min allows PE MPI to bypass parameter checking on all send and receive operations.
• deb (debug) checking is intended for developing applications, and can significantly slow performance.
• min (for minimum mode) feedback.should only be used with well tested applications because a bug in an application running with min will not provide useful error

• CORE   Default: specifies that each MPI task runs on a single physical processor core.
• CORE:n  Specifies the number of processor cores to which the threads of an MPI task are constrained (one thread per core), typically n is the number of OMP_NUM_THREADS (should be used for OpenMP or hybrid jobs)
• CPU  Specifies that each MPI task runs on a single logical CPU.
• CPU:n Specifies the number of of logical CPUs to which the threads of an MPI task are constrained (one thread per cpu), typically n is the number of OMP_NUM_THREADS
• MCM Specifies that the tasks are allocated in a round-robin fashion among the sockets of a node, should only be use if not all cores of a node are not used.
• list-of-numbers  Specifies that the tasks are assigned on a round-robin basis to this set of sockets
• -1  Specifies that no affinity request will be made (disables task affinity).

CPU or CORE will generate  a CPU mask with one or two entries.

CPU:n will generatea CPU which contains all CPU but will set KMP_AFFINITY different for each MPI task, e.g.
KMP_AFFINITY="proclist=[0-5],exclicit"

nnnnn (byte)
nnnK (kB)
nnM (MB)
nnG (GB)

Can also be specified in the form MP_BUFFER_MEM=M1,M2, where M1 specifies the amount of pre-allocated memory. M2 specifies an upper bound on the amount of early arrival buffer memory

Number of
Tasks     MP_EAGER_LIMIT
------------------------
   1 to  256       32768
 257 to  512       16384
 513 to 1024        8192
1025 to 2048        4096
2049 to 4096        2048
4097 to 8192        1024

MPI uses the MP_BUFFER_MEM and the MP_EAGER_LIMIT values that are selected for a job to determine how many complete point-to-point messages, each with a size that is equal to or less than the eager_limit, can be sent eagerly from every task of the job to a single task, without causing the single target to run out of buffer space. This is done by allocating to each sending task a number of message credits for each target. The sending task will consume one message credit for each eager send to a particular target. It will get that credit back after the message has been matched at that target. The following equation is used to calculate the number of credits to be allocated:

MP_BUFFER_MEM / (MP_PROCS * MAX(MP_EAGER_LIMIT, 64))

MPI uses this equation to ensure that there are at least two credits for each  target. If needed, MPI reduces the initially selected value of MP_EAGER_LIMIT, or increases the initially selected value of MP_BUFFER_MEM, in order to achieve this minimum threshold of two credits for each target.

MPI_WAIT_MODE set to nopoll may reduce CPU consumption for applications that post a receive call on a separate thread, and that receive call does not expect an immediate message arrival. Also, using MPI_WAIT_MODE set to nopoll may increase delay between message arrival and the blocking call's return. It is recommended that MP_CSS_INTERRUPT be set to yes when the nopoll wait is selected, so that the system wait can be interrupted by the arrival of a message. Otherwise, the nopoll wait is interrupted at the timing interval set by MP_POLLING_INTERVAL

User Space is an unreliable packet transport (packets may be dropped during transport without an error being reported), the message dispatcher manages packet acknowledgment and retransmission with a sliding window protocol. This message dispatcher is also run on a hidden thread once every few hundred milliseconds and, if environment variable MP_CSS_INTERRUPT is set, upon notification of packet arrival.

To specify whether or not arriving packets generate interrupts. Using this environment variable may provide better performance for certain applications. Setting this variable explicitly will suppress the MPI-directed switching of interrupt mode, leaving the user in control for the rest of the run.

For more information, refer to the MPI_FILE_OPEN and MPI_WIN_CREATE subroutines in IBM Parallel Environment: MPI Subroutine Reference.

See Determining which nodes will participate in parallel file I/O for more information.

nnnnn
nnnK (where:K = 1024 bytes)
nnM (where: M = 1024*1024 bytes)

nnnn
nnnK (where K=1024 bytes)
nnnM (where M=1024*1024 bytes)

It is highly recommended to limit the amount of information written for large parallel application by setting it to STDERR.