pSeries or RS/6000
workstation environment, without the ssp.clients fileset
installed, you can only use AIX authentication.
The IBM Parallel Environment for AIX (PE) software lets you develop, debug, analyze, tune, and execute parallel applications written in Fortran, C, and C++. PE conforms to existing standards like UNIX(R) and MPI. The PE runs on either an IBM RS/6000 SP (SP) machine, or an AIX workstation cluster.
PE consists of the following:
The purpose of the Parallel Operating Environment (POE) is to allow you to develop and execute your parallel applications across multiple processors, called nodes. When using POE, there is a single node (a workstation) that is called the home node that manages interactions with users.
POE transparently manages the allocation of remote nodes where your parallel application actually runs. It also handles the various requests and communication between the home node and the remote nodes via the underlying network.
This approach eases the transition from serial to parallel programming by hiding the differences, and allowing you to continue using standard AIX tools and techniques. You have to tell POE what remote nodes to use (more on that in a moment), but once you have, POE does the rest.
When we say processor node, we're talking about a physical entity or location that's defined to the network. It can be a standalone machine, or a processor node within an IBM RS/6000 SP (SP) frame. From POE's point of view, a node is a node... it doesn't make much difference.
If you're using a Symmetric Multiprocessor (SMP) system, it's important to know that, although an SMP node has more than one processing unit, it is still considered, and referred to as, a processor node.
Before getting underway, you should check that you have addressed the items covered in this section.
Whoever installed POE (this was probably your System Administrator but may have been you or someone else) should have verified that it was installed successfully by running the Installation Verification Program (IVP). The IBM Parallel Environment for AIX: Installation Guide discusses the IVP, which verifies installation for both threaded and non-threaded environments.
The IVP tests to see if POE is able to do the following:
Before you can run your job, you must first have access to the compute resources in your system. Here are some things to think about:
Note that if you're using LoadLeveler to submit POE jobs, it is LoadLeveler, not POE that handles user authorization. As a result, if you are using LoadLeveler to submit jobs, the following sections on user authorization do not apply to you.
POE uses the PSSP Security Services functions for performing user authorization and authentication. Your system administrator is responsible for defining and maintaining the security methods. For more information, see IBM Parallel Environment for AIX: Installation Guide.
POE supports the following user authentication methods, which are based on the PSSP Security Services methods. The system administrator sets the user authentication method with the PSSP chauthts command.
Note: If you're using LoadLeveler to submit POE jobs, which includes all user space applications, then LoadLeveler is responsible for the security authentication. The security function in POE is not invoked when POE is run under LoadLeveler.
User authentication is set on the POE remote nodes, but the method that is set on the POE home node is also significant, particularly with DCE authentication. Refer to IBM Parallel Environment for AIX: Installation Guide for specific details on mixing security methods in your system.
When you run POE in a standalone IBM pSeries or RS/6000
workstation environment, without the ssp.clients fileset
installed, you can only use AIX authentication.
You can use the lsauthts command to check the authentication method in use. See IBM AIX Parallel System Support Programs: Commands and Technical Reference (SA22-7351-02) for more information on the chauthts and lsauthts commands.
You must have remote execution authority on all the nodes in the system that you will use for parallel execution. Your system administrator should:
or
/etc/hosts.equiv is checked first, and, if the home node and user/machine name don't appear there, it then looks to .rhosts.
You can verify that you have remote execution authority by running a remote shell from the workstation where you intend to submit parallel jobs. For example, to test whether you have remote execution authority on 202r1n10, try the following command:
$ rsh 202r1n10 hostname
The response to this should be the remote host name. If it isn't the remote host name, or the command cannot run, you'll have to see your system administrator. Issue this command for every remote host on which you plan to have POE execute your job.
Refer to IBM Parallel Environment for AIX: Installation Guide for more detailed information.
When DCE authentication is enabled, POE expects a valid set of DCE credentials in order to submit parallel jobs.
When both DCE and Compatibility methods are enabled, POE first tries to use DCE authentication. If DCE authentication is unsuccessful, POE then uses AIX authentication.
In order to use DCE, POE requires the following:
For more information on DCE user authentication, refer to IBM Parallel Environment for AIX: Installation Guide and IBM Parallel Environment for AIX: Operation and Use, Vol. 1.
One way to tell POE where to run your program is by using a host list file. The host list file is generally in your current working directory, but you can move it anywhere you like by specifying certain parameters. This file can be given any name, but the default name is host.list. Many people use host.list as the name to avoid having to specify another parameter (as we'll discuss later). This file contains one of two different kinds of information; node names or pool numbers (a pool can also be designated by a string).
Node names refer to the hosts on which parallel jobs may be run. They may be specified as Domain Names (as long as those Domain Names can be resolved from the workstation where you submit the job) or as Internet addresses. Each host goes on a separate line in the host list file.
Here's an example of a host list file that specifies the node names on which four tasks will run:
202r1n10.hpssl.kgn.ibm.com 202r1n11.hpssl.kgn.ibm.com 202r1n09.hpssl.kgn.ibm.com 202r1n12.hpssl.kgn.ibm.com
Once you've checked all the items in Before you start, you're ready to run the Parallel Operating Environment. At this point, you can view POE as a way to run commands and programs on multiple nodes from a single point. Remember that these commands and programs are really running on the remote nodes. If you ask POE to perform some operation on a remote node, everything necessary to perform that operation must be available on the remote node. More on this in a moment.
Note that there are two ways to influence the way your parallel program is executed; with environment variables or command-line option flags. You can set environment variables at the beginning of your session to influence each program that you execute. You could also get the same effect by specifying the related command-line flag when you invoke POE, but its influence only lasts for that particular program execution. For the most part, this book shows you how to use the command-line option flags to influence the way your program executes. Running POE with environment variables gives you some high-level information, but you may also want to refer to IBM Parallel Environment for AIX: Operation and Use, Vol. 1 to learn more about using environment variables.
One more thing. In the following sections, we show you how to run a parallel job by requesting that POE use nodes in a host list file. A little later, we'll tell you how to use a host list file to request nodes from LoadLeveler.
The poe command enables you to load and execute programs on remote nodes. The syntax is:
poe [program] [options]
When you invoke poe, it allocates processor nodes for each task and initializes the local environment. It then loads your program and reproduces your local shell environment on each processor node. POE also passes the user program arguments to each remote node.
The simplest thing to do with POE is to run an AIX command. When you try these examples on your system, use a host list file that contains the node names (as opposed to a pool number). The reason for this will be discussed a little later. These examples also assume at least a four-node parallel environment. If you have more than four nodes, feel free to use more. If you have fewer than four nodes, it's okay to duplicate lines. This example assumes that your file is called host.list, and is in the directory from which you're submitting the parallel job. If either of these conditions are not true, POE will not find the host list file unless you use the -hostfile option (covered later.... one thing at a time!).
The -procs 4 option tells POE to run this command on four nodes. It will use the first four in the host list file.
$ poe hostname -procs 4 202r1n10.hpssl.kgn.ibm.com 202r1n11.hpssl.kgn.ibm.com 202r1n09.hpssl.kgn.ibm.com 202r1n12.hpssl.kgn.ibm.com
What you see is the output from the hostname command run on each of the remote nodes. POE has taken care of submitting the command to each node, collecting the standard output and standard error from each remote node, and sending it back to your workstation. One thing that you don't see is which task is responsible for each line of output. In a simple example like this, it isn't that important. If, however, you had many lines of output from each node, you'd want to know which task was responsible for each line of output. To do that, you use the -labelio option:
$ poe hostname -procs 4 -labelio yes 1:202r1n10.hpssl.kgn.ibm.com 2:202r1n11.hpssl.kgn.ibm.com 0:202r1n09.hpssl.kgn.ibm.com 3:202r1n12.hpssl.kgn.ibm.com
This time, notice how each line starts with a number and a colon. Notice also that the numbering started at 0 (zero). The number is the task id that the line of output came from (it is also the line number in the host list file that identifies the host which generated this output). Now we can use this parameter to identify lines from a command that generates more output.
Try this command:
$ poe cat /etc/motd -procs 2 -labelio yes
You should see something similar to this:
0:******************************************************************************* 0:* * 0:* Welcome to IBM AIX Version 5.1 on 202r1n09.hpssl.kgn.ibm.com * 0:* * 0:******************************************************************************* 0:* * 0:* Message of the Day: Never drink more than 3 Pan * 0:* Galactic Gargle Blasters unless you are a 50 ton maga * 0:* elephant with nemona. * 0:* * 1:******************************************************************************* 1:* * 1:* Welcome to IBM AIX Version 5.1 on 202r1n10.hpssl.kgn.ibm.com * 1:* * 1:******************************************************************************* 1:* * 1:* * 1:* Message of the Day: Never drink more than 3 Pan * 1:* Galactic Gargle Blasters unless you are a 50 ton maga * 1:* elephant with nemona. * 1:* * 1:* * 1:******************************************************************************* 0:* * 0:* * 0:* * 0:*******************************************************************************
The cat command is listing the contents of the file /etc/motd on each of the remote nodes. But notice how the output from each of the remote nodes is intermingled? This is because as soon as a buffer is full on the remote node, POE sends it back to your workstation for display (in case you had any doubts that these commands were really being executed in parallel). The result is the jumbled mess that can be difficult to interpret. Fortunately, we can ask POE to clear things up with the -stdoutmode parameter.
Try this command:
$ poe cat /etc/motd -procs 2 -labelio yes -stdoutmode ordered
You should see something similar to this:
0:******************************************************************************* 0:* * 0:* Welcome to IBM AIX Version 5.1 on 202r1n09.hpssl.kgn.ibm.com * 0:* * 0:******************************************************************************* 0:* * 0:* * 0:* Message of the Day: Never drink more than 3 Pan * 0:* Galactic Gargle Blasters unless you are a 50 ton maga * 0:* elephant with nemona. * 0:* * 0:* * 0:******************************************************************************* 1:******************************************************************************* 1:* * 1:* Welcome to IBM AIX Version 5.1 on 202r1n10.hpssl.kgn.ibm.com * 1:* * 1:******************************************************************************* 1:* * 1:* * 1:* Message of the Day: Never drink more than 3 Pan * 1:* Galactic Gargle Blasters unless you are a 50 ton maga * 1:* elephant with nemona. * 1:* * 1:* * 1:*******************************************************************************
This time, POE holds onto all the output until the jobs either finish or POE itself runs out of space. If the jobs finish, POE displays the output from each remote node together. If POE runs out of space, it prints everything, and then starts a new page of output. You get less of a sense of the parallel nature of your program, but it's easier to understand. Note that the -stdoutmode option consumes a significant amount of system resources, which may affect performance.
By the way, if you're getting tired of typing the same command line options over and over again, you can set them as environment variables so you don't have to put them on the command line. The environment variable names are the same as the command line option names (without the leading dash), but they start with MP_, all in upper case. For example, the environment variable name for the -procs option is MP_PROCS,
and for the -labelio option it's MP_LABELIO. If we set these two variables like this:
$ export MP_PROCS=2 $ export MP_LABELIO=yes
we can then run our /etc/motd program with two processes and labeled output, without specifying either with the poe command.
Try this command;
$ poe cat /etc/motd -stdoutmode ordered
You should see something similar to this:
0:******************************************************************************* 0:* * 0:* Welcome to IBM AIX Version 5.1 on pe03.pok.ibm.com * 0:* * 0:******************************************************************************* 0:* * 0:* * 0:* Message of the Day: Never drink more than 3 Pan * 0:* Galactic Gargle Blasters unless you are a 50 ton maga * 0:* elephant with nemona. * 0:* * 0:* * 0:******************************************************************************* 1:******************************************************************************* 1:* * 1:* Welcome to IBM AIX Version 5.1 on pe03.pok.ibm.com * 1:* * 1:******************************************************************************* 1:* * 1:* * 1:* Message of the Day: Never drink more than 3 Pan * 1:* Galactic Gargle Blasters unless you are a 50 ton maga * 1:* elephant with nemona. * 1:* * 1:* * 1:*******************************************************************************
In the previous example, notice how the program ran with two processes, and the output was labeled.
Now, just so you can see that your environment variable setting lasts for the duration of your session, try running the command below, without specifying the number of processes or labeled I/O.
$ poe hostname 0:202r1n09.hpssl.kgn.ibm.com 1:202r1n10.hpssl.kgn.ibm.com
Notice how the program still ran with two processes and you got labeled output?
Now let's try overriding the environment variables we just set. To do this, we'll use command line options when we run POE. Try running the following command:
$ poe hostname -procs 4 -labelio no 202r1n09.hpssl.kgn.ibm.com 202r1n12.hpssl.kgn.ibm.com 202r1n11.hpssl.kgn.ibm.com 202r1n10.hpssl.kgn.ibm.com
This time, notice that the program ran with four processes and that the output wasn't labeled? No matter what the environment variables have been set to, you can always override them when you run POE.
To show that this was a temporary override of the environment variable settings, try running the following command again, without specifying any command line options.
$ poe hostname 0:202r1n09.hpssl.kgn.ibm.com 1:202r1n10.hpssl.kgn.ibm.com
Once again, the program ran with two processes, and the output was labeled.
All this is fine, but you probably have your own programs that you want to (eventually) run in parallel. We're going to talk about creating parallel programs in a little more detail in Chapter 2, The answer is 42. Right now we'll cover compiling a program for POE. You can compile almost any Fortran, C, or C++ program for execution under POE.
According to The Hitchhiker's Guide to the Galaxy, Vogon poetry is the third worst in the Universe. In fact, it's so bad that the Vogons subject Arthur Dent and Ford Prefect to a poetry reading as a form of torture. Some people may think that compiling a parallel program is just as painful as Vogon poetry, but as you'll see, it's really quite simple.
Before compiling, you should verify that the following has happened:
See IBM Parallel Environment for AIX: MPI Programming Guide for information on compilation restrictions for POE.
To show you how compiling works, we've selected the Hello World program. Here it is in C:
/************************************************************************
*
* Hello World C Example
*
* To compile:
* mpcc -o hello_world_c hello_world.c
*
************************************************************************/
#include<stdlib.h>
#include<stdio.h>
/* Basic program to demonstrate compilation and execution techniques */
int main()
{
printf("Hello, World!\n");
exit(0);
}
c*********************************************************************** c* c* Hello World Fortran Example c* c* To compile: c* mpxlf -o hello_world_f hello_world.f c* c*********************************************************************** c ------------------------------------------------------------------ c Basic program to demonstrate compilation and execution techniques c ------------------------------------------------------------------ c program hello implicit none write(6,*)'Hello, World!' stop end
To compile these programs, you just invoke the appropriate compiler script:
$ mpcc -o hello_world_c hello_world.c $ mpxlf -o hello_world_f hello_world.f ** main === End of Compilation 1 === 1501-510 Compilation successful for file hello_world.f.
mpcc, mpCC, and mpxlf are POE scripts that link the parallel libraries that allow your programs to run in parallel. mpcc, mpCC, and mpxlf are for compiling non-threaded programs. Just as there is a version of the cc command called cc_r, that's used for threaded programs, there is also a script called mpcc_r (and also mpCC_r, and mpxlf_r) for compiling threaded message passing programs. mpcc_r generates thread-aware code by linking in the threaded version of MPI, including the threaded POE utility library. These threaded libraries are located in the same subdirectory as the non-threaded libraries.
All the compiler scripts accept all the same options that the non-parallel compilers do, as well as some options specific to POE. For a complete list of all parallel-specific compilation options, see IBM Parallel Environment for AIX: Operation and Use, Vol. 1.
Running one of the POE compiler scripts, as we've shown you, creates an executable version of your source program that takes advantage of POE. However, before POE can run your program, you need to make sure that it's accessible on each remote node. You can do this by either copying it there, or by mounting the file system that your program is in to each remote node.
Figure 1. Output from mpcc/mpxlf
Here's the output of the C program (threaded or non-threaded):
$ poe hello_world_c -procs 4 Hello, World! Hello, World! Hello, World! Hello, World!
And here's the output of the Fortran program:
$ poe hello_world_f -procs 4 Hello, World! Hello, World! Hello, World! Hello, World!
There are a number of options (command line flags) that you may want to specify when invoking POE. These options are covered in full detail in IBM Parallel Environment for AIX: Operation and Use, Vol. 1 but here are the ones you'll most likely need to be familiar with at this stage.
When you set -procs, you're telling POE how many tasks your program will run. You can also set the MP_PROCS environment variable to do this (-procs can be used to temporarily override it).
The default host list file used by POE to allocate nodes is called host.list. You can specify a file other than host.list by setting the -hostfile or -hfile options when invoking POE. You can also set the MP_HOSTFILE environment variable
to do this (-hostfile and -hfile can be used to temporarily override it).
You can set the -labelio option when invoking POE so that the output from the parallel tasks of your program are labeled by task id. This becomes especially useful when you're running a parallel program and your output is unordered. When you have output that is labeled output, you can easily determine which message the task returned.
You can also set the MP_LABELIO environment variable
to do this (-labelio can be used to temporarily override it).
You can use the -infolevel or -ilevel options to specify the level of messages you want from POE. There are different levels of informational, warning, and error messages, plus several debugging levels. Note that the -infolevel option consumes a significant amount of system resources. Use it with care. You can also set the MP_INFOLEVEL environment variable to do this
(-infolevel and -ilevel can be used to temporarily override it).
The -pmdlog option lets you specify that diagnostic messages should be logged to a file in /tmp on each of the remote nodes of your partition. These diagnostic logs are particularly useful for isolating the cause of abnormal termination. Note that the -infolevel option consumes a significant amount of system resources. Use it with care. You can also set the MP_PMDLOG environment variable to do this
(-pmdlog can be used to temporarily override it).
The -stdoutmode option lets you specify how you want the output data from each task in your program to be displayed. When you set this option to ordered, the output data from each parallel task is written to its own buffer, and later, all buffers are flushed, in task order, to STDOUT. We showed you how this works in some of the examples in this section. Note that using the -infolevel option consumes a significant amount of system resources, which may affect performance. You can also set the MP_STDOUTMODE environment variable to do this (-stdoutmode can be used to temporarily override it).
So far, we've explicitly specified to POE the set of nodes on which to run our parallel application. We did this by creating a list of hosts in a file called host.list, in the directory from which we submitted the parallel job. In the absence of any other instructions, POE selected host names out of this file until it had as many as the number of processes we told POE to use (with the -procs option).
Another way to tell POE which hosts to use is with LoadLeveler. LoadLeveler can be used to manage jobs on a networked cluster of pSeries or RS/6000 workstations, which may or may not include nodes of an IBM RS/6000 SP.
LoadLeveler is used to allocate nodes, one job at a time. This is necessary if your parallel application is communicating directly over the SP Switch. With the -euilib command line option (or the
MP_EUILIB environment variable), you can specify how you want to do message passing. This option lets you specify the message passing subsystem library implementation, IP or User Space (US), that you wish to use. See IBM Parallel Environment for AIX: Operation and Use, Vol. 1 for more information. With LoadLeveler, you can also dedicate the parallel nodes to a single job, so there's no conflict or contention for resources. LoadLeveler allocates nodes from either the host list file, or from a predefined pool, which the System Administrator usually sets up.
So how do you know who's allocating the nodes and where they're being allocated from? First of all, you must always have a host list file
or use the MP_RMPOOL environment variable or -rmpool command line option (unless you're using the MP_LLFILE environment variable or the -llfile command line option).
See IBM Parallel Environment for AIX: Operation and Use, Vol. 1 for more information.
As we've already mentioned, the default for the host list file is a file named host.list in the directory from which the job is submitted. This default may be overridden by the -hostfile command line option or the MP_HOSTFILE environment variable.
For example, the following command:
$ poe hostname -procs 4 -hostfile $HOME/myHosts
would use a file called myHosts, located in the home directory. If the value of the -hostfile parameter does not start with a slash (/), it is taken as relative to the current directory. If the value starts with a slash (/), it is taken as a fully-qualified file name.
For specific examples of how a System Administrator defines pools, see IBM LoadLeveler for AIX: Using and Administering (SA22-7311). Note, however, that there's another way to designate the pool on which you want your program to run. If myHosts didn't contain any pool numbers, you could use the:
Note: If a host list file, named host_list, exists, it will override anything you specify with the MP_RMPOOL environment variable or the -rmpool command line option. You must set MP_HOSTFILE or -hostfile to NULL in order for MP_RMPOOL or -rmpool to work.
For more information about the MP_RMPOOL environment variable or the -rmpool command line option, see IBM Parallel Environment for AIX: Operation and Use, Vol. 1
If the myHosts file contains actual host names, but you want to use the SP Switch directly for communication, LoadLeveler will only allocate the nodes that are listed in myHosts. LoadLeveler keeps track of which parallel jobs are using the switch. Since it allows more than one job at a time to use the switch, LoadLeveler makes sure that only the allowed number of tasks actually use it. If the host list file contains actual host names, but you don't want to use the SP Switch directly for communication, POE allocates the nodes from those listed in the host list file.
As we said before, you can't have both host names and pool IDs in the same host list file.
Your program executes exactly the same way, regardless of whether POE or LoadLeveler allocated the nodes. In the following example, the host list file contains a pool number which causes the job management system to allocate nodes. However, the output is identical to Figure 1, where POE allocated the nodes from the host list file.
$ poe hello_world_c -procs 4 -hostfile pool.list Hello, World! Hello, World! Hello, World! Hello, World!
So, if the output looks the same, regardless of how your nodes are allocated, how do you skeptics know whether LoadLeveler was really used? Well, POE knows a lot that it ordinarily doesn't tell you. If you coax it with the -infolevel option, POE will tell you more information than you ever wanted to know. Read on...
You can control the level of messages you get from POE as your program executes by using the -infolevel option of POE. The default setting is 1 (normal), which says that warning and error messages from POE will be written to STDERR. However, you can use this option to get more information about how your program executes. For example, with -infolevel set to 2, you see a couple of different things. First, you'll see a message that says that POE has contacted LoadLeveler. Following that, you'll see messages that indicate which nodes LoadLeveler passed back to POE for use.
For a description of the various -infolevel settings, see IBM Parallel Environment for AIX: Operation and Use, Vol. 1
Here's the hello world program again:
$poe hello_world_c -resd yes -hostfile ./host.list -procs 2 -labelio yes -infolevel 2
You should see output similar to the following:
INFO: 0031-364 Contacting LoadLeveler to set and query information for interactive job INFO: 0031-380 LoadLeveler step ID is c162n04.ppd.pok.ibm.com.2.0 INFO: 0031-118 Host c162n04.ppd.pok.ibm.com requested for task 0 INFO: 0031-118 Host c162n04.ppd.pok.ibm.com requested for task 1 0:INFO: 0031-724 Executing program: <hello_world_c> 1:INFO: 0031-724 Executing program: <hello_world_c> 0:INFO: 0031-619 32bit(UNKNOWN) `Apr 24 2001 15:42:51 MPCI library was compiled at Tue Apr 24 15:42:51 2001 0: INFO: 0031-425 MPI initialization has completed 1:Hello, world! 0:Hello, world! 0:INFO: 0031-306 pm_atexit: pm_exit_value is 0. 1:INFO: 0031-306 pm_atexit: pm_exit_value is 0. INFO: 0031-656 I/O file STDOUT closed by task 0 INFO: 0031-656 I/O file STDERR closed by task 0 INFO: 0031-656 I/O file STDOUT closed by task 1 INFO: 0031-656 I/O file STDERR closed by task 1 INFO: 0031-251 task 0 exited: rc=0 INFO: 0031-251 task 1 exited: rc=0 INFO: 0031-639 Exit status from pm_respond = 0
With -infolevel set to 2, you also see messages from each node that indicate the executable they're running and what the return code from the executable is. In the example above, you can differentiate between the -infolevel messages that come from POE itself and the messages that come from the remote nodes, because the remote nodes are prefixed with their task ID. If we didn't set -infolevel, we would see only the output of the executable (hello world!, in the previous example), interspersed with POE output from remote nodes.
With -infolevel set to 3, you get even more information. In the following example, we use the host list file that contains host names again (as opposed to a Pool ID), when we invoke POE.
Look at the following output. In this case, POE tells us that it's opening the host list file, the nodes it found in the file (along with their Internet addresses), the parameters to the executable being run, and the values of some of the POE parameters.
$poe hello_world_c -resd yes -hostfile ./host.list -procs 2 -labelio yes -ilevel 3
You should see output similar to the following:
INFO: DEBUG_LEVEL changed from 0 to 1 D1<L1>: Open of file ./host.list successful D1<L1>: mp_euilib = ip D1<L1>: task 0 c162n04.ppd.pok.ibm.com 9.114.98.68 10 D1<L1>: task 1 c162n04.ppd.pok.ibm.com 9.114.98.68 10 D1<L1>: node allocation strategy = 2 INFO: 0031-364 Contacting LoadLeveler to set and query information for interactive job D1<L1>: Job Command String: #@ job_type = parallel #@ environment = COPY_ALL #@ node_usage = shared #@ class = parallel #@ queue INFO: 0031-380 LoadLeveler step ID is c162n04.ppd.pok.ibm.com.3.0 INFO: 0031-118 Host c162n04.ppd.pok.ibm.com requested for task 0 INFO: 0031-118 Host c162n04.ppd.pok.ibm.com requested for task 1 D1<L1>: Entering pm_contact, jobid is 0 D1<L1>: Jobid = 991362386 D1<L1>: Spawning /etc/pmdv3 on all nodes D1<L1>: 1 master nodes D1<L1>: Socket file descriptor for master 0 (c162n04.ppd.pok.ibm.com) is 5 D1<L1>: Leaving pm_contact, jobid is 991362386 0:INFO: 0031-724 Executing program: <hello_world> 1:INFO: 0031-724 Executing program: <hello_world> 1:INFO: DEBUG_LEVEL changed from 0 to 1 1:D1<L1>: In mp_main, mp_main will not be checkpointable 1:D1<L1>: mp_euilib is <ip> 1:D1<L1>: Executing _mp_init_msg_passing() from mp_main()... 0:INFO: DEBUG_LEVEL changed from 0 to 1 0:D1<L1>: In mp_main, mp_main will not be checkpointable 0:D1<L1>: mp_euilib is <ip> 0:D1<L1>: Executing _mp_init_msg_passing() from mp_main()... 1:D1<L1>: cssAdapterType is <1> D1<L1>: init_data for task 1: <9.114.98.68:44278> 0:INFO: 0031-619 32bit(UNKNOWN) `Apr 24 2001 15:42:51 0: 0:D1<L1>: cssAdapterType is <1> D1<L1>: init_data for task 0: <9.114.98.68:44279> INFO: 0031-425 MPI initialization has completed 0:D1<L1>: mp_css_interrupt is <0> 1:D1<L1>: mp_css_interrupt is <0> 1:D1<L1>: Task < 128: MP_SYNC_ON_CONNECT is defaulted to YES. 1: 1:D1<L1>: About to call mpci_connect 0:D1<L1>: Task < 128: MP_SYNC_ON_CONNECT is defaulted to YES. 0: 0:D1<L1>: About to call mpci_connect 0:D1<L1>: Elapsed time for mpci_connect: 0 seconds 1:D1<L1>: Elapsed time for mpci_connect: 0 seconds 1:D1<L1>: _css_init: adapter address = 0 1: 0:D1<L1>: _css_init: adapter address = 0 0: 1:D1<L1>: _css_init: rc from HPSOclk_init is 1 1: 1:D1<L1>: About to call _ccl_init 0:D1<L1>: _css_init: rc from HPSOclk_init is 1 0: 0:D1<L1>: About to call _ccl_init 0:D1<L1>: Elapsed time for _ccl_init: 0 seconds 1:D1<L1>: Elapsed time for _ccl_init: 0 seconds 1:Hello, world! 0:Hello, world! 0:INFO: 0031-306 pm_atexit: pm_exit_value is 0. 1:INFO: 0031-306 pm_atexit: pm_exit_value is 0. INFO: 0031-656 I/O file STDOUT closed by task 0 INFO: 0031-656 I/O file STDERR closed by task 0 D1<L1>: Accounting data from task 1 for source 1: D1<L1>: Accounting data from task 0 for source 0: INFO: 0031-656 I/O file STDOUT closed by task 1 INFO: 0031-656 I/O file STDERR closed by task 1 INFO: 0031-251 task 0 exited: rc=0 INFO: 0031-251 task 1 exited: rc=0 D1<L1>: All remote tasks have exited: maxx_errcode = 0 INFO: 0031-639 Exit status from pm_respond = 0 D1<L1>: Maximum return code from user = 0
The -infolevel messages give you more information about what's happening on the home node, but if you want to see what's happening on the remote nodes, you need to use the -pmdlog option. If you set -pmdlog to a value of yes, a log is written to each of the remote nodes that tells you what POE did while running each task.
If you issue the following command, a file is written in /tmp, of each remote node, called mplog.jobid.taskid,
$ poe hello_world -procs 4 -pmdlog yes
If -infolevel is set to 3 or higher, The job ID will be displayed in the output. If you don't know what the job ID is, it's probably the most recent log file. If you're sharing the node with other POE users, the job ID will be one of the most recent log files (but you own the file, so you should be able to tell).
Here's a sample log file. Note that in this example, all four tasks are running on the same node. For more information about how POE runs with multiple tasks on the same node, see Appendix A, A sample program to illustrate messages.
AIX Parallel Environment pmd3 version @(#) 99/07/11 13:19:38
The ID of this process is 27714
The version of this pmd for version checking is 3100
The hostname of this node is c162n04.ppd.pok.ibm.com
The short hostname of this node is c162n04
The taskid of this task is 0
HOMENAME: c162n04.ppd.pok.ibm.com
USERID: 1079
USERNAME: voe3
GROUPID: 100
GROUPNAME: usr
PWD: /u/voe3/pfc
PRIORITY: 0
NPROCS: 4
PMDLOG: 1
NEWJOB: 0
PDBX: 0
AFSTOKEN: 5765-D93 AIX Parallel Environment
LIBPATH: /usr/lpp/ppe.poe/lib/ip
VERSION (of home node): 3100
JOBID: 991362729
ENVC recv'd
envc: 34
envc is 34
env[0] = _=/usr/bin/poe
env[1] = MANPATH=/usr/lpp/LoadL/full/man:/usr/lpp/LoadL/so/man
env[2] = LANG=en_US
env[3] = LOGIN=voe3
env[4] = PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/u/voe3/bin:/usr/bin/X11:/sbin:/usr/local/bin:.
env[5] = LC__FASTMSG=true
env[6] = CGI_DIRECTORY=/var/docsearch/cgi-bin
env[7] = HISTFILE=/u/voe3/.sh_history/sh_history_22260
env[8] = LOGNAME=voe3
env[9] = MAIL=/usr/spool/mail/voe3
env[10] = LOCPATH=/usr/lib/nls/loc
env[11] = DOCUMENT_SERVER_MACHINE_NAME=localhost
env[12] = USER=voe3
env[13] = AUTHSTATE=compat
env[14] = DEFAULT_BROWSER=netscape
env[15] = SHELL=/bin/ksh
env[16] = ODMDIR=/etc/objrepos
env[17] = DOCUMENT_SERVER_PORT=49213
env[18] = HOME=/u/voe3
env[19] = TERM=aixterm
env[20] = MAILMSG=[YOU HAVE NEW MAIL]
env[21] = PWD=/u/voe3/pfc
env[22] = DOCUMENT_DIRECTORY=/usr/docsearch/html
env[23] = TZ=EST5EDT
env[24] = ENV=/u/voe3/.kshrc
env[25] = A__z=! LOGNAME
env[26] = NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
env[27] = MP_PROCS=4
env[28] = MP_PMDLOG=YES
env[29] = MP_EUIDEVICE=en0
env[30] = MP_PGMMODEL=SPMD
env[31] = MP_ISATTY_STDIN=1
env[32] = MP_ISATTY_STDOUT=1
env[33] = MP_ISATTY_STDERR=1
Couldn't open /etc/poe.limits
MASTERS: 1
TASKS: 4:0:1:2:3
Total number of tasks is 4
Task id for task 1 is 0
Task id for task 2 is 1
Task id for task 3 is 2
Task id for task 4 is 3
TASK_ENV: 0:1 MP_CHILD_INET_ADDR=9.114.98.68 1:1 MP_CHILD_INET_ADDR=9.114.98.68 2:1
Number of environment variables is 1
Environment specific data for task 1, task id 0 :
-- MP_CHILD_INET_ADDR=9.114.98.68
Number of environment variables is 1
Environment specific data for task 2, task id 1 :
-- MP_CHILD_INET_ADDR=9.114.98.68
Number of environment variables is 1
Environment specific data for task 3, task id 2 :
-- MP_CHILD_INET_ADDR=9.114.98.68
Number of environment variables is 1
Environment specific data for task 4, task id 3 :
-- MP_CHILD_INET_ADDR=9.114.98.68
Initial data msg received and parsed
Info level = 1
Before spsec_start...taking timings, current TOD is 991338838
Compatibility authentication method enabled.
Doing ruserok() user validation
User validation complete
About to do user root chk
User root check complete
spkeyfuncs not found, continuing....
ident_match not found, continuing....
task information parsed
STDOUT socket SO_SNDBUF set to 370688
STDOUT socket SO_RCVBUF set to 65160
main thread id is 1 before Setup signal handler for termination.
newjob is 0.
msg read, type is 13
string = <hello_world_c >
SSM_CMD_STR recv'd
command string is <hello_world_c >
0: pm_putargs: argc = 1, k = 1
1: pm_putargs: argc = 1, k = 1
2: pm_putargs: argc = 1, k = 1
3: pm_putargs: argc = 1, k = 1
SSM_CMD_STR parsed
SSM_EXT_DEBUG msg, type is 46
child pipes created
child: pipes successfully duped for task 0
0: MP_COMMON_TASKS is <3:1:2:3>
0: partition id is <991362729>
child: pipes successfully duped for task 1
1: MP_COMMON_TASKS is <3:0:2:3>
1: partition id is <991362729>
child: pipes successfully duped for task 2
2: MP_COMMON_TASKS is <3:0:1:3>
2: partition id is <991362729>
child: pipes successfully duped for task 3
after initgroups (*group_struct).gr_gid = 100
after initgroups (*group_struct).gr_name = usr
3: MP_COMMON_TASKS is <3:0:1:2>
3: partition id is <991362729>
pmd child: core limit is 1073741312, hard limit is 9223372036854775807
pmd child: rss limit is 33554432, hard limit is 9223372036854775807
pmd child: stack limit is 33554432, hard limit is 9223372036854775807
pmd child: data segment limit is 134217728, hard limit is 9223372036854775807
pmd child: cpu time limit is 9223372036854775807, hard limit is 9223372036854775807
pmd child: file size limit is 9223372036854775807, hard limit is 9223372036854775807
0: (*group_struct).gr_gid = 100
0: (*group_struct).gr_name = usr
parent: task 0 forked, child pid is 19822
attach data sent for task 0
1: (*group_struct).gr_gid = 100
1: (*group_struct).gr_name = usr
0: userid, groupid and cwd set!
0: current directory is /u/voe3/pfc
0: about to start the user's program
0: argument list:
argv[0] for task 0 = hello_world_c
argv[1] (in hex) = 0
child: environment for task 0:
task_env[0] = MP_CHILD_INET_ADDR=9.114.98.68
child: common environment data for all tasks:
env[0] = _=/usr/bin/poe
env[1] = MANPATH=/usr/lpp/LoadL/full/man:/usr/lpp/LoadL/so/man
env[2] = LANG=en_US
env[3] = LOGIN=voe3
env[4] = PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/u/voe3/bin:/sbin:/usr/local/bin:.
env[5] = LC__FASTMSG=true
env[6] = CGI_DIRECTORY=/var/docsearch/cgi-bin
env[7] = HISTFILE=/u/voe3/.sh_history/sh_history_22260
env[8] = LOGNAME=voe3
env[9] = MAIL=/usr/spool/mail/voe3
env[10] = LOCPATH=/usr/lib/nls/loc
env[11] = DOCUMENT_SERVER_MACHINE_NAME=localhost
env[12] = USER=voe3
env[13] = AUTHSTATE=compat
env[14] = DEFAULT_BROWSER=netscape
env[15] = SHELL=/bin/ksh
env[16] = ODMDIR=/etc/objrepos
env[17] = DOCUMENT_SERVER_PORT=49213
env[18] = HOME=/u/voe3
env[19] = TERM=aixterm
env[20] = MAILMSG=[YOU HAVE NEW MAIL]
env[21] = PWD=/u/voe3/pfc
env[22] = DOCUMENT_DIRECTORY=/usr/docsearch/html
env[23] = TZ=EST5EDT
env[24] = ENV=/u/voe3/.kshrc
env[25] = A__z=! LOGNAME
env[26] = NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat
env[27] = MP_PROCS=4
env[28] = MP_PMDLOG=YES
env[29] = MP_EUIDEVICE=en0
env[30] = MP_PGMMODEL=SPMD
env[31] = MP_ISATTY_STDIN=1
env[32] = MP_ISATTY_STDOUT=1
env[33] = MP_ISATTY_STDERR=1
0: LIBPATH = /usr/lpp/ppe.poe/lib/ip
crid returned from mkcrid function is 91.
parent: task 1 forked, child pid is 29122
attach data sent for task 1
1: userid, groupid and cwd set!
1: current directory is /u/voe3/pfc
1: about to start the user's program
1: argument list:
argv[0] for task 1 = hello_world_c
argv[1] (in hex) = 0
child: environment for task 1:
task_env[0] = MP_CHILD_INET_ADDR=9.114.98.68
1: LIBPATH = /usr/lpp/ppe.poe/lib/ip
parent: task 2 forked, child pid is 26112
attach data sent for task 2
crid returned from mkcrid function is 275.
parent: task 3 forked, child pid is 14498
attach data sent for task 3
2: (*group_struct).gr_gid = 100
2: (*group_struct).gr_name = usr
2: userid, groupid and cwd set!
2: current directory is /u/voe3/pfc
2: about to start the user's program
2: argument list:
argv[0] for task 2 = hello_world_c
argv[1] (in hex) = 0
child: environment for task 2:
task_env[0] = MP_CHILD_INET_ADDR=9.114.98.68
2: LIBPATH = /usr/lpp/ppe.poe/lib/ip
crid returned from mkcrid function is 515.
3: (*group_struct).gr_gid = 100
3: (*group_struct).gr_name = usr
3: userid, groupid and cwd set!
3: current directory is /u/voe3/pfc
3: about to start the user's program
3: argument list:
argv[0] for task 3 = hello_world_c
argv[1] (in hex) = 0
child: environment for task 3:
task_env[0] = MP_CHILD_INET_ADDR=9.114.98.68
3: LIBPATH = /usr/lpp/ppe.poe/lib/ip
crid returned from mkcrid function is 771.
select: rc = 3
pulse is on, curr_time is 991338838, send_time is 0, select time is 600
pulse sent at 991338838 count is 0
0: pmd parent: cntl pipe read OK:
0: pmd parent: type: 26, srce: 0, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 19822
1: pmd parent: cntl pipe read OK:
1: pmd parent: type: 26, srce: 1, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 29122
2: pmd parent: cntl pipe read OK:
2: pmd parent: type: 26, srce: 2, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 26112
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 5
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
in pmd select, SSM_read ok, SSM_type=34.
pulse received at 991338838 received count is 0
0: pmd parent: cntl pipe read OK:
0: pmd parent: type: 47, srce: 0, dest: -2, bytes: 5
parent: childs version is 3100.
parent: home node version is 3100.
parent: this pmd version is 3100.
1: pmd parent: cntl pipe read OK:
1: pmd parent: type: 47, srce: 1, dest: -2, bytes: 5
parent: childs version is 3100.
parent: home node version is 3100.
parent: this pmd version is 3100.
2: pmd parent: cntl pipe read OK:
2: pmd parent: type: 47, srce: 2, dest: -2, bytes: 5
parent: childs version is 3100.
parent: home node version is 3100.
parent: this pmd version is 3100.
3: pmd parent: cntl pipe read OK:
3: pmd parent: type: 26, srce: 3, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 14498
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
3: pmd parent: cntl pipe read OK:
3: pmd parent: type: 47, srce: 3, dest: -2, bytes: 5
parent: childs version is 3100.
parent: home node version is 3100.
parent: this pmd version is 3100.
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
1: pmd parent: cntl pipe read OK:
1: pmd parent: type: 23, srce: 1, dest: -1, bytes: 18
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 2
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
2: pmd parent: cntl pipe read OK:
2: pmd parent: type: 23, srce: 2, dest: -1, bytes: 18
3: pmd parent: cntl pipe read OK:
3: pmd parent: type: 23, srce: 3, dest: -1, bytes: 18
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
0: pmd parent: cntl pipe read OK:
0: pmd parent: type: 23, srce: 0, dest: -1, bytes: 18
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
in pmd select, SSM_read ok, SSM_type=29.
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
0: pmd parent: cntl pipe read OK:
0: pmd parent: type: 44, srce: 0, dest: -1, bytes: 3
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 3
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
1: pmd parent: cntl pipe read OK:
1: pmd parent: type: 44, srce: 1, dest: -1, bytes: 3
2: pmd parent: cntl pipe read OK:
2: pmd parent: type: 44, srce: 2, dest: -1, bytes: 3
3: pmd parent: cntl pipe read OK:
3: pmd parent: type: 44, srce: 3, dest: -1, bytes: 3
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
in pmd select, SSM_read ok, SSM_type=3.
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 3
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
0: pmd parent: cntl pipe read OK:
0: pmd parent: type: 26, srce: 0, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 19822
1: pmd parent: cntl pipe read OK:
1: pmd parent: type: 26, srce: 1, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 29122
2: pmd parent: cntl pipe read OK:
2: pmd parent: type: 26, srce: 2, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 26112
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
3: pmd parent: cntl pipe read OK:
3: pmd parent: type: 26, srce: 3, dest: -2, bytes: 6
parent: SSM_CHILD_PID: 14498
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 3
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
pmd parent: STDOUT read OK for task 0
0: STDOUT: Hello, world!
pmd parent: STDOUT read OK for task 1
1: STDOUT: Hello, world!
pmd parent: STDOUT read OK for task 2
2: STDOUT: Hello, world!
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 2
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
0: pmd parent: cntl pipe read OK:
0: pmd parent: type: 17, srce: 0, dest: -1, bytes: 2
1: pmd parent: cntl pipe read OK:
1: pmd parent: type: 17, srce: 1, dest: -1, bytes: 2
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 2
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
2: pmd parent: cntl pipe read OK:
2: pmd parent: type: 17, srce: 2, dest: -1, bytes: 2
pmd parent: STDOUT read OK for task 3
3: STDOUT: Hello, world!
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
3: pmd parent: cntl pipe read OK:
3: pmd parent: type: 17, srce: 3, dest: -1, bytes: 2
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 1
pulse is on, curr_time is 991338838, send_time is 991338838, select time is 600
in pmd select, SSM_read ok, SSM_type=5.
select is not timeout, select_tval->tv_sec=600,select_tval->tv_usec=0.
select: rc = 12
pulse is on, curr_time is 991338844, send_time is 991338838, select time is 600
pmd send SSM_IO_CLOSED to poe for stdout_open
0: count = 0 on stderr
pmd send SSM_IO_CLOSED to poe for stderr_open
pmd send SSM_IO_CLOSED to poe for stdout_open
1: count = 0 on stderr
pmd send SSM_IO_CLOSED to poe for stderr_open
pmd send SSM_IO_CLOSED to poe for stdout_open
2: count = 0 on stderr
pmd send SSM_IO_CLOSED to poe for stderr_open
pmd send SSM_IO_CLOSED to poe for stdout_open
3: count = 0 on stderr
pmd send SSM_IO_CLOSED to poe for stderr_open
select is not timeout, select_tval->tv_sec=594,select_tval->tv_usec=0.
in pmd signal handler for task 3, signal 20
3: wait status is 00000000
Exiting child for task 3, PID: 14498
err_data for task 3 is 0
1: wait status is 00000000
Exiting child for task 1, PID: 29122
err_data for task 1 is 0
0: wait status is 00000000
Exiting child for task 0, PID: 19822
err_data for task 0 is 0
2: wait status is 00000000
Exiting child for task 2, PID: 26112
err_data for task 2 is 0
in pmd signal handler, wait returned -1...
parent: child exited and all pipes closed for all tasks
err_data for task 0 is 0
err_data for task 1 is 0
err_data for task 2 is 0
err_data for task 3 is 0
pmd_exit reached!, exit code is 0
Appendix A, A sample program to illustrate messages includes an example of setting -infolevel to 6, and explains the important lines of output.