View figure.
Often, once a program is running correctly, you'd like to keep tabs on its progress. Frequently, in a sequential program, this is done by printing to standard output. However, if you remember from Chapter , Hitching a lift on the Vogon constructor ship, standard output from all the tasks is interleaved, and it is difficult to follow the progress of just one task. If you set the MP_STDOUTMODE environment variable to ordered, you can't see how the progress of one task relates to another. In addition, normal output is not a blocking operation. This means that a task that writes a message will continue processing so that by the time you see the message, the task is well beyond that point. This makes it difficult to understand the true state of the parallel application, and it's especially difficult to correlate the states of two tasks from their progress messages. One way to synchronize the state of a parallel task with its output messages is to use the Program Marker Array (pmarray).
Note: If you are unfamiliar with the Program Marker Array, you may find it helpful to refer to IBM Parallel Environment for AIX: Operation and Use, Vol. 1 for more information.
The Program Marker Array consists of two components: the display function, pmarray, and the instrumentation call, mpc_marker. When pmarray is running, it shows a display that looks like Figure 2, below.
Figure 2. Program Marker Array
View figure.
Each row of colored squares is associated with one task, which can change the color of any of the lights in its row with the mpc_marker call. The declaration looks like this in Fortran:
MP_MARKER (INTEGER LIGHT, INTEGER COLOR, CHARACTER STRING)
And it looks like this in C:
void mpc_marker(int light, int color, char *str)
This call accepts two integer values and a character string. The first parameter, light, controls which light in the pmarray is being modified. You can have up to 100 lights for each task. The second parameter, color, specifies the color to which you are setting the light. There are 100 colors available. The third parameter is a string of up to 80 characters that is a message shown in the text area of the pmarray display.
Before you start the parallel application, you need to tell pmarray how many lights to use, as well as how many tasks there will be. You do this with the MP_PMLIGHTS and the MP_PROCS environment variables.
$ export MP_PROCS=4 $ export MP_PMLIGHTS=16
If the parallel application is started from an X-Windows environment where pmarray is running, the output square of pmarray, for the task that made the call in the position specified by the light parameter, changes to the color specified by the color parameter. The character string is displayed in a text output region for the task. In addition to providing a quick graphical representation of the progress of the application, the output to pmarray is synchronized with the task that generates it. The task will not proceed until it has been informed that the data has been sent to pmarray. This gives you a much more current view of the state of each task.
The example below shows how you can use pmarray to track the progress of an application. This program doesn't do anything useful, but there's an inner loop that's executed 16 times, and an outer loop that is executed based on an input parameter. On each pass through the inner loop, the mpc_marker call is made to color each square in the task's pmarray row according to the color of the index for the outer loop. On the first pass through the inner loop, each of the 16 squares will be colored with color 0. On the second pass, they will be colored with color 1. On each pass through the outer loop, the task will be delayed by the number of seconds equal to its task number. Thus, task 0 will quickly finish, but task 4 will take a while to finish. The color of the squares for a task indicate how far they are through the outer loop. The square that is actually changing color is the position in the inner loop. In addition, a text message is updated on each pass through the outer loop.
/************************************************************************
*
* Demonstration of use of pmarray
*
* To compile:
* mpcc -g -o use_pmarray use_pmarray.c
*
************************************************************************/
#include<stdlib.h>
#include<stdio.h>
#include<mpi.h>
#include<time.h>
int main(int argc, char **argv)
{
int i, j;
int inner_loops = 16, outer_loops;
int me;
char buffer[256];
time_t start, now;
MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD,&me;);;
if(argc>1) outer_loops = atoi(argv[1]);
if(outer_loops<1) outer_loops = 16;
for(i=0;i<outer_loops;i++)
{
/* Create message that will be shown in pmarray text area */
sprintf(buffer,"Task %d performing loop %d of %d",me,i,outer_loops);
printf("%s\n",buffer);
for(j=0;j<inner_loops;j++)
{
/* pmarray light shows which outer loop we are in */
/* color of light shows which inner loop we are in */
/* text in buffer is created in outer loop */
mpc_marker(i,5*j,buffer);
}
/* Pause for a number of seconds determined by which */
/* task this is. sleep(me) cannot be used because */
/* underlying communication mechanism uses a regular */
/* timer interrupt that interrupts the sleep call */
/* Instead, we'll look at the time we start waiting */
/* and then loop until the difference between the */
/* time we started and the current time is equal to */
/* the task id */
time(&start);
time(&now);
while(difftime(now,start)<(double)me)
{
time(&now);
}
}
MPI_Finalize();
return 0;
}
Before running this example, you need to start pmarray, telling it how many lights to use. You do this with the MP_PMLIGHTS environment variable.
In our example, if we wanted to run the program with eight outer loops, we would set MP_PMLIGHTS to 8 before running the program.
Although it's not as flexible or extensible as print statements, pmarray allows you to send three pieces of information back to the home node for presentation. The three pieces are the light number, the color, and the text string. It also ensures that the presentation is synchronized as closely to the task state as possible. We recommend that if you use pmarray for debugging, you define a consistent strategy for your application and stick with it. For example, you may want to use color to indicate state (initializing, active, disabled, terminated), and light number to indicate module or subsystem. You can configure pmarray with as many lights as will fit on the display (up to 100 per task).