User Information Home
Compute Resources
Software
Data
Security
Allocations
Consulting
Training
Strategic Applications Program

mpiP at NCSA

The mpiP library for MPI profiling is available on NCSA's Tungsten (Xeon) and Mercury (TeraGrid Itanium 2) clusters and on the NCSA SGI Altix (Cobalt).

mpiP is an MPI profiling library from Lawrence Livermore National Laboratory that provides an easy-to-use way of measuring overall message-passing statistics for your MPI application. mpiP is not a tracing or logging library that records information about each individual message sent between processors: instead, it collects statistics about communication volume and wall clock time spent within MPI for your application. Additionally, mpiP records the call site information for MPI calls - this allows you to identify the exact spot within your application where measured MPI time occurred and can be very helpful to isolate the areas of your application that are dominated by communication time.

The output of mpiP is a single text file with the suffix ".mpiP" written from MPI task 0 at the time your application calls MPI_Finalize(). Please refer to the mpiP documentation listed below or to the mpiP project web site for examples of the information that the mpiP output file contains.

---

Basic Use at NCSA

mpiP is installed on NCSA's Tungsten cluster and on Cobalt in the directory:

/usr/apps/tools/mpip

mpiP is installed on NCSA's Mercury cluster in the directory:

/usr/projects/perftools/mpip

To use mpiP, you'll only need to relink your application with the mpiP profiling library. You should select the correct library based on the MPI implementation you are using by including one of the subdirectories lib/cmpi, lib/mpich, or lib/mpt in the library search path used at link time (-L option). You'll also want to include the GNU libbfd and libiberty libraries that mpiP uses. You do not need to include the library -lintl at NCSA. On Cobalt, you should also include the math library libm.

Note: call site identification on Mercury and Cobalt requires the Hewlett-Packard version of the library "libunwind".

To link your program with mpiP on Mercury, you should include the following library in your link line and you must use the full path name:

/usr/projects/perftools/libunwind/lib/libunwind.a

To link your program with mpiP on Cobalt, you should include the following library in your link line and you must use the full path name:

/usr/apps/tools/libunwind/lib/libunwind.a

If you'd like mpiP to translate program addresses to source code locations within your application, you'll want to include the option -g when you compile your program. If you don't include this option, mpiP will still work, but will only display raw program counter values where MPI calls occurred.

Here are example link lines that can be used to link the C program myprog for mpiP profiling:

ChaMPIon/Pro

cmpicc -o myprog myprog.o \
	-L/usr/apps/tools/mpip/lib/cmpi -lmpiP -lbfd -liberty

MPICH (including MPICH-GM)

mpicc -o myprog myprog.o \
	-L/usr/apps/tools/mpip/lib/mpich -lmpiP -lbfd -liberty

MPICH (Mercury with H-P's libunwind)

mpicc -o myprog myprog.o \
	-L/usr/projects/perftools/mpip/lib/mpich -lmpiP -lbfd -liberty \
	/usr/projects/perftools/libunwind/lib/libunwind.a

MPT (Cobalt with H-P's libunwind)

icc -o myprog myprog.o \
	-L/usr/apps/tools/mpip/lib/mpt -lmpiP -lbfd -liberty -lm \
	/usr/apps/tools/libunwind/lib/libunwind.a -lmpi

Once you have linked your program with mpiP, you can run your application as you normally would, using either mpirun or cmpirun as appropriate.

---

To Learn More

The mpiP project web site provides a great deal of information about the software as well as examples of use. You can also find documentation in the subdirectory doc within the mpiP installation directory.