NCSA SGI Altix System: Programming Environment

Compilers

Introduction
Compiler Commands
Vendor Documentation
Useful Compiler/Linker Options

Libraries and Application Software

Intel's Math Kernel Library (MKL)
SGI's Scientific Computing Software Library (scsl)
Locally Installed Software
Hierarchical Data Format (HDF) Library

1. Compilers

1.1 Introduction

NCSA supports the Intel compilers on the SGI Altix system. (The GNU compilers are also available, but they generally produce slower code.)

1.2 Compiler Commands

To compile and link a serial program in Fortran, C, and C++ enter:

  ifort myprog.f
  icc myprog.c
  icpc myprog.cc

To compile and link an OpenMP program in Fortran and C, use the -openmp option:

  ifort -openmp myprog.f
  icc -openmp myprog.c

Before you run an OpenMP program, set the environment variable OMP_NUM_THREADS to the number of thtreads you want. For example, to run program a.out interactively with two threads:

  setenv OMP_NUM_THREADS 2
  ./a.out

Before you run an MPI OpenMP hybrid program, set the environment variable MPI_OPENMP_INTEROP to 1 or "true" so that OpenMP threads are allowed to use their own processors along with each MPI rank.

  setenv OMP_NUM_THREADS 2
  setenv MPI_OPENMP_INTEROP true
  mpirun -np 4 ./a.out_openmp

The following environment variables may also be useful in running your OpenMP programs:

OMP_SCHEDULE	Sets the schedule type and (optionally) the chunk size for DO and PARALLEL DO loops declared with a schedule of RUNTIME. The default is STATIC.
KMP_LIBRARY	sets the run-time execution mode. The default is `throughput`, but it can be set to `turnaround` so worker threads do not yield while waiting for work.
KMP_STACKSIZE	Sets the number of bytes to allocate for the stack of each parallel thread. You can use a suffix k, m, or g to specify kilobytes, megabytes or gigabytes. The default is 4m.

To compile and link an MPI program in Fortran, C, and C++ add -lmpi at the end of the compile command:

  ifort myprog.f -lmpi
  icc myprog.c -lmpi
  icpc myprog.cc -lmpi -lmpi++

When you run an MPI program, you need to specify the number of processors with the -np option. For example, to run program a.out on two processors:

  mpirun -np 2 a.out

Notes:

The MPICH compile commands (mpicc, mpif90, etc.) are not supported on the SGI Altix system.
The default behavior of icc for the C preprocessor (CPP) macros is to appear as if it is the GNU gcc 3.0 compiler. To list what macros are being passed to the preprocessor, use the -v option. Use the -no-gcc option to turn off these GNU CPP macros.
For information about running OpenMP and MPI programs in a batch job, see Running Jobs.

1.3 Vendor Documentation

1.4 Useful Compiler/Linker Options

Compatibility Options

-fpp[n]: runs the Fortran preprocessor on source files prior to compilation.
-assume byterecl: Many compilers assume that in the OPEN statement, the record length (recl=) of direct-access unformatted files is specified in bytes. With the Intel 8.X/9.X compilers the default is in four-byte words. If you use the option above, it assumes the record length is specified in bytes.
-Dname[=value]: specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).
-w90, -w95: suppress messages about use of non-standard Fortran
-i8: set default KIND of integer variables is 8
-integer_size size: (8.X/9.X compilers) specifies the default size of integer and logical variables (size: 16, 32, 64)
-r8: set default size of REAL to 8 bytes
-real_size size: (8.X/9.X compilers) specify the size of REAL and COMPLEX declarations, constants, functions, and intrinsics (size: 32, 64, 128)
-save: save all variables (static allocation)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

  csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
  sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

-g: produce symbolic debug information in object file (implies -O0)
-CB: array bounds checking
-fpen: (8.X/9.X compilers) specifies behavior on floating point exceptions (n = 0, 1 or 3)
-zero: implicitly initialize all data to zero
-u: equivalent to having "IMPLICIT NONE" throughout the code
-traceback: (Fortran 8.X/9.X compiler) allow the display of source file traceback information at runtime when a severe error occurs.

Optimization

-O2: enable optimizations (DEFAULT)
-O1: optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit
-O3: enable -O2 plus more aggressive optimizations that may not improve performance for all programs
-O0: disable optimizations
-O: same as -O2
-ipo: enable multi-file IP optimizations (between files)
-ftz: flush denormal values to zero when the application is in the gradual underflow mode. By default, the compiler lets results to gradually underflow. Flushing the denormal values to zero may improve performance of your application. Use this option if the denormal values are not critical to application behavior.
-mp: maintain floating point precision (disables some optimizations)
-mp1: improve floating-point precision (speed impact is less than -mp)
-tpp2: Optimize for Intel Itanium 2 processors. (default)

Parallel Programming Options

-openmp: tells the parallelizer to generate multithreaded code based on OpenMP directives.
-parallel: tells the auto-parallelizer to generate multithreaded code for loops that can be safely executed in parallel. To use this option, you must also specify -O2 or -O3.
-lmpi: link with MPI libraries. This is required for MPI programs. It must be specified at the end of the compile command.

Other Options

-v

display verbose information about each compiler phase.

-V

Displays the compiler version information

-sox

Tells the compiler to save the compiler options and version number in the executable. The default is -no-sox. To retrieve the information:

        strings --all a.out | grep comment:

2. Libraries and Application Software

2.1 Intel Math Kernel Library

The Math Kernel Library (MKL) contains the complete set of functions from the basic linear algebra subprograms (BLAS), the extended BLAS (sparse), and the complete set of LAPACK routines. In addition, there is a set of fast Fourier transforms in single- and double-precision, real and complex data types with both Fortran and C interfaces. The library also includes the cblas interfaces, which allow the C programmer to access all the functionality of the BLAS without considering C-Fortran issues.

See Intel Math Kernel Library for detailed information, including release notes and reference manual.

The libraries are installed under ${MKL_HOME}/lib/64. To link to the libraries:

For blas: -L${MKL_HOME}/lib/64 -lmkl -lguide -lpthread
For lapack: -L${MKL_HOME}/lib/64 -lmkl_lapack -lmkl -lguide -lpthread

Note: Some MKL routines can run in parallel (using OpenMP) if the environment variable $OMP_NUM_THREADS is set to the number of threads. To ensure only one thread is being used in MKL routines set the value of $MKL_SERIAL to YES (this is the default).

2.2 Scientific Computing Software Library

SGI's Scientific Computing Software Library contains routines to do the following:

signal processing (FFT, convolution, correlation)
linear equation solvers for real and complex sparse systems.
random number generators
vector-vector, matrix-vector and matrix-matrix linear algebra subprograms
LAPACK routines

See the scsl man page or SCSL User's Guide (PDF) for more information.

To link SCSL serial: -lscs
To link SCSL threaded: -lscs_mp

The SGI Scientific Computing Software Library also provides a distributed shared memory library (SDSM) that contains the following routines:

BLACS (Basic Linear Algebra Communication Subprograms)
SCALAPACK (Scalable LAPACK)
PBLAS (Parallel BLAS)

See the sdsm, blacs, scalapack man pages for more information.

For serial SDSM and SCSL: -lsdsm -lscs¹
For threaded SDSM and SCSL: -lsdsm_mp -lscs_mp²

¹ MKL can be used with the SDSM library. Replace -lscs with the libraries for MKL Lapack above.
² Using the threaded version of SDSM and SCSL with an MPI code enables hybrid parallelism. Users should carefully review the MPT documentation (man mpi) to determine optimal mechinism for launching such jobs.

2.3 Locally Installed Software

NCSA has a variety of applications software available in different areas of concentration, including chemistry, computational fluid dynamics, mathematics, solid mechanics, and visualization. An online list is maintained in the Software Repository. The list is organized by system and by area of science. If you have any questions about available software, contact the appropriate software coordinator or the staff in Consulting Services. The /usr/apps directory contains third party applications with most located in subdirectories according to the applications areas.

2.3 Hierarchical Data Format (HDF) Library

HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. There are two HDF formats, HDF (4.x and previous releases) and HDF5. These formats are completely different and NOT compatible.

HDF/HDF5 is available on NCSA HPC systems via the TeraGrid Coordinated TeraGrid Software and Services (CTSS). Use SoftEnv for information on accessing the software. Information on support is available at the HDF Support Issues page.

The HDF Home Page has detailed information on HDF and HDF5, including documentation, tutorials, examples, and FAQs.

Programming Environment on NCSA SGI Altix System