NCSA Xeon Cluster: Programming Environment

Compilers

Introduction
Compiler Commands
Vendor Documentation
Useful Compiler/Linker Options

Libraries and Application Software

Intel's Math Kernel Library (MKL)
Locally Installed Software
Hierarchical Data Format (HDF) Library

1. Compilers

1.1 Introduction

NCSA supports the Intel compilers and ChaMPIon/Pro MPI on the Tungsten cluster. The GNU compilers are also available, but they generally produce slower code. Since the cluster uses 32-bit Xeon processors, each process may access no more than 2 Gbytes of memory. A batch job is also subject to a limit of 3 Gbytes (size of physical memory) per dual-processor node.

1.2 Compiler Commands

To compile and link a serial program in Fortran and C, enter:

  ifort myprog.f
  icc myprog.c

To compile and link an MPI program in Fortran 77, Fortran 90, C and C++, use the following commands:

  cmpifc myprog.f
  cmpif90c myprog.f90
  cmpicc myprog.c
  cmpic++ myprog.C

See the man pages for the above commands for information about options.

To run an MPI program, use the cmpirun command in a batch job. See Running Jobs for more information.

Notes:

The MPICH commands (mpicc, mpif90, mpirun, etc.) are not supported with ChaMPIon/Pro.
The default behavior of icc for the C preprocessor (CPP) macros is to appear as if it is the GNU gcc 3.0 compiler. To list what macros are being passed to the preprocessor, use the -v option. Use the -no-gcc option to turn off these GNU CPP macros.
If you have problems with ChaMPIon/Pro MPI, try using generic MPICH.
Large File Support: Tungsten uses Large File Support (LFS) for files larger than 2 GB. The compiler options to use for LFS are:
```
      -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE64
```

1.3 Vendor Documentation

1.4 Useful Compiler/Linker Options

Compatibility Options

-auto: puts static variables (including arrays) on the run-time stack which is limited by the stacksize limit to a maximum of 2 Gbytes. Without this option, the code and all static data are limited to about 1 Gbyte.
-assume byterecl: Many compilers assume that in the OPEN statement, the record length (recl=) of direct-access unformatted files is specified in bytes. With the Intel 8.X and 9.X compilers the default is in four-byte words. If you use the option above, it assumes the record length is specified in bytes.
-fpp[n]: runs the Fortran preprocessor on source files prior to compilation.
-Dname[=value]: specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-fpp).
-w90, -w95: suppress messages about use of non-standard Fortran
-i8: set default KIND of integer variables is 8
-integer_size size: (8.X/9.X compilers) specifies the default size of integer and logical variables (size: 16, 32, 64)
-r8: set default size of REAL to 8 bytes
-real_size size: (8.X/9.X compilers) specify the size of REAL and COMPLEX declarations, constants, functions, and intrinsics (size: 32, 64, 128)
-save: save all variables (static allocation)
-Vaxlib: (Fortran) link to the portability library libpepcf90.a. This includes functions such as dtime, etime, rand, exit, etc. For a complete list of these functions see Chapter 2, "Portability Functions" in the Intel Fortran Libraries Reference Manual. (automatically included with 8.X/9.X compilers)

Little endian to big endian conversion

This is intended for Fortran unformatted input/output operations. This enables the development and processing of files with big-endian data organization on the Intel processors, which use little endian. It is implemented as an environment variable F_UFMTENDIAN. The syntax is:

  csh/tcsh: setenv  F_UFMTENDIAN u[,u]...
  sh/bash:  export  F_UFMTENDIAN=u[,u]...

where u[,u]... are unit numbers of those files that are to be treated as big endian.

Detecting Programming Errors

-g: produce symbolic debug information in object file (implies -O0)
-CB: array bounds checking
-fpen: (8.X/9.X compilers) specifies behavior on floating point exceptions (n = 0, 1 or 3)
-zero: implicitly initialize all data to zero
-u: equivalent to having "IMPLICIT NONE" throughout the code
-traceback: (Fortran 8.X/9.X compiler) allow the display of source file traceback information at runtime when a severe error occurs.

Optimization

-O2: enable optimizations (DEFAULT)
-O1: optimize for maximum speed, but disable some optimizations which increase code size for a small speed benefit
-O3: enable -O2 plus more aggressive optimizations that may not improve performance for all programs
-O0: disable optimizations
-O: same as -O2
-ipo: enable multi-file IP optimizations (between files)
-mp: maintain floating point precision (disables some optimizations)
-mp1: improve floating-point precision (speed impact is less than -mp)
-tpp7: optimize for Intel Pentium 4/Xeon processors (default)
-xW: enable SSE/SSEII vectorization
-vec_report3: tells about successful and unsuccessful vectorization

Other Options

-v

display verbose information about each compiler phase.

-V

displays the compiler version information

-sox

Tells the compiler to save the compiler options and version number in the executable. The default is -no-sox. To retrieve the information:

        strings --all a.out | grep comment:

Notes

Caution: Any code built with the -align option cannot make calls to system or compiler libraries that use long long, double, or long double in their interfaces.

2. Libraries and Application Software

2.1 Intel Math Kernel Library

The Math Kernel Library (MKL) contains the complete set of functions from the basic linear algebra subprograms (BLAS), the extended BLAS (sparse), and the complete set of LAPACK routines. In addition, there is a set of fast Fourier transforms in single- and double-precision, real and complex data types with both Fortran and C interfaces. The library also includes the cblas interfaces, which allow the C programmer to access all the functionality of the BLAS without considering C-Fortran issues.

See Intel Math Kernel Library for detailed information, including release notes and reference manual.

The libraries are installed under ${MKL_HOME}/lib/32. To link to the libraries:

For blas: -L${MKL_HOME}/lib/32 -lmkl -lguide -lpthread
For lapack: -L${MKL_HOME}/lib/32 -lmkl_lapack -lmkl -lguide -lpthread

Note: Some MKL routines can run in parallel (using OpenMP) if the environment variable $OMP_NUM_THREADS is set to the number of threads. To ensure only one thread is being used in MKL routines set the value of $MKL_SERIAL to YES (this is the default).

2.2 Locally Installed Software

NCSA has a variety of applications software available in different areas of concentration, including chemistry, computational fluid dynamics, mathematics, solid mechanics, and visualization. An online list is maintained in the Software Repository. The list is organized by system and by area of science. If you have any questions about available software, contact the appropriate software coordinator or the staff in Consulting Services. The /usr/apps directory contains third party applications with most located in subdirectories according to the applications areas.

2.3 Hierarchical Data Format (HDF) Library

HDF is a library and platform independent data format for the storage and exchange of scientific data. It includes Fortran and C calling interfaces, and utilities for analyzing and converting HDF data files. There are two HDF formats, HDF (4.x and previous releases) and HDF5. These formats are completely different and NOT compatible.

HDF/HDF5 is available on NCSA HPC systems via the TeraGrid Coordinated TeraGrid Software and Services (CTSS). Use SoftEnv for information on accessing the software. Information on support is available at the HDF Support Issues page.

The HDF Home Page has detailed information on HDF and HDF5, including documentation, tutorials, examples, and FAQs.

Programming Environment on NCSA Xeon Cluster