>
 
NCSA Home
Contact Us | Intranet | Search

Transition from the Origin2000

 

There are many issues to consider when making the transition from the SGI Origin 2000 to the IBM p690. They range from the obvious; all progams will need to be recompiled and all batch scripts will need to be rewritten, to the subtle; dtime needs to be replaced by dtime_. This document provides a source for information on these issues and will be extended as new issues arise.

Contents

System Comparison

Hardware Overview
Memory Architecture
Software

File Handling

Transferring files from the Origin 2000 to IBM p690
Binary Compatibility

Programming / Compilers

Compiler Names v Compiler Flags
Compiling OpenMP Programs
f90 file suffix
Automatic Promotion to REAL*8
Useful FORTRAN Compiler flags
Porting Programs to XL FORTRAN
Porting Programs from GNU C/C++
Default Addressing mode
Data Segment Size
Optimization Levels
Math Libraries
Fast Math
Automatic Parallelization
Timing Routines
Tool to check makefiles
Fortran End of File Handling

Running Programs

Running MPI Programs
Running OpenMP Programs

Batch System

Comparison of Batch Systems
Automatic Saving of Files to Mass Storage
Batch System Command Line Directives
Comparison of Batch Scripts


  1. System Comparison

  2. Hardware Overview
       IBM p690  SGI Origin 2000
     Processors  Power4  R10000
     Number of Processors  384  1512
     Speed  1.3 GHz  0.195 & 0.25 GHz
     Total Memory  1.5 TB  0.614 TB
     Peak Performance  2 TFlops  0.66 TFlops


    Memory Architecture
       IBM p690  SGI Origin 2000
     Level 1 (Data)  32 Kbytes  32 Kbytes
     Level 1 (Instruction)  64 Kbytes  32 Kbytes
     Level 2  1.44 Mbytes  4 Mbytes
     Level 3  128 MBytes  
     Main Memory  64-256 Gbytes  14-74 Gbytes


    Software Overview
       IBM p690  SGI Origin 2000
     OS  AIX 5.1  IRIX 6.5
     Compilers  XL, Visual Age  MIPSpro
     Batch System  Loadleveler  Lsbatch
     Main Math Libraries  ESSL, MASS  SCSL
     Programming Paradigms  MPI, OpenMP, SHMEM  NativeMP, OpenMP, MPI, PVM, SHMEM
     MPI Runtime Environment  poe  mpirun


  3. File Handling

    • Transferring Files
      There are three ways you can transfer files from modi4 to p690:
      1. You can use scp with the "-r" option to transfer all files from modi4 directly to copper.

      2. You can tar up your modi4 home directory and transfer by kerberos ftp to p690.

      3. You can tar up your home directory and transfer to unitree for example msscmd "tar cvf modi4.tar .". Then from copper you can transfer files back from unitree for example msscmd "tar xvf modi4.tar".


    • Binary Compatibility

      The IBM p690 and SGI Origin2000 use the same format for Fortran unformatted files, so unformatted files written on the Origin can be read on the IBM p690 and visa versa.


  4. Programming / Compilers

    • Compiler names v compiler flags
      The Origin and IBM p690 compilers are very different.  Where the Origin uses flags for different parallel programming (MPI, openMP etc.), the p690 uses different compiler names.  For example to compile an MPI program:
    • Origin   % f77 mpi.f -lmpi

      IBM p690  % mpxlf mpi.f

      A full list of compiler names and uses can be found at :

      p690 Compilers


    • Compiling OpenMP Programs

      To compile an openMP program you need to use the the complier with the _r suffix and include the flag -qsmp=omp:

      Origin   % f77 -mp openmp.f

      p690  % xlf_r -qsmp=omp openmp.f


    • f90 suffix

      By default the IBM compilers do not associate the .f90 suffix with FORTRAN.

      Cu12:~/consult/codes/single106% xlf90 hello.f90

      xlf90: 1501-218 file hello.f90 contains an incorrect file suffix
      ld: 0711-715 ERROR: File hello.f90 cannot be processed.
      The file must be an object file, an import file, or an archive.


      You can get around this problem by using the -qsuffix=f=f90 compiler flag:

      Cu12:~/consult/codes/single110%xlf90 -qsuffix=f=f90 hello.f90

      ** hello   === End of Compilation 1 ===
      1501-510  Compilation successful for file hello.f90.


      Alternatively you can rename .f90 files to .f files:

      Cu12:~/consult/codes/single107% cp hello.f90 hello_f90.f Cu12:~/consult/codes/single108% xlf90 hello_f90.f

      ** hello   === End of Compilation 1 ===
      1501-510  Compilation successful for file hello_f90.f.


    • Automatic Promotion to REAL*8

      On the Origin2000, -r8 uses REAL(KIND=8) and COMPLEX(KIND=8) for real and complex variables, respectively.

      On the IBM p690, there is no equivalent compiler option that does exactly the same thing. -qrealsize=8 sets the size of the default REAL, DOUBLE PRECISION, COMPLEX and DOUBLE COMPLEX values to 8 bytes. -qautodbl=dbl4 promotes floating-point objects that are single precision to 8 bytes. (Note: variables explicitly declared as REAL*4 are also promoted.)


    • Useful FORTRAN Compiler flags

      The following Fortran compiler option may be useful:

      -qinitauto : initializes all variables to zero at compile time


    • Porting Programs to XL FORTRAN

      This link contains a useful discussion on porting programs to XL FORTRAN

      Porting FORTRAN programs


    • Porting Programs from GNU C/C++

      Page 19 of this AIX C document contains a useful discussion on porting programs from GNU C/C++

      Porting GNU c/C++ programs


    • Addressing mode

      The default on the IBM p690 is 32-bit (default on Origin is 64-bit)

      To use 64-bit on IBM, compile with -q64 -qwarn64

      ( more infomation)

    • Data Segment Size

      When using 32-bit mode the default data segment size (static, common, allocatable variables) is 256 MBytes

      Use this option to raise to 2 GB: -bmaxdata:0x80000000


    • Optimization Levels

      Optimization options (-O, -O3, -O4, etc.) are different on the p690. See the compiler man page(s) for details.

      Description of Optimization levels for FORTRAN


    • Vendor Supported Math Libraries

      On the Origin these are SCSL. On the p690 they are ESSL and PESSL.

      Both SCSL and ESSL provide BLAS, LAPACK and FFT routines. In addition, PESSL also proides PBLAS, ScaLAPACK and BLACS.



      ESSL (Engineering and Scientific Subroutine Library)

      PESSL (Parallel ESSL)


    • Fast Math Functions

      The Origin uses fastm for simple math functions. The p690 uses MASS.

      MASS (Math Acceleration Subsystem)


    • Automatic Parallelization

      Automatic parallelization on the Origin uses the -ipo compiler flag.

      On the p690 use -qsmp instead.


    • Timing Routines

      Routines such as dtime, etime, and flush, need to have a trailing underscore (_).


    • Tool to check makefiles

      There is a utility to check your makefiles named check_irix_makefiles. It will check for makefiles starting from the current directory, and show the FILENAME:LINE_NUMBER:TEXT for each Irix-specific option and display suggested AIX XL compiler options to use when porting.

      Cu12:~/bin125% check_irix_makefiles
      ./makefile:4: LIBS        = -lmpi
              * discard -lmpi , use: "mpxlf_r, mpcc_r, ..."
      ./makefile:5: CC          = cc -64 -mp -O3
              * replace -64 with -q64 -qwarn64
      ./makefile:5: CC          = cc -64 -mp -O3
              * replace -mp with -qsmp=omp
      ./makefile:6: CLINKER     = cc -64
              * replace -64 with -q64 -qwarn64
      ./makefile:8: CPPFLAGS    = -DnoCHECK
              * [fortran only: replace -D with -WF,-D]
      
              * .f90 suffix found on fortran files, use: -qsuffix=f=f90
      Cu12:~/bin126%
      
    • FORTRAN End of File Handling

      XL FORTRAN does not interpret an end-of-file as an error. Therefore constructs such as:

      ERR=stmt_label 

      will be ignored on reaaching the end of a file read as it does not consider that an error has occurred. Instead you can use the specific end of file specifier:

      END=stmt_label

  5. Running Programs

    • MPI

      Where the Origin uses mpirun to run parallel jobs the p690 uses poe:


      Origin % mpirun np n a.out {program arguments}


      IBM p690 % poe a.out {program arguments} -procs n

      POE (Parallel Operating Environment)

    • OpenMP

      To run an openmp program, set the environment variable OMP_NUM_THREADS to the desired number of threads, then enter the executable name and any necessary arguments at the shell prompt as with a serial program.

      % setenv OMP_NUM_THREADS 2

      % openmp.out


  6. Batch System

    • LoadLeveler

      The IBM p690 uses the LoadLevel batch system. A comparison of Origin and p690 commands is given in the following table:

      Batch Commands
      Function LSF (SGI Origin 2000) LoadLeveler (IBM p690)
       Submit a Job  bsub  llsubmit
       View Job Status  bjobs  llq
       Delete a Job  bkill  llcancel
       View Completed Jobs  busage  llsummary

      More information on the batch system can be found at :

      LoadLeveler

      and sample batch scripts are available at:

      /usr/local/doc/ll/

    • Automatic Saving of Files to Mass Storage

      The comamnd msscmd -b has been replaced by saveafterjob


    • Command Line Directives

      Command line directives are not allowed in p690 batch scripts.

    • Batch Scripts
      Sample MPI Batch Scripts

      LSF (SGI Origin 2000)

      LoadLeveler (IBM p690)

      #!/bin/csh
      

       

      #@ shell = /usr/bin/tcsh
      
      #BSUB -M 500m
      

      Specify 500 megabytes of physical memory

      #@ resources = ConsumableCpus(1) ConsumableMemory(500Mb)
      
      #BSUB -n 4
      

      Specify 4 threads/processes

      #@ tasks_per_node = 4
      
      #BSUB -W 1:00
      

      Specify job run time limit of 1 hour

      #@ wall_clock_limit = 1:00:00
      
      #BSUB -o testjob.o
      

      Store the standard output and standard error (optional)

      #@ output = $(job_name).$(jobid).out
      
      #@ error = $(job_name).$(jobid).err
      
      #BSUB -N
      

      Send mail (optional)

      #@ notification = complete
      
      #BSUB -J testjob
      

      Specify job name (optional)

      #@ job_name = testjob
      
      #BSUB -P abc
      

      Charge job to project abc.
      Not yet available in LoadLeveler

       

       

      Specify type of job

      #@ job_type = parallel
      

       

      Export environment variables

      #@ environment = COPY_ALL
      

       

      Specify job class

      #@ class = batch
      

       

      This MUST be the last LoadLevel directive

      #@ queue
      

      ######################

      End of batch manager directives

      #######################

      cd $SCR
      

      Change to machine-local scratch directory

      cd $SCR
      
      msscmd -b "tar cf test.output.tar test.output*"
      

      Save files after the job completes.

      saveafterjob "tar cf test.output.tar test.output*"
      
      msscmd cd test,get a.out,get test.input
      

      Get executable and input file from mass storage.

      msscmd cd test,get a.out,get test.input
      
      chmod u+x a.out
      

      Make program executable

      chmod u+x a.out
      
      mpirun -np $BSUB_NUMTHREADS ./a.out < test.input
      

      Run mpi program

      poe ./a.out < test.input