Development2

Hartree Centre Logo

Back to Contents Page

Last modified: 26/08/2015

Software Development on Invicta, Iden, Napier and Neale

This page contains details of setting your environment and compiling code on the Blue Wonder systems Invicta, Iden, Napier and Neale.

Using environment modules

We use environment modules to ensure you get the correct shell environment for the tasks you want to do. To see which modules are available, type:

module avail

This will present you with a list like this:

------------------------------------------- /gpfs/packages/Modules/3.2.9/modulefiles --------------------------------------------
dot         module-cvs  module-info modules     null        use.own

------------------------------------------ /gpfs/packages/Modules/modulefiles/packages ------------------------------------------
amber/12(default)               code_saturne-gcc/3.2.0          ls-dyna/971-r4.2.1              star-ccm+/8.04.010
amber/license                   code_saturne-gcc/3.2.1          ls-dyna/971-r6.1.0(default)     star-ccm+/8.06.007
ansys/14.5(default)             cp2k/2.3                        ls-dyna-ompi/971-r6.1.0         star-ccm+/license
ansys/license                   delft3d/12Jun13                 lsprepost/4.0                   telemac/6.1
castep/6.0.1                    dlclassic/1.9                   namd/2.9                        telemac-gcc/6.3
castep/6.1                      dlpoly/4.04                     nwchem/6.1.1                    vasp/5.2.12(default)
castep/6.1.1                    dlpoly/4.05(default)            nwchem/6.3(default)             vasp/license
castep/7.0.1(default)           dynamis/3.1.0(default)          oasys/oa10                      vorpal/5.2.2
chemshell/3.5.0                 dynamis/license                 octopus/4.0.1                   vorpal/6.0.0(default)
code_aster/11.4                 genesis/2.0                     openfoam-gcc/1.7.1              vorpal/license
code_saturne/3.0.0              getm/2.2.2                      openfoam-gcc/2.1.1              wps/3.4.1
code_saturne/3.0.1(default)     gromacs/4.5.5                   pelegant-gcc/25.1.0             wrf/3.4.1
code_saturne/3.0.1_dbg          gulp/4.0                        salome-meca
code_saturne-gcc/3.0.1(default) lammps/13Sep13(default)         siesta/3.1
code_saturne-gcc/3.0.3          lammps/28Oct12                  star-ccm+/7.06.012(default)

------------------------------------------- /gpfs/packages/Modules/modulefiles/other --------------------------------------------
cgns/3.1.4                intel/comp/14.0.1         libxml2-gcc/2.9.1         parmetis/4.0.3
cgns-gcc/3.1.4            intel/license             med/3.0.6                 parmetis-gcc/4.0.3
fftw/2.1.5                intel/mkl/10.3.0          med-gcc/3.0.6             pmpi/8.3.0
fftw/3.3.2                intel/mkl/10.3.6(default) mumps/4.10.0              scalapack/2.0.2
gsl-gcc/1.15              intel/mkl/11.0.1          netcdf/4.0.1              scotch/6.0.0
hdf5/1.8.9                intel/mkl/11.1.2          netcdf/4.2.1.1            scotch-gcc/6.0.0
hdf5-gcc/1.8.9            intel/mpi/4.0.3(default)  netcdf-gcc/4.2.1.1        zlib/1.2.7
intel/comp/12.0           intel/mpi/4.1.0           openmpi/1.6               zlib-gcc/1.2.7
intel/comp/12.1(default)  intel/mpi/4.1.3           openmpi-gcc/1.6
intel/comp/13.0.1         libxml2/2.9.1             papi/5.1.0

----------------------------------------- /gpfs/packages/Modules/modulefiles/production -----------------------------------------
R/3.0.1                   octave/3.6.4              tau/2.22                  vmd/1.9.1
cmake/2.8.9               paraview/3.98.1           totalview/8.12.0(default) x10/2.4.0
gcc/4.8.1                 perl5/5.16.2              totalview/license
java/1.6                  python/2.7.5              utilities
memoryscape/3.4.0         scalasca/1.4.2            vampir/8.1.0

Note that some modules are suffixed "-gcc". This means that the application they refer to was compiled with the GNU compiler suite. Applications without this suffix can be assumed to have been compiled with the Intel Compiler suite.

To load a module, type something like this:

module load intel/mpi

or:

module add intel/mpi

If a particular application has multiple versions available, you will get the version marked as "default", or the most recent. If you want to choose a different version, specify it, like this:

module add intel/mpi/10.3.0

All module loads are silent, so if you want to see some more information about a particular module, use:

module help hdf5

which will return something like this:

----------- Module Specific Help for 'hdf5/1.8.9' -----------------

	hdf5 - loads the hdf5 1.8.9 environment

	Version 1.8.9

To see which modules you have already loaded, you can do:

module list

And to remove a module from your environment, do:

module rm hdf5

Finally, you can remove all modules at once with:

module purge

So, for example:

ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module li
No Modulefiles Currently Loaded.
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module add hdf5
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module add zlib
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module add netcdf
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module li
Currently Loaded Modulefiles:
  1) hdf5/1.8.9       2) zlib/1.2.7       3) netcdf/4.2.1.1
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module rm hdf5
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module li
Currently Loaded Modulefiles:
  1) zlib/1.2.7       2) netcdf/4.2.1.1
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module purge
ida6c42:/gpfs/home/package_build/build/djc87-build/idplx/imb $ module li
No Modulefiles Currently Loaded.

Compiling with gcc and OpenMPI

Load the openmpi-gcc module

module load openmpi-gcc

You can then use the standard mpicc, mpif90, mpiCC commands as on other systems. A reference guide is available here: external link: http://www.open-mpi.org/faq/?category=mpi-apps

For more information see external link: Performance Optimisation Guide

Compiling with Intel Compilers and Intel MPI

Load the intel/comp and intel/mpi modules. You may also want to use the Intel Math Kernel libraries.

module add intel/comp intel/mpi intel/mkl

Use the mpiicc, mpiifort, and mpiicpc wrappers. Getting started and reference guides are available here: external link: http://software.intel.com/en-us/en-us/articles/intel-mpi-library-documentation. For more information see external link: Performance Optimisation Guide

For information about linking with MKL libraries, you could try the link line advisor: external link: http://software.intel.com/sites/products/mkl/ .

Using the default MKL-10.3

Use "module load intel/mkl/10.3.6".

In the examples below, if OMP is not required just remove the -liomp5 library. These examples also assume the F95 interfaces are required.

Multi-threaded LAPACK and BLAS with static linking.

-I$(MKLROOT)/include/intel64/lp64 -I$(MKLROOT)/include
$(MKLROOT)/lib/intel64/libmkl_blas95_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_lapack95_lp64.a \
  -Wl,--start-group  $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_intel_thread.a \
  $(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group \
  -liomp5 -lpthread -lm

Multi-threaded LAPACK and BLAS with dynamic linking.

-I$(MKLROOT)/include/intel64/lp64 -I$(MKLROOT)/include
-L$(MKLROOT)/lib/intel64 $(MKLROOT)/lib/intel64/libmkl_blas95_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_lapack95_lp64.a -lmkl_intel_lp64 \
  -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -lm

Multi-threaded ScaLAPACK with Intel MPI and static linking. Example, SIESTA executable linked with this option is 19.4MB.

-I$(MKLROOT)/include/intel64/lp64 -I$(MKLROOT)/include
$(MKLROOT)/lib/intel64/libmkl_blas95_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_lapack95_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_scalapack_lp64.a \
  -Wl,--start-group  $(MKLROOT)/lib/intel64/libmkl_intel_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_intel_thread.a \
  $(MKLROOT)/lib/intel64/libmkl_core.a \
  $(MKLROOT)/lib/intel64/libmkl_blacs_intelmpi_lp64.a \
  -Wl,--end-group -liomp5 -lpthread -lm

Multi-threaded ScaLAPACK with Intel MPI and dynamic linking. Example, SIESTA executable linked with this option is 6.7MB.

-I$(MKLROOT)/include/intel64/lp64 -I$(MKLROOT)/include
-L$(MKLROOT)/lib/intel64 $(MKLROOT)/lib/intel64/libmkl_blas95_lp64.a \
  $(MKLROOT)/lib/intel64/libmkl_lapack95_lp64.a -lmkl_scalapack_lp64 \
  -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core \
  -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm

Performance Optimisation

See external link: Performance Optimisation Guide

For more information about linking with MKL libraries, you could try the link line advisor: external link: http://software.intel.com/sites/products/mkl/ .

Back to Contents Page