Development

Hartree Centre Logo

Back to Contents Page

Software Development on the Blue Gene/Q

Last modified: 9/12/2015

Quick Links

Using environment modules to access compilers

For the IBM compiler suite, use:

module add ibmmpi

For the GNU compilers, use:

module add gccmpi

Where is the software on the BG/Q?

IBM refer to software as "drivers". The following notes may be useful. /bgsys/drivers/ppcfloor - This is the root of the BG system software release, and it contains files for the front ends, the I/O nodes and the compute nodes. We probably don't want to make this the system root for the back end, because it would allow other types of binaries into the build.

Note: /bgsys/drivers/ppcfloor is a soft link /bgsys/drivers/V1R2M2/ppc64 or similar to allow for software upgrades. Do not refer to V1R2M2 directly as it may change. For the Compute Nodes, it is believed that there is no formal documentation about where you should look to find node system libraries. However, the IBM RedBook sg247948.pdf may be useful (See RedBooks resource folder on the community portal). There are the MPI compiler scripts in /bgsys/drivers/ppcfloor/comm/xl/bin and comm/gcc/bin. These were written at Argonne and are now included with the BG driver distribution, and they have to link against all the BG system libraries. Note that in most cases it is preferable to use these wrappers rather than invoking the raw compiler, see external link: http://www.bnl.gov/bluegene/content/guide/bgq/compileinvokeq.shtml . The wrappers implicitly contain a definitive list of libraries, so you can invoke with -show use all the args you get with that command.

This narrows it down to a few directories as follows.

/bgsys/drivers/ppcfloor/comm/default/lib
/bgsys/drivers/ppcfloor/comm/sys/lib
/bgsys/drivers/ppcfloor/spi

Those should be the only backend directories that contain libraries used by apps. The Compute Node Kernel (CNK) OS is minimal and SPI is the System Programming Interface to the run-time kernel of the CNK. If you look in the COMM directories, they contain MPI libs and various lower level comm APIs like DCMF, which BG uses. For GNU, the back end compiler run time libs are in /bgsys/drivers/ppcfloor/gnu-linux .

If you are using GCC compilers these can be found in locations like: /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgp-linux-gcc which is the same as: /bgsys/drivers/ppcfloor/gnu-linux/powerpc64-bgp-linux/bin/gcc .

Note: location and names of of libraries changed at release V1R2M1.

Current installed Version

Blue Joule was updated in Jan'2015 to V1R2M2 which is compatible with the current software stack:

Where is the documentation?

Some of the directories mentioned above contain a doc which has a collection of HTML files. Relevant IBM documentation (including Redbooks) can be found at external link: http://community.hartree.stfc.ac.uk/access/content/group/admin/BG_Q/. Other documentation is available from IBM InfoCenter - external link: http://pic.dhe.ibm.com/infocenter/compbg/v121v141/index.jsp . We have created an environment module called ibmmpi which will set the environment variables and library paths for you, just do

module load ibmmpi.

Using XL Compilers on the BG/Q.

It is important to understand that the BG/Q uses cross compilation, i.e. the login node is a different architecture to the CNs. This introduces several complications:

  • tools cannot auto-detect the target system;
  • the build process has to be told the location of include files and libraries for the target system;
  • usually the executables don't run on the build platform.

To compile a parallel F90 application we suggest the following basic setup (but you can simply use the ibmmpi module instead).

The flush() calls need to be converted to flush_(), so tried to use the linker option but resorted to editing the source instead.

FC=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90
FFLAGS0= -O3
FFLAGS= -O1
# linker options to use for parallel jobs
LD=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90
# this next line did not work
LDFLAGS=-brename:flush,flush_ macdep_in_stata.f90

The full XL compiler manuals are on line

Some information about building shared libraries is here: external link: http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/UserInfo/SharedLibraries.html

Using GNU configure

It is also possible to cross compile using the GNU configure tool and as many applications are installed with configure it is useful to know how to do it. There are two important keywords -build=powerpc64-bgq-linux-gnu and -host=powerpc64-bgq-linux . The former is the login node platform and the second is the CN (host) node platform. For instance to compile FFTW we used the following.

../configure --build=powerpc64-bgq-linux-gnu --host=powerpc64-bgq-linux \
--prefix=$INSTALLDIR --enable-mpi --enable-openmp --disable-shared \
CC=mpixlc_r F77=mpixlf90_r MPICC=mpixlc_r MPICXX=mpixlcxx_r MPIFC=mpixlf90_r MPILIBS=" " \
CFLAGS=-O3 FCFLAGS=-O3 LDFLAGS=-qnostaticlink

make -j 4
make install

Note: you will find "make check" won't work because the tests are cross compiled for BG/Q. You will have to run each one via the batch queue to test. Some environment variables that might need setting to use configure and other tools are:

CC - C compiler command - e.g. export CC="/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlc_r"

CFLAGS - C compiler flags - export CFLAGS="-I/bgsys/drivers/ppcfloor -I/bgsys/drivers/ppcfloor/spi/include/kernel/cnk"

LDFLAGS - linker flags, e.g. export LDFLAGS="-L/bgsys/drivers/ppcfloor/spi/lib"

LIBS - libraries to pass to the linker, e.g. export LIBS="-lSPI_cnk -lpthread -lrt"

CPP - C preprocessor

CPPFLAGS - (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a non-standard directory <include dir>

Using CMAKE

The CMake Web site guide is here: external link: http://www.vtk.org/Wiki/CMake_Cross_Compiling For cross compilation using CMAKE you need to specify the node environment, the choice of compiler, and whether the binary should be statically or dynamically linked. There are pre-configured platform toolchain configurations to do this. You can add definitions to the beginning of your application's CMakeLists.txt file as follows.

set(CMAKE_SYSTEM_NAME BlueGeneQ-static)
set(CMAKE_C_COMPILER bgxlc_r)
set(CMAKE_CXX_COMPILER bgclC_r)

BlueGeneQ-dynamic is also available as are the GCC compilers. CMAKE will search for the files in /gpfs/packages/cmake/2.8.9/share/cmake-2/Modules/Platform/${CMAKE_SYSTEM_NAME}.cmake etc.

Note that at time of writing there are some issues with cmake. These are being investigated.

Using ESSL

ESSL is IBM's optimised Engineering and Scientific Subroutine Library. There is a special version of theis for BG/Q. You can use it as follows, examples show C and Fortran linkage. Note you must use the thread safe compiler (with _r).

module load ibmmpi ibmessl

# C example
mpicc_r -o test.out call_essl.c -L$ESSL_LIB -lesslbg

# Fortran example single threaded
mpixlf90_r -qsmp -o test.out call_essl.f90 -L$ESSL_LIB -lesslbg

# Fortran example, multi-threaded with -qsmp flag
mpixlf90_r -qsmp -o test.out call_essl.f90 -L$ESSL_LIB -lesslsmpbg

You can also use the MASS library in a similar way, but it is automatically included if using the -qhot option.

Using libtool

There are some possible issues with using libtool on the BlueGene/Q. These are described here: external link: http://comments.gmane.org/gmane.comp.gnu.libtool.general/11872 . Note that we currently have libtool-2.2.6b on the front end node, but other software will use libtool-2.4.2.

It may be necessary to add "LDFLAGS='-nostaticlink'" onto your configure options and optionally '-nostaticlink=libgcc'.

Further Information

Here's the Brookhaven Web site external link: http://www.bnl.gov/bluegene/content/guide/bgq/compileinvokeq.shtml and ANL Wiki page which may be useful external link: https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q .

Material from the IBM training course held at the Hartree Centre in 2013 can be found here: external link: http://community.hartree.stfc.ac.uk/access/content/group/admin/HPC%20Training/BG_Q%20training%20course%20February%202013/ .

A useful tutorial for software on the LLNL Sequoia Blue Gene/Q can be found at external link: https://computing.llnl.gov/tutorials/bgq .

Back to Contents Page