Managed Software

Hartree Centre Logo

Back to Contents Page

Managed Software

Last modified 14/12/2018

Users of Hartree Centre systems can expect a range of software to be centrally installed and managed by Centre staff.

The current status and updates external link: can be seen here.

Quick Links

Introduction to Environment Modules

"Environment Modules" is an open source software management utility that has been chosen to manage the centrally available software on the Hartree Centre systems. The main reference for Environment Modules is the SourceForge site: external link: http://modules.sourceforge.net/index.html .

The idea of Modules is to make it easier to use the managed software. Here is a brief guide to using modules.

To see the software available, use the "module avail" command on a login node. For instance, on the Blue Gene/Q this gives the following.

-bash-4.1$ module avail

------------------------------ /gpfs/packages/Modules/modulefiles/production -------------------------------
cmake/2.8.9               ibmessl                   memoryscape/3.3.0         tau/2.22
gccmpi                    ibmmpi                    prace                     totalview/8.11.0(default)
hpctools                  ibmmpi.ndebug             prace_test                totalview/license
ibm-frontend              make/3.82                 scalasca/1.4.2            utilities

…

The format of the listed packages is generally <package name>/<package version>. For example, dlpoly/4.03 is version 4.03 of DL_POLY. We may in some cases include the name of the compiler used for the package, but the default will be compiled with the latest IBM XL suite on the Blue Gene and the latest Intel suite on the iDataPlex.

The centrally installed and maintained software is located under the /gpfs/packages file system. Note that whilst the path looks the same, this is a different directory mount on each machine. On Blue Gene it is structured as follows. For software compiled with the XL compilers for the compute nodes see /gpfs/packages/ibm; for that compiled with the GCC compilers for the compute nodes see /gpfs/packages/gcc and for utilities that run on the login node see /gpfs/packages/utilities .

We intend to structure the modules as follows.

  • production - associated with loading a default environment for compilation, runtime or utilities on the login node;
  • packages/<name>/<version> - whole applications compiled for the compute nodes using XL (BG/Q) or Intel suite (X86 systems);
  • packages/<name-gcc>/<version> - whole applications compiled for the compute nodes using GCC suite;
  • other/<name>/<version> - library or middleware compiled for the compute nodes using XL (BG/Q) or Intel suite (X86 systems);
  • other/<name-gcc>/<version> - library or middleware compiled for the compute nodes using GCC suite.

To get information about a particular piece of software, use the command "module help".

-bash-4.1$ module help lammps/28Oct12 

----------- Module Specific Help for 'lammps/28Oct12' -------------

        Adds LAMMPS-28Oct12 executable lmp_bgq to your environment
        Compiled with XL compiler suite

        Version 28Oct12

"module whatis" will give an abbreviated form of this information.

"module show" gives further information about what paths will be set, e.g.

-bash-4.1$ module show ibmessl
-------------------------------------------------------------------
/gpfs/packages/Modules/modulefiles/production/ibmessl:

module-whatis    adds IBM ESSL to your environment 
conflict         gcc 
prepend-path     PATH /bgsys/ibm_essl/prod/opt/ibmmath/essl/5.1 
setenv           ESSL_LIB /bgsys/ibm_essl/prod/opt/ibmmath/lib64 
prepend-path     LD_RUN_PATH /bgsys/ibm_essl/prod/opt/ibmmath/lib64 
prepend-path     LD_RUN_PATH /gpfs/packages/ibm/fftw/essl_wrapper/5.1/lib64 
-------------------------------------------------------------------

To use a particular piece of software, a user must first load the appropriate module as in the following example.

-bash-4.1$ module load fftw/3.3.2

To see what packages you currently have loaded, use "module list".

-bash-4.1$ module list

Currently Loaded Modulefiles:
  1) ibmmpi        2) cmake/2.8.9   3) fftw/3.3.2

Organisation of Modules

There are currently independent collections of modules for Fairthorpe, Panther, Paragon and Scafell Pike. These are invoked using meta-modules called use.fairthorpe, use.panther, use.paragon and use.scafellpike. This organisation looks like this:

Modules Architecture

Using Modules in Job Scripts

The module command is implemented as a shell function and modifies the environment of your running shell. Shell functions are stored internally in the shell process and are not transferred to your job's environment with the COPY_ALL setting. You must therefore re-define the module function in your job script. In order to get access to the module command from within a LoadLeveler script, you need to source the appropriate profile scripts as follows.

  • "source /etc/profile.d/modules.sh" for bourne-derived shells;
  • "source /etc/profile.d/modules.csh" for csh-derived shells.

Note: This method does not work on the Power-8 systems, see Jobs on Panther and Paragon. Instead, you need to module load use.panther or module load use.paragon before submitting the job.

Installed Software

Any user may install 3rd party software themselves if they possess an appropriate license. The Hartree Centre Technology Support Team can centrally install requested 3rd party software if the software's license conditions permit this. The purchasing of 3rd party software for The Hartree Centre will be decided on a case by case basis and will be determined by such factors as cost, number of requestors, etc.

Each item of installed software will have an associated module. The set of modules at the time of writing is as below.

Scafell Pike

Last updated: 14/12/2018

----------------- /lustre/scafellpike/local/apps/Modules/modulefiles/packages -----------------
   castep/16.11              gnuplot/5.0.3                  plumed/2.4.1
   cfdem-gcc/3.8.0           gromacs/5.1.4                  repast-hpc/2.2.0
   cfx/18.1                  gromacs/2016.4          (D)    salome-meca/2018.0
   cfx/19.2           (D)    gromacs/2018.1                 singularity/2.5.2
   code_saturne/5.0.4        gulp/4.5                       spacy/2.0.3
   dlmeso/2.6                hello_world/1.1                star-ccm+/11.04.010
   dlpoly/4.08               lammps/11Aug17                 star-ccm+/12.06.010
   dlpoly/4.09        (D)    liggghts-gcc-impi/3.8.0        star-ccm+/13.02.013 (D)
   espresso/6.2.1            ls-dyna/r9.1.0                 star-ccm+/13.04.011
   espresso/6.3       (D)    ls-dyna/r10.1.0         (D)    star-cd/4.28.033
   fenics/2018.1.0           namd-gcc/2.12                  star-cd/4.30.014    (D)
   fluent/18.1        (D)    notebook/5.2.1                 vasp/5.3.3
   fluent/19.2               nwchem/6.8
   fluidity/30Mar18          openfoam-gcc/5.0

------------------ /lustre/scafellpike/local/apps/Modules/modulefiles/other -------------------
   atlas/3.10.3            hdf5-gcc7/1.10.1                  openmpi-gcc/3.1.3
   boost-gcc/1.59.0        hdf5/1.10.1                       openmpi-gcc6/2.1.1
   boost-gcc/1.65.0 (D)    intel-training/anaconda           openmpi-gcc7/2.1.1
   boost/1.65.0            intel-training/intel_tools        openmpi/2.1.1
   bull/fastio             intel-training/2018        (D)    petsc-gcc/3.6.4
   eigen3-gcc/3.2.2        mpich-gcc/3.2.1                   petsc/3.6.4
   eigen3-gcc/3.3.5 (D)    mpich-gcc7/3.2.1                  petsc/3.7.6
   esmf/7.0.1              mvapich2-gcc/2.2                  petsc/3.7.7        (D)
   fftw2/2.1.5             netcdf/4.4.1                      plasma-intel/2.8.0
   fftw3/3.3.6             openmpi-gcc/1.6.5                 scalapack/2.0.2
   fluidity-gcc/4.x        openmpi-gcc/1.10.7                starpu-gcc/1.2.6
   gsl/2.4                 openmpi-gcc/2.1.1          (D)    starpu-gcc7/1.2.6
   hdf5-gcc/1.10.1         openmpi-gcc/3.1.1

---------------- /lustre/scafellpike/local/apps/Modules/modulefiles/production ----------------
   clang/5.0.0           intel/18.2.199            licenses/star-ccm+
   clang/6.0.0    (D)    intel/18.3.222            licenses/star-cd   (D)
   cmake/3.7.1    (D)    intel/19.0.117            lmod/7.6
   cmake/3.9.1           intel/19.1.144            mysql/5.7.17
   dos2unix/7.4          intel_mpi/latest          nano/2.5.3
   eclipse/4.7           intel_mpi/5.1.3           papi/5.6.0
   emacs/25.2            intel_mpi/17.4.196        perl5/5.26.0
   gcc/4.9.2             intel_mpi/18.0.128 (D)    pgi/17.4
   gcc/4.9.4      (D)    intel_mpi/18.1.163        psyclone/1.4.1
   gcc5/5.4.0            intel_mpi/18.2.199        python2/anaconda
   gcc6/6.4.0            intel_mpi/18.3.222        python2/2.7.13     (D)
   gcc7/7.2.0            intel_mpi/19.0.117        python3/anaconda
   gcc8/8.1.0            intel_mpi/19.1.144        python3/3.6.2      (D)
   gdb/8.1               intel_vtune/2017          tau/2.26.3
   intel/latest          intel_vtune/2018   (D)    utilities
   intel/15.1.133        intel_vtune/2019          utilities-gcc
   intel/16.4.258        java/1.7                  vim/8.1
   intel/17.4.196        java/1.8           (D)    xalt/1.1.2         (L)
   intel/18.0.128 (D)    licenses/ansys
   intel/18.1.163        licenses/intel

------------------ /lustre/scafellpike/local/apps/lmod/lmod/modulefiles/Core ------------------
   StdEnv     (L)    module-info    settarg/7.6       use.panther            use.viz
   dot               modules        use.benchmarks    use.paragon            use.xcs
   module-git        null           use.own           use.scafellpike (L)

Viceroy -- Visualisation on Scafell Pike

Last updated: 14/12/2018

Additional visualisation packages are available on the XRV interactive login nodes.

JADE

Last updated: 14/12/2018

Panther Power-8 Cluster

Last updated: 14/12/2018

------------------- /gpfs/panther/local/apps/Modules/modulefiles/production -------------------
   R/3.4.3                 gcc7/7.1.0                       pgi/16.10           (D)
   ant/1.10.1              gdb/7.11                         pgi/17.4
   camfort/0.901           gperftools/2.6.1                 python2/anaconda
   clang/coral0.2          ibm/13.1.3                       python2/2.7.8       (D)
   clang/3.8.0      (D)    ibm/13.1.5                (D)    python3/anaconda
   cmake/3.4.1             ibmmpi/xl.debug                  python3/3.6.0       (D)
   cmake/3.6.2             ibmmpi/xl.perf            (D)    spectrum_mpi/10.1.0
   cmake/3.7.1             ibmmpi/xl.trace                  swig/3.0.10
   cmake/3.10.2     (D)    java/8.0                         totalview/16.07.11
   cuda/7.5                lmod/6.5                         utilities
   cuda/8.0         (D)    maven/3.5.2                      utilities-fen
   cuda/9.0                mysql/5.7.17                     utilities-gcc
   cuda/9.1                nano/2.5.3                       valgrind-gcc/3.12.0
   cuda/9.2                ofc/27Oct16                      viz
   extrae-gcc/3.4.3        papi-gcc/5.5.1                   xalt/1.1.2          (L)
   gcc6/test               papi/5.4.1                       xlc/16.1.1.0-180913
   gcc6/6.1.1              perl5/5.20.3-thread-multi        xlf/16.1.1.0-180913
   gcc6/6.3.0       (D)    perl5/5.20.3              (D)

-------------------- /gpfs/panther/local/apps/Modules/modulefiles/packages --------------------
   beast2/2.4.5                  infernal/1.1.2              sklearn/0.18
   biobuilds/2016.11             interproscan/5.26.64        spacy/2.0.3
   caffe/20Oct16                 keras/1.1.0                 spark/2.0.0
   castep/16.11                  keras/2.0.3          (D)    spark/2.1.0
   cgns-gcc/3.3                  lammps/17Nov16              spark/2.1.1            (D)
   code_saturne-gcc/4.2.2        metis/5.1.0                 tensorflow/ibm
   code_saturne-gcc/5.0.4 (D)    namd/2.11            (D)    tensorflow/1.0.1
   code_saturne/4.2.2            namd/2.12b                  tensorflow/1.4.1       (D)
   cuda-convnet2/Nervana         notebook/4.2.0              tflearn/0.3
   cuda-convnet2/10Oct16  (D)    notebook/5.0.0       (D)    theano/0.7.0
   dlmeso/2.6                    opencv-gcc/3.1              theano/0.8.2
   dlpoly/4.08                   openfoam-gcc7/v1806         theano/0.9.0           (D)
   foam-extend-gcc/4.0           openfoam-gcc7/5.0    (D)    utils4cfd-gcc/unstable
   geopy/1.11.0                  ranger/1.8.1                vim/8.0
   gromacs-gcc/5.1.4             ruby/2.4.2
   gromacs-gcc/2016.1     (D)    scala/2.12.1

--------------------- /gpfs/panther/local/apps/Modules/modulefiles/other ----------------------
   atlas-gcc/3.10.3        hdf5-gcc/1.8.11         openmpi-gcc/1.10.2 (g,O)
   boost-gcc/1.62.0        hdf5-gcc/1.10.0  (D)    openmpi-gcc/2.0.2  (g,D)
   boost-gcc/1.65.1 (D)    hdf5/1.10.1             openmpi-gcc/3.0.0  (g)
   fftw2/2.1.5             ibmessl/5.4             openmpi-pgi/1.10.2
   fftw3-gcc/3.3.4         ibmpessl/5.2            openmpi-pgi/2.0.2  (D)
   fftw3/3.3.5             mvapich2-gcc/2.2        openmpi/2.0.2
   git/2.9.5               mvapich2-pgi/2.2        openmpi/2.1.1      (D)
   google-gcc              mvapich2/2.2a    (O)    plplot-gcc/5.11.1
   gsl-gcc/2.4             netcdf-gcc/4.3.3        plplot/5.11.1
   hdf5-gcc/serial         netcdf/4.3.3            protoc-gcc/3.1.0

----------------- /gpfs/panther/local/apps/Modules/modulefiles/visualisation ------------------
   gnuplot/5.0.3             paraview/5.2.0        qt/5.10.1            vtk/6.1.0
   gnuplot/5.2.2      (D)    paraview/5.4.1 (D)    utilities-viz        vtk/6.3.0
   graphviz/2.38.0           qt/4.8.6       (D)    visit/2.12.3         vtk/7.1.0 (D)
   paraview-egl/5.4.1        qt/5.4.0              visit/2.13.2  (D)
   paraview-egl/5.5.2 (D)    qt/5.9.1              vmd/1.9.3

------------------- /gpfs/fairthorpe/local/apps/lmod/lmod/modulefiles/Core --------------------
   StdEnv     (L)    module-info        use.fairthorpe          use.paragon.dev (E)
   dot               modules            use.own                 use.scafellpike
   lmod/7.7   (D)    null               use.panther      (L)
   lsf-utils         settarg/6.5        use.panther.orig
   module-git        settarg/7.7 (D)    use.paragon

  Where:
   O:  Obsolete
   g:  built for GPU
   L:  Module is loaded
   E:  Experimental
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 …" to search for all possible modules matching any of the
"keys".

Paragon Power-8 Cluster

Last updated: 14/12/2018

------------------- /gpfs/paragon/local/apps/Modules/modulefiles/production -------------------
   camfort/0.901        ibmmpi/xl.debug                  python2/2.7.8       (D)
   clang/3.8.1          ibmmpi/xl.perf            (D)    python3/anaconda
   cmake/3.4.1          ibmmpi/xl.trace                  python3/3.6.0       (D)
   cmake/3.6.2          java/8.0                         spectrum_mpi/10.1.0
   cmake/3.7.1          lmod/6.5                         swig/3.0.10
   cmake/3.10.2  (D)    mysql/5.7.17                     utilities
   cuda/8.0      (D)    nano/2.5.3                       utilities-fen
   cuda/9.0             ofc/27Oct16                      utilities-gcc
   cuda/9.2             papi-gcc/5.5.1                   xalt/1.1.2          (L)
   gcc/5.4.0            papi/5.4.1                       xlc/16.1.1.0-180913
   gcc6/6.4.0           perl5/5.20.3-thread-multi        xlc/16.1.1.0-181102 (D)
   gcc7/7.1.0           perl5/5.20.3              (D)    xlf/16.1.1.0-180913
   ibm/13.1.3           pgi/17.4                         xlf/16.1.1.0-181102 (D)
   ibm/13.1.5    (D)    python2/anaconda

-------------------- /gpfs/paragon/local/apps/Modules/modulefiles/packages --------------------
   amgx-gcc/unstable                  gromacs-gcc/2016.1       (D)    opencv-gcc/2.4.0
   biobuilds/2016.11                  gromacs/2016.1                  openfoam-gcc7/v1806
   caffe/bvlc                         hdf5-parallel-gcc/1.10.2        openfoam-gcc7/5.0   (D)
   caffe/ibm                   (D)    keras/1.2.2                     ranger/1.8.1
   castep/16.11                       keras/2.0.3              (D)    spark/2.1.0.1
   code_saturne-gcc/gpu-beta          lammps/17Nov16                  spark/2.1.1         (D)
   code_saturne-gcc/nogpu-beta        lammps/22Aug18           (D)    tensorflow/ibm      (D)
   code_saturne-gcc/5.0.4      (D)    namd/2.12                       tensorflow/1.0.1
   dlpoly/4.08                        namd/2018_10_10          (D)    tensorflow/1.2.1
   go/1.9.2                           nltk/3.2.5                      tensorflow/1.4.0
   gromacs-gcc/5.1.4                  openai_gym/0.10.8               theano/0.9.0

--------------------- /gpfs/paragon/local/apps/Modules/modulefiles/other ----------------------
   boost-gcc/1.53.0    hdf5/1.10.1         netcdf/4.3.3             openmpi-gcc/3.1.2  (g,D)
   fftw2/2.1.5         ibmessl/5.4         netcdf/4.4.1      (D)    openmpi-pgi/1.10.2
   fftw3-gcc/3.3.4     ibmpessl/5.2        node.js/10.10.0          openmpi/2.0.2
   fftw3/3.3.5         mvapich2-gcc/2.2    openmpi-gcc/2.0.2
   git/2.9.5           mvapich2/2.2        openmpi-gcc/3.0.0 (g)

----------------- /gpfs/paragon/local/apps/Modules/modulefiles/visualisation ------------------
   gnuplot/5.2.2    paraview-egl/5.5.0    paraview-egl/5.5.2    paraview-egl/5.6.0 (D)    viz

------------------- /gpfs/fairthorpe/local/apps/lmod/lmod/modulefiles/Core --------------------
   StdEnv     (L)    module-info        use.fairthorpe          use.paragon.dev (E)
   dot               modules            use.own                 use.scafellpike
   lmod/7.7   (D)    null               use.panther
   lsf-utils         settarg/6.5        use.panther.orig
   module-git        settarg/7.7 (D)    use.paragon      (L)

  Where:
   g:  built for GPU
   L:  Module is loaded
   E:  Experimental
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 …" to search for all possible modules matching any of the
"keys".

Visual Computing Facilities

Details typically subject to customer requirements.

Brunner-Mond Training Suite

Details typically subject to customer requirements.

Python Packages

Python packages have been installed following requests to the Helpdesk. Some have their own modules, e.g. Theano, PyWT. Pip list reports the following.

Scafell Pike

Last updated 14/12/2018

python/2.7.13 backports.functools-lru-cache (1.4) cycler (0.10.0) en-core-web-sm (2.0.0) fparser (0.0.5) matplotlib (2.1.0) MySQL-python (1.2.5) nose (1.3.7) numpy (1.13.1) pandas (0.21.0) pip (9.0.1) pyparsing (2.2.0) python-dateutil (2.6.1) pytz (2017.3) scipy (0.19.1) setuptools (36.3.0) six (1.11.0) subprocess32 (3.5.0rc1) virtualenv (16.0.0)

python/3.6.2

cycler 0.10.0 kiwisolver 1.0.1 matplotlib 3.0.0 numpy 1.13.1 pip 18.1 pyparsing 2.2.1 python-dateutil 2.7.3 scipy 0.19.1 setuptools 28.8.0 six 1.11.0 virtualenv 16.0.0

Panther

Last updated: 14/12/2018

python2/2.7.8

alabaster (0.7.10) appdirs (1.4.3) attrs (17.2.0) Automat (0.6.0) Babel (2.3.4) backports.shutil-get-terminal-size (1.0.0) backports.ssl-match-hostname (3.5.0.1) buildbot (0.8.12) buildbot-slave (0.8.12) configparser (3.5.0) constantly (15.1.0) cycler (0.10.0) Cython (0.25.2) decorator (4.0.11) docutils (0.13.1) entrypoints (0.2.2) enum34 (1.1.6) findspark (1.1.0) functools32 (3.2.3.post2) futures (3.1.1) h5py (2.7.0) imagesize (0.7.1) incremental (17.5.0) ipyparallel (6.0.2) ipython (5.3.0) ipython-genutils (0.2.0) Jinja2 (2.9.5) jupyter (1.0.0) jupyter-core (4.3.0) lxml (3.6.1) MarkupSafe (0.23) matplotlib (2.0.0) mpi4py (2.0.0) MySQL-python (1.2.5) nose (1.3.7) numpy (1.13.3) packaging (16.8) pandas (0.21.0) pandocfilters (1.4.1) pathlib2 (2.2.1) pexpect (4.2.1) pickleshare (0.7.4) PIL (1.1.7) Pillow (3.4.2) pip (9.0.1) prompt-toolkit (1.0.14) protobuf (3.2.0) ptyprocess (0.5.1) Pygments (2.2.0) pyparsing (2.2.0) pysqlite (2.8.2) python-dateutil (2.6.0) pytz (2016.4) requests (2.13.0) scandir (1.5) scipy (0.17.0) setuptools (21.0.0) simplegeneric (0.8.1) six (1.10.0) snowballstemmer (1.2.1) Sphinx (1.5.3) SQLAlchemy (0.7.10) sqlalchemy-migrate (0.7.2) subprocess32 (3.2.7) Tempita (0.5.3.dev0) traitlets (4.3.2) Twisted (17.1.0) virtualenv (15.1.0) wcwidth (0.1.7) wheel (0.29.0) zope.interface (4.4.1)

python3/3.6.0

appdirs (1.4.3) attrs (17.4.0) chaospy (2.2.5) clawpack (5.4.0) cycler (0.10.0) decorator (4.2.1) gimmik (2.1) h5py (2.6.0) Mako (1.0.7) MarkupSafe (1.0) matplotlib (2.0.0) mpi4py (3.0.0) networkx (2.0) numpy (1.14.0) pip (9.0.1) pluggy (0.6.0) py (1.5.2) pyparsing (2.2.0) pytest (3.3.2) python-dateutil (2.6.0) pytools (2017.6) pytz (2017.2) scipy (0.19.0rc2) setuptools (38.4.0) six (1.11.0) virtualenv (15.1.0)

Paragon

Last updated: 14/12/2017

python/2.7.8

appdirs (1.4.0) Cython (0.25.2) enum34 (1.1.6) glog (0.3.1) nose (1.3.7) numpy (1.14.1) packaging (16.8) PIL (1.1.7) Pillow (3.4.2) pip (9.0.1) protobuf (3.5.2) pyparsing (2.1.10) python-gflags (3.1.2) scipy (0.17.0) setuptools (38.2.3) six (1.10.0) virtualenv (15.1.0) wheel (0.30.0)

HOWTOs

Separate HOWTO pages will be provided for each installed application explaining its capabilities, license and other conditions, how to invoke it and how to submit jobs to the various resources. The list will be maintained on the portal under the "Hartree Community/ Wiki" pages and some sample files will be kept on the systems under the packages area.

Go to HOWTOs Page

Back to Contents Page