Maxeler User Guide

Hartree Centre Logo

Back to Contents Page

Maxeler System 21/3/2014

Using the Hartree EEC Data Flow Engine "deLorean" from Maxeler Technologies

This system comprises a small number of compute nodes (for compiling for DFE) and of DFEs (for running on the actual FPGA hardware).

Given this is a Hartree EEC Research Machine, there is less support than on Hartree Production Machines (such as Scafell Pike). There is also limited software applications installed (use "module avail")

PLEASE NOTE that delorean is due to be retired at the end of 2019

Quick Links

Accessing the Maxeler System

To access the Maxeler system you will need to be involved in an agreed research project. If you would like to propose a project, please contact the Hartree Helpdesk(e-Mail: hartree@stfc.ac.uk). Enquiries will be passed to Michael Bane who manages the Energy Efficient Research Computing Programme.

To log on to the Maxeler use:

ssh -X fpga.eecr.hartree.stfc.ac.uk

If logging in remotely you may find the X2go software helps remove some lag for X applications such as maxIDE. (Also see bottom of this page!)

Either of these will connect you to the login node which is maxlogin01 - the IP address is 193.62.123.17.

In addition to the (virtual) login node the system has 5x Xeon CPU host nodes plus 5x Maxeler DFE nodes. You will need to use the SGE batch system to obtain an interactive session on a compute node from where you should compile and submit to DFE.

Useful Commands

maxtop - this command is used to query the status of the DFEs which you have access to. Either set SLIC_CONF first (see below) or use

maxtop -r 192.168.74.198

This is the IP address associated with "maxorch" orchestrator (which handles reservations of DFEs etc)

qlogin - requests an interactive shell on a compute node

qhost - shows what CPU hosts are configured for batch scheduling. "qhost -q" shows which queues are available.

qstat - query the status of your jobs on the Grid Engine batch queue. Type "qstat -help" for more information.

For more information on SGE batch system, see (eg) reference manuals available from University of Liverpool, see: external link: https://arc.liv.ac.uk/trac/SGE and external link: http://arc.liv.ac.uk/SGE/htmlman/manuals.html.

Installed Software

The Maxeler system has its own PanFS (Panasas) disk file system.

The Maxeler compiler suite, tools and example programs are installed here: /panfs/panfs.maxeler/maxeler/maxcompiler-2014.1.1 (UPDATE: v2016 is now the default) Before using the compiler or tools you should source the settings.sh or settings.csh setup script in that directory.

Programming, Compiling & Running

overview

The Data Flow Engine is concerned with processing long streams of data - there is a large latency, but one running it will take input and produce output each clock cycle.

There are various methods available to programme FPGAs (eg VHDL, VeriLog). The Maxeler system provides a nice interface to FPGA programming that abstracts coding a little away from the hardware. Typically, one has

Building code for the DFE takes substantial time (~30 mins or a few hours) so there is a Simulator available for checking compilation and testing/debugging code. It is highly recommended to use the Simulator to ensure everything works as required before re-compiling but with the DFE as the target architecture.

The host code is linked to a library called SLiC which interacts with the MaxelerOS library to schedule kernels to DFE nodes. DFE nodes are connected to the host CPUs via a PCIe bus and have their own memory. The MaxCompiler converts code for the DFE into kernels which can directly modify the hardware (the Field Programmable Gate Arrays) to process the data stream as required.

The host application code is typically written in C, C++ or Fortran90 although other languages should be possible. This is compiled and linked to the SliC and MaxelerOS libraries in a standard way. Certain parts of the application will be chosen for acceleration (usually after some profiling) and will then need to be re-written in a Java like data-flow syntax to be processed into kernels by the MaxCompiler. This can be quite a lengthy process but could yield huge benefits.

Building and Running using the Command Line

The key steps are:

And after a successful build you can run the application on one or more DFEs. To run on a single DFE (still within the original 'qlogin' session):

You can ignore the initial warning about library mismatch.

Building and Running using the GUI "MaxIDE"

Many people find that the best way to start on the system is to use the MaxIDE and start a new "MaxCompiler Project".

NB we recently changed some scripts on delorean so please let the Hartree Helpdesk know of any issues with these (old) instructions.

To compile and run on the simulator (ie you do not need a DFE):

source /panfs/panfs.maxeler/maxeler/maxcompiler-2014.1.1/settings.sh
qlogin 
maxide

This will ask you you to confirm which workspace you want to use for project files; it can be set as the default. When the IDE has launched you will have access to the MaxCOmpiler documentation, tutorials and sample programs. You can import tutorial projects and modify them. There is also a Project Wizard for setting up an entirely new application.

Key things to note:

  • the definition of the Kernel as defined in its Java ".maxj" file drives the prototype argument list that will be called from the host C code
  • when working remotely you may find the interface slow to respond. If you prefer you can download a Maxeler VM from their "MyMaxeler" site and use that for the Simulation target. You will need to use delorean for compiling and running on the DFEs.

To target a DFE for compilation and running you will need to set some flags before running maxide, specifically

qlogin
source /panfs/panfs.maxeler/maxeler/maxcompiler-2016.1.1/settings.sh 
export PATH=/panfs/panfs.maxeler/maxeler/altera/13.1/quartus/bin:$PATH
export DISPLAY=:99
maxorch -r 192.168.74.198 -c reserve -i $USER -t MAIA
export SLIC_CONF='default_engine_resource=$USER^192.168.74.198'
maxide

Example Codes

Here is a very simple example of a code which takes a stream of numbers x(:) and computes y(:)=x(:)*x(:)+30.

Host code (.c).

MyKernel (.maxj).

Manager (.maxj).

A real application is typically developed, built and run using the MaxIDE interface which provides templates and greatly simplifies many of the steps including both simulation to check correctness and kernel to hardware mapping.

X2goserver

X2goserver has been installed on DeLorean. It allows for faster X-forwarding of the MaxIDE program.

To use the service, download the x2goclient (for Windows, Linux and Mac) from - external link: http://wiki.x2go.org/doku.php/download:start

Once you load x2goclient, add your user credentials and the hostname of the machine (fpga.eecr.hartree.stfc.ac.uk).

Where it says 'Session type' select 'Single application' Add to the command '/usr/bin/start_maxide'

Once you connect, MaxIDE should load. You may need to sort out some configuration settings such as X windows application.

Further Information

For general information, the Maxeler Technologies Web site is here: external link: https://www.maxeler.com .

This page explains the principe of Dataflow Computing: external link: https://www.maxeler.com/technology/dataflow-computing .

When logged onto the Maxeler system you will find examples and documentation as follows:

  • MaxCompiler documentation: /panfs/panfs.maxeler/maxeler/maxcompiler-2014.1.1/docs
  • Sample codes and tutorials: /panfs/panfs.maxeler/maxeler/maxcompiler-2014.1.1/examples

For more detailed technical information and documentation please use MaxIDE or ask to join the moderated Maxeler work site: external link: http://community.hartree.stfc.ac.uk/portal/site/maxeler .