Visualisation2

Remote Visualisation

This page explains our approach to Visual Data Analysis or Remote Visualisation. It includes use of the XRV facility on Scafell Pike, the Panther and Paragon interactive nodes and also the Hartree Centre Visual Computing Suite.

See also Visualisation for information about how our previous facilities were used.

Terminology and VirtualGL

We refer to "remote visualisation" in the case that images are generated by rendering without copying the data elsewhere. In the case of Scafell Pike there are special purpose nodes with nVidia GPUs which have acess to the Lustre file system. These are referred to as XRV nodes (their purpose is to run the eXtreme Remote Visualisation software). On Panther and Paragon all compute nodes have GPUs and can access the local GPFS file systems.

To explain in more detail how VirtualGL works on Scafell Pike, Panther and Paragon, also see background information from: external link: https://en.wikipedia.org/wiki/VirtualGL .

VGL Transport

Figure 1: The VGL Transport with a Remote 2D X Server (i.e. your client desktop machine)

Note that in X11 terminology the remote machine is the one with the display (the client in Figure 1) and the local one is the server doing the rendering (application server). When using the VGL Transport, the 3D rendering occurs on the application server, but the 2D rendering occurs on the client machine, i.e. your desktop. VirtualGL compresses the rendered images from the 3D application and sends them as a video stream to the client, which decompresses and displays the video stream in real time.

The "interactive" Script

This example shows how to log into Scafell Pike from desktop or Viz Suite Linux machine or Ubuntu shell on Windows-10 (you will need the VirtualGL client installed):

For Panther and Paragon use either 193.62.123.226 or 193.62.123.227.

The next steps in using VirtualGL are greatly simplified with the "interactive" script written by Colin Morey. E.g. to start a remote visualisation session on Scafell Pike with vglconnect do:

"interactive -q XRV -W 180"

This opens an Xterm window from which tasks can be started, including vglrun.

Note that by default the script will target the interactive queue on each cluster. On Panther and Paragon the interactive nodes have attached GPUs, on Scafell Pike only the XRV nodes have GPUs. The default wall clock time is 40 minutes.

For more options do "interactive -h".

There are a few simple checks you can do to see that all is OK. Firstly check the X11 server is running on the GPU by typing "nvidia-smi". This should show something like the following - note the Xorg process is running:

| NVIDIA-SMI 384.66                 Driver Version: 384.66                    |
+                                                                             +
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                                                             |
|   0  Tesla K40m          Off  | 00000000:82:00.0 Off |                    0 |
| N/A   30C    P8    21W / 235W |      7MiB / 11439MiB |      0%      Default |
+                                                                             +
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|                                                                             |
|    0     19733    G   Xorg                                             6MiB |

Now check that VirtualGL can use the GPU for rendering:

"vglrun -d :0.0 glxinfo | grep renderer" should show:

OpenGL rendered string: Tesla K40m/PCI2/SSE2

This should be the same GPU sype as above. If for any reason you don't get the expected output please submit an incident ticket via the Service Now Portal

How is this using VirtualGL?

VirtualGL opens an SSH port with X11 forwarding plus another port for the 3D content. If you login into an interactive node or XRV node you can connect tunnels to these using VGL_PORT etc. The following shows one way to do this.

ParaView with VirtualGL

The following uses the above steps to log into Scafell Pike and launch ParaView. First log in and access an XRV node.

Now launch ParaView.

Yes it really is that easy!

OSPRay with VirtualGL

Intel OSPRay is an open source, scalable and portable ray tracing engine for high performance and high fidelity visualisation on Intel Architecture CPUs. Here we are using it on the KNL partition of Scafell Pike.

The purpose of OSPRay is to provide an open, powerful and easy to use rendering library that allows one to build applications that use ray tracing based rendering for interactive applications (including both surface and volume based visualisations). OSPRay is completely CPU based, and runs on anything from laptops, to workstations, to compute nodes in HPC systems. We will still use the VirtualGL transport to ship the images out.

Firstly use the "interactive" script to allocate the required resources and connect to a master process on an XRV node.

"interactive -v -q parallelDebugSP -k 64 -W 60"

This connects one process to an XRV node and 64 processes to KNL nodes (i.e. 1 node) for 60 minutes (note that the parallelDebugSP queue permits a mixture of architectures).

Once the script returns an Xterm you can launch OSPRay with Intel MPI is then as follows:

Note this is only an example and ths OSPRay command should be tailored to your data sets and options.

Possible Issues

When you have finished using the interactive or XRV nodes please log off because they are a limited resource. Check that your interactive job has really finished e.g. do "bjobs". If there is anything still running it will show up as follows:

JOBID      USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
12103239   my-acco RUN   XRV        hcxlogin1   sqg1cintr24 *ash -norc Mar 29 11:39

Do "bkill 12103239" to kill it.

Also check that the "interactive" script is not still running, it may start several processes. "ps" shows:

   PID TTY          TIME CMD
 10373 pts/21   00:00:01 bash
 22912 pts/21   00:00:00 interactive
 22951 pts/21   00:00:00 interactive
 22952 pts/21   00:00:00 interactive
 22953 pts/21   00:00:00 interactive
 24645 pts/21   00:00:00 ps

To close then do "killall interactive". This should not happen unless the lSF job hit its wall clock limit.

Alternative Methods 1: Virtual Frame Buffers

Xvfb is an option, e.g. for making snapshots during an application run, but it may not be appropriate because OpenGL won't benefit from hardware acceleration on the GPUs, and FBOs (frame buffer objects) aren't supported, which are necessary for CUDA inter-operability with OpenGL. Nevertheless, this might be a workable option where simply using X11 transport isn't acceptable because its too slow, or where users can't do X11 forwarding. You can do this on Scafell Pike:

Then copy to a machine with a real display and do "xwud -in xwdout"

Alternative Methods 2: XRV

This will be covered in a future update to this document.