Major Rivanna Upgrade Approaching

A major operating system upgrade is scheduled for Rivanna on October 10, 2017. The core system will move from Centos 6.8 to Centos 7.3. Most compilers and community applications will be upgraded as well. In addition, Rivanna will gain several new nodes with hardware accelerators. We will offer 10 nodes with 4 NVIDIA K80 GPGPUs, 4 nodes with 4 NVIDIA P100 GPGPUs, and 8 nodes of Intel Knight’s Landing Many-Integrated Core processors with 64 cores/256 threads per node.

Users will be permitted to log on to a test environment, rivanna4.hpc.virginia.edu, starting October 2. This is a limited environment consisting of several older nodes and the new accelerator nodes and should be used to test your software in the new environment. Some changes to the partitions on this testbed are a standard queue that replaces the economy and serial queues, and a new queue knl for Knight’s Landing. The GPGPUs are in the gpu queue as before, but you must use the "gres" option to request that you wish to use the hardware.  If you need a specific gpu architecture, k80 or p100 is requested by a type, either k80 or p100, e.g.

#SBATCH –p gpu
#SBATCH --gres=gpu:p100

For more than one gpu on a node, use

#SBATCH --gres=gpu:p100:2

The maximum number of boards on a node is 4.

Finally, all the older, out-of-warranty hardware that made up the “economy” partition has been retired from Rivanna and replaced by 25 new nodes with 28 cores and 256GB RAM per node in the standard queue. 

Please read the new documentation for updates to SLURM.

At the upgrade each user home directory will be migrated to a new home directory with 50GB of space. Please be aware that this is no longer the ITS Home Directory Service. All your files will have been copied by the time you first log in, but new files you create will not be updated on the old home directory. Should you need to access your old home directory, it will be available at /sfs/blue/yourid.  The new home directory will have snapshots but no full backups.  If full backups are essential you can lease storage from ITS.

As part of the upgrade we will switch from the older environment modules system we have been using to the newer lmod system. Most of the commands will remain the same, but lmod is more oriented to a hierarchical system and will not allow incompatible compilers and libraries to be loaded.

Python users: the modules are

anaconda2

for Python 2.7 or

anaconda3

for Python 3.   These are the only Python distributions on the new Rivanna software stack.  If you installed your own packages with pip install --user, please check whether they work; you may need to reinstall them.

R users: if you installed your own libraries you will probably have to reinstall them, since R has been built with a newer compiler.

If you compile your own code you will need to recompile.  Please read the new documentation for the modules system.

If you use FastX, please be sure you have a recent client (Version 2.082 or later). We are using Gnome 3 with Classic view.  You will have multiple workspaces available in the lower right.  Log out through the down-pointing arrow in the upper right, or simply terminate your FastX session.