Matlab

MATLAB is an integrated technical computing environment from the MathWorks that combines array-based numeric computation, advanced graphics and visualization, and a high-level programming language. Separately licensed toolboxes provide additional domain-specific functionality.

Mathworks provides MATLAB examples and tutorials for all experience levels here

Using MATLAB on Rivanna

You must always set up your environment in order to use MATLAB.  To load the most recent installed version run

module load matlab

You can work in the MATLAB desktop on the Rivanna frontend nodes; we recommend FastX for this application.  However, the time and memory that a job can use on the frontends are limited, so for longer jobs you should submit your job to compute nodes through SLURM.

If your Matlab job requires user interactions via the Matlab interface, you should use ijob to request an interactive job. For more information, see the documentation.

Submitting a SLURM Batch Job using a single core

Once your program is debugged, we recommend running in batch mode when possible. This runs the job in the background on a compute node. Write a script similar to the following:

#!/bin/bash
#SBATCH --nodes 1   #Number of nodes
#SBATCH --ntasks-per-node 1   #Number of cores
#SBATCH -t 01:00:00
#SBATCH -o output_filename
#SBATCH -p standard 
#SBATCH -A mygroup

module load matlab
matlab -nojvm -nodisplay -nosplash -singleCompThread -r "Mymain(myvar1s);exit"

The options -nojmv and -nodisplay suppress the Desktop and any attempt to run a graphical display. The -singleCompThread option is to ensure that all MATLAB built-in functions use only one thread. Some MATLAB functions are capable of running on multiple cores, but may not do so very efficiently. To use them in multicore mode you must request the appropriate number of cores and parallelize your core as shown below, but you will be charged SUs for all cores whether they are used effectively or not. Unless you are sure you can use the cores effectively it's generally best to restrict your job to one core.

The ;exit is very important to ensure that the job terminates when the computation is done.

Running Multiple Single-Core Matlab Jobs Using Slurm Job Arrays

The following slurm script shows how to run 10 single core Matlab jobs using slurm job arrays.

#!/bin/bash
# The slurm script file runParallelMultiple.slurm runs 
# multiple single core Matlab jobs using s Slurm job array 

#SBATCH --array=1-10
#SBATCH -p standard
#SBATCH -A hpc_build
#SBATCH --time=00:30:00
#SBATCH --mail-type=end
#SBATCH --mail-user=teh1m@virginia.edu
#SBATCH --job-name=runMultiple
#SBATCH --output=runMultiple_%A_%a.out
#SBATCH --error=runMultiple_%A_%a.err
#SBATCH --ntasks=1

# Load Matlab environment
module load matlab

# Create variable for slurm job and task ids
slurmArrayID="${SLURM_ARRAY_JOB_ID}${SLURM_ARRAY_TASK_ID}"
export slurmArrayID

% Create a temporary directory on scratch for any Job related files
mkdir -p /scratch/teh1m/slurmJobs/$slurmArrayID

# Shell script to sample output of top command
# while your job runs on compute node (optional)
#./sampleTop2.sh teh1m $slurmArrayID 15 &

# Run Matlab parallel program 
matlab -nodisplay -nosplash -singleCompThread -r "pcalc2(2000,${slurmArrayID});exit;" 

# remove workspace
rm -rf /scratch/teh1m/slurmJobs/$slurmArrayID

where the example code pcalc2.m is shown below. Note that passing the slurmArrayID variable allows the function to save output to a job-specific filenames.

function  [a]=pcalc2(nloop,jobid)
% Example using the parfor construct to calculate the maximum eignevalue
% of a random 300x300 matrix nloops times
% nloop: Number of time parfor loop should run
% jobid: Slurm job id  for save saving output to job specific file

if ischar(nloop) % checking data type on first input
    nloop=str2num(nloop);
end
if ischar(jobid) % checking data type on second input
    jobid=str2num(jobid);
end

% preallocate output array
a=zeros(nloop, 1);
% TIME CONSUMING LOOP
tic;
parfor i=1:nloop
    a(i)=FunctionTakesLongTime();
end
time=toc  % timing loop

% save output to a file
save(['pcalc_' num2str(jobid) '.out'],'time','a','-ascii')
end

function max_eig=FunctionTakesLongTime() 
% Computation intensive calculation dependent on matrix size
max_eig=max(abs(eig(rand(300))));
end

Running MATLAB in Parallel on Rivanna

If you have a job that can be structured to run across multiple cores, you can greatly speed up the time to your results. The Parallel Computing Toolkit allows you to distribute for loops over multiple cores using parfor and other parallel constructs in MATLAB. For more information on using the Parallel Computing Toolbox in MATLAB see the MathWorks documentation.

Parallel Matlab on a Single Compute Node

The example function pcalc2.m above uses a  parallel for loop (parfor) in MATLAB. To run your parallel MATLAB code across multiple cores on one compute node, you can use a slurm script similar to the following:

#!/bin/bash
#SBATCH -p standard
#SBATCH -A hpc_build
#SBATCH --time=12:00:00
#SBATCH --output=parforTest1.out
#SBATCH --error=parforTest.err
#SBATCH --nodes=1                #Number of nodes
#SBATCH --ntasks-per-node=20     #Number of cores per node

# Load Matlab environment
module load matlab

# Run Matlab parallel program program
matlab -nodisplay -nosplash  \
-r "parpool('local',19);pcalc2(2000,${SLURM_JOB_ID});exit;" 

Notice that the local profile is used. You may need to open MATLAB and edit the local parallel profile to specify a new maximum number of workers (the default is 20). Additionally you should specify the pool to have one less worker than the number of cores requested from SLURM. This is because the main MATLAB instance will be running on one of the cores requested.

Running Multiple Single-Node Parallel Matlab Jobs Using Slurm Job Arrays

#!/bin/bash
# The slurm script file runParallelMultiple.slurm runs 
# multiple parallel Matlabjobs using a Slurm job array 

#SBATCH --array=1-10
#SBATCH -p standard
#SBATCH -A hpc_build
#SBATCH --time=00:30:00
#SBATCH --mail-type=end
#SBATCH --mail-user=teh1m@virginia.edu
#SBATCH --job-name=runMultiple
#SBATCH --output=runMultiple_%A_%a.out
#SBATCH --error=runMultiple_%A_%a.err
#SBATCH --ntasks=20

# Load Matlab environment
module load matlab

# Create variable for slurm job and task ids
slurmArrayID="${SLURM_ARRAY_JOB_ID}${SLURM_ARRAY_TASK_ID}"
export slurmArrayID

% Create a temporary directory on scratch for any Job related files
mkdir -p /scratch/teh1m/slurmJobs/$slurmArrayID

# Shell script to sample output of top command
# while your job runs on compute node (optional)
#./sampleTop2.sh teh1m $slurmArrayID 15 &

# Set workers to one less that number of tasks (leave 1 for mater process)
numWorkers=$((SLURM_NTASKS-1))
export numWorkers

# Run Matlab parallel program 
matlab -nodisplay -nosplash -r "setPool1;pcalc2(10000,${slurmArrayID});exit;" 

# remove workspace
rm -rf /scratch/teh1m/slurmJobs/$slurmArrayID
% Script setPool1.m 

% create a local cluster object
pc = parcluster('local');

% explicitly set the JobStorageLocation to the temp directory that was created in your sbatch script
pc.JobStorageLocation = strcat('/scratch/teh1m/slurmJobs/', getenv('slurmArrayID'));

% start the matlabpool with maximum available workers
% control how many workers by setting ntasks in your sbatch script
parpool(pc, str2num(getenv('numWorkers')))

Parallel Matlab on Multiple Compute Nodes

To run Matlab parallel jobs that require more cores than are available on one compute node, you can launch the Matlab desktop on one of the Rivanna login nodes. The following MATLAB setup script will create the cluster profile for your account on Rivanna:

% The following set of commands are for running parallel Matlab programs across compute nodes
% of the Rivanna cluster using Matlab R2017a or R2017b. 

% For running parallel Matlab programs using on one compute node using multiple 
% cores, use the slurm script runParallelNode.slurm

% Load the module for Matlab from the xlinux command line.
module load matlab

% The following commands are executed from within Matlab 

configCluster  % set up initial configuration for running multi-node Matlab parallel jobs 
               % on Rivanna. This just needs to be done once, and its saved in Matlab’s 
               % parallel profiles 


c = parcluster; % Create a cluster object based on the profile
c.AdditionalProperties.Account = 'hpc_build' % account to charge job to
c.AdditionalProperties.QueueName = 'parallel' % queue to submit job to
c.AdditionalProperties.WallTime = '24:00:00' % amount of wall time needed for job
c.saveProfile
c.AdditionalProperties  % confirm above properties are set

% Additional configuration commands

% email address for Slurm to send email
c.AdditionalProperties.EmailAddress ='teh1m@virginia.edu'
% send email when job ends
c.AdditionalProperties.AdditionalSubmitArgs ='--mail-type=end'

Once this configuration is complete you can submit jobs to the cluster using the following commands:

c=parcluster
% Launch Matlab parallel code across two compute nodes 
j=c.batch(@pcalc2,1,{600,101},'pool',39); % Launch batch job to cluster 

% Arguments of c.batch in order

% Handle of function containing parallel code to run
% Number of function outputs
% Cell array of function inputs
% Setting of a pool of matlab workers
% Number of Matlab workers. There are 40 cores on two nodes 
% so use 39 for workers and one for master

% There are 256 MDCS licenses, so don’t use more that two nodes 
% per parallel job

% Commands to use to get back debug information if a job crashes

% For parallel jobs (i.e. calling batch with pool>0)
j.Parent.getDebugLog(j)

% Get the state of the job
j.State

% Don't return command prompt until job finishes
j.wait

% Get output from the job
j.fetchOutputs

Utilizing GPUs with Matlab

General guidelines on requesting GPUs on Rivanna.

Once your job has been granted its allocated GPUs, you can use the gpuDevice function to initialize a specific GPU for use with Matlab functions that can utilize the architecture of GPUs. For more information see the MathWorks documentation on GPU Computing in Matlab.

Compiling Your MATLAB Code

The number of licenses for MATLAB is limited; this is especially true for some of the specialty toolkits.  If you compile your code, you can run it without requiring any licenses.  This helps both you (since you will not need to worry about whether a license is available) and other users (by keeping licenses free). 

Compiling Matlab code to run on a Single Compute Core

The first requirement is that all your code must be in functions; none can be a script.  If you have a master script, turn it into a function called, for example, main.m.  To compile the example function pcalc2.m above, load the module for Matlab and then from within Matlab, type,

>> mcc -mv pcalc2.m

The compilation creates these files

 run_pcalc2.sh
 readme.txt
 pcalc2

 

Then run the compiled executable in the slurm file runSingleNode.slurm below.

#!/bin/bash
# Example slurm script for running a compiled serial Matlab code.

#SBATCH -p standard
#SBATCH -A hpc_build
#SBATCH --time=12:00:00
#SBATCH --output=parforTest1.out
#SBATCH --error=parforTest1.err
#SBATCH --mail-type=end
#SBATCH --mail-user=teh1m@virginia.edu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1

# Generate job-specific location for runtime cache
export MCR_CACHE_ROOT=/scratch/$USER/$SLURM_JOB_ID

# Run Matlab serial program program as standalone
 ./run_pcalc2.sh /apps/software/standard/core/matlab/R2017a/MATLAB_Runtime/v92 \
 2000 $SLURM_JOB_ID

When you are running many jobs from a compiled code, they will share a location to cache libraries and other data, which by default is .mcrCache in your top-level home directory.  Since all your jobs are trying to use the same files, corruption can occur.  We recommend that you separate your cache locations for each job.  This can be accomplished by adding to your SLURM script the line

export MCR_CACHE_ROOT=/scratch/$USER/$SLURM_JOB_ID

Running Multiple Single-Node Compiled Parallel Matlab Jobs Using Slurm Job Arrays

The following slurm script launches multiple compiled parallel matlab codes with each running on the cores of one compute node, spread across multiple compute nodes.

#!/bin/bash
# The slurm script file runParallelMultiple.slurm runs
# multiple parallel Matlab jobs, either from within Matlab or as a
# complied executable, using Slurm job array

#SBATCH --array=1-10
#SBATCH -p standard
#SBATCH -A hpc_build
#SBATCH --time=00:30:00
#SBATCH --mail-type=end
#SBATCH --mail-user=teh1m@virginia.edu
#SBATCH --job-name=runMultiple
#SBATCH --output=runMultiple_%A_%a.out
#SBATCH --error=runMultiple_%A_%a.err
#SBATCH --ntasks=20

# Load Matlab environment
module load matlab

# Create variable for slurm job and task ids
slurmArrayID="${SLURM_ARRAY_JOB_ID}${SLURM_ARRAY_TASK_ID}"
export slurmArrayID

% Create a temporary directory on scratch
mkdir -p /scratch/teh1m/slurmJobs/$slurmArrayID

# Shell script to sample output of top command
# while your job runs on compute node (optional)
# ./sampleTop2.sh teh1m $slurmArrayID 15 &

# Set workers to one less that number of tasks (leave 1 for mater process)
numWorkers=$((SLURM_NTASKS-1))
export numWorkers

# Run Matlab parallel program
# matlab -nodisplay -nosplash -r "setPool1;pcalc2(50000,${slurmArrayID});exit;"

# Run Matlab parallel compiled program
./run_parallelpcalc2.sh /apps/software/standard/core/matlab/R2017a/MATLAB_Runtime/v92 \
50000 $slurmArrayID

# remove workspace
rm -rf /scratch/teh1m/slurmJobs/$slurmArrayID

This is the matlab wrapper function that actually gets compiled.

function [a]=parallelpcalc2(nloop,jobid)
% Function wrapper for compiled parallel pcalc1
% nloop is the number of iterations passed to pcalc2.m
% jobid is the slurm job id, also passed to pcalc2.m

setmcruserdata('ParallelProfile','local.settings');
setPool1 % sets up pool of local workers

% Call parallelized code
[a]=pcalc2(nloop,jobid);
end

 

% Script setPool1.m 

% create a local cluster object
pc = parcluster('local');

% explicitly set the JobStorageLocation to the temp directory that was created in your sbatch script
pc.JobStorageLocation = strcat('/scratch/teh1m/slurmJobs/', getenv('slurmArrayID'));

% start the matlabpool with maximum available workers
% control how many workers by setting ntasks in your sbatch script
parpool(pc, str2num(getenv('numWorkers')))