Resource Allocation

Resource allocation in HPC is the set of resources you are given to use for your jobs. A resource in this context is defined as a given number of CPU cores or GPUs, a certain amount of CPU or GPU usage per second, or a given amount of RAM.

There are several approaches to ensuring fair resource management, which include:

  • First In, First Out (FIFO)
  • Backfill (smallest job first)
  • Fairshare, which dynamically adjusts priority based on users’ recent usage.

Slurm

Resource allocation on Monsoon is handled by a system called slurm. Slurm grants access to resources over a given block of time, called an allocation. When you submit a job to Monsoon, you are actually requesting a given set of resources (e.g 1 CPU core, 1GB of RAM, and 1 GPU) over a given time span (e.g 1 hour). Slurm then schedules these blocks through a priority system, described below.

If a job is canceled or finishes early, a gap is created in the schedule between that job and the one that follows it. Depending on the job, this means Slurm can sometimes place a job into a much sooner time slot than it was originally scheduled for. This process is known as backfilling.

Fairshare

Fairshare is a value between 0 and 1 that is calculated based on a user’s recent resource usage. The higher the fairshare value is, the higher the priority for scheduling jobs. The more resources a user has used recently, the lower their fairshare value will be. This means that users who have used fewer resources recently will have a higher priority for their jobs to be scheduled.

More information on the fairshare algorithm can be found in the slurm documentation.

LevelFS

LevelFS is a value that represents the level of fairness in resource allocation. The ranges of LevelFS levels and their meanings are as follows:

  • LevelFS < 1: High priority for scheduling jobs
  • LevelFS 1 – 10: Moderate priority for scheduling jobs
  • LevelFS > 10: Low priority for scheduling jobs

TRESRunMins

Separately from fairshare, Monsoon also implements a system for limiting jobs called TRESRunMins, which determines the upper bound on how many CPU-minutes a given Slurm account (e.g faculty member or classroom) can use.

This is a variable in Slurm that represents a number assigned to an account (or QoS) which limits the total number of remaining CPU minutes which your running jobs can occupy. Having this feature enabled on Monsoon helps with:

  • Flexible resource limiting
  • Staggering jobs
  • Increasing cluster utilization
  • More accurate resource requests

The current value for the limit is 3,000,000. This value is sometimes increased as cluster utilization drops, which allows folks to use the idle cores. To calculate the TRESRunMins for your jobs, multiply the number of CPUs being used by the time limit remaining, then multiply that number by the total number of jobs you are running, like so:

tresrunmins = sumofjobs( cpus * time remaining )

Examples:

500000 = 24x 1cpu, 2 week jobs

500000 = 49x 1 cpu, 1 week jobs

500000 = 347x 1 cpu, 1 day jobs

500000 = 21x 16 cpu, 1 day jobs

500000 = 130x 16 cpu, 4 hr jobs

500000 = 520x 4 cpu, 4 hr jobs

To see the current TRESRunMins for a single account or all accounts, use

sshare -l -A <account name> # single account
sshare -l # all accounts

In the output, the pertinent column will be labeled CPURunMins and will be the farthest to the right. This number changes dynamically as jobs change state.

Checking your Fairshare and LevelFS

The sshare command can be used to check your fairshare and LevelFS values. This command provides information about the resource usage and priority of users in the SLURM system.

$ sshare -Po user,fairshare,levelfs | grep $USER
abc123|0.456209|4.588063

In this example, abc123 has a fairshare value of 0.456209 and a LevelFS value of 4.588063.

A fairshare value of 0.456209 means a moderate amount of resources were used recently. A LevelFS value of 4.588063 indicates a moderate priority for scheduling jobs.