{"id":4051,"date":"2026-08-19T21:33:26","date_gmt":"2026-08-19T21:33:26","guid":{"rendered":"https:\/\/in.nau.edu\/arc\/?page_id=4051"},"modified":"2026-08-19T21:33:26","modified_gmt":"2026-08-19T21:33:26","slug":"resource-allocation","status":"publish","type":"page","link":"https:\/\/in.nau.edu\/arc\/resource-allocation\/","title":{"rendered":"Resource Allocation"},"content":{"rendered":"<h1>Resource Allocation<\/h1>\n<p><strong>Resource allocation<\/strong> in HPC is the set of resources you are given to use for your jobs. A <strong>resource<\/strong> in this context is defined as a given number of CPU cores or GPUs, a certain amount of CPU or GPU usage per second, or a given amount of RAM.<\/p>\n<p>There are several approaches to ensuring fair resource management, which include:<\/p>\n<ul>\n<li><strong>F<\/strong>irst <strong>I<\/strong>n, <strong>F<\/strong>irst <strong>O<\/strong>ut (FIFO)<\/li>\n<li><strong>Backfill<\/strong> (smallest job first)<\/li>\n<li><strong>Fairshare<\/strong>, which dynamically adjusts priority based on users&#8217; recent usage.<\/li>\n<\/ul>\n<h2>Slurm<\/h2>\n<p>Resource allocation on Monsoon is handled by a system called <a href=\"https:\/\/slurm.schedmd.com\/overview.html\" target=\"_blank\" rel=\"noopener\">slurm<\/a>. Slurm grants access to resources over a given block of time, called an allocation. When you submit a job to Monsoon, you are actually requesting a given set of resources (e.g 1 CPU core, 1GB of RAM, and 1 GPU) over a given time span (e.g 1 hour). Slurm then schedules these blocks through a priority system, described below.<\/p>\n<p>If a job is canceled or finishes early, a gap is created in the schedule between that job and the one that follows it. Depending on the job, this means Slurm can sometimes place a job into a much sooner time slot than it was originally scheduled for. This process is known as\u00a0<strong>backfilling<\/strong>.<\/p>\n<h2>Fairshare<\/h2>\n<p><strong>Fairshare<\/strong> is a value between 0 and 1 that is calculated based on a user&#8217;s recent resource usage. The\u00a0<em>higher<\/em> the fairshare value is, the\u00a0<em>higher<\/em> the priority for scheduling jobs. The\u00a0<em>more<\/em> resources a user has used recently, the\u00a0<em>lower<\/em> their fairshare value will be. This means that users who have used fewer resources recently will have a higher priority for their jobs to be scheduled.<\/p>\n<p>More information on the fairshare algorithm can be found in the <a href=\"https:\/\/slurm.schedmd.com\/fair_tree.html#algorithm\" target=\"_blank\" rel=\"noopener\">slurm documentation<\/a>.<\/p>\n<h2>LevelFS<\/h2>\n<p><strong>LevelFS<\/strong> is a value that represents the level of fairness in resource allocation. The ranges of LevelFS levels and their meanings are as follows:<\/p>\n<ul>\n<li>LevelFS &lt; 1: High priority for scheduling jobs<\/li>\n<li>LevelFS 1 &#8211; 10: Moderate priority for scheduling jobs<\/li>\n<li>LevelFS &gt; 10: Low priority for scheduling jobs<\/li>\n<\/ul>\n<h2>TRESRunMins<\/h2>\n<p>Separately from fairshare, Monsoon also implements a system for limiting jobs called TRESRunMins, which determines the upper bound on how many CPU-minutes a given Slurm account (e.g faculty member or classroom) can use.<\/p>\n<p>This is a variable in Slurm that represents a number assigned to an account (or QoS) which limits the total number of remaining CPU minutes which your running jobs can occupy. Having this feature enabled on Monsoon helps with:<\/p>\n<ul>\n<li>Flexible resource limiting<\/li>\n<li>Staggering jobs<\/li>\n<li>Increasing cluster utilization<\/li>\n<li>More accurate resource requests<\/li>\n<\/ul>\n<p>The current value for the limit is 3,000,000. This value is sometimes increased as cluster utilization drops, which allows folks to use the idle cores. To calculate the TRESRunMins for your jobs, multiply the number of CPUs being used by the time limit remaining, then multiply that number by the total number of jobs you are running, like so:<\/p>\n<p>tresrunmins = sumofjobs( cpus * time remaining )<\/p>\n<p>Examples:<\/p>\n<p>500000 = 24x 1cpu, 2 week jobs<\/p>\n<p>500000 = 49x 1 cpu, 1 week jobs<\/p>\n<p>500000 = 347x 1 cpu, 1 day jobs<\/p>\n<p>500000 = 21x 16 cpu, 1 day jobs<\/p>\n<p>500000 = 130x 16 cpu, 4 hr jobs<\/p>\n<p>500000 = 520x 4 cpu, 4 hr jobs<\/p>\n<p>To see the current TRESRunMins for a single account or all accounts, use<\/p>\n<p style=\"padding-left: 30px\">sshare -l -A &lt;account name&gt; # single account<br \/>\nsshare -l # all accounts<\/p>\n<p>In the output, the pertinent column will be labeled CPURunMins and will be the farthest to the right. This number changes dynamically as jobs change state.<\/p>\n<h2>Checking your Fairshare and LevelFS<\/h2>\n<p>The <span style=\"font-size: 16px;font-family: monospace;border: 1px solid;border-radius: 4px;padding: 0px 4px 0px;border-color: #BBBBBB\">sshare<\/span> command can be used to check your fairshare and LevelFS values. This command provides information about the resource usage and priority of users in the SLURM system.<\/p>\n<p><code>$ sshare -Po user,fairshare,levelfs | grep $USER<br \/>\nabc123|0.456209|4.588063<br \/>\n<\/code><\/p>\n<p>In this example, <span style=\"font-size: 16px;font-family: monospace;border: 1px solid;border-radius: 4px;padding: 0px 4px 0px;border-color: #BBBBBB\">abc123<\/span> has a fairshare value of <span style=\"font-size: 16px;font-family: monospace;border: 1px solid;border-radius: 4px;padding: 0px 4px 0px;border-color: #BBBBBB\">0.456209<\/span> and a LevelFS value of <span style=\"font-size: 16px;font-family: monospace;border: 1px solid;border-radius: 4px;padding: 0px 4px 0px;border-color: #BBBBBB\">4.588063<\/span>.<\/p>\n<p>A fairshare value of <span style=\"font-size: 16px;font-family: monospace;border: 1px solid;border-radius: 4px;padding: 0px 4px 0px;border-color: #BBBBBB\">0.456209<\/span> means a moderate amount of resources were used recently. A LevelFS value of <span style=\"font-size: 16px;font-family: monospace;border: 1px solid;border-radius: 4px;padding: 0px 4px 0px;border-color: #BBBBBB\">4.588063<\/span> indicates a moderate priority for scheduling jobs.<\/p>\n<div class=\"notranslate\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Resource Allocation Resource allocation in HPC is the set of resources you are given to use for your jobs. A resource in this context is defined as a given number of CPU cores or GPUs, a certain amount of CPU or GPU usage per second, or a given amount of RAM. There are several approaches [&hellip;]<\/p>\n","protected":false},"author":2964,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"ring_central_script_selection":"","footnotes":""},"class_list":["post-4051","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/4051","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/users\/2964"}],"replies":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/comments?post=4051"}],"version-history":[{"count":3,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/4051\/revisions"}],"predecessor-version":[{"id":4117,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/4051\/revisions\/4117"}],"wp:attachment":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/media?parent=4051"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}