In these cases, as we have learned thus far, we would have to manually rerun the sbatch command multiple times for each of the aforementioned scenarios. Fortunately, SLURM allows us to automate this procedure using job arrays. Each array is considered to be one \u201carray job\u201d that has a specific ID. Each element of the array is one array task, which has it\u2019s own sub-ID. For example, if your \u201carray job\u201d ID was 1212985, your first \u201carray task\u201d that runs would have an ID of 1212985_0.<\/p>\n\n
Example 1 Accordion Closed<\/span><\/h4>\n <\/span>\n <\/div>\n <\/a>\n \n \n\nIn this first simple example, we have several commands that we would like to run as separate jobs. As we have learned so far, we would have to individually run all of these commands, or create a script that runs each command in a different job step. If we did this, the jobs could take a while because they will be run in sequence. If we use a job array we can ensure that the jobs run in parallel.<\/p>\n
First, we make a file called “commandlist” that stores all of the commands we would like slurm to run:<\/p>\n
commandlist<\/strong><\/p>\nsleep 5
\nsleep 4
\nsleep 8
\nsleep 3
\nsleep 6<\/p>\n
Next, we create a bash script called “command_array.sh” that utilizes the SBATCH ––array feature (line numbers have been added for readability):<\/p>\n
command_array.sh<\/strong><\/p>\n1 #!\/bin\/sh
\n2 #SBATCH ––job-name=”command-array”
\n4 #SBATCH ––output=”\/scratch\/mkg52\/command_array-%A_%a.out”
\n5 #SBATCH ––workdir=”\/home\/mkg52″
\n6 #SBATCH ––array=1-5
\n7
\n8 command=$(sed –n “$SLURM_ARRAY_TASK_ID”p commandlist)
\n9 srun $command<\/p>\n
There are three lines in particular in this script that make use of the job array feature:<\/p>\n
\n- Line 6 tells SLURM to create an array of 5 items, numbered 1 through 5. This should be changed to match the number of jobs you need to run. In our case, we want this range to match the number of commands in our “commandlist” file.<\/li>\n
- Line 8 utilizes one of SLURM’s built in variables, called SLURM_ARRAY_TASK_ID. This accesses the specific task ID of the current task in the job array (e.g. 1 for the first task) and can be used like any bash variable. In this example, “sed” is being used to get the contents of a particular line in the “commandlist” file using SLURM_ARRAY_TASK_ID. For the first task, the “command” variable will be “sleep 5”.<\/li>\n
- Line 4 uses a shorthand method of accessing the job array ID and array task ID and embedding them into the name of the output file. The “%A” represents the SLURM_ARRAY_JOB_ID variable (e.g. 1212985) and the “%a” represents the SLURM_ARRAY_TASK_ID variable (e.g. 1). This would generate an output file similar to “command_array-1212985_1.out” for the first element of the array.<\/li>\n<\/ul>\n
To run the job array, simply use sbatch like you would with any other script. (The creation of the array is taken care of by SLURM when it sees the ––array option)<\/p>\n
[mkg52@wind ~ ]$ sbatch command_array.sh<\/p>\n
You can see all of the jobs in the array running at once (or at least as many as SLURM will allow based on your priority or fairshare) using the “squeue” command:<\/p>\n
![\"Screen](\"https:\/\/in.nau.edu\/wp-content\/uploads\/sites\/208\/2018\/01\/Screen-Shot-2018-01-18-at-2.07.30-PM-944x576.png\")
\n<\/p><\/body><\/html>\n\n <\/div>\n<\/div>\n\n\n
In this first simple example, we have several commands that we would like to run as separate jobs. As we have learned so far, we would have to individually run all of these commands, or create a script that runs each command in a different job step. If we did this, the jobs could take a while because they will be run in sequence. If we use a job array we can ensure that the jobs run in parallel.<\/p>\n
First, we make a file called “commandlist” that stores all of the commands we would like slurm to run:<\/p>\n
commandlist<\/strong><\/p>\n sleep 5 Next, we create a bash script called “command_array.sh” that utilizes the SBATCH ––array feature (line numbers have been added for readability):<\/p>\n command_array.sh<\/strong><\/p>\n 1 #!\/bin\/sh There are three lines in particular in this script that make use of the job array feature:<\/p>\n To run the job array, simply use sbatch like you would with any other script. (The creation of the array is taken care of by SLURM when it sees the ––array option)<\/p>\n [mkg52@wind ~ ]$ sbatch command_array.sh<\/p>\n You can see all of the jobs in the array running at once (or at least as many as SLURM will allow based on your priority or fairshare) using the “squeue” command:<\/p>\n
\nsleep 4
\nsleep 8
\nsleep 3
\nsleep 6<\/p>\n
\n2 #SBATCH ––job-name=”command-array”
\n4 #SBATCH ––output=”\/scratch\/mkg52\/command_array-%A_%a.out”
\n5 #SBATCH ––workdir=”\/home\/mkg52″
\n6 #SBATCH ––array=1-5
\n7
\n8 command=$(sed –n “$SLURM_ARRAY_TASK_ID”p commandlist)
\n9 srun $command<\/p>\n\n
\n<\/p><\/body><\/html>\n\n <\/div>\n<\/div>\n\n\n
![\"sbatch](\"https:\/\/in.nau.edu\/wp-content\/uploads\/sites\/208\/2018\/01\/sbatch-job-array-triangle-1108x600.png\")
<\/p>\n![\"Job](\"https:\/\/in.nau.edu\/wp-content\/uploads\/sites\/208\/2018\/01\/Job-Array-example-2-job-array-result-960x186.png\")
<\/p>\n<\/body><\/html>\n\n <\/div>\n<\/div>\n\n\n![\"\"](\"https:\/\/in.nau.edu\/wp-content\/uploads\/sites\/208\/2018\/01\/Job-Array-Example-3-filelist-960x198.png\")
<\/p>\n![\"Job](\"https:\/\/in.nau.edu\/wp-content\/uploads\/sites\/208\/2018\/01\/Job-Array-example-3-output-960x176.png\")
<\/p>\n