Job arrays allow you to leverage SLURM\u2019s ability to create multiple jobs from one script. Many of the situations where this is useful include:<\/p>\n
In these cases, as we have learned thus far, we would have to manually rerun the sbatch command multiple times for each of the aforementioned scenarios. Fortunately, SLURM allows us to automate this procedure using job arrays. Each array is considered to be one \u201carray job\u201d that has a specific ID. Each element of the array is one array task, which has it\u2019s own sub-ID. For example, if your \u201carray job\u201d ID was 1212985<\/span>, your first \u201carray task\u201d that runs would have an ID of 1212985_0<\/span>.<\/p>\n NOTE:<\/strong><\/p>\n Take note that if you use these examples that you will need to replace NAUID<\/span> with your own id, and that you will need to take out the line numbers. Also, we use a directory called playground and you will need to create one as well or it will not run.<\/p>\n If your input files are numerically sequenced, each individual (sub-)job in the array read a single, unique input file: one array (sub-)job for each input file. (E.g.: a job-array set to range from 1-4 would utilize one file, from file-1 through file-4, over its total of 4 sub-jobs.)<\/p>\n To set-up this type of job-array, a researcher could set the start-index\/end-index of the job-array to match the iterating part of the input files; or they could sequentially rename existing input files. Then, by using $SLURM_ARRAY_TASK_ID<\/span> as part of a filename, each sub-job could operate on its own separate input file. For the 4-files example: The job-script for the overall array-job would then just run whatever number-crunching tasks were necessary on a derived filename:<\/p>\n If necessary input files aren’t sequentially named, it’s still quite easy to set-up a job-array to iterate through the entire set, with just a bit of basic shell-scripting.<\/p>\n Helpful BASH scripting tips:<\/b><\/p>\n Again, the job-script for the overall array-job would then just run whatever number-crunching tasks were necessary on a derived filename:<\/p>\nExamples<\/h2>\n
Iteratively named input\/reference files<\/h3>\n
\n$ ls -1 ~\/data
\nsample-1.dat
\nsample-2.dat
\nsample-3.dat
\nsample-4.dat<\/code><\/p>\n#SBATCH --array=1-4\r\n\r\ntaskfile=\"~\/data\/sample-$SLURM_ARRAY_TASK_ID.dat\"\r\nsrun analyze_datafile \"$taskfile\"\r\nsrun process_datafile \"$taskfile\"\r\n<\/code><\/pre>\nIrregularly named input\/reference files<\/h3>\n
\n
\nvariable_name=$()<\/span><\/li>\n
\ncommand_1 | command_2<\/span><\/li>\n
\n<piped_input> | awk “NR==#”<\/span>
\nor<\/i>
\nawk “NR==#” <input_file><\/span><\/li>\n<\/ul>\n<\/div>\n$ ls -1 ~\/data\/
\nsample_2021_01.dat
\nsample_2021_07.dat
\nsample_2022_01.dat
\nsample_2022_07.dat
\n$ filelist=$( ls -1 ~\/data\/ )
\n$ echo \"$filelist\" | awk \"NR==1\"
\nsample_2021_01.dat
\n$ echo \"$filelist\" | awk \"NR==2\"
\nsample_2021_07.dat
\n$ echo \"$filelist\" | awk \"NR==3\"
\nsample_2022_01.dat
\n$ echo \"$filelist\" | awk \"NR==4\"
\nsample_2022_07.dat<\/code><\/p>\n#SBATCH --array=1-4\r\n\r\nfilelist=$( ls -1 ~\/data\/ )\r\ntaskfile=$( echo \"$filelist\" | awk \"NR==$SLURM_ARRAY_TASK_ID\" )\r\nsrun analyze_datafile \"$taskfile\"\r\nsrun process_datafile \"$taskfile\"<\/code><\/pre>\nSingle input source with varying parameters<\/h3>\n
$ cat ~\/coords.txt
\n35.19N,111.6W
\n33.57N,112.1W
\n32.15N,110.9W
\n$ cat ~\/coords.txt | awk \"NR==1\"
\n35.19N,111.6W
\n$ cat ~\/coords.txt | awk \"NR==2\"
\n33.57N,112.1W
\n$ cat ~\/coords.txt | awk \"NR==3\"
\n32.15N,110.9W
\n$ taskcoord=$(cat ~\/coords.txt | awk \"NR==$SLURM_ARRAY_TASK_ID\")<\/code><\/p>\nOld Examples<\/h2>\n\n
\n