Job Arrays

There may be times that you would like to send many files as input to a program. Instead of having to do this one at a time, you can set up a job array to do this automatically. In this next example, we will be using a shell script called exercise1.sh that takes input files that are numbered and puts the data into an output file.

Exercise1.sh
#!/bin/bash

#SBATCH --job-name=exercise1
#SBATCH --time=00:01:00
#SBATCH --mem=1
#SBATCH --input=/scratch/NAUID/playground/files/input_%a.csv
#SBATCH --output=/scratch/NAUID/playground/output_%a.txt
#SBATCH --chdir=/scratch/NAUID/playground/files
#SBATCH --array=1-3

filename="input_${SLURM_ARRAY_TASK_ID}.csv"
srun echo "File: $filename"
srun cat $filename

In this example there are two lines that we use which are very important to be able to access the files. In line 6 we specify the input file. Since our files are labeled in a formatting of input_x.csv, then we can specify our input files using input_%a.csv. The %a specification denotes the job array ID (index) number. In line 12 we use a variable called SLURM_ARRAY_TASK_ID. This variable is similar to %a and is used to denote the usage of the job array ID (index) number.

Similar to example 1, in this example we will be taking in different input files from a specified directory. Instead of reading the files that are labeled with sequential numbers, we will be reading files that are labeled nonnumerical titles using a file list. In this next example, we will be using a shell script called exercise2.sh that takes input files from a directory that are nonnumerical and puts the data into an output file.

File_list.txt
avondale.csv
flagstaff.csv
gilbert.csv
goodyear.csv
phoenix.csv
williams.csv Exercise2.sh #!/bin/bash #SBATCH --job-name=exercise2 #SBATCH --time=00:01:00 #SBATCH --mem=1 #SBATCH --output=/scratch/NAUID/playground/output_%a #SBATCH --chdir=/scratch/NAUID/playground/ #SBATCH --array=1-6 file=$(awk "NR==${SLURM_ARRAY_TASK_ID}" file_list.txt) srun echo "Data from $file: " srun cat ./cityfiles/$file

In this example there is one line that makes this different from example 1. In line 10 we are using the awk command. We are using awk to to read the file line by line, where NR is equivalent to the job array ID (index) number and NR is equivalent to the number of records seen so far (line number). Line 12 is then using the file name that we got from line 10 and writing the data to our output file.

Taking what we learned in example 2, we can take the input files and run them against multiple programs using a command list. This can be used in the chance that you need to take your data files  and run them against multiple programs. In the example below, we are taking our input files and running them against the following scripts: add_numbers, multiply_numbers, and subtract_numbers. In the programs we are using we pass through three numbers and compute calculations.

command_list
add_numbers
multiply_numbers
subtract_numbers
 add_numbers #!/bin/env python3

import sys

data=sys.argv[1]
data=data.split(",")
num1=data[0]
num2=data[1]
num3=data[2]
total=int(num1)+int(num2)+int(num3)
print(total)
 multiply_numbers #!/bin/env python3

import sys
data=sys.argv[1]
data=data.split(",")
num1=data[0]
num2=data[1]
num3=data[2]
total=int(num1)*int(num2)*int(num3)
print(total)
 subtract_numbers #!/bin/env python3

import sys

data=sys.argv[1]
data=data.split(",")
num1=data[0]
num2=data[1]
num3=data[2]
total=int(num1)-int(num2)-int(num3)
print(total)
 Exercise3.sh #!/bin/bash

#SBATCH --job-name=exercise3
#SBATCH --time=00:01:00
#SBATCH --mem=1
#SBATCH --output=/scratch/NAUID/playground/output_%a
#SBATCH --chdir=/scratch/NAUID/playground
#SBATCH --array=1-6

module load python/3.latest
file=$(awk "NR==${SLURM_ARRAY_TASK_ID}" file_list.txt)  
srun echo "File: $file"
for line in `cat command_list`; do
    echo "Script: $line"
    for data in `cat ./cityfiles/$file`; do
        srun python $line $data
    done
done

In the example above, similar to example 2 line 12 is obtaining the file names from the file_list. From there, starting at line 17 we have started a loop that gets each line of the command_list and grabs the program to run. Then line 20 grabs each line of data in our input files and passes it to the program as seen in line 21. Once the job runs you will see output files like the one seen below.

Output_1
File: avondale.csv
Script: add_numbers
369
639
1893
Script: multiply_numbers
1860867
9663597
251239591
Script: subtract_numbers
-123
-213
-631

Input_1.csv
x1,x2,x3
123,345,12
213,230,127
543,345,209
631,183,324
345,789,901
Avondale.csv
123,345,12
213,230,127
631,183,324