Dependent jobs

If you have reasons to perform analysis on files in a certain order, you can tell SLURM that ensuing jobs depend on the previous jobs completing successfully. This is done with the ‐‐dependency option. The syntax for this is:

––dependency=afterok:<job id>

where <job id> is the ID of the job that must be completed before the current job can start. For example, if you had a job currently running that had an ID of 123456 and wanted to submit another job that wouldn’t run until 123456 was finished, you would submit your new job as follows:

[abc123@wind ~]$ sbatch ––dependency=afterok:123456 <analysis program>

It is important to note that when you set the ‐‐dependency option on a job, that job will never start unless the dependent job completed successfully (see FAQ for more information).

In the previous job array example, all of our jobs ran at once. Let’s do the example again, but this time we’ll use dependencies. Since we are only submitting 5 jobs, we could just submit them one at a time, manually placing a dependency on each one. However, most people will typically be submitting 10-100’s of jobs that all require dependencies, which will be very cumbersome to do by hand. The easiest way to accomplish this is with a “for loop”, which is demonstrated in the script below:

driver.sh

#!/bin/bash
INPUT="/scratch/abc123/bigdata/input/
FLAG=0
for file in $(find "$INPUT" -name "file*"); do
    # first time we can't depend on a job
    if [[ "$FLAG" -eq 0 ]]; then
        prev_id=$(sbatch analysis.sh "$file" output | awk '{ print $4 }')
        FLAG=1
    else
        prev_id=$(sbatch --dependency="afterok:$prev_id" analysis.sh "$file" output | awk '{ print $4 }')
    fi
done

The loop looks at every file in our input folder (file_1 through file_5) and runs the analysis.sh script. In order to create dependencies, we must get the ID of the previous job and set that as the dependency for the current job. The first time through the loop, the FLAG variable is 0, so we submit the job with no dependencies and then set FLAG to 1. Subsequent times through the loop, we add the previous job (the $prev_id variable) as a dependency to the current job. The bold text demonstrates this.
We start our jobs by running the driver.sh script.

[abc123@wind /scratch/abc123/bigdata]$ bash ./driver.sh

We can prove that our analysis files are being created in the correct order by monitoring the output directory every second.

[abc123@wind /scratch/abc123/bigdata] $ while [ 1 ]; do
ls output
sleep 1
done