If you have reasons to perform analysis on files in a certain order, you can tell SLURM that ensuing jobs depend on the previous jobs completing successfully. This is done with the\u00a0\u2010\u2010dependency<\/strong>\u00a0option. The syntax for this is:<\/p>\n where <job id><\/span> is the ID of the job that must be completed before the current job can start. For example, if you had a job currently running that had an ID of 123456<\/span> and wanted to submit another job that wouldn’t run until 123456<\/span> was finished, you would submit your new job as follows:<\/p>\n It is important to note that when you set the \u2010\u2010dependency<\/span> option on a job, that job will never start unless the dependent job completed successfully (see\u00a0FAQ\u00a0for more information).<\/p>\n In the\u00a0previous job array\u00a0example, all of our jobs ran at once. Let’s do the example again, but this time we’ll use dependencies. Since we are only submitting 5 jobs, we could just submit them one at a time, manually placing a dependency on each one. However, most people will typically be submitting 10-100’s of jobs that all require dependencies, which will be very cumbersome to do by hand. The easiest way to accomplish this is with a “for loop”, which is demonstrated in the script below:<\/p>\n driver.sh<\/strong><\/p>\n The loop looks at every file in our input folder (file_1<\/span> through file_5<\/span>) and runs the analysis.sh<\/span> script. In order to create dependencies, we must get the ID of the previous job and set that as the dependency for the current job. The first time through the loop, the FLAG<\/span> variable is 0<\/span>, so we submit the job with no dependencies and then set FLAG<\/span> to 1<\/span>. Subsequent times through the loop, we add the previous job (the $prev_id<\/span> variable) as a dependency to the current job. The bold text demonstrates this. We can prove that our analysis files are being created in the correct order by monitoring the output directory every second.<\/p>\n Dependent jobs If you have reasons to perform analysis on files in a certain order, you can tell SLURM that ensuing jobs depend on the previous jobs completing successfully. This is done with the\u00a0\u2010\u2010dependency\u00a0option. The syntax for this is: \u2013\u2013dependency=afterok:<job id> where <job id> is the ID of the job that must be completed before […]<\/p>\n","protected":false},"author":76,"featured_media":145,"parent":71,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"","_relevanssi_noindex_reason":"","ring_central_script_selection":"","footnotes":""},"class_list":["post-154","page","type-page","status-publish","has-post-thumbnail","hentry"],"_links":{"self":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/154","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/users\/76"}],"replies":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/comments?post=154"}],"version-history":[{"count":4,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/154\/revisions"}],"predecessor-version":[{"id":3809,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/154\/revisions\/3809"}],"up":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/pages\/71"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/media\/145"}],"wp:attachment":[{"href":"https:\/\/in.nau.edu\/arc\/wp-json\/wp\/v2\/media?parent=154"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\u2013\u2013dependency=afterok:<job id><\/code><\/pre>\n[abc123@wind ~]$ sbatch \u2013\u2013dependency=afterok:123456 <analysis program><\/code><\/pre>\n#!\/bin\/bash\r\nINPUT=\"\/scratch\/abc123\/bigdata\/input\/\"\r\nFLAG=0\r\nfor file in $(find \"$INPUT\" -name \"file*\"); do\r\n # first time we can't depend on a job\r\n if [[ \"$FLAG\" -eq 0 ]]; then\r\n prev_id=$(sbatch analysis.sh \"$file\" output | awk '{ print $4 }')\r\n FLAG=1\r\n else\r\n prev_id=$(sbatch --dependency=\"afterok:$prev_id\"<\/strong> analysis.sh \"$file\" output | awk '{ print $4 }')\r\n fi\r\ndone<\/code><\/pre>\n
\nWe start our jobs by running the driver.sh<\/span> script.<\/p>\n[abc123@wind \/scratch\/abc123\/bigdata]$ bash .\/driver.sh<\/code><\/pre>\n[abc123@wind \/scratch\/abc123\/bigdata] $ while [ 1 ]; do\r\nls output\r\nsleep 1\r\ndone<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"