Infrastructure/trickle - Revision history

Oe at 19:28, 2 March 2020

2020-03-02T19:28:03Z

Oe at 19:26, 2 March 2020

2020-03-02T19:26:34Z

Oe at 19:23, 2 March 2020

2020-03-02T19:23:16Z

Oe: Created page with "i am no big fan of the Abel arrayrun(1) facility. what i usually do is create a large file with as many command lines as i want to run jobs, each with whatever parameters tha..."

2019-04-13T14:47:14Z

Created page with "i am no big fan of the Abel arrayrun(1) facility. what i usually do is create a large file with as many command lines as i want to run jobs, each with whatever parameters tha..."

New page

i am no big fan of the Abel
arrayrun(1) facility. what i usually do is create a large file with
as many command lines as i want to run jobs, each with whatever
parameters that job requires. a silly example of such a master job
file could be something like

<pre>
for i in 0 1 2 3 4 5 6 7 8 9; do
for j in 0 1 2 3 4 5 6 7 8 9; do
echo "sbatch ${HOME}/echo.slurm ${i} ${j}";
done;
done > ~/echo.jobs
</pre>

assuming such a file, i have a script that ‘trickles’ through the
sequence of jobs, keeping up to some maximum limit of queue entries at
any point in time, and filling up the queue to the limit again as jobs
terminate. my idiom of setting into motion this process then goes as
follows:

<pre>
/projects/nlpl/operation/tools/trickle --start --limit 20 ~/echo.jobs
while true; do /projects/nlpl/operation/tools/trickle --limit 20
~/echo.jobs ; sleep 30; done
[19-02-16 15:00:37] trickle[20]: 20 jobs; 3 running; 0 new.
[19-02-16 15:01:07] trickle[20]: 17 jobs; 0 running; 3 new.
[19-02-16 15:01:38] trickle[23]: 17 jobs; 0 running; 3 new.
[19-02-16 15:02:10] trickle[26]: 20 jobs; 3 running; 0 new.
...
</pre>

the first integer is the pointer into the job sequence, 20 initially,
then at each step advancing by the number of new jobs submitted for
that call.

—just in case you might find this useful ... for all i know, this
script provides similar functionality to arrayrun(1), but i find it
more convenient to be able to pass each job its full command line
directly, without having to redirect on the job indices under
arrayrun(1) control.

@@ Line 21: / Line 21: @@
 <pre>
 /cluster/shared/nlpl/operation/tools/trickle --start --limit 20 ~/echo.jobs
-while true; do /cluster/shared/nlpl/operation/tools/trickle --limit 20
+while true; do
-~/echo.jobs ; sleep 30; done
+  /cluster/shared/nlpl/operation/tools/trickle --limit 20 ~/echo.jobs; sleep 30;
 [19-02-16 15:00:37] trickle[20]: 20 jobs; 3 running; 0 new.
 [19-02-16 15:01:07] trickle[20]: 17 jobs; 0 running; 3 new.

@@ Line 8: / Line 8: @@
 for i in 0 1 2 3 4 5 6 7 8 9; do
    for j in 0 1 2 3 4 5 6 7 8 9; do
-     echo "sbatch ${HOME}/echo.slurm ${i} ${j}";
+     echo "sbatch /cluster/shared/nlpl/operation/tools/echo.slurm ${i} ${j}";
    done;
 done > ~/echo.jobs

@@ Line 1: / Line 1: @@
-i am no big fan of the Abel
+i am no big fan of the SLURM
 arrayrun(1) facility.  what i usually do is create a large file with
 as many command lines as i want to run jobs, each with whatever
@@ Line 20: / Line 20: @@
 <pre>
-/projects/nlpl/operation/tools/trickle --start --limit 20 ~/echo.jobs
+/cluster/shared/nlpl/operation/tools/trickle --start --limit 20 ~/echo.jobs
-while true; do /projects/nlpl/operation/tools/trickle --limit 20
+while true; do /cluster/shared/nlpl/operation/tools/trickle --limit 20
 ~/echo.jobs ; sleep 30; done
 [19-02-16 15:00:37] trickle[20]: 20 jobs; 3 running; 0 new.