Table of Contents
LOGGING INTO ARC. 1
ACCESSING COMPUTE NODES ON ARC. 1
RUNNING JOBS INTERACTIVELY WITH THE SRUN COMMAND.. 3
SUBMITTING BATCH JOBS WITH THE SBATCH COMMAND.. 4
OTHER SLURM COMMANDS FOR JOB MANAGEMENT.. 8
RUNNING JOBS ON THE COMPUTE NODES ON ARC. 13
LOGGING INTO ARC
You can connect to Arc using your abc123 id and an SSH client/terminal. Two-factor authentication is required and hence, you would need the Duo application.
The hostname for accessing Arc is: login.arc.utsa.edu
The port number to connect is: 22
The complete SSH command is as follows, and abc123 in this command should be replaced with your actual abc123 id:
ssh -p 22 abc123@login.arc.utsa.edu
ACCESSING COMPUTE NODES ON ARC
This section covers an overview of the steps for requesting access to the compute nodes for running jobs interactively and in batch mode using the Slurm job-scheduler and workload manager. Slurm enables submitting, monitoring, and cancelling jobs on Arc.
There are multiple partitions or queues on Arc. Table 1 shows the queue-names, and the constraints on using them.
Table 1. Slurm job queues on Arc
You can get interactive access to a compute or GPU node by using the srun command from a login node.
If you are proceeding with running the commands below for the interactive session, please skip the steps for the batch job submission corresponding to each example.
Run the following command to get interactive access on a non-GPU compute node:
srun -p compute1 -n 1 -t 01:30:00 --pty bash
Run the following command to get interactive access on a GPU node:
srun -p gpu1v100 -n 1 -t 01:30:00 --pty bash
Note: There are multiple GPU and non-GPU partitions/queues on Arc. Please substitute the right partition/queue name (e.g., compute1 and gpu1v100) in the sruncommands above.
The Slurm sbatch command is used to submit a batch job to one of the Arc queues mentioned in Table 1 and following is its syntax:
[abc123@login003 ~]$ sbatch job_script.slurm
In the command above,job_script.slurmis a text file containing#SBATCHdirectives and shell commands that describe how the job that is being submitted will run. The details of the job script's contents depend on the type of job that you wish to run.
In the job script :(1) use#SBATCHdirectives to request computing resources (e.g., 2 nodes for 1 hour); and then (2) use regular Linux shell commands to specify the tasks that you would like to do once your job starts running. You can choose to run a single executable (of an application), or can combine different executables in the same job-script to run either one after another or simultaneously.
Table 2. Options to use with the #SBATCH directive in a Slurm job-script
|
Options
|
Full form of Option Name
|
Description
|
|
-J
|
--job-name
|
· Specify a name for the job
· This is a required option
|
|
-o
|
--output
|
· Instruct Slurm to write the job’s standard output to the file named after the -o flag
· This is a required option
|
|
-p
|
--partition
|
· Request a specific partition or queue for running the job
· This is a required option
|
|
-t
|
--time
|
· Specify the maximum anticipated run-time of the job
· This is a required option
|
|
-N
|
--nodes
|
· Specify the number of nodes required for the job
· This is a required option
|
|
-n
|
--ntasks
|
· The number of tasks that will be launched on a node
· The maximum number of tasks can be set as equal to the maximum number of physical cores in the node in a queue, or the maximum number of hyperthreads or virtual cores
· This is a required option
|
|
--mail-user=email_id
|
|
· Replace email_id with your actual email id to receive notifications related to your job start, end, and failure. This option is meant to be used in conjunction with the–mail-typeoption
· Using this option is not required
|
|
--mail-type=begin
|
|
· Here, a mail will be sent to notify the beginning of the job
· “begin” can be replaced with “end”, “fail”, or “all” to receive other notifications
· This option is meant to be used in conjunction with the –mail-user option
·Using this option is not required |
A sample batch job-script is shown in Listing 1 and the options are explained. Make sure you are connected to the Arc system (ssh -p 22 username@login.arc.utsa.edu) and are on the login node for submitting a batch job.
#!/bin/bash
#----------------------------------------------------
# Sample Slurm job script for Arc nodes
#SBATCH -J myjob # Job Name
#SBATCH -o myjob.o%j # Name of the stdout output file
#SBATCH -e myjob.e%j # Name of stderr error file
#SBATCH -p compute1 # Queue (partition) name
#SBATCH -N 1 # Total # of nodes (must be 1 for serial)
#SBATCH -n 1 # Total # of mpi tasks (should be 1 for serial)
#SBATCH -t 01:30:00 # Run time (hh:mm:ss)
#Put other commands such as module load if needed after this line (All commands must follow #SBATCH directives)
# Launch code...
./mycode.exe
Listing 1. Sample Slurm Job-Script
Note-1: If you want to activate the email notifications for tise job then you will need to add the following email in the Slurm job script shown above before you list any of the Linux shell commands to run the job.
#SBATCH --mail-type=ALL
#SBATCH --mail-user=username@utsa.edu
Note-2: The following Slurm job options are mandatory and must be included with every job:*[-n,-N,-t,-p]*
After you have submitted a job through Slurm - either interactively or through the batch job-script - there are additional commands that you can run to monitor or cancel the job, or create dependencies such as holding the execution of a future job till the current one has completed.
Table 3 : Overview of Other Job Management commands
|
Command
|
Syntax
|
Example
|
|
squeue
|
squeue -u uname
|
squeue -u abc123
Here abc123 is the username with which you log into Arc.
|
|
scancel
|
scancel jobid
|
scancel 8088547
Here8088547 is the job id that you received after your job was submitted.
|
|
sinfo
|
sinfo
|
sinfo
|
|
sbatch --dependency
|
sbatch --dependency=afterok:jobid jobscript
|
sbatch --dependency=afterok:8088568 new_job.slurm
Here8088568is the job id that you received after you submitted a previous job on which you would like to create a dependency in the current job.
|
|
scontrol
|
scontrol show job=jobid
|
scontrol show job=8090259
Here8090259is the job id that you received after you submitted a previous job, whose detailed information you wanted to check
|
|
sacct
|
sacct --starttime date
|
sacct --starttime 2021-07-10
Here2021-07-10 is the date on ar after which you want to see the jobs |
<span style="font-size: 18.0pt; font-family: 'Dosis
ExtraLight '; mso-fareast-font-family: 'Dosis ExtraLight '; mso-bidi-font-family: 'Dosis
ExtraLight '; color: #0c2340;" class='TMLhtml' >SQUEUE
You can check the status of your job - whether it is pending, running, or completing - by using the squeue command. This command can also help you find out the job ids associated with all your pending or running jobs so that you can use it with the scancelorsbatch --dependency command (its syntax is also shown in Table 3).
If you just enter thesqueue command and press enter, it will show you the list of all the jobs currently waiting or running on the system (and not necessarily yours):
$ squeue
JOBID PARTITION NAME USER ST TIMENODES NODELIST(REASON)
1429gpu2v100 test_job abc123 PD 0:00 2 (PartitionNodeLimit)
If you specify your username with the -u option to the squeuecommand then it will show you the status of all the jobs associated with your account:
$ squeue -u abc123
A sample output of the squeue command looks as follows:
JOBID PARTITION NAME USER ST TIMENODES NODELIST(REASON)
1429gpu2v100 test_job abc123 PD 0:00 2 (PartitionNodeLimit)
The column labeled "ST" displays each job's status:
<![if supportLists]>
● <![endif]>
"PD" means "Pending" (waiting)
<![if supportLists]>
● <![endif]>
"R" means "Running"
<![if supportLists]>
● <![endif]>
"CG" means "Completing" (or cleaning up after exiting the job script)
If you specify the--startoption to thesqueue command then it will show you the status of all the jobs associated with your account:
$ squeue --start
JOBID PARTITION NAME USER ST START_TIMENODES SCHEDNODES NODELIST(REASON)
1429gpu2v100 test_job abc123 PD N/A 2 (null) (PartitionNodeLimit)
<span style="font-size: 18.0pt; font-family: 'Dosis
ExtraLight '; mso-fareast-font-family: 'Dosis ExtraLight '; mso-bidi-font-family: 'Dosis
ExtraLight '; color: #0c2340;" class='TMLhtml' >SCANCEL
You can cancel a job as long as it is in the pending or running state using thescancel command. A sample of scancel command is shown in Table 3. You can find out the job ids associated with all your pending or running jobs by using the squeuecommand (its syntax is also shown in Table 3).
$ squeue -u abc123
JOBID PARTITION NAME USER ST TIMENODES NODELIST(REASON)
1429gpu2v100 test_job abc123 PD 0:00 2 (PartitionNodeLimit)
1440compute1 bash abc123R 4:11 1 c001
$ scancel 1440
<span style="font-size: 18.0pt; font-family: 'Dosis
ExtraLight '; mso-fareast-font-family: 'Dosis ExtraLight '; mso-bidi-font-family: 'Dosis
ExtraLight '; color: #0c2340;" class='TMLhtml' >SBATCH --DEPENDENCY
The sbatchcommand can be used to manage workflows that involve multiple steps such that the output of one step (or job) is the input to another step (or job) by using the--dependency option. The syntax and example of using this command is shown in Table 3. As an example, let us assume that we submit our first job (new_job1.slurm) as below and get a job id1441by the Slurm job scheduler.
$ sbatch new_job1.slurm
Submitted batch job 1441
If the completion of this job is required before we submit another job (new_job2.slurm), then we can create a dependency between this new job and the old job (1441) as follows:
$ sbatch --dependency=afterok:1441 new_job2.slurm
Submitted batch job 1442
<span style="font-size: 18.0pt; font-family: 'Dosis
ExtraLight '; mso-fareast-font-family: 'Dosis ExtraLight '; mso-bidi-font-family: 'Dosis
ExtraLight '; color: #0c2340;" class='TMLhtml' >SINFO
You can monitor the status of the different partitions or queues on Arc using the sinfo command. As an example, the following command will show the number of nodes that are Available, Idle, Other, and Total in each partition on Arc:
sinfo -S+P -o "%18P %8a %20F"
$ sinfo -S+P -o "%18P %8a %20F"
PARTITION AVAIL NODES(A/I/O/T)
bigmem up 0/2/0/2
compute1* up 0/65/0/65
compute2 up 0/25/0/25
compute3 up 0/0/0/0
computedev up 0/5/0/5
gpu1v100 up 0/30/0/30
gpu1vector up 0/0/0/0
gpu2a100 up 0/0/0/0
gpu2v100 up 0/5/0/5
gpudev up 0/2/0/2
softmatter up 0/20/0/20
testing up 0/3/1/4
<span style="font-size: 18.0pt; font-family: 'Dosis
ExtraLight '; mso-fareast-font-family: 'Dosis ExtraLight '; mso-bidi-font-family: 'Dosis
ExtraLight '; color: #0c2340;" class='TMLhtml' >SCONTROL
Thescontrolcommand provides detailed information about the configuration of a specific job. It is used to view or modify Slurm configuration including: job, job step, node, partition, reservation, and overall system configuration. As an example, let us assume that we have submitted a job (new_job1.slurm) as below and get a job id1441by the Slurm job scheduler.
$ sbatch new_job1.slurm
Submitted batch job 1441
We can see the detailed information about this job as follows:
$ scontrol show job=1441
JobId=1441 JobName =helloWorldC
UserId=abc123(860633341) GroupId =abc123(860633341) MCS_label=N/A
Priority=20681 Nice=0 Account=abc123 QOS=normal
JobState=COMPLETED Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag =1 Reboot=0 ExitCode =0:0
RunTime=00:00:00 TimeLimit =00:02:00 TimeMin =N/A
SubmitTime=2021-07-12T20:29:38 EligibleTime =2021-07-12T20:29:38
AccrueTime=2021-07-12T20:29:38
StartTime=2021-07-12T20:29:38 EndTime =2021-07-12T20:29:38 Deadline=N/A
PreemptTime=None SuspendTime =None SecsPreSuspend =0
LastSchedEval=2021-07-12T20:29:38
Partition=compute1 AllocNode:Sid=c001:19615
ReqNodeList=(null) ExcNodeList =(null)
NodeList=c002
BatchHost=c002
NumNodes=1 NumCPUs =80 NumTasks =1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=80,node=1,billing=80
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec =*
MinCPUsNode=1 MinMemoryNode =0 MinTmpDiskNode =0
Features=(null) DelayBoot =00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/home/abc123/documentation/serial_jobs/serialc/GNU/helloworldc.slurm
WorkDir=/home/abc123/documentation/serial_jobs/serialc/GNU
StdErr=/home/abc123/documentation/serial_jobs/serialc/GNU/helloWorldC.txt
StdIn=/dev/null
StdOut=/home/abc123/documentation/serial_jobs/serialc/GNU/helloWorldC.txt
Power=
<span style="font-size: 18.0pt; font-family: 'Dosis
ExtraLight '; mso-fareast-font-family: 'Dosis ExtraLight '; mso-bidi-font-family: 'Dosis
ExtraLight '; color: #0c2340;" class='TMLhtml' >SACCT
The sacctcommanddisplaysjob accounting data stored in the job accounting log file or Slurm database in a variety of forms for your analysis.This command displays information on jobs, job steps, status, and exit codes by default.You can tailor theoutputwith the use of the --format= option to specify the fields to be shown. As an example, following command will show all thejobs that started on or after the date passed as an argument to the starttime option:
$ sacct --starttime 2021-07-09
JobID JobNamePartition AccountAllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
1429 test_job gpu2v100 abc123 4 PENDING 0:0
1432 helloWorl+ compute1 abc123 80COMPLETED 0:0
1432.batch batch abc123 80COMPLETED 0:0
1432.extern extern abc123 80COMPLETED 0:0
1437 helloWorl+ compute1 abc123 80 FAILED 11:0
1437.batch batch abc123 80 FAILED 11:0
1437.extern extern abc123 80COMPLETED 0:0
A sample R script is shown below and all this code does is print “Hello World” to standard output.
Add execute permissions on the R script file as follows:
[username@login001]$ chmod +x program_name.r
The R programcan be run either in batch mode using a Slurm batch job-script or interactively on a compute node.
Running the script in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --pty bash
[username@c001]$ ml gnu8/8.3.0
[username@c001]$ ml R/3.6.1
[username@c001]$ Rscript program_name.r
"Hello World!!"
If you are currently on a compute node and would like to switch back to the login node then please enter the exit command as follows:
[username@c001]$ exit
Running the script in Batch-Mode: A sample Slurm batch job-script to run the serial program_name.r in batch mode is shown in Listing 2. This script should be run from a login node.
|
#!/bin/bash
#SBATCH -J program_name
#SBATCH -o program_name.txt
#SBATCH -p compute1
#SBATCH -t 00:05:00
#SBATCH -N 1
#SBATCH -n 1
ml gnu8/8.3.0
ml R/3.6.1
Rscript program_name.r |
Listing 2: Batch Job Script for R code (job_script16.slurm)
The job-script shown in Listing 2 can be submitted as follows:
[username@login001]$ sbatch job_script16.slurm
The output from the Slurm batch-job shown in Listing 2 can be checked by opening the output file as follows:
[username@login001]$ cat program_name.txt
"Hello World!!"
--
AdminUser - 28 Jul 2021
ARC
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.