 |
|
You are here: Foswiki>Sandbox Web>CC++FORTRANTesting (25 Jul 2022, AdminUser)Edit Attach
RUNNING C, C++, FORTRAN, PYTHON, AND R CODE - BOTH SERIAL AND PARALLEL MODES ARE COVERED WHERE APPLICABLE
This section covers the steps to run sample serial and parallel code in C, C++, Fortran, Python, and R using both interactive and batch job submission modes. Where relevant, separate steps for using Intel and GNU compilers are covered.
Please note that all code should be run only on the compute nodes either interactively or in batch mode. Please DO NOT run any code on the login nodes. Acceptable use of login nodes is as follows: installing code, file transfer, file editing, and job submission and monitoring.
1. CLONE THE GITHUB REPOSITORY
If you want to get a copy of all the programs and scripts used in the examples shown in this document then you can clone the GitHub repository for the examples using the following command:
git clone https://github.com/ritua2/documentation
If you cloned the GitHub repository, you can switch to the documentation directory with the following command to find the scripts and sample programs referred to throughout the document:
cddocumentation
Thedocumentation directory structure is shown below:
.
└── documentation
├── cuda_jobs
│ ├── cudac
│ │ ├── helloworldccuda.slurm
│ │ └── hello_world.cu
│ ├── cudacpp
│ │ ├── helloworldcppcuda.slurm
│ │ └── hello_world.cu
│ └── cudaf
│ ├── hello_world.cuf
│ └── helloworldfcuda.slurm
├── GNUparallel
│ ├── hello_world.c
│ ├── helloworldcgnup.slurm
│ └── myout.txt
├── mpi_jobs
│ ├── mpic
│ │ ├── hello_world.c
│ │ └── helloworldcmpi.slurm
│ ├── mpicpp
│ │ ├── hello_world.cpp
│ │ └── helloworldcppmpi.slurm
│ └── mpif
│ ├── hello_world.f90
│ └── helloworldfmpi.slurm
├── multiExes
│ ├── mpi_multiexes
│ │ ├── bye_world.c
│ │ ├── hello_world.c
│ │ └── helloworldcmpim.slurm
│ └── serial
│ ├── hello_world.c
│ └── helloworldcm.slurm
├── openmp_jobs
│ ├── openmpc
│ │ ├── GNU
│ │ │ ├── hello_world.c
│ │ │ └── helloworldcopenmp.slurm
│ │ └── Intel
│ │ ├── hello_world.c
│ │ └── helloworldcopenmpi.slurm
│ ├── openmpcpp
│ │ ├── GNU
│ │ │ ├── hello_world.cpp
│ │ │ └── helloworldopenmpcpp.slurm
│ │ └── Intel
│ │ ├── hello_world.cpp
│ │ └── helloworldopenmpcppi.slurm
│ └── openmpf
│ ├── GNU
│ │ ├── hello_world.f90
│ │ └── helloworldfopenmp.slurm
│ └── Intel
│ ├── hello_world.f90
│ └── helloworldfopenmpi.slurm
├── serial_jobs
│ ├── serialc
│ │ ├── GNU
│ │ │ ├── hello_world.c
│ │ │ └── helloworldc.slurm
│ │ └── Intel
│ │ ├── hello_world.c
│ │ └── helloworldci.slurm
│ ├── serialcpp
│ │ ├── GNU
│ │ │ ├── hello_world.cpp
│ │ │ └── helloworldcpp.slurm
│ │ └── Intel
│ │ ├── hello_world.cpp
│ │ └── helloworldcppi.slurm
│ ├── serialf
│ │ ├── GNU
│ │ │ ├── hello_world.f90
│ │ │ └── helloworldf.slurm
│ │ └── Intel
│ │ ├── hello_world.f90
│ │ └── helloworldfi.slurm
│ ├── serialpy
│ │ ├── hello_world.py
│ │ └── helloworldpy.slurm
│ └── serialR
│ ├── hello_world.r
│ └── helloworldr.slurm
When you switch to the subdirectories within the documentation folder, the Slurm job script files are available with the *.slurmextension. While the program files within the subdirectories have names beginning with hello_world.*, the corresponding files in the code listings shown in the rest of the document have names beginning with program_name.* .
If you do not want to clone the aforementioned GitHub repository, you should be able to copy the code shown in the listings into the respective files (for example: program_name.c) according to the instructions given.
2. COMPILING AND RUNNING A SAMPLE SERIAL C PROGRAM
A sample C program is shown in Listing 1. All this code does is print “Hello World!!” to standard output.
# include <stdio.h>int main(){printf("Hello World!!");return 0;}
Listing 1: Sample C program - (../documentation/serial_jobs/serialc/GNU/hello_world.c)
If you would like to compile the C example using the GNU C compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ gcc -o helloCOuthello_world.c
If you would like to compile the C example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icc -o helloCOuthello_world.c
The executable helloCOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --pty bash
[username@c001]$ ./helloCOut
Hello World!!
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloCOut corresponding to the serial program_name.c in batch mode is shown in Listing 2. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldC#SBATCH -o helloWorldC.txt#SBATCH -p compute1#SBATCH -t 00:02:00#SBATCH -N 1#SBATCH -n 1./helloCOut
Listing 2: Batch Job Script for C code (../documentation/serial_jobs/serialc/GN/helloworldc.slurm)
The job-script shown in Listing 2 can be submitted as follows:
[username@login001]$ sbatchhelloworldc.slurm
The output from the Slurm batch-job shown in Listing 2 can be checked by opening the output file as follows:
[username@login001]$ cathelloWorldC.txt
Hello World!!
3. COMPILING AND RUNNING A SAMPLE SERIAL C++ PROGRAM
A sample C++ program is shown in Listing 3. This program will print “Hello World” to standard output.
# include <iostream>using namespace std;int main(){cout<<"Hello World!!";return 0;}
Listing 3: Sample C++ program (../documentation/serial_jobs/serialcpp/GNU/hello_world.cpp)
If you would like to compile the C++ example using the GNU CPP compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ g++ -o helloCPPOut\hello_world.cpp
If you would like to compile the C++ example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icpc -o helloCPPiOut\hello_world.cpp
The executablehelloCPPOut or helloCPPiOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed directly on the terminal:
[username@login001]$srun -p compute1 -n 1 -t 00:05:00 --pty bash
[username@c001]$ ./helloCPPOut
Hello World!!
If you are currently on a compute node and would like to switch back to the login node then please enter the exit command as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloCPPOutcorresponding to the serial hello_world.cpp in batch mode is shown in Listing 4. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCPP#SBATCH -o helloWorldCPP.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1./helloCPPOut
Listing 4: Batch Job Script for C++ code (../documentation/serial_jobs/serialcpp/GNU/hello_world.cpp)
The job-script shown in Listing 4 can be submitted as follows:
[username@login001]$ sbatchhelloworldcpp.slurm
The output from the Slurm batch-job shown in Listing 4 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldCPP.txt
Hello World!!
4. COMPILING AND RUNNING A SAMPLE SERIAL FORTRAN PROGRAM
A sample Fortran program is shown in Listing 5. This program will print “Hello World!!” to the standard output.
Listing 5: Sample Fortran Program ( ../documentation/serial_jobs/serialf/GNU/hello_world.f90)program helloprint *, 'Hello, World!!'end program hello
If you would like to compile the Fortran example using the GNU Fortran compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ gfortran -o helloF90Out \hello_world.f90
If you would like to compile the Fortran example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ ifort -o helloF90iOut\Hello_world.f90.f90
The executableshelloF90Outor helloF90iOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --pty bash
[username@c001]$ ./helloF90Out
Hello World!!
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedexefilenamecorresponding to the serial program_name.f90 in batch mode is shown in Listing 6. This script should be run from a login node.
Listing 6: Batch Job Script for Fortran code (../ documentation/serial_jobs/serialf/GNU/helloworldf.slurm)#!/bin/bash#SBATCH -J helloWorldF90#SBATCH -o helloWorldF90.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1./helloF90Out
The job-script shown in Listing 6 can be submitted as follows:
[username@login001]$ sbatchhelloworldf.slurm
The output from the Slurm batch-job shown in Listing 6 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldF90.txt
Hello World!!
5. RUNNING A SAMPLE PYTHON PROGRAM USING PYTHON 3
A sample Python program is shown in Listing 7. This program will also print “Hello World!!” to standard output.
print('Hello World!!')
Listing 7: Sample Python program (../documentation/serial_jobs/serialpy/hello_world.py)
The Python programcan be run either in batch mode using a Slurm batch job-script or interactively on a compute node.
Running the code in Interactive-Mode: The program can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --pty bash
[username@c001]$ python3 hello_world.py
Hello World!!
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the code in Batch-Mode: A sample Slurm batch job-script to run the serial program_name.py in batch mode is shown in Listing 8. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldpy#SBATCH -o helloWorldpy.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1python3 hello_world.py
Listing 8: Batch Job Script for Python code ( ../documentation/serial_jobs/serialpy/hello_world.py)
The job-script shown in Listing 8 can be submitted as follows:
[username@login001]$ sbatchhelloworldpy.slurm
The output from the Slurm batch-job shown in Listing 8 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldpy.txt
Hello World!!
6. COMPILING AND RUNNING A SAMPLE C+MPI CODE IN PARALLEL MODE
A sample C + MPI program is shown in Listing 9. This program will print “Hello world from processor #, rank # out of # processors” to standard output. The “#” signs in the aforementioned quoted text will be replaced with the processor name, rank, and total number of MPI processes participating in the computation.
#include <mpi.h>#include <stdio.h>int main(int argc, char** argv) {// Initialize the MPI environmentMPI_Init(NULL, NULL);// Get the number of processesint world_size;MPI_Comm_size(MPI_COMM_WORLD, &world_size);// Get the rank of the processint world_rank;MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);// Get the name of the processorchar processor_name[MPI_MAX_PROCESSOR_NAME];int name_len;MPI_Get_processor_name(processor_name, &name_len);// Print off a hello world messageprintf("Hello world from processor %s, rank %d out of %d processors\n",processor_name, world_rank, world_size);// Finalize the MPI environment.MPI_Finalize();return 0;}
Listing 9: Sample C+MPI code (../documentation/mpi_jobs/mpic/hello_world.c)
If you would like to compile the C + MPI example using the Intel OneAPI compiler and MVAPICH2 MPI library, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ ml mvapich2
[username@login001]$ mpicc -o helloCMPIOut\
hello_world.c
The executablehelloCMPIOut can berun either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 12 -N 2 -t 00:05:00 --pty bash
[username@c001]$ mpirun -np 12 ./helloCMPIOut
Hello world from processor c002, rank 10 out of 12 processors
Hello world from processor c002, rank 8 out of 12 processors
Hello world from processor c002, rank 6 out of 12 processors
Hello world from processor c002, rank 9 out of 12 processors
Hello world from processor c002, rank 11 out of 12 processors
Hello world from processor c002, rank 7 out of 12 processors
Hello world from processor c001, rank 4 out of 12 processors
Hello world from processor c001, rank 1 out of 12 processors
Hello world from processor c001, rank 3 out of 12 processors
Hello world from processor c001, rank 5 out of 12 processors
Hello world from processor c001, rank 2 out of 12 processors
Hello world from processor c001, rank 0 out of 12 processors
If you are currently on a compute node and would like to switch back to the login node then please enter the exit command as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloCMPIOutcorresponding to the serial program_name.c in batch mode is shown in Listing 10. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloCMPIOut#SBATCH -o helloCMPIOut.txt#SBATCH -p compute1#SBATCH -t 00:10:00#SBATCH -N 2#SBATCH -n 12mpirun -np 12 ./helloCMPIOut
Listing 10: Batch Job Script for C+MPI code (job_script5.slurm)
The job-script shown in Listing 10 can be submitted as follows:
[username@login001]$ sbatchhelloworldcmpi.slurm
The output from the Slurm batch-job shown in Listing 10 can be checked by opening the output file as follows:
[username@login001]$ cat helloCMPIOut.txt
Hello world from processor c002, rank 10 out of 12 processors
Hello world from processor c002, rank 8 out of 12 processors
Hello world from processor c002, rank 6 out of 12 processors
Hello world from processor c002, rank 9 out of 12 processors
Hello world from processor c002, rank 11 out of 12 processors
Hello world from processor c002, rank 7 out of 12 processors
Hello world from processor c001, rank 4 out of 12 processors
Hello world from processor c001, rank 1 out of 12 processors
Hello world from processor c001, rank 3 out of 12 processors
Hello world from processor c001, rank 5 out of 12 processors
Hello world from processor c001, rank 2 out of 12 processors
Hello world from processor c001, rank 0 out of 12 processors
7. COMPILING AND RUNNING SAMPLE MPI PROGRAM IN C++
A sample C++ MPI program is shown in Listing 11. This program will print “Hello world from processor #, rank # out of # processors” to standard output. The “#” signs in the aforementioned quoted text will be replaced with the processor name, rank, and total number of MPI processes participating in the computation.
If you would like to compile the C++ example using the Intel OneAPI and MVAPICH2 MPI library, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ ml mvapich2
[username@login001]$ mpicxx -o helloCPPMPIOut\
hello_world.cpp
The executablehelloCPPMPIOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Listing 11: Sample MPI program with C++ (.. /documentation/mpi_jobs/mpicpp/hello_world.cpp)#include <mpi.h>#include <iostream>using namespace std;int main(int argc, char** argv) {// Initialize the MPI environmentMPI_Init(NULL, NULL);// Get the number of processesint world_size;MPI_Comm_size(MPI_COMM_WORLD, &world_size);// Get the rank of the processint world_rank;MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);// Get the name of the processorchar processor_name[MPI_MAX_PROCESSOR_NAME];int name_len;MPI_Get_processor_name(processor_name, &name_len);// Print off a hello world messagecout<<"Hello World from processor "<<processor_name<<", rank "<<world_rank<<"out of "<<world_size<<"processors\n";// Finalize the MPI environment.MPI_Finalize();return 0;}
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 12 -N 2 -t 00:05:00 --pty bash
[username@c001]$ mpirun -np 12 ./helloCPPMPIOut
Hello World from processor c001, rank 0out of 12processors
Hello World from processor c001, rank 1out of 12processors
Hello World from processor c001, rank 2out of 12processors
Hello World from processor c001, rank 5out of 12processors
Hello World from processor c001, rank 3out of 12processors
Hello World from processor c001, rank 4out of 12processors
Hello World from processor c002, rank 6out of 12processors
Hello World from processor c002, rank Hello World from processor c002, rank 8out of 12processors
7out of 12processors
Hello World from processor c002, rank 11out of 12processors
Hello World from processor c002, rank 10out of 12processors
Hello World from processor c002, rank 9out of 12processors
Note: It is common to see the output printed in a non-deterministic manner - in the example above the process rank 7 and rank 8 overlap each other in the writing step.
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloCPPMPIOutcorresponding to the serial program_name.cpp in batch mode is shown in Listing 12. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCPPMPI#SBATCH -o helloWorldCPPMPI.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 2#SBATCH -n 12mpirun -np 12 ./helloCPPMPIOut
Listing 12: Batch Job Script for MPI with C++ code (../documentation/mpi_jobs/mpicpp/helloworldcppmpi.slurm)
The job-script shown in Listing 12 can be submitted as follows:
[username@login001]$ sbatchhelloworldcppmpi.slurm
The output from the Slurm batch-job shown in Listing 12 can be checked by opening the output file as follows:
[username@login001]$ cat program_name.txt
Hello World from processor c001, rank 0out of 12processors
Hello World from processor c001, rank 1out of 12processors
Hello World from processor c001, rank 2out of 12processors
Hello World from processor c001, rank 5out of 12processors
Hello World from processor c001, rank 3out of 12processors
Hello World from processor c001, rank 4out of 12processors
Hello World from processor c002, rank 6out of 12processors
Hello World from processor c002, rank 7out of 12processors
Hello World from processor c002, rank 8out of 12processors
Hello World from processor c002, rank 11out of 12processors
Hello World from processor c002, rank 10out of 12processors
Hello World from processor c002, rank 9out of 12processors
Note: It is common to see the output printed in a non-deterministic manner - in the example above the process rank 7 and rank 8 overlap each other in the writing step.
8. COMPILING AND RUNNING A SAMPLE FORTRAN+MPI PROGRAM
A sample Fortran+MPI program is shown in Listing 13. This program will print “Hello world” to the output file as many times as there are MPI processes.
program helloinclude 'mpif.h'call MPI_INIT (ierr)print *, "Hello world"call MPI_FINALIZE (ierr)end program hello
Listing 13: Sample Fortran+MPI code (../documentation/mpi_jobs/mpif/hello_world.f90)
If you would like to compile the Fortran example using the Intel OneAPI and MVAPICH2 MPI library, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ ml mvapich2
[username@login001]$ mpiifort -o helloF90MPIOut \
hello_world.f90
The executable helloF90MPIOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 12 -N 2 -t 00:05:00 --pty bash
[username@c001]$ mpirun -np 12 ./ helloF90MPIOut
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
If you are currently on a compute node and would like to switch back to the login node then please enter theexitcommand as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloF90MPIOut corresponding to the serial program_name.f90 in batch mode is shown in Listing 14. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldF90MPI#SBATCH -o helloWorldF90MPI.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 2#SBATCH -n 12mpirun -np 12 ./helloF90MPIOut
Listing 14: Batch Job Script for Fortran+ MPI (../documentation/mpi_jobs/mpif/helloworldfmpi.slurm)
The job-script shown in Listing 14 can be submitted as follows:
[username@login001]$ sbatchhelloworldfmpi.slurm.slurm
The output from the Slurm batch-job shown in Listing 14 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldF90MPI.txt
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
9. COMPILING AND RUNNING A SAMPLE C+OPENMP PROGRAM
A sample C+OpenMP program is shown in Listing 15. All this code does is print“Hello World... from thread =”# where # is the thread number to standard output.
#include <omp.h>#include <stdio.h>#include <stdlib.h>int main(int argc, char* argv[]){// Beginning of parallel region#pragma omp parallel{printf("Hello World... from thread = %d\n",omp_get_thread_num());}// Ending of parallel regionreturn 0;}
Listing 15: Sample C+OpenMP program (../documentation/openmp_jobs/openmpc/GNU/hello_world.c)
If you would like to compile the C example using the GNU C compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ gcc -fopenmp -o helloCOpenMPOut\hello_world.c
If you would like to compile the C example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icc -qopenmp -o helloCOpenMPiOut\ hello_world.c
Note: Some compiler options are the same for both Intel and GNU (e.g. "-o"), while others are different (e.g. "-qopenmp" vs "-fopenmp")
The executablehelloCOpenMPOutor helloCOpenMPiOut can be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 4 -t 00:05:00 --pty bash
[username@c001]$ export OMP_NUM_THREADS=4
[username@c001]$ ./helloCOpenMPOut
Hello World... from thread = 2
Hello World... from thread = 0
Hello World... from thread = 1
Hello World... from thread = 3
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable in Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloCOpenMPOutcorresponding to the parallel program_name.c in batch mode is shown in Listing 16. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCOpenMP#SBATCH -o helloWorldCOpenMP.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 4export OMP_NUM_THREADS=4./helloCOpenMPOut
Listing 16: Batch Job Script for C+OpenMP (../documentation/openmp_jobs/openmpc/GNU/helloworldcopenmp.slurm)
The job-script shown in Listing 16 can be submitted as follows:
[username@login001]$ sbatchhelloworldcopenmp.slurm
The output from the Slurm batch-job shown in Listing 16 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldCOpenMP.txt
Hello World... from thread = 2
Hello World... from thread = 0
Hello World... from thread = 1
Hello World... from thread = 3
10. COMPILING AND RUNNING A SAMPLE OPENMP PROGRAM IN C++
A sample C++ program with OpenMP is shown in Listing 17. All this code does is print “Hello World... from thread =” # where # is the thread number to standard output.
Listing 17: Sample OpenMP program in C++ (../documentation/openmp_jobs/openmpcpp/GNU/hello_world.cpp)#include <omp.h>#include <iostream>#include <stdlib.h>using namespace std;int main(int argc, char* argv[]){// Beginning of parallel region#pragma omp parallel{cout<<"Hello World... from thread ="<< omp_get_thread_num();cout<<"\n";}// Ending of parallel regionreturn 0;}
If you would like to compile the C++ example using the GNU CPP compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ g++ -fopenmp -o helloOpenMPCPPOut\
hello_world.cpp
If you would like to compile the C++ example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icpc -qopenmp -o helloOpenMPCPPiOut\
hello_world.cpp
Note: Some compiler options are the same for both Intel and GNU (e.g. "-o"), while others are different (e.g. "-qopenmp" vs "-fopenmp")
The executablehelloOpenMPCPPOutand helloOpenMPCPPiOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 4 -t 00:05:00 --pty bash
[username@c001]$ export OMP_NUM_THREADS=4
[username@c001]$ ./helloOpenMPCPPiOut
Hello World... from thread =Hello World... from thread =Hello World... from thread =Hello World... from thread =103
2
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable in Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloOpenMPCPPiOutcorresponding to the parallel program_name.cpp in batch mode is shown in Listing 18. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloOpenMPCPPOut#SBATCH -o helloOpenMPCPPOut.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 4export OMP_NUM_THREADS=4./helloOpenMPCPPOut
Listing 18: Batch Job Script for OpenMP in C++ (job_script9.slurm)
The job-script shown in Listing 18 can be submitted as follows:
[username@login001]$ sbatchhelloworldopenmpcpp.slurm
The output from the Slurm batch-job shown in Listing 18 can be checked by opening the output file as follows:
[username@login001]$ cat helloOpenMPCPPOut.txt
Hello World... from thread =Hello World... from thread =Hello World... from thread =Hello World... from thread =103
2
11. COMPILING AND RUNNING A SAMPLE FORTRAN + OPENMP PROGRAM
A sample Fortran program is shown in Listing 19. All this code does is print to standard output “Hello from process: #”, where # represents different thread numbers.
PROGRAM Parallel_Hello_WorldUSE OMP_LIB!$OMP PARALLELprint *,"Hello from process: ", OMP_GET_THREAD_NUM()!$OMP END PARALLELEND
Listing 19: Sample Fortran+OpenMP program (../ documentation/openmp_jobs/openmpf/GNU/hello_world.f90)
If you would like to compile the Fortran example using the GNU Fortran compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ gfortran -fopenmp -o helloF90OpenMPOut hellow_world.f90
If you would like to compile the Fortran example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ ifort -qopenmp -o helloF90OpenMPiOut \hello_world.f90
The executablehelloF90OpenMPiOut can be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 4 -t 00:05:00 --pty bash
[username@c001]$ export OMP_NUM_THREADS=4
[username@c001]$ ./helloF90OpenMPiOut
Hello from process: 2
Hello from process: 0
Hello from process: 1
Hello from process: 3
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable in Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloF90OpenMPOutcorresponding to the parallel program_name.f90 in batch mode is shown in Listing 20. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloworldfopenmp#SBATCH -o helloworldfopenmp.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 4export OMP_NUM_THREADS=4./helloF90OpenMPOut
Listing 20: Batch Job Script for Fortran+OpenMP(../ documentation/openmp_jobs/openmpf/GNU/helloworldfopenmp.slurm)
The job-script shown in Listing 20 can be submitted as follows:
[username@login001]$ sbatchhelloworldfopenmp.slurm
The output from the Slurm batch-job shown in Listing 20 can be checked by opening the output file as follows:
[username@login001]$ cat helloworldfopenmp.txt
Hello from process: 2
Hello from process: 0
Hello from process: 1
Hello from process: 3
12. COMPILING AND RUNNING A SAMPLE C+CUDA PROGRAM
A sample C program is shown in Listing 21. All this code does is print“device count = #” to standard output, where # is replaced by the actual number of GPUs on the node.
#include <stdio.h>#include <cuda.h>#include <stdlib.h>#include <unistd.h>int main(){int count, err;cudaGetDeviceCount(&count);printf("device count = %d\n", count);if(err = cudaSetDevice(count-1)){printf("cudaSetDevice error, %d\n", err);}return 0;}
Listing 21: Sample C+Cuda program (../documentation/cuda_jobs/cudac/hello_world.cu)
If you would like to compile the C example using the Nvidia CUDA compiler, you can run the following commands:
[username@login001]$ ml cuda/toolkit/11.3
[username@login001]$ nvcc -o helloCcudaOut\hello_world.cu
The executablehelloCcudaOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a GPU node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a GPU node using the following set of commands:
Single GPU
[username@login001]$ srun -p gpu1v100 -n 1 -t 00:05:00 --pty bash
[username@gpu001]$ ./helloCcudaOut
device count = 1
Multiple GPU
[username@login001]$ srun -p gpu2v100 -n 1 -t 00:05:00 --pty bash
[username@gpu001]$ ./helloCcudaOut
device count = 2
If you are currently on a GPU node and would like to switch back to the login node then please enter the exit command as follows:
[username@gpu001]$ exit
Running the Executable in Batch-Mode: Sample Slurm batch job-scripts to run the executable namedhelloCcudaOutcorresponding to the C+CUDA program_name.cu in batch mode are shown in Listing 22 (single GPU) and Listing 23 (multiple GPU). These scripts should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCcuda#SBATCH -o helloWorldCcuda.txt#SBATCH -p gpu1v100#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1./helloCcudaOut
Listing 22: Batch Job Script for C+CUDA (../documentation/cuda_jobs/cudac/helloworldccuda.slurm)
#!/bin/bash#SBATCH -J helloWorldCcuda#SBATCH -o helloWorldCcuda.txt#SBATCH -p gpu2v100#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1#SBATCH --gres=gpu:v100:0./helloWorldCcuda
Listing 23: Batch Job Script for C+CUDA - This file is not included in the documentation download. Copy the contents of this example script to a file called helloWorldCcuda2.slurm in your directory.
The job-script shown in Listing 22 and 23 can be submitted as follows:
The job-script shown in Listing 22 and 23 can be submitted as follows:
[username@login001]$ sbatchhelloWorldCcuda.slurm
[username@login001]$ sbatchhelloWorldCcuda2.slurm
The output from the Slurm batch-job shown in Listing 22 and 23 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldCcuda.txt
Single GPU
device count = 1
Multiple GPU
device count = 2
13. COMPILING AND RUNNING A SAMPLE CUDA PROGRAM IN C++
A sample C++ program is shown in Listing 24. All this code does is print“device count = #” to standard output, where # is replaced by the actual number of GPUs on the node.
#include <iostream>#include <cuda.h>#include <stdlib.h>#include <unistd.h>using namespace std;int main(){int count, err;cudaGetDeviceCount(&count);cout<<"device count = "<<count<<"\n";if(err = cudaSetDevice(count-1)){cout<<"cudaSetDevice error, "<<err<<"\n";}return 0;}
Listing 24: Sample CUDA program in C++(../documentation/cuda_jobs/cudacpp/hello_world.cu)
If you would like to compile the C++ example using the Nvidia CUDA compiler, you can run the following commands:
[username@login001]$ ml cuda/toolkit/11.3
[username@login001]$ nvcc -o helloCPPcudaOut\hello_world.cu
The executablehelloCPPcudaOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a GPU node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a GPU node using the set of commands below.
Single GPU
[username@login001]$ srun -p gpu1v100 -n 1 -t 00:05:00 --pty bash
[username@gpu001]$ ./helloCPPcudaOut
device count = 1
Multiple GPU
[username@login001]$ srun -p gpu2v100 -n 1 -t 00:05:00 --pty bash
[username@gpu001]$ ./helloCPPcudaOut
device count = 2
If you are currently on a GPU node and would like to switch back to the login node then please enter theexit command as follows:
[username@gpu001]$ exit
Running the Executable in Batch-Mode: Sample Slurm batch job-scripts to run the executable named helloCPPcudaOutcorresponding to the CPP+CUDA program_name.cu in batch mode are shown in Listing 25 (single GPU) and Listing 26 (multiple GPU). These scripts should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCPPcuda#SBATCH -o helloWorldCPPcuda.txt#SBATCH -p gpu1v100#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1./helloWorldCPPcuda
Listing 25: Batch Job Script for CUDA in C++ (../documentation/cuda_jobs/cudacpp/helloworldcppcuda.slurm)
#!/bin/bash#SBATCH -J helloWorldCPPcuda#SBATCH -o helloWorldCPPcuda.txt#SBATCH -p gpu2v100#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1#SBATCH --gres=gpu:v100:0./helloWorldCPPcuda
Listing 26: Batch Job Script for CUDA in C++ - This file is not included in the documentation download. Copy the contents of this example script to a file called helloworldcppcuda2.slurm in your directory.
The job-script shown in Listing 25 and 26 can be submitted as follows:
[username@login001]$ sbatchhelloWorldCPPcuda.slurm
[username@login001]$ sbatchhelloWorldCPPcuda2.slurm
The output from the Slurm batch-job shown in Listing 25 and 26 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldCPPcuda.txt
Single GPU
device count = 1
Multiple GPU
device count = 2
14. COMPILING AND RUNNING A SAMPLE FORTRAN+CUDA PROGRAM
A sample Fortran code is shown in Listing 27. All this code does is check whether the data transfer is successful or not between Host CPU and device GPU.
program cpydatause cudaforimplicit noneinteger, parameter :: n = 256real :: a(n), b(n)real, device::a_d(n), b_d(n)a = 1.0a_d = ab_d = a_db = b_dif (all(a==b)) thenwrite(,) 'Test Passed'elsewrite(,) 'Test Failed'end ifend program cpydata
Listing 27: Sample Fortran+CUDA code (../documentation/cuda_jobs/cudaf/hello_world.cuf)
For compiling the Fortran example using the CUDA Fortran compiler, you can run the following commands:
[username@login001]$ ml cuda/toolkit/11.3
[username@login001]$ ml cuda/sdk/nvhpc/21.5
[username@login001]$ nvfortran -o helloWorldFcuda\
hello_world.cuf
The executable helloWorldFcudacan be run either in a batch mode using a Slurm batch job-script or interactively on a GPU node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a GPU node using the following set of commands :
[username@login001]$ srun -p gpu1v100 -n 1 -t 00:05:00 --pty bash
[username@gpu001]$ ./helloWorldFcuda
Test Passed
If you are currently on the GPU node, switch back to login node before running the job-script in batch mode using the following command:
[username@gpu001]$ exit
Running the Executable in Batch-Mode: A sample Slurm batch job-script to run the executable named helloWorldFcudacorresponding to the Fortran+CUDAprogram_name.cuf in batch mode is shown in Listing 28. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldFcuda#SBATCH -o helloWorldFcuda.txt#SBATCH -p gpu1v100#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1./helloWorldFcuda
Listing 28: Batch Job Script for fortran+cuda (../documentation/cuda_jobs/cudaf/helloworldfcuda.slurm)
The job-script shown in Listing 28 can be submitted as follows:
[username@login001]$ sbatchhelloworldfcuda.slurm
The output from the Slurm batch-job shown in Listing 28 can be checked by opening the output file as follows:
[username@login001]$ cat helloworldfcuda.txt
Test Passed
15. RUNNING A SAMPLE R SCRIPT
A sample R script is shown below and all this code does is print “Hello World!!” to standard output.
print("Hello World!!")
Add execute permissions on the R script file as follows:
[username@login001]$ chmod +x hello_world.r
The R programcan be run either in batch mode using a Slurm batch job-script or interactively on a compute node.
Running the script in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --pty bash
[username@c001]$ ml gnu8/8.3.0
[username@c001]$ ml R/3.6.1
[username@c001]$ Rscripthello_world.r
"Hello World!!"
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the script in Batch-Mode: A sample Slurm batch job-script to run the serial program_name.r in batch mode is shown in Listing 29. This script should be run from a login node.
#!/bin/bash#SBATCH -J program_name#SBATCH -o program_name.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 1ml gnu8/8.3.0ml R/3.6.1Rscripthello_world.r
Listing 29: Batch Job Script for R code (job_script16.slurm)
The job-script shown in Listing 29 can be submitted as follows:
[username@login001]$ sbatchhelloworldr.slurm
The output from the Slurm batch-job shown in Listing 29 can be checked by opening the output file as follows:
[username@login001]$ cat program_name.txt
"Hello World!!"
16. RUNNING MULTIPLE COPIES OF ONE OR MORE EXECUTABLES FROM THE SAME SLURM JOB
a. Bundling Multiple Executables of Serial Programs in One Job-Script: Let us assume that there are two program files named as program_name1.* and program_name2.* (e.g., program_name1.c and program_name2.cpp), two executable files named as exefilename1 and exefilename2 corresponding to the aforementioned to program files, and Slurm batch job as job_script*.slurm (e.g., job_script1.slurm and job_script2.slurm) throughout this section.
A sample C code is shown in Listing 30. All this code does is print“Hello World!!: #!” to standard output where # is replaced by the iteration number, and the iteration number depends upon the argument passed to the executable at run-time.
# include <stdio.h># include<stdlib.h>int main(int argc, char *argv[]){int n = atoi( argv[1]);for(int i=1;i<=n;i++){printf("Hello World!!: %d \n",i);}return 0;}
Listing 30: Sample C program (../documentation/multiExes/serial/hello_world.c)
If you would like to compile the C example using the GNU C compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ gcc -o helloCOuthello_world.c
If you would like to compile the C example using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icc -o helloCOutihello_world.c
The executableshelloCOutandhelloCOutibe run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 5 -t 00:05:00 --pty bash
[username@c001]$ ./helloCOut1& ./helloCOut2& \./helloCOut3& ./helloCOut4& ./helloCOut5
Hello World!!: 1
Hello World!!: 2
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 1
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 5
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable in Batch-Mode: A sample Slurm batch job-script to run the executable namedhelloCOut corresponding to the serial program_name.c in batch mode is shown in Listing 32. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCm#SBATCH -o helloWorldCm.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH -n 5./helloCOut1&./helloCOut2&./helloCOut3&./helloCOut4&./helloCOut5
Listing 32: Batch Job Script for C codes (../documentation/multiExes/serial/helloworldcm.slurm)
The job-script shown in Listing 32 can be submitted as follows:
[username@login001]$ sbatchhelloworldcm.slurm
The output from the Slurm batch-job shown in Listing 32 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldCm.txt
Hello World!!: 1
Hello World!!: 2
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 1
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 5
b. Bundling Multiple Executables of Parallel Programs in One Job-Script: A sample C code is shown in Listing 33. This code prints “Hello world from processor #, rank # out of # processors” , where # varies depending upon the number of MPI processes engaged in running the executable.
#include <mpi.h>#include <stdio.h>int main(int argc, char** argv) {// Initialize the MPI environmentMPI_Init(NULL, NULL);// Get the number of processesint world_size;MPI_Comm_size(MPI_COMM_WORLD, &world_size);// Get the rank of the processint world_rank;MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);// Get the name of the processorchar processor_name[MPI_MAX_PROCESSOR_NAME];int name_len;MPI_Get_processor_name(processor_name, &name_len);// Print off a hello world messageprintf("Hello world from processor %s, rank %d out of %d processors\n",processor_name, world_rank, world_size);// Finalize the MPI environment.MPI_Finalize();return 0;}
Listing 33: Sample C+MPI code (../documentation/multiExes/mpi_multiexes/hello_world.c)
Another sample C code is shown in Listing 34. This code prints “Bye world from processor #, rank # out of # processors” , where # varies depending upon the number of MPI processes engaged in running the executable.
#include <mpi.h>#include <stdio.h>int main(int argc, char** argv) {// Initialize the MPI environmentMPI_Init(NULL, NULL);// Get the number of processesint world_size;MPI_Comm_size(MPI_COMM_WORLD, &world_size);// Get the rank of the processint world_rank;MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);// Get the name of the processorchar processor_name[MPI_MAX_PROCESSOR_NAME];int name_len;MPI_Get_processor_name(processor_name, &name_len);// Print off a hello world messageprintf("Bye world from processor %s, rank %d out of %d processors\n",processor_name, world_rank, world_size);// Finalize the MPI environment.MPI_Finalize();return 0;}
Listing 34: Sample “Bye World” C+MPI code(../documentation/multiExes/mpi_multiexes/bye_world.c)
For compiling the C examples shown in Listings 33 and 34 using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ mpicc -o helloCMPIOut\hello_world.c
[username@login001]$ mpicc -o byeCMPIOut\bye_world.c
The executableshelloCMPIOutandbyeCMPIOutcan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executables in Interactive-Mode: The executables can be run simultaneously in the same interactive session using the set of commands below and the output will be displayed on the terminal.
[username@login001]$ srun -p compute1 -n 24 -N 2 -t 00:05:00 --pty bash
[username@c001]$ mpirun -np 12 ./helloCMPIOut& \mpirun -np 12 ./byeCMPIOut
Hello world from processor c002, rank 4 out of 12 processors
Hello world from processor c002, rank 7 out of 12 processors
Hello world from processor c002, rank 6 out of 12 processors
Hello world from processor c002, rank 8 out of 12 processors
Bye world from processor c002, rank 7 out of 12 processors
Hello world from processor c002, rank 3 out of 12 processors
Bye world from processor c002, rank 9 out of 12 processors
Bye world from processor c002, rank 11 out of 12 processors
Hello world from processor c002, rank 1 out of 12 processors
Bye world from processor c002, rank 4 out of 12 processors
Hello world from processor c002, rank 2 out of 12 processors
Hello world from processor c002, rank 11 out of 12 processors
Hello world from processor c002, rank 0 out of 12 processors
Bye world from processor c002, rank 5 out of 12 processors
Hello world from processor c002, rank 5 out of 12 processors
Bye world from processor c002, rank 8 out of 12 processors
Bye world from processor c002, rank 2 out of 12 processors
Bye world from processor c002, rank 10 out of 12 processors
Bye world from processor c002, rank 1 out of 12 processors
Hello world from processor c002, rank 10 out of 12 processors
Bye world from processor c002, rank 6 out of 12 processors
Hello world from processor c002, rank 9 out of 12 processors
Bye world from processor c002, rank 0 out of 12 processors
Bye world from processor c002, rank 3 out of 12 processors
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executables in Batch-Mode: A sample Slurm batch job-script to run the executables named helloCMPIOutandbyeCMPIOutcorresponding to the parallel program_name1.c and program_name2.c shown in Listings 33 and 34 respectively in batch mode is shown in Listing 35. This script should be run from a login node.
#!/bin/bash#SBATCH -J helloWorldCMPIm#SBATCH -o helloWorldCMPIm.txt#SBATCH -p compute1#SBATCH -t 00:10:00#SBATCH -N 2#SBATCH -n 24mpirun -np 12 ./helloCMPIOut&mpirun -np 12 ./byeCMPIOut
Listing 35: Batch Job Script for C+MPI codes (../ documentation/multiExes/mpi_multiexes/helloworldcmpim.slurm)
The job-script shown in Listing 35 can be submitted as follows:
[username@login001]$ sbatchhelloworldcmpim.slurm
The output from the Slurm batch-job shown in Listing 35 can be checked by opening the output file as follows:
[username@login001]$ cat helloWorldCMPIm.txt
Hello world from processor c002, rank 4 out of 12 processors
Hello world from processor c002, rank 7 out of 12 processors
Hello world from processor c002, rank 6 out of 12 processors
Hello world from processor c002, rank 8 out of 12 processors
Bye world from processor c002, rank 7 out of 12 processors
Hello world from processor c002, rank 3 out of 12 processors
Bye world from processor c002, rank 9 out of 12 processors
Bye world from processor c002, rank 11 out of 12 processors
Hello world from processor c002, rank 1 out of 12 processors
Bye world from processor c002, rank 4 out of 12 processors
Hello world from processor c002, rank 2 out of 12 processors
Hello world from processor c002, rank 11 out of 12 processors
Hello world from processor c002, rank 0 out of 12 processors
Bye world from processor c002, rank 5 out of 12 processors
Hello world from processor c002, rank 5 out of 12 processors
Bye world from processor c002, rank 8 out of 12 processors
Bye world from processor c002, rank 2 out of 12 processors
Bye world from processor c002, rank 10 out of 12 processors
Bye world from processor c002, rank 1 out of 12 processors
Hello world from processor c002, rank 10 out of 12 processors
Bye world from processor c002, rank 6 out of 12 processors
Hello world from processor c002, rank 9 out of 12 processors
Bye world from processor c002, rank 0 out of 12 processors
Bye world from processor c002, rank 3 out of 12 processors
17. RUNNING AN EXECUTABLE WITH MULTIPLE DIFFERENT PARAMETERS SIMULTANEOUSLY USING GPU PARALLEL
GNU Parallel is a tool that is used for running parameter-sweep applications. It helps in running multiple copies of an executable simultaneously with different parameters read from an input file and as a part of the same Slurm job. GNU Parallel is installed on Arc and can be used after loading theparallel/20210722module. For detailed information on this tool, please refer to the GNU Parallel documentation page.
The general syntax of the command for using the GNU Parallel tool is as follows:
parallel --joblog logfilename.txt ./exefilename {1} :::: paramfile.txt
In the command above, the option “--joblog” helps in creating a log file that can be used to resume the execution of the job in case it is interrupted. For resuming a job after interruption, the “--resume-failed” option should be used along with the previously saved log file as shown in the following command:
parallel --resume-failed --joblog logfilename.txt ./exefilename {1} :::: paramfile.txt
A sample C code that will serve as our parameter-sweep application is shown in Listing 36. This code prints “Hello World!!: #” multiple times to the standard output such that # is replaced by the iteration number, and the total number of iterations is determined by the argument passed at run-time.
# include <stdio.h># include<stdlib.h>int main(int argc, char *argv[]){int n = atoi( argv[1]);for(int i=1;i<=n;i++){printf("Hello World!!: %d \n",i);}return 0;}
Listing 36: Sample C code (../documentation/GNUparallel/hello_world.c)
If you would like to compile the example in listing 36 using the GNU C compiler, you can run the following commands:
[username@login001]$ ml gcc/11.2.0
[username@login001]$ gcc -o helloCOuthello_world.c
If you would like to compile the example in listing 36 using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icc -o helloCOuti\-std=c99 hello_world.c
The contents of the sample input file to provide arguments to the executable in the Slurm batch job script is shown in Listing 37.
[username@login001]$ cat myout.txt12345
Listing 37: Text file containing arguments for the executables
The executablehelloCOutandhelloCOutican be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --cpus-per-task=1 --pty bash
[username@c001]$ ml gcc/11.2.0
[username@c001]$ ml parallel/20210622
[username@c001]$ parallel --joblog logfilename.txt \ ./helloCOut{1} :::: myout.txt
Hello World!!: 1
Hello World!!: 1
Hello World!!: 2
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 5
If you would like to put the output in separate files for each instance, use the following Parallel option:
[username@c001]$ parallel --results outdir ./helloCOut{1} :::: myout.txt
The output files can be found in outdir/1/ directory:
[username@c001]$ cd outdir/1
[username@c001]$ ls
1 2 3 4 5 seq stderr stdout
[username@c001]$ cd 2
[username@c001]$ ls
seq stderr stdout
[username@c001]$ cat stdout
Hello World!!: 1
Hello World!!: 2
Logs from the parallel command above can be checked by opening the log file as follows:
[username@c001]$ cat logfilename.txt
SeqHostStarttimeJobRuntimeSendReceiveExitvalSignalCommand
1:1626380539.453 0.00301800./exefilename 1
2:1626380539.457 0.00903600./exefilename 2
3:1626380539.462 0.00905400./exefilename 3
4:1626380539.466 0.00807200./exefilename 4
5:1626380539.471 0.00609000./exefilename 5
6:1626380539.475 0.0060000./exefilename ''
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable Batch-Mode: A sample Slurm batch job-script to run the executable namedexefilenamecorresponding to the serial program_name.c in batch mode is shown in Listing 38. This script should be run from a login node.
#!/bin/sh#SBATCH -J helloWorldCgnup#SBATCH -o helloWorldCgnup.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH --cpus-per-task=1#SBATCH -n 1ml parallel/20210622parallel --joblog logfilename.txt ./helloCOut{1} :::: myout.txt
Listing 38: Batch Job Script for C code (../documentation/GNUparallel/helloworldcgnup.slurm)
The parallelcommand shown in Listing 38 will run thehelloCOutwith different parameters read from an input file named asmyfile.txt. This command will also create a log file named logfilename.txtthat can be used to resume the execution in case it is interrupted. This program will print “Hello World!!: #”, where # varies depending upon input from theparamfile.txt.
The job-script shown in Listing 38 can be submitted as follows :
[username@login001]$ sbatchhelloworldcgnup.slurm
The output from the Slurm batch-job shown in Listing 38 can be checked by displaying the contents of the output file as follows:
[username@login001]$ cat helloWorldCgnup.txt
Hello World!!: 1
Hello World!!: 1
Hello World!!: 2
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 1
Hello World!!: 2
Hello World!!: 3
Hello World!!: 4
Hello World!!: 5
Logs from the Slurm batch-job shown in Listing 38 can be checked by displaying the contents of the log file as follows:
[username@login001]$ cat logfilename.txt
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
1 :1626294133.948 0.00301800./exefilename 1
2 :1626294133.952 0.00903600./exefilename 2
3 :1626294133.957 0.00705400./exefilename 3
4 :1626294133.960 0.00707200./exefilename 4
5 :1626294133.964 0.00709000./exefilename 5
6 :1626294133.968 0.0060000./exefilename ''
18. COMPILING AND RUNNING AN EXAMPLE CODE TO FIND THE PRIME FACTORS OF GIVEN NUMBERS AT THE SAME TIME USING GNU PARALLEL
A sample C code that will serve as another example of parameter-sweep application is shown in Listing 39. This code prints “The prime factors of # are:*” multiple times to the standard output such that # is replaced by the input numbers, and * is replaced by the prime factors of that number.
#include <stdio.h>#include <math.h>#include <stdlib.h>int main(int argc, char** argv){int n,i;sscanf(argv[1],"%d",&n);printf(" The prime factors of %d are:", n);while (n%2 == 0){n = n/2;}for(i = 3; i <= (int)sqrt(n); i = i+2){while (n%i == 0){printf("%d ", i);n = n/i;}}if (n > 2)printf ("%d", n);printf("\n");return 0;}
Listing 39: Sample C code (program_name.c)
Note that this program is not provided in the documentation download. Copy the contents of the program to a file in your directory.
If you would like to compile the example in listing 39 using the GNU C compiler, you can run the following commands:
[username@login001]$ ml gnu8/8.3.0
[username@login001]$ gcc -o exefilenameprogram_name.c -lm
Note: The math library must be linked in when building the executable. The math library is named libm.so, and the -l command option assumes a lib prefix and .a or .so suffix.
If you would like to compile the example in listing 39 using the Intel OneAPI, you can run the following commands:
[username@login001]$ ml intel/oneapi/2021.2.0
[username@login001]$ icc -o exefilenameprogram_name.c
The contents of the sample input file to provide arguments to the executable in the Slurm batch job script is shown in Listing 40. This file is not included in the documentation download. Copy the contents of this file to a file in your directory using the name paramfile.txt.
[username@login001]$ cat paramfile.txt123456789612345678911234567899123456789512345678471234567941123456892712345789571234689630123457896012346789751235678933124567899813456789201234567892123456784412345679701234568951123457898112346789691235678949123457899312345678971234567898123456789112345679661234568918123457898312346789561235678921124567895513456789371234567895123456789512345678951234567988123456893612345789321234678946123567891312456789241234567896123456789212345678111234567990123456897912345789861678977111123456789612345678911234567899123456789512345678471234567941123456892712345789571234689630123457896012346789751235678933124567899813456789201234567892123456784412345679701234568951123457898112346789691235678949123457899312345678971234567898123456789112345679661234568918123457898312346789561235678921124567895513456789371234567895123456789512345678951234567988123456893612345789321234678946123567891312456789241456789841
Listing 40 : Text file containing arguments for the executables
The executableexefilenamecan be run either in a batch mode using a Slurm batch job-script or interactively on a compute node.
Running the Executable in Interactive-Mode: The executable can be run interactively on a compute node using the following set of commands and the output will be displayed on the terminal:
[username@login001]$ srun -p compute1 -n 1 -t 00:05:00 --cpus-per-task=1 --pty bash
[username@c001]$ ml gcc/11.2.0
[username@c001]$ ml parallel/20210622
[username@c001]$ parallel --joblog logfilename.txt ./exefilename {1} :::: paramfile.txt
The prime factors of 1234567896 are:3 83 619763
The prime factors of 1234567891 are:1234567891
The prime factors of 1234567899 are:3 3 3 3 109 139831
The prime factors of 1234567895 are:5 11 23 975943
The prime factors of 1234567847 are:47 251 104651
The prime factors of 1234567941 are:3 23 47 199 1913
The prime factors of 1234568927 are:491 1123 2239
The prime factors of 1234578957 are:3 139 2960621
The prime factors of 1234689630 are:3 5 37 1112333
The prime factors of 1234578960 are:3 3 5 19 90247
The prime factors of 1234678975 are:5 5 17 409 7103
The prime factors of 1235678933 are:23 23 2335877
The prime factors of 1245678998 are:622839499
If the above execution is interrupted due to any failures or the user exits or interrupts the execution, the option “--joblog” helps in creating a log file that can be used to resume the execution of the job.
Logs from the parallel command can be checked by opening the log file as follows:
[username@c001]$ cat logfilename.txt
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
1 : 1626460968.422 0.000 0 49 0 0 ./exefilename 1234567896
2 : 1626460968.426 0.003 0 48 0 0 ./exefilename 1234567891
3 : 1626460968.430 0.003 0 56 0 0 ./exefilename 1234567899
4 : 1626460968.433 0.006 0 52 0 0 ./exefilename 1234567895
5 : 1626460968.437 0.006 0 51 0 0 ./exefilename 1234567847
6 : 1626460968.439 0.009 0 54 0 0 ./exefilename 1234567941
7 : 1626460968.443 0.008 0 51 0 0 ./exefilename 1234568927
8 : 1626460968.448 0.003 0 51 0 0 ./exefilename 1234578957
9 : 1626460968.451 0.006 0 52 0 0 ./exefilename 1234689630
10 : 1626460968.455 0.009 0 52 0 0 ./exefilename 1234578960
11 : 1626460968.458 0.006 0 53 0 0 ./exefilename 1234678975
12 : 1626460968.461 0.006 0 51 0 0 ./exefilename 1235678933
13 : 1626460968.464 0.006 0 47 0 0 ./exefilename 1245678998
For resuming a job after interruption, the “--resume-failed” option should be used along with the previously saved log file as shown in the following command:
[username@c001]$ parallel --resume-failed --joblog logfilename.txt ./exefilename {1} :::: paramfile.txt
A snippet from the output on the command prompt is shown here:
The prime factors of 1345678920 are:3 3 3 3 5 67 6199
The prime factors of 1234567892 are:41 7527853
The prime factors of 1235678913 are:3 3 7 7 1009 2777
The prime factors of 1456789841 are:13 907 123551
The prime factors of 1245678924 are:3 7 181 81931
Logs from the parallel --resume-failed command can be checked by opening the log file as follows:
[username@c001]$ cat logfilename.txt
A snippet from the log file on the command prompt is shown here:
Seq Host StarttimeJobRuntime Send Receive Exitval Signal Command
1 : 1626460968.422 0.000 0 49 0 0 ./exefilename 1234567896
2 : 1626460968.426 0.003 0 48 0 0 ./exefilename 1234567891
3 : 1626460968.430 0.003 0 56 0 0 ./exefilename 1234567899
4 : 1626460968.433 0.006 0 52 0 0 ./exefilename 1234567895
5 : 1626460968.437 0.006 0 51 0 0 ./exefilename 1234567847
If you are currently on a compute node and would like to switch back to the login node then please enter theexit command as follows:
[username@c001]$ exit
Running the Executable in Batch-Mode: A sample Slurm batch job-script to run the executable namedexefilenamecorresponding to the serial program_name.c in batch mode is shown in Listing 41. This script should be run from a login node. This file is not included in the documentation download. Copy the contents of the batch script below to a file in your directory.
#!/bin/sh#SBATCH -J program_name#SBATCH -o program_name.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH --cpus-per-task=1#SBATCH -n 1ml gcc/11.2.0ml parallel/20210722parallel --joblog logfilename.txt ./exefilename {1} :::: paramfile.txt
Listing 41: Batch Job Script for C code (job_script20.slurm)
The job-script shown in Listing 41 can be submitted as follows:
[username@login001]$ sbatch job_script20.slurm
Assuming that the job-id corresponding to the submission of the job-script shown in Listing 41 is8088547, and this job is to be interrupted usingscancel, then the following command can be used:
[username@login001]$ scancel 8088547
The output from the Slurm batch-job shown in Listing 41 can be checked by displaying the contents of the output file as follows:
[username@login001]$ cat program_name.txt
The prime factors of 1234567896 are:3 83 619763
The prime factors of 1234567891 are:1234567891
The prime factors of 1234567899 are:3 3 3 3 109 139831
The prime factors of 1234567895 are:5 11 23 975943
The prime factors of 1234567847 are:47 251 104651
The prime factors of 1234567941 are:3 23 47 199 1913
The prime factors of 1234568927 are:491 1123 2239
The prime factors of 1234578957 are:3 139 2960621
The prime factors of 1234689630 are:3 5 37 1112333
The prime factors of 1234578960 are:3 3 5 19 90247
The prime factors of 1234678975 are:5 5 17 409 7103
The prime factors of 1235678933 are:23 23 2335877
The prime factors of 1245678998 are:622839499
slurmstepd: error: * JOB $SLURM_JOBID ON c001 CANCELLED AT 2021-07-16T13:10:17 ***
If the above execution is interrupted due to any failure or the user exits or interrupts the execution, the option “--joblog” helps in creating a log file that can be used to resume the execution of the job.
Logs from the parallel command can be checked by opening the log file as follows:
[username@login001]$ cat logfilename.txt
Seq Host Starttime JobRuntime Send Receive Exitval Signal Command
1 : 1626460968.422 0.000 0 49 0 0 ./exefilename 1234567896
2 : 1626460968.426 0.003 0 48 0 0 ./exefilename 1234567891
3 : 1626460968.430 0.003 0 56 0 0 ./exefilename 1234567899
4 : 1626460968.433 0.006 0 52 0 0 ./exefilename 1234567895
5 : 1626460968.437 0.006 0 51 0 0 ./exefilename 1234567847
6 : 1626460968.439 0.009 0 54 0 0 ./exefilename 1234567941
7 : 1626460968.443 0.008 0 51 0 0 ./exefilename 1234568927
8 : 1626460968.448 0.003 0 51 0 0 ./exefilename 1234578957
9 : 1626460968.451 0.006 0 52 0 0 ./exefilename 1234689630
10 : 1626460968.455 0.009 0 52 0 0 ./exefilename 1234578960
11 : 1626460968.458 0.006 0 53 0 0 ./exefilename 1234678975
12 : 1626460968.461 0.006 0 51 0 0 ./exefilename 1235678933
13 : 1626460968.464 0.006 0 47 0 0 ./exefilename 1245678998
For resuming a job after interruption, the “--resume-failed” option should be used along with the previously saved log file.
A sample Slurm batch job-script to run the executable namedexefilenamecorresponding to the serialprogram_name.cin batch mode after restarting from interruption is shown in Listing 42. This script should be run from a login node.
#!/bin/sh#SBATCH -J program_name#SBATCH -o program_name.txt#SBATCH -p compute1#SBATCH -t 00:05:00#SBATCH -N 1#SBATCH --cpus-per-task=1#SBATCH -n 1ml gnu8/8.3.0ml parallel/20210622parallel --resume-failed --joblog logfilename.txt ./exefilename {1} :::: paramfile.txt
Listing 42: Batch Job Script for C code (job_script21.slurm)
The job-script shown in Listing 42 can be submitted as follows:
[username@login001]$ sbatch job_script21.slurm
The output from the Slurm batch-job shown in Listing 42 can be checked by displaying the contents of the output file as follows:
[username@login001]$ cat program_name.txt
A snippet from the output file is shown here:
The prime factors of 1345678920 are:3 3 3 3 5 67 6199
The prime factors of 1234567892 are:41 7527853
The prime factors of 1234567844 are:101 1277 2393
The prime factors of 1234567970 are:5 73 1691189
The prime factors of 1234568951 are:7 11 17 943139
Logs from the parallel --resume-failed command can be checked by opening the log file as follows:
[username@login001]$ cat logfilename.txt
A snippet from the log file on the command prompt is shown here:
Seq Host StarttimeJobRuntime Send Receive Exitval Signal Command
1 : 1626460968.422 0.000 0 49 0 0 ./exefilename 1234567896
2 : 1626460968.426 0.003 0 48 0 0 ./exefilename 1234567891
3 : 1626460968.430 0.003 0 56 0 0 ./exefilename 1234567899
4 : 1626460968.433 0.006 0 52 0 0 ./exefilename 1234567895
5 : 1626460968.437 0.006 0 51 0 0 ./exefilename 1234567847
-- AdminUser - 17 Aug 2021
SCTASK0015757
Edit | Attach | Print version | History: r2 < r1 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r2 - 25 Jul 2022, AdminUser
ARC
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding Foswiki? Send feedback