Once resources have been allocated through PBS, users have the option of serially running commands on the allocated resources’ head node or across all the resources in the allocated resource pool.

Serial

Batch Script

The executable portion of batch scripts is interpreted by the shell specified on the first line of the script. If a shell is not specified, the submitting user’s default shell will be used. This portion of the script may contain comments, shell commands, executable scripts, and compiled executables. These can be used in combination to, for example, navigate file systems, set up job execution, run executables, and even submit other batch jobs.

Batch Interactive

While running in interactive mode, the submitting user’s default shell will be used.

Parallel

By default, commands will executed on the head node. The mpirun command is used to execute a job on one or more compute nodes. Smoky’s layout should be kept in mind when running a job using mpirun. Smoky’s current layout consists of four quad-core sockets per node. The PBS node option requests compute nodes. The PBS ppn option requests cores.

mpirun accepts the following common options:

-display-map	Can be used to view layout
-npernode	Number of cores per node
-n	Number of total cores

Note: If you do not specify the number of tasks to mpirun, the system will default to 1.

OpenMP

Memory is shared within a node. Threads cannot span across nodes. To run a code with 8 MPI tasks and 16 threads per task, you would use the following:

export OMP_NUM_THREADS=16
mpirun -n 8 -npernode1 ./a.out

NOTE: csh/tcsh users should replace the first line with

setenv OMP_NUM_THREADS 16

The mpirun command specifies 8 tasks (-n 8) and asks for one task per socket (-npernode1). The OMP_NUM_THREADS environment variable tells the system to spawn 16 threads per MPI task.

Task Layout

The default MPI task layout on the system is round-robin.

For example,

mpirun -n 4 a.out

will run the MPI executable a.out on a total of four cores, two cores on two compute nodes. The MPI tasks will be allocated in the following sequential fashion:

Compute Node 0			Compute Node 1
core 0	core 1		core 0	core 1
0	1		2	3

The layout order can be changed using the environment variable . See man intro_mpi for more information. Task layout can be seen by setting the environment PMI_DEBUG variable to 1.