Job Scheduling & Resource Allocation

Job Scheduling and Resource Allocation

The Hopper Cluster is a shared resource among the many Auburn University principal investors (PIs). The goal is to maximize utilization of the computing resources while guaranteeing each PI access to the resources that they have purchased. This requires special software to schedule jobs and manage the resources.

Hopper uses the Torque Resource Manager with the Moab scheduler (aka ‘Torque/Moab’).

Torque is an open source fork of OpenPBS, which is maintained by Adaptive Computing. Even further deconstructed, OpenPBS itself is an open source implementation of the Portable Batch System.

Torque serves as Hopper’s Resource Manager, and essentially handles the counting and grouping of available resources in the cluster.

Moab serves as the Workload Manager, and performs the majority of decision making for where and when jobs are allocated.

Hopper implements a shared-maximum model of scheduling which guarantees that each PI has access to the resources that they have purchased, while also providing extra computational power through leveraging underutilized processing power. This model relies heavily on Moab “reservations” which are similar to traditional queues, but are defined in terms of ownership. Moab reservations grant access to the system’s global pool of researchers when unused, but will initiate job preemption when they are eventually requested by the owner.

If your job(s) consume more than your share (or your sponsor’s share) of available resources, they have a high chance of being preempted. Therefore, cluster researchers are encouraged to be mindful of their primary allocation of cores, the system load, and the current demand from fellow researchers when requesting resources from the Workload Manager using the commands described below.

TorqueMoab provide commands that give users information about resource availability in order to obtain quicker job turnaround times and to more fully utilize the system. Deep familiarity with these commands is essential in gleaning useful work from the machine.

Job Submission Overview

Hopper’s recommended methods for job submission and monitoring is through the use of the Torque “q” based commands, specifically “qsub”.

TorqueMoab provide a multitude of commands that will enable you to (in addition to others) instruct the cluster on what code and data to act upon, poll and assign available resources, and estimate and specify how long a given job should run.

Determining optimal values for “qsub” parameters comes with experience, and it is usually best to submit test jobs to get an estimate for the number of processors and wall time.

Familiarity with the qsub command, specifically its expected syntax, input, and behavior, is crucial for getting work done on Hopper.

How to Submit a Job Job submission is accomplished using the Torque ‘qsub’ command. This command includes numerous directives which are used to specify resource requirements and other attributes for jobs. Torque directives can be in a batch script as header lines (#PBS) or as command-line options to the qsub command.

The general form of the qsub command:

qsub –l <resource-list> -m abe –M <email addr> <job script>

Job Submission Options

qsub Option	Description
-q queue	Specifies a queue
-l resource_list	Defines resources required by the job
-m abe	Sends email if the job is (a) aborted, when it (b) begins, and when it (e) ends
-W x=FLAGS:ADVRES:	Specifies a reservation

Resource List

Resource	Descripton
mem	Max amount of physical memory used by the job
nodes	Number of nodes for exclusive use by the job
ppn	Number of processors per node allocated for the job
walltime	Time from job start to job completion. ( Default: 2 days/Maximum: 90 days )
reservation	Dedicated resources ( nodes ) based on your group.

Examples

Example 1:

This job submission requests 40 processors on two nodes for the job ‘test.sh’ and 20 hr of walltime. It will also email ‘nouser’ when the job begins and ends or if the job is aborted. Since no queue is specified, the general queue is used as it is the default.

$ qsub -l nodes=2:ppn=20,walltime=20:00:00 -m abe –M nouser@auburn.edu test.sh

Example 2:

This job requests a node with 200MB of available memory in the gen28 queue. Since no walltime is indicated, the job will get the two day default walltime.

$ qsub -q gen28 -l mem=200mb /home/user/script.sh

Example 3:

This job specifies a reservation.

$ qsub -l nodes=2:ppn=20,walltime=20:00:00 -W x=FLAGS:ADVRES:hpctest_lab.56273

In this example, the reservation id is hpcadmin_lab.56273. Your reservation id will be different. Please run the showres command to determine your reservation id.

Example 4:

This job specifies a reservation using rsub.

$ rsub -l nodes=2:ppn=20,walltime=20:00:00 -m abe –M nouser@auburn.edu test.sh

This example does the same thing as the previous example: it specifies a reservation in the job submission. However, it does so by means of a wrapper script ‘rsub’ that finds your correct reservation and includes it without you having to do so yourself.

Testing

Compile and run a test job on login node first before submitting a job to the queue.

To see how your job will run on a compute node, exactly as it will using the scheduler, you can use an interactive job in the debug queue:

$ qsub -q debug -l nodes=1:ppn=8 -I -V

Common Resource Commands

Moab Command	Description
pbsnodes	Detailed node information
pbsnodes -l free	Available nodes
showres -n	Nodes assigned to you
showbf	Shows what resources are available for immediate use

Examples

What is the estimated start time for a job requiring 20 processors for 30 minutes?

What resources are available for immediate use?

Note: Tasks are equivalent to processors in this context.

Monitor Jobs

To monitor the status of a queued or running job, use the qstat command:

Example: Display running jobs

$ qstat –r

qstat Option	Description
-u user_list	Displays jobs for users listed in user_list
-a	Displays all jobs
-r	Displays running jobs
-f jobid	Displays the full listing of jobs
-n	Displays nodes allocated to jobs

To display information about active, eligible and blocked jobs, use the showq command:

$ showq

To display detailed job state information and diagnostic output for a specified job, use the checkjob command:

$ checkjob -v <jobid>

To cancel a job:

$ canceljob <jobid>

In some cases, a job can get stuck and does not respond to the canceljob command, if you have a job that refuses to die, you can try:

$ mjobctl -F <jobid>

Monitor Resources

Use the checkquota command to check your disk space usage.

$ checkquota

To see if you have files that are scheduled to expire soon

$ expiredfiles