jacked.in

Moab / Torque Primer Part 2: Resources

2009-11-08T23:53:00Z

The main purpose of Torque and Moab is to manage, monitor and schedule resources. At a very high level the most obvious types of resources are compute nodes. Each of these compute nodes, however, contributes resources at a much finer level to the resource pool and resources monitors such as Torque often monitor processors, main memory, storage, etc. In addition to these rather common resource types, nodes may also be associated with other more specific resource types, such as network bandwidth, software licenses or power consumption. However, for a general understanding considering compute nodes and processors as resources is sufficient.

Torque is responsible for monitoring the resources provided by compute nodes. Each compute node runs a small daemon program called pbs_mom which collects all relevant resource information on the compute node it is running on and sends this information to Torque’s pbs_server which typically runs on the headnode of the cluster. The pbs_server then stores this information in an internal database which can be used by schedulers such as Moab to get real-time information about the availability of resources.

Torque provides access to its list of available resources through the pbsnodes command. Without any command-line arguments the command lists all nodes that Torque knows about and the resources it monitors for each node:

$ pbsnodes
compute-0-0.local
     state = job-exclusive
     np = 8
     ntype = cluster
     jobs = 0/45.cluster.local, 1/46.cluster.local, 2/47.cluster.local, 3/48.cluster.local, 4/49.cluster.local, 5/50.cluster.local, 6/51.cluster.local, 7/52.cluster.local
     status = opsys=linux,uname=Linux compute-0-0.local 2.6.9-42.ELsmp #1 SMP Tue Aug 15 10:35:26 BST 2006 x86_64,sessions=9878 9881 9882 9883 9889 9918 9924 9927,nsessions=8,nusers=1,idletime=16682405,totmem=12258160kb,availmem=10771408kb,physmem=8161596kb,ncpus=8,loadave=9.43,netload=1074777221096,state=free,jobs=31.cluster.local 46.cluster.local 45.cluster.local 47.cluster.local 49.cluster.local 48.cluster.local 50.cluster.local 51.cluster.local 52.cluster.local,varattr=,rectime=1250282323

...

compute-0-3.local
     state = free
     np = 8
     ntype = cluster
     status = opsys=linux,uname=Linux compute-0-3.local 2.6.9-42.ELsmp #1 SMP Tue Aug 15 10:35:26 BST 2006 x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=27314517,totmem=12258140kb,availmem=11926516kb,physmem=8161576kb,ncpus=8,loadave=0.00,netload=4289316359815,state=free,jobs=972217.cluster.local,varattr=,rectime=1250282329

For each node it lists the node’s state, the number of processors provided by the node, as well as other attributes such as available memory or jobs currently executed on the node. In most cases the default pbsnodes output is too verbose and a number of command-line switches can help filtering the information:

pbsnodes -l -N lists all nodes marked with states offline, down, or unknown and respective user-defined comments associates with these nodes:

$ pbsnodes -l -n
compute-0-0.local   offline                    reimaging
compute-0-5.local   offline                    blinking HDD
compute-0-7.local   down,offline               replacing CPU

pbsnodes -l free lists all nodes that are free, i.e. which can potentially run jobs:

$ pbsnodes -l free
compute-0-1.local   free
compute-0-2.local   free
compute-0-3.local   free
compute-0-4.local   free
compute-0-6.local   free

In addition to reporting node states, pbsnodes may be used to set a node state manually. This is useful for maintenance or debugging of hard-/software problems. pbsnodes -o [nodename] is used to mark a node as offline. This allows jobs that may currently be running on this node to complete, but no new jobs will be allocated to the node. In Moab terminology the node is considered to be drained if it is marked as offline. Additionally, when putting a node into an offline state, a comment or note should always be provided, such that the reason for taking the node offline can be easily determined. The user-defined comment can be set with the -N command line option:

$ pbsnodes -o -N "installing new hard-drive" compute-0-5.local
$ pbsnodes -l -n
compute-0-5.local   offline                    installing new hard-drive

Once a node can be placed back into the pool of available resources its offline state needs to be cleared. This is done by using the -c option to pbsnodes. At the same time the comment can be removed from the node using -N "":

$ pbsnodes -c -N "" compute-0-5.local
$ pbsnodes -l -n free
...
compute-0-5.local    free
...
$ pbsnodes compute-0-5.local
compute-0-5.local
     state = free
     np = 8
     ntype = cluster
     status = opsys=linux,uname=Linux compute-0-5.local 2.6.9-42.ELsmp #1 SMP Tue Aug 15 10:35:26 BST 2006 x86_64,sessions=18901,nsessions=1,nusers=1,idletime=36154599,totmem=12258140kb,availmem=11888652kb,physmem=8161576kb,ncpus=8,loadave=0.01,netload=3655128653370,state=free,varattr=,rectime=1250283674

In addition to Torque’s tools to manage nodes, Moab also provides tools to gain an insight into Moab’s view at the nodes and the resources they provide. This is done using the checknode command.

The checknode command is only used to report information about a node and cannot be used to modify any of the node’s parameters. A typical command output for a busy node looks like:

$ checknode compute-0-3.local
node compute-0-3.local
State:      Busy  (in current state for 00:32:26)
Configured Resources: PROCS: 8  MEM: 7970M  SWAP: 11G  DISK: 1M
Utilized   Resources: PROCS: 8  SWAP: 1189M
Dedicated  Resources: PROCS: 8
  MTBF(longterm):   INFINITY  MTBF(24h):   INFINITY
Vars:       RACK,SLOT,Rack
Opsys:      linux     Arch:      ---   
Speed:      1.00      CPULoad:   8.080
Network Load: 10.35 kB/s
Flags:      rmdetected
Network:    DEFAULT
Classes:    [queue1 8:8][queue2 0:8][queue3 8:8]
RM[cluster] TYPE=PBS  STATE=Busy
EffNodeAccessPolicy: SHARED

Total Time: 78:12:29:09  Up: 78:12:04:28 (99.98%)  Active: 13:20:15:23 (17.63%)

Reservations:
  res1.609x8  User  9:51:12 -> 11:51:12 (2:00:00)
    Blocked Resources@9:51:12     Procs: 8/8 (100.00%)  Mem: 0/7970 (0.00%)  Swap: 0/11970 (0.00%)  Disk: 0/1 (0.00%)
  66x8  Job:Running  -00:32:26 -> 1:27:34 (2:00:00)
  res1.614x8  User  1:09:51:12 -> 1:11:51:12 (2:00:00)
    Blocked Resources@1:09:51:12  Procs: 8/8 (100.00%)  Mem: 0/7970 (0.00%)  Swap: 0/11970 (0.00%)  Disk: 0/1 (0.00%)
Jobs:        66

The output provides a number of details about the compute node including information on what resources are available on the node (Configured), which of those are currently in use (Utilized), and which resources have been dedicated to a job (Dedicated). Note that a node may show a utilization of resources that are not explicitly dedicated. In the case above, the resources that are dedicated to a job are 8 CPUs, however, the node also reports a utilization of 1189 MB of swap space. In most cases this is ok until the utilized resources reach the limit of physically available resources or the load of the system dramatically exceeds the number of available processors. In the ideal case a job should always only utilize at most the resources that are dedicated to it and if this is not the case the job’s owner should be informed to adjust the resources requested by any subsequent similar jobs.

The checknode output additionally provides an overview of any current and future resource reservations. The example above shows reservations for 2 future standing reservations (res1) and one current reservation for a job (66). One can identify current reservations by their start time being negative.

Aside of typical computation resources such as CPU, main memory and disk, Torque and Moab allow user-defined resources to be associated with nodes. Such resources can for example be special hardware devices, a limited number of software licenses or application specific resource measures (e.g. Dreamhost’s Conueries). These kinds of resources are typically defined through the NODECFG directive in Moab’s configuration file.

Moab / Torque Primer Part 1: Submitting Jobs

2009-08-26T04:38:00Z

Traditionally, the most common way of submitting jobs to a cluster is from the command-line or a shell script by using the qsub command. qsub is part of Torque and submits jobs into Torque’s job queue. When Torque and Moab are used in combination, then Moab receives the information about the job through Torque automatically which is typically completely oblivious to the user.

For the user to control various properties of the job and have control over what the job does, qsub takes a so-called submission script as input. The submission script is typically a normal shell script written by the user that describes which commands should be executed once the job gets to run. The submission script can be passed to qsub either through standard input or as a file specified as a command-line argument:

$ cat << EOF > test_job.sh
sleep 60
hostname
EOF
$ qsub test_job.sh

is equivalent to:

$ echo "sleep 60 ; hostname" | qsub

In either case, Torque makes an internal copy of the script and uses this copy to actually run the job. In most cases the user would write the submission script as a file as such scripts are typically rather complex.

In addition to the submission script qsub also accepts arguments that specify certain properties about the job, such as the job’s name, its expected runtime, the priority or into which queue it should be submitted. Those properties can either be specified as command-line arguments to the qsub command or specified within the job’s submission script using a special syntax. For example, to submit a job with name “testjob”, the name of the job can either be specified as a command-line argument:

$ cat << EOF | qsub -N testjob
sleep 60
hostname
EOF

or as part of the submission script:

$ cat << EOF | qsub
#PBS -N testjob
sleep 60
hostname
EOF

Note, the #PBS directive in the submission script. It instructs Torque to interpret the following parameters as if they where specified on the command-line.

Please see the qsub manpage for information on other command-line arguments and additional information.

Moab / Torque Primer: A Gentle Introduction to Job Management

2009-08-26T03:34:00Z

Moab and Torque are two software packages that work closely together and are used in combination at many HPC sites:

Torque is an Open Source resource manager which is responsible for collecting status and health information from compute nodes and keeps track of jobs running in the system. It is also responsible for spawning the actual executables that are associated with a jobs, e.g. running the executable on the corresponding compute node.
Moab is a commercial scheduler product developed by ~~Cluster Resources Inc.~~ Adaptive Computing which is responsible for allocating resources to jobs that are requesting resources. It does so by collecting all the information that Torque (or other resource managers) can provide about currently running jobs, available nodes and other resources. Once Moab has scheduled resources for a job, it instructs Torque to execute the job on the allocated resources.

This collection of articles is intended as a very basic introduction to resource and job management with Moab and Torque. It provides a high-level practical starting point for new HPC system administrators who want to become more familiar with these systems. It is not intended to provide comprehensive and detailed descriptions of all of the systems’ features or their configuration.

The articles are organized in parts which will be posted as they become available:

Submitting Jobs
Resources
Queues
Reservations
Job Status
Modifying Jobs
Canceling Jobs

Decipher OSM log messages

2009-08-05T23:55:00Z

The last post described a situation where the OpenSM Infiniband (IB) subnet manager was logging hundreds of messages per second to its log file /tmp/osm.log and proposed an intermediate solution for preventing it to fill up the file system. This post is about tracking down the root of the issue.

Supposedly the OpenSM log file contains messages like:

Aug 04 23:27:02 910870 [40A04960] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:131 Producer:2 from LID:0x008B Port 6 TID:0x000000002c20f6fa
Aug 04 23:27:02 910896 [40A04960] -> __osm_trap_rcv_process_request: ERR 3804: Received trap 1091363 times consecutively
Aug 04 23:27:02 912507 [40401960] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:131 Producer:2 from LID:0x008B Port 6 TID:0x000000002c20f6fb
Aug 04 23:27:02 912533 [40401960] -> __osm_trap_rcv_process_request: ERR 3804: Received trap 1091364 times consecutively
Aug 04 23:27:02 914191 [40602960] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:131 Producer:2 from LID:0x008B Port 6 TID:0x000000002c20f6fc
Aug 04 23:27:02 914212 [40602960] -> __osm_trap_rcv_process_request: ERR 3804: Received trap 1091365 times consecutively

These messages are rather cryptic and by themselves not particularly helpful. However, they contain information about the origin of the error: Producer:2 from LID:0x008B Port 6. This obscure pair of LID and port actually refers to a port on the IB switch, which is reporting the error to the subnet manager. Now, one can log in to the IB switch and try to figure which physical port and cable on the IB switch are associated with the given (LID, port) pair. Then a trip to the server room and digging among numerous IB cables may reveal the machine that is the cause of all this trouble.

Alternatively, one can open the software toolbox and pull out ibdiagnet — a tool that is part of the OpenIB/OFED distribution. ibdiagnet provides a number of useful functions to debug Infiniband networks and, in addition to general IB network path information, it conveniently provides a mapping from IB switch ports to machine hostnames for all IB host interfaces that are reachable. Even though it can only report that mapping for interfaces that are reachable it can still be used to identify interfaces that are offline assuming the IB cabling was following some predictable pattern.

When running ibdiagnet without any command-line arguments it will run a number of diagnostics and leave a couple of files in /tmp. A detailed list of the files can be found in the ibdiagnet manpage. The file /tmp/ibdiagnet.lst provides a list of all active ports in the IB fabric, including ports that are internal to the switch. Additionally, for any host ports that are active it will show the hostname configured for the corresponding host. This information is used to eventually identify the IB host that causes the troubles:

...
{ SW Ports:18 ... Chip A} LID:008B PN:02 } { CA ... {compute-0-3 HCA-1} LID:008F PN:01 } PHY=4x LOG=ACT SPD=5
{ SW Ports:18 ... Chip A} LID:008B PN:04 } { CA ... {compute-0-5 HCA-1} LID:0002 PN:01 } PHY=4x LOG=ACT SPD=5
{ SW Ports:18 ... Chip A} LID:008B PN:05 } { CA ... {compute-0-6 HCA-1} LID:0006 PN:01 } PHY=4x LOG=ACT SPD=5
{ SW Ports:18 ... Chip A} LID:008B PN:07 } { CA ... {compute-0-8 HCA-1} LID:0004 PN:01 } PHY=4x LOG=ACT SPD=5
{ SW Ports:18 ... Chip A} LID:008B PN:08 } { CA ... {compute-0-9 HCA-1} LID:0005 PN:01 } PHY=4x LOG=ACT SPD=5
...

Since /tmp/osm.log explicitly refers to switch LID 0x8B Port 6, it can be easily determined that the entry for LID 0x8B Port 6 is missing. The nodes are connected to the switch in a particular order and following that wiring pattern LID 0x8B Port 6 would be connected with compute-0-7, the presumed trouble maker. A quick check on the node indeed revealed a kernel panic which prevented the IB driver to initialize the host interface correctly.

Obviously, doing such investigation every single time there is a problem with the IB network may become tedious. So using the information collected by ibdiagnet to create an up-to-date chart for (LID, port) to hostname mappings may be a good idea.