Torque Job Submission

R2D2, the star wars cluster, consists of 30 dual processor, 64 bit, Opteron, computational nodes, with routing to 3 of the jedi computers as well. These jedis, 3, 4, and 5 are similar to the other computational nodes, except that they have 4 processors instead of 2, so that they are exceptional for cases where the software is multithreaded.

Interactive use of the star wars cluster is prohibited, with a few exceptions, such as cluster monitoring software and CCP4i_6.0.1, which has the queue submission commands built in. All other jobs should be submitted with the qsub command. The qsub command has a different implementation on r2d2 than on queen, which necessitates these instructions. It is advisable to review the instructions for the hive, and use it when the star wars cluster is busy.

In order submit jobs it is necessary to ssh to r2d2. If you have trouble with this step, consult our general computing and software documentation, or see the facility manager.

Using the Torque implementation of qsub, which we have on r2d2, jobs can now be routed to the machines which are best suited in terms of memory requirements and multi-threading capabilities, as in the following examples.

qsub -q lowmem
qsub -q medmem
qsub -q himem

qsub -q @jedi_master

qsub -q @jedi_master3
qsub -q @jedi_master4
qsub -q @jedi_master5

Add the name of your command script to one of those commands in order to submit it to the appropriate queue, and feel free to let me know if you hit any snags. We are ready to help you get underway with the cluster.

The high priority default queue is lowmem, so if you have special requirements, other queues should be selected. Here are examples of some commonly used queue queries which are adapted to the new situation.

qstat -q
qstat -q midmem
qstat -q @midmem
qstat -q @jedi_master5

Note the server line in the output. The -q option works with qsub too, but I think qdel must be run on the destination server. Entries with an @ in front are destination queues. Master is now a dedicated routing queue server, so jobs disappear from the master queue and are passed along to destination queues.

Here are some job query commands.

qstat -na @lowmem
qstat -na @midmem
qstat -na @himem
qstat -na @jedi_master3
qstat -na @jedi_master4
qstat -na @jedi_master5

Here are some pbsnodes examples.

pbsnodes -a -s midmem
pbsnodes -a -s jedi_master4

Together with our recent gigabit and parallel file access upgrades, now beowulf now provides a fast and up to date clustering tool. It was quite some work, but if you have any further questions or suggestions, please feel free to pass them along.

document created by Michael L. Love, and maintained by the Crystallography facility