Submitting you job at ABCC

Do not run your job interactively (unless you want Karol to hate you). You have to use a script (say runme.csh) that contains all the instructions to run the program on their cluster, and to transfer the files back and forth between the ABCC machine you connect to (e.g., nciiris.ncifcrf.gov) and the node of the cluster on which you job is running (which you find out by typing qstat -ans). A typical ``runme.csh'' script, that you submit to the queue using qsub runme.csh, looks like this:

#PBS -S /bin/csh
#PBS -N job_name
#PBS -l pcput=20:00:00,cput=20:00:20,pvmem=400mb,ncpus=2
rcp nciiris:your_input_files /usr/tmp/$user/.
source /usr/local/fbscapp/accelrys/cshrc
set CHARMM=/usr/local/fbscapp/accelrys/I2000.1/mpi6m3/charmm.exe
set input="stream=your_input_file"
cd /usr/tmp/$user/
mpirun -np 2 -cpr $CHARMM $input < stream.inp > your_output_file
rcp /usr/tmp/$user/your_output_files nciiris:/usr/tmp/$user

A few comments :

• You define the number of processors to be used by CHARMM via the -np flag to the mpirun command. Remember, to keep this number and the number of processors in the PBS limits line (the ncpus=flag in the #PBS line) in sync.
• You need to request much more memory for parallel MPI runs than for single processor runs. A 4 processor run may require even more than 4 times the memory for a single processor run. However, since the NCI systems have large memory installed, this should not be a problem.
• There are three Accelrys precompiled executables: charmm.exe, charmm.large, and charmm.xxlarge. If charmm.exe is not big enough for you, replace it with charmm.large or charmm.xxlarge in the line:

  set CHARMM=/usr/local/fbscapp/accelrys/I2000.1/mpi6m3/charmm.exe
  

• charmm.large (and expecially charmm.xxlarge) will require substantially more memory than charmm.exe. When combined with extra memory required for parallel runs, the memory demand may be prohibitive for charmm.xxlarge on some systems or memory limits available on your account may not be large enough. You should probably contact Karol, if you encounter this problem.
• If you want other (and probably better) ways to transfer the files to and from the node of the cluster, look at /etc/pbs-intro, as suggested by the login message at ABCC.