List of the most useful GE commands:

Grid Engine Command What It Does

qstat

List all jobs on the cluster (running, waiting to run, etc ).

qstat -u $USER

List only your jobs ( $USER ).

qstat -s r

List all running jobs (job status "r/Rr").

qstat -s p

List all pending jobs ( waiting ) to run.

qstat -s z

List jobs that have completed ( last 50 jobs ).

qstat -q queue

List all jobs for queue running & waiting to run. ( ie qstat -q pub64 )

qstat -F -j job_id

Get Full details on job ( ie qstat -F 12345678 ).

qsub script

Submit a job script to the cluster.

qsub -hold_jid x,y,z

Submit job with a hold until jobs x,y and z complete.

qdel job_id

Delete a job (job_id) from the cluster.

qrsh

To request an Interactive node with one core (defaults to interactive queue)

qrsh -q pub64 -pe openmp 64

Request an Interactive compute node on the public queue (pub64) with all 64 cores.

qalter -q pub64,free64 job-id

Modify (alter) waiting job_id to use both free64 & pub64 queues. (This only works for jobs waiting to run).

qalter -p -100 job-id

Modify (alter) waiting job_id priority lower to -100. You cannot increase priority, only lower them.

qhold job_id

Hold a job.

qrls job_id

Release (Unhold) Job.

qmod -sj job_id

Suspend job. (if job is using checkpoint, it will not suspend but will migrate to another node instead).

qmod -sj job_id.nnn

Same as above but for job array element nnn ).

qmod -usj job_id

Unsuspend job.

Core Binding:

Important Core binding is currently semi-broken.

What is Core Binding?

Following description taken from here:

  • Core binding refers to the capability that Grid Engine is able to dispatch jobs (scripts or binary programs) to specific cores on an host not only to the host itself. The difference is that when a job is dispatched to an host then usually the operating scheduler places the jobs processes to sockets/cores depending on the OS scheduler policies. From time to time the process is moved from one CPU core to another, which can even include jumps to other sockets. For some applications (like HPC jobs, graphic rendering, or benchmarks) the movement between cores has visible influence in the overall job run-times. The reason for this is that caches are invalidated and must refilled again and that on NUMA architectures the memory access times differs depending on which core the job is running and on which memory socket memory was allocated. Also, in case of overallocation of an host (more slots/processes than compute cores), the run-times of the jobs (even when they are identical) varies a lot. For example when you submit 4 jobs on a dual socket machine (with 2 cores each) but each job spawns 4 threads, the run-time of the longest running job comparing to the run-time of the shortest job can be a multiple (even when the jobs are identical!).

  • With the core binding feature it is now possible to dispatch the jobs to specific cores, which means in case of Linux that the job gets a core bit mask which tells the OS scheduler that the job is only allowed to run on specific cores. When you are now submitting the 4 jobs (each with 4 threads) to a 4 core host and you tell GE during submission time that you want to have 1 core for each job (-binding linear:1) then each of the job gets one core exclusively (with the exception of OS internal processes). So all 4 jobs running then on a different core where they never can be moved away by the operating system scheduler. When you are comparing now the job runtimes you will get very similar results. In many cases the program execution times are shorter because of the better caching behavior. When you are using a NUMA machine exclusively for a job, which has less slots (processes/threads) than cores you can also speed up your program by forcing it to run on different sockets. This increases the total amount of used cache.

On 5/6/14 Linear Core Binding was activated on the HPC Cluster cluster-wide. This change should be transparent to users in that no modifications are needed to current Grid Engine scripts.

When you now request cores on HPC, you will run with requested cores only on a node. So if you request 16-cores, your program will only be able to consume 16-cores and no more. If you run in serial mode (default 1-core) your program will run on 1 core-only.

Core binding should help HPC jobs that don’t behave nicely in that if the job tries to use more cores than it has requested, it will not be able to. Programs such as MATLAB are notorious for trying to use as many cores as it can possibly get on a node regardless of how many cores have been requested. With core-binding this issue will no longer exist (also true for ANY program on HPC that tries to use more cores than allocated)

Core Binding: Workers (cores) to Process/Threads

Normally you request as many cores as the program is going to consume so that you have a one-to-one ratio (one-process to one-core).

Running four 16-core OpenMP jobs on a 64-core node will yield a load of around 64 (4*16) which is correct. One worker (one core) to one process/thread. Same thing if you run one 64-core OpenMP job on a 64-core node (a load of around 64 which is one-core to one-process).

For programs that use more processes / spawn more threads than cores have been requested, the program will be restricted to requested cores via core-binding. This will add an artificial load on the node which is normal but will not actually over-load the node.

Core Binding: Programs the vary in core-usage

There are programs/jobs that run serially (1-core) for a while and then speed up using all available cores on a node. So we have a dilemma: How many cores do we request?

  • Requesting all 64-cores on a 64-core node is a waste of cores because while the job runs serially (1-core), the other 63-cores will be idle and not in use.

  • Requesting 1-core only is also problematic because when the program speeds up using all 64-cores, it will over-load the node.

The solution here is to find the average core-usage over time and request that amount (error on the side of smaller core-count). For example start by requesting 8-cores and see how the job performs. When the job tries to use all available cores on a 64-core node, core-binding will limit the job to only 8-cores. Core-binding keeps the node from over-loading even-though there may be an artificially high load on the node for a while.