The HPC Cluster is a large collection of mostly high-end nodes (AMD & Intel 64-cores, 48-cores, and 32-cores nodes with a few GPU nodes). The nodes can be login, interactive, computational, ionodes, or a mixture.

HPC uses Son of Grid Engine (GE for short) to manage all of the resources (the nodes). You will be using GE to request nodes via queues. You will never access nodes directly since GE takes care of managing all of the resources to ensure that work is properly distributed across the cluster.

Types of Queues:

The HPC Cluster has 2 main types of queues:

  • Public Queues and

  • Private Queues

Public Queues:

The Public queues are queues that all users on HPC have access to. All public queues have a maximum wall-clock run-time limit and this is done to keep any one user from running forever. Limits ensure that other users will get to run as well.

Note ALL Public queues have a 3-day (72 hour) max wall-clock run time limit. When the limit is reached, jobs are automatically killed unless you are running with HPC CheckPoint

Private Queues:

The Private queues are queues that are only accessible to specific HPC users and/or HPC groups. These queues point to nodes that have been purchased by individual research groups or schools. If you are interested to see the owners of HPC nodes, the list is here:

Note All (most) private queues on HPC do NOT have run-time limits.

Which Queues do I have access to?

HPC has some 60+ queues. When you run queue or q on HPC, only the queues you are authorized to use will be displayed. The command:

$ queue

or the simpler short name of

$ q

Will display both Private (if any) and Public queues you are allowed to use.

Note If you believe that you should have access to a particular queue not listed by q, please email hpc-support@uci.edu with the details.

You can also see what queues other users have access to by specifying the HPC userid after the q command. For example:

$ q jfarran

Will show the HPC queues user jfarran has access to. Being able to see who has access to what queues can be helpful at times.

Understanding the output of q:

The output of q is actually fairly straight forward but we will cover it here as well.

=========================================================================+
|                              HPC QUEUES                                |
|                      http://hpc.oit.uci.edu/queues                     |
|========================================================================+
| Joseph Farran [ jfarran ]
| HPC Groups: [ staff bio ]
+========================================================================+

Joseph, you have access to the following HPC Queues:

PRIVATE QUEUES:
+-------------------+-------+--------+---------+-----------+-----------+
|                   | Cores |        |         |           | Available |
|        Queue      | Per   | Total  |  Total  | Available | FULL      |
|        Name       | Node  | Nodes  |  Cores  | Cores     | Nodes     |
+-------------------+-------+--------+---------+-----------+-----------+
          abio         64       19      1216        632          9
           bio         64        7       448        366          5

PUBLIC QUEUES:
+-------------------+-------+--------+---------+-----------+-----------+
|                   | Cores |        |         |           | Available |
|        Queue      | Per   | Total  |  Total  | Available | FULL      |
|        Name       | Node  | Nodes  |  Cores  | Cores     | Nodes     |
+-------------------+-------+--------+---------+-----------+-----------+
        free64         64       78      4992        839         13
        free48         48        3       144         18          0
        free32         32        1        32          0          0     <-- Queue full.
       free32i         32        1        32         24          0
       free24i         24        1        24          0          0     <-- Queue full.
           gpu         24        1        24         19          0
   interactive        128        1       128        118          0
         pub64         64        9       576        242          3
        ionode          1        9         9          8          8

The output above tells us the following:

  • Joseph Farran [ jfarran ] - This is the HPC user name and UCInetID / Login ID.

  • HPC Groups: [ staff bio ] - These are the HPC Linux groups the user belongs to. Users can belong to one or more HPC groups.

Next comes two listing, one for PRIVATE QUEUES and another for PUBLIC QUEUES. These represent the actual queues this user has access to. You can only access queues on HPC which you are authorized to use. If a particular queue is not listed, then you do not have access to it.

Q Column What It Means

Queue Name

This is the name of the queue you have access to. You can request this queue with GE using the -q option. For example qrsh -q free64

Cores Per Node

Number of cores each node has. 64 means that each node has 64 cores.

Total Nodes

Number of total nodes the queue has.

Total Cores

Number of total cores the queue has. For example the free64 queue has 78 nodes and each node has 64 cores, so this queue has a total of 4,992 cores (78 * 64).

Available Cores

Number of available cores that you can RUN WITH RIGHT NOW. Maybe because the scheduler may be reserving cores for a large MPI job.

Available FULL Nodes

Total whole nodes that are currently available ( idle ) that you can RUN WITH RIGHT NOW. This is an approximation and can be off.

Looking at the free64 queue list above, of the 4,992 cores, 839 cores are currently available for you to use right now (again maybe becuse the scheduler may be reserving cores for a large MPI job).

What happens when a Queue is full?

We hit critical mass and an explosion happens :-).

When a queue is full, the "← Queue full" indicator is shown at the end of the queue line. For example:

free32         32        1        32          0          0     <-- Queue full.

A full queue means that all cores on that queue are currently in use by jobs (0 cores available)

When a queue is full, you can submit jobs to the queue but your jobs will not run right away. Your jobs will sit in the queue until cores become available for work. When cores become available, your jobs will then run if you are next in line waiting to run.

We are using Grid Engine functional FairShare setup to make sure everyone has a chance to run. With FairShare jobs from users are interlaced so that jobs are released in an orderly and fair manner.

For example if User-A submits 1,000 jobs and User-B submits 100 jobs, the first 100 jobs from each user will be released in an interlaced manner.

Grid Engine Scheduler Limits:

When you use the q command on HPC, you may get something similar to this:

*** Grid Engine Scheduler LIMITS placed on the following HPC queue(s) ***

 limit                filter
slots=1961/4000       users jfarran queues free64

The listing above is saying that user jfarran is currently using 1,961 cores out of 4000 limit on the free64 queue. Even though the free64 queue has 4,992 possible cores, we do place some limits on the public queues to keep any one user from taking over the entire cluster. The above limits are adjusted by the system administrator at various times to keep the cluster running optimally.

When you reach the limit you can keep submitting jobs but new jobs will be queued-up. In other words, the limits are how many concurrent jobs you can have running at any one time and after that, new jobs will wait in the queue.

How can I see all HPC jobs?

To see a listing of ALL jobs on HPC Cluster running, waiting to run, etc use qstat. For example:

$ qstat

To see just your jobs:

$ qstat -u $USER

To see only the running jobs:

$ qstat -r

To see the manual of qstat:

$ man qstat

Understanding qstat output:

Here is partial listing of qstat and a short description of what the columns mean.

job-ID  prior   name     user   state   submit/start   queue              slots task

176930 0.3416 SHEAR.04  user1     r     03/18/2014   tjr@compute-4-7       256
180106 0.9749 q_0.1_p18 user2     r     03/18/2014   free64@compute-3-3      1
180727 0.1833 Si30nm_11 user3     r     03/19/2014   pub64@compute-7-1      64
180729 0.9452 getN43.tc user4     qw    03/20/2014
180740 0.1437 pwfa      user5     S     03/17/2014   free64@compute-8-13    32
180742 0.0289 blackscho user6     Rr    03/20/2014   pub64@compute-7-6       4
180777 0.4999 CMAQ      user7     dr    03/20/2014   air64@compute-8-3      32
182721 0.0928 run_TEMet user8     r     03/20/2014   bsg@compute-2-8         1   1
182721 0.0928 run_TEMet user8     r     03/20/2014   bsg@compute-2-8         1   2
182721 0.0928 run_TEMet user8     r     03/20/2014   bsg@compute-2-8         1   3
182721 0.0928 run_TEMet user8     r     03/20/2014   bsg@compute-2-8         1   4
182721 0.0928 run_TEMet user8     r     03/20/2014   bsg@compute-2-8         1   5
Queue Column What is means

Job-ID

The job id. All HPC jobs have unique job ID. You use the job id whenever you need to reference a particular job ( like qdel 182721 ).

prior

Job priority. Grid Engine functional FairShare has been enabled on HPC to make sure everyone waiting to run get’s equal and fair access. Note that this is only used by jobs waiting to run and has NO effect on jobs currently runnning!

name

Name of the job.

user

The user login id.

state

The state of the job. See below for an explanation of some of the run states.

submit/start

When the job was submitted. When the job actually starts, the field is updated.

queue

The queue and compute name this job in running under.

slots

The number of cores the job has requested.

task

Task ID if this is a job array. Field will be empty if this is not a job array.

  • User1 is running (r) on private tjr queue with 256 cores

  • User2 is running (r) on public free64 queue with 1 core.

  • User3 is running (r) on public pub64 queue using 64 cores.

  • User4 is waiting to run (qw) queue wait.

  • User5 job has been suspended (S) ( see http://hpc.oit.uci.edu/free-queue ).

  • User6 job is Re-Running (Rr) after it was CheckPointed ( see http://hpc.oit.uci.edu/checkpoint ).

  • User7 job is being deleted (dr).

  • User8 is running (r) a job-array with 1-core (slots) per job with array (task) going from 1-5.

What is interactive Queue?

When you log into HPC (hpc.oit.uci.edu), you are connected to one of two possible login nodes.

The login nodes are simple basic nodes (16 cores) that are meant for doing very light work like:

  • Editing files (emacs, vi)

  • Submitting jobs (qsub)

  • Removing jobs (qdel)

  • Checking on job status (qstat)

If you need to do more extensive work like compiling & testing your code, running interactive programs like Mathematica, etc, you will need to use an interactive node. You request an interactive node with:

$ qrsh -q interactive

Or simply by doing qrsh since this is the default queue on HPC:

$ qrsh

The interactive node should be used for things like:

  • Compiling.

  • Testing your program to make sure all works before embarking on a large run.

  • Running one or two interactive programs like Matlab, Mathematica, vmd, etc.

When you are ready for more serious work, like running multiple instances of your program, large runs, etc, you will need to request compute nodes.

There is a entire web page dedicated on how to run jobs on HPC and that web page can be found at:

What are the Free Queues?

What is the pub64 Queue?

These are public 64-core nodes which are available to all HPC users. These are nodes that were purchased by OIT (actually nodes that we got from a process known as SEP) for everyone to use.

What is the difference between pub64 and free64 queue?

The main difference between the public pub64 queue and the public free64 queue, is that:

  • Jobs on the public pub64 queue will NEVER SUSPEND.

  • Jobs on the public free64 queue can be suspended ( see here )

Note ALL Public queues have a 3-day (72 hour) max wall-clock run time limit. When the limit is reached, jobs are automatically killed unless you are running with HPC CheckPoint
Note When a job is suspended, the wall-clock timer keeps on counting. So if your job ran for say 10 minutes and then got suspended. The job will get killed after 3 days regardless of how many minutes of computation it got.

What is ionode Queue?

When you need to move large amount of data (I/O) to & from HPC, meaning moving data outside of the HPC cluster, you will want to use one or more ionodes. You request an ionode with:

$ qrsh -q ionode

You can then start your sftp, scp, rsync, filezilla, etc, file transfer. With an ionode you can access systems outside of the HPC cluster.

  • By using an IO node you will have a full dedicated GigE Ethernet connection and your data transfer will be that much faster.

  • If you do NOT use an ionode, you will be competing with all HPC users login I/O traffic on the login nodes and so your data transfers will be that much slower.

Please remember to release the ionodes when done (type exit).

What is the gpu Queue?