The HPC Cluster consists of several private and public data storage servers with private disk space being the majority. For a complete list of data storage servers on the cluster please visit this link.
As of mid 2016 the cluster has approx ~1.5 petabyte of cumulative disk space between all data servers and the demand for additional disk space keeps increasing so having some kind of backup has become critical.
Due to the fact that we cannot backup everything on the cluster, HPC Selective Backup was created so that you can choose what is important and have it automatically saved. The physical location of the backup server is different from the HPC location for extra protection in the event of a fire, etc.
Quick Start Guide
The Selective Backup process will automatically start saving your home directory as well as some public and private disk spaces. There is nothing for you to do if you like the defaults, however, we highly recommend that you request email notifications (see below) to make sure things are working and perform some spot checks of what you think is being saved to make sure your data is indeed being backed-up.
How does it work?
The HPC Selective Backup process is based on rsync in conjunction with GNU Parallel. The combination of both rsync and GNU Parallel maximizes the network throughput and server capabilities in order to backup hundreds of user accounts from dozens of public and private HPC file-systems.
Users manage their Selective Backup options via two dot-files located in their home directory (/data/user/$USER) called:
These are text files that can be edited with your favorite editor. The .hpc-selective-backup file must be edited with the paths of the directories and files to be saved in order of priority (i.e: first line contains the file/dir most precious, second line has the second most precious file/dir to be saved etc.). The objects are backed in sequential order as they are listed. That way, if a user runs out of selective disk quota before all objects they have listed in .hpc-selective-backup have been backed up, at least they will hopefully have their most prized data saved.
The file .hpc-selective-backup-exclude can be edited with file extension and file/dir names of objects you do not want to be backed up. For more information on rsync exclude patterns please see the "ANCHORING INCLUDE/EXCLUDE PATTERNS” section of man rsync.
What is my disk quota?
Since not all data on HPC is critical and since we cannot backup everything due to the huge file-systems on HPC, every user on receives 1TB of backup disk space.
|Every user on HPC receives 1TB of backup disk space.|
This is your backup disk quota. 1TB should be enough for most users but if it is not enough for your needs, you can purchase additional backup space over the 1TB limit. For pricing please email HPC support firstname.lastname@example.org
What happens when I go over my backup quota?
If you go past your backup disk quota then backups stops for your account. Nothing else can be saved. The rsync’s will complete in error as no new data can be written to the backup server since you reached your limit.
|When you reach your backup quota limit, nothing further can be saved to the backup server.|
How much of my backup quota have I consumed?
Instructions coming up soon.
Managing your .hpc-selective-backup text file
Your ~/.hpc-selective-backup configuration file is initially pre-configured for you to include your home directory (/data/users/$USER), your public data directory /pub/$USER and any other directories such as /bio/$USER, /som/$USER, etc, if they exist for your account.
You can edit the configuration file to add, remove, or change the order of the directories to be saved. If you have additional private disk space on HPC at /share/something/$USER, you can add those as well.
Here is what a typical .hpc-selective-backup file looks like:
##################################################### ## HPC Selective Backup Configuration File ##################################################### # # Un-comment any 'HPC_' line below to select that option. # HPC_SEND_EMAIL_SUMMARY # HPC_SEND_EMAIL_ON_ERROR # HPC_KEEP_DELETED=14 # # Specify directories or files to backup, one per line. # All paths must begin with root forward slash "/". # The order is important as the saves are processed in # the sequence you specify below. # /data/users/jfarran /pub/jfarran/project1 /pub/jfarran/project2 /share/tw/jfarran/data-file
Anything after the pound sign "#" are comments and are ignored by the backup process. Comments are included in the configuration file to help you understand how it works.
The configuration file has two sections:
Directories / Files to Backup
Here you select what backup options you prefer. All backup options start with "HPC_". To select an option simply un-comment it (remove the pound sign in front). For example if you wish to receive email notifications in the event errors are encountered with your saves, edit your configuration file and change the line:
And the backup process will now send you an email if any rsync completes in error. If no errors are found, no email will be sent.
The following table lists all available "HPC_" options and what they do:
|Selective Backup Option||What It Does|
Sends you daily email summaries of your saves.
Default is NO summary email notifications.
You will receive an email if any RSYNC completes in error. Error being anything other than zero exit status from rsync. Please consult the rsync man page for error values and meaning.
If no errors are found with your rsync’s, no email will be sent.
Default is NO email notifications.
Keep deleted files on the backup server for X days where X is a number anywhere from 0 to 90 days. Deleted files are files you removed from the source location.
Please see "Where are DELETED files kept?" below for the location of your deleted files.
Default is 14 days.
Selecting Directories / Files to Backup
This is the main section of your backup configuration file. Here is where you specify the FULL PATH of what you want to save starting from the root "/" down to the directory or file you want to backup.
The rules are as follows:
One line per directory or file you want to backup.
You have to specify the full path starting from the top "/" (root). Any path not starting with root "/" will get ignored since that is not a valid full path.
Trailling "/" in a path will get removed.
No wild-cards, back-tics, control characters nor parsing is allowed. This keeps our security risks to a minimum.
You can specify any private or public directory or file that belongs to you (one per line).
There is no limit on the number of directories or files that you can specify (please keep it reasonable).
Please do NOT specify any scratch location such as /tmp, /scratch/$USER, /fast_scratch/$USER, etc.
The RSYNCs saves are done with the "-C" (--cvs-exclude) option to eliminate the saving of unwanted file types. See rsync man page for details of what this option eliminates.
Any file less than 5 bytes in size will NOT be saved to prevent our backup server from saving ZOT data (zillions of small files)
The order of the directories / files to save matters because saves are done in sequential order and if you go over your backup quota then nothing else from that point on will be able to be saved.
We also keep a list of several file-types that we are excluding from being saved to the backup server. To see the entire exclude list:
For a list of the rsync options we are using on HPC Selective Backup, please see:
Please note that the above rules are not static and we will be updating them periodically as the need arise. So if you discover a particular type of file that is not being saved, please consult the HPC global exclusions lists above before contacting HPC support.
If we are excluding a particular file type from being saved that you need backed-up please email HPC support.
Managing your .hpc-selective-backup-exclude text file
You may edit this text file to describe what type of files you want to exclude from being saved to the backup server. Wild-cards are allowed or any format as long as it adheres to rsync syntax. One entry per line.
The HPC rsync process uses the --exclude-from=FILE to read your ~/.hpc-selective-backup-exclude file. It is your responsibility to ensure that this file uses valid rsync syntax.
Here is sample .hpc-selective-backup-exclude file:
readme.txt README.txt processing.*
You can see if the exclusion process is working by reviewing your HPC backup logs at:
Why is the order of items to save important?
Since this is a Selective Backup System as opposed to a Full Backup System, the order of what to backup is important when you go over your backup disk quota.
The list of what you specify to backup is processed sequentially, one line at a time and NOT in parallel. This is done on purpose because you may run out of backup disk quota mid-way into your saves and when you run out of disk quota NOTHING further can be saved. Therefore you have the option of moving the directories or files that are most important to you up to the top of the list.
For example using the backup list shown before, we have:
/data/users/jfarran /pub/jfarran/project1 /pub/jfarran/project2 /share/tw/jfarran/data-file
Let’s say that "project2" directory is critical to my work followed by "data-file", then my home directory and I don’t need anything else saved. I edit my ~/.hpc-selective-backup file as follows:
/pub/jfarran/project2 /share/tw/jfarran/data-file /data/users/jfarran
Now when backups are ran "/pub/jfarran/project2" is the first item to be saved by the backup system. Assuming I am still under disk quota, next will be "/share/tw/jfarran/data-file" and if I am still under quota, finally my home directory.
The moment you run out of backup disk quota, nothing else will be able to be saved. Please note that when you run out of disk quota, the order will not matter until you are under disk quota again.
|When you reach your backup quota limit, nothing further can be saved to the backup server!|
I reached my backup disk quota, what can I do?
You have two options:
Edit your backup configuration file ~/.hpc-selective-backup and:
Remove items to backup. Remove enough items to be below your backup quota.
Reduced the number of days to keep DELETED files on the backup server by changing the "HPC_KEEP_DELETED=" value. Deleted files stored on the backup server counts towards your backup quota.
Add file types you don’t want saved in the exclusion configuation file ~/.hpc-selective-backup-exclude to reduce the amount of data to be saved.
Purchase additional backup disk space. Contact email@example.com for pricing.
Generally speaking you will want to only backup critical data such as scripts, programs, etc and not data you can get from other sources specially large data-sets.
How often are backups performed?
Backups are started at midnight with time limits per user to ensure that saves do not continue into the next day.
Where are my backup logs?
Your Selective Backup rsync logs are available for the past 30 days at:
Where yyyy-mmdd is the year, month, and day of when the system rsync process started. This is your complete rsync log including the actual rsync command that was issued on your behalf.
Can I retrieve my backup data?
Yes you can browse and retrieve any data the belongs to you from the HPC interactive node at location:
It is your responsibility to browse your saved files periodically to ensure that everything you wish to have backed-up is in fact being saved to the backup server. Further you may want to request email notifications of when saves complete in error to alert you of failed saves.
|It is YOUR responsibility to periodically check your saved data to ensure everything you think is being saved is in fact on the backup server. Please do NOT assume saves are working.|
|Request email notifications for when your saves complete in error|
Where are DELETED files kept?
When you remove a file from your source location, the rsync process will ALSO remove said file(s) on the backup server. This is the normal way rsync works - it mirrors the data from the source location to the destination location.
The HPC Selective process allows you to specify how long you wish to keep deleted files on the backup server after you removed them at the source. Your deleted files are kept by date at:
NB: This dir can only be accessed from the Interactive node (compute-1-13). You can use qrsh to start a session there.
Think of DELETED-FILES directory as your backup Trash-can from which you can get old deleted files up to the number of days you specified.
You can control how many days to keep deleted files via your HPC configuration file. Your deleted files count towards your backup disk quota as they are kept on the back server.
Questions & Answers
QUESTION: Is my data backed-up any other location?
ANSWER: YES! Check with the NSA. They have your data and a lot more.