Beyond rsync

rdiff-backup is almost a superset of rsync in many ways. It claims to combine the "best features of a mirror and an incremental backup", and if you are forced to pick only one tool to do your backups, this is probably the best one.

The features that make rdiff-backup really stand out are:

1. Much saner logic for file inclusions and exclusions compared to rsync, while still using the same wonderful difference algorithm to reduce network traffic.
2. Keeps the latest version just like rsync does (like a mirror), but also keeps older versions as compressed (and reversed) rsync-style diffs within a new directory tree called rdiff-backup-data on the target disk. What rsync does only for network bandwidth, rdiff-backup extends to disk space also! Very cool...
3. Very easy to retrieve older versions.
4. Just as easy to purge really old versions that you no longer care about!

How is it used?

• An excellent and easy to follow introduction to this powerful tool is the "examples" page at http://www.nongnu.org/rdiff-backup/examples.html

• This is a tool that is just begging to be run from cron, so one way to take a backup is:
  1. create a list of files, say /root/rd-b.lst. Some illustrative parts of my list are given below:

             /var/spool/cron
             - /usr/local/share/vim
             /usr/local
             - /home/sitaram/.mozilla/firefox/44kffhfm.default/Cache
             /home/sitaram/.mozilla
             /home/sitaram/.jpilot
             /home/sitaram/.freemind
             /home/sitaram/.darrc
             /home/sitaram/bin
             /home/sitaram/.bashrc
             /etc
             - **
         

     IMPORTANT: Paths in the include/exclude list should start with either **/ or the source directory name. Using ./ won't work unless the source directory was given as ".".

     Each candidate file/dir in the source tree is tested against the list above. Checking stops as soon as a match is found. If the matched pattern is prefixed with a "-" sign, the file/dir and all its subdirectories/files are excluded. Otherwise, they are included.

     This is why exclusions at a deeper level are listed before any parent that must be included. Notice, for example, how the cache directory is excluded first, and then the mozilla directory is included. This is because I want to back up the whole of the mozilla directory except the cache.

     If a file does not match any pattern, the default is to include it. Since we don't (normally) want that, the last line lists "**" as an exclusion pattern. "**" is a special pattern that matches any character, including a "/", so think of it as an ultra-wildcard.

  2. run this command or put it in cron:

     rdiff-backup --include-globbing-filelist /root/rd-b.lst / /bigfs/rd-b.data

     If you want, you can use a user@host::path syntax (note the double colon) to put it on another machine.

• Restoring it is just as simple. You can use plain cp or scp to get the latest version (remember this is a mirror). Or you can do this:

  rdiff-backup -r now user@host::/bigfs/rd-b.data/home/sitaram/bin bin

  will restore the bin directory. As you can see, you can restore a sub-tree of the full backup set if you need to.

• You can get older versions too. Replace the now in the above command with something like 3M which means "3 months", or 5D for 5 days, or 6W for 6 weeks, or combinations of these, or specify an exact date in several possible ways. Clearly plain cp will not get you older versions. Read the manual for details.

• Finally, the one feature that makes it all worthwhile is the --remove-older-than option. I have an extra command in my cron job that cleans out versions older than 3 weeks:

  rdiff-backup --remove-older-than 3W /bigfs/rd-b.data

  This makes the whole thing pretty much zero-maintenance as long as you are directing the backups to some other machine with enough disk space.

When is it useful?

When you have enough space on another machine you trust, or, failing that, on a second hard disk on your own machine, this is the best way to backup your stuff. Pretty much fire-and-forget!!

If you have enough free disk, even backing up to the same hard disk may be worthwhile, because you're at least protected from human error or software bugs causing lost data.

It is so efficient in both time and space that you can probably take a backup as often as you like. Let cron do it once a day, but by all means take another one manually at the end of a long hard day's work!

You still need to do the following:

• check the /root/rd-b.lst file periodically to ensure that everything you want really is being backed up. This is not necessary if you have asked rdiff-backup to backup the entire machine, of course!

• if the "remote" machine you're backing up to is not that "remote", (example, same building, so a fire or a flood could wipe out both!), maybe you should find some other means to do an offsite backup :-) dar (next article!) with a DVD writer is a good solution for most people.

What are the downsides?

Nothing major really, but you need to remember that this is a mirror, not a backup in the normal sense. For instance, you can't encrypt a mirror, so you should only use machines you trust as the targets.

dar, the subject of the next article, allows encryption, offsite/offline backups, and other features besides, but then of course it's not a mirror, and it can't use the rsync algorithm so it's not as efficient! You can't have everything in one software, as I said in the first article!

What next?

Well, the next chapter: A true backup solution -- dar