Tips for backup, mirroring, and synchronisation

^{This document has the following sections:

    Tips for backup, mirroring, and synchronisation

        Temporary backups

                    Features and uses

                    Downsides

        Plain mirror, local or remote

                    Features and uses

                    Downsides

        better than rsync

                    Features and uses

                    Downsides

        real backups

                    Features and uses

                    Downsides

                    Additional notes

        Synchronisation -- when files can change on both sides

                    Features and uses

                    Sample profile 1

                    Sample profile 2

Temporary backups

tar uvf rescue.tar file1 file2

tar xvf rescue.tar file-you-want

tar tvf rescue.tar file-you-want | nl

tar xvf rescue.tar --occurrence=7 file-you-want

Note: the timestamps reflect when the file was last changed, not when
the file was backed up.

Features and uses

cleaner than cluttering with ".orig", "backup-", etc.

take a backup you think you will not really need but want to
be safe anyway

Downsides

grows indefinitely; periodic cleanup required

must be local and uncompressed.

NOT for long term backups; if you need to restore, it'd better be soon!

Plain mirror, local or remote

rsync --progress -avz sources user@remote:

rsync --progress -avz user@remote:sources .

works almost like cp or scp

Features and uses

plain mirror; no more and no less!

low bandwidth usage

lots and lots of other options

Downsides

too many options :-)

BIG WARNING: stupid design error that breaks the "think like cp"
idea. If you put a trailing slash on any path name (local or remote), it
takes it as path/*. In the following commands, think what cp
would do if you appended a * wherever there is a trailing /.

(wrong) rsync -avz sources/ user@remote:

(wrong) rsync -avzP user@remote:sources sources

(right) rsync -avzP user@remote:sources/ sources

(right) rsync -avzP user@remote:sources .

better than rsync

http://www.nongnu.org/rdiff-backup/examples.html

rdiff-backup --include-globbing-filelist incl.lst / /target-dir

Target can be user@host::path (note the double colon).

rdiff-backup -r now user@host::/bigfs/rd-b.data/home/sitaram/bin bin

now in the above command can be something like 3M or 5D or
date/time etc.

rdiff-backup --remove-older-than 3M /bigfs/rd-b.data

A sample incl.lst looks like:

/var/spool/cron
- /usr/local/share/vim
/usr/local
- /home/sitaram/.mozilla/firefox/44kffhfm.default/Cache
/home/sitaram/.mozilla
- **

IMPORTANT: Paths in the include/exclude list should start with either **/
or the source directory name. Using ./ won't work unless the
source directory was given as ".".

Files are matched against the patterns in the order given; the first match
stops further checking. Hence the order of the cache line and the
mozilla line, for example.

Also, the default is to "include", so we have an ultra-wildcard exclude at the
end.

Features and uses

much saner include/exclude logic than rsync

uses rsync algo for bandwidth AND for storing old versions!

easy retrieval/purging of older versions

fire-and-forget if you have enough space and trust the other machine

backup as often as you like; hardly any overhead even with multiple
changes

Downsides

slightly more space than original

no compression on "current" version, no encryption at all

real backups

dar -h gives you a pretty good summary of the options.

dar -c /bigfs/dar-data -R sources -g dir-to-save -P dir-to-save/subdir-not-to-save

-R defaults to "."

-g and -P must be relative to -R.

Order doesn't matter; anything matching a -P pattern is excluded. If
there are no -g patterns, the entire -R is considered, otherwise,
only files matching the -g patterns are considered.

dar -l dar-archive-base-name

you have to remove the .1.dar at the end!

-as lists only files actually saved in a differential backup

dar -x dar-archive-base-name

-k and -r are useful if you want to play safe; dar -h for
details

For both -l and -x, -I gets you just the files you want, without
worrying about the full path.

Features and uses

can be compressed (-y), encrypted (-K:), and offline (everything a
mirror isn't!)

differential backups (-A old_dar_file)

can back up in slices (-s), pausing (-p) to let you burn each
slice to a CD. Make backups with very little free space!

can extract (-C) a "catalog" of offline archives and manage the
catalogs

good for long term backups on removable media or on untrusted computers

files anywhere in dar archives are directly accessible, unlike tar. This
has implications for security and space. But you can use a dar archive
like "near-line" storage, e.g., for /usr/share/doc

Downsides

Not good for frequent backups of frequently changing files; wastes too
much space. Mitigate it a little by using dar_manager with -s.

Additional notes

The kdar GUI supports some of dar's options, not all. Not very useful
except to show you a graphical tree of an archive file and let you
interactively diff/test/restore one/some files.

Here's my ~/.darrc

create:
# I care more about preserving ctime than atime
-aa
# bzip2 compress
-y
# I dont care about extended attributes in user...
-u
# ...and root namespaces
-U
# minimum 512 bytes in order to compress a file
-m 256

# ignore case for following flags
-an
# do not compress following extensions, case insensitive due to above
-Z "*.avi"
-Z "*.bz2"
# -Z "*.gif"
-Z "*.gz"
-Z "*.jar"
-Z "*.jpeg"
-Z "*.jpg"
-Z "*.mov"
-Z "*.mp3"
-Z "*.mpg"
# -Z "*.png"
-Z "*.tgz"
-Z "*.zip"

# set case sensitivity back to default
-acase

extract:
# use -O only for ordinary users' ~/.darrc, not for root
-O
-u
-U

-B lets you specify any options, but is best used to specify an
include/exclude list (example below):

-g /var/www
-g /var/spool/cron
-g /var/named
--prune /var/mailman/archives
-g /var/mailman

dar always needs the first and the last slice to restore a file.

be sure to extract catalogs and save locally, and use dar_manager
regularly

dar_manager -C my_dmd (once only)

dar_manager -B my_dmd -A 2006-07-01-full_catalog (for each archive)

dar_manager -B my_dmd -r files-to-restore

try making the slices half the size of a CD and burning both on the same
CS :-)

Synchronisation -- when files can change on both sides

Unison has a nice GUI. Use it.

Features and uses

you can synch A-B, then B-C, and back, to make A-C in sync

ad hoc, very efficient, interactive way to compare directories

uses the rsync algorithm for network transfers

you can see a diff between two files

you can interactively add to its preferences

you can also ignore files with certain names, regardless of where the
name occurs. Example:

ignore = Name {CVS,*.cmo}

Sample profile 1

label = sync the a780 flash disk
include default
perms = 0o0400
pretendwin = true
root = /home/sitaram/a780-backup/
root = /mnt/removable

My default.prf is:

# Unison preferences file
height = 40
diff = gvim -d

Sample profile 2

Note that the order of ignore (exclude) and ignorenot (include) does not
matter.

label = RC files and other settings

include default

root = /home/sitaram
root = ssh://sitaram@remote.host

# "ignore = Path something/*" works, but "ignore = Path *" doesn't
# (this is documented!)
ignore = Path ?*
ignore = Path .*

ignorenot = Path .*bashrc
ignorenot = Path .darrc
ignorenot = Path .vimrc
ignorenot = Path .sunrise
ignore = Path .sunrise/cache}