Server Backups

The servers merlin and gandalf use disk-based backups where the backups from each are sent to a USB disk on gandalf. The basic backup is handled by rdiff-backup, and the backup itself is managed by backupninja.

What is Backed Up

The configs in /etc/backup.d/ give the details, but basically everything except /proc, /sys, /dev, /tmp, and the backup directories is backed up. For things like trac and subversion which use a database, a consistent snapshot is first made into /var/backups/ - the original directories are then excluded from the backup.

Where it is Backed Up to

merlin's backup is on gandalf in /backup/merlin/; gandalf's backup is on gandalf too in /backup/gandalf/. /backup is mounted on an external USB disk.

When it is Backed Up

Daily at 3 AM and 4 AM on gandalf and merlin, respectively. See /etc/backupninja.conf for details.

Remote Access Setup

In order for merlin to be able to store the backups on gandalf, gandalf has an account named backup which merlin uses to log in. The account has no password and only allows ssh access via public key. Login is further restricted to the ip-address of merlin and to running the rdiff-backup command on the backup directory.

The access details can be found in ~backup/.ssh/authorized_keys and in ~root/.ssh/config (the backups are run as root).

Packages

Fedora Core 6 provides an older version of rdiff-backup (1.0.5) and no backupninja package. Hence both were built ourselves (on merlin - see /usr/src/redhat/... for the SRPM's and RPM's). The rdiff-backup 1.1.7 rpm is built from the latest tarball and a suitably modified spec file from the fc6 1.0.5 rpm. The backupninja rpm was created based on an rpm spec found on pbone.net and on the spec in the latest svn of backupninja. In addition, because of fedora-core related bugs in 0.9.4 a set of patches has been applied.

Restoring a Backup

Since rdiff-backup produces a layout that exactly mirrors what is backed up, the latest version of everything is always directly accessible on gandalf in /backup/<machine>/. However, because the backup from merlin is run as user backup, the file ownerships are not correct for /backup/merlin/.... But these can be restored using rdiff-backup in restore mode. For details see 'man rdiff-backup', but here are two example commands to restore a backup of /etc/sysconfig from 5 days ago on merlin and on gandalf, respectively:

    rdiff-backup -r 5D backup@gandalf-backup::/backup/merlin/etc/sysconfig /tmp/sysconfig
    rdiff-backup -r 5D /backup/gandalf/etc/sysconfig /tmp/sysconfig

Note also that the backups for trac and svn are in /var/backups/, not in their original /home/... because we use take a hot snapshot of the live stuff.

Other Notes

rdiff-backup basically only does incremental backups. However, since the previous backups are stored as reverse-diffs, and since it uses the rsync lib for the diffs, full backups are not really needed.

Old revisions are purged after 12 months.