Backup on Linux: rsnapshot vs. rdiff-backup (vs. Time Machine)

This is my OLD blog. I've copied this post over to my NEW blog at:

http://www.saltycrane.com/blog/2008/02/backup-on-linux-rsnapshot-vs-rdiff/

You should be redirected in 2 seconds.

Apple's release of Leopard and the included backup utility, Time Machine, has generated a lot of talk about backups recently. I will admit Time Machine is pretty cool and believe that it is a bit more than a glorified GUI on top of an existing *nix tool as some have claimed. However, the core functionality is very similar to the command-line tool, rsnapshot, which is itself based on a rsync script by Mike Rubel. Time Machine added a couple of features and a GUI to make it easy to use. Since I prefer the command line over GUIs most of the time anyways, rsnapshot seemed perfect for me.

To be complete, I researched a number of other backup utilities for Linux. Dirvish and flyback were out because I prefer the command line and they didn't seem to offer anything more than rsnapshot. Scripting rsync wouln't get me anything more than rsnapshot either, plus it would be more work. In the end, I eliminated all but rsnapshot and another command line tool called rdiff-backup. Rdiff-backup has some advantages over rsnapshot (and Time Machine) because it stores compressed deltas of each version of a file instead of a complete copy of the file each time it changes. This is not a big deal for small files, but for large files that change often, it makes a significant difference. However, the big disadvantage of rdiff-backup, for me, was the inablility to perform different levels of backup, such as hourly, daily, weekly, monthly, etc. Depending on the needs of the user, this could negate the space saving advantage by needing to keep a large number of snapshots.

I ended up choosing rsnapshot over rdiff-backup because of this last reason. It seems rdiff-backup is closer to a version control tool and rsnapshot closer to a traditional backup solution. It would be great to create a hybrid of the two tools to gain the advantages of each. I started to dig into the source of rdiff-backup (python source makes me happy), but I didn't want to get too sidetracked with another project. For now, I am using Mercurial to version control my /etc and /home/.* config files and rsnapshot as a broader, general purpose backup tool.

Here is my comparison of rsnapshot and rdiff-backup:

Similar Features	both use an rsync-like algorithm to transfer data (rsnapshot actually uses rsync; rdiff-backup uses the python librsync library) both can be used over ssh (though rsnapshot cannot push over ssh without some extra scripting) both use a simple copy of the source for the current backup
Written in	rsnapshot is written in Perl; rdiff-backup is written in Python and C
Size	rdiff-backup stores previous versions as compressed deltas to the current version similar to a version control system. rsnapshot uses actual files and hardlinks to save space. For small files, storage size is similar. For large files that change often, such as logfiles, databases, etc., rdiff-backup requires significantly less space for a given number of versions.
Speed	rdiff-backup is slower than rsnapshot
Metadata	rdiff-backup stores file metadata, such as ownership, permissions, and dates, separately.
Transparency	For rsnapshot, all versions of the backup are accessible as plain files. For rdiff-backup, only the current backup is accessible as plain files. Previous versions are stored as rdiff deltas.
Backup levels	rsnapshot supports multiple levels of backup such as monthly, weekly, and daily. rdiff-backup can only delete snapshots earlier than a given date; it cannot delete snapshots in between two dates.
Community	Based on the number of responses to my post on the mailing lists (rsnapshot: 6, rdiff-backup: 0), rsnapshot has a more active community.

[Read the full post...]

backing up with rsync

This is my OLD blog. I've copied this post over to my NEW blog at:

http://www.saltycrane.com/blog/2007/08/backing-up-with-rsync/

You should be redirected in 2 seconds.

Here is a python script using rsync to backup my Users directory in Vista to an external hard drive.

import os

cmd = "rsync -avz --exclude '/AppData/'" + \
      "/cygdrive/c/Users/sofeng" + \
      "/cygdrive/f/backup/Users"
os.system(cmd)

Why not to use the backup program that comes with your external hard drive:

Do not, whatever you do, feed your valuable data to a program that is going to save it in a file format that can only be read by that program, or by that kind of computer. Because when the program can’t or the computer can’t, you’re out of options. --Tim Bray, Protecting Your Data

Why to use rsync: rsync -essh -rtpvz rocks. --Mark Pilgrim, Essentials


[Read the full post...]