Free Wisdom Online: Backup with rsync and EC2

Don't put anything here

Page Content

Page Name (for URL)

Page Title

Breadcrumb Text

This is going to be very nerdy, and probably not at all interesting for those
who haven't at least _heard_ of Amazon S3, Elastic Compute Cloud (EC2) and
rsync.  But in case you are curious, here is a brief summary.
In addition to selling books, Amazon.com has in the last few years been
a pioneer in the area of [cloud computing][cloud].  First they started
offering fairly cheap replicated storage "in the cloud" ([Amazon S3][s3]).
From very beginning S3 looked like potentially a great option for backing
up stuff - at least for the nerds. However, the interface to it was too
limited to be practical.  Then Amazon started offering computing power
in the cloud ([EC2][ec2]).  As I've been revisiting my backup strategy
recently, I started thinking about the possibility of running rsync on
EC2 and backing up to S3 through it.  [rsync][rsync] is a clever UNIX
program that makes it possible to very
efficiently synchronize files between too computers.  Basically, the two
computers talk about what each of them has, figure out what files changed
and only send those (or even parts of them).  This means that if you've got
gigabytes of stuff and change a few files here and there, you can let rsync
figure out what has changed and update this.  And now you can also use
rsync to back up stuff into Amazon's storage.  But at this point it's not yet
for the faint of heart.

So, the idea is to be able to do the following steps:

1. Create an machine instance with EC2
2. Attach a volume to it.
3. Mount the volume.
4. rsync some directories to the volume mounted on the remote machine
5. Unmount and detach the volume.
6. Terminate the machine.
7. Tell Amazon to create a snapshot into S3 for extra security.

This backup solution ends up costing $.10 per hour while rsync is running (which would be quite a few hours the first time, but quick after that) + $.10 per month 
per gigabyte of allocated space + $.15 per month per gigabyte
of the snapshot size.  I haven't gotten by first bill yet, but I expect it
to come out to about $3 a month for backing up around 10GB.  (If you only backup a gigabyte of data, you should be paying under $1 a month.)

While the setup process is a little tedious, once you are done, you should
be able to do backup with a script that would look like this:

### Setup EC Tools - replace with your paths and keys
    export EC2_HOME=~/apps/ec2tools  
    PATH=$EC2_HOME/bin:$PATH
    export EC2_PRIVATE_KEY=~/.ec2/pk-FLKSO8372HSJALPSQJHFKS92387DKADG.pem 
    export EC2_CERT=~/.ec2/cert-FLKKLK0SL2GEWE8SJANJSIEUSJDKQS83.pem
    export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre/

### Attach and mount the volume
    export EC2_VOLUME=vol-1134d178
    ec2-attach-volume $EC2_VOLUME -i $EC2_INSTANCE -d /dev/sdh
    ssh -i id_rsa-gsg-keypair root@$EC2_HOST \
        "mkdir /mnt/data-store && mount /dev/sdh /mnt/data-store"

### Run rsync
    rsync -e "ssh -i id_rsa-gsg-keypair" -avz /some/local/directory/ \
        root@$EC2_HOST:/mnt/data-store/directory/ > out.txt

### Clean up
    ssh -i id_rsa-gsg-keypair root@$EC2_HOST "umount /mnt/data-store"
    ec2-detach-volume $EC2_VOLUME -i $EC2_INSTANCE
    ec2-terminate-instances $EC2_INSTANCE

Those instructions are provided without any warranty, use at your own risk.

## Setup Step 1: Sign Up for the Services

Before you can use Amazon Web Services, you need to sign up for an account at
<http://aws.amazon.com/>.

This will give you two keys that you will need for accessing the services:
your "access key" and your "secrete access key". Then sign up for two
services: [Elastic Compute Clode][ec2] (EC2) and [S3][s3].  As part of
signing up for EC2, you will create an X.509 certificate.  You will need to
save your private and public keys in ~/.ec2.  All this is described in more
detail in AWS "Getting Started Guide" under [Setting up an account][account].

## Setup Step 2: Install EC2 Tools

The process of setting up the tools is described under
[Setting up the Tools][tools] in AWS "Getting Started Guide".  Just do what
that page says.

## Setup Step 3: Create an EBS Volume

[Elastic Block Storage][ebs] is a storage system that can be mounted on EC2 servers.
Unlike [S3][s3], EBS storage is not replicated and is somewhat less reliable,
though presumably still better than what most hosting companies would offer
you (or than what you would expect from an average hard drive).
It's also a little cheaper than S3 ($0.10 per GB per month rather than
$0.15).  On the other hand, you need to decide upfront how big it's going to
be and you get charged for what you allocate.  While in theory you can mount
S3 buckets as directories, having tried a few tools, I found all fo them
fragile.  So, my recommendation is to use EBS for mountable storage and
create S3 snapshots for additional reliability.  (See below.)

To create a 1GB storage block:

ec2-create-volume -s 1 -z us-east-1a

This will give you back a volume ID, something like "vol-5f34d136".  Save it
in an environment variable:

export EC2_VOLUME=vol-5f34d136

(If you later need this id again, you can look it up with
ec2-describe-volumes.)

For a larger block (and you will probably need more than a gigabyte
eventually), pass a different size (in GB) with the -s parameter.
You can make volumes of up to 1 terabyte.  Amazon will charge you $0.10 per
GB per month, so a 10GB volume will be $1 per month.

The second parameter ("-z us-east-1a") is the "availability zone" where the
storage would be created.  We'll have to use the same one later when creating
the machine instance.

Once you create the storage you will be charged $0.10 per gigabyte per month
for it until you destroy it (with "ec2-delete-volume").

## Setup Step 4: Create an Instance and Format the Volume

This only needs to be done once: we'll create a machine instance so that we
could format the volume we just created.

First of all, you'll need to set up a key to log into an EC2 machine.  Follow
the section called "Generating an SSH Keypair" in 
[Running an Instance][running].  "Getting Started Guide" tells you how to
create an instance of the default machine.  You can do what they tell you or
create an instance of Ubuntu by using this AMI instead: `ami-179e7a7e`.  In
either case, you will specify the same availability zone as your volume (for
instance with "-z us-east-1a").

Instead of doing all this manually as explained in the Guide, you can use export, tr and grep to
save the instance ID and the machine's host name into environment variables.

First the instance ID:

export EC2_INSTANCE=`$EC2_HOME/bin/ec2-run-instances ami-179e7a7e \
        -k gsg-keypair -z us-east-1a | tr '\t' '\n' | grep '^i-'`
    echo $EC2_INSTANCE

Then the host name, after waiting 20 seconds or so:
 
    export EC2_HOST=`$EC2_HOME/bin/ec2-describe-instances | grep $EC2_INSTANCE \
        | tr '\t' '\n' | grep amazonaws.com`
    echo $EC2_HOST

If the last commands didn't give you anything, wait a little more and try
the last two commands again.

Now attach the volume:

ec2-attach-volume $EC2_VOLUME -i $EC2_INSTANCE -d /dev/sdh

Then use SSH to format the drive:

ssh -i id_rsa-gsg-keypair root@$EC2_HOST "mkfs -t ext3 /dev/sdh"

(You shouldn't be asked for a password at this point, if you are, go back to
"Generating an SSH Keypair" in [Running an Instance][running].)

We could now continue using the same instance to do backup, but to make
things simpler, let's terminate it now and make a new one in the next section:

ec2-detach-volume $EC2_VOLUME -i $EC2_INSTANCE
    ec2-describe-volumes | grep ATTACHMENT # This should return nothing.
    ec2-terminate-instances $EC2_INSTANCE

## Actual Backup

To do the actual backup, you'll need to use the following commands.
(Those commands will need to be used every time you want to do backup.)

First, create an instance of Ubuntu and attach the volume to it.  Nothing new
here, we already did this during setup.

export EC2_INSTANCE=`ec2-run-instances ami-179e7a7e -k gsg-keypair \
        -z us-east-1a | tr '\t' '\n' | grep '^i-'`
    echo $EC2_INSTANCE
    export EC2_HOST=`ec2-describe-instances | grep $EC2_INSTANCE \
        | tr '\t' '\n' | grep amazonaws.com`
    echo $EC2_HOST
    ec2-attach-volume $EC2_VOLUME -i $EC2_INSTANCE -d /dev/sdh

Now mount the volume as "/mnt/data-store" (you can call it anything you want,
though):

ssh -i id_rsa-gsg-keypair root@$EC2_HOST \
        "mkdir /mnt/data-store && mount /dev/sdh /mnt/data-store"

Now you can do the actual backup:

rsync -e "ssh -i id_rsa-gsg-keypair" -avz /some/local/directory/ \
        root@$EC2_HOST:/mnt/data-store/directory/ > out.txt

Try a small directory first.  Note that at this point rsync will be copying
all the files, since the remote server doesn't have any of them.

Then unmount the storage:

ssh -i id_rsa-gsg-keypair root@$EC2_HOST "umount /mnt/data-store"
   
Note that you _have_ to unmount the volume to avoid corrupting it.  So, check
that it actually got unmounted properly:

ssh -i id_rsa-gsg-keypair root@$EC2_HOST "ls /mnt/data-store/

This should return nothing.

Finally, detach the volume and terminate the instance:

ec2-detach-volume $EC2_VOLUME -i $EC2_INSTANCE
    ec2-describe-volumes | grep ATTACHMENT # should return nothing
    ec2-terminate-instances $EC2_INSTANCE

Check that you don't have any instances running:

ec2-describe-instances | grep INSTANCE | grep -v terminated

At this point, your stuff should be backed up on the volume, for which you will be
billed $0.10 per gigabyte per _month_.  However, your machine instance (which costs
you $0.10 per _hour_) should be off, and thus not costing you anything.  On
a good day, when rsync takes less than an hour to run and doesn't transmit too
much data, the whole backup procedure will cost you just a little over a dime.

## Creating a Snapshot

Amazon documentation says that EBS storage should be more reliable than your
typical hard drive, but it's not as reliable as S3.  So, for extra security,
you would want to take a snapshot of your volume and save it in S3:

ec2-create-snapshot vol-5f34d136

This will schedule the snapshot, which you can later use to re-create the
volume -- see [Elastic Block Store Feature Guide][ebs].

[cloud]: http://en.wikipedia.org/wiki/Cloud_computing
[ec2]: http://aws.amazon.com/ec2
[s3]: http://aws.amazon.com/s3 
[account]: http://docs.amazonwebservices.com/AWSEC2/2008-05-05/GettingStartedGuide/account.html
[tools]: http://docs.amazonwebservices.com/AWSEC2/2008-05-05/GettingStartedGuide/setting-up-your-tools.html
[running]: http://docs.amazonwebservices.com/AWSEC2/2008-05-05/GettingStartedGuide/running-an-instance.html
[ebs]: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1667&ref=featured
[rsync]: http://en.wikipedia.org/wiki/Rsync

Advanced Fields

HTML Fields

Main

Head

Body

Header

Logo

Content Template

Page

Sidebar

Footer

Tags Allowed for XSSFilter

HTTP Fields

Cache-Control

Expires

Guru Fields

Templates

Translations

Fields

Edit UI

Admin Edit UI

Don't put anything here

A summary of your changes

Don't put anything here

Edit Summary

Don't put anything here