Utility Computing Meets Personal Backups

Short Version

AWS is designed for applications which could benefit from scale-on-demand scheme. If your application doesn’t need this, then it might not make sense economically.

Long Version


The Joys and Sorrows Of Owning Shiny Metal Boxes

One of the SATA drives on my home file server died a horrible death this week. The server had been acting funny all week, giving me the occasional strange I/O error. Finally, the system would not boot , instead showing the familiar “Insert CD:” error that shows on the screen when the boot drive fails. Attempting to auto-recognize the first SATA drive from the BIOS made the computer think for awhile, and then come back without finding anything. Unfortunately, this box was also running slimserver which provides access to my music collection from my Squeezebox. No streaming music for the BBQ I’m hosting this weekend (we’ll have to switch to the redundant iPod + boombox).

I originally built the computer as a tiny fileserver-on-the-cheap for things like my videos, music collection, and family photo album. At the time, 2 years ago, I had been looking at SOHO NAS systems for a while, but just couldn’t get over how much they cost. Also, for the price I thought I deserved unfettered access to the underlying server to put whatever I wanted on it. Whenever I see some new device with Linux on it, the hacker in me wants access to bend it to my will. For the server, I used the tiniest enclosure I could find that fit microATX, which at the time was the Antec 1380, a small cube about the size of 2 shoe boxes. One slight design flaw was putting a powerful (but inexpensive) CPU in the box: an AMD dual-core 3800. Yes, I succumbed to market forces and, in the popular venacular, “biggie sized it.” While I got the CPU at a good price, it ran a bit hot. In retrospect I wonder if this extra heat caused my drive to fail.

If I had to buy my own equipment today, I’d probably go with the Thecus 5200 (dedicated SOHO NAS). Let’s face it, owning tiny, shiny boxes packed with technology is fun. But it’s also expensive, and eventually, everything breaks.

Cloud Computing To The Rescue?

A natural response to owning things that break is to not own them anymore, which brings us to Utility or Cloud Computing. I had preferred the earlier term “Utility Computing,” as I thought this moniker more accurate, although Cloud Computing does sound cooler. Eventually the coolest sounding name wins mindshare and we are forced to use it.

Also, a problem with the term Utility Computing is that the the analogy of “computing power as electricity” is limited. Utility Computing is less like electricity, more like lego pieces. Utility, er rather Cloud Computing already has a variety of different payment models, and service types, each with different features which are important for solving different types of problems. One important feature to emerge is the ability to scale-on-demand, which is being pioneered by companies like Amazon, with their Web Services (AWS).

Scale-on-demand (my own term) is a really cool feature, which is part of the basic strategy for Amazon Web Services. The basic argument is if one must own/rent equipment to cover the maximal use, then the equipment will not be fully utilized. Consider a company whose normal utilization is 1/10th of periodic spikes in activity, say around noon. In own/rent models, the company would have to have 10 times the equipment just to handle the spikes, even if the spikes only lasted for 1 hour. Calculations show the average utilization would be far lower => 9 + 24 computer hours => 33/24 = 1.375. So we need 1.375 computers instead of 10, if we could spread our load out consistently over time.

This shows that there is room for a middleman who aggregates computing resources to make a profit. If the middleman can effectively spread out utilization spikes so that the utilization pattern for a large number of companies approaches the average utilization, then machines can approach 100% utilization. This is exactly the approach that Amazon is taking with its various services which make up AWS.

AWS has many individual component services which each have different applications. Those of you who work in the Internet space, or distributed computing will recognize many of the pieces of scalable Internet storage – entity storage, mySQL database instances, key value persisted storage, virtual instances (ala Xen), durable queues. There are already many startup companies who have built their entire infrastructure using AWS.

Are Personal Backups Using S3 Cost Effective?

I’ve seen a lot of blog posts about doing personal backups to S3, which one must admit sounds extremely cool at first glance. S3 is Amazon’s Simple Storage Service, which allows one to write, read and delete objects of up to 5GB. The objects are retrieved from buckets you define via the bucket id and object name.

The best way to do incremental backups is to look at the state of what is backed up, and then produce a delta of what has changed. This way, we’d only send the portions of the files that had changed. This delta system works extremely well for many types of files. And it turns out there is a well-known and dependable tool which does this well: rsync. Rsync remotely calculates a delta, and sends only the minimal set of file updates via ssh. There are also some clever minimal backup packages that use Unix hard links to provide “minimal space snapshots.” This technique is so cool and simple, but it does rely on hard links, which are not available to every OS (but all *nix have them). If hard links aren’t available in a particular OS/filesystem, rsync can be trivially used to maintain a single up-to-date snapshot (a cron job invoking rsync is all it takes).

Note that rsync should be installed on any Linux host from a web hosting company, so if you have ssh access, you can use this backup technique, with almost no setup required. Here are some links:

1. Rsync – file copy utility which computes and transmits minimal deltas to update.
2. Rsnapshot – a backup program based on rsync and written in perl which implements the minimal space snapshot idea with hard links.

Next, lets look at the cost analysis. Remember that the Amazon Web Services can help us save money, but are designed from the standpoint of providing scale-on-demand ability versus purchasing for peak utilization. We are essentially “buying insurance” against usage spikes. But our simple backup application is completely predictable, and will never have any usage spikes. This is for our home backups, not to provide scalable backups for all the consumers out there on the Web. The backups are easy to schedule so that there are never any spikes, and thus we can completely predict our usage. Not only that, but backups are not latency sensitive – I just want to make sure they happen every night. I don’t really care if they take an extra 10 minutes.

S3 costs $0.15 per GB per month for storage. For 500 GB, this would be $75 a month. Web hosting services provide a variety of monthly plans from about $11, and give between 500 and 2500 GB. In fact, there are services which offer “unlimited storage.” Unlimited storage is simply another play on the averages. If you have enough customers, you can give people unlimited storage, because your average storage will be extremely low. It’s like an all-you-can-eat buffet: the average person is paying for the people loading their plates.

Thus, purely from a cost perspective, storage for backups from a web hosting company should be 1/8th – 1/34th the cost of Amazon S3.

One final issue to look at is availability. Amazon has an actual SLA for S3 which guarantees 99.9% uptime, or you will get reduced rate for the period of the outage. This is actually pretty cool as they will return from 10%-25% of your money for that month. Keep in mind this is good, as the incentives will push Amazon to get better and better, but it’s no guarantee of stability. Once they have a “really bad month,” the incentive to keep the month good goes away. After all, under the Amazon S3 Service Level Agreement they could be down the entire month (0% uptime), but still charge you 75% for storage fees. In extremis this sounds ridiculous, but this is a problem in the structure of most SLA’s, not just Amazon’s.

In the month of July, Amazon S3 had an 8 hour outage due to a data corruption problem. Amazon CTO Werner Vogels mentions in his blog the root cause was single bit corruption of internal state messages that are distributed via Gossip techniques. It’s good to keep in mind that the core technologies for AWS are new and so have a few minor kinks to work out. While Amazon does not have to rely on software upgrades to existing services for revenue, they still have to provide missing features for their customers. So I would expect errors like this to abate over time, but not completely vanish. In a distributed system, any error, regardless how small, becomes serious through amplification.

This puts them at 98.88% for the month of July, just for this event. To their credit, the response was very professional, swift and public. CEO Jeff Bezos even talked about it. To put this in perspective, service companies usually apologize in private, and try to make it up when the monthly SLA reports are given to customers. In my mind this is one of the things, besides the technology, which puts Amazon way out in front in the race to define this industry. I especially like the Amazon AWS dashboard which shows their history of outages.

The web hosting companies have no SLAs for most self-serve customers (although upscale web hosting, which costs more, may have SLAs for businesses). Uptime is measured by third party companies. It’s not clear how the measurements are taken or what the relationships are from the hosting companies. I have to believe the hosting companies are providing advertising money to these uptime companies, which doesn’t bode well for the objectivity of the measuring company. Still, it is not hard to find companies with uptime of 99.5%. This would equate to 2 hours of downtime a month, which for a backup solution, seems completely acceptable, especially at 1/8th the price. This does not take into account data transfer, or computational costs, which would be free for the web hosting company, but translate into even higher S3 costs.

Also note you can have a secondary web hosting service, or even a different server with the same service, to get a much better availability through trivial redundancy. Note that two hosting servers would give availability of 99.9975%, which is 1 minute of downtime a month, and far better than Amazon, at 1/4th the cost.

Note that either of these solutions is superior to owning your own hardware, as Jeremy Zawodny calculates. Also, if you are interested in software which backs up to S3, Jeremy has done the legwork for you.

Amazon S3 (and AWS) really shines as a model for scale-on-demand. This makes a lot of sense for things like a website with a video that becomes popular overnight, and needs to scale up to meet a rush of users. But the scale-on-demand terms approach loses the pricing advantage when the max utilization is close to the average utilization. For long term storage this tends to be true, as it grows very slowly. I’m thinking S3 is designed more for scaling I/O reads and writes, which are costly features our backup solution won’t use.

Don’t get me wrong, I think S3 and AWS in general are the coolest thing since sliced bread, but they don’t make sense for every project (especially where scalability isn’t important). We’ll come back to some cool uses of AWS services in future articles where the model does make sense.

Giving Dreamhost A Try

I’d heard good things about Dreamhost, so I’m giving them my business. Setting up backups was trivial, rsync was already installed so it worked on my first rsync command attempt. We’ll see how the availability is over the coming months.

In Today’s Post

I provide a simple template I use to start every command-line utility I write in Perl. We’ll also make command-line option parsing easy using Getopt::Long, as well as make encapsulated documentation using POD.

Perl, the Swiss Army Chainsaw

Perl is an excellent language for system scripting. There are those who criticize Perl for being “syntactically excessive.” However, as languages evolve expressive power they often become more complicated. Remaining backward compatible requires carrying around a certain amount of syntactic baggage.

I’m pragmatic in that I’d choose a language that was powerful with a few eccentricities over one that was perfect and less useful, or one that had stopped evolving. Languages are a living thing – they need to adapt to changes in the environment or they die.

Perl will always be alive and well because it’s just too damn useful. Larry Wall, the creator of Perl, has described his creation variously as “duct tape for the Web” and a “Swiss Army chainsaw”. Perl’s hallmark is flexibility, which is just what’s required to glue things together that weren’t necessarily designed to be glued together.

In a future post I’ll show you a style of Perl programming that works well for me and provides just the right amount of OOP without going overboard. This is actually simpler to do than you might think.

Unix Command Line Utilities Are Reusable, System-Level Building Blocks

1. One of the reasons Unix is so successful is the philosophy of “doing one thing really well.” Command-line utilities are designed to address a single problem well.
2. A second reason Unix is so successful is the “building block approach” – simple, well designed components can be connected together in various ways to solve a larger problem.
3. Unix command-line utilities should be thought of as reusable, system-level building blocks.
4. Write a command-line utility rather than a script. This is a conceptual task, which requires a few simple steps. One can often alter a script to turn it into a command-line utility.

Note, when I say Unix here, I mean “Unix and all it’s related descendants”, such as Linux (I’m old school, deal). I typically work with Linux (RHEL4, FC8) and FreeBSD (4 and 6).

I differentiate between scripts and command-line utilities. I think of a script as being designed for a single purpose, whereas a command-line utility was (should be) a building block. This is largely conceptual, and I realize it’s not a hard and fast rule, but it tends to be true.

I can’t count the number of times I’ve had to rewrite something someone wrote, just because they didn’t take the time to think of things from the perspective of reusability. The template below solves a few problems in this area.

Parsing Options

Command-line utilities often have a plethora of options that fine tune their behavior. For example the “ls” command has 36 “single character” options on OS X. The reason for having so many options is related to the design principles I’ve already mentioned: do one thing really well and make it so no one has to ever do it again (i.e. make it a building block). “ls” is designed to be used interactively, as well as produce output that is consumed by other programs, and so modifying it’s output via options is important (less so today since languages like perl have their own internal methods of listing files, like globbing).

One of the annoying parts of writing a command-line utility is parsing options. Options may have arguments, be different types, have valid and invalid ranges, and they need error checking. It’s also useful to be able to print a list of them from the program (using -h, or –help).

The perl library Getopt::Long solves the problems I’ve just described in an elegant way. In addition it’s got lots of extra bells and whistles for doing things like “repeated arguments”. The template below uses Getopt::Long and demonstrates it’s power.

What does this code do?

The code below is a “starting template”. I have a directory called “templates” which are simple program starting points for different purposes. When I want to write a command-line utility, I copy the template and make changes.

1. Command-line option parsing through Getopt::Long.
2. Internal documentation which can be programatically displayed, or formatted by other programs. This is done with Perl’s POD format (Plain Old Documentation).
3. Basic error handling.
4. As a template, its purpose is to jog my memory so I don’t have to try to remember how to use Getopt::Long, or the correct format for POD. Thus the code as-is doesn’t do too much (we’ll see how to use it below).

Starting Template: Unix Command Line Utility

#!/usr/bin/perl

# Use this "starter template" as a starting point for your
# own cmdline utilities.  Shows how to use Getopt::Long for
# option processing and POD for documentation.
#
# http://rogerbush.wordpress.com/2008/02/19/
# command-line-programs-and-perl-getoptlong/

use strict;
use warnings;

use Getopt::Long;
use Pod::Usage;

sub main
{
    my %options;

    GetOptions
        (\%options,
         'help|?',
         'length=i',
         'filename=s',
         'addr=s@',
         'man') or pod2usage (2);

    pod2usage (1) if $options{help};
    pod2usage (-exitstatus => 0, -verbose => 2)
        if $options{man};

    if ($options{length})
    {
        pod2usage ("--length must be > 0.")
            unless $options{length} > 0;

        print "length is $options{length}\n";
    }

    print "filename is $options{filename}\n"
        if $options{filename};

    if ($options{addr})
    {
        foreach my $a (@{${options{addr}}})
        {
            print "addr: $a\n";
        }
    }
}

main ();
__END__

=head1 NAME

perlcmdline - A do nothing, starter template for implementing
command-line utilities.

=head1 SYNOPSIS

perlcmdline [options]

"perlcmdline --help" will list options.  "perlcmdline --man"
will show docs.

=head1 OPTIONS

=over 4

=item B<--help>

Print a brief help message and exits.

=item B<--man>

Prints the manual page and exits.

=item B<--length positive-integer>

A "do nothing" length value which must be an int > 0.

=item B<--filename input-filename>

A "do nothing" string value.

=item B<--addr hostname-or-ip>

An internet address.  This option may be used multiple
times in a single command to specify multiple addresses.

=back

=head1 AUTHOR

Roger Bush, rogerbush8 at yahoo dot com

=head1 COPYRIGHT

Copyright (c) 2007 Roger Bush. All rights reserved. This
program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=cut

1. Getopt takes many forms.  In the form I'm using, I pass
   in a hash to hold all of the variables rather than assigning
   values to loose variables.
2. Getopt allows other types of inputs, such as hashes,
   flags, and several other types.  For a full list see the
   reference at the end of this post.
3. If a value is not specified, it will not be defined in the
   options hash (e.g. if ( ! defined $options{foo})).
4. The expression @{$options{addr}} evaluates to an array
   (it dereferences the array ref at $options{addr}.  Similarly
   %{hash_ref} would dereference a hash.
5. __END__ tells perl where the code has ended.  This always
   precedes the docs at the end which are called "POD" (Plain
   Old Documentation).  Perl's POD has special directives which
   start a new line with a "=name-of-directive".  See ref on
   POD at end of blog for more.
6. =over, and =back are POD indenting commands.  over
   indents the given number of spaces, and back goes back
   to the indenting setup prior to the over statement.
7. pod2usage prints out the SYNOPSIS and potentially
   the options in the POD section, when given a single
   numeric argument (1 is Usage, 2 is Options).  It can also
   be provided a hash of options affecting its behavior.

Time to give the program a try. Let’s try an unknown option to see how it handles it:

$ ./perlcmdline --foo
Unknown option: foo
Usage:
    perlcmdline [options]

    "perlcmdline --help" will list options. "perlcmdline --man"
    will show docs.

It told us that “foo” was an “Unknown option” and then it printed out usage from the SYNOPSIS (POD) section. This is the behavior of “pod2usage(2)”. Note the helpful “–help” and “–man” info we added to SYNOPSIS.

Now let’s try the “–length” option without the argument (it requires a number):

$ ./perlcmdline --length
Option length requires an argument
Usage:
    perlcmdline [options]

    "perlcmdline --help" will list options. "perlcmdline --man"
    will show docs.

Now let’s try a length that is equal to zero (we explicitly check for this and exit from the command-line):

$ ./perlcmdline --length 0
--length must be > 0.
Usage:
    perlcmdline [options]

    "perlcmdline --help" will list options. "perlcmdline --man"
    will show docs.

Now let’s try misspelling length (we’ll truncate it):

$ ./perlcmdline --len 5
length is 5

Whoa, what happened there? It turns out that Getopt automatically supports abbreviation. It will use the string to match the longest match possible.

Let’s try “–help”:

./perlcmdline --help
Usage:
    perlcmdline [options]

    "perlcmdline --help" will list options. "perlcmdline --man"
    will show docs.

Options:
    --help
        Print a brief help message and exits.

    --man
        Prints the manual page and exits.

    --length positive-integer
        A "do nothing" length value which must be an int > 0.

    --filename input-filename
        A "do nothing" string value.

    --addr hostname-or-ip
        An internet address.  This option may be used multiple
        times in a single command to specify multiple addresses.

Very cool. The options listed under (POD) OPTIONS got printed this time, due to pod2usage(1). Try “–man” on your own.

Now, let’s try some legitimate options.

$ ./perlcmdline --filename bar --length 2 --addr 127.0.0.1
--addr 192.168.0.1
length is 2
filename is bar
addr: 127.0.0.1
addr: 192.168.0.1

Note that the multiple options specified using –addr worked properly, as well as the other options.

Conclusion

Now you’ve got a template for creating command-line utilities that have embedded documentation and handle options with ease. You should copy it and use it as a starting point whenever you want to write a script or command-line utility. We’ve alsoseen the basics of using Getopt::Long, as well as basic POD, and pod2usage.

There are a few more important aspects to writing good command-line utilities that I’ll save for other posts.

References

1. Getopt::Long docs at perldoc.perl.org
2. pod2usage docs at perldoc.perl.org
3. POD docs at perldoc.perl.org