Mike Sisk

s3cmd with Multiple Accounts

2011-08-09T00:00:00Z

Recently I've been doing a lot of work involving Amazon Simple Storage Service (aka Amazon S3).

And while tools like Panic's Transmit, the Firefox S3Fox extension, or even Amazon's own S3 Management Console make it easy to use, sometimes you really need a command-line tool.

There's a lot of good tools out there, but the one I've been using is s3cmd. This tool is done in Python, has been around awhile and is well documented. Installation on Linux or OS X is simple as is its configuration. And as a longtime Unix command-line user it's syntax is simple and Unixy. Some examples:

To list your buckets:

~ $ s3cmd ls
2010-04-28 23:50  s3://g5-images
2011-01-21 06:42  s3://g5-mongodb-backup
2011-03-21 21:23  s3://g5-mysql-backup
2010-06-03 17:45  s3://g5-west-images
2010-09-02 15:57  s3://g5engineering

List the size of a bucket with "human readable" units:

~ $ s3cmd du -H s3://g5-mongodb-backup
1132G    s3://g5-mongodb-backup/

List the contents of a bucket:

~ $ s3cmd ls s3://g5-mongodb-backup
2011-08-08 14:43 3273232889   s3://g5-mongodb-backup/mongodb.2011-08-08-06.tar.gz
2011-08-08 21:12 3290592536   s3://g5-mongodb-backup/mongodb.2011-08-08-12.tar.gz
2011-08-09 03:16 3302734859   s3://g5-mongodb-backup/mongodb.2011-08-08-18.tar.gz
2011-08-09 09:09 3308369423   s3://g5-mongodb-backup/mongodb.2011-08-09-00.tar.gz
2011-08-09 14:51 3285753739   s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz

Show the MD5 hash of an asset:

~ $ s3cmd ls --list-md5 s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz
2011-08-09 14:51 3285753739   07747e3de16138799d9fe1846436a3ce  \
    s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz

Transferring a file to a bucket uses the get and put commands. And if you forget an option or need a reminder of usage the very complete s3cmd --help output will likely be all the help you need.

One problem I have with most tools for AWS is managing multiple accounts. Most of these tools assume you have just one account, but I work with multiple accounts and switching between them can be cumbersome.

Here's how I work with multiple AWS accounts using s3cmd.

By default s3cmd puts its configuration file in ~/.s3cfg, but you can override this and specify a configuration file with the -c option.

What I do is create a separate config file with the appropriate credentials for each account I work with and give them unique names:

~ $ ls -1 .s3cfg*
.s3cfg-g5
.s3cfg-tcp

Another option is to keep the credentials for the account you use most often in the standard ~/.s3cfg file and use the -c option when/if you need another account. I don't like this option because it's too easy to mistakenly use the wrong account. For example, without a ~/.s3cfg this is what happens when I use s3cmd without specifying a configuration:

~  $ s3cmd ls
ERROR: /Users/mike/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

So, what to do? Using the -c all the time is a PITA. Answer: use Bash aliases! Here's a subset of the s3cmd aliases I have in my ~/.bashrc file:

# s3cmd aliases for different s3 accounts
alias s3g5='s3cmd -c ~/.s3cfg-g5'
alias s3tcp='s3cmd -c ~/.s3cfg-tcp'

Now, to list the buckets in my personal account I just do:

~ $ s3tcp ls
2011-07-01 06:10  s3://mikesisk-img
2011-07-05 23:16  s3://www.tcpipranch.com
2011-07-01 22:55  s3://www.watch4rocks.com

And I can still pass arguments:

~ $ s3tcp -H --list-md5 ls s3://mikesisk-img/me.jpg
2011-07-01 06:09         5k  13d7c86bccd8915dd93b085985305394  \
    s3://mikesisk-img/me.jpg

Just keep in mind that calls to bash aliases from scripts and cronjobs might not work. Plus it's bad form and will come back to bite you one of these days. Just use the long form with -c in these places and keep the aliases for your own interactive command-line usage.

Joplin Tornado

2011-07-01T00:00:00Z

Some pictures from my recent trip back to my hometown of Joplin, Missouri.

Cron and Sewing Needles

2011-02-16T00:00:00Z

Sometimes, even after decades of experience, you still screw up.

Consider this cron entry I put in last night:

# Backup MongoDB every 6 hours, zip it up, and rsync it.
* */6 * * * ~/bin/backup_mongo.sh

I wanted this to run the backup script for MongoDB every six hours. Instead, I got it running every minute for an hour every six hours. You'd think I'd know better considering I put the next cron in correctly:

# Remove MongoDB backups that are more than 24-hours old.
00 02 * * * find /db/backup -mtime +1 -exec rm -f {} \;

What I meant to do is this:

# Backup MongoDB every 6 hours, zip it up, and rsync it.
00 */6 * * * ~/bin/backup_mongo.sh

Luckily we host our infrastructure at Engine Yard and their staff noticed the CPU spike on this server at midnight and fixed the cron.

Which brings up another point: name your scripts appropriately. In this case a quick scan of cron revealed this script was running a backup and doing that every six hours makes sense. If the script was just named mongo, it's conceivable it could have been a metric collection script that runs every minute for an hour every six hours.

So what do sewing needles have to do with cron? I'm working from home this week and had just finished that MongoDB backup script and was putting it in cron when my daughter came running (Ok, make that limping) into my office with a large sewing needle in-bedded in the arch of her foot. I quickly saved the cron entry to take care of that problem and didn't go back to check my work.

Moral of the story: whenever you set up a new cron job it's a good idea to watch it run and see if it's doing what you think it is. Especially if you think you know what you're doing.

Driving

2011-02-13T00:00:00Z

I drive a lot.

I live in the Willamette Valley but drive to work in Bend, Oregon on the high desert – that's 130 miles each way over the Cascades at Santiam Pass twice a week. Generally I work from home on Monday and Friday and I'm in the office Tuesday though Thursday.

Until recently I kept an apartment over in Bend, but I gave it up and either stay at a hotel or just crash in the office. I tend to work long hours and with a recent rent, electric and cable-modem rate increase I just wasn't getting much value out of the apartment.

And so I drive. A lot of folks think I'm crazy, but I love to drive. And frankly, I've had to commute to work in some horrific traffic (I've commuted in LA, Boston, and San Francisco), so the scenic lightly traveled roads over the Cascades is a very pleasant drive in comparison.

Of course, weather can be a problem in the winter. Santiam Pass can get pretty nasty at times, but ODOT does a good job keeping it clear.

It helps that I drive a Jeep with snow tires and chains in reserve. I don't worry about getting up and down the pass as much as I do about someone in a hurry running into me. I've not seen many cars in the ditch but I have seen a lot of SUVs.

In the summer time with the Jeep I can take some interesting shortcuts. The picture above is one I took heading home this past summer.

What do I do to pass the time during six hours of driving every week? Listen to podcasts. There's so much to listen to that I would never have time unless I had this block of time.

Some vi tips

2011-02-10T00:00:00Z

Today I got hit with a last-minute request to update our Nginx redirect map with data provided in a Google docs spreadsheet. Normally I only have to do one or two redirect rules at a time. But today I got hit with 120 rules - each of which needed data from two cells in the spreadsheet. Doing it manually would require 240 cut-n-paste operations - not fun and error prone. Oh, and I only had 30-minutes to get this done and up in production.

We do these sorts of redirects when a customer changes their domain name. In this particular case the customer had a number of locations under one domain and they're now splitting each location out into its own domain. But we don't want visitors to the sites getting 404s due to the URL changing so we put in redirect rules to rewrite the request and forward the visitor to the new domain.

There's a bunch of ways to do this, but this is how I did it.

First, a quick explanation of the Nginx redirect map format. There's not much to it:

old.example.com    new.example.com;

That's the old URL, a space or spaces (or tab), the new URL and a semicolon terminating the line.

In this case, each row of the spreadsheet with the redirect data had 7 columns: the name of the property being redirected, and three URLs that needed to be redirected with the destination (i.e. Each location in this case only has three pages so there's only three redirects each). Luckily the order of the data is just what I needed for the map.

The first thing I did was a CSV export of the data and opened it up in vi.

The CSV export contained title information for the columns I don't need so let's just delete those right off.

Ok, now the data is ready to be processed into something I can use. In another stroke of luck we can see each redirect pair is separated by two commas since the spreadsheet contained an empty column between the three pairs for each location. This will make things much easier.

First, let's get each redirect on its own line in the file. We can search and replace for the domain name to be redirected since they're all the same. We'll search for the domain and replace it as the first thing on its own line with:

:%s:www.myfavoriteapartment.com:\rwww.myfavoriteapartment.com:g

The % does all lines in the file; the s is for search; the \r is vi-speak for newline; and the g at the end is for global so it'll process the whole line rather than the first match it finds on that line. By habit I use : for the separator in the search and replace command; you can also (and most people do) use /.

Ok, we're getting there. Next let's get rid of the double commas and – since that's always at the end of the redirect destination – put a semicolon at the end as Nginx requires:

:%s:,,:;:g

Now we need to deal with the first column of data from the spreadsheet – the name of the location. We don't need this information for the map file so let's just delete it. Each location name is on its own line at this point and ends with a comma. So let's search for lines ending with a comma and delete 'em:

:g:,$:d

In this case, the g is for a global operation on all lines; the ,$ matches all lines with a , at the end of a line (the $); and the d is for delete.

All we have left is to replace the single remaining comma on each line that separates the source and destination URLs. Nginx requires that the separation here be one or more spaces. I typically use a tab (although I probably shouldn't – it makes the map file look messy) so let's do this:

:%s:,:^I:g

The ^I is the tab character. Nowadays you can usually just press the tab key and vi will put in the ^I, but in the old days you had to do control-v and control-i to get a tab.

And that's it.

Now all we got to do is save the file and do a copy and paste of its contents into the map file on our production configuration. Of course, you'll want to scroll through the file and make sure it looks correct and do a nginx configtest before nginx reload to make sure it's valid.