22 January 2008

Amazon EC2 as a Build and Development Environment

How many people are faced with using machines where they're not given root access, it's a headache to get additional software installed and you're required to access those systems using a slow VPN? Although that list is short, there are a number of impedances to making any real progress in there. So to short-circuit the political morass, I took a look at Amazon EC2 and so far I am quite pleased.

Just before the holidays, I took a day to dig in to Amazon EC2 and I must say that having the ability to quickly (less than 30 mins) spin up a machine image, sized to my liking where I have total control of the operating system and the software installed on it is exactly what I needed as a developer.

It took me a couple hours to walk through and understand all the steps to create an image, but it was more than worth it. I did quickly discover that the docs don't tell you that you must size the image to your liking when it's created (see the ec2-bundle-vol utility's --size option). The default image size is 1GB (which is pretty small) and the maximum size is 10GB - I now use 10GB :-). The other thing I discovered was that shutting down an image will clobber *everything* you've installed on that image. This caused me to have to recreate my image a couple times after installing a bunch of software and configuring the image to my liking.

There are certain limitations like I've mentioned above with the storage, no static IPs assigned to images, inability to hack the kernel, and other things. The storage shortcoming I'm already working around by looking into using an Amazon S3 bucket as a filesystem to easily store stuff. The other stuff is really not critical for me so far because I'm just using it as a build and development environment. Maybe if I decide to host real apps on an image someday I'll worry more about them. But for the time being, providing me an environment over which I have total control and only need SSH to access is a major leg up.

There is a cost involved but it's minimal when compared to hosting a machine in a facility. If you need a machine where you have total control and a low cost, I encourage you to take a look at Amazon EC2.


  1. -S3 is not a real 'local' FS, and the tricks to make it look like one are dubious. Dont try using it like that. Instead, do your builds to the transient filestore (/mnt). After you've built, publish to S3...a POST can do it if the library has S3 authentication support.

  2. Take a look at EBS. You can have a permanent "disk" that you mount on whatever instance you are running, that you can snapshot and clone at any time and which survives the demise of any instance it is attached to.

    Sounds ideal for your needs.

    Also, Elastic IP's allow you to assign a permanent IP address to an instance. Indeed, you can even take the instance down, start another one and attach the Elastic IP to the new instance. Again, sounds like just what you want.

  3. Bruce, I bumped into your article on using Amazon EC2 for build and dev while searching for businesses that may make use of my current startup technology. We've just released a new product LabSlice (http://LabSlice.com) which extends EC2 into a self-service, policy controlled and sharable lab environment. I'm not sure if it's 100% aligned with your thoughts about this space, but I'd love to hear your feedback. The tool is currently in Beta and is free for use once you register an account.