Progressive Data Solutions

What Second Life can teach your datacenter about scaling Web apps

Tue, 02 Feb 2010 21:19:20 -0700

What Second Life can teach your datacenter about scaling Web apps

Scaling lessons from Second Life

Tue, 02 Feb 2010 18:35:18 -0700

Scaling lessons from Second Life:

Comments

Generating Thousands of PDFs on EC2 with Ruby « Rails Dog

Wed, 23 Dec 2009 20:35:35 -0700

Generating Thousands of PDFs on EC2 with Ruby « Rails Dog

Hadoop on Rails

Thu, 10 Dec 2009 12:32:54 -0700

A few months ago, I wrote an article about using Ruby with Hadoop, and more specifically, the Amazon Elastic MapReduce (EMR) service. I hope some of you found that article helpful.

I figured it was time to post a follow-up - using Rails with Hadoop. Much of my work over the past few months has been building a system to efficiently store, process, and display large amounts of log data. Naturally I wanted to use Rails, but I also knew that I needed to use EMR. I was tasked with building the system myself, so rather than spend more time building and maintaining a Hadoop cluster, I opted to use EMR up-front and focus on the entire process.

So what have I done with Hadoop and Rails? Essentially, I’ve built a system that processes large amounts of log data end-to-end with Ruby, Hadoop/Pig, and Rails.

So what’s going on in this image? Following is a description of each step. Note that all Ruby scripts / processing are done within a Rails app, often with script/runner to enable access to the apps data model:

Log data is collected and stored in S3. A Ruby script on an EC2 instance, started by cron, downloads the log data for the previous day, consolidates the many separate files into one, and then compress the file using bzip2 compression.
A Ruby script sends the compressed file back to S3, storing it in a new bucket.
The compressed file is also sent to the Rackspace CloudFiles service, for off-site backup.
After log file consolidation and backup is complete, a Ruby script starts an Elastic MapReduce job.
Data for the job, created during steps 1 and 2, is transferred from S3 to the temporary Hadoop cluster created by Elastic MapReduce. The data is processed using a Pig script which is also stored in an S3 bucket.
Results of the EMR processing are stored in S3, in a separate bucket.
Later, after the Elastic MapReduce job is complete, the output is downloaded via a Ruby script to the tmp/ directory within the Rails app.
Once the data is downloaded, it is processed within the context of the Rails app, and loaded into a MySQL database residing on Amazon’s Relational Database Service.

Notice that each step makes use of Ruby and / or Rails. Ruby really is the glue that holds this system together, and it’s a very powerful glue. A lot of what I am doing is date-specific, and Ruby’s date library and methods make parsing and handling dates much easier (than using shell scrips).

The other language used in this system, Pig, is used to filter, count, and group the large datasets. Once Pig has done its work, running on EMR, the output is just a series of text files that are parsed by Ruby, then stored in MySQL using ActiveRecord relationships. Hadoop / Pig does the heavy lifting, while Ruby / Rails controls everything.

Each part of the system is designed to grow as needed. If log combination and compression is taking too long, it can be modified to run on a larger more powerful EC2 instance. Once that process gets too big for EC2, it could be moved into its own EMR process, using as many machines as necessary.

Likewise, if the Hadoop/Pig processing takes too long, more machines can be added by adjusting one line in the controlling script. Even the MySQL storage can be increased or moved to a more powerful server if needed, thanks to RDS and its simple API.

The biggest challenge in getting this system up and running was learning Pig. Once you understand that Pig is really for filtering, grouping, and counting data, you realize its power. Pig is not Turing Complete, so it can be challenging to solve problems with it. For instance, there are no loops, which can make certain types of problems difficult to solve. I worked around some problems I encountered by moving some of the processing into the Rails app.

There are issues with this system that will need to be addressed soon. The log file combination and compression will need to be improved. I’ll probably switch from bzip2 compression to using splitable LZO, as detailed in this article. Twitter is doing some pretty cool things with Pig / Hadoop and they make a strong case for using LZO. Another issue I’ll be looking at soon is how to streamline the EMR job process. I’m adding more jobs and at some point I’ll have to abstract what I’m doing into some sort of framework. There’s just too much code duplication there.

Let me know if you have questions.

Production Rails Tuning with Passenger: PassengerMaxProcesses

Tue, 08 Dec 2009 16:34:22 -0700

Production Rails Tuning with Passenger: PassengerMaxProcesses:

tuning phusion passenger performance

masterkain's ImageMagick-sl at master - GitHub

Thu, 03 Dec 2009 22:15:50 -0700

masterkain's ImageMagick-sl at master - GitHub:

working build script for ImageMagick on Snow Leopard

Passing environment variables to Ruby from Phusion Passenger « Phusion Corporate Blog

Thu, 03 Dec 2009 18:05:04 -0700

Passing environment variables to Ruby from Phusion Passenger « Phusion Corporate Blog:

getting mini_magick gem and ImageMagick to work with passenger

Pollux: Automatically Organize and Fix Your Music Library

Thu, 03 Dec 2009 10:11:49 -0700

Pollux: Automatically Organize and Fix Your Music Library:

free app to clean itunes library

TidySongs.com – Tidy Up Your Music!

Thu, 03 Dec 2009 10:11:49 -0700

TidySongs.com – Tidy Up Your Music!:

clean itunes library

ActiveJS: The Cross Platform JavaScript MVC

Wed, 02 Dec 2009 14:58:33 -0700

ActiveJS: The Cross Platform JavaScript MVC:

ActiveRecord-like library for browsers

R A C K A M O L E

Sat, 28 Nov 2009 21:04:14 -0700

R A C K A M O L E:

rack app for tracking site usage

Interview with Ezra Zygmuntowicz – Engine Yard

Tue, 24 Nov 2009 12:01:37 -0700

Interview with Ezra Zygmuntowicz – Engine Yard:

interview with Ezra about the history of EngineYard

Neven Mrgan's tumbl

Tue, 24 Nov 2009 12:01:37 -0700

Neven Mrgan's tumbl:

glyphboard for iphone

#haml

Fri, 20 Nov 2009 15:56:23 -0700

#haml:

haml for rails

Apple's Mistake

Fri, 20 Nov 2009 11:56:47 -0700

Apple's Mistake:

“Software isn’t like music or books. It’s too complicated for a third party to act as an intermediary between developer and user. And yet that’s what Apple is trying to be with the App Store: a…