Progressive Data Solutions

Hadoop on Rails

A few months ago, I wrote an article about using Ruby with Hadoop, and more specifically, the Amazon Elastic MapReduce (EMR) service. I hope some of you found that article helpful.

I figured it was time to post a follow-up - using Rails with Hadoop. Much of my work over the past few months has been building a system to efficiently store, process, and display large amounts of log data. Naturally I wanted to use Rails, but I also knew that I needed to use EMR. I was tasked with building the system myself, so rather than spend more time building and maintaining a Hadoop cluster, I opted to use EMR up-front and focus on the entire process.

So what have I done with Hadoop and Rails? Essentially, I’ve built a system that processes large amounts of log data end-to-end with Ruby, Hadoop/Pig, and Rails.

So what’s going on in this image? Following is a description of each step. Note that all Ruby scripts / processing are done within a Rails app, often with script/runner to enable access to the apps data model:

1. Log data is collected and stored in S3. A Ruby script on an EC2 instance, started by cron, downloads the log data for the previous day, consolidates the many separate files into one, and then compress the file using bzip2 compression.
2. A Ruby script sends the compressed file back to S3, storing it in a new bucket.
3. The compressed file is also sent to the Rackspace CloudFiles service, for off-site backup.
4. After log file consolidation and backup is complete, a Ruby script starts an Elastic MapReduce job.
5. Data for the job, created during steps 1 and 2, is transferred from S3 to the temporary Hadoop cluster created by Elastic MapReduce. The data is processed using a Pig script which is also stored in an S3 bucket.
6. Results of the EMR processing are stored in S3, in a separate bucket.
7. Later, after the Elastic MapReduce job is complete, the output is downloaded via a Ruby script to the tmp/ directory within the Rails app.
8. Once the data is downloaded, it is processed within the context of the Rails app, and loaded into a MySQL database residing on Amazon’s Relational Database Service.

Notice that each step makes use of Ruby and / or Rails. Ruby really is the glue that holds this system together, and it’s a very powerful glue. A lot of what I am doing is date-specific, and Ruby’s date library and methods make parsing and handling dates much easier (than using shell scrips).

The other language used in this system, Pig, is used to filter, count, and group the large datasets. Once Pig has done its work, running on EMR, the output is just a series of text files that are parsed by Ruby, then stored in MySQL using ActiveRecord relationships. Hadoop / Pig does the heavy lifting, while Ruby / Rails controls everything.

Each part of the system is designed to grow as needed. If log combination and compression is taking too long, it can be modified to run on a larger more powerful EC2 instance. Once that process gets too big for EC2, it could be moved into its own EMR process, using as many machines as necessary.

Likewise, if the Hadoop/Pig processing takes too long, more machines can be added by adjusting one line in the controlling script. Even the MySQL storage can be increased or moved to a more powerful server if needed, thanks to RDS and its simple API.

The biggest challenge in getting this system up and running was learning Pig. Once you understand that Pig is really for filtering, grouping, and counting data, you realize its power. Pig is not Turing Complete, so it can be challenging to solve problems with it. For instance, there are no loops, which can make certain types of problems difficult to solve. I worked around some problems I encountered by moving some of the processing into the Rails app.

There are issues with this system that will need to be addressed soon. The log file combination and compression will need to be improved. I’ll probably switch from bzip2 compression to using splitable LZO, as detailed in this article. Twitter is doing some pretty cool things with Pig / Hadoop and they make a strong case for using LZO. Another issue I’ll be looking at soon is how to streamline the EMR job process. I’m adding more jobs and at some point I’ll have to abstract what I’m doing into some sort of framework. There’s just too much code duplication there.

Let me know if you have questions.