A Hadoop Success Story: Horizontally Scaling Our Data Pipeline

1 Comment

With 2000% growth in 2011, a data pipeline bursting at the seams, and 24/7/365 uptime, AppNexus engineers faced a task likened to changing the engine of a 747 in midflight. In order to horizontally scale the pipeline and data reporting, we adapted a variety of technologies, with Hadoop, HBase, and Hive composing the core elements.

At a presentation for the NYC Hadoop Meetup on February 28, 2012, Director of Engineering Ersin Yilmaz and Senior Software Engineer Sateesh Lakkarsu shared lessons learned in terms of hardware application, monitoring, rollback, failover nodes, controlling memory allocation, and day-to-day fire drills. A recording of the presentation is below:

This entry was posted in Architecture, Back-end Feature, Infrastructure. Bookmark the permalink.

1 Comment
  • Amey Jadiye

    This is AWSOME what these guys did !!!!!! :-)