Project Hydra – Scaling an API Globally



Posted on April 10, 2012 by Larry

At AppNexus we provide our clients with a RESTful API that allows people to manage all of their data in our adserving echosystem.  The API is built on a LAMP stack with memcache sprinkled in to help out with session management and cacheing some commonly used objects.  We have the typical cloud backend with various servers behind a load balancer with all of our servers housed in our East Coast data center.

When the time came for global expansion, we needed to spool up instances in our West Coast and European data centers.

Our Expectations:

Using our cloud backend, we could just spool up some virtual machines for Apache, memcache, and MySQL slaves replicating from the master in the East Coast.  GET requests to the API will use the local MySQL slave and memcaches while POST/PUT/DELETE calls will go to the master database in the East Coast. Pretty damn simple, right?

Wrong!  You didn’t think it would be that easy, did you?  
Continue reading…

comments (1)

Introduction to Lock-Free Algorithms

Posted on February 23, 2012 by sbahra

Slides are at http://concurrencykit.org/presentations/lockfree_introduction/.

Recently I’ve been interested in formalizing a course structure for the practical application of non-blocking data structure design and implementation. As part of this effort, I will be giving public Tech Talks here at AppNexus. The first talk was “Introduction to Lock-Free Algorithms” and it went reasonably well. There are a lot of improvements to be made to the introduction course, primarily because I think I attempted to cover too much at once. The current deck of slides also fails to illustrate linearizability sufficiently.

I plan on providing a majorly retrofitted implementation of this presentation some time in the near future (the next loop of the introduction class) but in the mean time I have made the current deck of slides available at http://concurrencykit.org/presentations/lockfree_introduction/. For the curious, I used reveal.js to create the presentation. When viewing the presentation, please note the controls in the top right which indicate in which direction you should go. Use the arrow keys to navigate and prefer going down to going left/right.

If you’re interested in learning more about the goals of the course (to be decomposed into several talks), please read more. The relevant skillshare page is http://www.skillshare.com/Introduction-to-Lock-Free-Algorithms/592151380.

Continue reading…

comments (7)

AppNexus CEO Talks Cache Update Cycles

Posted on February 19, 2012 by Emily

When you’re making programmatic purchases in real time, it’s important to be able to update your preferences as fast as possible. An ad industry blog post this week by AppNexus CEO (and experienced software engineer) Brian O’Kelley talks about how AppNexus cache updates were designed to quickly push user changes into production.

“Instead of sending the whole file out to each box on every update cycle, we only push out the changes. This dramatically reduces the amount of data that we have to send out each cache cycle, and means that we can update our caches as frequently as we want to. At the moment, any change you make through AppNexus Console or the AppNexus API is live on every server in three minutes or less.”

Read more at How Real Time Is Your RTB Platform.

comments (0)