Another installment of our ongoing series of tech talks has just hit the youtubes. Paul Khuong, previously of this fame, brings us “Software Transactional Memory is Simple” which takes us into the innards of the AppNexus real time platform and how we do configuration data updates in such a low latency environment.
Baby’s first memory allocator – background
Let me start off by saying that one really isn’t supposed to write
custom memory allocators. Berger, Zorn, and McKinley thorougly
debunked the need for them by the vast majority of the population in
. If you’re looking for a speedy allocator, projects like
and dlmalloc have done such
a great job improving on glibc without changing the
interface that you shouldn’t even have to think about your memory
allocator unless you’re doing something hugely performance
sensitive. Just throw jemalloc on the case and get a free performance
improvement, then don’t bother with it again. That was our theory when
we moved to jemalloc at AppNexus in August of 2011, and it continues
to be the theory. But then we built our own allocator anyway. This is
the story of why and how.
The Signals Intelligence group (SIGINT) within AppNexus Data Science has the responsibility of identifying and stopping invalid traffic as quickly as possible. One technique they use to achieve this very quickly is collecting, aggregating and acting on streaming data using Kafka and Spark Streaming.
Watch this video to learn how AppNexus use these systems, some of the data science findings, the challenges and tribulations they’ve had to overcome, and how you can put these techniques into practice yourself.
An Open Call to Submit Papers for AppNexus Optimize NYC
We’ve always believed that creating a better Internet is a task far greater than the efforts of any given company or entity. That was the intention behind hosting our first-ever AppNexus Optimize which took place in London earlier this summer. Catherine Williams, our Chief Data Scientist, had this to say in her opening remarks for the event:
“We want to give you deeper insight into [AppNexus’] technology and tools, and share with you our learnings and best practices. But it’s not just a one-way street. We [also] want to hear from you… We want you to share with us your own learnings and best practices, your challenges and successes. And we’re hoping for you to learn from each other as well.”
Traditions tend to develop quickly here at AppNexus; we’re excited to announce that we’ll be hosting our second Optimize event, this time in New York City on November 3, 2015. We’ll be discussing everything from the impact of ad fraud across our industry to how AppNexus is bursting the video cost bubble with a set of new video buying capabilities. In addition to product announcements and technical breakouts, we’re offering the broader tech community an opportunity to “Razzle Dazzle” us with their own projects and solutions that power the tech that powers ad tech. We actively invite, challenge, welcome, and encourage all engineers, technologists, data scientists, and developers from outside AppNexus to submit projects that demonstrate the possibilities for advanced real-time data integrations of all kinds. Those who submit their papers by Friday, October 23 will have a chance to present their findings on the main stage at AppNexus Optimize NYC.
We look forward to hearing from you all!
The Actor Model is a computational model for designing concurrent, distributed systems around the principal of self-contained Actors which operate on sending and receiving messages. While the idea has been around since the mid to late 70’s, it is now gaining more traction with frameworks and languages such as Akka and Erlang, which share many similar principals.
See John Murray, Senior Software Engineer at AppNexus, serve up an introduction to Actor Model principals and concepts and give a look at how we can construct parallel systems using the Akka framework.
Learn more about upcoming Tech Talks at AppNexus here http://go.appnexus.com/TechTalks.html
We experienced a recent outage here at AppNexus surrounding an ephemeral user session store. This is a story of how visibly correct code can still fail in unexpected ways.
The video we posted on June 23rd, 2015 (link here) introduced our efforts to open up our big data platform to other teams within AppNexus. Data Platform as a Service (DPaaS) is our internal offering that allows other teams at AppNexus to run analytics upon our wealth of data. Our users want to be confident that they’re using the platform safely and appropriately. They only want to see the jobs and resources that are relevant to them, and not to impact other users nor mainline production processes. As operators of the platform, we want to ensure the safety and stability of the system as a whole and reasonable isolation between our users.
This clearly points to requiring an AAA solution - authentication, authorization and accounting.
- Authentication - identifying an entity acting upon your system
- Authorization - allowing/disallowing that entity to perform actions
- Accounting - keeping track of which entities have performed which actions
Practically speaking, we will be tackling these one at a time in the natural order listed above. Each item has its own complexity and intricacies, and this post will discuss those around the first A - authentication.
The AppNexus Mobile Advertising SDK provides developers a fast and convenient way to monetize their apps. It’s a well documented and throughly tested open-source code with direct engineering support. While implementing the Android native ad solution, I ran across a puzzling issue that I’d like to share my investigation and hopefully save other Android developers some time in the future. For those who are not familiar with native advertising, the IAB has a very clear video.
The issue was that a registration call to bond a native ad response with an Android native view would fail silently, even though the debugging tool showed that the code was executed correctly.
To simplify it, let’s pretend you’re building an app that turns on/off a flashlight, in which there’re two runnables
turnOffTheLight. You would assume that if we call
view.post(turnOffTheLight) first and then call
view.post(turnOnTheLight), the light should be on. However, in the test run, the light is actually off after execution. I put break points and stepped through the code,
turnOffTheLight was indeed posted first. Then what happened?
I downloaded the source code of Android SDK, stepped into the method
post() and found this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
It turns out that, if the view is not attached to the window, the runnable will be put in the RunQueue of the view hierachy - The run queue is used to enqueue pending work from Views when no Handler is attached. The work is executed during the next call to performTraversals on the thread.
Go back to the scenario above, when posting
turnOffTheLight the view was not attached but was attached when posting
turnOnTheLight is posted to UI thread to be executed immediately and
turnOffTheLight is not executed till the next performTraversals is called. The solution is very simple, post both runnables to the UI thread directly using the following method:
1 2 3 4 5 6 7
In the end, this is not actually a bug in the app’s code, it’s more of a rare use case exposed a slient inconvenient behavior of an Android convenience method, that the call to APIs must be done in the proper sequence to get the correct result. Sharing here with Android developers who might run into this weird situation too.
The Data Platform Engineering Team at AppNexus has been utilizing Hadoop in production for the last 4+ years. Growth of data volume as well as the number of customer use cases supported by Hadoop’s infrastructure has grown exponentially since we adopted the Hadoop stack. More specifically, in 2012, we were processing 10 terabytes of data per day, today we process over 170 terabytes per day.
We evaluated various commercial and open source solutions to reduce our storage foot print, improve Hadoop utilization, and unlock YARN’s multi-tenancy promises.
Our talk covers:
- Architecture of AppNexus Data Platform: Data ingestion, processing, and how our customers consume the data.
- Complex use cases supported by MapReduce, Vertica and Spark streaming.
- Overview of how we are offering our Data Platform As A Service to AppNexus’ business units where teams can build and manage their own YARN application deployments.
Understanding the low level behavior of our applications is one of the most useful skills for working on high throughput systems at AppNexus. We currently process around 4 million requests per second with extremely strict latency requirements, so being able to identify and correct inefficiencies at the instruction level can yield significant performance gains. More importantly, being able to work in assembly engenders feelings of supreme confidence and consummate pride (citation needed).
Here are some of our team’s favorite (and/or least favorite) x86 instructions described through comparisons to Taylor Swift songs.