tech blog

AppNexus is today’s most powerful, open, and customizable ad tech platform. Advertising’s largest and most innovative companies build their businesses on AppNexus.

Software Transactional Memory is Simple

| Comments

Another installment of our ongoing series of tech talks has just hit the youtubes. Paul Khuong, previously of this fame, brings us “Software Transactional Memory is Simple” which takes us into the innards of the AppNexus real time platform and how we do configuration data updates in such a low latency environment.

AppNexus’ real-time adserving stack is built on non-blocking concurrency control, which is how we achieve sub 1% timeout rates. In practice, it’s easy to misuse these techniques, which can result in crashes or angry clients. In this tech talk AppNexus Principal Engineer Paul Khuong talks about software transactional memory, how to implement it (simply), and learn about the specialized non-blocking STM we use here at AppNexus. Sign up to our MeetUp group to stay up to date with upcoming tech talks: http://www.meetup.com/TechTalks-AppNexus-NYC/

pool_party – Transactional allocation for short-lived requests

| Comments

Baby’s first memory allocator – background

Let me start off by saying that one really isn’t supposed to write custom memory allocators. Berger, Zorn, and McKinley thorougly debunked the need for them by the vast majority of the population in [1]. If you’re looking for a speedy allocator, projects like jemalloc, tcmalloc, and dlmalloc have done such a great job improving on glibc without changing the malloc/free interface that you shouldn’t even have to think about your memory allocator unless you’re doing something hugely performance sensitive. Just throw jemalloc on the case and get a free performance improvement, then don’t bother with it again. That was our theory when we moved to jemalloc at AppNexus in August of 2011, and it continues to be the theory. But then we built our own allocator anyway. This is the story of why and how.

Stopping Invalid Traffic using Spark Streaming, Kafka, and Science!

| Comments

The Signals Intelligence group (SIGINT) within AppNexus Data Science has the responsibility of identifying and stopping invalid traffic as quickly as possible. One technique they use to achieve this very quickly is collecting, aggregating and acting on streaming data using Kafka and Spark Streaming.

Watch this video to learn how AppNexus use these systems, some of the data science findings, the challenges and tribulations they’ve had to overcome, and how you can put these techniques into practice yourself.

The Signals Intelligence group (SIGINT) within AppNexus Data Science has the responsibility of identifying and stopping invalid traffic as quickly as possible. One technique they use to achieve this very quickly is collecting, aggregating and acting on streaming data using Kafka and Spark Streaming. Watch this video to learn how AppNexus use these systems, some of the data science findings, the challenges and tribulations they’ve had to overcome, and how you can put these techniques into practice yourself.

Optimize - Call for Speakers

| Comments

An Open Call to Submit Papers for AppNexus Optimize NYC

We’ve always believed that creating a better Internet is a task far greater than the efforts of any given company or entity. That was the intention behind hosting our first-ever AppNexus Optimize which took place in London earlier this summer. Catherine Williams, our Chief Data Scientist, had this to say in her opening remarks for the event:

“We want to give you deeper insight into [AppNexus’] technology and tools, and share with you our learnings and best practices. But it’s not just a one-way street. We [also] want to hear from you… We want you to share with us your own learnings and best practices, your challenges and successes. And we’re hoping for you to learn from each other as well.”

Traditions tend to develop quickly here at AppNexus; we’re excited to announce that we’ll be hosting our second Optimize event, this time in New York City on November 3, 2015. We’ll be discussing everything from the impact of ad fraud across our industry to how AppNexus is bursting the video cost bubble with a set of new video buying capabilities. In addition to product announcements and technical breakouts, we’re offering the broader tech community an opportunity to “Razzle Dazzle” us with their own projects and solutions that power the tech that powers ad tech. We actively invite, challenge, welcome, and encourage all engineers, technologists, data scientists, and developers from outside AppNexus to submit projects that demonstrate the possibilities for advanced real-time data integrations of all kinds. Those who submit their papers by Friday, October 23 will have a chance to present their findings on the main stage at AppNexus Optimize NYC.

We look forward to hearing from you all!

Submit your paper

Introduction to the Actor Model for Concurrent Computation

| Comments

The Actor Model is a computational model for designing concurrent, distributed systems around the principal of self-contained Actors which operate on sending and receiving messages. While the idea has been around since the mid to late 70’s, it is now gaining more traction with frameworks and languages such as Akka and Erlang, which share many similar principals.

See John Murray, Senior Software Engineer at AppNexus, serve up an introduction to Actor Model principals and concepts and give a look at how we can construct parallel systems using the Akka framework.

Learn more about upcoming Tech Talks at AppNexus here http://go.appnexus.com/TechTalks.html

In this tech talk, John Murray, Senior Software Engineer at AppNexus, serves up an introduction to Actor Model principals and concepts and gives a look at how we can construct parallel systems using the Akka framework. Actor Models is a computational model for designing concurrent, distributed systems around the principal of self-contained Actors which operate on sending and receiving messages. While the idea has been around since the mid to late 70’s, it is now gaining more traction with frameworks such as Akka and Celluloid as well as languages such as Go, which share many similar principals. Learn more about events at AppNexus at www.appnexus.com/razzledazzle.

Securing our Big Data Platform Part 1

| Comments

Overview

The video we posted on June 23rd, 2015 (link here) introduced our efforts to open up our big data platform to other teams within AppNexus. Data Platform as a Service (DPaaS) is our internal offering that allows other teams at AppNexus to run analytics upon our wealth of data. Our users want to be confident that they’re using the platform safely and appropriately. They only want to see the jobs and resources that are relevant to them, and not to impact other users nor mainline production processes. As operators of the platform, we want to ensure the safety and stability of the system as a whole and reasonable isolation between our users.

This clearly points to requiring an AAA solution - authentication, authorization and accounting.

  • Authentication - identifying an entity acting upon your system
  • Authorization - allowing/disallowing that entity to perform actions
  • Accounting - keeping track of which entities have performed which actions

Practically speaking, we will be tackling these one at a time in the natural order listed above. Each item has its own complexity and intricacies, and this post will discuss those around the first A - authentication.

Weird Android bug

| Comments

The AppNexus Mobile Advertising SDK provides developers a fast and convenient way to monetize their apps. It’s a well documented and throughly tested open-source code with direct engineering support. While implementing the Android native ad solution, I ran across a puzzling issue that I’d like to share my investigation and hopefully save other Android developers some time in the future. For those who are not familiar with native advertising, the IAB has a very clear video.

The issue was that a registration call to bond a native ad response with an Android native view would fail silently, even though the debugging tool showed that the code was executed correctly.

To simplify it, let’s pretend you’re building an app that turns on/off a flashlight, in which there’re two runnables turnOnTheLight and turnOffTheLight. You would assume that if we call view.post(turnOffTheLight) first and then call view.post(turnOnTheLight), the light should be on. However, in the test run, the light is actually off after execution. I put break points and stepped through the code, turnOffTheLight was indeed posted first. Then what happened?

I downloaded the source code of Android SDK, stepped into the method post() and found this:

Android SDK source code from View.javaSource
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
/**
 * <p>Causes the Runnable to be added to the message queue.
 * The runnable will be run on the user interface thread.</p>
 *
 * @param action The Runnable that will be executed.
 *
 * @return Returns true if the Runnable was successfully placed in to the
 *         message queue.  Returns false on failure, usually because the
 *         looper processing the message queue is exiting.
 *
 * @see #postDelayed
 * @see #removeCallbacks
 */
public boolean post(Runnable action) {
    final AttachInfo attachInfo = mAttachInfo;
    if (attachInfo != null) {
        return attachInfo.mHandler.post(action);
    }
    // Assume that post will succeed later
    ViewRootImpl.getRunQueue().post(action);
    return true;
}

It turns out that, if the view is not attached to the window, the runnable will be put in the RunQueue of the view hierachy - The run queue is used to enqueue pending work from Views when no Handler is attached. The work is executed during the next call to performTraversals on the thread.

Go back to the scenario above, when posting turnOffTheLight the view was not attached but was attached when posting turnOnTheLight. Thus, turnOnTheLight is posted to UI thread to be executed immediately and turnOffTheLight is not executed till the next performTraversals is called. The solution is very simple, post both runnables to the UI thread directly using the following method:

AppNexus SDK source codeSource
1
2
3
4
5
6
7
 Handler handler = new Handler(Looper.getMainLooper());
 handler.post(new Runnable() {
        @Override
        public void run() {
            // code
        }
});

In the end, this is not actually a bug in the app’s code, it’s more of a rare use case exposed a slient inconvenient behavior of an Android convenience method, that the call to APIs must be done in the proper sequence to get the correct result. Sharing here with Android developers who might run into this weird situation too.

Taming Big Data

| Comments

The Data Platform Engineering Team at AppNexus has been utilizing Hadoop in production for the last 4+ years. Growth of data volume as well as the number of customer use cases supported by Hadoop’s infrastructure has grown exponentially since we adopted the Hadoop stack. More specifically, in 2012, we were processing 10 terabytes of data per day, today we process over 170 terabytes per day.

We evaluated various commercial and open source solutions to reduce our storage foot print, improve Hadoop utilization, and unlock YARN’s multi-tenancy promises.

Our talk covers:

  • Architecture of AppNexus Data Platform: Data ingestion, processing, and how our customers consume the data.
  • Complex use cases supported by MapReduce, Vertica and Spark streaming.
  • Overview of how we are offering our Data Platform As A Service to AppNexus’ business units where teams can build and manage their own YARN application deployments.
Watch the video to learn how the AppNexus Data Platform Engineering Team has utilized Hadoop in production for the last 4+ years. Join our NYC Tech community MeetUp to stay up to date on future tech talks: http://www.meetup.com/TechTalks-AppNexus-NYC/ AppNexus’ advertising systems generate over 175 TB of data every day, which makes running a mission-critical data pipeline with tight SLAs an extremely challenging endeavor. Learn how the team evaluated various commercial and open source solutions to reduce our storage foot print, improved Hadoop utilization and unlocked YARN’s multi-tenancy promises. Learn more about

x Eighty Swift

| Comments

Understanding the low level behavior of our applications is one of the most useful skills for working on high throughput systems at AppNexus. We currently process around 4 million requests per second with extremely strict latency requirements, so being able to identify and correct inefficiencies at the instruction level can yield significant performance gains. More importantly, being able to work in assembly engenders feelings of supreme confidence and consummate pride (citation needed).

Here are some of our team’s favorite (and/or least favorite) x86 instructions described through comparisons to Taylor Swift songs.

-><-