Think hiring the best engineers is tough? What about keeping engineers engaged once they are a part of your organization? Watch the latest installment of our Tech Talks series to hear how Joel Spolsky, founder and CEO of Stack Overflow, and Igor Shindel, CIO of AppNexus, think about these issues and discuss what it takes to build and manage a world-class engineering organization.
As we’ve seen in our past couple of posts – Securing our Big Data Platform: Part 1 and Securing our Data Platform: Part 2 – we’ve been hard at work beefing up the security of our Big Data Platform. We had one other important piece of the puzzle to solve before we dotted our i’s on this security effort – Delegation.
Delegation is the process of letting a user act on the behalf of another user. Delegation allows user X to perform actions on behalf of user Y by granting the privileges of user Y upon user X. Thus, through delegation we are able to account for a user’s actions in our services.
The latest installment of our ongoing Tech Talk @ AppNexus videos is now live on YouTube. Watch John Wittrock, Senior Software Engineer on the Real-Time Platform team, tell a variety of disaster stories that illustrate the challenges of and lessons learned from supporting our large-scaled distributed backend that serves millions of requests a second across thousands of servers.
Last week a security researcher, Randy Westergren, pointed out in a blog post a vulnerability from cross-site scripting (XSS) that was common across our industry and impacted AppNexus. We’ve since rolled out a patch to the immediate issue and encourage you to read Brian’s blogpost addressing this and publisher security in the broader context.
Another in our “Tech Talks” series @ AppNexus.
Lambda architectures for Big Data sprung out of the concept that stream processing is fast but inaccurate. So you needed to have a fast but less accurate streaming path along with a slower, accurate batch path. With the advent of technologies like Ratsoda and Samza, this is no longer the case.
When last we saw our heroes, they were battling Hadoop authentication in DPaaS. But our intrepid knights were able to overcome that crafty foe and move on to the next obstacle: authorization. And that’s where our story continues…
Authorization is the control of access to resources or actions for an entity that’s requesting them. Implementing a proper authorization layer within our system is what will give our users the peace of mind that they are in control of their data & processes and are properly isolated from one another. As the operators of the system, we gain the same peace of mind and more. We get more control over the use of our finite resources. Better yet, we can take ourselves out of the critical path of our users’ needs — more automatic security means more self-service. That’s a big win-win.
Data Science … Translating Theory into the Real World
In today’s post, I will unite the theory behind a machine learning technique with a real-world industry problem. I’ll describe one problem we needed to solve, how we solved it by building a new tool, and how you can also leverage it!
Hopefully after reading this, you’ll have a better idea how we use data at AppNexus. Or, maybe you can sympathize with me after encountering yet another memory issue, and this tool will help alleviate your pain!
Pyrobuf is an alternative to Google’s
Python Protobuf library that is written in Cython and that offers better
performance (roughly 1.5-2x faster), Python 3 support, and simple
serialization/deserialization to JSON and native Python dictionaries.
Since Pyrobuf’s only installation requirements are Cython, Jinja2, and setuptools,
its also much easier to install than Google’s library, and should work as a drop-in
replacement. Pyrobuf parses the same
.proto specs as the Google library and
generates Python modules (in
Pyrobuf started as just one part of a larger Python “port” of a C library that we use for serializing and deserializing data between a variety of formats. We plan to open-source this library in the future, but in the mean time we saw such good results with the Protobuf portion of the library that it seemed worthwhile to release it on its own.
Another installment of our ongoing series of tech talks has just hit the youtubes. Paul Khuong, previously of this fame, brings us “Software Transactional Memory is Simple” which takes us into the innards of the AppNexus real time platform and how we do configuration data updates in such a low latency environment.
Baby’s first memory allocator – background
Let me start off by saying that one really isn’t supposed to write
custom memory allocators. Berger, Zorn, and McKinley thorougly
debunked the need for them by the vast majority of the population in
. If you’re looking for a speedy allocator, projects like
and dlmalloc have done such
a great job improving on glibc without changing the
interface that you shouldn’t even have to think about your memory
allocator unless you’re doing something hugely performance
sensitive. Just throw jemalloc on the case and get a free performance
improvement, then don’t bother with it again. That was our theory when
we moved to jemalloc at AppNexus in August of 2011, and it continues
to be the theory. But then we built our own allocator anyway. This is
the story of why and how.