Watch this tech talk to learn how our DevOps team has built a scalable infrastructure (using Docker and other technologies) to automate the creation of scaled-down replicas of the production AppNexus platform and make them available company-wide as an on-demand service. Featuring AppNexus Engineering Team Lead Leevar Williams.
Think hiring the best engineers is tough? What about keeping engineers engaged once they are a part of your organization? Watch the latest installment of our Tech Talks series to hear how Joel Spolsky, founder and CEO of Stack Overflow, and Igor Shindel, CIO of AppNexus, think about these issues and discuss what it takes to build and manage a world-class engineering organization.
As we’ve seen in our past couple of posts – Securing our Big Data Platform: Part 1 and Securing our Data Platform: Part 2 – we’ve been hard at work beefing up the security of our Big Data Platform. We had one other important piece of the puzzle to solve before we dotted our i’s on this security effort – Delegation.
Delegation is the process of letting a user act on the behalf of another user. Delegation allows user X to perform actions on behalf of user Y by granting the privileges of user Y upon user X. Thus, through delegation we are able to account for a user’s actions in our services.
The latest installment of our ongoing Tech Talk @ AppNexus videos is now live on YouTube. Watch John Wittrock, Senior Software Engineer on the Real-Time Platform team, tell a variety of disaster stories that illustrate the challenges of and lessons learned from supporting our large-scaled distributed backend that serves millions of requests a second across thousands of servers.
Last week a security researcher, Randy Westergren, pointed out in a blog post a vulnerability from cross-site scripting (XSS) that was common across our industry and impacted AppNexus. We’ve since rolled out a patch to the immediate issue and encourage you to read Brian’s blogpost addressing this and publisher security in the broader context.
Another in our “Tech Talks” series @ AppNexus.
Lambda architectures for Big Data sprung out of the concept that stream processing is fast but inaccurate. So you needed to have a fast but less accurate streaming path along with a slower, accurate batch path. With the advent of technologies like Ratsoda and Samza, this is no longer the case.
When last we saw our heroes, they were battling Hadoop authentication in DPaaS. But our intrepid knights were able to overcome that crafty foe and move on to the next obstacle: authorization. And that’s where our story continues…
Authorization is the control of access to resources or actions for an entity that’s requesting them. Implementing a proper authorization layer within our system is what will give our users the peace of mind that they are in control of their data & processes and are properly isolated from one another. As the operators of the system, we gain the same peace of mind and more. We get more control over the use of our finite resources. Better yet, we can take ourselves out of the critical path of our users’ needs — more automatic security means more self-service. That’s a big win-win.
Data Science … Translating Theory into the Real World
In today’s post, I will unite the theory behind a machine learning technique with a real-world industry problem. I’ll describe one problem we needed to solve, how we solved it by building a new tool, and how you can also leverage it!
Hopefully after reading this, you’ll have a better idea how we use data at AppNexus. Or, maybe you can sympathize with me after encountering yet another memory issue, and this tool will help alleviate your pain!
Pyrobuf is an alternative to Google’s
Python Protobuf library that is written in Cython and that offers better
performance (roughly 1.5-2x faster), Python 3 support, and simple
serialization/deserialization to JSON and native Python dictionaries.
Since Pyrobuf’s only installation requirements are Cython, Jinja2, and setuptools,
its also much easier to install than Google’s library, and should work as a drop-in
replacement. Pyrobuf parses the same
.proto specs as the Google library and
generates Python modules (in
Pyrobuf started as just one part of a larger Python “port” of a C library that we use for serializing and deserializing data between a variety of formats. We plan to open-source this library in the future, but in the mean time we saw such good results with the Protobuf portion of the library that it seemed worthwhile to release it on its own.
Another installment of our ongoing series of tech talks has just hit the youtubes. Paul Khuong, previously of this fame, brings us “Software Transactional Memory is Simple” which takes us into the innards of the AppNexus real time platform and how we do configuration data updates in such a low latency environment.