These days, the guys and gals on the data pipeline team find themselves in an interesting position: wrangling 20TB of billing data and records into databases and data warehouses every single day. Our optimization team couldn’t be more pleased: the more data they have to work with, the better and more fine-tuned their bidding algorithms can be. However, this also presents an immense technical challenge for the rest of the company. Not only do we store all of this data, but we also provide reports for clients that let them know how their campaigns are doing on our platform, and we ship these reports once an hour. Finally, all of this data has to be redundant. No one would be happy if a datacenter’s worth of logs went missing.

To solve the storage and network transport problems induced by our growth, we built a system called Packrat. Its function is simple: receive data in rows, maybe store a copy, and then ship it out elsewhere.  Packrat is developed and maintained by the ad server team, since we interface so closely with our core ad server and bidding engine, but Packrat’s real purpose is to move the data somewhere it can be effectively dealt with: the data team. To further our data redundancy effort, each datacenter (three globally) restreams all its data to another. As you might imagine, we transmit a tremendous amount of data all over the world, and network bandwidth is a huge factor in capacity planning and our overall growth.

When I started at AppNexus, I was told that one of my big projects for the summer would be compressing large portions of our data before it crosses the wire.  Continue reading…