Today we’re excited to announce the release of a new UI feature we call Campaign Monitor. The process that brought us here began at LUXr with a simple idea: users who manage campaigns in our interface need a tool that gives them insight into possible issues that hurt campaign health. We set out to see what we could do to help.

To design a useful tool, you first need to understand three things: who will use it, what they need it to do, and why. This spring we met with a bunch of campaign managers at partner ad networks (the who) and learned that they must keep track of delivery and performance on tons of advertising campaigns simultaneously (the what). Their advertisers usually specify a timeline, a budget, targeting, and goals for each campaign. If things don’t go well, the advertisers might reallocate the next quarter’s campaign budgets to other higher performing networks (the why).

As we talked to our users, patterns began to emerge. Many users had built very similar spreadsheets using Excel, SalesForce, or their own internal interfaces. Their spreadsheets compared actual data from our system to the expected performance or delivery targets for each campaign. Users would manually enter items’ settings into the spreadsheets and pull several reports in our interface each morning to populate the data. Upon seeing items with severe delivery or performance issues, they would go to our interface to troubleshoot, then return to their spreadsheet to find the next underperforming item. This back-and-forth would continue throughout the day, every day.

Round and Round We Go

The next step was to start prototyping a tool that would help campaign managers monitor items more efficiently and effectively. During our campaign manager interviews one metric kept appearing, a KPI (key performance indicator) that essentially slices the lifetime budget into daily goals and compares expected to actual delivery to-date. We hoped this ‘magic number’ would enable us to summarize line item and campaign health for users at a glance, so we created an extremely simple prototype that we could present to the campaign managers we’d just visited to see what they thought.

What The… Health?

We called our first wireframe-like prototype Campaign Triage because it was designed to bring the most troubled items to campaign managers’ attention. Users could toggle through preset views that would sort and filter the objects displayed based on what factors they cared about most (“troubled campaigns”, “ending soon”, “setup issues”). Health was indicated by a colored dot and the ‘magic number’ that showed how far off-target a campaign was. All data, alerts, and calls to action were completely made up.

Above, prototype round 1: Campaign Triage

While campaign managers liked some aspects of the rudimentary Campaign Triage prototype, the layout and health indicator were not particularly well received. The tool wouldn’t enable users to do what their spreadsheets did. One user told us, “Depending on the number of campaigns folks had, it could get rather cluttered to take this approach rather than something like a table where you can just sort. But at the same time I do like how this gets really granular and points out things like trafficking parameters that are a bit tough to quantify in a table format.” Though we weren’t close to the right solution for our users, the interest and ideas this prototype generated encouraged us to take another cut.

Spreadsheet a Bit Thin

The second prototype we built looked a bit more like the spreadsheets campaign managers work with every day. It provided raw lifetime stats in addition to that ‘magic’ pacing number. Expanding the rows displayed detailed information, and a large graph compared expected to actual cumulative delivery. Each row also had a small sparkline that preserved the at-a-glance quality of the health indicator dots while adding an extra layer of information about performance over time.

Above, prototype round 2: Spreadsheet view

When we put this prototype in front of our users, we still heard confusion about the pacing numbers. “Can you give me an example of what this pacing number would represent?” one user asked. Campaign managers explained that health was not simply about a single number — campaigns or line items could have revenue or media cost budgets or impression budgets, and sometimes performance metrics mattered more than delivery. But all was not lost; one user kept our hopes up by saying, “I like where it’s going, I think it could definitely be helpful to see things at a glance.”

Raising the Bars

Before returning to the drawing board for our next cut, we took a step back and mulled over all the feedback we’d received. It was clear we needed to abandon our assumption about that single ‘magic number’ that indicated campaign health. One metric was not sufficient to capture all the information that matters to campaign managers. Sunlight broke through the clouds when we found we could add sparkline-inspired pacing bars wherever interesting data appeared for a campaign or line item. These pacing bars compared where a campaign was (the colored fill) to where it should be (the dark grey line) in the context of the campaign’s total budget (the full bar).

In order to be convincing, the prototype also needed realistic data. We created a spreadsheet that looked very similar to those our users created and pulled a series of reports from our production database. A little tweaking and obfuscation enabled us to fill our prototype with lifelike numbers.

Above, prototype round 3: Campaign Monitor

Finally we had a design that really resonated with users. Adding real-life data enabled campaign managers to see and interpret the stories that the campaigns were telling, and with the pacing bars they could now imagine how the tool could actually help them. We knew we were definitely onto something when a user told us, “I’m glad you guys are showing me this, because this is something that we’ve been waiting for for a few years. I mean, working with [other ad platform] for all those years, we never had anything like this. Having the external Excel sheets and always keeping them updated has always been a huge load on our side, so having this within the UI will save our team a ton of time.” Success!

The Alpha Bet

Now that we felt confident about what Campaign Monitor should look like, we needed to know how actual campaigns would display with the settings and data that could be stored across many objects in our system. The numbers in our prototype were idealized because we had the luxury of choosing campaigns and line items that were properly set up with defined budgets and flight dates, and we were able to manually gather data and settings from different multiple levels to populate the rest of the data that would produce the pacing bars. Unfortunately for the Campaign Monitor project, setting budgets and flight dates on campaigns in our system is a best practice, but our UI does not require these settings be added upon object creation. In addition, it could be challenging for the UI to synthesize data and settings from multiple levels on the backend. We started to understand why this tool hadn’t been built before.

To see what we were up against we took a leap of faith and transformed our design from a prototype into a fully functioning feature. Our entire UI team embarked on a two day off-site hackathon to build a real working version the tool into our alpha environment where we could view actual production data in live accounts. The UI team had a functional interface up and running by the end of those two days, and they spent another few weeks cleaning it up and fixing bugs.

Above, Campaign Monitor in Alpha, with production data

Seeing live campaigns and line items in our alpha environment highlighted the gap between our existing capabilities and what we hoped to show. As we feared, many objects in our database were missing the data necessary to visualize pacing and performance.

Over the next few weeks we worked to get more visualizable data into the alpha version of Campaign Monitor. Before we built the tool, new line items and campaigns defaulted to a null start date. Many users did not bother to pick a true start date for items that were intended to start immediately, even when an end date was defined. As a result, for lots of items we could not calculate how many total days were in a flight, much less how far into flight a campaign might be. This prevented us from showing many useful bars, so we decided to change our datepicker UI module to require a start date. In the new module, objects would by default start on today’s date rather than “immediately” or null. We also began to keep track of the first run date for campaigns and line items, which we could use in place of a true start date in a pinch.

Another source of inadequate visualization was related to our flexible object hierarchy. In AppNexus, a parent line item controls the revenue and goals the ad network receives from the advertiser, while child campaigns represent different bidding strategies the campaign managers set up to obtain the advertiser’s performance and delivery goals. Users commonly set budgets and flight dates at only one of these levels to reduce complexity and the potential for error. As a result, certain settings would need to be shared between the levels in Campaign Monitor even though they were stored on different objects in the database. This required a lot of client-side calculations, which we opted to perform asynchronously after page load.

Over time we incorporated most of the visualizations from our prototype into the alpha version of Campaign Monitor and soon we were ready to for users to dive in and try it out. In July we released the tool into a limited availability test with many of the clients who had participated in our early research and user testing phases. The test gave this select group of users exclusive access to Campaign Monitor in the UI, and we asked them to use it and provide feedback by email and phone on a regular basis. Over six weeks we gathered complaints, requests, and ideas from our test participants, and we rapidly incorporated their input into the tool.

Having campaign managers using Campaign Monitor daily brought to our attention all kinds of usability and performance issues and highlighted gaps in the tool that prevented it from being as helpful as we’d hoped. Luckily, we had anticipated the ongoing feedback and had set aside additional development cycles to continue to improve Campaign Monitor. Over the next six weeks we polished the layout and built all kinds of new features: we enabled users to toggle between lifetime and yesterday’s data for more granular insight into delivery trends; we redid the filters section and incorporated a few new ways to filter objects; we refined the alerts and included a new “paused” status to indicate that a setting had made an item ineligible to serve; we introduced new graphs and tweaked the existing ones; we added pagination; and we continued to polish the visualizations and interactions to provide a great experience.

The comments we heard throughout the limited availability test confirmed our hope that we were building something that would make campaign managers’ jobs a bit easier. Over the six weeks, users told us things like…

“That’s HUGE! This is now going to become my go-to.”
“Currently we track everything in-house manually in Excel (it’s like the Stone Ages here) but this could replace what we’re doing with Excel right now. It’ll certainly make our lives a lot easier.”
“I don’t really go into Excel anymore. Not for this stuff anyway.”
“I think this is really nice, it sums everything up.”
“When do you think you’d be able to roll this out?”

Campaign Monitor is the kind of feature that we could keep improving forever, but we wanted everyone who uses AppNexus Console to have access to it as soon as possible. We knew it was time when we felt we had a satisfactory basic tool that could be incorporated into a campaign manager’s workflow in place of their external spreadsheets. And so today, we’re happy to announce the release of Campaign Monitor to all our users. We hope the feedback will keep coming and we plan to incorporate tons of exciting new features into future generations of Campaign Monitor.

Read more about Campaign Monitor from Ari Paparo, SVP of Product on our company blog.

The End Result