Curl Up and Test: Putting the “Real-Time” in Real-Time Platform

Leave a comment

This post is part of our Summer Intern Blog Series! Each of our most recent class of interns wrote a blog post on their biggest accomplishments and lessons of the summer. This installment comes from Emma, a rising senior at Columbia University majoring in Computer Science and minoring in Psychology. She loves baking, dogs, and movies.

During my summer at AppNexus, I worked on developing OCD!

Now, for those of you unfamiliar with AppNexus’ love of backronyms, do not be alarmed. I did not work on developing Obsessive Compulsive Disorder. Rather, I was developing the new Object Change Delivery system (OCD) as part of the Real-Time Platform (RTP) API team.

When an AppNexus client creates or changes an advertising campaign, the information needs to be distributed throughout the AppNexus platform. Currently, when requests enter the API, they are stored in a persistent database, and a system called Batches delivers them to our ad serving services. The current system faces several challenges including batch stalls, a single shared database, and differing validation standards. OCD is designed to be an improved way of capturing and processing RTP object changes and distributing them into the RTP apps that are involved in serving advertisement requests. In other words, OCD allows our system to process changes to campaigns more efficiently.

My overarching summer project was to help develop, test, and expand OCD. My work was divided into two main subcategories: OCD-Core – working on adding features and improving the core OCD pipeline, and OCD-Conversion – writing the code to enable new object sets to be brought across from Batches and processed through OCD. Among other things, I improved the algorithm to update posted objects in the OCD-Store, added the capability to post multiple objects in a list to OCD, added metrics, and optimized our object conversion code.

A critical part of the development process is ensuring quality through testing. OCD will be a lynch pin in our flow, so a problem here is a problem for everyone. Testing ensures that we catch any bugs within our code and that new updates or optimizations still produce the same, desired end results as the old code. For example, when we improved the algorithm to update objects currently stored in the OCD-Store – a database used to store all non-deleted OCD supported objects in production – we needed to test to make sure that the end state of the OCD-Store was still identical to the previous algorithm’s output. An integral part of my project was to create a local testing suite that would allow us to test the entire OCD pipeline. This means testing that the initial object posted to OCD is properly validated and transformed before being inserted into the OCD-Store, and then correctly distributed to the RTP Apps like Impbus.

With a simple initial request of “add tests to verify insertion mechanisms,” I was given the opportunity and freedom to design and implement the test suite. As a new system with no existing means of systematically and automatically testing OCD features, I was working with a blank canvas. As I began to design and discuss with my manager what we wanted to achieve with the test suite, the scope quickly grew. We discussed the ultimate desired end goal of end-to-end testing from our client facing API through OCD and ultimately to the RTP apps.

One of the most important things I learned this summer is the difference between the deliverable and the ultimate desired outcome. Having the broader picture in mind is key, and in a perfect world, the deliverable and the desired outcome would be the same. However, when starting a new project from scratch, focusing first on a smaller scope is critical to produce a working program. Programming is an iterative process. The first step should never be the last. It is about building the initial framework and then iterating and developing the program from there. The initial job of producing an automated test suite to verify our insertion mechanism into OCD-API is where I started.

Each test consists of a series of requests sent to the OCD-API with a JSON payload that is then transformed in the App Datamodel Worker (ADW), inserted into the OCD-store, collected by OCD-Collector, and finally uploaded to an Impbus. There are several independent services that run within OCD, each with a distinct role. To keep these services isolated from each other during testing, the test suite uses Docker Compose to coordinate multiple separate Docker containers. Working with Docker for the first time this summer has been a great experience. It is perfect for testing because it is quick and easy to bring up and down an entire development environment that consists of multiple interacting services, each running in a configurable environment. More importantly, it ensures repeatability of our tests. Multiple developers on different machines can run the same series of tests with the confidence that they will all receive the same result regardless of their system-specific configuration. With the environment setup, running the tests is as a simple as executing a script.

Now, the summer is nearly over, and I have built on and added to my original testing framework to support newly-added features and objects brought across to OCD. From the initial build, which could only run a small set of tests for a single, hardcoded top-level object, it has grown to easily support running tests on multiple tests object sets and testing for a wider range of features. The suite has been by far the most important and useful tool I have used to validate my own work as I have altered and optimized code. I can now launch an entire testing environment by running two simple scripts to test my work, whereas previously I’d have had to manually set up, curl, and compare output. This new level of efficiency has allowed my team and I to have caught bugs that would otherwise have been missed due to human error. In that way, I have become a demonstrably more productive programmer as a result of this project.

Filed under AppNexus Updates, Intern Blog.

Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *