This post is part of our Summer Intern Blog Series! Each of our most recent class of interns wrote a blog post on their biggest accomplishments and lessons of the summer. This installment comes from Mason, a rising senior at Harvard University who studies computer science and MBB (Mind, Brain, and Behavior). He is a music nerd, tennis fanatic, and amateur comedy-writer who hosts a weekly radio show during the school year called “Smoother Things.”
Allow me to introduce the work I did this summer with a quick metaphor.
Meet Jim. He’s tall, broad-shouldered, slightly scruffy and wears his shoulder-length brown hair in a neat ponytail. Jim is an extremely useful guy to know because he’s the man in charge of driving the dump truck. See, whenever employees at AppNexus need something, they simply give Jim a ring and tell him what they want. Dutiful Jim then loads his dump truck up with the desired objects and drives them directly to the employees’ front door.
Jim can handle all kinds of requests, but unfortunately, he has a fatal flaw: if an employee asks for too much, Jim will still do his best to satisfy that query, even if it means the dump truck buckles under the weight of the desired objects. And once the dump truck breaks down, no one else can make requests until it’s been repaired. As you might expect, this downtime causes an AppNexus-wide weeping and gnashing of teeth (read: a very costly dip in efficiency).
This is the problem I was tasked with solving during my ten weeks at AppNexus. In reality, each “phone call” is a request for a specific object or group of objects (made by hitting a specific internal URL) that live on the cache of one of our ImpBus or Bidder machines. Jim is a “dump handler,” the code that contains the logic for carrying out the delivery of said requests for cached information. The dump truck is the buffer (memory space) in which this information is stored for transportation. Lastly, the dump truck’s breakdown is equivalent to one of our machines crashing.
Now, this is a simplified analogy (for instance, there would truly be many Jims, each for a different kind of desired object—one Jim for deals, one for creatives, one for blacklisted IPs, and so on). But it’s a good place to start in understanding the central problem I set out to solve. I was lucky enough to have the chance to approach this issue from two angles, spending half of my internship on each part.
First, I decided to build a mostly front-end application that would streamline the process of making requests and serve as a first line of defense against the dreaded crash. As it turns out, the URLs required to make any given cache dump request are fairly convoluted—instead of easy phone calls to Jim, the requests are more like lengthy messages in Morse code. Thus, I set out to create an app that would allow users to select the type of object they wanted from a drop-down, specify the object’s ID (or select all objects), and receive a correctly formatted JSON response, painlessly. Because my chief goal this summer was to learn, I decided to build the application in Django/Python, which I’d never used before. After a few days of digesting the framework and spec’ing out the project, I started building. One week later, I had the first working version of my application. The tool was bare-bones and only worked for ImpBus dump handlers. But I had a tangible creation in my hands—a truly exciting milestone.
Next, I iterated on this prototype. I sent the tool to employees across engineering who used the dump handlers and asked for their feedback – basically, “what would make your life easier?” Addressing their responses, I added Bidder functionality, the ability to select specific ImpBus/Bidder instances, and sessions (saving past lookups). As for the first line of defense, I needed to determine the handlers that resulted in consistent crashes, so I wrote a script that sent requests to all handlers and carefully tracked those that caused the most trouble. I then added a ban on these handlers so that an employee would have to manually create the request URL (circumventing my application) to generate a “dangerous” request. With each new feature, I felt my comfort with and confidence in working with Django grow.
Then, halfway through the internship, I completely switched gears. I realized the app had reached peak utility and that it was now time to dig deep into our C source code to solve the root problem. With the help of my manager, we arrived at a solution: give Jim a wheelbarrow. If Jim were to deliver a very large request in many smaller loads rather than one large one, there would be no risk of a breakdown. This is called “chunked encoding,” an HTTP mechanism that transfers information in, well, chunks.
So, I started over from square one. Thus began weeks of poring through ImpBus and libevent HTTP source code, carefully stepping through programs with GDB, and leveraging other debugging tools like Valgrind and ASAN to determine exactly how everything related to the dump handlers fit together. I learned so much that even though progress was relatively slow, the work was satisfying: each day meant another technique acquired or another piece of the system understood. Finally, after weeks of exploring numerous promising yet ultimately unsuccessful implementations, I finally arrived at a working solution – and unbelievably satisfying event! I first got this implementation to work with blacklist_ip—proving the method was viable—and then moved on to the city and ad_profile handlers, among others. Ultimately, the goal is to generalize this solution so that we ultimately can use chunked encoding for all handlers. It’s a challenge I would love to tackle if I had another ten weeks.
During my time at AppNexus, I had the incredible opportunity to approach a challenging problem from two disparate angles and absorb twice as much in the process. I was provided with a high level of independence and ownership over what I did every day, making each achievement all the sweeter. Furthermore, my manager was instrumental in providing a perfect balance of guidance throughout—enough help that I kept moving forward and enough distance that I learned as much as possible. And he’s just one example of the scores of kind and whip-smart AppNexus employees I’ve had the pleasure of getting to know. It’s been a phenomenal summer, and I’m not yet ready for it to end.