The Path from Legacy to the Future: How DoubleVerify Transitioned from F5 to HAProxy Enterprise
More than 5 billion requests per day.
Providing services to the largest companies in the world.
12 years of experience in the industry.
About DoubleVerify
DoubleVerify monitors the quality of ad impressions and gives its customers instant feedback so they can ensure that they’re getting the most value out of their advertising campaigns. The platform provides dashboards that show how many impressions were authentic, viewed by real people, and presented in a brand-safe environment, meaning appearing next to suitable content. It also checks websites for fraud, which protects its customers from practices such as stacked and hidden ads, URL masking, and impression laundering.
Results at a Glance
The Challenge
In 2016, with rapid growth, the company began to have concerns about the ballooning price tag of its load balancing infrastructure. Like clockwork, every two years, the team had been compelled to purchase bigger and more expensive hardware, moving from Citrix to F5 and spending millions of dollars in an arms race to support the popularity of the DoubleVerify platform. By 2016, they were already seeing more than 500,000 requests per second. They needed to find a solution that would allow them to continue to scale, but without the astronomical costs.
Other concerns also cropped up. For one, the existing load balancers were inflexible. They could not be installed onto commodity hardware, which meant having no control over the operating system or specs. Designed as one-size-fits-all, the servers were ill-suited for DoubleVerify’s type of traffic, which was made up of many, small messages exchanged quickly. Making configuration changes on the fly to the hardware servers was impossible.
Another problem was that the Citrix and F5 load balancers lacked the operational metrics the team needed. That meant limited insight into their request traffic and the inability to see trends. Without this, they could only make assumptions about their infrastructure. They needed accurate, timely data.
Wally Barnes III and Oren Alexandroni, during their HAProxyConf 2019 talk
The Objectives
If they were to replace the servers, their wish list was this:
Scalability. Any new technology would have to match the company’s growth in a cost-efficient way. In fact, they had already set their sights on supporting a billion requests per day. At the rate they were going, they would reach that number within a year. The system would have to accommodate frequent configuration changes and integrate well with their CI/CD process.
Speed. Collecting data in real-time is critical to the success of the DoubleVerify platform, and the window for doing so is while an ad is rendering. Therefore, the flow of data from the client, through the load balancers, and to the backend servers must happen without slowing down the web page.
Observability. They needed access to metrics and logging so that they could make better decisions about their infrastructure using timely and accurate data. If successful, they could proactively build-out capacity, reduce troubleshooting time, and optimize their applications. The icing on the cake would be to find a load balancer that could be reconfiguration on the fly, without downtime.
The Solution
DoubleVerify chose HAProxy Enterprise to replace their F5 VIPRION load balancers. Instead of needing to buy a few powerful, but very expensive, servers as they had done in the past, the team opted for horizontal scaling, which means employing cheaper, commodity servers and spreading the load across them evenly. Horizontal scaling is more cost-efficient in the long run and it can support a much larger volume of traffic because new nodes can be added as needed.
They tuned HAProxy Enterprise using its multi-process and CPU mapping features. The former lets you run multiple HAProxy processes under a parent process. The latter lets you pin specific work to each of those processes, dedicating them to HTTP processing, HTTPS processing, frontend work, and backend work. This optimizes those tasks and makes it easier to pinpoint problem areas. Other features, such as Keep-Alive for reusing existing connections and leveraging the Runtime API to enable and disable servers on the fly, further optimized the load balancing tier.
DoubleVerify opted for horizontal scaling, which means employing cheaper, commodity servers and spreading the load across them evenly
The team gained better observability over their traffic. HAProxy Enterprise reports verbose logging and metrics, which the DoubleVerify team use to guide their decision making. “When we were behind F5, we didn’t have insight,” said Wally Barnes, Senior Systems Engineer. “It’s amazing when you pull open the curtain.”.
The Results
DoubleVerify reached its goal of processing a billion requests per day and in 2019, they were seeing five times that amount. With HAProxy Enterprise, they are confident that they could accommodate even more. By switching to HAProxy Enterprise, DoubleVerify also reduced their costs to a tenth of what they were paying before.
Their new load balancing tier gives them redundancy, so their engineers can add or remove backend servers without affecting customers. The load balancers themselves can be upgraded and maintained without downtime, and the configuration files are managed across DoubleVerify’s global network using automation tools like Ansible. Their move from legacy, hardware load balancers to the HAProxy Enterprise software load balancer helped support the meteoric rise of their brand management platform.
Interested to learn more about HAProxy use cases? Explore our Success Stories page.