A major outcome of IT/Ops evolution is realizing the importance of a reverse proxy, such as HAProxy, in achieving modern application delivery.
This is the first post in a series about HAProxy’s role in building a modern systems architecture that relies on cloud-native technology such as Docker containers and Kubernetes. Containers have revolutionized how software is deployed, allowing the microservice pattern to flourish and enabling self-healing, autoscaling applications. HAProxy is an intelligent load balancer that adds high performance, observability, security, and many other features to the mix.
Over the past few years, countless changes have left their mark on the IT/Ops world. Organizations now benefit from on-demand cloud infrastructures, container technology, microservices, and continuous delivery. One major outcome of all of this is those reverse proxies are now recognized as a critical element when it comes to achieving modern application delivery. That’s because they form the bridge from traditional, inflexible infrastructure to dynamic, distributed environments where reverse proxies provide routing, security, observability, and other key capabilities.
What is a reverse proxy? A reverse proxy, in essence, receives requests from clients as they enter your network and routes them to application servers, databases, and other services. Reverse proxies serve several critical functions above and beyond simply routing traffic. They often include features such as load balancing, SSL termination, compression, and can act as a shield against DDoS and application attacks.
HAProxy is a fast and widely used reverse proxy. HAProxy was first released in 2001 by Willy Tarreau. Since then, it has evolved to meet the most prevalent use cases for modern applications while still maintaining its high performance and strict adherence to clean code. It’s one of the most battle-tested pieces of software out there and is used by countless organizations around the world.
In this blog post, you’ll see examples of recent trends in the industry and learn how HAProxy is being shaped to accommodate these changes.
#1 Run Anywhere
The first requirement for a reverse proxy in today’s age is that it can run anywhere. IT operators often run multiple layers of proxies. For example, a proxy may be placed at the edge of the network as a TCP-based load balancer. From there, it can direct incoming traffic to the correct region or zone. This type of proxy may be a hardware appliance or run as software and is often used to load balance another layer of reverse proxies. Beyond this point, there may be proxies that act as API Gateways and/or HTTP-based load balancers.
With the rise of containers and Kubernetes, proxies are now used as part of Kubernetes Ingress Controller implementations and hook directly into Kubernetes to watch for changes and adjust routing rules on the fly. Also, within a container network, there may be a service mesh, which is a web of services interconnected by reverse proxies.
A modern proxy should excel in a wide range of environments so that the same technical skills can be used across a technology stack. HAProxy’s run-anywhere philosophy is important for adapting to the layered architectures of modern systems. It is built to run on Linux and can run on commodity hardware, in a virtual machine, in a Docker container, or as a Kubernetes Ingress Controller. For example, the HAProxy Kubernetes Ingress Controller runs as a service inside a Kubernetes cluster and routes traffic based on rules you can define in Kubernetes YAML files. It automatically detects new pods and has the power to terminate SSL, rate limit requests, and whitelist IP addresses.
#2 High Performance
In a modern infrastructure stack, a request may travel through multiple proxy layers. That means that the latency introduced by each proxy is compounded. What may have been an acceptable amount of delay when running a single proxy becomes a magnified source of pain. That’s why it’s crucial that every component runs as efficiently as possible.
In HAProxy, performance is an important metric that determines how new features are added and whether they pass the bar. In fact, odd-numbered versions tend to focus on bug fixes and tuning performance to meet a certain threshold. Recent updates optimized HAProxy’s internal scheduler, overhauled the way that HTTP messages are represented, and defaulted HAProxy to run across multiple threads. Check out some benchmarks of HAProxy compared to other proxies.
When HAProxy load balances traffic across a cluster of backend servers, it can improve the overall response time for the client. Its various load balancing algorithms allow you to select the optimal server to handle each request. They include round-robin, fewest connections, and consistent hashing, among others. In Test Driving Power of Two Random Choices Load Balancing, we measured the requests per second, response time, and maximum server load of five load balancing algorithms.
#3 Configuration Made Easy for Humans & Machines
Proxies are responsible for a lot and that can lead to some complicated configurations. Also, when a proxy manages more services, its configuration grows in lockstep. The answer to this problem is, of course, automation. A good proxy can be managed through a number of tools, depending on the environment. However, at the same time, it should be possible (and easy) for an actual human being to open the file and read it.
The HAProxy configuration file, typically found at /etc/haproxy/haproxy.cfg for HAProxy or /etc/hapee-2.0/hapee-lb.cfg for HAProxy Enterprise, is the brain of the software. This file manages nearly all of the runtime behavior. It boasts an impressive list of capabilities, nearly all of which you can toggle on or off and fine-tune. When HAProxy starts, it validates the file, displays errors, and avoids loading the configuration if it’s incorrect. Consider this example that we put together to showcase the new features in the 2.0 and 2.1 releases. Many complex behaviors are included, but the file is still easy to read.
The file can be created with HTTP commands by using the HAProxy Data Plane API. There’s also a Go library that lets you do this from code, which is the strategy used by the HAProxy Kubernetes Ingress Controller. Or, it can be built up through automation templating tools like consul-template.
You can read about the fundamentals of the HAProxy configuration file in our blog post, The Four Essential Sections of an HAProxy Configuration.
#4 Zero-Downtime Configuration Changes
In dynamic environments like Kubernetes, backend servers may be created and destroyed frequently. Whenever that happens, the proxy must refresh its list of endpoints. A modern proxy must support configuration updates without dropping connections, even when there are many active clients and frequent configuration changes. HAProxy has always been good at this. In fact, by using HAProxy’s server-template feature and the Runtime API, it’s possible to add and remove servers dynamically and avoid reloads altogether.
Historically, HAProxy spun up a new process and switched traffic over when a reload was required, allowing the old process to finish up existing connections. However, for reasons described in Truly Seamless Reloads with HAProxy, a tiny window existed between the switch from the old to the new process, during which time a client might try to connect and not succeed. The problem is more obvious in busy, highly volatile microservice environments where the configuration is reloaded many times per second.
HAProxy version 1.8 introduced several important changes, but one that relates particularly to configuration reloads is hitless reloads. Hitless reloads allow the HAProxy process to pass active file descriptors, which hold the listener sockets, to a new process when the old process is shut down. This hand-off means that zero connections are dropped when HAProxy is reloaded.
The HAProxy Runtime API was updated to manage this transfer between processes. You can enable this feature with one line: Add expose-fd listeners to a stats socket line in the global section of your configuration, as shown:
| stats socket /var/run/haproxy.sock mode 600 expose-fd listeners level user |
Now, you can have thousands of clients connecting while you reload the configuration multiple times per second without dropping a single client. You can read more about it in our blog post, Hitless Reloads with HAProxy – HOWTO. With HAProxy Enterprise, you get the Update Module, which allows HAProxy to update the contents of ACL and Map files without a reload so that changing settings like routes or URL rewrites don’t require a reload either.
#5 Dynamic Service Discovery
Organizations adopt DevOps practices because they want to accelerate the process of releasing software. That naturally leads to applications being deployed more often, which causes the application’s IP addresses to change frequently when using containers and immutable infrastructure. The same can happen as servers are automatically scaled up or down on-demand.
Instead of manually editing your HAProxy configuration, you can use server templates and DNS service discovery to detect and add server definitions automatically. When creating a server template, specify the number of server slots that should be created when HAProxy starts. Then, when you scale up your infrastructure, these server slots are filled in and join the load-balancing rotation. Since slots for those servers have already been allocated, no reload is required.
Server templates are ideal for dynamic container environments; They allow your system to expand and contract without human intervention, which lessens toil and reduces errors. Typically, templates are combined with DNS service discovery, in which changes to DNS records trigger automatic changes to the templated server definitions. You can read more about it in our blog post, DNS for Service Discovery in HAProxy.
#6 A Modern REST API
Automation commands a towering influence over IT these days. People want to spin up components with the click of a button—or at least the typing of a command. APIs accessible via well-known protocols like HTTP are definitely the way to go.
The HAProxy Data Plane API was introduced with HAProxy 2.0 and gives you the ability to configure nearly every aspect of HAProxy through a modern HTTP REST interface. Commands can be grouped into transactions and then applied as atomic actions, avoiding any half-states in the configuration. Each change is versioned, which catches collisions between two people making updates at the same time. Also, before changes are applied, the configuration is checked for syntax errors in case a manual edit of the file was made. All in all, several mechanisms are ingrained into the API to ensure that the configuration stays valid.
The Data Plane API is a project maintained separately from HAProxy. That makes it possible to evolve at a natural velocity for the project. Since its introduction, it has seen several valuable contributions and is evolving rapidly. You can learn more about the API by reading our blog post, The New HAProxy Data Plane API. Or, find its GitHub repository here.
#7 Real-Time Observability
When you combine vast, distributed systems with automatic scaling and dynamic server changes, you get a system that is much more complex than a traditional monolith application. You need a way to maintain a watchful eye over the maelstrom.
Modern proxies must provide superior introspection into the routing and health of upstream services. Detailed logging of requests and responses is needed to troubleshoot network problems and audit the links between microservices. In addition, metrics that track the volume of requests, the number of clients waiting in queues, and the percentage of good vs bad responses are needed to build reports, set up thresholds for alerts, and plan for future server capacity.
HAProxy provides detailed logs that include:
detailed timing data for each phase of each request
counters that track queue size, concurrent connections, etc.
information about how the request was routed
HTTP headers, status codes, cookie values, etc.
termination statuses that show how a connection ended
In addition to logs, HAProxy also exposes many metrics. You can view them with either the built-in dashboard, which is called the HAProxy Stats page, or by calling the HAProxy Runtime API. Metrics can be formatted as CSV or JSON for easy parsing. HAProxy also supports Prometheus metrics and is well integrated with metrics scraping tools like Telegraf or Collectd. Metrics can then be visualized in Grafana to display rich, time-series data.
Read our blog posts, Introduction to HAProxy Logging and Exploring the HAProxy Stats Page to learn more about HAProxy’s observability. With HAProxy Enterprise, you get access to the Real-Time Dashboard, which allows you to aggregate data from across a cluster of active-active load balancers.
#8 Focus on Security
A modern proxy must be secure. As the conduit that traffic flows through, it needs to have mechanisms in place to detect and stop security threats. No matter where they’re placed within your infrastructure, proxies become critical security checkpoints.
HAProxy is secure from the ground up. After it starts, it can immediately drop root permissions and run as a non-privileged user. That way, should a hacker gain access to the proxy’s running process, they would have access to very little beyond that. It has built-in support for running inside a Linux chroot; It also doesn’t access the filesystem after it starts, preventing a whole class of vulnerabilities. HAProxy also supports SSL/TLS and can verify client certificates for authentication and authorization.
More advanced security capabilities, including rate limiting, analysis of a client’s behavior across requests, whitelisting/blacklisting IP addresses, and the ability to tarpit attackers, can be enabled with the flexible configuration language. These complex features demonstrate how the thoughtful construction of the language’s syntax was well worth the effort.
Read more about a specific use case, HAProxy’s bot protection, in our blog post, Bot Protection with HAProxy. Or, learn about how HAProxy mitigates DDoS attacks in our blog post, Application-Layer DDoS Attack Protection with HAProxy.
With HAProxy Enterprise, there are more advanced protection mechanisms, including the HAProxy Web Application Firewall, which repels application-layer attacks. The HAProxy Enterprise WAF supports three modes: SQLi/XSS only, ModSecurity mode, and whitelist-only mode, so you can choose the level that’s right for you.
#9 Extensibility
No environment is exactly the same as another. Organizations build complex systems that often need custom add-ons and solutions. Since proxies are the glue that binds it all, it’s helpful if some functionality can be incorporated there, especially in order to add functionality at the edge of a network or between services in a service mesh.
You can write Lua scripts that extend HAProxy’s runtime behavior. Use this to add new fetches, converters, actions, services, or tasks. Read our blog post, 5 Ways to Extend HAProxy with Lua, to learn more.
You can also connect HAProxy to external programs that inspect traffic as it passes. The Stream Processing Offload Engine lets you add filters in your HAProxy configuration that hook into these programs. Read our blog post, Extending HAProxy with the Stream Processing Offload Engine, to learn more.
Conclusion
In this blog post, you learned how HAProxy is constantly evolving to meet the demands of modern IT. As an open-source, software-based proxy, it thrives in various environments. It has methods for adapting to dynamic server changes, can reload without dropping a connection, is serious about security, and provides verbose logs and metrics. The changing IT landscape continues to shape it, driving its evolution above and beyond other types of proxies.
Want to stay up to date on similar topics? Subscribe to this blog! You can also follow us on Twitter and join the conversation on Slack.
HAProxy Enterprise combines HAProxy Community, the world’s fastest and most widely used open-source load balancer and application delivery controller, with enterprise-class features, services, and premium support. It is a powerful product tailored to the goals, requirements, and infrastructure of modern IT. Contact us to learn more and sign up for a free trial!
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
