Announcing HAProxy 2.8

HAProxy 2.8 is now available, and HAProxy Enterprise 2.8 will be released later this year.

Register for the live webinar HAProxy 2.8 Feature Roundup (also available in French) to learn more about this release.

HAProxy 2.8 contains features that build upon HAProxy's flexibility, performance, and programmability. This version gives you more control over how to tune the load balancer for your use case, while also setting default values to make life easier. Performance enhancements include HTTP compression for requests, a simple way to combine listener shards and thread groups, and built-in OCSP stapling. Anyone using Lua to customize HAProxy will be pleased with the new event framework, queues, and functions for retrieving server information in Lua code.

HAProxy 2.8 cements HAProxy’s reputation as the world’s fastest software load balancer, and extends HAProxy’s versatility for new and experienced users alike, making HAProxy the best choice for delivering modern applications at enterprise-scale.

This release came together through the efforts of all the community members who got involved. A release like this requires feature requests, bug reports, forum discussions, code submissions, QA tests, and documentation! In other words, this project is fueled by people like you! If you're interested in joining this vibrant community, it can be found on GitHub, Slack, Discourse, and the HAProxy mailing list.

How to Get HAProxy 2.8

You can install HAProxy version 2.8 in any of the following ways:

• Run it as a Docker container. View the Docker installation instructions.

• Compile it from source. View the compilation instructions.

Email Alerts With Lua

This syntax for configuring email alerts in HAProxy stays the same in HAProxy 2.8, but the underlying implementation, previously compiled into HAProxy's C code, is now duplicated as a Lua module. As a Lua module, you can more easily edit the code to customize the format of these emails and add extra, relevant information, such as links to operating manuals and on-call personnel. Because Lua is a scripting language, you can edit the file with a text editor, save, restart the HAProxy service, and the changes will be in effect.

HAProxy can notify you by email when important server state changes occur, such as when HAProxy removes a server from load balancing due to failing health checks or adds it back due to passing health checks. The mailers section in a HAProxy configuration lists the addresses of your SMTP mail servers.

Read More: How to Create Lua Mailers in HAProxy

A simple example of a mailers section:

Then, to enable notifications, you would add email-alert directives to your defaults:

To enable the Lua incarnation of this feature, and disable the C implementation, download the mailers.lua file from the examples folder in the GitHub repository and then add a lua-load directive to the global section of your configuration:

New Event Framework in Lua

If you extend HAProxy's functionality by writing a Lua module, that module can now take advantage of the new HAProxy Lua event framework. Lua code can already hook into HAProxy in a number of ways, including:

• as actions that authenticate, validate, or alter requests and responses on the fly

• services that intercept traffic and immediately return a response

• tasks that run continuously in the background

• custom fetches and converters that capture and transform data as it passes through.

The event framework augments all of that by letting you subscribe to events and then invoke a callback function when those events fire.

The email alerts Lua module mentioned before is a good example of how to leverage events. Essentially, there is a new function named core.event_sub(), which lets you subscribe to events that occur across all servers, and also a function named Server.event_sub(), which lets you subscribe to events related to a specific backend server. There's also an unsub() function for unsubscribing from events.

The following event types exist:

Queues in Lua

With HAProxy 2.8, your Lua code can utilize queues, first-in-first-out storage, for transferring data between coroutines. Lua does not provide such a structure on its own.

The email alerts Lua module provides an example of using a queue. It adds alert messages to a queue when a server event fires. Then, a separate background task fetches messages from the queue to send them as email notifications. Central to this feature is the core.queue() function, which returns a Queue object.

Lua Functions for Server Information

Custom Lua modules can leverage new functions for accessing information about backend servers. The table below gives a brief description:

A Timeout for Lua Execution

The directive tune.lua.burst-timeout squashes the problem of Lua scripts that take too long to execute due to inefficient code. It kills them after a period of time, with a default period of one second. This safeguards the load balancer from locking up while waiting for a script to complete and could help locate custom Lua modules in need of optimization. You can disable this by setting the value to zero.

Disable the Default Resolvers Section for the HTTP Client

You can now disable the creation of the default resolvers section, which could be helpful to anyone not making use of Lua modules that use the HTTP client.

Since version 2.6, HAProxy has generated an invisible DNS resolvers section to support domain name resolution for the Lua HTTP client. This default resolver parses the /etc/resolv.conf file on Linux to give generic DNS resolution support to Lua modules that use core.httpclient() for making HTTP requests.

You can disable this by adding the global configuration option  httpclient.resolvers.disabled:

HTTP Compression of Requests & Responses

HAProxy 2.8 expands its HTTP compression feature so that you can now compress requests in addition to responses. Request compression aids the transfer of large data sets to backend servers, which can save you money in cloud environments where you pay for data sent between regions. The new syntax has you specify whether you want to compress requests, responses, or both. In the example below, we want to compress JSON sent to the server, so we configure HAProxy to compress requests that have a Content-Type header set to application/json.

The backend server must have the ability to decompress the messages.

Signing Algorithms for TLS

You can now specify which signing algorithms HAProxy supports for digital signatures used in TLS handshakes and client certificate authentication. When using client certificate authentication, the client sends a certificate to HAProxy as proof of identity. The certificate itself will have been digitally signed by a CA your organization trusts. Additionally, the client will use its private key to sign a CertificateVerify message, proving that it is the rightful owner of the certificate. Digital signatures also come into play during the TLS handshake, when the server (HAProxy) signs the ephemeral key exchanged with the client.

Two new global directives allow HAProxy to tell the client which cryptographic algorithms it supports when producing digital signatures, enabling you to disallow weak cryptography such as SHA-1, when supporting older clients is not a concern.

Additionally, you can set supported algorithms on individual bind lines in frontend sections by using the client-sigalgs and sigalgs arguments.

Default ALPN Values

When you do not set the alpn argument on a bind line, HAProxy 2.8 now sets a default value of h2,http/1.1 for you. Also, bind lines that use the QUIC protocol will get a default ALPN value of h3.

ALPN (Application-layer Protocol Negotiation) is a TLS extension that lets a client and HAProxy agree on a mutually supported protocol to use for communication. It's how a client can switch from HTTP/1.1 to HTTP/2 without switching to a different port. Exchanging this info during the TLS handshake saves a network round trip.

The new default values tell the client that HAProxy prefers HTTP/2 and HTTP/3, but can fall back to HTTP.1.1 if the client doesn't support them. You can also disable ALPN by adding the no-alpn argument to the bind line.

OCSP Stapling

HAProxy 2.8 includes improved support for OCSP stapling, making it a built-in feature of HAProxy. Previously, this functionality could be achieved only through custom scripts that called the HAProxy Runtime API's set ssl ocsp-response method.

OCSP (Online Certificate Status Protocol) provides an efficient way for clients to check the revocation status of TLS certificates. HAProxy supports OCSP stapling wherein it sends revocation status information to clients preemptively. To configure this feature, you would set ocsp-update to on on a certificate line in a CRT list to enable automatic updates to that certificate's OCSP file.

Two new global directives give you control over how often OCSP responses are updated. The tune.ssl.ocsp-update.mindelay directive sets the minimal interval between updates, while tune.ssl.ocsp-update.maxdelay sets the maximum interval.

There are also new Runtime API commands: update ssl ocsp-response for requesting an updated OCSP response immediately and show ssl ocsp-updates for viewing the expected time of the next update and the status of the last update.

The show ssl ocsp-response method got a refresh that allows you to view OCSP response data in text format or as a base64-encoded string.

WolfSSL Support

A lot of progress has been made to support WolfSSL as a TLS encryption library. WolfSSL has built-in support for the QUIC protocol and shows good performance. HAProxy engineers continue to work with WolfSSL engineers to iron out missing features, and you can expect a fully working version in the near future. Check the install guide in GitHub to learn details about which WolfSSL features are complete and which are in progress.

HTTP Forwarded Header

A new option adds a Forwarded header to requests, making it simple to pass the client's source IP address and information about the HAProxy server that handled the request to backend servers. 

The option forwarded directive enables the header, and you can configure whether to set the header's fields automatically or define expressions that generate the values in custom ways. For example, you could obfuscate the for field, which contains the client's source IP address, for privacy.

In addition to generating a Forwarded header, this version adds new converters for validating and extracting information from a Forwarded header found on the incoming request. The header is defined in RFC 7239, which explains the naming convention:

New converters:

Set the Number of Sticky Counters

It is now possible to change the number of stick table sticky counters available within your HAProxy configuration.

HAProxy's in-memory storage, called stick tables, have a finite number of slots in which to store connection tracking information. Historically, that number defaulted to 3 and was a compile-time constant, which you could not change later. Now, just set the tune.stick-counters directive in the global section. Note that the higher the number, the more memory HAProxy will consume per HTTP request.

HTTP Actions

HAProxy 2.8 gains the following http-after-response actions:

Additionally, all action directives now have an sc-add-gpc action, which increments a general-purpose counter. General-purpose counters are useful for defining custom counters stored in stick tables. For example, you could count each time a client attempts to log into your website as a way of detecting login brute forcing.

Tuning HTTP/2 Performance

New and updated global directives help you get the best performance when using the HTTP/2 protocol. 

Defaults for Listener Sharding

HAProxy 2.5 introduced a shortcut called shards for defining multiple listeners (bind lines) in a frontend. The shards argument on a bind line replicates the listener so that threads can pull work from multiple listeners for the same application at once, reducing contention. This is especially useful on servers with many cores/threads in order to make the best use of multithreading.

In HAProxy 2.8, a new, global directive called tune.listener.default-shards helps you set the optimal number of listener shards. Instead of using the shards argument on a bind line, this global directive sets the number of listeners to match either the number of threads or the number of thread groups. Or, you can set it to one listener per process. The per-thread-group option is the default. 

If you want to override the global default, you can set the mode directly on the bind line. When you don't set the shards argument to a number, it expects a value of either by-thread or by-group.

Thread groups, which became a stable feature in HAProxy 2.7, allow you to assign threads to groups. For example, two groups, each with 64 threads, and then assign each group to handle incoming connections on a listener. Having control over the number of threads in a group and then setting tune.listener.default-shards to allocate a listener for each group provides a streamlined way to optimize for multithreading.

Other Performance Tuning Options

This version adds or updates directives for tuning HAProxy's performance on your system.

• The cpu-map argument on a bind line can now define multiple, comma-separated CPU sets (pairings of CPU number and thread ranges).

• The global directive tune.listener.multi-queue gives you more control over how work is distributed among threads assigned to a listener. The default value, on, sends work to the least busy thread. Setting this to fair distributes connections to threads in a round-robin rotation, which suits traffic consisting of short-lived connections.

• Each thread is allocated an amount of memory that will be kept hot in the local cache. Although the default amount of 512 KB should suit most, the new tune.memory.hot-size global directive lets you change it if needed.

• The thread argument on a bind line now accepts multiple, comma-separated sets of threads allowed to process incoming connections for the listener, instead of just a single set.

• An experimental, global directive called tune.disable-fast-forward allows you to disable data fast-forwarding. You can also set this using the HAProxy startup argument, -dF. Use this only when directed by project developers to do so.

Fetching URL & Body Parameters

The fetch method for getting a parameter from an HTTP request body, req.body_param, now accepts an additional parameter, the letter i, for finding the parameter by name but ignoring the case. Similarly, the url_param and urlp_val fetch methods also accepts the i parameter for matching a URL parameter name while ignoring the case. This can be the crucial ingredient for making HAProxy's behavior consistent with a backend server's behavior in how they both interpret the same parameter.

Also, the new param converter extracts name-value parameters from a string. It offers an alternative to using the urlp fetch, but with the ability to set the expected delimiter to something other than an ampersand.

Getting the Round-Trip Time

Two new fetch methods, bc_rtt and bc_rttvar, return the round-trip time between HAProxy and a backend server, which can be helpful for troubleshooting. You can configure these to return the value in milliseconds or microseconds.

Runtime API Updates

HAProxy's Runtime API received updates to several of its methods:

• When using the Runtime API interactively via the prompt command, prompt now takes an optional parameter, timed. This causes the process's uptime to be displayed as part of the prompt. This timer can alert you to the fact that the process didn't reload when you expected it to and is running an older configuration, for example.

• The show quic method that was introduced in version 2.7 displays information about all active QUIC frontend connections. You can now change the display format to either oneline or full, depending on how verbose you want the output to be. With the QUIC protocol being implemented entirely in userland, as opposed to lower in the network stack, having built-in debugging tools is crucial.

Read More:

• Dynamic Configuration With the HAProxy Runtime API

• Enable QUIC Load Balancing on HAProxy

Preprocessor Directives

Preprocessor directives enable you to change what goes into your HAProxy configuration based on conditional logic. For example, if you wanted to include a set of configuration lines only if the version of HAProxy supported them, you could add an if statement that checked for that.

HAProxy 2.8 adds two new preprocessor functions:

• The strstr function returns true if a given string is found within another string.

• The enabled function returns true if the passed in option is enabled at runtime. Options you can check for are: POLL, EPOLL, KQUEUE, EVPORTS, SPLICE, GETADDRINFO, REUSEPORT, FAST-FORWARD, and SERVER-SSL-VERIFY-NONE.

Let's Encrypt

The Let's Encrypt client, acme.sh, provides a simple way to integrate with Let's Encrypt for getting TLS certificates. As a UNIX shell script, it has no dependencies on programming languages like Python. As a part of this release, HAProxy now integrates with acme.sh to enable populating TLS certificates in HAProxy. We will publish a separate blog post soon on how to configure this integration.

Contributors

HAProxy 2.8 has been made possible by a long list of contributors, all providing invaluable work and collaboration. Contribution to the project comes in all forms, from design choice discussions, bug reporting, testing development releases, and maintaining documentation, to assisting users on both Discourse and the mailing list, classifying Coverity reports, reviewing patches, and contributing code. With a contributor list too long to include here, please know that the community appreciates each and every one of you who made 2.8 possible!