Announcing HAProxy 2.5

Watch our webinar to learn more about this release.

HAProxy 2.5 is now available! It adds improvements to a number of areas including better usability around setting variables, more descriptive error reporting and logging, and enhanced HTTP and WebSocket support. The HAProxy Runtime API has expanded its coverage of SSL-related commands and now includes the ability to add and remove CA files and revocation lists on-the-fly. The Lua integration now supports an HTTP client for initiating HTTP requests, and there is early support for QUIC and HTTP/3.

This release was truly a community effort and could not have been made possible without all of the hard work from everyone involved in active discussions on the mailing list and the HAProxy project on GitHub.The HAProxy community provides code submissions covering new functionality and bug fixes, documentation improvements, quality assurance testing, continuous integration environments, bug reports, and much more. Everyone has done their part to make this release possible! If you’d like to join this amazing community, you can find it on GitHub, Slack, Discourse, and the HAProxy mailing list.

In the following sections, you will find a full list of changes included in this version.

Upcoming QUIC and HTTP/3 Support Milestone

You can receive HTTP/3 requests and process them on your HAProxy instance or forward them to an HTTP/1, HTTP/2, or FastCGI backend server.

The feature is in the pre-experimental stage, meant only for development. Error processing is still very limited.

You can build HAProxy with QUIC support to get a first glimpse into the inner workings of the functionality, using the QUICTLS SSL library instead of OpenSSL.

Dynamic Servers

Version 2.4 introduced the ability to create servers on the fly using the Runtime API, with some limitations (no check, track, slowstart, error-limit, ssl, or observe keywords). In contrast, this version provides full support, notably adding support for health checks and SSL/TLS. You can also remove any server via the API.

You can, for example, manage the lifecycle of a new server as follows:

1) Access the socat utility to connect to the Runtime API, then enter the prompt command:

2) Switch to experimental mode:

3) Dynamically add the dynserv server to the be_app backend:

4) Enable the server:

5) Enable health check for the server:

6) Put the server into maintenance mode:

7) Delete the server:

Usability

HAProxy 2.5 improves usability in a number of ways.

Append strings to a variable easily

The set-var-fmt action accepts a log-format string instead of a sample expression.

You can now create a variable composed of multiple values through a one-liner:

Before this, you would have had to write at least four lines of code to achieve the same result  in order to set multiple variables and then concatenate them together:

The tcp-request connection directive now supports set-var-fmt and set-var actions, which allows you to set a variable when a client connects. These variables can be scoped as proc or sess to be process-scoped or session-scoped variables, respectively.

Request rules in default sections

You can now place tcp-request and http-request rules in named defaults sections and have them processed by frontends and backends that explicitly depend on them. You can thus declare generic rules in defaults sections while keeping section-specific rules in frontend or backend sections.

In the snippet below, we’ve moved an http-request redirect rule that redirects HTTP to HTTPS to a named defaults section, which the frontend inherits. Any other frontends that inherit from this defaults section will also redirect HTTP to HTTPS.

Dark mode for the statistics page

To reduce eye strain, the statistics page now supports a dark theme. A dark or light theme is chosen based on your operating system preferences (it uses the prefers-color-scheme CSS media feature to do this).

Improved startup error reporting

Error reporting at startup was improved. It now takes into account such errors as bind errors about Maximum Segment Size per interface.

Simplified HTTPS log format

The new option httpslog provides additional functionalities as compared to the existing option httplog. It provides information about the SSL/TLS frontend connection, such as which ciphers were used, errors, etc.

Number of entries in the summary

To help you decide whether to dump all ACL or map keys, the show map and show acl Runtime API commands now report the number of entries on the summary output.

Better control over threads

The outdated process keyword is now deprecated on bind lines. The more convenient and durable thread keyword replaces it.

Recall that HAProxy’s nbproc directive let specify how many child processes to run under HAProxy, which helped to scale out the load balancer’s capacity. The process argument on the bind line existed to let you fine tune exactly how each of these child processes would be utilized, allowing you to assign them to specific bind lines.

When nbthread was introduced, which allows you to run multiple threads within a single HAProxy process, it solved many of the drawbacks inherent in a multi-process model by switching to a multithreaded model instead. However, until now, there wasn’t a way to control which threads would be assigned to handle incoming connections to a specific bind line, like there had been with the process keyword for multiple processes. The thread argument fills in that gap.

Let’s say that you have two bind lines, each listening for incoming connections on a unique IP address. Each is meant to serve a different application.

Here, the thread argument assigns a range of threads to each listener, which reduces contention that can occur when many threads compete to handle incoming connections. In essence, we’ve given each listener its own group of threads that are dedicated to it.

In many cases, you can leave all of this up to HAProxy. If you do not set nbthread, HAProxy will set the number of worker threads to match the machine’s number of available CPU cores. If you do not  set the thread argument on the bind line, connections will be distributed evenly to whichever threads have the fewest active connections. Controlling how threads are assigned with the thread argument is useful, however, when there are many threads and contention is likely to be high.

The thread argument also allows you to assign a thread-group to a listener, which will make it easy to assign groups of threads, but this feature is nascent and currently, you can create only one thread-group in HAProxy.

Better management for high connection rates

When HAProxy runs in multithreading mode, which is the default, each thread creates its own scheduler that pulls connections off of bind lines (listeners) and handles them. On systems with many threads, wherein all schedulers compete to handle new connections on the same listener, contention can cause performance to drop.

To keep performance at its highest level on operating systems that support several listeners on the same socket address, you can manually replicate bind lines in the frontend section. Threads will distribute the work of handling incoming connections across both listeners, which both serve the same application.

You can use the new shards option as a shortcut syntax. This option automatically replicates listeners any number of times and evenly distributes them among available threads to provide the best performance.

The previous snippet is equivalent to:

Graceful stop of all proxies

A new global grace directive replaces the deprecated per-proxy grace directive. The new directive keeps all proxies operational for some time after a soft stop.

Frontends which are monitored by an external device such as a Layer 4 load balancer or a dynamic router remain up for the time needed by the device to detect the failure.

Observability

HAProxy gives you best-in-class observability into the traffic that passes through it. It gives you statistics on the health of servers. It measures error rates, tracks queue lengths, and counts connections. 100 measurements display on the HAProxy Stats page or are fetched by the Runtime API’s show stat command.

Display what is being processed for each session

The show sess all command now displays for each session what file, line numbers, rules, and filters are currently processed.

Display statistics about a stopping process

You can collect the last statistics of a stopping process. The statistics page now always lists stopped proxies during reloads. It only overlooks internal proxies.

Display free memory in thread-local caches

The show pools Runtime API command displays what part of the used value represents free memory in thread-local caches.

Retrieve backend connection errors

You can easily troubleshoot failed connections, even on backend servers you do not manage directly: new bc_err and bc_err_str sample fetch functions help retrieve backend connection error codes and strings.

Log connection-level information even in case of a handshake error

You can log some of the available connection-level information even in case of handshake error through SSL sample fetch functions that keep a reference to the SSL context.

You can get actionable information by using a combination of frontend and backend SSL fetch functions such as:

• ssl_bc_err and ssl_fc_err, which returns an error code,

• ssl_bc_err_str and ssl_fc_err_str, which returns the description of the error, and,

• some connection-level fetch functions such as fc_err and bc_err.

Display version information and enabled options

You can easily check the version and enabled options through a new -cc command-line option. This option allows HAProxy to execute an expression made of predicates.

Refer back to the HAProxy 2.4 release, which introduced pre-processor conditionals that let you check whether a feature was enabled, that the version of HAProxy was at least some minimum, or that an environment variable was set. Now, you can perform those same checks before starting HAProxy by using the -cc flag.

Below, we check whether the Prometheus feature is enabled and then check the command’s exit status. If the status is 0, then the condition is evaluated to true (i.e. Prometheus is enabled).

Enhanced conditions

The configuration file .if, .elif, and endif conditions now support expressions with AND, OR, NOT, and parenthesis. These expressions are also supported on the command-line -cc argument.

Retrieve the type of SSL library at runtime

You can retrieve the type of SSL library in command-line -cc rules through the ssllibnamestartswith configuration predicate.

Ignore closed connections in logs

You can use the http-ignore-probes directive to ignore in log files HTTP/1.1 connections that are closed when servers accept HTTP/2.

Protocol Interoperability

HAProxy 2.5 has enhanced support for various protocols.

Enhanced reliability of ACLs relying on the Host header field

The port is stripped from absolute HTTP/1 requests and standard HTTP/2 requests. This conforms to the RFC3986 scheme-based normalization rules.

Any :80 when the scheme is HTTP, and any :443 when the scheme is HTTPS, are stripped both from the URI and from the Host field. This solves issues related to some browsers that were accidentally emitting :443 in the HTTP/2 websocket requests. It also increases the reliability of ACLs relying on the Host header field.

HTTP/1 updates

The latest HTTP/1 specifications suggest that Transfer-Encoding should not appear in HTTP/1.0: it could be abused, depending on how other intermediaries parse it. Thus:

• A request or response featuring a Transfer-Encoding header is now the last one on the connection.

• A request containing both Content-Length and Transfer-Encoding is the last one on a connection.

• The TE header is sanitized to make sure not to advertise unsupported encodings to the server; unsupported encodings can thus be safely rejected.

Disable bootstrapping WebSockets with HTTP/2 for newer browsers

You can disable the RFC8441 extension when new browsers do not support it through the new global directive h2-workaround-bogus-websocket-clients.

This directive prevents HAProxy from advertising support for the extended HTTP/2 CONNECT method. It makes browsers use a separate HTTP/1 connection for WebSocket.

New sample fetch functions to retrieve source and destinations

You can now retrieve the original source address of an incoming proxy that connects using the PROXY protocol.

New sample fetch functions allow retrieving source and destination addresses at various levels (e.g. fc_src for the connection).

The set-src and set-src-port actions were also added to the tcp-request content rulesets.

Stick Tables and Peers

Stick tables functionality has been updated in several ways.

Data accuracy

Peers now ignore any update on the conn_cur counter from other peers. This ensures local data is accurate and reflects actual traffic.

Peers can still push their local values to other peers, such as HAProxy Enterprise Stick Table Aggregator so that third-party components can use the information.

Store multiple variables in a single key

Stick tables now support arrays of GPC counters and GPT tags. You can store up to 100 indexes in a single variable.

Performance Improvements

HAProxy’s performance has been optimized in the following ways.

Threading Optimizations

Performance on specific architectures has improved by 2 to 5%, thanks to such optimizations as relaxed memory barriers on x86 platforms, and a lockless memory pool implementation on other architectures.

Increased performance on HTTP/1 small chunk parser

The HTTP/1 chunk parser is now faster than ever. It makes HAProxy even less sensitive to small chunks.

Much faster connection dequeuing on multi-thread

Significant queue speedup on multi-thread processes. Performance gains of up to 90% were measured on an 8-core machine thanks to significant locking reduction when delivering traffic to saturated servers.

Much faster DNS responses processing

DNS response processing was revamped and is now significantly faster.

SSL/TLS Enhancements

HAProxy 2.5 updates support of the SSL and TLS protocols.

OpenSSL 3.0

OpenSSL 3.0 is fully supported, using the deprecated API. To allow compiling with QUIC support, QuicTLS is also supported.

SSL/TLS-based client fingerprinting

You can now easily detect spoofed user-agents or grant access only to known user-agents. A set of new sample fetch functions makes it easy to retrieve SSL/TLS records from a client hello message. You can then hash them, and convert them to a format compatible with the JA3 method for creating SSL/TLS client fingerprints.

Display OCSP response information from the CLI

You can display OCSP response information from the CLI through the show ssl cert and show ssl ocsp-response commands, and when displaying the details of a certificate.

Update Certificate Authorities and Certificate Revocation Lists at runtime

You can now use the Runtime API to add, edit, show or remove entries from certificate authority (CA) and certificate revocation list (CRL) files used on bind and server lines.

$ echo -e "set ssl crl-file crlfile.pem <<\n$(cat rootCRL.pem)\n" | \
socat /var/run/haproxy.stat -
$ echo "commit ssl crl-file crlfile.pem" | socat /var/run/haproxy.stat -

$ echo "show ssl crl-file" | socat /var/run/haproxy.master -
# transaction
*crlfile.pem
# filename
crlfile.pem
$ echo "show ssl crl-file crlfile.pem" | socat /var/run/haproxy.master -
Filename: /home/jdoe/work/haproxy/reg-tests/ssl/crlfile.pem
Status: Used
Certificate Revocation List #1:
Version 1
Signature Algorithm: sha256WithRSAEncryption
Issuer: /C=FR/O=HAProxy Technologies/CN=Intermediate CA2
Last Update: Apr 23 14:45:39 2021 GMT
Next Update: Sep 8 14:45:39 2048 GMT
Revoked Certificates:
Serial Number: 1008
Revocation Date: Apr 23 14:45:36 2021 GMT
Certificate Revocation List #2:
Version 1
Signature Algorithm: sha256WithRSAEncryption
Issuer: /C=FR/O=HAProxy Technologies/CN=Root CA
Last Update: Apr 23 14:30:44 2021 GMT
Next Update: Sep 8 14:30:44 2048 GMT
No Revoked Certificates.

$ echo -e "set ssl ca-file cafile.pem <<\n$(cat rootCA.crt)\n" | \
socat /var/run/haproxy.stat -
$ echo "commit ssl ca-file cafile.pem" | socat /var/run/haproxy.stat -

$ echo "show ssl ca-file" | socat /var/run/haproxy.master -
# transaction
*cafile.crt - 2 certificate(s)
# filename
cafile.crt - 1 certificate(s)
$ echo "show ssl ca-file cafile.crt" | socat /var/run/haproxy.master -
Filename: /home/jdoe/work/haproxy/reg-tests/ssl/setcafileca2.crt
Status: Used
Certificate #1:
Serial: 11A4D2200DC84376E7D233CAFF39DF44BF8D1211
notBefore: Apr 1 07:40:53 2021 GMT
notAfter: Aug 17 07:40:53 2048 GMT
Subject Alternative Name:
Algorithm: RSA4096
SHA1 FingerPrint: A111EF0FEFCDE11D47FE3F33ADCA8435EBEA4864
Subject: /C=FR/ST=Some-State/O=HAProxy Technologies/CN=HAProxy Technologies CA
Issuer: /C=FR/ST=Some-State/O=HAProxy Technologies/CN=HAProxy Technologies CA
$ echo "show ssl ca-file *cafile.crt:2" | socat /var/run/haproxy.master -
Filename: */home/jdoe/work/haproxy/reg-tests/ssl/setcafileca2.crt
Status: Unused
Certificate #2:
Serial: 587A1CE5ED855040A0C82BF255FF300ADB7C8136
[...]

HAProxy’s Lua integration has new features.

Filter HTTP or TCP sessions via Lua scripts

You can now inspect or modify TCP or HTTP content through Lua scripts. This feature is currently highly experimental and must not be used in production, as the API might change.

Native HTTP client

You can now easily initiate HTTP requests from Lua through the new httpclient class. This class supports the Transfer-Encoding header, HTTP/1, HTTP/2, etc.

You can, for example, fetch an object thus:

Miscellaneous Improvements

The following miscellaneous improvements are included in this release.

JSON Web tokens integrity check

You can now check the integrity of the claims contained within signed JWTs through HAProxy. If a token is signed through a public and private key pair, you can be confident that only the party that owns the private key signed it. You can also extract information from the JWTs and add them to headers or use them in rules, for example, to apply rate limiting depending on user names.

Skip redirect rules for empty target URLs

The new ignore-empty option skips HTTP redirect rules if the target URL is empty. This is useful when retrieving URLs from a map file, for example:

That way, URLs that belong to the map file will match and be redirected. URLs that do not belong to the map file continue to the next rule. Empty URLs are not redirected.

Improved FreeBSD support

Linux-specific features have been ported to FreeBSD: set-dumpable, automatic executable path retrieval, etc.

Improved OpenBSD support

OpenBSD now supports the set-mark action.

HTTP actions now available in TCP mode

Some HTTP actions are now available to tcp-request content (e.g. set-log-level, set-mark, set-tos, set-nice, set-src, and set-src-port).

Display the HAProxy version and path

The HAProxy version and executable path are displayed before the first warning or error. This helps figure out that a script launched the wrong HAProxy version or why a certain configuration error is reported.

Deprecated and Removed Directives and Fetch Functions

The following directives were already deprecated and are no longer available:

• grace

• http-tunnel

• nbproc

• no option http-use-htx

• option forceclose

• option http_proxy

• set-cookie()

• tune.chksize

Contributors

We would like to thank each and every contributor who was involved in this release. Contributors help in various forms such as discussing design choices, testing development releases, reporting detailed bugs, helping users on Discourse and the mailing list, managing issue trackers and CI, classifying Coverity reports, maintaining the documentation, operating some of the infrastructure components used by the project, reviewing patches, and contributing code.

The following list doesn’t do justice to all of the amazing people who offer their time to the project, but we wanted to give a special shout out to new contributors.

New contributors

• Anubhav

• Daniel Black

• Jaroslaw Rzeszótko

• Jenny Cheung

• Jonathon Lacher

• Kunal Gangakhedkar

• Mark Mullan

• Marno Krahmer

• Vishnu

Regular contributors

• Alexandar Lazic

• Amaury Denoyelle

• Björn Jacke

• Christopher Faulet

• David Carlier

• Dirkjan Bussink

• Dragan Dosen

• Émeric Brun

• Frédéric Lécaille

• Ilya Shipitsin

• John Roesler

• Marcin Deranek

• Maximilian Mader

• Miroslav Zagorac

• Olivier Houchard

• Rémi Tricot Le Breton

• Thayne McCombs

• Thierry Fournier

• Tim Düsterhus

• William Dauchy

• William Lallemand

• Willy Tarreau