HAProxy 2.1: Supercharged Performance & Streamlined Codebase

In HAProxy 2.1, you will see significant performance gains in key areas, exciting new features including Dynamic SSL Certificate Updates, FastCGI, and Native Protocol Tracing, and a streamlined codebase.

HAProxy 2.1 is a technical release that is focused on improving the core codebase and developing better debugging and observability tools for contributors, support engineers, and developers. You will see a significant performance increase in multiple areas. This is due to optimizations such as scheduler improvements for dealing with high connection rates and increased efficiency from removing the FD cache. This version also lays the foundation for new and exciting features coming in HAProxy 2.2. With that said, there are still some amazing features that made it into this release, including Dynamic SSL Certificate Updates, FastCGI, and Native Protocol Tracing!

We’ve put together a complete HAProxy 2.1 configuration, which allows you to follow along and get started with the latest features. You will find the latest Docker images here. This release is, of course, made possible by the work of our open-source community members. They provide code submissions for new functionality and bug fixes, do quality assurance testing, host continuous integration environments, report bugs, and work on documentation. If you’d like to join this amazing community, find it on Slack, Discourse, and the HAProxy mailing list.

Dynamic SSL Certificate Updates

SSL certificate information has been centralized so that if the same certificate file is referenced by multiple bind lines, it is only loaded once. This means fewer system calls and a substantially faster startup time when loading the same certificate on multiple lines. There are some users who load tens of thousands or even hundreds of thousands of certificates multiple times and they will see a visible improvement.

We have also added the ability to update SSL certificates using the Runtime API. Use the set ssl cert and commit ssl cert commands. Here’s an example:

FastCGI

A direct outcome of supporting multiplexed connections everywhere was being within striking distance of supporting FastCGI. It became so tempting that we had to give it a try, and we succeeded! The new FastCGI support in 2.1 can help you improve resource management by eliminating superfluous components.

For example, you can remove components such as web servers deployed solely to serve as HTTP-to-FCGI gateways. Such components add a small amount of latency and obstruct HAProxy’s view of the backend application. Without them, HAProxy has a clear view of the application’s response times and can use it to operate at full speed without overloading it.

A new section named fcgi-app defines parameters for communicating with a FastCGI application. A normal backend section can then relay requests to it by including the use-fcgi-app directive and setting proto to fcgi on the server lines.

Native Protocol Tracing

HAProxy has a great track record of providing diagnostic and observability information that can be used during debugging of problematic requests and responses. To make it even better, a new tracing infrastructure has been integrated, allowing systems engineers and developers to collect low-level trace messages from HAProxy without having to recompile it.

Traces can be enabled by using Runtime API commands and will report diagnostic information at varying levels of detail. The messages can be sent to stdio, log servers, or to ring buffers, which are rotating log. In the following example, we use the show trace command to see supported trace sources and their current states:

There are a few variants of the trace command. When given a source and level, it enables debug tracing for the given source (e.g. h2) and filters down to events for the specified level. Use the sink variant to configure where trace messages are sent (e.g. stdout and stderr). Use the start/stop/pause variant to start, stop and pause tracing.

The following options are available for the level. It is recommended to only use “user” unless you have a reason otherwise.

Level	Description
user	This will report information that is suitable for use by a regular HAProxy user who wants to observe their traffic.
proto	In addition to what is reported at the “user” level, this also displays protocol-level updates. This can, for example, be the frame types or HTTP headers after decoding.
state	In addition to what is reported at the “proto” level, this will also display state transitions (or failed transitions) that happen in parsers, so this will show attempts to perform an operation while the “proto” level only shows the final operation.
data	In addition to what is reported at the “state” level, it will also include data transfers between the various layers.
developer	It reports everything available, which can include advanced information such as “breaking out of this loop” that are only relevant to a developer trying to understand a bug that only happens once in a while in the field. Function names are only reported at this level.

The following example starts tracing for HTTP/2 messages at the user level and sends them to stderr:

It can also be handy to see events from all sources at once. This release introduces an in-memory ring buffer (1MB in size) that lets you see messages regardless of their configured sinks. Simply issuing a show events buf0 from the Runtime API will show everything and adding the –w –N flags after it will work like tail -f.

In the following example, show events is called from an interactive shell:

To change the level, you would use:

Removal of the File Descriptor Cache

The file descriptor (FD) cache has been completely removed. This was a leftover from the very old “speculative I/O” mode intended to save a lot of syscalls in callbacks by keeping track of the last polling status. With recent improvements, it’s not needed anymore. This change has shown an increase in performance of up to ~20% on some artificially tailored workloads. However, you can realistically expect to see a 5-10% improvement in a real production environment.

Scheduler Improvements

A part of the internal scheduler was improved to support waking up tasks belonging to another thread. This was a necessary step in order to deal with idle inter-thread connections but also allowed us to simplify and speed up other parts of the code. The scheduler now uses a combination of a locked and a lock-free list to regain 5-10% performance on workloads involving high connection rates.

Defaulted HTTP Representation to HTX

HAProxy 2.1 officially removes support for the legacy HTTP mode and now supports the Native HTTP Representation only, also known as HTX. HTX was introduced in version 1.9 and became the default in version 2.0. No configuration change is required unless you try to specify no option http-use-htx in which case you’ll get an error.

[WARNING] 326/104833 (24011) : parsing [haproxy.cfg:9]: option ‘http-use-htx’ is deprecated and ignored. The HTX mode is now the only supported mode.

Several changes and bug fixes have come to HTX since the release of 2.0. Notably, it was reported by users that some legacy applications required case-sensitive header names. There is no standard case for header names, because as stated in RFC7230, they are case insensitive. To alleviate this challenge, and help these legacy applications seamlessly transition to modern architecture, the new h1-case-adjust directive was introduced. When used, HAProxy will replace the given header name with the value on the right:

If you have many headers that require adjusting, use h1-case-adjust-file to load them directly from a file. Each line in the file should contain a lower-case header name and its new value.

Because these directives are set in the global section, they are visible to all proxy sections. However, you must enable their use for specific proxies and server pools by adding an option h1-case-adjust-bogus-client directive to a frontend or listen section and/or an option h1-case-adjust-bogus-server to a backend. These should only be used as a temporary workaround for the time it takes to fix the client or server. Clients and servers that need such workarounds might be vulnerable to content smuggling attacks and must absolutely be fixed. They may also break randomly if you add other layers or proxies that don’t support uppercase names. Note that this feature has been backported to version 2.0.10.

Another update relates to the implementation of HTTP/2. In the legacy HTTP mode, HTTP/2 was implemented on top of HTTP/1. HTTP/2 requests were turned into HTTP/1 requests in “origin form”:

GET /path/to/file + Host header

But HTTP/2 clients are encouraged to use absolute form:

GET scheme://authority/path/to/file

Our conversion always used the origin form, which resulted in the loss of the HTTP or HTTPS scheme on end-to-end transfers. Now that HTX is the only internal representation, it was possible to keep the request in its original form (typically absolute for HTTP/2 and origin for HTTP/1) and preserve all elements end-to-end.

One visible effect is that logs will now show the scheme, such as “GET https://authority/path” instead of “GET /path”, since the URI really is this. Some will find this better, others may be annoyed, but it’s still possible to change the format if desired. What matters is that we do not lose the scheme anymore. This will also make it easier for users to build redirects that include the scheme.

Fetches and Converters

New fetch methods have been added, as described in the following table:

Name	Description
srv_name	Returns a string containing the name of the server that processed the request. It can be useful to return this to the client for debugging purposes.
srv_queue	Takes an input, either a server name or <backend>/<server> format, and returns the number of queued sessions on that server.
fc_pp_authority	Returns the PP2_TYPE_AUTHORITY Type-Length-Value (TLV) sent by the client in the PROXY protocol header. See https://www.haproxy.org/download/2.1/doc/proxy-protocol.txt
uuid	Returns a universally unique identifier (UUID) following the RFC4122 standard. Currently, only version 4 is supported.

A new converter, sha2(number_of_bits), is now available for generating a checksum for a binary string using the SHA-2 cryptographic hash function. The result is a binary value with a byte length equal to number_of_bits / 8. The number_of_bits parameter can be set to 224, 256, 384, or 512. The default is 256.

The SHA-2 function is more secure and reliable than MD5 or SHA-1 because it is more resistant to hash collision attacks. You might use it to sign a cookie and verify the signature later by concatenating the value with a secret key before hashing.

Deprecated Configuration Options

HAProxy attempts to be backwards compatible across versions. However, there comes a time when certain directives are due to be phased out as they are no longer relevant or better methods have been introduced. One of these changes is the removal of rsprep and reqrep, which have been emitting deprecation warnings in previous versions for a long time.

These are very old directives that operate upon a whole HTTP line to modify HTTP headers or the URI. They require emulating a dummy request when dealing with HTTP/2 and HTX, which isn’t ideal. It also presented the challenge of not being able to use case-sensitivity on values without having the name. All req and rsp directives should be replaced with http-request replace-header, http-request replace-uri, and http-response replace-header.

Also, several deprecated options and directives have been officially removed and will present a fatal error when used. See the table below for the newer directives to use instead.

Removed	Replacement
block	http-request deny
reqrep	http-request <replace-uri|replace-header>
rsprep	http-response replace-header
clitimeout	timeout client
contimeout	timeout connect
srvtimeout	timeout server
redispatch	option redispatch
resolution__poolsize	NONE
option independant-streams	option independent-streams

Strict Limits Setting

HAProxy communicates with the operating system to increase various system limits as needed, such as the file descriptor limit to account for maxconn settings. In some cases, HAProxy may not be able to increase these limits and so displays a warning. However, there are times when you may want to force HAProxy to abort at startup if it can not get the required limits.

As an example, you may require high limits in production and if there is an error in achieving this your production traffic could be affected. To alleviate this problem, you can add a new option strict-limits to the global section. If HAProxy fails to increase the limits set by setrlimit()—typically the file descriptor limit—then HAProxy will fail to start. This setting will be turned on by default in version 2.3.

Debugging for Developers

The debug dev Runtime API command that was previously only available when building with -DDEBUG_DEV is now always available, but only shown when expert-mode is on. This is sometimes needed by developers to extract information about a sick session or to perform fault injection. Do not try to use them in production without being invited to do so, you’ll very likely crash your process before you understand what you did.

Version Info Shows Links

Users can now get a better view of the status of the code they’re using by running haproxy -v, as shown here:

This gives you a link to a page listing the issues relevant for the version of HAProxy you are running. This makes it easy to see whether or not you are affected by a bug.

Runtime API Field Descriptions

The show info and show stat Runtime API commands accept a new parameter, desc, that adds a short description to each field.

Prometheus Improvements

In certain environments, exporting all of the metrics available by Prometheus can result in a large amount of unnecessary data. A method was developed to allow for exporting only specific sets of data. In HAProxy 2.1 you can now pass the scope parameter within the query string to filter exported metrics. It accepts the following values:

• global

• frontend

• backend

• server

• * (all)

A scope parameter with no value will filter out all scopes and nothing will be returned. If multiple scope parameters are set, they are parsed in the order they appear in the query string. Here are a few examples:

Prometheus URI	Output
/metrics?scope=server	server metrics will be exported
/metrics?scope=frontend&scope=backend	Frontend and backend metrics will be exported
/metrics?scope=*&scope=	no metrics will be exported
/metrics?scope=&scope=global	global metrics will be exported

Miscellaneous Improvements

Several miscellaneous improvements have been made:

• The storage of the server-state global file was moved to a tree, which provides much faster reloads.

• http-request / http-response sc-set-gpt0 now accept an expression.

• resolve-opts now accepts ignore-weight so that when generating servers with DNS SRV records, server weights can be set dynamically with agent health checks or the Runtime API and not be subsequently reset by DNS SRV weights.

• The Stats page can now be exported in JSON format. This can be accessed by appending “/;json” to the URI.

• The send-proxy-v2 directive can now send the PP2_TYPE_AUTHORITY value, allowing it to chain layers using SNI.

• Added support for the user and group directives to the program Process Manager section.

• Connections should now take significantly less memory as the source and destination addresses are now dynamically allocated as needed. That translates to 128 to 256 bytes saved per connection and per side in the common case.

Conclusion

With all of the architecture improvements, refactoring, and removal of obsolete code, you can expect better performance in some areas, improved debugging capabilities, and even more responsive developers thanks to their ability to work on a cleaner and more manageable codebase. Keep a watch on our blog for other news as we turn our attention back to adding new and amazing features!

Curious about HAProxy Enterprise? Contact us to learn more and get your HAProxy Enterprise free trial. Follow us on Twitter and join the conversation on Slack!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.