If you are using HAProxy 2.0 or newer, it is important that you update to the latest version. A vulnerability was found that makes it possible to abuse the HTTP/2 parser, allowing an attacker to prepend hostnames to a request, append top-level domains to an existing domain, and inject invalid characters through the :method pseudo-header. Willy Tarreau has also announced this on the mailing list here.
Timeline
5 August 2021 19:00 UTC – James Kettle in his presentation HTTP/2: The Sequel is Always Worse describes a new class of vulnerabilities that could cause desync attacks on some proxy servers during the HTTP/2 to HTTP/1 translation. This presentation prompts Willy Tarreau, chief maintainer of HAProxy, to assess whether HAProxy is affected.
6 August 2021 00:43 UTC – A link to the article is shared on the HTTP working group
6 August 2021 02:00 UTC – Willy reads the article and begins checking the code.
6 August 2021 03:43 UTC – Willy concludes that HAProxy is fine, and responds to the HTTP working group that he suspects this would only affect older implementations as modern ones are expectedly safe regarding this.
7 August 2021 22:19 UTC – HAProxy contributor Tim Düsterhus contacts Willy reporting that he found that :path and :scheme can be abused, but only when the backend is HTTP/2, which differs from the findings in the article. He also supplies a reproducer.
8 August 2021 05:28 UTC – Willy responds to Tim with a proposed fix for review and testing. A few exchanges happen over the weekend on cosmetic details of the patch.
10 August 2021 20:15 UTC – Tim reports that he could also abuse the :method field, in this case assuming a vulnerable HTTP/1 server is located behind HAProxy.
10 August 2021 20:28 UTC – Willy proposes a fix for the method and overlooks a detail resulting in a longer discussion with Tim.
11 August 2021 15:15 UTC – After a mistaken belief that it is possible to conduct a smuggling attack, the decision is taken to contact distro maintainers and keep the issue under embargo for the end of the week with a release planned for the following Tuesday. Distro maintainers instantly respond favorably, confirming being able to deliver timely fixes if patches are provided quickly.
12 August 2021 07:30 UTC – Willy shares a preliminary batch of tested backports with distros, though the details of the impact in each branch are still under investigation, and the patch description is still under editing as the knowledge of the problem is refined.
16 August 2021 08:40 UTC – Willy signals distros that there’s finally no request smuggling attack, it was a mistaken analysis resulting from modifications needed to test the request injection. Commit messages are updated.
17 August 2021 15:00 UTC – The embargo is lifted, stable versions with the fix are released to the public.
Vulnerability Overview
Tim Düsterhus discovered several issues within HAProxy’s handling of HTTP/2. The first allows an attacker to abuse the scheme (http:// and https://) to prepend a prefix hostname to requests that are forwarded to HTTP/2 backend servers, which causes the web servers to see a different hostname from the one that HAProxy saw, possibly bypassing some filtering performed at the proxy layer. When a web server is in HTTP/1 mode, the scheme is dropped and so it has no effect.
The second issue allows the :path pseudo-header to appear as part of the host header if it does not start with / or *. An attacker could use this to append a top-level domain to the existing domain and bypass hostname checks. Similarly, this one will be ignored if the request is forwarded to an HTTP/1 backend server, so only HTTP/2 to HTTP/2 communications are affected.
The third is a corner case within the current HTTP/2 specification, which makes the host header prevail over the :authority, causing rules applied to the host header to not necessarily match what the backend server sees once the authority is used to reconstruct the host header (or the host header dropped depending on the version, but this is the same). The new HTTP/2 specification, which is still being written, updates this by mandating that the host is always ignored when authority is present, which will solve this problem in the long term.
Affected Versions & Remediation
The following section lists the affected versions, fixed version, and potential workarounds where available. We recommend that you upgrade immediately if you are using any of these.
Affected Versions | Fixed Versions |
HAProxy 2.0 versions <= 2.0.23 | HAProxy 2.0.24 |
HAProxy 2.2 versions <= 2.2.15 | HAProxy 2.2.16 |
HAProxy 2.3 versions <= 2.3.12 | HAProxy 2.3.13 |
HAProxy 2.4 versions <= 2.4.2 | HAProxy 2.4.3 |
HAProxy Enterprise 2.0r1 versions <= 1.0.0-234.1213 | HAProxy Enterprise 2.0r1 >= 234.1215 |
HAProxy Enterprise 2.2r1 versions <= 1.0.0-240.455 | HAProxy Enterprise 2.2r1 >= 240.490 |
HAProxy Enterprise 2.3r1 versions <= 1.0.0-239.297 | HAProxy Enterprise 2.3r1 >= 241.329 |
HAProxy ALOHA 11 versions <= 11.5.10 | HAProxy ALOHA 11.5.11 |
HAProxy ALOHA 12 versions <= 12.5.2 | HAProxy ALOHA 12.5.3 |
HAProxy ALOHA 13 versions <= 13.0.4 | HAProxy ALOHA 13.0.5 |
Workarounds
It’s important to note that the only issue that affects HTTP/1 communication with backend servers involves the use of an HTTP/1 server that fails to properly validate a request line. At the time of writing, we’re not aware of any such server failing to properly parse an HTTP/1 request among the mainstream servers such as Apache, NGINX, or Varnish. This means that most users who cannot upgrade can simply disable HTTP/2 communications with their backend servers if it was enabled. This is done by making sure that neither proto h2 nor alpn h2 is present on the server lines.
Users who are using less common servers and who fear these might process invalid requests should consider the alternative workarounds below.
If you are not able to update right away, you can apply the following rules to mitigate the issues. These should be added to your frontend.
| http-request reject if { method -m reg [^A-Z0-9] } | |
| http-request set-uri %[url] |
Alternatively, you can remove alpn h2,http/1.1 from your bind line and add the following to your defaults section:
| option disable-h2-upgrade |
Acknowledgments
We would like to thank Tim Düsterhus for his efforts in identifying these issues and his responsible disclosure of them.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
