CitrixBleed, or CVE-2023-4966, is now an infamous security vulnerability affecting Citrix NetScaler that allows attackers to hijack user sessions by stealing session authentication tokens.
Unfortunately, it has affected many NetScaler customers including Xfinity, which lost data for 36 million customers as a result of CitrixBleed.
There is no way to protect against CitrixBleed by configuring the NetScaler WAF to detect and block it. The vulnerability affects the NetScaler appliance itself, so you must update every instance and also kill all existing sessions to patch the vulnerability. This is far from ideal and many customers are looking for easier ways to protect themselves.
In this post, we will show how you can use an HAProxy Enterprise load balancer to protect against CitrixBleed by placing it in front of your NetScaler instance(s).
Background on the CitrixBleed exploit
Let’s explore how CitrixBleed works at a high level. If an attacker sends a request with a HTTP Host header over 24,812 characters in length to a vulnerable endpoint in NetScaler, the appliance will dump the memory contents back to the attacker. One of the vulnerable endpoints is the path for OAuth discovery: /oauth/idp/.well-known/openid-configuration.
The resulting memory dump might contain information about sessions currently saved in the appliance. This information could allow the attacker to access the NetScaler appliance without any additional credentials.
This happens because the HTTP Host header generates the payload for NetScaler’s OAuth configuration but lacks sufficient checks to prevent buffer overflow.
Bleepingcomputer has a good article with a description of the attack.
Using HAProxy to protect against CitrixBleed
By default, HAProxy blocks requests with Headers that are too long, so it is not affected by this issue. We can use HAProxy to protect NetScaler from the CitrixBleed vulnerability.
In this example, we will:
set up HAProxy Enterprise in front of a Citrix NetScaler
protect your infrastructure by default with HAProxy Enterprise
configure HAProxy Enterprise to accept larger headers for maximum compatibility with NetScaler
configure HAProxy Enterprise to reject and log any HTTP Host header over 24k characters (safely below the 24,812 character limit)
identify attacks using the observability tools in HAProxy Fusion Control Plane.
Set up HAProxy Enterprise in front of Citrix NetScaler
In this architecture, we use HAProxy Enterprise to receive and filter external traffic first, passing only safe and legitimate traffic to NetScaler.
In addition, I always add the HAProxy Fusion Control Plane because it greatly simplifies the setup and provides default observability, allowing us to review any possible attack attempts.
Protect your infrastructure by default with HAProxy Enterprise
HAProxy Enterprise’s lower limit for HTTP headers means the NetScaler is now protected from attacks hoping to exploit the CitrixBleed vulnerability. HAProxy will reject long HTTP headers with a 400 status code.
However, since we want to log and block the attacks, instead of rejecting them outright, we will create some additional configuration on HAProxy Enterprise.
Configure HAProxy Enterprise to accept larger headers
For maximum compatibility with NetScaler’s acceptance of larger HTTP headers, we will configure HAProxy Enterprise also to accept larger headers, overriding the default.
Global section
Increase the total bufsize size to allow larger headers. This is technically not needed at all since HAProxy would just reject the long headers by default (with a 400 status code), but we want to be able to log the attacks as well, instead of rejecting them outright.
| tune.bufsize 64000 |
Increasing bufsize increases the memory used by each request, so ensure you set this value to an appropriate amount.
Configure HAProxy Enterprise to reject and log any HTTP Host header over 24k characters
While HAProxy protects against this attack by default, for logging purposes, we will create a specific deny rule that applies an easily identifiable status code when rejecting these requests.
Load Balancing configuration
| frontend webserver | |
| mode http | |
| bind *:443 ssl crt /var/lib/dataplaneapi/storage/ssl/certificate.pem | |
| http-request deny deny_status 413 if { hdr_len(Host) gt 24000 } | |
| default_backend webserver_backend | |
| backend webserver_backend | |
| server srv1 <netscaler IP address> |
Notice this line:
| http-request deny deny_status 413 if { hdr_len(Host) gt 24000 } |
This line does two things:
Detects any request where the Host header is over 24,000 characters long (this is close to the 24,812 buffer limit that NetScaler suffers from and large enough to consider a request as malicious)
Reject those requests with a status code “413 - Payload too large”.
In this case, we are protecting the Host header only, but HAProxy’s defaults protect the backend with other headers as well.
You can choose your own status code, but 413 is useful because it’s unique enough to be very easy to spot in your logs – see how we do that below.
Identify attacks using HAProxy Fusion Control Plane
HAProxy Fusion provides a single pane of glass for centralized management, monitoring, and automation of HAProxy Enterprise. With HAProxy Fusion installed, you can easily monitor for any attacks detected by your HAProxy Enterprise deployment.
Let’s use the HAProxy Fusion UI and look into the Request Explorer for requests where the response code is 413 and the URI is the vulnerable OAuth URL.
Here, we can easily identify the matching requests and see the request details and logs.
Summary
In this blog post, you’ve learned how to use HAProxy to block possible CitrixBleed attacks and how to monitor for any attacks using HAProxy Fusion.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
