We've added new details explaining how the HTTP/2 Rapid Reset Attack works, what that looks like in HAProxy, and how steps we've previously taken protect our users today. Read on to learn more.
The vulnerability CVE-2023-44487 found in the HTTP/2 protocol could allow a denial of service attack against web servers, reverse proxies, or other software processing HTTP/2 traffic.
We are following the developments of this situation, but have concluded that our products are not affected. Specifically, our products are safeguarded by code developed in HAProxy 1.9, released in 2018. This applies to HAProxy, HAProxy Enterprise, HAProxy Enterprise Kubernetes Ingress Controller, and HAProxy ALOHA.
After rigorous testing, we have confirmed that our implementation of the HTTP/2 protocol can handle the Rapid Reset Attack without increasing the resource usage or compromising the parallelism of the protocol. We will continue to test and monitor, but the supported versions of our products are not vulnerable to the known attack vectors.
Why is HAProxy unaffected?
CVE-2023-44487 leverages the HTTP/2 protocol—and specifically the stream multiplexing feature—to abuse the HTTP/2 request cancellation function. Since the protocol allows the client to unilaterally make cancellations and send massive request volumes with little personal resource cost, the door is open for abuse. The HTTP/2 protocol does implement a request concurrency limit (HAProxy defaults to 100 active streams per connection, but this is configurable), and servers must automatically reject any client attempts to open a stream past this limit using an RST_STREAM frame. Other active streams on the connection are unaffected by this.
Problems arise when clients abuse the request cancellation feature to quickly reset an unbounded number of streams. The protocol only considers open streams counted at the connection level, and since reset streams switch to a "closed" state, they do not count towards the limit. As a result, opening and closing streams quickly doesn't increase the total number of streams.
Servers that start asynchronous processing for each stream without accounting for these resets (and their dead parent connections) can accumulate an excessive process backlog. This leads to heavy resource use, complete memory consumption, and can exhaust all available connections. When weaponized in a DoS attack, this can cause massive slowdowns or complete service interruptions. It also renders new, incoming requests useless.
Since the messages needed to create and reset a stream are just a few bytes apiece, attackers find this method particularly interesting. Taking down a server that is not properly protected takes very little bandwidth. Allocating 100,000 new streams per second only requires roughly 25 Mbps of attack traffic—without even needing to process return traffic.
HAProxy is protected by design
These issues impact many load balancers, so how do we counteract them? Resilience to malicious client behavior is a high priority at HAProxy. HAProxy 1.9 and later have laid protective groundwork by strengthening how we handle HTTP/2 protocol stream multiplexing:
Instead of only counting known, established streams at the protocol level, HAProxy counts allocated resources. HAProxy will count a stream as present until its resources are completely released. This also helps enforce the streaming limit.
Until HAProxy dips below the configured stream limit again, new stream creation remains pending—regular timeouts eventually apply and the stream is cut if the situation does not resolve itself. This can occur during an attack.
Measures like these prevent Rapid Reset from impacting HAProxy and the servers behind it any more than legitimate traffic would.
The HAProxy 1.9 code commit has been carried forward through each successive version of HAProxy, HAProxy Enterprise, HAProxy Enterprise Kubernetes Ingress Controller, and HAProxy ALOHA. Our products have remained proactively immune to attacks like HTTP/2 Rapid Reset for the past five years without showing any signs of vulnerability.
How can we know that we are not affected?
In response to CVE-2023-44487, we've performed internal testing using a variety of test cases and simulated attacks.
We tested client requests on an AMD EPYC 74F3 server, containing 24 cores at 3GHz. This is the same server we used for demos at HAProxyConf. We reached 800,000 requests per second at saturation and performance was good.
Next, we ran a comparative test between the number of calls to different functions inside the process while under attack and under h2load. We used 24 clients for each simulation while enforcing a cap of approximately 2.2 million requests. If our simulated attack were successful, we'd expect these calls to vary quite noticeably. However, those numbers in HAProxy were practically identical—signaling unsuccessful attack penetration. Architecture will make a small difference here, but not enough to concern us.
Overall, attackers cannot leverage this attack as a low-cost way to take down HAProxy nor the servers it protects. Attackers would need significant resources to build such an attack. Because of this, other more common attack methods might seem equally appealing—yet HAProxy's usual protective mechanisms are highly effective and poised to prevent these attacks.
HAProxy remains resistant to HTTP/2 Rapid Reset
Based on prior development and recent testing, HAProxy remains unaffected by the HTTP/2 Rapid Reset Attack. Our performance isn't impacted by DoS attacks that leverage stream multiplexing, and the servers behind HAProxy won't receive any abnormal boosts in traffic. The resources needed to launch an attack are far too great.
Our customers can rest assured that we always develop our products with resource optimization in mind, which in this case explains why we are unaffected by CPU, memory, and general resource issues.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
