Security researcher Felix Wilhelm, working with Google’s Project Zero, has disclosed a critical vulnerability in HAProxy’s HTTP/2 HPACK decoder in versions 1.8 and above.
Wilhelm has disclosed critical vulnerabilities in other popular products such as Xen, Hyper-V, IBM GPFS, and FireEye’s MPS. Project Zero is a team of security analysts employed by Google that is tasked with finding zero-day vulnerabilities. The project was announced in July 2014 and has discovered many vulnerabilities from a wide range of vendors including Apple, Cloudflare, and Microsoft. Wilhelm has published his own write-up here.
In many cases, bugs such as this one are discovered through fuzzing, static analysis, and manual code reviews. Fuzzing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks. Traditionally this is done using open source tools such as American Fuzzy Lop (AFL) and Google’s tool clusterfuzz. However, even with such tools available, fuzzing is no easy task, especially when it comes to targeting a threaded, event-driven, non-blocking engine such as HAProxy. Static analysis is the analysis of software that is performed without actually executing the program. According to Felix’s paper Tracing Privileged Memory Accesses to Discover Software Vulnerabilities, “techniques that are used mainly by compilers for performing safe optimizations of source code can also be used to discover security vulnerabilities”. Static analysis can also be quite tedious as it involves crawling through each discovered item and eliminating false positives from the report and testing the severity of each. In short, while there are tools and processes available to help discover vulnerabilities in software it is certainly not an easy task and takes countless hours of dedication, validation and a tremendous effort to locate valid bugs in modern software.
Timeline
24 March 18:37 UTC: Felix Wilhelm contacts Willy Tarreau, HAProxy’s lead developer, privately with a detailed report and a reproducer.
24 March 18:50 UTC: Tarreau confirms receipt and starts looking at the issue
25 March 8:31 UTC: Tarreau informs distros about an upcoming H2/HPACK vulnerability
27 March 16:38 UTC: Tarreau proposes a candidate fix to Wilhelm
27 March 17:10 UTC: Tarreau notifies distros about possible workarounds
29 March 08:32 UTC: Tarreau sends the candidate patches to distros and proposes a coordinated release date for April 2nd 13:00 UTC
30 March 11:38 UTC: all distros have already agreed on the CRD
30 March 16:46 UTC: Wilhelm confirms the fix
31 March 05:59 UTC: CVE ID assigned by the Debian team
2 April 13:00 UTC: new packages published, making the vulnerability public
Vulnerability Overview
As mentioned above, Wilhelm discovered a critical vulnerability in HAProxy’s HTTP/2 HPACK decoder that can be exploited to cause an out-of-bounds memory write potentially leading to corruption of data, a crash, or code execution. Specifically, the vulnerable code was located in HAProxy’s HPACK table management code. HAProxy’s lead developer, Willy Tarreau, has provided a detailed outline within the commit message explaining how this vulnerability came to be.
“The HPACK header table is implemented as a wrapping list inside a contiguous area. Header names and values are stored from right to left while indexes are stored from left to right. When there’s no more room to store a new one, we wrap it to the right again or possibly defragment it if needed. The condition to use the right part (called the tailroom) or the left part (called the headroom) depends on the location of the last inserted header. After wrapping happens, the code is forced to stick to the tailroom by pretending there’s no more headroom, so that the size fit always fails. The problem is that nothing prevents storing a header with an empty name and empty value, resulting in a total size of zero bytes, which satisfies the condition to use the headroom. Doing this in a wrapped buffer results in changing the “front” header index and causing miscalculations on the available size and the addresses of the next headers. This may even allow an overwrite to some parts of the index, opening up the possibility to perform arbitrary writes into a 32-bit relative address space.
This patch fixes the issue by making sure the headroom is considered only when the buffer does not wrap, instead of relying on the zero size.”
This vulnerability has been assigned CVE-2020-11100.
Potential Impact
Writing to arbitrary memory locations may sometimes be used to grant special permissions, modify data or even trick the program into following different code paths, particularly when the program’s stack is overwritten. Attempts to exploit this bug may lead to a denial of service by crashing the HAProxy process. The risk of using it to read or modify data in memory within the program is low. This bug only allows for reaching memory areas within 32 bits from the beginning of the HPACK table, which doesn’t allow it to touch the stack on 64-bit systems. In addition, the likelihood that it is used to overwrite a known location is extremely low because the address of each connection’s HPACK table depends on the thread that accepted the connection and on the traffic that precedes this connection, all of which are not under the attacker’s control. Even in a lab setting, when sending the attack as the very first request, some extra uncertainty happens because modern operating systems randomize functions and variables’ addresses via a mechanism called ASLR (Address Space Layout Randomization) which provides random addresses to functions, variables, and libraries.
Affected Versions & Remediation
The following is the list of affected versions, fixed version, and potential workarounds where available. It is recommended to upgrade immediately if you are using any of these with HTTP/2 enabled.
If you are using HAProxy 2.0 or 2.1 you must upgrade or apply the relevant mitigations where available as these versions default to HTX which automatically upgrades HTTP/1 to HTTP/2.
Affected Versions | Fixed Versions | Workarounds (where possible) |
|---|---|---|
HAProxy 1.8.0 – 1.8.24 HAProxy Enterprise 1.8r1 1.0.0-186.251 – 193.716 HAProxy Enterprise 1.8r2 2.0.0-190.714 – 205.1000 ALOHA 10.0.0 – 10.0.14 ALOHA 10.5.0 – 10.5.12 | HAProxy 1.8.25+ HAProxy Enterprise 1.8r2 2.0.0-205.1048+ ALOHA 10.5.13+ | Remove |
HAProxy 1.9.0 – 1.9.14 HAProxy Enterprise 1.9r1 1.0.0-197.290 – 208.876 HAProxy ALOHA 11.0.0 – 11.0.7 | HAProxy 1.9.15+ HAProxy Enterprise 1.9r1 1.0.0-213.948+ HAProxy ALOHA 11.0.8+ | Remove |
HAProxy 2.0.0 – 2.0.13 HAProxy Enterprise 2.0r1 1.0.0-204.260 – 219.645 HAProxy ALOHA 11.5.0 – 11.5.3 | HAProxy 2.0.14+ HAProxy Enterprise 2.0r1 1.0.0-220.698+ HAProxy ALOHA 11.5.4+ | Disable HTX by adding |
HAProxy 2.1.0 – 2.1.3 HAProxy Enterprise 2.1r1 1.0.0-217.0 – 221.38 | HAProxy 2.1.4+ HAProxy Enterprise 2.1r1 1.0.0-221.93+ | NO WORK AROUND AVAILABLE! MUST UPGRADE! |
HAProxy Enterprise customers can follow the below upgrade documentation to ensure you are on the latest version:
HAProxy Enterprise 1.8r2: https://www.haproxy.com/documentation/hapee/1-8r2/upgrade/
HAProxy Enterprise 1.9r1: https://www.haproxy.com/documentation/hapee/1-9r1/upgrade/
HAProxy Enterprise 2.0r1: https://www.haproxy.com/documentation/hapee/2-0r1/upgrade/
HAProxy Enterprise 2.1r1: https://www.haproxy.com/documentation/hapee/2-1r1/upgrade/
Optionally you can contact our support for help.
For ALOHA 10.0 and above customers, we recommend that you reach out to support for assistance in obtaining the latest image.
HAProxy Edge has been automatically upgraded prior to the announcement.
If you have any questions at all about this please feel free to reach out directly to our support team.
Acknowledgments
We would like to really thank Felix Wilhelm for his responsible disclosure and his timely help in addressing this issue. Felix also kindly agreed to slightly delay the publication of his full analysis of this bug to help protect users while they deploy the necessary updates on their systems. Full details will be released on April 20th, but users are urged to upgrade and are reminded that attackers will figure out the issue by themselves without waiting for the full disclosure.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
