HAProxy Technologies 2024 . All rights reserved. https://www.haproxy.com/feed en https://www.haproxy.com daily 1 https://cdn.haproxy.com/assets/our_logos/feedicon-xl.png <![CDATA[HAProxy Technologies]]> https://www.haproxy.com/feed 128 128 <![CDATA[Lasting Impressions and Technical Tidbits From AWS re:Invent 2024]]> https://www.haproxy.com/blog/lasting-impressions-and-technical-tidbits-from-aws-reinvent-2024 Fri, 13 Dec 2024 07:58:00 +0000 https://www.haproxy.com/blog/lasting-impressions-and-technical-tidbits-from-aws-reinvent-2024 ]]> AWS re:Invent 2024 has officially wrapped up, but not everything that happens in Vegas stays in Vegas. We're still gushing over the reception our booth team received from attendees—and we were excited to see even more organizations going all-in on application delivery and security. Naturally, AWS integration was a constant backdrop throughout the week as we showcased the HAProxy platform. 

As always, our interactions with booth visitors—both new and familiar—defined the team's AWS re:Invent experience. The constant buzz generated by tens of thousands of attendees was incredibly energizing. However, re:Invent was also a strong reminder that the tech world remains smaller (and more human) than expected, despite the conference's staggering scale. 

Here are some key takeaways from our five days spent alongside AWS and application delivery enthusiasts.

Getting cozy with our booth visitors

]]> ]]> AWS re:Invent means a lot to us. It's an opportunity to check the pulse of the tech community, connect with attendees, and showcase our recent product development wins to a diverse audience. Our booth was absolutely swamped with visitors (a problem we're happy to have!) all hoping to see the latest and greatest updates from HAProxy Technologies. 

We'd like to give two big shoutouts coming out of AWS re:Invent. First, a massive thank you to our booth visitors! Seeing old and new faces was fantastic, and your patience as we worked our way through the crowd was truly appreciated.

]]> ]]> And to whomever anonymously screamed, "I love HAProxy!" while passing by: we love you too! 

Attendees showed just how amazing the tech community is—and proved that a hunger for whiteboard sessions and hands-on demos isn't easily sated.

]]> ]]> Second, our booth team tirelessly answered as many questions as humanly possible. The team went above and beyond to make each visitor feel like an HAProxy community member. Together, they navigated the crowd with relative ease, demonstrating impressive load-balancing aptitude.

What were attendees talking about?

AWS re:Invent rightfully draws plenty of questions about the AWS platform and how integrated solutions from other vendors help users conquer challenges. Many visitors also asked how we measure up against competitors such as F5 Networks and VMware/Avi Networks. Having the chance to explore our unique advantages and how far we've come was invaluable. 

We also often heard the following questions: 

  • What does HAProxy do?

  • How do you compare to AWS Application Load Balancer (ALB) and NGINX? 

  • Do you have alternatives available to replace [insert infrastructure component]?

  • How do I win that LEGO R2D2?

Across the entire conference, plenty of chatter centered on longstanding AWS topics such as Auto Scaling support and Amazon Route 53. Since most attendees rely on AWS to power some portion of their infrastructure, understanding how HAProxy fits into that equation is critical. For example, HAProxy Fusion's latest updates allowed us to showcase our products much more easily and comprehensively than before. 

Today's trending topics such as Kubernetes, application security, API gateways, AI, and automation were also front and center. We heard and facilitated conversations covering the entire tech stack, from databases to support for the IoT's Message Queuing Telemetry Transport (MQTT) protocol. Name any topic related to application delivery and security—we likely heard rumblings about it. 

Lastly, we appreciated the glowing feedback (and the multiple verbal “thumbs up” comments) that we received from HAProxy users! We're incredibly flattered that you took the time to share your thoughts following some deeply technical (and lengthy) conversations.

HAProxy hits the AWS stage

We ourselves dove into security quite heavily—not only at the booth, but also on stage. HAProxy Technologies' Jakub Suchy, Director of Solutions Engineering, gave a thought-provoking lightning talk covering no-compromise security in AWS with HAProxy:

]]> Thanks so much for your engagement! Being able to give a talk at re:Invent is always a great privilege, and we look forward to presenting the next big development in secure application delivery.

Come see us next year!

AWS re:Invent 2024 was incredible. Our chats helped us better understand the evolving needs of the AWS community. We were thrilled to share our expertise as G2-recognized leaders in the categories of Load Balancing, API Management, Web Application Firewall (WAF), and DDoS Protection (with the awards to prove it!). Your feedback remains consistent with our G2 performance, and we're committed to keeping it that way. 

We'll continue to monitor the changing AWS landscape and bolster our platform support. We look forward to seeing you again at AWS re:Invent 2025, also in Las Vegas, from December 1st to December 6th. Stay tuned as more details become available!

Last but not least, we're thrilled to keep planning for HAProxyConf 2025 next summer in San Francisco. HAProxyConf celebrates the thriving community that's helped make HAProxy One the world’s fastest application delivery and security platform. Over two-plus days, expert speakers will share best practices and real-world use cases highlighting HAProxy's next-gen approach to high-performance application delivery and security. Check out the HAProxyConf official website to learn more and stay updated. And please, don't forget to answer our call for papers!

Want to learn more about HAProxy and AWS?

To dive a little deeper into our AWS support, check out these helpful resources:

]]> Lasting Impressions and Technical Tidbits From AWS re:Invent 2024 appeared first on HAProxy Technologies.]]>
<![CDATA[Announcing HAProxy ALOHA 16.5]]> https://www.haproxy.com/blog/announcing-haproxy-aloha-16-5 Wed, 27 Nov 2024 00:21:00 +0000 https://www.haproxy.com/blog/announcing-haproxy-aloha-16-5 ]]> HAProxy ALOHA 16.5 is now available, and we’re delighted to share that this release includes one of the cornerstone security features announced earlier this year—the new Bot Management Module. HAProxy ALOHA customers will also benefit from the new Network Management CLI, secure Wireguard VPN synchronization between appliances, updated root filesystem packages, and the features announced in open source HAProxy 3.0.

New to HAProxy ALOHA?

HAProxy ALOHA provides high-performance load balancing for TCP, UDP, QUIC, and HTTP-based applications; SSL processing; PacketShield DDoS protection; bot management; and a next-generation WAF. HAProxy ALOHA combines the performance, reliability, and flexibility of our open-source core (HAProxy – the most widely used software load balancer) with a convenient hardware or virtual appliance, an intuitive GUI, and world-class support. HAProxy ALOHA benefits from next-generation security layers powered by threat intelligence from HAProxy Edge and enhanced by machine learning.

What’s new?

HAProxy ALOHA 16.5 includes exclusive new features plus many of the features from the community version of HAProxy 3.0. For the full list of features, read the release notes for HAProxy ALOHA 16.5.

New in HAProxy ALOHA 16.5 are the following important features:

  • The new HAProxy Enterprise Bot Management Module provides fast, reliable, and flexible identification and categorization of bots attempting to access websites or applications, with 100% local processing for low latency and no external dependencies.

  • The new Network Management CLI (netctl) allows customers to automate the management of network interfaces directly from the appliance itself. It operates as an abstraction layer that allows users to configure the network stack of the HAProxy ALOHA load balancer using a simple command-line tool.

  • The new Wireguard VPN feature empowers customers to securely synchronize configurations between HAProxy ALOHA servers across the Internet or internal networks, making it easier to maintain consistency and manage configurations across appliances through an encrypted UDP tunnel that ensures data is protected when traveling between servers.

  • Updates to the root filesystem packages, including libraries, binaries, scripts, and all embedded components improve stability, security, and functionality.

We announced the release of HAProxy 3.0 in May 2024, which included improved simplicity, reliability, security, and flexibility. Many of the features from HAProxy 3.0 are now available in HAProxy ALOHA 16.5.

Some of the biggest community features include:

We outline every community feature in detail in, “Reviewing Every New Feature in HAProxy 3.0”.

Ready to upgrade?

To start the upgrade procedure, visit the installation instructions for HAProxy ALOHA 16.5.

New bot management makes identifying bots and categorizing your traffic a breeze

Our customers have implemented some impressive bot management strategies using HAProxy ALOHA’s tools for traffic profiling, tracking, and filtering. Now, it’s even easier to use HAProxy ALOHA as a powerful alternative to a separate bot management solution. The new Bot Management Module provides fast, reliable, and flexible bot identification and categorization with low latency and deep integration with HAProxy ALOHA’s multi-layered security controls. 

Why bot management?

From DoS attacks to content scraping, the risks from bot traffic are growing yearly. Failure to identify and block malicious bots could result in downtime, data theft, fraud, and more, affecting an organization’s reputation and revenue. Additionally, bot traffic can significantly increase resource use, which increases operational costs and could affect application performance for legitimate human users. 

To combat the rising risks, we wanted to make effective bot management more accessible and more powerful. In HAProxy ALOHA 16.5, customers now have access to the new Bot Management Module, a new weapon in their arsenal against malicious bots.

What can you do with the new Bot Management Module?

HAProxy ALOHA’s new Bot Management Module works out-of-the-box to identify traffic accurately, categorizing it as human, suspicious, bot, verified crawler (search engines), or verified bot/tool/app (non-browser). 

You can combine accurate bot identification with the other powerful layers in the security suite (including the next-generation HAProxy Enterprise WAF) to create customizable, high-performance, and low latency bot management and rate limiting strategies—from simple to advanced.

Why should you use the new Bot Management Module?

Three reasons:

  • Fast performance eliminates latency and ensures rapid bot identification and enforcement of bot management policies even under heavy load (e.g., DoS attack). 

  • Reliable bot management with a simple architecture reduces complexity and keeps your data local and secure.

  • Flexible and customizable bot management shares intelligence with other powerful security layers for smarter, more holistic decision-making and enforcement. 

For most users, we expect the simple answer to be: why wouldn’t you use it? 🙂 You can enable it in moments, and since it’s built into the firmware of HAProxy ALOHA—the plug-and-play hardware or virtual load balancer—it works quickly and efficiently even under heavy load. 

But the real question for many customers is: why use this instead of one of the market-leading bot management solutions? 

Unfortunately, bot management solutions often come with significant compromises (not even counting the extra cost).

  • Latency: solutions that pass requests through an additional layer, sometimes in a different network location, add latency (in addition to the often-quoted processing time) that affects the user experience.

  • Complexity: solutions that require a constant or frequent connection to the vendor’s cloud (for example, for automatic updates to the detection algorithm) introduce complexity and an additional point of failure, putting reliability and data privacy at risk. 

  • Lack of integration: solutions without deep integration with other security layers, such as with the WAF and anomaly detection layers, make decisions with incomplete information and do not give users the flexibility to enhance and customize their bot management strategy.

HAProxy ALOHA’s new Bot Management Module uses reputational signals and scoring based on HAProxy Technologies’ security expertise, data science, and large real-world datasets to identify traffic accurately. Our data science team uses the threat intelligence data provided by HAProxy Edge to train our security models with machine learning, resulting in extremely accurate and efficient detection algorithms for bots and other threats – without relying on static lists and regex-based attack signatures.

Importantly, all the detection, processing, and enforcement is local to the appliance. It does not add additional layers to the request path and does not require an external connection. This minimizes latency, maximizes reliability, and gives you the flexibility to deploy anywhere you like—such as in air-gapped environments.

With deep integration with HAProxy ALOHA’s multi-layered security, you can customize your organization’s bot management to meet your unique needs and traffic profile. You can customize your enforcement policies with options including blocking, tarpitting, challenging, and rate limiting.

But how good is it at identifying bots? While this is hard to test in a benchmark scenario, in real-world deployments with early adopters on HAProxy Enterprise, the Bot Management Module helped a top eCommerce website handling 300,000 requests per second identify heavy amounts of suspicious traffic and avoid crippling outages. As much as 20% of traffic was identified as anomalous, which their previous system had accepted without raising any security concerns.

Now that the HAProxy Enterprise Bot Management Module has come to HAProxy ALOHA, our appliance customers can benefit from its fast, reliable, and flexible bot management capabilities to protect their business and reputation and reduce the resource cost of serving requests from unwanted bots.

New Network Management CLI puts more power in your hands

Previously, administrators could only configure the HAProxy ALOHA network stack using:

  • traditional command-line operations such as ip route and ip rules within the Linux command line—an effective approach that can be complex and require advanced networking knowledge; or

  • the HAProxy ALOHA Services tab—a simpler and more widely used method for our appliance customers.

While these approaches are effective, they focus exclusively on appliance-specific networking configurations. With modern environments increasingly blending hardware, software, cloud, on-premises systems, containers, and virtual machines, it became imperative that we introduce a more open and powerful networking alternative.

With the release of HAProxy ALOHA 16.5, we’ve introduced the new Network Management CLI (netctl), a first-of-its-kind feature in the HAProxy product stack. The Network Management CLI redefines how users configure their networks, harnessing the power of the Network API for an easier and more consistent approach directly from the appliance.

What are the benefits of the Network Management CLI

Netctl isn’t just a simple command-line utility—it's an interface that interacts directly with the Linux Network API, giving users access to a powerful networking suite. It centralizes and abstracts networking commands, like ip route, ip address, and ip link, into a single interface. 

The Network Management CLI offers familiar and intuitive functionality for users accustomed to the Network Manager on Linux distributions. With netctl, you can program and manage the network environment directly from your HAProxy ALOHA appliance, making previously complex tasks, like creating link aggregations, defining VLANs, or managing IP routing, more accessible.

With the new Network Management CLI, HAProxy ALOHA users can:

  • Eliminate complexity by abstracting complex network tasks into simple CLI calls, enabling users to easily configure advanced setups such as link aggregation, virtual local area network (VLAN) over bridges, and virtual router redundancy protocol (VRRP) over VLANs.

  • Gain greater flexibility by providing a unified way to manage network settings without needing to switch between multiple tools or rely on extensive, manual command sequences.

  • Save time and avoid mistakes by streamlining the network setup process, reducing the manual effort required to implement complex setups, and minimizing the risk of human error.

The Network Management CLI demonstrates HAProxy Technologies’ commitment to providing extensive tools to its users over other offerings. It further enhances HAProxy ALOHA’s plug-and-play capabilities with a feature that now handles network configuration.

Enhanced reliability with updated network configuration scripts

In HAProxy ALOHA 16.5, we’ve also updated the network-scripts and config.rc to better support the Network API—which will manage the network stack of HAProxy ALOHA. This brings users more benefits beyond the Network Management CLI, including improved reliability and more efficient configuration.

With the updated network configuration scripts, users will benefit from:

  • Seamless rollback support by reverting to the previous configuration versions in the case of errors, ensuring continuity without requiring an appliance restart.

  • Streamlined VRRP configuration by automatically managing VRRP settings on interfaces, reducing complexity and minimizing misconfiguration.

  • Improved interface management by resolving issues such as deleting virtual interfaces.

New Wireguard VPN secures synchronization between appliances

In distributed environments, synchronizing configurations between appliances over a network can risk exposing sensitive data to potential security threats.

In HAProxy ALOHA 16.5, we’ve introduced Wireguard VPN, a powerful new feature that secures the way HAProxy ALOHA appliances in the same or different data centers communicate over a network.

Why Wireguard VPN?

When HAProxy ALOHA appliances operate in different data centers, synchronizing configuration can pose a risk if the appliances are not interconnected with a dedicated, private connection. While it’s possible to synchronize changes over the internet, this approach could lead to data being intercepted during transmission.

In HAProxy ALOHA 16.5, Wireguard VPN addresses this by providing a fully encrypted UDP tunnel of communication, ensuring that configuration data remains private and secure. Even in scenarios where data centers are interconnected, Wireguard VPN offers HAProxy ALOHA customers enhanced protection by encrypting all configuration data transmitted between the two appliances. This new secure tunnel ensures that bad actors monitoring your network cannot discover sensitive information about your HAProxy ALOHA deployment.

Enhanced stability and security with root filesystem updates

In HAProxy ALOHA 16.5, the root filesystem packages, including libraries, binaries, scripts, and all embedded components, have been updated to the latest version. This update inherits the maintenance of all the embedded open source projects, as well as security and functional fixes.

By updating the root filesystem, HAProxy ALOHA provides users with a more robust and reliable user experience.

Upgrade to HAProxy ALOHA 16.5

When you are ready to upgrade to HAProxy ALOHA 16.5, follow the link below.

Product

Release Notes

Install Instructions

Free Trial

HAProxy ALOHA 16.5

Release Notes

Installation of HAProxy ALOHA 16.5

HAProxy ALOHA Free Trial


]]> Announcing HAProxy ALOHA 16.5 appeared first on HAProxy Technologies.]]>
<![CDATA[Announcing HAProxy 3.1]]> https://www.haproxy.com/blog/announcing-haproxy-3-1 Tue, 26 Nov 2024 00:15:00 +0000 https://www.haproxy.com/blog/announcing-haproxy-3-1 ]]> Back in the spotlight, HAProxy 3.1 builds on its history of innovation, delivering improvements that ensure it remains the go-to open source load balancer! As usual, all the features announced in this release will be incorporated into the next enterprise release (HAProxy Enterprise 3.1).

HAProxy 3.1 brings improvements to observability, reliability, performance, and flexibility. So everything we love about HAProxy is now even better! Continual refinement of the things that matter most to HAProxy users is what helps HAProxy remain the G2 category leader in API management, container networking, DDoS protection, web application firewall (WAF), and load balancing.

In this blog post, we'll cover the changes in a short and digestible format. For a deeper dive into what’s new in version 3.1, subscribe to our blog to make sure you don’t miss part 2 (coming soon)!

Watch our webinar HAProxy 3.1: Feature Roundup and listen to our experts as we examine new features and updates and participate in the live Q&A. 

New to HAProxy?

HAProxy is the world’s fastest and most widely used software load balancer. It provides high availability, load balancing, and best-in-class SSL processing for TCP, QUIC, and HTTP-based applications.

HAProxy is the open source core that powers HAProxy One, the world’s fastest application delivery and security platform. The platform consists of a flexible data plane (HAProxy Enterprise and HAProxy ALOHA) for TCP, UDP, QUIC and HTTP traffic; a scalable control plane (HAProxy Fusion); and a secure edge network (HAProxy Edge).

HAProxy is trusted by leading companies and cloud providers to simplify, scale, and secure modern applications, APIs, and AI services in any environment.

How to get HAProxy 3.1

You can install HAProxy version 3.1 in any of the following ways:

Install the Linux packages for Ubuntu / Debian.

Run it as a Docker container. View the Docker installation instructions.

Compile it from source. View the compilation instructions.

]]> ]]> Major changes

First, let's cover the most important changes in HAProxy 3.1. These changes substantially modify how things were done in previous versions or introduce entirely new capabilities.

Log profiles

HAProxy logs have always been known as a treasure trove of information, offering extreme flexibility in customizing the log format. While this approach has served engineers well for years, evolving needs from DevOps and SecOps teams have pushed this feature to the next level. With HAProxy 3.1, you can now use log profile, a new configuration area designed to define the log format used at different stages of a transaction. These stages include key steps like accept, request, connect, response, close, error, or even any.

Each log profile is linked to a specific log destination server, which brings several advantages:

  • Tailored log formats per destination: No need to rely on post-processing at the syslog server before forwarding log entries again.

  • Logging at multiple stages: Capturing logs at various steps of the transaction simplifies troubleshooting and provides deeper insight into what’s happening.

Additionally, you can pair log profiles with the new do-log action, which lets you generate even more detailed logs as traffic flows through HAProxy. This gives you even greater control and visibility over your infrastructure.

Traces

Traces provide detailed debug messages about how different subsystems are running. They give you a way to dive deeply into diagnosing problems, offering valuable insights when dealing with complex issues. While traces have been part of HAProxy for a while, they were considered experimental and primarily used by developers. With HAProxy 3.1, traces are now officially supported (GA) and much easier to use—though they remain a tool designed for advanced debugging scenarios.

You can enable traces for various subsystems, including h1, h2, h3, quic, qmux, fcgi, spop, peers, check, and more. Traces now have their own dedicated configuration section and can even be controlled using the HAProxy Runtime API.

We’ll be publishing a dedicated blog post soon to walk you through everything you can do with traces, so stay tuned—it’s a game-changer for troubleshooting!

Rework of SPOE

Stream Processing Offloading Engine (SPOE) enables administrators, DevOps, and SecOps teams to implement custom functions at the proxy layer using any programming language. However, as HAProxy’s codebase has evolved, maintaining the original SPOE implementation became a bit more complex.

With HAProxy 3.1, SPOE has been updated to fully support HAProxy’s modern architecture, allowing greater efficiency in building and managing custom functions. It’s now implemented as a “mux”, which allows for fine-grained management of SPOP (the SPOE Protocol) through a new backend mode called mode spop. This update brings several benefits:

  • Support for load-balancing algorithms: You can now apply any load-balancing strategy to SPOP backends, optimizing traffic distribution.

  • Connection sharing between threads: Idle connections can be shared, improving efficiency on the server side and response times on the agent side.

Rest assured, backward compatibility has been a priority. If you’ve built SPOA (Agents) in previous versions of HAProxy, they’ll continue to work just fine with HAProxy 3.1.

HTTP/2 Performance

In the HTTP/2 protocol, each stream has a window size: this is the maximum volume of data that can be transferred before requiring an acknowledgement. By increasing this window size dynamically during a stream’s lifetime, performance can be significantly improved.

With HAProxy 3.1, this process is now automatic. HAProxy adjusts the per-stream window size for optimal efficiency, using dedicated buffers for each stream alongside a shared buffer pool. This enhancement delivers a dramatic boost in performance:

  • POST uploads are now up to 20x faster.

  • Head-of-line blocking is reduced when downloading from HTTP/2 servers, improving responsiveness.

For even more control, a couple of new tuning parameters have been introduced: tune.h2.fe.rxbuf for frontends and tune.h2.be.rxbuf for backends, allowing you to further extend the default settings to match your specific needs.

New Master/Worker Model

HAProxy 3.1 introduces significant improvements to the master/worker model, making the separation of roles cleaner and more efficient.

In previous versions, the master process was responsible for parsing the configuration and then undoing it before handing it over to the workers. This approach occasionally led to issues like inconsistencies during reloads or file descriptor leaks.

Now, the master’s role is limited to starting the worker processes. The workers handle configuration parsing and perform their tasks independently, resulting in more consistent operation across reloads.

Noteworthy changes

Beyond the major changes, HAProxy 3.1 includes several changes that simplify the configuration, improve performance, or extend existing functionality.

  • New quic-initial: Introduces actions to execute during the QUIC handshake, similar to tcp-request actions for TCP. The currently supported actions are reject, accept, dgram-drop (for a silent drop), and send-retry (to force a retry when in 0-RTT for example). These actions help prevent abuse or enforce source-based filtering so that the client cannot even engage in a handshake.

  • New set-retries action: Available in tcp-request and http-request rules, this action allows HAProxy to dynamically change the number of desired retries at runtime. This is particularly useful for adapting HAProxy’s behavior to specific parts of an application, or for learning the desired number of retries from client-side information.

  • New when(condition) converter: Evaluates the condition, and if true passes the input sample as-is; otherwise it returns nothing. This allows emitting extra debugging information or triggering some actions only when some conditions are met. This prevents filling the logs with unnecessary information and can be combined with the bs.debug_str and fs.debug_str fetches to help developers better understand a problem.

  • New bs.debug_str / fs.debug_str fetches: Report useful debugging information from HAProxy’s internal layers. Use these for debugging purposes. You can trigger them with the when() converter above.

  • New last_entity / waiting_entity fetches: Indicate what waiting operation was interrupted by a timeout or an error. For example, it may help detect problems with Lua scripts or SPOA subsystems like compression, or even detect when a full body was not received. This can also report the last rule that was evaluated by HAProxy because of an accept / redirect or deny for example.

  • QUIC pacing: Smooths packet distribution over time to avoid overflowing buffers on the path to client applications. (e.g. browser). While it incurs higher CPU usage, it can improve performance by 10–20x on lossy networks or with slow clients. This feature is currently experimental.

  • New bbr congestion algorithm for QUIC: bbr (Bottleneck Bandwidth and Round-trip propagation time) uses network measurements, including packet loss, to model the path and determine optimal data transfer speeds. This allows higher throughput and lower queueing than other congestion algorithms on lossy networks or weak clients. bbr relies on the pacing mechanism and as such is also currently experimental.

  • Option httpchk now supports a Host header: This is one of the oldest checks in HAProxy and it now officially supports a Host header, eliminating the need for a workaround with fake strings in the httpchk line.

  • New server’s init-state: allows a server to remain down (not on) at startup or when leaving maintenance until the first health check succeeds.

Deprecated features

Several features are now marked as deprecated and will trigger warnings unless you set the expose-deprecated-directives global option. Unless noted otherwise, these features will be removed from version 3.3. Most of the deprecated features have a replacement.

  • The program section is marked as deprecated in this release and will be removed in HAProxy 3.3. 

  • The opentracing filter will be marked as deprecated in HAProxy 3.3 and will be removed in HAProxy 3.5.

  • Another noticeable change is to block duplicate names between various families of proxies (frontend/listen/backend/defaults/log-forward etc) and between servers. Duplicates are now properly detected and will report a deprecation warning in 3.1, indicating a breakage in 3.3.

  • The legacy C-based mailers are also deprecated in 3.1 and will be removed in 3.3. The customizable Lua mailers introduced in HAProxy 2.8 will then be the only way to set up mailers.

Breaking changes

HAProxy 3.1 did not introduce any breaking changes. However, to improve communication about future changes, the R&D team created a Wiki page. This page provides a summary of upcoming breaking changes and the releases in which they are planned, helping you stay informed and prepared: https://github.com/haproxy/wiki/wiki/Breaking-changes.

Conclusion

As always, none of this would be possible without the amazing HAProxy community. Your feedback, suggestions, code commits, testing, and documentation benefits millions of users worldwide as they use HAProxy to master their application traffic. To everyone who contributes – thank you.

Subscribe to our blog below and stay tuned for further deep dives on the latest updates from HAProxy 3.1. And in case you missed it, catch up with the new features we announced earlier this month in HAProxy Enterprise 3.0.

Ready to upgrade to HAProxy 3.1? Here’s how to get started.

Last but not least, we're thrilled to kick off HAProxyConf 2025 next summer in San Francisco.HAProxyConf celebrates the thriving community that's helped make HAProxy One the world's fastest application delivery and security platform. Over 2+ days, expert speakers will share best practices and real-world use cases that highlight HAProxy's next-gen approach to high-performance application delivery and security. Check out the HAProxyConf official website to learn more, stay updated, and answer our call for papers!

]]> Announcing HAProxy 3.1 appeared first on HAProxy Technologies.]]>
<![CDATA[KubeCon NA 2024: Service Discovery, Security, and AI—Oh My!]]> https://www.haproxy.com/blog/kubecon-na-2024-service-discovery-security-and-ai-oh-my Thu, 21 Nov 2024 09:31:00 +0000 https://www.haproxy.com/blog/kubecon-na-2024-service-discovery-security-and-ai-oh-my ]]> Though KubeCon North America 2024 has officially come to a close, the CNCF's flagship event has left us buzzing with residual excitement. After all, waving goodbye to a crowd of 9,200+ attendees is never easy—especially with Salt Lake City's snow-capped mountains towering impressively in the background. 

However, it was our conversations with HAProxy booth visitors that truly stole the show. From the casual hello to the deeply technical dive, attendees eagerly shared their Kubernetes experiences and infrastructure challenges. Exploring these obstacles and corresponding tech trends was a tremendously rewarding experience. Our booth team lives for those "aha" moments where solutions meet user problems head-on and emerge triumphant. 

Here's what we've learned from our four days alongside DevOps professionals, engineers, architects, and fellow K8s enthusiasts.

We've seen how Kubernetes (and open source software) constantly changes at HAProxy—as do the trending topics around it. Unsurprisingly, KubeCon North America 2024 demonstrated that such tech trends are as ephemeral as K8s pods. Our conversations with booth visitors repeatedly touched on some key topics:

AI and ML

We noticed an accelerated shift in interest this year towards AI/ML, which isn't shocking since 58% of organizations are actively experimenting with large language models (LLMs). And since today's experiments will likely be tomorrow's deployments, organizations are rightfully weighing load balancing options for AI/ML workloads. New ML training models and uses are emerging each day, and vendors like us have taken note. In fact, roughly 50% of KubeCon booths incorporated AI messaging in some way, shape, or form! 

We fielded multiple related questions and observed an uptick in Kubernetes adoption specifically to support these applications. AI/ML is an exciting frontier we're eagerly exploring, which is why HAProxy One offers AI/API gateway support. Because our multi-cluster routing support is so capable, we're well positioned to support these bleeding-edge technologies as they mature.

Application security

Unsurprisingly, security remains a hot-button issue for the vast majority of organizations. KubeCon also demonstrated that while web application firewalls (WAFs) and bot management are critical, new and novel approaches to DDoS mitigation have garnered heavy interest. Customization and performance have become differentiators as users demand more from their security suites.

While K8s can do plenty on its own, security features such as HAProxy’s stick tables and HAProxy Enterprise's Global Profiling Engine (GPE) provide highly-effective supplemental protection against application-layer attacks. You can even deploy a stick table purely for bot-labeled requests to further reduce false positives. 

The CAPTCHA module provides another layer of actionable protection, The module supports reCAPTCHA v2, reCAPTCHA v3, reCAPTCHA Enterprise, hCaptcha, Friendly Captcha (frCaptcha), and Turnstile. We chatted with many attendees who were eager to fill their existing security gaps with these features.

Service discovery

Service discovery has always been integral to uncovering active Kubernetes services and pushing configuration changes on the fly. While the concept isn't new, software vendors offer slightly unique flavors of service discovery with varied levels of automation and performance. 

We chatted with many booth visitors who enthusiastically discussed their discoverability needs while expressing a desire for improved scalability for their K8s services. Our team showcased our high-performance service discovery (introduced in HAProxy Fusion 1.2 and improved in HAProxy Fusion 1.3) to great fanfare in Salt Lake City—fueling our fire to further improve the feature. 

With the power to automatically generate over 100,000 lines of HAProxy configuration in seconds, HAProxy Fusion service discovery is a perfect match for large-scale deployments. We're thrilled to continually improve upon this key Kubernetes capability and showcase our progress at the next KubeCon. Don't miss it!

Answering your K8s questions

]]> ]]> KubeCon was a massive convergence of ideas and curiosities, all flowing like a firehose without an off valve. We quickly learned that vendor lock-in remains a primary concern for plenty of organizations, and simplicity is a guiding principle for many grappling with Kubernetes complexities. Attendees also perked up for some exciting new development avenues, such as egress gateways and TLS-based SNI allowlisting using access control lists (which one of our customers is already doing successfully). 

Naturally, attention soon turned to us and our development roadmap. Visitors kept us occupied with numerous questions about our platform and vision behind the evolving HAProxy One platform. Here are responses to some common questions our booth team received:

Does HAProxy offer Ingress functionality?

Yes! HAProxy's comprehensive Kubernetes solution includes Kubernetes Ingress support for organizations requiring simple setup, low resource use, high performance, and cost efficiency. 

Ingress control exists alongside our unique approach to intelligent external load balancing, multi-cluster routing, and blue-green deployments. Deploy HAProxy Enterprise Kubernetes Ingress Controller independently or together with other components in our K8s solution, according to your load balancing needs. 

These features are fulfilled by different products within HAProxy One, so we'd love to chat and determine what best fits your needs.

With Ingress' development "frozen," is Gateway API the way forward?

We're continually evaluating Gateway API support in HAProxy One and plan to bring Gateway API to HAProxy Fusion Control Plane. We want to support as many customer use cases (and preferences) as possible. We also anticipate that organizations will increasingly migrate away from Ingress to an alternative solution. 

Our Kubernetes solution enables external load balancing and multi-cluster routing without the added complexities of Gateway API. HAProxy enables you to route traffic directly from your external HAProxy Enterprise nodes to your Kubernetes pods without having to use Ingress or Gateway API at all. 

There's no management overhead with vendor-specific policies, nor a need to install additional custom resource definitions (CRDs) unless that helps your use case. Using HAProxy Fusion to automate direct-to-pod load balancing also eliminates a network hop normally associated with querying Gateway API. This reduces latency for massive-scale K8s applications while removing a potential point of failure. 

We're truly excited to see how HAProxy Enterprise and HAProxy Fusion service discovery can help our customers' applications perform better at massive scale. This approach is future-proof and a great next step for users should Ingress reach end-of-life.

Thanks for engaging with us

As always, your questions excited us, challenged us, and have even inspired us to redefine what's possible with Kubernetes and HAProxy One. It was great catching up with fresh and familiar faces alike. We never get tired of taking visitors through live demos or deep whiteboard sessions—blending plenty of technical knowledge with a little artistic flair. 

The HAProxy community also deserves a gigantic shoutout for its willingness to share valuable feedback. KubeCon left us (happily!) drowning in a tidal wave of G2 reviews, which help us improve HAProxy One and prioritize popular feature requests. Please, keep those opinions coming and your voices loud!

Come see us next year!

KubeCon North America 2024 blew us away. Our conversations have helped us better understand the evolving needs of the K8s community and better position ourselves as a leader in container networking. 

It's now time to flip the page to KubeCon Europe 2025 and KubeCon North America 2025. We can't wait to unveil some exciting new developments and see how the Kubernetes landscape changes. We'll also be at AWS re:Invent 2024 in Las Vegas, from December 2nd to December 6th. Come see us at Booth 571!

Last but not least, we're thrilled to kick off HAProxyConf 2025 next summer in San Francisco. HAProxyConf celebrates the thriving community that's helped make HAProxy One the world's fastest application delivery and security platform. Over 2+ days, expert speakers will share best practices and real-world use cases that highlight HAProxy's next-gen approach to high-performance application delivery and security. Check out the HAProxyConf official website to learn more, stay updated, and answer our call for papers! 

Want to learn more about HAProxy and Kubernetes?

To dive a little deeper into our Kubernetes solutions and story, check out these helpful resources: 

Our products and HAProxy One—the world’s fastest application delivery and security platform—are always evolving. Stay tuned for important updates and development milestones! Thank you for another fantastic KubeCon.

]]> KubeCon NA 2024: Service Discovery, Security, and AI—Oh My! appeared first on HAProxy Technologies.]]>
<![CDATA[Announcing HAProxy Enterprise 3.0]]> https://www.haproxy.com/blog/announcing-haproxy-enterprise-3-0 Thu, 14 Nov 2024 00:00:00 +0000 https://www.haproxy.com/blog/announcing-haproxy-enterprise-3-0 ]]> HAProxy Enterprise 3.0 is now available. This release extends HAProxy Enterprise’s legendary performance and flexibility and builds upon its cornerstone features. The HAProxy Enterprise WAF is even more powerful, the Global Profiling Engine is more dynamic and performant, UDP load balancing is simpler and more observable, HTTPS performance is improved, and we have added new CAPTCHA and SAML single sign-on modules.

New to HAProxy Enterprise?

HAProxy Enterprise provides high-performance load balancing, can serve as an API gateway, and provides Kubernetes routing and ingress, TLS offloading, bot management, global rate limiting, and a next-generation WAF. HAProxy Enterprise combines the performance, reliability, and flexibility of our open-source core (HAProxy – the most widely used software load balancer) with ultra-low-latency security layers and world-class support. HAProxy Enterprise benefits from full-lifecycle management, monitoring, and automation (provided by HAProxy Fusion), and next-generation security layers powered by threat intelligence from HAProxy Edge and enhanced by machine learning.

To learn more, contact our sales team for a demonstration or request a free trial.

What’s new?

HAProxy Enterprise 3.0 includes new enterprise features plus all the features from the community version of HAProxy 3.0. For the full list of features, read the release notes for HAProxy Enterprise 3.0.

HAProxy Fusion will support HAProxy Enterprise 3.0 soon.

New in HAProxy Enterprise 3.0 are the following important features:

  • Strengthened HAProxy Enterprise WAF robustness and security precision. The next-generation HAProxy Enterprise WAF powered by our Intelligent WAF Engine is now even better at detecting disguised threats with new features such as base64 decoding and the ability to process requests without Content-Type.

  • A more dynamic and performant Global Profiling Engine. The Global Profiling Engine has been upgraded with dynamic peer support, enabling load balancers to connect to it without explicitly being added to the GPE configuration file. The ability to learn peers dynamically results in a lower memory footprint due to peer and session reuse.

  • Improved HTTPS performance and reliability. We’ve improved HTTPS performance as a result of redistributing and defaulting to OpenSSL 1.1.1.

  • New logging capabilities and simplified configuration with the HAProxy Enterprise UDP Module. The HAProxy Enterprise UDP Module now provides logging capabilities for enhanced observability, along with simplified configuration with support for the default-server directive.

  • A new CAPTCHA module. The new CAPTCHA module supports more providers and enables simpler configuration management. The supported modes include reCAPTCHA v2, reCAPTCHA v3, reCAPTCHA Enterprise, hCaptcha, Friendly Captcha (frCaptcha), and Turnstile.

  • A new SAML module. The new SAML single sign-on module is now embedded in HAProxy Enterprise as a native module and is easier to configure.

We announced the release of HAProxy 3.0 in May 2024, which included improved simplicity, reliability, security, and flexibility. The features from HAProxy 3.0 are now available in HAProxy Enterprise 3.0.

Some of these biggest community features include:

  • crt-store feature. Separates certificate storage from frontend use, simplifying and scaling SSL/TLS certificate management.

  • Enhanced HTTP/2 stack. Adds the option to limit and track glitchy HTTP/2 connections. HAProxy’s ability to handle the HTTP/2 CONTINUATION Flood demonstrates its resilience with this type of connection.

  • Persistent stats after reloads. Stats are preserved using the Runtime API command dump stats-file and the stats-file directive, provided proxy objects have assigned GUIDs.

  • Machine-readable logs. Supports JSON and CBOR formats for easier log management and system interoperability.

  • Improved stick table performance. Lock contention reduced by sharing data across smaller, individual tables with separate locks.

  • Differentiated Services field support. Allows classification and traffic prioritization by setting the DS field on both frontend and backend connections via set-fc-tos and set-bc-tos actions.

  • Virtual ACL and map files. Enables in-memory ACL and map file representations using the virt@ prefix, avoiding filesystem searches.

We outline every community feature in detail in, “Reviewing Every New Feature in HAProxy 3.0”.

Ready to upgrade?

When you are ready to start the upgrade procedure, go to the upgrade instructions for HAProxy Enterprise.

]]> ]]> Delivering greater robustness and precision with HAProxy Enterprise WAF

In the last release, we introduced the next-generation HAProxy Enterprise WAF, powered by the Intelligent WAF Engine. This unique engine delivers exceptional accuracy, zero-day threat detection, ultra-low latency, and simple management. Now, in HAProxy Enterprise 3.0, we’ve further enhanced its robustness and security precision.

With the addition of new features, the Intelligent WAF Engine is even more capable of detecting obfuscated threats. These updates strengthen the already powerful HAProxy Enterprise WAF, providing enhanced security against sophisticated attacks and improved accuracy in identifying disguised attacks.

We’ve previously discussed the incredible accuracy of the HAProxy Enterprise WAF, which achieved a true-positive rate of 99.61%, comfortably beating the category average. With the release of HAProxy Enterprise 3.0, the true-positive rate has climbed to 99.84%, tested using open source WAF benchmark data. False-negatives that were already virtually eliminated are now approaching zero.

Additionally, the HAProxy Enterprise WAF continues to deliver a robust true-negative rate of 97.124%, resulting in a balanced accuracy of 98.48%. With the false-positive rate remaining low at 2.876%, these metrics underscore the consistent and reliable performance of the HAProxy Enterprise WAF.

What’s new in the HAProxy Enterprise WAF?

The new capabilities of the HAProxy Enterprise WAF include:

  • Support for base64 decoding to better identify threats that use base64 encoding to obfuscate malicious payloads.

  • The ability to parse requests without a Content-Type to inspect malformed requests and minimize false positives.

  • Support for atomic ruleset updates through the Runtime API, eliminating the need for external tools and reducing complexity and the likelihood of error-prone updates.

  • Prometheus exporter metrics that make monitoring more efficient, including the total number of HTTP requests processed and blocked by an HAProxy Enterprise WAF instance.

With HAProxy Enterprise 3.0, the HAProxy Enterprise WAF delivers superior detection of deceptive threats and reliability, surpassing other vendor solutions that struggle with complex, evasive attacks.

For organizations seeking a solid web application firewall, HAProxy Enterprise WAF offers a robust defense that enhances your infrastructure’s security.

]]> ]]> Upgraded Global Profiling Engine brings enhanced scalability and performance

The Global Profiling Engine helps customers maintain a unified view of client activity across an HAProxy Enterprise cluster. By collecting and analyzing stick table data from all nodes, the Global Profiling Engine offers real-time insight into current and historical client behavior. This data is then shared across the load balancers, enabling informed decision-making such as rate limiting to manage traffic effectively.

In HAProxy Enterprise 3.0, we upgraded the Global Profiling Engine, which now offers dynamic peer support and a much lower memory footprint. This upgrade brings enhanced scalability and improved performance to clients.

What is dynamic peer support in the Global Profiling Engine?

With dynamic peers, load balancers can now connect to the Global Profiling Engine without explicitly being added to the configuration. This means that when new nodes are added or removed from a cluster, they can seamlessly connect or disconnect to the Global Profiling Engine, with all data and configuration automatically shared between them.

Dynamic peer support ensures that each node in a cluster can instantly synchronize data about client behavior and traffic patterns, without the need for administrators to manually configure and manage peer support. This makes it easier to enforce rate limiting policies at a global level and enables customers to make real-time, informed decisions as their system scales, offering cluster-wide data tracking and aggregation—now more dynamic and efficient than ever.

Dynamic peer support also brings customers better memory management due to peer reuse and session reuse. Using the same resources multiple times minimizes memory allocation, resulting in a much lower memory footprint.

Ultimately, the upgraded Global Profiling Engine is a more resource-efficient and scalable solution—and we hope customers take advantage of its dynamic capabilities.

Enhanced TLS performance with OpenSSL optimization

HAProxy Enterprise allows customers to encrypt traffic between the load balancer, clients, and backend servers using TLS.

With the release of HAProxy Enterprise 3.0, TLS performance has been optimized by switching from OpenSSL 3.X to OpenSSL 1.1.1 as the default for relevant operating systems. While this may be a notable change for some customers, the OpenSSL optimization will ultimately bring better performance and reliability for their systems.

]]> ]]> Simplified and more observable UDP load balancing

Customers love the HAProxy Enterprise UDP Module because it delivers fast, reliable UDP proxying and load balancing. By unifying UDP, TCP, and HTTP load balancing under a single solution, HAProxy Enterprise simplifies infrastructure management and eliminates the need for multiple products from other vendors.

Now with the release of HAProxy Enterprise 3.0, there’s more to love about the UDP module. When load balancing UDP traffic, customers now have access to logging capabilities for enhanced observability, along with support for the default-server directive, making configuration easier than before.

Basic logging can be enabled by specifying the log keyword and its arguments in the udp-lb section. Currently, the log output format contains the source and destination addresses, bytes received and sent, the instance name, and the server—and we plan to expand capabilities further in the future.

To learn more about configuring logging for UDP load balancing, visit our documentation.

Previously, configuring UDP load balancing in HAProxy Enterprise required manually specifying each server, which required more time and effort, especially when managing a large number of servers. But now, with the default-server directive, customers can specify these settings once and apply them uniformly across multiple servers. The end result is a more streamlined and simpler configuration process. 

Our documentation outlines how to take advantage of this new directive to improve your workflow.

This enhancement, along with logging capabilities, further strengthens the HAProxy Enterprise UDP Module, which already delivers best-in-class UDP performance compared to other software load balancers. With these updates, customers gain not only a highly performant and scalable UDP proxying and load balancing solution but also one that offers enhanced observability and simplified configuration management.

New CAPTCHA and SAML modules

HAProxy Enterprise 3.0 brings two new native modules to customers:

  • The CAPTCHA Module

  • The SAML Module

Both of these modules, while having different functions, simplify HAProxy Enterprise configuration for customers.

New CAPTCHA Module

This release introduces a new CAPTCHA module that simplifies configuration while extending support to more CAPTCHA providers, including Google reCAPTCHA Enterprise, the biggest one.

Some of the supported modes include:

  • reCAPTCHA v2

  • reCAPTCHA v3

  • reCAPTCHA Enterprise

  • hCaptcha

  • Friendly Captcha (frCaptcha)

  • Turnstile

Similar to the previous implementation, the new CAPTCHA module presents a challenge page to clients to determine if the user is a human. The only difference this time is that the new CAPTCHA module is now embedded in HAProxy Enterprise as a native module. This results in a module that supports more CAPTCHA providers beyond Google reCAPTCHA, can easily integrate with other providers not listed above, and is much simpler to configure.

The previous reCAPTCHA module required customers to configure the module through an extra configuration file and to add more to hapee-lb.cfg. With the new CAPTCHA module, a new section is added to the hapee-lb.cfg where all the settings go—a much simpler, streamlined process to verify that clients are humans.

In HAProxy Enterprise 3.0, implementing a CAPTCHA solution is simpler than ever, making it easier to integrate CAPTCHA verification into your HAProxy Enterprise setup without compromising security.

New SAML Module

This release also includes a new Security Assertion Markup Language (SAML) module, which provides single sign-on to any web application behind HAProxy Enterprise.

Previously, SAML was supported through an SPOE Agent, but with HAProxy Enterprise 3.0, the SAML module is now running in HAProxy Enterprise, greatly simplifying configuration. With the new SAML module, customers no longer have to configure the module in a separate SPOA configuration and can instead merge the configuration into hapee-lb.cfg.

Upgrade to HAProxy Enterprise 3.0

When you are ready to upgrade to HAProxy Enterprise 3.0, follow the link below.

Product

Release Notes

Install Instructions

HAProxy Enterprise 3.0

Release Notes

Installation of HAProxy Enterprise 3.0

Try HAProxy Enterprise 3.0

The world’s leading companies and cloud providers trust HAProxy Technologies to protect their applications and APIs. High-performing teams delivering mission-critical applications and APIs need the most secure, reliable, and efficient application delivery engine available. HAProxy Enterprise’s no-compromise approach to secure application delivery empowers organizations to deliver next-level enterprise scale and innovation.

There has never been a better time to start using HAProxy Enterprise. Request a free trial of HAProxy Enterprise and see for yourself.

]]> Announcing HAProxy Enterprise 3.0 appeared first on HAProxy Technologies.]]>
<![CDATA[Nearly 90% of our AI Crawler Traffic is From TikTok Parent Bytedance – Lessons Learned]]> https://www.haproxy.com/blog/nearly-90-of-our-ai-crawler-traffic-is-from-tiktok-parent-bytedance-lessons-learned Thu, 31 Oct 2024 10:01:00 +0000 https://www.haproxy.com/blog/nearly-90-of-our-ai-crawler-traffic-is-from-tiktok-parent-bytedance-lessons-learned ]]> This month, Fortune.com reported that TikTok’s web scraper — known as Bytespider — is aggressively sucking up content to fuel generative AI models. We noticed the same thing when looking at bot management analytics produced by HAProxy Edge — our global network that we ourselves use to serve traffic for haproxy.com. Some of the numbers we are seeing are fairly shocking, so let’s review the traffic sources and where they originate.

Our own measurements, collected by HAProxy Edge and filtered to traffic for haproxy.com, show a few interesting figures:

  • Nearly 1% of our total traffic comes from AI crawlers

  • Close to 90% of that traffic is from Bytespider, by Bytedance (the parent company of TikTok)

]]> ]]> While Bytespider is currently the most prevalent AI crawler, showing that Bytedance is currently the top source, we have previously observed others (such as ClaudeBot) taking the top spot. AI crawler activity, like all traffic, changes over time.

What does AI traffic mean for us – and you?

While we are primarily a technology company, we also consider ourselves to be a content company; we invest in original, human-authored content — such as documentation or blogs that provide helpful information to our users and wider audience.

Content-scraping bots existed long before LLMs started crawling the web for generative AI applications, and they have usually been considered undesirable visitors on content-heavy websites. Many businesses would not consent to the scraping and possible re-use of their content, in full or in part, by a third party. 

However, AI crawlers used by LLMs come with unique risks and opportunities.

  1. On one hand, an LLM might re-use the original content in full, or with some modification, or remixed with other content at the level of an LLM token (roughly the level of a single word). It is unlikely that a user will know where the original content came from. In cases where an LLM “hallucinates”, a user might receive inaccurate information, for example when requesting code or configuration instructions.

  2. On the other hand, with many users turning to AI chatbots as an alternative to traditional search engines, this is becoming an important channel for discovery and awareness. Businesses might want their brand or product information to be supplied by chatbots in response to user queries. For example, if a user asks for a list of relevant products, a business might want their product to be included in the list, along with features and benefits.

While we don’t limit AI crawlers on our website right now, we will have to make a decision whether to continue to allow them or not. Other businesses running content-heavy public websites will likely find themselves having to make the same decision: to protect the value of their content, or to allow the dissemination of information about their brand and products via these new channels.

What can you do to protect your content from AI crawlers?

If bots and the risk of content replication pose a threat to your business, you need a strategy to mitigate this risk and a technology solution that enables you to implement it.

A common method of disallowing bots is to use the robots.txt file on your website domain. However, some AI crawlers (including Bytespider) don’t identify themselves transparently; they try to pretend to be real users and ignore instructions in robots.txt. It is for this reason that we — like the Fortune.com article — describe the crawling as “aggressive”. It is not only a matter of scale but also the way it is being done. 

Therefore, any technical solution for managing AI crawlers and scrapers must be capable of accurately identifying such bots, even when they are designed to be hard to distinguish from humans.

HAProxy Enterprise customers already benefit from the HAProxy Enterprise Bot Management Module, announced in version 2.9. This technology combines a simple and efficient method for identifying and classifying bots with HAProxy’s legendary flexibility, to support a range of bot management strategies — such as blocking, rate limiting, or challenging via CAPTCHA. 

Our guide, How to Reliably Block AI Crawlers Using HAProxy Enterprise, shows you how to identify and block these bots (either individually or as a category) using a few lines of configuration on HAProxy Enterprise. Other providers, such as our friends at Cloudflare, recently provided a similar solution.

Where does our data come from, and how do we use it to improve bot management?

Our traffic statistics from HAProxy Edge show that the scale of AI crawler traffic is significant and growing fast. Let’s talk about where our data comes from and how we use it.

HAProxy Edge provides a globally distributed application delivery network (ADN) that provides fully managed application services, accelerated content delivery, and a secure partition between external traffic and your network.

By analyzing the traffic connecting to websites and applications hosted on HAProxy Edge (which includes haproxy.com), we can build a picture of global traffic trends. We can also filter these traffic metrics to show AI crawlers. Our bot management technology performs rapid identification and classification of bots (and humans), including identification of known AI crawlers such as:

  • Bytespider (TikTok)

  • OpenAI search bot and ChatGPT variants

  • PerplexityBot

  • Google AI crawler

  • ClaudeBot

  • Others

Our data science team uses the threat intelligence data provided by HAProxy Edge to train our security models with the use of machine learning, resulting in extremely accurate and efficient detection algorithms for bots and other threats – without relying on static lists and regex-based attack signatures. We use these algorithms to power the security layers in HAProxy Edge itself and HAProxy Enterprise and HAProxy Fusion. This includes the HAProxy Enterprise WAF (powered by the Intelligent WAF Engine) and the HAProxy Enterprise Bot Management Module.

For businesses looking for fully managed application services, HAProxy Edge provides bot management and other security features, backed by HAProxy Technologies’ authority on all aspects of the load balancing and traffic control stack. Contact us if you’d like a demo or a trial.

]]> Nearly 90% of our AI Crawler Traffic is From TikTok Parent Bytedance – Lessons Learned appeared first on HAProxy Technologies.]]>
<![CDATA[Announcing HAProxy Fusion 1.3]]> https://www.haproxy.com/blog/announcing-haproxy-fusion-1-3 Tue, 29 Oct 2024 01:00:00 +0000 https://www.haproxy.com/blog/announcing-haproxy-fusion-1-3 ]]> HAProxy Fusion 1.3 is now available—making it even simpler to adopt modern, scalable application delivery. New custom dashboards, high-performance Kubernetes service discovery, and optimized workflows bolster HAProxy Fusion's observability and flexibility.

New to HAProxy Fusion?

HAProxy Fusion provides full-lifecycle management, monitoring, and automation of multi-cluster, multi-cloud, and multi-team HAProxy Enterprise deployments. HAProxy Fusion combines a high-performance control plane with a modern GUI and API (with 100% coverage), enterprise administration, a comprehensive observability suite, and infrastructure integrations including AWS, Kubernetes, Consul, and Prometheus. HAProxy Fusion and HAProxy Enterprise benefit from next-generation security layers powered by threat intelligence from HAProxy Edge and enhanced by machine learning.

Together, this flexible data plane, scalable control plane, and secure edge network form HAProxy One: the world’s fastest application delivery and security platform that is the G2 category leader in API management, container networking, DDoS protection, web application firewall (WAF), and load balancing.

To learn more, contact our sales team for a demonstration.

What's new?

HAProxy Fusion 1.3 is packed with major quality-of-life improvements. For the complete list of features, please view the HAProxy Fusion 1.3 release notes (customer login required). Key additions include:

  • A new, customizable Security dashboard complete with bot management and WAF monitoring statistics

  • An enhanced Access Logs dashboard with support for creating custom views into all your observability data

  • More powerful service discovery integration with Kubernetes and Consul, including support for filtering based on labels (AKA "tags") for services and namespaces

  • Improved configuration management, including a new fast backends feature for near-instant configuration generation from service discovery registries, with minimal overhead

  • A more powerful and searchable Request Explorer interface with added security details

  • Line-by-line or by-key .map file editing, letting you make edits with high concurrency of changes

  • An upgraded method for sending access logs from HAProxy Enterprise to HAProxy Fusion or your syslog servers

This only scratches the surface of what HAProxy Fusion 1.3 brings to the table. We'll dive into the above features to explain how they make it even easier to use HAProxy Fusion to manage large-scale deployments.

]]> Ready to upgrade?

When you're ready to start the upgrade process, please carefully read our HAProxy Fusion upgrade documentation (customer login required). We've also backported multiple features to HAProxy Fusion 1.2, for teams that are unable to upgrade to the new version right away.

]]> ]]> New Security dashboard improves observability for threat management

HAProxy Enterprise 2.9 constituted a major leap forward for multi-layered security in HAProxy, introducing the new HAProxy Enterprise WAF and HAProxy Enterprise Bot Management Module. Now, HAProxy Fusion 1.3 upgrades the monitoring capabilities for these layers to truly empower teams with the data needed for threat response. It also boosts transparency into how HAProxy Enterprise clusters operate. 

HAProxy Fusion 1.3 adds a customizable Security dashboard to the Dashboards picker under "Overview." You can now view a variety of contextual information for HAProxy Enterprise WAF and HAProxy Enterprise Bot Management Module, neatly organized by pane. You can drap, drop, resize, and remove these individual dashboard sections according to your preferences. That includes creating a unified view for both security layers (example shown below), or a focused view centered solely on the metrics you care about:

]]> ]]> HAProxy Fusion 1.3 lets you create a Security dashboard from scratch or save layout templates for later use. And for those using a custom layout, reverting to the default view takes just a few clicks. Users no longer have to dig through custom log data or stick table details, or leverage other tools to locate these useful metrics. Improved RBAC support also gives administrators granular control over who can view what.

New Access Logs dashboard brings customization and granularity

Active HAProxy Enterprise nodes track numerous performance metrics such as total requests, average requests, bandwidth in and out, and others. As its name suggests, the new Access Logs dashboard in HAProxy Fusion 1.3 can draw from your access logs—viewable in Request Explorer—to display much broader data than before. HTTP methods, IP addresses, and GeoIP metrics are just a few examples. 

New tools also make it easier to view the contextual data you care about. Users can choose a date (or date range) by zooming or dragging in the chart—or by using the calendar for historical data. These controls are identical to those present within the Native Statistics dashboard since HAProxy Fusion 1.2.

]]> ]]> To help you drill down even further, the Access Logs dashboard supports advanced data filtering by cluster, resource, backend, frontend, IP address, WAF field, and more. Each pane of the dashboard is also resizable and movable, letting you create the perfect bird’s-eye view of cluster performance and security. Users can create and share their own customized dashboards using hundreds of combinations of stats and visualization styles.

Finally, these HAProxy Enterprise statistics are now reviewable for longer periods of time. You can preserve basic metrics for months without worrying about storage requirements, as the quantity and complexity of collected stats are relatively low. HAProxy Fusion 1.3 gives you more control over your data.

]]> Service discovery becomes more capable

HAProxy Fusion 1.2 first added service discovery for Kubernetes (K8s) and Consul environments. You can use this to automatically identify backend services (such as Kubernetes pods) and configure external routing and load balancing in your HAProxy Enterprise nodes. 

HAProxy Fusion 1.3 builds on this foundation by bolstering Kubernetes infrastructure integration with new-and-improved service discovery features. You can now do the following in HAProxy Fusion 1.3: 

  • Define which K8s services are load balanced using labels on namespaces and services. This makes HAProxy Fusion service-aware within large-scale Kubernetes environments through Kubernetes-native configuration, and only pulls in the services you want.

  • Use the new fast backends feature to generate configuration for a very large number of Kubernetes or Consul services, with minimal overhead.

The optional fast backends transformer is an important addition for organizations that host numerous services in Kubernetes and thus scale out frequently. We've revamped the configuration code to reduce human error and slash configuration upload time—based on service discovery information—from minutes to mere seconds

Service discovery data can change (and become obsolete) every few seconds, so restoring earlier versions isn't very useful. However, we've maintained a reviewable historical record of previous runs for HAProxy Fusion users.

]]> ]]> This performance benefit of fast backends is most noticeable at massive scale. HAProxy Fusion can now dynamically generate over 100,000 lines of configuration in seconds without encountering bottlenecks. Any associated backends appear as normal HAProxy Enterprise backends in your HAProxy configuration, supplied directly by Kubernetes. 

You'll see this configuration populate in the newly added read-only portion of the editor. It lives alongside other sections delivered to your load balancers and available for use in configuration—such as HAProxy variables populated by HAProxy Fusion or other logging settings. While you can see all portions of your configuration in one unified view, this setup mitigates human error and helps simplify the complex task of managing Kubernetes.

Request Explorer interface displays richer data

Request Explorer has historically enabled users to retroactively view client access logs, and understand the request-response flow within HAProxy Enterprise. While these access logs provided useful request details, we've expanded this feature in HAProxy Fusion 1.3 by adding a dedicated dashboard page to help you review the following (and much more): 

  • Top 10 request paths

  • Response codes

  • GeoIP countries

  • Protocols used

Since logs feed data to Request Explorer, it's also possible to search by structured data using the familiar ctrl + f keybinding. We may continue to expand upon this overall feature list in future versions of HAProxy Fusion.

A new Security tab helps you drill down deeper

Request Explorer's expanded view is a great place to centrally inspect a wide range of HAProxy Enterprise cluster operational data, which is why we've expanded it to encompass security data. Teams use HAProxy Fusion to manage multi-layered security features such as HAProxy Enterprise WAF and HAProxy Enterprise Bot Management Module, so we've compiled data from both into a dedicated tab.

]]> ]]> HAProxy Fusion now displays a brief security summary and tells users what mitigating action was taken (if any) against a threat, based on your configuration. This section is bolstered by new sections on SSL, WAF, JavaScript challenge, and bot management—each demonstrating the intersection of client behavior and HAProxy Enterprise functionality.

Map files gain flexibility with fewer reloads

Many HAProxy Enterprise users use .map files to define key-value pairs, enabling features such as dynamic rate limiting and blue-green deployments. They also support Layer 7 routing and many other creative functions. However, the process of updating those files hasn't been very user-friendly. 

For example, .map files previously allowed for conflicting changes. HAProxy Fusion also required a complete file re-upload to apply miniscule edits. This was resource-intensive and error-prone for customers with massive files.

HAProxy Fusion 1.3 brings some important enhancements to boost .map file usability: 

  • Line-by-line and by-key editing are now supported through additional specialized API endpoints.

  • Changes to .map files submitted to HAProxy Fusion through the new editing API endpoints are applied to your HAProxy Enterprise nodes dynamically through Runtime API. These changes skip HAProxy reloads, yet your file is still uploaded to the filesystem for persistence. 

  • After users edit them, HAProxy Fusion can send multiple independent .map files to your load balancers in one transaction, with built-in error handling and reduced latency while applying multiple .map file changes.

]]> ]]> HAProxy Fusion is a central hub for teams, so we also wanted to make .map file editing more collaborative. You can now concurrently edit and merge changes, using several policies that govern mergers and conflict resolution. Additional controls are in place to enable smarter sending of updated files without overwhelming the system. Synchronicity between HAProxy Fusion users keeps everyone on the same page and displays the same file contents to all users, protecting against accidental deletion or duplication. 

If you want to track the file changes you've made, a new real-time changelog appears in the upper right (pictured above). HAProxy Fusion users can also use the API to make and apply .map file changes.

Conclusion

HAProxy Fusion 1.3 represents a mighty leap forward for HAProxy Fusion. New monitoring dashboards, high-performance service discovery, and .map file editing changes bring numerous usability improvements for HAProxy Fusion users. 

If you want to see HAProxy Fusion 1.3 in action, contact our sales team for a demonstration or watch our introductory webinar.

]]> Announcing HAProxy Fusion 1.3 appeared first on HAProxy Technologies.]]>
<![CDATA[Encoding HAProxy logs in machine-readable JSON or CBOR]]> https://www.haproxy.com/blog/encoding-haproxy-logs-in-machine-readable-json-or-cbor Thu, 24 Oct 2024 09:19:00 +0000 https://www.haproxy.com/blog/encoding-haproxy-logs-in-machine-readable-json-or-cbor ]]> Standardized logging formats are important for teams that rely on logging for observability, troubleshooting, and workflow integration. Using structured formats simplifies parsing and eliminates the need to interpret fields manually, ensuring consistency across logging formats. This reduces manual work, prevents brittleness from unstructured logs, and simplifies integration between teams that feed logs into a shared aggregation system.

Standardized logging formats have become essential for teams relying on integration to aggregate logs from heterogeneous technologies and tools within their organizations. HAProxy users can now address this need more effectively using standardized logging formats.

With the release of HAProxy 3.0, we introduced new JSON and CBOR encoding options. Users can now format log lines as JSON or CBOR, making them easier to parse. Furthermore, when formatting log lines as CBOR, users also have the option to specify binary (+bin) to reduce bandwidth.

Additionally, log format expressions were sometimes ambiguous about data type. Now, you can explicitly define data types for log format expression fields to improve consistency between logs.

These updates simplify log management and ensure better interoperability across systems. Furthermore, applications handling these structured logs benefit from enhanced performance, no longer needing to manually extract and format data from default logs.

Why did we add new encoding options to HAProxy?

HAProxy users have long wanted HAProxy logs to be in a standardized format to simplify the extraction of data, improve application performance, and ensure compatibility with existing logging facilities.

While default HAProxy logs are not standardized (often requiring custom parsers to extract relevant information), their design was intentional to accommodate earlier logging practices. In earlier versions of HAProxy, other structured formats such as JSON were not common yet, and HAProxy logs were primarily read directly by administrators with minimal need for parsing. As a result, HAProxy logs were designed to be less verbose and maintain backward compatibility.

However, logging needs have evolved. Users frequently manipulate the log format to make the output compliant with other systems. If users wanted HAProxy logs in a JSON or CBOR format, they had to convert them manually or use middleware to convert HAProxy logs to the desired format.

Furthermore, emitting structured logs with HAProxy was often difficult and problematic:

  • Quoting issues. The %{+Q} format often failed to quote string values consistently. For example, some string values (e.g., %tsc) were not quoted while others were. Also, numeric values leveraging sample expressions (e.g., %[expression]) were inconsistently quoted.

  • Hexadecimal representation. The %{+X} format, hexadecimal representation for integers, was not consistently applied across different log format aliases

  • Inconsistent null values. Null values were represented inconsistently, often as empty strings, sometimes as "-", and other times as -1. This inconsistency required treating everything as a string and manually adding quotes, rather than relying on %{+Q}, to ensure valid encoding (e.g., avoiding invalid encoding in JSON like {"foo":} or {"foo":-}).

  • String handling for numerical values. Some numerical values, such as %ms (padded output) or %rc / %B (leading characters), had to be handled as strings (for historical reasons).

HAProxy users had to spend additional effort creating custom parsers and handling log format issues manually. This resulted in increased complexity and inconsistencies in logging when trying to create JSON outputs, making it more difficult to achieve accurate and reliable log outputs.

The new JSON and CBOR encoding options address these issues by providing structured data logging options out-of-the-box.

  • JSON (JavaScript Object Notation) is a standard text-based format for storing and transporting data.

  • CBOR (Concise Binary Object Representation) is a binary data format often preferred for optimizing network bandwidth. Without specifying binary, CBOR is a hexadecimal format for interoperability purposes and can be more verbose than JSON and HAProxy’s default log format.

Together, these two encoding options give HAProxy users two widely used log formats, enabling better performance and integration with other systems and teams. The new log format encoders in HAProxy 3.0 provide a more standardized approach to logging, eliminating the need for custom parsers, middleware for conversions, and extensive log format adjustments. Users should find their logging setups are easier to maintain.

The advantages of HAProxy 3.0’s new encoding options

With HAProxy 3.0, users no longer require alternative solutions to convert HAProxy logs to JSON or CBOR. This native support simplifies the immense amount of logs generated by modern applications and microservices, making it easier for teams to operationalize and make sense of their data.

While alternatives like Syslog remain a standard, their lack of descriptiveness can make them less suitable for modern logging needs. JSON and CBOR offer more structure and interoperability. Even though these formats can be more verbose, the benefit of multi-team log consistency is a fair trade-off.

Encoding logs in JSON means you don’t need to manually craft a log yourself by “hacking” the log format. This makes logs encoded in JSON less error-prone and more consistent because manual adjustments are minimized. Furthermore, JSON-structured logs are easier to read and parse than default HAProxy logs, ultimately resulting in easier debugging and archiving experiences.

CBOR support in HAProxy 3.0 introduces a method for structuring logging that was previously unattainable through manual adjustments—it was never as simple as “hacking” the log format as it was for JSON. CBOR support improves interoperability with existing tools and products and reduces bandwidth usage by transmitting pure binary payload over the wire, provided the ring/syslog endpoint supports it.

]]> ]]> Practical examples of JSON and CBOR structured logs

JSON equivalent of "option httplog"

When enabling option httplog, HAProxy implicitly sets the proxy log-format directive to the default  HTTP access log formatted string, which can be accessed through the global environment variable named HAPROXY_HTTP_LOG_FMT.

Setting option httplog is equivalent to setting log-format to ${HAPROXY_HTTP_LOG_FMT} on an HTTP proxy:

]]> blog20241023-01.cfg]]> As of HAProxy 3.0, HAPROXY_HTTP_LOG_FMT is defined as the following variables, each variable representing a part of the request and response:

]]> blog20241023-02.cfg]]> As a reminder, enabling option httplog on a proxy will produce the following:

]]> blog20241023-03.cfg]]> The log payload, shown after the log header, comes with the limitations mentioned earlier regarding log consistency and parsing.

To generate JSON structured logs instead, we can create our own log format string based on HAPROXY_HTTP_LOG_FMT:

]]> blog20241023-04.cfg]]> If we configure HAProxy to use our own JSON log format:

]]> blog20241023-05.cfg]]> HAProxy will generate logs like this (log payload following the log header, everything after - -):

]]> blog20241023-06.cfg]]> You can verify that no information from option httplog is missing and that the result is fully JSON-compliant using a JSON validator.

CBOR (plain text) equivalent of “option httplog”

Let’s do the same thing for CBOR logs.

To generate CBOR (plain text) structured logs, we can create our own log-format string that enables the CBOR encoding option by setting %{+cbor}o:

]]> blog20241023-07.cfg]]> If we configure HAProxy to use our own CBOR log format:

]]> blog20241023-08.cfg]]> HAProxy will generate logs like this:

]]> blog20241023-09.cfg]]> You can verify CBOR compliance using the CBOR.me online tool.

]]> ]]> As demonstrated, leveraging HAProxy’s new JSON and CBOR encoding is significantly easier than encoding the log payload yourself.

Conclusion

The addition of JSON and CBOR encoding in HAProxy 3.0 streamlines log management and improves interoperability between systems. With these standardized formats, HAProxy reduces the complexity of log extraction and formatting, making it simpler for teams to maintain consistency across their infrastructure.

]]> Encoding HAProxy logs in machine-readable JSON or CBOR appeared first on HAProxy Technologies.]]>
<![CDATA[HAProxyConf 2025 Call for Papers Now Open]]> https://www.haproxy.com/blog/haproxyconf-2025-call-for-papers-now-open Tue, 17 Sep 2024 00:00:00 +0000 https://www.haproxy.com/blog/haproxyconf-2025-call-for-papers-now-open ]]> The wait is over! HAProxyConf is coming to San Francisco, California, from June 3 to 5, 2025.

The 2+ days event will take place in the Mission Bay Conference Center, where DevOps, networking, and security professionals, open source developers, community enthusiasts, and business leaders come together to discuss all things HAProxy! Presenters will share how they are addressing today's biggest application delivery and security challenges, how to get started or do more with HAProxy, and how they are using HAProxy to solve real-world problems while making a difference in their organizations. 

HAProxy is, first and foremost, a passionate community of open-source developers, who helped HAProxy reach version 3.0—a milestone release—earlier this year. We welcome everyone involved in the HAProxy project to join us and help encourage the next generation to get involved in building the future of HAProxy.

In addition to the main event, workshops will be held at the nearby Luma Hotel on June 3.

Got a great story, lesson, or innovation to share? From multi-layered security to infrastructure automation, and from API / AI gateways to Kubernetes, we want to hear from you! Submit your talk and become a part of HAProxyConf 2025's exciting lineup.

When is the conference?

HAProxyConf 2025 will happen from June 4 to 5, with pre-conference workshops on June 3.

Where will the conference be?

HAProxyConf 2025 will take place in San Francisco, California.

The main conference will be held at the Mission Bay Conference Center, and the workshops will be held at the nearby Luma Hotel.

Check the Call for Papers webpage to get the full details and submit your talk.

Looking for inspiration? Check out previous conference videos!

]]> HAProxyConf 2025 Call for Papers Now Open appeared first on HAProxy Technologies.]]>
<![CDATA[Announcing HAProxy Data Plane API 3.0]]> https://www.haproxy.com/blog/announcing-haproxy-data-plane-api-3-0 Tue, 10 Sep 2024 11:03:00 +0000 https://www.haproxy.com/blog/announcing-haproxy-data-plane-api-3-0 ]]> Watch our on-demand webinar to explore the exciting new features of our HAProxy Data Plane API 3.0 release!

HAProxy Data Plane API 3.0 is now available! The latest version is hosted on our GitHub releases page.

This release follows the recent HAProxy 3.0 release and incorporates its changes, along with some improvements and changes specific to the API.

HAProxy Data Plane 3.0 adds multiple breaking changes. We'll cover the impacts of these changes in detail to highlight how your implementation and usage of Data Plane API may be affected. Despite some initial growing pains, v3 provides a stronger foundation for future development as we continue to modernize and simplify the API to meet users' needs. We recommend reading carefully through this upcoming section. 

This release also makes key improvements to the API, which we'll also expand upon later: 

  • Users can now fetch a parent resource with all children embedded. 

  • We've added a PUT operation on index-based resources. 

  • Timeouts are now handled with your preferred time suffix. 

  • The API's state data is removed from the configuration file. User-defined settings are now the configuration's sole focus. 

  • Support has been added for every new keyword introduced in HAProxy 3.0. 

But that only scratches the surface. We're excited to show you the core functionality updates we've made in v3 and why they matter.

Breaking changes

This version introduces breaking changes described in this section.

Removed _version and data wrapper fields in responses

Each time the Data Plane API changes the load balancer configuration it increments the version of the file. This enables optimistic locking of the configuration to prevent a user from unintentionally overwriting another user's changes. In the initial design, each API endpoint returned JSON data with _version and data fields as top keys in the response object. Later, we also added a Configuration-Version HTTP header, making the configuration version available in both the wrapped response and the header. 

In v3, we've removed the redundant _version and data fields from the response and moved the JSON previously under data to the top—making it easier to parse. This simplifies the process of changing an existing resource. Users can leverage this flat structure to fetch the resource, change a few fields, and send back the same object without needing to unpack it from the data field.

Renamed the defaults resource

In the early stages of the Data Plane API, we intended to generate and manage only one defaults section within the HAProxy configuration by using the /v2/services/haproxy/configuration/defaults endpoint. This decision to manage just one was based on the fact that support for assigning defaults sections names—by which you could indicate which to inherit settings from—didn't originally exist. Assigning a name to a defaults section, although valid syntactically, had no functional meaning in HAProxy.

However, HAProxy 2.4 introduced keywords to the frontend and backend sections, giving meaning to these names and allowing users to create multiple, named defaults that frontends and backends could then inherit settings from. Since the defaults resource initially returned an object instead of a list, we introduced a new resource called named_defaults to maintain backward compatibility. This resource returned a list of defaults resources that names and could be referenced in backends and frontends sections. We kept the original defaults resource for backward compatibility as an object.

With v3 of the API, we're migrating named_defaults to simply defaults, breaking the old defaults behavior. The defaults resource now returns a list versus an object.

Moved child resources as nested resources

In v2, some resources represent lines within a section of HAProxy configuration, effectively making them child resources of the parent section resource. For example, the acl resource can be a child of a frontend or backend, and a server resource can be a child of a backend parent. 

You could previously fetch or update these resources in the API by identifying them with their index, which was the case for the acl resource. You could do the same by name for other resources, such as the servers resource. You'd then specify their parents using the parent_type and parent_name query-string parameters.

This process was unfortunately confusing, so we've redesigned child resources as nested resources in the URL. Because a child resource cannot simultaneously exist in multiple parent resources, it makes sense to include this in the parent resource's URL. This also boosts compatibility with many external tools. For example, you can build role-based access control (RBAC) rules more easily atop the new URL hierarchy without relying on query-string parameters.

Here's how the two implementations compare:

  • In v2, you'd see /services/haproxy/configuration/acls?parent_type=frontend&parent_name=my-example-frontend

  • In v3, you'll instead see /services/haproxy/configuration/frontends/my-example-frontend/acls

Removed explicit index field on index-based child resources

In v2 of the API, all index-based resources—which represent configuration lines that can be repeated within a section and are distinguished by the order of appearance—had an index field indicating the position of that specific resource. For example, acl resources get an index to indicate their order within the configuration.

We've removed that index field in v3, as we found it redundant and complicated to maintain when positions change. You can infer the ordering from the actual position in the returned list.

Removed support for HAProxy in multi-process mode

HAProxy previously had a feature that allowed it to run in multi-process mode. This mode has since been deprecated and removed following the introduction of multi-thread mode. In v2 of the API, we supported all keywords related to multi-process mode. Additionally, we supported multiple runtime APIs (one for each process mode), so all /runtime resources worked with multiple process support.

We've removed this support altogether in v3 alongside the following fields from the respective sections:

  • Backend resource: bind_process

  • Bind resource: process

  • Defaults resource: bind_process

  • Frontend resource: bind_process

  • Global resource: nbproc

In addition to removing these fields, /runtime endpoints have changed.

The stats resource at /v2/services/haproxy/stats/native used to return an array of arrays. The top-level array held per-process stats, with each array element containing an array of stat lines for each object (server, backend, frontend). Now that multi-process mode support is ending, /v3/services/haproxy/stats/native returns an object with a single stats array for the server, backend, or frontend object you've requested. Similarly, we removed one layer of arrays that returned process information on a per-process basis. This change affected /v2/services/haproxy/runtime/stick_table such that the process field is removed from responses that previously indicated from which process the stick table information was being fetched.

In addition to this, the query string parameter process was removed from both /v2/services/haproxy/runtime/stick_tables and /v2/services/haproxy/runtime/stick_table_entries resources.

Reorganized global resource fields

One thing we set out to do with v3 was reorganize the global resource fields. The global resource is our largest if you measure by the number of fields, and it primarily had a flat structure. To improve readability of this resource on the API, we reorganized the fields into nested objects.

While making this change, we tried to adhere to HAProxy's field naming conventions and followed documentation hints when deciding how best to structure the revised fields. We tried to maintain only one level of nested objects to avoid unnecessary complexity. Some fields remain in the root level of the object, and some options are split into debug_options and performance_options as specified in the Configuration Manual.

We've also added an ssl_options nested object to cover all SSL directives. Other objects like tune_ssl_options, tune_vars_options, and tune_zlib_options are all derived from tune.*.* options. Plus, the tune_options object covers other tuning global directives and acts as a catch-all. Meanwhile, http_client_options and fifty_one_degrees_options are similarly derived from the manual's dot notation.

For the full and detailed specification of fields, please see the global model specification within our GitHub project.

Removed deprecated fields and resources

As a major version release, Data Plane API 3.0 gave us the chance to clean up keywords previously marked as deprecated. Here's more information about the keywords we've removed in v3:

Global section

  • Removed the deprecated tune_ssl_default_dh_param field that can now be found in tune_ssl_options as default_dh_param

Backend section

  • Removed the deprecated http-check field. You can now specify multiple http_check resources on /v3/services/haproxy/configuration/backends/{parent_name}/http_checks.

  • Removed the http-keep-alive, http-server-close, and httpclose fields as they're mutually exclusive and configurable using the http_connection_mode field

HTTP request/response rules and TCP Request Rule resource

  • Removed the deprecated track_sc<stick_counter> field. In v2 we only supported 0, 1, and 2 stick counter indexes by hard-coding them in the field names. We've now added the track_sc_stick_counter field (specifying the stick counter index) which can exceed 2 if configured using the tune.stick_counters keyword in the global section. It defaults to 3 and debuted in HAProxy 2.8.

Service discovery resources

  • Removed the deprecated service-whitelist and service-blacklist fields, which we've replaced with service_allowlist and service_denylist, respectively

Further API Improvements

]]> ]]> We'll describe other notable improvements in this section.

Fetching a parent resource with all children embedded

While we've found that our granular approach to creating resources on the API is helpful, we're aware it can also add complexity. As a result, we've added an extra query string parameter full_section on all the section resources such as frontend and backend resources. This lets you fetch, create, or replace a complete section with all of its child resources.

This query string parameter works on both PUT and POST endpoints, so you can now edit or create the whole parent section with just one API call.

Added a PUT operation to index based resources

Previously, change automation on a list of index-based resources was complex for several reasons. First, you had to reconcile all index fields in each resource. This made it difficult to locate a resource if something was deleted and added, as its index could have changed. Second, large lists with multiple changes forced you to make numerous API calls.

With v3, we added a PUT operation to the list endpoints, letting you automatically replace the entire list of resources.

Handle timeout options with preferred units

We handle timeout options as integers in the API, since we want to represent numeric values as numbers wherever possible. However, since HAProxy supports strings by accepting time unit suffixes, we recalculated the submitted values to the default unit in v2 (usually milliseconds) and stored them as integers in the configuration file. This was challenging for those who still use and read the raw configuration file, as it changed their preferred inputs and made the file less readable.

In v3, we've introduced a new option in the Data Plane API configuration file called preffered_time_suffix where you can specify one of the options: nearest, none, ms, s, m, h, and d. Data Plane API will then honor your preferred time suffix, which is nearest by default, and calculate it to the smallest possible value with the corresponding suffix. When using none, your timeout keywords in the configuration file are written without the suffix while using the default value for the respective keyword.

For further info, we've added some documentation to our Open API specification by annotating such fields with x-duration: true and x-default-unit to indicate the standard time unit for that specific field.

Keep Data Plane API config file unchanged

Since Data Plane API lacks its own storage, it previously used its configuration file to store state—notably for service discovery and user configurations. This proved problematic when using Data Plane API with automation software like Ansible, Puppet, and others.

Data Plane API will no longer write to its own configuration file from v3 onward. We've introduced dataplane_storage_dir, where Data Plane API's state will be stored in specific JSON files for each feature. We chose JSON for easier readability and debugging.

Upon startup, state data will automatically migrate from the configuration file into JSON files located at /etc/haproxy/dataplane. For information on where the data goes, view our brief Data Plane API internal storage Readme on GitHub.

Added support for all new keywords added in HAProxy 3.0

Following the latest release of HAProxy, we're extending the configuration keyword support to include many new features in the load balancer. These keywords impact security, persistence, streaming, directives, and more. 

To view and understand the complete collection of keywords now available in HAProxy 3.0, check out our Reviewing Every New Feature in HAProxy 3.0 blog post.

Contributions

As always, we'd like to extend a massive thank you to everyone who helped accelerate Data Plane API's development and make v3 possible: 

Contributor

Area

Zlatko Bratkovic

BUG | BUILD | CLEANUP | FEATURE

Olivier Duclos

BUG | FEATURE | REORG | TEST

Helene Durand

BUG | BUILD | FEATURE | REORG

Vincent Gramer

BUG | BUILD

Libo Huang

BUG

Andjelko Iharos

FEATURE

Marko Juraga

BUG | BUILD | CLEANUP | FEATURE | TEST

Dinko Korunic

FEATURE | OPTIM

Aurélien Maigret

BUG

Robert Maticevic

BUG | BUILD | CLEANUP | FEATURE | OPTIM | TEST

Fabiano Parente

BUG

Dario Tranchitella

BUG | REORG

Csaba Tűz

FEATURE

George Vine

BUG | FEATURE

Data Plane API 3.0 constitutes a major step forward in functionality and maintainability for Data Plane API. We're excited to bring you even more features and enhancements soon in response to your feedback. 

For more information including installation and upgrade instructions, visit our Data Plane API documentation. You can also view our Data Plane API releases section on GitHub for updated release notes and a changelog.

]]> Announcing HAProxy Data Plane API 3.0 appeared first on HAProxy Technologies.]]>