How SNCF Connect Built Its Own CDN with HAProxy

The Challenge

More than a decade ago, when originally operating under its old moniker of Oui.sncf, the site was utilizing a three-tier system with an Apache web server, a Weblogic application server, and Oracle databases. Faced with a lack of observability over this infrastructure, the team utilized the community version of HAProxy to help them monitor error statistics, before switching to the supported HAProxy Enterprise software in front of their web applications several years later.

Then, in 2014, faced with the rising costs of their off-the-shelf Content Delivery Network provider Akamai, engineers Antonin Mellier and Nicolas Besin began to reconsider their options. Their growing bandwidth needs were making the pricing prohibitive, and they realized that many of the optional features they were paying for, like real-time log analysis and SSL certificates, could be implemented for free into a custom CDN solution.

The team thus decided to go about constructing their own Content Delivery Network, with HAProxy Enterprise at its core.

The Objectives

The chief objective of Nicolas and Antonin’s switch to a custom CDN was to keep the latency benefits of a system of cache servers in their network, while eliminating third parties from its structure. By responding to end user requests in the place of origin, Content Delivery Networks minimize delay in loading web page content, while also providing protections against high traffic peaks and DDoS attacks. Given the reduced bandwidth costs, it is also significantly cheaper than traditional hosting.

The team needed a non-traditional DNS product for their CDN as well. So, after finalizing the choice for their bare metal CDN hosts, Nicolas and Antonin went with GeoDNS, offering them the ability to assign IPs based on the location of clients in addition to weighted shuffle for different loads on each of the edge servers. They also added a separate, managed DNS provider in Dyn in order to increase their global presence. This multiple-DNS solution provided them with a failover in case of emergency and the ability to distribute traffic between the two solutions.

The Solution

With the infrastructure now in place, SNCF Connect implemented their HAProxy Enterprise load balancers at the edge of their CDN, with Varnish as their caching solution. Having the HAProxy Enterprise instance at the edge of the CDN gave Nicolas and Antonin a standardized view of their inputs and outputs as data was routed through, making it easier to determine error and cache ratios, as well as to locate the source of potential problems from either inside or outside their CDN network.

In order to manage their configuration files, the team developed their own application to act as a Configuration Management Database. For each site or application that uses the network, the CMDB has stored parameters for features like caching rules, IP addresses and ports to be used, as well as HTTP/2 or IPv6 activation. These files were to be generated in ruby and subsequently deployed by Ansible.

The backend of SNCF Connect’s CDN was then routed through their HAProxy Enterprise instances to one of their two datacenters, each treated by the load balancers as a destination server. For each application, a predetermined percentage of traffic is routed to one of the two destinations, with cookies used for session persistence when the application is not stateless. Access Control Lists are also used in certain instances, especially when retrieving media from a dedicated server, or the parts of the website which are hosted in the public cloud on AWS.

The Results

The team at SNCF Connect also takes advantage of the unrivaled observability of their HAProxy Enterprise load balancers. By using Prometheus to collect metrics across their different components, they are in turn able to display and analyze the data in their Promviz and Grafana visualizers. Another HAProxy innovation that they utilize are stick tables, enabling them to determine the presence of abnormal IP behavior across their network.

The end result of Nicolas and Antonin’s move to a self-operated CDN meant a two-thirds drop in their operating costs compared to their previous paid model. This even after an 80% bandwidth increase in the years since its implementation. The content cache also means only 10% of requests need to reach their origin servers, meaning a better website experience for their customers and less strain on the internal network.

What HAProxy Enterprise Offers You

Whether you need superior observability, protection against web-based threats, or simply a load balancer that can integrate with a custom platform, HAProxy Enterprise has a lot to offer. While the team at SNCF Connect built their own CDN, you can deliver your own applications at hyperscale with HAProxy Edge, the CDN built upon HAProxy technology.

Interested to learn more about HAProxy use cases? Explore our Success Stories page.