Configure Syslog-ng to Log Readable HTTP URL From HAProxy

These tips are provided by Exosec. Exosec provides a very good monitoring product called POM, based on Nagios with very strong value added such as very simple administration, application monitoring, etc… For some of their project, they use either HAProxy or the ALOHA Load-Balancer (heh, what else???) and they export log entries into syslog-ng for storage and later analysis.

HAProxy’s Log

HAProxy’s logs are very powerful since they provide many information about the request and the status of the platform at the moment of the request. For a readable HAProxy’s log description, please consult the ALOHA memo dedicated to HAProxy’s HTTP log line description.

One of the weaknesses of the log line is that it logs only the path and the query string of each URL. No server name or protocol information. Well, HAProxy allows us to log the Host header, which is fine and there is a tild ‘~’ after the frontend name when the connection is made over SSL.

Using syslog-ng flexible configuration, we can re-order things to make HAProxy log the URL exactly like it was sent by the client, like: “http://www.domain.tld/url/path”

Configuration

HAProxy configuration

Note

This is a very minimalistic configuration, not recommended in production 🙂.

global
 log 127.0.0.1:514 local2

frontend ft_http
 bind 127.0.0.1:8080
 bind 127.0.0.1:8081 ssl crt /etc/haproxy/haproxy.pem
 option http-server-close
 mode http
 log global
 option httplog
 # Mandatory to build the URL:
 capture request header Host       len 32
 # Optional, just for statistics:
 capture request header User-Agent len 200
 default_backend bk_http

backend bk_http
 option http-server-close
 mode http
 log global
 option httplog

 server srv1 10.0.0.1:80

Syslog-ng configuration

The configuration below will reproduce an HAProxy log line but will replace the URL part with something more readable.

Note

If you capture a different number of HTTP headers in HAProxy (the current example contains 2 captured headers), you may have to update the parser p_haproxy_headersreq and the destination d_haproxy_full.

source s_loopback { syslog(ip(127.0.0.1) port(514) transport("udp")); };
destination d_haproxy_full {
     file("/var/log/haproxy.$YEAR-$MONTH-$DAY.log"
          template("$DATE $FULLHOST $PROGRAM: ${HAPROXY.CLIENT_IPPORT} [${HAPROXY.DATE}] ${HAPROXY.FRONTEND} ${HAPROXY.BACKEND} ${HAPROXY.TIME} ${HAPROXY.STATUS_CODE} ${HAPROXY.BYTES_READ} ${HAPROXY.COOKIE_REQ} ${HAPROXY.COOKIE_RESP} ${HAPROXY.TERM_STATE} ${HAPROXY.RUN_STATE} ${HAPROXY.QUEUE_STATE} {${HAPROXY.HEADERS_REQ}} "${HAPROXY.METHOD} ${HAPROXY.FRONTEND_PROTOCOL}://${HAPROXY.HOST}${HAPROXY.URL} ${HAPROXY.HTTP_VERSION}"n")
          group(adm) perm(0640) dir_perm(0750) template_escape(no)
         );
};
filter f_haproxy { program("haproxy"); };
filter f_frontend_ssl { match("~ "); };
rewrite r_set_frontend_protocol {
  set("http", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_haproxy)));
  set("https", value("HAPROXY.FRONTEND_PROTOCOL") condition(filter(f_frontend_ssl)));
};
parser p_haproxy {
  csv-parser(columns("HAPROXY.CLIENT_IPPORT", "HAPROXY.DATE",
                     "HAPROXY.FRONTEND", "HAPROXY.BACKEND",
                     "HAPROXY.TIME", "HAPROXY.STATUS_CODE",
                     "HAPROXY.BYTES_READ", "HAPROXY.COOKIE_REQ",
                     "HAPROXY.COOKIE_RESP", "HAPROXY.TERM_STATE",
                     "HAPROXY.RUN_STATE", "HAPROXY.QUEUE_STATE",
                     "HAPROXY.HEADERS_REQ", "HAPROXY.REQUEST")
             flags(escape-double-char,strip-whitespace)
             delimiters(" ")
             quote-pairs('""[]{}'));
};
parser p_haproxy_request {
  csv-parser(columns("HAPROXY.METHOD", "HAPROXY.URL",
                     "HAPROXY.HTTP_VERSION")
             delimiters(" ")
             flags(escape-none)
             template("${HAPROXY.REQUEST}"));
};
parser p_haproxy_headers_req {
  csv-parser(columns("HAPROXY.HOST", "HAPROXY.USER_AGENT")
             delimiters("|")
             flags(escape-none)
             template("${HAPROXY.HEADERS_REQ}"));
};
log {
  source(s_loopback);
  filter(f_haproxy);
  parser(p_haproxy);
  parser(p_haproxy_request);
  parser(p_haproxy_headers_req);
  rewrite(r_set_frontend_protocol);
  destination(d_haproxy_full);
};

This configuration is downloadable from HAProxy Technologies github: https://raw.github.com/exceliance/haproxy/master/logs/syslog-ng_full_http_url.conf

And the result would look like the below at the end of the logged line:

[...] "GET http://test.domain.tld/blah HTTP/1.1"
[...] "GET https://test.domain.tld/blah HTTP/1.1"
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.