Developing a strategy for collecting application-level logs necessitates stepping back and looking at the big picture. Engineers developing the applications may only see logging at its ground level: the code that writes the event to the log—for example a function that captures Warning: An interesting event has occurred! But where does that message go from there? What path does it travel to get to its destination?
In a sufficiently large architecture where you have many services and applications, possibly spanning multiple data centers and maybe even touching both on-prem and cloud-hosted environments, you may want to designate log collection points at different tiers. These collection points would ingest data from all the applications in a network, for example, or from all within a data center, before forwarding it to a higher tier such as a log aggregation server that is responsible for reporting on all data globally. That way, only the mid-tier collection points would need to have knowledge of where the log aggregation server lives, while the applications would need to know only about the mid-tier collection points, simplifying administration if the network ever changes.
In this blog post, I will describe HAProxy’s log forwarding support for the Syslog protocol, which is a standardized logging protocol baked into many devices, software, and programming languages. HAProxy can act as a log forwarder, standing in as a collection point that receives logs and relays them to a higher tier. Its ability to act as a remote Syslog server and client, along with its support for both UDP and TCP, make it a convenient choice for teams that host a variety of applications and environments.
Log Forwarding Over UDP
In version 2.3, HAProxy introduced a feature for receiving Syslog messages and forwarding them to another server. HAProxy can listen on either a TCP or UDP port, or both, and then send the message out via TCP or UDP. The UDP configuration is the simplest, so let’s start with that.
In the HAProxy configuration snippet below, a section named log-forward listens for incoming messages and forwards them to a server at 172.25.0.12.
| log-forward syslog | |
| # Accepts incoming TCP messages | |
| bind 0.0.0.0:514 | |
| # Accepts incoming UDP messages | |
| dgram-bind 0.0.0.0:514 | |
| # Sends outgoing messages via UDP | |
| log 172.25.0.12:514 local0 |
The bind line tells HAProxy to listen for incoming messages over TCP, and the dgram-bind line tells it to listen for incoming messages over UDP. By specifying 0.0.0.0 as the address, it listens on all available IP addresses assigned to the load balancer. We’re listening and sending using port 514, which is the standard Syslog port for both UDP and TCP. The facility code, local0, is used only if the received log event doesn’t already set a facility.
HAProxy supports two formats of the Syslog protocol, the older RFC3164 and the newer RFC5424, and it will relay whichever format it receives. However, you can choose which one to relay by setting format on the log line.
In this example, messages are sent out using the UDP protocol. The benefit of UDP is that it’s a fire-and-forget protocol, meaning that HAProxy doesn’t need to wait for a response from the target server, which avoids any slowdowns in performance. The disadvantage is that there is no guarantee that all log messages will arrive at their destination, such as if the destination server were to go offline. HAProxy isn’t able to health-check the server to know if it’s down. In many cases, losing non-critical logs here and there isn’t too important, but if you require 100% message delivery, consider sending outgoing messages over TCP instead.
Log Forwarding Over TCP
TCP guarantees that messages will reach their destination, but it expects HAProxy to wait for an acknowledgment from the server. To avoid slowdowns, HAProxy performs this while waiting in the background, storing queued up Syslog messages in a buffer until they’re sent and acknowledged.
In the configuration snippet below, we utilize a ring section to define a circular buffer in memory for storing log messages until they’re sent. The log line in the log-forward section then refers to this ring section rather than an IP address as in the UDP example.
| log-forward syslog | |
| bind 0.0.0.0:514 | |
| dgram-bind 0.0.0.0:514 | |
| log ring@logbuffer local0 | |
| ring logbuffer | |
| description "buffer for logs" | |
| format rfc5424 | |
| maxlen 1200 | |
| size 32764 | |
| timeout connect 5s | |
| timeout server 10s | |
| # Sends outgoing messages via TCP | |
| server logserver 172.25.0.12:514 check |
Together, the ring and log-forward sections enable relaying Syslog messages over TCP without creating a bottleneck. You can play with the maxlen and size parameters to change the maximum size of a Syslog message and the overall size of the buffer. Timeouts ensure that HAProxy won’t wait forever on an unresponsive server.
Read more:
Conclusion
In this blog post, you learned how HAProxy can act as a log collection point that ingests logs from multiple applications and then forwards them to a centralized log aggregation server. HAProxy’s ability to listen for both UDP and TCP Syslog messages helps it to integrate with a variety of software. By creating a tier of log collection points, you can simplify the administration of your logging infrastructure.
Interested to know when we publish content like this? Subscribe to our blog! You can also follow us on Twitter and join the conversation on Slack.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
