We've previously explained why some established connections could be dropped during reloads under high utilization, and how the problem has been solved in HAProxy Enterprise 1.8r1 and HAProxy 1.8. (“High utilization” typically means handling thousands of new connections per second and simultaneously reloading HAProxy processes multiple times per second.)
Since then, we've also noticed a great deal of user interest in actually enabling this feature in their configurations, so we have decided to post a quick and efficient HOWTO.
HAProxy 1.8 implements hitless reloads by transferring listening sockets from the old HAProxy instance to the new, reloaded one. This not only allows for the configuration to be updated but also for HAProxy to be upgraded without any dropped connections or downtime. In order for this to work, the old and the new processes need to be configured with the Runtime API option that will enable handing off the connections.
Making it work is very simple – the HAProxy configuration file needs to be updated, and in some cases, a command line option needs to be added.
HAProxy has released a patchset to enable seamless reloads of HAProxy without dropping packets in the process
Configuration File Update
To update the configuration file, simply add the option expose-fd listeners to an existing Runtime API socket definition, or add the whole line if you currently do not have it enabled:
| stats socket /var/run/haproxy.sock mode 600 expose-fd listeners level user |
Command Line Options Update
Please note that this step does not need to be performed if your HAProxy configuration already contains the directive master-worker, or if it is started with the option -W. That is the default with newer HAProxy versions running on GNU/Linux distributions which use systemd. In those cases, the option -x is added to new worker processes automatically if there is at least one socket with option expose-fd listeners set on it, and so no changes are necessary. If, however, your HAProxy command line options do need to be updated, simply add the option -x [socket_file] to the haproxy arguments, pointing it to the location of the Runtime API socket. For example:
| haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf $(cat /run/haproxy.pid) -x /var/run/haproxy.sock |
And that is all there is to enabling hitless reloads with HAProxy!
Availability
Hitless Reloads are available in HAProxy Enterprise 1.7r1 and above as well as HAProxy 1.8 and above.
For HAProxy 1.7 users who would like to take advantage of this feature without a major upgrade in your production environment, consider HAProxy Enterprise 1.7r1 which provides cutting edge, stable features backported from HAProxy 1.8.
Conclusion
We hope you have enjoyed this summarized blog post providing the HOWTO for enabling truly hitless reloads with HAProxy.
If you would like to run HAProxy in your infrastructure backed by enterprise support from HAProxy Technologies, please see our HAProxy Enterprise – Trial Version or contact us for expert advice.
Happy hitless reloading and stay tuned!
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
