In his HAProxyConf 2019 presentation, William Lallemand (Senior HAProxy Developer) shows how process management in HAProxy has evolved since the beginning of the project; With the advent of systemd, new techniques had to be developed so that users could reload HAProxy safely. The Master-Worker mode simplifies the management of HAProxy processes and introduces interesting features. This talk briefly reviews the history of process management in HAProxy and the CLI commands introduced through the evolution of the architecture.
Watch William’s presentation video or read the transcript below.
Explore more HAProxyConf 2019 talks in our User Spotlight Series.
Transcript
Thank you, Daniel! So, this talk will be a quick introduction to the process management in HAProxy or more precisely the Master/Worker mode. I won’t go into details, so if you want the details you can talk to me after the talk.
Little bit of history first. I won’t talk about everything, but the important thing here is the systemd wrapper, which was the ancestor of the Master/Worker mode. Basically it was an external binary, which was launching HAProxy and handling signals. It was made to be compatible which systemd because the daemon mode of HAProxy wasn’t compatible with it. In version 1.8, we wanted to reintegrate this feature within HAProxy, so we have only one binary to do it. And after that we enhanced it with the Master CLI and the program section.
To understand why we needed the Master/Worker Mode, I have to introduce the daemon mode and what are the flaws in it.
The daemon mode is the original mode used before systemd, so when you add a distribution with system file, we used this mode. It’s a simple design, basically it only launches as a process. It can be complex to monitor, because you have to monitor every single process. You can have many, and when you reload you can have many more. There is no Master process, so you cannot monitor only one and this is a problem with systemd.
This is the architecture of the daemon mode. For example, you have two processes in your configuration, so when you launch it, there are two processes.
And if you want to reload it, you have to launch a new process. This is interesting because with this method you can update the binary because you will launch a new process. The new process will load the configuration and it will send the soft stop signal, which is the -USR1 signal. The argument is -sf followed by the PID of the previous processes.
And then the answer it sends, it sends a signal. The new process will fork to a new process and you will probably still have the older processes because they still have jobs to finish.
After they finish their jobs, they will disappear. This is the problem with systemd, because systemd will consider that HAProxy left because of some processes left. Because with systemd, we need a main PID, and to have a main PID, you need main processes, a master process, and this is not the case in this mode.
.png)
So, we went on to work with systemd and we introduced the Master/Worker. It was inspired by the systemd wrapper, but simpler because it’s not an external binary anymore. You just have to launch HAProxy with an argument. It’s only one process to monitor, so you have your main PID for systemd. And there is a feature which is a Master CLI, which is basically a stats socket which is connected to the master process.
It’s the same configuration with nbproc 2. In this architecture you can see that there is a Master and two Workers.The Workers are still independent, they still don’t communicate with each other, but the Master is able to communicate with the Worker.
If you want to reload, you have to send the -USR2 signal, this is what is done in the systemd unit file.
Once you send this signal, the Master will reload itself.
We still need the ability to update the binary, so it will re-exec itself. It will load the new binary and then load the new configuration.
And after that all is the same as the daemon mode: it will send the soft stop signal, it will fork and then the previous process will disappear once they finish their job.
To launch in Master/Worker mode you simply put the -W argument on your command line. On the first line, you can see that it will be launched on the foreground, on the second line combined with -D argument it will be sent in the background. The last line is a -Ws argument which is a Master mode with systemd compatibility. It’s the ability to connect with sd_notify API to say to systemd: “Okay, I finished to load the configuration”, so you can have your systemctl status. To use it you need to modify the system unit file and put the Type=notify in the unit file.
Okay, so here is some optional configuration. For example, if you don’t want to use the -W argument you can use the master-worker keyword in the global section.
There is a no-exit-on-failure keyword, which is the opposite of the exit-on-failure ability of the Master/Worker. In fact, if there is a problem with one of the Workers, for example, if it quits for whatever reason, the Master will kill every other Worker. So, it guarantees you to have a complete HAProxy instance which is working. If you want to debug on either one of the Workers, which can be still there, if another one segfaults or if there is an Out of Memory, you can use this keyword.
The second keyword is the mworker-max-reloads. It will say to the Master to kill a Worker after a certain amount of reloads. For example, if you try to reload a Worker three times and if it’s still there because there is some stuff to do, it would be killed under force-reload. And you can achieve the same thing, but this time with hard-stop-after. So, after a certain amount of time it will be killed.
There is a new feature which is called the Master CLI. To understand why we made this, we need to understand what the flaws of the previous mode were. So, basically when you add two processes, you need it to have one CLI on each process, which is not really convenient.
The problem is that when you reload the processes, the CLI is bound on the new process and you won’t be able to access the previous processes.
The Master CLI is connected directly to all processes, so you can connect to any process: the new one and the old one.
The Master CLI is configurable with an HAProxy argument, so you don’t need to modify your configuration file.
You just have to put a -S with a bind line, which can be a path or an IP or you can use any keywords that are useful. They are the same keywords as the bind line in the HAProxy configuration. For example, you can see, there’s a uid, a gid, you can even use SSL, if you want. You can also use several Master CLIs. For example, you can use one CLI with a level,operator, one CLI with a level,user, which would be basically read-only.
There are a few new commands. The important ones are: the @ symbol and the show proc command and reload which does the same as kill -USR2.
The show proc command will show you every process. So you will have the <PID>, the <type> of process, the <relative PID> and the number of <reloads>. Also, there’s <uptime>. Something which can be important if you will try to update the binary – the <version>. For example, you can see that on the last line the version is not the same, because we reloaded it with a new version.
To enter in a process with the CLI, you need to use the @ prefix. For example, you can enter @1, which will enter in the process whose relative PID is 1. There you can type anything, like if you were on a normal stats socket. To kill it, you just have to type @ without anything. And if you want to enter in an old process, you can’t use the relative PID, so you have to use the real PID.
The last feature which was made in the Master/Worker are Programs. It’s the ability to launch an external binary at HAProxy startup. It’s simple. This is not systemd, so it will just launch binary and send a signal on the reload. But don’t expect dependencies and the complicated stuff. If you need that, use systemd.
The configuration is simple. This is an example where I will launch an SPOA agent. You can use any argument on the program. And the option start-on-reload will permit you to relaunch the binary upon an HAProxy reload. The different behavior is to send a -USR1 signal and probably your program will kill the -USR1 signal, so you want to probably restart it. This option will achieve this.