High availability
Active/Standby clustering
In an active-standby cluster, one HAProxy Enterprise node receives traffic while a second node is put on standby. In case the active node fails, the standby takes over. With HAProxy Enterprise, you configure an active-standby cluster by installing the VRRP module.
The HAProxy Enterprise VRRP module assigns a virtual, static IP address to your load balancer node. If the node fails, the IP address will instantly transfer to a backup node, avoiding any break in service. Your router can continue to send traffic to the same IP address for the load balancer, never knowing that there was a failover. The module utilizes a stable version of Keepalived, which implements the Virtual Router Redundancy Protocol (VRRP).
In this guide, we will set up two load balancers: one active and the other on standby.
Installation Jump to heading
Follow the steps on your active and backup load balancers.
Active load balancer Jump to heading
On the active load balancer, follow these steps:
-
Install the VRRP module using your system’s package manager:
nixsudo apt-get install hapee-extras-vrrpnixsudo apt-get install hapee-extras-vrrpnixsudo yum install hapee-extras-vrrpnixsudo yum install hapee-extras-vrrpnixsudo zypper install hapee-extras-vrrpnixsudo zypper install hapee-extras-vrrpnixsudo pkg install hapee-extras-vrrpnixsudo pkg install hapee-extras-vrrp -
Edit the file
/etc/hapee-extras/hapee-vrrp.cfg
.-
In the
vrrp_instance
section, change theinterface
value frometh0
to the name of a network interface that receives traffic on the server. This is the same interface that you the bind HAProxy Enterprise’sfrontend
to. For example, if traffic reaches the load balancer using interfaceenp0s8
. Set this line tointerface enp0s8
. -
Use the same interface name in the
track_interface
block. -
Change the IP address listed in the
virtual_ipaddress_excluded
block to the virtual IP address you’d like to assign to this interface. HAProxy Enterprise will bind to this address to receive traffic. If needed, add more IP addresses here, each on its own line. The new address(es) should fall within the interface’s IP subnet, but should not be assigned to any server already.
hapee-vrrp.cfgtextvrrp_instance vrrp_1 {interface enp0s8 # Change network interface namestate MASTERvirtual_router_id 51priority 101virtual_ipaddress_excluded {192.168.50.10 # New IP address}track_interface {enp0s8 weight -2 # Change network interface name}track_script {chk_sshdchk_lb}}hapee-vrrp.cfgtextvrrp_instance vrrp_1 {interface enp0s8 # Change network interface namestate MASTERvirtual_router_id 51priority 101virtual_ipaddress_excluded {192.168.50.10 # New IP address}track_interface {enp0s8 weight -2 # Change network interface name}track_script {chk_sshdchk_lb}} -
-
Enable and start the
hapee-extras-vrrp
service:nixsudo systemctl enable hapee-extras-vrrpsudo systemctl start hapee-extras-vrrpnixsudo systemctl enable hapee-extras-vrrpsudo systemctl start hapee-extras-vrrp -
Allow the server to bind to the virtual IP address.
Edit the file
/etc/sysctl.conf
and add thenet.ipv4_ip_nonlocal_bind=1
directive, which allows the server to accept connections for IP addresses that are not bound to any of its interfaces, enabling the use of a floating, virtual IP.sysctl.confininet.ipv4.ip_nonlocal_bind=1sysctl.confininet.ipv4.ip_nonlocal_bind=1 -
Configure your
bind
line in the HAProxy Enterprise configuration to use the virtual IP address.haproxyfrontend myfrontendmode httpbind 192.168.50.10:80default_backend web_servershaproxyfrontend myfrontendmode httpbind 192.168.50.10:80default_backend web_servers -
Reboot the server.
Backup load balancer Jump to heading
On the backup load balancer, follow these steps:
-
Install the VRRP module using your system’s package manager:
nixsudo apt-get install hapee-extras-vrrpnixsudo apt-get install hapee-extras-vrrpnixsudo yum install hapee-extras-vrrpnixsudo yum install hapee-extras-vrrpnixsudo zypper install hapee-extras-vrrpnixsudo zypper install hapee-extras-vrrpnixsudo pkg install hapee-extras-vrrpnixsudo pkg install hapee-extras-vrrp -
Edit the VRRP configuration
/etc/hapee-extras/hapee-vrrp.cfg
. It must be almost exactly the same as the file on the active load balancer, except:-
change the
priority
field in thevrrp_instance
block to have a lower number than the active node; The load balancer with the highest priority is promoted to be the active node. For example, if the priority on the active node is 101, then set the backup node’s priority to 100. -
change
state
toBACKUP
.
hapee-vrrp.cfgtextvrrp_instance vrrp_1 {interface enp0s8state BACKUPvirtual_router_id 51priority 100 # A lower priority value than the primaryvirtual_ipaddress_excluded {192.168.50.10 # Same IP address as on primary}track_interface {enp0s8 weight -2}track_script {chk_sshdchk_lb}}hapee-vrrp.cfgtextvrrp_instance vrrp_1 {interface enp0s8state BACKUPvirtual_router_id 51priority 100 # A lower priority value than the primaryvirtual_ipaddress_excluded {192.168.50.10 # Same IP address as on primary}track_interface {enp0s8 weight -2}track_script {chk_sshdchk_lb}} -
-
Enable and start the
hapee-extras-vrrp
service:nixsudo systemctl enable hapee-extras-vrrpsudo systemctl start hapee-extras-vrrpnixsudo systemctl enable hapee-extras-vrrpsudo systemctl start hapee-extras-vrrp -
Allow the server to bind to the virtual IP address.
Edit the file
/etc/sysctl.conf
and add thenet.ipv4_ip_nonlocal_bind=1
directive, which allows the server to accept connections for IP addresses that are not bound to any of its interfaces, enabling the use of a floating, virtual IP.sysctl.confininet.ipv4.ip_nonlocal_bind=1sysctl.confininet.ipv4.ip_nonlocal_bind=1 -
Configure your
bind
line in the HAProxy Enterprise configuration to use the virtual IP address.haproxyfrontend myfrontendmode httpbind 192.168.50.10:80default_backend web_servershaproxyfrontend myfrontendmode httpbind 192.168.50.10:80default_backend web_servers -
Reboot the server.
Verify the setup Jump to heading
To check that the VRRP service is running, use systemctl status
:
nix
sudo systemctl status hapee-extras-vrrp --no-pager
nix
sudo systemctl status hapee-extras-vrrp --no-pager
outputtext
● hapee-extras-vrrp.service - HAPEE VRRP : VRRP daemon (Keepalived)Loaded: loaded (/lib/systemd/system/hapee-extras-vrrp.service; enabled; vendor preset: enabled)Active: active (running) since Wed 2023-10-18 18:01:59 UTC; 10min agoMain PID: 1191 (hapee-vrrp)Tasks: 2 (limit: 4555)Memory: 1.5MCPU: 4.632sCGroup: /system.slice/hapee-extras-vrrp.service├─1191 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid└─1192 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pidOct 18 18:02:01 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) forcing a new MASTER electionOct 18 18:02:02 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Transition to MASTER STATEOct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Entering MASTER STATEOct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) setting protocol E-VIPs.Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Sending gratuitous ARPs on eth0 for 172.1.1.4
outputtext
● hapee-extras-vrrp.service - HAPEE VRRP : VRRP daemon (Keepalived)Loaded: loaded (/lib/systemd/system/hapee-extras-vrrp.service; enabled; vendor preset: enabled)Active: active (running) since Wed 2023-10-18 18:01:59 UTC; 10min agoMain PID: 1191 (hapee-vrrp)Tasks: 2 (limit: 4555)Memory: 1.5MCPU: 4.632sCGroup: /system.slice/hapee-extras-vrrp.service├─1191 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pid└─1192 /opt/hapee-extras/sbin/hapee-vrrp -D -f /etc/hapee-extras/hapee-vrrp.cfg -p /var/run/hapee-extras/hapee-vrrp.pidOct 18 18:02:01 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) forcing a new MASTER electionOct 18 18:02:02 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Transition to MASTER STATEOct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Entering MASTER STATEOct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) setting protocol E-VIPs.Oct 18 18:02:03 hapee01 Keepalived_vrrp[1192]: VRRP_Instance(vrrp_1) Sending gratuitous ARPs on eth0 for 172.1.1.4
Manual failover Jump to heading
To activate the standby load balancer while simultaneously putting the active load balancer into standby mode:
-
Edit the VRRP configuration
/etc/hapee-extras/hapee-vrrp.cfg
on the active load balancer. -
Lower the
priority
value to less than the standby load balancer’spriority
. This will put the active load balancer into the standby state and simultaneously activate the standby load balancer. For example, change it from 101 to 90. -
Restart the
hapee-extras-vrrp
service.Reverse these actions to restore the original state.
Failover triggers Jump to heading
The following events can trigger a failover:
- The active node lowers its weight below one of the backup nodes due to a failed health check.
- A backup node is reconfigured with a weight larger than the current active node.
- The active node stops emitting its heartbeat packet to the cluster.
VRRP health check scripts Jump to heading
In the VRRP configuration, the vrrp_script
blocks define health checks that can trigger a failover. By default, our configuration checks whether the SSH daemon is running and whether HAProxy Enterprise is running. You may add more checks.
Consider this block, which checks the status of the SSH daemon every five seconds. If the service is not available, the VRRP instance’s weight (i.e. its priority) is reduced by 4 points:
hapee-vrrp.cfgtext
vrrp_script chk_sshd {script "pkill -0 sshd" # pkill -0 is cheaper than pidofinterval 5 # check every 5 secondsweight -4 # remove 4 points of prio if missingfall 2 # check twice before setting downrise 1 # check once before setting up}
hapee-vrrp.cfgtext
vrrp_script chk_sshd {script "pkill -0 sshd" # pkill -0 is cheaper than pidofinterval 5 # check every 5 secondsweight -4 # remove 4 points of prio if missingfall 2 # check twice before setting downrise 1 # check once before setting up}
The HAProxy Enterprise health check is similar, except that if the hapee-lb
process is running, it adds 6 points. The reason is, in case one load balancer
has SSH working but not hapee-lb
, while the other has hapee-lb
but not SSH, the one with hapee-lb
will become the active node, even though it is not manageable remotely.
hapee-vrrp.cfgtext
vrrp_script chk_lb {script "pkill -0 hapee-lb" # pkill -0 is cheaper than pidofinterval 1 # check every secondweight 6 # add 6 points of prio if presentfall 2 # check twice before setting downrise 1 # check once before setting up}
hapee-vrrp.cfgtext
vrrp_script chk_lb {script "pkill -0 hapee-lb" # pkill -0 is cheaper than pidofinterval 1 # check every secondweight 6 # add 6 points of prio if presentfall 2 # check twice before setting downrise 1 # check once before setting up}
The following fields define a vrrp_script
block:
Name | Argument type | Description |
---|---|---|
script |
string | Command to run; Running pkill -0 checks if any process with the given name is running. If it returns 0, the check passes; If it returns 1, the check fails. |
interval |
integer | Interval (in seconds) between two checks. |
weight |
integer | Points to add or remove to instance weight. |
fall |
integer | Number of consecutive negative checks before considered as down. |
rise |
integer | Number of consecutive positive checks before considered as up. |
VRRP instance reference Jump to heading
The following fields define a vrrp_instance
block:
Name | Argument type | Description |
---|---|---|
vrrp_instance |
string | Describes a new VRRP instance. |
interface |
string | Interface managed by the VRRP module. |
state |
string | State at startup until VRRP priority negotiation completes. Can be either MASTER or BACKUP. |
virtual_router_id |
integer | VRRP instance identifier which must be common to all nodes of the same cluster. Its value ranges from 0 to 255. |
priority |
integer | Weight of local node in this VRRP instance. NOTE This will not work on cloud provider networks that disable IP multicast. |
virtual_ipaddress |
string | List of virtual IP addresses to add or remove when the state changes to active or backup. All VRRP nodes in a cluster must own the same IP. NOTE This will not work on cloud provider networks that disable IP multicast. |
virtual_ipaddress_excluded |
string | Same as virtual_ipaddress ; but IPs are not announced in the VRRP heartbeat packet, which reduces bandwidth usage when sending packets. |
track_interface |
string | Tracks interface status and updates priority accordingly. |
track_script |
string | Runs the health check scripts and updates priority accordingly. |
Do you have any suggestions on how we can improve the content of this page?