HAProxy on AWS: Best Practices Part 2

There is such a thing as too many layers. There’s a restaurant in town that serves the most outrageously layered sandwiches. I’m talking burgers topped with cheese, ham, onion rings, and guacamole. It’s the same with load balancers: layering one proxy over another proxy—it starts to seem like overkill.

As you learned in the previous article in this series, when hosting your infrastructure in AWS, high availability can be achieved by placing AWS Elastic Load Balancing (ELB) in front of two HAProxy load balancers. It’s a way to load balance the load balancers, so to speak.

Yet, something about it gives you a bit of heartburn. “Can’t I just use HAProxy without these extra layers?” you wonder.

Yes, you can!

In this post, you’ll remove the dependency on ELB. A pair of active-active HAProxy Enterprise load balancers will receive Internet traffic directly. Both will receive requests and forward them to backend servers, creating redundancy without the need for additional, downstream proxies.

Placing HAProxy at the edge of your AWS infrastructure is possible without involving Elastic Load Balancing (ELB). In this article, we’ll discuss how. This blog post is part of a series. See Part 1 and Part 3.

High Availability Architecture

Let’s get a quick overview of how the experiment will look. You’ll have two highly available EC2 instances running HAProxy Enterprise. Each will be assigned an Elastic Network Interface (ENI) that’s associated with two private IP addresses. On each machine, one private IP address will be mapped to a publicly routable Elastic IP address (EIP). The other is held in reserve. In the event that the other HAProxy instance fails, its EIP will be mapped to this unused private IP on the healthy instance. When the failed node recovers, the EIP will return to its original ENI.

The load balancers will serve traffic to backend node.js websites. Using node.js makes it easy to create simple, one-off web applications.

In the event that one of the load balancers fails, you’ll use the AWS Command Line Interface (CLI) to dynamically reassign its Elastic IP address to the other node. Instead of calling the CLI commands yourself, though, you will let Heartbeat do that. Heartbeat is a tool that comes from the Linux-HA project and provides high-availability clustering. Heartbeat will send UDP heartbeat messages to your nodes and dynamically failover the EIP to the other node if one peer can’t reach the other.

This model requires you to create Identity and Access Management (IAM) policies in AWS to be able to both view and modify EIP assignments dynamically. You’ll see how to automate the whole setup by using Terraform and Ansible.

Infrastructure Creation with Terraform

You’ll create the environment in AWS by using Terraform. Terraform lets you write declarative configuration files that automate the process of creating infrastructure in AWS. In this way, you get a repeatable process for setting up and tearing down your cloud-based infrastructure. Download the full example project on GitHub. To run Terraform, navigate into the example project directory and run:

The resources being created include the following:

• Two HAProxy Enterprise load balancers

• Three node.js web applications

• The network including VPC, subnet, Internet gateway, and route table

• Security groups for allowing some types of traffic to reach our nodes

• IAM policy permissions

• Public and private IP address associations

While the network and instance creation with Terraform is quite straightforward and likely doesn’t need further explanation, we will show the excerpt that selects the most recent Ubuntu-based HAProxy Enterprise image from the AWS Marketplace (with “483gxnuft87jy44d3q8n4kvt1” being the HAPEE AWS Marketplace Product ID).

The same strategy is used for our node.js web servers, but using a regular Ubuntu 16.04 image (with “099720109477” being the Canonical AWS Marketplace Owner ID).

In order for your nodes to communicate with one another and the outside world, you must define security groups that act as firewalls with only specific ports open. This is handled by Terraform. The security group for the HAProxy nodes includes ingress rules for the following:

• SSH (port 22) from anywhere

• HTTP/HTTPS (ports 80 and 443) from anywhere

• HAProxy Real-Time Dashboard UI (ports 9022 and 9023) from anywhere

• Heartbeat unicast (port udp/694) from inside the network

• ICMP Type 3, Code 4 (MTU Discovery) from anywhere

The security group for the node.js backends will need to permit the following:

• SSH (port 22) from anywhere

• HTTP (port 80) from the load balancers

Another security precaution that AWS implements is restricting who can call various AWS API methods. Since your EC2 instances will use Heartbeat to list network interfaces and associate, dissociate, and release EIPs, Terraform must set up IAM policies that allow these actions.

The following Terraform configuration creates an IAM role with the necessary permissions and assigns it to an instance profile. This instance profile is later assigned to the HAProxy EC2 instances.

While we’re on the subject of IAM permissions, note that when you apply your Terraform configuration, the user account that you connect to AWS with needs to have certain privileges. In particular, because you are dynamically giving your load balancers the ability to view and change network settings, your Terraform user account must have the rights to grant those permissions. It will need access to the following actions, which can be set through the AWS console:

Post-creation Configuration with Ansible

Right after creating the initial stack with Terraform, you must run Ansible to ensure that all of your instances have up-to-date software, required configuration, secondary IP addresses, helper scripts, etc. Ansible is an automation tool for provisioning servers with required software and settings. The ansible-playbook command connects to your AWS EC2 instances and applies for the Ansible roles.

You’ll need to set up an SSH key-pair beforehand so that you can connect. This is set in the variables.tf file for Terraform and ansible.cfg for Ansible. Also, note that the Jinja2 templating engine requires the Python package jmespath to be installed on your workstation. You can install it as follows:

Here’s a summary of the Ansible roles, describing their purpose:

Ansible roles applied to HAProxy instances

Role name	Purpose
secondary-ip	Ensures that each HAPEE instance is able to configure a secondary private IP on boot, as that doesn’t happen by default on Amazon EC2.
ec2facts	Gathers ENI and EIP facts for further use in Heartbeat EIP helper scripts.
hapee-lb	Auto-generates the hapee-lb.cfg configuration file from a Jinja2 template and populates private IPs in the backend server definition.
heartbeat	Handles complete Heartbeat installation with all prerequisites and configuration (ha.cf, authkeys, haresources, updateEIP1 and updateEIP2 Heartbeat Resource Agents) on both load-balancer nodes; note that helper scripts differ between load-balancer nodes.

Ansible role applied to web backend nodes

Role name	Purpose
nodejs	Handles installation and configuration of the node.js HTTP server.

In the upcoming sections, we’ll explain these roles in detail.

secondary-ip

In order for a load balancer to reassign a failed peer’s EIP to itself, you need to have a secondary private IP address to associate with it. EIPs are assigned either to the whole ENI or to an individual private IP address. We are choosing the latter because dealing with multiple ENIs causes more overhead.

Every EC2 instance gets one private IP by default. Adding a second is a two-step process. First, you’ll use the ec2_eni module from the development version of Ansible to add a secondary private address to the existing ENI.

The next step is to query AWS for instance metadata to find out what address was assigned. In the Amazon Linux distribution, a specific ec2-net-utils package would handle this step, ensuring that, if enabled, a secondary private IP address keeps being refreshed with DHCP. Sadly this doesn’t happen automatically with any other Linux distribution, so we must use a rudimentary shell script to do so. The secondary-ip Ansible role copies this shell script to the load balancer node and sets it to run as a service.

The script is as follows:

Now after the service starts up, each load balancer will have two private IP addresses assigned to their respective ENIs.

ec2facts

Prior to applying for the heartbeat role, the ec2facts role gathers some information about the EC2 ENIs and EIPs. This includes information about interface IDs, allocation IDs, and private addresses that are needed when configuring Heartbeat.

Although the example project gets many of its variables from ec2.py, which is used for creating a dynamic inventory, it doesn’t populate all of the information you need. So, the example project relies on two other development-version Ansible modules: ec2_eni_facts and ec2_eip_facts. Together, these generate the rest of the necessary variables.

hapee-lb

The hapee-lb role generates a complete HAProxy Enterprise configuration file, hapee-lb.cfg, that sets up round-robin HTTP load-balancing over all of the backend web servers. It uses a fairly extensible Jinja2 template, which you can further customize to cover specific configuration cases such as a number of threads to run depending on the number of CPU cores, and so on:

When this template is rendered, the listen section will include all of the addresses of your web servers, as in the following example:

heartbeat

The real magic of the project is setting up high availability between your load balancers through Heartbeat. Furthermore, Heartbeat associates the elastic IP addresses with the private IP addresses on the HAProxy nodes.

This role is the most complex of all, as several files must be rendered from templates. The Heartbeat configuration file, ha.cf, needs to list all private IP addresses in use as well as the default gateway, which it pings to avoid split-brain situations. Its Ansible template looks like this:

The rendered file will look like this:

We are using unicast communication, as in a cloud environment using multicast is not a feasible option unless we delve into various overlays that make it happen. For our purposes, using unicast is good enough. When the template is rendered, each private IP address will be listed on its own line, prefixed with ucast eth0.

In the haresources file, we list the primary node for each EIP resource and the helper scripts, updateEIP1 and updateEIP2, that map the EIPs to private addresses. The helper scripts are basically just wrappers over the AWS CLI:

Let’s review possible scenarios that the updateEIP Resource Agent scripts will handle. Note that these are just regular LSB init scripts.

• No failure scenario:

  • updateEIP1 on the first node associates EIP1 to primary ENI, primary private IP

  • updateEIP2 on the second node allocates EIP2 to primary ENI, primary private IP

• Failure scenario:

  • updateEIP2 on the first node (failure on the second node) associates EIP2 to primary ENI, secondary private IP

  • updateEIP1 on the second node (failure on the first node) associates EIP1 to primary ENI, secondary private IP

Let’s take a look at the template for the updateEIP1 script:

The updateEIP2 script is the same as updateEIP1, except that first_eip is switched for second_eip. Both scripts have standard and required actions for a Resource Agent: The start command will associate an EIP to a primary ENI and primary or secondary private address, depending on whether it is an EIP belonging to a remote peer or to a local instance; the stop command will disassociate an EIP; the status command just shows an EIP association status.

The End Result

Once these Ansible roles are executed, you’ll have two HAProxy Enterprise load balancers that accept traffic directly over publicly routed Elastic IP addresses. Remember to update your DNS settings to point to both EIPs.

These instances are able to monitor one another and recover their peer’s EIP if the peer is stopped due to maintenance or failure. This allows you to put HAProxy at the front lines of your cloud infrastructure without the need for Elastic Load Balancing.

Conclusion

This concludes our HAProxy Enterprise Heartbeat HA example. Doesn’t it feel good to remove some extra layers from your AWS infrastructure? As a reminder, all of the example code is available and contains complete Terraform and Ansible configurations. Use it for testing and to contribute further improvements!

In the next part of this blog series, we will replace Heartbeat with Keepalived/VRRP, which is our standard enterprise-recommended setup. Please leave comments below! Contact us to learn more about HAProxy Enterprise or sign up for a free trial.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.