Introduction to HAProxy Maps

An HAProxy map file stores key-value pairs and is the starting point for some inventive behavior including dynamic rate limiting and blue-green deployments.

Dictionaries. Maps. Hashes. Associative arrays. Can you imagine life without these wonderful key-value data structures? The mad, dystopian world it would be? They’re just sort of there when needed: a trusty tool, never too far out of reach.

So maybe you’re not entirely shocked that they’re counted among the HAProxy load balancer’s extensive feature set. They’re called maps and are built-in and ready to roll. What you’re probably not expecting are the imaginative tasks that you can tackle with them.

Want to set up blue-green deployments? Maybe you’d like to set up rate limits by URL path? How about just dynamically switching which backend servers are used for a domain? It’s all done with maps!

In this post, you’ll learn how to create a map file, store it on your system, reference it in your HAProxy configuration, and update it in real time. You’ll also see some useful scenarios in which to put your new knowledge to good use.

Getting Started

Before considering the fascinating things you can do with a map file, let’s wrap our minds around what a map file is.

The map file

Everything starts by creating a map file. Fire up your favorite text editor and create a file named hosts.map. Then add the following lines to it:

A few things to note about the structure of this file:

• It’s plain text

• A key begins each line (e.g. static.example.com)

• A value comes after a key, separated by at least one space (e.g. be_static)

• Empty lines and extra whitespace between words are ignored

• Comments must begin with a hash sign and must be on their own line

A map file stores key-value pairs. HAProxy uses them as a lookup table, such as to find out which backend to route a client to based on the value of the Host header. The benefit of storing this association in a file rather than in the HAProxy configuration itself is the ability to change those values dynamically.

Next, transfer this file to a server where you have an instance of HAProxy that you don’t mind experimenting with and place it into a directory of your choice. In these examples, since we’re using HAProxy Enterprise, we’ll store it under /etc/hapee-1.8/maps. Map files are loaded by HAProxy when it starts, although, as you’ll see, they can be modified during runtime without a reload.

Map converters

To give you an idea of what you can do with map files, let’s look at using one to find the correct backend pool of servers where users should be sent. You will use the hosts.map file that you created previously to look up which backend should be used based on a given domain name.

Begin by editing your haproxy.cfg file. As you will see, you will add a map converter that reads the map file and returns a backend name.

A converter is a directive placed into your HAProxy configuration that takes in an input and returns an associated output. There are various types of converters. For example, you might use the lower converter to change a given string to lowercase or url_dec to URL decode it.

In the following example, the input is a string literal example.com and the map converter looks up that key in the map file, hosts.map.

The first row in hosts.map that has example.com as a key will have its value returned. Notice how the input, str(example.com), begins the expression and is separated from the converter with a comma.

When this expression is evaluated at runtime, it will be converted to the line use_backend be_static, which directs requests to the be_static pool of servers. Of course, rather than passing in a hardcoded string like example.com, you can send in the value of an HTTP header or a URL parameter. The next example uses the value of the Host header as the input.

The map converter takes up to two arguments. The first is the path to your map file. The second, optional argument declares a default value that will be used if no matching key is found. So, in this case, if there’s no match, be_static will be used. If the input matches multiple items in the map file, HAProxy will return the first one.

The map converter looks for an exact match in the file, but there are a few variants that provide opportunities for a partial match. The most commonly used are summarized here:

map_beg	Looks for entries in the map file that match the beginning of the input (e.g. an input of “abcd” would match “a” in the file).
map_end	Looks for entries in the map file that match the end of the input (e.g. an input of “abcd” would match “d” in the file). Unlike the other match modes, this doesn’t perform ebtree lookups and instead checks each line.
map_sub	Looks for entries in the map file that make up a substring of the sample (e.g. an input of “abcd” would match “ab” or “c” in the file). Unlike the other match modes, this doesn’t perform ebtree lookups and instead checks each line.
map_ip	This takes the input as an IP address and looks it up in the map. If the map has masks (such as 192.168.0.0/16) in it then any IP in the range will match it.This is the default if the input type is an IP address.
map_reg	This reads the samples in the map as regular expressions and will match if the regular expression matches.Unlike the other match modes, this doesn’t perform ebtree lookups and instead checks each line.
map_str	An alias for map. This takes a string as an input, matches on the whole key, and returns a value as a string.

Modifying the Values

Much of the value of map files comes from your ability to modify them dynamically. This allows you to, for example, change the flow of traffic from one backend to another, such as for maintenance.

There are four ways to change the value that we get back from a map file. First, you can change the values by editing the file directly. This is a simple way to accomplish the task but does require a reload of HAProxy. This is a good choice if you’re using a configuration management tool like Puppet or Ansible.

The second way is provided with HAProxy Enterprise via the lb-update module, which you’ll really appreciate if you’re running a cluster of load balancers. It allows you to update maps within multiple instances of HAProxy at once by watching a map file hosted at a URL at a defined interval.

A third way to edit the file’s contents is by using the Runtime API. The API provides all of the necessary CRUD operations for creating, removing, updating, and deleting rows from the map in memory, without needing to reload HAProxy. There’s also a simple technique for saving your changes to disk, which you’ll see later in this post.

A fourth way is with the http-request set-map directive in your HAProxy configuration file. This gives you the opportunity to update map entries based on URL parameters in the request. It’s easy to turn this into a convenient HTTP-based interface for making map file changes from a remote client.

In the next few sections, you’ll get some guidance on how to use these techniques.

Editing the file directly

A straightforward way to change the values you get back from your map file is to change the file itself. Open the file and make any modifications you need: adding rows, removing others, changing the values of existing rows. However, know that HAProxy only reads the file when it’s starting up and then loads it into memory. Refreshing the file, then, means reloading HAProxy.

Thanks to hitless reloads introduced in HAProxy Enterprise 1.8r1 and HAProxy 1.8, you can trigger a reload without dropping active connections. Read our blog post Hitless Reloads with HAProxy – HOWTO for an explanation on how to use this feature.

This approach will work with configuration management tools like Puppet, which allow you to distribute changes to your servers at a set interval. Be sure to reload HAProxy to pick up the changes.

Editing with the lb-update module

Although configuration management tools allow you to update the servers in your cluster, they can be a heavy solution that requires administration. An alternative is using the lb-update module to keep each replica of HAProxy within your cluster in sync. The lb-update module instructs HAProxy Enterprise to retrieve the contents of the map file from a URL at a defined interval. The module will automatically check for updates as frequently as configured. This is especially useful when there are a lot of processes and/or servers in a cluster that need the updated files.

Below is a sample of a dynamic-update section that manages updating the hosts.map file from a URL. You’d add an update directive for each map file that you want to watch.

See the HAProxy Enterprise documentation for detailed usage instructions or contact us to learn more.

Editing with the Runtime API

Looking back at our previous blog post, Dynamic Configuration with the HAProxy Runtime API, you’ll see that there are several API methods available for updating an existing map file.

The table below summarizes them.

API method	Description
show map	Lists available map files or displays a map file’s contents.
get map	Reports the keys and values matching a given input.
set map	Modifies a map entry.
add map	Adds a map entry.
del map	Deletes a map entry.
clear map	Deletes all entries from a map file.

Without any parameters, show map lists the map files that are loaded into memory. If you give it the path to a particular file, it will display its contents. In the following example, we use it to display the key-value pairs inside hosts.map.

The first column is the location of the entry and is typically ignored. The second column is the key to be matched and the third is the value. We can easily add and remove entries via the Runtime API. To remove an entry from the map file, use del map. Note that this only removes it from memory and not from the actual file.

You can also delete all entries with clear map:

Add a new key and value with add map:

Change an existing entry with set map:

Using show map, we can get the contents of the file, filter it to only the second and third columns with awk, and then save the in-memory representation back to disk:

Actions can also be chained together with semicolons, which makes it easy to script changes and save the result:

Editing with http-request set-map

Suppose you didn’t want to go about editing files by hand or using the Runtime API. Instead, you wanted to be able to make an HTTP request with a certain URL parameter and have that update your map file. In that case, http-request set-map is your go-to.

This allows the use of fetches, converters, and ACLs to decide when and how to change a map during runtime. In addition to set-map, there’s also del-map, which allows you to remove map entries in the same way. As with the runtime API, these changes also only apply to the process that the request ends up on.

Pass the map file’s path to set-map and follow it with a key and value, separated by spaces, that you want to add or update. Both the key and value support the log-format notation, so you can specify them as plain strings or use fetches and converters. For example, to add a new entry to the hosts.map file, but only if the source address falls within the 192.168.122.0/24 range, you can use a configuration like this:

This will allow you to make web requests such as http://192.168.122.64/map/add?domain=example.com&backend=be_static for a quick and easy way to update your maps. If the entry already exists, it will be updated. Notice that you can use http-request deny deny_status 200 to prevent the request from going to your backend servers.

The http-request del-map command is followed by the key to remove from the map file.

Using the show map technique you saw earlier, you might schedule a cron job to save your map files every few minutes. However, if you need to replicate these changes across multiple instances of HAProxy, using one of the other approaches will be a better bet.

DID YOU KNOWAnother way to control when to set or delete an entry is to check the method of the request and then set an entry if it’s POST or PUT. If it’s DELETE, delete an entry.

Putting it into Practice

We’ve seen how to use the Host header to look up a key in a map file and choose a backend to use. Let’s see some other ways to use maps.

A blue-green deployment

Suppose you wanted to implement a blue-green deployment wherein you’re able to deploy a new release of your web application onto a set of staging servers and then swap them with a set of production servers. You could create a file called bluegreen.map and add a single entry:

In this scenario, the be_blue backend contains your set of currently active, production servers. Here is your HAProxy configuration file:

After you deploy a new version of your application to the be_green servers and test it, you can use the Runtime API to swap the active be_blue servers with the be_green servers, causing your be_green servers to become active in production.

Now your traffic will be directed away from your be_blue servers and to your be_green servers. This, unlike a rolling deployment, ensures that all of your users are migrated to the new version of your application at the same time.

Rate limiting by URL path

For this example, you will set rate limits for your website. Using a map file lets you set different limits for different URLs. For example, URLs that begin with /api/routeA may allow a higher request rate than those that begin with /api/routeB.

Add a map file called rates.map and add the following entries:

Consider the following frontend, wherein the current request rate for each client is measured over 10 seconds. A URL path like /api/routeA/someFunction would allow up to four requests per second (40 requests / 10 seconds = 4 rps).

Here, the stick-table definition records client request rates over ten seconds. Note that we are tracking clients using the base32+src fetch method, which is a combination of the Host header, URL path, and source IP address. This allows us to track each client’s request rate on a per-path basis. The base32+src value is stored in the stick table as binary data.

Then, two variables are set with http-request set-var. The first, req.rate_limit, is set to the predefined rate limit for the current path from the rates.map file. The second, req.request_rate, is set to the client’s current request rate.

The ACL rate_abuse does a calculation to see whether the client’s request rate is higher than the limit for this path. It does this by subtracting the request rate from the request limit and checking whether the difference is less than zero. If it is, the http-request deny directive responds with 429 Too Many Requests.

Conclusion

Now that you’ve seen a few of the possibilities, consider reaching for your trusty tool, maps, the next time you run into a problem where it can help. What would you use maps for? Leave us a comment here or start a conversation with us on Twitter!

Planning to update maps across a cluster of HAProxy load balancers? Try out a free trial of HAProxy Enterprise so that you can take advantage of the lb-update module. Being an HAProxy Enterprise customer also includes expert technical support, so we can help you plan the map rules that will solve your specific problems.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.