HAProxy 3.1 is now the latest version. Learn more
13th of October 2015, Willy Terrau announced the release of HAProxy 1.6.0, after 16 months of development. A total of 1156 commits from 59 people were committed since the release of 1.5.0.
You can find the official announcement here. In his mail, Willy detailed all the features that have been added to this release. The purpose of this blog is to highlight a few of them, providing the benefits and some configuration examples.
Most of the features below were already backported and integrated into our HAProxy Enterprise and ALOHA.
HAProxy Enterprise is our open source version of HAProxy based on the HAProxy community stable branch, where we backport many features from the dev branch and we package it to make the most stable, reliable, advanced, and secured version of HAProxy.
It also comes with third-party software to fill the gap between a simple HAProxy process and a load-balancer (VRRP, syslog, SNMP, Route Health Injection, etc…). The cherry on the cake, we provide enterprise support.
It’s 2015, Let’s Use QUOTE in the Configuration File
Those who've used HAProxy for a long time will be happy to know that the ‘\ ‘ (backslash-space) sequence is an old painful souvenir with 1.6.
You can now write:
reqirep "^Host: www.(.*)" "Host: foobar\1"or
option httpchk GET / "HTTP/1.1\r\nHost: www.domain.com\r\nConnection: close"Lua Scripting
Maybe the biggest change that occurred is the integration of Lua. Quote from Lua’s website: “Lua is a powerful, fast, lightweight, embeddable scripting language.“.
Now everyone has the ability to extend HAProxy by writing and running their own Lua scripts. No need to write C code, maintain patches, etc…
If some Lua snippets are very popular, we may write the equivalent feature in C and make it available in HAProxy mainline.
One of the biggest challenges Thierry faced when integrating Lua is to give it the ability to propose non-blocking processing of Lua code and non-blocking network socket management.
HAProxy requires Lua 5.3 or above.
With Lua, we can add new functions to the following HAProxy elements:
Service
Action
Sample-fetch
Converter
Compiling HAProxy and Lua
Installing Lua 5.3
cd /usr/src
curl -R -O http://www.lua.org/ftp/lua-5.3.0.tar.gz
tar zxf lua-5.3.0.tar.gz
cd lua-5.3.0
make linux
sudo make INSTALL_TOP=/opt/lua53 installLua 5.3 library and include files are now installed in /opt/lua53.
Compiling HAProxy with Lua support:
make TARGET=linux2628 USE_OPENSSL=1 USE_PCRE=1 USE_LUA=1 LUA_LIB=/opt/lua53/lib/ LUA_INC=/opt/lua53/include/HAProxy/Lua simple "Hello world!" example
A simple Hello world! in Lua could be written like this:
The Lua code in a file called hello_world.lua:
core.register_service("hello_world", "tcp", function(applet)
applet:send("hello world\n")
end)The HAProxy configuration:
global
lua-load hello_world.lualisten proxy
bind 127.0.0.1:10001
tcp-request content use-service lua.hello_worldCaptures
HAProxy‘s running context is very important when writing configuration. In HAProxy, each context is isolated. IE: you can’t use a request header when processing the response.
With HAProxy 1.6, this is now possible: you can declare capture slots, store data in it and use it at any time during a session.
defaults
mode http
frontend f_myapp
bind :9001
declare capture request len 32 # id=0 to store Host header
declare capture request len 64 # id=1 to store User-Agent header
http-request capture req.hdr(Host) id 0
http-request capture req.hdr(User-Agent) id 1
default_backend b_myapp
backend b_myapp
http-response set-header Your-Host %[capture.req.hdr(0)]
http-response set-header Your-User-Agent %[capture.req.hdr(1)]
server s1 10.0.0.3:4444 checkTwo new headers are inserted in the response:Your-Host: 127.0.0.1:9001
Your-User-Agent: curl/7.44.0
Multiprocess, peers, and stick-tables
In HAProxy 1.5, we introduced “peers” to synchronize stick-table content between HAProxy servers. This feature was not compatible with multi-process mode.
We can now, in 1.6, synchronize the content of the table to stick to one process. It allows creating configurations for massive SSL processing pointing to a single backend stuck on a single process where we can use stick tables and synchronize its content.
peers article
peer itchy 127.0.0.1:1023
global
pidfile /tmp/haproxy.pid
nbproc 3
defaults
mode http
frontend f_scalessl
bind-process 1,2
bind :9001 ssl crt /home/bassmann/haproxy/ssl/server.pem
default_backend bk_lo
backend bk_lo
bind-process 1,2
server f_myapp unix@/tmp/f_myapp send-proxy-v2
frontend f_myapp
bind-process 3
bind unix@/tmp/f_myapp accept-proxy
default_backend b_myapp
backend b_myapp
bind-process 3
stick-table type ip size 10k peers article
stick on src
server s1 10.0.0.3:4444 checkLog
log-tag
It is now possible to position a syslog tag per process, frontend or backend. The purpose is to ease the job of syslog servers when classifying logs. If no log-tag is provided, the default value is the program name.
Example applied to the configuration snippet right above:
frontend f_scalessl
log-tag SSL
[...]
frontend f_myapp
log-tag CLEAR
[...]New log format variables
New log format variables:
%HM: HTTP method (ex: POST)
%HP: HTTP request URI without query string (path)
%HQ: HTTP request URI query string (ex: ?bar=baz)
%HU: HTTP request URI (ex: /foo?bar=baz)
%HV: HTTP version (ex: HTTP/1.0)
Server IP resolution using DNS at runtime
HAProxy 1.5 performed DNS resolution when parsing configuration, in a synchronous mode and using the glibc (hence /etc/resolv.conf file).
Now, HAProxy can perform DNS resolution at runtime, in an asynchronous way, and update server IP on the fly. This is very convenient in environments like Docker or Amazon Web Service where server IPs can be changed at any time.
Configuration example applied to docker. A dnsmasq is used as an interface between /etc/hosts file (where docker stores server IPs) and HAProxy:
resolvers docker
nameserver dnsmasq 127.0.0.1:53
defaults
mode http
log global
option httplog
frontend f_myapp
bind :80
default_backend b_myapp
backend b_myapp
server s1 nginx1:80 check resolvers docker resolve-prefer ipv4Then, let’s restart s1 with the command “docker restart nginx1” and let’s have a look at the magic in the logs:(...) haproxy[15]: b_myapp/nginx1 changed its IP from 172.16.0.4 to 172.16.0.6 by docker/dnsmasq.
HTTP rules
New HTTP rules:
http-request: capture, set-method, set-uri, set-map, set-var, track-scX, sc-inc-gpc0, sc-set-gpt0, silent-drop
http-response: capture, set-map, set-var, sc-inc-gpc0, sc-set-gpt0, silent-drop, redirect
Variables
We often used HTTP header fields to store temporary data in HAProxy. With 1.6, you can now define variables. A variable is available for a scope: session, transaction (request or response), request, response.
A variable name is prefixed by its scope (sess, txn, req, res), a dot ‘.’ and a tag composed by ‘a-z’, ‘A-Z’, ‘0-9’, and ‘_’.
Let’s rewrite the capture example using variables:
global
# variables memory consumption, in bytes
tune.vars.global-max-size 1048576
tune.vars.reqres-max-size 512
tune.vars.sess-max-size 2048
tune.vars.txn-max-size 256
defaults
mode http
frontend f_myapp
bind :9001
http-request set-var(txn.host) req.hdr(Host)
http-request set-var(txn.ua) req.hdr(User-Agent)
default_backend b_myapp
backend b_myapp
http-response set-header Your-Host %[var(txn.host)]
http-response set-header Your-User-Agent %[var(txn.ua)]
server s1 10.0.0.3:4444 checkMailers
Now, HAProxy can send emails when the server states change (mainly goes DOWN).
mailers mymailers
mailer smtp1 192.168.0.1:587
mailer smtp2 192.168.0.2:587
backend mybackend
mode tcp
balance roundrobin
email-alert mailers mymailers
email-alert from test1@horms.org
email-alert to test2@horms.org
server srv1 192.168.0.30:80
server srv2 192.168.0.31:80Processing of HTTP request body
Until 1.5 included, HAProxy could process only HTTP request headers. It can now access request body.
Simply enable the statement below in your frontend or backend to give HAProxy this ability:
option http-buffer-request
Protection against slow-POST attacks
Slow-POST attacks are like the slowlorys, except that HTTP headers are sent quickly, but body requests are sent very slowly.
Once enabled, the timeout HTTP-request parameters also apply to the POSTED data.
Fetch methods
A few new fetch methods now exist to play with the body: req.body, req.body_param, req.body_len, req.body_size, etc…
Short example to detect “SELECT *” string in a request POST body, and of course how to deny it:
defaults
mode http
frontend f_mywaf
bind :9001
option http-buffer-request
http-request deny if { req.body -m reg "SELECT \*" }
default_backend b_myapp
backend b_myapp
server s1 10.0.0.3:4444 checkNew converters
1.5 introduced the converters, but only a very few of them were available. 1.6 adds many of them. The list is long, but let’s give the most important ones: json, in_table, field, regsub, table_* (to access counters from stick-tables),...
Device Identification
Through our company, we have some customers who want us to integrate into HAProxy the ability to detect device type and characteristics and report it to the backend server. We got a couple of contributions from 2 companies' experts in this domain: 51 degrees and deviceatlas.
You can now load those libraries in HAProxy in order to fully qualify the client's capabilities and set up some headers your application server can rely on to adapt content delivered to the client or let the varnish cache server use it to cache multiple flavor of the same object based on client capabilities.
More on this blog later on how to integrate each product.
Seamless Server States
Prior to 1.6, when being reloaded, HAProxy considers all the servers are UP until the first check is performed.
Since 1.6, we can dump server states into a flat-file right before performing the reload and let the new process know where the states are stored. That way, the old and new processes own exactly the same server states (hence seamless).
The following information is reported:
server IP address when resolved by DNS
operational state (UP/DOWN/…)
administrative state (MAINT/DRAIN/…)
Weight (including slowstart relative weight)
health check status
rise / fall current counter
check state (ENABLED/PAUSED/…)
agent check state (ENABLED/PAUSED/…)
The state could be applied globally (all server found) or per backend.
Simple HAProxy configuration:
global
stats socket /tmp/socket
server-state-file /tmp/server_state
backend bk
load-server-state-from-file global
server s1 10.0.0.3:4444 check weight 11
server s2 10.0.0.4:4444 check weight 12Before reloading HAProxy, we save the server states using the following command:
socat /tmp/socket - <<< "show servers state" > /tmp/server_state
Here is the content of /tmp/server_state file:
1
# <field names skipped for the blog article>
1 bk 1 s1 10.0.0.3 2 0 11 11 4 6 3 4 6 0 0
1 bk 2 s2 10.0.0.4 2 0 12 12 4 6 3 4 6 0 0Now, let’s proceed with reloading as usual. Of course, the best option is to export the server states using the init script.
External Check
HAProxy can run a script to perform complicated health checks. Just be aware of the security concerns when enabling this feature!
Configuration:
global
external-check
backend b_myapp
external-check path "/usr/bin:/bin"
external-check command /bin/true
server s1 10.0.0.3:4444 checkTLS / SSL
Some features introduced here may need a recent OpenSSL library.
Detection of ECDSA-able clients
This has already been documented in this blog: Serving ECC and RSA certificates on the same IP with HAProxy
SSL certificate forgery on the fly
Since 1.6, HAProxy can forge SSL certificates on the fly! Yes, you can use HAProxy with your company’s CA to inspect content.
Support of Certificate Transparency (RFC6962) TLS extension
When loading PEM files, HAProxy also checks for the presence of a file at the same path suffixed by “.sctl”. If such a file is found, support for Certificate Transparency (RFC6962) TLS extension is enabled. The file must contain a valid Signed Certificate Timestamp List, as described in RFC. The file is parsed to check basic syntax, but no signatures are verified.
TLS Tickets key load through the stats socket
A new stats socket command is available to update TLS ticket keys at runtime. The new key is used for encryption/decryption while the old ones are used for decryption only.
Server side SNI
Many application servers now take benefit from Server Name Indication TLS extension.
Example:
backend b_myapp_ssl
mode http
server s1 10.0.0.3:4444 check ssl sni req.hdr(Host)Peers v2
The peers protocol used to synchronize data from stick-tables between two HAProxy servers can now synchronize more than just the sample and the server id.
It can synchronize all the tracked counters. Note that each node pushes its local counter to a peer. So this must be used for safe reload and server failover only.
Don’t expect to see 10 HAProxy servers to sync and aggregate counters in real-time.
That said, this protocol has been extended to support different data type, so we may see more features soon relying on it.
HTTP connection sharing
On the road to HTTP/2, HAProxy must be able to support connection pooling. And on the road to the connection pools, we have the ability to share a server-side connection between multiple clients.
The server-side connection may be used by multiple clients until the owner (the client-side connection) of this session dies. Then new connections may be established.
The new keyword is http-reuse and has different level of sharing connections:
never: no connections are shared
safe: first request of each client is sent over its own connection, subsequent requests may use another connection. It works like a regular HTTP keepalive
aggressive: send a request to connections that has proved reliably support connection reuse (no quick connection close after a response has been sent)
always: send requests to established connections, whatever happens. If the server was closing the connection in the meantime, the request is lost, and the client must resend it.
Get Rid of 408 in Logs
Simply use the new option:
option http-ignore-probes.
