Docker is a nice tool to handle containers: it allows building and running your apps in a simple and efficient way.
When used in production together with HAProxy, DevOps teams face a big challenge: how to followup a container IP change when restarting a container?
This blog article aims at giving the first answer to this question.
The version of docker used for this article is 1.8.1 (very important, since docker’s default behavior has changed in 1.9.0…)
HAProxy, Webapp & Docker Diagram
The diagram below shows how docker runs on my laptop:
A docker0 network interface, with IP 172.16.0.1
all containers run in the subnet 172.16.0.0/16
+-------------------------------------host--------------------------------------+
| |
| +-----------------------docker-----------------------+ 172.16.0.0/16 |
| | | docker0: 172.16.0.1 |
| | +---------+ +---------+ +----------+ | |
| | | HAProxy | | appsrv1 | | rsyslogd | | |
| | +---------+ +---------+ +----------+ | |
| | | |
| +----------------------------------------------------+ |
| |
+-------------------------------------------------------------------------------+The IP address associated with docker0 will be used to export some services.
In this article, we’ll start up 3 containers:
rsyslogd: where HAProxy will send all its logs
appsrv1: our application server, which may be restarted at any time
HAProxy: our load balancer, which must follow up appsrv1‘s IP
Building & Running the Lab
Building
First, we need rsyslogd and HAProxy containers. They can be built from the following Dockerfiles:
Then run:
docker build -t blog:haproxy_dns ~/tmp/haproxy/blog/haproxy_docker_dns_link/blog_haproxy_dns/
docker build -t blog:rsyslogd ~/tmp/haproxy/blog/haproxy_docker_dns_link/blog_rsyslogd/I consider appsrv container as yours: it’s your application.
Starting up our lab
Docker assigns IPs to containers in the order they are started up, incrementing the last byte for each new container.
To make it simpler, let’s restart docker first, so our container IPs are predictable:
sudo /etc/rc.d/docker restartThen let’s start up our rsyslogd container:
docker run --detach --name rsyslogd --hostname=rsyslogd \
--publish=172.16.0.1:8514:8514/udp \
blog:rsyslogdAnd let’s attach a terminal to it:
docker attach rsyslogdNow, run appsrv container as appsrv1:
docker run --detach --name appsrv1 --hostname=appsrv1 demo:appsrvAnd finally, let’s start HAProxy, with a docker link to appsrv1:
docker run --detach --name haproxy --hostname=haproxy \
--link appsrv1:appsrv1 \
blog:haproxy_dnsDocker Links, /Etc/Hosts File Updated and DNS
When using the ”–link” option, docker creates a new entry in the containers /etc/hosts file with the IP address and name provided by the ”link” directive.
Docker will also update this file when the remote container (here appsrv1) IP address is changed (IE when restarting the container).
If you’re familiar with HAProxy, you already know it doesn’t do file system IOs at run time. Furthermore, HAProxy doesn’t use /etc/hosts file directly. The glibc might use it when HAProxy asks for DNS resolution when parsing the configuration file. (read below for DNS resolution at runtime)
That said, if appsrv1 IP gets changed, then /etc/hosts file is updated accordingly, then HAProxy is not aware of the change and the application may fail.
A quick solution would be to reload HAProxy process in its container, to force it to take into account the new IP.
A more reliable solution is to use HAProxy 1.6 DNS resolution capability to follow up the IP change. With this purpose in mind, we added 2 tools to our HAProxy container:
dnsmasq: tiny software which can act as a DNS server that takes /etc/hosts file as its database
inotifytools: watch changes on /etc/hosts file and force dnsmasq to reload it when necessary
I guess now you got it:
when appsrv1 is restarted, then docker gives it a new IP
Docker populates then this IP address into all /etc/hosts file required (those using ”link” directives)
Once populated, inotify tool detect the file change and triggers a dnsmasq reload
HAProxy periodically (can be configured) probes DNS and will get the new IP address quickly from dnsmasq
Docker Container Restart and HAProxy Followup in Action
At this stage, we should have a container attached to rsyslogd and we should be able to see HAProxy logging. Let’s give it a try:
curl http://172.16.0.4/
Nov 17 09:29:09 172.16.0.1 haproxy[10]: 172.16.0.1:55093 [17/Nov/2015:09:29:09.729] f_myapp b_myapp/appsrv1 0/0/0/1/1 200 858 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"Now, let’s consider your dev team delivered a new version of your application, so you build it and need to restart its running container:
docker restart appsrv1and voilà:
==> /var/log/haproxy/events <==
Nov 17 09:29:29 172.16.0.1 haproxy[10]: b_myapp/appsrv1 changed its IP from 172.16.0.3 to 172.16.0.5 by docker/dnsmasq.
Nov 17 09:29:29 172.16.0.1 haproxy[10]: b_myapp/appsrv1 changed its IP from 172.16.0.3 to 172.16.0.5 by docker/dnsmasq.Let’s test the application again:
curl http://172.16.0.4/
Nov 17 09:29:31 172.16.0.1 haproxy[10]: 172.16.0.1:59450 [17/Nov/2015:09:29:31.013] f_myapp b_myapp/appsrv1 0/0/0/0/0 200 858 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"Limitations
There are a few limitations to this mechanism:
it’s painful to maintain a ”link” directive when you have 10s or 100s or more of containers
the host computer and computers in the host network can’t easily access our containers because we don’t know their IPs, and their hostnames are resolved in the HAProxy container only
if we want to add more appserver in HAProxy‘s farm, we still need to restart HAProxy‘s container (and update configuration accordingly)
To fix some of the issues above, we can dedicate a container to perform DNS resolution within our docker world and deliver responses to any running containers or hosts in the network. We’ll see that in a next blog article
Links
