Extend HAProxy With Stream Processing Offload Engine (SPOE)

The HAProxy Stream Processing Offload Engine filter enables you to extend HAProxy in any language without modifying its core codebase.

Imagine you’re living in the distant future and you’re stomping around town in the latest 15-foot tall mech robot. You’ve outfitted it in the latest gadgetry. It’s fast, agile and responsive. When the new arm cannon upgrade becomes available next month, you’ll be able to augment your robot by simply heading to the nearest shop and having the cannons snapped into place.

Although this is straight out of a sci-fi movie, there are some clear parallels to the work we’ve been doing at HAProxy Technologies. HAProxy is the world’s fastest and most widely used software load balancer. Its popularity is due to its high performance, reliability, and flexible configuration language. Now, we want to provide the ability to extend it using any programming language without the complexity of modifying its core codebase. Want to add traffic shadowing capabilities? Or maybe you’d like to route real-time data to a Machine Learning system that can perform some advanced analysis?

In order to expand the possibilities of what HAProxy can do, the Stream Processing Offload Engine (SPOE) was invented. It provides a way to send live traffic to external programs that can process the data out-of-band, not unlike snapping an upgrade onto your mech robot. In this blog post, you’ll be taken on a tour of the SPOE and, we hope, become inspired by your own futuristic possibilities.

A Bit of History

The SPOE has been increasingly refined over the past few versions. It was added as a feature in HAProxy 1.7. It matured in HAProxy 1.8 and 1.9. During upcoming versions, you’ll see it further improved and enriched.

In HAProxy 1.7, the concept of filters was introduced. Filters provide hooks into specific phases of the load balancer’s traffic processing. They allow HAProxy to be extended with code that can intercept real-time, streaming data at various points during the request/response lifecycle. Filters are written using the C programming language and compiled into HAProxy.

In version 1.7, HAProxy Technologies put it to use by rewriting the HTTP compression logic as a filter. In 1.8, an HTTP cache for small objects was added. Its storage is built on the stream processing capabilities of a filter. During that version, we also created a new kind of filter called an SPOE filter. The idea was to make extending HAProxy easier and more accessible, with the specific aim of allowing you to do so with any programming language. An SPOE filter forwards streaming data to an external program.

There are plenty of reasons why you might want to stream load balancer data to an external program. You might not want to get involved in developing HAProxy’s own source code. Or, you might do it to take advantage of a particular programming language’s libraries and features. Or, you may need to develop a proprietary extension that you’re unable to open source.

In the next few sections, we’ll introduce you to the SPOE filter, explain its architecture, and give you a high-level overview of how it works.

Stream Processing Setup

To understand how an SPOE filter works, you should become familiar with three terms:

• Stream Processing Offload Engine (SPOE)

• Stream Processing Offload Agent (SPOA)

• Stream Processing Offload Protocol (SPOP)

The Stream Processing Offload Engine is a type of filter that gives you access to real-time data from HAProxy. It allows work to be offloaded to self-hosted components. A component that does the work is called a Stream Processing Offload Agent. It receives data from an SPOE filter. Data is exchanged between the filter and the agent via a new, binary protocol over TCP called the Stream Processing Offload Protocol.

The following diagram shows how an SPOE interacts with an SPOA via the SPOP. For the rest of this article, I’ll use these acronyms sparingly and stick to the terms: engine, agent and protocol.

HAProxy configuration

Creating an engine begins with adding a filter directive to your HAProxy configuration file. This turns the stream processing on for a particular proxy (e.g. frontend or listen) or backend section. You can give the filter a name to differentiate it from other engines, although this is not mandatory.

Giving it a name allows you to configure several engines that points to the same SPOE configuration file. Otherwise, the configuration file will be dedicated to a single engine. The filter directive links to a separate SPOE config file that specifies how your engine will communicate with the agents and which load balancer events to hook into. You’ll also need to add a backend section containing the agent servers.

Here’s an example that uses a filter directive to stream data to a fictitious service that performs IP reputation scoring on every new client connection.

In this example, the ip-reputation engine will watch traffic coming into the myproxy frontend. The iprep.conf file configures where to send the data. Ultimately, it’s forwarded to an agent in the agents backend, which does a calculation and sets a variable named ip_score. The variable is then used in a tcp-request content reject rule to deny clients that have an IP reputation score lower than 20.

HAProxy attaches the prefix iprep to variables that the agents set. Prefixes are used to avoid name clashes between engines and are defined in the SPOE configuration file. Currently, a large part of an agent’s power comes from its ability to feed data back to HAProxy through the use of variables.

SPOE configuration

Here’s how iprep.conf looks:

The full SPOE configuration syntax is documented within the HAProxy git repository, but let’s go over the highlights. At the top of the file is a scope enclosed in square brackets. A scope lets you tell one engine from another when they’re both configured in the same file. It matches the filter engine name that you set in the HAProxy configuration file. Each named engine gets its own scope like this. If you didn’t assign a name, then you wouldn’t need the scope declaration.

The spoe-agent line begins a part of the file that defines how the engine communicates with its agents. The timeout lines indicate how long HAProxy will wait for an agent to respond during the initial handshake when the connection has been idle, and when processing a message. The use-backend directive tells the engine which backend in the HAProxy configuration contains the agents. That’s where the stream of events will be sent.

HAProxy does not send all of the available information about a connection or request to an agent. In many cases, that would lead to information overload. Instead, you have to decide which pieces of data are important to you. The spoe-agent section sets a list of space-delimited message types via a messages directive. The message types are defined in separate spoe-message sections. Inside each, an args line lists the message’s arguments, which map back to HAProxy fetch methods (e.g. src). Multiple arguments are separated by spaces. It’s okay to pass arguments that don’t have a key (e.g. src instead of ip=src).

The event line tells the engine when to intercept data and send a message. The following events are used to trigger sending messages:

Event	Meaning
on-client-session	Triggered when a new client session is created. This event is only available for SPOE filters declared in a frontend or a listen section.
on-frontend-tcp-request	Triggered just before the evaluation of tcp-request content rules in a frontend. This event is only available for SPOE filters declared in a frontend or a listen section.
on-backend-tcp-request	Triggered just before the evaluation of tcp-request content rules in a backend. This event is skipped for SPOE filters declared in a listen section.
on-frontend-http-request	Triggered just before the evaluation of http-request rules in a frontend. This event is only available for SPOE filters declared in a frontend or a listen section.
on-backend-http-request	Triggered just before the evaluation of http-request rules in a backend. This event is skipped for SPOE filters declared in a listen section.
on-server-sessions	Triggered when a session with the server is established.
on-tcp-response	Triggered just before the evaluation of tcp-response content rules.
on-http-response	Triggered just before the evaluation of http-response rules.

 You can, as in this example, append an ACL if statement to control when to, or when not to, invoke sending the message. Here, we’ve whitelisted some IP addresses for which we don’t want to run the IP reputation service. Without an ACL, a message is sent every time the event fires.

After an agent processes a message, it can send back a list of actions that HAProxy should apply. Currently, you can tell HAProxy to set or unset a variable. Future versions will add many other kinds of actions.

Agent code

A Stream Processing Offload Agent can be written in any language and would listen on a TCP port and speak to HAProxy using the Stream Processing Offload Protocol. The provided example IP reputation service is written in C and randomly assigns a score to each IP address and returns a variable named ip_score. We won’t go into detail about the code. Instead, we’ll give a high-level overview of the protocol. However, you can check out the following libraries too:

• C

• .NET Core

• Golang (another exists here from Andrew Privalov)

• Lua

• Python

The Stream Processing Offload Protocol

The communication between HAProxy and the agents happens over a binary protocol called the Stream Processing Offload Protocol (SPOP). You can read the official documentation on the HAProxy project’s website.

The basic unit of the protocol is a frame. Frames package up data, with each frame being a certain type and serving a particular purpose. Frames are prefixed with four bytes that tell the total frame length. That’s followed by a single byte that identifies the frame type. Next, there is some metadata, which contains flags, a stream ID, and a frame ID. Then comes the frame’s payload.

Here is the list of currently supported frame types:

Type (ID)	Description
0 – UNSET	Used for all frames except the first when a payload is fragmented.
1 – HAPROXY-HELLO	Sent by HAProxy when it opens a connection to an agent.
2 – HAPROXY-DISCONNECT	Sent by HAProxy when it wants to close the connection.
3 – NOTIFY	Sent by HAProxy to pass information to an agent.
101 – AGENT-HELLO	Reply to a HAPROXY-HELLO frame, when the connection is established.
102 – AGENT-DISCONNECT	Sent by an agent just before closing the connection.
103 – ACK	Sent to acknowledge a NOTIFY frame.

 

A connection between HAProxy and an agent is established when HAProxy sends a HAPROXY-HELLO frame and the agent responds with an AGENT-HELLO. During this handshake, the protocol version to use and the parameters of the transaction (e.g. maximum frame size and supported capabilities) are communicated within the payload. At this stage, the stream ID and frame ID should both be set to zero.

If for any reason, an incompatibility is detected during the handshake, HAProxy will send a HAPROXY-DISCONNECT frame. This frame contains a status code that the agent can use to diagnose what went wrong. The agent is responsible for then replying with an AGENT-DISCONNECT frame and closing the connection. It can also initiate a disconnect from its end. In that case, it simply closes the connection and doesn’t need to wait for a response from HAProxy.

If both sides agree to proceed, the connection is considered established and ready to use. At that point, HAProxy can begin sending NOTIFY frames that contain SPOE messages, as set up in the SPOE configuration’s messages list. These frames have stream IDs and frame IDs. When the agent code parses the frames, it looks for messages with names that it knows how to process (e.g. check-client-ip). It acknowledges each frame with an ACK frame, which includes a matching stream ID. The payload of an ACK may list actions for HAProxy to perform.

Frames cannot exceed the maximum size that was negotiated during the initial handshake. However, it is possible to divvy large payloads up as fragments. This capability is announced during the handshake. A fragmented message is sent as frames that have the same stream ID and frame ID. Its last frame adds a FIN flag.

The payload of a NOTIFY frame is perhaps the most interesting because it contains the real-time traffic information that you’re streaming to the agent. For each spoe-message section that you’ve added to your SPOE configuration file, a message in a NOTIFY frame will be sent when the corresponding event is triggered. Also, if you’ve attached multiple types of messages to the same event, they’ll all be sent.

For example, in iprep.conf, we had defined a message named check-client-ip:

When the on-client-session event is triggered, which happens when a client first connects to HAProxy (and, in this case, if the client is not whitelisted), then HAProxy sends a NOTIFY frame that contains the message’s name and arguments in its payload. So, in this example, the name check-client-ip along with the client’s source IP, which is recorded via the src fetch method, are included in the message.

Actions

After an agent has finished processing the data that HAProxy has sent to it, you have the option of instructing the load balancer to perform an action. At this time, you’re able to either set or unset a variable in HAProxy. You can then add ACL statements to your HAProxy configuration file that check the variable and later perform an operation, such as choosing a backend or denying the request.

Our fictitious IP reputation service checks the client’s IP address and, based on that, sets a variable that contains their score. We then use that variable to decide whether to allow or deny the connection. Actions are included in ACK frames. The payload of an ACK can contain a list of actions, such as set-var. When using set-var and unset-var, you’ll assign a scope (e.g. session) to the variable to control the phase of the request/response lifecycle in which the variable will live.

In our previous example, the HAProxy configuration reads the sess.iprep.ip_score variable that was set by the agent.

If its value is less than 20, the connection is rejected. You can also log variables, like this:

Then, the variable will be written to the access log. You’ll find it between curly braces, as in this example where the IP reputation score is 92:

Or you might set them as response headers and view them in your browser.

SPOP Connections Management

HAProxy can reuse existing, open connections to an agent. By default, once the connection is established, it is locked during the time of a NOTIFY / ACK exchange. So, only one stream uses it at any particular time. However, it’s marked as idle once the ACK frame is received. Then, other streams can use it.

This functionality is similar to how HTTP Keep-Alive works when HAProxy connects to a backend server. It’s the easiest way to handle reusing an open connection, but there’s a way to get better performance: During the protocol handshake, support for pipelining can be negotiated. Pipelining is the ability for a peer to decouple NOTIFY and ACK frames. Or, in other words, many streams will be able to share the same connection simultaneously. It works much like HTTP/2 multiplexing.

It’s also possible to enable asynchronous mode. In this state, any live connection can be used to transport frames for the same message. So, a NOTIFY frame could go out on one connection and the ACK could come back over a different connection, which offers performance benefits. See the official documentation for more information.

Conclusion

In this blog post, you learned about the Stream Processing Offload Engine, its protocol, the Stream Processing Offload Protocol, and the Stream Processing Offload Agents that receive and process messages from HAProxy. The agents can also instruct HAProxy on actions to perform. With these tools, you’re able to extend HAProxy using any programming language by hooking into events that fire during a session. This allows you to stream real-time data to applications that augment your load balancer with extra functionality. In upcoming posts, we’ll dig into some examples of using an SPOE filter to add new functionality to HAProxy, including the implementation of traffic shadowing.

HAProxy Enterprise includes a robust and cutting-edge codebase, an enterprise suite of add-ons, expert support, and professional services. It even includes a single-sign-on module, which uses an SPOE filter, that authenticates your clients with an LDAP server. Contact us to learn more or get your HAProxy Enterprise free trial today!

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.