PacketShield provides stateful packet filtering and protects your network against DDoS.
Packet floods are DDoS attacks in which large numbers of packets are sent to a target with the goal of saturating the target’s resources. They can heavily impact the system’s networking stack and generate heavy CPU load due to the processing they induce. Floods can also raise memory usage and fill TCP session tables with illegitimate, forged packets. Add the extra processing generated on userland applications if the sessions can reach that level. All of this can crash a service or system even before the available bandwidth is exhausted.
PacketShield is patented software that provides stateful packet filtering and protects your network against DDoS. It blocks illegitimate packets before they need to be processed by the kernel. This allows services that PacketShield protects to stay fully operational under attack.
PacketShield was designed to protect the local networking stack of the machine where an HAProxy process is running, such as the ALOHA load balancer. Having PacketShield on an internet-facing load balancer not only protects the load balancer but also all of the backend servers behind it. PacketShield has been demonstrated to outperform firewall software such as Netfilter when it comes to packet floods.
In this blog post, we’ll describe why having anti-DDoS software like PacketShield is important, the history of how it came to be, and how it works to protect you from attacks.
How Much Can It Stop?
PacketShield can filter packets at a line rate for a 10 Gigabit link. To understand the performance needs of packet filtering you have to know some details. On a 10 Gigabit link, you can reach 14.88 million packets per second. That means that to handle a flood on such a link you’d have to process each packet within 67 nanoseconds (1 / 14.88M).
Compare that with the latency of actual CPUs and RAM:
Access the L2 CPU cache | 3-4 ns |
Access the L3 CPU cache | 10-20 ns |
Access RAM | 65-85ns |
You’ll notice that the performance bottleneck to filtering on a 10 Gigabit link is not only the CPU frequency or the number of cores, but also the memory latency. For that reason, PacketShield is optimized to use the processor cache and avoid accessing memory. It’s been implemented as a kernel module using native code. It isn’t possible to implement it using recent technologies, such as eBPF, which doesn’t offer the flexibility of native code.
The aim is to filter packets at the lowest level—even lower than the system’s network stack mechanisms of Netfilter/conntrack or the kernel’s SYN cookies. This permits more efficient protection against DDoS attacks without impacting legitimate traffic. PacketShield processing is performed in the NIC’s driver code to be as close as possible to the hardware.
PacketShield has two layers of defense. It can process packets using stateless filtering, which is based on static rules and protocol sanitization. Or, it can do smart, stateful filtering, in which the state is maintained for either TCP or certain UDP protocols, such as DNS.
How It Works
To achieve its superior DDoS protection, PacketShield first uses stateless filtering to drop packets that are not RFC compliant or that present invalid checksums. It can also drop packets using static rules based on the source IP’s subnet, the network protocol, the destination address or port, or more advanced rules based on the IP’s geographic location.
After passing the stateless filters, stateful filtering handles any TCP floods (i.e. ACK, RST and SYN), as well as any DNS reflection attacks. At the driver level, it can generate responses to incoming SYN packets, completely bypassing the system’s network stack. It generates SYN cookies that it is able to match the ACK response to and allow the valid sessions to pass during SYN flood attacks. It also drops any packet that is not recognized as being part of a valid session.
To aid stateful filtering at this level, PacketShield builds its own image of the system’s session table. This table is filled by inspecting packets to learn established TCP sessions or to match DNS responses to issued DNS requests. The session table is optimized to allow for fast insertions and lookups. By doing this, PacketShield offers a flood filtering solution with absolutely no false-positive drops of valid session packets.
By analyzing the valid sessions, PacketShield learns normal values for TTL for a given subnet. It can then detect if a TTL value appears the suspect in a packet, drop this packet and keep a signature to recognize it in the future.
There’s also a configurable throttle to limit the number of ICMP requests passed to the kernel. A surge protection mechanism is available to limit the global number of packets submitted to the system, and a random drop of empty ACK packets is configurable to stop an ACK storm.
PacketShield Countermeasures
All of these countermeasures can either be enabled manually or activated automatically based on per second thresholds. Furthermore, each rule, threshold, and countermeasure can be configured per destination address. PacketShield is fully hot configurable via a sysfs interface (/sys/packetshield). Configuration changes don’t need a reload or iface reset and they are applied immediately. This is a great asset when you need to react and quickly apply changes during a heavy attack.
The PacketShield sysfs interface allows you to perform packet captures at line rate with no impact on traffic, even during heavy floods. In addition, it provides real-time statistics, showing counters that increase depending on how a packet has been qualified and processed by PacketShield. Those values can be monitored by an external tool to draw graphs or launch alerts.
How We Got Here
Although DDoS attacks are a looming threat to businesses, existing defenses could not stand up to large-scale packet floods. We resolved to tackle the problem. After all, both the HAProxy and ALOHA load balancers are utilized by some of the busiest websites in the world; We had often helped these organizations fight complex DDoS attacks, which gave us a unique perspective.
At HAProxy Technologies, we studied several solutions including NFqueue, DPDK, and the netmap framework. We began with NFqueue, but quickly abandoned it for performance reasons. It operates too high in the kernel networking stack. DPDK, on the other hand, could meet the performance requirements, but our goal was to protect services that run locally and DPDK doesn’t provide a way to forward packets to the local stack.
So, the first PacketShield prototype was done using the netmap framework. The netmap framework provides an interface for extracting a packet from a NIC and passing it to a userland application. After processing, the application can drop this packet, forward it to another NIC, or deliver it to the network stack.
However, NIC descriptors, which are representations of packets, have to be mapped to netmap descriptors and then to the kernel’s stack descriptors (skb). The same, in the opposite order, happens for outgoing packets. Switching between formats causes some metadata to be lost and prevents the use of some hardware optimizations of the NICs, such as Generic Receive Offload and hardware checksums.
Without those hardware optimizations, we only achieved forwarding 2.5 Gb/s of traffic through PacketShield based on netmap. We decided to improve this by patching netmap, adding metadata, and reimplementing those optimizations in the NIC driver. Doing this, we raised performance to 8 Gb/s, but we still hadn’t yet saturated a 10 Gigabit link.
We discovered a bottleneck. The netmap framework performs very well when filtering packets and forwarding them to another NIC. However, it uses the same kernel API as a TAP tunnel would to process and deliver packets to the local stack, which was the cause of the performance problems
Switching to NDIV
We decided to bypass the issues we encountered with netmap by implementing our own framework. Our idea was to patch drivers and add hooks to submit incoming packet descriptors to another registered kernel module, which could then choose to let it pass to the networking stack, drop it, or respond directly on the same NIC.
A hook was also added to inspect outgoing packets. Another hook was then added to aggregate counters and data among multiple CPUs or to process tasks per batch of packets, which allowed us to process each packet without locks. The Network Diverter framework (NDIV) had been born.
Version 2 of PacketShield relies on NDIV. This means it doesn’t suffer from switching between the userland and the kernel, and there’s no copying of data or bottleneck for legitimate traffic. The NDIV driver patches have evolved since kernel version 4.19 and now rely directly on the eXpress Data Path (XDP) framework code, which is very similar to NDIV when it comes to processing incoming packets.
NDIV currently supports the following Linux drivers:
mvneta
igb
e1000
e1000
ixgbe
i40e
mlx5
You’ll find the drivers and NDIV core patchsets on Willy Tarreau’s space on Kernel.org. Truly, any drivers supporting XDP could easily be enhanced to support NDIV. These days, we work to promote the benefits of using native code to process packets at a high rate, such as by presenting its advantages at the Netdev Conferences.
Getting Started with PacketShield
PacketShield is included with the ALOHA load balancer, so you can get DDoS protection integrated with a powerful, plug-and-play appliance. It is available as either rack-mounted hardware or as a virtual image that’s supported by multiple hypervisors.
Very often, people need simple solutions to complex problems. They need secure, reliable infrastructure with no hassle. The ALOHA load balancer takes the guesswork out of finding the right hardware specs or the right system tunings when setting up L4 and L7 load balancing. It’s as though we packaged all of our expert knowledge and best practices into a self-contained system and tied it neatly with a bow.
With the ALOHA load balancer, PacketShield can be enabled easily by using the built-in GUI or via the CLI. You’ll find everything you need in the ALOHA User Guide.
Conclusion
PacketShield is a unique solution, operating strictly in software, that allows stateful filtering of packets at line rate on a 10 Gigabit NIC with nearly no impact on legitimate traffic. It has been demonstrated to outperform other firewall software, due to its use of the NDIV framework. NDIV was developed by HAProxy Technologies when it became clear that existing packet filtering frameworks could not perform at the level required to stop today’s large-scale packet floods.
Stay tuned. We will be following up with news about how PacketShield can protect against even larger-scale floods in an upcoming blog post.
Want to stay in the loop on similar topics? Subscribe to this blog! You can also follow us on Twitter and join the conversation on Slack.
Contact us to learn more about ALOHA and PacketShield. You can also sign up to receive a free trial version of the ALOHA virtual appliance.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
