Enterprise modules

Traffic mirroring

On this page

Available since

  • HAProxy Enterprise 1.9r1

HAProxy Enterprise can mirror traffic to a different environment, even one on a different network. Traffic mirroring, also called traffic shadowing, can be useful for copying live production traffic to another environment for such purposes as:

  • QA
  • Staging
  • Auditing
  • Network analytics
  • Security applications (such as IDS)

Traffic mirroring has almost no impact on clients because the load balancer does not wait for a response from the mirrored environment. The mirroring process is basically “fire and forget”, where requests are copied to the mirrored environment and forgotten. Only HTTP traffic can be mirrored.

Architecture Jump to heading

A mirroring deployment consists of these components:

  • The server where the load balancer resides.
  • The Stream Processing Offload Engine (SPOE) mirroring engine, also running on the load balancer server.
  • The production web servers.
  • The mirror server that receives the copied traffic sent by the mirroring engine. This is the server running the mirrored services described previously, such as QA, auditing, or analytics.

Architecture of Sample Mirroring Deployment

The data flow occurs as follows:

  1. The user client sends a request to the load balancer frontend.
  2. The frontend sends the request to the regular (production) backends and mirror backends.
  3. The regular backend processes the request normally, sending the request to the production web servers.
  4. The mirror backend copies the request, sending it to the SPOE mirror engine.
  5. The SPOE mirror engine sends the request to the mirror server in the secondary environment used for testing, auditing, or other purposes.

Configure traffic mirroring Jump to heading

To configure mirroring of traffic:

  1. Install the required packages on the HAProxy Enterprise node:

    nix
    sudo apt-get install hapee-extras-spoa-mirror
    nix
    sudo apt-get install hapee-extras-spoa-mirror
    nix
    sudo yum install hapee-extras-spoa-mirror
    nix
    sudo yum install hapee-extras-spoa-mirror
    nix
    sudo zypper install hapee-extras-spoa-mirror
    nix
    sudo zypper install hapee-extras-spoa-mirror
    nix
    sudo pkg install hapee-extras-spoa-mirror
    nix
    sudo pkg install hapee-extras-spoa-mirror

  2. Configure HAProxy Enterprise to send traffic to the agent. Add a filter spoe directive to your frontend, as shown:

    haproxy
    # Production frontend
    frontend fe_main
      mode http
      bind :80
      option http-buffer-request
      default_backend be_servers
      filter spoe  engine mirror  config /etc/hapee-extras/hapee-mirror-spoe.cfg
    haproxy
    # Production frontend
    frontend fe_main
      mode http
      bind :80
      option http-buffer-request
      default_backend be_servers
      filter spoe  engine mirror  config /etc/hapee-extras/hapee-mirror-spoe.cfg

    This directive specifies the mirror engine name and the mirror configuration file name. We cover these items in a later section.

  3. In addition to the backend that specifies your production servers, add a backend that specifies the server and port of the SPOE mirror agent. Here’s an example:

    haproxy
    # Mirror agents
    backend mirroragents
      mode tcp
      balance roundrobin
      timeout connect 5s
      timeout server 10s
      option spop-check
      server spoe-mirror-agent1 127.0.0.1:12345 check
    haproxy
    # Mirror agents
    backend mirroragents
      mode tcp
      balance roundrobin
      timeout connect 5s
      timeout server 10s
      option spop-check
      server spoe-mirror-agent1 127.0.0.1:12345 check

    This example also uses option spop-check, which enables health checking of SPOE agents via the SPOE protocol.

  4. Edit the mirror engine configuration file, /etc/hapee-extras/hapee-mirror-spoe.cfg, specified by the filter spoe directive in the HAProxy Enterprise configuration file.

    Add the following content to the mirror engine configuration file:

    hapee-mirror-spoe.cfg
    ini
    [mirror]
    spoe-agent mirror
      log global
      messages mirror
      use-backend mirroragents
      timeout hello 500ms
      timeout idle 5s
      timeout processing 5s
    spoe-message mirror
      args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
      event on-frontend-http-request
    hapee-mirror-spoe.cfg
    ini
    [mirror]
    spoe-agent mirror
      log global
      messages mirror
      use-backend mirroragents
      timeout hello 500ms
      timeout idle 5s
      timeout processing 5s
    spoe-message mirror
      args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
      event on-frontend-http-request

    This file configures how HAProxy Enterprise communicates with the SPOE mirror agent.

  5. The agent mirrors data to http://localhost:10100/ by default. To change this URI, edit the configuration file:

    • On Debian/Ubuntu, /etc/default/hapee-extras-spoa-mirror
    • On Alma/Oracle/Redhat/Rocky, /etc/sysconfig/hapee-extras-spoa-mirror

    Modify the -u option in the MIRROR_OPTIONS environment variable to send traffic to a new URL. You can specify a domain or an IP address. The port is optional. You may define only one destination URL. Examples:

    hapee-extras-spoa-mirror
    text
    # Set destination URL to an FQDN without port
    MIRROR_OPTIONS="-D -r0 -uhttp://mirror.mysite.com/"
    # Set destination URL to an IP address with port
    MIRROR_OPTIONS="-D -r0 -uhttp://192.168.41.10:10100"
    # Set destination URL to localhost with port
    MIRROR_OPTIONS="-D -r0 -uhttp://localhost:8100/"
    hapee-extras-spoa-mirror
    text
    # Set destination URL to an FQDN without port
    MIRROR_OPTIONS="-D -r0 -uhttp://mirror.mysite.com/"
    # Set destination URL to an IP address with port
    MIRROR_OPTIONS="-D -r0 -uhttp://192.168.41.10:10100"
    # Set destination URL to localhost with port
    MIRROR_OPTIONS="-D -r0 -uhttp://localhost:8100/"

  6. Enable the mirror agent:

    nix
    sudo systemctl enable hapee-extras-spoa-mirror
    nix
    sudo systemctl enable hapee-extras-spoa-mirror

  7. Restart the mirror agent and HAProxy Enterprise:

    nix
    sudo systemctl restart hapee-extras-spoa-mirror
    sudo systemctl restart hapee-2.8-lb
    nix
    sudo systemctl restart hapee-extras-spoa-mirror
    sudo systemctl restart hapee-2.8-lb

Logging Jump to heading

Mirrored requests are logged to the file /var/log/hapee-2.8r1/lb-access-<date>.log by default.

An example log statement is shown below:

lb-access-20231205.log
text
Aug 25 17:48:36 node1 hapee-lb[215242]: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=707 st=0 0/13/8/0/22 1/1 0/0 0/1
lb-access-20231205.log
text
Aug 25 17:48:36 node1 hapee-lb[215242]: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=707 st=0 0/13/8/0/22 1/1 0/0 0/1

An st (status code) value of 0 indicates success.

Tune mirrored traffic Jump to heading

There are several ways to tune the traffic that is mirrored.

  • Filtering
  • Sampling
  • Mapping key-value pairs
  • Making runtime changes using the Data Plane API

Filtering Jump to heading

You can add an ACL that limits the requests that are captured. For instance, if you only want to mirror traffic for requests to the /search feature on your site, you would ignore all requests except those that have a URL path beginning with /search, as shown:

hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  event on-frontend-http-request if { path_beg /search }
hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  event on-frontend-http-request if { path_beg /search }

You can also define named ACLs that do the same thing:

hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  acl is_search path_beg /search
  event on-frontend-http-request if is_search
hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  acl is_search path_beg /search
  event on-frontend-http-request if is_search

Sampling Jump to heading

Suppose you don’t want to capture all traffic but rather only a portion of it. You would add an ACL that collects a random sample of requests. In the next example, we generate a random number between 1 and 100 and only mirror the request if that number is less than or equal to 10:

hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  acl is_search path_beg /search
  event on-frontend-http-request if { rand(100) le 10 }
hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  acl is_search path_beg /search
  event on-frontend-http-request if { rand(100) le 10 }

Mapping key-value pairs Jump to heading

Your ACL statements can also check values from map files. For example, you can switch mirroring on or off by using a map file that contains a key-value pair like mirroring on. Then, check the map file from your hapee-mirror-spoe.cfg file like this:

hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  acl mirroring_on str(mirroring),map(/etc/hapee-2.8/mirroring.map) -m str on
  event on-frontend-http-request if mirroring_on
hapee-mirror-spoe.cfg
ini
spoe-message mirror-msg
  args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
  acl mirroring_on str(mirroring),map(/etc/hapee-2.8/mirroring.map) -m str on
  event on-frontend-http-request if mirroring_on

Use the HAProxy Enterprise Runtime API to change the value in the map file to off.

nix
# Change mirroring to off
echo "set map /etc/hapee-2.8/mirroring.map mirroring off" | nc 127.0.0.1 9999
# Show current value
echo "show map /etc/hapee-2.8/mirroring.map mirroring" | nc 127.0.0.1 9999
nix
# Change mirroring to off
echo "set map /etc/hapee-2.8/mirroring.map mirroring off" | nc 127.0.0.1 9999
# Show current value
echo "show map /etc/hapee-2.8/mirroring.map mirroring" | nc 127.0.0.1 9999

Make changes using the Data Plane API Jump to heading

You can also use the Data Plane API to add or remove filter spoe lines from the HAProxy Enterprise configuration file dynamically. In the following example, we show the existing filters, then add a new one, and then remove it:

Show existing filters:

nix
curl -X GET \
  --user admin:mypassword \
  "http://localhost:5555/v1/services/haproxy/configuration/filters?parent_name=fe_main&parent_type=frontend"
nix
curl -X GET \
  --user admin:mypassword \
  "http://localhost:5555/v1/services/haproxy/configuration/filters?parent_name=fe_main&parent_type=frontend"

Add a filter line:

nix
curl -X POST \
  --user admin:mypassword \
  -H "Content-Type: application/json" \
  -d '{"id": 0, "spoe_config":"/etc/hapee-2.8/spoa.conf", "spoe_engine":"mirror", "type":"spoe"}' \
  "http://localhost:5555/v1/services/haproxy/configuration/filters?parent_name=fe_main&parent_type=frontend&version=1"
nix
curl -X POST \
  --user admin:mypassword \
  -H "Content-Type: application/json" \
  -d '{"id": 0, "spoe_config":"/etc/hapee-2.8/spoa.conf", "spoe_engine":"mirror", "type":"spoe"}' \
  "http://localhost:5555/v1/services/haproxy/configuration/filters?parent_name=fe_main&parent_type=frontend&version=1"
output
text
{"id":0,"spoe_config":"/etc/hapee-2.8/spoa.conf","spoe_engine":"mirror","type":"spoe"}
output
text
{"id":0,"spoe_config":"/etc/hapee-2.8/spoa.conf","spoe_engine":"mirror","type":"spoe"}

Remove a filter line:

nix
curl -X DELETE \
  --user admin:mypassword \
  -H "Content-Type: application/json" \
  "http://localhost:5555/v1/services/haproxy/configuration/filters/0?parent_name=fe_main&parent_type=frontend&version=2"
nix
curl -X DELETE \
  --user admin:mypassword \
  -H "Content-Type: application/json" \
  "http://localhost:5555/v1/services/haproxy/configuration/filters/0?parent_name=fe_main&parent_type=frontend&version=2"

Tip

Here are a few ways to get the most out of traffic mirroring.

  • After setting up monitoring, compare the errors you get from your production servers with those you get from the new version to which you’re mirroring traffic. Having a monitoring strategy in place will be key to validating a release.
  • Make sure the feature you’re testing has URL paths and parameters that match the existing feature so that it is forward compatible with mirrored traffic. Forward compatibility may be a valuable test in and of itself.

Troubleshooting Jump to heading

Traffic mirroring in HAProxy Enterprise allows you to replicate network traffic to a separate destination for monitoring and analysis purposes. Here are some suggestions to help you diagnose and resolve common problems with traffic mirroring functionality.

Confirm the HAProxy Enterprise version Jump to heading

Make sure you are using a version of HAProxy Enterprise that supports traffic mirroring. The mirroring feature was introduced in version 1.9.0, so if you’re using an older version, consider upgrading to a compatible release.

nix
/opt/hapee-2.8/sbin/hapee-lb -v
nix
/opt/hapee-2.8/sbin/hapee-lb -v
output
text
HAProxy version 2.8.0-1.0.0-310.374 2023/11/20 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: https://www.haproxy.com/documentation/hapee/2-8r1/onepage/changelog/#2.8.0
Running on: Linux 5.10.0-22-amd64 #1 SMP Debian 5.10.178-3 (2023-04-22) x86_64
output
text
HAProxy version 2.8.0-1.0.0-310.374 2023/11/20 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: https://www.haproxy.com/documentation/hapee/2-8r1/onepage/changelog/#2.8.0
Running on: Linux 5.10.0-22-amd64 #1 SMP Debian 5.10.178-3 (2023-04-22) x86_64

Check network connectivity Jump to heading

Ensure that the destination IP address and port specified for mirroring are reachable from the HAProxy Enterprise server. Verify the network connectivity between the two systems using tools like ping or telnet. If there are any firewalls, security groups, or access control lists in place, ensure they allow traffic between HAProxy Enterprise and the mirroring destination.

Monitor resource utilization Jump to heading

Traffic mirroring can be resource-intensive, especially if the mirrored traffic volume is significant. Monitor the resource utilization of the HAProxy Enterprise, including CPU, memory, and network usage. Ensure that the server has enough capacity to handle the additional load caused by mirroring.

Check SPOE mirror agent activity status Jump to heading

Check if the agent is running on the system:

nix
ps aux | grep spoa | grep -v grep
nix
ps aux | grep spoa | grep -v grep
output
text
hapee-m+   63734  0.0  0.2 756464  5900 ? Sl 12:24 0:00 /opt/hapee-extras/bin/hapee-spoa-mirror 
 -D -r0 -uhttp://localhost:10100/ --logfile=a:/var/log/hapee-mirror.log -F /var/run/hapee-extras/hapee-spoa-mirror.pid
output
text
hapee-m+   63734  0.0  0.2 756464  5900 ? Sl 12:24 0:00 /opt/hapee-extras/bin/hapee-spoa-mirror 
 -D -r0 -uhttp://localhost:10100/ --logfile=a:/var/log/hapee-mirror.log -F /var/run/hapee-extras/hapee-spoa-mirror.pid

You can also check via the system status command:

nix
systemctl status hapee-extras-spoa-mirror
nix
systemctl status hapee-extras-spoa-mirror
output
text
● hapee-extras-spoa-mirror.service - LSB: HAPEE HTTP requests replicator
  Loaded: loaded (/etc/init.d/hapee-extras-spoa-mirror; generated)
  Active: active (running) since Thu 2023-06-01 19:48:27 UTC; 23h ago
    Docs: man:systemd-sysv-generator(8)
Main PID: 32783 (hapee-spoa-mirr)
    Tasks: 11 (limit: 9165)
  Memory: 2.9M
output
text
● hapee-extras-spoa-mirror.service - LSB: HAPEE HTTP requests replicator
  Loaded: loaded (/etc/init.d/hapee-extras-spoa-mirror; generated)
  Active: active (running) since Thu 2023-06-01 19:48:27 UTC; 23h ago
    Docs: man:systemd-sysv-generator(8)
Main PID: 32783 (hapee-spoa-mirr)
    Tasks: 11 (limit: 9165)
  Memory: 2.9M

If the process is in an error state, you can run journalctl -xe| grep -A3 -B3 -i spoa for more details.

Check the logs Jump to heading

  1. Search the /var/log/hapee-2.8/ logs for any issues. For example, st=0 means a successful response.

    lb-access-20231205.log
    ini
    May 31 16:59:51 ip-172-31-13-131 hapee-lb[713]:: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=52 st=0 0/0/0/0/0 1/1 0/0 1/25
    lb-access-20231205.log
    ini
    May 31 16:59:51 ip-172-31-13-131 hapee-lb[713]:: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=52 st=0 0/0/0/0/0 1/1 0/0 1/25

  2. Enable logging for the mirroring agent:

    nix
    sudo touch /var/log/hapee-mirror.log
    sudo chown hapee-mirror:hapee /var/log/hapee-mirror.log
    nix
    sudo touch /var/log/hapee-mirror.log
    sudo chown hapee-mirror:hapee /var/log/hapee-mirror.log

  3. Use your editor to add --logfile=a: to the startup script:

    • On Debian/Ubuntu, /etc/default/hapee-extras-spoa-mirror
    • On Alma/Oracle/Redhat/Rocky, /etc/sysconfig/hapee-extras-spoa-mirror

    This option determines the mode of logging, which allows opening and writing at end-of-file. If a capital letter is used for the mode, then line buffering is used when writing to the log file.

    hapee-extras-spoa-mirror
    ini
    MIRROR_OPTIONS="-D -r0 -uhttp://10.0.1.4:8080/ --logfile=A:/var/log/hapee-mirror.log"
    hapee-extras-spoa-mirror
    ini
    MIRROR_OPTIONS="-D -r0 -uhttp://10.0.1.4:8080/ --logfile=A:/var/log/hapee-mirror.log"

    The mirror agent will fail to start if mode is used without a defined log file.

Traffic mirroring reference Jump to heading

The SPOE mirror engine uses Stream Processing Offload Protocol (SPOP). The file /etc/hapee-extras/hapee-mirror-spoe.cfg configures how HAProxy Enterprise communicates with the SPOE mirror agent.

hapee-mirror-spoe.cfg
ini
[mirror]
spoe-agent mirror
    log global
    messages mirror
    use-backend mirroragents
    timeout hello 500ms
    timeout idle 5s
    timeout processing 5s
spoe-message mirror
    args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
    event on-frontend-http-request
hapee-mirror-spoe.cfg
ini
[mirror]
spoe-agent mirror
    log global
    messages mirror
    use-backend mirroragents
    timeout hello 500ms
    timeout idle 5s
    timeout processing 5s
spoe-message mirror
    args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
    event on-frontend-http-request

It supports the following directives in the spoe-agent section:

Directive Description
[*name*] The file begins with an engine name, mirror, in square brackets. As mentioned, this name must match the engine parameter value set on the filter spoe directive in the HAProxy Enterprise configuration.
log global This line means that events, such as when HAProxy Enterprise sends data, will be logged to the same output defined by the log statement in the global section of the HAProxy Enterprise configuration.
messages This line is a space-delimited list of labels that match up with spoe-message sections.
use-backend This line specifies which backend in the HAProxy Enterprise configuration holds the mirror agents.
timeout hello This setting limits how long HAProxy Enterprise will wait for an agent to acknowledge a connection.
timeout idle This setting limits how long HAProxy Enterprise will wait for an agent to close an idle connection.
timeout processing This setting limits how long an agent is allowed to process an event.

A spoe-message section defines which HAProxy Enterprise fetch methods will be used to capture data to send to the agents. The label here, mirror, is expected by this particular agent. For traffic mirroring, we capture the following:

  • the HTTP method
  • the URL path
  • the version of HTTP
  • all HTTP headers
  • the request body (note that this requires option http-buffer-request in the HAProxy Enterprise configuration)

Data is sent every time the on-frontend-http-request event fires, which is before the evaluation of http-request rules on the frontend side.

The options supported by hapee-spoa-mirror can be found using -h or --help:

nix
/opt/hapee-extras/bin/hapee-spoa-mirror -h
nix
/opt/hapee-extras/bin/hapee-spoa-mirror -h
output
text
Usage: hapee-spoa-mirror { -h --help }
  hapee-spoa-mirror { -V --version }
  hapee-spoa-mirror { -r --runtime=TIME } [OPTION]...
  Options are:
  -a, --address=NAME              Specify the address to listen on (default: "0.0.0.0").
  -B, --libev-backend=TYPE        Specify the libev backend type (default: AUTO).
  -b, --connection-backlog=VALUE  Specify the connection backlog size (default: 10).
  -c, --capability=NAME           Enable the support of the specified capability.
  -D, --daemonize                 Run this program as a daemon.
  -F, --pidfile=FILE              Specifies a file to write the process-id to.
  -h, --help                      Show this text.
  -i, --monitor-interval=TIME     Set the monitor interval (default: 5.00s).
  -l, --logfile=[MODE:]FILE       Log all messages to logfile (default: stdout/stderr).
  -m, --max-frame-size=VALUE      Specify the maximum frame size (default: 16384 bytes).
  -n, --num-workers=VALUE         Specify the number of workers (default: 10).
  -p, --port=VALUE                Specify the port to listen on (default: 12345).
  -r, --runtime=TIME              Run this program for the specified time (0 = unlimited).
  -t, --processing-delay=TIME     Set a delay to process a message (default: 0).
  -u, --mirror-url=URL            Specify the URL for the HTTP mirroring.
  -I, --mirror-interface=NAME     Specify the interface/address for outgoing connections.
  -P, --mirror-local-port=VALUE   Specify the local port range for outgoing connections.
  -V, --version                   Show program version.
output
text
Usage: hapee-spoa-mirror { -h --help }
  hapee-spoa-mirror { -V --version }
  hapee-spoa-mirror { -r --runtime=TIME } [OPTION]...
  Options are:
  -a, --address=NAME              Specify the address to listen on (default: "0.0.0.0").
  -B, --libev-backend=TYPE        Specify the libev backend type (default: AUTO).
  -b, --connection-backlog=VALUE  Specify the connection backlog size (default: 10).
  -c, --capability=NAME           Enable the support of the specified capability.
  -D, --daemonize                 Run this program as a daemon.
  -F, --pidfile=FILE              Specifies a file to write the process-id to.
  -h, --help                      Show this text.
  -i, --monitor-interval=TIME     Set the monitor interval (default: 5.00s).
  -l, --logfile=[MODE:]FILE       Log all messages to logfile (default: stdout/stderr).
  -m, --max-frame-size=VALUE      Specify the maximum frame size (default: 16384 bytes).
  -n, --num-workers=VALUE         Specify the number of workers (default: 10).
  -p, --port=VALUE                Specify the port to listen on (default: 12345).
  -r, --runtime=TIME              Run this program for the specified time (0 = unlimited).
  -t, --processing-delay=TIME     Set a delay to process a message (default: 0).
  -u, --mirror-url=URL            Specify the URL for the HTTP mirroring.
  -I, --mirror-interface=NAME     Specify the interface/address for outgoing connections.
  -P, --mirror-local-port=VALUE   Specify the local port range for outgoing connections.
  -V, --version                   Show program version.

Supported libev backends: select, poll, epoll, linuxaio, iouring.

Supported capabilities: fragmentation, pipelining, async.

Allowed logging file opening modes: a, w. The a mode allows opening or creating file for writing at end-of-file. The w mode allows truncating the file to zero length or creating a new file. If a capital letter is used for the mode, then line buffering is used when writing to the log file.

The time delay/interval is specified in milliseconds by default, but can be in any other unit if the number is suffixed by a unit (us, ms, s, m, h, d).

Traffic mirroring log reference Jump to heading

SPOE mirror agent activity is logged using HAProxy Enterprise’s logger. Mirrored requests are logged to the file /var/log/hapee-2.8/lb-access-<date>.log by default. A message is emitted for each mirrored request. Depending on the status code, the log level will be different. In the normal case, when no error occurred, the message is logged with the level LOG_NOTICE. If an error occurred, the message is logged with the level LOG_WARNING.

Consider the following example log message for a mirrored request:

An example log statement is shown below:

lb-access-20231205.log
text
Aug 25 17:48:36 node1 hapee-lb[215242]: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=707 st=0 0/13/8/0/22 1/1 0/0 0/1
lb-access-20231205.log
text
Aug 25 17:48:36 node1 hapee-lb[215242]: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=707 st=0 0/13/8/0/22 1/1 0/0 0/1

In this example, the mirror agent logged an event named on-frontend-http-request with a stream-id of 707. Its status code of 0 indicates it was successful. One event was processed and had zero errors.

Mirror agent log messages follow this format:

lb-access-20231205.log
text
SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUS-CODE reqT/qT/wT/resT/pT <idles>/<applets> <nb_sending>/<nb_waiting> <nb_error>/<nb_processed>
lb-access-20231205.log
text
SPOE: [AGENT] <TYPE:NAME> sid=STREAM-ID st=STATUS-CODE reqT/qT/wT/resT/pT <idles>/<applets> <nb_sending>/<nb_waiting> <nb_error>/<nb_processed>
Log message item Description
AGENT The agent name. It is mirror for the mirror agent.
TYPE For mirrored requests this is EVENT.
NAME The event name.
STREAM-ID The unique integer id of the stream.
STATUS_CODE The request’s status code. A status code of 0 indicates success. Other status codes include: 1: I/O error; 2: A timeout occurred; 3: Frame is too big; 4: Invalid frame received; 5: Version value not found; 6: max-frame-size value not found; 7: Capabilities value not found; 8: Unsupported version; 9: max-frame-size too big or too small; 10: Payload fragmentation is not supported; 11: Invalid interlaced frames; 12: frame-id not found (it does not match any referenced frame); 13: Resource allocation error; 99: An unknown error occurred
reqT/qT/wT/resT/pT These represent the following time events: reqT : The encoding time. It includes ACLs processing time, if applicable. For fragmented frames, it is the sum of all fragments. qT: The delay before the request leaves the sending queue. For fragmented frames, it is the sum of all fragments. wT: The delay before the response is received. Fragmentation is not supported. resT: The delay in processing the response. Fragmentation is not supported. pT: The delay in processing the event; it is the latency added by the SPOE processing. It is more or less the sum of the other values. For all of these time events, -1 means the process was interrupted. For example, -1 for the queue time means the request never left the queue. Note that for fragmented frames it is harder to know when an interruption occurred.
<idle> The number of idle SPOE applets.
<applets> The number of SPOE applets.
<nb_sending> The number of streams waiting to send data.
<nb_waiting> The number of streams waiting for an ack.
<nb_error> The number of processing errors.
<nb_processed> The number of events processed.

Do you have any suggestions on how we can improve the content of this page?

Previous page SNMP Next page Integrations
© 2024 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences