Forwarding TLS traffic to staging using spoa-mirror

Help!
1

Hi. I need to enable traffic mirroring from the production server to staging. We had this working in the past, but I’m unable to restore the setup, probably because of technical changes that happened in the meantime.

In the normal workflow, we do TLS termination in HAProxy and forward in plain HTTP to backend servers. I believe I’m not managing TLS correctly at SPOE level, but I cannot find what’s wrong.

I forward spoa-mirror traffic back in HAProxy again to alter the Host and Location headers, replacing production hosts with staging ones.

global
    master-worker
    mworker-max-reloads 3
    set-dumpable
    log 127.0.0.1   local0 info
    log 127.0.0.1   local1 notice
    maxconn 25000
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM
    ssl-default-bind-curves X25519:secp521r1:secp384r1:prime256v1
    tune.ssl.default-dh-param 2048

defaults
    log global
    mode        http
    option      httplog
    option      dontlognullmonitor probes)
    option      forwardfor
    option      log-separate-errors
    option      http-server-close
    retries     3
    timeout connect 20s
    timeout client 100s
    timeout server 100s
    timeout http-keep-alive 80s

frontend web
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    http-request del-header Proxy
    http-request add-header X-Forwarded-Proto https
    http-response add-header X-Frame-Options DENY
    filter spoe  engine mirror  config /etc/haproxy/mirror.cfg

program mirror
   command /usr/local/bin/spoa-mirror --runtime 0 --mirror-url "http://127.0.0.1:9000" --logfile A:/var/log/haproxy-mirror.log

# We receive back SPOA's traffic and alter headers
frontend spoa_frontend
    log global
    bind *:9000
    http-request replace-value Location ^my.domain.es(.*) staging.my.domain.es\1
    http-request del-header Proxy
    http-request add-header X-Forwarded-Proto http
    http-request add-header X-Is-Mirrored yes
    use_backend spoa_backend

# SPOA destination
backend mirroragents
    mode tcp
    balance roundrobin
    timeout connect 5s
    timeout server 1m
    server agent1 127.0.0.1:12345

# Backend server where the SPOA mirrored traffic is delivered
backend spoa_backend
    mode http
    http-request set-header Host staging.my.domain.es
    http-request replace-value Location ^https://my.domain.es(.*)$ http://staging.my.domain.es\1
    server stagingsrv stagingsrv:80 check inter 2000 rise 2 fall 3

And this is mirror.cfg

[mirror]
spoe-agent mirror
    log global
    messages mirror
    timeout hello 500ms
    timeout idle 10s
    timeout processing 100ms
    use-backend mirroragents

spoe-message mirror
    args arg_method=method arg_path=url arg_ver=req.ver arg_hdrs=req.hdrs_bin arg_body=req.body
    acl is_prod                hdr(host)  -m  str  my.domain.es
    event on-frontend-http-request if is_prod

This is the log of a request performed with curl:

==> haproxy-mirror.log <==
[ 7][ 9366.125010] "GET http://127.0.0.1:9000https://my.domain.es/ HTTP/???" 0 0/0 0.000

==> haproxy-http.log <==
Feb 28 14:29:26 localhost haproxy[376208]: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=40796 st=0 0/1/0/0/2 1/1 0/0 0/10
Feb 28 14:29:26 localhost haproxy[376208]: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=40796 st=0 0/1/0/0/2 1/1 0/0 0/10
Feb 28 14:29:27 localhost haproxy[376208]: 15.24.10.11:60855 [28/Feb/2024:14:29:26.783] web~ backend_http_cluster/spain-01 2/0/1/549/567 200 14425 - - --NI 1/1/0/0/0 0/0 "GET https://my.domain.es/ HTTP/2.0"

I’m puzzled by the HTTP/??? output of spoa-mirror, that I found in curl.c for CURL_HTTP_VERSION_NONE.

Anyway, in the logs we never get a record for the spoa_frontend, that means the mirror agent is never forwarding the request there. But I haven’t found a verbose option for spoa-mirror…

thanks