I have haproxy in http mode sitting in front of a java application server. The application uses TLS with client auth, with certificates being forwarded to the application as a HTTP header. For some very small number of requests (e.g. 15 out of 80k), the header with the client certificate is not added (or not processed) by the server. When this happens, the timing of the requests suggest that HTTP 1.1 pipelining is in use (or I’m not reading the logs correctly).
I have the following log sequence for a a failed request (the last request with the 401 status indicates that the client certificate header is missing in the request):
May 19 16:25:46 test haproxy[1796]: 10.23.1.173:60851 [19/May/2023:16:25:46.435] HTTPS~ test_Nodes/test_srv 49/0/4/7/61 200 617 - - ---- 5/5/0/1/0 0/0 {0,“/CN=user1”} “POST /test/test/routing/FlowHeartbeatAction.do HTTP/1.1”
May 19 16:25:58 test haproxy[1796]: 10.23.1.173:60851 [19/May/2023:16:25:46.485] HTTPS~ test_Nodes/test_srv 12331/0/11/6/12348 200 1055 - - ---- 1/1/1/2/0 0/0 {0,“/CN=user1”} “POST /test/core/dashboard/SystemStatus.do HTTP/1.1”
May 19 16:25:58 test haproxy[1796]: 10.23.1.173:60851 [19/May/2023:16:25:58.816] HTTPS~ test_Nodes/test_srv 0/0/11/16/27 200 1050 - - ---- 1/1/0/1/0 0/0 {0,“/CN=user1”} “POST /test/core/alert/ComputeSummaryAlertAction.do HTTP/1.1”
May 19 16:26:14 test haproxy[1796]: 10.23.1.173:60851 [19/May/2023:16:25:58.816] HTTPS~ test_Nodes/test_srv 15479/0/11/11/15502 401 5471 - - ---- 1/1/0/1/0 0/0 {0,“/CN=user1”} “POST /test/test/routing/InjectMessageRun.do HTTP/1.1”
The IP and port combination (10.23.1.173:60851) shows that http keep-alive is active. Moreover, the the last two requests have the same time [19/May/2023:16:25:58.816] - not sure how that is possible (sometimes I see a millisecond difference).
I have the following timeout settings:
defaults
mode http
log global
option httplog
option dontlognull
option http-keep-alive
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 5s
timeout queue 10s
timeout connect 5s
timeout client 30s
timeout server 5m
timeout http-keep-alive 1s
timeout check 5s
timeout client-fin 1s
timeout server-fin 1s
timeout tunnel 5m
log-format “%ci:%cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs {%[ssl_c_verify],%{+Q}[ssl_c_s_dn]} %{+Q}r”
The header with the certificate is added in the backend section using
http-request set-header X-ClientCert %{+Q}[ssl_c_der,base64]
Has anyone got any idea what is going on? I don’t understand how the last two requests have the same time and the same IP:port combination nor why the last request does not have the right http header set.