Hi,
I recently updated the version of haproxy from 2.0.25 to 2.0.31 to get rid of some vulnerability.
It seemed okay and nothing special after updating, but about one or two weeks later, one of haproxy process suddenly consumes almost 100% of cpu. this high cpu usage doesn`t stay at a specific cpu, it keeps moving to other cpu(got 48 core). and this status continues after i remove all the traffic.
I googled for a long time but got nowhere… so help me please (I want to solve this problem without version updating)
let me share some debugging information!
Current version information
HA-Proxy version 2.0.31-c8b1c15 2023/02/14 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -DMAX_SESS_STKCTR=20
OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_PTHREAD_PSHARED=1 USE_REGPARM=1 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_TFO=1 USE_SYSTEMD=1
Feature list : -51DEGREES +ACCEPT4 -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL +EPOLL -EVPORTS +FUTEX +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MY_ACCEPT4 -MY_EPOLL -MY_SPLICE +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +PCRE -PCRE2 -PCRE2_JIT +PCRE_JIT +POLL +PRCTL -PRIVATE_CACHE +PTHREAD_PSHARED +REGPARM +RT -SLZ +STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -VSYSCALL -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=48).
Built with OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
Running on OpenSSL version : OpenSSL 1.1.1t 7 Feb 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with PCRE version : 8.44 2020-02-12
Running on PCRE version : 8.44 2020-02-12
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
<default> : mode=HTX side=FE|BE mux=H1
<default> : mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
haproxy.cfg
####################################################################################################
global
log 127.0.0.1 local2 notice
pidfile /var/run/haproxy.pid
maxconn 500000
tune.bufsize 16384
user haproxy
group haproxy
daemon
unix-bind prefix /var/run/ mode 660 user haproxy group haproxy
spread-checks 5
tune.ssl.cachesize 400000
tune.ssl.lifetime 86000
tune.ssl.maxrecord 2859
tune.ssl.default-dh-param 2048
ssl-default-bind-options no-sslv3
# cipher suites
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
# for tls1.3
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
nbproc 1
nbthread 48
# http/1 header case adjustment
h1-case-adjust-file /etc/haproxy/h1-case-adjust-list.txt
# CPU
cpu-map 1/all 0-
stats socket /var/run/haproxy.stat1 process 1 level admin expose-fd listeners
server-state-file /etc/haproxy/server_state
####################################################################################################
defaults
mode http
log global
maxconn 500000
option httplog
option dontlognull
option log-separate-errors
option log-health-checks
option forwardfor except 127.0.0.1
#option http-server-close
option redispatch
option tcp-smart-accept
option tcp-smart-connect
option abortonclose
option splice-auto
option http-use-htx
option h1-case-adjust-bogus-client
retries 3
timeout http-request 15s
timeout queue 10s
timeout connect 3s
timeout client 100s
timeout server 100s
timeout check 1s
timeout http-keep-alive 5s
load-server-state-from-file global
####################################################################################################
frontend HTTPS
bind *:443 ssl crt-list /etc/haproxy/ssl/cert-list.txt tls-ticket-keys /etc/haproxy/ssl/tls-ticket.key alpn h2,http/1.1 npn h2,http/1.1
backlog 100000
timeout http-keep-alive 100s
# HSTS
# rspadd Strict-Transport-Security:\ max-age=31536000
http-request set-header X-Forwarded-Proto https if !{ hdr(X-Forwarded-Proto) -m found }
http-request set-header X-Forwarded-Host %[req.hdr(host)]
http-request deny deny_status 403 if !{ req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m found }
http-request deny deny_status 403 if { req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m str http } { hdr(X-Forwarded-Proto) -m str https }
stick-table type string len 2048 size 100k expire 10m store http_req_rate(10s),bytes_out_rate(60s)
http-request track-sc0 path
use_backend CACHE_POOL if { sc0_http_req_rate gt 10 } { req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m found }
use_backend CACHE_POOL_HASH
capture request header Host len 64
log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[ssl_fc_sni] %[ssl_fc_is_resumed] %[ssl_fc_alpn] %[ssl_fc_npn]"
####################################################################################################
listen Stats
bind *:8500 process 1
stats enable
stats realm Haproxy\ Statistics
stats uri /
stats auth bluetiger:qorentks
stats admin if TRUE
stats refresh 60s
####################################################################################################
frontend HTTP
bind *:80
http-request redirect scheme https code 301 if { req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m str https }
http-request set-header X-Forwarded-Proto http
http-request allow if { hdr(host) -i stat.haproxy }
http-request deny deny_status 403 if !{ req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m found }
http-request deny deny_status 403 if { req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m str http } { hdr(X-Forwarded-Proto) -m str https }
stick-table type string len 2048 size 100k expire 10m store http_req_rate(10s),bytes_out_rate(60s)
http-request track-sc0 path
use_backend STAT if { hdr(host) -i stat.haproxy }
use_backend RR_POOL if { sc0_http_req_rate gt 10 } { req.hdr(host),lower,map_beg(/etc/haproxy/DomainToAccept.map) -m found }
use_backend HASH_POOL
capture request header Host len 64
log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
####################################################################################################
backend STAT
server STAT 127.0.0.1:9090 weight 10 check inter 10s
####################################################################################################
backend HASH_POOL
balance uri
hash-type consistent
option log-health-checks
http-reuse safe
server x.x.x.1 x.x.x.1:8080 weight 10 check inter 1s
server x.x.x.2 x.x.x.2:8080 weight 10 check inter 1s
server x.x.x.3 x.x.x.3:8080 weight 10 check inter 1s
####################################################################################################
backend RR_POOL
option log-health-checks
http-reuse safe
server x.x.x.1 x.x.x.1:8080 weight 10 check inter 1s
server x.x.x.2 x.x.x.2:8080 weight 10 check inter 1s
server x.x.x.3 x.x.x.3:8080 weight 10 check inter 1s
####################################################################################################
strace -tt -p
strace -tt -p 169075
strace: Process 169075 attached
18:23:47.072216 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=687996000}) = 0
18:23:47.072558 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688039247}) = 0
18:23:47.072693 epoll_wait(5, [], 200, 172) = 0
18:23:47.245032 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688135477}) = 0
18:23:47.245225 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688169765}) = 0
18:23:47.245326 epoll_wait(5, [], 200, 10) = 0
18:23:47.255525 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688237475}) = 0
18:23:47.255663 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688268626}) = 0
18:23:47.255781 epoll_wait(5, [], 200, 149) = 0
18:23:47.405125 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688341412}) = 0
18:23:47.405264 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688370830}) = 0
18:23:47.405392 epoll_wait(5, 0x2327860, 200, 47) = -1 EINTR (Interrupted system call)
18:23:47.426563 --- SIGALRM {si_signo=SIGALRM, si_code=SI_TIMER, si_timerid=0x22, si_overrun=0, si_value={int=17, ptr=0x7f9200000011}} ---
18:23:47.426647 clock_gettime(0xffffffffffeb5bd6 /* CLOCK_??? */, {tv_sec=97365, tv_nsec=821814677}) = 0
18:23:47.426780 timer_settime(34, 0, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=1, tv_nsec=0}}, NULL) = 0
18:23:47.426885 rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
18:23:47.426992 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688588561}) = 0
18:23:47.427063 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688605798}) = 0
18:23:47.427130 epoll_wait(5, [], 200, 26) = 0
18:23:47.453302 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688659565}) = 0
18:23:47.453403 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688685802}) = 0
18:23:47.453517 epoll_wait(5, [], 200, 16) = 0
18:23:47.469702 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688751286}) = 0
18:23:47.469876 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688791795}) = 0
18:23:47.469965 epoll_wait(5, [], 200, 19) = 0
18:23:47.489138 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688854976}) = 0
18:23:47.489260 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688885230}) = 0
18:23:47.489379 epoll_wait(5, [], 200, 10) = 0
18:23:47.499644 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=688995558}) = 0
18:23:47.499825 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689024501}) = 0
18:23:47.499951 epoll_wait(5, [], 200, 49) = 0
18:23:47.549145 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689095901}) = 0
18:23:47.549309 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689136616}) = 0
18:23:47.549408 epoll_wait(5, [], 200, 10) = 0
18:23:47.559589 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689201456}) = 0
18:23:47.559697 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689229570}) = 0
18:23:47.559829 epoll_wait(5, [], 200, 28) = 0
18:23:47.588054 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689307597}) = 0
18:23:47.588221 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689350627}) = 0
18:23:47.588338 epoll_wait(5, [], 200, 44) = 0
18:23:47.632618 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689425838}) = 0
18:23:47.632826 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689501817}) = 0
18:23:47.632928 epoll_wait(5, [], 200, 233) = 0
18:23:47.866397 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689584365}) = 0
18:23:47.866560 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689617080}) = 0
18:23:47.866687 epoll_wait(5, [], 200, 33) = 0
18:23:47.899940 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689692940}) = 0
18:23:47.900065 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689722038}) = 0
18:23:47.900235 epoll_wait(5, [], 200, 313) = 0
18:23:48.213693 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689800393}) = 0
18:23:48.213836 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689833393}) = 0
18:23:48.213936 epoll_wait(5, [], 200, 31) = 0
18:23:48.245131 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689907734}) = 0
18:23:48.245315 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=689943453}) = 0
18:23:48.245411 epoll_wait(5, [], 200, 9) = 0
18:23:48.254623 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690007500}) = 0
18:23:48.254779 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690047248}) = 0
18:23:48.254880 epoll_wait(5, [], 200, 22) = 0
18:23:48.277127 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690126175}) = 0
18:23:48.277282 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690152426}) = 0
18:23:48.277406 epoll_wait(5, [], 200, 4) = 0
18:23:48.281659 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690234663}) = 0
18:23:48.281802 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690265234}) = 0
18:23:48.281954 epoll_wait(5, [], 200, 123) = 0
18:23:48.405323 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690353301}) = 0
18:23:48.405492 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690387879}) = 0
18:23:48.405588 epoll_wait(5, [], 200, 21) = 0
18:23:48.426819 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690455145}) = 0
18:23:48.426930 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690485220}) = 0
18:23:48.427052 epoll_wait(5, 0x2327860, 200, 22) = -1 EINTR (Interrupted system call)
18:23:48.429306 --- SIGALRM {si_signo=SIGALRM, si_code=SI_TIMER, si_timerid=0x22, si_overrun=0, si_value={int=17, ptr=0x7f9200000011}} ---
18:23:48.429402 clock_gettime(0xffffffffffeb5bd6 /* CLOCK_??? */, {tv_sec=97366, tv_nsec=822187902}) = 0
18:23:48.429513 timer_settime(34, 0, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=1, tv_nsec=0}}, NULL) = 0
18:23:48.429607 rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
18:23:48.429703 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690626300}) = 0
18:23:48.429788 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690643060}) = 0
18:23:48.429869 epoll_wait(5, [], 200, 19) = 0
18:23:48.449024 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690694174}) = 0
18:23:48.449119 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690727866}) = 0
18:23:48.449256 epoll_wait(5, [], 200, 39) = 0
18:23:48.488547 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690804985}) = 0
18:23:48.488708 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690838812}) = 0
18:23:48.488824 epoll_wait(5, [], 200, 28) = 0
18:23:48.517138 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690918906}) = 0
18:23:48.517303 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=690950767}) = 0
18:23:48.517427 epoll_wait(5, [], 200, 3) = 0
18:23:48.520689 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=691018793}) = 0
18:23:48.520822 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=691046710}) = 0
18:23:48.520954 epoll_wait(5, [], 200, 34) = 0
18:23:48.555260 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=691140734}) = 0
18:23:48.555413 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=1023, tv_nsec=691174656}) = 0
18:23:48.555508 epoll_wait(5, ^Cstrace: Process 169075 detached
show fd
4 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [lc] cnext=-3 cprev=0 tmask=0x1 umask=0x0 owner=0x2006410 iocb=0x5bcd80(mworker_accept_wrapper)
6 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x2 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
8 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lC] cnext=-3 cprev=-2 tmask=0xffffffffffff umask=0x0 owner=0x200a0a0 iocb=0x5730e0(listener_accept) l.st=RDY fe=GLOBAL
10 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lC] cnext=-3 cprev=-2 tmask=0xffffffffffff umask=0x0 owner=0x200c160 iocb=0x5730e0(listener_accept) l.st=RDY fe=HTTPS
11 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [lC] cnext=-3 cprev=0 tmask=0xffffffffffff umask=0x0 owner=0x2231290 iocb=0x5730e0(listener_accept) l.st=RDY fe=Stats
12 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lC] cnext=-3 cprev=-2 tmask=0xffffffffffff umask=0x0 owner=0x2233670 iocb=0x5730e0(listener_accept) l.st=RDY fe=HTTP
14 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x4 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
17 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x8 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
20 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x10 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
23 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x40 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
26 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x100 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
29 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x200 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
32 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x800 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
35 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x2000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
38 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x8000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
41 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x40000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
44 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x200000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
47 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x400000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
50 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
53 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x4000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
56 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x8000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
59 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x20000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
62 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x100000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
65 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x400000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
68 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x800000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
71 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
74 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x2000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
77 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x8000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
80 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x10000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
83 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x40000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
86 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x80000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
89 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x200000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
92 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x800000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
95 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x20 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
98 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x80 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
101 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x400 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
104 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
107 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x4000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
110 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x10000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
113 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x20000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
116 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x100000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
119 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x800000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
122 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x2000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
125 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x40000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
128 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x80000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
131 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x200000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
134 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x4000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
137 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x100000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
140 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x400000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
143 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x1 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
145 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x80000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
148 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x10000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
151 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x20000000000 umask=0x0 owner=0x597850 iocb=0x597850(poller_pipe_io_handler)
154 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [lc] cnext=-3 cprev=-2 tmask=0x800 umask=0x0 owner=0x7f9214312750 iocb=0x587e90(conn_fd_handler) back=0 cflg=0x00241300 fe=GLOBAL mux=PASS ctx=0x7f91941afc70
913 : st=0x22(R:pRa W:pRa) ev=0x1d(HEOpI) [lc] cnext=-3 cprev=-2 tmask=0x20000 umask=0x0 owner=0x7f916817e220 iocb=0x587e90(conn_fd_handler) back=0 cflg=0x80301300 fe=HTTP mux=H1 ctx=0x7f91f007ec50 h1c.flg=0x40 .sub=2 .ibuf=16@0x7f916823b390+0/16384 .obuf=0@(nil)+0/0 h1s=0x7f9228306710 h1s.flg=0x4010 .req.state=MSG_DONE .res.state=MSG_DATA .meth=GET status=200 .cs.flg=0x00004000 .cs.data=0x7f921c0354f8
968 : st=0x22(R:pRa W:pRa) ev=0x01(heopI) [Lc] cnext=-3 cprev=-2 tmask=0x20000 umask=0x0 owner=0x7f91c4081a60 iocb=0x587e90(conn_fd_handler) back=1 cflg=0x00202b00 sv=CACHE_POOL_HASH/183.111.124.40 mux=H1 ctx=0x7f9168139f30 h1c.flg=0x0 .sub=0 .ibuf=0@(nil)+0/0 .obuf=0@(nil)+0/0 h1s=0x7f91a4019ab0 h1s.flg=0x4210 .req.state=MSG_DONE .res.state=MSG_DATA .meth=GET status=200 .cs.flg=0x00008000 .cs.data=0x7f921c035550
perf record 1
perf record -F 99 -p 169075 -a -- sleep 60
Warning:
PID/TID switch overriding SYSTEM
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.875 MB perf.data (12165 samples) ]
perf report --stdio
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 12K of event 'cycles:ppp'
# Event count (approx.): 209035839845
#
# Overhead Command Shared Object Symbol
# ........ ....... ................. ......................................
#
34.82% haproxy haproxy [.] process_runnable_tasks
14.87% haproxy haproxy [.] h1_rcv_buf
11.74% haproxy haproxy [.] si_cs_recv
8.57% haproxy haproxy [.] stream_int_notify
4.78% haproxy haproxy [.] si_cs_process
4.57% haproxy haproxy [.] h1_wake_stream_for_recv
3.98% haproxy [kernel.kallsyms] [k] sysret_check
2.89% haproxy haproxy [.] h1_io_cb
2.77% haproxy haproxy [.] h1_process
1.73% haproxy haproxy [.] __pool_put_to_cache
1.33% haproxy haproxy [.] si_cs_io_cb
0.83% haproxy haproxy [.] h1_refresh_timeout
0.46% haproxy [vdso] [.] __vdso_clock_gettime
0.40% haproxy haproxy [.] h1_subscribe
0.35% haproxy [kernel.kallsyms] [k] update_curr
0.26% haproxy [kernel.kallsyms] [k] sys_clock_gettime
0.25% haproxy haproxy [.] _do_poll
0.24% haproxy [kernel.kallsyms] [k] sys_epoll_wait
0.23% haproxy [kernel.kallsyms] [k] _raw_spin_lock_irqsave
0.22% haproxy [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.21% haproxy [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore
0.20% haproxy [kernel.kallsyms] [k] native_sched_clock