Command limit when using runtime API?

[gosko]

Hi, I am using the runtime API to add/remove/modify entries in map files as described in this blog post, and it works fine for small numbers of commands at a time, but if there are too many commands at once I usually get an error after a few hundred commands have been processed:

2022/02/01 03:15:38 socat[29468] E write(5, 0x55f4b129ee30, 67): Broken pipe

This happens when using echo | socat from the command line, for example:

for n in {1..500}; do echo "add map /etc/haproxy/test.map /test$n -"; done | socat stdio /run/haproxy.stat

And also if using the following perl code:

    foreach my $kr (sort keys %filemap) {
        push( @changes, "add map $mapfile $kr " . $filemap{$kr} );
    }

    # open new connection to haproxy socket
    my $writesock = new IO::Socket::UNIX (
        Peer => '/run/haproxy.stat',
        Type => SOCK_STREAM,
    );
    if ( ! $writesock ) {
        syslog('warning',"unable to connect to haproxy socket");
        return;
    }
    print $writesock join("\n",@changes) . "\n";
    close $writesock;

If I add a sleep(3) before the close $writesock in the perl code the issue seems to go away, but I wonder if there’s something else I should be doing.

\Should I be limiting the number of commands given at once, or doing something else with the socket before closing it?

[lukastribus]

For now: yes, a small sleep would probably be good.

Improvements have just been applied to the development tree and will trickle down to stable branches at some point:

Crash when sending multiple "del map" commands at once

[gosko]

Great, thanks.

Should I be rate-limiting my commands over the socket? The reporter of that issue said he is sending 50 del map commands per second but I haven’t seen that mentioned in the haproxy docs or blog. I don’t want these map updates to have a negative impact on the server’s performance or stability.

[lukastribus]

The current haproxy behavior it’s a bug, that’s why it’s not documented.

You can rate-limited the commands for now as a workaround, until you can use fixed code.

[gosko]

If I’m running the fixed code, will it be necessary to rate-limit the commands? These map file updates are not something we’re running in production yet so I would rather wait for the haproxy update than add a rate-limiting feature to my code if it won’t be needed in a month or two.

[lukastribus]

No, rate-limiting commands will not be necessary long term, if something doesn’t work, it’s a bug that fill be fixed.

If you are running in the lab, you can try the just released v2.6-dev1, this is a development snapshot but contains all the runtime API fixes.

The fixes will definitely be backported but with the dev snapshot you can try now.

[gosko]

Excellent, thanks very much for the quick responses.

[gosko]

Hi, I saw that based on the changelog Willy’s fixes have now made it into 2.4.13, but this issue still seems to be present in 2.4.13. Is that expected?

for n in {1..5000}; do echo "add map /etc/haproxy/test.map /test$n -"; done | socat stdio /run/haproxy.stat
2022/02/18 17:16:48 socat[17569] E write(5, 0x560d34be9e30, 528): Broken pipe

This is with ‘HAProxy version 2.4.13-095275f, released 2022/02/16’

[lukastribus]

Can you file a new bug on github please?

Issues · haproxy/haproxy