Trouble with starting the Data Plane API

Help!
1

Hi there!

I am attempting to run the HAProxy Data Plane API from a Docker container in Kubernetes and I have gotten to the point where the app is crashing with no logs whatsoever.

At first it was complaining about not having access to write in any of the directories which I could validate based on the permissions of the folder and everything belonging to the root user and group. I was able to surpass this by changing every directory to /tmp.

Now, being past all of those errors, whenever the Data Plane API starts up it will take down the entire running instance and I’m not sure why or how to fix it. I have attached the logs and my configuration at the bottom of this post.

I do not have permission to bind to privileged ports which is why I am binding to 8080 and 4443, although I perform a mapping to that port through its k8s service.

I am deploying this using helm and running the latest version of HAProxy.

Here is a copy of my config:

[NOTICE]   (1) : New program 'api' (8) forked
[NOTICE]   (1) : New worker (9) forked
[NOTICE]   (1) : Loading success.
configuration file /tmp/dataplaneapi.yml does not exists, creating one
[NOTICE]   (1) : haproxy version is 2.8.2-61a0f57
[ALERT]    (1) : Current program 'api' (8) exited with code 1 (Exit)
[ALERT]    (1) : exit-on-failure: killing every processes with SIGTERM
[ALERT]    (1) : Current worker (9) exited with code 143 (Terminated)
[WARNING]  (1) : All workers exited. Exiting... (1)
Stream closed EOF  default/haproxy-6f9f7c6b6d-jwj5l (haproxy)


global
    log stdout format raw local0 debug
    maxconn 1024
    master-worker
    stats socket /tmp/haproxy.sock mode 666 level admin

  userlist haproxy-dataplaneapi
    user admin insecure-password mypassword

  program api
    command /usr/local/bin/dataplaneapi --host 0.0.0.0 --port 5555 --haproxy-bin /usr/sbin/haproxy -f /tmp/dataplaneapi.yml --config-file /usr/local/etc/haproxy/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --reload-delay 5 --restart-cmd "kill -SIGUSR2 1" --userlist haproxy-dataplaneapi --maps-dir /tmp --backups-dir /tmp --ssl-certs-dir /tmp --general-storage-dir /tmp --spoe-dir /tmp --cluster-tls-dir /tmp --spoe-transaction-dir /tmp
    no option start-on-reload

  defaults
    mode http
    log global
    timeout client 60s
    timeout connect 60s
    timeout server 60s

  frontend fe_main
    bind :8080
    default_backend default_response

  backend default_response
    http-request return content-type "text/plain" string "Hello World!"

Any help is greatly appreciated! Thank you.

2

I have a similar error message happen and when i run the container again it successfully starts. I’m using docker compose up to start it up, and I wonder if that dataplaneapi.yaml file is getting created once and already present on subsequent start ups of the container. Because if I do docker system prune -a and then docker compose up, the problem comes back

3

Try using the --scheme=http flag in your dataplaneapi invocation in command /usr/local/bin/dataplaneapi ...

Short explanation:
Faced the exact same problem, the dataplaneapi subprocess just exiting without any info. I eventually found that the default scheme (protocol(s) to be used by the dataplaneapi’s listener) was changed from [http] to [http, https] but the inclusion of https in the list requires extra flags for the command (–tls-key, --tls-certificate).

Longer explanation:
Without any logs, I managed to find this by using strace, ltrace, then gdb on that dataplaneapi invocation. Note: catch syscall 231 means stop when encountering exit_group system calls which I saw were happening right before the process exited without logs:

gdb --args /usr/bin/dataplaneapi --host 0.0.0.0 --port 5555 --log-level=info --haproxy-bin /usr/sbin/haproxy --config-file /usr/local/etc/haproxy/haproxy.cfg --reload-cmd="/bin/bash /bin/haproxy_reload" --restart-cmd="/bin/bash /bin/haproxy_reload" --reload-delay 10 --userlist haproxy-dataplaneapi --ssl-certs-dir=/tmp/ssl --spoe-dir=/tmp/spoe
...
(gdb) catch syscall 231
(gdb) run
... 
(gdb) bt
#0  runtime.exit () at runtime/sys_linux_amd64.s:55
#1  0x0000000000473c30 in syscall.Exit (code=-38) at runtime/runtime.go:132
#2  0x00000000004f9d58 in os.Exit (code=1) at os/proc.go:77
#3  0x0000000001229130 in github.com/haproxytech/dataplaneapi.(*Server).Fatalf (s=<optimized out>, f=..., args=...) at github.com/haproxytech/dataplaneapi/server.go:136
#4  0x000000000122a33f in github.com/haproxytech/dataplaneapi.(*Server).Serve (s=0xc0003d0000, err=...) at github.com/haproxytech/dataplaneapi/server.go:312
#5  0x000000000122fa07 in main.startServer (cfg=0xc000266008, cancelDebugServer={void (void)} 0xc00219ff08, reload=...) at github.com/haproxytech/dataplaneapi/cmd/dataplaneapi/main.go:248
#6  0x000000000122d9ca in main.main () at github.com/haproxytech/dataplaneapi/cmd/dataplaneapi/main.go:63

Hmm… server.go:312: dataplaneapi/server.go at v2.9.1 · haproxytech/dataplaneapi · GitHub

s.Fatalf("the required flags `--tls-certificate` and `--tls-key` were not specified")

Wtf, why does it need TLS stuff?
That line is inside an if block: dataplaneapi/server.go at v2.9.1 · haproxytech/dataplaneapi · GitHub

if s.hasScheme(schemeHTTPS) {
  ...

Then it took even more time to find where the default value for --scheme comes from: MINOR: add the https scheme to the API spec · haproxytech/client-native@8fa8615 · GitHub

There we have it. It was changed from just http to http AND https but the addition of https needs TLS-related cli flags.

There are so many things I don’t understand, like:

  • why does it not log that it is missing the --tls... flags?
  • why was this default changed without adding checks or warnings that new flags are now required?

At least adding --scheme=http made it work for us. Cost me 2 days of debugging and headaches. You are not alone!