Keepass Sync with WebDAV server results in Bad Gateway 502 when using HAProxy

Help!
1

Hi,

I’m currently facing an annoying issue with HAproxy and my (Synology) WebDAV server, running behind a linux firewall (IPFire).

I’m using Keepass (latest version) on Win10 Pro. Keepass successfully loads a file from my internal WebDAV server w/o any issues, accessing the file with https://webdav.mydomain.de/webdav/pw.kdbx. This traffic is passing the firewall with a running HAProxy service just fine.

keepass -> www -> firewall with haproxy -> LAN -> WebDav (Port 5005)

However, when saving any modification in this password file to the WebDAV again, this results in a bad gateway 502 error.

I noticed that Keepass first successfully creates a temporary file and when trying to move this temp file to the original one, this finally results in the 502 error.

As you will probably notice above, I access the file in question with https and my WebDAV server is running on port 80. SSL termination is done by HAProxy using a LE cert.

At the time of this error, the haproxy log file reads, pls. see last line.

Jun 19 14:49:30 localhost haproxy[25037]: 123.456.78.90:54146 [19/Jun/2020:14:49:30.427] http_https~ webdav_server/webdav01 0/0/1/1/2 401 612 - - --NI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "GET /webdav/pw.kdbx HTTP/1.1"
Jun 19 14:49:32 localhost haproxy[25037]: 123.456.78.90:54146 [19/Jun/2020:14:49:30.441] http_https~ webdav_server/webdav01 1/0/0/825/1883 200 3336890 - - --NI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "GET /webdav/pw.kdbx HTTP/1.1"
Jun 19 14:49:39 localhost haproxy[25037]: 123.456.78.90:54146 [19/Jun/2020:14:49:38.485] http_https~ webdav_server/webdav01 0/0/1/628/1430 200 3336890 - - --NI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "GET /webdav/pw.kdbx HTTP/1.1"
Jun 19 14:49:42 localhost haproxy[25037]: 123.456.78.90:54146 [19/Jun/2020:14:49:41.381] http_https~ webdav_server/webdav01 0/0/1/1365/1366 201 438 - - --NI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "PUT /webdav/pw.kdbx.tmp HTTP/1.1"
Jun 19 14:49:43 localhost haproxy[25037]: 123.456.78.90:54146 [19/Jun/2020:14:49:42.757] http_https~ webdav_server/webdav01 0/0/1/669/686 200 290052 - - CDNI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "GET /webdav/pw.kdbx HTTP/1.1"
Jun 19 14:49:44 localhost haproxy[25037]: 123.456.78.90:54166 [19/Jun/2020:14:49:43.523] http_https~ webdav_server/webdav01 0/0/1/727/728 204 129 - - --NI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "DELETE /webdav/pw.kdbx HTTP/1.1"
Jun 19 14:49:44 localhost haproxy[25037]: 123.456.78.90:54166 [19/Jun/2020:14:49:44.259] http_https~ webdav_server/webdav01 1/0/0/676/677 502 406 - - --NI 1/1/0/0/0 0/0 {webdav.mydomain.de|} "MOVE /webdav/pw.kdbx.tmp HTTP/1.1"

What’s interesting: when using the same https URL from within a Android tablet, using one of the abvailable file explorers that is capable of the WebDAV protocol, all is fine! Which means, I can download any file from the server, create, delete any files and folders w/o any issues.

IMO, the WebDAV server is not the cause if this problem. Maybe HAProxy or maybe Keepass at the end.

I’ve done a further test: I’ve created a port forwarding in the firewall to let Keepass reach the WebDAV server in LAN without passing HAProxy, using the URL http://123.456.78.90/webdav/pw.kdbx to access the file. Guess what? Keepass successfully saved the modified file without any error.

I found out alread when removing those lines from config file

http-request set-header X-Forwarded-Proto https 
redirect scheme https code 301 if !{ ssl_fc }

Keepass works as expected. Of course I’m now using the http protocol instead of desired https.

Now I’m clueless! Any hints on how to get rid of this problem, is highly appreciated!

Below is my current haproxy.cfg

cu,
Michael

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local1
        
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        nobody
    group       nobody
    daemon
    
    tune.ssl.default-dh-param 2048
    #tune.maxrewrite 4096
    #tune.http.maxhdr 202

    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    
    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  forwardfor except 127.0.0.0/8
    option                  redispatch except 172.0.0.0/8 
    retries                 3
    timeout http-request    30s
    timeout queue           1m
    timeout connect         30s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 30s
    timeout check           30s
    maxconn                 3000
 
#---------------------------------------------------------------------
# Frontend Configuration
#---------------------------------------------------------------------
frontend http_https
    bind 172.17.0.2:80
    
    #Add available LE certs 
    bind 172.17.0.2:443 ssl crt /etc/haproxy/certs/webdav.mydomain.de.pem 

    mode http
    
    #---------------------
    #HAProxy handles SSL
    #---------------------
    #X-Forwarded-Proto for SSL offloading - needed for 
    http-request set-header X-Forwarded-Proto https
    redirect scheme https code 301 if !{ ssl_fc }

    #Logging
    capture request header host len 40
    capture request header cookie len 20
    
    #Default log format, unchanged
    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
        
	       
    http-response set-header Strict-Transport-Security max-age=31536000
	
    # X-Content-Type-Options
    http-response set-header X-Content-Type-Options nosniff
	
    # X-Xss-Protection (for Chrome, Safari, IE)
    http-response set-header X-Xss-Protection 1;\ mode=block
	
    # X-Frame-Options (DENY or SELF)
    http-response set-header X-Frame-Options DENY
	
    # X-Robots-Tag to not index our site
    http-response set-header X-Robots-Tag none
	
    # Delete Server Header
    http-response del-header Server
	
    #Instruct clients to not sniff for Content-Type   
    http-response set-header X-Content-Type-Options: nosniff
	
    #Leaving HTTPS to HTTP page permit sniffing to find out actual HTTPS URLs
    http-response set-header Referrer-Policy no-referrer-when-downgrade


#---------------------------------------------------------------------
#Backend Configuration
#---------------------------------------------------------------------
    acl is_webdav_domain hdr_beg(host) -i webdav.mydomain.de
    
    #----WEBDAV----
    acl is_webdav_path path -i /webdav/
	
    http-request set-path /webdav%[path] if is_webdav_domain !is_webdav_path
    use_backend webdav_server if is_webdav_domain
      
    #default
    default_backend no_match

	  
#---------------------------------------------------------------------
# Backend WEBDAV
#---------------------------------------------------------------------
backend webdav_server
    balance leastconn
    cookie WEBDAVSERVER insert indirect nocache
    http-check disable-on-404
    http-check expect status 401
    option httpchk GET /webdav

    server webdav01 192.168.6.96:5005 cookie webdav01 inter 60s
	
#---------------------------------------------------------------------
# Backend: No Match
#---------------------------------------------------------------------
backend no_match
   http-request deny deny_status 400
2

So which one of the two is it?

Configure the admin socket and use show errors to have more information about this:

https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.3-show%20errors

You could also try to set option accept-invalid-http-response
https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4-option%20accept-invalid-http-response

3

As removing those lines disables http → https redirection, it does not matter at the end. I still want to use https though, hence I need those lines. I was just a proof of concept that basically HAProxy works with WebDAV server. Sbdy already told me that this server probably runs nginx if this matters.

This option atually did not change the game :neutral_face:

Is there an alternative to socat? My firewall actually does not offer this command. Maybe nc. Did already try but could not connect to any socket - I’m not the Linux Pro, though. :grinning:

4

It matters to pinpoint the root cause if you found a configuration option that actually impacts this.

You are saying that Keepass works as expected if you remove those two configuration lines. So I’m not sure if you are saying that Keeppass works:

  • when using HTTP instead of HTTPS through haproxy
  • when haproxy doesn’t not redirct to HTTPS (but you are using HTTPS anyway)
  • when haproxy does not send the X-Forwarded-Proto header
  • or whether you don’t actually know

No, not that I’m aware.

And alternative to the admin socket may be to capture the entire unencrypted HTTP traffic between haproxy and your backend server, so we can check how the request and response look like. So on your firewall run:

tcpdump -ps0 -i ethX -w keepass-unencrypted-backend-traffic.cap host 192.168.6.96 and port 5005

Adjust interface, filename and capture filter as needed, reproduce the problem and analyze the capture in wireshark or provide it here.

The keepass file would be encrypted of course, so we would not see the actual content, but you can also create a new, empty keepass database file just to reproduce the problem and share the data without actually sharing your real database.

5

With the help of others, I got a solution for the missing socat command:

echo “show errors -1 response” | ncat --unixsock /var/run/haproxy.sock

I’ve modified my haproxy.cfg and added these lines before executing the command above:

stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 2m

Unfortunately this did reveal - nothing no single line, no error at all.

So, I continued with a new empty Keepass database and started the dump right before pressing the save button. When this error occurred, I stop tracing. Here is a Dropbox link with the cap file Dropbox - File Deleted - Simplify your life

@lukastribus would you be willing to analyze this file for me? I certainly do not have the skills for this task.

For the rest of the questions:

when using HTTP instead of HTTPS through haproxy

Yes, this actually works as expected by removing both lines from haproxy.cfg which forces me to use http instead of https of course.

when haproxy doesn’t not redirct to HTTPS (but you are using HTTPS anyway)

No, by removing both lines I’m now using http protocol

when haproxy does not send the X-Forwarded-Proto header

This does not matter at all.

Actually only by removing both lines and using http, is the solution for my issue. Unfortunately I will loose SSL encryption then.

Michael

6

The failing HTTP transaction is this:

MOVE /webdav/Database.kdbx.tmp HTTP/1.1
destination: https://webdav.mydomain.de/webdav/Database.kdbx
authorization: Basic <omitted>
host: webdav.mydomain.de
x-forwarded-proto: https
x-forwarded-for: <omitted>
connection: close

HTTP/1.1 502 Bad Gateway
Date: Tue, 23 Jun 2020 18:13:53 GMT
Server: Apache
Content-Length: 255
Connection: close
Content-Type: text/html; charset=ISO-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>Destination URI refers to different scheme or port (https://hostname:443)
(want: http://hostname:5005)</p>
</body></html>

As the MOVE request requires a destination header that includes the entire scheme and hostname, it looks like the backend doesn’t like the fact that it’s HTTPS.

You can try modifying https to http in that particular request header:

http-request replace-header destination ^https(.+) http\1

If it still doesn’t work, try replacing the entire hostname with what the backend apparently wants:

http-request replace-header destination https://webdav.mydomain.de/(.+) http://hostname:5005/\1
1 Like
7

Thanks Sir! :laughing:
This was the perfect solution to my issue! The Keepass file now saves as expected.

If you do not mind, a last question for my configuration. As you may see in my haproxy.cfg above, I’m setting a new path, if the source URL does not contain the necessary path /webdav/ in its initial request:

acl is_webdav_path path -i /webdav/
http-request set-path /webdav%[path] if is_webdav_domain !is_webdav_path

When using a URL like https://webdav.mydomain.de/password.kdbx to again request a file from the appropriate server, this config line above should result in a URL https://webdav.mydomain.da/webdav/password.kdbx.

This currently works - until Keepass tries to save the database again. I’m now getting the error 405 - method not allowed.

OTH, if using the “correct” path in the URL, like https://webdav.mydomain.de/webdav/password.kdbx all runs well. Notice the path /webdav/ in the middle.

Hence, I guess there is still some magic necessary to get this part running as well, correct. Do you have another hint how to deal with this?

Edit: Next issue arised! When using your solution http-request replace-header destination ^https(.+) http\1 the http to https redirection does not work anymore :face_with_thermometer:
The error that appears now is 401 - not authorized when accessing the file on the WebDav server.

8

What did was a workaround, and clearly a bad idea, given that we introduced two new problems.

I suggest your remove all workarounds and work on the backend site, to actually support https and a non-root path.