I am facing significant delays in some specific cases of the requests between servers. Below is the architecture which I am following.
Server A & Server B Configuration:
- Public Traffic received at HaProxy having two backend services with Round Robin as load balancing method.
- These services are running in Docker containers.
- I am using php:7.4-apache with lumen
Scenario
- Server A receives a Request from Client.
- Server A initiates the request to server B
- Server B requests back to Server A (Without giving Response to Server A) for additional details because Server A has integration with 3rd Party API for some information.
- Server B gets the data from 3rd Party API and respond back to Server A.
- Server A respond back to client.
In the above scenario I am facing significant delays. The total time to start and end the request is like 20-35 seconds. Without containerized environment the total time is just 4-5 seconds.
HaProxy Configuration
global
log /dev/log local0
log localhost local1 notice
maxconn 2000
daemon
ssl-server-verify none
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
retries 3
timeout connect 5000
timeout client 50000
timeout server 50000
frontend http-in
bind *:80
bind *:443 ssl crt /ssl/example.com.pem
http-request redirect scheme https unless { ssl_fc }
default_backend webservers
option forwardfor
backend webservers
stats enable
stats auth username:password
stats uri /haproxy?stats
balance roundrobin
option httpchk
option http-server-close
option forwardfor
http-response set-header X-Frame-Options "DENY"
http-response set-header X-XSS-Protection 1;mode=block
http-response set-header Permissions-Policy fullscreen=();geolocation=()
http-response set-header X-Content-Type-Options "nosniff"
http-response set-header Strict-Transport-Security max-age=31536000;includeSubDomains;preload
http-response set-header Referrer-Policy no-referrer-when-downgrade
http-response set-header Public-Key-Pins "pin-sha256=\"pin1\"; pin-sha256=\"pin2\"; pin-sha256=\"pin3\"; pin-sha256=\"pin4\"; max-age=5184000; includeSubDomains"
server apache1 web_app:443 check maxconn 20 ssl
server apache2 web_app2:443 check maxconn 20 ssl
Docker Compose File
services:
web_app:
image: repo_url/web_app_image:latest
container_name: webb_app
env_file:
- .env
extra_hosts:
- "host1:IP_1"
- "host2:IP_2"
- "host2:IP_3"
volumes:
- ./app-data:/var/www/storage
- ./apache/logs:/var/log/apache2
- ./apache2/ssl:/etc/apache2/ssl
- ./ssl/openssl.cnf:/etc/ssl/openssl.cnf
networks:
- web_app_net
web_app2:
image: repo_url/web_app_image:latest
container_name: webb_app2
env_file:
- .env
extra_hosts:
- "host1:IP_1"
- "host2:IP_2"
- "host2:IP_#"
volumes:
- ./app-data:/var/www/storage
- ./apache/logs:/var/log/apache2
- ./apache2/ssl:/etc/apache2/ssl
- ./ssl/openssl.cnf:/etc/ssl/openssl.cnf
networks:
- web_app_net
haproxy:
image: repo_url/web_app_image/haproxy:latest
container_name: haproxy
volumes:
- ./haproxy/conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
- ./haproxy/ssl:/ssl
ports:
- 80:80
- 443:443
networks:
- web_app_net
networks:
web_app_net:
driver: bridge
ipam:
config:
- subnet: 172.29.0.0/24
Dockerfile for web app
ARG DOCKER_REG
FROM ${DOCKER_REG}/php:7.4-apache
MAINTAINER some_email@some_domain.com
# Copy composer.lock and composer.json
COPY composer.lock composer.json /var/www/
# Set working directory
WORKDIR /var/www
# Install dependencies
RUN apt-get update && apt-get install -y \
build-essential \
mariadb-client \
libpng-dev \
libjpeg62-turbo-dev \
libfreetype6-dev \
libgmp-dev \
locales \
zip \
jpegoptim optipng pngquant gifsicle \
vim \
unzip \
git \
curl \
libxml2-dev \
libcurl4-openssl-dev \
pkg-config \
libssl-dev
# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
# Install extensions
#with 7.4 version
RUN docker-php-ext-install pdo_mysql exif pcntl bcmath gmp
RUN docker-php-ext-configure gd --with-freetype=/usr/include/ --with-jpeg=/usr/include/
RUN docker-php-ext-install gd
RUN a2enmod rewrite headers ssl
# Install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
# Add user for laravel application
RUN groupadd -g 1000 www
RUN useradd -u 1000 -ms /bin/bash -g www www
COPY apache/ssl /etc/apache2/ssl
COPY apache/config/emoney_apache-ssl.conf /etc/apache2/sites-enabled/emoney_apache-ssl.conf
COPY apache/config/emoney_apache.conf /etc/apache2/sites-enabled/emoney_apache.conf
RUN rm -rf /etc/apache2/sites-enabled/000-default.conf
# Copy existing application directory contents
COPY . /var/www
# Copy existing application directory permissions
COPY --chown=www:www . /var/www
RUN composer self-update 1.10.10
RUN composer install
# Change current user to www
#USER www
RUN chown -R $USER:www-data /var/www/storage && \
chmod -R 775 /var/www/storage
NOTE: I am only getting the delays in circular requests when it comes to containerized/docker environment. Otherwise with Apache simple one node everything works perfectly fine. I need suggestions from experts to overcome this situation in production architecture.
I also need suggestion either it is good to use above architecture in production or not. Plus if there is a better way to handle the cyclic requests at proxy level or container level or apache level I would love to hear the community suggestions.
Regards, Nasir