Facing a weird delay in responses HaProxy | Apache | Docker Container

I am facing significant delays in some specific cases of the requests between servers. Below is the architecture which I am following.

Server A & Server B Configuration:

  1. Public Traffic received at HaProxy having two backend services with Round Robin as load balancing method.
  2. These services are running in Docker containers.
  3. I am using php:7.4-apache with lumen

Scenario

  1. Server A receives a Request from Client.
  2. Server A initiates the request to server B
  3. Server B requests back to Server A (Without giving Response to Server A) for additional details because Server A has integration with 3rd Party API for some information.
  4. Server B gets the data from 3rd Party API and respond back to Server A.
  5. Server A respond back to client.

In the above scenario I am facing significant delays. The total time to start and end the request is like 20-35 seconds. Without containerized environment the total time is just 4-5 seconds.

HaProxy Configuration

global
    log /dev/log local0
    log localhost local1 notice
    maxconn 2000
    daemon
    ssl-server-verify none

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    retries 3
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend http-in
    bind *:80
    bind *:443 ssl crt /ssl/example.com.pem
    http-request redirect scheme https unless { ssl_fc }
    default_backend webservers
    option forwardfor

backend webservers
    stats enable
    stats auth username:password
    stats uri /haproxy?stats
    balance roundrobin
    option httpchk
    option http-server-close
    option forwardfor
    http-response set-header X-Frame-Options "DENY"
    http-response set-header X-XSS-Protection 1;mode=block
    http-response set-header Permissions-Policy fullscreen=();geolocation=()
    http-response set-header X-Content-Type-Options "nosniff"
    http-response set-header Strict-Transport-Security max-age=31536000;includeSubDomains;preload
    http-response set-header Referrer-Policy no-referrer-when-downgrade
    http-response set-header Public-Key-Pins "pin-sha256=\"pin1\"; pin-sha256=\"pin2\"; pin-sha256=\"pin3\"; pin-sha256=\"pin4\"; max-age=5184000; includeSubDomains"
    server apache1 web_app:443 check maxconn 20 ssl
    server apache2 web_app2:443 check maxconn 20 ssl

Docker Compose File

services:
  web_app:
    image: repo_url/web_app_image:latest
    container_name: webb_app
    env_file:
      - .env
    extra_hosts:
      - "host1:IP_1"
      - "host2:IP_2"
      - "host2:IP_3"
    volumes:
      - ./app-data:/var/www/storage
      - ./apache/logs:/var/log/apache2
      - ./apache2/ssl:/etc/apache2/ssl
      - ./ssl/openssl.cnf:/etc/ssl/openssl.cnf
    networks:
      - web_app_net
  web_app2:
    image: repo_url/web_app_image:latest
    container_name: webb_app2
    env_file:
      - .env
    extra_hosts:
      - "host1:IP_1"
      - "host2:IP_2"
      - "host2:IP_#"
    volumes:
      - ./app-data:/var/www/storage
      - ./apache/logs:/var/log/apache2
      - ./apache2/ssl:/etc/apache2/ssl
      - ./ssl/openssl.cnf:/etc/ssl/openssl.cnf
    networks:
      - web_app_net
  haproxy:
    image: repo_url/web_app_image/haproxy:latest
    container_name: haproxy
    volumes:
      - ./haproxy/conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
      - ./haproxy/ssl:/ssl
    ports:
      - 80:80
      - 443:443
    networks:
      - web_app_net
networks:
  web_app_net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.29.0.0/24

Dockerfile for web app

ARG DOCKER_REG
FROM ${DOCKER_REG}/php:7.4-apache
MAINTAINER some_email@some_domain.com
# Copy composer.lock and composer.json
COPY composer.lock composer.json /var/www/

# Set working directory
WORKDIR /var/www

# Install dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    mariadb-client \
    libpng-dev \
    libjpeg62-turbo-dev \
    libfreetype6-dev \
    libgmp-dev \
    locales \
    zip \
    jpegoptim optipng pngquant gifsicle \
    vim \
    unzip \
    git \
    curl \
    libxml2-dev \
    libcurl4-openssl-dev \
    pkg-config \
    libssl-dev

# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Install extensions
#with 7.4 version
RUN docker-php-ext-install pdo_mysql exif pcntl bcmath gmp
RUN docker-php-ext-configure gd --with-freetype=/usr/include/ --with-jpeg=/usr/include/
RUN docker-php-ext-install gd

RUN a2enmod rewrite headers ssl
# Install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Add user for laravel application
RUN groupadd -g 1000 www
RUN useradd -u 1000 -ms /bin/bash -g www www

COPY apache/ssl /etc/apache2/ssl
COPY apache/config/emoney_apache-ssl.conf /etc/apache2/sites-enabled/emoney_apache-ssl.conf
COPY apache/config/emoney_apache.conf /etc/apache2/sites-enabled/emoney_apache.conf
RUN rm -rf /etc/apache2/sites-enabled/000-default.conf
# Copy existing application directory contents
COPY . /var/www

# Copy existing application directory permissions
COPY --chown=www:www . /var/www

RUN composer self-update 1.10.10

RUN composer install

# Change current user to www
#USER www

RUN chown -R $USER:www-data /var/www/storage && \
    chmod -R 775 /var/www/storage

NOTE: I am only getting the delays in circular requests when it comes to containerized/docker environment. Otherwise with Apache simple one node everything works perfectly fine. I need suggestions from experts to overcome this situation in production architecture.

I also need suggestion either it is good to use above architecture in production or not. Plus if there is a better way to handle the cyclic requests at proxy level or container level or apache level I would love to hear the community suggestions.

Regards, Nasir