Pardon me if this is not the correct section, but I couldn’t find an HAProxy-specific topic to post this on.
Percona offers an HAProxy package, which can be installed with:
yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm
percona-release setup ppg16
dnf install percona-haproxy.x86_64
When using it along keepalived, one may also want to enable binding to any IP address:
sysctl -w net.ipv4.ip_nonlocal_bind=1
setsebool -P haproxy_connect_any=1
Here is a small /etc/haproxy/haproxy.cfg that can help us reproduce the problem:
global
chroot /var/lib/haproxy
log /dev/log local0
maxconn 4096
daemon
user haproxy
group haproxy
stats socket /var/lib/haproxy/stats level admin
pidfile /var/run/haproxy.pid
defaults
log global
option dontlognull
timeout http-request 60s
timeout queue 1m
timeout connect 60s
timeout client 5m
timeout server 5m
timeout http-keep-alive 30s
timeout check 60s
mode http
balance source
listen my-listener-here
bind 192.168.248.239:6443
mode tcp
server master1 my-backend:6443 check inter 1s
If we run a systemctl status haproxy.service
we can see the service is running.
Now let’s restart the machine:
# reboot
Now, upon the machine’s reboot, we can see the status is now failed:
# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/haproxy.service.d
└─override.conf
Active: failed (Result: exit-code) since Tue 2025-01-28 15:07:26 CET; 15s ago
Process: 1195 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=1/FAILURE)
gen 28 15:07:26 myhostname systemd[1]: Starting HAProxy Load Balancer...
gen 28 15:07:26 myhostname systemd[1]: haproxy.service: Control process exited, code=exited status=1
gen 28 15:07:26 myhostname systemd[1]: haproxy.service: Failed with result 'exit-code'.
gen 28 15:07:26 myhostname systemd[1]: Failed to start HAProxy Load Balancer.
The ExecStartPre failed with status=1/exit code 1.
Let us override the service to troubleshoot the issue with strace:
export SYSTEMD_EDITOR="vim"
systemctl edit haproxy.service
We will add the following lines:
[Service]
ExecStartPre=
ExecStartPre=/bin/strace /usr/sbin/haproxy -f $CONFIG -c -q
Now let’s once again reboot the machine:
# reboot
The service is once again failed. Let’s take a look at the journalctl logs:
gen 28 15:10:54 myhostname strace[1199]: connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.248.239")}, 16) = -1 ENETUNREACH (Network is unreachable)
gen 28 15:10:54 myhostname strace[1199]: close(4) = 0
gen 28 15:10:54 myhostname strace[1199]: socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 4
gen 28 15:10:54 myhostname strace[1199]: setsockopt(4, SOL_IP, IP_RECVERR, [1], 4) = 0
gen 28 15:10:54 myhostname strace[1199]: connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.248.239")}, 16) = -1 ENETUNREACH (Network is unreachable)
gen 28 15:10:54 myhostname strace[1199]: close(4) = 0
gen 28 15:10:54 myhostname strace[1199]: socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 4
gen 28 15:10:54 myhostname strace[1199]: setsockopt(4, SOL_IP, IP_RECVERR, [1], 4) = 0
gen 28 15:10:54 myhostname strace[1199]: connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.248.239")}, 16) = -1 ENETUNREACH (Network is unreachable)
gen 28 15:10:54 myhostname strace[1199]: close(4) = 0
gen 28 15:10:54 myhostname strace[1199]: rt_sigprocmask(SIG_BLOCK, [HUP USR1 USR2 PIPE ALRM CHLD TSTP URG VTALRM PROF WINCH IO], [], 8) = 0
gen 28 15:10:54 myhostname strace[1199]: uname({sysname="Linux", nodename="myhostname", ...}) = 0
gen 28 15:10:54 myhostname strace[1199]: rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gen 28 15:10:54 myhostname strace[1199]: exit_group(1) = ?
gen 28 15:10:54 myhostname strace[1199]: +++ exited with 1 +++
gen 28 15:10:54 myhostname systemd[1]: haproxy.service: Control process exited, code=exited status=1
gen 28 15:10:54 myhostname systemd[1]: haproxy.service: Failed with result 'exit-code'.
gen 28 15:10:54 myhostname systemd[1]: Failed to start HAProxy Load Balancer.
Clearly, it cannot resolve it’s own name, and this seems to be the problem.
We can verify this by having haproxy NOT fail upon missed DNS resolution in haproxy.cfg, like so:
global
chroot /var/lib/haproxy
log /dev/log local0
...
defaults
# never fail on address resolution
default-server init-addr last,libc,none
log global
...
Now, let us reboot once again.
Now, we can see that the haproxy service is running:
# systemctl status haproxy.service
...
Active: active (running) since Tue 2025-01-28 15:15:35 CET; 18s ago
...
So, why does this happen?
If we take a look at another service’s unit file, for example keepalived’s service unit:
# cat /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
It is waiting for network-online.target before starting.
Instead, our HAProxy service is waiting for network.target:
# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target
To confirm this is the problem, we can override the service configuration like so:
# export SYSTEMD_EDITOR="vim"
# systemctl edit haproxy.service
[Unit]
After=network-online.target
Wants=network-online.target
With this configuration, upon reboot HAProxy will happily start.
I hope this article helps someone with the same problem save some time.
@perconafriends do you think such a modification would be possible/a good default configuration for the haproxy package?
It seems the haproxy package is reused across projects so I don’t have the complete picture in mind right now.
More information can be found on the various network targets here NetworkTarget
Best regards!