# Percona HAProxy fails on machine reboot (failure in resolving DNS names)

**URL:** <https://forums.percona.com/t/percona-haproxy-fails-on-machine-reboot-failure-in-resolving-dns-names/36385>\
**Category:** Percona XtraDB Cluster\
**Created:** [January 28, 2025, 5:00pm UTC](https://forums.percona.com/t/percona-haproxy-fails-on-machine-reboot-failure-in-resolving-dns-names/36385 "2025-01-28T17:00:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![SuperHammer](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SuperHammer](https://forums.percona.com/u/SuperHammer)\
**Post date:** [January 28, 2025, 5:00pm UTC](https://forums.percona.com/t/percona-haproxy-fails-on-machine-reboot-failure-in-resolving-dns-names/36385/1 "2025-01-28T17:00:06Z")

</div>

Pardon me if this is not the correct section, but I couldn’t find an HAProxy-specific topic to post this on.

Percona offers an HAProxy package, which can be installed with:  
yum install [https://repo.percona.com/yum/percona-release-latest.noarch.rpm](https://repo.percona.com/yum/percona-release-latest.noarch.rpm)  
percona-release setup ppg16  
dnf install percona-haproxy.x86\_64

When using it along keepalived, one may also want to enable binding to any IP address:  
sysctl -w net.ipv4.ip\_nonlocal\_bind=1  
setsebool -P haproxy\_connect\_any=1

Here is a small /etc/haproxy/haproxy.cfg that can help us reproduce the problem:

```auto
global
        chroot /var/lib/haproxy
        log /dev/log local0
        maxconn 4096
        daemon
        user haproxy
        group haproxy
        stats socket /var/lib/haproxy/stats level admin
        pidfile /var/run/haproxy.pid

defaults
        log global
        option dontlognull
        timeout http-request 60s
        timeout queue 1m
        timeout connect 60s
        timeout client 5m
        timeout server 5m
        timeout http-keep-alive 30s
        timeout check 60s
        mode http
        balance source

listen my-listener-here
  bind 192.168.248.239:6443
  mode tcp
  server master1 my-backend:6443 check inter 1s

```

If we run a `systemctl status haproxy.service` we can see the service is **running**.

Now let’s restart the machine:

```auto
# reboot

```

Now, upon the machine’s reboot, we can see the status is now **failed** :

```auto
# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/haproxy.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Tue 2025-01-28 15:07:26 CET; 15s ago
  Process: 1195 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=1/FAILURE)

gen 28 15:07:26 myhostname systemd[1]: Starting HAProxy Load Balancer...
gen 28 15:07:26 myhostname systemd[1]: haproxy.service: Control process exited, code=exited status=1
gen 28 15:07:26 myhostname systemd[1]: haproxy.service: Failed with result 'exit-code'.
gen 28 15:07:26 myhostname systemd[1]: Failed to start HAProxy Load Balancer.

```

The **ExecStartPre** failed with status=1/exit code 1.

Let us override the service to troubleshoot the issue with strace:

```auto
export SYSTEMD_EDITOR="vim"
systemctl edit haproxy.service

```

We will add the following lines:

```auto
[Service]
ExecStartPre=
ExecStartPre=/bin/strace /usr/sbin/haproxy -f $CONFIG -c -q

```

Now let’s once again reboot the machine:

```auto
# reboot

```

The service is once again failed. Let’s take a look at the journalctl logs:

```auto
gen 28 15:10:54 myhostname strace[1199]: connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.248.239")}, 16) = -1 ENETUNREACH (Network is unreachable)
gen 28 15:10:54 myhostname strace[1199]: close(4) = 0
gen 28 15:10:54 myhostname strace[1199]: socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 4
gen 28 15:10:54 myhostname strace[1199]: setsockopt(4, SOL_IP, IP_RECVERR, [1], 4) = 0
gen 28 15:10:54 myhostname strace[1199]: connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.248.239")}, 16) = -1 ENETUNREACH (Network is unreachable)
gen 28 15:10:54 myhostname strace[1199]: close(4) = 0
gen 28 15:10:54 myhostname strace[1199]: socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 4
gen 28 15:10:54 myhostname strace[1199]: setsockopt(4, SOL_IP, IP_RECVERR, [1], 4) = 0
gen 28 15:10:54 myhostname strace[1199]: connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.248.239")}, 16) = -1 ENETUNREACH (Network is unreachable)
gen 28 15:10:54 myhostname strace[1199]: close(4) = 0
gen 28 15:10:54 myhostname strace[1199]: rt_sigprocmask(SIG_BLOCK, [HUP USR1 USR2 PIPE ALRM CHLD TSTP URG VTALRM PROF WINCH IO], [], 8) = 0
gen 28 15:10:54 myhostname strace[1199]: uname({sysname="Linux", nodename="myhostname", ...}) = 0
gen 28 15:10:54 myhostname strace[1199]: rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gen 28 15:10:54 myhostname strace[1199]: exit_group(1) = ?
gen 28 15:10:54 myhostname strace[1199]: +++ exited with 1 +++
gen 28 15:10:54 myhostname systemd[1]: haproxy.service: Control process exited, code=exited status=1
gen 28 15:10:54 myhostname systemd[1]: haproxy.service: Failed with result 'exit-code'.
gen 28 15:10:54 myhostname systemd[1]: Failed to start HAProxy Load Balancer.

```

Clearly, it cannot resolve it’s own name, and this seems to be the problem.

We can verify this by having haproxy NOT fail upon missed DNS resolution in haproxy.cfg, like so:

```auto
global
        chroot /var/lib/haproxy
        log /dev/log local0
...
defaults
        # never fail on address resolution
        default-server init-addr last,libc,none
        log global
...

```

Now, let us reboot once again.

Now, we can see that the haproxy service is **running** :

```auto
# systemctl status haproxy.service
...
   Active: active (running) since Tue 2025-01-28 15:15:35 CET; 18s ago
...

```

So, why does this happen?

If we take a look at another service’s unit file, for example keepalived’s service unit:

```auto
# cat /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target

```

It is waiting for **network-online.target** before starting.  
Instead, our HAProxy service is waiting for **network.target** :

```auto
# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target

```

To confirm this is the problem, we can override the service configuration like so:

```auto
# export SYSTEMD_EDITOR="vim"
# systemctl edit haproxy.service

```

```auto
[Unit]
After=network-online.target
Wants=network-online.target

```

With this configuration, upon reboot HAProxy will happily start.

I hope this article helps someone with the same problem save some time.

@perconafriends do you think such a modification would be possible/a good default configuration for the haproxy package?

It seems the haproxy package is reused across projects so I don’t have the complete picture in mind right now.

More information can be found on the various network targets here [NetworkTarget](https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/)

Best regards!

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [January 28, 2025, 6:28pm UTC](https://forums.percona.com/t/percona-haproxy-fails-on-machine-reboot-failure-in-resolving-dns-names/36385/2 "2025-01-28T18:28:06Z")

</div>

Hey there @SuperHammer,  
Thank you for this research/investigation. I have opened a JIRA with our team for verification as a bug fix.  
[https://perconadev.atlassian.net/browse/DISTMYSQL-503](https://perconadev.atlassian.net/browse/DISTMYSQL-503)
