ProxySQL removed all servers from it's database

Hello,

ProxySQL removed all servers from it’s database and added them again 2 days later automatically.
The environment is a Kubenetes Cluster with one Control Plane and three Worker Nodes (v1.24.2) and
pxc-operator (1.11.0).

This is the log when removing started:

Oct 9, 2022 @ 13:34:06.173	2022-10-09 11:34:06 MySQL_Monitor.cpp:1840:monitor_galera_thread(): [ERROR] Got error. mmsd 0x7f9cdf657080 , MYSQL 0x7f9cdea07d00 , FD 45 : The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.171	2022-10-09T11:34:06.170+0000 I  NETWORK  [conn815381] end connection 127.0.0.1:49980 (17 connections now open)	unifi	 - 	 - 	10.32.239.78/32	mongodb-0

And these are the logs afterwards:

Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 [INFO] Dumping mysql_servers: ALL	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 12 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 12 , address: db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 10 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 10 , address: db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 10 , address: db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 13 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 11 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 11 , address: db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	HID: 11 , address: db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 [INFO] Dumping current MySQL Servers structures for hostgroup ALL	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 [INFO] Creating new server in HG 13 : db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com:3306 , gtid_port=0, weight=1000, status=0	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	+--------------+---------------------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	| 13           | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         | 0           | 0         | 1000   | 0      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	+--------------+---------------------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	| hostgroup_id | hostname                                                                  | port | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer | gtid_port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	+--------------+---------------------------------------------------------------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 [INFO] Dumping mysql_servers JOIN mysql_servers_incoming	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 MySQL_HostGroups_Manager.cpp:1681:commit(): [WARNING] Removed server at address 140310606971520, hostgroup 11, address db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 MySQL_HostGroups_Manager.cpp:1681:commit(): [WARNING] Removed server at address 140310606971680, hostgroup 10, address db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 MySQL_HostGroups_Manager.cpp:1681:commit(): [WARNING] Removed server at address 140310606971840, hostgroup 12, address db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	+-----------------+--------------+---------------------------------------------------------------------------+------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	| 140310606971520 | 11           | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	| 140310606971680 | 10           | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	| 140310606971840 | 12           | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	+-----------------+--------------+---------------------------------------------------------------------------+------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	| mem_pointer     | hostgroup_id | hostname                                                                  | port |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	+-----------------+--------------+---------------------------------------------------------------------------+------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.175	2022-10-09 11:34:06 [INFO] Dumping mysql_servers LEFT JOIN mysql_servers_incoming	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+--------------+---------------------------------------------------------------------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 11           | db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000    | 1      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 13           | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000    | 0      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 10           | db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000000 | 0      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 10           | db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000    | 0      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 12           | db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000    | 0      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 11           | db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0         | 1000000 | 0      | 0           | 1000            | 0                   | 0       | 0              |         |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+--------------+---------------------------------------------------------------------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| hostgroup_id | hostname                                                                  | port | gtid_port | weight  | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+--------------+---------------------------------------------------------------------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	2022-10-09 11:34:06 [INFO] Dumping mysql_servers_incoming	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+-----+---------------------------------------------------------------------------+------+------+---------+--------+-----+-----------+---------+-----+---------+---------+-----------------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 11  | db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000    | 1      | 0   | 1000      | 0       | 0   | 0       |         | 140311480056352 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 11  | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000    | 1      | 0   | 1000      | 0       | 0   | 0       |         | 140310606971520 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 10  | db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000000 | 0      | 0   | 1000      | 0       | 0   | 0       |         | 140311631496448 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 10  | db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000    | 0      | 0   | 1000      | 0       | 0   | 0       |         | 140311721542272 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 10  | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000    | 0      | 0   | 1000      | 0       | 0   | 0       |         | 140310606971680 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 12  | db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000    | 0      | 0   | 1000      | 0       | 0   | 0       |         | 140311631496608 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 12  | db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000    | 0      | 0   | 1000      | 0       | 0   | 0       |         | 140310606971840 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| 11  | db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 0    | 1000000 | 0      | 0   | 1000      | 0       | 0   | 0       |         | 140310689678592 |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+-----+---------------------------------------------------------------------------+------+------+---------+--------+-----+-----------+---------+-----+---------+---------+-----------------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| hid | hostname                                                                  | port | gtid | weight  | status | cmp | max_conns | max_lag | ssl | max_lat | comment | mem_pointer     |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+-----+---------------------------------------------------------------------------+------+------+---------+--------+-----+-----------+---------+-----+---------+---------+-----------------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	2022-10-09 11:34:06 [INFO] Dumping mysql_servers: ALL	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 12 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 12 , address: db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 10 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 10 , address: db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 10 , address: db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 11 , address: db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 11 , address: db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	HID: 11 , address: db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com , port: 3306 , gtid_port: 0 , weight: 1000000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	2022-10-09 11:34:06 [INFO] Dumping current MySQL Servers structures for hostgroup ALL	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+---------------------------------------------------------------------------+------+------------------+------------+-------------------+-----------+------------------------+-------------------+--------------+----------------------+---------------------------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| db-pxc-db-pxc-2.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 1665315246173774 | 0          | 0                 | 1         | 0                      | 0                 | 1            | 1                    | 1                               | 0              | The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior. |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| db-pxc-db-pxc-1.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 1665315245174091 | 10998      | 1                 | 0         | 0                      | 4                 | 0            | 0                    | 0                               | 0              |                                                                                                                                          |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| db-pxc-db-pxc-0.db-pxc-db-pxc.mysql.svc.marvin.fra1.k8s.hundertserver.com | 3306 | 1665315245173503 | 6948       | 1                 | 0         | 0                      | 4                 | 0            | 0                    | 0                               | 0              |                                                                                                                                          |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+---------------------------------------------------------------------------+------+------------------+------------+-------------------+-----------+------------------------+-------------------+--------------+----------------------+---------------------------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	| hostname                                                                  | port | start_time       | check_time | primary_partition | read_only | wsrep_local_recv_queue | wsrep_local_state | wsrep_desync | wsrep_reject_queries | wsrep_sst_donor_rejects_queries | pxc_maint_mode | error                                                                                                                                    |	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	+---------------------------------------------------------------------------+------+------------------+------------+-------------------+-----------+------------------------+-------------------+--------------+----------------------+---------------------------------+----------------+------------------------------------------------------------------------------------------------------------------------------------------+	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2
	Oct 9, 2022 @ 13:34:06.174	2022-10-09 11:34:06 [INFO] Galera: Node status changed by ProxySQL, dumping all galera nodes status:	mysql	 - 	 - 	10.32.193.197/32	db-pxc-db-proxysql-2

Also this log looks weird:

Oct 9, 2022 @ 13:34:06.559 - MySQL_HostGroups_Manager.cpp:5548:update_galera_set_writer(): [WARNING] Galera: too many writers in HG 11. Max=1, current=2
1 Like

Hi @bdriton13 welcome to the Percona forums!

By default ProxySQL will write to just one writer in your writer HG.

This looks normal for MySQL, if there is no activity on the connection then the server will terminate client connections. But it should not remove the backend from the hostgroup.
Can you share with us the logs for when the instances were removed? It isn’t clear why all servers would be evicted from runtime_mysql_servers

1 Like