Logrotate fails after the first day on new version 8.0.35-27.1

MySQL & MariaDB Percona XtraDB Cluster 8.x
,
1

After using the newly released versions of the XtraDB Operator and Cluster for the first day, I encountered an issue where the logs would not rotate anymore. Here are the versions I deployed:

  • XtraDB Operator: percona/percona-xtradb-cluster-operator:1.14.0
  • XtraDB Cluster: percona/percona-xtradb-cluster:8.0.35-27.1
  • Logs/Logcollector: percona/percona-xtradb-cluster-operator:1.14.0-logcollector

I deployed these using the pxc-db and pxc-operator helm charts from the official Percona chart repo.

Initially, the logcollector worked as expected, forwarding the collected logs to stdout, allowing our Grafana/Loki system to pick them up. However, after one day, we encountered the following error:

[logrotate] executing: sh -c logrotate -s /opt/percona/logrotate/logrotate.status /opt/percona/logrotate/logrotate-mysql.conf;/usr/bin/find /var/lib/mysql/ -name GRA_*.log -mtime +7 -delete
[logrotate] error: error running shared postrotate script for '/var/lib/mysql/[!G]*.log '

Upon inspecting the postrotate script, I found:

cat /usr/local/bin/postrotate-mysql.sh
#!/bin/bash

set -o errexit

PXC_SERVER_PORT='3306'
MONITOR_USER='monitor'
TIMEOUT=10
MYSQL_CMDLINE="/usr/bin/timeout $TIMEOUT /usr/bin/mysql -nNE -u$MONITOR_USER -h127.0.0.1 -P$PXC_SERVER_PORT"

export MYSQL_PWD=${MONITOR_PASSWORD}

# Check if the audit plugin is loaded
audit_plugin_loaded=$($MYSQL_CMDLINE -e "SHOW PLUGINS" | grep -c 'audit_log' || true)
if [ $audit_plugin_loaded -gt 0 ]; then
    $MYSQL_CMDLINE -e 'FLUSH NO_WRITE_TO_BINLOG ERROR LOGS;SET GLOBAL audit_log_flush=1;'
else
    $MYSQL_CMDLINE -e 'FLUSH NO_WRITE_TO_BINLOG ERROR LOGS;'
fi

However, I did not identify any obvious issues with it. The monitor password is correctly set. It’s possible that a timeout occurred, but it’s difficult to say for sure without more information. Are there any more error logs I could look up?

Following this error, the /var/lib/mysql/mysqld-error.log file, the one without number suffix, was deleted. As a result, the logcollector no longer printed logs to stdout.

Is this a recognized issue with the new version? I have made no changes to the database configurations.

Thank you for your help.

2

Hi @Philipp_Malkmus, I can’t reproduce this issue :frowning: Could you please provide your CR? Maybe you have some specific configuration.

3

Thank you for trying to help.
You mean the XtraDBCluster Custom Ressource? Sure, here it is.
cr.txt

4

@Philipp_Malkmus I see the problem, and it does not connect with the latest release. I think we had this bug when we added support for proxy protocol. The postrotate-mysql.sh does not support it. Please create the task, and we will fix it.

5

You can remove proxy_protocol_networks option and script will work.

6

Ooh, interesting. I must have overlooked that since on our other cluster we don’t use proxy protocol because it also does not seem to work with kubectl port forwarding.
Thanks a lot!

How do I create a task for you?

7

Please use this jira project Jira - Percona JIRA

8

@Philipp_Malkmus, I have fixed it. K8SPXC-1364 adopt postrotate-mysql.sh for proxy protocol by hors · Pull Request #947 · percona/percona-docker · GitHub You can build a custom image for logcollector and use it with proxy protocol.