Hello everybody,
i wanted to inform you guys about our experience when updating to 8.0.36 which was released a few days ago.
We did as usual a apt update && apt dist-upgrade which also installed the update of percona-server-server version 8.0.36. Soon after the upgrade we discovered that our slaves were out of sync.
Here is what seems to have happened:
- apt/dpkp installed the update
- the update overwrote our
/etc/mysql/my.cnfwith a symlink to default configuration (PS-9062 update-alternatives switches my.cnf to /etc/mysql/my.cnf.fall… · percona/percona-server@d06c470 · GitHub) - the mysql service was started with default configuration
- mysql started to write binlogs beginning from 0/1
- after some time our config management (puppet) kicked in and overwrote
/etc/mysql/my.cnfto contain the config - mysql service started to write binlogs again into the previous files
- at some point the slave replication failed because of binlog index error or something like that (don’t have the exact naming)
The mentioned change in 8.0.36 might also affect other config management systems if they used to use /etc/mysql/my.cnf as config file and don’t use symlinks.
- puppetlabs/puppetlabs-mysql/blob/main/manifests/params.pp#L176 (puppet)
- geerlingguy/ansible-role-mysql/blob/master/vars/Debian.yml#L11 (ansible)
- saltstack/salt/blob/master/doc/topics/development/conventions/formulas.rst#abstracting-static-defaults-into-a-lookup-table (salt - couldn’t find the default values only the documentation)
All of these are github links but i can’t put more than 2 links into a post.
It would be nice if next time you don’t force something like that or do it in a major version even if you don’t comply to sem versioning.
I hope others read this post before they install the update to ensure that their setup receives the needed change before the replication breaks 
Best regards,
Stephan