Startup time more than 7hrs each time post upgrade to 8.0.22 from 8.0.16 (mysql enterprise)

Hi,

We recently upgraded 900GB database with nearly 75k tables from mysql enterprise version 8.0.16 to 8.0.22. post upgrade we noticed the server startup time is about 7hrs. this is the same time that server takes each time of the stop/start.

upon checking information_schema.innodb_tablespaces, noticed all object’s SPACE_VERSION points to 8.0.16 instead of 8.0.22. so the assumption is that mysql is trying to rebuild/upgrade all objects each time of the restart and hence taking more than 7hrs for the server to come online

so as part of investigation, we had performed

mysqld --defaults-file=/etc/mysql/.my.cnf --upgrade=FORCE &

but no luck. it took still the same time to come online and so the next restart. SPACE VERSION of all objects still refers 8.0.16 instead of 8.0.22

Upgrade procedure followed at first time highlevel:

  1. shutdown server
  2. upgrade binaries to 8.0.22 (uninstall 8.0.16)
  3. bring up database server (assuming that there is not manual upgrade and server start should auto perform the upgrade post to 8.0.16) - took 7hrs to come online which is expected for the first time

upgrade procedure followed at second time (manual upgrade with force option)

  1. shutdown server
  2. use --upgrade=force option to upgrade at startup - took 7hrs to come online
  3. use mysqladmin to startup (becuase systemclt is not identifiying this startup)
  4. start server normally with systemctl - took 7hrs to come online again

Looking for assistance what might be fix and how to proceed further as there is not option to downgrade back to 8.0.16

Regards,

Hi Venug,

Could you check error.log and post/attach the startup sequence?
if numa enabled, pre populating the buffer pool can take some time (although it should be less than 7 hours) .
And also you are right that upgrading to a newer version will automatically run mysql_upgrade and it might be trying to rebiuld some tables. Let’s check last 2 startup sequences from error.log to review this

Last, let me ask you. Do you have any pre production environment or test server with the same data set size where you can test and restart freely?

Regards

1 Like