Cannot start container using docker-compose.yml

I get the following error after building the application using docker compose and doing a health check.

Error logs:

2023-07-11T11:55:46.103479Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2023-07-11T11:55:46.104303Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2023-07-11T11:55:46.104372Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-25) starting as process 1
2023-07-11T11:55:46.108486Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-07-11T11:55:46.219110Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysqld: Table 'mysql.plugin' doesn't exist
2023-07-11T11:55:46.274063Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.
2023-07-11T11:55:46.274148Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.275131Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.275245Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.275312Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.275438Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.275506Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.275554Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.293780Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2023-07-11T11:55:46.305971Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2023-07-11T11:55:46.310056Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-07-11T11:55:46.310084Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-07-11T11:55:46.315784Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2023-07-11T11:55:46.315881Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist
2023-07-11T11:55:46.315890Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.
2023-07-11T11:55:46.316069Z 0 [ERROR] [MY-010326] [Server] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
2023-07-11T11:55:46.316096Z 0 [ERROR] [MY-010952] [Server] The privilege system failed to initialize correctly. For complete instructions on how to upgrade MySQL to a new version please see the 'Upgrading MySQL' section from the MySQL manual.
2023-07-11T11:55:46.316143Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-07-11T11:55:47.843700Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-25)  Percona Server (GPL), Release 25, Revision 60c9e2c5.

docker-compose.yml:

app-mysql:
    image: percona/percona-server:8.0.33-aarch64
    container_name: app-mysql
    #    restart: unless-stopped
    env_file:
      - app-env.txt
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    ports:
      - "3306:3306"
    healthcheck:
      test: [ "CMD", "mysqladmin", "--socket=/var/lib/mysql/mysql.sock", "--user=root", "--password=root",  "ping", "-h", "localhost" ]
      interval: 10s
      timeout: 20s
      retries: 10
    networks:
      app-network:
        aliases:
          - app-mysql.docker

Looks like you didn’t --initialize the database? It seems to be missing the required system tables.

Hi Matthew, thanks for the reply. Now after initialization, I get a different error which is:

2023-07-12T11:44:09.693180Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2023-07-12T11:44:09.694538Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2023-07-12T11:44:09.694627Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-25) starting as process 1
2023-07-12T11:44:09.696032Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2023-07-12T11:44:09.699977Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-07-12T11:44:09.739122Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 5.7.32-35, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2023-07-12T11:44:09.739140Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2023-07-12T11:44:10.714275Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2023-07-12T11:44:10.714530Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-07-12T11:44:10.714548Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-07-12T11:44:10.715091Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-25)  Percona Server (GPL), Release 25, Revision 60c9e2c5.

where it is asking me to update MySQL 5 but I am using the following image percona/percona-server:8.0.33-25-aarch64 which is supposed to run with MySQL 8.

Would it be possible for you to try a fresh container without compose? Just a normal docker:
https://hub.docker.com/r/percona/percona-server

I tried that, and weirdly enough, it works without issues when executing the Dockerfile. This issue is mainly related to docker-compose. Is there a way to get around the fact that our whole infrastructure uses docker-compose in the CICD pipeline for delivery?

Update:
I tried with percona/percona-server:8.0.32, and the issue persists. It seems to be an InnoDB related issue when using docker compose. Even after building it with buildx and push the image to docker hub and then pulling for usage, the issue persists.

It’s not an InnoDB issue. The issue is with however compose creates/initializes containers vs how regular docker does it. The fact that you used the same image in both cases proves that; docker works, compose doesn’t. So there’s something fishy going on with compose that is different vs normal docker. I’ll highlight this post with Percona’s docker people and see if anyone has experience with compose for hints.

Worked fine for me. My version of compose didn’t support healthcheck, so I removed that part. I also removed the command: you had since those are the defaults, no point in specifying them again, and I just used basic networking.

$ docker-compose --version
docker-compose version 1.24.1, build 4667896b

$ cat docker-compose.yml
app-mysql:
    image: percona/percona-server:8.0.33
    container_name: app-mysql
    ports:
      - "3306:3306"
    environment:
      - MYSQL_RANDOM_ROOT_PASSWORD=yes

$ docker-compose up
Pulling app-mysql (percona/percona-server:8.0.33)...
8.0.33: Pulling from percona/percona-server
56f7a0abdb77: Pull complete
863d7efe4e31: Pull complete
800b80b4a09e: Pull complete
7c8b01f72d04: Pull complete
0df8274d000d: Pull complete
e58f774faeea: Pull complete
Digest: sha256:2a3d9ef658cc7ea90040ad38f8614c57125a5d5d9c053c7fb3369a63f1c56b40
Status: Downloaded newer image for percona/percona-server:8.0.33
Creating app-mysql ... done
Attaching to app-mysql
app-mysql    | Initializing database
app-mysql    | 2023-07-13T14:47:46.399940Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
app-mysql    | 2023-07-13T14:47:46.400093Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.33-25) initializing of server in progress as process 14
app-mysql    | 2023-07-13T14:47:46.452698Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
app-mysql    | 2023-07-13T14:47:51.908190Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
app-mysql    | 2023-07-13T14:48:09.631776Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
app-mysql    | Database initialized
app-mysql    | MySQL init process in progress...
app-mysql    | 2023-07-13T14:48:42.190144Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
app-mysql    | 2023-07-13T14:48:42.191399Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
app-mysql    | 2023-07-13T14:48:42.191455Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-25) starting as process 61
app-mysql    | 2023-07-13T14:48:42.299492Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
app-mysql    | 2023-07-13T14:48:42.901979Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
app-mysql    | MySQL init process in progress...
app-mysql    | MySQL init process in progress...
app-mysql    | 2023-07-13T14:48:44.154539Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
app-mysql    | 2023-07-13T14:48:44.154577Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
app-mysql    | 2023-07-13T14:48:44.215711Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/lib/mysql/mysqlx.sock
app-mysql    | 2023-07-13T14:48:44.215840Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-25'  socket: '/var/lib/mysql/mysql.sock'  port: 0  Percona Server (GPL), Release 25, Revision 60c9e2c5.
app-mysql    | mysql: [Warning] Using a password on the command line interface can be insecure.
app-mysql    | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
app-mysql    | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
app-mysql    | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
app-mysql    | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
app-mysql    | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
app-mysql    | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
app-mysql    | GENERATED ROOT PASSWORD: YviNS4kYgid3xTEGUhTYxOjIzABx
app-mysql    | mysql: [Warning] Using a password on the command line interface can be insecure.
app-mysql    |
app-mysql    | 2023-07-13T14:48:48.837989Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.33-25).
app-mysql    | 2023-07-13T14:48:57.642725Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-25)  Percona Server (GPL), Release 25, Revision 60c9e2c5.
app-mysql    |
app-mysql    | MySQL init process done. Ready for start up.
app-mysql    |
app-mysql    | 2023-07-13T14:48:57.927626Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
app-mysql    | 2023-07-13T14:48:57.928854Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
app-mysql    | 2023-07-13T14:48:57.928911Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-25) starting as process 1
app-mysql    | 2023-07-13T14:48:57.935501Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
app-mysql    | 2023-07-13T14:48:58.901382Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
app-mysql    | 2023-07-13T14:48:59.943588Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
app-mysql    | 2023-07-13T14:48:59.943636Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
app-mysql    | 2023-07-13T14:48:59.985322Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/lib/mysql/mysqlx.sock
app-mysql    | 2023-07-13T14:48:59.985379Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-25'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Percona Server (GPL), Release 25, Revision 60c9e2c5.

Hi Matthew,
thanks a lot for the reply. The snippet you provided worked great and since I am using M1 silicon, I had to change it this:

Thanks again.