Bug in entry entrypoint.sh in docker image based setup?

Martin_Laager · January 12, 2023, 10:55am

Hey.

Did some testing with percona-xtradb-cluster:8.0 docker image setup with 3 hosts.
Got it bootstrapped and running (wsrep_cluster_size was 3 and wsrep_local_state_comment was synced) and also imported some sql dump for failover testing.

All was good until my 2 last tests.

Shut down container in the last host (node3). And deleted grastate.dat to test SST (case when something happens with one node and i want to get it synced from donor).
This test passed, it found the donor and synced data via SST and cluster was OK again.
Shut down container in node 3 again but this time deleted all data from data dir.
When I started the container it also found the donor and started the SST process, but at the same time some init process was started that looped from 120->0 and then exited. Which killed the whole mysql container including the SST process which did not finish. This process that ends with exit happens every time i’l try to start the container.

docker compose logs for speciffic subject

content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.115103Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on JOINER side (wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.193.163.32:4444' --datadir '/var/lib/mysql/' --basedir '/
usr/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '77' --mysqld-version '8.0.30-22.1'   '' )
content-api-v3-mysql-mysql-1  | + for i in {120..0}
content-api-v3-mysql-mysql-1  | ++ mysql --protocol=socket -uroot -hlocalhost --socket=/tmp/mysql.sock --password= -s
content-api-v3-mysql-mysql-1  | ++ echo 'SELECT variable_value FROM performance_schema.global_status WHERE variable_name='\''wsrep_local_state_comment'\'''
content-api-v3-mysql-mysql-1  | + wsrep_local_state=
content-api-v3-mysql-mysql-1  | + true
content-api-v3-mysql-mysql-1  | + '[' '' = Synced ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process in progress...'
content-api-v3-mysql-mysql-1  | MySQL init process in progress...
content-api-v3-mysql-mysql-1  | + sleep 1
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.912910Z 1 [Note] [MY-000000] [WSREP] Prepared SST request: xtrabackup-v2|10.193.163.32:4444/xtrabackup_sst//1
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.912981Z 1 [Note] [MY-000000] [Galera] Check if state gap can be serviced using IST
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.913275Z 1 [Note] [MY-000000] [Galera] Local UUID: 00000000-0000-0000-0000-000000000000 != Group UUID: a136dbb6-90dd-11ed-b937-87c4dfb409b4
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.913308Z 1 [Note] [MY-000000] [Galera] ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 42914, STRv: 3
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.913992Z 1 [Note] [MY-000000] [Galera] IST receiver addr using ssl://10.193.163.32:4568
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.914065Z 1 [Note] [MY-000000] [Galera] IST receiver bind using ssl://0.0.0.0:4568
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.914441Z 1 [Note] [MY-000000] [Galera] IST receiver using ssl
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.914758Z 1 [Note] [MY-000000] [Galera] Prepared IST receiver for 0-42914, listening at: ssl://0.0.0.0:4568
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.915899Z 0 [Note] [MY-000000] [Galera] Member 1.0 (10.193.163.32) requested state transfer from '*any*'. Selected 0.0 (10.193.163.31)(SYNCED) as donor.
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.916083Z 0 [Note] [MY-000000] [Galera] Shifting PRIMARY -> JOINER (TO: 42914)
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.916363Z 1 [Note] [MY-000000] [Galera] Requesting state transfer: success, donor: 0
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.916385Z 1 [Note] [MY-000000] [Galera] Resetting GCache seqno map due to different histories.
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:12.916394Z 1 [Note] [MY-000000] [Galera] GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> a136dbb6-90dd-11ed-b937-87c4dfb409b4:42914
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:13.755016Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
content-api-v3-mysql-mysql-1  | 2023-01-12T10:12:13.776483Z 0 [Note] [MY-000000] [WSREP-SST] ............Waiting for SST streaming to complete!
content-api-v3-mysql-mysql-1  | + for i in {120..0}
content-api-v3-mysql-mysql-1  | ++ echo 'SELECT variable_value FROM performance_schema.global_status WHERE variable_name='\''wsrep_local_state_comment'\'''
content-api-v3-mysql-mysql-1  | ++ mysql --protocol=socket -uroot -hlocalhost --socket=/tmp/mysql.sock --password= -s
content-api-v3-mysql-mysql-1  | + wsrep_local_state=
content-api-v3-mysql-mysql-1  | + true
content-api-v3-mysql-mysql-1  | + '[' '' = Synced ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process in progress...'
content-api-v3-mysql-mysql-1  | MySQL init process in progress...
content-api-v3-mysql-mysql-1  | + sleep 1
content-api-v3-mysql-mysql-1  | + for i in {120..0}
content-api-v3-mysql-mysql-1  | ++ echo 'SELECT variable_value FROM performance_schema.global_status WHERE variable_name='\''wsrep_local_state_comment'\'''
content-api-v3-mysql-mysql-1  | ++ mysql --protocol=socket -uroot -hlocalhost --socket=/tmp/mysql.sock --password= -s
content-api-v3-mysql-mysql-1  | + wsrep_local_state=
content-api-v3-mysql-mysql-1  | + true
content-api-v3-mysql-mysql-1  | + '[' '' = Synced ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process in progress...'
content-api-v3-mysql-mysql-1  | + sleep 1
content-api-v3-mysql-mysql-1  | MySQL init process in progress...
content-api-v3-mysql-mysql-1  | + for i in {120..0}
content-api-v3-mysql-mysql-1  | ++ echo 'SELECT variable_value FROM performance_schema.global_status WHERE variable_name='\''wsrep_local_state_comment'\'''
content-api-v3-mysql-mysql-1  | ++ mysql --protocol=socket -uroot -hlocalhost --socket=/tmp/mysql.sock --password= -s
content-api-v3-mysql-mysql-1  | + wsrep_local_state=
content-api-v3-mysql-mysql-1  | + true
content-api-v3-mysql-mysql-1  | + '[' '' = Synced ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process in progress...'
content-api-v3-mysql-mysql-1  | MySQL init process in progress...
content-api-v3-mysql-mysql-1  | + sleep 1
content-api-v3-mysql-mysql-1  | + for i in {120..0}
content-api-v3-mysql-mysql-1  | ++ echo 'SELECT variable_value FROM performance_schema.global_status WHERE variable_name='\''wsrep_local_state_comment'\'''
content-api-v3-mysql-mysql-1  | ++ mysql --protocol=socket -uroot -hlocalhost --socket=/tmp/mysql.sock --password= -s
content-api-v3-mysql-mysql-1  | + wsrep_local_state=
content-api-v3-mysql-mysql-1  | + true
content-api-v3-mysql-mysql-1  | + '[' '' = Synced ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process in progress...'
content-api-v3-mysql-mysql-1  | + sleep 1
content-api-v3-mysql-mysql-1  | MySQL init process in progress...
content-api-v3-mysql-mysql-1  | + for i in {120..0}
content-api-v3-mysql-mysql-1  | ++ echo 'SELECT variable_value FROM performance_schema.global_status WHERE variable_name='\''wsrep_local_state_comment'\'''
content-api-v3-mysql-mysql-1  | ++ mysql --protocol=socket -uroot -hlocalhost --socket=/tmp/mysql.sock --password= -s
content-api-v3-mysql-mysql-1  | + wsrep_local_state=
content-api-v3-mysql-mysql-1  | + true
content-api-v3-mysql-mysql-1  | + '[' '' = Synced ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process in progress...'
content-api-v3-mysql-mysql-1  | MySQL init process in progress...
content-api-v3-mysql-mysql-1  | + sleep 1
......
content-api-v3-mysql-mysql-1  | + '[' 0 = 0 ']'
content-api-v3-mysql-mysql-1  | + echo 'MySQL init process failed.'
content-api-v3-mysql-mysql-1  | MySQL init process failed.
content-api-v3-mysql-mysql-1  | + exit 1
content-api-v3-mysql-mysql-1 exited with code 1

Is there a problem in my setup or there is some bug in entrypoint.sh script ?

Evgeniy_Patlan · January 13, 2023, 9:33am

Hi @Martin_Laager
Thanks for your report.
I suggest you creating ticket in jira.percona.com in Percona-XtraDB-Cluster project and we will investigate this issue

Ivaylo_Korakov · February 28, 2023, 7:34pm

Hi,

I have been doing some digging around and came across your issue with remarkably similar behaviour from the init process. Just for someone in the future here is a link to that one:
https://forums.percona.com/t/pxc-cluster-not-starting-in-kind/20042/12