If sometimes the Bootstrap starts but the other node doesnt start mysql
Jun 02 11:46:06 morpheus-db-01node.domain.local setroubleshoot[4756]: AnalyzeThread.run(): Set alarm timeout to 10
Jun 02 11:46:11 morpheus-db-01node.domain.local mysql-systemd[4383]: /usr/bin/mysql-systemd: line 140: kill: (4382) - No such process
Jun 02 11:46:11 morpheus-db-01node.domain.local mysql-systemd[4383]: ERROR! mysqld_safe with PID 4382 has already exited: FAILURE
Jun 02 11:46:11 morpheus-db-01node.domain.local systemd[1]: mysql@bootstrap.service: Control process exited, code=exited status=1
Jun 02 11:46:11 morpheus-db-01node.domain.local mysql-systemd[4777]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Jun 02 11:46:11 morpheus-db-01node.domain.local mysql-systemd[4777]: WARNING: mysql may be already dead
Jun 02 11:46:11 morpheus-db-01node.domain.local systemd[1]: mysql@bootstrap.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit mysql@bootstrap.service has entered the 'failed' state with result 'exit-code'.
Jun 02 11:46:11 morpheus-db-01node.domain.local systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
my.cnf file
[client]
socket=/var/lib/mysql/mysql.sock
[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem
[mysqld]
server-id=2
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800
######## wsrep ###############
# Certificate
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”
# Path to Galera library
wsrep_provider=/usr/lib64/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
wsrep_cluster_address=gcomm://192.168.0.61,192.168.0.62,192.168.0.63
# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW
# Slave thread to use
wsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2
# Node IP address
wsrep_node_address=192.168.0.62
# Cluster name
wsrep_cluster_name=morpheusdb-cluster
#If wsrep_node_name is not specified, then system hostname will be used
wsrep_node_name=morpheus-db-02node
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=PERMISSIVE
# SST method
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sstuser:Password123!
Please help
1 Like
Need to see mysql’s error log, not the output from systemctl
1 Like
2021-06-02T12:51:37.997202Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-06-02T12:51:37.999296Z 0 [Warning] WSREP: Node is not a cluster node. Disabling pxc_strict_mode
2021-06-02T12:51:38.351380Z 0 [Warning] InnoDB: New log files created, LSN=2755557
2021-06-02T12:51:38.424979Z 0 [Warning] CA certificate ca.pem is self signed.
2021-06-02T12:51:38.434866Z 0 [Warning] WSREP: Initial position was provided by configuration or SST. Avoid overriding this position.
2021-06-02T12:57:25.509953Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-06-02T12:57:25.524259Z 0 [Warning] WSREP: Node is not a cluster node. Disabling pxc_strict_mode
2021-06-02T12:57:25.734472Z 0 [Warning] CA certificate ca.pem is self signed.
2021-06-02T12:57:25.803910Z 0 [Warning] WSREP: Initial position was provided by configuration or SST. Avoid overriding this position.
1 Like
Hi @sghose
It is only a small part of log which contain only warning messages.
Could you please attach more log context. Also as it is issue with systemd please attach syslog or messages?
1 Like
I re-installed all the 3 nodes but bootstrap failed, set seLinux to permissive
[root@morpheus-db-01node mysql]# sealert -l 8c39c1f5-5475-4479-9194-72d0a6cf9276 | less
auditctl -w /etc/shadow -p w
Try to recreate AVC. Then execute
ausearch -m avc -ts recent
If you see PATH record check ownership/permissions on file, and fix it,
otherwise report as a bugzilla.
***** Plugin catchall (9.59 confidence) suggests **************************
If you believe that mysqld should have the dac_override capability by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
ausearch -c ‘mysqld’ --raw | audit2allow -M my-mysqld
semodule -X 300 -i my-mysqld.pp
Additional Information:
Source Context system_u:system_r:mysqld_t:s0
Target Context system_u:system_r:mysqld_t:s0
Target Objects Unknown [ capability ]
Source mysqld
Source Path mysqld
Port
Host morpheus-db-01node.domain.local
Source RPM Packages
Target RPM Packages
SELinux Policy RPM selinux-policy-targeted-3.14.3-67.el8.noarch
Local Policy RPM selinux-policy-targeted-3.14.3-67.el8.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Permissive
Host Name morpheus-db-01node.domain.local
Platform Linux morpheus-db-01node.domain.local
4.18.0-305.el8.x86_64 #1 SMP Thu Apr 29 08:54:30
EDT 2021 x86_64 x86_64
Alert Count 2
First Seen 2021-06-03 10:26:31 EDT
Last Seen 2021-06-03 10:51:30 EDT
Local ID 8c39c1f5-5475-4479-9194-72d0a6cf9276
Raw Audit Messages
type=AVC msg=audit(1622731890.964:232): avc: denied { dac_override } for pid=5343 comm=“mysqld” capability=1 scontext=system_u:system_r:mysqld_t:s0 tcontext=system_u:system_r:mysqld_t:s0 tclass=capability permissive=1
Hash: mysqld,mysqld_t,mysqld_t,capability,dac_override
1 Like
As soon as I mv the my.cnf file to a my.cnf.bak in all the nodes the mysql starts fine… So is something wrong in the my.cnf file
2 Likes