# ERROR! MySQL (Percona XtraDB Cluster) is not running, but PID file exists **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [December 23, 2015, 5:41pm UTC](https://forums.percona.com/t/error-mysql-percona-xtradb-cluster-is-not-running-but-pid-file-exists/4631 "2015-12-23T17:41:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![adrci](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@adrci](https://forums.percona.com/u/adrci)\ **Post date:** [December 23, 2015, 5:41pm UTC](https://forums.percona.com/t/error-mysql-percona-xtradb-cluster-is-not-running-but-pid-file-exists/4631/1 "2015-12-23T17:41:42Z")
I am trying to startup node two of a three node percona cluster. I am by no means a mysql database administrator, but maybe someone can identify what happened based on this error: * * * Thread pointer: 0x7f483c000990 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong… stack\_bottom = 7f48523b8988 thread\_stack 0x40000 /usr/sbin/mysqld(my\_print\_stacktrace+0x35)[0x7ee025] /usr/sbin/mysqld(handle\_fatal\_signal+0x4b4)[0x6c0db4] /lib64/libpthread.so.0(+0xf710)[0x7f4865b44710] /lib64/libc.so.6(gsignal+0x35)[0x7f48641a0625] /lib64/libc.so.6(abort+0x175)[0x7f48641a1e05] /usr/sbin/mysqld[0x8f7239] /usr/sbin/mysqld[0x8f85b9] /usr/sbin/mysqld[0x8fc28b] /usr/sbin/mysqld[0x8fc53b] /usr/sbin/mysqld[0x82eaf3] /usr/sbin/mysqld[0x831fe4] /usr/sbin/mysqld[0x801c74] /usr/sbin/mysqld(\_ZN14Rows\_log\_event8find\_rowEPK14Relay\_log\_info+0x1e4)[0x766574] /usr/sbin/mysqld(\_ZN21Update\_rows\_log\_event11do\_exec\_rowEPK14Relay\_log\_info+0xa5)[0x766a95] /usr/sbin/mysqld(\_ZN14Rows\_log\_event14do\_apply\_eventEPK14Relay\_log\_info+0x267)[0x76d1c7] /usr/sbin/mysqld(\_Z14wsrep\_apply\_cbPvPKvmjPK14wsrep\_trx\_meta+0x6a5)[0x67d0b5] /usr/lib64/libgalera\_smm.so(+0x1a3699)[0x7f48613ce699] /usr/lib64/libgalera\_smm.so(\_ZN6galera13ReplicatorSMM9apply\_trxEPvPNS\_9TrxHandleE+0x273)[0x7f48613cfc13] /usr/lib64/libgalera\_smm.so(\_ZN6galera13ReplicatorSMM11process\_trxEPvPNS\_9TrxHandleE+0x45)[0x7f48613d04e5] /usr/lib64/libgalera\_smm.so(\_ZN6galera15GcsActionSource8dispatchEPvRK10gcs\_actionRb+0x2dc)[0x7f48613aa86c] /usr/lib64/libgalera\_smm.so(\_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x7f48613aaf13] /usr/lib64/libgalera\_smm.so(\_ZN6galera13ReplicatorSMM10async\_recvEPv+0x93)[0x7f48613ca1e3] /usr/lib64/libgalera\_smm.so(galera\_recv+0x23)[0x7f48613dd5d3] /usr/sbin/mysqld[0x67df11] /usr/sbin/mysqld(start\_wsrep\_THD+0x2ee)[0x5215de] /lib64/libpthread.so.0(+0x79d1)[0x7f4865b3c9d1] /lib64/libc.so.6(clone+0x6d)[0x7f48642568fd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 2 Status: NOT\_KILLED You may download the Percona XtraDB Cluster operations manual by visiting [url][http://www.percona.com/software/percona-xtradb-cluster/[/url]](http://www.percona.com/software/percona-xtradb-cluster/%5B/url%5D). You may find information in the manual which will help you identify the cause of the crash. 151116 01:03:18 mysqld\_safe Number of processes running now: 0 151116 01:03:18 mysqld\_safe WSREP: not restarting wsrep node automatically 151116 01:03:18 mysqld\_safe mysqld from pid file /var/lib/mysql/web.web.com .pid ended * * * Looking at the database service on node 2: I looked at the service [root@db subsys]# ps ax | grep mysql 25218 pts/0 S+ 0:00 grep mysql * * * When I try to start the service back up, I get this error message: [root@db2 mysql]# sudo /etc/init.d/mysql start ERROR! MySQL (Percona XtraDB Cluster) is not running, but PID file exists * * * I could not find the specific answer how to fix this issue online (only regarding locked PIDs). Does anyone know what I should do to fix this issue? Thanks! ---
**Author:** ![jrivera](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jrivera/32/13_2.png) [@jrivera](https://forums.percona.com/u/jrivera)\ **Post date:** [December 23, 2015, 10:18pm UTC](https://forums.percona.com/t/error-mysql-percona-xtradb-cluster-is-not-running-but-pid-file-exists/4631/2 "2015-12-23T22:18:52Z")
provide output of “SHOW STATUS LIKE ‘wsrep%’” from bootstrapped node, also attach a copy of my.cnf and full error log from node2 (use pastebin or gist) ---
**Author:** ![adrci](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@adrci](https://forums.percona.com/u/adrci)\ **Post date:** [December 24, 2015, 9:45am UTC](https://forums.percona.com/t/error-mysql-percona-xtradb-cluster-is-not-running-but-pid-file-exists/4631/3 "2015-12-24T09:45:05Z")
[TABLE=“class: table\_results ajax pma\_table”] [TR] [TD] [/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_local\_state\_uuid[/TD] [TD=“class: data text”]a8e8a277-6f03-11e2-0800-5896d9f10d3c[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_protocol\_version[/TD] [TD=“class: data text”]4[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_last\_committed[/TD] [TD=“class: data text”]93828567[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_replicated[/TD] [TD=“class: data text”]3305034[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_replicated\_bytes[/TD] [TD=“class: data text”]7215828937[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_received[/TD] [TD=“class: data text”]4690946[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_received\_bytes[/TD] [TD=“class: data text”]10461552800[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_local\_commits[/TD] [TD=“class: data text”]3300961[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_local\_cert\_failures[/TD] [TD=“class: data text”]3027[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_local\_replays[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_local\_send\_queue[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_local\_send\_queue\_avg[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_local\_recv\_queue[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_local\_recv\_queue\_avg[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_flow\_control\_paused[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_flow\_control\_sent[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_flow\_control\_recv[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_cert\_deps\_distance[/TD] [TD=“class: data text”]87.147799[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_apply\_oooe[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_apply\_oool[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_apply\_window[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_commit\_oooe[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_commit\_oool[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_commit\_window[/TD] [TD=“class: data text”]0.000000[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_local\_state[/TD] [TD=“class: data text”]4[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_local\_state\_comment[/TD] [TD=“class: data text”]Synced[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_cert\_index\_size[/TD] [TD=“class: data text”]1293[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_causal\_reads[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_incoming\_addresses[/TD] [TD=“class: data text”]111.111.111.11:1111,111.111.111.111:111[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_cluster\_conf\_id[/TD] [TD=“class: data text”]15[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_cluster\_size[/TD] [TD=“class: data text”]2[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_cluster\_state\_uuid[/TD] [TD=“class: data text”]a8e8a277-6f03-11e2-0800-5896d9f10d3c[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_cluster\_status[/TD] [TD=“class: data text”]Primary[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_connected[/TD] [TD=“class: data text”]ON[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_local\_bf\_aborts[/TD] [TD=“class: data text”]181[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_local\_index[/TD] [TD=“class: data text”]0[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_provider\_name[/TD] [TD=“class: data text”]Galera[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_provider\_vendor[/TD] [TD=“class: data text”]Codership Oy \[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_provider\_version[/TD] [TD=“class: data text”]2.11(r318911d)[/TD] [/TR] [TR=“class: even”] [TD=“class: data not\_null text”]wsrep\_ready[/TD] [TD=“class: data text”]ON[/TD] [/TR] [TR=“class: odd”] [TD=“class: data not\_null text”]wsrep\_thread\_count[/TD] [TD=“class: data text”]9[/TD] [/TR] [/TABLE] [root@db02 etc]# cat my.cnf # The following options will be passed to all MySQL clients [client] #password = ################ port = 3306 socket = /var/lib/mysql/mysql.sock # The MySQL server [mysqld] port = 3306 socket = /var/lib/mysql/mysql.sock # Set up SSL ssl-ca = /etc/mysql/certs/ca-cert.pem ssl-cert = /etc/mysql/certs/server-cert.pem ssl-key = /etc/mysql/certs/server-key.pem ssl-cipher = AES256-GCM-SHA384:AES256-SHA256:AES256-SHA skip-external-locking log-error = /var/log/mysql/mysqld.log slow\_query\_log\_file = /var/log/mysql/mysql-slow.log key\_buffer\_size = 32M max\_allowed\_packet = 16M table\_open\_cache = 2048 thread\_cache\_size = 50 query\_cache\_type = 0 query\_cache\_size = 0 table\_definition\_cache = 4096 tmp\_table\_size = 32M max\_heap\_table\_size = 32M long\_query\_time = 2 slow\_query\_log = 1 wait\_timeout = 30 interactive\_timeout = 300 max\_connections = 450 open\_files\_limit = 65535 innodb\_stats\_on\_metadata = 0 # Percona Settings wsrep\_cluster\_address = gcomm:(deleted this part) wsrep\_provider = /usr/lib64/libgalera\_smm.so wsrep\_slave\_threads = 8 wsrep\_sst\_method = xtrabackup-v2 wsrep\_sst\_auth = wsrep\_cluster\_name = the\_web wsrep\_node\_name = web2 wsrep\_provider\_options = “pc.weight=2;socket.ssl\_cert=/etc/mysql/certs/server-cert.pem;socket.ssl\_key=/etc/mysql/certs/server-key.pem;socket.ssl\_ca=/etc/mysql/certs/ca-cert.pem;” innodb\_autoinc\_lock\_mode = 2 innodb\_locks\_unsafe\_for\_binlog = 1 datadir = /var/lib/mysql binlog\_format=ROW log\_slave\_updates # Replication Master Server (default) # binary logging is required for replication #log-bin=mysql-bin # required unique id between 1 and 2^32 - 1 # defaults to 1 if master-host is not set # but will not function as a master if omitted server-id = 2 innodb\_buffer\_pool\_size = 10G innodb\_log\_file\_size = 256M [mysqldump] quick max\_allowed\_packet = 16M [mysql] no-auto-rehash # Remove the next comment character if you are not familiar with SQL #safe-updates [myisamchk] key\_buffer\_size = 256M sort\_buffer\_size = 256M read\_buffer = 2M write\_buffer = 2M [mysqlhotcopy] interactive-timeout [sst] encrypt=3 tkey=/etc/mysql/certs/server-key.pem tcert=/etc/mysql/certs/server-cert.pem ---
**Author:** ![adrci](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@adrci](https://forums.percona.com/u/adrci)\ **Post date:** [December 24, 2015, 10:18am UTC](https://forums.percona.com/t/error-mysql-percona-xtradb-cluster-is-not-running-but-pid-file-exists/4631/4 "2015-12-24T10:18:30Z")
Issue was solved by renaming the PID file: [root@db2 mysql]# cd /var/lib/mysql [root@db2 mysql]# ps -ef | grep 6511 [root@db2 mysql]# fuser -u web2.web.com.pid [root@db2 mysql]# mv web2.web.com.pid web2.web.com.pid.old [root@db2 mysql]# sudo /etc/init.d/mysql start