Error The server quit without updating PID file

Hello,
I have Percona XtraDB Cluster 5.6 with Ubuntu Server 14.04.1 LTS.
I have 3 nodes. Everything worked perfectly for 5 months
Today I find all 3 nodes with mysql server stopped.
When I try to start mysql service I receive this error:

  • Starting MySQL (Percona XtraDB Cluster) database server mysqld
  • The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

This error occour on all nodes.
Can you help me ?

My my.cnf file have this configuration:

MySQL Configuration

[mysql]

CLIENT

port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld]

GENERAL

user = mysql
default_storage_engine = InnoDB
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
bind-address = 0.0.0.0

MyISAM

key-buffer-size = 32M
myisam-recover = FORCE,BACKUP

SAFETY

max-allowed-packet = 16M
max-connect-errors = 1000000

DATA STORAGE

#basedir = /usr
datadir = /var/lib/mysql
#tmpdir = /tmp

BINARY LOGGING

log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 3
sync-binlog = 1

CACHES AND LIMITS

tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 4096

INNODB

innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 256M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 2G

LOGGING

log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log

Percona XtraDB Cluster Settings

wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_cluster_address = gcomm://10.11.20.78,10.11.20.79,10.11.20.80
binlog_format = ROW
wsrep_node_address = 10.12.20.80
wsrep_sst_method = xtrabackup-v2
wsrep_cluster_name = cluster-mysql
wsrep_sst_auth = “xxxxx”

!includedir /etc/mysql/conf.d/

Tnx
Manuel

This is the same error I get since I installed Percona XtraDB cluster and looking for some useful tips!

I move my.cnf to another directory and mysql service start succesfully, but without my configuration, optimization and cluster :expressionless:
With my.cnf (which worked perfectly for 5 months), mysql not start :frowning:

Manuel

I try to reinstall server with fresh ubuntu server 14.04.1 LTS installation.
I follow this guide: http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/ubuntu_howto.html
When I try to start first node (/etc/init.d/mysql bootstrap-pxc) for bootstrapping cluster, service not start and I receive this error:

  • Bootstrapping Percona XtraDB Cluster database server mysqld
  • The server quit without updating PID file (/var/lib/mysql/cls-mysql1-db1.pid).
    [fail]

In servername.err I find many many “[ERROR] InnoDB: Unable to lock ./ibdata1, error: 11”

2014-12-06 16:32:48 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:48 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:48 3422 [Note] InnoDB: Retrying to lock the first data file
2014-12-06 16:32:48 3422 [Note] WSREP: handshake failed, my group: ‘mysql_cluster’, peer group: ‘cluster-mysql1’
2014-12-06 16:32:49 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:49 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:50 3422 [Note] WSREP: handshake failed, my group: ‘mysql_cluster’, peer group: ‘cluster-mysql1’
2014-12-06 16:32:50 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:50 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:51 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:51 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:51 3422 [Note] WSREP: handshake failed, my group: ‘mysql_cluster’, peer group: ‘cluster-mysql1’
2014-12-06 16:32:52 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:52 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:53 3422 [Note] WSREP: handshake failed, my group: ‘mysql_cluster’, peer group: ‘cluster-mysql1’
2014-12-06 16:32:53 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:53 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:54 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:54 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:54 3422 [Note] WSREP: handshake failed, my group: ‘mysql_cluster’, peer group: ‘cluster-mysql1’
2014-12-06 16:32:55 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:55 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2014-12-06 16:32:56 3422 [Note] WSREP: handshake failed, my group: ‘mysql_cluster’, peer group: ‘cluster-mysql1’
2014-12-06 16:32:56 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2014-12-06 16:32:56 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.

I had a similar problem on Ubuntu, which I solved by deleting the /etc/init/mysql.conf upstart script

I had the same problem on ubuntu LTS 14.04 server 64 bits, any help with this isue?

Best regards.

2014-12-06 16:32:56 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11

I would like to ask you to check whether there’s any mysql process left running; you can do that with:

$ ps auxww | grep mysql

For that error possible cause is you already have mysqld running. You can kill mysqld process and try to restart.

In scenario, where all galera nodes stopped then try to bootstrap first node once it’s up you can restart remaining nodes via calling mysqld daemon script as below:
$ /etc/init.d/mysql start

I foresee this will start remaining nodes of the cluster via SST/IST.

In case, If it’s gives you error

  • Starting MySQL (Percona XtraDB Cluster) database server mysqld
  • The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

Please share the relevant entries from mysql error log to identify what causing the issue. Also, share the installed xtradb cluster packages details from all failed nodes.

$ dpkg -l | grep -i “percona”

And my.cnf from failed node.

Moreover, please refer to http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/ubuntu_howto.html

Hello, here the info

Node 1

ps auxww | grep mysql
root 5598 0.0 0.0 14276 900 pts/0 R+ 17:59 0:00 grep --color=auto mysql

/etc/init.d/mysql bootstrap-pxc

Percona XtraDB Cluster database server mysqld
[ OK ]
root@XtraDB-1:~# ps auxww | grep mysql
root 5645 0.0 0.0 4444 732 pts/0 S 18:01 0:00 /bin/sh /usr/bin/mysqld_safe --wsrep-new-cluster
mysql 6021 1.1 23.8 1726452 489124 pts/0 Sl 18:01 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera_smm.so --wsrep-new-cluster --log-error=/var/lib/mysql/XtraDB-1.err --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=848a69b3-9aa9-11e4-af58-cbd501ad4269:2
root 6493 0.0 0.0 14276 900 pts/0 R+ 18:02 0:00 grep --color=auto mysql

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> show status like ‘wsrep%’;
±-----------------------------±-------------------------------------+
| Variable_name | Value |
±-----------------------------±-------------------------------------+
| wsrep_local_state_uuid | 848a69b3-9aa9-11e4-af58-cbd501ad4269 |
| wsrep_protocol_version | 6 |
| wsrep_last_committed | 2 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 7 |
| wsrep_received_bytes | 562 |
| wsrep_local_commits | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 2 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.142857 |
| wsrep_local_cached_downto | 18446744073709551615 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 0 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_incoming_addresses | 192.168.2.70:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | 89754819-9c30-11e4-8d83-2e6a7a990f01 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | 848a69b3-9aa9-11e4-af58-cbd501ad4269 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.8(rf6147dd) |
| wsrep_ready | ON |
±-----------------------------±-----------

in the 2nd node

[mysqld]

datadir=/var/lib/mysql
user=mysql

Path to Galera library

wsrep_provider=/usr/lib/libgalera_smm.so

Cluster connection URL contains the IPs of node#1, node#2 and node#3

wsrep_cluster_address=gcomm://192.168.2.70,192.168.2.71,192.168.2.72

In order for Galera to work correctly binlog format should be ROW

binlog_format=ROW

MyISAM storage engine has only experimental support

default_storage_engine=InnoDB

This changes how InnoDB autoincrement locks are managed and is a requirement for Galera

innodb_autoinc_lock_mode=2

Node #2 address

wsrep_node_address=192.168.2.71

SST method

wsrep_sst_method=xtrabackup

Cluster name

wsrep_cluster_name=my_ubuntu_cluster

Authentication for SST method

wsrep_sst_auth=“sstuser:s3cretPass”

#user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

ps auxww | grep mysql
root 2963 0.0 0.0 14276 896 pts/0 R+ 18:09 0:00 grep --color=auto mysql

/etc/init.d/mysql start

  • Starting MySQL (Percona XtraDB Cluster) database server mysqld * State transfer in progress, setting sleep higher mysqld * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
    [fail]

dpkg -l | grep -i “percona”
ii percona-xtrabackup 2.2.7-5050-1.trusty amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-56 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster with Galera
ii percona-xtradb-cluster-client-5.6 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common-5.6 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-3 3.8.3390.trusty amd64 Metapackage for latest version of galera3.
ii percona-xtradb-cluster-galera-3.x 3.8.3390.trusty amd64 Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.6 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster database server binaries

syslog

/etc/init.d/mysql[2399]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
Jan 14 18:10:26 XtraDB-2 /etc/init.d/mysql[3006]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid

thanks!

Best!

SST method

wsrep_sst_method=xtrabackup

Please update wsrep_sst_method to xtrabackup-v2 which is recommended SST method since PXC 5.5 You can read more here http://www.percona.com/doc/percona-xtradb-cluster/5.6/manual/xtrabackup_sst.html

  • Starting MySQL (Percona XtraDB Cluster) database server mysqld * State transfer in progress, setting sleep higher mysqld * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

Make sure you have correct ownership/permission on MySQL data directory and on /var/run/mysqld/ directory. Make sure AppArmor disabled as per http://www.percona.com/blog/2012/12/20/percona-xtradb-cluster-selinux-is-not-always-the-culprit/
Further, please share innobackup.backup.log should be in donor datadir (first node of cluster which is running). This post is nice to identify the possible causes for SST http://www.percona.com/blog/2014/12/30/diagnosing-sst-errors-with-percona-xtradb-cluster-for-mysql/

In case, If it still fails to start after above mentioned checklist. Please attach the complete error log from failed node.

Hello.

It works! I just change the

SST method

wsrep_sst_method=xtrabackup-v2

But when i reboot the VMs stop working.

apparmor is not installed

and the permissions are ok.

thanks!

Hello

When execute

service mysql stop

  • Stopping MySQL (Percona XtraDB Cluster) mysqld [ OK ]

but mysql service still ON ?

root@XtraDB-1:/var/lib/mysql# ps aux | grep mysql
root 1324 0.0 0.0 4444 728 pts/2 S 19:16 0:00 /bin/sh /usr/bin/mysqld_safe
mysql 1648 0.1 22.0 1060292 451380 pts/2 Sl 19:16 0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera_smm.so --log-error=/var/lib/mysql/XtraDB-1.err --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=848a69b3-9aa9-11e4-af58-cbd501ad4269:5
root 2360 0.0 0.0 14276 896 pts/2 S+ 19:39 0:00 grep --color=auto mysql

Hello

I setup a new test emviroment with ubuntu 12.04 LTS 64 BITS, same error with xtradb 5.6. I follow this http://www.percona.com/doc/percona-xtradb-cluster/5.6/installation/apt_repo.html

and

http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/ubuntu_howto.html

SAME ERROR !!

best regards.

Hello,

You could see the similar problem solved @ https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/32360-xtradb-cluster-loads-but-no-pid-file-mysql-wont-load

I was going through the similar situation on an ubuntu 12.04. Commands followed a) sudo echo “manual” > /etc/init/mysql.override
b) sudo reboot
c) sudo /etc/init.d/mysql bootstrap-pxc

best regards,

Facing same problem while restart mysql service. .

MySQL server PID file could not be found! [FAILED]
Starting MySQL…The server quit without updating PID file [FAILED]tabase/editorial/primary/xyz.pid).

Can Anyone help me with this.