Not the answer you need?
Register and ask your own question!

Error The server quit without updating PID file

ManuelRighiManuelRighi EntrantCurrent User Role Beginner
Hello,
I have Percona XtraDB Cluster 5.6 with Ubuntu Server 14.04.1 LTS.
I have 3 nodes. Everything worked perfectly for 5 months
Today I find all 3 nodes with mysql server stopped.
When I try to start mysql service I receive this error:

* Starting MySQL (Percona XtraDB Cluster) database server mysqld
* The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

This error occour on all nodes.
Can you help me ?

My my.cnf file have this configuration:


#
# MySQL Configuration
#

[mysql]

# CLIENT #
port = 3306
socket = /var/run/mysqld/mysqld.sock


[mysqld]

# GENERAL #
user = mysql
default_storage_engine = InnoDB
socket = /var/run/mysqld/mysqld.sock
pid-file = /var/run/mysqld/mysqld.pid
bind-address = 0.0.0.0

# MyISAM #
key-buffer-size = 32M
myisam-recover = FORCE,BACKUP


# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000


# DATA STORAGE #
#basedir = /usr
datadir = /var/lib/mysql
#tmpdir = /tmp


# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 3
sync-binlog = 1


# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 500
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 4096


# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 256M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 2G


# LOGGING #
log-error = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes = 1
slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log


# Percona XtraDB Cluster Settings #
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_cluster_address = gcomm://10.11.20.78,10.11.20.79,10.11.20.80
binlog_format = ROW
wsrep_node_address = 10.12.20.80
wsrep_sst_method = xtrabackup-v2
wsrep_cluster_name = cluster-mysql
wsrep_sst_auth = "xxxxx"


!includedir /etc/mysql/conf.d/




Tnx
Manuel

Comments

  • jithjith Entrant Current User Role Beginner
    This is the same error I get since I installed Percona XtraDB cluster and looking for some useful tips!
  • ManuelRighiManuelRighi Entrant Current User Role Beginner
    I move my.cnf to another directory and mysql service start succesfully, but without my configuration, optimization and cluster :|
    With my.cnf (which worked perfectly for 5 months), mysql not start :(

    Manuel
  • ManuelRighiManuelRighi Entrant Current User Role Beginner
    I try to reinstall server with fresh ubuntu server 14.04.1 LTS installation.
    I follow this guide: http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/ubuntu_howto.html
    When I try to start first node (/etc/init.d/mysql bootstrap-pxc) for bootstrapping cluster, service not start and I receive this error:

    * Bootstrapping Percona XtraDB Cluster database server mysqld
    * The server quit without updating PID file (/var/lib/mysql/cls-mysql1-db1.pid).
    [fail]

    In servername.err I find many many "[ERROR] InnoDB: Unable to lock ./ibdata1, error: 11"


    2014-12-06 16:32:48 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:48 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:48 3422 [Note] InnoDB: Retrying to lock the first data file
    2014-12-06 16:32:48 3422 [Note] WSREP: handshake failed, my group: 'mysql_cluster', peer group: 'cluster-mysql1'
    2014-12-06 16:32:49 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:49 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:50 3422 [Note] WSREP: handshake failed, my group: 'mysql_cluster', peer group: 'cluster-mysql1'
    2014-12-06 16:32:50 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:50 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:51 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:51 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:51 3422 [Note] WSREP: handshake failed, my group: 'mysql_cluster', peer group: 'cluster-mysql1'
    2014-12-06 16:32:52 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:52 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:53 3422 [Note] WSREP: handshake failed, my group: 'mysql_cluster', peer group: 'cluster-mysql1'
    2014-12-06 16:32:53 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:53 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:54 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:54 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:54 3422 [Note] WSREP: handshake failed, my group: 'mysql_cluster', peer group: 'cluster-mysql1'
    2014-12-06 16:32:55 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:55 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
    2014-12-06 16:32:56 3422 [Note] WSREP: handshake failed, my group: 'mysql_cluster', peer group: 'cluster-mysql1'
    2014-12-06 16:32:56 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
    2014-12-06 16:32:56 3422 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
  • ldoldo Entrant Current User Role Beginner
    I had a similar problem on Ubuntu, which I solved by deleting the /etc/init/mysql.conf upstart script
  • maxpaomaxpao Entrant Inactive User Role Beginner
    I had the same problem on ubuntu LTS 14.04 server 64 bits, any help with this isue?

    Best regards.
  • mirfanmirfan Database Administrator Inactive User Role Beginner
    >> 2014-12-06 16:32:56 3422 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11

    I would like to ask you to check whether there's any mysql process left running; you can do that with:

    $ ps auxww | grep mysql

    For that error possible cause is you already have mysqld running. You can kill mysqld process and try to restart.

    In scenario, where all galera nodes stopped then try to bootstrap first node once it's up you can restart remaining nodes via calling mysqld daemon script as below:
    $ /etc/init.d/mysql start

    I foresee this will start remaining nodes of the cluster via SST/IST.

    In case, If it's gives you error
    >> * Starting MySQL (Percona XtraDB Cluster) database server mysqld
    >> * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

    Please share the relevant entries from mysql error log to identify what causing the issue. Also, share the installed xtradb cluster packages details from all failed nodes.

    $ dpkg -l | grep -i "percona"

    And my.cnf from failed node.

    Moreover, please refer to http://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/ubuntu_howto.html
  • maxpaomaxpao Entrant Inactive User Role Beginner
    Hello, here the info

    Node 1

    ps auxww | grep mysql
    root 5598 0.0 0.0 14276 900 pts/0 R+ 17:59 0:00 grep --color=auto mysql

    /etc/init.d/mysql bootstrap-pxc




    Percona XtraDB Cluster database server mysqld
    [ OK ]
    [email protected]:~# ps auxww | grep mysql
    root 5645 0.0 0.0 4444 732 pts/0 S 18:01 0:00 /bin/sh /usr/bin/mysqld_safe --wsrep-new-cluster
    mysql 6021 1.1 23.8 1726452 489124 pts/0 Sl 18:01 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera_smm.so --wsrep-new-cluster --log-error=/var/lib/mysql/XtraDB-1.err --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=848a69b3-9aa9-11e4-af58-cbd501ad4269:2
    root 6493 0.0 0.0 14276 900 pts/0 R+ 18:02 0:00 grep --color=auto mysql


    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

    mysql> show status like 'wsrep%';
    +
    +
    +
    | Variable_name | Value |
    +
    +
    +
    | wsrep_local_state_uuid | 848a69b3-9aa9-11e4-af58-cbd501ad4269 |
    | wsrep_protocol_version | 6 |
    | wsrep_last_committed | 2 |
    | wsrep_replicated | 0 |
    | wsrep_replicated_bytes | 0 |
    | wsrep_repl_keys | 0 |
    | wsrep_repl_keys_bytes | 0 |
    | wsrep_repl_data_bytes | 0 |
    | wsrep_repl_other_bytes | 0 |
    | wsrep_received | 7 |
    | wsrep_received_bytes | 562 |
    | wsrep_local_commits | 0 |
    | wsrep_local_cert_failures | 0 |
    | wsrep_local_replays | 0 |
    | wsrep_local_send_queue | 0 |
    | wsrep_local_send_queue_max | 1 |
    | wsrep_local_send_queue_min | 0 |
    | wsrep_local_send_queue_avg | 0.000000 |
    | wsrep_local_recv_queue | 0 |
    | wsrep_local_recv_queue_max | 2 |
    | wsrep_local_recv_queue_min | 0 |
    | wsrep_local_recv_queue_avg | 0.142857 |
    | wsrep_local_cached_downto | 18446744073709551615 |
    | wsrep_flow_control_paused_ns | 0 |
    | wsrep_flow_control_paused | 0.000000 |
    | wsrep_flow_control_sent | 0 |
    | wsrep_flow_control_recv | 0 |
    | wsrep_cert_deps_distance | 0.000000 |
    | wsrep_apply_oooe | 0.000000 |
    | wsrep_apply_oool | 0.000000 |
    | wsrep_apply_window | 0.000000 |
    | wsrep_commit_oooe | 0.000000 |
    | wsrep_commit_oool | 0.000000 |
    | wsrep_commit_window | 0.000000 |
    | wsrep_local_state | 4 |
    | wsrep_local_state_comment | Synced |
    | wsrep_cert_index_size | 0 |
    | wsrep_causal_reads | 0 |
    | wsrep_cert_interval | 0.000000 |
    | wsrep_incoming_addresses | 192.168.2.70:3306 |
    | wsrep_evs_delayed | |
    | wsrep_evs_evict_list | |
    | wsrep_evs_repl_latency | 0/0/0/0/0 |
    | wsrep_evs_state | OPERATIONAL |
    | wsrep_gcomm_uuid | 89754819-9c30-11e4-8d83-2e6a7a990f01 |
    | wsrep_cluster_conf_id | 3 |
    | wsrep_cluster_size | 1 |
    | wsrep_cluster_state_uuid | 848a69b3-9aa9-11e4-af58-cbd501ad4269 |
    | wsrep_cluster_status | Primary |
    | wsrep_connected | ON |
    | wsrep_local_bf_aborts | 0 |
    | wsrep_local_index | 0 |
    | wsrep_provider_name | Galera |
    | wsrep_provider_vendor | Codership Oy <[email protected]> |
    | wsrep_provider_version | 3.8(rf6147dd) |
    | wsrep_ready | ON |
    +
    +



    in the 2nd node

    [mysqld]

    datadir=/var/lib/mysql
    user=mysql

    # Path to Galera library
    wsrep_provider=/usr/lib/libgalera_smm.so

    # Cluster connection URL contains the IPs of node#1, node#2 and node#3
    wsrep_cluster_address=gcomm://192.168.2.70,192.168.2.71,192.168.2.72

    # In order for Galera to work correctly binlog format should be ROW
    binlog_format=ROW

    # MyISAM storage engine has only experimental support
    default_storage_engine=InnoDB

    # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
    innodb_autoinc_lock_mode=2

    # Node #2 address
    wsrep_node_address=192.168.2.71

    # SST method
    wsrep_sst_method=xtrabackup

    # Cluster name
    wsrep_cluster_name=my_ubuntu_cluster

    # Authentication for SST method
    wsrep_sst_auth="sstuser:s3cretPass"


    #user = mysql
    pid-file = /var/run/mysqld/mysqld.pid
    socket = /var/run/mysqld/mysqld.sock
    port = 3306
    basedir = /usr
    datadir = /var/lib/mysql
    tmpdir = /tmp
    lc-messages-dir = /usr/share/mysql
    skip-external-locking


    ..................................


    ps auxww | grep mysql
    root 2963 0.0 0.0 14276 896 pts/0 R+ 18:09 0:00 grep --color=auto mysql


    /etc/init.d/mysql start
    * Starting MySQL (Percona XtraDB Cluster) database server mysqld * State transfer in progress, setting sleep higher mysqld * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
    [fail]


    dpkg -l | grep -i "percona"
    ii percona-xtrabackup 2.2.7-5050-1.trusty amd64 Open source backup tool for InnoDB and XtraDB
    ii percona-xtradb-cluster-56 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster with Galera
    ii percona-xtradb-cluster-client-5.6 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster database client binaries
    ii percona-xtradb-cluster-common-5.6 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
    ii percona-xtradb-cluster-galera-3 3.8.3390.trusty amd64 Metapackage for latest version of galera3.
    ii percona-xtradb-cluster-galera-3.x 3.8.3390.trusty amd64 Galera components of Percona XtraDB Cluster
    ii percona-xtradb-cluster-server-5.6 5.6.21-25.8-938.trusty amd64 Percona XtraDB Cluster database server binaries


    syslog

    /etc/init.d/mysql[2399]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid
    Jan 14 18:10:26 XtraDB-2 /etc/init.d/mysql[3006]: MySQL PID not found, pid_file detected/guessed: /var/run/mysqld/mysqld.pid



    thanks!

    Best!
  • mirfanmirfan Database Administrator Inactive User Role Beginner
    >> # SST method
    >> wsrep_sst_method=xtrabackup

    Please update wsrep_sst_method to xtrabackup-v2 which is recommended SST method since PXC 5.5 You can read more here http://www.percona.com/doc/percona-xtradb-cluster/5.6/manual/xtrabackup_sst.html

    >> * Starting MySQL (Percona XtraDB Cluster) database server mysqld * State transfer in progress, setting sleep higher mysqld * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).

    Make sure you have correct ownership/permission on MySQL data directory and on /var/run/mysqld/ directory. Make sure AppArmor disabled as per http://www.percona.com/blog/2012/12/20/percona-xtradb-cluster-selinux-is-not-always-the-culprit/
    Further, please share innobackup.backup.log should be in donor datadir (first node of cluster which is running). This post is nice to identify the possible causes for SST http://www.percona.com/blog/2014/12/30/diagnosing-sst-errors-with-percona-xtradb-cluster-for-mysql/

    In case, If it still fails to start after above mentioned checklist. Please attach the complete error log from failed node.
  • maxpaomaxpao Entrant Inactive User Role Beginner
    Hello.

    It works! I just change the

    # SST method
    wsrep_sst_method=xtrabackup-v2


    But when i reboot the VMs stop working.

    apparmor is not installed

    and the permissions are ok.

    thanks!
  • maxpaomaxpao Entrant Inactive User Role Beginner
    Hello

    When execute

    service mysql stop
    * Stopping MySQL (Percona XtraDB Cluster) mysqld [ OK ]

    but mysql service still ON ?

    [email protected]:/var/lib/mysql# ps aux | grep mysql
    root 1324 0.0 0.0 4444 728 pts/2 S 19:16 0:00 /bin/sh /usr/bin/mysqld_safe
    mysql 1648 0.1 22.0 1060292 451380 pts/2 Sl 19:16 0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-provider=/usr/lib/libgalera_smm.so --log-error=/var/lib/mysql/XtraDB-1.err --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=848a69b3-9aa9-11e4-af58-cbd501ad4269:5
    root 2360 0.0 0.0 14276 896 pts/2 S+ 19:39 0:00 grep --color=auto mysql
  • maxpaomaxpao Entrant Inactive User Role Beginner
    Hello

    I setup a new test emviroment with ubuntu 12.04 LTS 64 BITS, same error with xtradb 5.6. I follow this http://www.percona.com/doc/percona-xtradb-cluster/5.6/installation/apt_repo.html

    and

    http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/ubuntu_howto.html


    SAME ERROR !!

    best regards.
  • siyasiya Entrant Current User Role Beginner
    Hello,

    You could see the similar problem solved @ https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/32360-xtradb-cluster-loads-but-no-pid-file-mysql-wont-load

    I was going through the similar situation on an ubuntu 12.04. Commands followed a) sudo echo "manual" > /etc/init/mysql.override
    b) sudo reboot
    c) sudo /etc/init.d/mysql bootstrap-pxc

    best regards,
  • gauravgaurav Entrant Current User Role Beginner
    Facing same problem while restart mysql service. .

    MySQL server PID file could not be found! [FAILED]
    Starting MySQL...The server quit without updating PID file [FAILED]tabase/editorial/primary/xyz.pid).

    Can Anyone help me with this.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.