Pidfile is not always created on percona-server start

MySQL & MariaDB Other MySQL® Questions
1

Hi all

I’m using percona-server, latests squeeze 5.1 version.

I’m also using monit to monitor mysqld process. What I’ve relized is that mysqld does not always create the pidfile under /var/run/mysqld, monit thinks that it’s down and tries to restart it or gives up. It also stops all the services that depend on mysql.

Monit is not the issue. Manually starting mysqld via init.d script confirms that the pidfile is sometimes and sometimes not created.

2

This sounds like something else on the system is interacting with the server. Usually when the pid-file disappears, it’s something like the following scenario:

  1. mysqld starts and writes the pid file
  2. something tries to start mysqld again, and it fails because of the existing instance; but it removes the pid file when it shuts down
  3. you notice the pid file is gone

There can be many variations on such scenarios, but that’s the general kind of thing I usually see.

3

Hi Baron

It might be a race condition between mysql starting and monit trying to start it at the same time. I’ll try to reproduce it without monit.

thanks.

4

Yes, looks like mysqld_safe removes the pidfile. I took a look to the script and saw this:

If there exists an old pid file, check if the daemon is already running

Note: The switches to ‘ps’ may depend on your operating system

if test -f “$pid_file”
then
PID=cat "$pid_file"
if /bin/kill -0 $PID > /dev/null 2> /dev/null
then
if /bin/ps wwwp $PID | grep -v " grep" | grep -v mysqld_safe | grep – “$MYSQLD” > /dev/null
then # The pid contains a mysqld process
log_error “A mysqld process already exists”
exit 1
fi
fi
rm -f “$pid_file”
if test -f “$pid_file”
then
log_error “Fatal error: Can’t remove the pid file:
$pid_file
Please remove it manually and start $0 again;
mysqld daemon not started”
exit 1
fi
fi

During the startup, looks like monit wants to start mysql at the same time mysql is in start process.

In the race condition looks like mysql has created the pidfile but the process is not yet listed in ps. So the pidfile is detected as old pidfile and removed.

Can this remove be safely deleted from the script? If mysql is started and an old pidfile exists, wouldn’t mysql overwrite it?

5

I have a similar problem with startup from sysv init.d script. When I run /etc/init.d/mysql start, mysql fails to write a pid file. Actually, doing some debugging inside the init.d script shows that mysql starts, but fails to spawn the first child:

Starting MySQL (Percona Server)/usr/bin/mysqld_safe --datadir=/ssd/mysqldata --pid-file=/ssd/mysqldata/test.com.pid >/dev/null 2>&1 &
/ssd/mysqldata/test.com.pid

This is the results of ‘ps ax | grep mysql’ run inside the “wait_for_pid” function in the init script:

2100 pts/0 S+ 0:00 /bin/sh /etc/init.d/mysql start
2109 ? R 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/ssd/mysqldata --pid-file=/ssd/mysqldata/test.com.pid
2111 pts/0 S+ 0:00 grep mysql

this is an ls of the data directory where there should be a pid file

ibdata1 ib_logfile1 performance_schema RPM_UPGRADE_MARKER-LAST sj-slow.log
ib_logfile0 mysql RPM_UPGRADE_HISTORY test.com.err test

however, running the exact same command at the command line starts mysql just fine AND creates the pid file:

/usr/bin/mysqld_safe --datadir=/ssd/mysqldata --pid-file=/ssd/mysqldata/test.com.pid >/dev/null 2>&1 &

the error log shows nothing, this is a fresh server with nothing else running on it.