Dual MySQL on the same server boot issue

Hello
I have an little issue with Percona-Server-server-56 Version : 5.6.21 Release : rel70.1.el6 Arch : x86_64
I have created 2 my.cnf with defaults values as follows:

# cat /etc/my.cnf

[client]
#password = your_password
port = 3306
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/vm-template.pid
log-error = /var/lib/mysql/vm-template.err

[mysqld]
user = mysql
basedir = /usr
datadir = /var/lib/mysql
port = 3306
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/vm-template.pid
log-error = /var/lib/mysql/vm-template.err
log-bin = /var/lib/mysql/mysql-bin
innodb_data_home_dir = /var/lib/mysql
# cat /etc/my1.cnf
# my.cnf for secondary mysql server

[client]
#password = your_password
port = 3307
socket = /var/lib/mysql1/mysql1.sock
pid-file = /var/lib/mysql1/vm-template1.pid
log-error = /var/lib/mysql1/vm-template1.err

[mysqld]
user = mysql
basedir = /usr
datadir = /var/lib/mysql1
port = 3307
socket = /var/lib/mysql1/mysql1.sock
pid-file = /var/lib/mysql1/vm-template1.pid
log-error = /var/lib/mysql1/vm-template1.err
log-bin = /var/lib/mysql1/mysql-bin1
innodb_data_home_dir = /var/lib/mysql1

And i created an start up script for the second instance. Both servers works without issues if i start them manualy

# /etc/init.d/mysqld start
Starting MySQL (Percona Server). SUCCESS!
# /etc/init.d/mysqld1 start
Starting MySQL1 (Percona Server). SUCCESS!
# /etc/init.d/mysqld status
SUCCESS! MySQL (Percona Server) running (2620)
# /etc/init.d/mysqld1 status
SUCCESS! MySQL1 (Percona Server) running (2875)

but when i reboot the server i get this in the boot log

Starting MySQL (Percona Server) SUCCESS!
Starting MySQL1 (Percona Server). ERROR! The server quit without updating PID file (/var/lib/mysql1/vm-template1.pid).

The question is what i did wrong

Thank you in advance

Hi GosthMan;

What does the error log say (/var/lib/mysql1/vm-template1.err)?

-Scott

Hello scott,
The issue is that i get nothing on the error log, only the normal shutdown.
No other error messages, just like the second start up script check the pid of main server instead of the secondary

2014-12-15 17:31:46 2471 [Note] /usr/sbin/mysqld: Normal shutdown

2014-12-15 17:31:46 2471 [Note] Giving 0 client threads a chance to die gracefully
2014-12-15 17:31:46 2471 [Note] Event Scheduler: Purging the queue. 0 events
2014-12-15 17:31:46 2471 [Note] Shutting down slave threads
2014-12-15 17:31:46 2471 [Note] Forcefully disconnecting 0 remaining clients
2014-12-15 17:31:46 2471 [Note] Binlog end
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'partition'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'ARCHIVE'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'BLACKHOLE'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_METRICS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CMPMEM'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_CMP'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_LOCKS'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'INNODB_TRX'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'XTRADB_RSEG'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2014-12-15 17:31:46 2471 [Note] Shutting down plugin 'InnoDB'
2014-12-15 17:31:46 2471 [Note] InnoDB: FTS optimize thread exiting.
2014-12-15 17:31:46 2471 [Note] InnoDB: Starting shutdown...
2014-12-15 17:31:47 2471 [Note] InnoDB: Shutdown completed; log sequence number 1626017
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'CSV'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'MyISAM'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'MRG_MYISAM'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'MEMORY'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'sha256_password'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'mysql_old_password'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'mysql_native_password'
2014-12-15 17:31:47 2471 [Note] Shutting down plugin 'binlog'
2014-12-15 17:31:47 2471 [Note] /usr/sbin/mysqld: Shutdown complete

141215 17:31:47 mysqld_safe mysqld from pid file /var/lib/mysql1/vm12029011.pid ended

Hi GosthMan;

The fact that the log shows only the shutdown seems suspicious, as it should mention it trying to start up.

When you start both of them manually, does each of their error logs show them starting up?

And what does the ownership/permissions look like on your my.cnf and my1.cnf files?

-Scott

Hello
Yes if i start them manually they start without issues and the logs of both mysql reflect the start up as normal, the only time when appears the error is on server reboot, also the servers works without issues.
permissions

-rw-r--r--. 1 root root 413 Dec 11 17:30 my1.cnf
-rw-r--r--. 1 root root 349 Dec 11 17:29 my.cnf

Hi GosthMan;

Are you using SELinux or AppArmor on the server? And when you are starting the instances manually, are you doing so as root?

Basically it seems like on boot it’s either not reading your my1.cnf correctly (since there is no message in the error log) or there is a permissions issue (hence SELinux or AppArmor). Odd part is that it outputs the correct PID, so that leads me more towards a permissions issue.

-Scott

dam i forgot to check that.
You was right, on the instance copy the selinux got reactivated.

type=AVC msg=audit(1418665859.308:32): avc: denied { append } for pid=2410 comm="mysqld" path="/var/lib/mysql1/vm-template.err" dev=dm-0 ino=51 scontext=system_u:system_r:mysqld_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file

Thank you to point me to that direction

Hi GosthMan;

No problem; glad you got it working!

-Scott