Alex,
Thanks so much for the explanation of what’s going wrong. I tried with and without the tcp:// scheme for “gmcast.listen_addr” with the same unfortunate outcome.
I simplified the configuration as you suggested to the one below:
[mysqld]datadir=/var/lib/mysqluser=mysqlbinlog_format=ROWwsrep_debug=1wsrep_provider=/usr/lib/libgalera_smm.sowsrep_cluster_address="gcomm://"wsrep_slave_threads=2wsrep_node_name=node1wsrep_node_address=10.254.167.5wsrep_cluster_name=clusterwsrep_sst_method=rsyncbind_address=0.0.0.0default_storage_engine=InnoDBinnodb_locks_unsafe_for_binlog=1innodb_autoinc_lock_mode=2max_connections=10max_allowed_packet=32Mtable_cache=2048thread_cache_size=32query_cache_size=256Minnodb_buffer_pool_size=128Msort_buffer_size=64M read_rnd_buffer_size=2Minnodb_log_file_size = 64Minnodb_log_buffer_size = 8Mlog-bin=/mnt/mysql-binlogs/mysql-binlog-bin=/mnt/mysql-binlogs/mysql-bin.indexrelay-log=/mnt/mysql-binlogs/mysql-relay-binrelay-log-index=/mnt/mysql-binlogs/mysql-relay-bin.index
It terminated with the same outcome:
/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql120607 12:07:56 [Note] Flashcache bypass: disabled120607 12:07:56 [Note] Flashcache setup error is : ioctl failed120607 12:07:56 [Note] WSREP: Read nil XID from storage engines, skipping position init120607 12:07:56 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so’120607 12:07:56 [Note] WSREP: wsrep_load(): Galera 2.1dev(r112) by Codership Oy <info@codership.com> loaded succesfully.120607 12:07:56 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1120607 12:07:56 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’.120607 12:07:56 [Note] WSREP: Passing config to GCS: base_host = 10.254.167.5; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2147483647; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3120607 12:07:57 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1120607 12:07:57 [Note] WSREP: wsrep_sst_grab()120607 12:07:57 [Note] WSREP: Start replication120607 12:07:57 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1terminate called after throwing an instance of 'gu::NotFound’19:07:57 UTC - mysqld got signal 6 ;This could be because you hit a bug. It is also possible that this binaryor one of the libraries it was linked against is corrupt, improperly built,or misconfigured. This error can also be caused by malfunctioning hardware.We will try our best to scrape up some info that will hopefully helpdiagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.key_buffer_size=0read_buffer_size=131072max_used_connections=0max_threads=10thread_count=0connection_count=0It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 656721 K bytes of memoryHope that’s ok; if not, decrease some variables in the equation.Thread pointer: 0x0Attempting backtrace. You can use the following information to find outwhere mysqld died. If you see no messages after this, something wentterribly wrong…stack_bottom = 0 thread_stack 0x30000/usr/sbin/mysqld(my_print_stacktrace+0x33)[0x8409693]/usr/sbin/mysqld(handle_fatal_signal+0x48c)[0x82d309c][0xc5d420]/lib/i686/nosegneg/libc.so.6(abort+0x101)[0x941a21]/usr/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x150)[0x20c4d0]/usr/lib/libstdc++.so.6[0x209f35]/usr/lib/libstdc++.so.6[0x209f72]/usr/lib/libstdc++.so.6[0x20a0aa]/usr/lib/libgalera_smm.so(_ZNK2gu3URI10get_optionERKSs+0xf6)[0x4c73e6]/usr/lib/libgalera_smm.so(_ZN9GCommConnC2ERKN2gu3URIERNS0_6ConfigE+0x25d)[0x5b8d2d]/usr/lib/libgalera_smm.so(gcs_gcomm_create+0xd4)[0x5b41b4]/usr/lib/libgalera_smm.so(gcs_backend_init+0xa9)[0x5a1cb9]/usr/lib/libgalera_smm.so(gcs_core_open+0x6f)[0x5a7eaf]/usr/lib/libgalera_smm.so(gcs_open+0x2c8)[0x5aeb58]/usr/lib/libgalera_smm.so(ZN6galera13ReplicatorSMM7connectERKSsS2_S2+0x296)[0x5f97d6]/usr/lib/libgalera_smm.so(galera_connect+0xae)[0x6146ae]/usr/sbin/mysqld(_Z23wsrep_start_replicationv+0x107)[0x8289cb7]/usr/sbin/mysqld(_Z18wsrep_init_startupb+0x7c)[0x828aa9c]/usr/sbin/mysqld[0x813b069]/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0xa22)[0x813d402]/usr/sbin/mysqld(main+0x27)[0x8130e27]/lib/i686/nosegneg/libc.so.6(__libc_start_main+0xdc)[0x92ce9c]/usr/sbin/mysqld[0x8130d41]The manual page at MySQL :: MySQL 8.0 Reference Manual :: B.3.3.3 What to Do If MySQL Keeps Crashing containsinformation that should help you find out what is causing the crash.
I noticed that it’s always in the “get_option” call that it appears to crash, so I agree with you that there’s some “fatal error in configuration”!
As for why we’re running CentOS 5.4, it’s just due to a very large infrastructure already built on top of it. With over 26G of rpms comprised of 12K packages it’s a project we’re pushing off! With that said, much of the reason for all the packages is we’ve upgraded the OS well beyond 5.4 by backporting RPMs. Ofcourse, libstdc++ is not one of them =).
If upgrading to 5.8 might alleviate some of these issues, I can try it out for these particular servers. Also, fwiw, this is on EC2.
-Erik