Wresp breakdown WSREP: fork() failed: 12 (Cannot allocate memory)

simonhoenscheidsimonhoenscheid EntrantActive Member Participant
We use percona cluster on debian wheezy. last cluster was build two weeks ago and worked properly.
Today we tried startuing a new one and get memory errors from wresp.


dpkg -l|grep galera
ii percona-xtradb-cluster-galera-3 3.9.3494.wheezy amd64 Metapackage for latest version of galera3.
ii percona-xtradb-cluster-galera-3.x 3.9.3494.wheezy amd64 Galera components of Percona XtraDB Cluster


150310 17:48:52 mysqld_safe Starting mysqld_using_numa daemon with databases from /data/mysql
150310 17:48:52 mysqld_safe WSREP: Running position recovery with --log_error='/data/mysql/wsrep_recovery.35GNao' --pid-file='/data/mysql/galeranode-02-recover.pid'
2015-03-10 17:48:55 0 [Warning] Using unique option prefix sort_buffer instead of sort_buffer_size is deprecated and will be removed in a future release. Please use the full
name instead.
2015-03-10 17:48:55 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the fu
ll name instead.
2015-03-10 17:48:55 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
2015-03-10 17:48:55 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more d
etails).
150310 17:48:58 mysqld_safe WSREP: Recovered position fafe51f0-c743-11e4-9db3-765772f3dde5:15
Log of wsrep recovery (--wsrep-recover):
2015-03-10 17:48:55 15254 [Warning] Using unique option prefix myisam_recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please
use the full name instead.
2015-03-10 17:48:55 15254 [Warning] option 'innodb-log-files-in-group': unsigned value 1 adjusted to 2
InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
InnoDB: The log block size is set to 4096.
2015-03-10 17:48:55 7f563d5f5720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the optio
n innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2015-03-10 17:48:55 7f563d5f5720 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTE
D transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
2015-03-10 17:48:55 15254 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-03-10 17:48:55 15254 [Note] InnoDB: The InnoDB memory heap is disabled
2015-03-10 17:48:55 15254 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-03-10 17:48:55 15254 [Note] InnoDB: Memory barrier is not used
2015-03-10 17:48:55 15254 [Note] InnoDB: Compressed tables use zlib 1.2.7
2015-03-10 17:48:55 15254 [Note] InnoDB: Using Linux native AIO
2015-03-10 17:48:55 15254 [Note] InnoDB: Not using CPU crc32 instructions
2015-03-10 17:48:55 15254 [Note] InnoDB: Initializing buffer pool, size = 5.5G
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
InnoDB HugeTLB: Warning: Using conventional memory pool
2015-03-10 17:48:55 15254 [Note] InnoDB: Completed initialization of buffer pool
2015-03-10 17:48:55 15254 [Note] InnoDB: Highest supported file format is Barracuda.
2015-03-10 17:48:55 15254 [Note] InnoDB: 128 rollback segment(s) are active.
2015-03-10 17:48:55 15254 [Note] InnoDB: Waiting for purge to start
2015-03-10 17:48:55 15254 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 1647917
2015-03-10 17:48:55 15254 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_port=9998'
2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_port_wr=9999'
2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_threads=16'
2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_threads_wr=1'
2015-03-10 17:48:56 15254 [Note] RSA private key file not found: /data/mysql//private_key.pem. Some authentication plugins will not work.
2015-03-10 17:48:56 15254 [Note] RSA public key file not found: /data/mysql//public_key.pem. Some authentication plugins will not work.
2015-03-10 17:48:56 15254 [Note] Server hostname (bind-address): '*'; port: 33306
2015-03-10 17:48:56 15254 [Note] IPv6 is available.
2015-03-10 17:48:56 15254 [Note] - '::' resolves to '::';
2015-03-10 17:48:56 15254 [Note] Server socket created on IP: '::'.
2015-03-10 17:48:56 15254 [Note] WSREP: Recovered position: fafe51f0-c743-11e4-9db3-765772f3dde5:15
2015-03-10 17:48:56 15254 [Note] Binlog end
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'partition'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'BLACKHOLE'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_METRICS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_TRX'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'XTRADB_RSEG'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'InnoDB'
2015-03-10 17:48:56 15254 [Note] InnoDB: FTS optimize thread exiting.
2015-03-10 17:48:56 15254 [Note] InnoDB: Starting shutdown...
2015-03-10 17:48:58 15254 [Note] InnoDB: Shutdown completed; log sequence number 1647927
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'ARCHIVE'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'MEMORY'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'MyISAM'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'CSV'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'MRG_MYISAM'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'sha256_password'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'mysql_old_password'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'mysql_native_password'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'wsrep'
2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'binlog'
2015-03-10 17:48:58 15254 [Note] /usr/sbin/mysqld: Shutdown complete

Comments

  • simonhoenscheidsimonhoenscheid Entrant Active Member Participant
    2015-03-10 17:49:03 0 [Warning] Using unique option prefix sort_buffer instead of sort_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
    2015-03-10 17:49:03 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the full name instead.
    2015-03-10 17:49:03 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
    2015-03-10 17:49:03 0 [Note] WSREP: wsrep_start_position var submitted: 'fafe51f0-c743-11e4-9db3-765772f3dde5:15'
    2015-03-10 17:49:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2015-03-10 17:49:03 15327 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2015-03-10 17:49:03 15327 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
    2015-03-10 17:49:03 15327 [Note] WSREP: wsrep_load(): Galera 3.9(r93aca2d) by Codership Oy <[email protected]> loaded successfully.
    2015-03-10 17:49:03 15327 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
    2015-03-10 17:49:03 15327 [Note] WSREP: Found saved state: fafe51f0-c743-11e4-9db3-765772f3dde5:-1
    2015-03-10 17:49:03 15327 [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/; base_host = 172.16.123.205; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT10S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 4G; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false;
    2015-03-10 17:49:03 15327 [Note] WSREP: Service thread queue flushed.
    2015-03-10 17:49:03 15327 [Note] WSREP: Assign initial position for certification: 15, protocol version: -1
    2015-03-10 17:49:03 15327 [Note] WSREP: wsrep_sst_grab()
    2015-03-10 17:49:03 15327 [Note] WSREP: Start replication
    2015-03-10 17:49:03 15327 [Note] WSREP: Setting initial position to fafe51f0-c743-11e4-9db3-765772f3dde5:15
    2015-03-10 17:49:03 15327 [Note] WSREP: protonet asio version 0
    2015-03-10 17:49:03 15327 [Note] WSREP: Using CRC-32C for message checksums.
    2015-03-10 17:49:03 15327 [Note] WSREP: backend: asio
    2015-03-10 17:49:03 15327 [Warning] WSREP: access file(/data/mysql//gvwstate.dat) failed(No such file or directory)
    2015-03-10 17:49:03 15327 [Note] WSREP: restore pc from disk failed
    2015-03-10 17:49:03 15327 [Note] WSREP: GMCast version 0
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2015-03-10 17:49:03 15327 [Note] WSREP: EVS version 0
    2015-03-10 17:49:03 15327 [Note] WSREP: gcomm: connecting to group 'sso-uat', peer '172.16.123.204:,172.16.123.205:,172.16.123.206:,172.16.123.207:'
    2015-03-10 17:49:03 15327 [Warning] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') address 'tcp://172.16.123.205:4567' points to own listening address, blacklisting
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') address 'tcp://172.16.123.205:4567' pointing to uuid 5bcb2aad is blacklisted, skipping
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
    2015-03-10 17:49:03 15327 [Note] WSREP: declaring f844ae82 at tcp://172.16.123.204:4567 stable
    2015-03-10 17:49:03 15327 [Note] WSREP: Node f844ae82 state prim
    2015-03-10 17:49:03 15327 [Note] WSREP: view(view_id(PRIM,5bcb2aad,6) memb {
    5bcb2aad,0
    f844ae82,0
    } joined {
    } left {
    } partitioned {
    })
    2015-03-10 17:49:03 15327 [Note] WSREP: save pc into disk
    2015-03-10 17:49:03 15327 [Note] WSREP: discarding pending addr without UUID: tcp://172.16.123.206:4567
    2015-03-10 17:49:03 15327 [Note] WSREP: discarding pending addr without UUID: tcp://172.16.123.207:4567
    2015-03-10 17:49:04 15327 [Note] WSREP: gcomm: connected
    2015-03-10 17:49:04 15327 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2015-03-10 17:49:04 15327 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2015-03-10 17:49:04 15327 [Note] WSREP: Opened channel 'sso-uat'
    2015-03-10 17:49:04 15327 [Note] WSREP: Waiting for SST to complete.
    2015-03-10 17:49:04 15327 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE EXCHANGE: sent state msg: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE EXCHANGE: got state msg: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2 from 0 (galeranode-02)
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE EXCHANGE: got state msg: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2 from 1 (galeranode-01)
    2015-03-10 17:49:04 15327 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 5,
    members = 1/2 (joined/total),
    act_id = 17,
    last_appl. = -1,
    protocols = 0/7/3 (gcs/repl/appl),
    group UUID = 377d1d1f-c723-11e4-be0f-f205bd80e309
    2015-03-10 17:49:04 15327 [Note] WSREP: Flow-control interval: [23, 23]
    2015-03-10 17:49:04 15327 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 17)
    2015-03-10 17:49:04 15327 [Note] WSREP: State transfer required:
    Group state: 377d1d1f-c723-11e4-be0f-f205bd80e309:17
    Local state: fafe51f0-c743-11e4-9db3-765772f3dde5:15
    2015-03-10 17:49:04 15327 [Note] WSREP: New cluster view: global state: 377d1d1f-c723-11e4-be0f-f205bd80e309:17, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3
    2015-03-10 17:49:04 15327 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2015-03-10 17:49:04 15327 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.123.205' --auth 'root:Uz6oebu6' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15327' '' '
    2015-03-10 17:49:04 15327 [ERROR] WSREP: fork() failed: 12 (Cannot allocate memory)
    2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to execute: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.123.205' --auth 'root:Uz6oebu6' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15327' '' : 12 (Cannot allocate memory)
    2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
    2015-03-10 17:49:04 15327 [ERROR] Aborting
    2015-03-10 17:49:06 15327 [Note] WSREP: Closing send monitor...
    2015-03-10 17:49:06 15327 [Note] WSREP: Closed send monitor.
    2015-03-10 17:49:06 15327 [Note] WSREP: gcomm: terminating thread
    2015-03-10 17:49:06 15327 [Note] WSREP: gcomm: joining thread
    2015-03-10 17:49:06 15327 [Note] WSREP: gcomm: closing backend
    2015-03-10 17:49:06 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2015-03-10 17:49:07 15327 [Note] WSREP: view(view_id(NON_PRIM,5bcb2aad,6) memb {
    5bcb2aad,0
    } joined {
    } left {
    } partitioned {
    f844ae82,0
    })
    2015-03-10 17:49:07 15327 [Note] WSREP: view((empty))
    2015-03-10 17:49:07 15327 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2015-03-10 17:49:07 15327 [Note] WSREP: gcomm: closed
    2015-03-10 17:49:07 15327 [Note] WSREP: Flow-control interval: [16, 16]
    2015-03-10 17:49:07 15327 [Note] WSREP: Received NON-PRIMARY.
    2015-03-10 17:49:07 15327 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 17)
    2015-03-10 17:49:07 15327 [Note] WSREP: Received self-leave message.
    2015-03-10 17:49:07 15327 [Note] WSREP: Flow-control interval: [0, 0]
    2015-03-10 17:49:07 15327 [Note] WSREP: Received SELF-LEAVE. Closing connection.
    2015-03-10 17:49:07 15327 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 17)
    2015-03-10 17:49:07 15327 [Note] WSREP: RECV thread exiting 0: Success
    2015-03-10 17:49:07 15327 [Note] WSREP: recv_thread() joined.
    2015-03-10 17:49:07 15327 [Note] WSREP: Closing replication queue.
    2015-03-10 17:49:07 15327 [Note] WSREP: Closing slave action queue.
    2015-03-10 17:49:07 15327 [Note] WSREP: Service disconnected.
    2015-03-10 17:49:07 15327 [Note] WSREP: rollbacker thread exiting
    2015-03-10 17:49:08 15327 [Note] WSREP: Some threads may fail to exit.
    2015-03-10 17:49:08 15327 [Note] Binlog end
    2015-03-10 17:49:08 15327 [Note] /usr/sbin/mysqld: Shutdown complete

    Error in my_thread_global_end(): 1 threads didn't exit
    150310 17:49:13 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended


    Anybody some hints?

    Thx in advance

    Simon
  • jriverajrivera Percona Support Engineer Percona
    Can you also post error log from the DONOR node and post relevant wsrep_* variables for further analysis
  • wagnerbianchiwagnerbianchi Remote DBA Active Member Advisor
    2015-03-10 17:49:04 15327 [ERROR] WSREP: fork() failed: 12 (Cannot allocate memory)
    2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to execute: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.123.205' --auth 'root:Uz6oebu6' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15327' '' : 12 (Cannot allocate memory)
    2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
    2015-03-10 17:49:04 15327 [ERROR] Aborting

    It seems that the server where this is running has a small space in RAM and when xtrabackup comes in to prepare a backup, it use to allocate 100MB RAM by default and consequently, failing by a lack of RAM space. It's going to consider the size of dataset being copied to the backup set. Just to confirm, is this a small VM or similar thing?

    Additionally, the fork() called out my attention: http://en.wikipedia.org/wiki/Fork_%28system_call%29

    Cheers,
  • HAGBARDHAGBARD Entrant Active Member Participant
    I had the same problem as you, I've solved enabling SWAP in the DONOR and other nodes.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.