Hi, I do DevOps and I help the sys-admins with our infrastructure, also I do Software Architecture, team lead for the company I work for. Just another geek with decades chasing a Keyboard.Over the weekend the sys-admins contacted me because Nagios started reporting alerts about a Node down (Percona XtraDB services: Too many connections), and even we have another 3 servers on the cluster, and we have redundant (3X) Percona ProxySQL(s) with KeepAlive each one and a floating IP, some how the login page was not working properly, the sys-admin rebooted some VMs, and eventually the login page started working, then I don’t have all the logs that I might need to solve this problem.
Here, what I think it happened based on the logs and some Grafana history, the node 2 a.k.a MySQL2 (it is a Percona XtraDB of 4 nodes MySQL 5.7), how I was saying the Node 2 ran out of swap, and it crashes the services, I guess?When I SSH’ed into the box, it looked okay, the MySQL service was down, and I couldn’t bring it up, I had to delete the content of MySQL DATA DIR, and I just rebooted the VM, it came back and performed SST, and it joined the cluster.
But I was wondering, why even if the VM has some kind of service crashed, why the Percona ProxySQL might not have worked properly? It is possible the fact when his Node 2 crashed: it was made the rest of the Cluster malfunctioned?
Here is the data I have, I will appreciate if somebody can give me some lights.
root@tor-proxy1:~ # proxysql -V
ProxySQL version 1.4.14-percona-1.1, codename Truls
root@tor-proxy1:~ # keepalived -v
Keepalived v1.2.13 (05/28,2014)
About the Node 2 a.k.a: Tor-SQL2
root@tor-sql2:~ # mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 164958
Server version: 5.7.26-29-57-log Percona XtraDB Cluster (GPL), Release rel29, Revision 03540a3, WSREP version 31.37, wsrep_31.37
…
--------------
mysql Ver 14.14 Distrib 5.7.26-29, for debian-linux-gnu (x86_64) using 6.3
root@tor-sql2:~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.10 (jessie)
Release: 8.10
Codename: jessie
root@tor-sql2:~ # free -m
total used free shared buffers cached
Mem: 12031 11878 153 7 143 6569
-/+ buffers/cache: 5165 6866
Swap: 1287 53 1234
__________________________________________________________________
root@tor-sql2:~ # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
The log files from the Node 2:
2020-08-09T09:02:41.467244-05:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 17801ms. The settings might not be optimal. (flushed=11, during the time.)
2020-08-09T09:02:41.493441-05:00 0 [Warning] WSREP: last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT17.3006S), skipping check
2020-08-09T11:30:16.648914-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (338 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154778 OS file writes, 302500140 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:18.655204-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (660 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154778 OS file writes, 302500140 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:20.660979-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (982 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154778 OS file writes, 302500140 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:22.667209-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (1304 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154781 OS file writes, 302500143 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:24.675075-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (1626 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154781 OS file writes, 302500143 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:26.681053-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (1948 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:28.686826-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (2270 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
=====================================
2020-08-09 11:30:28 0x7fb342548700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 52 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 966241 srv_active, 0 srv_shutdown, 326081 srv_idle
srv_master_thread log flush and writes: 1292011
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 17376417
–Thread 140408109606656 has waited at btr0cur.cc line 4038 for 7.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x40e3678 created in file btr0sea.cc line 230
a writer (thread id 140408109606656) has reserved it in mode wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file btr0sea.ic line 128
Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-debian-binary/label_exp/min-jessie-x64/percona-xtradb-cluster-5.7-5.7.26-31.37/storage/innobase/btr/btr0cur.cc line 4038
–Thread 140407239509760 has waited at btr0sea.ic line 90 for 14.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x40e33b8 created in file btr0sea.cc line 230
a writer (thread id 140407239509760) has reserved it in mode wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file btr0sea.ic line 128
Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-debian-binary/label_exp/min-jessie-x64/percona-xtradb-cluster-5.7-5.7.26-31.37/storage/innobase/btr/btr0sea.cc line 1368
–Thread 140407916005120 has waited at btr0sea.cc line 1368 for 14.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x40e37d8 created in file btr0sea.cc line 230
a writer (thread id 140407916005120) has reserved it in mode wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file btr0sea.cc line 1271
Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-debian-binary/label_exp/min-jessie-x64/percona-xtradb-cluster-5.7-5.7.26-31.37/storage/innobase/btr/btr0sea.cc line 1368
–Thread 140408008324864 has waited at btr0sea.cc line 1368 for 14.00 seconds the semaphore:
X-lock (wait_ex) on RW-latch at 0x40e30f8 created in file btr0sea.cc line 230
a writer (thread id 140408008324864) has reserved it in mode wait exclusive
number of readers 1, waiters flag 0, lock_word: ffffffffffffffff
Last time read locked in file btr0sea.cc line 1271
Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-debian-binary/label_exp/min-jessie-x64/percona-xtradb-cluster-5.7-5.7.26-31.37/storage/innobase/btr/btr0sea.cc line 1368
OS WAIT ARRAY INFO: signal count 15031214
RW-shared spins 0, rounds 13508226, OS waits 4860916
RW-excl spins 0, rounds 261056790, OS waits 7311277
RW-sx spins 721591, rounds 15930790, OS waits 263517
Spin rounds per wait: 13508226.00 RW-shared, 261056790.00 RW-excl, 22.08 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 5617640631
Purge done for trx’s n:o < 5615381499 undo n:o < 0 state: running but idle
History list length 12133
LIST OF TRANSACTIONS FOR EACH SESSION:
—TRANSACTION 421888215678896, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
—TRANSACTION 5617640630, ACTIVE (PREPARED) 4 sec
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1
MySQL thread id 4, OS thread handle 140412749170432, query id 319455015 wsrep: committing write set (4962596669)
—TRANSACTION 5617640629, ACTIVE (PREPARED) 7 sec
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1
MySQL thread id 8, OS thread handle 140407805470464, query id 319455003 wsrep: committing write set (4962596668)
—TRANSACTION 5617640628, ACTIVE (PREPARED) 7 sec
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1
MySQL thread id 3, OS thread handle 140412749436672, query id 319455001 wsrep: committing write set (4962596667)
—TRANSACTION 5617640627, ACTIVE 7 sec
2 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 2
—TRANSACTION 5617640626, ACTIVE (PREPARED) 7 sec
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1
MySQL thread id 10, OS thread handle 140407805204224, query id 319454998 wsrep: committing write set (4962596666)
—TRANSACTION 5617640625, ACTIVE (PREPARED) 14 sec
2 lock struct(s), heap size 1136, 1 row lock(s), undo log entries 1
MySQL thread id 2, OS thread handle 140413268973312, query id 319454979 wsrep: committing write set (4962596665)
—TRANSACTION 5617640624, ACTIVE (PREPARED) 14 sec
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1
MySQL thread id 5, OS thread handle 140407805736704, query id 319454977 wsrep: committing write set (4962596664)
—TRANSACTION 5617640623, ACTIVE (PREPARED) 14 sec
1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1
MySQL thread id 7, OS thread handle 140412748904192, query id 319454975 wsrep: committing write set (4962596663)
—TRANSACTION 5617640622, ACTIVE 15 sec truncating table
mysql tables in use 1, locked 1
0 lock struct(s), heap size 1136, 0 row lock(s)
MySQL thread id 2561292, OS thread handle 140407241639680, query id 319454972 10.102.70.174 aperture System lock
TRUNCATE TABLE fd_tmp_ldap1
—TRANSACTION 5617640609, ACTIVE 27 sec fetching rows
mysql tables in use 1, locked 1
177591 lock struct(s), heap size 15818960, 919989 row lock(s), undo log entries 5
MySQL thread id 2745810, OS thread handle 140407230191360, query id 319454747 10.102.70.174 lms updating
DELETE FROM email WHERE modifiedtime < 1581435001 AND due < 1596987001
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: complete io for buf page (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 119, 0] ,
ibuf aio reads:, log i/o’s:, sync i/o’s:
Pending flushes (fsync) log: 0; buffer pool: 0
146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs
196.32 reads/s, 8200 avg bytes/read, 10.31 writes/s, 3.44 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 11840, seg size 11842, 1051097 merges
merged operations:
insert 1598223, delete mark 5873836, delete 677102
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 1182691, node heap has 2 buffer(s)
Hash table size 1182691, node heap has 4 buffer(s)
2020-08-09T11:30:30.692927-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (2592 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:32.698516-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (2914 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:34.704623-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (3236 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:36.708689-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (3570 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:38.714826-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (3892 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:40.721373-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (4214 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:42.727709-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (4536 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:44.734275-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (4858 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:44.746485-05:00 2937175 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:45.358929-05:00 2937176 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:45.680491-05:00 2937177 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:45.715803-05:00 2937178 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:45.715929-05:00 2937178 [Note] Aborted connection 2937178 to db: ‘unconnected’ user: ‘monitor3’ host: ‘10.102.70.173’ (Got an error reading communication packets)
2020-08-09T11:30:45.812135-05:00 2937179 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:46.087777-05:00 2937180 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:46.087979-05:00 2937180 [Note] Aborted connection 2937180 to db: ‘unconnected’ user: ‘monitor’ host: ‘10.102.70.174’ (Got an error reading communication packets)
2020-08-09T11:30:46.740813-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (5180 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:46.812271-05:00 2937181 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:46.899405-05:00 2937182 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:46.899552-05:00 2937182 [Note] Aborted connection 2937182 to db: ‘unconnected’ user: ‘monitor2’ host: ‘10.102.70.172’ (Got an error reading communication packets)
2020-08-09T11:30:47.812369-05:00 2937183 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:48.746573-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (5502 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:48.806408-05:00 2937184 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:49.805439-05:00 2937185 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:50.737774-05:00 2937186 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:50.752886-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (5824 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:50.805025-05:00 2937187 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:51.808104-05:00 2937188 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:52.760478-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (6146 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:52.808385-05:00 2937189 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:53.816095-05:00 2937190 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:54.766828-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (6468 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:54.814007-05:00 2937191 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:55.741529-05:00 2937192 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:55.812944-05:00 2937193 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:56.770006-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (6802 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:56.815299-05:00 2937194 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:57.811408-05:00 2937195 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:58.001755-05:00 2937196 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:58.001892-05:00 2937196 [Note] Aborted connection 2937196 to db: ‘unconnected’ user: ‘monitor3’ host: ‘10.102.70.173’ (Got an error reading communication packets)
2020-08-09T11:30:58.604508-05:00 2937197 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:58.604679-05:00 2937197 [Note] Aborted connection 2937197 to db: ‘unconnected’ user: ‘monitor’ host: ‘10.102.70.174’ (Got an error reading communication packets)
2020-08-09T11:30:58.777042-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (7124 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:30:58.817325-05:00 2937198 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:59.211006-05:00 2937199 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:30:59.211163-05:00 2937199 [Note] Aborted connection 2937199 to db: ‘unconnected’ user: ‘monitor2’ host: ‘10.102.70.172’ (Got an error reading communication packets)
2020-08-09T11:30:59.823100-05:00 2937200 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:00.739620-05:00 2937201 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:00.784097-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (7446 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:00.818248-05:00 2937202 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:01.820261-05:00 2937203 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:02.790029-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (7768 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:02.821015-05:00 2937204 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:03.812563-05:00 2937205 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:04.796097-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (8090 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:04.815490-05:00 2937206 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:05.743219-05:00 2937207 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:05.817437-05:00 2937208 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:06.802501-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (8412 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:06.817723-05:00 2937209 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:07.818845-05:00 2937210 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:08.808556-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (8734 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:08.816944-05:00 2937211 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:09.818613-05:00 2937212 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:10.607234-05:00 2937213 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:10.607438-05:00 2937213 [Note] Aborted connection 2937213 to db: ‘unconnected’ user: ‘monitor’ host: ‘10.102.70.174’ (Got an error reading communication packets)
2020-08-09T11:31:10.741465-05:00 2937214 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:10.748530-05:00 2937215 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:10.748664-05:00 2937215 [Note] Aborted connection 2937215 to db: ‘unconnected’ user: ‘monitor3’ host: ‘10.102.70.173’ (Got an error reading communication packets)
2020-08-09T11:31:10.814790-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (9056 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:10.818273-05:00 2937216 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:11.480883-05:00 2937217 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:11.481098-05:00 2937217 [Note] Aborted connection 2937217 to db: ‘unconnected’ user: ‘monitor2’ host: ‘10.102.70.172’ (Got an error reading communication packets)
2020-08-09T11:31:11.823392-05:00 2937218 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:12.821082-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (9378 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:12.822412-05:00 2937219 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:13.819080-05:00 2937220 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:14.815462-05:00 2937221 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:14.826821-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (9700 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:15.736319-05:00 2937222 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:15.817551-05:00 2937223 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:16.817435-05:00 2937224 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:16.830492-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (10034 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:17.817909-05:00 2937225 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:18.818582-05:00 2937226 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:18.836593-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (10356 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:19.823237-05:00 2937227 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:20.736808-05:00 2937228 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:20.816503-05:00 2937229 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:20.842492-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (10678 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:21.819867-05:00 2937230 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:22.818683-05:00 2937231 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:22.848854-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (11000 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:23.486339-05:00 2937232 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:23.486519-05:00 2937232 [Note] Aborted connection 2937232 to db: ‘unconnected’ user: ‘monitor3’ host: ‘10.102.70.173’ (Got an error reading communication packets)
2020-08-09T11:31:23.603763-05:00 2937233 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:23.603916-05:00 2937233 [Note] Aborted connection 2937233 to db: ‘unconnected’ user: ‘monitor’ host: ‘10.102.70.174’ (Got an error reading communication packets)
2020-08-09T11:31:23.751724-05:00 2937234 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:23.751910-05:00 2937234 [Note] Aborted connection 2937234 to db: ‘unconnected’ user: ‘monitor2’ host: ‘10.102.70.172’ (Got an error reading communication packets)
2020-08-09T11:31:23.820917-05:00 2937235 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:24.819117-05:00 2937236 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:24.854548-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (11322 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:25.737616-05:00 2937237 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:25.821179-05:00 2937238 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:26.823662-05:00 2937239 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:26.861294-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (11644 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:27.820431-05:00 2937241 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:28.822084-05:00 2937243 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:28.867814-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (11966 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:29.822387-05:00 2937244 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:30.737574-05:00 2937245 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:30.820891-05:00 2937246 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:30.873245-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (12288 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:31.248814-05:00 2937248 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:31.823468-05:00 2937249 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:32.824814-05:00 2937250 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:32.878824-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (12610 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:33.822865-05:00 2937252 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:34.822149-05:00 2937253 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:34.884769-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (12932 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:35.292191-05:00 2937254 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:35.735221-05:00 2937255 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:35.822817-05:00 2937256 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:36.021546-05:00 2937257 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:36.021789-05:00 2937257 [Note] Aborted connection 2937257 to db: ‘unconnected’ user: ‘monitor2’ host: ‘10.102.70.172’ (Got an error reading communication packets)
2020-08-09T11:31:36.118438-05:00 2937258 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:36.118573-05:00 2937258 [Note] Aborted connection 2937258 to db: ‘unconnected’ user: ‘monitor’ host: ‘10.102.70.174’ (Got an error reading communication packets)
2020-08-09T11:31:36.231120-05:00 2937259 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:36.231254-05:00 2937259 [Note] Aborted connection 2937259 to db: ‘unconnected’ user: ‘monitor3’ host: ‘10.102.70.173’ (Got an error reading communication packets)
2020-08-09T11:31:36.823414-05:00 2937260 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:36.888261-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (13266 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:37.825485-05:00 2937261 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:38.824950-05:00 2937262 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:38.895052-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (13588 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:39.824811-05:00 2937263 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:40.735169-05:00 2937264 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:40.825603-05:00 2937265 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:40.900933-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (13910 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:41.823850-05:00 2937266 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:42.826044-05:00 2937267 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:42.906306-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (14232 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:43.827545-05:00 2937268 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:44.824393-05:00 2937269 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:44.913031-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (14554 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-08-09T11:31:45.737092-05:00 2937270 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:45.826777-05:00 2937271 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:46.828557-05:00 2937272 [Note] WSREP: monitor wait failed for causal read (repl.causal_read_timeout): : 110 (Connection timed out)
at galerautils/src/gu_lock.hpp:wait():97
2020-08-09T11:31:46.919826-05:00 2745810 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (14876 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 146332592 OS file reads, 311154782 OS file writes, 302500144 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
2020-0