# One of my nodes is behind

**URL:** <https://forums.percona.com/t/one-of-my-nodes-is-behind/16821>\
**Category:** Percona XtraDB Cluster 8.x\
**Created:** [July 30, 2022, 9:47am UTC](https://forums.percona.com/t/one-of-my-nodes-is-behind/16821 "2022-07-30T09:47:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![danarashad](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@danarashad](https://forums.percona.com/u/danarashad)\
**Post date:** [July 30, 2022, 9:47am UTC](https://forums.percona.com/t/one-of-my-nodes-is-behind/16821/1 "2022-07-30T09:47:51Z")

</div>

I have a 5 node cluster. I run a statement on all of the servers to do a basic check. One table should all have the same date\_and\_time. However one of the nodes does not have that time. I have a multimaster setup. However my loading scripts are only writing to node1. I use `show status like 'wsrep_clu%';` and they are all reporting the same

```
  wsrep_cluster_weight 5                                    
  wsrep_cluster_capabilities
  wsrep_cluster_conf_id 53                                   
  wsrep_cluster_size 5                                    
  wsrep_cluster_state_uuid 76174b3b-01de-11ed-a719-bb52abb919c1 
  wsrep_cluster_status Primary                              

```

Using show processlist is showing a lot of I guess in process replications.

```
  db: NULL
  Command: Query
  Time: 9758
  State: wsrep: committed write set (623924)
   Info: NULL
  Time_ms: 11820

     Id: 8
     User: event_scheduler
     Host: localhost
      db: NULL
      Command: Daemon
     Time: 1446964
    State: Waiting on empty queue
     Info: NULL
    Time_ms: 1446964376
   Rows_sent: 0
   Rows_examined: 0

       Id: 11                                                                                                                                                     
      User: system user 
     Host:
       db: store1
     Command: Query
     Time: 9739
      State: wsrep: updating row for write-set (623925)
      Info: NULL
     Time_ms: 359
      Rows_sent: 0
    Rows_examined: 0

    User: system user
     Host:
       db: NULL
    Command: Query
     Time: 9760
    State: wsrep: committed write set (623922)
     Info: NULL
     Time_ms: 12429
    Rows_sent: 0
  Rows_examined: 0

```

Is there anyway to view the `wsrep: committed write set (###)`. To determine what exactly is the issue?

Also since the node is stuck and not accepting replications (or think it’s up to date) is it safe to just restart the node.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [July 31, 2022, 9:16pm UTC](https://forums.percona.com/t/one-of-my-nodes-is-behind/16821/2 "2022-07-31T21:16:08Z")

</div>

Restarting the node won’t change anything. In PXC/Galera, replication of events is synchronous, but apply is async. Verify that all 5 nodes have the exact same specifications, and exact same my.cnf parameters. Are all 5 nodes on the same LAN? I suggest installing and configuring PMM to correlate all 5 nodes onto the same graphs, that way you can see if one is hitting CPU/disk/memory issues.

---

<div class="post-metadata">

**Author:** ![danarashad](https://avatars.discourse-cdn.com/v4/letter/d/7ab992/32.png) [@danarashad](https://forums.percona.com/u/danarashad)\
**Post date:** [August 8, 2022, 11:42am UTC](https://forums.percona.com/t/one-of-my-nodes-is-behind/16821/3 "2022-08-08T11:42:17Z")

</div>

All are on the same lan, my.cnf parameters are the same with a few exceptions `wsrep_sst_donor,wsrep_cluster_address` those of course exclude the node’s ip and name in the donor. The server spec’s are slightly different on two of the nodes. But none of the boxes are maxing CPU/disk/memory when the node is not in sync.