Hi,
I have confiugred Percona XtraDB Cluster with 3 nodes, and it seems that after couple of minutes the data nodes are not synced anymore. Only after starting the cluster manally again, the data nodes are up-to-date.
Configuration of my.cnf(simialr at all nodes):
# Template my.cnf for PXC# Edit to your requirements.
[client]socket=/var/lib/mysql/mysql.sock
[mysqld]server-id=1datadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.socklog-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pidssl-ca=/etc/mysql/certs/ca.pemssl-cert=/etc/mysql/certs/server-cert.pemssl-key=/etc/mysql/certs/server-key.pem# Binary log expiration period is 604800 seconds, which equals 7 daysbinlog_expire_logs_seconds=604800
######## wsrep ################ Path to Galera librarywsrep_provider=/usr/lib64/galera4/libgalera_smm.so
# Cluster connection URL contains IPs of nodes#If no IP is found, this implies that a new cluster needs to be created,#in order to do that you need to bootstrap this nodewsrep_cluster_address=gcomm://x,y,z
# In order for Galera to work correctly binlog format should be ROWbinlog_format=ROW
# Slave thread to usewsrep_slave_threads=8
wsrep_log_conflicts
# This changes how InnoDB autoincrement locks are managed and is a requirement for Galerainnodb_autoinc_lock_mode=2
# Node IP addresswsrep_node_address=y# Cluster namewsrep_cluster_name=pxc_vault_cluster
#If wsrep_node_name is not specified, then system hostname will be usedwsrep_node_name=pxc_vault_node02
#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTERpxc_strict_mode=PERMISSIVE
# SST methodwsrep_sst_method=xtrabackup-v2pxc-encrypt-cluster-traffic=OFF
and the result of the wresp:
±---------------------------------±-----------------------------------------------------------------------------------------------------------------------------------------------+| Variable_name | Value |±---------------------------------±-----------------------------------------------------------------------------------------------------------------------------------------------+| wsrep_local_state_uuid | 9f5ec88f-9ec0-11ea-afec-0b7f88e7dceb || wsrep_protocol_version | 10 || wsrep_last_applied | 679 || wsrep_last_committed | 679 || wsrep_monitor_status (L/A/C) | [ (674, 674), (679, 679), (679, 679) ] || wsrep_replicated | 207 || wsrep_replicated_bytes | 97192 || wsrep_repl_keys | 621 || wsrep_repl_keys_bytes | 9936 || wsrep_repl_data_bytes | 73482 || wsrep_repl_other_bytes | 0 || wsrep_received | 466 || wsrep_received_bytes | 229000 || wsrep_local_commits | 207 || wsrep_local_cert_failures | 0 || wsrep_local_replays | 0 || wsrep_local_send_queue | 0 || wsrep_local_send_queue_max | 2 || wsrep_local_send_queue_min | 0 || wsrep_local_send_queue_avg | 0.00465116 || wsrep_local_recv_queue | 0 || wsrep_local_recv_queue_max | 1 || wsrep_local_recv_queue_min | 0 || wsrep_local_recv_queue_avg | 0 || wsrep_local_cached_downto | 3 || wsrep_flow_control_paused_ns | 0 || wsrep_flow_control_paused | 0 || wsrep_flow_control_sent | 0 || wsrep_flow_control_recv | 0 || wsrep_flow_control_interval | [ 173, 173 ] || wsrep_flow_control_interval_low | 173 || wsrep_flow_control_interval_high | 173 || wsrep_flow_control_status | OFF || wsrep_cert_deps_distance | 1 || wsrep_apply_oooe | 0 || wsrep_apply_oool | 0 || wsrep_apply_window | 1.0045 || wsrep_commit_oooe | 0 || wsrep_commit_oool | 0 || wsrep_commit_window | 1 || wsrep_local_state | 4 || wsrep_local_state_comment | Synced || wsrep_cert_index_size | 3 || wsrep_cert_bucket_count | 47 || wsrep_gcache_pool_size | 347680 || wsrep_causal_reads | 0 || wsrep_cert_interval | 0 || wsrep_open_transactions | 0 || wsrep_open_connections | 0 || wsrep_ist_receive_status | || wsrep_ist_receive_seqno_start | 0 || wsrep_ist_receive_seqno_current | 0 || wsrep_ist_receive_seqno_end | 0 || wsrep_incoming_addresses | y:3306,x:3306,z:3306 || wsrep_cluster_weight | 3 || wsrep_desync_count | 0 || wsrep_evs_delayed | || wsrep_evs_evict_list | || wsrep_evs_repl_latency | 0/0/0/0/0 || wsrep_evs_state | OPERATIONAL || wsrep_gcomm_uuid | 1b5247b8-9f2c-11ea-b80e-d33ece4dbfc9 || wsrep_cluster_capabilities | || wsrep_cluster_conf_id | 13 || wsrep_cluster_size | 3 || wsrep_cluster_state_uuid | 9f5ec88f-9ec0-11ea-afec-0b7f88e7dceb || wsrep_cluster_status | Primary || wsrep_connected | ON || wsrep_local_bf_aborts | 0 || wsrep_local_index | 0 || wsrep_provider_capabilities | :MULTI_MASTER:CERTIFICATION:PARALLEL_APPLYING:TRX_REPLAY:ISOLATION:PAUSE:CAUSAL_READS:INCREMENTAL_WRITESET:UNORDERED:PREORDERED:STREAMING:NBO: || wsrep_provider_name | Galera || wsrep_provider_vendor | Codership Oy <info@codership.com> || wsrep_provider_version | 4.3(r752664d) || wsrep_ready | ON || wsrep_thread_count | 9 |±---------------------------------±-----------------------------------------------------------------------------------------------------------------------------------------------+
What do you mean data is out of sync. Can you specifically show what you do, what behavior you expect vs what behavior you actually see.
Hi Peter,
Thanks for answering.
I am running 3 hashicorp vault applications.
Each node application is connected to local mysql database.
The 3 databases are in the Percona XtraDB cluster.
Data replication between nodes works OK for ~2 minutes only after all vault applications startup .
Then the data stop syncing between the nodes.
For example, in the vault application:
n1 - write secret ‘kuku’ -> n2, n3 has secret ‘kuku’
wait for two minutes
n1 - write secret ‘kuku2’ -> n2, n3 don’t receive the new secret.
Thanks,
Noy
How did you identify what the data not propagating properly on Percona XtraDB Level is the cause of Vault malfunction ?
Can you validate and see different data on the nodes ?