Percona XtraDB cluster not available after bootstrap the first node

percona_mm · October 9, 2024, 4:19pm

We follow the documentation to install Percona XtraDB Cluster on Debian, we install successfully on 3 nodes.
Then we configure nodes for write-set replication.
We bootstrap the first node, checking the error log, we have this warnings:

2024-10-09T10:25:27.819236Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-10-09T10:25:27.819299Z 0 [Warning] [MY-000000] [WSREP] Node is not a cluster node. Disabling pxc_strict_mode
2024-10-09T10:25:27.821423Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.37-29.1) starting as process 32939
2024-10-09T10:25:27.842394Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

Checking the processes, we have:

root@10:~# ps -ef | grep sql
mysql 32939 1 2 11:25 ? 00:00:06 /usr/sbin/mysqld --wsrep-new-cluster --wsrep_start_position=ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:5

Checking the cluster status:

root@10:~# clustercheck
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57

Percona XtraDB Cluster Node is not synced or non-PRIM.

We have cluster size 0, cluster_status disconnected
Also we just have 14 rows instead of 40 that are refered on documentation.

Config file:

# Template my.cnf for PXC
# Edit to your requirements.
[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
server-id=1
datadir=/var/lib/mysql
user=mysql
socket=/var/run/mysqld/mysqld.sock
log-error=/var/log/mysql/error.log
pid-file=/var/run/mysqld/mysqld.pid
wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem”

[sst]
encrypt=4
ssl-key=server-key.pem
ssl-ca=ca.pem
ssl-cert=server-cert.pem

# Binary log expiration period is 604800 seconds, which equals 7 days
binlog_expire_logs_seconds=604800

######## wsrep ###############
# Path to Galera library
wsrep_provider=/usr/lib/galera4/libgalera_smm.so

# Cluster connection URL contains IPs of nodes
#If no IP is found, this implies that a new cluster needs to be created,
#in order to do that you need to bootstrap this node
# wsrep_cluster_address=gcomm://
wsrep_cluster_address=gcomm://ip1,ip2,ip3

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# Slave thread to use
wsrep_slave_threads=8

wsrep_log_conflicts

default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node IP address
wsrep_node_address=ip1

# Cluster name
wsrep_cluster_name=pxc-cluster

#If wsrep_node_name is not specified,  then system hostname will be used
wsrep_node_name=pxc1

#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER
pxc_strict_mode=ENFORCING

# SST method
wsrep_sst_method=xtrabackup-v2

# For debug
wsrep_debug=1

We put wsrep_debug=1, but we don´t receive more information on error log.

We don´t have firewall active.

Please advice
Thanks

matthewb · October 9, 2024, 5:01pm

You ran systemctl start mysql@bootstrap ? There should have been a lot more info the log file than what you show above.

percona_mm · October 10, 2024, 7:32am

Yes, I ran systemctl start mysql@bootstrap.
Here is the complete log:
INFO: Skipping wsrep-recover for ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:5 pair
INFO: Assigning ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:5 to wsrep_start_position
2024-10-10T07:31:10.224209Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-10-10T07:31:10.224292Z 0 [Warning] [MY-000000] [WSREP] Node is not a cluster node. Disabling pxc_strict_mode
2024-10-10T07:31:10.226419Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.37-29.1) starting as process 36748
2024-10-10T07:31:10.247349Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-10-10T07:31:10.247411Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-10-10T07:31:10.255770Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-10-10T07:31:10.538969Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-10-10T07:31:10.705258Z 1 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-10-10T07:31:10.720911Z 1 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-10-10T07:31:10.924888Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2024-10-10T07:31:10.924978Z 0 [Note] [MY-000000] [Galera] Loading provider none initial position: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:5
2024-10-10T07:31:10.925087Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘none’
2024-10-10T07:31:10.925125Z 0 [Note] [MY-000000] [Galera] Symbol ‘wsrep_node_isolation_mode_set_v1’ not found from provider
2024-10-10T07:31:10.932164Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: ‘::’ port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-10-10T07:31:10.932253Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.37-29.1’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel29, Revision d29a325, WSREP version 26.1.4.3.

Thks

matthewb · October 10, 2024, 1:29pm

Your my.cnf is bad. You started a new [sst] section and all parameters that come after this are tied to that section. Move the sst section to the bottom so that all those wsrep_ params are part of the mysqld section

percona_mm · October 10, 2024, 4:59pm

Thank you so much Matthewb, I move the sst block to bottom of configuration file and now we have the expected output when run “show status like ‘wsrep%’;”

However the output of clustercheck is the same:
root@10:/etc/mysql# clustercheck
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57
Percona XtraDB Cluster Node is not synced or non-PRIM.

I add the second node, starting mysql “systemctl start mysql”

When I verify the expected output of “show status like ‘wsrep%’;”, all status are correct except the wsrep_cluster_size which maintains the value 1, instead of 2.

Please advice.
Thanks

matthewb · October 10, 2024, 6:44pm

The clustercheck script looks at this value:

SHOW STATUS LIKE 'wsrep_local_state';

What do you have for your first node?

Provide some logs? From both nodes?

percona_mm · October 11, 2024, 10:40am

On the first node we have value 4, on documentation is the expected output.

Log for first node, we have now Galera connection errors:

INFO: Skipping wsrep-recover for ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6 pair
INFO: Assigning ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6 to wsrep_start_position
2024-10-10T15:59:09.904912Z 0 [Warning] [MY-011070] [Server] ‘binlog_format’ is deprecated and will be removed in a future release.
2024-10-10T15:59:09.904933Z 0 [Warning] [MY-011068] [Server] The syntax ‘wsrep_slave_threads’ is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
2024-10-10T15:59:09.907721Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-10-10T15:59:09.909879Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.37-29.1) starting as process 1977
2024-10-10T15:59:09.917122Z 0 [Note] [MY-000000] [WSREP] Server initial position: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6
2024-10-10T15:59:09.930621Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-10-10T15:59:09.930678Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-10-10T15:59:09.931544Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6
2024-10-10T15:59:09.931575Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘/usr/lib/galera4/libgalera_smm.so’
2024-10-10T15:59:09.932626Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.20(8a53c64) by Codership Oy info@codership.com (modified by Percona https://percona.com/) loaded successfully.
2024-10-10T15:59:09.932729Z 0 [Note] [MY-000000] [Galera] Resolved symbol ‘wsrep_node_isolation_mode_set_v1’
2024-10-10T15:59:09.932790Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2024-10-10T15:59:09.933172Z 0 [ERROR] [MY-000000] [Galera] Unrecognized parameter ‘”socket.ssl_key’
2024-10-10T15:59:09.933269Z 0 [Note] [MY-000000] [Galera] not using SSL compression
2024-10-10T15:59:09.938483Z 0 [Note] [MY-000000] [Galera] Found saved state: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6, safe_to_bootstrap: 1
2024-10-10T15:59:09.938678Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9
Seqno: 1 - 6
Offset: 1280
Synced: 1
EncVersion: 1
Encrypted: 0
MasterKeyConst UUID: ec40b2a1-7fd0-11ef-bcff-2e79ecdbe7d9
MasterKey UUID: 00000000-0000-0000-0000-000000000000
MasterKey ID: 0
2024-10-10T15:59:09.938709Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9, offset: 1280
2024-10-10T15:59:09.938854Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% (0/134217752 bytes) complete.
2024-10-10T15:59:09.938932Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 100.0% (134217752/134217752 bytes) complete.
2024-10-10T15:59:09.938956Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-6
2024-10-10T15:59:09.938999Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% (0/1336 bytes) complete.
2024-10-10T15:59:09.939043Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 100.0% (1336/1336 bytes) complete.
2024-10-10T15:59:09.939062Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 0/6 locked buffers
2024-10-10T15:59:09.939078Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 134216392/134217728
2024-10-10T15:59:09.990930Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.81.50.21; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2024-10-10T15:59:10.013203Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-10-10T15:59:10.013370Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6, protocol version: -1
2024-10-10T15:59:10.013520Z 0 [Note] [MY-000000] [WSREP] refresh_provider_options: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.81.50.21; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; ist.recv_addr = 10.81.50.21; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2024-10-10T15:59:10.013581Z 0 [Note] [MY-000000] [WSREP] SR storage init for: table
2024-10-10T15:59:10.013612Z 0 [Note] [MY-000000] [WSREP] wsrep_start_replication
2024-10-10T15:59:10.013640Z 0 [Note] [MY-000000] [WSREP] Starting replication
2024-10-10T15:59:10.013680Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
2024-10-10T15:59:10.013721Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6
2024-10-10T15:59:10.013874Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2024-10-10T15:59:10.019261Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2024-10-10T15:59:10.019350Z 0 [Note] [MY-000000] [Galera] backend: asio
2024-10-10T15:59:10.019470Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2024-10-10T15:59:10.019848Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-10-10T15:59:10.019907Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-10-10T15:59:10.020363Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2024-10-10T15:59:10.020837Z 0 [Note] [MY-000000] [Galera] (962fecf1-9f8a, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567
2024-10-10T15:59:10.020873Z 0 [Note] [MY-000000] [Galera] (962fecf1-9f8a, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1
2024-10-10T15:59:10.021637Z 0 [Note] [MY-000000] [Galera] EVS version 1
2024-10-10T15:59:10.021926Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘pxc-cluster’
2024-10-10T15:59:10.022143Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
2024-10-10T15:59:10.023027Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1
2024-10-10T15:59:10.023075Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1
2024-10-10T15:59:10.023372Z 0 [Note] [MY-000000] [Galera] Node 962fecf1-9f8a state primary
2024-10-10T15:59:10.023532Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node

view (view_id(PRIM,962fecf1-9f8a,1)
memb {
962fecf1-9f8a,0
}
joined {
}
left {
}
partitioned {
}
)
2024-10-10T15:59:10.023561Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-10-10T15:59:10.039196Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://10.81.50.21:4567
2024-10-10T15:59:10.039229Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7f1af80058b0
2024-10-10T15:59:10.039308Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://10.81.50.22:4567
2024-10-10T15:59:10.039435Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7f1af8003450
2024-10-10T15:59:10.039474Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://10.81.50.23:4567
2024-10-10T15:59:10.039491Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7f1af8001550
2024-10-10T15:59:10.039694Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-10-10T15:59:10.039795Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-10-10T15:59:10.039759Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-10T15:59:10.040030Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2024-10-10T15:59:10.040210Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2024-10-10T15:59:10.040173Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-10T15:59:10.040395Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-10T15:59:10.040597Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-10-10T15:59:10.040781Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 963311e9-8720-11ef-ae89-a664f6e4ef33
2024-10-10T15:59:10.040878Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 963311e9-8720-11ef-ae89-a664f6e4ef33
2024-10-10T15:59:10.040956Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 963311e9-8720-11ef-ae89-a664f6e4ef33 from 0 (pxc1)
2024-10-10T15:59:10.041035Z 0 [Note] [MY-000000] [Galera] Quorum results:
version = 6,
component = PRIMARY,
conf_id = 0,
members = 1/1 (primary/total),
act_id = 6,
last_appl. = 6,
protocols = 4/11/4 (gcs/repl/appl),
vote policy= 0,
group UUID = ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9
2024-10-10T15:59:10.041155Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-10-10T15:59:10.041202Z 0 [Note] [MY-000000] [Galera] Restored state OPEN → JOINED (7)
2024-10-10T15:59:10.041281Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc1) synced with group.
2024-10-10T15:59:10.041327Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 7)
2024-10-10T15:59:10.041701Z 1 [Note] [MY-000000] [WSREP] wsrep running threads now: 1
2024-10-10T15:59:10.041739Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
2024-10-10T15:59:10.041761Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: End of file
2024-10-10T15:59:10.041713Z 2 [Note] [MY-000000] [WSREP] wsrep running threads now: 2
2024-10-10T15:59:10.042030Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
2024-10-10T15:59:10.042077Z 2 [Note] [MY-000000] [Galera] ####### processing CC 7, local, ordered
2024-10-10T15:59:10.042127Z 2 [Note] [MY-000000] [Galera] Maybe drain monitors from 6 upto current CC event 7 upto:6
2024-10-10T15:59:10.042162Z 2 [Note] [MY-000000] [Galera] Drain monitors from 6 up to 6
2024-10-10T15:59:10.042200Z 2 [Note] [MY-000000] [Galera] Process first view: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9 my uuid: 962fecf1-8720-11ef-9f8a-5331c45733e8
2024-10-10T15:59:10.042269Z 2 [Note] [MY-000000] [Galera] Server pxc1 connected to cluster at position ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:7 with ID 962fecf1-8720-11ef-9f8a-5331c45733e8
2024-10-10T15:59:10.042305Z 2 [Note] [MY-000000] [Galera] server pxc1 state change: disconnected → connected
2024-10-10T15:59:10.042332Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected → connected
2024-10-10T15:59:10.042357Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.042411Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 962fecf1-8720-11ef-9f8a-5331c45733e8
2024-10-10T15:59:10.042443Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 11), state transfer needed: no
2024-10-10T15:59:10.042505Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-10-10T15:59:10.042555Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-10-10T15:59:10.042581Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 11 (6)
2024-10-10T15:59:10.042612Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 → 7
2024-10-10T15:59:10.042645Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-10-10T15:59:10.043948Z 2 [Note] [MY-000000] [Galera] ================================================
View:
id: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:7
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: 962fecf1-8720-11ef-9f8a-5331c45733e8, pxc1

2024-10-10T15:59:10.044050Z 2 [Note] [MY-000000] [Galera] server pxc1 state change: connected → joiner
2024-10-10T15:59:10.044092Z 2 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2024-10-10T15:59:10.044126Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.044190Z 2 [Note] [MY-000000] [Galera] server pxc1 state change: joiner → initializing
2024-10-10T15:59:10.044207Z 2 [Note] [MY-000000] [WSREP] Server status change joiner → initializing
2024-10-10T15:59:10.044218Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.051847Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-10-10T15:59:10.356344Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-10-10T15:59:10.561487Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-10-10T15:59:10.561592Z 3 [Note] [MY-000000] [WSREP] SQL: 202 CREATE TABLE IF NOT EXISTS mysql.wsrep_cluster(cluster_uuid CHAR(36) PRIMARY KEY,view_id BIGINT NOT NULL,view_seqno BIGINT NOT NULL,protocol_version INT NOT NULL,capabilities INT NOT NULL) ENGINE=InnoDB thd: 3
2024-10-10T15:59:10.564411Z 3 [Note] [MY-000000] [WSREP] SQL: 203 CREATE TABLE IF NOT EXISTS mysql.wsrep_cluster_members(node_uuid CHAR(36) PRIMARY KEY,cluster_uuid CHAR(36) NOT NULL,node_name CHAR(32) NOT NULL,node_incoming_address VARCHAR(256) NOT NULL) ENGINE=InnoDB thd: 3
2024-10-10T15:59:10.565504Z 3 [Note] [MY-000000] [WSREP] SQL: 199 CREATE TABLE IF NOT EXISTS mysql.wsrep_streaming_log(node_uuid CHAR(36), trx_id BIGINT, seqno BIGINT, flags INT NOT NULL, frag LONGBLOB NOT NULL, PRIMARY KEY (node_uuid, trx_id, seqno)) ENGINE=InnoDB thd: 3
2024-10-10T15:59:10.572762Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-10-10T15:59:10.578095Z 0 [Note] [MY-000000] [WSREP] wsrep_plugin_init()
2024-10-10T15:59:10.629684Z 0 [Note] [MY-000000] [WSREP] Initial TC log open: binlog
2024-10-10T15:59:10.705318Z 0 [Note] [MY-000000] [WSREP] SYSTEM_THREAD_COMPRESS_GTID_TABLE declared as non wsrep_on
2024-10-10T15:59:10.773312Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2024-10-10T15:59:10.773359Z 0 [Note] [MY-000000] [Galera] Server initialized
2024-10-10T15:59:10.773377Z 0 [Note] [MY-000000] [Galera] server pxc1 state change: initializing → initialized
2024-10-10T15:59:10.773390Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized
2024-10-10T15:59:10.773429Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.773477Z 2 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2024-10-10T15:59:10.773495Z 2 [Note] [MY-000000] [WSREP] Setting WSREPXid (InnoDB): 00000000-0000-0000-0000-000000000000:-1
2024-10-10T15:59:10.773531Z 2 [Note] [MY-000000] [WSREP] Updating WSREPXid: 00000000-0000-0000-0000-000000000000:-1
2024-10-10T15:59:10.780527Z 9 [Note] [MY-000000] [WSREP] Recovered cluster id ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9
2024-10-10T15:59:10.781485Z 12 [Note] [MY-000000] [WSREP] wsrep running threads now: 4
2024-10-10T15:59:10.781730Z 2 [Note] [MY-000000] [Galera] server pxc1 state change: initialized → joined
2024-10-10T15:59:10.781829Z 2 [Note] [MY-000000] [WSREP] Server status change initialized → joined
2024-10-10T15:59:10.781879Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.781936Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.780386Z 11 [Note] [MY-000000] [WSREP] wsrep running threads now: 3
2024-10-10T15:59:10.784220Z 2 [Note] [MY-000000] [WSREP] Restored cluster view:
id: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:6
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: -1
members(1):
0: 3133c5be-8716-11ef-b777-3b2188aded94, pxc1

2024-10-10T15:59:10.784332Z 2 [Note] [MY-000000] [WSREP] Storing cluster view:
id: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:7
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: 962fecf1-8720-11ef-9f8a-5331c45733e8, pxc1

2024-10-10T15:59:10.784400Z 2 [Note] [MY-000000] [WSREP] SQL: 31 DELETE FROM mysql.wsrep_cluster thd: 2
2024-10-10T15:59:10.784591Z 2 [Note] [MY-000000] [WSREP] SQL: 39 DELETE FROM mysql.wsrep_cluster_members thd: 2
2024-10-10T15:59:10.784981Z 12 [Note] [MY-000000] [WSREP] Starting applier thread 12
2024-10-10T15:59:10.785305Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2024-10-10T15:59:10.785534Z 13 [Note] [MY-000000] [WSREP] wsrep running threads now: 5
2024-10-10T15:59:10.786000Z 2 [Note] [MY-000000] [WSREP] Setting WSREPXid (InnoDB): ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:7
2024-10-10T15:59:10.786116Z 2 [Note] [MY-000000] [WSREP] Updating WSREPXid: ec4c4e8d-7fd0-11ef-8d17-af9e54bcf7c9:7
2024-10-10T15:59:10.786667Z 13 [Note] [MY-000000] [WSREP] Starting applier thread 13
2024-10-10T15:59:10.787310Z 14 [Note] [MY-000000] [WSREP] wsrep running threads now: 6
2024-10-10T15:59:10.787401Z 14 [Note] [MY-000000] [WSREP] Starting applier thread 14
2024-10-10T15:59:10.787546Z 15 [Note] [MY-000000] [WSREP] wsrep running threads now: 7
2024-10-10T15:59:10.787673Z 15 [Note] [MY-000000] [WSREP] Starting applier thread 15
2024-10-10T15:59:10.787864Z 16 [Note] [MY-000000] [WSREP] wsrep running threads now: 8
2024-10-10T15:59:10.787963Z 16 [Note] [MY-000000] [WSREP] Starting applier thread 16
2024-10-10T15:59:10.788127Z 17 [Note] [MY-000000] [WSREP] wsrep running threads now: 9
2024-10-10T15:59:10.788221Z 17 [Note] [MY-000000] [WSREP] Starting applier thread 17
2024-10-10T15:59:10.788980Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 7
2024-10-10T15:59:10.789082Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 7
2024-10-10T15:59:10.789132Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-10-10T15:59:10.789206Z 2 [Note] [MY-000000] [Galera] Server pxc1 synced with group
2024-10-10T15:59:10.789285Z 2 [Note] [MY-000000] [Galera] server pxc1 state change: joined → synced
2024-10-10T15:59:10.789341Z 2 [Note] [MY-000000] [WSREP] Server status change joined → synced
2024-10-10T15:59:10.789383Z 2 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2024-10-10T15:59:10.789440Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T15:59:10.790854Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: ‘::’ port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-10-10T15:59:10.791129Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.37-29.1’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel29, Revision d29a325, WSREP version 26.1.4.3.

2024-10-10T15:59:10.791350Z 18 [Note] [MY-000000] [WSREP] Compress GTID table thread will run with wsrep_on=0/false
2024-10-10T15:59:45.299200Z 19 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ‘‘mysql_native_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’
2024-10-10T16:40:39.347807Z 21 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ‘‘mysql_native_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’
2024-10-10T17:14:05.730300Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:14:07.223546Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:14:08.723872Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
…
2024-10-10T17:27:09.956208Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:27:11.456813Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:27:12.956566Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:50:23.427994Z 22 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ‘‘mysql_native_password’ is deprecated and will be removed in a future release. Please use caching_sha2_password instead’

percona_mm · October 11, 2024, 10:41am

Log of second node:

INFO: Skipping wsrep-recover for 0ceacf31-7ffd-11ef-9263-8321177cfaee:6 pair
INFO: Assigning 0ceacf31-7ffd-11ef-9263-8321177cfaee:6 to wsrep_start_position
2024-10-10T16:46:57.030104Z 0 [Warning] [MY-011070] [Server] ‘binlog_format’ is deprecated and will be removed in a future release.
2024-10-10T16:46:57.030129Z 0 [Warning] [MY-011068] [Server] The syntax ‘wsrep_slave_threads’ is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
2024-10-10T16:46:57.032831Z 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-log-path: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2024-10-10T16:46:57.035018Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.37-29.1) starting as process 7065
2024-10-10T16:46:57.042191Z 0 [Note] [MY-000000] [WSREP] Server initial position: 0ceacf31-7ffd-11ef-9263-8321177cfaee:6
2024-10-10T16:46:57.056190Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-10-10T16:46:57.056261Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-10-10T16:46:57.057141Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: 0ceacf31-7ffd-11ef-9263-8321177cfaee:6
2024-10-10T16:46:57.057180Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library ‘/usr/lib/galera4/libgalera_smm.so’
2024-10-10T16:46:57.058344Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.20(8a53c64) by Codership Oy info@codership.com (modified by Percona https://percona.com/) loaded successfully.
2024-10-10T16:46:57.058381Z 0 [Note] [MY-000000] [Galera] Resolved symbol ‘wsrep_node_isolation_mode_set_v1’
2024-10-10T16:46:57.058435Z 0 [Note] [MY-000000] [Galera] CRC-32C: using 64-bit x86 acceleration.
2024-10-10T16:46:57.058839Z 0 [ERROR] [MY-000000] [Galera] Unrecognized parameter ‘”socket.ssl_key’
2024-10-10T16:46:57.058942Z 0 [Note] [MY-000000] [Galera] not using SSL compression
2024-10-10T16:46:57.064326Z 0 [Note] [MY-000000] [Galera] Found saved state: 0ceacf31-7ffd-11ef-9263-8321177cfaee:6, safe_to_bootstrap: 1
2024-10-10T16:46:57.064540Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
Version: 2
UUID: 0ceacf31-7ffd-11ef-9263-8321177cfaee
Seqno: 1 - 6
Offset: 1280
Synced: 1
EncVersion: 1
Encrypted: 0
MasterKeyConst UUID: 0ce32e13-7ffd-11ef-8feb-97e20f1df0a4
MasterKey UUID: 00000000-0000-0000-0000-000000000000
MasterKey ID: 0
2024-10-10T16:46:57.064570Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: 0ceacf31-7ffd-11ef-9263-8321177cfaee, offset: 1280
2024-10-10T16:46:57.064705Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 0.0% (0/134217752 bytes) complete.
2024-10-10T16:46:57.064788Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan… 100.0% (134217752/134217752 bytes) complete.
2024-10-10T16:46:57.064811Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-6
2024-10-10T16:46:57.064853Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 0.0% (0/1336 bytes) complete.
2024-10-10T16:46:57.064888Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan… 100.0% (1336/1336 bytes) complete.
2024-10-10T16:46:57.064908Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found 0/6 locked buffers
2024-10-10T16:46:57.064924Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: free space: 134216392/134217728
2024-10-10T16:46:57.066970Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.81.50.22; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2024-10-10T16:46:57.088595Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-10-10T16:46:57.088928Z 0 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 0ceacf31-7ffd-11ef-9263-8321177cfaee:6, protocol version: -1
2024-10-10T16:46:57.089065Z 0 [Note] [MY-000000] [WSREP] refresh_provider_options: allocator.disk_pages_encryption = no; allocator.encryption_cache_page_size = 32K; allocator.encryption_cache_size = 16777216; base_dir = /var/lib/mysql/; base_host = 10.81.50.22; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.encryption = no; gcache.encryption_cache_page_size = 32K; gcache.encryption_cache_size = 16777216; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_auto_evict_threshold = 0.75; gcs.fc_auto_evict_window = 0; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; ist.recv_addr = 10.81.50.22; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.wait_restored_prim_timeout = PT0S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 11; socket.checksum = 2; socket.recv_buf_size = auto; socket.send_buf_size = auto; socket.ssl = YES; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_key = server-key.pem; socket.ssl_reload = 1;
2024-10-10T16:46:57.089094Z 0 [Note] [MY-000000] [WSREP] SR storage init for: table
2024-10-10T16:46:57.089106Z 0 [Note] [MY-000000] [WSREP] wsrep_start_replication
2024-10-10T16:46:57.089117Z 0 [Note] [MY-000000] [WSREP] Starting replication
2024-10-10T16:46:57.089140Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
2024-10-10T16:46:57.089164Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 0ceacf31-7ffd-11ef-9263-8321177cfaee:6
2024-10-10T16:46:57.089282Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
2024-10-10T16:46:57.094717Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
2024-10-10T16:46:57.094761Z 0 [Note] [MY-000000] [Galera] backend: asio
2024-10-10T16:46:57.094860Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0
2024-10-10T16:46:57.095303Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown
2024-10-10T16:46:57.095474Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown
2024-10-10T16:46:57.095891Z 0 [Note] [MY-000000] [Galera] GMCast version 0
2024-10-10T16:46:57.096139Z 0 [Note] [MY-000000] [Galera] (4318ee36-874b, ‘ssl://0.0.0.0:4567’) listening at ssl://0.0.0.0:4567
2024-10-10T16:46:57.096233Z 0 [Note] [MY-000000] [Galera] (4318ee36-874b, ‘ssl://0.0.0.0:4567’) multicast: , ttl: 1
2024-10-10T16:46:57.096968Z 0 [Note] [MY-000000] [Galera] EVS version 1
2024-10-10T16:46:57.097223Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group ‘pxc-cluster’
2024-10-10T16:46:57.097374Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
2024-10-10T16:46:57.097840Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 → 1
2024-10-10T16:46:57.097964Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 → 1
2024-10-10T16:46:57.098104Z 0 [Note] [MY-000000] [Galera] Node 4318ee36-874b state primary
2024-10-10T16:46:57.098221Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,4318ee36-874b,1)
memb {
4318ee36-874b,0
}
joined {
}
left {
}
partitioned {
}
)
2024-10-10T16:46:57.098312Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-10-10T16:46:57.100641Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-10-10T16:46:57.100726Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-10-10T16:46:57.101349Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2024-10-10T16:46:57.101376Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2024-10-10T16:46:57.101661Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-10-10T16:46:57.101969Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 4319e375-8727-11ef-9dd9-fb77b415a2e0
2024-10-10T16:46:57.101987Z 1 [Note] [MY-000000] [WSREP] wsrep running threads now: 1
2024-10-10T16:46:57.102206Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 4319e375-8727-11ef-9dd9-fb77b415a2e0
2024-10-10T16:46:57.102442Z 1 [Note] [MY-000000] [WSREP] Starting applier thread 1
2024-10-10T16:46:57.102076Z 2 [Note] [MY-000000] [WSREP] wsrep running threads now: 2
2024-10-10T16:46:57.102538Z 2 [Note] [MY-000000] [WSREP] Starting rollbacker thread 2
2024-10-10T16:46:57.102761Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 4319e375-8727-11ef-9dd9-fb77b415a2e0 from 0 (pxc2)
2024-10-10T16:46:57.102932Z 0 [Note] [MY-000000] [Galera] Quorum results:

    version    = 6,
    component  = PRIMARY,
    conf_id    = 0,
    members    = 1/1 (primary/total),
    act_id     = 6,
    last_appl. = 6,
    protocols  = 4/11/4 (gcs/repl/appl),
    vote policy= 0,
    group UUID = 0ceacf31-7ffd-11ef-9263-8321177cfaee

2024-10-10T16:46:57.103233Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2024-10-10T16:46:57.103402Z 0 [Note] [MY-000000] [Galera] Restored state OPEN → JOINED (7)
2024-10-10T16:46:57.103590Z 0 [Note] [MY-000000] [Galera] Member 0.0 (pxc2) synced with group.
2024-10-10T16:46:57.103625Z 1 [Note] [MY-000000] [Galera] ####### processing CC 7, local, ordered
2024-10-10T16:46:57.103781Z 1 [Note] [MY-000000] [Galera] Maybe drain monitors from 6 upto current CC event 7 upto:6
2024-10-10T16:46:57.103806Z 1 [Note] [MY-000000] [Galera] Drain monitors from 6 up to 6
2024-10-10T16:46:57.103833Z 1 [Note] [MY-000000] [Galera] Process first view: 0ceacf31-7ffd-11ef-9263-8321177cfaee my uuid: 4318ee36-8727-11ef-874b-16fcc3a3b3c9
2024-10-10T16:46:57.103889Z 1 [Note] [MY-000000] [Galera] Server pxc2 connected to cluster at position 0ceacf31-7ffd-11ef-9263-8321177cfaee:7 with ID 4318ee36-8727-11ef-874b-16fcc3a3b3c9
2024-10-10T16:46:57.103911Z 1 [Note] [MY-000000] [Galera] server pxc2 state change: disconnected → connected
2024-10-10T16:46:57.103927Z 1 [Note] [MY-000000] [WSREP] Server status change disconnected → connected
2024-10-10T16:46:57.103951Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T16:46:57.104017Z 1 [Note] [MY-000000] [Galera] ####### My UUID: 4318ee36-8727-11ef-874b-16fcc3a3b3c9
2024-10-10T16:46:57.104052Z 1 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 11), state transfer needed: no
2024-10-10T16:46:57.104085Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-10-10T16:46:57.104113Z 1 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2024-10-10T16:46:57.104141Z 1 [Note] [MY-000000] [Galera] REPL Protocols: 11 (6)
2024-10-10T16:46:57.104161Z 1 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 → 7
2024-10-10T16:46:57.104189Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2024-10-10T16:46:57.103752Z 0 [Note] [MY-000000] [Galera] Shifting JOINED → SYNCED (TO: 7)
2024-10-10T16:46:57.127791Z 1 [Note] [MY-000000] [Galera] ================================================
View:
id: 0ceacf31-7ffd-11ef-9263-8321177cfaee:7
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: 4318ee36-8727-11ef-874b-16fcc3a3b3c9, pxc2

2024-10-10T16:46:57.127828Z 1 [Note] [MY-000000] [Galera] server pxc2 state change: connected → joiner
2024-10-10T16:46:57.127842Z 1 [Note] [MY-000000] [WSREP] Server status change connected → joiner
2024-10-10T16:46:57.127854Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T16:46:57.127878Z 1 [Note] [MY-000000] [Galera] server pxc2 state change: joiner → initializing
2024-10-10T16:46:57.127894Z 1 [Note] [MY-000000] [WSREP] Server status change joiner → initializing
2024-10-10T16:46:57.127904Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T16:46:57.136235Z 3 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-10-10T16:46:57.443192Z 3 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-10-10T16:46:57.607793Z 3 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2024-10-10T16:46:57.607842Z 3 [Note] [MY-000000] [WSREP] SQL: 202 CREATE TABLE IF NOT EXISTS mysql.wsrep_cluster(cluster_uuid CHAR(36) PRIMARY KEY,view_id BIGINT NOT NULL,view_seqno BIGINT NOT NULL,protocol_version INT NOT NULL,capabilities INT NOT NULL) ENGINE=InnoDB thd: 3
2024-10-10T16:46:57.609220Z 3 [Note] [MY-000000] [WSREP] SQL: 203 CREATE TABLE IF NOT EXISTS mysql.wsrep_cluster_members(node_uuid CHAR(36) PRIMARY KEY,cluster_uuid CHAR(36) NOT NULL,node_name CHAR(32) NOT NULL,node_incoming_address VARCHAR(256) NOT NULL) ENGINE=InnoDB thd: 3
2024-10-10T16:46:57.610242Z 3 [Note] [MY-000000] [WSREP] SQL: 199 CREATE TABLE IF NOT EXISTS mysql.wsrep_streaming_log(node_uuid CHAR(36), trx_id BIGINT, seqno BIGINT, flags INT NOT NULL, frag LONGBLOB NOT NULL, PRIMARY KEY (node_uuid, trx_id, seqno)) ENGINE=InnoDB thd: 3
2024-10-10T16:46:57.619032Z 3 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2024-10-10T16:46:57.624510Z 0 [Note] [MY-000000] [WSREP] wsrep_plugin_init()
2024-10-10T16:46:57.676210Z 0 [Note] [MY-000000] [WSREP] Initial TC log open: binlog
2024-10-10T16:46:57.783007Z 0 [Note] [MY-000000] [WSREP] SYSTEM_THREAD_COMPRESS_GTID_TABLE declared as non wsrep_on
2024-10-10T16:46:57.830860Z 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2024-10-10T16:46:57.830907Z 0 [Note] [MY-000000] [Galera] Server initialized
2024-10-10T16:46:57.830925Z 0 [Note] [MY-000000] [Galera] server pxc2 state change: initializing → initialized
2024-10-10T16:46:57.830938Z 0 [Note] [MY-000000] [WSREP] Server status change initializing → initialized
2024-10-10T16:46:57.830959Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T16:46:57.831022Z 1 [Note] [MY-000000] [Galera] Bootstrapping a new cluster, setting initial position to 00000000-0000-0000-0000-000000000000:-1
2024-10-10T16:46:57.831053Z 1 [Note] [MY-000000] [WSREP] Setting WSREPXid (InnoDB): 00000000-0000-0000-0000-000000000000:-1
2024-10-10T16:46:57.831070Z 1 [Note] [MY-000000] [WSREP] Updating WSREPXid: 00000000-0000-0000-0000-000000000000:-1
2024-10-10T16:46:57.834718Z 12 [Note] [MY-000000] [WSREP] wsrep running threads now: 4
2024-10-10T16:46:57.834850Z 12 [Note] [MY-000000] [WSREP] Starting applier thread 12
2024-10-10T16:46:57.834968Z 13 [Note] [MY-000000] [WSREP] wsrep running threads now: 5
2024-10-10T16:46:57.835036Z 13 [Note] [MY-000000] [WSREP] Starting applier thread 13
2024-10-10T16:46:57.835336Z 15 [Note] [MY-000000] [WSREP] wsrep running threads now: 7
2024-10-10T16:46:57.835400Z 15 [Note] [MY-000000] [WSREP] Starting applier thread 15
2024-10-10T16:46:57.835470Z 16 [Note] [MY-000000] [WSREP] wsrep running threads now: 8
2024-10-10T16:46:57.835511Z 16 [Note] [MY-000000] [WSREP] Starting applier thread 16
2024-10-10T16:46:57.835448Z 14 [Note] [MY-000000] [WSREP] wsrep running threads now: 6
2024-10-10T16:46:57.835593Z 14 [Note] [MY-000000] [WSREP] Starting applier thread 14
2024-10-10T16:46:57.837384Z 17 [Note] [MY-000000] [WSREP] wsrep running threads now: 9
2024-10-10T16:46:57.837459Z 17 [Note] [MY-000000] [WSREP] Starting applier thread 17
2024-10-10T16:46:57.837617Z 10 [Note] [MY-000000] [WSREP] Recovered cluster id 0ceacf31-7ffd-11ef-9263-8321177cfaee
2024-10-10T16:46:57.834733Z 11 [Note] [MY-000000] [WSREP] wsrep running threads now: 4
2024-10-10T16:46:57.838619Z 1 [Note] [MY-000000] [Galera] server pxc2 state change: initialized → joined
2024-10-10T16:46:57.838651Z 1 [Note] [MY-000000] [WSREP] Server status change initialized → joined
2024-10-10T16:46:57.838665Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T16:46:57.838694Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T16:46:57.840655Z 11 [Note] [MY-000000] [WSREP] Starting applier thread 11
2024-10-10T16:46:57.840913Z 1 [Note] [MY-000000] [WSREP] Restored cluster view:
id: 0ceacf31-7ffd-11ef-9263-8321177cfaee:6
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: -1
members(1):
0: ff51a450-8723-11ef-b847-af1c95d74f32, pxc2

2024-10-10T16:46:57.840970Z 1 [Note] [MY-000000] [WSREP] Storing cluster view:
id: 0ceacf31-7ffd-11ef-9263-8321177cfaee:7
status: primary
protocol_version: 4
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
final: no
own_index: 0
members(1):
0: 4318ee36-8727-11ef-874b-16fcc3a3b3c9, pxc2

2024-10-10T16:46:57.841005Z 1 [Note] [MY-000000] [WSREP] SQL: 31 DELETE FROM mysql.wsrep_cluster thd: 1
2024-10-10T16:46:57.841213Z 1 [Note] [MY-000000] [WSREP] SQL: 39 DELETE FROM mysql.wsrep_cluster_members thd: 1
2024-10-10T16:46:57.843746Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: ‘::’ port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-10-10T16:46:57.843845Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: ‘8.0.37-29.1’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 Percona XtraDB Cluster (GPL), Release rel29, Revision d29a325, WSREP version 26.1.4.3.
2024-10-10T16:46:57.844033Z 18 [Note] [MY-000000] [WSREP] Compress GTID table thread will run with wsrep_on=0/false
2024-10-10T16:46:57.851540Z 1 [Note] [MY-000000] [WSREP] Setting WSREPXid (InnoDB): 0ceacf31-7ffd-11ef-9263-8321177cfaee:7
2024-10-10T16:46:57.851739Z 1 [Note] [MY-000000] [WSREP] Updating WSREPXid: 0ceacf31-7ffd-11ef-9263-8321177cfaee:7
2024-10-10T16:46:57.853013Z 1 [Note] [MY-000000] [Galera] Recording CC from group: 7
2024-10-10T16:46:57.853211Z 1 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 7
2024-10-10T16:46:57.853374Z 1 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2024-10-10T16:46:57.853571Z 1 [Note] [MY-000000] [Galera] Server pxc2 synced with group
2024-10-10T16:46:57.853733Z 1 [Note] [MY-000000] [Galera] server pxc2 state change: joined → synced
2024-10-10T16:46:57.853878Z 1 [Note] [MY-000000] [WSREP] Server status change joined → synced
2024-10-10T16:46:57.854048Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections
2024-10-10T16:46:57.854203Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2024-10-10T17:14:05.727747Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:14:07.723097Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:14:09.223360Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
…
2024-10-10T17:27:08.954542Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:27:10.455345Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error
2024-10-10T17:27:11.956129Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: tlsv1 alert decrypt error

Thanks

matthewb · October 11, 2024, 2:14pm

Both of your logs have this. You have SSL issues. I recommend disabling all SSL functionality until you’ve established the cluster.

percona_mm · October 11, 2024, 3:04pm

We commented all the lines refering to ssl on configuration file, but the output of clustercheck is the same.

root@10:/etc/mysql# clustercheck
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57
Percona XtraDB Cluster Node is not synced or non-PRIM.

Please advice.
Thanks

matthewb · October 11, 2024, 5:15pm

The output of the clustercheck script is the last thing for you to look at. You need to be reading the error logs and looking for problems to correect.

percona_mm · October 15, 2024, 4:28pm

We disabled the SSL functionality and now we have less Galera errors in establishing connections, but still have this errors on node 1:

2024-10-15T15:36:06.612555Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-10-15T15:36:06.614152Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://10.81.50.21:4567
2024-10-15T15:36:06.614174Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7f039c0058b0
2024-10-15T15:36:06.614225Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://10.81.50.22:4567
2024-10-15T15:36:06.614240Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7f039c003450
2024-10-15T15:36:06.614266Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: ssl://10.81.50.23:4567
2024-10-15T15:36:06.614281Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7f039c001550
2024-10-15T15:36:06.614318Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-10-15T15:36:06.614365Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-15T15:36:06.614374Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-10-15T15:36:06.614386Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-15T15:36:06.614409Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-15T15:36:06.614476Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2024-10-15T15:36:06.614502Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2024-10-15T15:36:06.614600Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: End of file
2024-10-15T15:36:06.614829Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-10-15T15:36:06.614953Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 31adcacb-8b0b-11ef-8da3-d2403b55fe15
2024-10-15T15:36:06.614976Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 31adcacb-8b0b-11ef-8da3-d2403b55fe15
2024-10-15T15:36:06.615004Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 31adcacb-8b0b-11ef-8da3-d2403b55fe15 from 0 (pxc1)

Besides the error.log exists any other file where we can understand these errors ?
Please advice.

Thanks

matthewb · October 16, 2024, 5:55pm

It still indicates SSL usage in your error log (ssl://…) Set pxc-encrypt-cluster-traffic=OFF

percona_mm · October 18, 2024, 9:44am

Thanks for your reply.
We set pxc-encrypt-cluster-traffic to OFF and we have similar errors not ssl, but tcp:

2024-10-17T10:22:19.026304Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2024-10-17T10:22:19.028058Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://10.81.50.21:4567
2024-10-17T10:22:19.028093Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7fb9380058b0
2024-10-17T10:22:19.028143Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://10.81.50.22:4567
2024-10-17T10:22:19.028165Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7fb938003450
2024-10-17T10:22:19.028333Z 0 [Note] [MY-000000] [Galera] discarding pending addr without UUID: tcp://10.81.50.23:4567
2024-10-17T10:22:19.028354Z 0 [Note] [MY-000000] [Galera] discarding pending addr proto entry 0x7fb938001550
2024-10-17T10:22:19.028533Z 0 [Note] [MY-000000] [Galera] gcomm: connected
2024-10-17T10:22:19.028575Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
2024-10-17T10:22:19.029453Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-17T10:22:19.029613Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-17T10:22:19.029728Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: Operation aborted.
2024-10-17T10:22:19.029497Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED → OPEN (TO: 0)
2024-10-17T10:22:19.029941Z 0 [Note] [MY-000000] [Galera] Opened channel ‘pxc-cluster’
2024-10-17T10:22:19.030123Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2024-10-17T10:22:19.030335Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: b063de9b-8c71-11ef-9457-d3bd37e21b54
2024-10-17T10:22:19.030455Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: b063de9b-8c71-11ef-9457-d3bd37e21b54
2024-10-17T10:22:19.030820Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: b063de9b-8c71-11ef-9457-d3bd37e21b54 from 0 (pxc1)

Please advice.

matthewb · October 18, 2024, 12:09pm

Ensure ports 4444, 4567, 4568, and 3306 are open between all nodes.

dogs_Cute · October 19, 2024, 2:17am

You can check the settings of firewalld, selinux and AppArmor.

percona_mm · October 25, 2024, 1:02pm

Thanks for your replies.
We disable the sst, setting pxc-encrypt-cluster-traffic=OFF and the 2 other nodes joined the cluster without problems.
The replication tests were successful.

But when execute the clustercheck still receive the same message:

HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57
Percona XtraDB Cluster Node is not synced or non-PRIM.

matthewb · October 25, 2024, 2:40pm

You disabled SST? I’m not sure how you did that. If you don’t have SST, then nodes cannot sync if gcache is too small.

Ok. To clarify, all 3 nodes are together, and when you create a table on node1, you see the table on node3?

Using the same username/password as the clustercheck script, manually connect to the local node and run 'SHOW GLOBAL STATUS LIKE ‘wsrep%’ and verify this account has correct privileges.

percona_mm · October 25, 2024, 3:22pm

Thanks for your reply.
Sorry, I made a mistake, what we disabled was the SSL, setting pxc-encrypt-cluster-traffic=OFF
Yes, when we create a table on node 1, we see it on node 2 and node 3, we made all tests that are on your documentation to verify replication.
We executed clustercheck with root and connecting with root user on mysql we see the information expected, such as:
wsrep_cluster_size 3
wsrep_local_state 4
wsrep_local_state_comment Synced
wsrep_incoming_addresses 10.X.X.26:3306,10.X.X.24:3306,10.X.X.25:3306
wsrep_cluster_status Primary |
wsrep_connected ON
Any other wsrep variable that we should check ?

matthewb · October 25, 2024, 3:29pm

github.com

percona/percona-xtradb-cluster/blob/8.0/scripts/clustercheck.sh

#!/bin/bash 
#
# Script to make a proxy (ie HAProxy) capable of monitoring Percona XtraDB Cluster nodes properly
#
# Authors:
# Raghavendra Prabhu <raghavendra.prabhu@percona.com>
# Olaf van Zandwijk <olaf.vanzandwijk@nedap.com>
#
# Based on the original script from Unai Rodriguez and Olaf (https://github.com/olafz/percona-clustercheck)
#
# Grant privileges required:
# GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword!';

if [[ $1 == '-h' || $1 == '--help' ]];then
    echo "Usage: $0 <user> <pass> <available_when_donor=0|1> <log_file> <available_when_readonly=0|1> <defaults_extra_file>"
    exit
fi

MYSQL_USERNAME="${1-clustercheckuser}" 
MYSQL_PASSWORD="${2-clustercheckpassword!}"

This file has been truncated. show original

In your terminal, execute set -x, this will enable bash debugging. Then manually execute the check script. Examine the values for the variables. Determine what is not checking correctly. Provide all output here.

Topic		Replies	Views
Problem with starting bootstrap-pxc on fresh installation Debian 9.9 Percona XtraDB Cluster 5.x	1	855	July 22, 2019
Problem in adding 2nd node to Percona Xtradb cluster 8.0.35 Percona XtraDB Cluster 8.x troubleshooting , mysql , percona	2	338	April 2, 2024
New node is not able to join the cluster Percona XtraDB Cluster 8.x	1	1402	May 7, 2020
can't start percona cluster 8.019 for the first time. Percona XtraDB Cluster 8.x	9	1582	August 10, 2020
Setup Percona XtraDB Cluster with one node Percona XtraDB Cluster 5.x mysql , percona	2	1337	June 30, 2021

Percona XtraDB cluster not available after bootstrap the first node

Related topics