SST and Audit plugin (Failed but did not stop the process)

This is something I do not think I have seen before… I am unsure if this is a bug or if something else is going on.

Recently I had a server that needed a SST done on it (had been shut down for longer than the cache size) .
Everything went fine during the sst and brought itself back up and in the cluster properly. No Errors. I happened to notice that I was not recording items to the audit.log file and that it was empty. Doing a show plugins did not show the audit plugin enabled… As I did not have the server taking active traffic I took a Chance and loaded the plugin again. Another one of the servers immediately went offline and traffic moved to another system (I have a 5 Node Cluster) at that point in time I said ok… Not doing that again… As I now had two of 5 servers that were down. Here is the weird part… I “assumed” that maybe the first server that had just done a SST maybe something else had happened and I kept that one down.

I brought back up the Second server that went offline. While it came up it was in a out of sync process. Instead of taking the chance of fixing it and something else not going on correctly I had it init a new SST. (Simple means that I generally do is shut it down… remove the gcache file… Start it back up and it starts a SST from scratch again… Guess what… The plugin was not loaded on that one as well.

Looking though the post processing log (attached) I noticed the plugin was also not loaded. I figured ok maybe the first load it didnt pick it up… so I stopped it. Touched the audit.log file chown mysql:mysql audit.log and started it back up… Still nothing…

All servers are 8.0.26-16
The only thing I noticed is the xtrabackup does not match the server version
xtrabackup version 8.0.33-27 based on MySQL server 8.0.33

I downgraded to percona-xtrabackup-80-8.0.26-18.1.el7 on the crashed node and tried again. (Currently pending results)

I “Thought” that upgrading the xtrabackup would not cause a issue but does that need to match exactly the same version of the server?


---- Starting the MySQL server used for post-processing ----
2023-07-08T18:55:43.433722-04:00 0 [Warning] [MY-011068] [Server] The syntax 'log_slow_slave_statements' is deprecated and will be removed in a future release. Please use log_slow_replica_statements instead.
2023-07-08T18:55:43.433784-04:00 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2023-07-08T18:55:43.433797-04:00 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2023-07-08T18:55:43.433899-04:00 0 [Warning] [MY-011068] [Server] The syntax 'slave_load_tmpdir' is deprecated and will be removed in a future release. Please use replica_load_tmpdir instead.
2023-07-08T18:55:43.433919-04:00 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2023-07-08T18:55:43.433955-04:00 0 [Warning] [MY-011068] [Server] The syntax 'wsrep_slave_threads' is deprecated and will be removed in a future release. Please use wsrep_applier_threads instead.
2023-07-08T18:55:43.434008-04:00 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead.
2023-07-08T18:55:43.434074-04:00 0 [Warning] [MY-010084] [Server] options --log-slow-admin-statements, --log-queries-not-using-indexes and --log-slow-replica-statements have no effect if --slow-query-log is not set
2023-07-08T18:55:43.434098-04:00 0 [Warning] [MY-010915] [Server] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2023-07-08T18:55:43.434135-04:00 0 [Warning] [MY-010097] [Server] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2023-07-08T18:55:43.434200-04:00 0 [Warning] [MY-000000] [WSREP] Node is not a cluster node. Disabling pxc_strict_mode
2023-07-08T18:55:43.436403-04:00 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2023-07-08T18:55:43.436420-04:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.26-16.1) starting as process 59895
2023-07-08T18:55:43.440717-04:00 0 [Note] [MY-010163] [Server] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=tdclpbt1vd003-bin' to avoid this problem.
2023-07-08T18:55:43.440993-04:00 0 [Note] [MY-010306] [Server] Skipping generation of SSL certificates as --auto_generate_certs is set to OFF.
2023-07-08T18:55:43.441210-04:00 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_main. See below for the description of exact issue.
2023-07-08T18:55:43.441290-04:00 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL_CTX_new failed
2023-07-08T18:55:43.446050-04:00 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2023-07-08T18:55:43.448053-04:00 0 [Warning] [MY-010075] [Server] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 91695abe-1de2-11ee-8734-0050569851d1.
2023-07-08T18:55:43.451641-04:00 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2023-07-08T18:55:43.452038-04:00 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2023-07-08T18:55:43.453964-04:00 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-07-08T18:55:43.454126-04:00 1 [Note] [MY-013547] [InnoDB] Atomic write disabled
2023-07-08T18:55:43.454261-04:00 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2023-07-08T18:55:43.454390-04:00 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2023-07-08T18:55:43.454487-04:00 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2023-07-08T18:55:43.454593-04:00 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.11
2023-07-08T18:55:43.458120-04:00 1 [Note] [MY-013251] [InnoDB] Number of pools: 1
2023-07-08T18:55:43.458408-04:00 1 [Note] [MY-012951] [InnoDB] Using CPU crc32 instructions
2023-07-08T18:55:43.458907-04:00 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2023-07-08T18:55:43.459079-04:00 1 [Note] [MY-012204] [InnoDB] Scanning './'
2023-07-08T18:55:44.190539-04:00 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 1103 files.
2023-07-08T18:55:44.200285-04:00 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 30.000000G, instances = 30, chunk size =128.000000M
2023-07-08T18:55:45.022824-04:00 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2023-07-08T18:55:45.135016-04:00 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner and LRU manager thread priority can be changed. See the man page of setpriority().
2023-07-08T18:55:45.374801-04:00 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 81034095779340, whereas checkpoint_lsn = 81034095779350 and start_lsn = 81034095779328
2023-07-08T18:55:45.438853-04:00 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2023-07-08T18:55:45.439546-04:00 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ...
2023-07-08T18:55:45.439897-04:00 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2023-07-08T18:55:45.441888-04:00 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2023-07-08T18:55:45.444370-04:00 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2023-07-08T18:55:45.447220-04:00 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2023-07-08T18:55:45.447611-04:00 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 9675494676
2023-07-08T18:55:45.628575-04:00 1 [Note] [MY-011825] [InnoDB] Parallel initialization of rseg complete
2023-07-08T18:55:45.628857-04:00 1 [Note] [MY-011825] [InnoDB] Time taken to initialize rseg using 4 thread: 181262 ms.
2023-07-08T18:55:45.631164-04:00 1 [Note] [MY-012255] [InnoDB] Removed temporary tablespace data file: "ibtmp1"
2023-07-08T18:55:45.631388-04:00 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables
2023-07-08T18:55:45.631670-04:00 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-07-08T18:55:45.656311-04:00 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB.
2023-07-08T18:55:45.656786-04:00 1 [Note] [MY-013627] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/'
2023-07-08T18:55:45.670524-04:00 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2023-07-08T18:55:45.671651-04:00 1 [Note] [MY-012976] [InnoDB] Percona XtraDB ( 8.0.26-16 started; log sequence number 81034095779360
2023-07-08T18:55:45.672924-04:00 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-07-08T18:55:45.694966-04:00 1 [Note] [MY-011089] [Server] Data dictionary restarting version '80023'.
2023-07-08T18:55:45.971954-04:00 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2023-07-08T18:55:46.016551-04:00 1 [Warning] [MY-012351] [InnoDB] Tablespace 20914, name 'SWINFO/#sql-63e3_8', file './SWINFO/#sql-63e3_8.ibd' is missing!
2023-07-08T18:55:46.056831-04:00 1 [Note] [MY-012356] [InnoDB] Scanned 1106 tablespaces. Validated 1105. Found 1 missing.
2023-07-08T18:55:46.086434-04:00 1 [Note] [MY-000000] [WSREP] wsrep_init_schema_and_SR (nil)
2023-07-08T18:55:46.138779-04:00 1 [System] [MY-000000] [WSREP] PXC upgrade completed successfully
2023-07-08T18:55:46.139328-04:00 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2023-07-08T18:55:46.154954-04:00 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. socket: '/var/lib/mysql/mysqlx.sock''
mysqld: File '/app/mysql/log/audit.log' not found (OS errno 2 - No such file or directory)
2023-07-08T18:55:47.155623-04:00 0 [ERROR] [MY-000000] [Server] Plugin audit_log reported: 'Cannot open file /app/mysql/log/audit.log.'
2023-07-08T18:55:47.155971-04:00 0 [ERROR] [MY-000000] [Server] Plugin audit_log reported: 'Error: No such file or directory'
2023-07-08T18:55:47.156211-04:00 0 [ERROR] [MY-010202] [Server] Plugin 'audit_log' init function returned error.
2023-07-08T18:55:47.156440-04:00 0 [ERROR] [MY-010734] [Server] Plugin 'audit_log' registration as a AUDIT failed.
2023-07-08T18:55:47.199166-04:00 0 [Note] [MY-010902] [Server] Thread priority attribute setting in Resource Group SQL shall be ignored due to unsupported platform or insufficient privilege.
2023-07-08T18:55:47.218589-04:00 0 [Note] [MY-012487] [InnoDB] DDL log recovery : begin
2023-07-08T18:55:47.219555-04:00 0 [Note] [MY-012488] [InnoDB] DDL log recovery : end
2023-07-08T18:55:47.220378-04:00 0 [Note] [MY-011946] [InnoDB] Loading buffer pool(s) from /app/mysql/data/ib_buffer_pool
2023-07-08T18:55:47.232904-04:00 0 [Note] [MY-010306] [Server] Skipping generation of SSL certificates as --auto_generate_certs is set to OFF.
2023-07-08T18:55:47.235067-04:00 0 [Warning] [MY-010068] [Server] CA certificate /app/mysql/certs/ca.pem is self signed.
2023-07-08T18:55:47.235391-04:00 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-07-08T18:55:47.235697-04:00 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --sha256_password_auto_generate_rsa_keys as key files are present in data directory.
2023-07-08T18:55:47.236033-04:00 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --caching_sha2_password_auto_generate_rsa_keys as key files are present in data directory.
2023-07-08T18:55:47.238916-04:00 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/app' in the path is accessible to all OS users. Consider choosing a different directory.
2023-07-08T18:55:47.304772-04:00 0 [Note] [MY-000000] [WSREP] Initialized wsrep sidno 2
2023-07-08T18:55:47.305148-04:00 5 [Note] [MY-010051] [Server] Event Scheduler: scheduler thread started with id 5
2023-07-08T18:55:47.305516-04:00 0 [Note] [MY-000000] [Galera] Loading provider none initial position: 00000000-0000-0000-0000-000000000000:-1
2023-07-08T18:55:47.305828-04:00 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library 'none'
2023-07-08T18:55:47.306667-04:00 0 [Note] [MY-010137] [Server] Execution of init_file '/dev/stdin' started.
2023-07-08T18:55:47.342195-04:00 0 [Note] [MY-010138] [Server] Execution of init_file '/dev/stdin' ended.
2023-07-08T18:55:47.342349-04:00 0 [Note] [MY-011240] [Server] Plugin mysqlx reported: 'Using SSL configuration from MySQL Server'
2023-07-08T18:55:47.343032-04:00 0 [Note] [MY-011243] [Server] Plugin mysqlx reported: 'Using OpenSSL for TLS connections'
2023-07-08T18:55:47.343245-04:00 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/lib/mysql/mysqlx.sock
2023-07-08T18:55:47.343339-04:00 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.26-16.1'  socket: '/app/mysql/data/upgrd.BGu9/my.sock'  port: 0  Percona XtraDB Cluster (GPL), Release rel16, Revision b141904, WSREP version 26.4.3.
2023-07-08T18:55:47.373817-04:00 11 [System] [MY-013172] [Server] Received SHUTDOWN from user mysql.pxc.sst.user. Shutting down mysqld (Version: 8.0.26-16.1).
2023-07-08T18:55:47.374512-04:00 0 [Note] [MY-010067] [Server] Giving 1 client threads a chance to die gracefully
2023-07-08T18:55:47.374555-04:00 0 [Note] [MY-010117] [Server] Shutting down slave threads
2023-07-08T18:55:47.374574-04:00 0 [Note] [MY-010054] [Server] Event Scheduler: Killing the scheduler thread, thread id 5
2023-07-08T18:55:47.374592-04:00 0 [Note] [MY-010050] [Server] Event Scheduler: Waiting for the scheduler thread to reply
2023-07-08T18:55:47.374676-04:00 0 [Note] [MY-010048] [Server] Event Scheduler: Stopped
2023-07-08T18:55:47.374693-04:00 0 [Note] [MY-010118] [Server] Forcefully disconnecting 0 remaining clients
2023-07-08T18:55:47.374727-04:00 0 [Note] [MY-010043] [Server] Event Scheduler: Purging the queue. 0 events
2023-07-08T18:55:47.376024-04:00 0 [Note] [MY-012330] [InnoDB] FTS optimize thread exiting.
2023-07-08T18:55:48.073212-04:00 0 [Note] [MY-010120] [Server] Binlog end
2023-07-08T18:55:48.075462-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'authentication_ldap_simple'
2023-07-08T18:55:48.075496-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysqlx'
2023-07-08T18:55:48.075914-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysqlx_cache_cleaner'
2023-07-08T18:55:48.075931-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'ngram'
2023-07-08T18:55:48.075939-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'BLACKHOLE'
2023-07-08T18:55:48.075948-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'ARCHIVE'
2023-07-08T18:55:48.075955-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'TempTable'
2023-07-08T18:55:48.075962-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'MRG_MYISAM'
2023-07-08T18:55:48.075969-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'MyISAM'
2023-07-08T18:55:48.075983-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CHANGED_PAGES'
2023-07-08T18:55:48.075991-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESPACES_SCRUBBING'
2023-07-08T18:55:48.075998-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESPACES_ENCRYPTION'
2023-07-08T18:55:48.076005-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_SESSION_TEMP_TABLESPACES'
2023-07-08T18:55:48.076011-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CACHED_INDEXES'
2023-07-08T18:55:48.076018-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_VIRTUAL'
2023-07-08T18:55:48.076025-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_COLUMNS'
2023-07-08T18:55:48.076031-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESPACES'
2023-07-08T18:55:48.076038-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_INDEXES'
2023-07-08T18:55:48.076044-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLESTATS'
2023-07-08T18:55:48.076050-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TABLES'
2023-07-08T18:55:48.076057-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2023-07-08T18:55:48.076063-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2023-07-08T18:55:48.076070-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_CONFIG'
2023-07-08T18:55:48.076076-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2023-07-08T18:55:48.076083-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_DELETED'
2023-07-08T18:55:48.076089-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2023-07-08T18:55:48.076096-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_METRICS'
2023-07-08T18:55:48.076102-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2023-07-08T18:55:48.076109-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2023-07-08T18:55:48.076115-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2023-07-08T18:55:48.076121-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_BUFFER_PAGE'
2023-07-08T18:55:48.076128-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2023-07-08T18:55:48.076134-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2023-07-08T18:55:48.076140-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMPMEM_RESET'
2023-07-08T18:55:48.076153-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMPMEM'
2023-07-08T18:55:48.076159-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP_RESET'
2023-07-08T18:55:48.076166-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_CMP'
2023-07-08T18:55:48.076172-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'INNODB_TRX'
2023-07-08T18:55:48.076178-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'InnoDB'
2023-07-08T18:55:48.076207-04:00 0 [Note] [MY-013072] [InnoDB] Starting shutdown...
2023-07-08T18:55:48.076452-04:00 0 [Note] [MY-011946] [InnoDB] Buffer pool(s) load completed at 230708 18:55:48
2023-07-08T18:55:48.076505-04:00 0 [Note] [MY-011944] [InnoDB] Dumping buffer pool(s) to /app/mysql/data/ib_buffer_pool
2023-07-08T18:55:48.077058-04:00 0 [Note] [MY-011944] [InnoDB] Buffer pool(s) dump completed at 230708 18:55:48
2023-07-08T18:55:48.191331-04:00 0 [Note] [MY-013084] [InnoDB] Log background threads are being closed...
2023-07-08T18:55:49.666343-04:00 0 [Note] [MY-012980] [InnoDB] Shutdown completed; log sequence number 81034095827326
2023-07-08T18:55:49.666540-04:00 0 [Note] [MY-012255] [InnoDB] Removed temporary tablespace data file: "ibtmp1"
2023-07-08T18:55:49.666573-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'MEMORY'
2023-07-08T18:55:49.666585-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'CSV'
2023-07-08T18:55:49.666595-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'PERFORMANCE_SCHEMA'
2023-07-08T18:55:49.666637-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'wsrep'
2023-07-08T18:55:49.666645-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'daemon_keyring_proxy_plugin'
2023-07-08T18:55:49.666662-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'sha2_cache_cleaner'
2023-07-08T18:55:49.666673-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'caching_sha2_password'
2023-07-08T18:55:49.666684-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'sha256_password'
2023-07-08T18:55:49.666692-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'mysql_native_password'
2023-07-08T18:55:49.667661-04:00 0 [Note] [MY-010733] [Server] Shutting down plugin 'binlog'
2023-07-08T18:55:49.668883-04:00 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.26-16.1)  Percona XtraDB Cluster (GPL), Release rel16, Revision b141904, WSREP version 26.4.3.
---- Stopped the MySQL server used for post-processing ----

Ok here is the issue… has to be something weird on my side…

I noticed the log file says /app/mysql/log/audit.log

even though the my.cnf gives the full path of audit-log-file = /app/mysql/data/audit.log (Both systems)

I searched both the donor and the initial server and can not find any reference to /app/mysql/log/

Create /app/mysql/log/audit.log
Set permissions
Now the audit file is in the proper location