I use operator to create a new pxc cluster for test. I tried to delete the pvc on pod-0 and then kill the pod-0.
After testing, the data of pod-0 will be recovered correctly. However, the pitr service is modified from binlog synchronization based on pod-0 to pod-1. After that, I find that it cannot work normally. The errors are similar to the following
percona panic: runtime error: index out of range [19] with length 19
After checking the codes, I found that it was an error in filterBinLogs function, but I can’t see which line it was.
After confirmation, it seems that deleting the original s3 bucket data can solve this problem. At present, I think pitr (1.10.0) must remain in the same pod, is there a solution to this problem in actual production?
1 Like
Hi @cdmikechen !
I’m not able to reproduce your issue, I have tried several times to delete pvc and kill pods on which binlog collector was running and it was switching to another node without issues.
My “timeBetweenUploads” variable is set to 60 - what value did you use?
Can you share your cr.yaml?
Did you maybe purge binlogs or have some low value for “binlog_expire_logs_seconds” mysql variable?
Check my output below.
As you can see binlog collector is switching nodes and there are no errors.
Also I have done the restore from this and it seemed to work.
2022/06/03 15:10:07 run binlog collector
2022/06/03 15:10:08 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:10:08 Starting to process binlog with name binlog.000002
2022/06/03 15:10:09 Successfully written binlog file binlog.000002 to s3 with name binlog_1654268851_8cc9c27b22b15c7df59418a40a691a7b
2022/06/03 15:10:09 Starting to process binlog with name binlog.000003
2022/06/03 15:10:10 Successfully written binlog file binlog.000003 to s3 with name binlog_1654268909_8fb15a61ab6c705fe4047d080a4c7dad
2022/06/03 15:11:12 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:11:12 Starting to process binlog with name binlog.000004
2022/06/03 15:11:20 Successfully written binlog file binlog.000004 to s3 with name binlog_1654269008_d1ba5eeca104af7caa4f1c64011ff596
2022/06/03 15:12:22 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:12:22 Starting to process binlog with name binlog.000005
2022/06/03 15:12:49 Successfully written binlog file binlog.000005 to s3 with name binlog_1654269072_9b91b06596f42c88d5e9b8681cfa5c3f
2022/06/03 15:13:51 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:13:51 No binlogs to upload
2022/06/03 15:14:52 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:14:53 No binlogs to upload
2022/06/03 15:15:54 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:15:54 No binlogs to upload
2022/06/03 15:16:55 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:16:56 No binlogs to upload
2022/06/03 15:17:57 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:17:58 No binlogs to upload
2022/06/03 15:18:59 Reading binlogs from pxc with hostname= cluster1-pxc-1.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:18:59 Starting to process binlog with name binlog.000006
2022/06/03 15:19:34 Successfully written binlog file binlog.000006 to s3 with name binlog_1654268980_af6d5e53a80f7fafbb139b711c1bd67b
2022/06/03 15:20:35 Reading binlogs from pxc with hostname= cluster1-pxc-1.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:20:36 Starting to process binlog with name binlog.000007
2022/06/03 15:20:47 Successfully written binlog file binlog.000007 to s3 with name binlog_1654269539_b1e4ff28453c438254a60b69e641bb8b
2022/06/03 15:21:48 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:21:49 Starting to process binlog with name binlog.000009
2022/06/03 15:22:11 Successfully written binlog file binlog.000009 to s3 with name binlog_1654269268_cdc45372b750f4670cb5c7c8b658a4ba
2022/06/03 15:23:13 Reading binlogs from pxc with hostname= cluster1-pxc-1.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:23:13 Starting to process binlog with name binlog.000007
2022/06/03 15:23:23 Successfully written binlog file binlog.000007 to s3 with name binlog_1654269539_b1e4ff28453c438254a60b69e641bb8b
2022/06/03 15:23:23 Starting to process binlog with name binlog.000008
2022/06/03 15:23:36 Successfully written binlog file binlog.000008 to s3 with name binlog_1654269635_1fad82802aa09b6221a37497b5fe744c
2022/06/03 15:24:37 Reading binlogs from pxc with hostname= cluster1-pxc-1.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:24:38 No binlogs to upload
2022/06/03 15:25:39 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:25:39 Starting to process binlog with name binlog.000009
2022/06/03 15:26:02 Successfully written binlog file binlog.000009 to s3 with name binlog_1654269268_cdc45372b750f4670cb5c7c8b658a4ba
2022/06/03 15:26:02 Starting to process binlog with name binlog.000010
2022/06/03 15:26:12 Successfully written binlog file binlog.000010 to s3 with name binlog_1654269708_4abe6b0abe59514e67d6ac639b5f1901
2022/06/03 15:27:14 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:27:14 Starting to process binlog with name binlog.000011
2022/06/03 15:27:24 Successfully written binlog file binlog.000011 to s3 with name binlog_1654269939_b23fd8b2a56c3e7ac0bdc5e51af93bba
2022/06/03 15:28:25 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:28:26 No binlogs to upload
2022/06/03 15:29:27 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:29:27 No binlogs to upload
2022/06/03 15:30:29 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:30:29 No binlogs to upload
2022/06/03 15:31:30 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:31:31 No binlogs to upload
2022/06/03 15:32:32 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:32:32 No binlogs to upload
2022/06/03 15:33:34 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:33:34 No binlogs to upload
2022/06/03 15:34:35 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:34:36 No binlogs to upload
2022/06/03 15:35:37 Reading binlogs from pxc with hostname= cluster1-pxc-2.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:35:37 No binlogs to upload
2022/06/03 15:36:38 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:36:39 Starting to process binlog with name binlog.000011
2022/06/03 15:37:00 Successfully written binlog file binlog.000011 to s3 with name binlog_1654269732_9f06f3a2e634b17252bc61b3739cca1a
2022/06/03 15:38:01 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:38:02 Starting to process binlog with name binlog.000012
2022/06/03 15:40:02 Successfully written binlog file binlog.000012 to s3 with name binlog_1654270598_ac2b6a1db626f40c1f8555d5aea65c23
2022/06/03 15:41:04 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:41:04 No binlogs to upload
2022/06/03 15:42:06 Reading binlogs from pxc with hostname= cluster1-pxc-0.cluster1-pxc.pxc-test.svc.cluster.local
2022/06/03 15:42:06 No binlogs to upload
1 Like