I am trying to restore my backup and facing some issue

[Aditya1523]

When I am trying to do backup and restore procedure through perconaxtradbclusterrestore i am getting some error (attached).

[lalit.choudhary]

Hello @Aditya1523

Error look similar to Xtrabackup fails with errors on check point file

You should check below points,

• Check the backup log for any warnings or errors.
• Check backup directory permission and if you have/are able to access the xtrabackup_checkpoints file.

[Aditya1523]

Hi lalit,
I am trying to restore my data from s3 backup (cloudflare) where i have a bucket name mysql-backup-dev and there are three things backed up: pxccluster-2024-04-22-02:00:41-full.sst_info/
pxccluster-2024-04-22-02:00:41-full/
[pxccluster-2024-02-09-02:00:53-full.md5], and I have taken the resource destination as pxccluster-2024-04-22-02:00:41-full

yaml file for restoration:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterRestore
metadata:
name: restore1
spec:
pxcCluster: pxc-cluster
backupSource:
destination: s3://mysql-backup-dev/pxccluster-2024-04-21-02:00:30-full
s3:
bucket: mysql-backup-dev
credentialsSecret: s3-backup-secret
region: Asia-Pacific
endpointUrl: https://4b924987231b504ca149.r5.cloudflarestorage.com
After doing perconaxtradbclusterrestore pods are going into shut down state and never coming nac

[Aditya1523]

@lalit.choudhary Can you help me on this I am stuck,
Thanks.

[matthewb]

@Aditya1523,
The error log says “xbcloud: Probe failed. Please check your credentials and endpoint settings” It looks like your configuration is not correct.

[Aditya1523]

Hi Matthew, Is there any way to connect with you on Zoom or anything so that I can debug this issue, as per my knowledge the secrets I used are correct.

[matthewb]

Hello @Aditya1523,
You would need to engage with a support contract in order to connect on Zoom.

[Aditya1523]

I am using the correct credentials for secrets for s3, and I am providing my YAML file can you tell me where I am wrong I have tried everything to correct my YAML, but no luck, Location:Western Europe (WEUR) so I provided in region:eu-west-1

apiVersion: v1
kind: Secret
metadata:
name: s3-backup-secrets
namespace: pxc
immutable: false
stringData:
AWS_ACCESS_KEY_ID: HJNFJRyMjljNmZiNmQ1R
AWS_SECRET_ACCESS_KEY: HFNJFNJNFJ2YzkwZGU0MGMGRjMTMxYzVmMjg4YjFlYmNhNTdkYjMHFNJFNJDN

[Aditya1523]

I can restore the backup but after running for some time getting an error Cannot write to the standard output.

[matthewb]

@Aditya1523,
We will need to see the complete log output.

[Aditya1523]

Hi, the previous issue was resolved that happened because of insufficient disk space, and i have completed the restoration process successfully, BUT after the restoration its giving me error pod is not able to start and showing the below error:

Liveness probe failed: ERROR 1045 (28000): Access denied for user 'monitor'@'pxc-cluster-pxc-0.pxc-cluster-pxc.pxc.svc.cluster.local' (using password: YES) + [[ -n '' ]] + exit 1

[matthewb]

Please create a monitoring user for the probes to function:

CREATE USER monitor@'%' IDENTIFIED BY 'password';

You can find the password for the monitor user in the K8S secrets file

[Aditya1523]

This i need to create before restore right? because after restore I am not able to login into mysql to run this command.

bash-4.4$ mysql -h pxc-cluster-pxc.pxc.svc.cluster.local -P 3306 -u root -p
Enter password: 
ERROR 2005 (HY000): Unknown MySQL server host 'pxc-cluster-pxc.pxc.svc.cluster.local' (-2)
bash-4.4$ 

[matthewb]

No, create after restore. If you created before restore, then the restore would erase the user.

kubectl get svc to find the correct endpoint for your cluster.

[Aditya1523]

Hi matthewb, how can i create after restore it is giving me an error in login so i am not able to get inside the mysql so that i can run CREATE USER monitor@‘%’ IDENTIFIED BY ‘password’; this command.

[matthewb]

You’re using the wrong hostname. Determine what the correct hostname is for your PXC install. Did you run the command I suggested above?

[Aditya1523]

Yes I used the command and I am sure the hostname is correct because before restore I used the same hostname and it was working fine but after restore its showing unknown server host.
And In the screenshot cluster IP is showing none but In my running integration cluster the cluster -IP is also showing none which i think normal behaviour.
What i suspect is when the pxc cluster is created after restoration it is done by operator and it is causing the issue.

[Aditya1523]

I can see before restore procedure the endpoint of services are present but after restore the endpoint of service is showing null, thats why it is not able to detect the service or host.
Note: Before restore I am able to execute the same host and login but after it is not allowed.

[Aditya1523]

Hi matthewb any update?

[Aditya1523]

Hi, after restoration I am not able to login to database because i think the permission for /var/lib/mysql is not mysql I used cmd chown -R mysql:mysql /var/lib/mysql but its giving me error

chown: changing ownership of 'mysql/pxc-entrypoint.sh': Operation not permitted
chown: changing ownership of 'mysql/unsafe-bootstrap.sh': Operation not permitted
chown: changing ownership of 'mysql/pxc-configure-pxc.sh': Operation not permitted
chown: changing ownership of 'mysql/liveness-check.sh': Operation not permitted
chown: changing ownership of 'mysql/readiness-check.sh': Operation not permitted
chown: changing ownership of 'mysql/peer-list': Operation not permitted
chown: changing ownership of 'mysql/get-pxc-state': Operation not permitted
chown: changing ownership of 'mysql': Operation not permitted

I can not use sudo because I am inside pod and sudo is not installed can you suggest any docs or how to proceed further.