Using the percona operator to deploy a mongodb cluster on k8s which runs in azure. I’m now adding a backup configuration per the instructions and provisioned an azure storage account and blob. The k8s deployment update goes fine, but checking the logs of the backup agent, it seems that it might not be connecting to the storage? I’ll also add this is a new mongodb cluster with no data in it at the moment.
kubectl logs ads-mongodb-stage-psm-cfg-0 backup-agent -f
+ '[' pbm-agent = pbm-agent ']'
++ mktemp
+ OUT=/tmp/tmp.JGDosQ2dHa
++ mktemp
+ OUT_CFG=/tmp/tmp.3V9eT9V61n
+ timeout=5
+ for i in '{1..10}'
+ '[' TRUE ']'
+ echo 'waiting for sharded scluster'
+ set +o xtrace
waiting for sharded scluster
2022-03-30T02:00:25.791+0000 I NETWORK [js] Starting new replica set monitor for cfg/localhost:27017
2022-03-30T02:00:25.792+0000 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to localhost:27017
2022-03-30T02:00:25.799+0000 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to ads-mongodb-stage-psm-cfg-2.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017
2022-03-30T02:00:25.811+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for cfg is cfg/ads-mongodb-stage-psm-cfg-0.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017,ads-mongodb-stage-psm-cfg-1.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017,ads-mongodb-stage-psm-cfg-2.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017
2022-03-30T02:00:25.811+0000 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to ads-mongodb-stage-psm-cfg-1.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017
2022-03-30T02:00:25.811+0000 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to ads-mongodb-stage-psm-cfg-0.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017
2022-03-30T02:00:25.897+0000 E QUERY [js] uncaught exception: TypeError: db.isMaster(...).$configServerState is undefined :
@(shell eval):1:1
2022-03-30T02:00:25.897+0000 E - [main] exiting with code -4
+ exit_status=0
+ set +o xtrace
2
+ exit_status_cfg=0
++ grep -E '^Timestamp\([0-9]+, [0-9]+\)$' /tmp/tmp.JGDosQ2dHa
+ ts=
++ grep -E '^2$' /tmp/tmp.3V9eT9V61n
+ isCfg=2
+ [[ 0 == 0 ]]
+ [[ -n '' ]]
+ [[ 0 == 0 ]]
+ [[ -n 2 ]]
+ break
+ rm /tmp/tmp.JGDosQ2dHa
+ exec pbm-agent
2022-03-30T02:00:26.000+0000 I starting PITR routine
2022-03-30T02:00:26.000+0000 I pbm-agent:
Version: 1.6.1
Platform: linux/amd64
GitCommit: ec060aa090b6702b0f5938685fc88bbfa309660b
GitBranch: release-1.6.1
BuildTime: 2021-12-08_16:40_UTC
GoVersion: go1.16.11
2022-03-30T02:00:26.000+0000 I node: cfg/ads-mongodb-stage-psm-cfg-0.ads-mongodb-stage-psm-cfg.ns-team-an-ads.svc.cluster.local:27017
2022-03-30T02:00:26.000+0000 I listening for the commands
2022-03-30T02:00:31.000+0000 W [agentCheckup] get current storage status: query mongo: mongo: no documents in result
2022-03-30T02:00:31.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2022-03-30T02:00:36.000+0000 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
The last log line just repeats.
The storage account is configured to prevent public access, which should only block anonymous access, otherwise Allow access from is set to all networks.