Manage pbm-agent backups from the pmm-server user interface

MongoDB Percona Backup for MongoDB
1

Hello everyone!

I’m trying to manage my MongoDB backups from the PMM (Percona Monitoring and Management) User Interface, rather than having to use the pbm CLI. Basically, I would like to do the typical operations: list backups, create a new backup (on demand), schedule backups, etc from the user interface. Looking at the documentation in the Percona’s website, it seems to be possible.

But when I try to create a new Backup (Backup → All DBaaSBackups → Create backup) and fill in all the fields I get the following error message: “software “mongodb” is not installed: incompatible service”.

backups
backups2051×1273 233 KB

I have the tipical setup: 1 PMM-Server (version 2.36.0), and 3 Percona MongoDB nodes (version 5.0.15) as a ReplicaSet. On each of the MongoDB nodes I have a PMM-Client (version 2.36.0) connected to the PMM-Server, and a PBM-Agent (version 2.1.0) connected to the Mongo Cluster. Everything seems to be configured correctly.

I think I have applied all steps described in the documentation, and I still cannot make it work. Am I missing something?

Thanks in advance.
Ernesto.

1 Like
2

Please note that I’m not using DBaaS (Kubernetes operator, etc). I deployed everything manually.

Are Backups from PMM only available for DBaaS?

Regards.

3

Hi @ernestojpg ,

Firstly, backups from PMM are currently available for all MongoDB ReplicaSet cluster deployment.
From the error it seems that the MongoDB service might not be running on that node. Kindly verify it.
Also kindly share the rs.status output along with the inventory list from PMM.

Regards,
Parag

1 Like
4

Hello @Parag_Bhayani !

Thanks for your response.

This is the result of a “pmm-admin status” in the MongoDB Node 1:

bash-5.1$ pmm-admin status
Agent ID : /agent_id/c69459c4-74f2-466b-9a0a-331fd99fd622
Node ID  : /node_id/62f97d5c-ca26-464a-8b59-9d874ba72ccd
Node name: mongodb1

PMM Server:
	URL    : https://172.31.29.27:443/
	Version: 2.36.0

PMM Client:
	Connected        : true
	Time drift       : 373.269µs
	Latency          : 630.557µs
	Connection uptime: 100
	pmm-admin version: 2.36.0
	pmm-agent version: 2.36.0
Agents:
	/agent_id/665da036-49b3-4513-a4da-881b0ec9a46d mongodb_profiler_agent Running 0
	/agent_id/703f3a41-bb35-4b0d-bb02-0130da68798b vmagent Running 42000
	/agent_id/7c84cad5-bdca-4526-a9e9-eaaa7fc53115 mongodb_exporter Running 42002
	/agent_id/cc2d85b0-86c5-4387-9a78-fe4156c17d27 node_exporter Running 42001

This is the result of a “pmm-admin list” in the MongoDB Node 1:

bash-5.1$ pmm-admin list
Service type        Service name        Address and port        Service ID
MongoDB             mongodb1            127.0.0.1:27017         /service_id/7466daeb-6fbf-477c-a296-ceb95df4e621

Agent type                    Status           Metrics Mode        Agent ID                                              Service ID                                              Port
pmm_agent                     Connected                            /agent_id/c69459c4-74f2-466b-9a0a-331fd99fd622                                                                0
node_exporter                 Running          push                /agent_id/cc2d85b0-86c5-4387-9a78-fe4156c17d27                                                                42001
mongodb_exporter              Running          push                /agent_id/7c84cad5-bdca-4526-a9e9-eaaa7fc53115        /service_id/7466daeb-6fbf-477c-a296-ceb95df4e621        42002
mongodb_profiler_agent        Running                              /agent_id/665da036-49b3-4513-a4da-881b0ec9a46d        /service_id/7466daeb-6fbf-477c-a296-ceb95df4e621        0
vmagent                       Running          push                /agent_id/703f3a41-bb35-4b0d-bb02-0130da68798b                                                                42000

This is the result of a “rs.status()” in the MongoDB node 1:

{
	"set" : "myReplicaSet",
	"date" : ISODate("2023-05-02T15:12:36.113Z"),
	"myState" : 1,
	"term" : NumberLong(1),
	"syncSourceHost" : "",
	"syncSourceId" : -1,
	"heartbeatIntervalMillis" : NumberLong(2000),
	"majorityVoteCount" : 2,
	"writeMajorityCount" : 2,
	"votingMembersCount" : 3,
	"writableVotingMembersCount" : 3,
	"optimes" : {
		"lastCommittedOpTime" : {
			"ts" : Timestamp(1683040355, 1),
			"t" : NumberLong(1)
		},
		"lastCommittedWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
		"readConcernMajorityOpTime" : {
			"ts" : Timestamp(1683040355, 1),
			"t" : NumberLong(1)
		},
		"appliedOpTime" : {
			"ts" : Timestamp(1683040355, 1),
			"t" : NumberLong(1)
		},
		"durableOpTime" : {
			"ts" : Timestamp(1683040355, 1),
			"t" : NumberLong(1)
		},
		"lastAppliedWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
		"lastDurableWallTime" : ISODate("2023-05-02T15:12:35.851Z")
	},
	"lastStableRecoveryTimestamp" : Timestamp(1683040310, 1),
	"electionCandidateMetrics" : {
		"lastElectionReason" : "electionTimeout",
		"lastElectionDate" : ISODate("2023-04-26T14:52:08.891Z"),
		"electionTerm" : NumberLong(1),
		"lastCommittedOpTimeAtElection" : {
			"ts" : Timestamp(1682520718, 1),
			"t" : NumberLong(-1)
		},
		"lastSeenOpTimeAtElection" : {
			"ts" : Timestamp(1682520718, 1),
			"t" : NumberLong(-1)
		},
		"numVotesNeeded" : 2,
		"priorityAtElection" : 1,
		"electionTimeoutMillis" : NumberLong(10000),
		"numCatchUpOps" : NumberLong(0),
		"newTermStartDate" : ISODate("2023-04-26T14:52:08.941Z"),
		"wMajorityWriteAvailabilityDate" : ISODate("2023-04-26T14:52:09.462Z")
	},
	"members" : [
		{
			"_id" : 0,
			"name" : "172.31.31.78:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 519972,
			"optime" : {
				"ts" : Timestamp(1683040355, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2023-05-02T15:12:35Z"),
			"lastAppliedWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
			"lastDurableWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
			"syncSourceHost" : "",
			"syncSourceId" : -1,
			"infoMessage" : "",
			"electionTime" : Timestamp(1682520728, 1),
			"electionDate" : ISODate("2023-04-26T14:52:08Z"),
			"configVersion" : 1,
			"configTerm" : 1,
			"self" : true,
			"lastHeartbeatMessage" : ""
		},
		{
			"_id" : 1,
			"name" : "172.31.30.30:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 519637,
			"optime" : {
				"ts" : Timestamp(1683040352, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1683040352, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2023-05-02T15:12:32Z"),
			"optimeDurableDate" : ISODate("2023-05-02T15:12:32Z"),
			"lastAppliedWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
			"lastDurableWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
			"lastHeartbeat" : ISODate("2023-05-02T15:12:34.238Z"),
			"lastHeartbeatRecv" : ISODate("2023-05-02T15:12:34.522Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "172.31.31.78:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 1,
			"configTerm" : 1
		},
		{
			"_id" : 2,
			"name" : "172.31.25.63:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 519637,
			"optime" : {
				"ts" : Timestamp(1683040355, 1),
				"t" : NumberLong(1)
			},
			"optimeDurable" : {
				"ts" : Timestamp(1683040355, 1),
				"t" : NumberLong(1)
			},
			"optimeDate" : ISODate("2023-05-02T15:12:35Z"),
			"optimeDurableDate" : ISODate("2023-05-02T15:12:35Z"),
			"lastAppliedWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
			"lastDurableWallTime" : ISODate("2023-05-02T15:12:35.851Z"),
			"lastHeartbeat" : ISODate("2023-05-02T15:12:36.050Z"),
			"lastHeartbeatRecv" : ISODate("2023-05-02T15:12:34.663Z"),
			"pingMs" : NumberLong(0),
			"lastHeartbeatMessage" : "",
			"syncSourceHost" : "172.31.31.78:27017",
			"syncSourceId" : 0,
			"infoMessage" : "",
			"configVersion" : 1,
			"configTerm" : 1
		}
	],
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1683040355, 1),
		"signature" : {
			"hash" : BinData(0,"Q89GURIwKWC6NxezL7kVV/7mIcM="),
			"keyId" : NumberLong("7226371501602111492")
		}
	},
	"operationTime" : Timestamp(1683040355, 1)
}

These are the Services in PMM Inventory:

services
services2808×1426 426 KB

And these are the Nodes in the PMM Inventory:

nodes
nodes2812×1158 240 KB

Also note that in each MongoDB Node we have these 3 services running inside Docker:

  • percona/percona-server-mongodb:5.0.15 running isolated, but exposing port 27017
  • percona/pmm-client:2.36.0 running in network_mode: host
  • percona/percona-backup-mongodb:2.1.0 running in network_mode: host

Do pmm-client or pbm-agent require to call the “mongo” service directly, or everything happens through the standard 27017 port?

Thank you very much.
Ernesto.

5

Hi @ernestojpg ,

Have you enabled the ENABLE_BACKUP_MANAGEMENT=1 parameter when you ran the docker run command?

For setups where PMM Server runs as a Docker container, enable backup features at container creation time by adding -e ENABLE_BACKUP_MANAGEMENT=1 to your docker run command.

Kindly refer MongoDB backup prerequisites -

Regards,
Parag

6

Hello @Parag_Bhayani !

I think Backup Management is enabled by default in the latest versions of PMM Server, but yes, I added the option, just in case. This is the docker-compose I use:

version: "3.9"
services:
  pmm-server:
    image: percona/pmm-server:2.36.0
    network_mode: "host"
    restart: unless-stopped
    volumes:
      - ${PWD}/data:/srv
    environment:
      - ENABLE_BACKUP_MANAGEMENT=1

I can see the Backups section in PMM Server. What I don’t know is why it says “DBaaSBackups” everywhere if I’m not using DBaaS…

backups
backups1962×1296 189 KB

I think I followed all prerequisites in that document. I don’t know what I’m missing here …

Thanks.
Ernesto.

7

Just to let you know, when I try to create a new Backup I see that error in PMM Server, but I also see this error in the logs:

==> pmm-managed.log <==
INFO[2023-05-02T16:04:42.933+00:00] Starting RPC /backup.v1.Backups/StartBackup ...  request=0cec3377-e903-11ed-8ae6-067cb96f2d37
WARN[2023-05-02T16:04:42.940+00:00] RPC /backup.v1.Backups/StartBackup done in 7.489881ms with gRPC error: rpc error: code = FailedPrecondition desc = software "mongodb" is not installed: incompatible service  request=0cec3377-e903-11ed-8ae6-067cb96f2d37

Regards.
Ernesto.

8

Hello @ernestojpg. Currently PMM Backup management doesn’t support dockerized configurations. It’s required to have pmm-agent, pbm and mongod installed on the same host node. You’re getting error because pmm-agent cannot find installed mongod in the same environment.

Technically you can run backups for dockerized setups, but in that case all of the mentioned software should be installed in the same container and services should be managed by systemd.

9

Hi @Pavel_Khripkov ,

Thanks for your response. I will try to install the software normally in the machine without Docker and let you know.

But it is really a pity. I though that all the components were working fine in Docker and Kubernetes. How do backups work in the Percona Operator for MongoDB (Kubernetes) described here then? About backups - Percona Operator for MongoDB

Regards.
Ernesto.

10

To restore database from physical backup, it requires to restart mongod service after copying files back. When we’re using operators, we can stop existing containers and replace them with new ones. For regular backups we’re not managing containers. So we cannot restart database services to meet the requirements.