QAN API is Down after upgrade the PMM server from 2.44.0 to 3.2.0 version

Percona Monitoring and Management (PMM)
7

@nurlan, I have resolved the issue by following the steps outlined below.

Note: Prior to attempting the PMM 3.2.0 upgrade, I had already taken a backup of PMM 2.44.0 from the production server.

  • Production Server: prdpmm101
  • Temporary Server: tmppmm101

After the PMM 3.2.0 upgrade on the production server failed, I decided to restore the PMM 2.44.0 backup to the temporary server, perform the upgrade there, and then migrate the working setup back to the production environment.

Steps taken to resolve the PMM 3.2.0 upgrade issue:

  1. Restored the PMM 2.44.0 backup on the temporary server tmppmm101.
  2. Upgraded the temporary server to PMM 3.2.0.
  3. Verified the status of all services using the PMM Health Check dashboard.
  4. This time, the Query Analytics (QAN) functionality came up without issues.
  5. After confirming the upgrade was successful, I took a backup of PMM 3.2.0 from the temporary server.
  6. Copied the PMM 3.2.0 backup from tmppmm101 to the production server prdpmm101.
  7. Performed a fresh installation of PMM 3.2.0 on the production server.
  8. Restored the PMM 3.2.0 backup to the production server.
  9. Post-restore, I observed that the internal PostgreSQL monitoring was showing issues — specifically, the agent status was displayed as Down on the Service Summary dashboard.
  10. I reported this issue in the Percona forum:
    PMM3 | PostgreSQL | Inside container FATAL: role does not exist
  11. To address the PostgreSQL role issue, it is essential to follow these steps:
  • After the fresh installation of PMM 3.2.0 but before restoring the backup, note down the usernames and passwords from the agents table for both postgres_exporter and qan-postgresql-pgstatements-agent.
  1. Once the backup restoration is complete, update the PostgreSQL agent usernames and passwords in the production server using the values noted in step 11.

UPDATE agents SET username = ‘XXXXXXX’,password = ‘XXXXXXX’,updated_at = NOW() WHERE agent_id = ‘XXXXXXX’ AND agent_type = ‘postgres_exporter’;

UPDATE agents SET username = ‘XXXXXXX’,password = ‘XXXXXXX’,updated_at = NOW() WHERE agent_id = ‘XXXXXXX’ AND agent_type = ‘qan-postgresql-pgstatements-agent’;

  1. Verify the PostgreSQL agent status in the PMM Service Summary dashboard.
  2. If all services, including the PostgreSQL monitoring, show as Up, the backup restoration and upgrade process on the production server can be considered successful.