# PMM on AWS fails when adding RDS MySQL remote instance **URL:** \ **Category:** PMM 2.x\ **Created:** [July 18, 2021, 2:31pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432 "2021-07-18T14:31:54Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 18, 2021, 2:31pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/1 "2021-07-18T14:31:55Z")
We have PMM running nicely on an EC2 instance and I can login as the admin. But, after I fill in all the RDS details in the setup form and click on ‘Add service’, I get the following error: > pmm-agent with ID “pmm-server” is not currently connected I ran pmm-admin status on the EC2 instance and got the following: > Agent ID: /agent\_id/agent-id-string > Node ID : /node\_id/node-id-string > > PMM Server: > URL : [https://IPaddress:443/](https://IPaddress:443/) > Version: 2.19.0 > > PMM Client: > Connected : true > Latency : 289.252µs > pmm-admin version: 2.19.0 > pmm-agent version: 2.19.0 > > Agents: > /agent\_id/agent-id-string node\_exporter Running > /agent\_id/agent-id-string vmagent Running So, everything seems fine, but I am unable to add the remote RDS mysql instance to start the monitoring. ☹ Any help would be appreciated. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 19, 2021, 9:36am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/2 "2021-07-19T09:36:25Z")
After a fruitless search, I came across this article: > **[Cross-cloud PMM: which TCP ports to open](https://dev.to/aws-heroes/pmm-server-on-aws-which-tcp-ports-to-open-2df9)** > > By Franck Pachot . I recently installed Percona Monitoring & Management on AWS (free tier) and here is how to monitor an instance on another cloud (OCI), in order to show which TCP port must be opened. PMM server I installed PMM from the AWS... Based on that, I logged into my AWS EC2 instance running PMM and hit the suggested url: [https://IPaddress/prometheus/targets](https://IPaddress/prometheus/targets) and found that several node\_exporter\_agents and postgres\_exporter\_agents are all in the DOWN state with the following common error: > cannot read data: cannot scrape “[http://127.0.0.1:0/metrics?collect[]=bonding&collect[]=entropy&collect[]=textfile.lr&collect[]=uname](http://127.0.0.1:0/metrics?collect%5B%5D=bonding&collect%5B%5D=entropy&collect%5B%5D=textfile.lr&collect%5B%5D=uname)”: Get “[http://127.0.0.1:0/metrics?collect[]=bonding&collect[]=entropy&collect[]=textfile.lr&collect[]=uname](http://127.0.0.1:0/metrics?collect%5B%5D=bonding&collect%5B%5D=entropy&collect%5B%5D=textfile.lr&collect%5B%5D=uname)”: dial tcp4 127.0.0.1:0: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses Could someone please give a clue on how to fix this? Of course, the specific metric varies, but the above error is common to all the exporter agents that rae down. Note: The EC2 instance is in a subnet that doesn’t have IPv6. I doubt that’s the issue, though. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 20, 2021, 4:33am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/3 "2021-07-20T04:33:19Z")
Sorry to bump this up, but could someone at Percona please throw some light on a way to fix this error? I am wondering if this has something to do with the ingress/egress rules of the security group that Percona’s install process created - we went with ‘seller recommended settings’ and that opened up just Port 80, 443 and 22. ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [July 20, 2021, 12:31pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/4 "2021-07-20T12:31:45Z")
So the PMM instance is running on EC2 installed direct from Marketplace and you’re trying to add RDS remote instance monitoring through the UI’s Add Instance option and on the final step you get the error? The basics of how remote instance monitoring works is that PMM Server ALWAYS gets its data from a client…most of the time the client is installed on remote nodes and communicates back to the server but in remote monitoring’s instance, the client actually resides on the pmm-server itself (hence the 127.0.0.1 as the dial address). This says to me that the agent inside the pmm-server isn’t running. Easy to verify with by getting a prompt on the machine and running (as root) `supervisorctl status` and look at the pmm-agent entry to see if it’s running. if it’s not you can dig into /srv/logs/pmm-agent.log to see if there’s any hint as to why it couldn’t start and we can hopefully troubleshoot from there. if it is running then I’d be supicious about internal communication and we might have to get basic looking at config files `/usr/local/percona/pmm2/config/pmm-agent.yaml` and start telneting to ports etc. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 20, 2021, 12:59pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/5 "2021-07-20T12:59:28Z")
Thanks! Yes, the error is in the very last step. I ran `supervisorctl status` and got the following results. It looks like the pmm agent is running: > [admin@pmm ~]$ sudo supervisorctl status > alertmanager RUNNING pid 1512, uptime 0:00:58 > clickhouse RUNNING pid 1506, uptime 0:00:58 > cron RUNNING pid 1509, uptime 0:00:58 > dashboard-upgrade EXITED Jul 20 12:49 PM > dbaas-controller STOPPED Not started > grafana RUNNING pid 1507, uptime 0:00:58 > nginx RUNNING pid 1508, uptime 0:00:58 > pmm-agent RUNNING pid 1516, uptime 0:00:58 > pmm-managed RUNNING pid 1515, uptime 0:00:58 > pmm-update-perform STOPPED Not started > postgresql RUNNING pid 1505, uptime 0:00:58 > prometheus STOPPED Not started > qan-api2 RUNNING pid 1560, uptime 0:00:38 > victoriametrics RUNNING pid 1510, uptime 0:00:58 > vmalert RUNNING pid 1511, uptime 0:00:58 > [admin@pmm ~]$ As pmm-agent is running, based on your suggestion, I looked at `/usr/local/percona/pmm2/config/pmm-agent.yaml` and found the following: > id: /agent\_id/9b751828-3ffc-49b4-8d52-a421441a39e3 > listen-address: 127.0.0.1 > listen-port: 7777 > server: Does this mean Port 7777 has to be opened up on the EC2 instance? ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [July 20, 2021, 1:49pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/6 "2021-07-20T13:49:42Z")
No you shouldn’t have to…do me a favor and install telnet `yum install telnet` and then execute `telnet 127.0.0.1 7777` if it’s successful you’ll just get an “escape character is ‘^]’” or similar (hit ctrl+] and type quit if you’re not familiar with telnet). but I don’t think pmm-sever tries to leverage 7777 (although it may…I’ll ask internally). because you’re making a call for ‘metrics’ that’s pmm-server trying to talk to agent and it should say something like 127.0.0.1:4200x. Take a look at the following page: PMM → PMM Inventory → Agents tab…can you send a screenshot of that? you should not have to enable any ports internally…it should just work so it seems a config is messed up somewhere…just not sure where. Gonna see if anyone else has ideas… ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 20, 2021, 2:17pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/7 "2021-07-20T14:17:20Z")
![Screen Shot 2021-07-20 at 7.43.01 PM](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/9/968cc43acc3086691eaecd3309049cf3f61f82bb.png) I installed telnet and got the following for the telnet command: > [admin@pmm ~]$ telnet 127.0.0.1 7777 > Trying 127.0.0.1… > Connected to 127.0.0.1. > Escape character is ‘^]’. Attaching a screenshot of the Agents tab under PMM inventory. Thanks, again, for the assistance. ---
**Author:** ![nurlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nurlan/32/1523_2.png) [@nurlan](https://forums.percona.com/u/nurlan)\ **Post date:** [July 21, 2021, 8:13am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/8 "2021-07-21T08:13:47Z")
Could you tell us what have you done after installing PMM Server? Did you install client or update pmm-agent.yaml manually? `id` in your pmm-agent.yaml should be `pmm-server` to work correctly. Thank you. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 21, 2021, 9:01am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/9 "2021-07-21T09:01:41Z")
Sure. I only remember doing the following after setting up PMM: 1. Ran the following command with the updated password for the PMM admin user: > pmm-admin config --server-insecure-tls --server-url=[https://admin](https://admin):newpasswd@:443 1. Modified the hostname of the PMM ec2 instance by `sudo hostnamectl set-hostname hostname`, added a DNS record at Route53 to make this the public DNS, updated the reverse DNS of the Elastic IP to point to this and rebooted the instance. 2. Created an SSL for this hostname through ACM, but didn’t deploy it as it would require an ELB in front and I didn’t want to add costs. That’s it. So, pretty much the only thing I did on the PMM server was to run the config command a second time. P.S: Just checked .bash\_history of the PMM server and confirmed that #1 & #2 above are the only commands I ran on the PMM server before running into this problem. ---
**Author:** ![nurlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nurlan/32/1523_2.png) [@nurlan](https://forums.percona.com/u/nurlan)\ **Post date:** [July 21, 2021, 9:26am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/10 "2021-07-21T09:26:28Z")
Thank you for your answer Looks like ```auto pmm-admin config --server-insecure-tls --server-url=https://admin:newpasswd@:443 ``` overwrote `pmm-agent.yaml`, that’s why you don’t have pmm-agent with id `pmm-server` which is required to monitor remote services. Now you should update your `pmm-agent.yaml` and replace id there with `pmm-server` to make it work. Then just restart pmm-agent service. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 21, 2021, 10:48am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/11 "2021-07-21T10:48:41Z")
Thanks. I went ahead and modified `/usr/local/percona/pmm2/config/pmm-agent.yaml`, replacing the id there as below: > Updated by `pmm-agent setup`. > #id: /agent\_id/9b751828-3ffc-49b4-8d52-a421441a39e3 > id: pmm-server > listen-address: 127.0.0.1 > listen-port: 7777 Not sure if this is the right way to go about it as I get the same error when adding the RDS MySQL instance for monitoring. pmm-agent is running fine after the restart. ---
**Author:** ![nurlan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nurlan/32/1523_2.png) [@nurlan](https://forums.percona.com/u/nurlan)\ **Post date:** [July 21, 2021, 1:36pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/12 "2021-07-21T13:36:14Z")
Could you share `Inventory page`? ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 21, 2021, 1:46pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/13 "2021-07-21T13:46:10Z")
Thanks. Sure, I am attaching screenshots of each of the tabs (Services, Agents, Nodes) of the Inventory page. One thing I noticed is that, in the Nodes tab, the agent points to the internal hostname and private IP address of the EC2 instance. I don’t know if that’s how it’s supposed to be. Just to clarify again, I have followed verbatim the instructions on these two pages: 1. [AWS Marketplace - Percona Monitoring and Management](https://www.percona.com/doc/percona-monitoring-and-management/2.x/setting-up/server/aws.html) 2. [Amazon RDS - Percona Monitoring and Management](https://www.percona.com/doc/percona-monitoring-and-management/2.x/setting-up/client/aws.html) ![Screen Shot 2021-07-21 at 7.07.08 PM](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/f/ffcb29d21caec8ae36ed194141db46283a232230.png) ![Screen Shot 2021-07-21 at 7.07.24 PM](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/a/a746f23a956c45096f5faccf60cead53e9f414e8.png) ![Screen Shot 2021-07-21 at 7.07.47 PM](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/8/8760005305ed596ca246954fd0bff58410e4b27a.png) ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [July 21, 2021, 3:38pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/14 "2021-07-21T15:38:01Z")
Hmmm…I think our instructions may have been unclear and need to be fixed! the part that says: authentication between PMM Server and PMM Clients - you will re-use these credentials when configuring PMM Client for the first time on a server, for example: ```auto pmm-admin config --server-insecure-tls --server-url=https://admin:admin@:443 ``` is NOT meant to be run on your newly installed PMM server…that’s only needed if you install pmm-client on _another_ server and want your new PMM instance to also monitor it and DB’s on it. (trying to get that documentation updated ASAP) running that command locally is what broke PMM’s connection with itself…at this point it may be easier to blow the PMM instance away and recreate it from scratch since that pmm-admin command not only changes the pmm-agent.yaml file but attempts to register it in the internal database as well. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 21, 2021, 4:18pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/15 "2021-07-21T16:18:45Z")
> [@steve.hoffman](#): > > Hmmm…I think our instructions may have been unclear and need to be fixed! I was wondering if that might be the case! 😀 Another part that would be good to fix is the set of commands for creating the pmm user (@Vadim_Yalovets helped me with that) on the RDS instance. They don’t work on MySQL 8.x.x. Or, maybe a distinction could be made between 5.7.x and 8.x.x. Thanks to all the Percona staff who pitched in to try and help me on this post. Appreciate your efforts very much. 👍 I will go ahead and create a new instance. ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [July 21, 2021, 4:38pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/16 "2021-07-21T16:38:17Z")
We are on it! Thanks for the feedback and keep us posted! ---
**Author:** ![steve.hoffman](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/steve.hoffman/32/66_2.png) [@steve.hoffman](https://forums.percona.com/u/steve.hoffman)\ **Post date:** [July 21, 2021, 5:07pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/17 "2021-07-21T17:07:28Z")
Not sure why I can’t edit my post but submitted updates to both issues so once they’re approved hopefully no other runs into this 😉 ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 21, 2021, 5:35pm UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/18 "2021-07-21T17:35:03Z")
Thanks for acting quickly on this. ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 22, 2021, 2:40am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/19 "2021-07-22T02:40:08Z")
> [@steve.hoffman](#): > > Hmmm…I think our instructions may have been unclear and need to be fixed! For what it’s worth, I noticed the same error in the documentation for PMM 1.x on AWS. It may be useful to correct that, too: > **[Percona legacy documentation](https://docs.percona.com/legacy-documentation/)** > > You should upgrade to the most recent version for documentation support. The documentation for product versions that are either outdated or have reached their end-of-life is available in a PDF format. Outdated documentation is defined as the... ---
**Author:** ![fuzzybase](https://avatars.discourse-cdn.com/v4/letter/f/f6c823/32.png) [@fuzzybase](https://forums.percona.com/u/fuzzybase)\ **Post date:** [July 30, 2021, 3:58am UTC](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432/20 "2021-07-30T03:58:54Z")
> [@steve.hoffman](#): > > Not sure why I can’t edit my post but submitted updates to both issues so once they’re approved hopefully no other runs into this Checked just now and the old documentation still persists. Hope it can be corrected soon. [Next page](https://forums.percona.com/t/pmm-on-aws-fails-when-adding-rds-mysql-remote-instance/11432.md?page=2)