We have PMM running nicely on an EC2 instance and I can login as the admin. But, after I fill in all the RDS details in the setup form and click on ‘Add service’, I get the following error:
pmm-agent with ID “pmm-server” is not currently connected
I ran pmm-admin status on the EC2 instance and got the following:
Agent ID: /agent_id/agent-id-string
Node ID : /node_id/node-id-string
Could someone please give a clue on how to fix this?
Of course, the specific metric varies, but the above error is common to all the exporter agents that rae down. Note: The EC2 instance is in a subnet that doesn’t have IPv6. I doubt that’s the issue, though.
Sorry to bump this up, but could someone at Percona please throw some light on a way to fix this error?
I am wondering if this has something to do with the ingress/egress rules of the security group that Percona’s install process created - we went with ‘seller recommended settings’ and that opened up just Port 80, 443 and 22.
So the PMM instance is running on EC2 installed direct from Marketplace and you’re trying to add RDS remote instance monitoring through the UI’s Add Instance option and on the final step you get the error?
The basics of how remote instance monitoring works is that PMM Server ALWAYS gets its data from a client…most of the time the client is installed on remote nodes and communicates back to the server but in remote monitoring’s instance, the client actually resides on the pmm-server itself (hence the 127.0.0.1 as the dial address). This says to me that the agent inside the pmm-server isn’t running. Easy to verify with by getting a prompt on the machine and running (as root) supervisorctl status and look at the pmm-agent entry to see if it’s running.
if it’s not you can dig into /srv/logs/pmm-agent.log to see if there’s any hint as to why it couldn’t start and we can hopefully troubleshoot from there. if it is running then I’d be supicious about internal communication and we might have to get basic looking at config files /usr/local/percona/pmm2/config/pmm-agent.yaml and start telneting to ports etc.
No you shouldn’t have to…do me a favor and install telnet yum install telnet and then execute telnet 127.0.0.1 7777 if it’s successful you’ll just get an “escape character is ‘^]’” or similar (hit ctrl+] and type quit if you’re not familiar with telnet). but I don’t think pmm-sever tries to leverage 7777 (although it may…I’ll ask internally). because you’re making a call for ‘metrics’ that’s pmm-server trying to talk to agent and it should say something like 127.0.0.1:4200x.
Take a look at the following page: PMM → PMM Inventory → Agents tab…can you send a screenshot of that? you should not have to enable any ports internally…it should just work so it seems a config is messed up somewhere…just not sure where. Gonna see if anyone else has ideas…
Could you tell us what have you done after installing PMM Server? Did you install client or update pmm-agent.yaml manually? id in your pmm-agent.yaml should be pmm-server to work correctly.
Modified the hostname of the PMM ec2 instance by sudo hostnamectl set-hostname hostname, added a DNS record at Route53 to make this the public DNS, updated the reverse DNS of the Elastic IP to point to this and rebooted the instance.
Created an SSL for this hostname through ACM, but didn’t deploy it as it would require an ELB in front and I didn’t want to add costs.
That’s it. So, pretty much the only thing I did on the PMM server was to run the config command a second time.
P.S: Just checked .bash_history of the PMM server and confirmed that #1 & #2 above are the only commands I ran on the PMM server before running into this problem.
overwrote pmm-agent.yaml, that’s why you don’t have pmm-agent with id pmm-server which is required to monitor remote services.
Now you should update your pmm-agent.yaml and replace id there with pmm-server to make it work. Then just restart pmm-agent service.
Not sure if this is the right way to go about it as I get the same error when adding the RDS MySQL instance for monitoring. pmm-agent is running fine after the restart.
Thanks. Sure, I am attaching screenshots of each of the tabs (Services, Agents, Nodes) of the Inventory page.
One thing I noticed is that, in the Nodes tab, the agent points to the internal hostname and private IP address of the EC2 instance. I don’t know if that’s how it’s supposed to be.
Just to clarify again, I have followed verbatim the instructions on these two pages:
Hmmm…I think our instructions may have been unclear and need to be fixed!
the part that says:
authentication between PMM Server and PMM Clients - you will re-use these credentials when configuring PMM Client for the first time on a server, for example:
is NOT meant to be run on your newly installed PMM server…that’s only needed if you install pmm-client on another server and want your new PMM instance to also monitor it and DB’s on it. (trying to get that documentation updated ASAP)
running that command locally is what broke PMM’s connection with itself…at this point it may be easier to blow the PMM instance away and recreate it from scratch since that pmm-admin command not only changes the pmm-agent.yaml file but attempts to register it in the internal database as well.
Another part that would be good to fix is the set of commands for creating the pmm user (@Vadim_Yalovets helped me with that) on the RDS instance. They don’t work on MySQL 8.x.x. Or, maybe a distinction could be made between 5.7.x and 8.x.x.
Thanks to all the Percona staff who pitched in to try and help me on this post. Appreciate your efforts very much.
