Sometime in the last few weeks, we lost the ability to register clients. We are using Docker image pmm2 v.2.9.1 with pp2-client 2.11.0-6 on CentOS 7. The account used is both a Grafana admin and PMM admin and member of the Org. We receive this error with --debug:
URL:“https;//xx.xx.xx.xx:443/v1/management/Node/Register”, Err:(*errors.errorString)(0xc0004603c0)
When not using debug flag:
Failed to register pmm-agent on PMM server: Post “https://xx.xx.xx.xx:443/v1/management/Node/Register”: Forbidden.
LDAP is enabled but has not interfered prior to this point and the account used is local.
TBH, I’ve never tried to use an account other than the original “admin” account to register nodes. it’s actually a special permission set to grafana which is more powerful than the group/role of admin (in the UI login as a user you’ve made an admin and compare menu options on the left to when you login as the original “admin” user, you’ll see MORE options as admin then you will as an admin group member)
Since we use quite a bit of grafana’s auth system underneath the hood I wouldn’t be surprised if the call is protected/limited to the admin user vs the admin role/group.
@steve.hoffman - Thanks for your quick response. Please note that the registration account is both a Grafana and PMM admin.
We have registered many clients using the registration account. But just to check, I used the admin account and received the same response.
Is it possible that pmm-server 2.9.1 and the pmm2-client 2.11.0-6 are incompatible? I can upgrade our Docker image but have a few more pressing items at this time.
Hi,
I have just tested a setup procedure with mentioned versions and can confirm that instances are compatible.
Could you check logs on your PMM server side?
grep “Node/Register” /srv/logs/pmm-managed.log
@adivinho - Nothing in the logs. I attempted registration while tailing the logs and nothing. I am able to connect from the client by “openssl s_client -connect 10.xxx.xxx.xxx:443”. This client was previously registered when CentOS 6 and was replaced by a CentOS 7 instance. I used --force, to no benefit. The message was the same:
Failed to register pmm-agent on PMM Server: Post “https://10.xxx.xxx.xxx:443/v1/management/Node/Register”: Forbidden.
@adivinho - I was able to register another client that also been recently converted to C7. I’ll try to determine the difference and update.
Is it possible it’s routing that’s causing the conflict (I know I’m reaching here) but can you verify that your client is, in fact, talking to the right server. I’ve seen some weird stuff like firewalls reject from management networks (returns a forbidden…just not from the place you’d be expecting it to come from), I’ve seen systems configured only to answer to a certain IP range and then some silly little route statement (or even in one case, a fat-fingered subnet mask) send traffic out the back door, around the neighborhood before getting routed back the wrong way…(I said it was a stretch, just trying to think creatively here!)
I guess the easiest way is to use telnet or lynx (or if it has a UI, a browser) to see if you can get to the login page of the pmm-server from that client…at least then we’d be confident that it’s not the network. From there I’d have to see if there I’d be checking fail2ban logs (been bitten there more than once) on both client and server), iptables rules, selinux or firewalld (pulled out a ton of hair until I learned that’s in CentOS8 enabled by default).
Sorry I’m not more help!
It seems our customer set http_proxy, https_proxy, HTTP_PROXY and HTTPS_PROXY, sending the connection attempt out through Squid. I unset them temporarily, registered and re-exported them.
It might be worth implementing a --noproxy option in pmm-admin to ignore any environment variables set.
