We’ve recently installed an upgrade from PMM to PMM 3.8.0 and I’m having issues registering any clients with the new server. No matter what I put for the auth string in “pmm-admin config” it fails with:
[root@dv1medpglabdb1 mrolfs]# pmm-admin config --server-insecure-tls --server-url=https://:PASSWORD@:443
Checking local pmm-agent status…
pmm-agent is running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Internal server error.
Please check username and password.
- I can telnet to the server on 443
- I can log into the server web interface
- It fails no matter what user I put, and I’ve tested all user logins from the UI
- It also fails if I use a service account and put "https://service_token:glsa_@:443
- Running it in debug I get this output which from what I can tell basically says the same thing as “auth failure” with an HTTP 401 returned (I’ll include that output down below)
- I’m also having issues in the UI saving some specific settings, especially the “Check for updates” switch which nets me “Updates are configured via PMM_ENABLE_UPDATES environment variable” and “Request failed with status code 400” errors when I attempt to save
HALP ME PERCONA WAN KENOBI YOU’RE MY ONLY HOPE
Debug output:
[root@dv1medpglabdb1 mrolfs]# pmm-admin config --server-insecure-tls --server-url=https://<USERNAME>:<PASSWORD>@<SERVER ADDRESS>:443 --debug
DEBUG 2026-08-20 16:00:26.734779871Z: Running: pmm-agent --server-address=<SERVER ADDRESS>:443 --server-username=<USERNAME> --server-password=<PASSWORD> --listen-port=7777 --server-insecure-tls --log-level=warn --debug --log-lines-count=1024 setup --metrics-mode=auto <LOCAL IP> generic <LOCAL SERVER HOSTNAME>
DEBUG 2026-08-20 16:00:33.018239105Z: Result: &commands.configResult{Warning:"", Output:"Checking local pmm-agent status...\npmm-agent is running.\nRegistering pmm-agent on PMM Server...\nFailed to register pmm-agent on PMM Server: Internal server error.\nPlease check username and password."}
DEBUG 2026-08-20 16:00:33.018898324Z: Error: &exec.ExitError{ProcessState:(*os.ProcessState)(0x3541f87ce018), Stderr:[]uint8{<HEX CODES THAT MADE THE POST FAIL WITH TOO MANY CHARACTERS}}
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Internal server error.
Please check username and password.
DEBUG 2026-08-20 16:00:33.019207807Z: exit status 1, stderr:
time="2026-08-20T16:00:26.759+00:00" level=info msg="Temporary directory will default to /usr/local/percona/pmm/tmp" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Nomad data directory will default to /usr/local/percona/pmm/data/nomad" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/tools/pt-summary as a path to Percona Toolkit pt-summary" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/tools/pt-pg-summary as a path to Percona Toolkit pt-pg-summary" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/tools/pt-mongodb-summary as a path to Percona Toolkit pt-mongodb-summary" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/tools/pt-mysql-summary as a path to Percona Toolkit pt-mysql-summary" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/tools/nomad as a path to Nomad binary" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/node_exporter as a path to node_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/mongodb_exporter as a path to mongodb_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/postgres_exporter as a path to postgres_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/valkey_exporter as a path to valkey_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/azure_exporter as a path to azure_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/mysqld_exporter as a path to mysqld_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/proxysql_exporter as a path to proxysql_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/rds_exporter as a path to rds_exporter" component=setup
time="2026-08-20T16:00:26.760+00:00" level=info msg="Using /usr/local/percona/pmm/exporters/vmagent as a path to vmagent" component=setup
time="2026-08-20T16:00:26.761+00:00" level=debug msg="POST /local/Status HTTP/1.1\r\nHost: 127.0.0.1:7777\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 3\r\nAccept: application/json\r\nContent-Type: application/json\r\nAccept-Encoding: gzip\r\n\r\n{}\n\n" component=setup
time="2026-08-20T16:00:26.776+00:00" level=debug msg="HTTP/1.1 200 OK\r\nContent-Length: 231\r\nContent-Type: application/json\r\nDate: Thu, 20 Aug 2026 16:00:26 GMT\r\nGrpc-Metadata-Content-Type: application/grpc\r\n\r\n{\n \"agent_id\": \"\",\n \"runs_on_node_id\": \"\",\n \"node_name\": \"\",\n \"server_info\": null,\n \"agents_info\": [],\n \"config_filepath\": \"/usr/local/percona/pmm/config/pmm-agent.yaml\",\n \"agent_version\": \"3.8.0\",\n \"connection_uptime\": 0\n}\n" component=setup
time="2026-08-20T16:00:26.777+00:00" level=debug msg="Status error: <nil>" component=setup
time="2026-08-20T16:00:26.777+00:00" level=debug msg="POST /v1/management/nodes HTTP/1.1\r\nHost: <SERVER ADDRESS>:443\r\nUser-Agent: Go-http-client/1.1\r\nContent-Length: 231\r\nAccept: application/json\r\nAuthorization: Basic Y2xpZW50LXJlZ2lzdGVyOnphcTEyd3N4Y2RlMzRyZnY=\r\nContent-Type: application/json\r\nAccept-Encoding: gzip\r\n\r\n{\"node_type\":\"NODE_TYPE_GENERIC_NODE\",\"node_name\":\"<LOCAL HOSTNAME>\",\"address\":\"<LOCAL IP>\",\"machine_id\":\"56da652e05d844cc8f589dadfab293c8\",\"distro\":\"linux\",\"metrics_mode\":\"METRICS_MODE_AUTO\",\"disable_collectors\":null}\n\n" component=setup
time="2026-08-20T16:00:33.015+00:00" level=debug msg="HTTP/1.1 401 Unauthorized\r\nContent-Length: 80\r\nConnection: keep-alive\r\nContent-Type: application/json\r\nDate: Thu, 20 Aug 2026 16:00:33 GMT\r\nStrict-Transport-Security: max-age=31536000; includeSubDomains\r\n\r\n{\"code\":13,\"error\":\"Internal server error.\",\"message\":\"Internal server error.\"}\n\n" component=setup
time="2026-08-20T16:00:33.015+00:00" level=debug msg="Register error: &management_service.RegisterNodeDefault{_statusCode:401, Payload:(*management_service.RegisterNodeDefaultBody)(0x2c2c4b8f72f0)}" component=setup