Pmm-admin status code 404 (text/html)

Hi,
i’m new to the forrum but i am using Pmm-server for several years now and walked thourh many user posts here befor posting this one.

I was on my way to add new pmm-client (mysql metrics) to my pmm server when i come over an what i think is an incompatibility issue :

pmm-admin config --server ip-adress --server-user user --server-password password

OK, PMM server is alive.

PMM Server      | ip-adress (password-protected)
Client Name     | server_name
Client Address  | client-ip-adress

sudo pmm-admin add mysql --user pmm --password password
status code 404 (text/html)

**to get more information about this error i addes the --verbose flag**

sudo pmm-admin --verbose add mysql --user pmm --password password
2021/02/16 11:38:48 request:
> GET /qan-api/ping HTTP/1.1
> Host: ip-adress
> User-Agent: Go-http-client/1.1
> Authorization: Basic YXRobWluOmF0aG1pbg==
> Accept-Encoding: gzip
>
>
2021/02/16 11:38:48 response:
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: Content-Type,Authorization
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Origin: *
< Connection: keep-alive
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 16 Feb 2021 10:38:48 GMT
< Server: nginx
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Percona-Qan-Api-Version: 1.3.2-1710050805.582bb9c
< Content-Length: 0
<
<
2021/02/16 11:38:48 request:
> GET /v1/status/leader HTTP/1.1
> Host: ip-adress
> User-Agent: Go-http-client/1.1
> Authorization: Basic YXRobWluOmF0aG1pbg==
> Accept-Encoding: gzip
>
>
2021/02/16 11:38:48 response:
< HTTP/1.1 200 OK
< Content-Length: 16
< Connection: keep-alive
< Content-Type: application/json
< Date: Tue, 16 Feb 2021 10:38:48 GMT
< Server: nginx
< X-Remote-Ip: ip-adress
< X-Server-Time: Tuesday, 16-Feb-2021 10:38:48 GMT
<
< "127.0.0.1:8300"
2021/02/16 11:38:48 request:
> GET /qan-api/ping HTTP/1.1
> Host: ip-adress
> User-Agent: Go-http-client/1.1
> Accept-Encoding: gzip
>
>
2021/02/16 11:38:48 response:
< HTTP/1.1 401 Unauthorized
< Content-Length: 188
< Connection: keep-alive
< Content-Type: text/html
< Date: Tue, 16 Feb 2021 10:38:48 GMT
< Server: nginx
< Www-Authenticate: Basic realm="on"
<
< <html>
< <head><title>401 Authorization Required</title></head>
< <body bgcolor="white">
< <center><h1>401 Authorization Required</h1></center>
< <hr><center>nginx</center>
< </body>
< </html>
<
2021/02/16 11:38:48 request:
> GET /managed/v1/version HTTP/1.1
> Host: ip-adress
> User-Agent: Go-http-client/1.1
> Authorization: Basic YXRobWluOmF0aG1pbg==
> Accept-Encoding: gzip
>
>
2021/02/16 11:38:48 response:
< HTTP/1.1 404 Not Found
< Transfer-Encoding: chunked
< Connection: keep-alive
< Content-Type: text/html
< Date: Tue, 16 Feb 2021 10:38:48 GMT
< Server: nginx
<
< a2
< <html>
< <head><title>404 Not Found</title></head>
< <body bgcolor="white">
< <center><h1>404 Not Found</h1></center>
< <hr><center>nginx</center>
< </body>
< </html>
<
< 0
<
<
status code 404 (text/html)
 **pmm-admin networkcheck returns the same error : ** 
status code 404 (text/html)

**Here is some information about the configuration :** 

pmm-admin info
pmm-admin 1.17.4

PMM Server      | server-ip (password-protected)
Client Name     | host_name
Client Address  | host ip
Service Manager | linux-systemd

Go Version      | 1.10.1
Runtime Info    | linux/amd64

**The pmm-admin version of server is** 

pmm-admin -v
1.3.1

On Homepage it shown : v1.3.2

My firs question is : is pmm-client v1.17.4 compatible with server version 1.3.1 ?

If not where can i download older/archived version of pmm-client for debian based systems ?

After a few hours of seraching on internet i was not able to get older version of pmm-client.

fortunately i keept the package on one of my client servers so i was able to make some tests :

lets try with the older version of pmm-client :

dpkg -i pmm-client_1.10.0-1.xenial_amd64.deb

pmm-admin config --server ip^-adress --server-user user --server-password password
OK, PMM server is alive.

PMM Server      | server-ip (password-protected)
Client Name     | hostname
Client Address  | client-ip

sudo pmm-admin add mysql --user pmm --password password
[linux:metrics] OK, now monitoring this system.
[mysql:metrics] OK, now monitoring MySQL metrics using DSN root:***@unix(/var/run/mysqld/mysqld.sock)
[mysql:queries] OK, now monitoring MySQL queries from slowlog using DSN root:***@unix(/var/run/mysqld/mysqld.sock)

So from here whe can see that with the older version of pmm-client it seems to work

 pmm-admin list
pmm-admin 1.10.0

PMM Server      | server-ip (password-protected)
Client Name     | hostname
Client Address  | client-ip
Service Manager | linux-systemd

-------------- ----- ----------- -------- ------------------------------------------- ------------------------------------------
SERVICE TYPE   NAME  LOCAL PORT  RUNNING  DATA SOURCE                                 OPTIONS
-------------- ----- ----------- -------- ------------------------------------------- ------------------------------------------
mysql:queries  hostname  -           YES      root:***@unix(/var/run/mysqld/mysqld.sock)  query_source=slowlog, query_examples=true
linux:metrics  hostname  42000       YES      -
mysql:metrics  hostname  42002       YES      root:***@unix(/var/run/mysqld/mysqld.sock)

Whe can see that ther is a warning of version compatibility :

Error getting a list of external metrics: status code 404 (text/html).
Please check versions of your PMM Server and PMM Client.

As i have serveral older linx-based systems that works with older versions of the pmm-client, what is the good stratégy to get the whole thing work together ?

if i update pmm-server, will i found most recent pmm1 clients for older linux versions ?
is pmm2 compatble with older linux version

Any good advice would be helpfull :slight_smile:

Hope this post can help some one encountering same issue.

There is another issuer you can encounter when you var using pmm-client 1.10.0-1

I encountered the following error when setting up Mysql Metrics :

Please note that pmm-serve ↔ Client setup is all working fine :

pmm-admin check-network
PMM Network Status

Server Address | server-ip
Client Address | client-ip

  • System Time
    NTP Server (0.pool.ntp.org) | 2021-02-16 12:16:04 +0100 CET
    PMM Server | 2021-02-16 11:16:04 +0000 GMT
    PMM Client | 2021-02-16 12:16:04 +0100 CET
    PMM Server Time Drift | OK
    PMM Client Time Drift | OK
    PMM Client to PMM Server Time Drift | OK

  • Connection: Client → Server


SERVER SERVICE STATUS


Consul API OK
Prometheus API OK
Query Analytics API OK

pmm-admin add mysql --user pmm --password password
bash: !228: event not found

This issue is due to special characters in Mysql_User password : for me it was @ and !

Hope this helps

1 Like

Here is another problem from another server i want to add mysql metrics

pmm-admin add mysql --user pmm --password password
[linux:metrics] OK, already monitoring this system.
[mysql] 2021/02/16 12:41:09 packets.go:33: unexpected EOF
[mysql] 2021/02/16 12:41:09 packets.go:130: write unix @->/var/run/mysqld/mysqlx.sock: write: broken pipe
[mysql] 2021/02/16 12:41:39 packets.go:33: unexpected EOF
[mysql] 2021/02/16 12:41:39 packets.go:130: write unix @->/var/run/mysqld/mysqlx.sock: write: broken pipe

The solution to this one is to indicate the mysql socket file like :

pmm-admin add mysql --user pmm --password password --socket path_to_mysql_socket_file

refer to [mysqld] section of you configuration file

Regards

1 Like

All pmm-client versions are available on our site

BTW you have not mentioned what older linux-based systems you have got.

Percona provides .deb packages for 64-bit versions of the following distributions:

  • Debian 8 (jessie)
  • Debian 9 (stretch)
  • Ubuntu 14.04 LTS (Trusty Tahr)
  • Ubuntu 16.04 LTS (Xenial Xerus)
  • Ubuntu 16.10 (Yakkety Yak)
  • Ubuntu 17.10 (Artful Aardvark)
  • Ubuntu 18.04 (Bionic Beaver)
1 Like

Hi ,

Thanks for reply,

BTW you have not mentioned what older linux-based systems you have got.

Unfortunatly i was not able to find this page by Googling arround, may be my fault ?
The Oldest system i still need to run is Ubuntu 14 LTS that runs the pmm-client 1.10.0

unless I am mistaken if I update my pmm-server this system will no longer be supported ?

Are there backports repos for backwards compatibility?

Kind regards

1 Like
1 Like

Hi just some updates on other error encountered with pmm V1

I installed Pmm-client v1.10 on Ubuntu 20 and encountered the following problems

[mysql:metrics] Cannot connect to MySQL: MySQL server does not support required protocol 41+

when i tried to use socket option i got the following

[mysql:metrics] Cannot connect to MySQL: this user requires mysql native password authentication.

The solution is to set mysql server back to musql_native_password as defauly auth plugin

nano /etc/mysql/mysql.conf.d/mysqld.cnf

[mysqld]

default-authentication-plugin = mysql_native_password

1 Like

The next problem was that mysql:metrics does not registre properly

Error adding MySQL queries: problem with agent registration on QAN API: exit status 1

getting the following log output

cat /var/log/pmm-mysql-metrics-42002.log
time=“2022-01-28T11:49:01+01:00” level=info msg=“Starting mysqld_exporter (version=, branch=, revision=)” source=“mysqld_exporter.go:288”
time=“2022-01-28T11:49:01+01:00” level=info msg=“Build context (go=go1.9.4, user=, date=)” source=“mysqld_exporter.go:289”
time=“2022-01-28T11:49:01+01:00” level=fatal msg=“failed reading ini file: open .my.cnf: no such file or directory” source=“mysqld_exporter.go:296”

the solution is to midify the mysql_exporter service by specifying the authetification mysql client section in .my.cnf file

1 create a .my.cnf file like
[client]
user = pmm
password = [pmm mysql password]

modify the mysql_exporter service

nano /etc/systemd/system/pmm-mysql-metrics-42002.service

ExecStart=/bin/sh -c '/usr/local/percona/pmm-client/mysqld_exporter -config.my-cnf=/etc/pmm-mysql/.my.cnf

systemctl daemon-reload
systemctl restart pmm-mysql-metrics-42002

pmm-client# pmm-admin list
mysql:metrics helgi 42002 YES pmm:***@unix(/var/run/mysqld/mysqld.sock)

Hope This could help some other percona fanatics who still need v1 to work on more recent systems :slight_smile:

1 Like