# Pmm client crashes

**URL:** <https://forums.percona.com/t/pmm-client-crashes/5320>\
**Category:** PMM 1.x\
**Created:** [December 26, 2016, 11:47am UTC](https://forums.percona.com/t/pmm-client-crashes/5320 "2016-12-26T11:47:12Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [December 26, 2016, 11:47am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/1 "2016-12-26T11:47:12Z")

</div>

1. pmm client mysql:metrics crashes every now and then. I have already created a superuser for it on the mysql host. It has even grant permission.  
How do I check logs on client side to find out what is going wrong.  
[root@XYZ sbin]# pmm-admin list  
pmm-admin 1.0.6  
PMM Server | XX.YY.ZZ.AA  
Client Name | ABC  
Client Address | AA.BB.CC.DD  
Service manager | unix-systemv

* * *

SERVICE TYPE NAME CIENT PORT RUNNING DATA SOURCE OPTIONS

* * *

linux:metrics ABC 42000 NO -  
mysql:queries ABC 42001 YES root:_**@unix(/export/home/mysql/mysql.sock) query\_source=slowlog  
mysql:metrics ABC 42002 NO pmm:**_@unix(/export/home/mysql/mysql.sock)

Error Says  
time=“2016-10-18T14:29:31Z” level=info msg=“Starting mysqld\_exporter (version=1.0.5, branch=master, revision=3c95bb2c647443430db2337627d5f677665f41c7)” source=“mysqld\_exporter.go:631”

time=“2016-10-18T14:29:31Z” level=info msg=“Build context (go=go1.7.1, user=, date=)” source=“mysqld\_exporter.go:632”

time=“2016-10-18T14:29:31Z” level=info msg=“Listening on :42002” source=“mysqld\_exporter.go:666”

time=“2016-10-18T14:29:31Z” level=fatal msg=“listen tcp :42002: bind: cannot assign requested address” source=“mysqld\_exporter.go:667”

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [December 27, 2016, 4:51am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/2 "2016-12-27T04:51:30Z")

</div>

> [@kooshals;46828](#):
>
> listen tcp :42002: bind: cannot assign requested address

Is selinux disabled?  
according to documentation it should be disabled - [https://www.percona.com/doc/percona-monitoring-and-management/install.html#installing-pmm-client](https://www.percona.com/doc/percona-monitoring-and-management/install.html#installing-pmm-client)

---

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [December 27, 2016, 6:00am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/3 "2016-12-27T06:00:16Z")

</div>

Yes Selinux is disabled. I think its the problem with address binding on client.  
How do I use -bind-address option in new 1.0.7 client?

---

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [December 27, 2016, 6:04am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/4 "2016-12-27T06:04:52Z")

</div>

Here is the exact current status

CLIENT  
[root@XYZ bin]# ./pmm-admin list  
pmm-admin 1.0.7

PMM Server | Z.Z.Z.Z  
Client Name | [XYZ.COMPANY.com](http://XYZ.COMPANY.com)  
Client Address | X.X.X.X  
Service Manager | unix-systemv

* * *

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

* * *

mysql:queries [XYZ.COMPANY.com](http://XYZ.COMPANY.com) - YES root:_**@unix(/export/home/mysql/mysql.sock) query\_source=slowlog  
linux:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) 42000 NO -  
mysql:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) 42002 NO root:**_@unix(/export/home/mysql/mysql.sock)

[root@XYZ bin]# ./pmm-admin check-network  
PMM Network Status

Server Address | Z.Z.Z.Z  
Client Address | X.X.X.X

- System Time  
Server | 2016-12-27 02:39:08 -0800 PST  
Client | 2016-12-27 02:39:08 -0800 PST  
Time Drift | OK

- Connection: Client → Server

* * *

SERVER SERVICE STATUS

* * *

Consul API OK  
Prometheus API OK  
Query Analytics API OK

Connection duration | 71.493201ms  
Request duration | 75.272449ms  
Full round trip | 146.76565ms

- Connection: Client ← Server

* * *

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

* * *

linux:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) X.X.X.X:42000 DOWN - -  
mysql:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) X.X.X.X:42002 DOWN - -

ERROR in THE LOG  
time=“2016-12-27T02:38:15-08:00” level=info msg=“Build context (go=go1.7.4, user=, date=)” source=“mysqld\_exporter.go:680”  
time=“2016-12-27T02:38:15-08:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld\_exporter.go:724”  
time=“2016-12-27T02:38:15-08:00” level=info msg=“Listening on X.X.X.X:42002” source=“mysqld\_exporter.go:727”  
time=“2016-12-27T02:38:37-08:00” level=fatal msg=“listen tcp X.X.X.X:42002: bind: cannot assign requested address” source=“mysqld\_exporter.go:777”

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [December 27, 2016, 6:26am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/5 "2016-12-27T06:26:31Z")

</div>

> [@kooshals;46839](#):
>
> I think its the problem with address binding on client.

We should understand the reason why it is impossible to open port on this machine.  
possible reasons:  
[LIST]  
[_]port already busy by another application  
[_]selinux does not allow open this port  
[_]no IPv4 addresses on host  
[_]binding to IP which is not exists on this host (for example binding to public IP address on AWS instance instead of private IP)  
[\*]etc  
[/LIST]

> [@kooshals;46839](#):
>
> How do I use -bind-address option in new 1.0.7 client?

It is required to use exactly the same version of PMM client and server.  
So please try to update both PMM client and server to version 1.0.7

---

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [December 27, 2016, 9:09am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/6 "2016-12-27T09:09:19Z")

</div>

Here is the exact command I used to add the server to client. Also both are running 1.0.7  
pmm-admin config --server ELASTIC-IP-OF-AWS-MACHINE –bind-address PUBLIC-IP-OF-CLIENT --client-address INTERNAL-IP-OF-CLIENT

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [December 27, 2016, 11:45am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/7 "2016-12-27T11:45:06Z")

</div>

If server and client are in the same “Subnet ID” it is enough to configure access via internal IP only.

```auto
pmm-admin config --server INTERNAL-IP-OF-SERVER

```

If server and client are in different AWS Subnet IDs

```auto
pmm-admin config \
--server ELASTIC-IP-OF-SERVER \
--bind-address INTERNAL-IP-OF-CLIENT \
--client-address PUBLIC-IP-OF-CLIENT

```

according to help

```auto
# pmm-admin config --help
This command configures pmm-admin to communicate with PMM server.
...
Flags:
--bind-address string bind address, also local/private address that is mapped from client address via NAT/port forwarding (defaults to the client address)
--client-address string client address, also remote/public address for this system (if omitted it will be automatically detected by asking server)

```

---

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [January 3, 2017, 2:15am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/8 "2017-01-03T02:15:53Z")

</div>

Well the bind-address option worked with client 1.0.7. Also both client and server are running 1.0.7.  
While client is in our company DC, server is in AWS vpc.

Following is the config command that I used.  
pmm-admin config --server PUBLIC-IP of Server --bind-address Internal ip of client --client-address Public ip of client

pmm-admin list shows that all metrics are working fine.  
pmm-admin list  
pmm-admin 1.0.7

PMM Server | PUBLIC-IP of Server  
Client Name | [XYZ.COMPANY.com](http://XYZ.COMPANY.com)  
Client Address | Internal ip of client  
Service Manager | unix-systemv

* * *

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

* * *

mysql:queries [XYZ.COMPANY.com](http://XYZ.COMPANY.com) - YES root:_**@unix(/export/home/mysql/mysql.sock) query\_source=slowlog  
linux:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) 42000 YES -  
mysql:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) 42002 YES root:**_@unix(/export/home/mysql/mysql.sock)

But check-network shows that server-\> client has issue.  
pmm-admin check-network  
PMM Network Status

Server Address | PUBLIC-IP of Server  
Client Address | Public ip of client (Internal ip of client )

- System Time  
Server | 2017-01-02 23:28:12 -0800 PST  
Client | 2017-01-02 23:28:12 -0800 PST  
Time Drift | OK

- Connection: Client → Server

* * *

SERVER SERVICE STATUS

* * *

Consul API OK  
Prometheus API OK  
Query Analytics API OK

Connection duration | 72.795236ms  
Request duration | 83.901484ms  
Full round trip | 156.69672ms

- Connection: Client ← Server

* * *

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

* * *

linux:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) Public ip of client–\>Internal ip of client:42000 DOWN YES -  
mysql:metrics [XYZ.COMPANY.com](http://XYZ.COMPANY.com) Public ip of client–\>Internal ip of client:42002 DOWN YES -

Last log lines in the following logs on client machine  
pmm-mysql-metrics-42002.log  
time=“2017-01-02T23:29:17-08:00” level=info msg=“Starting mysqld\_exporter (version=1.0.7, branch=master, revision=4a4c53fb313fb1883bcbd464f53c83f73b336100)” source=“mysqld\_exporter.go:679”  
time=“2017-01-02T23:29:17-08:00” level=info msg=“Build context (go=go1.7.4, user=, date=)” source=“mysqld\_exporter.go:680”  
time=“2017-01-02T23:29:17-08:00” level=info msg=“HTTPS/TLS is enabled” source=“mysqld\_exporter.go:724”  
time=“2017-01-02T23:29:17-08:00” level=info msg=“Listening on Internal ip of client:42002” source=“mysqld\_exporter.go:727”  
2017/01/02 23:29:44 http: TLS handshake error from Internal ip of client:47658: tls: first record does not look like a TLS handshake

pmm-linux-metrics-42000.log  
time=“2017-01-02T23:29:16-08:00” level=info msg=“HTTPS/TLS is enabled” source=“node\_exporter.go:235”  
time=“2017-01-02T23:29:16-08:00” level=info msg=“Listening on Internal ip of client:42000” source=“node\_exporter.go:238”  
2017/01/02 23:29:44 http: TLS handshake error from Internal ip of client:50902: tls: first record does not look like a TLS handshake

Could you please let me know what is happening here? Client still doesnt show up the grafana.

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [January 3, 2017, 4:07am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/9 "2017-01-03T04:07:01Z")

</div>

according to shown output - client have been added correctly.  
now it is needed to understand why server cannot connect to client.  
possible reasons:  
[LIST]  
[_]internal client firewall  
[_]AWS security group rules  
[/LIST] it is needed to open 42000 (for linux:metrics) and 42002 (for mysql:metrics) ports from PMM Server to PMM Client.  
the following commands should should connect to port without errors.

```auto
# on PMM Client
telnet PRIVATE-IP-OF-CLIENT 42000
telnet PUBLIC-IP-OF-CLIENT 42000

# on PMM Server
telnet PUBLIC-IP-OF-CLIENT 42000

```

---

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [January 3, 2017, 4:20am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/10 "2017-01-03T04:20:42Z")

</div>

Ok so I think it is something to do with firewall of client inside our company datacenter. Can you please confirm.

ON CLIENT  
[root@HOSTNAME pmm-client-1.0.7]# telnet INTERNAL\_IP\_OF\_CLIENT 42000  
Trying INTERNAL\_IP\_OF\_CLIENT …  
Connected to [XYZ.COMPANY.com](http://XYZ.COMPANY.com) (INTERNAL\_IP\_OF\_CLIENT ).  
Escape character is ‘^]’.

[root@HOSTNAME pmm-client-1.0.7]# telnet PUBLIC\_IP\_OF\_CLIENT 42000  
Trying PUBLIC\_IP\_OF\_CLIENT …  
^C

[root@HOSTNAME pmm-client-1.0.7]# netstat -tanpu | grep 420  
tcp 0 0 INTERNAL\_IP\_OF\_CLIENT :42000 0.0.0.0:\* LISTEN 15416/node\_exporter  
tcp 0 0 INTERNAL\_IP\_OF\_CLIENT :42002 0.0.0.0:\* LISTEN 15438/mysqld\_export

ON SERVER  
[root@HOSTNAME-user]# telnet PUBLIC\_IP\_OF\_CLIENT 42000  
Trying PUBLIC\_IP\_OF\_CLIENT …  
^C

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [January 3, 2017, 4:27am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/11 "2017-01-03T04:27:21Z")

</div>

yes, it is definitely firewall issue  
client and server cannot connect to PUBLIC\_IP\_OF\_CLIENT 42000

---

<div class="post-metadata">

**Author:** ![kooshals](https://avatars.discourse-cdn.com/v4/letter/k/ea666f/32.png) [@kooshals](https://forums.percona.com/u/kooshals)\
**Post date:** [January 3, 2017, 4:28am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/12 "2017-01-03T04:28:19Z")

</div>

Thanks for confirming. i will try to get that fixed. Any other test we can do from server to check its connectivity with client other than telnet?

---

<div class="post-metadata">

**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\
**Post date:** [January 3, 2017, 4:57am UTC](https://forums.percona.com/t/pmm-client-crashes/5320/13 "2017-01-03T04:57:26Z")

</div>

If telnet is not installed, it is possible to use curl or openssl tool

```auto
curl https://PUBLIC_IP_OF_CLIENT:42000/metrics --insecure
# or
(echo -e "GET /metrics HTTP/1.0\r\n"; sleep 1) | openssl s_client -connect PUBLIC_IP_OF_CLIENT:42000

```