OVA import and console login via root

Percona Monitoring and Management (PMM) PMM 1.x
1

We imported the PMM server via the OVA option (1.7 client and server).
The Admin who helped with the import now needs to login as root to reset some DHCP settings (I was told)
I saw another posting that references root login via console (and there was another posting about login with ssh keys)
The password mentioned in the posting does not work for me (is the the web console? or some other console)
Can you please clarify how to login to the OVA server
Additionally at the client end I see as below - which I suspect after reading is related to some firewall port issue
The client has no firewall ports enabled. So We likely need to check at the server end.
Please advise

After the import able to logon to the console - but on the client check-network shows :

  • Connection: Client ← Server

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

linux:metrics a301-6362-0259.ldn.xxxxxxxxx.com 10.26.140.213:42000 DOWN YES YES
mysql:metrics a301-6362-0259.ldn.xxxxxxxxx.com 10.26.140.213:42002 DOWN YES YES
And

pmm-admin list

pmm-admin 1.7.0

PMM Server | 10.10.136.113 (password-protected)
Client Name | a301-6362-0259.ldn.xxxxxxxxx.com
Client Address | 10.26.140.213
Service Manager | unix-systemv

SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS

mysql:queries a301-6362-0259.ldn.xxxxxxxxx.com - YES root:@unix(/app/ipsoft/mysql/mysql.sock) query_source=slowlog, query_examples=true
linux:metrics a301-6362-0259.ldn.xxxxxxxxx.com 42000 YES -
mysql:metrics a301-6362-0259.ldn.xxxxxxxxx.com 42002 YES root:@unix(/app/ipsoft/mysql/mysql.sock) tablestats=OFF

2

Update: Login issue resolved.
There appeared to be an ntp issue - also resolved
However I am still getting the DOWN error
On the PMM server I see the following - Please advise

Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: Percona Monitoring and Management [ip address]
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: ++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: 2018-02-26 17:49:04,681 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: 2018-02-26 17:49:04,681 - util.py[WARNING]: Running module scripts-per-boot (<module ‘cloudinit.config.cc_scripts_per_boot’ from ‘/usr/lib/python2.7/site-…t.pyc’>) failed
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: Cloud-init v. 0.7.9 finished at Mon, 26 Feb 2018 17:49:04 +0000. Datasource DataSourceNone. Up 283.27 seconds
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: 2018-02-26 17:49:04,698 - cc_final_message.py[WARNING]: Used fallback datasource
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: cloud-final.service: main process exited, code=exited, status=1/FAILURE
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: Failed to start Execute cloud user/final scripts.
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: Unit cloud-final.service entered failed state.
Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: cloud-final.service failed.

3

Hi,

Could you please double check if you can connect from pmm-server to client’s exporters? You can do that with netcat (nc) / telnet / curl:


# telnet 10.26.140.213 42000
# curl 10.26.140.213:42000/metrics

Also, could you please check what Prometheus is logging on PMM-server?


# grep prometheus /var/log/messages|tail -n 500

Regarding cloud-init warnings:
Did you edit the cloud-init (/etc/cloud/cloud.cfg) configuration file inside VM?

4

telnet 10.26.140.213 42000 - [COLOR=#FF0000]F

curl 10.26.140.213:42000/metrics - [COLOR=#FF0000]F

— Note there are no filewalls
Output on both hosts pmm-server and client

# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Did you edit the cloud init file - NO
However, after seeing some posting related to “context deadline exceeded” [URL]https://github.com/prometheus/prometheus/issues/1438[/URL] I changed the [COLOR=#24292E]

Sample of output from the Prometheus grep

Feb 27 14:10:21 a301-9173-4370 prometheus: time="2018-02-27T14:10:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.58633ms." source="persistence.go:665"
Feb 27 14:15:21 a301-9173-4370 prometheus: time="2018-02-27T14:15:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:15:21 a301-9173-4370 prometheus: time="2018-02-27T14:15:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.27228ms." source="persistence.go:665"
Feb 27 14:20:21 a301-9173-4370 prometheus: time="2018-02-27T14:20:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:20:21 a301-9173-4370 prometheus: time="2018-02-27T14:20:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.430995ms." source="persistence.go:665"
Feb 27 14:25:21 a301-9173-4370 prometheus: time="2018-02-27T14:25:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:25:21 a301-9173-4370 prometheus: time="2018-02-27T14:25:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.538615ms." source="persistence.go:665"
Feb 27 14:30:21 a301-9173-4370 prometheus: time="2018-02-27T14:30:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:30:21 a301-9173-4370 prometheus: time="2018-02-27T14:30:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.393278ms." source="persistence.go:665"
Feb 27 14:34:59 a301-9173-4370 prometheus: time="2018-02-27T14:34:59Z" level=info msg="Completed full maintenance sweep through 1454 in-memory fingerprints in 4h2m20.180974991s." source="storage.go:1400"
Feb 27 14:35:21 a301-9173-4370 prometheus: time="2018-02-27T14:35:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:35:21 a301-9173-4370 prometheus: time="2018-02-27T14:35:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.905051ms." source="persistence.go:665"
Feb 27 14:40:21 a301-9173-4370 prometheus: time="2018-02-27T14:40:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:40:21 a301-9173-4370 prometheus: time="2018-02-27T14:40:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.826741ms." source="persistence.go:665"
Feb 27 14:45:21 a301-9173-4370 prometheus: time="2018-02-27T14:45:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:45:21 a301-9173-4370 prometheus: time="2018-02-27T14:45:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.553154ms." source="persistence.go:665"
Feb 27 14:50:21 a301-9173-4370 prometheus: time="2018-02-27T14:50:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:50:21 a301-9173-4370 prometheus: time="2018-02-27T14:50:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.791571ms." source="persistence.go:665"
Feb 27 14:55:21 a301-9173-4370 prometheus: time="2018-02-27T14:55:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
Feb 27 14:55:21 a301-9173-4370 prometheus: time="2018-02-27T14:55:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.206815ms." source="persistence.go:665"
5

Is there a way for me to change the port setup from 4200x to some other value… Looks like there is a subnet issue preventing access to 4200x

6

Just noting for anyone else reading this
As documented in the troubleshooting guide (no idea why I didn’t get any response here) you can specify the port with the add command (not documented )
Example:

pmm-admin add linux:metrics --service-port 42004

Though interestingly now I have the sql metrics showing as running but the Linux metrics as down
With no response I am wondering what next :slight_smile:

mysql:metrics a301-1234-5678.***.*********.com 10.26.156.93:42002 OK YES YES
linux:metrics a301--1234-5678.***.**********.com 10.26.156.93:42004 [COLOR=#B22222][B] DOWN [/B] YES YES
7

Hello,

it clearly looks like a network issue.

I’d suggest you verifying one more time if you are able to connect to 10.26.156.93:42004. This is HTTP, so you can just use curl to check if 10.26.156.93:42004/metrics responds.
Note, that you can check what is the state of Prometheus’ exporters through [url]http://PMM/prometheus/targets[/url].

8

The issue appears to be intermittent. It resets and gets to Ok and then fails again
I suspect that it is related to point mentioned in the FAQ - [url]Percona Monitoring and Management
The linux:metrics and mongodb:metrics services are set up to collect metrics with 1 second resolution
Though there does not appear to be an answer to how to rectify it for an OVA based implementation

9

Hi rpande

Can you share the contents of /var/log/pmm-mysql-metrics* please
Also the endpoints for mysqld_exporter have three valid URLs:

<ip-address>:42002/metrics-lr
<ip-address>:42002/metrics-mr
<ip-address>:42002/metrics-hr

Scraping just /metrics won’t work
For example:

curl -s -k https://1.2.3.4:42002:/metrics-hr

Should return a large list.