Not the answer you need?
Register and ask your own question!

OVA import and console login via root

rpanderpande EntrantCurrent User Role Beginner
We imported the PMM server via the OVA option (1.7 client and server).
The Admin who helped with the import now needs to login as root to reset some DHCP settings (I was told)
I saw another posting that references root login via console (and there was another posting about login with ssh keys)
The password mentioned in the posting does not work for me (is the the web console? or some other console)
Can you please clarify how to login to the OVA server
Additionally at the client end I see as below - which I suspect after reading is related to some firewall port issue
The client has no firewall ports enabled. So We likely need to check at the server end.
Please advise

After the import able to logon to the console - but on the client check-network shows :

* Connection: Client <-- Server





SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD





linux:metrics a301-6362-0259.ldn.xxxxxxxxx.com 10.26.140.213:42000 DOWN YES YES
mysql:metrics a301-6362-0259.ldn.xxxxxxxxx.com 10.26.140.213:42002 DOWN YES YES
And
# pmm-admin list
pmm-admin 1.7.0

PMM Server | 10.10.136.113 (password-protected)
Client Name | a301-6362-0259.ldn.xxxxxxxxx.com
Client Address | 10.26.140.213
Service Manager | unix-systemv






SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS





mysql:queries a301-6362-0259.ldn.xxxxxxxxx.com - YES root:***@unix(/app/ipsoft/mysql/mysql.sock) query_source=slowlog, query_examples=true
linux:metrics a301-6362-0259.ldn.xxxxxxxxx.com 42000 YES -
mysql:metrics a301-6362-0259.ldn.xxxxxxxxx.com 42002 YES root:***@unix(/app/ipsoft/mysql/mysql.sock) tablestats=OFF

Comments

  • rpanderpande Entrant Current User Role Beginner
    Update: Login issue resolved.
    There appeared to be an ntp issue - also resolved
    However I am still getting the DOWN error
    On the PMM server I see the following - Please advise

    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: Percona Monitoring and Management [ip address]
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: ++++++++++++++++++++++++++++++++++++++++++++++++++ ++++++++++++++++++++++++++++++
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: 2018-02-26 17:49:04,681 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: 2018-02-26 17:49:04,681 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python2.7/site-...t.pyc'>) failed
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: Cloud-init v. 0.7.9 finished at Mon, 26 Feb 2018 17:49:04 +0000. Datasource DataSourceNone. Up 283.27 seconds
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com cloud-init[1430]: 2018-02-26 17:49:04,698 - cc_final_message.py[WARNING]: Used fallback datasource
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: cloud-final.service: main process exited, code=exited, status=1/FAILURE
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: Failed to start Execute cloud user/final scripts.
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: Unit cloud-final.service entered failed state.
    Feb 26 17:49:04 a301-9173-4370.ldn.xxxxxxxxx.com systemd[1]: cloud-final.service failed.
  • IMPIMP Percona Percona Staff Role
    Hi,

    Could you please double check if you can connect from pmm-server to client's exporters? You can do that with netcat (nc) / telnet / curl:
    # telnet 10.26.140.213 42000
    # curl 10.26.140.213:42000/metrics
    

    Also, could you please check what Prometheus is logging on PMM-server?
    # grep prometheus /var/log/messages|tail -n 500
    

    Regarding cloud-init warnings:
    Did you edit the cloud-init (/etc/cloud/cloud.cfg) configuration file inside VM?
  • rpanderpande Entrant Current User Role Beginner
    # telnet 10.26.140.213 42000 - F
    # curl 10.26.140.213:42000/metrics - F



    --- Note there are no filewalls
    Output on both hosts pmm-server and client
    # iptables --list
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    

    Did you edit the cloud init file - NO
    However, after seeing some posting related to "context deadline exceeded" https://github.com/prometheus/prometheus/issues/1438 I changed the

    Sample of output from the Prometheus grep
    Feb 27 14:10:21 a301-9173-4370 prometheus: time="2018-02-27T14:10:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.58633ms." source="persistence.go:665"
    Feb 27 14:15:21 a301-9173-4370 prometheus: time="2018-02-27T14:15:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:15:21 a301-9173-4370 prometheus: time="2018-02-27T14:15:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.27228ms." source="persistence.go:665"
    Feb 27 14:20:21 a301-9173-4370 prometheus: time="2018-02-27T14:20:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:20:21 a301-9173-4370 prometheus: time="2018-02-27T14:20:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.430995ms." source="persistence.go:665"
    Feb 27 14:25:21 a301-9173-4370 prometheus: time="2018-02-27T14:25:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:25:21 a301-9173-4370 prometheus: time="2018-02-27T14:25:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.538615ms." source="persistence.go:665"
    Feb 27 14:30:21 a301-9173-4370 prometheus: time="2018-02-27T14:30:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:30:21 a301-9173-4370 prometheus: time="2018-02-27T14:30:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.393278ms." source="persistence.go:665"
    Feb 27 14:34:59 a301-9173-4370 prometheus: time="2018-02-27T14:34:59Z" level=info msg="Completed full maintenance sweep through 1454 in-memory fingerprints in 4h2m20.180974991s." source="storage.go:1400"
    Feb 27 14:35:21 a301-9173-4370 prometheus: time="2018-02-27T14:35:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:35:21 a301-9173-4370 prometheus: time="2018-02-27T14:35:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.905051ms." source="persistence.go:665"
    Feb 27 14:40:21 a301-9173-4370 prometheus: time="2018-02-27T14:40:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:40:21 a301-9173-4370 prometheus: time="2018-02-27T14:40:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.826741ms." source="persistence.go:665"
    Feb 27 14:45:21 a301-9173-4370 prometheus: time="2018-02-27T14:45:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:45:21 a301-9173-4370 prometheus: time="2018-02-27T14:45:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 17.553154ms." source="persistence.go:665"
    Feb 27 14:50:21 a301-9173-4370 prometheus: time="2018-02-27T14:50:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:50:21 a301-9173-4370 prometheus: time="2018-02-27T14:50:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.791571ms." source="persistence.go:665"
    Feb 27 14:55:21 a301-9173-4370 prometheus: time="2018-02-27T14:55:21Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
    Feb 27 14:55:21 a301-9173-4370 prometheus: time="2018-02-27T14:55:21Z" level=info msg="Done checkpointing in-memory metrics and chunks in 16.206815ms." source="persistence.go:665"
    
  • rpanderpande Entrant Current User Role Beginner
    Is there a way for me to change the port setup from 4200x to some other value.. Looks like there is a subnet issue preventing access to 4200x
  • rpanderpande Entrant Current User Role Beginner
    Just noting for anyone else reading this
    As documented in the troubleshooting guide (no idea why I didn't get any response here) you can specify the port with the add command (not documented )
    Example:
    pmm-admin add linux:metrics --service-port 42004
    
    Though interestingly now I have the sql metrics showing as running but the Linux metrics as down
    With no response I am wondering what next :)
    mysql:metrics  a301-1234-5678.***.*********.com  10.26.156.93:42002  OK      YES        YES
    linux:metrics  a301--1234-5678.***.**********.com  10.26.156.93:42004 [COLOR=#B22222][B] DOWN  [/B]  YES        YES
    
  • IMPIMP Percona Percona Staff Role
    Hello,

    it clearly looks like a network issue.

    I'd suggest you verifying one more time if you are able to connect to 10.26.156.93:42004. This is HTTP, so you can just use curl to check if 10.26.156.93:42004/metrics responds.
    Note, that you can check what is the state of Prometheus' exporters through http://PMM/prometheus/targets.
  • rpanderpande Entrant Current User Role Beginner
    The issue appears to be intermittent. It resets and gets to Ok and then fails again
    I suspect that it is related to point mentioned in the FAQ - https://www.percona.com/doc/percona-monitoring-and-management/faq.html
    The linux:metrics and mongodb:metrics services are set up to collect metrics with 1 second resolution
    Though there does not appear to be an answer to how to rectify it for an OVA based implementation
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi rpande

    Can you share the contents of /var/log/pmm-mysql-metrics* please
    Also the endpoints for mysqld_exporter have three valid URLs:
    <ip-address>:42002/metrics-lr
    <ip-address>:42002/metrics-mr
    <ip-address>:42002/metrics-hr
    
    Scraping just /metrics won't work
    For example:
    curl -s -k https://1.2.3.4:42002:/metrics-hr
    
    Should return a large list.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.