Not the answer you need?
Register and ask your own question!

Memory Usage graph reports error when >68 hosts are selected

ac2ac2 ContributorInactive User Role Beginner
Our PMM server monitors 16 hosts and 59 MySQL instances. After adding all the linux:metrics and mysql:metrics source types, we have detected that the Memory Usage graph in the Cross Server Graphs dashboard started reporting an error when all the hosts were selected.
Adding one by one we were able to restrict the problem to the number of hosts (order doesn't matter), basically when the number is > 68, the graph fails consistently with the following error:

Message:
a.data.data is undefined Stack trace:
b/this.query/</<@http://mysql-hxvm7-monitor-005:8000/graph/public/app/plugins/datasource/prometheus/datasource.js?bust=1486481238956:4:1767 [email][email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js[/email]:48:28473 [email][email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js[/email]:49:29427 b/this.query/<@http://mysql-hxvm7-monitor-005:8000/graph/public/app/plugins/datasource/prometheus/datasource.js?bust=1486481238956:4:1644 [email][email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js[/email]:38:29545 h/<@http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js:38:29717 xc/this.$get</[email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js:39:5353 xc/this.$get</[email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js:39:3835 xc/this.$get</[email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js:39:5638 x/i<@http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js:39:1804 [email][email protected]http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js[/email]:37:15602 jb/k.defer/c<@http://mysql-hxvm7-monitor-005:8000/graph/public/app/boot.85c49108.js:37:17066
All the other graphs seem to work as expected.

Is there any limitation in place we are not aware of?
We tried to dig into the Docker container logs, but we didn't find any clue.

Could you please help?

Thanks
Regards,
Alessio
«1

Comments

  • ac2ac2 Contributor Inactive User Role Beginner
    I should add, we label the hosts with the short hostname string, i.e.:

    pmm-admin add linux:metrics mysql-vm-0NN

    while the instances on the hosts are labelled as follows:

    pmm-admin add mysql --user <USER> --password <PWD> --query-source perfschema --socket <SOCKET> mysql-vm-0NN_<PORT>

    The Host drop down list in Graphana displays all the labels. When >68 labels are selected the error is triggered.
  • MykolaMykola Percona Percona Staff Role
    What is pmm-server version?
    What is you browser and version?
    Is issue reproducible in another browser (firefox)?
  • ac2ac2 Contributor Inactive User Role Beginner
    Both server and clients are 1.0.7/

    I have tried with Google Chrome and Firefox (latest versions) on OS X, RHEL6 and Fedora 24 without any noticeable change. The Memory Usage graph always reports the same error if >68 hosts are selected.

    Is there any other check I can perform?

    Thanks
  • MykolaMykola Percona Percona Staff Role
    can you enable "Developer Tools" in Chrome (Network tab) and try to catch response for 67 hosts and for 68?
    and share in same way? (pastebin, dropbox etc)
    I'll try to compare them.
  • ac2ac2 Contributor Inactive User Role Beginner
    Just to clarify. Are you looking for the HAR file that can be generated or a screen snaphot of the Network tab?

    Thanks
  • MykolaMykola Percona Percona Staff Role
    HAR please :)
  • ac2ac2 Contributor Inactive User Role Beginner
    Could you please try to download the HAR files for the "fail" and "success" conditions using the following URLs?

    https://drive.google.com/file/d/0B5FL9q-xRjLJZy1WNnRXRkV5Rms/view?usp=sharing
    https://drive.google.com/file/d/0B5FL9q-xRjLJVUxJRVlDU2lPcEU/view?usp=sharing

    Thanks
  • MykolaMykola Percona Percona Staff Role
    can you try to use the latest beta version of pmm-server?
    it has updated grafana to the latest version -

    I analyzed HAR files and it looks very strange.
    failed response size is exactly 200000 bytes...
    successful response size is much bigger - 6376716
    looks like some firewall or proxy cut packages if they bigger that 6.5 or 7 megabytes.
  • ac2ac2 Contributor Inactive User Role Beginner
    I can try with the Beta version later today.

    For what concerns the network issue, the MySQL and the PMM-servers are on the same network segment and no firewall is configured at any level.


    Thanks for your help
  • ac2ac2 Contributor Inactive User Role Beginner
    Just tried with the Docker image for version 1.1.0beta (containing Graphana 4.1.1) and the behavior is exactly the same as before.
  • ac2ac2 Contributor Inactive User Role Beginner
    Please disregard my comment on network, I misread you message. Client (browser) and PMM server are not on the same network segment.
  • MykolaMykola Percona Percona Staff Role
    Can you try to fetch exactly the same url (which you dumped to HAR file) locally on PMM Server and remotely from you computer?
    if result be the same (200000 bytes), it is definitely not network issue.
  • MykolaMykola Percona Percona Staff Role
    Chrome Developer tools has nice feature - "Copy as cURL".
    you can get simple curl command which just needed to run in shell.
  • ac2ac2 Contributor Inactive User Role Beginner
    I have done more than that. I have been able to temporary deploy firefox on the Docker host and the Memory Graph fails there as well, same as on any other browser from any other machine.
    Unless there is something weird on the container network stack, I would exclude a truncation of the TCP stream...
  • MykolaMykola Percona Percona Staff Role
    is "Load Average" graph is also affected?
    is "Memory Usage" still broken if open it alone in "fullscreen" mode (left click on title, "View" button) ?
    can you dump failed request one more time please?
  • ac2ac2 Contributor Inactive User Role Beginner
    Neither Load Average nor any other graph I have been able to check has problems excluding Memory Usage.

    The problem is present even in fullscreen mode.
  • MykolaMykola Percona Percona Staff Role
    Could you dump HAR file again?
    now in fullscreen mode
  • ac2ac2 Contributor Inactive User Role Beginner
    Here is the link: https://drive.google.com/file/d/0B5FL9q-xRjLJVUxJRVlDU2lPcEU/view?usp=sharing


    I will be traveling most of next week, so I don't know if I will be able to answer your message.

    Thanks a lot
  • MykolaMykola Percona Percona Staff Role
    it is old one :(
    can you make completely new dump with fail?
  • ac2ac2 Contributor Inactive User Role Beginner
    Sorry, I sent the link in a rush. Here is the correct one:

    https://drive.google.com/file/d/0B5FL9q-xRjLJVkN5c3NtTWhfa0k/view?usp=sharing
  • MykolaMykola Percona Percona Staff Role
    Hi ac2,

    it is fully empty :( size is 0 bytes
  • ac2ac2 Contributor Inactive User Role Beginner
    Sorry, the file was created on a remote connection while traveling...

    I have been back from holidays and generated a new file: https://drive.google.com/file/d/0B5FL9q-xRjLJWFc4WEZfQTI5RzA/view?usp=sharing

    Could you please check this?
  • MykolaMykola Percona Percona Staff Role
    is this issue connected to exact host or to any random 68 hosts?

    can you choose first 67 hosts in the list without issues?
    can you choose last 67 hosts in the list without issues?
    can you choose 67 hosts in the middle of the list without issues?
  • ac2ac2 Contributor Inactive User Role Beginner
    Hi Mykola,
    actually I made a mistake while counting, the critical threshold is 68, sorry for that. Basically, the memory graph works correctly with 68 hosts selected (top, bottom, middle or punched card style doesn't matter), fails with 69 onwards.
    I have tested version 1.1.1 without any luck, too.

    Thanks
  • ac2ac2 Contributor Inactive User Role Beginner
    We have 58 MySQL instances running on 16 MySQL VM, plus the PMM server is monitored, of course.

    I have tried to access data directly in Prometheus and I see all the metrics for all nodes without any trouble, hence the issue seems related to Grafana.

    Playing around selecting and deselecting hosts from the list, the problem appears "around" 68 hosts. Depending on the set, sometimes 67 is enough to trigger the problem, other times you need 69. So, it does not seem to be related to the number of hosts rather than the data volume to be accessed.

    Thanks
  • MykolaMykola Percona Percona Staff Role
    1. do you see any javascript errors in the browser javascript console?
    2. can you decrease resolution to 1/10
    it is needed to
    - login to grafana as Admin user
    - click on graph title
    - click "Edit"
    - choose "Metrics" tab
    - change Resolution value

    see screenshot
  • ac2ac2 Contributor Inactive User Role Beginner
    Sorry for the very late reply, been very busy during the past days.
    I have tried all the possible resolutions for the Memory trend graph, but I still get the error when >~68 hosts are selected from the list.
  • MykolaMykola Percona Percona Staff Role
    can you open "Developer Tools"->"Console" in your browser and reload page.
    do you see any javascript errors?
  • ac2ac2 Contributor Inactive User Role Beginner
    I have the following JS messages on the Console:

    TypeError: Cannot read property 'result' of undefined
    at eval (datasource.js?bust=1488533087258:4)
    at f (boot.b8fccc44.js:52)
    at Function.Pg (boot.b8fccc44.js:53)
    at eval (datasource.js?bust=1488533087258:4)
    at g (boot.b8fccc44.js:42)
    at boot.b8fccc44.js:42
    at o.$eval (boot.b8fccc44.js:42)
    at o.$digest (boot.b8fccc44.js:42)
    at o.$apply (boot.b8fccc44.js:42)
    at boot.b8fccc44.js:42
    (anonymous) @ boot.b8fccc44.js:42
    (anonymous) @ boot.b8fccc44.js:41
    g @ boot.b8fccc44.js:42
    (anonymous) @ boot.b8fccc44.js:42
    $eval @ boot.b8fccc44.js:42
    $digest @ boot.b8fccc44.js:42
    $apply @ boot.b8fccc44.js:42
    (anonymous) @ boot.b8fccc44.js:42
    f @ boot.b8fccc44.js:40
    (anonymous) @ boot.b8fccc44.js:41
    boot.b8fccc44.js:46

    Panel data error: TypeError: Cannot read property 'result' of undefined
    at eval (datasource.js?bust=1488533087258:4)
    at f (boot.b8fccc44.js:52)
    at Function.Pg (boot.b8fccc44.js:53)
    at eval (datasource.js?bust=1488533087258:4)
    at g (boot.b8fccc44.js:42)
    at boot.b8fccc44.js:42
    at o.$eval (boot.b8fccc44.js:42)
    at o.$digest (boot.b8fccc44.js:42)
    at o.$apply (boot.b8fccc44.js:42)
    at boot.b8fccc44.js:42
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.