invalid line for /proc/diskstats for sda

I found some server could not get the disk performance information . I checked the load then see this error . Is it because my system kernel version is too high causing exporter not to support the new /proc/diskstats format?

pmm-server 1.13.0
pmm-client 1.17.0

[root@node004027 fanboshi]# uname -a
Linux node004027 4.20.4-1.el7.elrepo.x86_64 #1 SMP Tue Jan 22 18:39:11 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@node004027 fanboshi]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)

[root@node004027 fanboshi]# tail -f /var/log/pmm-linux-metrics-42000.log
time=“2019-04-26T23:47:10+08:00” level=error msg=“ERROR: diskstats collector failed after 0.001108s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:47:15+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000929s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:47:20+08:00” level=error msg=“ERROR: diskstats collector failed after 0.001414s: invalid line for /proc/diskstats for sr0” source=“node_exporter.go:97”
time=“2019-04-26T23:47:25+08:00” level=error msg=“ERROR: diskstats collector failed after 0.002659s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:47:30+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000443s: invalid line for /proc/diskstats for sr0” source=“node_exporter.go:97”
2019/04/26 23:47:32 http: TLS handshake error from 192.168.4.27:30570: tls: first record does not look like a TLS handshake
2019/04/26 23:47:32 http: TLS handshake error from 192.168.4.27:30572: tls: first record does not look like a TLS handshake
time=“2019-04-26T23:47:35+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000193s: invalid line for /proc/diskstats for sdb” source=“node_exporter.go:97”
time=“2019-04-26T23:47:40+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000191s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:47:45+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000685s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:47:51+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000318s: invalid line for /proc/diskstats for sdb” source=“node_exporter.go:97”
time=“2019-04-26T23:47:55+08:00” level=error msg=“ERROR: diskstats collector failed after 0.001367s: invalid line for /proc/diskstats for sr0” source=“node_exporter.go:97”
time=“2019-04-26T23:48:00+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000331s: invalid line for /proc/diskstats for sr0” source=“node_exporter.go:97”
time=“2019-04-26T23:48:05+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000709s: invalid line for /proc/diskstats for sdb” source=“node_exporter.go:97”
time=“2019-04-26T23:48:10+08:00” level=error msg=“ERROR: diskstats collector failed after 0.001092s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:48:15+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000758s: invalid line for /proc/diskstats for sdb” source=“node_exporter.go:97”
time=“2019-04-26T23:48:20+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000793s: invalid line for /proc/diskstats for sdb” source=“node_exporter.go:97”
time=“2019-04-26T23:48:25+08:00” level=error msg=“ERROR: diskstats collector failed after 0.002234s: invalid line for /proc/diskstats for sda” source=“node_exporter.go:97”
time=“2019-04-26T23:48:30+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000508s: invalid line for /proc/diskstats for sr0” source=“node_exporter.go:97”
time=“2019-04-26T23:48:35+08:00” level=error msg=“ERROR: diskstats collector failed after 0.000851s: invalid line for /proc/diskstats for sdb” source=“node_exporter.go:97”

can anyone help?

1 Like

Hello again Fan…
Let me find out from the PMM team if they need a JIRA ticket for this one…

1 Like

Thanks alot

1 Like

Hello Fan,

pmm-client 1.17.0 is using node_exporter that can’t handle changes in the /proc/diskstats format.
The exporter has been updated in pmm2-client.

1 Like

Thank you @adivinho

1 Like

@Fan ,

Here is a link to the backported patch that fixes the problem in node_exporter version used in PMM 1.x:
https://github.com/shatteredsilicon/node_exporter/commit/cb2dd588eaadf44b53e51a932d44105ea5c14881

If you are using a RPM based distribution, you can get the .src.rpm to rebuild the binaries here:
https://dl.shatteredsilicon.net/pmm/SRPMS/

2 Likes

Thank you @gordan :slight_smile: :slight_smile:

1 Like