how to clean up space without effect the config

Hi

Centos 6.4 , docker 1.7.0 taking lot of space for volumes how to remove volumes without effect the config. please give me a tips

can you share output of the following commands?


docker inspect pmm-data | grep Destination
docker exec -it pmm-server du -m --max-depth 0 \
/opt/prometheus/data \
/opt/consul-data \
/var/lib/mysql \
/var/lib/grafana

Hi
Please find the below out put
xxxxx:/root # docker inspect pmm-data | grep Destination
xxxxx:/root # docker exec -it pmm-server du -m --max-depth 0 \

/opt/prometheus/data
/opt/consul-data
/var/lib/mysql
/var/lib/grafana
82443 /opt/prometheus/data
7 /opt/consul-data
1046 /var/lib/mysql
84 /var/lib/grafana

you can read more about how to control data retention in PMM FAQ - [url]Percona Monitoring and Management
you need to stop and remove the current pmm-server container and start new with METRICS_RETENTION option.
configuration should be saved if all 4 partitions located on pmm-data container.
unfortunately, i don’t see them in your the previous post. so can you share full output of “docker inspect pmm-data” command or just check it by yourself - all 4 partitions (/opt/prometheus/data, /opt/consul-data, /var/lib/mysql, /var/lib/ grafana ) should exists as volumes.

Hi
please find the below partitions details

xxxxx:/root # docker inspect pmm-data
[
{
“Id”: “6862de506cc7278440738442700e0f23389afbf45789e81fb f4faa070e0f2960”,
“Created”: “2017-04-19T09:47:38.268999705Z”,
“Path”: “/bin/true”,
“Args”: ,
“State”: {
“Running”: false,
“Paused”: false,
“Restarting”: false,
“OOMKilled”: false,
“Dead”: false,
“Pid”: 0,
“ExitCode”: 0,
“Error”: “”,
“StartedAt”: “0001-01-01T00:00:00Z”,
“FinishedAt”: “0001-01-01T00:00:00Z”
},
“Image”: “7c167ae7a6a7115cb7d186e32c056cf282d8b9e470f9d9ecc fe6ad12052f6364”,
“NetworkSettings”: {
“Bridge”: “”,
“EndpointID”: “”,
“Gateway”: “”,
“GlobalIPv6Address”: “”,
“GlobalIPv6PrefixLen”: 0,
“HairpinMode”: false,
“IPAddress”: “”,
“IPPrefixLen”: 0,
“IPv6Gateway”: “”,
“LinkLocalIPv6Address”: “”,
“LinkLocalIPv6PrefixLen”: 0,
“MacAddress”: “”,
“NetworkID”: “”,
“PortMapping”: null,
“Ports”: null,
“SandboxKey”: “”,
“SecondaryIPAddresses”: null,
“SecondaryIPv6Addresses”: null
},
“ResolvConfPath”: “”,
“HostnamePath”: “”,
“HostsPath”: “”,
“LogPath”: “”,
“Name”: “/pmm-data”,
“RestartCount”: 0,
“Driver”: “devicemapper”,
“ExecDriver”: “native-0.2”,
“MountLabel”: “”,
“ProcessLabel”: “”,
“Volumes”: {
“/opt/consul-data”: “/var/lib/docker/volumes/08c2c9bb0771276e914fb260b7f35064c6c3ba5f8d5f6be21c e6e721edbb6c37/_data”,
“/opt/prometheus/data”: “/var/lib/docker/volumes/b293eb813b20f5bf89908c1b65144541b7e0a0568c76428245 ff01d98343d02b/_data”,
“/var/lib/grafana”: “/var/lib/docker/volumes/e4762ce735069678e1f219aace49389f6b14bde32b16109c9f 6960be70e175e4/_data”,
“/var/lib/mysql”: “/var/lib/docker/volumes/b4ab762567a3f3b6628c4d923878f1712fc80cfe6289ecb0f1 973dd182bb9b6b/_data”
},
“VolumesRW”: {
“/opt/consul-data”: true,
“/opt/prometheus/data”: true,
“/var/lib/grafana”: true,
“/var/lib/mysql”: true
},
“AppArmorProfile”: “”,
“ExecIDs”: null,
“HostConfig”: {
“Binds”: null,
“ContainerIDFile”: “”,
“LxcConf”: ,
“Memory”: 0,
“MemorySwap”: 0,
“CpuShares”: 0,
“CpuPeriod”: 0,
“CpusetCpus”: “”,
“CpusetMems”: “”,
“CpuQuota”: 0,
“BlkioWeight”: 0,
“OomKillDisable”: false,
“Privileged”: false,
“PortBindings”: {},
“Links”: null,
“PublishAllPorts”: false,
“Dns”: null,
“DnsSearch”: null,
“ExtraHosts”: null,
“VolumesFrom”: null,
“Devices”: ,
“NetworkMode”: “bridge”,
“IpcMode”: “”,
“PidMode”: “”,
“UTSMode”: “”,
“CapAdd”: null,
“CapDrop”: null,
“RestartPolicy”: {
“Name”: “no”,
“MaximumRetryCount”: 0
},
“SecurityOpt”: null,
“ReadonlyRootfs”: false,
“Ulimits”: null,
“LogConfig”: {
“Type”: “json-file”,
“Config”: {}
},
“CgroupParent”: “”
},
“Config”: {
“Hostname”: “6862de506xx7”,
“Domainname”: “”,
“User”: “”,
“AttachStdin”: false,
“AttachStdout”: true,
“AttachStderr”: true,
“PortSpecs”: null,
“ExposedPorts”: {
“443/tcp”: {},
“80/tcp”: {}
},
“Tty”: false,
“OpenStdin”: false,
“StdinOnce”: false,
“Env”: [
“METRICS_MEMORY=25165824”,
“PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”
],
“Cmd”: [
“/bin/true”
],
“Image”: “percona/pmm-server:1.0.7”,
“Volumes”: {
“/opt/consul-data”: {},
“/opt/prometheus/data”: {},
“/var/lib/grafana”: {},
“/var/lib/mysql”: {}
},
“VolumeDriver”: “”,
“WorkingDir”: “/opt”,
“Entrypoint”: null,
“NetworkDisabled”: false,
“MacAddress”: “”,
“OnBuild”: null,
“Labels”: {}
}
}
]

xxxx:/root #
xxxx:/root # cd /var/lib/docker/
xxxxx:/var/lib/docker # ls
containers devicemapper graph init linkgraph.db repositories-devicemapper tmp trust volumes
xxxx:/var/lib/docker # cd volumes/
xxxx:/var/lib/docker/volumes # ls
02652f5e344f1e48b48ba09fdd2bf63d75b9bd7c0c9aebbec3 faf49473d9a230 b293eb813b20f5bf89908c1b65144541b7e0a0568c76428245 ff01d98343d02b
08c2c9bb0771276e914fb260b7f35064c6c3ba5f8d5f6be21c e6e721edbb6c37 b4ab762567a3f3b6628c4d923878f1712fc80cfe6289ecb0f1 973dd182bb9b6b
378b2a4a43c949304f4e270d5b1e7463db065b55f1df1fed49 1fa75ad9971c28 c1e0b1d2b6bb2b97ea133f2fae3e4f80cfbd41e76c168072e5 c4ea7b7e0528bc
39f96f21eaf777a4342a22ac3b82f9707fadb6426540750df6 b63534502227d6 e4762ce735069678e1f219aace49389f6b14bde32b16109c9f 6960be70e175e4
47bee98c87707e31a3c06ca5f656e0ba8ac0934e8a37e0c829 244d407c2e6959 e692c86a896063451cd3e2f66d6e2fa780920d1257283b910b d4d8d2182dfb2d
5d8c850058abe635aef2af1c491995fed18a97a523986d0dc8 c2468926b96edc e965ec43f198c4d0241e19b0eb25b1862c2511a8a81156ed24 7973e8b1c77524
xxxx:/var/lib/docker/volumes # du -sh *
30M 02652f5e344f1e48b48ba09fdd2bf63d75b9bd7c0c9aebbec3 faf49473d9a230
6.9M 08c2c9bb0771276e914fb260b7f35064c6c3ba5f8d5f6be21c e6e721edbb6c37
8.0K 378b2a4a43c949304f4e270d5b1e7463db065b55f1df1fed49 1fa75ad9971c28
68K 39f96f21eaf777a4342a22ac3b82f9707fadb6426540750df6 b63534502227d6
664K 47bee98c87707e31a3c06ca5f656e0ba8ac0934e8a37e0c829 244d407c2e6959
346M 5d8c850058abe635aef2af1c491995fed18a97a523986d0dc8 c2468926b96edc
77G b293eb813b20f5bf89908c1b65144541b7e0a0568c76428245 ff01d98343d02b
1.1G b4ab762567a3f3b6628c4d923878f1712fc80cfe6289ecb0f1 973dd182bb9b6b
8.0K c1e0b1d2b6bb2b97ea133f2fae3e4f80cfbd41e76c168072e5 c4ea7b7e0528bc
40M e4762ce735069678e1f219aace49389f6b14bde32b16109c9f 6960be70e175e4
316K e692c86a896063451cd3e2f66d6e2fa780920d1257283b910b d4d8d2182dfb2d
8.0K e965ec43f198c4d0241e19b0eb25b1862c2511a8a81156ed24 7973e8b1c77524
xxxxx:/var/lib/docker/volumes #

As I can see - all 4 directories located on pmm-data container (don’t remove it, all PMM Server data saved inside)
so it is safe to stop and remove the current pmm-server container and start new container with METRICS_RETENTION option (new container shouldn’t lose any data)
you can read more about data retention in PMM FAQ - [URL]Percona Monitoring and Management

I’ve got the same problem, but cannot change metric retention:

“docker run -d -p 80:80 -e METRICS_RETENTION=192h --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1.1.3
docker: Error response from daemon: Conflict. The container name “/pmm-server” is already in use by container d20f235b382e807bdd0530e792d4e7288ece8ecd3dad0378284ab7a95ec73440. You have to remove (or rename) that container to be able to reuse that name…”

  • i’ve removed container d20f235b382e807bdd0530e792d4e7288ece8ecd3dad0378284ab7a95ec73440, but the problem remain

Hi jackber,

I think pmm-server container still exists, please run the following commands before “docker run …” command.

docker stop pmm-server
docker rm pmm-server

after this procedure, pmm-server run to another docker volume, but after restart docker volumes remains full as before:

docker exec -ti pmm-server du -sh /opt/prometheus/data

13G /opt/prometheus/data

docker volume prune

WARNING! This will remove all volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0 B

Edit: for purge old data, is possible to delete “orphaned” directory in /var/lib/docker/volumes/0a643e7a71a4d46564916702MYVOLUME37b6c16e09f1fa88307/_data/ ?

It is not recommended to delete Prometheus database files.
in most cases, it is enough to change METRICS_RETENTION option and wait some time for retention.
can you check Prometheus directory size right now?
can you decrease METRICS_RETENTION to 24h and wait for 1 hour?
do you see any errors in ?

Hi
We unable get top 10 queries on PMM Query Analytics pls find the screen shot presently it showing only one call proc
We enable the performance_schema=ON and events_statements_history in my config and log_slow_rate_limit is 10
Databsae - mariadb10.0.29

Please provide any tips to solve the issue.

Hi there
Could you open up a new post for this? It’s off topic and likely that it will be overlooked if you leave it here.
Thanks.

hi, we flow the manual to upgrade our pmm from 1.13.0 to 1.15.0, we only change the METRICS_RETENTION from 2160h to 720h
after upgrade , the dashboard we made ourself can not use. see pic
and many client status are down

  • Connection: Client ← Server

SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD


linux:metrics 192.168.x.217_3306 10.1.2.217:42000 DOWN YES YES
mysql:metrics 192.168.x.217_3306 10.1.2.217:42002 OK YES YES
mysql:metrics 192.168.x.217_3307 10.1.2.217:42003 DOWN YES YES

the error messages are
2018/10/23 10:45:17 http: TLS handshake error from 10.1.x.217:30928: tls: first record does not look like a TLS handshake
2018/10/23 10:45:17 http: TLS handshake error from 10.1.x.217:30930: tls: first record does not look like a TLS handshake

and Space has not decreased and has grown
[root@node002143 fanboshi]# docker exec -it da7527c49789 /bin/bash
[root@da7527c49789 opt]# du -sh .
947G .
[root@da7527c49789 opt]# du -sh .
948G .

it’s is not the first time we upgrade pmm, last time is 1.4.0 , same issue, pmm can not use…

docker pull percona/pmm-server:1.5.0

Trying to pull repository docker.io/percona/pmm-server
1.5.0: Pulling from docker.io/percona/pmm-server
d9aaf4d82f24: Pull complete
54ab986d7dcd: Pull complete
36f5b3413c2f: Pull complete
279a7247971e: Pull complete
97441d251040: Pull complete
129df0c8e62e: Pull complete
d77f180519c3: Pull complete
a592d5939a35: Pull complete
e27020c41da0: Pull complete
Digest: sha256:63d8b8960b627dabb055a034250bdc5ea378fe1e891 69718c6836d8a98a63737
Status: Downloaded newer image for docker.io/percona/pmm-server:1.5.0

docker stop pmm-server

pmm-server

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3dfd23e87a5f db_monitor “./run.sh” 10 days ago Up 10 days 5555/tcp, 8793/tcp, 0.0.0.0:8088->8080/tcp blissful_noether
d4713948ec45 percona/pmm-server:latest “/opt/entrypoint.sh” 2 months ago Exited (137) 9 seconds ago pmm-server
46f83a2d7c9e percona/pmm-server:latest “/bin/true” 2 months ago Created pmm-data

docker rename pmm-server pmm-server-backup

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3dfd23e87a5f db_monitor “./run.sh” 10 days ago Up 10 days 5555/tcp, 8793/tcp, 0.0.0.0:8088->8080/tcp blissful_noether
d4713948ec45 percona/pmm-server:latest “/opt/entrypoint.sh” 2 months ago Exited (137) 22 seconds ago pmm-server-backup
46f83a2d7c9e percona/pmm-server:latest “/bin/true” 2 months ago Created pmm-data

docker images

REPOSITORY TAG IMAGE ID CREATED SIZE
db_monitor latest d11a621223e7 10 days ago 1.05 GB
docker.io/puckel/docker-airflow latest 333ad7f5b3aa 7 weeks ago 1.05 GB
docker.io/percona/pmm-server latest c32da3dfb2ae 2 months ago 971 MB
docker.io/percona/pmm-server 1.5.0 51bfaa37cb1e 11 months ago 1.33 GB

docker run -d \

-p 8080:80
–volumes-from pmm-data
–name pmm-server
-e SERVER_USER=olduser
-e SERVER_PASSWORD=olduserpass
-e METRICS_RETENTION=720h \ –change from 2160h to 720h
-e METRICS_RESOLUTION=5s
-e DISABLE_TELEMETRY=true
–restart always
percona/pmm-server:1.5.0
da7527c49789325c1250e73856d4614b59c96d0fb6e4b44179 4e9ae019aeb408

docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da7527c49789 percona/pmm-server:1.5.0 “/opt/entrypoint.sh” 4 seconds ago Up 3 seconds 443/tcp, 0.0.0.0:8080->80/tcp pmm-server
3dfd23e87a5f db_monitor “./run.sh” 10 days ago Up 10 days 5555/tcp, 8793/tcp, 0.0.0.0:8088->8080/tcp blissful_noether
d4713948ec45 percona/pmm-server:latest “/opt/entrypoint.sh” 2 months ago Exited (137) 5 minutes ago pmm-server-backup
46f83a2d7c9e percona/pmm-server:latest “/bin/true” 2 months ago Created pmm-data

–note added by Percona Community Manager, going forward using pmm-server:n rather than pmm-server:latest is best practice, our advice changed since this post was written. In this case pmm-server:1 would have been appropriate for these code examples.

we rollback to 1.13.0 , everthing goes ok. clients STATUS are OK , and the dashboard we create works right.
is it right that if we only want to change the METRICS_RETENTION we can manualy modify /etc/supervisord.d/pmm.ini /etc/sysconfig/prometheus /etc/sysconfig/prometheus1

if we set the --storage.tsdb.retention=2160h, and run pmm three month, then we change --storage.tsdb.retention to 720 , will the first and second month data be delete?
@Mykola

hi Mykola , after a few hours , the old data had been cleaned

[root@d4713948ec45 opt]# date
Tue Oct 23 12:22:48 CST 2018
[root@d4713948ec45 opt]# du -sh .
954G .

[root@d4713948ec45 opt]# du -sh .
508G .
Tue Oct 23 15:06:09 CST 2018

Hi Fan

Could you take a look in the file /var/log/prometheus.log and /var/log/prometheus1.log for any notable events - particularly whether an instance is restarting.
What state do the endpoints say they are in? Are the metric series empty also for those instances that are marked as down? Look at /prometheus/targets .

Could you share the JSON for the dashboard with me please, and I will review?

We should move this conversation to a new thread