Not the answer you need?
Register and ask your own question!

how to clean up space without effect the config

Vishwanath KristappaVishwanath Kristappa ContributorCurrent User Role Contributor
Hi

Centos 6.4 , docker 1.7.0 taking lot of space for volumes how to remove volumes without effect the config. please give me a tips

Comments

  • MykolaMykola Percona Percona Staff Role
    can you share output of the following commands?
    docker inspect pmm-data | grep Destination
    docker exec -it pmm-server du -m --max-depth 0 \
        /opt/prometheus/data \
        /opt/consul-data \
        /var/lib/mysql \
        /var/lib/grafana
    
  • Vishwanath KristappaVishwanath Kristappa Contributor Current User Role Contributor
    Hi
    Please find the below out put
    xxxxx:/root # docker inspect pmm-data | grep Destination
    xxxxx:/root # docker exec -it pmm-server du -m --max-depth 0 \
    > /opt/prometheus/data \
    > /opt/consul-data \
    > /var/lib/mysql \
    > /var/lib/grafana
    82443 /opt/prometheus/data
    7 /opt/consul-data
    1046 /var/lib/mysql
    84 /var/lib/grafana
  • MykolaMykola Percona Percona Staff Role
    you can read more about how to control data retention in PMM FAQ - https://www.percona.com/doc/percona-monitoring-and-management/faq.html#how-to-control-data-retention-for-prometheus
    you need to stop and remove the current pmm-server container and start new with METRICS_RETENTION option.
    configuration should be saved if all 4 partitions located on pmm-data container.
    unfortunately, i don't see them in your the previous post. so can you share full output of "docker inspect pmm-data" command or just check it by yourself - all 4 partitions (/opt/prometheus/data, /opt/consul-data, /var/lib/mysql, /var/lib/ grafana ) should exists as volumes.
  • Vishwanath KristappaVishwanath Kristappa Contributor Current User Role Contributor
    Hi
    please find the below partitions details

    xxxxx:/root # docker inspect pmm-data
    [
    {
    "Id": "6862de506cc7278440738442700e0f23389afbf45789e81fb f4faa070e0f2960",
    "Created": "2017-04-19T09:47:38.268999705Z",
    "Path": "/bin/true",
    "Args": [],
    "State": {
    "Running": false,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 0,
    "ExitCode": 0,
    "Error": "",
    "StartedAt": "0001-01-01T00:00:00Z",
    "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "7c167ae7a6a7115cb7d186e32c056cf282d8b9e470f9d9ecc fe6ad12052f6364",
    "NetworkSettings": {
    "Bridge": "",
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "HairpinMode": false,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "MacAddress": "",
    "NetworkID": "",
    "PortMapping": null,
    "Ports": null,
    "SandboxKey": "",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null
    },
    "ResolvConfPath": "",
    "HostnamePath": "",
    "HostsPath": "",
    "LogPath": "",
    "Name": "/pmm-data",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "",
    "ProcessLabel": "",
    "Volumes": {
    "/opt/consul-data": "/var/lib/docker/volumes/08c2c9bb0771276e914fb260b7f35064c6c3ba5f8d5f6be21c e6e721edbb6c37/_data",
    "/opt/prometheus/data": "/var/lib/docker/volumes/b293eb813b20f5bf89908c1b65144541b7e0a0568c76428245 ff01d98343d02b/_data",
    "/var/lib/grafana": "/var/lib/docker/volumes/e4762ce735069678e1f219aace49389f6b14bde32b16109c9f 6960be70e175e4/_data",
    "/var/lib/mysql": "/var/lib/docker/volumes/b4ab762567a3f3b6628c4d923878f1712fc80cfe6289ecb0f1 973dd182bb9b6b/_data"
    },
    "VolumesRW": {
    "/opt/consul-data": true,
    "/opt/prometheus/data": true,
    "/var/lib/grafana": true,
    "/var/lib/mysql": true
    },
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LxcConf": [],
    "Memory": 0,
    "MemorySwap": 0,
    "CpuShares": 0,
    "CpuPeriod": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "CpuQuota": 0,
    "BlkioWeight": 0,
    "OomKillDisable": false,
    "Privileged": false,
    "PortBindings": {},
    "Links": null,
    "PublishAllPorts": false,
    "Dns": null,
    "DnsSearch": null,
    "ExtraHosts": null,
    "VolumesFrom": null,
    "Devices": [],
    "NetworkMode": "bridge",
    "IpcMode": "",
    "PidMode": "",
    "UTSMode": "",
    "CapAdd": null,
    "CapDrop": null,
    "RestartPolicy": {
    "Name": "no",
    "MaximumRetryCount": 0
    },
    "SecurityOpt": null,
    "ReadonlyRootfs": false,
    "Ulimits": null,
    "LogConfig": {
    "Type": "json-file",
    "Config": {}
    },
    "CgroupParent": ""
    },
    "Config": {
    "Hostname": "6862de506xx7",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": true,
    "AttachStderr": true,
    "PortSpecs": null,
    "ExposedPorts": {
    "443/tcp": {},
    "80/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
    "METRICS_MEMORY=25165824",
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
    "/bin/true"
    ],
    "Image": "percona/pmm-server:1.0.7",
    "Volumes": {
    "/opt/consul-data": {},
    "/opt/prometheus/data": {},
    "/var/lib/grafana": {},
    "/var/lib/mysql": {}
    },
    "VolumeDriver": "",
    "WorkingDir": "/opt",
    "Entrypoint": null,
    "NetworkDisabled": false,
    "MacAddress": "",
    "OnBuild": null,
    "Labels": {}
    }
    }
    ]

    xxxx:/root #
    xxxx:/root # cd /var/lib/docker/
    xxxxx:/var/lib/docker # ls
    containers devicemapper graph init linkgraph.db repositories-devicemapper tmp trust volumes
    xxxx:/var/lib/docker # cd volumes/
    xxxx:/var/lib/docker/volumes # ls
    02652f5e344f1e48b48ba09fdd2bf63d75b9bd7c0c9aebbec3 faf49473d9a230 b293eb813b20f5bf89908c1b65144541b7e0a0568c76428245 ff01d98343d02b
    08c2c9bb0771276e914fb260b7f35064c6c3ba5f8d5f6be21c e6e721edbb6c37 b4ab762567a3f3b6628c4d923878f1712fc80cfe6289ecb0f1 973dd182bb9b6b
    378b2a4a43c949304f4e270d5b1e7463db065b55f1df1fed49 1fa75ad9971c28 c1e0b1d2b6bb2b97ea133f2fae3e4f80cfbd41e76c168072e5 c4ea7b7e0528bc
    39f96f21eaf777a4342a22ac3b82f9707fadb6426540750df6 b63534502227d6 e4762ce735069678e1f219aace49389f6b14bde32b16109c9f 6960be70e175e4
    47bee98c87707e31a3c06ca5f656e0ba8ac0934e8a37e0c829 244d407c2e6959 e692c86a896063451cd3e2f66d6e2fa780920d1257283b910b d4d8d2182dfb2d
    5d8c850058abe635aef2af1c491995fed18a97a523986d0dc8 c2468926b96edc e965ec43f198c4d0241e19b0eb25b1862c2511a8a81156ed24 7973e8b1c77524
    xxxx:/var/lib/docker/volumes # du -sh *
    30M 02652f5e344f1e48b48ba09fdd2bf63d75b9bd7c0c9aebbec3 faf49473d9a230
    6.9M 08c2c9bb0771276e914fb260b7f35064c6c3ba5f8d5f6be21c e6e721edbb6c37
    8.0K 378b2a4a43c949304f4e270d5b1e7463db065b55f1df1fed49 1fa75ad9971c28
    68K 39f96f21eaf777a4342a22ac3b82f9707fadb6426540750df6 b63534502227d6
    664K 47bee98c87707e31a3c06ca5f656e0ba8ac0934e8a37e0c829 244d407c2e6959
    346M 5d8c850058abe635aef2af1c491995fed18a97a523986d0dc8 c2468926b96edc
    77G b293eb813b20f5bf89908c1b65144541b7e0a0568c76428245 ff01d98343d02b
    1.1G b4ab762567a3f3b6628c4d923878f1712fc80cfe6289ecb0f1 973dd182bb9b6b
    8.0K c1e0b1d2b6bb2b97ea133f2fae3e4f80cfbd41e76c168072e5 c4ea7b7e0528bc
    40M e4762ce735069678e1f219aace49389f6b14bde32b16109c9f 6960be70e175e4
    316K e692c86a896063451cd3e2f66d6e2fa780920d1257283b910b d4d8d2182dfb2d
    8.0K e965ec43f198c4d0241e19b0eb25b1862c2511a8a81156ed24 7973e8b1c77524
    xxxxx:/var/lib/docker/volumes #
  • MykolaMykola Percona Percona Staff Role
    As I can see - all 4 directories located on pmm-data container (don't remove it, all PMM Server data saved inside)
    so it is safe to stop and remove the current pmm-server container and start new container with METRICS_RETENTION option (new container shouldn't lose any data)
    you can read more about data retention in PMM FAQ - https://www.percona.com/doc/percona-monitoring-and-management/faq.html#how-to-control-data-retention-for-prometheus
  • jackberjackber Entrant Inactive User Role Novice
    I've got the same problem, but cannot change metric retention:

    "docker run -d -p 80:80 -e METRICS_RETENTION=192h --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:1.1.3
    docker: Error response from daemon: Conflict. The container name "/pmm-server" is already in use by container d20f235b382e807bdd0530e792d4e7288ece8ecd3dad0378284ab7a95ec73440. You have to remove (or rename) that container to be able to reuse that name.."


    - i've removed container d20f235b382e807bdd0530e792d4e7288ece8ecd3dad0378284ab7a95ec73440, but the problem remain
  • MykolaMykola Percona Percona Staff Role
    Hi jackber,

    I think pmm-server container still exists, please run the following commands before "docker run ..." command.
    docker stop pmm-server
    docker rm pmm-server
    
  • jackberjackber Entrant Inactive User Role Novice
    after this procedure, pmm-server run to another docker volume, but after restart docker volumes remains full as before:

    # docker exec -ti pmm-server du -sh /opt/prometheus/data
    13G /opt/prometheus/data

    # docker volume prune
    WARNING! This will remove all volumes not used by at least one container.
    Are you sure you want to continue? [y/N] y
    Total reclaimed space: 0 B
  • jackberjackber Entrant Inactive User Role Novice
    Edit: for purge old data, is possible to delete "orphaned" directory in /var/lib/docker/volumes/0a643e7a71a4d46564916702MYVOLUME37b6c16e09f1fa88307/_data/ ?
  • MykolaMykola Percona Percona Staff Role
    It is not recommended to delete Prometheus database files.
    in most cases, it is enough to change METRICS_RETENTION option and wait some time for retention.
    can you check Prometheus directory size right now?
    can you decrease METRICS_RETENTION to 24h and wait for 1 hour?
    do you see any errors in ?
  • Vishwanath KristappaVishwanath Kristappa Contributor Current User Role Contributor
    Hi
    We unable get top 10 queries on PMM Query Analytics pls find the screen shot presently it showing only one call proc
    We enable the performance_schema=ON and events_statements_history in my config and log_slow_rate_limit is 10
    Databsae - mariadb10.0.29

    Please provide any tips to solve the issue.
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Patron
    Hi there
    Could you open up a new post for this? It's off topic and likely that it will be overlooked if you leave it here.
    Thanks.
  • FanFan Contributor Current User Role Patron
    hi, we flow the manual to upgrade our pmm from 1.13.0 to 1.15.0, we only change the METRICS_RETENTION from 2160h to 720h
    after upgrade , the dashboard we made ourself can not use. see pic
    and many client status are down
    * Connection: Client <-- Server





    SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD





    linux:metrics 192.168.x.217_3306 10.1.2.217:42000 DOWN YES YES
    mysql:metrics 192.168.x.217_3306 10.1.2.217:42002 OK YES YES
    mysql:metrics 192.168.x.217_3307 10.1.2.217:42003 DOWN YES YES

    the error messages are
    2018/10/23 10:45:17 http: TLS handshake error from 10.1.x.217:30928: tls: first record does not look like a TLS handshake
    2018/10/23 10:45:17 http: TLS handshake error from 10.1.x.217:30930: tls: first record does not look like a TLS handshake


    and Space has not decreased and has grown
    [[email protected] fanboshi]# docker exec -it da7527c49789 /bin/bash
    [[email protected] opt]# du -sh .
    947G .
    [[email protected] opt]# du -sh .
    948G .


    it's is not the first time we upgrade pmm, last time is 1.4.0 , same issue, pmm can not use.....
  • FanFan Contributor Current User Role Patron
    # docker pull percona/pmm-server:1.5.0
    Trying to pull repository docker.io/percona/pmm-server ...
    1.5.0: Pulling from docker.io/percona/pmm-server
    d9aaf4d82f24: Pull complete
    54ab986d7dcd: Pull complete
    36f5b3413c2f: Pull complete
    279a7247971e: Pull complete
    97441d251040: Pull complete
    129df0c8e62e: Pull complete
    d77f180519c3: Pull complete
    a592d5939a35: Pull complete
    e27020c41da0: Pull complete
    Digest: sha256:63d8b8960b627dabb055a034250bdc5ea378fe1e891 69718c6836d8a98a63737
    Status: Downloaded newer image for docker.io/percona/pmm-server:1.5.0

    # docker stop pmm-server
    pmm-server
    # docker ps -a
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    3dfd23e87a5f db_monitor "./run.sh" 10 days ago Up 10 days 5555/tcp, 8793/tcp, 0.0.0.0:8088->8080/tcp blissful_noether
    d4713948ec45 percona/pmm-server:latest "/opt/entrypoint.sh" 2 months ago Exited (137) 9 seconds ago pmm-server
    46f83a2d7c9e percona/pmm-server:latest "/bin/true" 2 months ago Created pmm-data
    # docker rename pmm-server pmm-server-backup
    # docker ps -a
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    3dfd23e87a5f db_monitor "./run.sh" 10 days ago Up 10 days 5555/tcp, 8793/tcp, 0.0.0.0:8088->8080/tcp blissful_noether
    d4713948ec45 percona/pmm-server:latest "/opt/entrypoint.sh" 2 months ago Exited (137) 22 seconds ago pmm-server-backup
    46f83a2d7c9e percona/pmm-server:latest "/bin/true" 2 months ago Created pmm-data
    # docker images
    REPOSITORY TAG IMAGE ID CREATED SIZE
    db_monitor latest d11a621223e7 10 days ago 1.05 GB
    docker.io/puckel/docker-airflow latest 333ad7f5b3aa 7 weeks ago 1.05 GB
    docker.io/percona/pmm-server latest c32da3dfb2ae 2 months ago 971 MB
    docker.io/percona/pmm-server 1.5.0 51bfaa37cb1e 11 months ago 1.33 GB
    # docker run -d \
    > -p 8080:80 \
    > --volumes-from pmm-data \
    > --name pmm-server \
    > -e SERVER_USER=olduser \
    > -e SERVER_PASSWORD=olduserpass \
    > -e METRICS_RETENTION=720h \ --change from 2160h to 720h
    > -e METRICS_RESOLUTION=5s \
    > -e DISABLE_TELEMETRY=true \
    > --restart always \
    > percona/pmm-server:1.5.0
    da7527c49789325c1250e73856d4614b59c96d0fb6e4b44179 4e9ae019aeb408
    # docker ps -a
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    da7527c49789 percona/pmm-server:1.5.0 "/opt/entrypoint.sh" 4 seconds ago Up 3 seconds 443/tcp, 0.0.0.0:8080->80/tcp pmm-server
    3dfd23e87a5f db_monitor "./run.sh" 10 days ago Up 10 days 5555/tcp, 8793/tcp, 0.0.0.0:8088->8080/tcp blissful_noether
    d4713948ec45 percona/pmm-server:latest "/opt/entrypoint.sh" 2 months ago Exited (137) 5 minutes ago pmm-server-backup
    46f83a2d7c9e percona/pmm-server:latest "/bin/true" 2 months ago Created pmm-data


    --note added by Percona Community Manager, going forward using pmm-server:n rather than pmm-server:latest is best practice, our advice changed since this post was written. In this case pmm-server:1 would have been appropriate for these code examples.
  • FanFan Contributor Current User Role Patron
    we rollback to 1.13.0 , everthing goes ok. clients STATUS are OK , and the dashboard we create works right.
    is it right that if we only want to change the METRICS_RETENTION we can manualy modify /etc/supervisord.d/pmm.ini /etc/sysconfig/prometheus /etc/sysconfig/prometheus1

    if we set the --storage.tsdb.retention=2160h, and run pmm three month, then we change --storage.tsdb.retention to 720 , will the first and second month data be delete?
    @Mykola
  • FanFan Contributor Current User Role Patron
    Fan wrote: »
    we rollback to 1.13.0 , everthing goes ok. clients STATUS are OK , and the dashboard we create works right.
    is it right that if we only want to change the METRICS_RETENTION we can manualy modify /etc/supervisord.d/pmm.ini /etc/sysconfig/prometheus /etc/sysconfig/prometheus1

    if we set the --storage.tsdb.retention=2160h, and run pmm three month, then we change --storage.tsdb.retention to 720 , will the first and second month data be delete?
    @Mykola

    hi Mykola , after a few hours , the old data had been cleaned

    [[email protected] opt]# date
    Tue Oct 23 12:22:48 CST 2018
    [[email protected] opt]# du -sh .
    954G .

    [[email protected] opt]# du -sh .
    508G .
    Tue Oct 23 15:06:09 CST 2018
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi Fan

    Could you take a look in the file /var/log/prometheus.log and /var/log/prometheus1.log for any notable events - particularly whether an instance is restarting.
    What state do the endpoints say they are in? Are the metric series empty also for those instances that are marked as down? Look at <pmmserver>/prometheus/targets .

    Could you share the JSON for the dashboard with me please, and I will review?

    We should move this conversation to a new thread
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.