# Nginx logs filling up volume **URL:** \ **Category:** Percona Monitoring and Management (PMM)\ **Tags:** pmm\ **Created:** [August 11, 2021, 3:04pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740 "2021-08-11T15:04:00Z")\ **Posts on this page:** 14\ **Page:** 2
**Author:** ![nikita.beletskii](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nikita.beletskii/32/3784_2.png) [@nikita.beletskii](https://forums.percona.com/u/nikita.beletskii)\ **Post date:** [August 25, 2021, 8:31am UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/21 "2021-08-25T08:31:40Z")
> [@dbray\_sd](#): > > Now I guess I’ll sit back and wait the ## hours to see if the logs rotate? Or, is there anything else you want me to check? Yep! It should be fine now. We rotate nginx logs after 50M. ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [August 25, 2021, 7:48pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/22 "2021-08-25T19:48:35Z")
Apologies, but something still doesn’t seem right. Logs haven’t rotated yet, even after restarting docker/containerd. I also checked the above pmm-admin --version command, and I still see: ```auto ProjectName: pmm-admin Version: 2.14.0 PMMVersion: 2.14.0 Timestamp: 2021-01-28 12:36:42 (UTC) FullCommit: cdf593c6774c3c239aaf22fff27e9e7aa1364a60 ``` I was expecting that to now read 2.20.0, same as I see in the web app. Right? ---
**Author:** ![nikita.beletskii](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nikita.beletskii/32/3784_2.png) [@nikita.beletskii](https://forums.percona.com/u/nikita.beletskii)\ **Post date:** [August 26, 2021, 10:40pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/23 "2021-08-26T22:40:06Z")
Yes, it should be 2.20… To be honest it looks very suspicious because we run updates on each pmm-update run… Could you pls attache pmm-update-perform.log again? ```auto docker cp pmm-server:/srv/logs/pmm-update-perform.log . ``` ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [August 27, 2021, 10:33am UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/24 "2021-08-27T10:33:47Z")
Attached is the updated log: [pmm-update-perform.log](https://forums.percona.com/uploads/short-url/pFvypLiwA6hxSqSDNdRILZ32IBd.log) (18.9 KB) ---
**Author:** ![nikita.beletskii](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/nikita.beletskii/32/3784_2.png) [@nikita.beletskii](https://forums.percona.com/u/nikita.beletskii)\ **Post date:** [September 2, 2021, 1:58pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/25 "2021-09-02T13:58:40Z")
OK, I tried to reproduce this issue and couldn’t ☹ My suggestion: 1. Let’s make backup again: [Percona Monitoring and Management](https://www.percona.com/doc/percona-monitoring-and-management/1.x/deploy/server/docker.backing-up.html) 2. Upgrade system by hand: ```auto yum upgrade ``` 1. and run upgrade PMM: ```auto /usr/sbin/pmm-update -perform -playbook=/usr/share/pmm-update/ansible/playbook/tasks/update.yml ``` ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [September 2, 2021, 5:28pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/26 "2021-09-02T17:28:31Z")
OK, clearly I’m doing something wrong…and my apologies for not knowing exactly what it is. I’m not the best when it comes to Docker. Here is what I did: ```auto $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 12fd55c0de21 7599815450fd "/opt/entrypoint.sh" 6 months ago Up 4 minutes (healthy) 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp pmm-server sudo docker run -it 7599815450fd /bin/bash [root@9baa6e6a11e3 opt]# [root@9baa6e6a11e3 opt]# rpm -qa | grep pmm pmm-managed-2.14.0-17.2101281011.787c431.el7.x86_64 pmm2-client-2.14.0-6.el7.x86_64 pmm-server-2.14.0-21.2101281241.cb737e6.el7.noarch pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch [root@9baa6e6a11e3 opt]# yum -y upgrade ...... [root@9baa6e6a11e3 opt]# rpm -qa | grep pmm pmm-managed-2.21.0-17.2108101848.05b1420.el7.x86_64 pmm-server-2.21.0-24.2108162024.afbc5c6.el7.noarch pmm2-client-2.21.0-6.el7.x86_64 pmm-update-2.21.0-61.2108261434.5ec7f7f.el7.noarch [root@9baa6e6a11e3 opt]# exit $ sudo docker run -it 7599815450fd /bin/bash [root@c0b7d8549a80 opt]# rpm -qa | grep pmm pmm-managed-2.14.0-17.2101281011.787c431.el7.x86_64 pmm2-client-2.14.0-6.el7.x86_64 pmm-server-2.14.0-21.2101281241.cb737e6.el7.noarch pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch ``` What am I doing wrong? ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [September 2, 2021, 8:04pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/27 "2021-09-02T20:04:34Z")
Hi, It feels like the “rollback” to v2.14.0 is due to a file system caveat, it’s just not very clear where it is coming from… Quite intriguing, let’s see if we can get more info to help investigate: 1/ Have you mounted any volumes to your docker container? It would help to see how you launched the PMM Server docker container. 2/ Which file system is your docker using (docker info)? 3/ Let us know about any particular configuration settings you may have applied to PMM Server Alex ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [September 3, 2021, 11:42am UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/28 "2021-09-03T11:42:26Z")
Hey Alex, I originally followed the simple steps documented here: [Installing Percona Monitoring and Management (PMM) Software - Percona](https://www.percona.com/software/pmm/quickstart#) This was all done on a CentOS 8 server. ```auto sudo dnf -y install docker-ce docker-ce-cli containerd.io sudo systemctl enable --now docker containerd sudo docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true sudo docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2 ``` Nightly, I backup everything (to a NFS share) with: `sudo docker cp pmm-data:/srv /backups/latest/` Other than that, I’ve made the necessary tweaks within the web app itself, but haven’t done anything else I can think of specifically to the OS nor docker itself. Here’s the details to your questions: > 1/ Have you mounted any volumes to your docker container? It would help to see how you launched the PMM Server docker container. No, not that I can recall. I checked bash\_history and everything, and I see no references to mount or anything similar. > 2/ Which file system is your docker using (docker info)? Using the default xfs filesystem CentOS 8 installs by default, and located in the default /var/lib/docker directory. > 3/ Let us know about any particular configuration settings you may have applied to PMM Server Nothing comes to mind. All the customization I’ve done has all been within the web app itself. Thanks you, I appreciate your help with this. I have no problem removing everything and starting over. I would just need to know exactly which files to restore, so that my customizations within the web app come back. ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [September 3, 2021, 8:32pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/29 "2021-09-03T20:32:08Z")
Hi, Thanks a lot for providing such extensive information. Your provisioning commands do look good and the “rollback” failure has most certainly nothing to do with how you bootstrap PMM Server. Now the only thing we can try is to tear down the container and try to spin it back up again. Your data is safe since it’s persisted to the `pmm-data` container. In your case the commands would be: To tear down: ```auto sudo docker stop pmm-server && sudo docker rm pmm-server ``` To spin up (same as before): ```auto sudo docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2 ``` It may well be that docker engine is misbehaving and that could potentially help put it back on track. You may want to reboot the machine before spinning up PMM again. Can’t wait to hear what it does 🙂 ) Alex ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [September 7, 2021, 12:36pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/30 "2021-09-07T12:36:14Z")
Sorry for the delays, it was a holiday for us yesterday. > [@ademidoff](#): > > Can’t wait to hear what it does Careful what you ask for 🙂 This is what it did. All the commands worked without error. Only slight change I had to make was the ports, so that iptables would NAT properly: `sudo docker run --detach -p 8080:80 -p 8443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2` I have to use 8080 and 8443, but no issue there. When I brought the web app up, it did initially say: **Current version: 2.15.0 February 28, 2020** So I went ahead and performed the in place upgrade, using the button within the GUI to upgrade, and I’m attaching the latest [pmm-update-perform.log](https://forums.percona.com/uploads/short-url/gBaPlsJC8cn8qN0FInEbm9qqfkE.log) (23.5 KB) However, the issue is not resolved, and now I have a little more confusion. It appears I now have two separate instances available and/or running: ```auto $ sudo docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5c3fcb9fc3f4 percona/pmm-server:2 "/bin/bash" 8 minutes ago Exited (130) 3 minutes ago charming_neumann 140fa15cdc82 7599815450fd "/bin/bash" 11 minutes ago Exited (0) 10 minutes ago quirky_noyce 1fd26d7c5d00 percona/pmm-server:2 "/opt/entrypoint.sh" 16 minutes ago Up 16 minutes (healthy) 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp pmm-server c0b7d8549a80 7599815450fd "/bin/bash" 4 days ago Exited (0) 4 days ago cool_liskov 9baa6e6a11e3 7599815450fd "/bin/bash" 4 days ago Exited (130) 4 days ago great_curie f6a56e4bd24f 7599815450fd "/bin/bash" 4 days ago Exited (0) 4 days ago infallible_brahmagupta 4b1c5a1f35cd 7599815450fd "/opt/entrypoint.sh" 4 days ago Exited (0) 4 days ago wonderful_austin 90de0a423713 7599815450fd "/bin/bash" 4 days ago Exited (0) 4 days ago beautiful_goodall 29d085c1c8fa 7599815450fd "/bin/bash" 4 days ago Exited (130) 4 days ago inspiring_germain 29cbf33a7b18 7599815450fd "/bin/bash" 11 days ago Exited (0) 11 days ago ecstatic_tesla f7bb17da2f4f 7599815450fd "/bin/bash" 11 days ago Exited (130) 11 days ago silly_swartz fa8e60a0a1d6 7599815450fd "/bin/bash" 12 days ago Exited (0) 11 days ago keen_ride 0d58ba4410e7 7599815450fd "/bin/bash" 13 days ago Exited (42) 13 days ago nice_booth a7d924df186e 7599815450fd "/bin/bash" 13 days ago Exited (130) 13 days ago pensive_engelbart 18fe481014f1 7599815450fd "/bin/bash" 2 weeks ago Exited (1) 2 weeks ago bold_noyce 02285cadb6a5 7599815450fd "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago boring_heyrovsky 8b97d47777f1 7599815450fd "/bin/bash" 2 weeks ago Exited (130) 2 weeks ago mystifying_chandrasekhar 4a082189086f 7599815450fd "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago affectionate_hermann f2c7ffed801b 7599815450fd "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago busy_goldwasser 7c0838f7b2f2 7599815450fd "/bin/true" 6 months ago Created pmm-data ``` So now if I check the old command I used to run, I see: ```auto $ sudo docker run -it 7599815450fd /bin/bash [root@c7d9295f9a9e opt]# rpm -qa | grep pmm pmm-managed-2.14.0-17.2101281011.787c431.el7.x86_64 pmm2-client-2.14.0-6.el7.x86_64 pmm-server-2.14.0-21.2101281241.cb737e6.el7.noarch pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch ``` However, if I check the new container that is now showing up, I see: ```auto $ sudo docker run -it percona/pmm-server:2 /bin/bash [root@ce2446d9cd98 opt]# rpm -qa | grep pmm pmm-server-2.15.0-21.2103010711.fdd2fb8.el7.noarch pmm-update-2.15.0-52.2103010712.c64c622.el7.noarch pmm-managed-2.15.0-17.2102251420.ec05154.el7.x86_64 pmm2-client-2.15.0-6.el7.x86_64 ``` However, the GUI does show: **Current version: 2.21.0 (August 25, 2021)** All very confusing. ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [September 7, 2021, 10:36pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/31 "2021-09-07T22:36:56Z")
Ok, now I’m starting to understand what this all means. Let’s break things down one by one. **1. What PMM Server version you are running** Judging by the log results you attached, it looks like you successfully migrated your PMM Server to the latest version which is 2.21.0 as of now. This is also confirmed by the UI since it show the current version number on the Upgrade panel. **2. How to check the rpm package versions inside the container** When you run this following command `sudo docker run -it percona/pmm-server:2 /bin/bash` to check for installed rpm package versions, it creates yet another PMM Server, a brand new one, off of the image version 2.15.0. When you get inside the docker conatiner, you shouldn’t be surprised to see old packages (2.15.0 in your case) since this is effectively the version of one of your docker images. This is expected, since those rpm packages come bundled with the docker image and their versions always match the image version. Now, to see which package versions are installed to your running PMM Server (its id is `1fd26d7c5d00 `), you could run the following command rather: ```auto % sudo docker exec -t pmm-server sh -c "rpm -qa | grep pmm" pmm-managed-2.22.0-17.2108310927.52bf11b.el7.x86_64 pmm2-client-2.22.0-6.el7.x86_64 pmm-server-2.22.0-24.2109010926.883130e.el7.noarch pmm-update-2.22.0-60.2108240517.4be8fda.el7.noarch ``` And I’m pretty confident your rpm versions should be `2.21.0` this time, otherwise we can call it `magic` 🙂 **3. Cleanup** Since those containers which are no longer running occupy much space (ca. 2 GB each) on your machine you may want to consider removing them with: ```auto sudo docker rm charming_neumann cool_liskov (... and more) ``` Hope this helps, otherwise please let us know. Alex ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [September 8, 2021, 11:23am UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/32 "2021-09-08T11:23:51Z")
Alex, I really appreciate your time and effort explaining all of this. It makes so much more sense now, and yes…finally…I see: ```auto $ sudo docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1fd26d7c5d00 percona/pmm-server:2 "/opt/entrypoint.sh" 24 hours ago Up 3 minutes (healthy) 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp pmm-server 7c0838f7b2f2 7599815450fd "/bin/true" 6 months ago Created pmm-data $ sudo docker exec -t pmm-server sh -c "rpm -qa | grep pmm" pmm-update-2.21.0-61.2108261434.5ec7f7f.el7.noarch pmm-server-2.21.0-24.2108162024.afbc5c6.el7.noarch pmm2-client-2.21.0-6.el7.x86_64 pmm-managed-2.21.0-17.2108101848.05b1420.el7.x86_64 ``` So now, with _alllllllll_ of that out of the way, I am still seeing a rather large log file: ```auto $ sudo ls -lah /var/lib/docker/volumes/37d310a60891d69b99d7f8ab76daa059c7211fdbd5e29374836d833a451178e2/_data/logs/ | grep nginx -rw-r--r-- 1 root root 1.4G Sep 7 11:29 nginx.access.log -rw-r--r-- 1 root root 537K Sep 7 11:29 nginx.error.log -rw-r--r-- 1 root root 27M Sep 8 11:15 nginx.log -rw-r--r-- 1 root root 51M Sep 7 21:43 nginx.log.1 -rw-r--r-- 1 root root 51M Sep 6 20:24 nginx.log.2 -rw-r--r-- 1 root root 127K Sep 7 11:31 nginx.startup.log ``` Should I assume the nginx.access.log should’ve already rotated. Or will that rotate tonight/tomorrow? Again, thank you so much, I really appreciate the help! ---
**Author:** ![ademidoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ademidoff/32/14234_2.png) [@ademidoff](https://forums.percona.com/u/ademidoff)\ **Post date:** [September 8, 2021, 12:25pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/33 "2021-09-08T12:25:34Z")
Wow, so excited to see we were able to help sort it out! To your question. After we refactored the log rotation, there should only be one log file per process, for example `nginx`’s logs are now being saved to `/srv/logs/nginx.log` only. Therefore, the server will no longer touch the other log files, namely `nginx.access.log`, `nginx.error.log` and `nginx.startup.log`, which is why they won’t be rotated. You’ll have to decide what to do with those: keep them around for some time or just remove them. One minor suggestion: it seems like to list a directory inside the container it’s much more straightforward and reliable to run this: ```auto sudo docker exec -t pmm-server ls -lah /srv/logs ``` Again, should you want to delete the files, you can do so by getting inside of `pmm-server` container first with: ```auto sudo docker exec -it pmm-server bash ``` and then `cd /srv/logs && rm nginx.*.log`. Alternatively, moving them out would entail copying to the host first (`docker cp pmm-server:/srv/logs/nginx.access.log .`) and then deleting. Alex ---
**Author:** ![dbray\_sd](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dbray_sd/32/4448_2.png) [@dbray\_sd](https://forums.percona.com/u/dbray_sd)\ **Post date:** [September 8, 2021, 12:41pm UTC](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740/34 "2021-09-08T12:41:20Z")
Makes perfect sense. I’d say from here I’ll be good to go. I really appreciate all your help Alex, you’re the best! [Previous page](https://forums.percona.com/t/nginx-logs-filling-up-volume/11740.md?page=1)