Yep! It should be fine now. We rotate nginx logs after 50M.
1 Like
Apologies, but something still doesnāt seem right. Logs havenāt rotated yet, even after restarting docker/containerd. I also checked the above pmm-admin --version command, and I still see:
ProjectName: pmm-admin
Version: 2.14.0
PMMVersion: 2.14.0
Timestamp: 2021-01-28 12:36:42 (UTC)
FullCommit: cdf593c6774c3c239aaf22fff27e9e7aa1364a60
I was expecting that to now read 2.20.0, same as I see in the web app. Right?
1 Like
Yes, it should be 2.20ā¦ To be honest it looks very suspicious because we run updates on each pmm-update runā¦
Could you pls attache pmm-update-perform.log again?
docker cp pmm-server:/srv/logs/pmm-update-perform.log .
1 Like
Attached is the updated log:
pmm-update-perform.log (18.9 KB)
2 Likes
OK, I tried to reproduce this issue and couldnāt 
My suggestion:
- Letās make backup again:
Percona Monitoring and Management - Upgrade system by hand:
yum upgrade
- and run upgrade PMM:
/usr/sbin/pmm-update -perform -playbook=/usr/share/pmm-update/ansible/playbook/tasks/update.yml
2 Likes
OK, clearly Iām doing something wrongā¦and my apologies for not knowing exactly what it is. Iām not the best when it comes to Docker. Here is what I did:
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
12fd55c0de21 7599815450fd "/opt/entrypoint.sh" 6 months ago Up 4 minutes (healthy) 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp pmm-server
sudo docker run -it 7599815450fd /bin/bash
[root@9baa6e6a11e3 opt]#
[root@9baa6e6a11e3 opt]# rpm -qa | grep pmm
pmm-managed-2.14.0-17.2101281011.787c431.el7.x86_64
pmm2-client-2.14.0-6.el7.x86_64
pmm-server-2.14.0-21.2101281241.cb737e6.el7.noarch
pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch
[root@9baa6e6a11e3 opt]# yum -y upgrade
......
[root@9baa6e6a11e3 opt]# rpm -qa | grep pmm
pmm-managed-2.21.0-17.2108101848.05b1420.el7.x86_64
pmm-server-2.21.0-24.2108162024.afbc5c6.el7.noarch
pmm2-client-2.21.0-6.el7.x86_64
pmm-update-2.21.0-61.2108261434.5ec7f7f.el7.noarch
[root@9baa6e6a11e3 opt]# exit
$ sudo docker run -it 7599815450fd /bin/bash
[root@c0b7d8549a80 opt]# rpm -qa | grep pmm
pmm-managed-2.14.0-17.2101281011.787c431.el7.x86_64
pmm2-client-2.14.0-6.el7.x86_64
pmm-server-2.14.0-21.2101281241.cb737e6.el7.noarch
pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch
What am I doing wrong?
1 Like
Hi,
It feels like the ārollbackā to v2.14.0 is due to a file system caveat, itās just not very clear where it is coming fromā¦
Quite intriguing, letās see if we can get more info to help investigate:
1/ Have you mounted any volumes to your docker container? It would help to see how you launched the PMM Server docker container.
2/ Which file system is your docker using (docker info)?
3/ Let us know about any particular configuration settings you may have applied to PMM Server
Alex
1 Like
Hey Alex,
I originally followed the simple steps documented here: Installing Percona Monitoring and Management (PMM) Software - Percona
This was all done on a CentOS 8 server.
sudo dnf -y install docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker containerd
sudo docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true
sudo docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2
Nightly, I backup everything (to a NFS share) with:
sudo docker cp pmm-data:/srv /backups/latest/
Other than that, Iāve made the necessary tweaks within the web app itself, but havenāt done anything else I can think of specifically to the OS nor docker itself.
Hereās the details to your questions:
1/ Have you mounted any volumes to your docker container? It would help to see how you launched the PMM Server docker container.
No, not that I can recall. I checked bash_history and everything, and I see no references to mount or anything similar.
2/ Which file system is your docker using (docker info)?
Using the default xfs filesystem CentOS 8 installs by default, and located in the default /var/lib/docker directory.
3/ Let us know about any particular configuration settings you may have applied to PMM Server
Nothing comes to mind. All the customization Iāve done has all been within the web app itself.
Thanks you, I appreciate your help with this. I have no problem removing everything and starting over. I would just need to know exactly which files to restore, so that my customizations within the web app come back.
1 Like
Hi,
Thanks a lot for providing such extensive information. Your provisioning commands do look good and the ārollbackā failure has most certainly nothing to do with how you bootstrap PMM Server.
Now the only thing we can try is to tear down the container and try to spin it back up again. Your data is safe since itās persisted to the pmm-data container. In your case the commands would be:
To tear down:
sudo docker stop pmm-server && sudo docker rm pmm-server
To spin up (same as before):
sudo docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2
It may well be that docker engine is misbehaving and that could potentially help put it back on track. You may want to reboot the machine before spinning up PMM again.
Canāt wait to hear what it does 
)
Alex
1 Like
Sorry for the delays, it was a holiday for us yesterday.
Careful what you ask for This is what it did.
All the commands worked without error. Only slight change I had to make was the ports, so that iptables would NAT properly:
sudo docker run --detach -p 8080:80 -p 8443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2
I have to use 8080 and 8443, but no issue there.
When I brought the web app up, it did initially say: Current version: 2.15.0 February 28, 2020
So I went ahead and performed the in place upgrade, using the button within the GUI to upgrade, and Iām attaching the latest pmm-update-perform.log (23.5 KB)
However, the issue is not resolved, and now I have a little more confusion. It appears I now have two separate instances available and/or running:
$ sudo docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5c3fcb9fc3f4 percona/pmm-server:2 "/bin/bash" 8 minutes ago Exited (130) 3 minutes ago charming_neumann
140fa15cdc82 7599815450fd "/bin/bash" 11 minutes ago Exited (0) 10 minutes ago quirky_noyce
1fd26d7c5d00 percona/pmm-server:2 "/opt/entrypoint.sh" 16 minutes ago Up 16 minutes (healthy) 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp pmm-server
c0b7d8549a80 7599815450fd "/bin/bash" 4 days ago Exited (0) 4 days ago cool_liskov
9baa6e6a11e3 7599815450fd "/bin/bash" 4 days ago Exited (130) 4 days ago great_curie
f6a56e4bd24f 7599815450fd "/bin/bash" 4 days ago Exited (0) 4 days ago infallible_brahmagupta
4b1c5a1f35cd 7599815450fd "/opt/entrypoint.sh" 4 days ago Exited (0) 4 days ago wonderful_austin
90de0a423713 7599815450fd "/bin/bash" 4 days ago Exited (0) 4 days ago beautiful_goodall
29d085c1c8fa 7599815450fd "/bin/bash" 4 days ago Exited (130) 4 days ago inspiring_germain
29cbf33a7b18 7599815450fd "/bin/bash" 11 days ago Exited (0) 11 days ago ecstatic_tesla
f7bb17da2f4f 7599815450fd "/bin/bash" 11 days ago Exited (130) 11 days ago silly_swartz
fa8e60a0a1d6 7599815450fd "/bin/bash" 12 days ago Exited (0) 11 days ago keen_ride
0d58ba4410e7 7599815450fd "/bin/bash" 13 days ago Exited (42) 13 days ago nice_booth
a7d924df186e 7599815450fd "/bin/bash" 13 days ago Exited (130) 13 days ago pensive_engelbart
18fe481014f1 7599815450fd "/bin/bash" 2 weeks ago Exited (1) 2 weeks ago bold_noyce
02285cadb6a5 7599815450fd "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago boring_heyrovsky
8b97d47777f1 7599815450fd "/bin/bash" 2 weeks ago Exited (130) 2 weeks ago mystifying_chandrasekhar
4a082189086f 7599815450fd "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago affectionate_hermann
f2c7ffed801b 7599815450fd "/bin/bash" 2 weeks ago Exited (0) 2 weeks ago busy_goldwasser
7c0838f7b2f2 7599815450fd "/bin/true" 6 months ago Created pmm-data
So now if I check the old command I used to run, I see:
$ sudo docker run -it 7599815450fd /bin/bash
[root@c7d9295f9a9e opt]# rpm -qa | grep pmm
pmm-managed-2.14.0-17.2101281011.787c431.el7.x86_64
pmm2-client-2.14.0-6.el7.x86_64
pmm-server-2.14.0-21.2101281241.cb737e6.el7.noarch
pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch
However, if I check the new container that is now showing up, I see:
$ sudo docker run -it percona/pmm-server:2 /bin/bash
[root@ce2446d9cd98 opt]# rpm -qa | grep pmm
pmm-server-2.15.0-21.2103010711.fdd2fb8.el7.noarch
pmm-update-2.15.0-52.2103010712.c64c622.el7.noarch
pmm-managed-2.15.0-17.2102251420.ec05154.el7.x86_64
pmm2-client-2.15.0-6.el7.x86_64
However, the GUI does show: Current version: 2.21.0 (August 25, 2021)
All very confusing.
1 Like
Ok, now Iām starting to understand what this all means. Letās break things down one by one.
1. What PMM Server version you are running
Judging by the log results you attached, it looks like you successfully migrated your PMM Server to the latest version which is 2.21.0 as of now. This is also confirmed by the UI since it show the current version number on the Upgrade panel.
2. How to check the rpm package versions inside the container
When you run this following command sudo docker run -it percona/pmm-server:2 /bin/bash to check for installed rpm package versions, it creates yet another PMM Server, a brand new one, off of the image version 2.15.0. When you get inside the docker conatiner, you shouldnāt be surprised to see old packages (2.15.0 in your case) since this is effectively the version of one of your docker images. This is expected, since those rpm packages come bundled with the docker image and their versions always match the image version.
Now, to see which package versions are installed to your running PMM Server (its id is 1fd26d7c5d00 ), you could run the following command rather:
% sudo docker exec -t pmm-server sh -c "rpm -qa | grep pmm"
pmm-managed-2.22.0-17.2108310927.52bf11b.el7.x86_64
pmm2-client-2.22.0-6.el7.x86_64
pmm-server-2.22.0-24.2109010926.883130e.el7.noarch
pmm-update-2.22.0-60.2108240517.4be8fda.el7.noarch
And Iām pretty confident your rpm versions should be 2.21.0 this time, otherwise we can call it magic
3. Cleanup
Since those containers which are no longer running occupy much space (ca. 2 GB each) on your machine you may want to consider removing them with:
sudo docker rm charming_neumann cool_liskov (... and more)
Hope this helps, otherwise please let us know.
Alex
1 Like
Alex,
I really appreciate your time and effort explaining all of this. It makes so much more sense now, and yesā¦finallyā¦I see:
$ sudo docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1fd26d7c5d00 percona/pmm-server:2 "/opt/entrypoint.sh" 24 hours ago Up 3 minutes (healthy) 0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp pmm-server
7c0838f7b2f2 7599815450fd "/bin/true" 6 months ago Created pmm-data
$ sudo docker exec -t pmm-server sh -c "rpm -qa | grep pmm"
pmm-update-2.21.0-61.2108261434.5ec7f7f.el7.noarch
pmm-server-2.21.0-24.2108162024.afbc5c6.el7.noarch
pmm2-client-2.21.0-6.el7.x86_64
pmm-managed-2.21.0-17.2108101848.05b1420.el7.x86_64
So now, with alllllllll of that out of the way, I am still seeing a rather large log file:
$ sudo ls -lah /var/lib/docker/volumes/37d310a60891d69b99d7f8ab76daa059c7211fdbd5e29374836d833a451178e2/_data/logs/ | grep nginx
-rw-r--r-- 1 root root 1.4G Sep 7 11:29 nginx.access.log
-rw-r--r-- 1 root root 537K Sep 7 11:29 nginx.error.log
-rw-r--r-- 1 root root 27M Sep 8 11:15 nginx.log
-rw-r--r-- 1 root root 51M Sep 7 21:43 nginx.log.1
-rw-r--r-- 1 root root 51M Sep 6 20:24 nginx.log.2
-rw-r--r-- 1 root root 127K Sep 7 11:31 nginx.startup.log
Should I assume the nginx.access.log shouldāve already rotated. Or will that rotate tonight/tomorrow?
Again, thank you so much, I really appreciate the help!
2 Likes
Wow, so excited to see we were able to help sort it out!
To your question. After we refactored the log rotation, there should only be one log file per process, for example nginxās logs are now being saved to /srv/logs/nginx.log only. Therefore, the server will no longer touch the other log files, namely nginx.access.log, nginx.error.log and nginx.startup.log, which is why they wonāt be rotated. Youāll have to decide what to do with those: keep them around for some time or just remove them.
One minor suggestion: it seems like to list a directory inside the container itās much more straightforward and reliable to run this:
sudo docker exec -t pmm-server ls -lah /srv/logs
Again, should you want to delete the files, you can do so by getting inside of pmm-server container first with:
sudo docker exec -it pmm-server bash
and then cd /srv/logs && rm nginx.*.log.
Alternatively, moving them out would entail copying to the host first (docker cp pmm-server:/srv/logs/nginx.access.log .) and then deleting.
Alex
1 Like
Makes perfect sense. Iād say from here Iāll be good to go. I really appreciate all your help Alex, youāre the best!
2 Likes