It appears that the Nginx logs are not rotating like the rest of the log files, and are therefor growing out of control.
sudo ls -lah /var/lib/docker/volumes/37d310a60891d69b99d7f8ab76daa059c7211fdbd5e29374836d833a451178e2/_data/logs/
total 1.7G
drwxrwxr-x 2 1000 1000 4.0K Aug 11 14:08 .
drwxr-xr-x 12 root root 189 Feb 16 19:07 ..
-rw-r--r-- 1 root root 14K Aug 2 17:52 alertmanager.log
-rw-r----- 1 root root 0 Feb 16 19:07 clickhouse-server.err.log
-rw-r----- 1 root root 704M Aug 11 14:52 clickhouse-server.log
-rw-r----- 1 root root 50M Jun 26 22:28 clickhouse-server.log.0.gz
-rw-r----- 1 root root 50M Apr 22 06:06 clickhouse-server.log.1.gz
-rw-r--r-- 1 root root 5.1K Aug 2 17:52 clickhouse-server.startup.log
-rw-r--r-- 1 root root 0 Feb 16 19:07 cron.log
-rw-r--r-- 1 root root 357K Aug 2 17:52 dashboard-upgrade.log
-rw-r--r-- 1 root root 1.4M Aug 11 10:56 grafana.log
-rw-r--r-- 1 root root 1.2G Aug 11 14:52 nginx.access.log
-rw-r--r-- 1 root root 800M Aug 11 10:56 nginx.error.log
-rw-r--r-- 1 root root 112 Aug 2 17:52 nginx.startup.log
-rw-r--r-- 1 root root 7.3M Aug 11 14:52 pmm-agent.log
-rw-r--r-- 1 root root 11M Jul 20 14:09 pmm-agent.log.1
-rw-r--r-- 1 root root 11M Jun 19 20:18 pmm-agent.log.2
-rw-r--r-- 1 root root 11M May 20 03:35 pmm-agent.log.3
-rw-r--r-- 1 root root 5.2M Aug 11 14:52 pmm-managed.log
-rw-r--r-- 1 root root 11M Aug 11 14:08 pmm-managed.log.1
-rw-r--r-- 1 root root 11M Aug 11 12:42 pmm-managed.log.2
-rw-r--r-- 1 root root 11M Aug 11 11:17 pmm-managed.log.3
-rw-r--r-- 1 root root 19K Jul 20 12:59 pmm-update-perform.log
-rw------- 1 26 26 11K Aug 2 17:52 postgresql.log
-rw-r--r-- 1 root root 12K Aug 2 17:52 postgresql.startup.log
-rw-r--r-- 1 root root 5.5M Aug 11 14:52 qan-api2.log
-rw-r--r-- 1 root root 11M Aug 10 05:51 qan-api2.log.1
-rw-r--r-- 1 root root 11M Aug 7 16:43 qan-api2.log.2
-rw-r--r-- 1 root root 11M Aug 5 03:37 qan-api2.log.3
-rw-r--r-- 1 root root 131K Aug 11 03:14 supervisord.log
-rw-r--r-- 1 root root 6.2M Aug 11 14:52 victoriametrics.log
-rw-r--r-- 1 root root 11M Aug 8 15:59 victoriametrics.log.1
-rw-r--r-- 1 root root 11M Aug 3 21:03 victoriametrics.log.2
-rw-r--r-- 1 root root 11M Jul 30 09:15 victoriametrics.log.3
-rw-r--r-- 1 root root 216K Aug 2 17:52 vmalert.lo
I’ve already read over Configure logging drivers | Docker Documentation and checked my /etc/docker/daemon.json, but since the other log files are rotating just fine, this seems to be more of a pmm issue, specific to Nginx.
Thanks!
Hi @dbray_sd thanks for posting to the Percona forums!
In fact if you upgrade to PMM 2.20 you’ll be able to take advantage of proper log rotation of all files in /srv/logs. Let us know your experiences!
https://jira.percona.com/browse/PMM-7627
1 Like
@Michael_Coburn wow, can’t believe I missed that there was an upgrade available. Usually I always apply updates before posting about an issue. Either way, I’ve upgraded to the latest 2.20, and waited for 24hrs, but I still see the large nginx.access.log and nginx.error.log files. Is there some additional steps I should perform after the update?
1 Like
Hi @dbray_sd
Have the logs been rotated? if not I’ll want to file a bug
1 Like
Just checked again, and no, the logs are not rotating. I restarted everything to be sure, waited another 24hrs, but the logs are still growing. Thank you.
1 Like
Hi @dbray_sd
I’m checking with the Engineers to see what’s going on. [PMM-7627] Consistent PMM Server log management - Percona JIRA should have fixed your use case
1 Like
Hi @dbray_sd,
We use a new file name for nginx logs in PMM 2.20 version:
/srv/logs/nginx.log
Instead:
nginx.startup.log
nginx.error.log
nginx.access.log
The only nginx.log will use and you can remove old log files. nginx.log also will rotate 
1 Like
@nikita.beletskii
Interesting. I’m running PMM 2.20, and I do see the nginx.log file, but it was last written to about a week ago, whereas the other log files are all recent:
$ sudo ls -lah /var/lib/docker/volumes/37d310a60891d69b99d7f8ab76daa059c7211fdbd5e29374836d833a451178e2/_data/logs/ | grep nginx
-rw-r--r-- 1 root root 724M Aug 17 16:21 nginx.access.log
-rw-r--r-- 1 root root 146K Aug 17 11:02 nginx.error.log
-rw-r--r-- 1 root root 336 Aug 11 19:01 nginx.log
-rw-r--r-- 1 root root 112 Aug 2 17:52 nginx.startup.log
The above is all in UTC time.
1 Like
Hmm…looks strange. Can you run following commands in container, pls:
cat /etc/supervisord.d/pmm.ini
ps -aux | grep nginx
cat /etc/nginx/nginx.conf | grep access_log
Thank you!
1 Like
Sure thing.
cat /etc/supervisord.d/pmm.ini
[unix_http_server]
chmod = 0700
username = dummy
password = dummy
[supervisord]
logfile = /srv/logs/supervisord.log
pidfile = /tmp/supervisord.pid
nodaemon = true
nocleanup = false
; TODO childlogdir = /srv/logs/supervisor
user = root
strip_ansi = false
[supervisorctl]
username = dummy
password = dummy
[program:postgresql]
priority = 1
command =
/usr/pgsql-11/bin/postgres
-D /srv/postgres
-c shared_preload_libraries=pg_stat_statements
-c pg_stat_statements.max=10000
-c pg_stat_statements.track=all
-c pg_stat_statements.save=off
user = postgres
autorestart = true
autostart = true
startretries = 10
startsecs = 1
stopsignal = INT ; Fast Shutdown mode
stopwaitsecs = 300
; postgresql.conf contains settings to log to /srv/logs/postgresql.log,
; so log startup messages only (if any) with supervisord
stdout_logfile = /srv/logs/postgresql.startup.log
stdout_logfile_maxbytes = 1MB
stdout_logfile_backups = 3
redirect_stderr = true
[program:clickhouse]
priority = 2
command = /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml
autorestart = true
autostart = true
startretries = 10
startsecs = 1
stopsignal = TERM
stopwaitsecs = 300
; config.xml contains settings to log to /srv/logs/clickhouse-server.log and .err.log,
; so log startup messages only (if any) with supervisord
stdout_logfile = /srv/logs/clickhouse-server.startup.log
stdout_logfile_maxbytes = 1MB
stdout_logfile_backups = 3
redirect_stderr = true
[program:grafana]
priority = 3
command =
/usr/sbin/grafana-server
--homepath=/usr/share/grafana
--config=/etc/grafana/grafana.ini
cfg:default.paths.data=/srv/grafana
cfg:default.paths.plugins=/var/lib/grafana/plugins
cfg:default.paths.logs=/srv/logs
cfg:default.log.mode=console
cfg:default.log.console.format=console
cfg:default.server.root_url="%%(protocol)s://%%(domain)s:%%(http_port)s/graph"
user = grafana
directory = /usr/share/grafana
environment = HOME=/usr/share/grafana
autorestart = true
autostart = true
startretries = 10
startsecs = 1
stopsignal = TERM
stopwaitsecs = 300
stdout_logfile = /srv/logs/grafana.log
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups = 3
redirect_stderr = true
[program:nginx]
priority = 4
command = nginx
autorestart = true
autostart = true
startretries = 10
startsecs = 1
stopsignal = TERM
stopwaitsecs = 10
; nginx.conf contains settings to log to /var/log/nginx/error.log and access.log,
; so log startup messages only (if any) with supervisord
stdout_logfile = /srv/logs/nginx.startup.log
stdout_logfile_maxbytes = 1MB
stdout_logfile_backups = 3
redirect_stderr = true
[program:cron]
priority = 5
command = /usr/sbin/crond -n
autorestart = true
autostart = true
startretries = 10
startsecs = 1
stopsignal = TERM
stopwaitsecs = 10
stdout_logfile = /srv/logs/cron.log
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups = 3
redirect_stderr = true
[program:dashboard-upgrade]
priority = 10
command = /usr/share/percona-dashboards/import-dashboards.py
autorestart = unexpected
exitcodes = 0
autostart = true
startretries = 10
startsecs = 0 ; it can exit quickly if there is nothing to do
stopsignal = TERM
stopwaitsecs = 10
stdout_logfile = /srv/logs/dashboard-upgrade.log
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups = 3
redirect_stderr = true
; [program:clickhouse_exporter]
; priority = 11
; command =
; /usr/local/percona/pmm-client/clickhouse_exporter
; -telemetry.address=localhost:9116
; stdout_logfile = /srv/logs/clickhouse_exporter.log
; stderr_logfile = /srv/logs/clickhouse_exporter.log
; autorestart = true
[program:pmm-managed]
priority = 14
command =
/usr/sbin/pmm-managed
--victoriametrics-config=/etc/victoriametrics-promscrape.yml
--victoriametrics-url=http://127.0.0.1:9090/prometheus
--postgres-name=pmm-managed
--postgres-username=pmm-managed
--postgres-password=pmm-managed
--supervisord-config-dir=/etc/supervisord.d
autorestart = true
autostart = true
startretries = 1000
startsecs = 1
stopsignal = TERM
stopwaitsecs = 300
stdout_logfile = /srv/logs/pmm-managed.log
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups = 3
redirect_stderr = true
priority = 15
command = /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
autorestart = true
autostart = true
startretries = 1000
startsecs = 1
stopsignal = TERM
stopwaitsecs = 10
stdout_logfile = /srv/logs/pmm-agent.log
stdout_logfile_maxbytes = 10MB
stdout_logfile_backups = 3
redirect_stderr = true
[program:pmm-update-perform]
command = /usr/sbin/pmm-update -perform -playbook=/usr/share/pmm-update/ansible/playbook/tasks/update.yml
directory = /
autorestart = unexpected
exitcodes = 0
autostart = false
startretries = 10
startsecs = 1
stopsignal = TERM
stopwaitsecs = 300
stdout_logfile = /srv/logs/pmm-update-perform.log
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups = 3
redirect_stderr = true
ps -aux | grep nginx
root 435 0.0 0.0 9108 876 pts/0 S+ 17:08 0:00 grep --color=auto nginx
grep access_log /etc/nginx/nginx.conf
access_log /srv/logs/nginx.access.log main;
1 Like
These files from PMM before 2.20 
Can you run another command, pls?
yum list installed | grep pmm
1 Like
That is so odd. I’m looking at the web app, and I see:
“Current version: 2.20.0 (July 26, 2021)”
Here is the yum output:
[root@8b97d47777f1 opt]# yum list installed | grep pmm
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
pmm-managed.x86_64 2.14.0-17.2101281011.787c431.el7 @local
pmm-server.noarch 2.14.0-21.2101281241.cb737e6.el7 @local
pmm-update.noarch 2.14.0-51.2101281242.f607c2c.el7 @local
pmm2-client.x86_64 2.14.0-6.el7 @local
1 Like
OK…We found why you have old files. Now we need to understand why you have an old version of the packages 
Can you run commands, pls:
docker cp pmm-server:/srv/logs/pmm-update-perform.log .
and attach pmm-update-perform.log file to the ticket.
Also, I need output from two commands:
yum repolist
cat /etc/yum.repos.d/percona-original-release.repo
Thank you!
1 Like
@nikita.beletskii
Here you go:
[root@8b97d47777f1 opt]# yum list installed | grep pmm
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
pmm-managed.x86_64 2.14.0-17.2101281011.787c431.el7 @local
pmm-server.noarch 2.14.0-21.2101281241.cb737e6.el7 @local
pmm-update.noarch 2.14.0-51.2101281242.f607c2c.el7 @local
pmm2-client.x86_64 2.14.0-6.el7 @local
[root@8b97d47777f1 opt]# yum repolist
Loaded plugins: changelog, fastestmirror, ovl
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Determining fastest mirrors
* base: mirror.mojohost.com
* epel: mirror.es.its.nyu.edu
* extras: mirror.ilumno.com
* updates: ftp.usf.edu
repo id repo name status
!base/7/x86_64 CentOS-7 - Base 10072
!epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 13514
!extras/7/x86_64 CentOS-7 - Extras 448
!nginx/x86_64 nginx repo 761
!percona-ppg-11 PostgreSQL YUM repository - x86_64 98
!percona-release-noarch/7 Percona Original release/noarch YUM repository 63
!percona-release-x86_64/7 Percona Original release/x86_64 YUM repository 2256
!pmm2-server PMM Server YUM repository - x86_64 191
!prel-release-noarch/7 Percona Release release/noarch YUM repository 3
!updates/7/x86_64 CentOS-7 - Updates 1216
repolist: 28622
[root@8b97d47777f1 opt]# cat /etc/yum.repos.d/percona-original-release.repo
#
# This repo is managed by "percona-release" utility, do not edit!
#
[percona-release-x86_64]
name = Percona Original release/x86_64 YUM repository
baseurl = http://repo.percona.com/percona/yum/release/$releasever/RPMS/x86_64
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
[percona-release-noarch]
name = Percona Original release/noarch YUM repository
baseurl = http://repo.percona.com/percona/yum/release/$releasever/RPMS/noarch
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
[percona-release-sources]
name = Percona Original release/sources YUM repository
baseurl = http://repo.percona.com/percona/yum/release/$releasever/SRPMS
enabled = 0
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
Let me know if you need that at the OS level.
I’ve attached the pmm-update-perform.log file.
pmm-update-perform.log (18.9 KB)
1 Like
Sorry, I missed notification from the forum 
As I see from your update log you should have 2.20 version:
Updated:
pmm-update.noarch 0:2.20.0-60.2107271101.d72f0db.el7
Beside, according to the logs, you updated from 2.19 to 2.20:
---> Package pmm-update.noarch 0:2.19.0-59.2106301054.58b0b89.el7 will be updated
---> Package pmm-update.noarch 0:2.20.0-60.2107271101.d72f0db.el7 will be an update
This string also looks suspicious because we have yum-cron in our image.
Can you run also these commands, pls:
pmm-admin --version
yum clean all
yum upgrade --assumeno
1 Like
I appreciate your attention to this issue. Here are the results:
pmm-admin --version
ProjectName: pmm-admin
Version: 2.14.0
PMMVersion: 2.14.0
Timestamp: 2021-01-28 12:36:42 (UTC)
FullCommit: cdf593c6774c3c239aaf22fff27e9e7aa1364a60
yum clean all
Loaded plugins: changelog, fastestmirror, ovl
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Cleaning repos: base epel extras nginx percona-ppg-11 percona-release-noarch percona-release-x86_64 pmm2-server
: prel-release-noarch updates
Cleaning up list of fastest mirrors
Other repos take up 5.3 M of disk space (use --verbose for details)
yum upgrade --assumeno
=====================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================
Updating:
ansible noarch 2.9.24-2.el7 epel 17 M
bind-license noarch 32:9.11.4-26.P2.el7_9.5 updates 91 k
dbaas-controller x86_64 2.20.0-1.2106281154.316d3d2.el7 pmm2-server 4.3 M
device-mapper x86_64 7:1.02.170-6.el7_9.5 updates 297 k
device-mapper-libs x86_64 7:1.02.170-6.el7_9.5 updates 325 k
glib2 x86_64 2.56.1-9.el7_9 updates 2.5 M
glibc x86_64 2.17-324.el7_9 updates 3.6 M
glibc-common x86_64 2.17-324.el7_9 updates 12 M
libblkid x86_64 2.23.2-65.el7_9.1 updates 183 k
libmount x86_64 2.23.2-65.el7_9.1 updates 185 k
libsmartcols x86_64 2.23.2-65.el7_9.1 updates 143 k
libuuid x86_64 2.23.2-65.el7_9.1 updates 84 k
nginx x86_64 1:1.21.1-1.el7.ngx nginx 790 k
nss x86_64 3.53.1-7.el7_9 updates 869 k
nss-sysinit x86_64 3.53.1-7.el7_9 updates 66 k
nss-tools x86_64 3.53.1-7.el7_9 updates 535 k
openldap x86_64 2.4.44-23.el7_9 updates 356 k
percona-alertmanager x86_64 0.21.0-3.el7 pmm2-server 12 M
percona-dashboards noarch 2.20.0-15.2107131605.75d67a4.el7 pmm2-server 32 M
percona-grafana x86_64 7.5.7-93.2107220953.908ff0c.el7 pmm2-server 41 M
percona-postgresql11 x86_64 1:11.12-2.el7 percona-ppg-11 1.7 M
percona-postgresql11-contrib x86_64 1:11.12-2.el7 percona-ppg-11 615 k
percona-postgresql11-libs x86_64 1:11.12-2.el7 percona-ppg-11 368 k
percona-postgresql11-server x86_64 1:11.12-2.el7 percona-ppg-11 4.8 M
percona-qan-api2 x86_64 2.20.0-15.2107120744.af14526.el7 pmm2-server 4.4 M
percona-release noarch 1.0-27 percona-release-noarch 20 k
percona-victoriametrics x86_64 1.60.0-1.el7 pmm2-server 7.9 M
perl x86_64 4:5.16.3-299.el7_9 updates 8.0 M
perl-Pod-Escapes noarch 1:1.04-299.el7_9 updates 52 k
perl-libs x86_64 4:5.16.3-299.el7_9 updates 690 k
perl-macros x86_64 4:5.16.3-299.el7_9 updates 44 k
pmm-managed x86_64 2.20.0-17.2107220951.3c660c3.el7 pmm2-server 16 M
pmm-server noarch 2.20.0-23.2107271100.e68715f.el7 pmm2-server 2.0 M
pmm-update noarch 2.20.0-60.2107271101.d72f0db.el7 pmm2-server 832 k
pmm2-client x86_64 2.20.0-6.el7 percona-release-x86_64 43 M
screen x86_64 4.1.0-0.27.20120314git3c2946.el7_9 updates 553 k
systemd x86_64 219-78.el7_9.3 updates 5.1 M
systemd-libs x86_64 219-78.el7_9.3 updates 418 k
systemd-sysv x86_64 219-78.el7_9.3 updates 97 k
unzip x86_64 6.0-22.el7_9 updates 171 k
util-linux x86_64 2.23.2-65.el7_9.1 updates 2.0 M
zlib x86_64 1.2.7-19.el7_9 updates 90 k
Transaction Summary
=====================================================================================================================
Upgrade 42 Packages
This is a little confusing. It appears that some things thinks it has been upgraded to the latest 2.20, however some do not.
1 Like
OK, let’s try to update PMM.
Could you do a backup before update:
It’ll be a big update and backup can be useful.
yum upgrade -y pmm-update
And after command will be finished:
/usr/sbin/pmm-update -perform -playbook=/usr/share/pmm-update/ansible/playbook/tasks/update.yml
1 Like
The yum upgrade completed without issue:
Running transaction
Updating : pmm-update-2.20.0-60.2107271101.d72f0db.el7.noarch 1/2
Cleanup : pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch 2/2
Verifying : pmm-update-2.20.0-60.2107271101.d72f0db.el7.noarch 1/2
Verifying : pmm-update-2.14.0-51.2101281242.f607c2c.el7.noarch 2/2
Updated:
pmm-update.noarch 0:2.20.0-60.2107271101.d72f0db.el7
Complete!
However the ansible script failed:
INFO[0001] pmm-update:
before update = {Version:2.20.0 FullVersion:2.20.0-60.2107271101.d72f0db.el7 BuildTime:2021-07-27 11:01:17 +0000 UTC Repo:pmm2-server}
after update = {Version:2.20.0 FullVersion:2.20.0-60.2107271101.d72f0db.el7 BuildTime:2021-07-27 11:01:17 +0000 UTC Repo:pmm2-server}
INFO[0001] Version did not change.
Starting "ansible-playbook --flush-cache /usr/share/pmm-update/ansible/playbook/tasks/update.yml" ...
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [detect /srv/pmm-distribution] ********************************************
ok: [localhost]
TASK [detect containers] *******************************************************
ok: [localhost]
TASK [set dbaas toggle] ********************************************************
ok: [localhost]
TASK [force container] *********************************************************
skipping: [localhost]
TASK [Configure systemd] *******************************************************
skipping: [localhost]
TASK [Remove old supervisord service confiuration] *****************************
skipping: [localhost]
TASK [Increase number of open files for jobs] **********************************
skipping: [localhost]
TASK [Reread supervisord configuration] ****************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["supervisorctl", "reread"], "delta": "0:00:00.732999", "end": "2021-08-24 18:12:43.944505", "msg": "non-zero retu
rn code", "rc": 2, "start": "2021-08-24 18:12:43.211506", "stderr": "", "stderr_lines": [], "stdout": "error: <class 'socket.error'>, [Errno 2] No such file or directory:
file: /usr/lib64/python2.7/socket.py line: 224", "stdout_lines": ["error: <class 'socket.error'>, [Errno 2] No such file or directory: file: /usr/lib64/python2.7/socket.
py line: 224"]}
PLAY RECAP *********************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=1 skipped=4 rescued=0 ignored=0
FATA[0008] RunPlaybook failed: exit status 2
1 Like
Did you change files in the container before this problem?
Most likely these error means that you don’t have supervisor socket in a container and it’s super weird because a supervisor is PID 1 in a container (it should be run always).
We tested updates between any versions on each release and I didn’t see problems like this…
Anyway, can you restart the container with PMM?
docker restart pmm-server
where pmm-server is container name and run Ansible command again:
/usr/sbin/pmm-update -perform -playbook=/usr/share/pmm-update/ansible/playbook/tasks/update.yml
1 Like
Nope, I’ve never changed any files manually.
The restart and rerun of the pmm-update command fixed the update.
Now I guess I’ll sit back and wait the ## hours to see if the logs rotate? Or, is there anything else you want me to check?
Thanks!
1 Like