Running PMM-Server in docker ends up in supervisord error

Mason_Cubes · January 17, 2022, 9:00am

I am trying to run the pmm-server like so.

docker run -d -p 80:80 -v /mnt/storage/percona/srv:/srv percona/pmm-server:2.25.0

The result I get is the following.

Error: The directory named as part of the path /srv/logs/supervisord.log does not exist
For help, use /usr/bin/supervisord -h

Update: If I manually create the logs directory. Then the issue is following.

percona_1  | TRAC[2022-01-17T09:21:24.407+00:00] ParseEnvVars: `HOSTNAME=2853578237bb`: k=`HOSTNAME` v=`2853578237bb` 
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`: k=`PATH` v=`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` 
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `PMM_DEBUG=1`: k=`PMM_DEBUG` v=`1` 
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `PWD=/opt`: k=`PWD` v=`/opt`   
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `PMM_TRACE=1`: k=`PMM_TRACE` v=`1` 
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `SHLVL=1`: k=`SHLVL` v=`1`     
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `HOME=/root`: k=`HOME` v=`/root` 
percona_1  | TRAC[2022-01-17T09:21:24.408+00:00] ParseEnvVars: `_=/usr/sbin/pmm-managed-init`: k=`_` v=`/usr/sbin/pmm-managed-init` 
percona_1  | 2022-01-17 09:21:24,594 INFO Included extra file "/etc/supervisord.d/alertmanager.ini" during parsing
percona_1  | 2022-01-17 09:21:24,594 INFO Included extra file "/etc/supervisord.d/dbaas-controller.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Included extra file "/etc/supervisord.d/grafana.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Included extra file "/etc/supervisord.d/pmm.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Included extra file "/etc/supervisord.d/prometheus.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Included extra file "/etc/supervisord.d/qan-api2.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Included extra file "/etc/supervisord.d/victoriametrics.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Included extra file "/etc/supervisord.d/vmalert.ini" during parsing
percona_1  | 2022-01-17 09:21:24,595 INFO Set uid to user 0 succeeded
percona_1  | 2022-01-17 09:21:24,610 INFO RPC interface 'supervisor' initialized
percona_1  | 2022-01-17 09:21:24,611 INFO supervisord started with pid 1
percona_1  | 2022-01-17 09:21:25,615 INFO spawned: 'pmm-update-perform-init' with pid 14
percona_1  | 2022-01-17 09:21:25,619 INFO spawned: 'postgresql' with pid 15
percona_1  | 2022-01-17 09:21:25,622 INFO spawned: 'clickhouse' with pid 16
percona_1  | 2022-01-17 09:21:25,625 INFO spawned: 'grafana' with pid 18
percona_1  | 2022-01-17 09:21:25,627 INFO spawned: 'nginx' with pid 26
percona_1  | 2022-01-17 09:21:25,630 INFO spawned: 'cron' with pid 27
percona_1  | 2022-01-17 09:21:25,633 INFO spawned: 'victoriametrics' with pid 28
percona_1  | 2022-01-17 09:21:25,636 INFO spawned: 'vmalert' with pid 29
percona_1  | 2022-01-17 09:21:25,638 INFO spawned: 'alertmanager' with pid 30
percona_1  | 2022-01-17 09:21:25,655 INFO spawned: 'qan-api2' with pid 37
percona_1  | 2022-01-17 09:21:25,658 INFO spawned: 'pmm-managed' with pid 41
percona_1  | 2022-01-17 09:21:25,662 INFO spawned: 'pmm-agent' with pid 47
percona_1  | 2022-01-17 09:21:25,668 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:25,669 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:25,747 INFO exited: qan-api2 (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:26,616 INFO success: pmm-update-perform-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,621 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,627 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,632 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,632 INFO success: victoriametrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,638 INFO success: vmalert entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,638 INFO success: alertmanager entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,661 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,661 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:26,675 INFO spawned: 'postgresql' with pid 142
percona_1  | 2022-01-17 09:21:26,678 INFO spawned: 'nginx' with pid 143
percona_1  | 2022-01-17 09:21:26,692 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:26,694 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:26,754 INFO spawned: 'qan-api2' with pid 145
percona_1  | 2022-01-17 09:21:27,681 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:27,793 INFO spawned: 'grafana' with pid 186
percona_1  | 2022-01-17 09:21:27,794 INFO success: qan-api2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:21:27,979 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:28,793 INFO spawned: 'postgresql' with pid 205
percona_1  | 2022-01-17 09:21:28,796 INFO spawned: 'nginx' with pid 206
percona_1  | 2022-01-17 09:21:28,816 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:28,823 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:29,795 INFO spawned: 'grafana' with pid 207
percona_1  | 2022-01-17 09:21:29,962 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:31,883 INFO spawned: 'postgresql' with pid 237
percona_1  | 2022-01-17 09:21:31,887 INFO spawned: 'nginx' with pid 238
percona_1  | 2022-01-17 09:21:31,904 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:31,912 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:32,797 INFO spawned: 'grafana' with pid 240
percona_1  | 2022-01-17 09:21:32,972 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:36,802 INFO spawned: 'postgresql' with pid 277
percona_1  | 2022-01-17 09:21:36,805 INFO spawned: 'grafana' with pid 278
percona_1  | 2022-01-17 09:21:36,807 INFO spawned: 'nginx' with pid 279
percona_1  | 2022-01-17 09:21:36,832 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:36,842 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:36,976 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:41,096 INFO spawned: 'grafana' with pid 306
percona_1  | 2022-01-17 09:21:41,258 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:42,780 INFO spawned: 'postgresql' with pid 318
percona_1  | 2022-01-17 09:21:42,783 INFO spawned: 'nginx' with pid 319
percona_1  | 2022-01-17 09:21:42,798 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:42,804 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:46,818 INFO spawned: 'grafana' with pid 365
percona_1  | 2022-01-17 09:21:46,986 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:48,820 INFO spawned: 'postgresql' with pid 378
percona_1  | 2022-01-17 09:21:48,823 INFO spawned: 'nginx' with pid 379
percona_1  | 2022-01-17 09:21:48,841 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:48,845 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:53,826 INFO spawned: 'grafana' with pid 413
percona_1  | 2022-01-17 09:21:53,975 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:21:56,831 INFO spawned: 'postgresql' with pid 435
percona_1  | 2022-01-17 09:21:56,834 INFO spawned: 'nginx' with pid 436
percona_1  | 2022-01-17 09:21:56,851 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:21:56,854 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:22:01,834 INFO spawned: 'grafana' with pid 447
percona_1  | 2022-01-17 09:22:02,008 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:22:05,299 INFO spawned: 'postgresql' with pid 472
percona_1  | 2022-01-17 09:22:05,303 INFO spawned: 'nginx' with pid 473
percona_1  | 2022-01-17 09:22:05,319 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:22:05,321 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:22:10,265 INFO spawned: 'grafana' with pid 722
percona_1  | 2022-01-17 09:22:10,429 INFO exited: grafana (exit status 1; not expected)
percona_1  | 2022-01-17 09:22:14,851 INFO spawned: 'postgresql' with pid 843
percona_1  | 2022-01-17 09:22:14,854 INFO spawned: 'nginx' with pid 844
percona_1  | 2022-01-17 09:22:14,871 INFO exited: postgresql (exit status 2; not expected)
percona_1  | 2022-01-17 09:22:14,876 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:22:19,555 INFO spawned: 'grafana' with pid 897
percona_1  | 2022-01-17 09:22:20,603 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
percona_1  | 2022-01-17 09:22:21,878 INFO waiting for grafana to stop
percona_1  | 2022-01-17 09:22:22,629 INFO stopped: grafana (exit status 0)
percona_1  | 2022-01-17 09:22:25,140 INFO spawned: 'postgresql' with pid 966
percona_1  | 2022-01-17 09:22:25,142 INFO spawned: 'nginx' with pid 967
percona_1  | 2022-01-17 09:22:25,156 INFO exited: nginx (exit status 1; not expected)
percona_1  | 2022-01-17 09:22:25,161 INFO gave up: nginx entered FATAL state, too many start retries too quickly

Vadim_Yalovets · January 17, 2022, 9:28am

Hi Mason,

Could you create a data volume and use it with PMM server?

docker create --volume /srv --name pmm-data percona/pmm-server:2.25.0 /bin/true

Sergey_Pronin · January 17, 2022, 9:55am

@Vadim_Yalovets good point!

@Mason_Cubes the problem that you get might be caused by permissions issue on /mnt/storage/percona/srv. In case if you specify the full path docker creates a bind mount vs named volume.
But better create the volume beforehand.

Mason_Cubes · January 17, 2022, 1:06pm

@Vadim_Yalovets That was not what I am trying to do.

@Sergey_Pronin I have given very permissive 0777 to the bind mounted folder and it still does not work. I want to bind mount to an external volume.

I figured out how to do bind mount correctly with all the permission issues and how by pass. It would be nice if you specify the per folder uid:gid some where in the documentation. There is some in the restore section of the documentation, but not fully.

Sergey_Pronin · January 17, 2022, 1:49pm

@Mason_Cubes so what was the solution for you?

Mason_Cubes · January 17, 2022, 5:21pm

@Sergey_Pronin The solution was to do,

The setup as per the documentation.( a. pmm-data container b. pmm-server running of the pmm-data )
Stop and remove the pmm-server ( docker stop pmm-server && docker rm pmm-server )
docker inspect pmm-data and find where the /srv is mapped on the disk (/var/lib/docker/volumes/container_volume_id/_data)
cp -r -p mapped_path_from_above *where_you_want_to_mount_data_path* ( -p is important as the docker cp command will not preserve the access rights on the data directories in the pmm-data container.
Remove the pmm-data container (docker rm pmm-data)
Use docker run command, docker-compose file and specify your custom data directory as mount binded path.

version             : "3.8"
services            :
  
  percona           :
    image           : percona/pmm-server:2.25.0
    networks        :
                    - dbnet
    expose          :
                    - "80"
                    - "443"
    volumes         :
                    - type  : bind
                      source: /mnt/storage/percona/srv
                      target: /srv
    environment     :
                    - PMM_DEBUG=1
                    - PMM_TRACE=1
networks:
  dbnet :

That should do it nicely.

Topic		Replies	Views
Pmm2-server 2.21 docker container unhealthy PMM 2.x	9	1660	September 12, 2021
Unable to start PMM PMM 1.x	0	1182	August 18, 2019
Unable to install PMM server on ubuntu 16 Percona Monitoring and Management (PMM) pmm	1	24	July 31, 2024
PMM-2.0 beta possible bug PMM 1.x	3	926	July 22, 2019
percona/pmm-server docker container createdb processes exit status 1 PMM 1.x	0	772	July 11, 2018

Running PMM-Server in docker ends up in supervisord error

Related topics