Modifying victoriametrics configs

Artem_Timchenko1 · September 13, 2022, 1:16pm

Hi. I use AWS AMI-based installation (PMM version 2.29.1) and I’m looking for some proper and reliable way of modifying underlying pmm-managed victoriametrics config. E.g. I want to modify search.maxSamplesPerQuery property. I can obviously modify /etc/supervisord.d/victoriametrics.ini directly, but I suspect it will be overwritten during update/pmm restart.

nurlan · September 13, 2022, 2:48pm

Hello @Artem_Timchenko1 ,
you can do it by passing environment variable into our docker container with prefix VM_. Here you can see how it works VictoriaMetrics · The High Performance Open Source Time Series Database & Monitoring Solution for Victoria metrics.

Please let me know if it works for you.

Artem_Timchenko1 · September 13, 2022, 2:57pm

Unfortunately I use AWS AMI, from the AWS Marketplace, so no Docker there under the hood.

steve.hoffman · September 13, 2022, 6:31pm

Good point but all is not lost! I think you could set the variables globally in /etc/profile.d/victoriametrics.sh (new file)

with export VM_search_maxSamplesPerQuery=x as the only line in the file then restart your PMM instance so the variable will be set and VM will read it on next start. (you can do it by hand if you like by running:

. /etc/profiled.d/victoriametrics.sh
supervisorctl restart victoriametrics

but you will need ssh/shell access to your AMI instance.

Artem_Timchenko1 · September 15, 2022, 6:54am

This doesn’t work, most probably due to all the processes being daemons, while /etc/profile.d seems to be used only for login sessions.

I did some tests, will left it here, maybe would be useful for someone.
The most reliable way is to add these VM_ variables to /etc/supervisord.conf, to environmant option in [supervisord] section. This will expose these vars to all supervisord-managed services, but it shouldn’t be a big issue. I haven’t found this file to be significantly modified by pmm-update ansible playbook right now.

Another way, which is kind of baggy and hacky is based on a supervisor bag, which allow to merge configurations for programs with the same name. For example, there is a supervisord file to manage VM /etc/supervisord.d/victoriametrics.ini, which has program definition [program:victoriametrics]. We can create some file like /etc/supervisord.d/victoriametrics-env.ini and define the next config there:
[program:victoriametrics]
environment = VM_search_maxSamplesPerQuery=‘XXX’
As a result both program definitions will be merged. But as I mentioned, it’s a bug in supervisor which is discussed here. Even though it hasn’t been fixed for 7 years already, there is no guarantee things will remain the same

Matthew_Dennison · December 20, 2023, 9:19pm

Is there a ticket on the pmm side to support easier customizations like this one? perhaps maybe mounting in a config file or supporting more env vars?

Roma_Novikov · December 21, 2023, 8:59pm

@Matthew_Dennison, we are looking to rework the AMI/OVF images to be based on docker/podman inside. So the env vars vill work in this case.
Will this be a good solution for your case?

Matthew_Dennison · December 21, 2023, 9:20pm

Would it be possible to ship this as mult containers as apposed to 1 large container? There are ton of processes inside the container and some update things on the running image. Making it decomposable would allow for more flexibility. An AMI/OVF wouldn’t help the --search.maxSamplesPerQuery is defined in the victoriametrics.ini (which is created / written during startup) and cli args take priority over env vars.

github.com

VictoriaMetrics/VictoriaMetrics/blob/01f9edda64496ff897dfc9f75f4ea0865a5cebdf/lib/envflag/envflag.go#L21


      
          
          var (
          	enable = flag.Bool("envflag.enable", false, "Whether to enable reading flags from environment variables in addition to the command line. "+
          		"Command line flag values have priority over values from environment vars. "+
          		"Flags are read only from the command line if this flag isn't set. See https://docs.victoriametrics.com/#environment-variables for more details")
          	prefix = flag.String("envflag.prefix", "", "Prefix for environment variables if -envflag.enable is set")
          )
          
          // Parse parses environment vars and command-line flags.
          //
          // Flags set via command-line override flags set via environment vars.
          //
          // This function must be called instead of flag.Parse() before using any flags in the program.
          func Parse() {
          	ParseFlagSet(flag.CommandLine, os.Args[1:])
          }
          
          // ParseFlagSet parses the given args into the given fs.
          func ParseFlagSet(fs *flag.FlagSet, args []string) {
          	args = expandArgs(args)
          	if err := fs.Parse(args); err != nil {

Topic		Replies	Views
How to query data from victoriametrics PMM 2.x	6	3750	January 9, 2023
How to add job on victoriametrics PMM 2.x closed-no-reply	0	726	May 3, 2022
Data Retention of PMM running under AWS PMM 1.x pmm	1	768	August 12, 2020
PMM AWS RDS MySQL Scraping PMM 1.x	2	541	August 21, 2018
How to extend victoriametrics-promscrape.yaml PMM 2.x	6	1295	July 29, 2022

Modifying victoriametrics configs

Related topics