Mysql and pmm-client

Hello,
I am using latest PMM version 2.28, and my pmm-client is registered with pmm-server and now when I try adding services, I am facing a problem, according to the pmm-admin documentation, --ssl-ca flag is mentioned and can be used for enabling ssl for mysql users, here’s the link for the documentation I am ref to:

but when I use that --ssl-ca flag I get the following error

pmm-admin: error: unknown long flag '--ssl-ca', try --help
pmm-admin add mysql --help-long
usage: pmm-admin [<flags>] <command> [<args> ...]

Version 2.28.0

Flags:
  -h, --help                     Show context-sensitive help (also try
                                 --help-long and --help-man)
      --server-url=SERVER-URL    PMM Server URL in
                                 `https://username:password@pmm-server-host/`
                                 format
      --server-insecure-tls      Skip PMM Server TLS certificate validation
      --debug                    Enable debug logging
      --trace                    Enable trace logging (implies debug)
      --pmm-agent-listen-port=7777  
                                 Set listen port of pmm-agent
      --json                     Enable JSON output
  -v, --version                  Show application version
      --socket=SOCKET            Path to MySQL socket
      --node-id=NODE-ID          Node ID (default is autodetected)
      --pmm-agent-id=PMM-AGENT-ID  
                                 The pmm-agent identifier which runs this
                                 instance (default is autodetected)
      --username="root"          MySQL username
      --password=PASSWORD        MySQL password
      --agent-password=AGENT-PASSWORD  
                                 Custom password for /metrics endpoint
      --query-source=slowlog     Source of SQL queries, one of: slowlog,
                                 perfschema, none (default: slowlog)
      --disable-queryexamples    Disable collection of query examples
      --size-slow-logs=SIZE-SLOW-LOGS  
                                 Rotate slow log file at this size (default:
                                 server-defined; negative value disables
                                 rotation). Ex.: 1GiB
      --disable-tablestats       Disable table statistics collection
      --disable-tablestats-limit=DISABLE-TABLESTATS-LIMIT  
                                 Table statistics collection will be disabled
                                 if there are more than specified number of
                                 tables (default: server-defined)
      --environment=ENVIRONMENT  Environment name
      --cluster=CLUSTER          Cluster name
      --replication-set=REPLICATION-SET  
                                 Replication set name
      --custom-labels=CUSTOM-LABELS  
                                 Custom user-assigned labels
      --skip-connection-check    Skip connection check
      --tls                      Use TLS to connect to the database
      --tls-skip-verify          Skip TLS certificates validation
      --tls-ca=TLS-CA            Path to certificate authority certificate
                                 file
      --tls-cert=TLS-CERT        Path to client certificate file
      --tls-key=TLS-KEY          Path to client key file
      --metrics-mode=auto        Metrics flow mode, can be push - agent will
                                 push metrics, pull - server scrape metrics
                                 from agent or auto - chosen by server.
      --disable-collectors=DISABLE-COLLECTORS  
                                 Comma-separated list of collector names to
                                 exclude from exporter
      --service-name=NAME        Service name (overrides positional argument)
      --host=HOST                Service hostname or IP address (overrides
                                 positional argument)
      --port=PORT                Service port number (overrides positional
                                 argument)

Args:
  [<name>]     Service name (autodetected default: stg-sanjana102-mysql)
  [<address>]  MySQL address and port (default: 127.0.0.1:3306)

Commands:
  help [<command>...]
    Show help


  annotate [<flags>] <text>
    Add an annotation to Grafana charts

    --tags=TAGS                  Tags to filter annotations. Multiple tags
                                 are separated by a comma
    --node                       Annotate current node
    --node-name=NODE-NAME        Name of node to annotate
    --service                    Annotate services of current node
    --service-name=SERVICE-NAME  Name of service to annotate

  config [<flags>] [<node-address>] [<node-type>] [<node-name>]
    Configure local pmm-agent

    --node-model=NODE-MODEL        Node model
    --region=REGION                Node region
    --az=AZ                        Node availability zone
    --force                        Remove Node with that name with all
                                   dependent Services and Agents if one exist
    --metrics-mode=auto            Metrics flow mode for agents
                                   node-exporter, can be push - agent will
                                   push metrics, pull - server scrape metrics
                                   from agent or auto - chosen by server.
    --disable-collectors=DISABLE-COLLECTORS  
                                   Comma-separated list of collector names to
                                   exclude from exporter
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels
    --paths-base=PATHS-BASE        Base path where all binaries, tools and
                                   collectors of PMM client are located

  list [<flags>]
    Show Services and Agents running on this Node

    --node-id=NODE-ID  Node ID (default is autodetected)

  status [<flags>]
    Show information about local pmm-agent

    --wait=WAIT  Time to wait for a successful response from pmm-agent

  summary [<flags>]
    Fetch system data for diagnostics

    --filename="summary_stg-sanjana102_2022_06_15_22_51_26.zip"  
                   Summary archive filename
    --skip-server  Skip fetching logs.zip from PMM Server
    --pprof        Include performance profiling data

  add external --listen-port=LISTEN-PORT [<flags>]
    Add External source of data (like a custom exporter running on a port) to
    the monitoring

    --service-name="stg-sanjana102-external"  
                                   Service name (autodetected default:
                                   stg-sanjana102-external)
    --agent-node-id=AGENT-NODE-ID  Node ID where agent runs (default is
                                   autodetected)
    --username=USERNAME            External username
    --password=PASSWORD            External password
    --scheme=http or https         Scheme to generate URI to exporter metrics
                                   endpoints
    --metrics-path=/metrics        Path under which metrics are exposed, used
                                   to generate URI.
    --listen-port=LISTEN-PORT      Listen port of external exporter for
                                   scraping metrics. (Required)
    --service-node-id=SERVICE-NODE-ID  
                                   Node ID where service runs (default is
                                   autodetected)
    --environment=prod             Environment name like 'production' or 'qa'
    --cluster=east-cluster         Cluster name
    --replication-set=rs1          Replication set name
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels. Example:
                                   region=east,app=app1
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server.
    --group="external"             Group name of external service (default:
                                   external)
    --skip-connection-check        Skip exporter connection checks

  add external-serverless [<flags>]
    Add External Service on Remote node to monitoring.

    Usage example: sudo pmm-admin add external-serverless
    --url=http://1.2.3.4:9093/metrics

    Also, individual parameters can be set instead of --url like: sudo
    pmm-admin add external-serverless --scheme=http --host=1.2.3.4
    --listen-port=9093 --metrics-path=/metrics --container-name=ddd
    --external-name=e125

    Notice that some parameters are mandatory depending on the context. For
    example, if you specify --url, --schema and other related parameters are
    not mandatory but, if you specify --host you must provide all other
    parameters needed to build the destination URL or even you can specify
    --address instead of host and port as individual parameters.

    --external-name=EXTERNAL-NAME  Service name
    --url=URL                      Full URL to exporter metrics endpoints
    --scheme=https                 Scheme to generate URL to exporter metrics
                                   endpoints
    --username=USERNAME            External username
    --password=PASSWORD            External password
    --address=1.2.3.4:9000         External exporter address and port
    --host=1.2.3.4                 External exporters hostname or IP address
    --listen-port=9999             Listen port of external exporter for
                                   scraping metrics.
    --metrics-path=/metrics        Path under which metrics are exposed, used
                                   to generate URL.
    --environment=testing          Environment name
    --cluster=CLUSTER              Cluster name
    --replication-set=rs1          Replication set name
    --custom-labels='app=myapp,region=s1'  
                                   Custom user-assigned labels
    --group="external"             Group name of external service (default:
                                   external)
    --machine-id=MACHINE-ID        Node machine-id
    --distro=DISTRO                Node OS distribution
    --container-id=CONTAINER-ID    Container ID
    --container-name=CONTAINER-NAME  
                                   Container name
    --node-model=NODE-MODEL        Node model
    --region=REGION                Node region
    --az=AZ                        Node availability zone
    --skip-connection-check        Skip exporter connection checks

  add haproxy --listen-port=LISTEN-PORT [<flags>] [<name>]
    Add HAProxy to monitoring

    --username=USERNAME            HAProxy username
    --password=PASSWORD            HAProxy password
    --scheme=http or https         Scheme to generate URI to exporter metrics
                                   endpoints
    --metrics-path=/metrics        Path under which metrics are exposed, used
                                   to generate URI
    --listen-port=LISTEN-PORT      Listen port of haproxy exposing the
                                   metrics for scraping metrics (Required)
    --node-id=NODE-ID              Node ID (default is autodetected)
    --environment=prod             Environment name like 'production' or 'qa'
    --cluster=east-cluster         Cluster name
    --replication-set=rs1          Replication set name
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels. Example:
                                   region=east,app=app1
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server
    --skip-connection-check        Skip connection check

  add mongodb [<flags>] [<name>] [<address>]
    Add MongoDB to monitoring

    --node-id=NODE-ID              Node ID (default is autodetected)
    --pmm-agent-id=PMM-AGENT-ID    The pmm-agent identifier which runs this
                                   instance (default is autodetected)
    --username=USERNAME            MongoDB username
    --password=PASSWORD            MongoDB password
    --agent-password=AGENT-PASSWORD  
                                   Custom password for /metrics endpoint
    --query-source=profiler        Source of queries, one of: profiler, none
                                   (default: profiler)
    --environment=ENVIRONMENT      Environment name
    --cluster=CLUSTER              Cluster name
    --replication-set=REPLICATION-SET  
                                   Replication set name
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels
    --skip-connection-check        Skip connection check
    --tls                          Use TLS to connect to the database
    --tls-skip-verify              Skip TLS certificates validation
    --tls-certificate-key-file=TLS-CERTIFICATE-KEY-FILE  
                                   Path to TLS certificate PEM file
    --tls-certificate-key-file-password=TLS-CERTIFICATE-KEY-FILE-PASSWORD  
                                   Password for certificate
    --tls-ca-file=TLS-CA-FILE      Path to certificate authority file
    --authentication-mechanism=AUTHENTICATION-MECHANISM  
                                   Authentication mechanism. Default is
                                   empty. Use MONGODB-X509 for ssl
                                   certificates
    --authentication-database=AUTHENTICATION-DATABASE  
                                   Authentication database. Default is empty.
                                   Use $external for ssl certificates
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server.
    --enable-all-collectors        Enable all collectors
    --disable-collectors=DISABLE-COLLECTORS  
                                   Comma-separated list of collector names to
                                   exclude from exporter
    --service-name=NAME            Service name (overrides positional
                                   argument)
    --host=HOST                    Service hostname or IP address (overrides
                                   positional argument)
    --port=PORT                    Service port number (overrides positional
                                   argument)
    --socket=SOCKET                Path to socket
    --stats-collections=STATS-COLLECTIONS  
                                   Collections for collstats & indexstats
    --max-collections-limit=-1     Disable collstats, dbstats, topmetrics and
                                   indexstats if there are more than <n>
                                   collections. 0: No limit. Default is -1,
                                   which let PMM automatically set this
                                   value.

  add mysql [<flags>] [<name>] [<address>]
    Add MySQL to monitoring

    --socket=SOCKET                Path to MySQL socket
    --node-id=NODE-ID              Node ID (default is autodetected)
    --pmm-agent-id=PMM-AGENT-ID    The pmm-agent identifier which runs this
                                   instance (default is autodetected)
    --username="root"              MySQL username
    --password=PASSWORD            MySQL password
    --agent-password=AGENT-PASSWORD  
                                   Custom password for /metrics endpoint
    --query-source=slowlog         Source of SQL queries, one of: slowlog,
                                   perfschema, none (default: slowlog)
    --disable-queryexamples        Disable collection of query examples
    --size-slow-logs=SIZE-SLOW-LOGS  
                                   Rotate slow log file at this size
                                   (default: server-defined; negative value
                                   disables rotation). Ex.: 1GiB
    --disable-tablestats           Disable table statistics collection
    --disable-tablestats-limit=DISABLE-TABLESTATS-LIMIT  
                                   Table statistics collection will be
                                   disabled if there are more than specified
                                   number of tables (default: server-defined)
    --environment=ENVIRONMENT      Environment name
    --cluster=CLUSTER              Cluster name
    --replication-set=REPLICATION-SET  
                                   Replication set name
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels
    --skip-connection-check        Skip connection check
    --tls                          Use TLS to connect to the database
    --tls-skip-verify              Skip TLS certificates validation
    --tls-ca=TLS-CA                Path to certificate authority certificate
                                   file
    --tls-cert=TLS-CERT            Path to client certificate file
    --tls-key=TLS-KEY              Path to client key file
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server.
    --disable-collectors=DISABLE-COLLECTORS  
                                   Comma-separated list of collector names to
                                   exclude from exporter
    --service-name=NAME            Service name (overrides positional
                                   argument)
    --host=HOST                    Service hostname or IP address (overrides
                                   positional argument)
    --port=PORT                    Service port number (overrides positional
                                   argument)

  add postgresql [<flags>] [<name>] [<address>]
    Add PostgreSQL to monitoring

    --socket=SOCKET                Path to socket
    --username="postgres"          PostgreSQL username
    --password=PASSWORD            PostgreSQL password
    --database=DATABASE            PostgreSQL database
    --agent-password=AGENT-PASSWORD  
                                   Custom password for /metrics endpoint
    --node-id=NODE-ID              Node ID (default is autodetected)
    --pmm-agent-id=PMM-AGENT-ID    The pmm-agent identifier which runs this
                                   instance (default is autodetected)
    --query-source=pgstatements    Source of SQL queries, one of:
                                   pgstatements, pgstatmonitor, none
                                   (default: pgstatements)
    --environment=ENVIRONMENT      Environment name
    --cluster=CLUSTER              Cluster name
    --replication-set=REPLICATION-SET  
                                   Replication set name
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels
    --skip-connection-check        Skip connection check
    --tls                          Use TLS to connect to the database
    --tls-ca-file=TLS-CA-FILE      TLS CA certificate file
    --tls-cert-file=TLS-CERT-FILE  TLS certificate file
    --tls-key-file=TLS-KEY-FILE    TLS certificate key file
    --tls-skip-verify              Skip TLS certificates validation
    --disable-queryexamples        Disable collection of query examples
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server.
    --disable-collectors=DISABLE-COLLECTORS  
                                   Comma-separated list of collector names to
                                   exclude from exporter
    --service-name=NAME            Service name (overrides positional
                                   argument)
    --host=HOST                    Service hostname or IP address (overrides
                                   positional argument)
    --port=PORT                    Service port number (overrides positional
                                   argument)

  add proxysql [<flags>] [<name>] [<address>]
    Add ProxySQL to monitoring

    --socket=SOCKET                Path to ProxySQL socket
    --node-id=NODE-ID              Node ID (default is autodetected)
    --pmm-agent-id=PMM-AGENT-ID    The pmm-agent identifier which runs this
                                   instance (default is autodetected)
    --username="admin"             ProxySQL username
    --password="admin"             ProxySQL password
    --agent-password=AGENT-PASSWORD  
                                   Custom password for /metrics endpoint
    --environment=ENVIRONMENT      Environment name
    --cluster=CLUSTER              Cluster name
    --replication-set=REPLICATION-SET  
                                   Replication set name
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels
    --skip-connection-check        Skip connection check
    --tls                          Use TLS to connect to the database
    --tls-skip-verify              Skip TLS certificates validation
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server.
    --disable-collectors=DISABLE-COLLECTORS  
                                   Comma-separated list of collector names to
                                   exclude from exporter
    --service-name=NAME            Service name (overrides positional
                                   argument)
    --host=HOST                    Service hostname or IP address (overrides
                                   positional argument)
    --port=PORT                    Service port number (overrides positional
                                   argument)

  register [<flags>] [<node-address>] [<node-type>] [<node-name>]
    Register current Node at PMM Server

    --machine-id="/machine_id/25f35630896d4d0eaa9f112143e011a0"  
                                   Node machine-id (default is autodetected)
    --distro="linux"               Node OS distribution (default is
                                   autodetected)
    --container-id=CONTAINER-ID    Container ID
    --container-name=CONTAINER-NAME  
                                   Container name
    --node-model=NODE-MODEL        Node model
    --region=REGION                Node region
    --az=AZ                        Node availability zone
    --custom-labels=CUSTOM-LABELS  Custom user-assigned labels
    --force                        Remove Node with that name with all
                                   dependent Services and Agents if one exist
    --metrics-mode=auto            Metrics flow mode, can be push - agent
                                   will push metrics, pull - server scrape
                                   metrics from agent or auto - chosen by
                                   server.
    --disable-collectors=DISABLE-COLLECTORS  
                                   Comma-separated list of collector names to
                                   exclude from exporter

  remove [<flags>] <service-type> [<service-name>]
    Remove Service from monitoring

    --service-id=SERVICE-ID  Service ID

  unregister [<flags>]
    Unregister current Node from PMM Server

    --force                Remove this node with all dependencies
    --node-name=NODE-NAME  Node name (autodetected default: stg-sanjana102)

So is --ssl-ca flag present for pmm 2.8?
Thanks

1 Like

--ssl-ca is deprecated and you should use --tls-ca instead. The docs need to be updated. The same certs can be used; Only the flag names are changed.

1 Like

okay, so if I use --tls-ca flag I am facing an issue,

pmm-admin add mysql --username=pmm --password=pass --tls-ca=/etc/mysql/certs/ca-certs.pem --tls-cert=/etc/mysql/certs/certificate.crt --tls-key=/etc/mysql/certs/certificate.key
Connection check failed: Error 1045: Access denied for user 'pmm'@'localhost' (using password: YES).

How could I resolve this without using the “--tls-skip-verify” flag?

1 Like

You need to add the self-signed CA to your operating system’s certificate database. Using the skip-verify DOES NOT decrease security in any way.

1 Like

But my ca-certs.pem is already present in the trusted trusted list of linux OS yet this error persists ,and also I do not understand how skip-verify is considered safe because the name of the flag itself says most of it, I am new to this and it would be helpful if you help me sort out this confusion.

1 Like

skip-verify is simply skipping the check that the CA is trusted. That’s all. Here’s an example: You meet your friend at a coffee shop. You don’t need to see his ID. You know it’s him. You trust that he is who he says he is. Therefore, you can skip the ID check. Skipping the ID check of your friend changes NOTHING about the conversation you’re about to have.

The skip-verify has NOTHING, absolutely NOTHING to do with the ENCRYPTION of the connection. The connection is 100% encrypted whether or not you trust the CA. Trusting the CA or skipping the verification makes NO DIFFERENCE to the encrypted state of the data. That’s why this is still 100% safe. You created the CA. You trust it. Therefore, you can tell PMM to skip the verification because YOU trust it.

1 Like

okay got it, one of the best explanations which cleared my confusion.
Thank you

1 Like