Good Day,
I have a K8s cluster configured in GCP and I am running pxc with 50GB of memory and 8 CPU pods.
However when I am running benchmark tests in mysql, the TPM is low and I can see all 8 cores are not being utilized in my testing, even with 20 Virtual users connected at the same time running thousand of queries.
Are there settings that I can tweak to influence how PXC/MYSQL uses the given resources?
I tried updating vars such as:
wsrep_applier_threads=8
wsrep_slave_threads=8
mysqlx_min_worker_threads = 4
The CPU resource never exceeds 3.9 CPU of 8
Any suggestions as to how to get the pxc cluster to utilize all of the resources requested?
Hello @Craig_Johnson ,
just to be sure - your container limits are set to more than 3.9 CPUs, right?
@Marco.Tusa any idea here?
@Sergey_Pronin
resources:
requests:
memory: 52G
cpu: 8
# ephemeral-storage: 1G
limits:
memory: 52G
cpu: 8
# ephemeral-storage: 1G
My disks are also SSD, so IO shouldn’t be a bottleneck.
What kind of benchmark tests? If CPU isn’t being used, and your disk I/O is low, and await% is low, then that means your benchmarks are not pushing enough traffic.
That wording makes me think 2,000-4,000 qps. That’s low. You should be in the upper tens of thousands of queries (15-19,000 qps), possibly into the 20s.
Keep increasing your benchmark until you see a bottleneck.
@matthewb
so I’m using hammerdb to benchmark pxc operator and mysql operator:
I have 20 users running 10 million queries over roughly 5mins
Mysql Operator achieves 14000 transactions per minute utilizing all the requested resources.
PXC achieves 6000 TPM with the same test, and its not using all CPU cores
So I am wondering if something is limiting PXC CPU usage?
Even query counts and drop database commands take unusually long to execute.
The Mysql Operator is running a single MySQL instance? Or a 3-node group replication?
Do you have any monitoring, like PMM, attached to all members of PXC? Where do you see bottlenecks?
Have you compared the my.cnf file of the PXC to the MySQL Operator setup? Buffer pools the same? flushing on commit the same? Sync binlog? Redo log size?
Update:
MySQL operator running on 3-node GR.
RE:PXC
I realized debug level was on ‘wsrep_debug=CLIENT’ in the mysqld configuration, which was affecting IO performance heavily.
I turned this off and I achieved 10K TPM.
Then I adjusted the below values to be 8 each:
wsrep_applier_threads=8
wsrep_slave_threads=8
With this the TPM rose to 22,0000, exceeding Mysql Operator.
Aside from what I changed with PXC, both operators are using default values for their configuration.
As you can see even though, the performance has increased significantly, the usage still seems low
PMM:
