Ubuntu 22.04, MySQL 8 and libjemalloc2

MySQL & MariaDB
1

Hello there.

I’m preparing to upgrade our Percona Server 5.7 cluster to Percona Server 8 and I figured that on the new machines, I would follow the advice in https://www.percona.com/blog/dont-start-using-your-mysql-server-until-youve-configured-your-os/

However, the Jemalloc stuff is giving me some headaches. When testing on lab servers, everything looks fine after installing libjemalloc2 (from Ubuntu repos) and Percona Server 5.7:

# dpkg -l |egrep "(percona|jem)" |awk '{print $2,$3}'
libjemalloc2:amd64 5.2.1-4ubuntu1
percona-release 1.0-27.generic
percona-server-client-5.7 5.7.44-48-1.jammy
percona-server-common-5.7 5.7.44-48-1.jammy
percona-server-server-5.7 5.7.44-48-1.jammy
percona-toolkit 3.2.1-1

# pt-mysql-summary | grep -A5 -i "memory management"
# Memory management library ##################################
jemalloc enabled in mysql config for process with id 9823
Using jemalloc from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
# The End ####################################################

# grep -i jem /proc/$(pidof mysqld)/maps
7ff2a4e71000-7ff2a4e77000 r--p 00000000 08:01 73744                      /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7ff2a4e77000-7ff2a4f14000 r-xp 00006000 08:01 73744                      /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7ff2a4f14000-7ff2a4f21000 r--p 000a3000 08:01 73744                      /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7ff2a4f21000-7ff2a4f22000 ---p 000b0000 08:01 73744                      /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7ff2a4f22000-7ff2a4f27000 r--p 000b0000 08:01 73744                      /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7ff2a4f27000-7ff2a4f28000 rw-p 000b5000 08:01 73744                      /usr/lib/x86_64-linux-gnu/libjemalloc.so.2

But, when I rip out Percona Server 5.7 and install version 8, jemalloc is no longer enabled:

# pt-mysql-summary | grep -A5 -i "memory management"
# Memory management library ##################################
jemalloc is not enabled in mysql config for process with id 12453
# The End ####################################################

After some searching, I ended up adding LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 to /etc/default/mysql and restarted mysqld, and now I get:

# grep -i jem /proc/$(pidof mysqld)/maps
7f26349a1000-7f26349a7000 r--p 00000000 fd:00 6820                       /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7f26349a7000-7f2634a44000 r-xp 00006000 fd:00 6820                       /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7f2634a44000-7f2634a51000 r--p 000a3000 fd:00 6820                       /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7f2634a51000-7f2634a52000 ---p 000b0000 fd:00 6820                       /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7f2634a52000-7f2634a57000 r--p 000b0000 fd:00 6820                       /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
7f2634a57000-7f2634a58000 rw-p 000b5000 fd:00 6820                       /usr/lib/x86_64-linux-gnu/libjemalloc.so.2

But pt-mysql-summary doesn’t say that it is enabled:


# pt-mysql-summary | grep -A5 -i "memory management"
# Memory management library ##################################
# The End ####################################################

Any ideas here? Is it enabled or not? The “system” appears to think so, but not the tool from percona-toolkit.

2

Can you share the content from

sudo systemctl edit mysql

It should have something like the below under [Service]

Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"

Also, make sure you are running pt-summary with the correct privileges user. Thanks

3

It appears to be a problem with the newer version of pt-mysql-summary in percona-toolkit-3.5.7-1 that I must have upgraded after switching to the ps-80 repo:

# pt-mysql-summary --version
pt-mysql-summary 3.5.7

# pt-mysql-summary | grep -A5 -i "memory management"
# Memory management library ##################################
# The End ####################################################

Because if I disable the ps-80 repo, uninstall percona-toolkit, switch back to the ps-57 repo and install percona-toolkit-3.2.1-1 I get:


# pt-mysql-summary --version
pt-mysql-summary 3.2.1

# pt-mysql-summary | grep -A5 -i "memory management"
# Memory management library ##################################
jemalloc enabled in mysql config for process with id 12332
Using jemalloc from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
# The End ####################################################
4

Redacting most of this reply because I apparently have a problem with reading comprehension…

Anyway, please check my reply about percona-toolkit versions above.

5

You can also check like this if Jemalloc is enabled,

cat /proc/$(pidof mysqld)/maps | grep -i jem

Can you post the output of that?

6

I did, in my first post (although I ran grep directly against maps instead of piping cat output through grep).

As I mentioned in my first reply, this appears to be an issue with the newer version of percona-toolkit. Version 3.5.7 cannot identify jemalloc as enabled, but if I downgrade to 3.2.1 it does.

7

Oh I see, I missed your first output.
I can reproduce it too.
It seems to be a bug in the newer version.
https://perconadev.atlassian.net/browse/PT-2322

8

Indeed. Thanks for reporting it. :slight_smile:

9

@GlitchCrab Please make sure to log into JIRA and ‘upvote’ and ‘watch’ the issue. Our eng teams tend to rank the next thing to work on using community feedback like this.

10

Managed to upvote it, but I couldn’t find a way to ‘watch’ it. Anyway, hope it will get fixed.

11

Oh god… I spent an entire evening experimenting with different MySQL versions. First, I added libjemalloc to my Dockerfile, then I noticed it was redundant because the official Percona Docker image already included it. I logged in here to create a new topic and found this one :melting_face:

Probable I faced the same issue. I didn’t try pt-mysql-summary 3.2.1 but everything else is the same for me:

pt-mysql-summary 3.5.5

pt-mysql-summary --host=127.0.0.1 -user=root | grep -A5 -i "memory management"
mysql: [Warning] Using a password on the command line interface can be insecure.
# Memory management library ##################################
# The End ####################################################

From container:

$ cat /proc/1/maps | grep -i jem
7fffffd2d000-7fffffd33000 r--p 00000000 00:23 1660080                    /usr/lib64/libjemalloc.so.2
7fffffd33000-7fffffda4000 r-xp 00006000 00:23 1660080                    /usr/lib64/libjemalloc.so.2
7fffffda4000-7fffffdaf000 r--p 00077000 00:23 1660080                    /usr/lib64/libjemalloc.so.2
7fffffdaf000-7fffffdb5000 r--p 00081000 00:23 1660080                    /usr/lib64/libjemalloc.so.2
7fffffdb5000-7fffffdb6000 rw-p 00087000 00:23 1660080                    /usr/lib64/libjemalloc.so.2