We’ve been using xtrabackup for several years and it turned out to be a great tool. However, each time we run a full backup, this tool eats all available RAM, making other services work slowly during a high traffic period. Is there any option available like “” but when running a full and not a prepare backup? Or any other option to run this process limiting RAM usage?
I’ve tried to set ulimit -v to assign limits on virtual memory on a ssh session when running a backup with the xtrabackup tool and still eats all available RAM.
Thanks, Carlos.
Hi Carlos
Can I get some more information from you about your set up please?
- what version of Percona XtraBackup are you on? If it’s not the latest version (2.4.9) would you be able to upgrade to the latest and retest for us?
- how much RAM do you have and how are you quantifying that Percona XtraBackup is eating it up?
- what environment are you operating?
- has it ALWAYS used up all the RAM or is this something new that you are seeing?
If you could update with some of these details, I’ll see if I can get more information for you. Can I just also check if you are a Support customer? Only if that’s the case, then we probably need to raise a ticket for you with customer support (rather than the open source forum)… [url]https://www.percona.com/services/support[/url]
Thanks!
Hi Lorraine,
- We are using Percona XtraBackup v 2.4.5; yes, we could upgrade to latest version (v 2.4.9) and retest; will let you know.
- We have a total of 122GB RAM; we have assigned RAM for the InnoDB buffers, services and OS; on low traffic, we usually get to have 35GB of free RAM, but after a couple of minutes we start to run the backup process, it begins to use all of this available RAM; we wish to control the amount of RAM assigned to this process during a FULL backup so it doesn’t uses the RAM reserved for other processes.
- We are using Ubuntu 16.04.1 LTS.
- It has been using up all available RAM since we started making live backups.
I’m not a Support customer.
Thanks, Carlos.
Hi Lorraine,
About using latest version (v 2.4.9); we have upgraded from v.2.4.5 yesterday and we still have same problem with memory RAM usage during a FULL backup.
Thanks, Carlos.
Hi Carlos,
When performing a full backup, xtrabackup allocates a fixed-size buffer for each of data copying threads (–parallel option). Buffer size depends on database physical page size. If compression or encryption is used, additional buffers will be allocated for that, also depends on --parallel option value and --encrypt-theads/–compress-threads options values.
xtrabackup using up 35G of RAM looks wrong to me. If --parallel option value you are using isn’t crazy high, then there might be some issue. Can you please share VIRT and RES numbers for running xtrabackup process and exact command you are using. It would also be great if you make a note of how memory consumption is changing over time. Is it growing or xtrabackup just allocates fixed amount at the beginning and releases at the end?
Hi Sergei,
We have never used --parallel option before, so we added it (with --parallel=4) to a new backup we started on 01-17 and through a script we were able to gather information about how memory consumption is changing over time, as you suggested. We notice that xtrabackup allocates a fixed amount at the beginning, increases it during the preparation phase (maybe because of --use-memory=2G) and releases it at the end, but take a look at the available RAM during that time, 19 hours and 55 minutes in a critical state; once the backup completes, everything returns to the state of how it was before running the full backup.
Commands that we are using:
start full backup
xtrabackup --user=USER --password=PASSWORD --parallel=4 --no-timestamp /backup
prepare backup
xtrabackup --user=USER --password=PASSWORD --use-memory=2G --apply-log /backup
and, innodb_page_size it’s 16KB.
What’s your opinion about this?
Thanks, Carlos.
Hi Carlos,
It looks like xtrabackup didn’t use much memory. Rather something else used it up. Since xtrabackup does a lot of copying, it could be that Linux kernel used memory for file cache. xtrabackup is trying to prevent polluting of kernel FS cache by using POSIX_FADV_DONTNEED. It worked back when we implemented it, but still worth checking if it works in your case. You can use vmstat to see where all this memory goes.
Hi Sergei,
Will inspect in more details what the Linux kernel could be doing during that proces then. Will check if the RAM usage has to do with the File Cache. Thanks!