problems with xtrabackup incrementals

We are having problems with incremental backups using xtrabackup.

In short, we are running the Percona server and trying to use innobackupex to make a daily full backup of all of our databases, then we would like to use innobackupex (fix released versions) or xtrabackup directly to make incremental backups.

rpm -qa | grep -i percona

Percona-Server-shared-compat-5.1.43-3
Percona-Server-server-51-5.1.47-rel11.2.53.rhel5
Percona-Server-client-51-5.1.47-rel11.2.53.rhel5

We have used various versions/revisions of the xtrabackup RPM taken from the Percona.com site (from 1.2.22 to version 156) and run into the following problem:

We follow the simple instructions from the “How To” guide on the xtrabackup page on the percona wiki:

innobackupex-1.5.1 /mnt/dbbackups/raw/

for the full backup, then either

xtrabackup --backup --target-dir=/mnt/dbbackups/raw/incremental/yyyy-mm-dd_hh-mm -ss --incremental-basedir=/mnt/dbbackups/raw/incremental/YYYY-MM -DD_HH-MM-SS

or

innobackupex-1.5.1 --incremental --incremental-basedir=/mnt/dbbackups/raw/YYYY-MM-DD_HH-MM-SS /mnt/dbbackups/raw/incremental

where YYYY-MM-DD_HH-MM-SS was assigned by innobackupex during the full backup and yyyy-mm-dd_hh-mm-ss is assigned by us (in the latter case, the timestamp-named directory for the delta files is generated by innobackupex)

Instead of just getting *.delta files for just the ibd files that were actually changed (on two small tables in two very small test databases), we get a 16K delta file for nearly every ibd file (tens of thousands of them) generated and copied (innodb_file_per_table). that happens in both cases.

If we use innobackupex, it finishes the xtrabackup portion of generating all these delta files, then prints messages that it’s about to start copying the other file types but instead craps out with a broken pipe error (I can get you the exact error next time i run another one as these are taking hours to complete).

I have scoured the bugs lists on the google groups pages and searched the percona forums and find no reference to these.

Is there some flag i’m leaving out or that was left out of the basic instructions on in the “how to” guide in the xtrabackup wiki on percona.com that is causing gigabytes of these delta files to be generated, turning what should be a small fairly quick process into a long, drawn out mess?

Thanks.

M Brian Ladner

EDIT: As an additional test I ran a full innobackupex on our database server (305GB, 3.5 hours), then immediately ran an incremental without making any changes to any tables. again, 16KB delta files for every ibd table are being copied into the incremental directory.

is this normal behavior? is it necessary or is there a workaround as it takes hours to run the incremental as well?