Pt-online schema change altering host IP; can't bypass replication checks

Polyglot Projects Percona Toolkit
1

Hello,

I’m trying to use pt-online-schema-change version 3.7.1 with MySQL 8.0.42 on AWS Aurora.

I’m having the same issue described here, where it’s changing my host string to an IP address I can’t access.

I’m tried seemingly every possible variation/combination of flags, but I can’t seem to bypass replication checks or stop it from running CHECK SLAVE STATUS; to pick up the write master host IP (which we can’t use). I can get it to work fine with --dry-run or if I resort to --force, but I’d prefer to keep the schema/data integrity checks intact.

Here are some of the flags I’ve tried, either alone or in combination with each other:

  • --recursion-method=none
  • --recursion-method=NONE
  • --recursion-method='dsn=D=percona,t=dsns' (pointing at empty table)
  • --recursion-method=hosts
  • --nocheck-replication-filters
  • --skip-check-replica-lag
  • --skip-check-slave-lag
  • --recurse=0

None of them seemed to make any difference in behavior with PTDEBUG=1, though they seemed to be getting parsed correctly. With --execute it just hangs until the connection eventually times out, though it can apparently do version checks and read variables just fine.

I would have created a ticket, but the exposed Jira portal is unresponsive to the point of being unusable.

2

Hi @Alex_Porter ,

Welcome to Percona community.

You’re right in disabling the replication check as it makes complete sense due to the way Aurora replicates.

The recursion method none should do the trick but I’d ask you to rerun the command with the options: --recursion-method=none --nocheck-replication-filters

Also, provide us with initial few lines atleast that shows the tool has identified the replicas and recusion-method flag is invalid along with full command.

Thanks,
K

3

Hey @kedarpercona , thank you for your quick reply!

I forgot to list --nocheck-replication-filters as part of my initial attempts. When I try that specific combination I get the same error.

Here’s a bit more information about exactly what I’m doing along with some debug output:

# Command
PTDEBUG=1 pt-online-schema-change \
   --recursion-method=none --nocheck-replication-filters \
   'h=aurora-mysql-cluster.cluster-*************.us-east-1.rds.amazonaws.com,P=3306,u=percona,p=***********,D=<schema>,t=<table>' \
   --alter 'FORCE' \
   --print \
   --execute

# Results 

# perl 5.042002
# Linux c95c46934ca6 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:40 UTC 2025 aarch64 GNU/Linux 
# Arguments: _[--recursion-method=none]_ _[--nocheck-replication-filters]_ _[h=aurora-mysql-cluster.cluster-*********.us-east-1.rds.amazonaws.com,P=3306,u=percona,p=********,D=<schema>,t=<table>]_ _[--alter]_ _[FORCE]_ _[--print]_ _[--execute]_

[...]

# OptionParser:607 16 Got option recursion-method = none
# OptionParser:607 16 Got option check-replication-filters = 0
# OptionParser:607 16 Got option alter = FORCE
# OptionParser:607 16 Got option print = 1
# OptionParser:607 16 Got option execute = 1

[...] // Connections work at first for version checks, etc:

# Percona::XtraDB::Cluster:8406 16 ip-<CORRECT HOST IP> SHOW VARIABLES LIKE 'wsrep_on'
# Percona::XtraDB::Cluster:8408 16 $VAR1 = undef;
# 
# VersionParser:2039 16 VersionParser got a dbh, trying to get the version
# VersionParser:2087 16 InnoDB support: DEFAULT
# VersionParser:2099 16 InnoDB version: 8.0.42
# pt_online_schema_change:9128 16 innodb_stats_persistent is ON, enabling --analyze-before-swap
# VersionParser:2039 16 VersionParser got a dbh, trying to get the version
# VersionParser:2087 16 InnoDB support: DEFAULT
# VersionParser:2099 16 InnoDB version: 8.0.42
# VersionParser:2039 16 VersionParser got a dbh, trying to get the version
# VersionParser:2087 16 InnoDB support: DEFAULT
# VersionParser:2099 16 InnoDB version: 8.0.42

// But then it checks replica status anyways and tries to use the inaccessible write master

# MasterSlave:4643 16 DBI::db=HASH(0xf6a83fba0d10) SHOW slave STATUS
# DSNParser:2187 16 Parsing h=<INACCESSIBLE WRITE MASTER IP>,P=3306

# DSNParser:2247 16 DSN string made from options: 
# DSNParser:2184 16 No DSN to parse
# DSNParser:2349 16 DBI:mysql:;host=<INACCESSIBLE WRITE MASTER IP>;port=3306;mysql_read_default_group=client   percona ********* AutoCommit=>1, mysql_enable_utf8=>0, PrintError=>0, ShowErrorStatement=>1, RaiseError=>1
4

We suggest you to remove --nocheck-replication-filters

This is not needed when you are setting --recursion-method to none. When we set --recursion-method=none, we are asking tool to not check replica at all.

Can you update your percona toolkit version to the latest and try just with --recursion-method=none?

There were some bugs related to replication connection in older version that was fixed. Just in case if you are hitting that. Always keep toolkit updated.

5

I’m afraid I get the same result with only --recursion-method=none.
Which is the most recent version? I’m using 3.7.1, but I’m not sure if it’s 3.7.1-3.

I’m installing via this endpoint to use only the online schema change tool:

wget -O /bin/pt-online-schema-change percona.com/get/pt-online-schema-change
6

The version is good.
I misread it as 3.7.0 and was referring to this bug https://perconadev.atlassian.net/browse/PT-2197
Which I think is not this case.
So, to test this further, I tried to use it on Aurora cluster with replica in different region.

Using the endpoint URL and same command as yours, I could complete the alter

No slaves found.  See --recursion-method if host ip-10-1-0-83 has slaves.
Not checking slave lag because no slaves were found and --check-replica-lag was not specified.
2026-06-22T22:57:45 Copied rows OK.
2026-06-22T22:57:45 Analyzing new table...
2026-06-22T22:57:45 Swapping tables...
RENAME TABLE `test`.`t1` TO `test`.`_t1_old`, `test`.`_t1_new` TO `test`.`t1`
2026-06-22T22:57:45 Swapped original and new tables OK.
2026-06-22T22:57:45 Dropping old table...
DROP TABLE IF EXISTS `test`.`_t1_old`
2026-06-22T22:57:45 Dropped old table `test`.`_t1_old` OK.
2026-06-22T22:57:45 Dropping triggers...
DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t1_del`
DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t1_upd`
DROP TRIGGER IF EXISTS `test`.`pt_osc_test_t1_ins`
2026-06-22T22:57:46 Dropped triggers OK.
Successfully altered `test`.`t1`.

Command used

./pt-online-schema-change \
  --recursion-method=none \
  --nocheck-replication-filters \
  h=<writer-endpoint>,P=3306,u=admin,p='<pass>',D=test,t=t1 \
  --alter 'FORCE' \
  --print \
  --execute

So the command is fine and should work.
Do you have any special characters in your password?

If yes, then add your password in between ' and check it once again.
Command ran with both cluster endpoint as well as writer endpoint and completes fine on same version 8.0.42.
Is there any external replica setup for this Aurora?

7

The password does have special chars, but I had wrapped the entire DSN string in single quotes. Specifically dollar signs in case those are particularly impactful.

I tried with just the password in single quotes like your example with the same result. For a quick sanity check, I logged in via a basic mysql client in the same container to rule out any connection issues. I’m on the same MySql version.

I don’t get any output at all without PTDEBUG=1, it just hangs.

Now that I look closer at the debug logs: It’s still running SHOW SLAVE STATUS; and using the IP address from Master_Host, but for some reason the password is mangled in this second set of connections from the dollar sign still being parsed bash style.

I can’t connect to the write master with a regular mysql client, so I think that’s ultimately the issue–not sure why it’s even running the slave status checks.

8

@Yunus Does the --recursion-method method flag have to be in a specific position among the args, or does ‘none’ have to be upper or lower case?

Following the command formatted exactly as you described, it still doesn’t seem to be honoring that option–it’s still running slave status checks like before.

Here’s a copy of the full debug output in case I missed something:

pt-online_trial1.txt (47.5 KB)

9

Hi Alex,

So we see that “No slaves found” is confirming that the replicas are not being probed. BUT itseems to be finding a master above this host?

Can you confirm if 10.179.30.23 is appearing in show replica status\G? Can you make sure of having connectivity to this host?

Can you confirm your architecture so that I don’t have to make assumptions?

Thanks,
Kedar

Cannot connect to MySQL: DBI connect(';host=10.179.30.23;port=3306;mysql_read_default_group=client','percona_tools',...) failed: Can't connect to server on '10.179.30.23' (115) at /bin/pt-online-schema-change line 2351.
18 at /bin/pt-online-schema-change line 8927.
10

@kedarpercona That’s correct–10.179.30.23 is the master IP address I mentioned in the initial issue.

It appears in SHOW REPLICA STATUS;, and I definitely do not have access to that host. Our cluster is managed by another team so I don’t have all the details or the ability to change permissions.

This is the same instance we use for writing data and DDL changes (definitely not read-only), so I’m not sure why I can’t bypass those write-master checks and tell pt-online schema change to just use that host directly. Is that not possible?

11

@Alex_Porter Thanks for sharing details.

What I think is that you are running pt-online-schema-change on a replica. Isn’t it? Which is not good and pt-online-schema-change has a guard to connect to source DSN and confirm

 9190       # Check if we are not a replica of the source server with ROW or MIXED base replication
 9191       if ( !$o->get('force') ) {
 9192          my $source = $ms->get_source_dsn($cxn->dbh(), $dsn, $dp);
 9193          if ( $source ) {
 9194             my $source_cxn = $make_cxn->(dsn => $source);
 9195 
 9196             # Check source
 9197             my $is_source_of = eval {
 9198                $ms->is_source_of($source_cxn->{dbh}, $cxn->{dbh});
 9199             };
 9200 
 9201             # We should not die if replica connected via tunnel or port redirection
 9202             if ( $EVAL_ERROR ) {
 9203                $EVAL_ERROR =~ m/The replica is connected to (\d+) but the source's port is \d+/;
 9204                if ( !$1 || $1 != $source->{P} ) {
 9205                   $is_source_of = 0;
 9206                }
 9207             }
 9208 
 9209             if ( $is_source_of) {
 9210                my $source_binlog_format = $source_cxn->dbh()->selectrow_arrayref("SHOW GLOBAL VARIABLES LIKE 'binlog_format'");
 9211                if ( uc @$source_binlog_format[1] ne 'STATEMENT' ) {
 9212                   _die("Server " . $dp->as_string($cxn->dsn())
 9213                   . " is a replica of " . $dp->as_string($source_cxn->dsn())
 9214                   . " running with binary log format "
 9215                   . "@${source_binlog_format[1]}, therefore we cannot guarantee "
 9216                   . "that all replication updates will be applied to the new table.\n"
 9217                   . "Exiting.\n"
 9218                   . "If you want to bypass this check, specify option --force.",
 9219                   NO_MINIMUM_REQUIREMENTS);
 9220                }
 9221             }
 9222          }
 9223       }
 9224

This part of code skips that check

Here is what doc mentions

–force

This option bypasses confirmation in case of using alter-foreign-keys-method = none, which might break foreign key constraints.

This option also allows to use option –where without options –no-drop-new-table and –no-swap-tables.

This option also allows to bypass the safety check that prevents the tool from running on replica that is replicating from a source with binary log format ROW or MIXED.

So we suggest you to not run it on replica only. If it uses GTID, it will create an errant GTID’s as well. Also do process on master if you are running this on replica.

Let us know if I am assuming incorrect.

12

Hi Alex,

As @Yunus noted below, you may need to use --force to skip that. Typically we avoid doing changes on replicas. We assume aim here is to defragment the table?

Yunus also mentioned about possible errant gtid risks. You might want to consider using --set-vars to disable binlogging this change.

tl;dr

  • generally avoid executing pt-osc on replicas
  • use --force
  • see if you need --set-vars

Thanks,
K

13

@kedarpercona @Yunus Thank you very much for your patience and help!

This explains the behavior much more clearly and gives me enough info to adjust our topology. It turns out we mistakenly had a stopped replication config left behind in our staging environment from an earlier server migration.