mandm
April 29, 2014, 9:43pm
1
I am trying to archive a 21 Million row table using pt-archiver on a 512MB Amazon EC2 instance
and it is running very slow
the table is actively written into by my applications users as we are archiving it, could this be slowing down the archiving?
here are my archiving options
pt-archiver -u $MyUSER -p$MyPASS --source h=$MyHOST,D=$DEST_DB,t=$DEST_TABLE --purge --primary-key-only --why-quit --progress 10000 --statistics --charset utf8 --retries 3 --commit-each --limit 3000 --where “REQUEST_TIME <= DATE_SUB(NOW(), INTERVAL 60 DAY)”
here is the log output when i kill the job after few hours
TIME ELAPSED COUNT
2014-04-29T15:46:07 0 0
2014-04-29T16:17:21 1874 10000
2014-04-29T16:36:18 3011 20000
2014-04-29T16:55:19 4152 30000
2014-04-29T17:14:17 5289 40000
2014-04-29T17:42:40 6992 50000
2014-04-29T18:11:04 8697 60000
2014-04-29T18:33:02 10015 70000
2014-04-29T18:52:03 11156 80000
2014-04-29T19:15:55 12588 90000
2014-04-29T19:33:00 13613 100000
2014-04-29T19:57:07 15059 110000
2014-04-29T20:25:27 16760 120000
2014-04-29T20:44:56 17929 130000
2014-04-29T21:06:48 19241 140000
2014-04-29T21:35:12 20945 150000
2014-04-29T21:52:58 22011 160000
2014-04-29T22:20:35 23667 170000
2014-04-29T22:39:26 24799 180000
mandm
May 1, 2014, 4:08pm
2
Another run and same scenario, it slows down after a particular point
TIME ELAPSED COUNT
2014-05-01T12:23:05 0 0
2014-05-01T12:23:08 2 10000
2014-05-01T12:23:10 4 20000
2014-05-01T12:23:13 7 30000
2014-05-01T12:23:15 9 40000
2014-05-01T12:23:17 12 50000
2014-05-01T12:23:20 14 60000
2014-05-01T12:23:22 17 70000
2014-05-01T12:23:25 19 80000
2014-05-01T12:23:27 22 90000
2014-05-01T12:23:30 24 100000
2014-05-01T12:23:32 27 110000
2014-05-01T12:23:35 29 120000
2014-05-01T12:23:48 42 130000
2014-05-01T12:24:12 66 140000
2014-05-01T12:24:36 90 150000
2014-05-01T12:25:01 116 160000
2014-05-01T12:25:26 140 170000
2014-05-01T12:25:49 164 180000
2014-05-01T12:26:14 188 190000
2014-05-01T12:26:38 212 200000
2014-05-01T12:27:02 237 210000
2014-05-01T12:27:27 261 220000
2014-05-01T12:27:51 285 230000
2014-05-01T12:28:15 309 240000
2014-05-01T12:28:39 334 250000
2014-05-01T12:29:04 359 260000
2014-05-01T12:29:28 383 270000
2014-05-01T12:29:53 407 280000
2014-05-01T12:30:17 432 290000
2014-05-01T12:30:42 456 300000
2014-05-01T12:31:06 481 310000
2014-05-01T12:31:31 505 320000
2014-05-01T12:31:55 529 330000
2014-05-01T13:12:23 2958 340000
2014-05-01T13:54:45 5499 350000
2014-05-01T14:32:16 7750 360000
2014-05-01T15:09:48 10002 370000
2014-05-01T15:40:48 11863 380000
2014-05-01T16:18:20 14114 390000