Amazon RDS MySQL: 99% system CPU, mysqld unexpectedly restarted, repeated failures resolved after Stop/Start — what could cause this?

MySQL & MariaDB Other MySQL® Questions
1

Hello,

I’m investigating a production incident involving Amazon RDS for MySQL and would appreciate help identifying its possible root cause.

On October 14, 2026, our RDS instance experienced a sudden and severe performance degradation. The database became unresponsive, causing our web application to exhaust all 200 Apache workers.

The most unusual finding is that OS-level CPU usage (system) reached approximately 99% for several minutes, while normal MySQL query processing appeared to stop.

Environment

  • Region: sa-east-1 (São Paulo)

  • Database engine: Amazon RDS for MySQL 8.4

  • Instance class during the incident: db.m6g.xlarge (4 vCPUs, 16 GiB RAM)

  • Application: Laravel/PHP running on EC2 Debian 11 with Apache prefork

  • Enhanced Monitoring: enabled

  • Database class is changed automatically three times daily to optimize costs. The morning change to db.m6g.xlarge occurs at 06:30 (UTC-3).

Incident timeline

All times below are São Paulo local time (UTC-3).

Time Observation
08:25:54 CPU system 1.5%, iowait 0.8%; normal operation
08:26:54 iowait increased to 13.3%
08:27:54 iowait reached 61.7%; CPU system 37.7%; network traffic dropped sharply
08:28:40 Apache reached MaxRequestWorkers=200
08:28:54 CPU system increased to 83.8%
08:29:54 CPU system reached 99.1%
08:30:54 CPU system 99.2%
08:31:54 CPU system 99.3%
08:32:54 CPU system 99.4%; mysqld was still present in process monitoring
08:33:25 mysqld no longer appeared in the process list; memory usage dropped sharply
08:33:43 MySQL error log recorded MySQL Server - start
08:36:03 MySQL reported mysqld: ready for connections

Important findings

1. Very high system CPU, not user CPU

At 08:30, Enhanced Monitoring reported:

  • os.cpuUtilization.system.avg: 99.2%

  • os.cpuUtilization.user.avg: 0.7%

  • os.cpuUtilization.wait.avg: 0.1%

This suggests that the CPU was predominantly consumed in kernel mode rather than normal MySQL query execution.

2. The operating system did not appear to reboot when mysqld restarted

Enhanced Monitoring reported:

  • 08:32:54 — OS uptime: 02:02:32

  • 08:33:25 — OS uptime: 02:03:03

The OS uptime continued increasing normally, even though the MySQL process subsequently restarted.

3. No manual reboot was requested during the initial incident

We did not manually reboot RDS at 08:33–08:36.

Our CloudTrail searches did not identify a RebootDBInstance API call or another RDS management event explaining this initial restart.

4. Swap was not in use

Enhanced Monitoring reported swap.total = 0, and the CloudWatch SwapUsage metric showed zero in the available samples.

FreeableMemory was low before the incident, but we have no evidence confirming an out-of-memory kill.

5. The incident occurred repeatedly

After the database recovered at approximately 08:36–08:39, similar failures occurred two or three more times between approximately 09:00 and 10:20.

Manually rebooting the RDS instance temporarily restored service. However, the database became unresponsive again after several minutes or tens of minutes.

6. Stop/Start provided a much longer recovery

We eventually stopped the RDS instance completely and started it again.

The procedure took approximately 30 minutes.

After this full Stop/Start cycle, our application remained stable for the entire afternoon, without further observed incidents.

I understand that Stop/Start and Reboot are different RDS operations, and that Stop/Start reprovisions the underlying compute instance. However, I cannot confirm whether an underlying physical host was changed.

Questions

  1. What could cause Amazon RDS MySQL to transition from high iowait (61.7%) to almost 100% CPU system usage within approximately two minutes?

  2. Could this behavior indicate a kernel-level issue, storage subsystem problem, memory-management issue, MySQL internal problem, or RDS infrastructure issue?

  3. What could cause mysqld to disappear and restart while the operating system uptime continues normally, without an identified user-initiated reboot?

  4. Why might repeated RDS reboots temporarily resolve the issue, while a complete Stop/Start provides sustained stability?

  5. Is there a way, using Enhanced Monitoring, CloudWatch Logs, Database Insights, or other tools available without a paid AWS Support plan, to determine whether mysqld was terminated by an internal watchdog, OOM killer, crash, or RDS management action?

  6. Could the automated DB instance class modification at 06:30 have contributed to the problem approximately two hours later, even though the instance initially operated normally?

  7. What additional metrics or diagnostic information should I preserve to investigate the root cause and prevent recurrence?

Available evidence

I have collected:

  • RDS Enhanced Monitoring (RDSOSMetrics) JSON covering the incident

  • CPU and I/O metrics from Database Insights

  • RDS MySQL error logs

  • CloudTrail event history

  • CloudWatch metrics, including database connections, memory and swap

  • Apache access/error logs showing application degradation

I can share sanitized excerpts or specific metric values if needed.

My main goal is to understand why the RDS instance experienced this severe OS-level CPU condition and why MySQL needed to restart, rather than simply mitigating the resulting Apache worker exhaustion.

Any technical guidance or similar experiences would be greatly appreciated.

Thank you!

2

Hi, sorry to hear you are having issues. Unfortunately RDS can be a black box sometimes. Have you tried opening a support ticket with AWS? they may be able to explain why the surge in %sys. Hard to say without direct access to the underlying operating system.

3

Thank you for your response!

Unfortunately, the company I work for doesn’t have a paid AWS Support plan, so we’re unable to open a technical support ticket.

I understand that identifying the root cause without access to the underlying operating system is difficult.

I appreciate your time and feedback!

4

| 08:27:54 | iowait reached 61.7%; CPU system 37.7%; network traffic dropped sharply |

This makes me think the underlying storage system had issues. I believe most RDS uses EBS as data volumes, which is network-based. If iowait increased, and network traffic dropped, that sounds like the EBS volume disappeared, or otherwise had issues.

Database class is changed automatically three times daily to optimize costs.

That is highly unusual. Every time this happens, your instance is being moved around to different physical hosts within AWS. How much money are you actually saving each month by changing the instance type 3x day?

5

Thank you for your analysis!

Your hypothesis about a possible EBS storage issue makes sense, especially considering the sudden increase in iowait.

Regarding the instance class changes, we automatically resize our RDS instance three times a day to match business hours and reduce costs. This strategy has resulted in over 30% savings on our RDS costs.

We’ve been using this approach, but I understand your concern about the frequent changes in the underlying infrastructure.

Interestingly, several RDS reboots only provided temporary recovery, while a complete Stop/Start resolved the issue, and the database remained stable afterward.

Unfortunately, we don’t have a paid AWS Support plan to investigate the underlying infrastructure.

Thanks again for your insights!