Midnight Alert Investigation: The Truth Behind 100% AWS EBS Disk IO Utilization
This article was last updated on: June 29, 2026 pm
At 3:30 AM (UTC time actually — evening for our US-based site, but normal working hours on our end, hehe), the phone rang. My heart sank: alerts at this hour are almost never good news. Sure enough, the on-call colleague reported that the application had become extremely slow, with massive request timeouts, and users were already complaining.
My first reaction: did some job kick off? Like a midnight log backup.
After checking around, I found that a log backup cronjob (compresses and writes first, then reads and transfers to S3 over the network) was indeed running. Just as I was about to blame it, I took a closer look — the log volume for this job wasn’t that large, not enough to bring down the service. But the problem was definitely happening, and that was the only obvious operation at that time.
Alright, time for the classic troubleshooting routine: check the infrastructure layer for bottlenecks first.
💥 Conventional Metrics All Normal?
Logged into the monitoring system (Prometheus + Grafana) and scanned the traditional trio:
- CPU: Low, only around 20%.
- Memory: Normal, around 50%.
- Network: Bandwidth usage around 1Gbps, while the EC2 instance’s baseline bandwidth is 3Gbps — far from saturated.
- Disk IOPS: The EC2 application uses a mounted gp3 volume with a baseline of 3000 IOPS, capable of bursting to around 12000. The read/write IOPS at the time was only around 500, well below the limit.
The data looked fine… so why was the application slow?
Wait.
There was one metric I overlooked — disk IO utilization.
│ 🐾 Note:
│ In Prometheus, node_disk_io_time_seconds or its rate() derivative (specifically: instance_device:node_disk_io_time_seconds:rate5m)
│ reflects the proportion of time the disk spends completing IO requests — think of it as the disk’s ‘busyness level’.
│ If this value stays at 100%, it means the disk is running at full capacity, IO requests are queuing up, and of course the application will lag.
I quickly checked this metric, and sure enough — straight to 100%!
This was contradictory: IOPS and network bandwidth weren’t saturated, so why was the disk at 100% busy?
🔍 The Hidden Culprit: gp3 Volume Throughput Limit
That’s when something clicked. I took a closer look at the network traffic and disk metrics.
- Network traffic: Calculated via node_network_receive_bytes_total, approximately 1Gbps.
- Disk throughput: Calculated via node_disk_bytes_total, also approximately 1Gbps — almost perfectly matching the NIC traffic.
Wait, what’s the throughput ceiling for a gp3 volume?
AWS documentation states it clearly: gp3 volume throughput is priced separately, with a baseline throughput of 125 MB/s (approximately 1 Gbps). Unlike gp2, if your IOPS and throughput exceed the baseline, you can use burst credits, but once credits are exhausted, throughput gets hard-capped at 1 Gbps.
The gp3 volume throughput ceiling defaults to 125 MB/s (approximately 1 Gbps) with no bursting capability. Unless you pay extra for additional throughput upfront.
│ 🤔 NIC traffic at 1 Gbps, disk throughput at 1Gbps? This has clearly hit the gp3 volume’s 1Gbps throughput limit.
The truth is simple: The log backup job generated massive disk writes/reads, instantly consuming the gp3 volume’s throughput capacity, causing all subsequent IO requests to be throttled. The application’s business requests didn’t have high IOPS, but they carried significant data volumes and got stuck in the queue. Disk IO utilization at 100% wasn’t due to insufficient IOPS — it was throughput being throttled.
Think of it this way: a delivery station (gp3 volume) has a fixed processing speed (throughput), and suddenly a huge truckload of packages arrives (log backup), occupying the entire conveyor belt. The VIP parcels (business requests) that come after can only wait in line, naturally causing timeouts.
🛠️ How to Confirm This Hypothesis?
To verify, I quickly checked the following:
- Compare network ingress traffic with disk write volume: Confirmed both were highly consistent, both reaching 1Gbps. This proves disk writes and network traffic are ‘the same flow’.
- Confirm gp3 volume throughput configuration: Checked the console and confirmed the volume’s throughput was indeed configured at the default 125MB/s (1Gbps).
🤔 What Did This Investigation Teach Us?
👍️ What We Did Well
- Comprehensive monitoring coverage: We deployed node_exporter, collecting disk IO utilization, IOPS, network bandwidth, and virtually all critical metrics — no blind spots.
- Checked all conventional metrics: Immediately investigated CPU, memory, IOPS, and bandwidth — a perfectly correct first response.
- Strong data correlation skills: Quickly comparing network traffic with disk throughput and spotting the matching values was the most critical step.
👎 Areas for Improvement
- Insufficient understanding of monitoring metrics: Only focused on whether “IOPS” and “bandwidth” were saturated, while overlooking IO utilization — the key metric that reveals “whether the application is queuing for IO”.
- Unfamiliar with AWS volume type characteristics: Although we eventually identified gp3 throttling, we lacked systematic knowledge of the IOPS/throughput/credit mechanisms across different volume types (gp3/gp2, etc.) during the investigation. It was gp3, and it truly burned us.
- 😱 This is why a “volume performance cheat sheet” is so important.
- Incomplete troubleshooting workflow: After “conventional metrics look normal”, we should have immediately pivoted to disk IO queue depth (instance_device:node_disk_io_time_weighted_seconds:rate5m) — the core metric for determining whether IO is truly ‘congested’ — rather than only looking at utilization.
💡 Final Thoughts: A Practical EBS Performance Troubleshooting Checklist
This case made me realize that many pitfalls in ‘cloud native’ and ‘cloud computing’ stem from a superficial understanding of the underlying infrastructure. For future investigations of similar issues, I recommend following this sequence:
- Observe symptoms: Application slow, timeouts.
- Scan hardware: CPU, memory, IOPS, bandwidth.
- ⚠️ If all these metrics are normal, immediately proceed to step three.
- Check IO queue: Examine the instance_device:node_disk_io_time_weighted_seconds:rate5m metric to determine if there’s IO congestion.
- If the IO queue is high but IOPS is low, it’s most likely a throughput or IO pattern issue.
- Compare traffic: Check NIC traffic against disk write/read throughput to see if they match.
- Verify hypothesis:
- Confirmed EBS throttling? Based on the EBS type, check either the BurstBalance metric or the IOPS and throughput metrics.
- Confirmed application IO pattern issue? Analyze which specific file’s reads/writes are causing high IO (e.g., logs in /var/log).
- Adjust configuration:
- EBS throttling: Upgrade volume type (e.g., gp3 → io2) or increase the throughput baseline.
- Application issue: Optimize log output (async, batching, reduce log level), adjust cronjob scheduling or stagger execution, throttle log backup, etc.
🤔 Further Thoughts on Cloud Services
After this incident, I specifically researched related materials and discovered that cloud service limitations go far beyond this.
While cloud services bring elasticity and scalability, network throttling (bandwidth limiting) is often overlooked and is a significant cause of slow application responses and stuttering. Even when CPU, memory, and other metrics appear normal, network throttling can dramatically increase latency through packet loss and retransmissions, causing service outages, performance instability, and potentially millions of dollars in lost revenue.
Core Issues
- Cloud providers (AWS, Azure, GCP, etc.) set bandwidth baselines based on instance size, triggering automatic throttling when exceeded (e.g., AWS reduces speed after burst credits are exhausted).
- Throttling rules are complex and opaque; traditional monitoring (e.g., measuring only RTT) cannot effectively detect them. NTP precision is insufficient, and PTP is costly.
Common Scenarios and Solutions
- Sustained high utilization: Upgrade instances, distribute load, implement intelligent caching.
- Microbursts: Application-level rate limiting, traffic shaping, high-granularity monitoring.
Future Strategies
- Implement real-time one-way latency monitoring.
- Design architectures for resilience, tolerating bandwidth limits.
- Balance instance size with network requirements to avoid over-provisioning.
- Establish performance baselines and proactively test.
- Core shift: Move from reactive auto-scaling to proactive automated latency self-healing, treating network throttling management as a business necessity rather than just an IT concern.
📝 Summary
When troubleshooting application issues, don’t just look at “how much water there is (IOPS)” — pay more attention to “how fast the water flows (throughput)” and “whether the pipe is clogged (IO utilization)”.