My REGXA VPS Had 94% CPU Steal With Zero Traffic
I am not writing this as a negative review of REGXA. I am not trying to tell anyone whether they should or should not use the provider. This is simply one of my days as a developer. I moved a normal workload to a KVM VPS with: 2 vCPU 2 GB RAM 60 GB NVMe Nginx was running. The backend was running. The machine was reachable. But it behaved as if it were completely overloaded. Requests accumulated. TLS operations became slow. Connections stayed open for far too long. Some requests eventually returned HTTP 504 after more than two minutes. My first assumption was simple: something inside my VPS was consuming all the CPU. Then I ran mpstat . Average CPU steal: 92.58% CPU 0 steal: 90.62% CPU 1 steal: 94.57% CPU user: 4.02% CPU system: 1.47% CPU iowait: 0.41% CPU idle: 0.53% That changed the investigation completely. My application was not consuming 90% of the processor. It was barely getting CPU time at all. CPU usage and CPU steal are very different things If an application is genuinely CPU-bound, I expect something like this: user: 80% system: 10% steal: 0% idle: 10% The VM is receiving CPU time and spending it on application or kernel work. My VPS looked roughly like this: user: 4% system: 1% steal: 93% idle: 1% %steal is time during which a virtual CPU is ready to run but the hypervisor does not schedule it. So when I say that CPU time was being taken away from the VPS, I mean it in that specific virtualization sense. I cannot prove that REGXA deliberately took physical cores assigned exclusively to me and gave them to another customer. Guest metrics cannot prove that. They also cannot prove intent. What they can show is much simpler: the VM had runnable work, but its vCPUs were repeatedly not being scheduled. Individual samples kept landing in the same range: 89.85% steal 92.16% steal 96.48% steal 94.87% steal 95.69% steal 95.95% steal 94.57% steal 93.75% steal CPU pressure pointed in the same direction: CPU PSI some avg10: 79.47 CPU PSI some avg60: 75.32 CPU PSI some avg300: 76.31 Load average: 5.85 / 5.75 / 5.73 This was a two-vCPU machine. There was clearly runnable work waiting for processor time. But I still wanted to rule out the most obvious explanation: my own workload. I removed all production traffic I moved the active workload to another server. Then I stopped all production traffic to the affected VPS. The request queues cleared. I repeated the CPU test. The result: Average CPU steal: 94.17% CPU 0 steal: 95.56% CPU 1 steal: 92.83% CPU user: 1.95% CPU system: 0.59% CPU iowait: 0.59% CPU idle: 2.18% CPU steal actually became worse. The important combination was: user: 1.95% system: 0.59% steal: 94.17% My applications were almost idle. Yet the VPS was still spending roughly 94% of measured CPU time waiting for the hypervisor. vmstat samples were still showing: 91% steal 98% steal 98% steal 97% steal 97% steal 95% steal 96% steal 95% steal At some points, as many as 15 runnable processes were waiting for CPU. That was the strongest experiment in the entire incident. Once the workload was gone, application CPU consumption could no longer explain what I was seeing. localhost was taking seconds Next I wanted to remove the external network from the equation. So I tested HTTPS through: 127.0.0.1 That removes: - public DNS; - my ISP; - geographic distance; - internet routing; - upstream transit; - the public network path to the server. The client and server were inside the same VM. Under production traffic, I ran ten localhost HTTPS requests. Four failed during the TLS handshake. The successful ones took: 29.30 s 22.77 s 12.25 s 11.87 s 11.12 s 9.40 s Some TLS handshakes alone took around nine seconds. For localhost. After removing production traffic, localhost improved, but remained extremely inconsistent: 0.061 s 0.745 s 0.830 s 0.873 s 1.010 s 1.117 s 1.121 s 2.188 s 3.355 s The same local operation could take: 61 ms and then more than: 3.3 seconds on another attempt. That is not the kind of result I would explain with public networking. The rest of the stack started backing up The application-level symptoms were also visible in socket state. At one point I had approximately: 450 established connections 122 orphaned connections 110 FIN-WAIT-1 33 CLOSE-WAIT The backend listen queue on localhost reached around: 14-15 The HTTPS listen queue reached around: 40 Nginx logged requests like: 142.857 s -> HTTP 504 138.902 s -> HTTP 504 135.064 s -> HTTP 504 129.819 s -> HTTP 504 128.657 s -> HTTP 504 Other requests remained open for roughly 67-130 seconds. I also saw: secure-connect timeout: ~35 s secure-connect timeout: ~41 s database network timeouts delayed TLS operations Looking at these individually could have sent me in completely different directions. A 504? Maybe Nginx. Database timeout? Maybe MongoDB. Slow TLS? Maybe networking. Growing queues? Maybe backend concurrency. But all of those systems depend on the process being scheduled. Increasing a timeout does not create CPU time. It only allows the request to wait longer. RAM and disk were not the bottleneck Memory looked fine: Total RAM: ~1.9 GiB Available RAM: ~1.0-1.1 GiB Swap used: ~1-2 MiB OOM events: none OOM killer: none Storage looked fine too: Filesystem: ~58 GB Used: ~11 GB Available: ~44 GB Usage: ~20% During the important CPU measurements, I/O wait remained below 1%. So I did not have: - exhausted memory; - heavy swapping; - OOM activity; - a full filesystem; - enough disk wait to explain the behavior. The number that kept dominating the system was still: %steal I compared it with another KVM VPS A useful diagnostic needs a control. So I ran the same style of test on another KVM VPS that was actively handling normal traffic. It reported: Average CPU steal: 0.02% CPU idle: 87.86% CPU PSI avg10: 0.29 CPU PSI avg60: 0.63 CPU PSI avg300: 0.49 Load average: 0.47 / 0.33 / 0.14 Ten localhost HTTPS requests completed between roughly: 37 ms and 69 ms Later, I tested another production VPS that genuinely was CPU-busy: CPU user: 61.71% CPU system: 5.08% CPU idle: 24.09% CPU steal: 0.13% That comparison made the difference very clear. Busy because my application is running user: high system: some steal: low Busy because the guest is not getting scheduled user: low system: low steal: extremely high Both can result in a slow website. But they require completely different debugging. REGXA later confirmed resource contention Until this point, I only had guest-side evidence. I could see what Linux was reporting. I could not see: - the physical host; - total host CPU load; - CPU quotas; - scheduler weights; - neighbouring VMs; - the provider's oversubscription level. REGXA could. Support eventually told me the VPS was running on: shared CPU infrastructure They explained that CPU resources were shared across multiple virtual machines and that performance could vary with physical-node load. More importantly, they explicitly said that the elevated CPU steal I was observing resulted from: resource contention on the underlying infrastructure They also said the Frankfurt infrastructure was experiencing particularly high demand. According to support, they could not change the CPU quotas or scheduling policy or provide additional dedicated CPU resources for that shared VPS. The proposed solution was to move the VPS to a less-loaded location. At that point, resource contention was no longer only my interpretation of mpstat . It was also the provider's explanation. The CPU wording was difficult to reconcile This part bothered me. The REGXA VPS material I had seen used wording around: Dedicated CPU Cores and guaranteed resources. But support described my VPS as using shared CPU infrastructure whose performance depended on physical-node load. I do not know the exact host configuration. I cannot tell whether the immediate mechanism was: - CPU overcommitment; - CPU quotas; - scheduler weighting; - throttling; - another hypervisor configuration; - or some combination. And again, I cannot prove intent. What I can say is: Under workload: CPU steal = 92.58% With production traffic removed: CPU steal = 94.17% And REGXA itself later attributed the elevated steal to resource contention on its shared infrastructure. That is enough evidence for me to document what happened without pretending I know more than I do. The refund became another part of the incident Once the infrastructure issue was clear, I did not want to keep moving the VPS around and testing different nodes. I wanted to cancel it. Initially, I was offered only a partial refund. And the proposed refund was not back to the card I had used. It was REGXA account credit. For me, provider credit is not the same thing as a refund if I am trying to stop using the provider. So I kept replying. I asked for: 100% of the payment to be returned to: the original payment method It took additional back-and-forth. Eventually, REGXA agreed and refunded the full amount to the original payment method, describing the full refund as an exception. I appreciate that they eventually returned everything. But having to keep pushing for the financial resolution after the infrastructure problem had already been acknowledged is also part of the experience. The debugging lesson was more useful than the hosting story The easiest mistake here would have been to keep optimizing my application. I could have: - increased Nginx timeouts; - changed keepalive settings; - reduced concurrency; - added retries; - tuned connection pools; - investigated MongoDB; - rewritten backend code. Some of those changes might have altered individual symptoms. None would have answered the real question: Why is the guest spending more than 90% of its CPU time waiting for the hypervisor? That problem lives below the application layer. My VPS smoke test now includes %steal I no longer consider a new VPS healthy just because: SSH works Nginx starts /health returns 200 I also check: mpstat -P ALL 1 20 vmstat 1 20 cat /proc/pressure/cpu And for a web server, I test localhost laten
Comments
No comments yet. Start the discussion.