Post-Incident Report: AMS VPS Cluster Outage

Date: 21st of August, 2025
Impact Period: ~3:00 PM – 4:08 PM CEST

At approximately 3:00 PM CEST, our AMS VPS cluster experienced an outage due to an unexpected loss of power on the A feed. Under normal conditions, the B feed should have automatically taken over, but this failover did not occur. Upon investigation by our on-site technician, we discovered that the B feed was overloaded. This issue was traced back to an error in the PDU reporting software, which incorrectly displayed the A and B groups as swapped in the monitoring output. This misreporting prevented proper balancing of the feeds.

Our on-site technician quickly re-balanced the power, allowing compute nodes to come back online. AMS1–6, 8–12 were restored within ~12 minutes, AMS7 required additional time, returning to full service at 4:08 PM due to unexpected OS behavior, which our team promptly debugged and resolved.

Following service restoration, we conducted a full audit of power distribution across all racks. During this review, four additional racks were found to be misreporting feed through snmp. These were immediately corrected to ensure accurate monitoring and to prevent recurrence of a similar issue.

We sincerely apologize for the inconvenience caused by this outage. Affected customers are eligible for service compensation; please contact our support team to arrange this. If you have any further questions or concerns, don’t hesitate to reach out

AMS07 has recovered as well, and an after-incident report will follow later.

If you are still experiencing issues with your service, please reach out to support.

We are working on resolving an issue with VPS node AMS07, all other nodes have recovered and are back online.

Most VPS nodes are back online.
If your VPS did not come online automatically you can boot it manually via the VPS manager.

We're currently resolving an outage on our VPS nodes, they should come back online shortly.