Back to overview
Degraded

Global Network Degraded

Aug 27 at 12:29am CEST
Affected services
Equinix AM5 - Amsterdam

Status Report Update State Resolved
Aug 27 at 02:30am CEST

The issues have been resolved, a detailed statement can be found below:

At 00:00 local time in Amsterdam, we identified widespread network problems. Upon conducting a thorough investigation, we noticed that these issues were isolated to the Amsterdam Location. Once this isolation was known, our immediate actions involved troubleshooting network configurations within our core routers. Simultaneously, we engaged in discussions with our upstream providers to ascertain the presence of any potential problems on their end.

After dedicating approximately 2 hours to network diagnostics, during which we meticulously examined configurations and errors across every individual rack, we successfully pinpointed the root cause. The disruption had originated from a bug within the Arista firmware, specifically related to GRE (Generic Routing Encapsulation). Notably, the network had undergone a GRE deployment for a customer project, with these alterations being implemented at 9:00 PM Amsterdam time. While the modifications themselves were innocuous and theoretically shouldn't have led to any disturbances, it appears that an inherent bug within the Arista firmware was triggered, resulting in the observed disruptions.

Promptly responding to the situation, we reverted the changes made at 9:00 PM, which led to an improvement in the network's performance and a return to normal traffic levels by 2:00 AM local time.

We will report this bug to Arista for a thorough investigation. As of now, the network has been restored to a stable state, and we are confident that measures are in place to prevent a recurrence of this issue.

It's worth highlighting that, even during this incident, the network exhibited a degree of resilience and managed to remain partially operational.

We sincerely apologize for any inconvenience this situation may have caused and trust that this explanation provides a transparent insight into the cause of the issue and how it was resolved and prevented from happening again.

Status Report Update State Updated
Aug 27 at 02:16am CEST

We have identified the cause of the issue and have made changes to resolve the issue, we are monitoring the network to confirm that the problem is actually resolved.

Status Report Update State Updated
Aug 27 at 01:07am CEST

We have identified that the issues are only being experienced in Amsterdam, we are working with upstream providers to resolve the issues.

Status Report Update State Created
Aug 27 at 12:29am CEST

We are currently experiencing global network issues and are investigating the issues, The network is still online but performance is degraded.