A new Google cloud outage struck the organisation, as Google Compute Engine (GSE) users experienced a brown out over the weekend, not unlike the service failure in February.
A brownout is defined as either an intentional or unintentional voltage drop in electricity.
Starting at 9:44 AM on Saturday 7th March, the outage lasted for 43 minutes and was caused by packet loss on engress network traffic. Thankfully for Google and its users, the incident was not as severe as February’s failure.
Users reported a variety of symptoms, including unusually slow responses and time- outs when attempting to connect to virtual machines.
Google provided the following explanation:
“The root cause of the packet loss was a configuration change introduced to the network stack designed to provide greater isolation between VMs and projects by capping the traffic volume allowed by an individual VM.
The configuration change had been tested prior to deployment to production without incident. However as it was introduced into the production environment it affected some VMs in an unexpected manner.”