Tag Archives: Outage

Google Cloud Outage

Leave a reply

An interesting day today as we found out.

GCP decided to have an off day and part of the London region decided to die on us. Part of our GKE cluster decided to die and terminate, degrading some of our services.

We also have a Mongo Atlas cluster which sits outside of our account, but also runs in GCP, and one of the nodes there also decided to die. Mongo Support are trying to recover it but are also pointing at Google’s outage as cause.

As of the time of writing, most things are back up, but the Mongo node (which, thankfully, is just an Analytics node, meaning it doesn’t serve live traffic) is the only node that is having trouble.

The issue is still running at this time and Atlas is showing

The Cloud Provider you’ve chosen is out of capacity for the selected database deployment configuration. We will continue to try and deploy on the chosen Cloud Provider.
If you keep experiencing issues, we recommend choosing a different Cloud Provider, region or instance size, and deploying your database again.

So we can’t even do anything about it.

The Register has an article about the outage too, and GCP’s status didn’t show any issues until around 9am — 2 hours after we started seeing issues.

Google’s Catch-22

Not often I post on problems at Google, but this is actually an interesting situation.

https://arstechnica.com/?p=1518703

Google had an outage the other week, and it knocked out several websites GitLab, Shopify and impacted others. Gsuite, Gmail, YouTube were affected, but not down.

There are some interesting lines in this article:

for an entire afternoon and into the night, the Internet was stuck in a crippling ouroboros: Google couldn’t fix its cloud, because Google’s cloud was broken.

Google says its engineers were aware of the problem within two minutes. And yet! “Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network,”

In short, Google Cloud broke due to congestion, Google couldn’t fix the problem because their tools required using the network that was now congested