An interesting day today as we found out.

GCP decided to have an off day and part of the London region decided to die on us. Part of our GKE cluster decided to die and terminate, degrading some of our services.

We also have a Mongo Atlas cluster which sits outside of our account, but also runs in GCP, and one of the nodes there also decided to die. Mongo Support are trying to recover it but are also pointing at Google’s outage as cause.

As of the time of writing, most things are back up, but the Mongo node (which, thankfully, is just an Analytics node, meaning it doesn’t serve live traffic) is the only node that is having trouble.

The issue is still running at this time and Atlas is showing

The Cloud Provider you’ve chosen is out of capacity for the selected database deployment configuration. We will continue to try and deploy on the chosen Cloud Provider.
If you keep experiencing issues, we recommend choosing a different Cloud Provider, region or instance size, and deploying your database again.

So we can’t even do anything about it.

The Register has an article about the outage too, and GCP’s status didn’t show any issues until around 9am — 2 hours after we started seeing issues.