Google Cloud Report Reveals Accidental Deletion of Customer Data

Google Cloud has publicly addressed an incident in which a misconfiguration during the setup of a Google Cloud VMware Engine (GCVE) private cloud led to the unintended deletion of Australian customer UniSuper’s data, including offsite backups.

The incident, stemming from a blank parameter left by Google operators during a deployment using an internal tool, resulted in the automatic deletion of UniSuper’s GCVE private cloud after a system-assigned one-year term.

“During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period,” reads the statement.

UniSuper, a major Australian university retirement fund managing over $125 billion in assets, experienced temporary disruption to its services due to the incident. However, thanks to redundant backups on another cloud platform, UniSuper was able to restore services within days, minimizing the overall impact.

Following the incident, Google Cloud and UniSuper teams worked around the clock to restore the lost data and services. UniSuper’s robust risk management strategy, which included redundant backups on another cloud platform, played a crucial role in the rapid recovery. The data backups stored in Google Cloud Storage (GCS) in the same region were not impacted, facilitating the swift restoration of services.

Google Cloud, in a commitment to transparency, has conducted an internal review and detailed the events leading to the deletion. The company has taken steps to ensure a similar incident cannot occur again, including the deprecation of the internal tool responsible and the correction of system behavior related to GCVE private cloud deletion.

Google Cloud deeply regrets the inconvenience caused to UniSuper and emphasizes the importance of close customer partnerships and robust risk management for rapid recovery in such situations. The incident underscores the need for stringent controls and comprehensive backup strategies in cloud environments.