A Google Cloud engineer accidentally disconnected all optical cables in the region
- A Google Cloud engineer disconnected all optical cables in 13 minutes
- The outage lasted four hours and affected the us-central1-b region
- Traffic dropped to 100%, virtual machines lost external connectivity
- Cause: violation of instructions during scheduled equipment maintenance
Google Cloud published a report on the causes of the partial service outage that occurred on September 1 in one of its infrastructure zones. The outage lasted four hours — from 07:41 to 11:52 Pacific Time, affecting the us-central1-b region and causing serious availability issues for cloud resources.
According to the company, the incident was caused by a simple human error. An engineer conducting scheduled maintenance on the data center equipment accidentally disconnected all optical cables connecting the routers. The peculiarity of the situation was that the process happened critically fast — all one hundred percent of the fiber-optic communication lines (FOCL) were disconnected in just 13 minutes, which did not allow monitoring systems and other engineers to timely notice the problem and stop the worker.
The scale of the damage turned out to be significant. When the routers lost connection, the traffic volume in the affected area dropped by 100 percent. Virtual machines hosted in one of the sub-zones of the us-central1-b region completely lost connection to the external network and the internet. However, they retained the ability to exchange data with each other, but users could not access their resources even for diagnosing the problem. Packet loss, of course, increased to critical levels.
The protection system did not activate.
Google Cloud emphasized that the data center architecture of the us-central1-b region was designed with multi-layer redundancy. The company uses router redundancy, separate power systems, and several independent optical communication lines. The infrastructure is designed to maintain operability even in the event of double and triple equipment failures. However, all these engineering solutions proved powerless against the direct physical disconnection of cables from the ports.
The company explicitly pointed out that the engineer violated a basic safety requirement and standard operating procedures — he did not read the instructions before starting work. The nature of the error was so obvious and unusual that even Google itself ironically described it as a "hyper-scale RTFM error" (RTFM stands for "Read The Manual").
Recovery and Consequences
Google engineers quickly identified the problem and began the recovery process. Traffic was redirected from the failed routers to functioning nodes in other parts of the region. Then, specialists physically restored all optical connections and returned the network to its original state. The incident was fully resolved by the end of the day.
This case once again confirms that the human factor remains one of the main causes of failures in cloud infrastructure and data centers around the world. In recent years, there have been numerous incidents caused by personnel errors — from accidental disconnection of network equipment to incorrect system configuration. Although the overall reliability of data centers has improved, the scale and impact of individual failures can be quite significant, affecting thousands of users and critical services.
Google Cloud pointed out the need to tighten control procedures and increase discipline during scheduled work to minimize the likelihood of similar incidents in the future. The company is also considering the introduction of additional technical barriers to prevent the simultaneous shutdown of all backup communication lines.
Source: 3DNews
