Many customers of iiNet, Australia's second largest internet provider, were disgruntled Monday to find the company's servers in Perth had succumbed to a brutal heat wave, leaving thousands offline for hours.
The shutdown came after temperatures hit 111 degrees Fahrenheit in Western Australia, and is the latest reminder that, despite modern technology, your internet connection is ultimately at the mercy of Mother Nature. In a statement, iiNet Chief Technology Officer, Mark Dioguardi said the company shut down a portion of its servers after air conditioners at its data center failed.
"Our Perth data centre was subject to a partial failure of both the mains and backup air-conditioning systems yesterday," the statement said. "This, along with the extreme heat necessitated a shut down to a portion of our servers as a precautionary measure."
Climate change is putting unprecedented stress on internet infrastructure
Kevin Leahy is a group general manager at global services provider Dimension Data, and has more than 30 years of experience in designing data centers. He said Monday's meltdown is indicative of broader issues affecting data centers as climate change puts unprecedented stress on internet infrastructure.
"At the end of the day, you're going to design data centers around certain limits of expectation, and the most important thing to figure out is, what are the limits? With global warming and things like that, it's hard to predict," he said.
Keeping servers cool is a major, energy-sucking component of internet infrastructure. According to Leahy, power and cooling is about 50 percent of the construction of a data center, and cooling comprises around half of that. Engineers are constantly trying to increase the efficiency of cooling systems. Leahy said more than a decade ago, people used to work alongside the machines until they discovered working remotely could decrease the overall temperature and allow equipment to run at a higher capacity. The design of cooling systems in data centers vary by location.
"How centers are designed in places like Perth or the Middle East is very different from how you design in Denver or Iceland," Leahy said. "When you're in a very hot location, you tend to use a lot of enclosed facilities with optimal cooling techniques, taking advantage of things like liquid cooling."
Ultimately, the systems are designed to cool the machines from inside the data center, not protect them from outside heat. Although meltdowns like Monday's are rare, they are not unheard of—similar outages occurred in Arizona in 2013 and in France in August 2009.
Leahy said Monday's disruption is ultimately the same as those caused by natural disasters, which, as we saw in the aftermath of Hurricane Sandy, leave data centers vulnerable to unpredictable environmental stress.
"It's unusual, but the result of the shutdown is not any different from those of a typhoon or a flood an earthquake or anything else—it's still a natural effect that causes a data center to not be able to deliver performance levels that are required," he said. "It's just another instance of not quite designing for the resilience that they need."
Australia's shutdown is one more than 100 times extreme weather has impacted iiNet service in the last year, including one other incident of heat-related disruption in January 2014.
"The majority of data centers have been fairly well designed for the extremes in the areas they are located, but these are temperatures that for the most part nobody has designed around," Leahy said. "They're predicting temperatures of 47 or 48 degrees Celsius [116 to 118 degrees Fahrenheit] in Perth in the next few days—no data center is designed for that."
In the face of changing environmental stress, organizations are making both physical and operational changes. After Hurricane Sandy shut down some servers for days, many organizations relocated their servers away from first level or basement level floors. Leahy said soon data centers will be designed to operate in an ambient external temperature of up to 122 degrees Fahrenheit.
Data centers are also becoming more dependent on cloud exchange—offsite, virtualized storage systems—to prepare for natural disasters and other external threats.
"You have to design in a way that you can reroute critical workloads away from areas of disaster," Leahy said. "You have to be sure you aren't just dependent on one location—whether it's a high temperature or tsunami or earthquake, you want to make sure your design of your service is based on data centers that are distributed so you aren't subjected to the extremes of whatever disasters are occurring."
As climate change affects weather patterns, the internet's infrastructure will have to get tougher to withstand increasingly extreme environmental events. I don't know about you, but if there's a heatwave, I want to be able to wait it out by playing Destiny.