Houston, you have a problem.
Saturday afternoon, a transformer explosion blew out three walls of the electrical equipment room at The Planet‘s primary data center in Houston. The Planet is the Web hosting provider for thousands of online businesses, including Xconomy. The explosion knocked out power to the entire facility, which houses some 9,000 servers hosting websites for 7,500 customers. No one was injured, and there was no damage to the servers, but the building’s main electrical conduits were destroyed, and restoring power has proved to be a struggle, judging from the company’s periodic—many say disappointingly sporadic—updates on the outage. The company started reactivating the 6,000 servers on the data center’s second floor yesterday, but 3,000 servers on the first floor, where the explosion occurred, are expected to be offline until Monday night at the earliest.
All of which is to explain why you got a blank page if you tried to visit Xconomy.com between about 5:00 pm Eastern time on Saturday and 1:00 am this morning. It was our longest outage since our launch last summer. And if not for the heroic efforts of our CTO emeritus Andrew Koyfman and executive editor and COO Rebecca Zacks, we’d still be offline. As the start of the business week approached last night, Andrew and Rebecca decided to move the entire site to servers at Media Temple, a Culver City, CA-based hosting provider we’ve used in the past. So far, the Media Temple servers are stable. We’re watching the situation at The Planet to see when it’s safe to move back (if we move back).
We feel for the folks at The Planet, who are working hard to deal with a situation that is surely as much of a nightmare scenario for them—they had redundant power systems at the site, but fire marshals wouldn’t allow them to be activated—as it is for the thousands of customers whose websites are still down. The event is a public-relations disaster as well as a physical one: the outage was featured on Slashdot on Sunday, and scores of customers have been complaining about the extended down time and The Planet’s reaction to it on the company’s customer forums.
Moving Xconomy.com to a different hosting provider wasn’t a choice we made lightly, since new server addresses have to propagate through the Internet’s Domain Name System (DNS) before anyone can find you on the Web. But it was the only way to make sure we’d be back online at the start of business today.
Most of us here are former print journalists, and we frequently remark on how thankful we are to be a fully online publication, meaning we’ve sidestepped all of the hassles and expenses of print publishing, including the ever-rising cost of printing, paper, and newsstand and postal distribution. But an event like this weekend’s is a harsh reminder that we’re never really free of the realities of the bricks-and-mortar world, which is, after all, home to the hardware makes Xconomy and millions of other websites go.
As Nicholas Carr recently remarked in a blog post on cloud computing: “The metaphor of ‘the cloud’ is a seductive one, but it’s also dangerous. It not only suggests that our new utility-computing system is detached from the physical (and political) realities of our planet, but it also lends to that system an empyrean glow. The metaphor sustains and extends the old idealistic belief in ‘cyberspace’ as a separate, more perfect realm in which the boundaries and constraints of the real world are erased.”
Tongue only partly in cheek, Carr (who will be a speaker at our June 24 forum on cloud computing) suggested a new name for the phenomenon: “miasma computing.” After this weekend, that name isn’t sounding so wrong to us.
