breakdowns, 37signals, laughing squid, rackspace, scott beale, top

Truck driver in Texas kills all the websites you really use

Remember the power mishap in July that brought down 365 Main, the San Francisco datacenter? A similar incident took place today at the Dallas datacenter of Rackspace, a San Antonio, Texas-based firm which serves several local Web outfits. Unlike the July outage, which killed all the websites we waste time with — LiveJournal, Craigslist, and so on — this one took out some sites which really mattered. Laughing Squid, Scott Beale's popular Web-hosting company, went down, taking a long list of customers with it, and 37signals, the maker of Web-based software, went out — a serious matter, since 37signals actually charges for using its software. So what exactly happened at Rackspace?

Like 365 Main, Rackspace was hit by a utility power outage on Sunday. Unlike 365 Main, Rackspace saw its generators kick in, and all was well. This evening, however, a truck drove into a power transformer, causing it to explode. Rackspace techs described this in an email to customers, with admirable sangfroid, as leading to "additional power issues." Further word from Rackspace is that the chillers that keep the servers cool lost power when the transformer blew. An unknown number of servers were taken offline to prevent damage from excessive heat. Currently, the chillers are back online and Rackspace techs are in the process of bringing all the affected customers back online.

Interestingly, as Scott Beale of Laughing Squid points out, "Rackspace does not have a status page or blog." Beale, who's using a status blog to keep his customers informed, later noted that Rackspace does have a "customer portal" — I guess that counts as a blog — which they eventually updated late tonight.

Update 7:54pm: Laughing Squid is back online once again. So is 37signals.

Followup:

7:12 PM on Mon Nov 12 2007
By Jordan Golson
44,924 views
10 comments

Comments

nirreskeya at 08:32 PM on 11/12/07
Reply by Email
*

Dumb fucking luck. I think they deserve a pass on this one. They were obviously well prepared for the expected power outage, and the truck driver probably counts as an Act Of God.
gweston at 09:47 PM on 11/12/07
Reply by Email
*

Thanks for the pass. Sorry we let you down.
Graham Weston, Chairman of Rackspace
tychay at 10:39 PM on 11/12/07
Reply by Email
*

Apparently "Acts of God" are one s/explosion/truck driver and two s/backup generator/cooling system/g away.

Face it. These things happen because the people who built and spec'd these data centers are no longer there to monitor and maintain them, irrespective of who is hosted on it or how the state voted in the last presidential election.

Next, you'll be demanding that Quantas run our data centers.
[www.airsafe.com]

These "perfect storms" seem to be happening a lot more frequently…
mariachi at 09:26 AM on 11/13/07
Reply by Email
*

"the truck driver probably counts as an Act Of God"

that's quite a truck driver!
lwatcdr at 09:27 AM on 11/13/07
Reply by Email
*

I am a Rackspace user. I must be lucky because I don't think any of our customers noticed the downtime or we where lucky and didn't have any.
Compared to where we used to host Rackspace is just great. There tech support is impressive and very helpful.
In my book all they have to do now is learn from these outages and not repeat them. That will earn a pass from me.
mjesales at 09:34 AM on 11/13/07
Reply by Email
*

Now I'm going to have to call mosso and find out if they went down. While they are not a rackspace - they are a rackspace company...
Snowcone at 09:37 AM on 11/13/07
Reply by Email
*

@tychay: As an ex-employee, Rackspace has top notch support personnel in place. Their monitoring systems are also top notch. I'd also challenge your assumption that you know anything about the people who built and spec'd the DCs that RS uses and how their daily operation works.
patchshorts at 09:48 AM on 11/13/07
Reply by Email
*

As a current employee of RS, and a Texan, I can say that there isn't a good way to design Data centers around Drunken rednecks in their trucks. The more rural the area in which the data center is built, the higher the Drunken Redneck threat goes up, and susceptibility of fiber cuts by drunken back hoe drivers for that matter.
quattro at 01:48 PM on 11/13/07
Reply by Email
*

did they forget these?

[www.imagecross.com]
mbundy at 03:12 PM on 11/13/07
Reply by Email
*

"Real" datacenters touch two power grids. "Real" data centers have generator backup for their chillers. Heck, they even have stored water on site in case you lose utility water to make up for evaporative loss. You get what you pay for. In the case of RS, it's still a pretty good deal.