Investigation continues into 365 Main's outage - Valleywag Comments

Investigation continues into 365 Main's outage

0x6772 — Fri, 27 Jul 2007 09:39:03 PDT

@rekoil, bullshit. There's no reason you can't shunt away from supplying the room on the up side but still simulate a service failure on the back side. I've done that, and so should 365 have been, although they obviously weren't.

It costs more to be able to isolate your generators (rather than insist on passing through them), but there's no reason you can't run the room of batteries like usual and the batteries of mains while pulling the gens out of the loop.

0x6772

Investigation continues into 365 Main's outage

rekoil — Thu, 26 Jul 2007 15:25:51 PDT

@synnik: The problem with testing backups is that while you do monthly tests as part of standard maintenance, it's not a *real* test - the load is manually transferred from utility power to generators and back again. This way if a failure is detected the building won't go dark The only *real* test of a backup system, and the automatic failover mechanisms, is to literally pull the plug, which data centers are understandably reluctant to do, given the consequences of a failure - exactly what we saw happen here.

rekoil

Investigation continues into 365 Main's outage

synnik — Thu, 26 Jul 2007 14:51:18 PDT

The unanswered question here -- how often do they test their generators' startup sequences?

The likelihood of that much failure at once either points to a lack of testing of the generators, or a lack of honesty in what happened.

Or just am amazing amount of bad luck. Anything is possible. Just highly unlikely. (Sherlock Holme's attitude on the highly unlikely notwithstanding.)

synnik

Investigation continues into 365 Main's outage

Rick — Thu, 26 Jul 2007 13:48:29 PDT

@Rick: Meant to say what will happen to the internets?

Rick

Investigation continues into 365 Main's outage

Rick — Thu, 26 Jul 2007 13:48:11 PDT

@deliriousnyc: My mind has been wandering similarly lately; this has only served to stoke the fires. What -will- happen when there's a big ole earthquake in SF? I'm sure this isn't the only datacenter in town.

Rick

Investigation continues into 365 Main's outage

DirtySpy — Thu, 26 Jul 2007 13:25:51 PDT

KANSAS! LOL! That's Dorothy Territory.
One big Tornado and the servers are on their way to OZ!

DirtySpy

Investigation continues into 365 Main's outage

deliriousnyc — Thu, 26 Jul 2007 12:31:16 PDT

I'm surprised that they keep these servers in San Francisco at all, at least without a back-up in a non-earthquake-prone location, like, say, Kansas. I'm sure SF is due for a good shaking any day now.

deliriousnyc