Fake Steve Jobs is Wired's Leander Kahney. Former editor Nick Denton's meticulous analysis of FSJ's prose leads him to believe a fellow ex-Brit is the pseudonymous Steve Jobs blogger. It isn't just wrong, it's patently ridiculous to Kahney's colleagues — the guy's just not mean enough.

Drunk employee crashes 365 Main's colocation site. A tipster uses credible inside knowledge of 365 Main's operations to pass new editor Owen Thomas a timely, credible story that turns out to be completely wrong. The story delivers 145,000 pageviews worth of wrongness. Lesson: Even autistic sysadmins can be full of baloney.

Mark Zuckerberg cashes out. A rumor published at midnight on a weekend during the holiday season gets sprayed all over the Net inside of an hour. Wall Street Journal reporter Kara Swisher shows up at my door in person to tell Owen he's a frigtard.

AdBrite is keeping the network's current customers, and, yes, moving them to BlackLabelAds. Which means, as of December 1, AdBrite will officially be in the porn business. AdBrite serves 678 million impressions on its regular network and 267 million impressions on AVNads.com, which means porn ads make up roughly 28 percent of AdBrite's business. That may decrease, of course, if AVN is successful at luring away customers. (AdBrite founder Philip Kaplan has not yet responded to a request for comment, but I'll update the item when he does.)

From the tone of AVN's press release about the split, it seems like the squabbling pair has someone else to blame for their troubles: 365 Main, the troubled datacenter in San Francisco whose backup power system failed during a July power outage. AdBrite's ad servers were among those brought down. AVN goes on at length about its plans to host its network in multiple datacenters, with 24/7 monitoring. One wonders: If 365 Main's failures led, ultimately, to the demise of this relationship, would AdBrite and AVN have a claim for lost revenues?

Of course, 365 Main's generators failed. The company blames a memory chip in a piece of electronics used to start the generators automatically. But aren't these generators tested monthly? 365 Main notes that the component in question is only used in two of its datacenters. No word on whether the faulty testing procedures are also common to all of its facilities, or just present in San Francisco.

And the kicker? 365 Main brags about the fact that it has "delivered 99.9942 percent uptime to customers," which sounds impressive until you do the math and realize that means the 365/7/24 facility is actually out of service, routinely, for nearly half an hour every year. Last month's outage, in other words, was all in a day's work for 365 Main. On top of that, consider this: It's a failure rate six times as high as the "five nines" standard 365 Main promised when it launched. 365? More like 364.98.

Here's the press release. I recommend you trust it as much as you do the "365" in 365 Main's name.

365 MAIN REPORTS ON ROOT CAUSE OF GENERATOR FAILURE

Company Implements Fix for All Affected Generators and Makes Information
about the Fix Available to Data Center Industry

SAN FRANCISCO, Calif., Aug. 1, 2007 - Data center developer and operator 365
Main Inc. is issuing information today that details the root cause behind
why back-up power generators in the company's San Francisco facility failed
to start during a PG&E power outage last week, resulting in approximately 40
percent of customers in the facility losing power to their equipment for up
to 45 minutes.

The Problem

At 1:47 p.m. on Tuesday, July 24, 365 Main's San Francisco data center was
impacted by a power surge caused when transformer breakers at a local PG&E
power station unexpectedly opened. PG&E has still not determined what caused
the breakers to open.

Typically when a power outage occurs, the outage triggers 365 Main's
rigorously maintained and tested back-up diesel generators to start-up and
take over providing power supply to customers. 365 Main's San Francisco
facility has ten 2.1 megawatt back-up generators to be used in the event of
a loss of utility power. Eight primary generators can successfully power the
building, with two generators available on stand-by in case there are any
failures with the primary eight.

However, following the power outage last week, three of 365 Main's 10
back-up power generators, manufactured by Hitec, failed to complete their
start sequence. A complete investigation of the incident began immediately.

Within hours of the incident, an international team of specialists was
deployed to 365 Main's San Francisco data center facility to join on-site
technicians and begin systematically testing the generators in search of a
root cause. After days of thorough testing around the clock, the team
discovered a weakness in an essential component of the back-up generator
system known as a DDEC (Detroit Diesel Electronic Controller).

The team discovered a setting in the DDEC that was not allowing the
component to correctly reset its memory. Erroneous data left in the DDEC's
memory subsequently caused misfiring or engine start failures when the
generators were called on to start during the power outage on July 24.

The Fix

The investigation team discovered DDEC issues on each of the failed Hitec
units and were able to successfully simulate failure. A fix was introduced
by altering the timing of a command to the DDEC component, allowing more
time between the engine shut-down command and the DDEC reset command. Once
this fix was introduced, the Hitec generators successfully passed more than
50 consecutive start-up sequence tests without incident.

The testing methodology was performed by Hitec specialists along with 365
Main's chief technician and staff. Specialists from Cupertino Electric were
present during all testing, and EYP Mission Critical Facilities will provide
independent verification of the findings the week of 8/6/07.

365 Main has implemented the DDEC fix in its San Francisco and El Segundo
facilities. Of the five data centers in 365 Main's portfolio, the San
Francisco and El Segundo facilities are the only ones with Hitec generators
containing DDECs. All other facilities feature other brands of generators
or have different models of Hitecs.

365 Main is sharing the discoveries of its investigation with other Hitec
customers. In addition, Hitec has expanded its preventative maintenance
procedures as a direct result of discoveries made during the 365 Main
investigation.

In the wake of the outage, 365 Main published an apology to customers and
daily updates directly from the investigation team meeting minutes, allowing
customers and the public at large to track progress. A complete archive of
these updates and more details about today's update are available at:
http://www.365main.com/status_update.html

Chris Dolan, president and CEO of 365 Main, said, "365 Main has a track
record of providing customers with data centers that are considered to be
among the world's finest. We extend our sincere apologies to customers who
were impacted by this incident. Addressing customer concerns is our top
priority. In the days since the incident occurred, we have identified and
corrected the root source of the problem and are taking steps to prevent
this type of problem from happening again. We are also making our
comprehensive findings available to other data centers to try to prevent the
same problem from recurring elsewhere."

Glenn Ellis, president and CEO of Hitec USA, also commented: "Our top
priority is taking steps to prevent this type of unforeseen incident from
occurring again. We sincerely apologize to 365 Main and its customers that
our generators failed to deliver the continuous power as designed."

365 Main's Track Record

Since its inception over five years ago, 365 Main has delivered 99.9967
percent power uptime to customers across its five-data-center portfolio.
This includes the outage experienced in San Francisco last week. 365 Main's
San Francisco facility has delivered 99.9942 percent uptime to customers
during the last five years, inclusive of last week's outage.

As part of their service level agreements with 365 Main, 365 Main customers
receive rent abatements (refunds) in the event that electrical power is
dropped in the section(s) of the data center where their servers are
located. 365 Main is honoring all service level agreements with affected
customers.

The very best report has come from an employee of a Fortune 500 tech company who was inside the datacenter at the time of the outage. He witnessed the chaos and confusion at 365 Main firsthand. Let's just say he was not impressed by how employees at the datacenter performed during the crisis. "You should be able to trust the power," says my informant. "That's what we pay them for." After the jump, his complete account.

FYI, I was in Colo 7 when the outage happened, preparing to kick off a very large upgrade to a very important set of gear in my employer's cage, and so when the power flickered off for a second, I nearly peed my pants. Not only did the power flip off, but there were incredible scary noises from the roof and the HVAC as stuff was turning on and off and generators spooled up and emergency lighting flipped on and off. I ran for the NOC, and got ahold of somebody who didn't know anything who I hung up on. Seconds after that, every door in the place burst open, and 365 Main folks were sprinting down the halls.

Colo 7 is on the top floor, and the only way to the roof seems to be through a stairwell in the big hallway up there outside Colo 7 and 8, so I watched for a bit while a coworker made the call to management. A lot of people went running by to the roof and towards the big PDUs in the center of Colo 7 and into the NOC/telco area, etc., and I'll tell you, I didn't see any drunk 365 Main people anywhere, just folks with an "OMFG, what just happened!!??" look on their face running around. Soon after that, other sysadmins began to trickle in, but we were too busy making sure that our own stuff was coming up to pay much attention. I've never seen so many people in 365 Main before.

Anyways, I saw the employees running around and flipping out and it was very exciting for everybody. Seems to me that if some guy had hit buttons, then a lot of activity would have been centered around the area with the buttons (possibly revolving around doing bodily harm to hypothetical drunk people), not the roof or the NOC or the PDUs or anything else that I saw the brown shirts doing.

So you might want to just kill that rumor. Really, don't you think that 365 Main would LOVE to be able to pin a quadruple failure of their equipment on some alcoholic? If it were a drunk guy doing it, it would make me a lot more confident in 365 Main, because it's easy to fire the guy and require breathalyzers to get into critical infrastructure rooms or whatever. As it is, I wonder what will happen next time we have to go off the grid.

The tip, which I printed verbatim, came in by AOL Instant Messenger — not the most trustworthy of mediums, I'll grant you, but one favored by sources trying to preserve their anonymity. Despite my misgivings, I trusted the tipster for the following reasons:

• He IM'd me at 2:14 p.m. Pacific, only 27 minutes after 365 Main lost power supply from PG&E, before anyone outside the datacenter knew details of the outage, claiming to have information from a friend who worked there.
• I pressed him for some details and established that his connection to 365 Main was plausible.
• He gave me a first and last name, and told me that he worked at a large software company. I established that there was a real person with that name who worked at such a company.

Contrary to what some blogtards have written, I didn't invent the tip, or make the tipster up. We had sustained IM and phone conversations. And my tipster, for what it's worth, continues to stand by his version of events. But I'm the ultimate blogtard for not checking him out more thoroughly. I've since learned that the tipster has associations with a band of hackers who delight in social engineering, the art of using technical means to get human beings to do their bidding. And if he belongs to such a band — which he denies — I played right into their hands.

Why? I'm told by people close to the hacker group that they do this kind of thing for sheer amusement. That they enjoy harassing people in the Web 2.0 world. That they have broken California's privacy laws by illegally recording telephone conversations, among other misdeeds. (I'm not going to name them, because they'd just enjoy the attention.)

Before you condemn me, though, let me say this: Yes, I'm an experienced reporter who ought to have known better. While I worded the post itself cautiously, the headline should have reflected that same caution. It didn't, and I apologize as well for that. But in running Valleywag, I'm experimenting in a new medium and a new style of journalism. Yes, I take tips over the transom. Yes, I post early and often, and factcheck, update, and correct as I go. And yes, this is why you read blogs.

And I also should say a bit about why I was inclined to believe the tip. 365 Main CEO Chris Dolan personally gave me a tour of his facilities a few years ago. He showed me the generators, the flywheels, the systems. He led me to believe that 365 Main could ride out an earthquake, with its supplies of fuel and water.

Based on what Dolan told me, I found it more plausible that an employee, acting maliciously, could take down 365 Main's power than something as insignificant as a power outage. Protection from power outages, after all, are precisely why companies put their servers in datacenters with supposedly redundant systems.

I hope no one will come away thinking that, because of my error, 365 Main should come away from this free from blame. Big questions remain about its outage. The company itself admits it still does not know why half of its generators failed. Four of those five generators failed because of "problems in [their] start sequence."

Think about that: Why would four generators fail, for the exact same reason, at the same time? I have reason to believe that, for reasons of his own, my tipster may have been trying to deceive me, and through me, you. But I ran with the tip because it had the ring of truth. Something very wrong happened at 365 Main on Tuesday, and we still don't know what it was.

365 Main has denied that there was any "employee misconduct." With its investigation admittedly incomplete, however, I'm surprised it would make such a claim. As I've pointed out, the datacenter's credibility is very low. (And before you say it, allow me: As is mine.) Customers are fleeing fast. My latest tip, if you care to believe it: Technorati plans to move 500 servers out of 365 Main by September.

There's a simple step 365 Main's management can take to staunch its customer losses: After finishing the investigation, post the video from the datacenter's 24-hour surveillance tapes to the Web. Show exactly how its employees behaved during the outage. Or they could release the tapes to me, and I'll review them, and post about what I saw.

Feel free to doubt me. After relying on a source I no longer trust, I deserve it. But until management at 365 Main concludes its investigation and releases its surveillance videos to back up their version of events, you should continue to doubt them, too.

From: "Marcy Maxwell"
To: "Engineering" ; "Security"
Sent: 7/25/07 5:08 PM
Subject: UPDATE: POWER EVENT - Fourth Notice

UPDATE: 5:00 P.M., Wednesday, July 25, 2007

A complete investigation of the power incident continues with several specialists and 365 Main employees working around the clock to address the incident.

Generator/Electrical Design Overview

The San Francisco facility has ten 2.1 MW back-up generators to be used in the event of a loss of utility. The electrical design is N+2, meaning 8 primary generators can successfully power the building (labeled 1-8), with 2 generators available on stand-by (labeled Back-up 1 and Back-up 2) in case there are any failures with the primary 8.

Each primary generator backs-up a corresponding colocation room, with generator 1 backing up colocation room 1, generator 2 backing up colocation room 2, and so on.

Series of Electrical Events

* The following is a description of the electrical events that took place in the San Francisco facility following the power surge on July 24, 2007:

* When the initial surge was detected at 1:47 p.m., the building's electrical system attempted to roll all colocation rooms to diesel generator power.

* Generator 1 detected a problem in its start sequence and shut itself down within 8-10 seconds. The cause of the start-up failure is still under investigation though engineers have narrowed the list of suspected components to 2-3 items. We are testing each of these suspected components to determine if service or replacement is the best option. Generator 1 was started manually by on-site engineers and reestablished stable diesel power by 2:24 p.m.

* After initial failure, Generator 1 attempted to pass its 732 kW load to Back-up 1, which also detected a problem in its start sequence. The exact cause of the Back-up 1 start sequence failure is also under investigation.

* After Generator 1 and Back-up 1 failed to carry the 732 kW, the load was transferred to Back-up 2 which correctly accepted the load as designed.

* Generator 3 started up and ran for 30 seconds before it too detected a problem in the start sequence and passed an additional 780 kW to Back-up 2 as designed.

* Generator 4 started up and ran for 2 seconds before detecting a problem in the start sequence, passing its 900 kW load on to Back-up 2. This 900kW brought the total load on Back-up 2 to over 2.4 MW, ultimately overloading the 2.1 MW Back-up 2 unit, causing it to fail. Generator 4 was manually started and brought back into operations at 2:22 p.m. Generator 4 was switched to utility operations at 7:05 a.m. on 7/25 to address an exhaust leak but is operational and available in the event of another outage.

* Generators 2, 5, 6, 7 and 8 all operated as designed and carried their respective loads appropriately.

* By 1:30 p.m. on Wednesday, July 25, after assurance from PG&E officials that utility power had been stable for at least 18+ continuous hours, 365 Main placed diesel engines back in standby and switched generators 2,5,6,7, 8 to utility power.

* Customers in colocation rooms 2, 4, 5, 6, 7 & 8 are once again powered by utility, and are backed up in an N+1 configuration with Back-up 2 generator available.

* Generators that had failed during the start-up sequence but were performing normally after manual start (1 & 3) continue to operate on diesel and will not be switched back to utility until the root causes of their respective failures are corrected.

Other Discoveries

* In addition to previously known affected colocation rooms 1, 3 and 4, we have discovered that several customers in colo room 7 were affected by a 490 millisecond outage caused when the dual power input PDUs in colo 7 experienced open circuits on both sources. A dedicated team of engineers is currently investigating the PDU issue.

Next Steps

* Determine exact cause of generator start-up failure and PDU issues through comprehensive testing methodology.

* Replacements for all suspected components have been ordered and are en route.

* Continue to run generators 1 & 3 on diesel power until automatic start-up failure root cause is corrected.

* Continue to update customers with details of the ongoing investigation.

Regards,

Marcy

Marcy Maxwell Vice President, Security 365 Main Inc. "The World's Finest Data Centers"

• 365 Main's facilities are supposed to be rock-solid, designed to ride out a major event like an earthquake. CEO Chris Dolan personally gave me a tour shortly after his team revamped the datacenter. Unless he was exaggerating to me then — and, one presumes, exaggerating to every customer he's since signed — a power outage shouldn't have taken 365 Main out.
• 365 Main has multiple colocation rooms, or "colos," in the center. Colos 3 and 4 — on the same floor, if memory serves — went down, while Colos 2 and 8 stayed up. Data Center Knowledge says that an additional, unspecified colo lost power. (According to a current customer, not all of 365 Main's colocation rooms are occupied, because the facility is constrained by power supply, not space.)
• Was there a drunk employee? Harris, the ever-so-believable 365 Main flack, is denying "employee misconduct." But that doesn't rule out someone else with access to the building tripping the emergency-power-off switch on the affected floor. Bad timing? Sure. Impossible coincidence? No.
• What caused the long lines outside 365 Main? Apparently 365 Main's customer-authentication system was down, forcing already-angry sysadmins to wait in line while guards checked IDs manually.
• Were customers' contracts breached? Almost certainly, if they negotiated any decent service-level agreements with 365 Main. Heard about any lawsuits filed or payments sought? Send in those tips.

Now, from commenter somafm, a highly detailed account of what he believes happened.

Here's what really went down at 365 Main:

365 Main, like all facilities built by AboveNet back in the day, doesn't have a battery backup UPS. Instead, they have these things called "CPS," or continuous power systems. What they are is very very large flywheels that sit between electric motors and generators. So the power from PG&E never directly touches 365 Main. PG&E power drives the motors which turn the flywheels which then turn the generators (or alternators, I don't remember the exact details) which in turn power the facility. There are 10 of these on their roof.

The flywheels (the CPS system) can run the generator at full load for up to 60 seconds according to the specs.

There are also 10 large diesel engines up on the roof as well, connected to these flywheels. If the power is out for more than 15 seconds, the generators start up, and clutch in and drive the flywheels. There are no generators in the basement. (There is a large fuel storage in the basement, and the fuel is pumped up to the roof. There are smaller fuel tanks on the roof as well. )

Here's what I think happened. Since there were several brief outages in a row before the power went out for good, it seems that the CPS (flywheel) systems weren't fully back up to speed when the next outage occurred. Since several of these grid power interruption happened in a row, and were shorter than the time required to trigger generator startup, the generators were not automatically started, BUT the CPS didn't have time to get back up to full capacity. By the 6th power glitch, there wasn't enough energy stored in the flywheels to keep the system going long enough for the diesel generators to start up and come to speed before switching over.

Why they just didn't manually switch on the generators at that point is beyond me.

So they had a brief power outage. By our logs, it looks like it was at the most 2 minutes, but probably closer to 20 seconds or so.

This afternoon a power outage in San Francisco affected the 365 Main St. data center. In the process of 6 cascading outages, one of the outages was not protected and reset systems in many of the colo facilities of that building. This resulted in the following:

- Some of our routers were momentarily down, causing network issues. These were resolved within minutes. Network issues would have been noticed in our San Francisco, San Jose, and Oakland facilities.

- DNS servers lost power and did not properly come back up. This has been resolved after about an hour of downtime and may have caused issues for many GNi customers that would appear as network issues

- Blades in the BC environment were reset as a result of the power loss. While all boxes seem to be back up we are investigating issues as they come in

- One of our SAN systems may have been affected. This is being checked on right now

If you have been experiencing network or DNS issues, please test your connections again. Note that blades in the DVB environment were not affected.

We apologize for this inconvenience. Once the current issues at hand are resolved, we will be investigating why the redundancy in our colocation power did not work as it should have, and we will be producing a postmortem report.

Some of the affected websites — most of which are back online — played it straight with customers, like Craigslist. Others offered the usual pack of lies websites trot out. AdBrite, for example, tried to claim that the outage was due to "scheduled maintenance," and RedEnvelope, the e-commerce gifts site which just today crowed about moving all of its Web operations to 365 Main, said the outage was a systems upgrade. Busted!

A source close to the company says:

Someone came in shitfaced drunk, got angry, went berserk, and fucked up a lot of stuff. There's an outage on 40 or so racks at minimum.

We're sure 365 Main will deny that such a thing could ever happen. And, conveniently, the neighborhood is having power troubles, too. But here's a question: When you have several levels of redundant power, what could bring your customers' servers down other than something like an employee physically ripping the plugs out of the wall? Or, with less effort, hitting the emergency-power-off switch that San Francisco's building codes require 365 Main install?

Update: Technorati's Dave Sifry just sent this email:

Folks,

I just wanted to let you know, it looks like San Francisco is having a MAJOR power event, with outages from the Financial district all the way down to Daly City. One of our colos at 365 Main Street has experienced a power outage (never mind that they always swear up and down that this kind of event can't possibly happen, oh no, they have multiple redundant systems and they charge us up the wazoo to make sure that we'll have business continuity, so of course, this isn't really happening, oh yes) however, our other data centers are all up and running, so we hope to be back up and running as quickly as possible.

I'll keep you all updated on progress, and I appreciate you bearing with us as we work our way through this...

Dave

Subsequent coverage:

• Angry mob gathers outside SF datacenter
  
• 365 Main outage causes aftershocks in Web world (photo gallery)
  
• 365 Main's credibility outage
  
• Investigation continues into 365 Main's outage
  
• Drunk editor kills the gossip item you care about