If you follow our business really closely, you probably know that we had a pretty big outage in the East Village on October 6th. You sure know about it if you happen to live in New York City and have a Twitter account – it was understandably all over the place.
The New York Times’ East Village blog mentioned this, naturally. Essentially, a number of nodes connected to this portion of fiber lost network connectivity, interrupting service to approximately 24,000 customers in the area.
Our VP of Engineering in New York City shared some photos of the mess with me, and I thought our customers would like to see what a large, lengthy, and unusual outage looks like.
First, here’s a few peeks at what’s under a New York City manhole:
It’s a mess of communications cables belonging not only to us but a variety of other tenants including Verizon, RCN, etc. As you may notice, these do not appear to be neatly labelled.
This is a look at the area where the fire actually occurred. Essentially, an underground steam pipe blew a leak that then caused a fire in the nearby electrical wiring. That fire melted our fiber-optic cable, which happened to be carrying our services to the aforementioned nodes. A node can service between 300 and 600 customers, depending on a variety of factors. You can see what one looks like and learn more here.
The multicolored wires are multi pair telephone wiring. Here’s a closer view of the hole, with the affected fiber cable running across the image:
I am told that the curly white stuff in there is the insulation from some of Verizon’s cables. This is the standard way that multi conductor phone wires are insulated. I want to meet the person who first discovered that ratty old drag queen wigs are excellent insulators. You KNOW they’ve got a few stories.
Here, you can see a cross-section of what the cable looked like before the fire:
I took a closeup with the macro lens on my iPhone:
Those multicolored things are a protective sheath around a single fiber-otic strand roughly the width of a human hair. Each strand carries video, broadband, and phone services to a node, which, again, can serve 300-600 homes. The colored sheaths are slathered with a sort of insulating goo inside the cable shell to help protect the fiber from moisture. The whole enterprise looks a little like an alien cyborg’s reproductive organ, doesn’t it?
Here’s what the cable looked like after being subjected to an electrical fire:
And again, with an iPhone macro closeup:
It took the better part of the day to identify the problem, dig in, seperate the cable out and splice in a new piece of fiber-optic cable. Each of those hair-width fibers has to be reconnected to precisely the correct wire, or else the whole thing doesn’t work. Imagine re-connecting a severed ponytail and you’ve got the basic idea.
This is both extremely unusual and business as usual for us. It’s extremely unusual because, well, freakish subterranean fires don’t happen that often. And while outages are an unfortunate part of running a big, complex system, one that affects that many people for that long is pretty rare. But it’s business as usual because, look: we’re doing business in the physical world. Things fall apart, and humans are all working against decay, chaos and factors beyond our control.
We built this system and we’re here to keep it moving when the going gets weird.