October 18, 2011

Thumbnail Photo

Share

       

Subterranean Fires and Melted Fiber-Optics: What A Large Outage Looks Like

If you follow our business really closely, you probably know that we had a pretty big outage in the East Village on October 6th. You sure know about it if you happen to live in New York City and have a Twitter account – it was understandably all over the place.

The New York Times’ East Village blog mentioned this, naturally. Essentially, a number of nodes connected to this portion of fiber lost network connectivity, interrupting service to approximately 24,000 customers in the area.

Our VP of Engineering in New York City shared some photos of the mess with me, and I thought our customers would like to see what a large, lengthy, and unusual outage looks like.

First, here’s a few peeks at what’s under a New York City manhole:

BIGhole

It’s a mess of communications cables belonging not only to us but a variety of other tenants including Verizon, RCN, etc. As you may notice, these do not appear to be neatly labelled.

This is a look at the area where the fire actually occurred. Essentially, an underground steam pipe blew a leak that then caused a fire in the nearby electrical wiring. That fire melted our fiber-optic cable, which happened to be carrying our services to the aforementioned nodes. A node can service between 300 and 600 customers, depending on a variety of factors. You can see what one looks like and learn more here.

hole7

The multicolored wires are multi pair telephone wiring. Here’s a closer view of the hole, with the affected fiber cable running across the image:

hole5

I am told that the curly white stuff in there is the insulation from some of Verizon’s cables. This is the standard way that multi conductor phone wires are insulated. I want to meet the person who first discovered that ratty old drag queen wigs are excellent insulators. You KNOW they’ve got a few stories.

Here, you can see a cross-section of what the cable looked like before the fire:

fiber_optic

I took a closeup with the macro lens on my iPhone:

healthy-endFIX

Those multicolored things are a protective sheath around a single fiber-otic strand roughly the width of a human hair. Each strand carries video, broadband, and phone services to a node, which, again, can serve 300-600 homes. The colored sheaths are slathered with a sort of insulating goo inside the cable shell to help protect the fiber from moisture. The whole enterprise looks a little like an alien cyborg’s reproductive organ, doesn’t it?

Here’s what the cable looked like after being subjected to an electrical fire:

burnt_end

And again, with an iPhone macro closeup:

burnt_end2

It took the better part of the day to identify the problem, dig in, seperate the cable out and splice in a new piece of fiber-optic cable. Each of those hair-width fibers has to be reconnected to precisely the correct wire, or else the whole thing doesn’t work. Imagine re-connecting a severed ponytail and you’ve got the basic idea.

This is both extremely unusual and business as usual for us. It’s extremely unusual because, well, freakish subterranean fires don’t happen that often. And while outages are an unfortunate part of running a big, complex system, one that affects that many people for that long is pretty rare. But it’s business as usual because, look: we’re doing business in the physical world. Things fall apart, and humans are all working against decay, chaos and factors beyond our control.

We built this system and we’re here to keep it moving when the going gets weird.

+- 30 Comments

  1. John's reply

    Why not make them fireproof.

  2. PSUSkier's reply

    @John – Because that would cost a crapload of money to insulate very thin strands of glass to the point that they would be shielded from fire. Multiply that by however many miles of stranded fiber optic runs are out there and you see that that is a hugely costly endeavor, especially considering how frequently something like this can happen.

  3. Karl H's reply

    This is a great look down below – it is amazing to think about how damn much we can move over fiber compared to copper these days.

  4. Mahdi's reply

    @PSUSkier What if you just insulate the sheath you pulled the strands of fiber through? Kind of like a fireproof insulated pipe?

  5. JimW's reply

    Wow. That would make sense on time. What about when TW comes to hook up the neighbor’s cable on Tuesday, knocks our cable, internet and phone in the process and then can’t come out until Friday to fix it unless there’s a cancellation in their schedule. New hookups apparently take precidence over 10+ years customers. We got a lot of there’s nothing we can do.

  6. Pete M's reply

    Great article – the general public need to realise that field techs are heroes! Always working their butts off to keep your connections alive.
    Greetings from Virgin Media in the UK!

  7. Spence C's reply

    Jeeeeee-bus! Who the heck is responsible for those manholes? If that is a city managed vault / manhole someone should be fired. I feel for you guys having to try to restore services with a crappy situation like that! I doubt those things get the care they need; I’m surprised you don’t have more outages with stuff like that happening.

  8. MJC's reply

    @Mahdi Goes back to cost-per-foot over, literally, miles. It is much more cost effective to use non-metallic flexible tubing with bundled cable and send a field tech team (usually two or three guys with spare cable/tubing and fusion splicers) when an event like this happen. Yes, downtime sucks, but there is no such thing as 100% uptime and these guys are good at their job so the downtime is minimal despite the amount of fiber optic they have to repair (keep in mind this stuff is glass, so even if the damage is only in a little 2′ section, chances are it has cracked a couple of feet down the line, something they have to find so they know how far back to cut the damaged cable).

    BTW – I hate that Kevlar insulation on super-books of telecom cable. Either the manhole looks like you buried a streetwalker or you look like you got attacked by shedding long haired dogs.

  9. paul's reply

    “I want to meet the person who first discovered that ratty old drag queen wigs are excellent insulators.” the LGBT community is going to have a field day with this one, wonder how quick before there’s a petition to get you canned. :: sigh ::

  10. Jeff Simmermon's reply

    Paul – before coming to Time Warner Cable, I actually managed AOL’s Gay & Lesbian community online. It was a tremendous, life-changing experience. And if there’s one thing I learned, it’s that the LGBT community has a pretty excellent sense of humor.

  11. Seth Rosenblum's reply

    Excellent look at the infrastructure, Thanks Jeff!

    I’m a little concerned as to why such a wide swath of your network has a single-point of failure though. With Manhattan’s density the way it is, wouldn’t it make more sense to build a mesh fiber network? That way when one underground link goes down, the others can compensate in capacity? Seems like that’s the big advantage Time Warner would have over Verizon’s nest of copper phone wires, dynamic routing.

  12. cableguy's reply

    Wow and I thought splicing fiber in the middle of the woods/easement was tough. That manhole is a mess. Great pictures…

  13. Jeff Simmermon's reply

    Seth – I’m so glad you liked the post. Unfortunately, the answers to your other questions are beyond both my area of expertise and my pay grade :)

  14. paul's reply

    jeff – i know, sorry if it came off that i took it that way, as a TG i dont, but i know how up tight some can be.

    but back on topic, im surprised that the fiber is color coded, as a legally blind / color blind individual i all ways wanted to go in to communications, but being color blind on top of being legally blind has made that impossible because of all the color coding (as far as im aware theres no tool to tell one the color of a wire) so im stuck.

    but i still find it fascinating.

  15. david's reply

    Did they credit customers for the outage? After this i jumped to fios and its much better.

  16. Brad Allen's reply

    I like the first respondent — “why not make them fireproof?” It’s a silly tangent to my consideration — just make the network redundant, by having multiple routes everywhere. Both approaches have one core concept in common: they cost more. The existant method of a single route to each node that isn’t super fireproof is cheaper. When a fault happens in a system like this, part of the society has less of some supply (in this case data in the east village), and the society as a whole continues unabated, and the part that had a loss for a while will catch up when its supply is restored. It’s the most cost effective way to run the society. Someday we may make it more civilized by making it more reliable and more redundant. Until then, it makes sense in the cost structures we are a part of.

  17. Spence C's reply

    Do you al have to foot the entire cost of the repair or does the offending service provider / utility eat some as well?

    @ Seth I’d imagine their Business Class users / folks with Service Level Agreements are on a mesh. I don’t think home use contracts have SLAs.

  18. Zidders's reply

    I came here because I was pissed at Time-Warner for something completely unrelated, I’m now a lot less pissed because of this;” I want to meet the person who first discovered that ratty old drag queen wigs are excellent insulators. You KNOW they’ve got a few stories.”

    and this “The colored sheaths are slathered with a sort of insulating goo inside the cable shell to help protect the fiber from moisture. The whole enterprise looks a little like an alien cyborg’s reproductive organ, doesn’t it?”

    Jeff, you’re ok in my book. Thank you for the laugh. I’d rather be in a good mood when I call customer service than a bad one.

  19. Dean's reply

    Wow! I live in the East Village and had no idea about this until right now. Must not have affected me as I am a heavy internet/tv user and would have noticed.

  20. Dennis Vergason's reply

    I would have been nice if the author knew more about the subject matter, the terminology and what the identified parts are. The commentary was juvenile and classless. I’ve been working in this industry for 27 years and while the information may be useful to someone that knows nothing about cable or specifically Fiber Optic cables, I was embarrassed by the inaccurate descriptions in the picture, especially if the writer works in the cable industry. Fiber Optic cabling is strands of glass that carry data by using light which is converted to RF signals with a Fiber Optic Node

  21. Jeff Simmermon's reply

    Dennis – Thanks for your note. I do in fact work in the cable industry – I’m in the PR department. The idea here was specifically to educate, entertain, and intrigue people who know nothing about fiber-optics, or really cable in general. I’m not sure how a description of a cable can embarrass you personally, but I suspect that it may have more to do with your personal baggage than my writing. However, if you have a more nuanced or informed take on what we’re seeing in these photos, I’m happy to check it against our engineering team and publish it in this post.

  22. Greg D.'s reply

    Dennis, actually Jeff did a very good job in what his focus was for this blog. As he mentioned, his focus is PR and uses an entertaining fashion towards folks that are not in our industry, which is the majority. I’m sure others on the technical side appreciate his candid descriptions, which I’ve heard good comments from others.

    The blog wasn’t about fiber architectures or what travels on those fibers or more exactly the type of impact on the fibers. Or even nodes and redundancy.

    The article was about communicating to the general public about a major outage and sharing with them the findings, including pictures. It also describes what happens to communications cables in this type of locked down environment. Showing pictures of what they look like after hot steamy water causes a fire and then burns the outer sheath from most of the cables in the area. If burned long enough, it can melt glass.

    Good thing the gas level in that manhole was at a low enough level that the fire didn’t cause an explosion. This is what I would be more concerned about, then being embarrassed. I’ve been embarrassed before, but that is when someone who thinks s/he knows the subject and still gets it wrong. Jeff never proclaimed to be an expert, except in PR.

    Fiber in general can withstand some level of fiber, but you loss the protective sheath as you probably already know. If enough heat is applied to fiber, it will melt.

    Customers tend to feel better about an outage when they know what is going on. Its a lot different when they are kept in the dark.

    Great article Jeff, keep it up.

  23. Kim Jones Williamson's reply

    I loved the article! I also have been in the cable industry for 27 years and I got a chuckle out of it!! Great descriptions!

  24. Tireta's reply

    This was very interesting-I feel more knowledgeable now and have a greater respect for our technicians!! Great Job

  25. Dennis Vergason's reply

    After much thought and reading your response I’d like to offer my apology. I should have read the article in the spirit that it was intended and frankly am embarrassed that I felt it necessary to comment at all,I think the “alien cyborg’s reproductive organ” part set me off,again I apologize, feel free to remove my comment.

  26. Andrew's reply

    This was a really good article and I wish more people read it! Thank you Jeff

  27. S. Peoples's reply

    Wow…great insight,if only we could share this with our customers..And when you write “Things fall apart, and humans are all working against decay, chaos and factors beyond our control.” How real is that?

  28. Steve R. Schmiedbauer's reply

    Jeff, I thought your article was on point for the targeted audience. The general public which is comprised of our customers really do not want a lot of technical data thrown their way when it comes to causation of outages, they just want it fixed and fast. The descriptions which you rendered here in your article were just enough to be informative without intimidation, and also your descriptions are that of a writer and not a technician, I enjoyed the pictures that you painted with your words.

    Great job, keep up the good work!
    Great Stuff, keep up the good work.

  29. Ron's reply

    This may be the best corporate blog I’ve ever seen – not just this article, but the entire site. I’m going to use this as an example of how to do it right in our blogging workshops.

Comments for this post are closed – all comment threads close after 14 days from the original post date.


If you are having trouble or need assistance with your phone, cable, or broadband service, please contact our Online Care Team at twcable.help@twcable.com, or at @twcablehelp on Twitter.