ZAP!

[Mark]

I think all 3 dell 2724 are in some way fried. Have plenty of dead ports, some with LEDs stuck "on", etc. Two were on UPS, one wasn't. Feel like the one that wasn't has more dead ports than the rest. Difficult to tell when you have dead ports on both ends of the wire! Obviously been testing ports with known working gear.

Interestingly, the ricoh mfp has a heavy duty surge protector that the cat5 is also running through. It wasn't effected.

Sent from my ADR6300 using Tapatalk 2

[Mark]

Not really resolved yet, no. Kind of a work in progress.

One thing I don't remember mentioning: when I came in on Saturday to see what was up, the fax@ server was blue screened with "hardware failure. Contact vendor" never seen that before, so just forced a power cycle and it came back up. Half way through the day today, the switchport it was connected to threw in the towel.

Which leads me to: I thought I had a rats nest of cables before. I don't even know what to call the mess I have now! I didn't even bother tracing the cable. I just yanked it and ran a new one around the rack to a working port, lol.

This is just a crazy mess. On top of it all, my car is in the shop again. Unloaded like hell on the garage owner for all the BS earlier this spring and made him take it to a dealership and he's footing the bill. Anyway, I pedaled to work today on a 20" kids bike so it would fit in the back of the van when Amanda came to get me. 30 min ride in at 5:45 central time!

What a day! Thanks for following along! Would have been cool (& helpful) had some of you been able to show up & lend a brain and a hand or two!

Cheers!

Sent from my ADR6300 using Tapatalk 2

[Hans Manhave]

Wondering if any of those switches have that tiny little paperclip hole to reset them and if a connection from a laptop with that special 'setup' cable could push a firmware reinstall.

[Jeff Golas]

Hmm...did you try connecting the on-board nics (the ones suspected bad) to a small linksys 5 port or anything other than the server room switches? Do the NICs still show as working in the OS?

Let's look at this logically...something had to happen. The reason I suspected loop was that I have a feeling that some switches, in loop status, may be so bogged down that the processor can't even switch the LEDs on/off properly. If it were me, knowing the network is already crawling, I'd just momentarily yank a complete LAG to see if everything else comes back to life, OR just consult management that you're taking down the network to find the issue. 10-20 minutes of down time to save 8 hours of frustration shouldn't be too hard to sell. I'd still isolate each switch just to try to find the problem though.

I've seen switches where ports died, and usually they're designed where a single chip runs a group of ports, so its certainly possible...but damn something catastrophic would have to have happen; I'd suspect an environment issue (everything baking in a VERY VERY hot room perhaps) before power related. The power supplies are most likely switchmode, and usually designed using cheapo parts that if over-voltaged, would just pop and go. The supplies themselves should have varistors on them that are the exact same 25c parts used in your $20 surge suppressors. Technical jargon aside I'd be really surprised if the power supply sent an overvoltage that damaged a chip. Not impossible by any means, I just think unlikely.

So let's look at other possibilities...an electrical anomaly, say a bad building ground, could certainly mess up a bunch of gear in a hurry, and another possibility that actually happened to us twice...losing a phase. Run a power supply at 60volts instead of 120 and all they do is make heat instead of juice. They really don't like that! Especially motors on air conditioning units. Was anything else other than computers/network gear messed up? I would think something that catastrophic would damage appliances and building systems as well (when we lost a phase our building AC had some failed motors).

Lightning strike? Very good possibility...although I'd suspect in that case you'd see physical damage..a little brown spot where the NIC chip once was LOL, or burnt transistors. I have see that before. Again I would think that would also damage other stuff, especially electronics like the phone system.

If you're still having issues tomorrow...get a meter and test some outlets, see if you still have 120v. (been there done that too...printers don't like running on 80 volts apparently).

Hope you get it all sorted out tomorrow!

[Bloody Jack Kidd]

That's a crazy one Mark - mysterious. Hope you can resolve it - i'm quite interested in the root cause if you ever figure that out.

[Mark]

I reset the switch with the most down ports. Remember, I took a laptop around and took the cable from a damaged pc and plug into the laptop, the laptop nic would light up if the port was good. That was one method used to determine if it was the pc nic or the switchport. Now, i didn't do this for every single pc, but for enough to satisfy my curiosity. Others I just traced and moved to a known good port.

I also don't think it is power supply damage. The switches are on a UPS except for the one with the most bad ports. The one not on a ups didn't even trip the surge protector. I also power cycled all switches more than once.

I don't think our building got hit. It's certainly not the tallest in the area and I highly doubt its the fastest path to ground. There is a cell tower behind some stuff across the street. Can see it clearly out the window (distance measurable in yards) and you know those things are grounded with expectations that they WILL get hit.

The guy downstairs had the same odd behavior with his switches.

I have 3 Dell, 1 ProCurve, and network printers are on a very old 10/100 Catalyst - the catalyst is completely fine. Still has the same dead ports that took it out of commission years ago (brought it back just for printers).

All 3 dell have same issues, different port. Not 100% sure about the ProCurve. It seems mostly working aside from two ports that must be dead. I power cycled ALL switches for good measure.

Also had to cycle the firewall twice, though it seems just fine now.

Sent from my ADR6300 using Tapatalk 2

[Mark]

That's a crazy one Mark - mysterious. Hope you can resolve it - i'm quite interested in the root cause if you ever figure that out.

Yeah, don't know how i will ever get a confident answer. Maybe the insurance company will once we file a claim which I suspect we will.

I was wondering if it couldn't be some how related to our proximity to a nearby lightening strike. Network cards are supposedly the most susceptible circuit next to power supplies - and again, doesn't seem logical to have come through power line. I didn't have much time to Google yesterday, but dis see a fee things mentioning how sensitive a nic could be.

Would be interested if anyone else finds some good info Googleing.

Sorry if I mentioned this already, but our VP/Principe was in the office on Saturday when she heard a boom so loud she did think the building was hit. Loudest she's ever heard in her life. But, power didn't go out, only internet did. Event logs on fax@ server show an unexpected shutdown at 12:58pm. Which correlates with the loud boom. Fax server connected to phone lines, network gear connected to phone company (ISP), Cisco gear seems un effected, Toshiba pbx unaffected EXCEPT that we had to unplug/replug a handful of phones, which get theory power from the pbx. Asterisk server also seemingly un effected & even has 8 pots lines directly to the Toshiba & accepts the pri directly from tds.

Lots of details but none of it makes any sense. How is it all connected?

CraZy.

Oh, & my workstation has been "loading personal settings" for 39 minutes this morning.

Sent from my ADR6300 using Tapatalk 2

[Billy Welsh]

We had a switch and some 3Com phones (which connect to LAN) get fried via a lightning strike that came in via phone lines as best as we could figure.  All components were on UPS but the LAN and phone lines were not run through these.

Similar situation in that logic did not apply - phone PBX was fine but some phones fried; computer switch fried but phone switch fine; some pc's were stubborn in reconnecting to LAN but eventually all did so after enough "rest" and reboots.

Happened at a branch office with no real on-site tech support, so could not spend a lot of time with detective work.  Sent an employee to local Best Buy or equivalent for new switch and got them back up as soon as possible walking employee through install over cell phone.  Small office that was on an old switch anyway, so very easy decision to just spend the $.

I did keep the fried switch I believe, with the intent of testing more intensely at a later date, but never did - still in storage.  Before we swapped it out there was some intermittent connectivity on some ports.

[Jan Regnier]

For what it's worth...I have been monitoring our power outages here the last couple of weeks because we did have a storm - power was out - went out 13 times in 5 days.  Never for a long time other than the storm outage but it concerns me that it is up and down so often. 

[Hans Manhave]

Loud boom could be a transformer blowing up. That is what lightning did to us. As we get residential power and all around us have power from the business road, I don't know who else was affected.  7000volt lines go to those things. Possibly a nearby unit blew and created a surge, either up or down.  Lightning striking a field knocked out a bunch of old fashioned serial ports and modems in the '80s.  Isn't it a lovely profession to be able to have these experiences?

[TrishaOurs]

YIKES...Good Luck with everything!

[Charlie Charbonneau]

It does sound like that lightning strike weaseled it's way into your network(and the guy downstairs) via your network instead of through power lines and played a little havoc. 

Best suggestion I would have would be to take everything down and bring it all back up one at a time to narrow down your trouble zones.  Can you get your hands on a new switch or borrow one somewhere?  Bypass your entire network and only include your firewall, server and your workstation?  See how that runs?   Then add another station to the new switch? Test connectivity/availability and move on? 

That would help you narrow down your trouble spots and get the working stations back up and working.  You may have been there done that, but if those switches are bad, you can hunt for days!   What does lightning do to Cat wiring?

[Jeff Zylstra]

It does sound like that lightning strike weaseled it's way into your network(and the guy downstairs) via your network instead of through power lines and played a little havoc. 

Best suggestion I would have would be to take everything down and bring it all back up one at a time to narrow down your trouble zones.  Can you get your hands on a new switch or borrow one somewhere?  Bypass your entire network and only include your firewall, server and your workstation?  See how that runs?   Then add another station to the new switch? Test connectivity/availability and move on? 

That would help you narrow down your trouble spots and get the working stations back up and working.  You may have been there done that, but if those switches are bad, you can hunt for days!   What does lightning do to Cat wiring?

I think Charlie has the right idea.  You don't have any choice but to strip your network down to the bare necessities and then add things until it fails.  If that all comes up OK, then maybe wireshark or some packet sniffing device/software might show some nefarious network traffic.  This reminds me of some of the virus hoax emails debunked on Snopes that overload and burn out your network in a shower of sparks!

Whatever happens, I would be leery to plug in too many "good" computers/nics into something that may be putting out volts rather than millivolts like it should be.  I've got a cheap ($10) Chinese network testing receiver/transmitter that has LEDs for the 8 wires and shows basic connectivity.  I would plug that in, but I would be hesitant to plug in anything that you think might be "good".  Just sayin'. 

After you get up and running, you may want to talk to an electrician about a whole building surge suppressor that goes on your electrical panel.   Once bitten, twice shy.  I hope your Amityville horror ends soon!

[Billy Welsh]

Just amazing how such a thin copper wire can carry enough current to fry everything. 

[Jeff Zylstra]

Just amazing how such a thin copper wire can carry enough current to fry everything.

It is.  Just think about how many volts your spark plug wires carry.  Used to be that the original GM electronic ignitions would put out close to 60,000 volts of juice, but they killed a couple of mechanics, so they cut them back (or so the story goes).  But all of that voltage runs through some pretty thin wire in your spark plug wires.  Low amperage, but high voltage.