ZAP!

[Mark]

Quite today because: tons of hardware fried on Saturday.  Dell 760 onboard NIC's either 100% toast or only working at 100Mbps depending on how lucky you were.  Got a bunch of people to bring in their laptops & chrome books and my TS is chugging along quite nicely. 

Tried adding a PCI NIC and the display would not come up.  Have two PCI slots and the graphics card is in one of them.  Short?  dunno.  Just damage.  Finally acquired some (they only had 5) USB to Ethernet adapters.  Working on those now, but of course I need to get online to grab drivers.

FUN STUFF!

[Jeff Golas]

Check your network wiring for noise - you have an amp you can listen in on the wire with? Something prob putting out tons of EMF, OR one switch may be causing a ground loop - check the power going to all your switches.

I would also check your electrical panels to make sure you don't have a ground wire glowing somewhere. Its happened.

[Mark]

I been tracing cat5 with a toner all day.  I hacve some bad ports.  replaceing with a nice quality Cisco.

[Jeff Golas]

PS - PCI port issue - I think a lot of the modern desktops are setup such that when you put in any card on the 1st PCI(x?) port...they automatically disable the onboard video for the flunkys that don't know any better.

[Jeff Golas]

Ah...just curious what brand? Can I take a guess? N##g##r?

[Jeff Zylstra]

I been tracing cat5 with a toner all day.  I hacve some bad ports.  replaceing with a nice quality Cisco.

Wait.... Was this one of those "POS" Dell switches that bit the dust?  Sounds suspiciously convenient to me!   

[Mark]

yes, POS Dell!!

Long story.  Golas, you do have me curious about noise on the line, but I knwo for sure the switches are all fubar.  At least the Dell are fubar, the HP should have that lifetime warranty for the ports?  Wonder if this is covered by that.

On the dell for example, I have one port with nothing plugged in but the left green LED is on.

[Jeff Golas]

So wait all your switches are toast? Like a Power surge issue or something? Is there a UPS or anything on front of them? Have you tried rebooting em for s--- and giggles? Sounds suspiciously like a network loop.

(Yes HP's have lifetime warranty, Dells I think are really "Foundry" switches I believe)

[Jeff Golas]

I couldn't think of the name of the tool, but an "inductive amp" would be able to show you if any of the cables have some serious noise on them or not (like laying on a flourescent light fixture).

I know when I checked mine, they all had a little noise one way or another, but if one has crazy amounts compared to the others, that could be culprit.

Call me if you want I'm curious to know what's going on - I'm on skype too gn et  ic  86  (remove spaces)

[Mark]

Our entire network is crawling.  Takes like 30 seconds to attach a PDF from a network share to an email.  Don't think Skype would work well! lol

It's a long story and I'm still trying to get everyone up.  All I know is whatever happened hit the network equipment only - switched and routers.  Even had to reboot my firewall twice.  Same thing happened to our tenant in the basement.  He lent me some PCI NICs earlier to test, but like I think I said -- didn't work out.  The NIC's were HOT when I [pulled them out too.  Well, it was just one but I'm not going to backspace and correct that

[Jeff Golas]

Really REEALLY sounding like a network loop! Are you still seeing activity lights or is everything solid on? If your switches are in a star config, and you can possibly play around, try disabling uplinks one at a time to see if everything suddenly comes back online. If so you narrowed it down to a switch...then you have to figure out which port(s) are looping.

I had the nic in our printer-enabled fax machine go stupid one day and took out the entire network, and I even think I had spanning tree enabled!

[Jeff Zylstra]

Sounds like the WOPR is running a Global Thermo-Nuclear War simulation.   

Sorry, couldn't resist that one.  Just watched War Games a couple weeks ago.

[Mark]

There was defiitly a loop at some point, but I have LAG between switches and I've triple checked them.  I have a crappy drawing showing which port is with which LAG and where it terminates.  Everything checks out in that respect (as of now).

I have lights on for empty ports and I have dead ports.  I can move a cable from a blinking for to a dark port and it never lights up.

[Hans Manhave]

I hope this is resolved?  Has any cause been determined?  I remember our lightning strike three years ago killed everything electronic, from switches to NICs, phone system, CCTV to simple magnetic entrance door locks.  Then today the finding of a weirdly behaving NIC in a machine that had "survived" that strike.   

[Mark]

PS - PCI port issue - I think a lot of the modern desktops are setup such that when you put in any card on the 1st PCI(x?) port...they automatically disable the onboard video for the flunkys that don't know any better.

Coming back to this thread now, finally with a barley pop in hand.

This was weird. Was testing my theory that it was in fact the on board nic that was dead. Put in a realtek pci nic in one of the PCs I had easy access to. Knew the cable was in a working port. No lights went on & still got the network cable is unplugged message. Took the realtek out & it was very hot. Switched it with the other one (3 com? Don't remember) and got lights, feeling good. But the monitors never came out of sleep and it didn't even seem like the computer booted properly. Yanked the card & it booted fine, still without a nic though.

I ended up buying out the closest best buy of some belkin 10/100 usb to Ethernet dongles and got 5 workstations back up. Have 5 more being overnighted with my catalyst. This will keep us crawling for now. All of the effected nics are running at 100mbs anyway.

Sent from my ADR6300 using Tapatalk 2

[Mark]

I think all 3 dell 2724 are in some way fried. Have plenty of dead ports, some with LEDs stuck "on", etc. Two were on UPS, one wasn't. Feel like the one that wasn't has more dead ports than the rest. Difficult to tell when you have dead ports on both ends of the wire! Obviously been testing ports with known working gear.

Interestingly, the ricoh mfp has a heavy duty surge protector that the cat5 is also running through. It wasn't effected.

Sent from my ADR6300 using Tapatalk 2

[Mark]

Not really resolved yet, no. Kind of a work in progress.

One thing I don't remember mentioning: when I came in on Saturday to see what was up, the fax@ server was blue screened with "hardware failure. Contact vendor" never seen that before, so just forced a power cycle and it came back up. Half way through the day today, the switchport it was connected to threw in the towel.

Which leads me to: I thought I had a rats nest of cables before. I don't even know what to call the mess I have now! I didn't even bother tracing the cable. I just yanked it and ran a new one around the rack to a working port, lol.

This is just a crazy mess. On top of it all, my car is in the shop again. Unloaded like hell on the garage owner for all the BS earlier this spring and made him take it to a dealership and he's footing the bill. Anyway, I pedaled to work today on a 20" kids bike so it would fit in the back of the van when Amanda came to get me. 30 min ride in at 5:45 central time!

What a day! Thanks for following along! Would have been cool (& helpful) had some of you been able to show up & lend a brain and a hand or two!

Cheers!

Sent from my ADR6300 using Tapatalk 2

[Hans Manhave]

Wondering if any of those switches have that tiny little paperclip hole to reset them and if a connection from a laptop with that special 'setup' cable could push a firmware reinstall.

[Jeff Golas]

Hmm...did you try connecting the on-board nics (the ones suspected bad) to a small linksys 5 port or anything other than the server room switches? Do the NICs still show as working in the OS?

Let's look at this logically...something had to happen. The reason I suspected loop was that I have a feeling that some switches, in loop status, may be so bogged down that the processor can't even switch the LEDs on/off properly. If it were me, knowing the network is already crawling, I'd just momentarily yank a complete LAG to see if everything else comes back to life, OR just consult management that you're taking down the network to find the issue. 10-20 minutes of down time to save 8 hours of frustration shouldn't be too hard to sell. I'd still isolate each switch just to try to find the problem though.

I've seen switches where ports died, and usually they're designed where a single chip runs a group of ports, so its certainly possible...but damn something catastrophic would have to have happen; I'd suspect an environment issue (everything baking in a VERY VERY hot room perhaps) before power related. The power supplies are most likely switchmode, and usually designed using cheapo parts that if over-voltaged, would just pop and go. The supplies themselves should have varistors on them that are the exact same 25c parts used in your $20 surge suppressors. Technical jargon aside I'd be really surprised if the power supply sent an overvoltage that damaged a chip. Not impossible by any means, I just think unlikely.

So let's look at other possibilities...an electrical anomaly, say a bad building ground, could certainly mess up a bunch of gear in a hurry, and another possibility that actually happened to us twice...losing a phase. Run a power supply at 60volts instead of 120 and all they do is make heat instead of juice. They really don't like that! Especially motors on air conditioning units. Was anything else other than computers/network gear messed up? I would think something that catastrophic would damage appliances and building systems as well (when we lost a phase our building AC had some failed motors).

Lightning strike? Very good possibility...although I'd suspect in that case you'd see physical damage..a little brown spot where the NIC chip once was LOL, or burnt transistors. I have see that before. Again I would think that would also damage other stuff, especially electronics like the phone system.

If you're still having issues tomorrow...get a meter and test some outlets, see if you still have 120v. (been there done that too...printers don't like running on 80 volts apparently).

Hope you get it all sorted out tomorrow!

[Bloody Jack Kidd]

That's a crazy one Mark - mysterious. Hope you can resolve it - i'm quite interested in the root cause if you ever figure that out.

[Mark]

I reset the switch with the most down ports. Remember, I took a laptop around and took the cable from a damaged pc and plug into the laptop, the laptop nic would light up if the port was good. That was one method used to determine if it was the pc nic or the switchport. Now, i didn't do this for every single pc, but for enough to satisfy my curiosity. Others I just traced and moved to a known good port.

I also don't think it is power supply damage. The switches are on a UPS except for the one with the most bad ports. The one not on a ups didn't even trip the surge protector. I also power cycled all switches more than once.

I don't think our building got hit. It's certainly not the tallest in the area and I highly doubt its the fastest path to ground. There is a cell tower behind some stuff across the street. Can see it clearly out the window (distance measurable in yards) and you know those things are grounded with expectations that they WILL get hit.

The guy downstairs had the same odd behavior with his switches.

I have 3 Dell, 1 ProCurve, and network printers are on a very old 10/100 Catalyst - the catalyst is completely fine. Still has the same dead ports that took it out of commission years ago (brought it back just for printers).

All 3 dell have same issues, different port. Not 100% sure about the ProCurve. It seems mostly working aside from two ports that must be dead. I power cycled ALL switches for good measure.

Also had to cycle the firewall twice, though it seems just fine now.

Sent from my ADR6300 using Tapatalk 2

[Mark]

That's a crazy one Mark - mysterious. Hope you can resolve it - i'm quite interested in the root cause if you ever figure that out.

Yeah, don't know how i will ever get a confident answer. Maybe the insurance company will once we file a claim which I suspect we will.

I was wondering if it couldn't be some how related to our proximity to a nearby lightening strike. Network cards are supposedly the most susceptible circuit next to power supplies - and again, doesn't seem logical to have come through power line. I didn't have much time to Google yesterday, but dis see a fee things mentioning how sensitive a nic could be.

Would be interested if anyone else finds some good info Googleing.

Sorry if I mentioned this already, but our VP/Principe was in the office on Saturday when she heard a boom so loud she did think the building was hit. Loudest she's ever heard in her life. But, power didn't go out, only internet did. Event logs on fax@ server show an unexpected shutdown at 12:58pm. Which correlates with the loud boom. Fax server connected to phone lines, network gear connected to phone company (ISP), Cisco gear seems un effected, Toshiba pbx unaffected EXCEPT that we had to unplug/replug a handful of phones, which get theory power from the pbx. Asterisk server also seemingly un effected & even has 8 pots lines directly to the Toshiba & accepts the pri directly from tds.

Lots of details but none of it makes any sense. How is it all connected?

CraZy.

Oh, & my workstation has been "loading personal settings" for 39 minutes this morning.

Sent from my ADR6300 using Tapatalk 2

[Billy Welsh]

We had a switch and some 3Com phones (which connect to LAN) get fried via a lightning strike that came in via phone lines as best as we could figure.  All components were on UPS but the LAN and phone lines were not run through these.

Similar situation in that logic did not apply - phone PBX was fine but some phones fried; computer switch fried but phone switch fine; some pc's were stubborn in reconnecting to LAN but eventually all did so after enough "rest" and reboots.

Happened at a branch office with no real on-site tech support, so could not spend a lot of time with detective work.  Sent an employee to local Best Buy or equivalent for new switch and got them back up as soon as possible walking employee through install over cell phone.  Small office that was on an old switch anyway, so very easy decision to just spend the $.

I did keep the fried switch I believe, with the intent of testing more intensely at a later date, but never did - still in storage.  Before we swapped it out there was some intermittent connectivity on some ports.

[Jan Regnier]

For what it's worth...I have been monitoring our power outages here the last couple of weeks because we did have a storm - power was out - went out 13 times in 5 days.  Never for a long time other than the storm outage but it concerns me that it is up and down so often. 

[Hans Manhave]

Loud boom could be a transformer blowing up. That is what lightning did to us. As we get residential power and all around us have power from the business road, I don't know who else was affected.  7000volt lines go to those things. Possibly a nearby unit blew and created a surge, either up or down.  Lightning striking a field knocked out a bunch of old fashioned serial ports and modems in the '80s.  Isn't it a lovely profession to be able to have these experiences?

[TrishaOurs]

YIKES...Good Luck with everything!

[Charlie Charbonneau]

It does sound like that lightning strike weaseled it's way into your network(and the guy downstairs) via your network instead of through power lines and played a little havoc. 

Best suggestion I would have would be to take everything down and bring it all back up one at a time to narrow down your trouble zones.  Can you get your hands on a new switch or borrow one somewhere?  Bypass your entire network and only include your firewall, server and your workstation?  See how that runs?   Then add another station to the new switch? Test connectivity/availability and move on? 

That would help you narrow down your trouble spots and get the working stations back up and working.  You may have been there done that, but if those switches are bad, you can hunt for days!   What does lightning do to Cat wiring?

[Jeff Zylstra]

It does sound like that lightning strike weaseled it's way into your network(and the guy downstairs) via your network instead of through power lines and played a little havoc. 

Best suggestion I would have would be to take everything down and bring it all back up one at a time to narrow down your trouble zones.  Can you get your hands on a new switch or borrow one somewhere?  Bypass your entire network and only include your firewall, server and your workstation?  See how that runs?   Then add another station to the new switch? Test connectivity/availability and move on? 

That would help you narrow down your trouble spots and get the working stations back up and working.  You may have been there done that, but if those switches are bad, you can hunt for days!   What does lightning do to Cat wiring?

I think Charlie has the right idea.  You don't have any choice but to strip your network down to the bare necessities and then add things until it fails.  If that all comes up OK, then maybe wireshark or some packet sniffing device/software might show some nefarious network traffic.  This reminds me of some of the virus hoax emails debunked on Snopes that overload and burn out your network in a shower of sparks!

Whatever happens, I would be leery to plug in too many "good" computers/nics into something that may be putting out volts rather than millivolts like it should be.  I've got a cheap ($10) Chinese network testing receiver/transmitter that has LEDs for the 8 wires and shows basic connectivity.  I would plug that in, but I would be hesitant to plug in anything that you think might be "good".  Just sayin'. 

After you get up and running, you may want to talk to an electrician about a whole building surge suppressor that goes on your electrical panel.   Once bitten, twice shy.  I hope your Amityville horror ends soon!

[Billy Welsh]

Just amazing how such a thin copper wire can carry enough current to fry everything. 

[Jeff Zylstra]

Just amazing how such a thin copper wire can carry enough current to fry everything.

It is.  Just think about how many volts your spark plug wires carry.  Used to be that the original GM electronic ignitions would put out close to 60,000 volts of juice, but they killed a couple of mechanics, so they cut them back (or so the story goes).  But all of that voltage runs through some pretty thin wire in your spark plug wires.  Low amperage, but high voltage.

[Jeff Golas]

I think some spark plug wires are now just carbon powder to make em flexible.

I'd still look at outlet voltages and consider grounding issues...

[Jeff Zylstra]

I think some spark plug wires are now just carbon powder to make em flexible.

I'd still look at outlet voltages and consider grounding issues...

Agreed on both points.  Especially the loss of a phase.  Our air conditioner blows a fuse for one of the phases about every 10 years.  Of course, it's usually about 95 degrees out when that happens.   Also, the tool and die place a block away gave us fits for a while when they'd use their new heavy press.  It would cause a brownout and all kinds of weird behavior with anything electrical.  They finally upgraded the transformer and we've had no issues since.

[Mark]

With the new switch in place, I have a new revelation. My workstation can only run at half duplex. That's 100mbps half duplex.

Love the info you can get from a managed switch. Makes my 40 minute login a little.clearer now. In fact, my profile is probably trashed due to this as well. I kept getting a registry recovery.

Oh well. Done all I could today.

The good news is we may likely move to VDI ASAP. Or after TENCon, then ASAP.

Sent from my ADR6300 using Tapatalk 2

[Jeff Golas]

Make sure your NIC drivers are up to date...amazing when you go back to a workstation that was rolled out 2 years ago to see drivers from 2009 (what used to be current) on them, only to find the nic drivers were updated a month ago lol.

In some other cases I've been able to get things to work better by specifying speed/duplex under the NIC properties instead of letting it auto-detect.

[Mark]

If I didn't have all this sudden and obvious damage with at least 16 machines, I'd do that.  But, it's clear there is damage and this aint software related so I'm not going to bother.

Off to Best Buy for 3 more USB to ethernet dongles.

[Bloody Jack Kidd]

Have you declared a state of emergency yet?

[Mark]

Not really.  We're running now.  No need for Agility to come in at the moment

[Jeff Zylstra]

And I was already to send you my pith helmet!

[Jeff Zylstra]

Everything squared away, Mark?  Disasters always fascinate the masses, especially when it's someone ELSE'S disaster.       Hope you're 100%.

[Mark]

Everyone is up and running but we haven't replaced anything yet. Gotta go through some motions before we do, and we are seriously leaning towards VDI so it may be a while before we do replace.

I'll definitly keep this thread updated.

Thanks!

Sent from my ADR6300 using Tapatalk 2

[Jeff Zylstra]

Glad to hear it.  I was under the impression that some of your network equipment was fried and non-functional, so I'm interested in hearing what the culprit was.

[Jeff Golas]

Would VDI be the answer though? Wouldn't the VDI hardware be just as suseptible to whatever happened as any regular desktop, the advantage being the desktop could accept USB nics (or even a PCI nic) whereas a thin/zero client might not?

[Mark]

@JZ: yes, we lost ports on switches so did get a new switch. We lost ports on workstations so they are using usb dongles for now.

@JG: VDI is just the route we want to go regardless. Zero clients will be cheaper & easier to replace than a full desktop & almost no setup time. They do have usb ports and with PCoIP you can scan & can attach devices locally as if it were a full pc.

I plan to put a surge protector between the ISP & our equipment as well.

Sent from my ADR6300 using Tapatalk 2