Are you refering to Local Control Networks over Ethernet? I have found
that those give plenty of warning as one can see the number of collisions
and missed packets go up as devices are added. I have found that most
PLCs are too slow to overload the network. The Control Logix and Ethernet
I/P is another story, but there is documentation provided by Rockwell that
tells one how to calculate the system through put in packets per second.
Recovery can be done by reducing the packet rate on offending modules.
The most common are HMIs that request data for items that are not on the
screen and at too fast a rate. Recovery can be instantaneous.
Another problem is trying to communicate with device that aren't on the bus.
This can use up connections and queue space that cause problems. It is best
to disable connections to devices that aren't responding to avoid this.
I have seen a Ethernet NIC fail in such a way that it continually tried
to transmit on a network which existed for HMI's/data collection.
Obviously this "broadcast storm" brought the network to a halt. It's the
only time I've seen a NIC (any kind of NIC, be it DH+, Modbus, or
Ethernet) fail. And it was sudden, with no warning.
How many nodes were on the LCN? How does the network run? Is it CSMA/CD?
Or deterministic? If we're going to be able to help you you'll have to
be able to give details about this Honeywell network. I'd guess your
problem was a "screamer" (bad NIC), but I've also seen sloppy HMI
projects that were looking for non-existent nodes or bad data locations
dramatically increase network traffic. Were any changes to the network
or program(s) made prior to the storm?
I have seen when the bandwidth got too high for LCN's. The data just
never got from one space to another, leaving lots of question marks on the
screens as everything timed out. It was a mess. I can provide some
references in private to you if you would like to chat with the fellows who
worked it out.
I have also seen one other problem with LCN's. It seems that they hum.
Nothing musical, but they hum at the same frequency a female rat hums when
she is in estrus. Male rats will mount her and bite her behind the ear
during the encounter. At any rate, rats, especially, um, excited ones,
aren't too picky, and quickly mounted the cable and bit it. Since the sound
was then loudest where he bit it, he returned several times until he had
chewed completely around one cable and almost completely around the backup.
That was one busy rat. The LCN cable was chewed down to the central wire
(it's really just a big coax cable). Once he got far enough on the backup,
everything went down.