Software Bug Contributed to Blackout Kevin Poulsen Feb 11 2004
A previously-unknown software flaw in a widely-deployed General Electric energy management system contributed to the devastating scope of the August 14th northeastern U.S. blackout, industry officials revealed this week.
[Unknown as it didn't exist until it was needed as a scapegoat in order to distract from the real reason. Sounds to me like FirstEnergy trying to deflect blame onto GE ]The bug in GE Energy's XA/21 system was discovered in an intensive code audit conducted by GE and a contractor in the weeks following the blackout, according to FirstEnergy Corp., the Ohio utility where investigators say the blackout began.
'It had never evidenced itself until that day," said spokesman Ralph DiNicola. "This fault was so deeply embedded, it took them weeks of poring through millions of lines of code and data to find it.'
[ Who is this contractor ? What code are they referring to here ? How were the tests conducted ? Did they include any other systems that were involved in the BlackOut ? How many of the SCADA units running on FirstEnergy were Microsoft Windows ?"FirstEnergy was aware the alarm system was broken, said company spokesman Ralph DiNicola. A functioning backup alarm at the Midwest Independent System Operator, a nonprofit power pool that oversees the region's electrical grid, was in place," DiNicola said.
The flaw was responsible for the alarm system failure at FirstEnergy's Akron, Ohio control center that was noted in a November report from the U.S.-Canadian task force investigating the blackout. The report blamed the then-unexplained computer failure for retarding FirstEnergy's ability to respond to events that lead to the outage, when quick action might have limited the blackout's spread.
Power system operators rely heavily on audible and on-screen alarms, plus alarm logs, to reveal any significant changes in their system's conditions," the report noted. FirstEnergy's operators "were working under a significant handicap without these tools. However, they were in further jeopardy because they did not know that they were operating without alarms, so that they did not realize that system conditions were changing.
[.. TRANSCRIPTS of telephone conversations .. include explicit mention of some unknown 'computer problems' at FirstEnergy, the Ohio utility thought to have triggered the regional power failures, in those preceding hours.Early on, a controller at the Midwest Independent System Operator asked his counterpart at FirstEnergy why it hadn't reacted to a transmission line outage. The utility's technician replied:
"We have no clue. Our computer is giving us fits, too. We don't even know the status of some of the stuff around us."
"I called you guys like 10 minutes ago, and I thought you were figuring out what was going on there."
"Well, we're trying to. Our computer is not happy. It's not cooperating either." ]
The cascading blackout eventually cut off electricity to 50 million people in eight states and Canada.
The blackout occurred at a time when the Blaster computer worm was wreaking havoc across the Internet. The timing triggered some speculation that the virus may have played a role in the outage -- a theory that gained credence after SecurityFocus reported that two systems at a nuclear power plant operated by FirstEnergy had been impacted by the Slammer worm earlier in the year.
[ "On January 25, 2003, Davis-Besse nuclear power plant was infected with the MS SQL Server 2000 worm. The infection caused data overload in the site network, resulting in the inability of the computers to communicate with each other."Instead, the XA/21 bug was triggered by a unique combination of events and alarm conditions on the equipment it was monitoring, DiNicola said. When a backup server kicked-in, it also failed, unable to handle the accumulation of unprocessed events that had queued up since the main system's failure.
Because the system failed silently, FirstEnergy's operators were unaware for over an hour that they were looking at outdated information on the status of their portion of the power grid, according to the November report.
[What were these 'unique combination of events and alarm conditions' ? ... This is Poulson in an earlier article about the earlier systems crash at a Nuclear Plant."In that article, Poulsen offers a detailed description of how another Microsoft worm, Slammer, crashed two Unix-based control systems at the Davis-Besse nuclear power plant in Northern Ohio also operated by FirstEnergy.
Poulsen reported that FirstEnergy engineers had bridged the nuclear plant's control network with FirstEnergy's corporate network -- a practice that is increasingly common among utility companies, according to industry and security experts."
"The root cause of the outage was linked to .. trees .. FirstEnergy says .. its role in the outage is overstated in the interim report"
[shuffle .. shuffle]from
"The existence of both internal and external links from SCADA systems to other systems introduced vulnerabilities."
Reliable, Field-Proven & Adaptable
The XA/21 transmission management system controls generation and the high voltage transmission network for optimal generation and transmission of power.
One of the industry's most advanced EMS/SCADA systems, the XA/21 system combines advanced open systems architecture with full graphics, power system application, historical information storage and retrieval and relational database technology.
With well over one million hours of online operation, the XA/21 system has improved utilities' bottom lines by helping to: .. ? Enhance operational efficiency ..
quote from Bill Gates, Feb 14 1998
" ... It would help me immensely to have a survey showing that 90 percent of developers believe that putting the browser into the OS makes sense. ... Ideally, we would have a survey like this done before I appear at the Senate on March 3rd."
Bill Gates Feb 2004 .. what he might have said :-D
'It would help me immensely to have a survey showing that the blackout was caused by Unix ... Ideally, we would have a survey like this done before I appear at the RSA Conference in Feb'