The worst part is cleaning up the mess produced by those people who
don't want to use the tools that already exist but want to reinvent
the wheel, to go inside and tinker with the low level programming
rather than looking at how to control the process.
Well, maybe I shouldn't say its the worst part, because it can be
quite profitiable for those of use doing the cleanup. ;-}
It might be profitable, but it can be a royal pain in the hind quarters. It
also exposes too many people to process risks if the changeover is on a live
system (as they all seem to be). Until the whole mess is cleaned up, every
time something fails we have to wake the super genius up and get him to fix
it (or fire him). I'd rather have a nice, uniform, off the shelf system
with strong vendor support.
Guys, Guys, no need to be harsh.
Have you seen the latest add in the 'Linux Journal' ? It seems the DOE
Bechtel, and others have recognized the need to make SCADA systems and
process controls SECURE....
They are actively seeking engineers and scientists with Controls and
Linux experience. (Hummm, I wonder why they just don't ask a few
Good_ole_boys from the ISA to solve their problems)?
Maybe the ISA is not quite up to snuff?
check it out, check it out:
I just don't think Microsoft et. al. will be providing a secure SCADA
systems, they cannot even keep kids out of their networks.....
Good_ole_boys from the ISA to solve their problems)?<<
Maybe you should look at the individuals who are behind the effort to
make SCADA and other critical process control systems secure.
Answer the ad and see who you will be working for.
Yeah, it is a pain. I remember once bitching to my boss who was very
impressed by one of these "I can do every thing so much better" guys. In
our town we have an annual cowboys and Indians parade (Calgary Stampede) and
everybody knows what follows the horses around. I told my boss, "I am sick
and tired of being the sweeper that follows Don XX around wherever he goes."
It took a few years but finally it sunk in.
Yeah, it is a pain. I remember once bitching to my boss who was very
Quite a pain. A company I worked for made the mistake of hiring people
who thought they could do a better job than established vendors by
writing their own programs in C, C+, etc., buying analog I/O boards,
etc. Each made a mess, didn't get the project finished, or, in the one
case where a project was completed and worked, left with no way for
anyone else to modify the control (no documentation, no way to easily
get replacement equipment, etc.) We went through about four of these
young geniuses before the company learned its lesson. Now, if you
apply for employement and talk about your skills in C, design of
operating systems, etc. it is the end of the interview. OTOH, if you
have a good background in PLCs, ladder logic, DCS configuration, and
other types of standard, vendor supported controls, you may well get a
Too many of us spent too much time cleaning up messes.
I just when I say that it might be profitable, but it is such a pain
in the butt, and often follows significant damage, that the work is
very unpleasant. There are better ways of making a living.
As for having to wake up super genius, by the time the cleanup begins
he is usually long gone, either having been fired or anticipating the
Real engineers don't reinvent the wheel for control problems. They use
proven equipment where vendor support will be around afte they have
left for other projects. On the other hand there are tinkerers. I
would not want a tinkerer involved in a project where significant
economic damage or even harm to humans could occur.
Control system work needs to be done by adults.
Funny thing is I usually have to make things work, where others have
failed (young and old alike). You definately are not talking about my
work, because I never leave anything incompleted. Furthermore, the
knowledge transfer, or training, is very easy with linux based
controls, because you get the sourcecode to the application, as well
as the RTOS the software runs on. I write code in ansi C and only
minimal parts in the native assembler of the microcontroller. That way
it is very easy to port to another PLC or another microcontroller.
Ever wonder why semiconductor vendor call devices that are design to
control things, microcontrollers?
TI, Motorola, Siemens, Microchip, Atmel, and many many other
semiconductor companies make 'microcontrollers'.... Go on, learn
something new as you just might find it, stimulating.
Have you guys seen Sixnet? They have a mixture of PLC and DCS modules.
The underlying RTOS is linux, in case you are interested...... I know
several large utilities that are switching the entire control
infrastructure to these 'embedded linux devices' or PLCs.....
After all, when a new PLC vendor enters the market, are they guilty of
the things you folks are blasting me about?
Look at the title of this thread, "Is ISA a rip-off?" Another person
asked that question. I tried to show alternatives, because many
persons in the controls business, are unemployed or underemployed.
Maybe this reader is dissatisfied with the ISA. I tried to show this
person hope and a way to learn about new things, including ways to
perform controls designs at a lower price than purchasing something
from a name vendor. Is it the best you folks can do to belittle
somebody who is trying to help those searching for something....
I hope that life is not cruel to you guys, or your smuge attitudes.
I also hope that the spirit of God finds you and unburdens your souls.
With all due respect for your programming prowness, do not be so sure I'm
not talking about your work. I've had to clean up after some rather fancy
programmers who thought they were hot stuff. What you regard as complete
could be woefully inadequate when the specs change after you thought you
were done. I don't want control engineers worrying about C programming of
basic loops. The vendor worries about that. We pay the vendor to worry
about that. Actually, control engineers shouldn't be worrying about much
programming at all, but rather on making the process they are trying to
control work properly. I would strongly consider firing any control
engineer who played with the assembly language of an industrial
microcontroller at a chemical plant. The damage done by reprogramming
those controllers, even if it is correctly done, can be significant.
If you want to make controllers, then you are looking at the right process.
If you want to do process control, you are asking for trouble. If you do
wish to do this, then make everything follow a good, rigid standard.
We might have a disagreement here on just what we are controlling. I would
like to control a chemical manufacturing facility. This typically uses
one or more DCS systems, a few PLC's, and a lot of off the shelf
transducers, transmitters, valves, etc. I expect them to work properly as
soon as they are installed. I can't take the time to program each of them.
I do have the time to write supervisory systems to make sure that they are
all given the correct setpoints, but that is a level at least two layers
removed from what you are talking about.
If you are thinking of making controllers, programming them, validating
them, and selling the standardized components of your system, then you are
considering doing the right thing.
When a PLC vendor enters the market, they will have completed all the stuff
you are talking about, and will provide their complete, validated product to
the world. It is up to them to work out all the compatibility issues,
architecture, etc. It is up to the users to purchase,the device, configure
it (with a short parameter list) and install it.
The vendor assumes a fantastic level of liability. I've seen negitiations
in which the vendor agreed to assume infinite liability for product failure.
The only way they can do this is to test the living daylights out of their
products. I would not want a plant control engineer trying to do that on
their own equipment.
I'm quite certain I seen 'attitudes' such as yours when computers(DCS)
and PLCs(microcontrollers with a RTOS and object oriented assember you
refer to as 'ladder logic') were first introduced to the
pneumatic/hyraulic/manual folks in the process industries. It's OK to
have your opinions of how to make things (like complicated chemical
processes) work, but, your 'way' is not the only valid way. Engineers
can learn process controls and how to program. In fact programming in
ladder logic is not much different than programming in assembler with
structured methodologies... Many people have caused damage in
industry, and this damage is not limited to (or even dominated by)
folks that do not follow your rigid professional beliefs.
Like 1131 or 61131? The linux community is leading, not following on
many standards. The threat the linux/openRTOS movements poise is that
companies and industries will not be locked into a product line from a
proprietary vendor. This paradigm shift is unconfortable to some,
most will benefit and adjust. Either way, it is enevitable. Linux runs
on IBM mainframes and can be used on a 16bit microprocessor. It has
been ported to more different processors that microsoft and WindRiver
and Greenhills and PSOS combined(yes you should learn about where
these PLC vendors actually get their RTOSes from).
Maybe in your mind, but not in reality. Control theory is control
theory. The underlying Microprocessors, Microcontrols, DSPs and FPGA
used to control Avionics, Chemical, Process, and medical processes,
such as spectroscopy, are the same. A tiny 8 bit microprocessor with a
transistorn and send out logic signals, and control the larges valve
around. Sensors and transducers work very well with microcontrollers
as well as $5,000 PLC that you seem to be comfortable using.
Haven't you ever wondered why NASA does not purchase a bunch of PLCs
to control the space shuttle with? Certainly there are chemical
processes on the shuttle that you would consider demanding?
This is all part and parcel with using embedded linux to perform
traditional control functions. Just because an engineer or technician
uses hardware and an embedded linux RTOS, does not mean they are going
to follow all sorts of bad methodologies. What you assume is mutually
exclusive is in fact complimentary.
You want to see a really bad idea for industrial controls? Let's look
at a historical mistake of biblical proportions made by microsoft:
Although this is a deprecated security hole, it was an unbelievably
stupid mistake by microsoft and the NSA. It is one of the primary
reasons Europe and Asia are leaving all things microsoft behind.
My friend (I do hope we will become friends, even pals...) you assume
to much. I have performed extensive work in the traditional controls
industry, product development, and communications security. These
fields are actually quite inter-twined. You assume these product
vendors are doing much, much more than they actually are. Just like
assuming microsoft will keep your networks secure, you assume too
much. Trust, but Verify, is the axiom of wise engineers and others.
If you cannot read the sourcecode that forms the RTOS on the PLC, then
you are assuming somebody, possible a foreigner with unknown
allegences, is doing what you want. You assume far too much.
Nothing was wrong with the World Trade Centers, until 11sep01.....
If you are planning on suing a vendor as your insurance premium, you
might want to rethink your strategies for catestrophic recovery
financing. Corporations are mostly worthless, as the real cash is such
out, and vendors have many many methods to basically say 'caveat
emptor', in a court of law.
NASA has a very large budget which most control departments don't have.
They also needed to move around a terriffic amount of specialized data in a
big hurry, analyze it, and act upon it properly. They basically wrote
their own DCS system on some VAXes. They then spent a lot of time verifying
the code, then built up the control systems on top of it. The whole thing
was tested for years before it ever left the ground. NASA (actually their
contractor) also had a small platoon of programmers working on the whole
Most chemical process are not that demanding. The basic controls we need
are pretty much available, with improvements coming out all the time. The
basic method is to figure out what component is needed in an area, buy it,
and install it.
If the controls are embedded, then my technician doesn't have to know
anything about them except where to connect the wires and what buttons to
push to make it work. Considering how much he has to do in a day, that's
just fine with both of us.
If I can tell that a particular device is Linux, or VMS, or whatever, and
the operators need to treat it accodingly, then that device is not suitable
for control. The device should look like a control device and be thought of
in process terms. I want operators to think of processes, not OS's.
Microsoft products are not used in the control systems I've worked on.
I'm assuming less than you might think. After writing a good deal of real
time supervisory controls and working with regulatory control systems I'm
familiar with what is supportable by normal plant personnel. I've seen too
many systems which worked great until the engineer who wrote it left
(voluntarily or otherwise). After that, we discovered several thousand,
sometimes tens of thousands of lines of spaghetti, hack jobs, kludges, and
workarounds. Entire systems had to be rewritten from the ground up, in some
cases rather quietly to avoid irritating the new manager with the idea that
his code was no longer used. No plant manager likes to be one personnel
loss away from disaster.
What is needed is for you to be able to walk away from whatever devices and
control systems you created, have a technician or another (not so qualified)
engineer walk in, take a quick look at 2 am, and be able to correct whatever
is wrong. If something is broken, he should be able to make a phone call
and get the right stuff in within a few hours. If your head instrument
electrician can't understand your system, you put the wrong system in.
I do trust and verify whatever systems I'm working on. Consider that a
typical PLC has between 20,000 and two million lines of code in its OS. Do
I really have to go through and verify each line? Should I schedule
structured walkthroughs, and FMA's on each OS subroutine? No. If I find
something wrong, who fixes it? Your suggestion would require a veritable
army of programmers at each plant. Going further, it would require some
very rigid standards to assure that this entire army programmed everything
the same way. Every plant would end up having a little different flavor of
everything, meaning that we couldn't take advantage of economies of scale.
Additionally, each device could end up being a "serial number 1".
In reality I have found bugs in the OS of several computers, including a
high end DCS system. When we notified them of the error, they investigated
it, thanked us, and gave us a patch in about 2 days. It would have taken me
weeks to attain the code familiarity I'd need to fix that problem. Instead,
I spent weeks working on matters more closely related to the process.
When we put in large control systems, there is a long checkout process to
verify that each control system component works properly. These focus just
on the control configuration, failsafes, device failures, etc. If we had to
go into how each microprocessor was programmed, it would take forever.
These infinite liability agreements are part and parcel of the trade. The
vendor must carry significant insurance to cover it.
James ( firstname.lastname@example.org) wrote:
: Haven't you ever wondered why NASA does not purchase a bunch of PLCs
: to control the space shuttle with? Certainly there are chemical
: processes on the shuttle that you would consider demanding?
The Shuttle launch system still uses Modcomp minis installed by IBM
back in the 1970s:
The shuttle launch prosessing system
The replacement project was killed in 2002:
NASA Kills 'Wounded' Launch System Upgrade at KSC
"...Instead of cutting the cost of running Launch Control Center computers
in half as NASA promised back in 1997, the independent review found
the new computer system -- even if could be completed -- was going to
cost $15 million more per year to run than the existing Launch
Processing System. So, NASA managers decided, there was no point in
throwing more money into the project..."
Note: email@example.com is invalid for email
If I had NASA's budget and schedule I, too, could produce a controller
cheaper than a PLC. If I could test everything as long as they do then I
wouldn't have to buy a pre-designed produt that's already been tested as
much as NASA does. If I had rocket scientits for maintenance techs I could
custom design everything.
But I don't and I won't.
things you folks are blasting me about?
Nope. They will have done testing and documentation to a degree far beyond
what I have time to do on a project. If not, they will soon find themselves
locked into the building management, car wash and model trian market.
Actually, nothing was ever wrong with the World Trade Center. What was
wrong was the sloppiest airport security I have ever seen (and I HAVE
travelled) that allowed twenty guys with four inch knives onto various
airplanes all on the same morning.
Nope. That's not how it works. It means that when I buy a new product the
vendor has the resources and the willingness to fly three guys in from Japan
the next morning to fix it. It means that if I buy a used product it is
well used. With your approach, every project is a Beta test; every service
call cuts into your personal income.
It also means the vendors have a reputation in this town. When Honeywell
couldn't deliver FF technology in PM hardware, as they had promised
Syncrude, FF was practically dead in this market for two years. Honeywell
itself is now finding it difficult to sell to grass roots projects. Final
result -- no more free Stampede tickets. They can't afford it any more.
Actually the other person complained because he thought ISA specs were
expensive and saw no other value in ISA. The fact is that ISA specs are
cheaper than a lot of others and I always get free drinks from the vendors
at every meeting. (Though probably no longer from the above mentioned.)
What a deal!
I've never used a PLC; every loop that I've closed has been inside of
some product into which a PLC wouldn't fit. But if you gave me one
machine that needed control, with a few square inches of panel space
into which I could punch a hole, and told me you needed control that
would be within the capabilities of a $5000 PLC, it would be
irresponsible of me to spend six man-months designing a custom board
with a $0.50 processor.
-- snip --
NASA does not purchase PLCs for two reasons: First, they need
controllers that are lighter, (usually) faster and have different form
factors than PLCs. Second, they need to validate the whole system to a
set of safety standards that require complete transparency of the
software involved. While a PLC may be reliable enough to pass muster
under DO-178 for a level-A criticality task, actually _proving_ it on a
virgin piece of software is usually more expensive than designing your
whole software suite to pass in the first place.
Tim, since James was speaking of Space Shuttle control, the point that you
missed is that PLCs are not radiation-hardened... and that is the single,
To rad-harden a PLC would require re-designing *every* part from scratch
(including the OS and software) so what is the point? NASA might as well
start from scratch, and if they are going to do that, they might as well
make it and test it to _their_ requirements. Also stuff designed to work up
there requires 100% up-time (not five-9's) and must work under some really
nasty environmental conditions too.
Personally, I think Industry could learn a lot from the Rocket Scientists...
Isn't the shuttle control system from the "bug-free" programming school? I
remember reading an article about the contractor. They would not accept
change-orders in the code without starting the project from scratch, IIRC.
The only bug that had been uncovered was in a place-holder for a robot arm
that was never installed.
I think my recollection for a change-order causing a start-from-scratch
condition might be somewhat exaggerated, but change orders are fully-
specced. The change to allow GPS navigation, for example, ran about 6500
lines of codes, and the spec sheet, written before any code changes ever
took place, ran 2,500 pages!
The knowledge that needs to be transferred is more on the order of a
naritive description of the logic, and logic diagrams (ladder logic or
gates) that a technician can easily follow. This should include
descriptions of logic conditions that will force a PID controller to
manual mode, automatic shutdown sequences, etc., including the reasons
for actions. A plant technician needs to be able to understand the
complete operation of the logic to determine what might be keeping a
pump from starting or a loop in manual. Other plant engineers, years
later, will need to be able to understand what part of the controls
should be modified when the process or process equipment is modified.
Nobody in the plant should need to look at the source code of the
operating system. That is for the control system vendor to worry
about. Control engineers and technicians in most plants typically
barely have enough time to stay knowledgable on the process, the
operations, and the control strategy. This is stuff the control
engineers have to deal with. The control system vendor can't deal with
that, but they can and do keep up with the RTOS and the type of
programming done in C.
Polytechforum.com is a website by engineers for engineers. It is not affiliated with any of manufacturers or vendors discussed here.
All logos and trade names are the property of their respective owners.