How much is process control like IT?

Hi, Interested in views on how amenable or otherwise an industrial process control environment is to management under 'IT' principles, ie. formal configuration and change management, project implementation using established IT methodologies. There's a definite sentiment around that it requires a different approach. Is this just due to culture and 'politics', or are there objective reasons why a different approach may be warranted?

Reply to
bruce varley
Loading thread data ...

The truth is always somewhere in the middle!

A lot do apply, some CAN apply and few should. In a control project the project should have IT input with controls project management.

Control systems normally need to be more real time than corp IT people want to allow. They should operate fast and in their own environment with limited / controlled ACCESS and data exchange with the rest of the system.

Formal c> Hi, Interested in views on how amenable or otherwise an industrial process

Reply to
Dennis Mchenney

ALSO:

Many IT people tend to be somewhat paranoid and often tend to make the system EASY for them to maintain, where as controls people HAVE to make it work for the end user as the users must ultimately OWN the system or it will be a big boat anchor.

I f> The truth is always somewhere in the middle!

Reply to
Dennis Mchenney

If you can turn the IT principles into solid Engineering Practice in a way that is appropriate to the industrial environment then they would apply reasonably well. Having been around a few different companies I can say that not all of them appreciated the need for keeping the design data safe and secure or well enough backed up. The IT and the Industrial Controls areas still have plenty to learn from each other and I can see that there is room for further and closer integration of the two paradigms. However, one will need to be selective in terms of the quality of what you elect to use to make sure that you retain the safety and security aspects of the overall design and process control systems.

Reply to
Paul E. Bennett

I'm not sure what you mean by "IT principals", particularly what you consider to be established IT methodologies.

However, to the extent that a control project is a great big software project with a number of different nodes all running simultaneously I agree that formal configuration and change management is essential. Unfortunately many organizations -- even ones that depend on the quality of their software -- don't support this sufficiently.

There was a thread on this some time back acknowledging this, with the proviso that there are times (like when a machine is broken and you need a fix NOW to keep the plant running) that you want to make a change but do NOT want it to contaminate your overall code base. Whether you accommodate this by writing "mystery code" or by branching the tree so the software is archived but not used in mainline code would have to be decided.

Reply to
Tim Wescott

I worked on a project for our corporate head quarters. We hung a control server on the network. Buggy at first, with in a week we were down to chasing the odd occurrence. Then some IT shumck decided to "upgrade all of the OS's connected to the network. Monday came, no one could get into the building. Security had to open every door, none of us had keys, just the key card. I got to work early, oh the sounds of not so silence. I got to the server. And the dummy decided to change the password so I could not even log in. He was at home, until the a car went and picked his bony little ass up, in pjs. (I loved it). He forgot the password he used. Now the president was standing in the room. I was chewing on a bagel and coffee. Until I could get access to the machine there was nothing I could do. Finally decision was made to reload the OS. Which I happened to have. I had to follow the procedures up until the decision was made. I loaded up the OS and my special back up CD's and all was normal by 10 am. Bony ass was in the IT managers office and did not come out most of the day. Seems he screwed with about half of the remote servers as well. I was asked how to stop this from happening again. I put our control machine behind another firewall and only the controls guys could get at it from then on. I never did see him again.

IT principals are great for the unknown uncoordinated masses that do not know the difference between copy and move. We had directories on the network that actually said " ( DO NOT DELETE )" in the titles. Controls are fine when they are working. Unless the hardware is changed there is not reason to mess with them. I have always separated controls from the office, cause production should not effect the office and the office should not effect the process.

I have crashed a couple of networks cause of the volume of data that was being requested.

My experience, controls on a separate network out of sight and mind of everyone.

Reply to
SQLit

Those IT principles that you carefully consider and select to use as an appropriate aid to creating a control system that is easier to use for the operators should be encouraged. Those that cannot stand the test of resilience, security and appropriateness should be left alone. Personnaly I would rather not have the latter sort in the office network either.

With a proper scheme of firewalling in place the office side should not affect the controls side but the office side should be allowed to view the data from the controls side (we do it successfully in my current place of work and the data is also viewable across the internet in the offices of association scientists around the world). I would agree that allowing control systems to be affected by anything outside of the firewall is not a good idea generally, however, future projects associated with my work-place have to face the challenge of allowing control of the plant from a very remote location over the network. MILS* style systems (which I believe the IT world is already geting to grips with) would seem to offer the sort of prinicpals that would be worth considering, especially when those controls will also have to be in real-time with moderately tight constraints on latency of operation).

This is obviously an area that the IT guys will need to look at properly. No network should crash due to too much data. Of course, it may depend on your definition of a network crash.

My, current daily, experience is of a controls system linked via the network to the office network. I can view the mimics and collected data of the plant that I am responsible for from my office. If I log on to the controls part of the system in a slightly differnt way I can actually take control of the plant from my office provided that persons in the control room authorise me doing so. It is also possible for me to have limited control from home, under the same control room peronnel authorisation scheme, but this is a facility that I haven't used yet.

Reply to
Paul E. Bennett

I understand the needs of many. But disagree violently with "remote control/out of sight or off sight " in a lot of situations. Shutting some thing off, no problem. Turn something on,,,,, now there is the rub. Is the process clear? Has some well intentioned fool placed a 12" crescent wrench in the works? I have seen that happen.

Monitoring the process ok, make small adjustments, ok, stopping ok, turn on, never. Just my experiences and I always error on the side of human life.

I once was working on a control upgrade in a power house and the manager kept forcing 15 kv breaker closed then open cause he had red lights on the control panel. He finally came down and looked at the situation, 20 guys working away. He said "OOOPS". I had placed this particular breaker in the test position, to prove my point. First time it closed it scared the hell out of us. Most of the guys knew you could remote operate the breaker but had forgotten. You will never guess who's idea the up grade was and who approved the work.............. You got it the manager forcing the breaker. His men went immediately to the union and all hell broke loose about the safety and control scheme. Keep it safe out there please. I hate reading about mistakes.

Reply to
SQLit

This is not always possible. When you do remotely control stuff it is important to put in place the proper safety devices and procedures. I do a lot of work with waste water and remote pump stations have to be controlled remotely. Careful design of the system is always necessary.

The big difference between IT and control systems is evident in the name. IT is all about data. You have to be careful not to let people from IT become the people that make the final decisions about controls because they do not have the experience with it. Even control designers make lots of mistakes and it would be good to have some sort of certification in place. Experience is key. No matter what you do with the design, the operators and maintenance people have to maintain some sort of safety protocol to insure good practice.

Paul Montgomery Progressive Gauging Inc.

Reply to
A. Paul Montgomery

The process is already, essentially, run remotely over network connections from the control room on-site. The plant is inside a very strong shielded containment where no person may enter during operation. Therefore, the personnel safety aspects are already taken care of. The remoter operation is therefore just one of degree. However, I understand your trepidation. Setting up for remote operation does require an enourmous amount of fore-thought.

You mean that there was no means of locking out remote operation while you upgraded the system. I would have ensured that there was one.

Reply to
Paul E. Bennett

...

Maintenance electricians and mechanics use padlocks on switches, brakes, and breakers; one per worker on each device. A good parallel system in software would have prevented the supervise from closing that breaker, but padlocks would have done the same.

formatting link
It should not be exceptionally difficult for a similar system to be implemented in software, with no device able to be energized until all workers have cleared their individual password-protected blocks on it.

....

Jerry

Reply to
Jerry Avins

Always remembering that the system includes the operators and maintenance staff as well as the equipment.

This is why HAZOP studies (and review meetings of such studies) are so improtant as part of the system development process. To develop a complex system that has safety implications requires that the people involved in the development have a wide range of appropriate skills and knowledge. The development process also needs to have enough rigour to ensure that small details that would make the system unsafe are nit missed out and that all potential problems are identified and dealt with properly before the design is completed.

Reply to
Paul E. Bennett

The majority of work on any machine automation I have every done is in the safety features. The safeties and interlocks are also the biggest issue in troubleshooting a broken system.

Reply to
A. Paul Montgomery

Best of all, the containment area for the plant has some very large and heavy shield doors and an airlock system and labyrinth. All doors must be proved closed and locked before the magnetic and high voltage power supplies can be energised. Nothing much, apart from vacuum pumping, can happen until the doors are locked closed.

I am, by the way, well aware of the tag-out devices used by electrical and mechanical maintenance personnel. We also use those at work. I even designed a tag-out lable that would attach through a padlock on which the authority to work on the equipment could also be declared (which also named the persons who authorised the work).

With respect to system software parallels to the tag-out, I mostly design in a specific structure of system architecture that supports isolation of parts of the system such that it will report un-available to the user interface. I like using at least one processor per actuator in the plant (quite often there are two - one for control and one for comms). I also use extra processors in group organiser/controller mode where the individual actuators are enabled by the group controller when the various interlocks that the group controller monitors are in the correct state.

The final layer of the control system is the user interface system, often based on mere office style computer terminals such as PC's or Sun workstations. This layer only provides a view into the system and allows only the specific control enabled to the person logged on at that terminal. The main user interface network is inside a strong firewall protected domain that provides a good degree of isolation from the office network and the internet connections. One-way portage allows operators to view the system status on their office PC (which allows them to get on with other work while waiting for other sections to ready their part of the next experiment). I already indicated that tunnels in the other direction can be allowed at the discretion of the control room staff (shift technicians and the engineer in charge) on a case by case basis. This is sometimes necessary to ensure that the expertise for specific parts of the plant are able to ensure that the plant continues to operate safely. Yes, this sometimes means shutting down one part or starting another item of equipment. Remember that most of my plant is within a very strong and secure containment wall and shield doors.

I guess that, in my work establishment, the IT staff are well aware of the impact of things they do to the system (although the effect is usually limited to just losing the remote user interface facilities - local controls are always still available).

Reply to
Paul E. Bennett

...

...

Paul,

I tossed my comments into the air, as it were. I certainly didn't undertake to advise you about something you know in far greater depth than I do.

Jerry

Reply to
Jerry Avins

I know Jerry, but my response was, I hope, for the elucidation of others.

Reply to
Paul E. Bennett

There are three (relevant) aspects to computer systems:

Confidentiality: information is limited to authorized people

Security: control is limited to authorized people

Reliability: the system is available when needed.

The relative ranking of the importance of each depends on one's "industry":

Spies: C, S, R IS: S, C, R Control: R, S, C

That said, the tools that IS departments use, such as formal configuration and change management, can be extremely useful. However, in my industry, the IS 'group' is corporate wide and multinational, while the controls group is strictly local. Thus the tools that IS find necessary are, to the control group, merely of interest (i.e., looks like a good idea, but we haven't the time, money, personnel, *or* need for it).

Steve Kassarjian

Reply to
steve

Process control is amenable to configuration and change management, and project implementation under existing process engineering principles and methods.

In certain industries--pharmaceuticals and nuclear power are the most extreme--there are established methodologies for the management of all phases of process engineering, including process control, which is just a part of process engineering.

Process engineering and process control are similar to IT in soom respects; different in other respects. It is not just a matter of culture and politics. Just one example that has been mentioned in other posts is the level of concern for reliability.

Reply to
John Shaw

Bruce,

I read through a lot of replies and see that you got a lot of personal experiences, expertise and suggestions. Yet, my reaction to your post was a bit different.

I can't tell exactly what you mean by "established IT methodologies". I was surprised that you called formal configuration and change management as "IT principles" as those kinds of things have been around in engineering and production environments long before we had a thing called "IT" as such.

Much depends on the environment that determines if formal configuration and change management are to be used. The emphasis here is on "formal" - which to me means there are multiple players and approval.

I can't think of a case where configuration management isn't important. To me this means that the current configuration of a system is documented (which at least implies that there may be a history of configurations as well). It also means that the rev. level (and change history perhaps) of a document is recorded near the first page or just inside the front cover if it's on paper.

[ :-) throw in DHCP and drive the anal configuration manager nuts! Better still, require a change order every time DHCP changes any assigned IP addresses. Job for life!]

The *process* through which the configuration changes is another matter - your "change management". When there is a one-off system with a single player and no management or organizational context then changes are done by that one person at will or according to some checklist or method that he/she has established or is required to follow. When there are many systems or copies of a product then there are surely others involved and more impacts on economics, etc.

Then there is the question of configuration changes on paper or database and configuration changes on real systems. The former might be the precursor to the latter or vice versa in the case of fixing systems in the field - the old "red lines on the shop drawings" situation. A change implies a new configuration needs to be documented.

I don't think it matters what label you put on it - IT or otherwise. I don't think it matters what tools or methods you use as long as they are effective and efficient for your situation. But, the label you put on it may matter to you in your organization! Reading between the lines I can imagine questions like: since this is an IT and software-centric situation, are we going to use CVS? or ..... ? I don't know how well or poorly that would line up with using PLCs or Wonderware.... I suppose one could always force the documents that show ladder diagrams and backups of an application program database to be put into a CVS context ...... Somehow I imagine there must be other ways. A reasonable question is - if not this, then *what*? You should be able to answer that question.

Fred

Reply to
Fred Marshall

PolyTech Forum website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.