So you might have noticed the lack of postings in the past couple of months…  Simply put, I have decided to take an opportunity outside Avid Solutions.  With that being said – “Level3” will continue, but have a new home (along with me).  I would like to thank Avid Solutions and all my peers for their support and assistance with the blog over the past year – matter of fact there are two other blogs created by others at Avid that I would encourage you to read:

• The ArchestrAnaut – a blog dedicated to all issues of the Wonderware Archestra System Platform

• Automate-ZEN – a blog created for the automation professional with aspirations to simplify their career by adding value and humor to their life.

You can find our new home for the Level3 blog at savigent.blogspot.com  – please remember to update your feeds!

- Jimmy Asher

I had a conversation the other day, in which somebody made the comment “well there is the S95 standard, but it really doesn’t do anything for end users…” [I think I just heard screams of pain].

It is true that the ISA-95 Standard is, well a standard… as such, it is geared to bring commonality to the way that we do things.  Furthermore it is broad to allow for flexibility within the space (because not all manufacturers are the same).  As a standard it has been adopted [or used as a marketing point] for Vendors that sell/develop software products, which allow for open systems that can/should be able to communicate to other systems.

However, there are additional ways that end users can benefit or use the standard directly.  First, make use of the Hierarchy Model outlined in Part 1.  Use this model to determine ownership of systems and dividing lines between groups within your organization (such as Engineering and IT).  Second, use the standard to help define user requirements for software design or specifications.  The standard provides a good listing of all activities and definitions that can help identify what might be needed and in scope verses not needed [or needed yet].  In the next posts I will go deeper into each of these.

“This initiative is not a Microsoft product-mandated system.  Rather, it seeks to develop an IT framework that allows for the easy flow of information across organizations.  ChemRA is based on a set of principles that map to the most common use cases of technology for users in the chemical and oil refining industries.” 

The five pillars of ChemRA are:  Natural User Experience, Application Interoperability, Enhanced Collaboration, Business Insight, Solid Infrastructure. 

The five pillars of ChemRA

For more information vist: http://www.microsoft.com/enterprise/industry/process-manufacturing/default.aspx and http://www.microsoft.com/enterprise/industry/process-manufacturing/solutions/chemra.aspx

This year some of topics slated for discussion are: The Cloud, Chance for Profits (the theme of turning problems into solutions to get us all thinking differently), Cut Through the Clutter (the good, the bad and the ugly about implementations) and Real Time (what is real time in my business?).  I providing key highlights in the weeks to come.  Check back tomorrow for a recap of an announcement from Microsoft.

For those not familiar, MESA (Manufacturing Enterprise Solutions Association) International is a global community of manufacturers, producers, industry leaders and solution providers who are focused on improving Operations Management capabilities through the effective application of technology solutions and best practices.

It was speculated, then later confirmed, that the target of the attack was the Natanz nuclear facility in Iran, specifically destroying centrifuges used to enrich uranium.  The attack appeared successful in that hundreds of centrifuges were damaged at the site, causing replacements to be needed, and stalling, for a while, enrichment programs.

Should we fear an attack like this? Not really.  This was a highly sophisticated attack, which probably had some sort of government backing and resources that are out of reach to the large majority of hackers.  If a group is dedicated and has sufficient funding and time, they will be able to break through your security.  Our job is to make that threshold so high that it is unreasonable for them to do so.

The only sure-fire way of locking down your system is to completely disconnect it from the outside and disallow physical access to the controllers/servers.  This, of course, is impossible to do as more and more systems are being tied together using SCADA software so that real-time monitoring can be used on the enterprise level.  Listed below are some simple strategies that are effective, and when used together, forms stronger security by creating a “defense in depth” strategy.  This is not an exhaustive list, but it will get you started heading in the right direction:

Segregate your networks

You should never have your control and business networks on the same network.  The security implications are obvious in that one malicious email attachment could bring down both networks, but there are also performance gains that can be achieved by separating the two.

Utilize DMZs

A DMZ (demilitarized zone) is a section of a network which can be accessed by both your control network and your business network. It provides an intermediate layer of security in that the business network can only access certain servers that reside in the DMZ, such as a data historian, and the control network can push data into this DMZ, but the control and business networks never speak directly.

Anti-Virus

Anti-virus (AV) integration into control system networks can be a tricky thing.  For anti-virus products to be effective, they need regular updates to stay on top of new attacks.  In a locked-down or validated system, patching is almost non-existent and anti-virus products would not get the updates they need.  Another problem with AV solutions is that vendors require certain files and folders to be excluded from scans in order for the products to play nicely together.  This can cause a system to lose responsiveness and AV effectiveness can be lost.

One way to utilize anti-virus products is to have it sitting on a gateway server, so that any files transferring in and out of the system must pass through and be scanned before being allowed into the main servers.  This server could also vet any USB drives or CDs that would be used on the other servers.

Deny Access by Default

Configuring firewalls between networks is something that many companies fail to do adequately.  Many configurations are rushed, leaving them incomplete with gaping security holes.  It’s akin to barricading your front door while leaving your window wide open.  The best policy is to deny all traffic by default, and only allow connections on an exception basis, a concept called ‘whitelisting’.  This may be time consuming, as you need to figure out exactly what traffic or programs are necessary to allow through the firewall, but it provides much better security overall.

Restrict Physical Access

You’d be surprised how many installations have very good IT infrastructure security, but allow anyone to be able to walk up to a cabinet in the field and hook up their laptop directly to the PLC or network switch.  Simple solutions, such as locking control panels, and allowing only certain pre-screened engineering laptops on the control network can increase security and stop the proliferation of harmful worms and viruses.

Disable USB/CD Autoplay

The original vector for Stuxnet was through infected USB drives that integrators took with them around the world and plugged into control systems.  It is good practice to disable Autoplay in Windows, so these infections are not spread through merely inserting a USB device.

To disable Autoplay on Windows XP:

1. Bring up the Run prompt using Win+R
2. Type gpedit.msc and press Ok.
3. Navigate to Local Computer Policy > Computer Configuration > Administrative Templates > System
4. Under the settings in the right-hand window pane, double-click Turn off Autoplay
5. Select the Enabled radio button and select All Drives from the drop-down menu to disable Autoplay on all drives.
6. Press OK.

I believe lasting legacy of Stuxnet will not be that of a new era of attacks on control systems, but an era of focusing more on the security of these systems. For too long has the industry relied on security through obscurity; it’s time to be more proactive in our security practices.

Below are some links for further reading about industrial control system security:

• www.tofinosecurity.com – ICS security devices and information
• http://www.digitalbond.com/ – Lots of good ICS information
• http://www.us-cert.gov/control_systems/ics-cert/ – US Government’s Industrial Control Systems Cyber Response Team

There is an easy and overlooked method to provide first steps to the question “Why did X happen? Or why did X not happen?”.   Not only is it an easy method, it is already there waiting for you to review…  The Windows Event Viewer!  OK, some of you are thinking, “Oh, that gee – so?”  For those of you who didn’t think of it, you are not alone… the Windows Event Viewer (or Dr. Watson for those reminiscing) provides detailed information about significant events on your computer. It can be helpful (and yet overlooked) when troubleshooting problems and errors with Windows and other programs.  A key point that I will make here is to look for information on the application and dependant technologies that the application uses:  DCOM, MSDTC and such for correlations.

Event Viewer

For a quick refresher, you can access the Event Viewer several ways, but I typically typically click Start, Run, and type eventvwr. There are typically three logs available:

• Application: applications running under Windows are supposed to log their events here.
• Security: when enabled Windows, can log a host of security-related events which are logged here.
• System: the operating system logs its events here.
• If you are really luckily, the MES or SCADA system you use might create its own heading too… (really, really lucky).

For those of you who haven’t bothered to look in the Event Viewer – don’t panic on your first view – there will be informational events logged aside from errors.  THIS IS NORMAL!  Another point is get a feel for what events are logged under NORMAL conditions so that you don’t chase a dead end for a missing printer driver (or somebody RDPing).

Now hop to it and check it out!

Before I became acquainted with the Cisco IOS(Internetwork Operating System), I made the same mistake many people do; if a port stops working and I can get my device working again by just moving the connection to another port, the port must be bad.  In my experience with Cisco switches, this is rarely the case.  However, there is a feature that is enabled by default on many Cisco devices called ErrDisable.  This feature is designed to detect network problems and stop them before the rest of the network is affected.  The default behavior is to disable the port in question until  someone intervenes.  In order to re-enable the port, an administrator would have to issue the shutdown command followed by the no shutdown command.  There is also a feature that allows the user to set a recovery interval for the errdisabled state.  If the recovery interval is set, the switch will, on a periodic basis, check the disabled port to see if the error condition still exists.  If the error condition has cleared, the port will be re-enabled.

The guidance provided by Cisco and Rockwell is to set the recovery interval using the errdisable recover interval seconds command.  In conjunction with the errdisable recovery cause errortype command, the recovery configuration can be very granular based on the type of error encountered.  Playing devil’s advocate, I could argue that configuration of the errdisable recovery feature may cause further problems unless the switch logs are being monitored on a regular basis.  Assuming you have an intermittent hardware problem such as a sloppy cable termination that is causing a link flap (a condition in which the physical link is broken more than 5 times in 10 seconds, easily caused by poor terminations and vibration in an industrial environment).  In this case, if errdisable recovery has been established, the problem may never be discovered until there is a catastrophic failure, resulting in manufacturing downtime as opposed to  scheduled maintenance.  My point is, just because recovery keeps data flowing in the short term, the asumption that no problem exists cannot be made.

Monitoring is essential to technology systems reliability, but that is a whole other topic.  Here is a document that outlines some of Cisco and Rockwell Automation’s guidelines for plantwide ethernet:  http://cisco.biz/en/US/docs/solutions/Verticals/CPwE/CPwE_DIG.pdf
Detailed information about the errdisabled state per Cisco’s documentation library:  http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00806cd87b.shtml

I presented on a project we completed a little while ago.  It’s a 10,000 foot overview of the facility, the MES system, and the ties to the ERP system.  It was a great project to be on from a programmer’s point of view.  There were LOTS of cool problems to come up with even cooler solutions to.

I thought both keynote speakers were very good.  I won’t recap their presentations because they’re actually posted on the web.  If you get a chance, their well worth a listen (see links below).

Jim Porter’s Keynote Speech
John Berra’s Keynote Speech

Dennis Brandl got dressed up in a #88 race car driver suit.  His presentation was on an S88 implementation for a pharma packaging line.

The Honeywell guys had a lot of interesting thoughts on visualization of the future & good UIs.  They tossed out there the possibility of using the Xbox Kinect as an operator interface.  They focused on the face that (A) it’s a pretty darn cool UI device & (B) it’s really cheap for what it can do.  They also tossed out there using an iPad as a tablet HMI.  The catch is that it would use the camera to scan a barcode on the equipment or recognize the equipment itself & automatically load the right screen.

A British company presented on their PAT software.  In short, it analyzes mounds of multi-variable historical data and present it to the operator to make decisions on at run time.  Seemed like a cool piece of software at any rate.

A guy from AB had a presentation on how they implement S88 from a spreadsheet.  Each “task” is a bit in a step.  Each step does one or more “tasks” in parallel.  While the concept is really cool, it still doesn’t escape from the fact that the logic hasn’t been simplified.  What it does buy you is a structure that is much easier to reconfigure for new products.

Dave Chappell had a good presentation how to apply the GAMP V-model to a project to help reduce & mitigate risks during various types of projects (i.e. pure implementation vs R&D and the spectrum in between).

Finally, WBF is coming out with a series of books that looked well put together.  I didn’t get a chance to even skim through them, but they should be pretty good.  You can check it out here.

In this “space” there are two sources of models that can assist:  MESA and ISA. 

MESA-11 Model

MESA Model, Version2.1

Next time I will introduce the S-95 standard written by ISA and adopted by ANSI.

85HJPA3WNFHA