Digital Bond's SCADA Security Portal

The Hidden Dangers of DNS

Reid W — Wed, 16 May 2012 15:56:14 +0000

DNS is probably the second most misunderstood protocol (the first being the control protocol du network), and that needs to change. I can’t claim to be anything close to a DNS expert, but am known to do neat tricks with it now and then.

A few years back I was lucky enough to catch Dan Kaminsky present at ToorCon on IP-over-DNS tunneling, which planted a seed with me about general protocol tunneling. He had just tinkered with doing DNS queries over SSH, itself tunneled over DNS (and proceeded to chug a cup filled with Cinnamon Toast Crunch and Mickey’s to celebrate, IIRC).

The technique is actually much older, and dates back to the late 90s/early 2000s, when hacker-types would dial-in to Microsoft PPP servers (normally used for software registration), and would tunnel to the Internet using the technique.

DNS can be a bit complicated, but there are some basic principles that should be understood about DNS where security is concerned.

Let’s take digitalbond.com. If you are reading this, you entered www.digitalbond.com into your web browser. Your web browser, in turn, had to figure out what IP address www.digitalbond.com belonged to.

To do that, your web browser first contacted your local DNS server. This DNS server probably had our www.digitalbond.com cached (we’re Digital Bond, after all). In the unlikely event that it did not have us cached, it would first try to figure out what DNS server was responsible for resolving digitalbond.com domains. You can do this yourself via a Unix command line with the command ‘host -t ns digitalbond.com’. On Windows, you would do this with the command ‘nslookup -querytype=ns digitalbond.com’.

This will give you a list of namservers responsible for resolving hosts within the digitalbond.com domain. Your local DNS server will then contact these nameservers in their listed order to figure out the IP address that belongs to ‘www’ in our domain to get you to www.digitalbond.com.

In our case, our nameservers are ns3151.dns.dyn.com, ns2137.dns.dyn.com, ns1128.dns.dyn.com, and ns4132.dns.dyn.com. Your local DNS server will then ask each of these servers what IP www.digitalbond.com is associated with. If the first query gets an answer, voilla, your local DNS server is done. If none of the four respond, your browser will show a cryptic error about host resolution.

The trick is that you generated some data: a request for ‘www’, and this exact request was sent, eventually, to dyn.com. Some arbitrary response, an IP address, came back.

DNS over IP tunneling works exactly like this, except that the hostname is something completely nonsensical-looking. So is the response. If digitalbond.com ran an IP-over-DNS tunnel, you’d see requests for hosts like qezxcvl239817lzkjcvzlq23.digitalbond.com, with a response of “Oh yeah, that’s at and by the way here’s some padding SLksdflzxjzlsalfwelakjczvnvvawe”. The domain being resolved and response provided are both just encoded IP traffic (which could be anything, TCP, UDP, other).

Blocking IP over DNS tunneling is easy on a control network — the control network shouldn’t need DNS at all, except for host resolution of local systems (and if it does need to resolve Internet-served names, sit down and have a tough conversation with your vendor about what, exactly, they mean when they say, “don’t plug our stuff in to the Internet”). Anything contacting the Internet and truly requiring DNS could be moved to a DMZ — it doesn’t belong on a control systems network.

Blocking it on commercial and corporate networks is trickier — I have yet to find a commercial product that can stop the technique while guaranteeing that it won’t interfere with normal users. Detecting it would be fairly easy: a flood of requests for various records on a single domain would be issued, and the flood could be stopped if not at least detected and alarmed upon. GoGo InFlight wireless, for example, appears to block IP-over-DNS tunneling, but only for users that haven’t paid yet. Connections work, sort of, but GoGo throttles the requests fairly quickly causing the tunnel to break down.

If you are able to resolve www.digitalbond.com or ay other internet address from inside your control systems network, chances are good that you are vulnerable to this method of communication. My weapon of choice is iodine, a free IP-over-DNS tunnel that allows for a password-protected server. If you want to try IP-over-DNS tunneling from your own control network but don’t have an internet-accessible server to try it on, drop me a note and I’ll give you access to a test domain. From it, you should be able to load up an internally-hosted web page.

Of course, the crafty have their wheels begin to turn: this technique is universal across bidirectional protocols. For example, I can (and maybe will) make an IP-over-Modbus/TCP tunnel. The tunnel could use Modbus read coil commands to send and receive arbitrary data to a malicious slave, say on the far side of a Layer 7 Modbus firewall…

Image by amagill

Another DHS Bungle or Risky Stratagem?

Dale G Peterson — Tue, 15 May 2012 13:06:53 +0000

DHS Control System Security Program (CSSP) actions in the natural gas pipeline alert get even stranger. They have either bungled helping natural gas pipeline companies to protect themselves or have some risky stratagem to take down an attacker and are willing to accept the collateral damage.

1) First, what it isn’t. There still is no evidence disclosed by DHS that the goal of spear-phishing attacks is any way ICS related except the natural gas pipeline companies have ICS. There has not been evidence that they are trying to collect ICS information or attack the ICS.

2) As reported by Mark Clayton of CS Monitor, Bob Huber and Jonathan Pollet confirmed that there were similarities between the “indicators of compromise” of the natural gas pipeline compromise and those of the attack on RSA and its token database. In simple terms, it is likely that the same person or group that attacked RSA and US defense contractors is attacking natural gas pipeline companies.

3) DHS has not told the public or the affected natural gas pipeline companies that the source of the attack is likely the same as the RSA attack. It appears that DHS did not know about the likely connection to the RSA attack until told by outside security professionals. At this time it’s impossible to say what they knew and when, but it is clear they chose not to share this information with the people being attacked.

4) If the natural gas pipeline companies had been told it was the same attacker and similar attack, they could have implemented more effective defenses and responses. The RSA attack has been studied and effective protective and detective measures developed. DHS could have even shared these security controls with all of the energy sector to limit or prevent additional attacks.

Even more troubling is the DHS advice “do not block or take mitigating action”. Was DHS planning to take immediate action and responsibility for purging the RSA attacker from the affected companies? Short of that, the advice puts the companies at risk. They could have said this is a skilled attacker, based on the RSA attack link, and your company is going to need a well planned approach to identify where the attacker is and how to expel him.

Based on past performance, bungle is the more likely answer. It’s not the goal of this blog or my articles to be a constant DHS bashing vehicle, but it would be nice to see them take on a serious issue in a reasonable way from start to finish. Except for issues completely under their control (training, vuln coordination with willing parties, hired assessments), DHS/INL seems incapable of having a rationale and effective approach to a complex issue with uncertainties.

A Request for a Competitive Process

Dale G Peterson — Mon, 14 May 2012 12:47:14 +0000

Guest author Sean McBride is the Director of Analysis and Co-founder of Critical Intelligence, a company that provides Cyber Situational Awareness and Threat Intelligence services for Industrial Control System Owner/Operators, Vendors and Government stakeholders.

One simplified explanation for the differing views of Dale Peterson and Bryan Owen, as seen in the comments here and here is based on simple economic analysis.

Bryan represents the security function of a highly successful software company. His position directly benefited from the subsidies the US government made available to the private sector through the INL. The INL-OSIsoft story, I know from several conversations with Bryan, is quite compelling. I wish the details were publicly available to serve as an example for how to work security into the product lifecycle – even when the process can be painful.

On the other hand, Dale represents a highly-specialized consulting firm whose thought leadership has publicly and credibly pushed for improvement over much of the past decade. His firm must compete under market forces to land every client. As such, government subsidies through the INL are irksome as his potential clientele turns there, obviously attracted by taxpayer help and good PR.

From a business perspective, if I were in OSIsoft shoes, NOT going to INL is a mistake. However, I have a hard time believing that a firm like OSIsoft, with a security leader as sharp as Bryan Owen leading the way, could not have found a similar (and in some cases superior) quality of assistance in the private market space.

Hence, the deeper issue I see in Dale’s repeated and sometimes stinging analyses is a request for leadership at some level (Ms. Menna, Mr. Weatherford, Mr. King, Mr. Lieberman, Ms. Collins, Mr. Schmidt) to address the questions:

If cyber security of critical infrastructure is as critical as we like to make it sound, isn’t it time to start a competitive process that encourages ICS-security innovation?
Doesn’t the near-decade-long “INL owns this space” mentality risk shutting the door on fresh approaches to ICS-security?
Are non-competitive contracts, vendor subsidies, and non-disclosure agreements providing the daylight necessary to gauge real progress?
Isn’t there some OMB guidance about National Labs competing with industry?

Can someone with direction-setting authority consciously consider those issues – before the next budget cycle – please?

Image by Rennett Stowe

ICS-CERT ≠ DHS CSSP; INL = DHS CSSP

Dale G Peterson — Thu, 10 May 2012 14:25:10 +0000

Let’s take a closer look at DHS since this is the week of DHS’s ICSJWG Spring Conference. Like many, I’m guilty of treating ICS-CERT as if they are THE DHS sponsored organization responsible for ICS security in the US Government. ICS-CERT is part of the DHS Control System Security Program (CSSP) and should be treated and evaluated as a CERT.

ICS-CERT does a fine job coordinating activities between researchers and companies. They are balanced and try to reach a compromise that satisfies both parties. It is a huge benefit for a researcher to be able to turn over findings to ICS-CERT and let them deal with the coordination. Just ask McCorkle and Rios who turned over a huge amount of HMI vulns. There have also been many cases where ICS-CERT knocking on a vendor’s door has gotten a response after the researcher was ignored.

While ICS-CERT has done well on coordination when the researcher and company cooperate, their products, alerts and advisories, are based and biased towards whatever the vendor has admitted or released. It’s not surprising that the vendor panel at ICSJWG had high praise for ICS-CERT. It’s also not surprising they support the vendor’s point of view since many are customers paying INL top dollar for ICS security services.

ICS-CERT has failed to use their ICS expertise or wealth of lab equipment in the ICS-CERT Alerts and Advisories. They have been a clipping service, reporting whatever information others have chose to make public and no more. The best example of this is the Beresford vulnerabilities where ICS-CERT had the Siemens equipment, must have known Dillon was right, and still went with the Siemens party line until it was no longer tenable. That ICS-CERT did not suffer a massive black eye when they completely missed the PLC attack portion of Stuxnet is still baffling. It’s so easy to get me ranting on this topic … and the point is that a CERT is just a portion of what DHS is responsible for in ICS security.

While ICS-CERT ≠ DHS CSSP, it is credible to say that ICS resources at Idaho National Labs (INL) is DHS CSSP. Marty Edwards, the DHS Director of CSSP was formerly in a similar role at INL. Marty still lives in Idaho and has his office at INL. ICS-CERT resources are from INL. The DHS CSSP program office has been staffed by INL contractors. The DHS training courses were developed and are delivered by INL. The fly away teams come from INL. It goes on and on.

PNNL and Sandia play bit roles in DHS ICS security compared with INL, and actual DHS employees unconnected to INL are anomalies and tend to only last a year or two.

A fair characterization is DHS has outsourced ICS security to INL.

Many people mistakenly believe that national labs and non-profits operate for the public good and not as businesses. Most try to maximize revenue like other companies and find ways to allocate and spend that money so it is not considered profit. The national labs are no different. In fact the rates at INL for things like training and support are much higher than commercial industry, both small and large commercial organizations, because the labs have played the game for many years and know how to establish and support huge rates without making a “profit”.

Sometime buy a manager at a national lab a beer and ask them how the operating company, Battelle Energy Alliance (BEA) for INL, makes money. You will hear a load of stories about all the tricks and restrictions preventing reasonable use of knowledge and resources available from the lab. They are a business, and the INL 10-year management contract was valued at $5B. This is not bad, but thinking of them as an altruistic organization is deeply flawed. In fact, INL has more, and gets away with more, conflicts of interest than any other organization in the ICS security space.

There is ICS security talent at INL. This is not the issue. They should be a resource available to the US Government, vendors, and owner/operators can consider to help with ICS security. INL shouldn’t be the DHS CSSP.

DHS is now almost 10 years old. Was the expectation of Congress and the various Administrations that DHS would outsource critical infrastructure ICS security? Is this going to continue? Should this continue based on the progress over the last ten years?

SCADACON (ICS Readiness Condition)

Dale G Peterson — Wed, 09 May 2012 13:02:13 +0000

There have been more than a few hysterical articles, also full of hysteria, in the press based on attack information provided by DHS. Wow, a number of large companies have been subject to a spear-phishing attack! ICS specific threat or attack information = 0.

This could be a precursor to a serious attack on pipeline or water ICS, but based on the information DHS has put out it is merely everyday life for a large corporation connected to the Internet with users who email and access web sites. It is odd that DHS plays up these issues and downplays the serious vulnerabilities that continue to go unfixed by vendors and unaddressed by owner/operators in the deployed ICS.

There is also a question as to what is the criteria for a DHS fly away team getting involved in a cyber incident. All it takes is any cyber attack on a company with some link to the critical infrastructure? The question I would have asked at during the ICSJWG case study on the Curran-Gardner water non-incident is “why did DHS get involved in a small outage in a small water utility?”. This is a topic for another article, but it ties together with the CSSP at DHS groping to prove its doing something in this space while still avoiding the contentious issues where their leadership is needed.

So after a bit of DHS bashing, and in the spirit of the DEFCON scale, here is our SCADACON Rating Scale:

SCADACON 5

A critical infrastructure company is receiving a wide variety of almost continuous attacks from the Internet. Attackers are banging against the corporate/Internet firewall. Attackers are sending email with malware attachments and links to compromised websites. SCADACON 5 is every day for any company or individual who connects to the Internet.

SCADACON 4

Your company is receiving some sort of targeted attack. This could be a spear-phishing or some other attack customized with company information to lure your employees into taking an action that will give an attacker a foothold. The attacker could have already succeeded and is enumerating the network or even gathering or corrupting any data besides specific information on the control system.

Most companies are living in SCADACON 4, and it is naive to believe you are not if you are a company of any size — critical infrastructure or not. The information in the dire warnings from DHS to date have been SCADACON 4. This is why they are hysteria from an ICS perspective. Based on the information DHS provided, the fact that they are pipeline or water system related is incidental.

SCADACON 3

Now we have reached an ICS specific attack level.

At SCADACON 3 an attacker from the Internet or partner/customer network is trying to gain access to a system on the corporate network that is allowed to communicate to the ICS, such as a database server, SCADA admin PC, or IT staff system responsible for ICS switches.

A good example would be an email, including a file or a link with an important bulletin on the DCS application in use, purporting to come from a known DCS vendor source and sent to a DCS admin.

SCADACON 3 is very similar to SCADACON 4 except the monitoring has detected that capturing ICS information or attacking the ICS a goal of the attack.

SCADACON 2

An attacker on the corporate network is trying to gather information on the ICS or compromise the ICS.

The attacker has achieved the first, not too difficult, step of gaining a presence on the corporate network. The attacker could be trying to access file servers or databases with configuration files, drawings or other helpful information in crafting an attack. The attacker could be trying to find the corporate/ICS firewall and then find a way through it to a server on the ICS DMZ or actual ICS. ICS protocols may be seen, ICS default credentials, known ICS vulns will be exploited or perhaps web server, database or other 3rd party software on the ICS server or workstation will be compromised.

At SCADACON 2 you should be disconnecting the ICS network from the corporate network. Yes, I said and meant air gap. You should also assume you are about to lose control and view and at least have your emergency response plans ready.

Admittedly, the SCADACON rating could jump directly from SCADACON 4 to 2 if the attacker chose to find the easiest, non ICS-specific way to gain a presence on the corporate network. However if we treat every corporate network attack on a critical infrastructure company as an attack on the critical infrastructure we will waste a lot of energy. It’s better to focus on identifying events that would lead to SCADACON 3 or 2. Monitoring and detection of ICS specific attacks is key, and fortunately it is an area that is getting increasing attention in the ICS security community.

SCADACON 1

The SCADA or DCS system has been compromised and an attacker is implementing a real time or future loss of control or loss of view attack. The owner/operator no longer has reliable control of the process. Safety, economic, environmental and other worst case impacts could be looming

The frightening thing is it is very easy to go from SCADACON 3 to SCADACON 1 given the complete lack of user or data authentication in the most of the devices that control and monitor a system.

I’m not suggesting anyone actually use this SCADACON scale, but hopefully it is useful in understanding what we are looking for in monitoring actual ICS attacks and useful data.

In my view, DHS / ICS-CERT should not even be issuing warnings until SCADACON 3, and if they cannot provide some level of detail about the ICS-specific nature of the attack it is crying wolf. There are legitimate concerns about protecting data and keeping promises to the companies that have shared the information, but even the following generic statements are examples that would not give any owner/operator identifying information away and still be helpful:

The attacker has targeted computers on the corporate network with access to the ICS network.
The attacker included control system information relevant to the ICS in the target company as part of a spear-phishing attack.
The attacker has attempted to gather information about the control system
The attacker was probing the network for ICS protocol ports
The attacker was attempting to login with ICS default credentials
The attacker was trying to find a way through the corporate / ICS firewall
The attacker had a rogue HMI / EWS attempting to issue commands to ICS devices
The attacker was using ICS specific Metasploit modules
The attacker was trying to load rogue ladder logic and firmware on a PLC

Wouldn’t that be useful to know?

Image by Manual Cernuda

Friday News & Notes

Dale G Peterson — Fri, 04 May 2012 13:50:59 +0000

ISA99 had a busy, well attended 3-day set of Working Group Meetings this week in Gaithersburg, MD. A lot of work gets done in these sessions, and it’s a testament to ISA99 they continue to get this level of participation and effort through many years of work. We hope to have some updates on the key decisions made in Gaithersburg this week.

Next week is the DHS ICSJWG Spring Meeting in Savannah, GA. It’s not a boycott, but unfortunately we won’t be in attendance at this edition. I decided early on to pass, since I attended the last two. Michael and Reid submitted papers, but were rejected. Later on they were asked to present, but by that time they were committed for work at a plant. We will be following the news and tweets from the event.

Tweet of the Week

Every company I know hates revealing attack data yet @Symantec claims "targeted attacks" are up 81%. http://t.co/uANyOvlG Come on guys.

2 May 2012 15:07 via TweetDeck ReplyRetweetFavorite

@jeffreycarr

Jeffrey Carr

Don’t forget to subscribe to this blog RSS feed and follow @digitalbond.com on twitter.

Worth Reading Articles

A lot of RuggedCom / Justin W. Clarke articles, but nothing new.

Critical Intelligence’s ICS Security Event Calendar Updates

Joe Weiss security presentation at ISA POWID Controls and Instrumentation Conference, May 16 in Austin, Texas

Critical Intelligence provides reports and other information products on Cyber Situational Awareness and Threat Intelligence services for Industrial Control System Owner/Operators, Vendors and Government stakeholders.

Image by pdugmoore2001

The Curious Incident of the Original Switch Manufacturer

Reid W — Wed, 02 May 2012 17:59:48 +0000

Dan Goodin at Ars Technica pointed out something very curious to me yesterday. RuggedCom recently took down their ‘Customers’ page, which includes a list of companies for which RuggedCom is the OEM. Fortunately various search engines keep caches of these things, including the Internet Wayback Machine™.

I have been fascinated with the OEM scene since listening to Sean McBride’s excellent presentation at S4 this past year . Like automotive manufacturers, ICS equipment vendors have some very interesting and sometimes very odd relationships with other vendors. Sometimes these are relationships with embedded OS and library (software), sometimes vendors make hardware for other vendors, and sometimes the relationships extend to both, with the OEM purchaser simply slapping a badge on the front…

In particular, RuggedCom had the following list of companies as OEM purchasers from their historical pages:

- ABB
- Areva
- Cooper Power
- General Electric
- Schweitzer Engineering
- Siemens

Dan and I are both curious (and probably Sean, too) if anyone has information on switch lines worth looking at that may be backdoor’d with Justin Clarke‘s RuggedOS vulnerability. I did grab firmwares for the various Siemens Scalance switches. While the logon banners for that switch are similar to the RuggedCom, the firmware is vxWorks running on a different CPU, and appears to be unrelated to RuggedCom’s firmware (at least, via 10 minute analysis).

If you have any info on any products in the vendor list above that run RuggedCom’s firmware, please drop me and/or Dan a line (and ICS-CERT, while you’re at it). He can be reached at dan dot goodin at arstechnica dot com / @dangoodin001. And if you own switches by any of those manufacturers and have the ability to run a quick Python script to gather some information, please get in touch…

Image by denverjeffrey

Demo: Uploading Rogue Ladder Logic to a Modicon PLC

Dale G Peterson — Wed, 02 May 2012 12:52:20 +0000

We want engineers and IT professionals in the critical infrastructure to demo the Project Basecamp Metasploit Modules. It’s a very easy and powerful demo for management and anyone else who is downplaying the fragility and insecurity of PLC’s. Here’s a video to show just how easy it is.

The first 7:35 of the video involves downloading and installing Metasploit. This is a bit tedious and can be skipped by most loyal blog readers, but it’s my fault it’s in the video. I asked Reid to show the whole process from download to exploit. Those of you who have never used Metasploit may benefit a bit from seeing just how easy it is to download and get started with this powerful tool.

After 7:35 the video gets very interesting beginning with the legitimate Unity software interface and uploading ladder logic to the Modicon Quantum mode. Then Reid shows how to find and use the Modicon Metasploit Modules from Project Basecamp. He demonstrates stopping the PLC and uploading rogue ladder logic, all in less than seven minutes.

See all the S4 2012 videos.

S4 Video: Holstein on Insider Threat Statistical Detection

Dale G Peterson — Mon, 30 Apr 2012 16:21:14 +0000

Dennis Holstein of Opus Consulting presented a Consequence Based Assessment Schema at S4 2012. The goal of the schema is to detect an insider attacks, and Dennis goes through the work he has been doing with the National Labs. It is a bit wonkish, like most statistical papers, but the goal of automated monitoring to detect insider attacks is worth the effort.

The presentation also highlights a lot of the peer work in this area.

Dennis also gives some thoughts on whether the ISA99 Security Assurance Levels (SALs) are achievable. He has been very active in this effort so the comments are worth hearing.

Check out our S4 page to see all of the S4 2012 presentations.

RuggedCom Owes its Customers an Explanation

Reid W — Mon, 30 Apr 2012 13:42:33 +0000

RuggedCom was first contacted by Justin Clarke in April 2011 concerning backdoor access to their switches and serial converters. Late on Friday, they announced that they would remove the account from their devices, and that the change would only take a few weeks.

From the press release (notably written by RuggedCom’s VP of Marketing), the backdoor sounds like code cruft left over from the development process. I smell something fishy, though. If the code is really just development cruft, it should be easy to remove. RuggedCom should have removed it when Justin Clarke contacted them a year ago, or should have removed it when ICS-CERT contacted them months ago. They did not, though. Take another look at Justin’s reported timeline:

Apr 2011 – Vendor notified directly
Jul 2011 – Vendor verbally acknowledges knowledge of backdoor,
and ceases communication.
Feb 11 2012 – US-CERT notified
Mar 12 2012 – Vendor responds to US-CERT.
Apr 06 2012 – Due to lack of further contact by vendor, CERT sets
public disclosure for April 13 2012
Apr 10 2012 – Vendor states they need another three weeks to alert
their customers, but not fix the vulnerability.
Apr 11 2012 – Clarification requested regarding need for additional three weeks.
Apr 23 2012 – No response from vendor.
Apr 23 2012 – This disclosure.

In medical parlance, RuggedCom addresses the symptom but not the disease in their press release. The disease, in this case, is a lack of a methodical development process that has any awareness of security. RuggedCom clearly does not include security as a part of its development lifecycle, at least not in their switch and serial converter lines. This ‘developer backdoor’ made it into release. Nobody and no process at RuggedCom stopped it, and RuggedCom has no process to address security concerns in already-released products. They were not going to fix it at all until Justin went full disclosure.

This is bad because RuggedCom’s product is not software, it is hardware and firmware. Upgrading a field-deployed device like this is expensive and can only be done at a time when entire networks of end devices (PLCs, RTUs, relays, etc) can be offline. That is not often. It is a cost that is passed on to RuggedCom’s customers in downtime and risk, and a cost that RuggedCom is no doubt seeing in overtime as its workers rush to recompile and test firmware on all hardware revisions (I certainly hope they take this step, anyway).

What I would like to see in RuggedCom’s response is something that tells me they are aware of the risk that they have placed on their customers, and that they are changing their development process to include security checks. I would like to see them start doing internal code reviews, and maybe even bring in a Justin Clarke as an external security consultant to check out their products. It is never too early in the development process to run things by a security guy or gal. Fixing a backdoor is far more expensive to do later.

In short, I would like to see all vendors of broken ICS equipment start asking why, or at least to give me some confidence that they are doing so in private. I think the root cause in this case is going to lie somewhere around the VP of Marketing making RuggedCom’s announcement — there either isn’t a ‘security’ group at RuggedCom or the head of it has no visibility and no mandate to provide information.

Image by spi516