And IT auditors congregate at the fantastic ISACA conferences and chapter events.  It’s one of the few conferences that have a good balance of IT risk and business risk.  I don’t know of any other conference where you can not only learn about application and network security, but also bone up on how to audit and secure SAP and PeopleSoft systems!

I’m a fan of this conference.  I usually like to make sure I attend the entire week.  After all, it’s been years since I’ve actually touched an SAP instance, and knowing more about SAP makes me feel smarter.

Image courtesy: copydesk.org (this is not a real pic from CACS conference, btw…)

Given the throngs of people at the Infosecurity Europe conference, I expected a similarly huge crowd at the ISACA North American CACS conference, held on April 27-May 1.  Alas, this wasn’t the case.

The last NA-CACS conference I was at was probably three years ago, when it was in Las Vegas at some huge hotel.  I’m guessing there were about 3500 people at that conference, which was one of the largest ISACA events I had been to.  This was around the same time as the huge buildup/panic around SOX-404.

This year, I’m estimated that there were only around 1300 attendees.  It was a fantastic program, with lots of senior practitioners, spanning information security, IT audit, even some chief audit executives speaking, and IT governance.

Given that this is probably one of the best put together curriculums, I think it’s unfortunate that it didn’t attract the numbers of Infosecurity Europe.  Why?  These are only my speculations:

• IT audit training budgets are shrinking, unlike the bushels of money being thrown around in information security
• ISACA is not effectively reaching the radar screens of information security practitioners

If true, this is too bad.  Information security could use a good dose of learnin’ about risk-based application of IT controls.

Based on attendance, you’d be forgiven if you thought it was 1999, during the middle of the dot-com boom.

Image courtesy: gem66/Flickr

Oh, wait.  We are in the middle of the security/compliance boom!  IMHO, this is made more amazing that it landed in the middle of one of most capital-starved periods for IT in nearly a decade.  If you think that all these hard-earned dollars are being spent on truly creating continuous compliance, this is money well spent.  Yay.

If you think that these capital dollars are being thrown at a huge Band-Aid, and that information security breaches will continue to occur, and that equal dollars will be need to be spent passing next year’s audit, then not so much.  Boo.

I observed the following…

• Tons of attendees: 12,500 visitors, according to their website.  And according to the fliers they passed out on Day 2, 4441 visitors on Day 1, representing a 7% increase over 2008.
• Tons of vendors: Holy cow.  Like in Chris Orr points out in his post about RSA, there were a ton of vendors there.  I’m guessing there were, I can hardly believe it, 700 vendors.  And most of them had basically the same messaging: security, compliance, controls.

It’s difficult to imagine talking to over 100 of these vendors, and keeping track of how they’re different.  I guess that would explain all the marketing investment on, umm, scantily clad people marching around with picket signs, lots of huge, flashing signs, big speakers blaring propaganda like in George Orwell’s 1984,  etc.

The Startling Contrast To Service Desk & IT Support Show

I got a wake-up call when I wandered over next door to the other conference being held at Earl’s Court, which was the Service Desk & IT Support Show.  Compared to Infosecurity Europe, it was like a ghost town.

I estimated that there were about 100 vendors, and maybe a total of 600 attendees.  It was a much, much smaller event.

This reinforced my conclusion that these economic times are starving ITIL projects, and that compliance deadlines are driving huge date-driven projects, which our industry is benefiting from.

Other Benefits Of Being In London

I got a chance to meet up with our UK-based colleagues, and hang out with my good buddy, Steve Chambers from VMware (@stevie_chambers).  He is one smart dude… (Who apparently can drive a sub-10m lap at Nurburgring!)

Oh, and I finally met Tom Howarth (@tom_howarth), who I thoroughly enjoyed talking to, about our respective journeys, the state of our vocation, using social media, etc.  Oh, and talked a lot about vWire evangelist Steve Beaver (@sbeaver), another great guy.

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

My talk title is “Controlling Virtualization Security Risks: Tips from the Experts”, but I will be presenting the results of almost 4 years of benchmarking work that I’ve done with Kevin Behr (@kevinbehr), IT Process Institute, SANS, Software Engineering Institute, Institute of Internal Auditors, and others.  The goal is to be able say what controls really impact information security and IT operational effectiveness.

Contact me if you want a copy of the slides.

Also, Adam has done a fantastic job assembling some very interesting talks from some of the best minds in the field.  You can see the track he’s assembled at his Emergent Chaos blog. And you can even download his entire track in .ics format, so you can view it in your calendar.

What a great guy!

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

As I mentioned in a previous blog entry, I’m a big fan of Hal.  I loved the work he’s done at places that had truly mission-critical environments, including at eBay, Cendant and Google.  He and I, along with Kevin Behr, share a common passion of how to deliver kick ass IT, or as I’ve called it over the years, have amazing IT kung fu.

The webinar went great, but I think we were both surprised by the number of questions that we got from the webinar attendees.  We had 22 questions get posted, of which we could only answer a couple.

So, we earlier today, we did a second webinar (post link), just answer some of these questions.  Over the next couple of weeks, I’ll be posting answers to some of them.

Question: How Should I Engage Internal Audit In The Change Management Process?

By the book, audit engagements has four distinct phases: planning, fieldwork, reporting and follow-up.  Life in IT management sucks when your time with auditors is dominated by preparing and undergoing audits as they do their fieldwork (imagine teams of auditors showing up with suitcases).  Or actually, far worse, when they’re walking you through their findings, extracting promises from you to have them fixed within 90 days in front of your boss.

Obviously, the way to reduce time spent in both of these areas is to have an effective change management processes, with both preventive controls (e.g., defined policies, defined authorization levels, defined consequences when people go around the process, etc.) and detective controls (e.g., monitoring and reconciliation controls like Tripwire).  This allows you to assert and substantiate that you have no unauthorized changes.

But, provided that you have these controls in place, there is also a less formal way that you can help increase auditors’ perception of controls assurance.  That’s to proactively reach out to internal audit, and offer them a standing invitation to join any of your change management meetings.

For them to even sit in even one change management meeting allows them to observe and formulate on the effectiveness of the process.  They will hopefully see how the meeting is being effectively run, how changes are evaluated and authorized, reviewed after their scheduled implementation, and how failures, exceptions and unauthorized changes are handled.

In auditor parlance, observation is one of the types of evidence that auditors can use to support their opinions on the effectiveness of controls.  (The other types of evidence include surveys, testing and independent sources.)

If the auditor observers that no one is showing up to the change management meetings, authorizations are rubber stamped without any real evaluation, unauthorized changes and unplanned outages are occurring regularly, then she will likely flag this as a potential high risk area.

However, if the auditor observes that the meetings are competently run, changes are documented and planned, authorization are thoughtful and considered, and unauthorized changes are quickly dealt with, then this is likely to be viewed as a lower risk area.  Consequently, they will likely spend less time in their fieldwork doing change control testing.

Contrast this to some organizations that spend hundreds, sometimes even thousands of hours, working in emergency projects to try to “clean house” before the auditors arrive to do their testing.  This is what leads to sometimes absurd behaviors, such as closing 6000 change control tickets in one day.

Hal noted during the webinar that this level of transparency is good to extend not only to audit, but business stakeholders as well.

So, to summarize.  Reach out proactively to your friendly internal IT auditor that you may have worked with in the past, ask for a meeting to share respective views of risks that the IT change control processes are designed to mitigate, offer to have someone on their team observe one of your change management meetings, sending them the relevant policies first.

This will help built a mutually respectful working relationship, help build an ongoing dialogue about risks, as well as provide transparency to them about how the change management process is being run.  If the change controls are actually working, this can dramatically reduce the amount of time the auditor spends in the fieldwork, reporting and followup phases of the audit.

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

Project Killer Kumquat is finally going to deliver the set of features that’s going to allow us to catch up to the competition.  We’ve had over 300 developers have been working on this project for nine months.  It’s been a death march for them.

This is one of those damned date-driven projects where senior management made some promise to Wall Street and customers that we were going to ship this week.

The developers were over two months late delivering their code.  But, instead of what rational people do, the business just said, “That’s okay.  We’ll just cut the time dedicated to the downstream tasks, like QA and Production Deployment.”

QA and Production Deployment.  I’m the QA Manager.  Between us and the deployment team, it’s like being stuck between the truck and the loading dock.  It sucks.

29 hours ago, the developers checked in all their code, and we started the QA testing.  Not only did things not go as planned, we now have a potential catastrophe on our hands.  This was supposed to be a damned 4 hour deployment, and we’re 29 hours in, with no end in sight.

I look blearily at the clock that says it’s 3am, and I regret the decision I made twelve hours ago not to cancel this whole damned release and initiate a rollback.  Now, it’s too late.  We’re in so deep that we’ll be lucky if we have everything running by the time the East Coast customer start trying to access the systems in three hours.

I just knew something really bad was going to happen when the deployment team kept saying, “I just need another hour”, and I had already given them five hours.  At some point, we should just put down the shovel and step away from the hole.

Now it’s pretty clear what happened.  And upon some reflection, and after taking a 15 minute walk outside to clear my head, I’m starting to think that this is what happened to us in our last release, too.  (But nowhere nearly as painful…)

28 hours ago, when we started testing, my team started finding failures left and right.  Which is what we expected, given all the corners that were cut by the developers because of deadlines.  But, for some of these issues, it took us hours to figure out whether it was a problem with the code, or something wrong with the QA environment, like an incorrectly configured OS, library, database, or variance between what we’re using and what Dev used.

And so, being the heroes that we are, once my team started finding the errors, we bent over backwards to fix them.  We changed mount points, we modified configuration settings, changed file permissions, modified database stored procedures, we added user accounts, etc…

The problem is, none of those changes were systematically replicated downstream to production.

In fact, our problem is right now, my team is so tired from 28 hours of firefighting, they can’t remember what they did to get things running.  (Jeez.  I’m looking at one of my guys trying to figure out what he had written on his hand eight hours ago to figure out what he did, but it’s long since faded.)

And so now, we’re repeating the whole firefight again, but this time in production.  And frankly, we’re now screwing up more stuff than we’re actually fixing.

But, actually, that’s not the worst part.  Some stuff is breaking because this happened in our last release, and all *those* changes weren’t systematically replicated into our Dev and QA environments!

Lessson: Preproduction changes must be captured, and systematically replicated on downstream systems (e.g., Production), as well as queued up to be replicated in upstream systems for the next release (e.g., Dev, Integration Test, etc.)

This is one of my favorite uses of Tripwire, which is to control pre-production environments, to ensure that we can quickly move releases into production, faster than ever, without introducing chaos and disruption to the production environment.  I’ll write more about this later.

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

“When is it acceptable to patch the QA environment ahead of the production environment?”

If you believe that the goal of QA is to test that application code will operate properly with the production databases and OSes, then the answer is…  Drum roll, please…

NEVER!

When the QA systems are patched ahead of the production environments, several undesirable things will likely result:

1. The code fails to function as designed in QA, because of differences in the OS, libraries, databases, etc.  When this occurs, QA must figure out whether it is a code failure or an environment failure.  Either way, QA is spending more time on non-productive work, most likely slipping the release schedule or forcing QA to skip tests to keep the schedule.
2. This situation is worse than the first.  Here, the code works in QA, and then the code is then deployed into production, which then fails spectacularly.  Now the problem isn’t that the QA schedule is slipping.  Now the problem is that a potentially mission-critical service is down, and we have a potential Sev 1 outage, requiring the best Ops, QA and Development people to figure out how to restore service.

In the second scenario, it appeared that QA did all the right things, but the deployment still failed.

My long-time collaborator Kevin Behr noted, “It sure does seem like the entire point of QA is inextricably linked to production though?”

Absolutely.

So, we have two winners.  The first winner is Jonathan Katz (Twitter handle @katzmandu), who wrote “[You should] have OS patches in-sync the release cycle; as you promote code you test the code + OS + app together.”  Perfect!

The second winner is @sabletek, who had a long answer that I won’t quote exactly.  But, what I believe he was stating very precisely is that you can patch QA early, if you are weighing the risk of patching production vs. delaying the patch to the next release and actively managing the QA vs. production variance risk.  He notes, “I’m making some large inherent assumptions regarding not only QA but patch management as well.”

Brilliant stated.

One honorable mention, because he pointed out a very good wording error in my question.  Nicholas Weaver (@lynxbat) writes, “Wouldn’t the answer be “always”? I mean, you have to QA the patch…”

Crap, yes, that is also true.  If you’re not testing application functionality, then yes, you should always deploy in QA before production.  But, I think @katzmandu gives the more complete answer: in order to predict outcomes, all the components have to be tested together.

Okay, enough theory.  To see what it feels like when this happens in a real life environment, see the next post!

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

He told me about an argument that came up between him and his QA manager.  This QA manager was already getting harassed by the rest of the business for delaying needed software releases and needing more QA engineers each year to keep up.

To make matters worse, they found out that one of their pre-production environments was accidentally patched ahead of the production environment.  Now they had to figure out what to do about it.

My buddy had made the following argument:

“My job is to protect the shareholders.  A bunch of us, including the QA manager, are always complaining about how the Development and QA environments were always too lax and open, and that we needed to lock them down.  That’s why we need to patch them — it just doesn’t feel right to rollback the patches.  We should keep the QA systems as is.”

After my buddy told me this story, I shared my opinion, but wasn’t able to verbalize why it was true.  But over the weekend, I couldn’t help thinking about this problem.  And then I remembered about a problem that I’m working on in two of the largest Internet companies, and suddenly the correct answer was obvious.

On Twitter, I posted this question, with a Visible Ops book to the best answer.

“When is it acceptable to patch the QA environment ahead of the production environment?”

I reveal the correct answer in the next post!

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

This webinar had an astonishingly high attendance rate, with 80% of people registered attending.  That’s more than 2x higher than average, which may indicate the urgency surrounding the first set of NERC compliance deadlines coming in June 2009, and the lack of specifics coming from the regulatory and enforcement agencies.

This will be the first several blog entries, where I will provide highlights and summaries of the information presented in the webinar.  In the next post, I will present some of the late breaking news from the NERC auditors, presented at the WECC conference a couple of weeks ago.  (WECC stands for Western Electricity Coordinating Council, whose members are the bulk power generation and transmission entities that must comply with NERC compliance requirements.)

In this article, I wanted to expound upon a very interesting question that was asked about working with internal audit:

“How should NERC compliance managers work with internal audit?  How should we be coordinating testing, does any of the testing work really need to be duplicated, etc.?  What about all the other compliance programs, like PCI, SOX-404, etc.?”

This is a terrific question, because the answer is that compliance managers may choose to do all or none of these, depending on how your view of the risk environment jibes with those of internal audit.  Therefore, probably one the best thing that compliance managers can do is to sit down with internal audit, and have a conversation about what they believe are the top compliance risks to the organization.

If internal audit and compliance managers have a common view of the organizational risk, all sorts of virtuous things may result.  Internal audit may:

• Make us aware of other compliance objectives that are relevant to the IT infrastructure in scope for NERC compliance (e.g., financial reporting objectives for SOX-404, protecting cardholder data for PCI DSS)
• Make us aware of compliance testing that has already been done for the IT infrastructure in scope for NERC compliance (i.e., so we can rely on the work of others to use as compliance evidence)
• Or even potentially use the work of internal audit to fulfil the compliance evidence requirements

In an era where we have “compliance programs du jour,” there are tremendous efficiencies to be gained scoping and testing controls for multiple compliance programs at once.  Internal audit is uniquely suited to help.   They have the lay of the land, having formulated what they believe are the major organizational risks into their audit plan, including contractual and regulatory risks.  And they also have to do work for each of them, to gain assurance that management is not asleep at the wheel.

This is a topic of real interesting to me, especially after I helped lead the Institute of Internal Auditors GAIT task force that developed and published the GAIT Principles and Methodology in January 2007, designed to help management appropriately scope the IT portions of SOX-404.

Effectively leading successful compliance programs require expertise on risks and controls.  Most often, the place where this is most often found is the auditors.  So make sure you reach out for their expertise!

Questions or comments?  Feel free to send me a note on Twitter!  I’m @RealGeneKim.

What made this article so interesting was that Hal pointed out something that’s seems to often be a blind spot for information security.  This risk is often hidden in plain sight, poses a genuine clear and present danger to the business and information security objectives, and one that is often overlooked.

This issue is change control. Hal writes very convincingly the following:

The information in the FBI Agent’s affidavit that really made me sit up and take notice was the following:

“On October 29, 2008, SK, [a Fannie Mae] senior Unix engineer, discovered malicious script embedded within a pre-existing, legitimate script… It was only by chance that SK scrolled down to the bottom of the legitimate script to discover the malicious script.”

In other words, Fannie Mae got very, very lucky here.  What I want to know is why Fannie Mae had to trust to luck to detect an attack that (again according to the FBI affidavit), “would have caused millions of dollars of damage and reduced if not shut down operations at [Fannie Mae] for at least a week.”

When a bunch of us were discussing Hal’s provocative article on Twitter, there were some interesting rebuttals and conclusions.  However, some of the conclusions just didn’t sit right with me.  Such as, “Humans as the best IDS (intrusion detection systems)!” and “That system administrator is a hero!”  These statements are all partly true, difficult to disagree with, but can lead to strange conclusions.  Something didnt’ seem right to me.

I picked up the phone and called Hal, who I’ve known since 1999, but we haven’t talked in about a year.  (Thank you, Twitter.)  And as we deconstructed the arguments, slowly, I started to understand what was bothering me, and why some of these critiques are just wrong — or just need to be verbalized more precisely.

It boils down to: preventing and detecting failures (whether operational or information security) can’t be the responsibility of the individual.  Instead, it must be the responsibility of the institution.

Just as trust is not really a control, neither is luck.

It’s difficult to disagree that humans are the best IDS (Hal and I discussed that the Cliff Stohl’s famous “Cuckoo’s Egg” story started when a penny discrepancy in a LLBL timesharing system prompted him to ask, “That’s funny.  Why is it off by $0.01?”  That led to discovery of a genuine espionage operation.).  And it’s difficult to argue that the outcomes of heroism is better than if there were no heroism.

But you can’t rely on trust or luck.

So, what would the thought process be to create or verify that we have an effective control environment, where the responsibility is in the institution, not an individual getting lucky?  I would think it would go something like this:

• We learn that certain IT services are required to be operating in order to conduct some critical business operations (e.g., 4000 servers running some mission-critical application)
• We identify that a key risk is that the IT service not being available (e.g., due to failure, sabotage, human error, etc.), causing business disruption
• We ask “what could go wrong in that IT service to cause that event?”  For instance, it may include:
  • Environment failure causes loss of functionality (e.g., power failure, external network failure)
  • Application or infrastructure change causes incorrect or loss of functionality (e.g., config file changing, new application release)
  • Malicious change or sabotage introduced (e.g., script added)
• We flip these risk statements around to craft our control objectives: let’s focus on change control
  • All changes are implemented following a change management process that identifies unauthorized or untested changes that are deployed into production:
• We then design the preventive and detective controls to achieve the control objectives
  • Preventive: we have a change management process that enforces authorization and testing requirements
  • Detective: All production changes are detected, which management reconciles to authorized changes (here’s where Tripwire fits in)
  • Corrective/Deterrent: Management takes decisive action when unauthorized or undocumented changes occur, ensuring “tone at the top” and accountability

(Remember: After benchmarking over 1000 IT organizations, the ITPI research shows that “detecting changes” and “defined consequences for intentional, unauthorized and undocumented changes” are two very accurate predictors of IT and information security performance!  Article from SEI/CMU here.)

Now, that’s a control environment that would make IT operations, information security and auditors happy!

Follow Hal Pomeranz and me on Twitter!  (@hal_pomeranz and @RealGeneKim).

I’ve come to believe that in order to safeguard the production IT environment, information security requires standardization and documentation.  It requires controls such as checklists, and continual control (and where possible, the reduction) of production variance.

This is a very good things, as it aligns information security very closely with some of the key objectives of release management, as defined by ITIL.  (This means that information security should be viewed as value-adding, instead of shrill, hysterical paranoids, always in the way of getting real work done.)

I’ve found that effective information security should strive to reduce variation in the production environment as much as possible. This may contradict what some information security theorists recommend that voice concerns about monocultures, advocating the supposedly inherent safety that diversity provides.

In the real world, it is difficult to achieve sustainable information security through random diversity. If we use a standardization strategy, we can rely on monitoring and reduction of configuration variance. If we rely on a random diversity strategy, we must rely on luck and obscurity.

A Thought Experiment

To test this conjecture, consider the following thought experiment. Suppose we had to inherit one of two undesirable scenarios. In Scenario A, we would inherit 1,000 servers supporting a given business process, configured identically but insecurely. In Scenario B, we have 1,000 servers supporting that same business process, but each server is configured randomly, but 50 percent are configured securely. Which scenario should we choose?

Some information security practitioners will choose Scenario B. They may give many reasons, including that of monocultures. For example, in biological systems, increased homogeneity in crops results in increased risk of catastrophic crop failures[2]. These information security practitioners may conclude from the biology analogy that the risk of disease is similar to the risk of unpatched and insecure infrastructure, making randomness better than consistency.

Benefits Of Standardization: Information Security And IT Operations

On the other hand, let’s explore Scenario A, which every high-performing IT organization would choose instead. High-performers emphatically point out that when every configuration is identical, then:

•    Our mean time to generate security fixes is lower because we have only one fix to generate.

•    We have higher confidence in our fix because we have high configuration mastery of our approved configuration.

•    Our mean time to test fixes is lower because we can build one testing environment that faithfully matches the configuration of the production environment.

•    Our change success rate is significantly higher because our changes are tested in an environment where we have high configuration mastery.

•    Our mean time to deploy fixes is likely much lower because we have a uniform production environment, even allowing us to use automated software distribution tools with high confidence.

The systems in Scenario A also have one very interesting attribute from an information security perspective. The fact that all servers are identical shows that the organization can keep systems in a defined state, as opposed to letting them drift apart over time.

Scenario A Is Orders Of Magnitude Better

Just how much more expensive in terms of time, effort, and cost is Scenario B over Scenario A? In this thought experiment, we may conclude that if the effort scales linearly with the number of configurations, then Scenario A would require 500 times less effort than Scenario B. That is an astonishing difference in effort (whether it is planned or unplanned), and shows how much more desirable Scenario A is when you have one well-understood configuration.

To achieve configuration standardization, we must integrate into the release management process. Production and information security checklists may exist, but manual performance of the tasks they specify can introduce configuration errors, either in the implementation or the verification phase. What’s more, releases may be deployed without error into the production environment, but undocumented or unauthorized changes implemented after release may cause configurations to drift from the approved builds. These configuration problems can cause subsequent problems in applying patches and changes.

(Some of this text adapted from the book, Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps, (written by myself, Paul Love and George Spafford).

PS:  Follow me on Twitter: I’m @RealGeneKim!