Very good point! But here’s another way of looking at it: as founder of one of the first managed security services companies, Bruce has a good idea of how hard it is to derive strategic information out of a mass of tactical data.

Log data and event data is very specific to the here and now and the target in question. It’s often entirely site policy centric. I.e.: does a “firewall deny” mean the same thing on my SCADA network as it does on my DMZ? No. How, then, do I separate those out, and apply meaning differently to them? In the environment of worms and robotic attacks, does 100,000 bad logins mean more than 10 bad logins? And, would a jump from 100,000 to 200,000 indicate an important trend, or just that we upgraded our bandwidth?

What winds up happening (another reason I reject risk management) is that values to those questions above get plugged in based on expert opinions. If the customer doesn’t have an expert who understands the significance of those factors to the site in question, you get nonsensical outputs if you try to plug the same factors from another site. At best, you are building an expert system - at worst you are building a site-specific expert system. If that’s the case, then skip the “risk management” label and just call it what it is: an expert opinion.

Bruce and I have both looked at enough logs and security data to be able to tell you how dramatically they change when the bad guys come up with something new. That’s also a serious problem for the notion that security data is going to be useful: at any given moment there can be huge discontinuities. I remember when Code Red came out, my IDS logs went from 10,000 events/day to 250,000 - so I turned off Code Red alerting because I knew it didn’t matter to me. How do you represent that kind of incident and suppression response in a threat model? You can’t. You can, perhaps, model that new forms of attacks come out every year - but there’s no historical data on how effective they are.

Zooming out to a macro level you might be able to say “at any given time, 40% of the windows machines in the world are vulnerable to 5-10 well-known attacks.” or something like that. So what? Any organization will assert that ‘that doesn’t apply to us because we have patch management!’ — and then you are left making guesses about the effectiveness of a particular organization’s patch management. Actually, nobody even tries that because the concept is laughable.

“The real problem is that business managers can’t value their IT assets, for all kinds of reasons, not the least of which is that information is an intangible that doesn’t acquire a consensus value until it participates in a market-based transaction.”

Exactly. Add to that the fact that the value of information fluctuates and the amount of effort an attacker might spend to get at it does, too. That’s where the creative attackers come in, again - and that can be at a tactical or strategic level; someone might suddenly find a day-zero into a database and instantly render a million dollars worth of security useless - or Google may announce they are giving away a free version of a core business process and the value of corporate information assets suddenly goes from millions to somewhere close to zero.

You can do “risk management” if you play roulette (probability says that there are only 2 ways to win: be the house, or be the state) but that’s because the rules don’t change and the probabilities are therefore fixed.

How do you know that we DO? If someone is producing some kind of statistic or model and suggesting that it should be used, effectively to predict the future, it should be shown to have more predictive power than randomness. I’m sure there are some IT security risk models that hold water, but I’d be prepared to bet that they’re the ones where the model is equal to obviousness. I.e.:
Statistician: “Our predictive weather models say it’s got a 90% chance of raining”
Engineer: “I just looked out the window and there’s a huge thundercloud with lightning and rain headed our way. Who needs your predictive model when the window works just fine?”

Eventually, you move into the territory typically held by psychic card readers: “someone famous will die in the next year.” For example, I could predict fairly confidently that there will be a major security breach at some bank or brokerage in the next year. The question is, does someone have models and numbers that would allow more detailed projections? If you think about it a little bit, you realize that “risk management” equates to predicting the future. That’s hard, but predicting the past is easy. Predicting the future is going to be like the past is a good bet and it works for Sylvia Browne but it’s not a basis for business decisions.

“there are plenty of scientific disciplines that have to deal in imprecise prior information, or evidence that’s fraught with uncertainty (what Ranum calls “squishy”, and what I’ve heard real honest to goodness physicists call “noisy”)”

Physics is a good example. Yes, you don’t know where an electron is, but you’ve got a mathematical model that works with a high degree of accuracy in spite of your ignorance. But you’ve constrained the problem to the point where the electron is, at least, in your experimental apparatus. Network security is a problem that involves a lot more variables - including an active, intelligent, creative, hostile power - the “enemy” if you will, does nothing BUT perturb your models. That’s what innovation in attack IS.

“These analysts are applying scientific method(s) and developing reasonable approaches to a very complex problem.”

Isaac Newton used the scientific method to do alchemy. I’m sure it was a perfectly reasonable approach; too bad it didn’t work. I know that’s just an example, but don’t make the mistake of thinking that the scientific method can allow people to predict the future. I’ve been through this game with models for risk a couple times in the past so I’m not entirely blowing smoke: guess what happens when the model fails? They change the model to match observable reality and say that’s “scientific.” They neglect the part where scientific theories show predictive power. That’s why I take quantum electrodynamics seriously and laugh at risk management. The physics works out to be able to predict cause and effect with a great deal of accuracy whereas the risk management happens to predict that the model is correct, given inputs that make the model correct.

“JPMC just visited our ISSA chapter claiming, like, a bajillion events an hour.”

I suspect JPMC’s event load has completely different meaning from, say, ebay’s event load - or mine. Just because there’s a lot of data doesn’t mean that you can easily find an underlying theory that unifies it.

“The boundaries of IT Risk losses are pretty well established by events that happen to public companies.”

Unfortunately, those events are often self-reported. Psychologists (the honest ones) can tell you about the problems of dealing with self-reported data. Pollsters (the honest ones) and statisticians can tell you about the problems of self-selected samples and sampling bias.

Just saying that there’s lots of data doesn’t help unless there’s a unifying theory of what the data means, that has some predictive power. If someone wanted to do science in this area, they would propose the theory first, then gather evidence that supported it. Statistical methods are tools to explore problems we don’t understand, looking for correlations and possible areas of significance that might allow us to build those theories. But, ultimately, since risk management is dealing with human behaviors you’d need a predictive behavioral model for it to be anything much better than astrology.

“I am left wondering if Bruce and Marcus were the right people to write about risk management in a mainstream publication.”

Yes and no. The best reply to that question, unfortunately, is fairly nasty and it’s PZ Myers’ “the courtier’s reply” ( http://scienceblogs.com/pharyngula/2006/12/the_courtiers_reply.php ) Essentially it’s that if your field of endeavor is so obviously fake that even non-expert outsiders can dismiss it - don’t attack the non-expert outsiders for playing outside their field: ask yourself “what’s so obviously wrong here.” If you ask the risk management faithful, of course they aren’t going to see that the emperor has no clothes - they can dismiss me as “not understanding risk management.” I’m fine with that. I don’t hear anyone successfully refuting my charges that, namely:
- Risk management inputs are estimates and the results are therefore questionable
- Risk management attempts to predict the future; that is hard
- Risk management is based on backward-looking statistics, which does not make sense in an environment where you’re up against a creative attacker

Thanks for your stimulating comments and blog posting,
mjr.

Thank you, that is a helpful perspective on the issue. I had a look at the final BS 10012 Section 4.4 and it does say:

“The organization shall implement a process for assessing the level of risk to individuals associated with the processing of their personal information.”

which sounds very like your version, but the draft is no longer available to compare.

>>whether we can consider more than one perspective simultaneously

Yes, it can. However there are drawbacks. What happens when you include a larger threat community and a larger asset group is that you lose fidelity in your results. In other words, the results tend to be less precise because there is a large inherent variability in the nature of the attacks from the threat communities against the assets considered. This is why we recommend using a very (or relatively) narrowly defined scenario for each analysis. (The truth is that this is the same for all risk assessment methodologies; namely, that fidelity is inversely proportionate to the size of the combined threat community and asset groups)

So, yes, FAIR can very effectively be used for PIAs. I’ve done this myself. It can also be used for things like BS 10012 PIMS in Section 4.4 (this is from my public comments draft version):

“The organization shall implement a system for assessing the level of risk to data subjects associated with the processing of their personal information.”

To do this from a FAIR perspective, there are probably a handful of threat agents that may act against PII. However, the asset may be carved into several groups to help gain greater fidelity and a better sense of what the organization’s exposure looks like. These may be name and health insurance numbers at rest on database servers. Or we may split it into those on DB2, those on Oracle, etc. and analyze what the risk looks like for each of the threat agents against those assets.

Does this help?

In the UK, the Information Commissioner’s Office encourages privacy impact assessments (PIAs) to be undertaken within a broader “[organisation] risk assessment” and I was just beginning to think how this might work with FAIR i.e. whether we can consider more than one perspective simultaneously.

http://www.ico.gov.uk/upload/documents/pia_handbook_html/html/10-fullbackground.html

I’m leaning towards assuming the risk assessments have to be undertaken independently, and then the output compared.

FAIR uses the following loss categories: Productivity, Response, Replacement, Competitive Advantage, Fines and Judgments, and Reputation.

The pivotal difference is that FAIR doesn’t incorporate loss scenarios as part and parcel of the asset inasmuch as that is defined for the whole analysis scenario.

Most risk assessment methodologies start with the asset and move to every type of loss scenario you can think of (What about nuclear fallout? What if the RAID controller fails?). FAIR starts with the asset (object group) and a threat community (TCom) which necessarily narrows the analysis to a more manageable (and far more believable) result. Once you know who the actor is, coming up with the losses in those six ways is much easier (relatively speaking) and far more believable. Also, FAIR practitioners aren’t afraid to run a lot of scenarios to get the analysis right.

Does this help? What are your thoughts about this?