Exploring Possibility Space

Look Papa! I'm on the Loopcast! -- Talking complexity, simulation, black swans, randomness, resilience, and institutional innovation

2020-05-06T00:07:00.000-07:00

If you have a spare 1 hr 40 min. *, you might want to listen my interview on the "Loopcast" podcast (below). The host is Sina Kashefipour (@rejectionking on Twitter).

* Personally, I think I sound better at 1.5X speed, but then again I listen to most podcasts at 1.5X speed.

In this podcast, I reference the following websites and resources:

Santa Fe Institute -- birthplace of Complexity Science. SFI on YouTube, and "Complexity" podcast.
Complexity Explorer -- coursework and resources for the study of complex systems,
NetLogo -- a free and open source platform for agent-based modeling and dynamical systems.
Ignorance and Uncertainty -- book by Michael Smithson, out-of-print but available used
Uncertainty and Risk: Multidisciplinary Perspectives -- editors Gabriele Bammer and Michael Smithson

CORRECTION

In the podcast, at 1: 09:00, I used the phrase "...30 sigma event..." and then went on to define the Greek character "sigma" σ

as the symbol used for "variance" in Statistics. That is incorrect.

Instead, "sigma" σ is "standard deviation", and "variance" is the square of standard deviation, or σ²

\sigma ^{2}

Also I made some errors on "one sigma", "two sigma", "three sigma"

for a Normal distribution .

Instead, it should be the " 68-95-99.7 rule ":

+/− one standard deviation ("one sigma") encompasses 68% of the probability mass
+/− two standard deviations ("two sigma") encompasses 95% of the probability mass
+/− three standard deviations ("three sigma") encompasses 99.7% of the probability mass

S4x20 Video: Lessons Learned from Norsk Hydro on Loss Estimation and Cyber Insurance

2020-05-04T19:03:00.000-07:00

I gave a talk at S4X20 in January on the Norsk Hydro ransomware attack. The full video has now been posted on YouTube:

Like all great presentations, it includes a Seinfeld reference :-)

Splattered Swan: Collateral Damage, Friendly Fire, and Mis-fired Mega-systems

2020-04-01T16:22:00.001-07:00

Like Curly from the "Three Stooges" said,
"I'm a victim of circumstances!"

There is a type of "...Swan" that is not surprising or extreme in its aggregate effect, but is extremely surprising to a particular entity that was considered to be outside the scope of the main process. A bomb is supposed to kill enemy soldiers, not your own soldiers.

I call this type "Splattered Swans".

Context: Rethinking "Black Swans"

This post is nineth in the series "Think You Understand Black Swans? Think Again". The "Black Swan event" metaphor is a conceptual mess.

Summary: It doesn't make sense to label any set of events as "Black Swans". It's not the events themselves, but instead they are processes that involve generating mechanisms, our evidence about them, and our method of reasoning that make them unexpected and surprising.

Definition

A "Splattered Swan" is a process where:

The generating process involves a very powerful force (i.e. penal, constraining, or damaging force) with less than perfect aim.
The evidence are official rules, specifications, or scope, or experience that is limited what is intended;
The method of reasoning are based on the assumption that the aim will be perfect and error-free, or that errors will be "well behaved".

Main Features

A Splattered Swan arises when a very powerful system is prone to misfiring in very bad ways, causing damage to some entities that are considered "safe" or "out of bounds" by normal reasoning. That these outcomes are extreme or surprising are basically due to failures to understand the total system and the ways it can fail.

Two key features of Splattered Swans are 1) critical error conditions are excluded from reasoning on principle and 2) those errors are potentially severe, even the first time. A lot of systems adapt by trial and error, but that only works of the magnitude of errors (i.e. aim) is relatively small and the magnitude of collateral damage is also relatively small. Consider airplane bombers from World War II aiming to kill enemy troops that are located near allied troops. Even though they had bomb sights, the bombers were notoriously inaccurate. With ordinary large bombs, the risk of "friendly fire" (i.e. killing your own troops) is high. If the bomber is carrying a single atomic bomb, then the risk of "friendly fire" becomes extremely high, because you only get one chance to aim and drop and there is no feedback from previous attempts. Plus the damage process is extreme. In the other direction, if there are several bombers, and the first bombers drop flairs instead of bombs, then the cost of error is small and the opportunity of corrective feedback has the potential to dramatically reduce the risk of "friendly fire".

Another important feature of Splattered Swans are the blind spots created by the "official" or "intended" definition of the system of interest. This can lead analysts and decision-makers to never even consider the possibility of collateral damage or unintended consequences.

One Example

"Offensive cyber" , a.k.a. "hack back" is an example from the domain of cyber security. There are many flavors of offensive cyber, but the most extreme involve damaging the targets, either physically or digitally or both. Such extreme attacks might also be considered acts of war, a.k.a. "cyber war". Putting aside the ethics or advisability of offensive cyber, there is immense potential for collateral damage. First, it might be hard or impossible to attribute a given attack to the "real" threat agents or groups (a.k.a. "Black Hat"). They might operate through affiliates, mask or disguise their tools and infrastructure, and might even intentionally implicate a different agent or group in the "Indicators of Compromise" and other forensic evidence. Even if you can correctly identify the attacking group, it may be hard to attack them in a way that doesn't also do harm to socially-important entities or resources (e.g. cloud computing resources, networks, etc.). Finally, in corner-case situations there is also a non-zero potential for self-harm, where an offensive cyber attack backfires on the "White Hat" attacker.

From a planning and on-going management viewpoint, it is much harder to anticipate and control the side-effects of cyber attacks than it is for physical attacks.

How to Cope with Splattered Swan

It is relatively simple to cope with Splattered Swan systems. Don't take the "official" or "intended" system as a strict definition of what behavior or outcomes are possible. Use Scenario Planning or "What If?" analysis to look outside the "official" or "intended" to identify potential for collateral damage.

Then look for ways to introduce error-correcting feedback or damage mitigations for the collateral damage. Another good mitigation is to reduce the intensity of the damage/punishment process.

Swarm-as-Swan: Surprising Emergent Order or Aggregate Action

2020-04-01T15:31:00.000-07:00

A flock of swans in swan-shaped formation

In complex systems with many interacting elements or agents, most of the time the actions of individual agents "average out" and remain local. In some systems and in some circumstances, surprising aggregate or emergent behavior patterns arise.

Emergence is a common characteristic of such systems. But that alone doesn't qualify them as a "Swan". It requires several other factors that could, under the right circumstances, yield very surprising or cataclysmic outcomes. I call this the "Swarm-as Swan", since swarming behavior (birds, fish, insects) is one well-known type of emergent phenomena, but this category is explicitly not limited to swarm phenomena.

Context: Rethinking "Black Swans"

This post is eighth in the series "Think You Understand Black Swans? Think Again". The "Black Swan event" metaphor is a conceptual mess.

Summary: It doesn't make sense to label any set of events as "Black Swans". It's not the events themselves, but instead they are processes that involve generating mechanisms, our evidence about them, and our method of reasoning that make them unexpected and surprising.

Definition

A "Swarm-as-Swan" is a process where:

The generating process involves a large-scale Complex Adaptive System that has regions in the state space where collective and/or emergent phenomena become dominant, leading to collective behavior that is dramatically different from the behavior in the "normal" regions of state space.
The evidence are patterns of system behavior and interaction at various scales (individual, group, collective) and especially surprisingly different patterns, including downward causation and varieties of self-organization and information processing;
The method of reasoning are mental models of the system, whether formal or informal, sophisticated or common sense, and the implications of those models on what behaviors are "normal" and expected vs. what is surprising.

Main Features

The field of Complexity Science has grown and blossomed over the last 30 years. This Wikipedia article gives a good summary, along with the central theme of Complex Adaptive Systems (CAS). From that article, the common characteristics of CAS:

The number of elements is sufficiently large that conventional descriptions (e.g. a system of differential equations) are not only impractical, but cease to assist in understanding the system. Moreover, the elements interact dynamically, and the interactions can be physical or involve the exchange of information
Such interactions are rich, i.e. any element or sub-system in the system is affected by and affects several other elements or sub-systems
The interactions are non-linear: small changes in inputs, physical interactions or stimuli can cause large effects or very significant changes in outputs
Interactions are primarily but not exclusively with immediate neighbors and the nature of the influence is modulated
Any interaction can feed back onto itself directly or after a number of intervening stages. Such feedback can vary in quality. This is known as recurrency
The overall behavior of the system of elements is not predicted by the behavior of the individual elements
Such systems may be open and it may be difficult or impossible to define system boundaries
Complex systems operate under far from equilibrium conditions. There has to be a constant flow of energy to maintain the organization of the system
Complex systems have a history. They evolve and their past is co-responsible for their present behavior
Elements in the system may be ignorant of the behavior of the system as a whole, responding only to the information or physical stimuli available to them locally

While all CAS are, in principle, capable of emergent behavior, not all are capable of big surprises in behavior that we require for our "...Swans" series. Roughly, there are three levels of emergent phenomena

Level 1: Emergent behavior -- behavior of large numbers of individuals becomes interdependent and mutually influencing, far beyond the range of causal and information interaction, including some downward causation where the collective shapes the individuals. Examples: flocks of birds, schools of fish.
Level 2: Emergent functional structures -- the formation of stable networks of individuals that constitute functional subsystems. "Functional" means they do some work beyond just collective behavior of Level 1. An excellent example is the "glider" phenomena in Conway's Game of Life (a type of cellular automata).

A single Gosper's glider gun creating "gliders"

Level 3: Emergence against a model -- similar to Level 2, but the "stable functional subsystems" have information processing and self-sustaining capabilities (including possibly metabolizing energy and repairing/regenerating structures). In a real sense, Level 3 systems "take on a life of their own", at least for an extended period. Example: emergent subsystems that function as regulators (e.g. thermostat), communication systems (e.g. encoding, decoding, transmission), pattern matching, optimization, etc. The human immune system has some of these capabilities.

As you move up these levels, the nature of emergent phenomena changes dramatically, from mildly surprising and interesting at Level 1 up to "HOLY COW" at Level 3. But our understanding of CAS is most at Level 1, some at Level 2, and only a little at Level 3. Put another way, if you create a hundred or thousand different CAS in a computational laboratory, most would only exhibit Level 1 emergent phenomena, a few would exhibit Level 2 emergent behavior, and only a very few, under narrow circumstances, would have Level 3 capabilities. (In his book A New Kind of Science, Stephen Wolfram did exactly this investigation for all simple cellular automata.)

Like all of the "...Swans", it's just as important to understand the evidence that we use to understand these systems (i.e. CAS), and also our methods of reasoning. It's the combination of all three that give rise to the surprising/shocking/extreme behavior that we associate with "...Swans".

The most common evidence people pay attention two is either individual-based behaviors and interactions and the most common collective behavior patterns and distributions. If the states of the CAS are in "low complexity" regions of the state space (i.e. not in one of the three Levels, above), then people may not even recognize that the CAS is capable of complex emergent phenomena. The reverse is also true. If the CAS is normally in a highly coherent, highly functional state then people may not observe or understand the micro-level behavior that supports that phenomena. The evidence we need most is the location of "phase transitions" in state space, where the CAS shifts dramatically from one regime to another. Unfortunately for us mortal humans, it's almost impossible to know in advance where the important phase transitions are in CAS, especially the Level 2 and 3 CAS.

Our methods of reasoning about CAS fall into three categories: 1) Intuitive (i.e patterns of "normal" behavior with small deviations, naïve causal models, "folk wisdom", etc.); 2) Linear Models (i.e. the standard tools of science up to ~1990); 3) Non-linear Models, including Agent-based Modeling.

Methods 1) and 2) are most common, and work well in "normal" circumstances, but are very prone to catastrophic failures of reasoning when the CAS enters a new, unfamiliar regime of emergent behavior. Method 3) is specifically designed to understand CAS in all their complexity, but they aren't a "magic bullet" that completely eliminate the potentials for surprise or extreme outcomes.

One huge difference between Method 2) "Linear Models" and Method 3) "Non-linear Models" is that the that Method 3) usually does not yield a forecast or prediction of system behavior in the same way that Method 2) does. Instead it can help us understand when and why the CAS will change regimes, which is still very useful information to understand potential surprises.

Examples

In the previous section I mentioned some illustrative examples. But here I'll mention two "biggies".

The modern economy has been characterized and studied as a Complex Adaptive System (CAS), especially to understand innovation and crises, especially societal/economic collapse. Some of the first books published by the Santa Fe Institute in the late `80s were titled "The Economy as a Complex Adaptive System".

Mass uprisings and mass revolutions are other classes of phenomena that benefit from study as CAS. I won't go into detail here, but if you are curious, you might read up on these cases:

How to Cope with Swarm-as-Swan

The first step is to recognize that the system you are dealing with has the characteristics of a Complex Adaptive System. Start with the Wikipedia page, then read some of the general references listed at the bottom. This will give you the basic knowledge plus some exposure to many types of CAS.

The second step is to characterize the types of emergent behavior and structures that are within the "possibility space" of the CAS. But stay away from "magical thinking" and "conspiracy theories".

The third step is to apply modeling tools that are appropriate to the complexity of the CAS. Linear models are fine for what they do, but don't try to use them to identify "phase transitions" from simple to complex behavior, etc.

If you aren't mathematically inclined or are not comfortable programming your own Agent-based Models (ABM), you can at least read books and papers that utilize these models and learn from the pros who built them and analyzed them. Even better, you can play with them yourself using the Model Library that comes with NetLogo (free and open source). Each model is controlled by sliders and buttons, and comes with documentation that guides you how to use it, how to interpret it, and how to explore it.

S4x20 Presentation

2020-01-21T08:21:00.002-08:00

I am presenting today, January 21, at the S4x20 conference, 3:30-4pm on the Main Stage.

Here are my slides and notes.

Here is a paper mentioned in the talk that gives details on breach impact estimation:

"How Bad Is It? – A Branching Activity Model to Estimate the Impact of Information Security Breaches"

Talk Like a Cyber Insurance Risk Analyst

2019-11-25T13:57:00.000-08:00

In a recent class on catastrophe risk modeling, I learned the definition of terms that are common in insurance but not so well understood elsewhere:

Peril
Exposure
Hazard
Ground-up Loss
Risk

Read on for definitions, ending with an analogy that, hopefully, ties them all together.

Peril

A 'peril' is a class of process in Nature or Society that causes damage and losses. It's not just the proximate mechanism (wind, water, fire, cyber breach), but instead the entire causal process. Thus, a hurricane is a different peril from tornado, and both are different from wide-spread "wind events".

In cyber risk, we might label every loss process as a sub-class of a generic "cyber" peril. You'll see that frequently in insurance industry publications, presentations, and legal documents. Things get trickier to label sub-types, because it is generally necessary to include threat actors, attack method, loss processes, etc. in the label.

At RMS, the way we handle this is to define "loss processes" that also include some reference to the other elements. Our loss processes include "Data Exfiltration", "Distributed Denial of Service", "Contagious Malware/Extortion", "Cloud Outage", and "Financial Transaction Theft". This allows us take a divide-and-conquer to the task of risk modeling.

Exposure

'Exposure' is the set of the factors that determine whether a particular "insured entity" (person, building, or organization) will suffer losses in any particular instance of a peril event. For physical perils, 'exposure' also includes most of the factors that determine the magnitude of 'ground-up loss', given a loss event. For physical perils, exposure is mostly determined by geographic location and other physical characteristics (building type, construction, elevation above ground floor in multi-story buildings, etc.)

In cyber risk, exposure is more complicated. At a base level, geography and demographic factors (sector, industry, size) do have some influence over whether a given organization will be attacked in a given type of cyber attack, and what magnitude of losses they might experience. But do to both the connectivity cyber space and the relative homogeneity, it's not impossible for determined threat actors to get from any "Point A" to any other "Point B".

Furthermore, many factors beyond geography, industry, and size determine the magnitude of losses for a given firm to a given cyber attack, including internal network topology, security and operational practices, and even business architecture.

At RMS we handle this complexity of exposure inside of our risk models for each loss process in what we call 'high resolution models'. For example, in the Financial Transaction Theft model for banks (e.g. SWIFT attacks, ATM jackpotting), we developed a high resolution model that includes attack campaign lifecycle, targeting strategy, among many other factors. We then ran used Monte Carlo simulation on high resolution model to generate estimates for exposure at aggregate levels -- geography, industry, and size.

In cyber risk, the concept of 'exposure' would encompass the security concept of 'vulnerability', but not exactly as information security (InfoSec) specialists use the term. In InfoSec, a 'vulnerability' is a specific flaw, weakness, or opportunity for malfunction in some specific software or hardware that can, conceivably, be harnessed by a threat actor to do bad things, directly or indirectly. In other words, InfoSec definition of 'vulnerability' focuses on the technical aspects and functions, and not so much on the operational or business aspects, which is what matters for cyber risk modeling. The InfoSec community has attempted to incorporate the operational and business aspects through scoring systems such as CVSS (Common Vulnerability Scoring System), but experience and research have show it to be not well suited for risk modeling.

Hazard

'Hazard' is is the probability that a given insured entity will suffer any loss due to a defined set of perils, in a defined time period. Most often, hazard is estimated for a broad set -- e.g. probability of loss in Florida for residential housing due to hurricane (wind + storm surge + local flooding) in a specified year's hurricane season. If "Hazard > 0.0" for a given residence, then there is some chance of loss because the given residence is exposed to at least some degree.

'Hazard' functions also contain information about the severity of the peril at each point (wind speed, flood level) but says nothing about the probability that losses will be greater than $0, or even relative loss magnitude. That's where 'exposure' comes in.

'Hazard' can also be estimated for a single event. This is what is being conveyed in the maps of particular hurricanes that show the "cone of uncertainty".

Ground-up Loss

'Ground-up loss' is the total dollar amount of losses for a given insurable entity or set of entities, before all insurance payments. It is usually calculated based on legal accounting rules, which may be different from financial reporting rules and tax accounting rules (and, of course, might be different from common sense accounting rules).

In cyber risk, 'ground-up loss' is generally any cost directly due to a breach event or campaign that someone has to pay out of pocket or might be recoverable in a lawsuit. This would include costs of PR, forensics, extra legal costs, and loss of intellectual property (usually valued at recovery cost, not market value). But it almost always excludes cost of "technical debt" -- IT and security spending that a firm should have been paying but didn't, but then had to spend post-breach to bring IT and security up to standard.

Of course, in 'ground-up loss' there is the distinction between 'first party' (the firm that got attacked) and 'third party' (people or firms who experienced losses, but were not directly attacked or responsible for security). For systemically important firms, there may even be losses attributed to 'public harm' in the context of class action lawsuits or regulatory action, as a catchall for all externalized costs.

Reputation harm, including brand damage, is tricky territory. While many experts assert that this is the largest category of cost in data exfiltration attacks, I personally would be surprised to see 'reputation harm' included in insurance contracts for cyber risk, except maybe if it narrowly defined.

The insurance industry uses 'ground-up loss' as a basis their calculations of insurance coverage, deductibles, limits, attachment points, and all the rest.

Risk

When used as a noun in the insurance industry, the term 'risk' combines all of the above elements:

Risk (noun) -- an insurable entity (property, person, organization, etc) with associated exposure(s) and hazard(s), for a given a set of perils (usually one, but could be more).

In contrast, in InfoSec the term 'risk' as a noun rarely includes all of these factors and instead can be interpreted as "something bad that might happen, associated with some IT assets, vulnerabilities, threats, etc." In a blog post, I call these "little 'r' risks', and I propose an alternative "Big 'R' risk" which is more compatible with the insurance industry definition.

An Analogy -- Gambling Casino

To help understand this view of risks, think of special kind of gambling casino.

Think of each insurance company as the "House" in a gambling game (e.g. Black Jack). "Nature" is the deck of cards. "Players" are insured entities -- e.g. firms. Every "bet" by Players is a premium payment (lets assume that players can only bet in fixed amounts).

Every time a Player losses a bet, the House pockets the money (premium). Every time the Player wins, the House (insurance company) pays the Player (the insured).

The House aims to make money in the long run by taking in more money than they pay out and -- very important -- not run out of money in the case of large "winning streaks" by Players.

The key to this analogy is that the House gets to pick and choose what types of bets (a.k.a. "risks") it will payoff for any given Player or set of Players. This is how our "Insurance Casino" is different from regular casinos, where every player is offered the same choice of bets and rules (roughly speaking; high rollers get more favorable rules and bets!).

To an insurance company, a 'risk' (noun) is a bet they are either willing to cover or not, depending on the needs and priorities of the insurance company itself, as a profit seeking, survival-oriented agent.

RESET: "Data-driven Security Smashup" will launch in Fall 2019

2019-06-14T20:24:00.001-07:00

Big change of plans for the "Data-driven Security Smashup":

We are canceling the live event in Las Vegas, August 3 - 5.

Instead, we aim to launch one or more Virtual Smashup projects in the Fall of 2019, followed by one or more live events early in 2020, perhaps one in the US and one in UK.

Why?

Basically, we ran out of time as we were trying to organize the event: sponsorship, organizer recruiting and on-boarding, Call for Participation, legal structure, venue. No fault to anyone. We started relatively late, and our standards are high. We didn't want to just throw it together and risk having things fall apart during the event.

Benefits

This new schedule gives us time to do it right, starting with the basics. For example, we will secure a "fiscal sponsorship" relationship so we have the legal, financial, and operational infrastructure to take donations, manage risk, and to spend money responsibly.

Another "basic" that needs attention is contact and relationship management for all the people who have expressed interest, asked questions, or need responses. This includes a dedicated website instead of this blog.

The new schedule gives us the lead time to recruit organizers and collaborators in academia, professional associations, industry, independent consultants, and government, both in US and internationally (mostly UK, Europe, Switzerland).

Personally, I'm not disappointed. The core idea is solid. Lots of interest. This change makes some space for some of my other priorities (dissertation!).

Stay tuned!

SIRAcon: "Probabilistic models of breach impact – combining theory and empirical data"

2019-05-15T18:52:00.002-07:00

Here are my slides from my SIRAcon talk.

Here's a slide that got a lot of attention. (Humorous, of course)

Announcing: Data-driven Security Smashup

2019-04-15T03:00:00.000-07:00

Data-driven Security Smashup

A Hackathon + Supercollider of Talent, Ideas, & Resources

Fall 2019

Las Vegas, NV; Saturday - Monday August 3-5, 2019

[updated June 14, 2019, see "RESET..." for more info]

Venue: rented house*, well off the Strip Working on it. Aiming for UNLV
Timing: just before B-Sides LV/Black Hat/Defcon
Organizers: Me, Jon Hawkes, plus 2-6 others to be named (interested? Contact me)
On-site capacity: ~30 30 - 60
Remote/virtual participation? Yes. Details TBD Also several Satellite locations
Call for Participation: coming soon, mid May
Call for Sponsorship: coming soon, mid May
Other locations: if this first Smashup goes well, we'd like to 'step-and-repeat' it soon in the EU, UK, Switzerland, elsewhere in US, and maybe more
Updates and news: follow @dds_smashup on Twitter

Summary

The Data-driven Security Smashup (DDS Smashup) is a combination of hackathon and ‘supercollider’ of talent, ideas, and resources, aiming for breakthrough innovations in data-driven cyber security, especially solutions to problems that span domains of people, process, technology, institutions, and culture.

Theory of the Case

Why a “Smashup” rather than a conventional hackathon or workshop? Because we think it has a good chance of breaking through the conceptual and cognitive barriers to innovation in data-driven security.

We aren’t short of creative ideas. More “whacks on the side of the head” won’t make much difference. Current and past approaches to innovation have been too simplistic. We also need something more than just ideas that bridge two different fields or methods. Example: quantitative risk methods and incentives from insurance adapted to security (incl. metrics). People have been whacking away at this innovation for 10 to 15 years, hoping it would be revolutionary. It hasn’t been. Why? Because a revolutionary breakthrough probably requires simultaneous, coupled innovation in three, four, or more fields at the same time, across two or more levels of the socio-technical ecosystem.

This calls for a social invention process that is supercharged to force inventive activity in areas that seem almost impossible, especially from any single field or discipline. That’s why we need a ‘supercollider’ of talent, ideas, and resources like the DDS Smashup.

Space of Possibilities

For purposes of discussing scope and interactions, the solutions could include one or more: 1) participatory games (real or virtual world, tabletop), 2) software tools, 3) data sets, analytics, or visualization, or 4) frameworks, taxonomy, or ontology. This diagram shows the space of possibilities, along with existing or potential ‘docking projects’ (a.k.a. accelerators). (This is preliminary, and subject to much change.)

(click to enlarge)

It’s Like a Hackathon...

Hackathons are events where diverse people quickly “hack together” solutions in a concentrated setting (time and place), working in teams of people who they normally wouldn’t work with. Each team will design and build one prototype solution to one specific problem. “design and build” may involve conceptualization (visualization, animation, schematics) and/or realization (building a working prototype in software and/or hardware). Teams then present their completed prototypes to a panel of judges who award prizes to the best designs. Most hackathons are convened in a physical space, but some are virtual. Often there are people who document the process, including live-action video, interviews, etc.

Hackathons mostly embrace “techno-optimism” as a philosophy and value system, with a strong bias toward pragmatic action. The focus is on ‘the art of the possible’, spurred by creativity and improvisation. Teams are free to base their solutions on their prefered value system as long as it is explicit and it is compatible with “techno-optimism”.

Teams own the rights to Intellectual Property they create during a hackathon, and thus could use it as a basis for commercial development afterward. However, there is no presumption or favor for commercialization or proprietary control of IP. IP sharing and open source are often the best path to further development of a prototype.

Hackathons are frequently sponsored by non-profit entities for community benefit, and sometimes by for-profit firms for either community benefit and/or proprietary interest.

The DDS Smashup will be a 3-day on-site and virtual event, hosted and facilitated in a specially designed venue. The first DDS Smashup will be in Las Vegas, NV on Saturday - Monday August 3-5, just before B-Sides Las Vegas. On-site capacity is limited to about 30 people.

Everyone who participates in DDS Smashup will be a “doer” and 100% committed for the duration, including days and evenings. We will recruit a diverse set of highly-qualified people, including people who may not have ever engaged with the information security community.

Hackathons are frequently sponsored by non-profit entities for community benefit, and sometimes by for-profit firms for either community benefit and/or proprietary interest.

...but Different from a Hackathon

Participation

Unlike a regular hackathon where anyone can attend, participation in DDS Smashup will be limited and selective following a diversity-oriented recruitment process. We aim to supercharge the talent pool.

Participants will not be equal in authority and power. There will be a core team of Organizers (about 3 - 5 people) who will “run the show” -- doing all the organizing and preparation, making major decisions, orchestrating and facilitating, vetting and selecting participants, etc.

The other 25 on-site participants will be selected based on what they can contribute to the event, including:

Commitment to and zeal for data-driven security innovation
Relevant ideas, approaches, tools, or methods (see Scope diagram)
Relevant talents, skills, and expertise
Diverse and divergent perspectives and experiences, especially “boundary spanning”

We haven’t set goals or limits on the number of remote participants, or whether we would support remote-only teams. For this first event, our priority is on the on-site participants and process. If we can recruit several organizers who will focus exclusively on managing and facilitating remote participation, then we can be more ambitious.

We will recruit participants from a wide range of communities and aim for a diverse set of participants (gender, age, ethnicity, culture, institutional affiliation, experience-level/seniority, …). That said, nobody will be there solely to represent a group or point of view. Everyone will be there to work, probably to the limits of their capability. We hope to have sponsorship to help cover travel and other expenses for students and other limited income people. Diversity isn’t an end in itself, but instead serves the goal of breakthrough innovation.

Ideation

In a normal hackathon, all ideation happens on-site and arises spontaneously from the teams. Pre-work is discouraged. DDS Smashup will be different. We aim to supercharge both the pool of ideas and also energy that can use ideas together.

At the DDS Smashup, teams will mostly work by building on, adapting, to connecting to one or more “docking projects” (a.k.a. accelerators) (see “Space of Possibilities” diagram, above). We won’t stop anyone who passionately wants to work from scratch, but they will be at a disadvantage in terms of getting something done in three days and also recruiting support for follow-on projects.

These “docking projects” have been developed, developed to some degree, tested and sometimes applied to practice. While they could all be improved in the usual sense, the biggest potential for innovation is to extend design and development in completely new directions, perhaps connecting several docking projects together.

Teams

In normal hackathons, teams are self-organized when the event starts, possibly by joining people who pitch ideas. DDS Smashup will be different.

Due to capacity limitations (30 people on-site, and maybe 30 to 100 virtually), we will probably limit the number of teams to 6. We will probably have most of the teams identified and partially formed before the event starts. (This is part of pre-work.) But we won’t lock in team membership until after the event starts.

We will also be flexible about participation on multiple teams, or merger of teams, teams splitting apart, and temporary multi-team collaborations. A huge benefit for having a small venue with relatively small number of participants is we can improvise like crazy this way.

Pre-work

In normal hackathons there is no pre-work. It’s often discouraged. DDS Smashup will be different.

All participants will need to do pre-work. We estimate this could be 5 to 30 hours of work over four to six weeks leading up to the DDS Smashup. The pre-work may involve learning a tool stack (e.g. NetLogo and extensions API for Agent-based Modeling), or learning about one or more of the docking projects, or doing background reading on Data-driven Security (incl. risk, economics, metrics, etc.).

We will be strict about this. Any participant who shows up without doing adequate pre-work will be turned away. It’s just not fair to everyone else.

Work Schedule

In a normal hackathon, both the schedule and venue often promote extreme work schedules, including ‘all-nighters’. DDS Smashup will be different.

DDS Smashup will be 3 days rather than 2. Quite a bit of work will be done prior to the event, and also many team members may be virtual participants, including in other time zones. Therefore all participants should be able to get 8 hours sleep per 24 period. Participants should be prepared to work 10 to 16 hours per day for the three days.
Food and Drink
Most hackathons provide food, and some only pizza and soda. DDS Smashup will be different.

DDS Smashup will have healthy, good quality catered food and drinks, including coffee, tea, and soft drinks, but no alcohol. But, to set expectations, it won’t be fancy gourmet food.
Judging and Prizes
In a normal hackathon, judges evaluate team presentations and then award prizes to the best teams. Participants are often motivated to win prizes and a competitive atmosphere can develop. DDS Smashup will be different.

Closing Dinner

On Monday evening, August 5, we hope to have all teams present at a dinner event (limit 60) that includes CISOs at B-Sides Las Vegas. (B-Sides LV has a CISO track). While this might be viewed as judging, the real purpose of the presentations is feedback and to stimulate further ideas for projects and research. Thus, even a ‘failed’ prototype might be fruitful for future work because of what was learned in the process of the project.

Sponsorship

In some normal hackathons, one or more sponsors have a proprietary interest in the process or products. Sometimes they want to promote their products or services (e.g. “build a public health app using XYZ’s API and tools…”), or they are hoping to commercialize one or more prototypes, or maybe they want to recruit new employees.

By contrast, in the DDS Smashup all sponsorships will be for community benefit and for research goals. There will be no marketing sponsors and no formal job recruiting.

All sponsor money will be handled through the B-Sides LV non-profit organization (or similar).

All sponsor money will go for direct expenses for the venue, equipment, networking services, food, office supplies, etc. No money will go to Organizers or Participants in the form of salaries or consulting fees. Students and other limited income Participants may receive travel and expense stipends.

Fine Print

Sorry if the following sounds harsh, but we aren't messing around.

Unlike other Information Security conferences in Las Vegas that include a lot of non-work activity -- networking, socializing, partying, drinking, pranking, and rampaging -- the DDS Smashup will be all work and only work. This excludes alcohol and drugs, but also friends, partners, pets, diversionary games or media, etc. Obviously, it also excludes sales and marketing activity by vendors or consultants.

Also: no "rock stars". Nobody participates just because they are famous or have a cool reputation. Nobody participates without doing necessary pre-work. Sponsors can't buy participation slots, either.

All Organizers and all participants are collectively responsible to make this a productive, inclusive event. Because how we treat each other matters and we want everyone to be on the same page, we will have a simple code of conduct. We are creating something new and thus we can't assume that existing set of norms or other Codes of Conduct are sufficient or universally understood.

This will be strenuous work. On-site participation is not appropriate for people who are feeling sick or are in frail health. Through remote/virtual participation, we can probably accommodate anyone who can be productive regardless of health or disability.

We might have a few sociologists or anthropologists documenting our process, including video and interviews. We'll get signed permission from everyone, with options to opt-out. Otherwise, nobody will be there to "just watch" or "hangout".

Like other hackathons, DDS Smashup is not an appropriate venue to explore the outer reaches of free speech and free expression, nor for debates on meta-issues like social justice, value systems, or the validity of the problem statements. Likewise, DDS Smashup is not appropriate for social process work such as discourse critiques, deconstruction, or consciousness raising. These are best done in other venues and settings.

Why Is Breakthrough Innovation in Cyber Security So Hard?

2019-04-14T09:48:00.000-07:00

Short answer: Innovation activities tend to focus on just a few pieces at a time, treating it as a simple problem. That doesn't create breakthroughs because the system* is too complicated.

* "system" = technology, information, people, processes, organizations, institutions, economics,...

In Sciences of the Artificial, Herbert Simon argued that most evolved systems (natural and artificial) were "partially decomposable" (if not fully decomposable) into units or subsystems that could be studied and understood in isolation. While cyber security is partially decomposable for many purposes, it is my conjecture that it is much less decomposable than we believe or desire.

What this means is that breakthrough innovations will depend on many, simultaneous inventions, including crossing system levels.

Innovation as Chemistry

One way to understand this is through the Theory of Autocatalytic Networks, borrowed from Biochemistry and imported into Sociology by John Padgett and Woody Powell in their book The Emergence of Organizations and Markets.

In chemistry, one type of molecule A has a catalytic effect on another B when the presence of A increases the production of B. “Autocatalysis” is a complete loop of catalytic relationships A➔B➔C➔A which creates a self-sustaining, self-stabilizing process. Autocatalytic loops can be simple or complicated, and some autocatalytic loops are built from sets of simple autocatalytic loops.

In human society, organizations, and technology, you can think of “molecules” as being functional capabilities, not just behaviors or processes, and think of “catalysis” as coupling, not just input-output connections. Coupling is ontological -- the definition and structure of B depends upon A, so if A changes, then B either has to change or is no longer functional. Think of “autocatalytic loop” as closure or completeness of functional coupling.

With this theoretical lens, we can define invention as the creation of new autocatalytic cycles, and sometimes breaking or modifying existing autocatalytic cycles. We might imagine the invention process as the “ecological soup” in which reactive processes take place, including experiments, accidents (happy or sad), and purposeful research.

The Appeal of Simple Solutions

This diagram shows how people usually think about making innovation in cyber security: "It’s a simple matter of inventing two or three things and connecting them together in creative ways and ... BANG! INNOVATION!"

Sadly, no.

A simple 3 molecule auto-catalytic network. Arrows are catalytic relations.

From the last few years, here are a few tag lines of so-called "breakthrough innovations" that were highly touted but didn't change things fundamentally:

"Provable Security"
"End-to-end encryption"
"AI and/or machine learning for anomaly detection/prevention/whatever"
"Cyber insurance for everyone"
"National Cyber Leap Year" -- my analysis

When the Going Gets Complicated

Here is what a breakthrough innovation will probably look like (but we don’t really know in detail):

A complicated autocatalytic network that doesn't easily decompose into simple networks.

A Practical Example

Let me move out of the theoretical and conceptual, and look at a practical example showing the interrelations between necessary inventions. (Please excuse me for skipping over many details and not providing references.)

It has been recognized for over 10 years that misaligned and missing incentives is a root cause, but ...
... this requires better security metrics, including aggregate metrics, ...
... including methods for quantifying risk and ...
... aggregating risk at a business unit or enterprise level in economic units (e.g. risk capital $) along with ...
... innovation in incentive instruments, because existing incentive instruments don't cut it. This all requires ...
... innovation in how we measure and manage security performance, including capability building and learning, which of course depends on...
... better models of adversarial innovation, because we may be in a Red Queen arms race with adversaries, so simple improvements may leave us falling behind.
All this has to translate into major changes in human behavior at all levels, so we need innovation in framing, nudges, norm-formation, etc.

I'll stop here, but notice that I haven't even brought in the technology and information aspects of cyber security.

To be clear, I am not arguing that everything is connected to everything else and no innovation is possible unless we research everything. That's too extreme and hopeless. I am just saying the breakthrough innovations will look much more like the bottom diagram than the top diagram.

Implications

Achieving this type of sophisticated set of interdependent inventions will probably require processes that “smash together” talent, ideas, and resources that don’t easily work together, in a concentrated way to sustain these complex loops.

A 12 Year Quest -- My Story

2019-03-31T11:58:00.002-07:00

On a quest, through the desert.

(credit: Assassin's Creed – Origins;
Thick Skin Side Quest –
Crocodile, Hyena, Vulture Locations)

Last week I started a new job as Principal Modeler for Cyber Risk at Risk Management Solutions (RMS). This is HUGE, coming after a 12 year quest that was far from easy or certain.

I don't normally post personal stories on this blog (or elsewhere) but today feels like the right time for this particular personal story. I'm writing this as a way of connecting to my community, many of whom have shared the ups and downs of this journey. I don't have any big lessons or advice. Even so, some readers may find this story instructive or inspirational, even indirectly. I hope so.

Caveats: In this post, I don't individually acknowledge and thank all the people who have helped me along the way. There are so many, so I will do that separately, both in one-on-one communications and later blog posts. I'm also going to discipline myself not to write about all the details, all the events, all the feelings along the way. That would be too long. I aim is to have a post that is readable and still specific enough to be meaningful.

Even so, it's a long blog post. If this suits you, the story continues below.

Prelude: A Holding Pattern

Prior to mid-2006, I was working as a solo consultant ("Meritology"), focusing on economic analysis of IT investments, but also jack-of-all-trades on what ever project came to me. I had left KMPG Consulting (later BearingPoint) in 2004, after 10 years. Even though I was a Senior Manager and doing well, I left for three reasons: 1) my young son (Being a good father to him was a top priority and I needed to avoid travel to make my schedule work for shared custody); 2) Big 6 consulting was sucking my soul and not getting me closer to fulfilling my personal mission; and 3) BearingPoint was a sinking ship and I didn't want to be the last rat off before it sank. (Sure enough, BearingPoint collapsed after going public in Dot.Com boom, via Sarbanes-Oxley violations, mass firing of executives. Finally it was sold off in pieces.)

Really, I was in a holding pattern, waiting and hoping for a new career direction to unfold. While it appeared in the post-bubble world that there would be a high demand for solid economic models of IT investments, that did not materialize. But the truth was that my heart and soul weren't really moved by that type of work, therefore I didn't push very hard for it.

What was I looking for? Some class of important business/social problem that centered on "qualitative complexity" -- i.e. complexity in the dynamic structure of the system, including emergent structure. I wanted to find a business/social problem that defeated conventional methods and therefore required breakthrough models and methods. If the problem was hard enough and compelling enough, then it would be the motive force to drive fundamental innovation along the lines of my mission.

Earlier in my career, I worked computer-aided engineering, computer-aided education, and computer-aided sales/marketing (a.k.a. Customer Relationship Management -- CRM) as candidate domains, but none of these worked out as a "motive force" for fundamental innovation.

Quest status: Not yet started.

Birth of a Calling

In mid-2006, a good friend from my social network asked what I did for work, and then said, "I do a lot of consulting for information security firms. They are desperate for ROI models. I can introduce you!" My first reaction: this should be easy compared to what I have been working on (e.g. knowledge management, full of intangibles, etc.). How wrong I was. Instead of "easy" I found a tangled nest of unsolved problems, even unsolved in theory. Some people even said, "Anyone who can solve this problem deserves a Nobel Prize". There were several 'blue ribbon' commissions and reports during 2001-2006 that framed the problems, pointed in the direction of solutions, and called for making R&D in this area a national priority (US).

A nearly-impossible socio-technical problem that is a national priority? SIGN ME UP! (see mission statement)

I then set on the task of gathering all the information I could find -- academic, industry, and government. When I had a critical mass, I printed it all out (~50 pages, reduced, double sided), made copies, and then tried to make connections with potential collaborators and sponsors.

Quest status: We are ON OUR WAY! But to where, exactly?

A Rough Start

How do you think I was received by the first people I talked to? Not very well. One of my first pitches was to CommerceNet in Palo Alto. They presented themselves as a sponsor/ incubator/ investor in breakthrough ideas. They were pioneers in e-commerce. They had just invested in 23-and-Me (personal DNA evaluation). They had expressed interest in security and privacy as a major theme. Good fit, I thought.

Through a connection, I was able to get invited to present at one of their regular weekly seminars. It was attended by four CommerceNet people, including a couple Principals, a couple of technical people affiliated with CommerceNet, and a couple of my friends.

I was only presenting an R&D opportunity, and suggesting an approach to research. I had no answers or solutions. Even so, nearly all of the questions were: "How are going to do this? What is your solution to that? How much have you prototyped?" There was zero excitement after the meeting.

I suspect that one reason for the lack of excitement or interest is that I lacked any credentials in their eyes. I wasn't a PhD. I wasn't a software guru. I wasn't a veteran of any hot startups. I did have a well-regarded person recommend me, but that only got me the seminar presentation.

I persisted, to no avail. I invited myself to a few CommerceNet social events, including the Christmas party, hoping to convince one of the Principals to champion this cause, or at least hold some kind of workshop with other local experts. Fail. I followed up with emails and then phone calls. Eventually, it was clear that they were shutting me out. Like in Hollywood when they say, "The producers decided to go another direction...".

Clearly, it wasn't going to be enough for me to simply frame the problem and invite other people to work on it and sponsor the research.

Quest status: Bleak. No clear goals. Plenty of people said it was impossible in principle ("anti-quants") or impractical in practice ("how are you going to solve that?). Zero prospect for me to earn money via any specific job title. No prospects for consulting either, given that I didn't yet have viable solution.

Starting as a "Nobody", Becoming a "Somebody"

Moving into 2007, I decided to attend several conferences and workshops on my own dollar, hoping to learn as much as possible about information security and the current state of research on risk and economics of security. I was a Nobody. The people I talked to were polite, but mostly they weren't sure why they should talk to me. Only a few kind people entertained long conversations where I could ask my newbie questions. I wasn't always sure who I should talk to because almost nobody was presenting on the topics that would interest me most.

One conference was in Paris, for Risk Managers in Financial Services. There wasn't any mention of information security risk ("cyber" wasn't yet a thing) in any of the keynotes or session presentations. I had assumed that someone in Financial Services would be quantifying risk of information security, given their quant risk capabilities and regulatory imperatives. But not so. I grew exasperated. At a reception, I managed to button-hole the Head of Operational Risk at Citicorp and asked him -- point blank -- "Do you have a working relationship with the Citicorp Chief Information Security Officer (CISO)? Do you exchange data and models that allow you to quantify risk associated with information security?" Long answers made short: No and no. "We should, but we don't."

Out of these conferences and workshops, I made a few contacts that blossomed into collaborative relationships. That led to my first real work product.

Quest status: On a trail, of sorts. Not alone anymore. Still zero career options, let alone opportunities.

First Real Product

In fall of 2007, I thought I had learned enough to formulate a specific, actionable research agenda, in contrast to my earlier "framing" and unspecific plea. That lead to the white paper: "Incentive-based Cyber Trust: A Call To Action". I posted on the Web a few places and also circulated it by email. I would have liked to publish it some place official but I didn't have academic credentials, plus I didn't know who might publish it. (It was too long and detailed to be published in any industry journal.)

What was the initial response to this "call to action"? Silence. Nothing. Not even criticism. I seriously doubt if many people even read it. (For what its worth, I think its a good read even today.)

But, I made some progress connecting to community and finding collaborators. I found and joined the securitymetrics.org email list. Then in February 2008, I presented at their mid-year conference, Metricon 1.5 in San Francisco. And for the first time, I received significant positive feedback on my ideas, including an early version of the Total Cost of Risk approach. Very enthusiastic response from a couple of security experts, a CISO, and a fellow who worked for the US government (FDIC).

Quest status: A stake in the ground. No longer a "Nobody". A community member, with some good encouragement. Still zero career options or opportunities.

Attempting to Make Something Happen

In 2008, US Department of Homeland Security (DHS) put out its first R&D solicitation for research on information security metrics and risk. It was open to both industry and academics (unlike National Science Foundation (NSF)). GREAT! I cooked up a grand plan: orchestrate a multi-organization, multi-sector research team. Attended my second Workshop on the Economics of Information Security (WEIS), again on my own dollar, and recruited a couple of key people I met the previous year.

We put together a really good "white paper" (a.k.a. short preliminary proposal) for the first stage of evaluation, and I was lead author.

Quest status: On the YELLOW BRICK ROAD! (or so it seemed). If we make it past the first round AND second round, I could finally get paid for some research work, starting a year or more from then.

But then...

For Want of a Click

Things turned really bad in the Fall of 2008. Financial crisis. No consulting projects (not even getting my calls returned). Job interviews canceled. Bigger than all of this was a huge family crisis.

But in some ways the worst thing was when I forgot to visit a DHS web page before the deadline and click "Yes" next to the question: "Will you be submitting a full proposal?". I missed it by two days. I contacted DHS immediately and appealed for an exception, and they said, "No". Was there any other options? "No." Can I talk to anyone else? "No."

It was my own dumb mistake. Sure, having such a requirement with a deadline is dumb, but it was right there, in writing, and I didn't pay attention to it.

Heartbreaking, for both me and my collaborators. I hate disappointing people and, boy, did I let them down.

With both personal and economic crises to deal with, it was time to hunker down.

Quest status: In a dark, cold place. No clear path forward. No career prospects.

Survival Mode, Then a New Path Appears

In 2009, it was all about surviving, financially and emotionally. Often, it was all I could do to get through a the day. Wasn't thinking or planning ahead. Going into Fall 2009, at least I had a couple of job interviews -- nothing related to information security metrics or risk.

"Why don't you go to grad school and get a PhD? That's what you really want to do, isn't it?" A good friend asked me these provocative questions while we were hanging out a picnic one weekend. (She had some magical intuition about this, even though her work and mine are miles apart and we only know each other socially.)

I had been considering PhD programs about every 5 years ever since I graduated college. It just wasn't right. Only after Complexity Science appeared in the late 80s did it seem like the academic world was aligning to my interests and intuition. Finally, in the early 2000s the field of Computational Social Science became a reality, and that seemed like a great fit for me.

I thought, "Why not apply, at least as a backup plan? I can wait to make the final decision after I get accepted to at least one school." So I did. I applied to Stanford, UC Berkeley, Carnegie Mellon University, and George Mason University. The latter was the only one with a Computational Social Science department, and also the only one without a direct connection to information security/risk.

I only got accepted at George Mason. I was one of two finalists at Stanford, but didn't get it. (There's a great story there, which I might tell later.) By this time -- Spring 2010 -- I still didn't have any job offers or solid opportunities lined up. So I accepted the offer from George Mason and I became a 52-year old grad student. Off to Fairfax, Virginia!

Quest status: Discovered a new path. To follow it, I had to drop nearly everything I was carrying and start over. I had some minimal income, and wouldn't be earning anything more for several years. But at least there was a path. Feeling more optimistic.

A 52-Year Old Grad Student

I'm going to skip over a lot here. Grad school courses and exam went really well, overall. It was complicated, though. I was seeing my son much less. I had also started a relationship before I left California, so for 2 years it was long distance. In principle, she was supportive of me being on this path in grad school, though in practice there was a lot of difficulty.

The best thing about it was that I felt like I was on the right path -- finally! -- after many years of wandering. I still couldn't see exactly where I was going after grad school (PhD) but I thought there would be more options than before. Plus information security metrics + quant risk was still growing in importance.

In 2011, the Society of Information Risk Analysts was formed by some colleagues from the security metrics community. I was an early member. Finally, there was a dedicated community (email list, and eventually conferences) of people focused on the challenges of quantifying probabilistic risk models for cyber security.

In summer of 2012 I moved back to California, continuing as full-time grad student and research assistant, working remotely. By 2013 my assistantship had run out, so I got a full-time job as Data Scientist at Zions Bancorp. That put the PhD program into part-time, but I "only" had to finish my dissertation.

Quest status: Slowly crawling up a long steep hill.

The "Black Dog"

I am skipping many details and events, but I will share something important. I started experiencing serious episodes of depression.

Burning the candle at both ends. Chronic sleep deprivation. Feeling like I was constantly disappointing people around me. And a bit too much alcohol. It started out feeling like just a funk, but eventually matured into the "Black Dog", in the words of Winston Churchill.

Other people in the InfoSec community have written in some detail about their experience with depression. Later I might write about mine. For now, I just want to say that it doesn't resemble ordinary sadness or "the blues". My mother died recently, and the sadness I felt during and after has no resemblance to full-blown depression. Best way I can describe it is "sinking into a whirlpool", except the whirlpool is a black hole. It distorts everything, even the acts of kindness of loved ones who want to help. Ordinary logic and reasoning doesn't work. Crazy stuff (like suicide) seem inevitable, the same way that some young people believe it is inevitable that they will marry and have kids some day.

Of course, depression just made everything worse, so it was self-reinforcing.

Lucky for me, I didn't need medication. It was enough to get good sleep, exercise, fresh air and sunshine, and time spent on diverting activities that had immediate satisfaction/reward, like organizing books or fixing things.

Sadly, my relationship ended, too, when we finally agreed we were not aligned in life goals and priorities.

Quest status: Keeping my head above water, but just barely.

Will I Ever Get Back on the Quest?

After so many years of surviving on on a grad student salary and savings, I couldn't continue. It would have been ideal to finish by PhD by then, but that didn't happen.

Just in time, I found a job at Zions Bancorp through connections I had made years earlier in the securitymetrics.org and SIRA communities. This came after a few job opportunities fell through at major forward-thinking Bay Area companies -- where I thought I was well-qualified or even over-qualified.

Zions was good for me in that they were accommodating regarding my request for work arrangement that would also support me taking time to do academic work. Even so, progress on my dissertation had slowed to a crawl and I had no good answer to: "When will you finish?"

By Summer of 2018 I was looking for new work, with the hope that I might find something that would be in alignment with my Mission and Quest. Alas, while I found tons of job openings for Data Scientists, even at information security companies, it seemed like they were only looking specialist Data Scientists -- statistics, machine learning, R + Python, big data, etc. I didn't see any job openings that required the sort of breadth and depth that I could offer. Furthermore, I didn't see strong evidence that any companies were committed to advancing the state of the art in "quant risk".

Quest status: So close, yet so far away.

Finally: The Promised Land

In early January 2019, I received an email from the hiring manger at Risk Management Solutions (RMS). I had previously applied for a Data Scientist job at RMS, based in London, but didn't hear anything after the initial phone screen with HR.

I had been following RMS since 2007, as they appeared to be one of the few companies with core competencies and market position to be a leader in "quant risk" for cyber security. But until ~4 years ago, they didn't have any resources or activity devoted to cyber risk. (It turns out that their insurance firm customers had not requested any cyber risk models until recently.)

Hallelujah! A new job position had been created that was much better suited to me. The hiring manager had already read my blog posts, followed me on Twitter, and seen a few presentation videos. (Why is this rare? Why don't hiring managers do on-line research on all their job candidates?)

It's still very early, but for now I'll just say that the company, the team, and the mission is very exciting and very aligned with my Quest.

Quest status: I'm so very thankful to have reached this place. It is a fulfillment of everything that has come before, including the original intuitions and inspiration. It is also a platform for new things, including things I haven't yet imagined.

Thanks to you, dear reader, for reading this long story

Postscript: Who Do I think I am?

Some might read this post as evidence that I have a "Messiah complex" or otherwise that I think I am special or entitled because I feel a "calling", etc. Or maybe that I am just arrogant. Anyone who takes on near-impossible goals, especially as a personal mission, is subject to such criticisms, which boil down to: Who do you think you are? Special? A genius?

Judge for yourself. I won't argue. I will say that for a few decades, I have studied the lives of people who were later called "great" or "genius", so I know about the complex psychology of committing your life to a "grand challenge" missions. I've studied the down sides, including ruined lives and mental disorders like obsessive attachment to delusions (a.k.a. "going crazy"). No doubt, there is something intrinsically "crazy", "arrogant", and unreasonable to think to yourself: "Yes, I think I can help solve this massive, unsolved problem that looks impossible and has defeated many other people." Reasonable, sensible people avoid such problems.

"The reasonable people adapt themselves to the world: the unreasonable person persists in trying to adapt the world to themselves. Therefore all [major] progress depends on the unreasonable person." - George Bernard Shaw (adapted)

Does Modern Portfolio Theory (MPT) apply to cyber security risks?

2018-12-18T07:00:00.000-08:00

Many months ago, my colleague David Severski asked on Twitter how Modern Portfolio Theory (MPT) does or does not apply to quantified cyber security risk:

I replied that I would blog on this "...soon". Ha! Almost four months later. Well, better late than never.

Short answer: No, MPT doesn't apply. Read on for explanations.

NOTE: "Cyber security risk" in this article is quantified risk -- probabilistic costs of loss events or probabilistic total costs of cyber security. Not talking about color-coded risk, categorical risk, or ordinal scores for risk. I don't ever talk about them, if I can help it.

A Short Summary of Modern Portfolio Theory (MPT)

This is a "For Dummies..." explanation, with only enough to serve our purposes. It's a sketch that leaves out many details and elaborations. See "Further Reading" at the end for more.

The Basics

Modern Portfolio Theory (MPT) is a sub-field in Economics of Investment and Finance.

Economists are not like you and me. They look at investments in a very particular way: Every investment is reduced to a cash flow -- a regular or irregular sequence of cash payments to or from an investor.

For example, if you buy a rental property for cash, the cash out is your initial investment, the cash in is the monthly rental income, with possibly cash from the sale after some years. Periodically you have cash out for maintenance costs, taxes, etc.

MPT adopts a general scheme for characterizing all investments so they can be compared uniformly. Two parameters completely characterize each investment: 1) average rate of return per year (mean return); and 2) average variability per year (variance, also called "risk"). Variance includes both upside and downside variations from the mean return.

The square root of variance is "standard deviation". These are the same two parameters that completely characterize the familiar Normal distribution (a.k.a. Gaussian):

Like the rest of Economic Finance, MPT assumes that everyone knows the mean return for all investments, or at least that nobody is systematically better or worse at estimating mean returns. All uncertainty is captured in the variance (or its square root, standard deviation).

No rational investor would ever choose a security with a negative mean return. Investors will only lose money if they have bad luck (i.e. variability on the downside wipes out the mean return) or if they foolishly over-pay for the asset.

A portfolio is an allocation/commitment of a pot of capital (i.e. cash money) to a set of investments, with an eye toward collecting future uncertain cash flows. All possible investments and portfolios can be plotted as points in a 2-dimensional space of mean return vs. variance ("risk"):

Due to market forces (the "pricing of risk"), investors cannot earn higher mean returns without also taking on higher variance ("risk"). All investors have the option of a "risk-free investment" which is the safest government bonds (currently USA).

Investors seek to assemble portfolios of investments that match their appetite for risk and reward. Variance ("risk") has two components: 1) "idiosyncratic" that is unique to each investment; and 2) "systematic" that is common to all investments. By pooling many, many investments, an investor can "diversify away" the idiosyncratic risk, and thereby improving their return/risk ratio.

Investments inside a firm (i.e. capital investments) are treated the same as investments in firms themselves via common stock. In fact, finance economists view firms as nothing more that bundles of cash flows.

Investments are assumed to be independent, but their returns may or may not be correlated. This can get complicated but here's a simple way to think about it. Imagine two rental properties, A and B, in the same metropolitan area. The returns on property A does not depend on whether you buy property B, and vice versa. It doesn't matter what sequence you buy them, or what sequence you sell them. That is independence. But the returns from property A and B will probably be correlated (rise and fall together, by the same percentage), because rents, maintenance costs, taxes, and resale prices are all governed by common processes.

Inside firms, economists assume that all dependencies between investments are, themselves, investable securities. These include options, futures, insurance, and so on.

MPT ignores many real-world factors, like borrowing limits, collateral for borrowing, budget constraints, and bankruptcy ("going bust").

What's the Big Deal About MPT?

The promise of MPT for investors is defining a unique set of optimal portfolios for each level of risk -- at the efficient frontier. Furthermore, it posits the existence of an optimal portfolio (the "tangency portfolio") that is both on the efficient frontier and also has the highest ratio of return to risk ("Sharp ratio") along the best possible Capital Allocation Line (CAL), which is a mix of a risky portfolio and the risk-free asset.

Why Might MPT Be Useful for Cyber Security Risk?

If you think of cyber security as a collection of distinct "risks" (what I call "Little-r risk" or "risk bricks"), maybe it is possible to use MPT to choose an optimal portfolio of cyber security risks. Maybe using MPT will also harmonize cyber security risk inside the general theoretical framework of economic finance. Maybe economists and Chief Financial Officers (CFOs) will be happier.

No. MPT Does Not Apply to Cyber Security Risk.

There are four reasons why cyber security does not fit the Modern Portfolio Theory (MPT) mold.

Operational Risk is not Investment Risk
It doesn't make sense to "invest" in cyber security risks
You can't ignore dependencies
"Big R Risk" is the best way to aggregate

I'll try to explain these as simply as possible.

1. Operational Risk is not Investment Risk

In the view of Enterprise Risk Management, cyber security risk is a subset of the broader category called Operational Risk, which includes supply chain disruptions, outages due to natural disasters, process and quality problems, fraud losses, etc. It is a mistake to try to bundle all these uncertain down-side costs into the variance of individual investments. They are distinct (random, uncertain) generative processes with distinct effects on cash flow.

The Only Reason Cyber Risk Exists is to Enable the Main Profit-making Activities

The profit-making assets (e.g. resources, equipment, people, projects, etc.) are primary. Cyber security is only a supporting player. It exists to serve and enable the primary profit-making assets. Your business has to take on cyber security risk to even have the possibility of realizing investment returns (and risk). In this way, it is more like your capital investment than it is the variable stream of investment returns.

Analogy: cyber security is like the backup singers or band in a group centered on a star -- e.g. the Pips in Gladys Knight and the Pips and Elvis Presley's backup band, TCB.

Even better analogy: cyber security is like the sound system and sound technicians for the band.

Many people notice that investment risk includes both upside and downside variability, while cyber security risk is all downside, all cost. This is because investors can take either side of the investment (in principle). For every borrower there is a lender. For every buyer of a stock there is a seller, including potentially short sellers. And so on.

What's the upside of cyber security risk? It's the profit-making assets that cyber security enables.

(Even though threat agents profit (sometimes) from your cyber security loss events, their profits do not count as "upside" to your company.)

Here's a very crude example:

Let's say you run a team of gamblers playing blackjack in a casino. It's a special casino where the house odds are negative, meaning that, on the whole and on average, gamblers make a profit. There is one catch. At random intervals, a random gambler is selected to play Russian Roulette with a labeled gun with an unknown number of chambers and unknown number of bullets. A player can't earn any money from gambling unless that gambler also risks playing Russian Roulette. If a gambler shoots a blank, they can report to you the label on the gun. If a gambler kills themself, you learn nothing. Even if you "buy" or "rent" your team members and replacements, it doesn't make since to bundle the cost and uncertainties of the Russian Roulette into the ups and downs of gambling winnings. Some other method for estimating and accounting for risk is needed.

Let's look our simplistic example of rental property:

The variability in returns is governed mostly by the "normal" ups and downs in maintenance costs, rents, resale values, etc. that are driven by the local and national economies. But there are other possible downside "shocks" like fire or natural disaster. In the absence of insurance or risk pooling, any one of these shocks could destroy an individual rental property, or even a whole portfolio of rental properties in the same local area. If the shock is big enough, it could wipe out an investor.

How would we compare these two rental properties?

Property A has high return and high cash flow variability ("investment risk"), but low risk of fire.
Property B has low return and low cash flow variability, but higher risk of fire.

Because destructive house fires are rare, it becomes important to know how long you intend to hold each property before selling. If you only hold Property B for a short time, maybe you can ignore the fire risk. But a short holding time also increases the chances you will get lower than normal return or experience higher than normal downside cashflow. You can't earn a return on A or B unless you take on some fire risk, so in that way it resembles your cash investment to buy the property.

The solution is to fully price the "cost of fire risk" in the form of insurance premiums you pay every month to cover the full cost of any fire. (This simple example works if even if you buy all your insurance through an external carrier, pool your risk among other investors, self-insure, or a combination of these risk finance methods.)

This solution is the essence of "Risk-adjusted Return on Capital" (RAROC) that is common in large banking and insurance companies. Unfortunately it is not well known or practiced in other industries.

(Skip this example if you understand the section, above)

Let's look at a very simple (and simplified) cyber security example.

Say you have a 100% on-line business with a single dedicated web server. Customers pay you a monthly fee to access and use the server through the Internet. You do all the development and maintenance on this server. You pay yourself a salary to keep up with the local cost of living. Other recurring costs include office rent, taxes, accounting services, electricity, and Internet services. You have some non-recurring costs for software, equipment, repairs, and security appliances and add-on software.

Maybe you want to get investments from friends or family to expand, so you need to estimate their return on investment and risk. Assuming you have reached "steady-state growth", you can forecast revenue and costs pretty well, though there are ups and downs each month, quarter, and year. Since you are a solo operator, part of the variability relates to your own capabilities, effort, and energy (including health). Your investors can compare your forecast return and risk to other solo operator businesses to get a good idea of a reasonable "price" (i.e. share of the business).

But what about potential losses due to cyber security breaches? Is your business is vulnerable to ransomware that might wipe out all your customer data and all your business data, too? What about the losses your customers might experience if their data is breached on your system? What about fines from regulators or lawsuits from customers? And the list goes on and on. In their frequency and severity, these loss events look completely different from the normal "ups and downs" of the business. It's a mistake to try to roll these into normal variability.

Instead, it's better to pay a monthly premium to cover the total cost of all non-recurring security costs -- a combination of self-insurance, commercial insurance, and contingent professional services. This monthly premium is different from your other monthly costs, because it goes into a reserve account for future use or investment in the mean time. (Investing the premiums is how insurance companies make profits.)

Plug in the cyber-insurance premium in a Risk-adjusted Return on Investment formula and you will be able to give your investors an accurate measure of investment risk and cyber security risk.

2. It doesn't make sense to "invest" in cyber security risks

Normally, we talk about "accepting" or "absorbing" or "mitigating" or "preparing for" operational risk. We don't "invest" in operational risk the same way as we do capital investments (e.g. assets, securities, projects, etc.).

This isn't just a quirk of language. Investment risk is chosen, while cyber security risk is imposed upon you just for the right to be in business. Investment risk is positively related to expected return. The same is not true for cyber security risk.

Someone might say: "But maybe we 'invest' in certain cyber risks when we decide to pursue certain lines of business or markets or business models. Can't we call these 'investments' under MPT?"

I don't think this works. You can't get an MPT-style return-risk model that bundles in cyber security risk into these different scenarios. In each scenario, you have to "pay the ante" of cyber security risk to play the game. Yes, you want to minimize the cost of risk through mitigations, controls, response & recovery, etc., but that's a separate dynamic from the normal "ups and downs" of the business.

3. You can't ignore dependencies

Above, I said that MPT treats each investment as an independent (but possibly correlated) cash flow, independent from all the other investments. That is most emphatically not true for cyber security investments and risks. Cyber security and associated loss events are run through with dependencies and contingencies.

Consider the significance of your company's business model and enterprise architecture (people, process, and technology), coupled with your information security architecture. These are platforms on which you build your business, and all the security decisions and investments you make here will determine the performance and results of everything else. Each individual cyber security risk ("Little r") is dependent on all these platform elements. Likewise, there is also dependence between "Little r risks". Somehow, we need a way to factor in all these dependencies and platform effects. That's where "Big R Risk" comes in.

4. "Big R Risk" is the best way to aggregate

Instead of trying to build a "portfolio" of many "Little r risk" bricks, I recommend a different approach called "Big R Risk". I've written a long blog post explaining it, so I won't go into depth here. Basically, the goal is to minimize Total Cost of Risk to enable the business. Total Cost of Risk (TCoR) is the sum of:

Total budgeted cyber security costs (direct and indirect, frequent incidents)
Self-insurance premiums (including commercial insurance, if any)
Allocation of disaster recovery and business continuity costs

Estimating these costs, and making them operational, takes a bit of doing, but is not out of the realm of possibility or feasibility. Every business should be able to estimate the first element -- Budgeted Costs -- using nothing but accounting resources and methods (e.g. activity-based costing). The trickiest one is 2) Self-insurance, but that's beyond our scope in this article.

NIST Cybersecurity Risk Management Conference

2018-11-08T06:59:00.004-08:00

I'm presenting today in a 45 minute session. It's a quick overview of previous topics, focused on the Ten Dimensions. The emphasis in this short presentation will be on defining what "performance" means and why managing performance in cyber security is not simply a matter of implementing a list of practices. Below are the slides and relevant blog posts.

Here is an Applicability Matrix I created that shows how the existing NIST CSF 1.1 applies to each of the Ten Dimensions. You'll notice that there are only a few blue squares, which indicates that the Ten Dimensions is a different way of carving up the space. This has plusses and minuses, of course. In the blog posts on the Ten Dimensions, I explain and justify. You'll also notice that some of the Ten Dimensions are poorly covered -- 3. Effective Design & Development; 8. Effective Agility and Learning (incl.. metrics); and 9. Optimize Total Cost of Risk.

Applicability Matrix. Rows = 10 Dimensions. Columns = NIST CSF.
Darker colors = more CSF items are applicable.

Slides
NIST CSF to 10 Dimensions spreadsheet with Applicability Matrix
Ten Dimensions of Cyber Security Performance (blog posts)
How to aggregate ground-truth metrics into a performance index (blog post)
Aggregating risk: Risk Management: Out with the Old, In with the New! (blog post)

Presentation: Navigating the Vast Ocean of Browser Fingerprints

2018-04-16T13:45:00.001-07:00

Here a PDF version of my BSides San Francisco presentation. (Today, Monday at 4:50pm)

COMING SOON: GitHub repo with Python and R code, plus sample data. Watch this space.

The Swan of No-Swan: Ambiguous Signals Tied To Cataclysmic Consequences

2018-03-07T13:29:00.004-08:00

What do you see? colored blocks, or a Black Swan, or both?
This is figure-ground reversal, a type of ambiguity.

We are in the middle of the 100th anniversary of the Great War (a.k.a. World War I). None of the great powers wanted a long total war. Yet the unthinkable happened anyway.

Surprisingly, historians are still struggling to understand what caused the war.

One of the biggest causal factors was ambiguous signals that precipitated cascading actions and reactions. When tied to cataclysmic consequences, this represents a distinct class of "Black Swan" systems.

(Here are some great lectures for those interested in a full analysis of causes of the Great War: Margret MacMillan, Michael Neiburg, Sean McMeekin)

Rethinking "Black Swans"

As I have mentioned at the start of this series, the "Black Swan event" metaphor is a conceptual mess. (This post is seventh in the series "Think You Understand Black Swans? Think Again".)

It doesn't make sense to label any set of events as "Black Swans". It's not the events themselves, but instead they are processes that involve generating mechanisms, our evidence about them, and our method of reasoning that make them unexpected and surprising.

Definition

A "Swan of No-Swan" is a process where:

The generating process is some set of large opposing forces that can be triggered by a set of signals or decisions tied to ambiguous signals;
The evidence are signals -- communications, interpreted actions, interpreted inaction, rhetoric/discourse, irreversible commitments -- that have ambiguous interpretations, either intentionally or unintentionally;
The method of reasoning either rational expectations (normative Decision Science) or biased expectations (Behavioral Psychology and Economics). The key feature is lack of attention or awareness that one might be mis-perceiving the signals, combined with a strategic preference for precautionary aggressiveness.

Main Features

First, let us recognize that ambiguity is pervasive in social, business, and political life. Ambiguous signals and communication have many pro-social functions: keeping our options open, saving face, avoiding insult or offense, optimistic interpretation of events, and so on. They are especially prevalent in the lead-up to major commitments -- romance+marriage in personal life and big ticket sales in commercial life.

Most of the time, ambiguity has a smoothing effect. It reduces the probability of extreme/rare events because of the flexibility of action and response associated with ambiguous signals. Therefore, most people would not associate ambiguous signals with any type of "Black Swan" phenomena.

But when tied to "large opposing forces", things change and that's why this deserves to be a separate type of Black Swan. Ambiguous signals become dangerous when they are linked to cataclysmic processes via certain types of reasoning processes. It's not rational vs. biased. Instead, it's committed self-confidence vs. self-aware fallibility. In committed self-confidence, there is lack of attention or awareness that one might be mis-perceiving the signals, combined with a strategic preference for precautionary aggressiveness. "Shoot first, ask questions later".

Examples

Military forces leading to total war are the obvious case, and most common in history. But we are now in a new age -- the Cyber Age! (Yes. I said it. Cyber) Here are some cyber examples.

Offensive cyber capabilities -- By "offensive" I mean everything from "hack back" to punitive or disabling cyber attacks on critical infrastructure. If it becomes common for nation states and various non-nation actors to develop and deploy offensive capabilities, they everyone faces the strategic dilemma as to when and how much to deploy/trigger each capability. This depends critically on the ability of each actor to detect and accurately interpret a wide variety of signals and evidence related to normal and abnormal activity, including breach events, threat actor attribution, signs of escalation, and so on. These are all swimming in ambiguity, including intentional ambiguity (spoofing, camouflage, etc.)
Remote kill switches -- What if Internet of Things (IOT) makers build "remote kill switches" in their devices? After all, we'd like to prevent our toaster, pacemaker, automobile, or drone from doing harm in the case when it starts malfunctioning catastrophically. Are there scenarios where one or more IOT manufacturers decide to remotely kill at the same time? What if their monitoring instruments make it appear that some threat actor(s) are making self-driving cars intentionally crash into crowds of people? Out of an abundance of caution, they might remotely kill the IOT devices to cut off the apparent disaster as it is unfolding. But maybe the threat actor is only spoofing the signals because they pwned the monitoring devices and infrastructure. Or maybe it's the precautionary action of some other IOT device system owner that is causing your monitoring system to go bonkers. I could go on but you get the idea.

How to Cope with Swan of No-Swan

It would be good to decouple the generating process if possible. Avoid the arms race to begin with. (Give peace a chance!)

Absent that, the best antidote is to treat evidence and signals pluralistically, which means avoiding the tendency to commit to one interpretation or another too early. This is very hard to do within one person or even one cohesive team. It's easier to assign different "lenses" to different people or teams who then proceed with their analysis and decision recommendations independently.

Finally, the decision makers who can "pull the trigger" should seek strategy alternatives to the preference for precautionary aggressiveness ("Shoot first, ask questions later"). While decision makers may feel like this is their only choice (and it may be), there is great advantage if more flexible alternatives can be found.

The Cyber Insurance Emperor Has No Clothes

2016-10-31T08:30:00.000-07:00

(Of course, the title is hyperbole and attention-seeking. Now that you are here, I hope you'll keep reading.)

(click to enlarge)

In the Hans Christian Anderson story, The Emperor's New Clothes, the collective delusion of the Emperor's grand clothes was burst by a young child who cried out: "But he has got nothing on!"

I don't mean that cyber insurance has no value or that it is a charade.

My main point: cyber insurance has the wrong clothes for the purposes and social value to which it aspires.

This blog post sketches the argument and evidence. I will be following up separately with more detailed and rigorous analysis (via computational modeling) that, I hope, will be publishable.

tl;dr: (switching metaphors)

As a driving force for better cyber risk management, today's cyber insurance is about as effective as eating soup with a fork.

(This is a long post. For readers who want to "cut to the chase", you can skip to the "Cyber Insurance is a Functional Misfit" section.)

There has been a ton of industry and academic articles on cyber insurance. On the academic side, nearly all of it has been economic analysis (mostly theoretical, some empirical). My lens is different: Organization Science. The focus is functional -- how do organizations actually work and make decisions.

This essay centers on cyber insurance in the USA, but except for some regulatory details, I believe that my argument holds for other regions.

Cyber Insurance as "Emperor"

The conventional wisdom in cyber security economics and policy circles is that the world would be a better and safer place if every firm (or most firms) bought cyber insurance. This idea is so widely accepted that what people mostly debate is "how do we promote cyber insurance as a market and industry?" and "what can we do to complement cyber insurance (e.g. standards, regulation, safe harbors, etc.)?" Certainly some people think that other institutions would work better (e.g. legal liability and tort law, regulatory mandates and penalties, technological solutions). But almost no one has come forward to proclaim that cyber insurance, in its present form, was not fit for purpose.

Thus, metaphorically, cyber insurance has become a conceptual "Emperor" to whom nearly everyone pays homage.

Like nearly all liability insurance, the putative economic function of cyber insurance is risk transfer from the insured parties to the insurers. But almost no one argues that the main problem with cyber security today is that insured parties (firms, consumers) are bearing excessive risk and therefore underinvest. This is not the desired social benefit has earned cyber insurance the honorific of "Emperor".

While most business insurance companies and lines operate on the value proposition of risk transfer, there is a subset that have been centered on risk reduction and best practices in addition to risk transfer. The exemplar is Hartford Steam Boiler, founded in 1866:

" 'the first company in America devoted primarily to industrial safety.' 'Hartford Standards' quickly became the specifications for boiler design, manufacture and maintenance."

In the same vein, the desired social benefits of cyber insurance are four-fold (paraphrasing):

"...provide financial incentives for firms to adopt best practices"
"...promote the adoption of standards that are compatible with best practices"
"Cyber insurance will be more agile than regulations, thereby more likely to promote good things and avoid bad unintended consequences."
"Insurance companies will build ever-better models of risk so that collectively we will be better able to quantify the costs and benefits of specific security practices."

Here is video from a panel discussion hosted by New America Foundation on Oct. 28, 2016. Robert Morgus (Policy Analyst, New America) gives a good, concise summary of cyber insurance, the current state, and (at 46:37), he enumerates the three of the four social benefits listed above.

(The purpose of the panel event was to promote their new book Cyber Insecurity: Navigating the Perils of the Next Information Age, which looks quite interesting.)

For more comprehensive summary, here is an hour-long video panel discussion that provides a comprehensive summary of cyber insurance and how it should improve cyber risk management:

https://www.youtube.com/watch?v=EqO9PoaFUbg

More panels, more conventional wisdom:

From DHS: "Cybersecurity Insurance Industry Readout Reports":

https://www.dhs.gov/publication/cybersecurity-insurance-reports

Cyber Insurance's "Fine Clothes"

The metaphorical "clothing" of cyber insurance are the capabilities associated with insurance companies and their functional ecosystem (i.e. value networks, legal and regulatory environment, professional societies, trade groups, academic programs and certifications, etc.). The claim that cyber insurance will deliver the desired social benefits is equivalent to saying that cyber insurance has the right capabilities, given the characteristics and job-to-be-done in the marketplace.

[EDIT: Moved "Analytic Framework" to the end]

Cyber Insurance is a Functional Misfit

This section aims to itemize how cyber insurance is a misfit, given the desired social benefits. First, what do firms need from cyber security and cyber insurance?

The Cyber Security "Job to Be Done"

The "job to be done" (see here and here) for firms and consumers is to make better decisions and investments regarding cyber security so that the expected benefits of "good things" outweighs the uncertain costs and worry over "bad things".

Compared to other major decisions involving risk (e.g. buying a house, building a factory, licensing a patent), cyber security decisions are both distributed (in time and space) and complex (feedback loops, nonlinearities, etc.). In medium to large organizations, there are hundreds or thousands of key decisions and decision makers that matter -- security staffing and training, IT architecture, vendor and outsourcing strategy and practices, business process design, hiring/staffing/performance incentives, and so on. And these go beyond what most people focus on: security products/appliances (e.g. firewalls, proxies, anti-malware, etc.) and security policies (e.g. password length/complexity, administrator rights, etc.).

The "job to be done" is to influence (a.k.a. "nudge") the decision and implementation process at all (or most) of these points. Nearly all firms have no idea how all these decisions influence each other and how they influence overall cyber security risk. (The Ten Dimension Framework that I have proposed is one approach to solving this problem: managing for cyber security performance.)

Cyber Insurance Misfits in Ten Ways

What cyber insurance does is governed by their existing capabilities and functions of insurance firms, and their supporting ecosystem. In terms of capabilities and routines, cyber insurance is not significantly different than other forms of business insurance. The contracts look similar, the sales process is similar, the pricing/underwriting is similar, the marketing is similar, and the regulatory oversight is similar.

[EDIT]
For readers interested in the theoretical/analytical framework, see the Appendix section at the end.

I will organize my argument for misfits under ten desiderata:

1. Cyber Insurance is Bought by and Sold to the Wrong People

Cyber insurance is bought by, and sold to, the same people who buy other types of business insurance. These folks are in Finance, Legal, or maybe the Corporate Risk department (if it exists). In any complex business-to-business sales process, the buyers are generally a network of people, taking different roles and having different interests, e.g. economic buyers, user buyers, technical buyers, and coaches (i.e. internal allies for the vendor's sales team). Many organizations even outsource the process of insurance buying because they lack the expertise internally. A high percentage of corporate insurance is sold through brokers, who try to serve the needs of insurance providers and consumers, though in reality their interests are closely tied to providers.

Whose budget does it come out of? Not out of the CIO's or CISO's budget, nor out of any Line of Business executive. It comes out of some corporate budget under the CFO. Maybe it is allocated to business units, but probably bundled in with all the other corporate allocations.

Whose capital budget gets affected? Nobody's. Whose headcount increases or decreases? Nobody's.

When all the contract details are being negotiated, and all the forms and checklists are being generated and processed, who (in the buying organization) is mapping all that back to decisions and investments that influence security? Nobody.

Turn it around. Look at the most significant decisions and investments that affect security. Who makes those decisions? Who influences them? Who monitors them and gives feedback? Now that you have that set of people in mind -- how many of those people have any involvement in or awareness of the cyber insurance buying process? Close to zero.

In terms of affordances (see Appendix), the insurance industry does not have a sales/demand creation capability that is suited to the diffuse, poorly organized buying network in most organizations. What do both organizations do? They engage with the affordances that the are compatible with (same old sellers and buyers), just to make transactions work.

As a consequence, sellers don't get the corrective feedback they need, and neither do buyers. None of the people who are most important for cyber security have anything to do with the cyber insurance purchase process, and vice versa. The cyber insurance that gets bought -- with all the contract complexities, limits, and premium cost -- has a remote connection to what is actually going on in the organization, especially regarding key decisions and investments. Result: the products, services, and prices don't face effective selective pressure (in the evolutionary sense), and therefore they don't improve and instead stay stuck in the swamp.

2. The Information Basis for Underwriting Decisions is Weak

Underwriting is the process of setting coverages, limits, conditions, and premium prices. On what basis do they make this decision for cyber insurance. In a large majority of cases, the information comes from questionnaires, checklists, audit results, or other qualitative evaluations, and only once per purchase cycle. (There are some insurance companies that are starting to incorporate ratings and scores by firms like Bitsight, RiskRecon, and Cyence. But it is not yet clear how this information will influence underwriting or premiums.)

I argue there are two sub-misfits. The first misfit is that none of the information that insurance companies get about a firm is contextualized. For example, what is the difference in risk between two hospitals: Hospital A that has two factor authentication for all users but also has many people with system administrator privileges; vs Hospital B that has very few system administrators but almost no two factor authentication? The answer (if there is one) depends on all the other information and security features of their organization. But the insurance company doesn't see any of the systemic nature of this. All they see are check boxes (yes/no), or categorical answers, or sometimes numbers.

The second misfit is that insurance underwriters are making course grained decisions based on a hodge-podge of qualitative information that may or may not have anything to do with risk posture.

Several years ago, I had a conversation with someone at a conference who was knowledgeable about underwriting and risk management practices at diversified insurance firms. He said, basically, that cyber insurance was being treated as a "long term development" market, and that firms offering policies today were hoping to have good models of risk ten or twenty years from now. In the mean time, cyber risk was being bundled in with all the other "developmental risks" and as long as it, alone, wasn't generating too much in the way of losses, it would be OK for the insurers. In other words, cyber insurance was being priced like it was the table stakes at a (long term) poker game. (See #4, below).

3. The Structure of Cyber Insurance Contracts is a Poor Fit

The structure of cyber insurance contracts reflects the interests and world view of insurance firms and their ecosystem. Like nearly every insurance contract these days, they are long and complex, with limits, exclusions, deductibles, co-pays, riders, footnotes, and exit clauses "up the wazoo".

While the putative economic value of cyber insurance is risk transfer, in reality insurance companies (and their reinsurers) do not want too much risk transferred, of the wrong kind (esp. cascading, correlated risk).

But most firms and nearly all consumers do not really benefit economically from risk transfer. What they want (implicitly) is risk pricing so they can factor that into today's decisions and investments. All of the complexity of cyber insurance contracts makes it harder, not easier, for firms to do risk pricing, taking into account ALL of their costs under ALL contingent circumstances.

Imagine you wanted to estimate how much you would weigh at the end of the week, given how much food and exercise you have during the week. You buy a digital fitness & diet tracking device. GREAT! But then you find out that it will only report calories consumed while sitting down and motionless, in solid form, and not more than once per hour. And it will only report exercise when the sun is up and the temperature is between 55 and 65 degrees. Everything else is on you. THANKS FOR NOTHING!

4. The Claims Process is Too Expensive, Too Uncertain, and Not Frequent Enough

The capability for processing claims (administrative, legal, financial, etc.) is a core competency of insurance firms. But it is tuned for infrequent use by any given insured party. The claims themselves are processed with a combination of "factory" (for those that fit standard patterns) and "hand-crafted" methods (for everything else). They claims are expensive for insurance firms to process, even if they end up denying a claim. Claims are also expensive for insured parties to file, because almost always it requires specialist resources at expensive hourly rates.

Rather than infrequent claims, what society needs is frequent reporting of breaches and even near-misses. Only then will we have data that is rich enough to model risk at all scales.

Then there are the uncertainties -- timing of resolution/payment and grant-vs-deny decision, not to mention any lawsuits or arbitration that might follow a denial of a large claim. There may even be "time bombs" hidden in the contracts that you discover too late.

Here's a somewhat hypothetical example.

Robert Morgus (in the video above) reported that 100% of the policies he surveyed have explicit exclusions for nation-state actors and terrorists ("acts of war" by non-state actors). But who has the burden of proof in any claim that the threat actor was definitely not a nation state or terrorist? This detail will be buried somewhere in the insurance contract and terms. It may be phrased in legalese so that non-specialists may not recognize it or fully understand it. Maybe it is ambiguous, or covered in some other blanket clauses.

Let's say your firm has a costly breach and files a big claim. But your insurance company denies it on the grounds that you have not sufficiently proved that the threat actor was not a nation state or terrorist. WTF? Like most firms, yours does not have sufficient capabilities in threat intelligence, digital forensics, and law enforcement/government agency relations to adequately do threat actor attribution. You are screwed.

Because insurance contracts are so complex (see #3, above), there may be dozens or hundreds of similar hidden "time bombs" that could lead to denial of a claim. While most people would expect some rate of claim denial for insurance, they mostly focus on what they, the insured, might do or not do that would lead to a denial. But there are other forces at work in cyber insurance that have less to do with a specific firm and claim, and everything to do with the norms, regulations, and strategies of diversified insurance businesses. Insurance firms have significant incentives to deny a specific claim if they think it will set an undesirable precedent for the rest of their portfolio and for future contracts.

[EDIT]
To clarify, I am not arguing that cyber insurance claims payments is any worse (or better) than any other line of business insurance, or that it should be better in order to perform it's risk transfer function. I am arguing that -- for the incentive system role (and risk information service role) -- the costs and uncertainties of claims processing is a detriment and a misfit, especially convoluted exclusions and other contractual clauses.

The misfit of the claims process is a specific instance of a more general misfit...

5. The Information Flow between Insured and Insurance Firms is Woefully Inadequate

Generally, what is the capability for insurance firms to take in information and process information (i.e. make decisions) from all of their customers (insured)? By the standards of the 21st century internet age, it's almost zero. Many insurance firms don't even have a direct relationship. Instead, they go through brokers. It's even worse for reinsurers, who (economically speaking) may be carrying the largest portion of economic risk in cyber. Typically, the information comes in once per purchase cycle, with perhaps a "re-up" annually.

[EDIT]
Every limit and gap in coverage is also a gap in information flow between insured parties and insurers. Referring back to the hypothetical example in #4, if all contracts deny coverage in cases of nation-state actors and terrorists threat agents, then how will insurers ever get enough information to distinguish those types of attacks from all the others?

Insurance firms don't learn about near misses, early warning signs, or even evidence that the information they may have received is incomplete, ambiguous, or erroneous. Even those insurance firms that are subscribing to real-time rating or monitoring services will have a hard time making use of this information in a way that changes the value proposition for insured parties. Why?

One reason is regulation. In the US, each of the 50 states has laws and regulatory authority over insurance, including definitions of what is and isn't insurance. Folks involved in financial innovation (e.g.. derivatives -- both OTC and traded -- and also real-money prediction markets) have run into this regulatory thicket. Insurance firms are on the inside, meaning that their ability to offer anything that looks different from "insurance" is quite limited.

6. The Cycle Time Between "Stimulus" and "Response" is Too Slow

Assume there is some new significant information about risk (being higher or lower) due to some event or signal. How long will it take for that information to percolate through the whole insurance ecosystem so that it is reflected in prices, coverage, or other relevant "incentives"? The answer is, at best, months, but more likely a year or more. Compare that to the response time of security markets, or even cash commodity markets, which respond in seconds to days.

Now, we should assume that this adjustment process is noisy and subject to overshoot or undershoot. How long will the system take to reach a new equilibrium? Five years? Ten years? And how many more "information shocks" will happen during that time? (Many)

For any adaptive system to maintain stability, it must have a response and settling time much shorter than the frequency of "shocks". Though it may be more responsive than the legal/regulatory system, the cyber insurance ecosystem is still too slow and unresponsive to be effective as a market information processing system.

7. Variation in Premiums and Coverage is a Noisy, Unreliable Signal

The folk wisdom about cyber insurance is that higher (lower) premiums and coverage will be strong signals and incentives relating to worse (better) cyber security practices and investments.

The reality is that there is only a loose connection between premium prices/coverage and real risk posture (if we can posit that such a reality exists).

Many parts of the insurance industry are cyclical -- notably Property & Casualty. The reasons are disputed (see here, here, here, and here, for example), but the consequences on incentives are clear. If premiums rise and fall 50% (or more) over the course of 6 years (typical cycle period) for reasons that have nothing to do with any individual insured firm, what signal does that firm get each year? Especially if the difference due to it's own security practices might be only 5% - 10%? And what is the signal when exclusions, deductions, and caps appear and disappear due to pressures on the insurance firms (and their reinsurers)?

[EDIT]
Specialty Insurance lines may be less cyclical than P&C. Even so, they are prone to "hard" and "soft" market conditions where premiums are higher or lower than they would be otherwise because insurers are either rushing into the market, or fleeing the market (or holding back due to tight capital conditions). My point is that these premium fluctuations could be much wider than premium differences due to incrementally better or worse security practices/policies for a given firm.

[EDIT]
Even if premium pricing is not noisy and unreliable, as an incentive system it is easily gamed by the people who manage the relationship with insurers and brokers. Let's say that your executive in charge is named Cheap Bastard (C.B. for short) and your firm has a strong incentive plan to minimize insurance premiums The easiest and surest way for C.B. to achieve the incentive goal is to under-insure -- deductibles that are too high, limits that are too low, and so on. After all, who knows what the right/best amount of insurance is for your firm? C.B. is the expert, right? Maybe your Board is too smart for that and closes that loophole by pre-specifying the basic features of coverage. No problem for C.B. because he can agree to dozens or more exclusions in the details of the contract. Only C.B. will know. The insurer will know, too, and will agree to much lower premiums while fitting the basic requirements.

Aside from the contract, there are several other points in processes where C.B. can act as gate keeper to limit or cut off spending that would trigger a premium increase. Let's say a big breach happens. C.B. choses not to hire the big name digital forensics firm, and instead hires brother-in-law's small firm. No expensive external PR firm or external law firm, either. Instead, shift blame and costs on to customers, suppliers, contractors, and especially specific employees who take the blame. Fire them in a very public way to terrorize the rest. Report authoritatively the Board that "no systemic problems were found and no additional breaches were detected." Case closed. Stonewall the media. Tough out the stock market's reaction. Don't file an insurance claim if possible, and therefore your firm won't be put into a "high risk pool" or other action that lead to significantly higher premiums. At the end of the year, C.B. declares victory and collects a big bonus check.

[EDIT]
Yes, that is a fairly extreme example of gaming the system. Every incentive system is susceptible to being gamed, and none is perfect. The purpose of this example is to highlight how the particular characteristics here provide affordances (see Appendix, below) for certain types of manipulation strategies.

[EDIT]
There are a host of less egregious strategies available to executives and key staff that would effectively muffle the "signal" that premium and coverage differentials might provide. For example, cyber insurance premiums could be bundled into a larger collection of risk and compliance-related costs, and this bundle becomes the basis for organization and individual metrics. Because cyber insurance premiums would be a small percentage of the total, any actual or potential variation would be muted. Analogy: like trying to sing while pressing a pillow into your face.

8. Risk Retention Needs to be Promoted, Not Risk Transfer

Most firms don't really need risk transfer. Then why are they paying for it (voluntarily or involuntarily)? Doesn't that take everyone's eye off of the real success factor: smart and effective risk retention.

Broadly speaking, society benefits when the parties best positioned to manage a risk retain that risk. Yes, society benefits when excess risk is transferred and diversified. But when it comes to the operations and practices that give rise to risk in the first place, society does not want that risk sloughed off onto other parties (insurers, customers, suppliers, employees, etc.).

Consider this scenario. Your company is offered two insurance policies with identical coverage but premiums of $\$$100K vs. $\$$50K, and the difference depends on you implementing a list of 45 best practices that you don't do now.

But then along comes an Angel benefactor who offers your firm $\$$50K to implement the 45 practices, and $\$$0 if you don't.

With the Angel, there is no hassle with contracts, deductibles, exclusions, caps, etc. and you don't have to pay $\$$50K per year for coverage you don't really want or need.

Wouldn't the Angel's incentive payment be a much simpler, clearer external incentive than cyber insurance? And your firm would be retaining the risk, which is what you should have been doing all along.

I'm not advocating this sort of direct payment incentive scheme, but I offer it to shine a spotlight the cumbersome, costly baggage that cyber insurance brings while riding the horse of "incentives for best practices".

9. The Risk Models that Insurance Firms are Developing are Not What The Rest of Us Need

What firms need to make better decisions is a risk model of their risk, given their unique circumstances and alternatives. But that is not what insurance firms are modeling.

Like all risk transfer insurance, insurers are modeling the risk associated with a portfolio of contracts. What they care most about is the risk of the whole portfolio over time, not what any individual firm experiences. (This is core to the business model of nearly all insurance businesses.) While they do care about the probability distribution of losses (claims) for firms in a given class or portfolio, what they care about much more are correlated losses that cause "excess claims" and "ruin" for the portfolio. This is where their best modeling minds and resources are working.

Now, there has been a trend in the last 10 years toward "predictive modeling" (e.g. see here) in casualty insurance, including some experimentation in real-time monitoring of automobile drivers and others. There is a possibility that cyber insurance could evolve in this direction, but given all the countervailing forces, it's not a sure bet.

The main message here is that it is the core competencies (a.k.a. capabilities) of the insurance industry, plus the regulatory structure that they operate in, that create clear incentives for them to create and innovate in risk models that serve the insurer and not the insured.

10. Cyber Insurance Doesn't Fit Networked, Interdependent Risk

For some firms and agencies, their biggest cyber risk is related to their being part of a service network -- i.e. critical infrastructure such as power, water, communications, etc. They also face nation-state threat agents and terrorist threat agents, both explicitly excluded in all cyber insurance policies. What they need is some way of pricing and pooling risk that embraces the interdependent nature of their risks, not shuns it.

So far, I have discussed cyber insurance as though the only type of insured party were an end-user of information technology. But what about all the product and service vendors of information technology? Many firms have side businesses or complementary services that are, in effect, information technology services. Supply chain cyber risk is well-recognized and widely studied. But it raises a whole new class of interdependence between firms in the network -- events, controlling parties, causation, and so on -- all of which are not well-matched by today's cyber insurance industry.

In Closing

To reiterate: my argument is not that cyber insurance is bad or broken as a traditional risk transfer institution. It may be fine for that function. But that is not the main reason that policy people are promoting it.

I'm also not painting insurance firms or insurance brokers as "bad guys" or somehow incompetent. They are working within a well established set of capabilities and institutions, doing what they are inclined to do.

That's all for now. Look for my follow up posts where I will dive into these issues in more detail, including some computational modeling.

Appendix: Analytic Framework for Functional Ecosystems

An analytic framework can help us structure the evidence and arguments, drawing from Organization Science and Ecological Psychology. This framework is an ontology for what I am calling "Functional Ecosystems" to contrast it with reproductive, energetic, or material ecosystems.

Actors (firms, people, etc.) are agents with purposes and values, usually because they are operating within personally and socially meaningful roles.

A function is the reason or purpose for doing something. "Function" establishes goals or metrics or indicators of good vs. bad performance, and also places the activity in some greater context of purpose or strategy.

A capability is a general ability to perform some function to some degree of excellence. Capabilities include knowledge, experience, and supporting resources (time, money, people, etc.).

A routine is any process, procedure, or algorithm that can be carried out in a step-by-step fashion to some conclusion. The term "routine" is not meant to imply "typical" or "unexceptional". Think of it as a "computer subroutine", where the "computer" happens to be some combination of people, process, and technology. Any given routine will be associated with one or more capabilities, and each capabilities will have a portfolio of supporting/relevant routines to carryout specific actions.

A characteristic is a feature of something (an object, a service, a person, an organization) that can be detected, sensed, and/or distinguished, and also has some relevance to the actor or agent.

(Crucial!)

An affordance is an interrelation between an actor and its environment (incl. other actors) that serves as a resource for action, or facilitates action, given the actor's capabilities, intentions, purposes, and attention. Affordance seems obvious when interacting with physical objects (it's not!), but it gets really interesting when interacting with signs, signals, and information (100% the case in cyber insurance).

Interrelations are more than interfaces or protocols for communication or action. They are meaning-creating interactions that make purposeful action possible, given the complexities and details of any given situation.

(Putting these all together)

Actors utilize capabilities to carry out their functions. Capabilities are enacted by selecting, configuring, and deploying specific routines, whose characteristics are an appropriate or serviceable match to the affordances that they perceive and engage with.

A Simple Example

You are probably familiar with the phrase: "Use the right tool for the job." Some of you may even know this saying: "Don't bring a knife to a gun fight". They both allude to the performance benefits of having tools that are a good fit to the job at hand. The following simple (simplistic) example will, hopefully, make all the analytic framework clear.

Imagine you are 4 years old. You are very hungry and want to eat. You have four types of familiar food in front of you:

Hot soup (broth-like with no solid chunks)
Chicken meat (whole breast)
A glass of cold milk
Green peas

You have three utensils available, and (for some reason) can't use your hands directly:

Fork
Spoon
Straw

The function you want to perform is "eat food". Your capabilities include 1) manual manipulation of hand-sized objects; and 2) eating. Your routines include: 1) grasp, 2) rotate, 3) aim, 4) press, 5) lift, and so on. The characteristics of the food include 1) phase (solid vs. liquid), 2) size, 3) density, 4) temperature, 5) odors, etc. The characteristics of the utensils include 1) weight, 2) sharpness, 3) receptacle capacity, 4) topology, etc.

The affordances arise through the interrelations between you (as actor and tool user), your chosen utensils, and the characteristics of the food that you try to eat, with a given technique (a.k.a. routines). Adults take this for granted because they are so habitual and unconscious, but young children (if not instructed) will poke around to try to find some sort of affordance that will fulfill their function.

Imagine you, as a four year old, try to eat the chicken meat with the straw. Maybe if you jab at it hard enough, you will gouge out some chicken meat inside the straw. If the straw is both stiff enough and sharp enough (characteristics), you might succeed in spearing the meat like a knife, and might then attempt to bring it to your mouth. Nearly all plastic straws will be too weak for this.

We can continue on with this exploration process, and eventually, you (4 year old) will learn which utensils work best for which types of food.

The fourth food type -- green peas -- are a very interesting case, because experience might reveal that all three utensils have sufficient characteristics to be used to eat peas (assuming the straw diameter is large enough). But while eating peas with a spoon is almost the same as eating soup (i.e. same capability and nearly the same routines), eating peas with a straw is very different than drinking soup or milk with the same straw. Notice also that almost no adults would ever consider eating peas with a straw because they are so acculturated in what utensils are used for what foods, and in the cultural norms than go against "rude" behavior of inappropriate utensil use (and sounds). Therefore, affordances are not simply what we see or believe to be possible or appropriate. But they also aren't purely "in the world". They require active engagement in order to come into being.

Orange TRUMPeter Swans: When What You Know Ain't So

2016-10-19T19:35:00.000-07:00

Was Donald J. Trump's political rise in 2015-2016 a "black swan" event? "Yes" is the answer asserted by Jack Shafer this Politico article. "No" is the answer from other writers, including David Atkins in this article on the Washington Monthly Political Animal Blog.

Orange Swan

My answer is "Yes", but not in the same way that other events are Black Swans. Orange Swans like the Trump phenomenon is fits this aphorism:

"It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so." -- attributed to Mark Twain

In other words, the signature characteristic of Orange Swans is delusion.

Rethinking "Black Swans"

As I have mentioned at the start of this series, the "Black Swan event" metaphor is a conceptual mess. (This post is sixth in the series "Think You Understand Black Swans? Think Again".)

It doesn't make sense to label any set of events as "Black Swans". It's not the events themselves, but instead they are processes that involve generating mechanisms, our evidence about them, and our method of reasoning that make them unexpected and surprising.

Definition

An "Orange Swan" is a process where:

The generating process is deterministic, either simple or complex, but where randomness in time does not play a significant role;
The evidence arises in two epochs: first, when the event is considered "extremely improbable" or "impossible", and second, when the event is happening or has happened;
The method of reasoning is "conventional wisdom" or punditry about causes and effects in this domain.

Main Features

The primary feature of Orange Swans is defying conventional wisdom, especially strongly held, widely accepted and acclaimed conventional wisdom. When an Orange Swan comes to pass, it reveals how the conventional wisdom was delusional, at least in those particular circumstances.

Some systems of conventional wisdom feature strong rules about what is possible and what is impossible, on principle. Simply, these rules are treated as though they are laws that govern our part of the Universe. It's helpful for us to remember that there was a time, not too long ago, that everything in the human realm was governed by conventional wisdom of one sort or another -- e.g. who got sick and who got well, the weather next week, crop success or failure, and the right/best form of religion, government, and social organization. Divination can be seen as systems of conventional wisdom overlaid by random draws. Even in the realm of sciences, conventional wisdom still has great power and influence on the social processes of science -- e.g. who gets admitted to grad school, who gets grants, who gets credit or blame, etc.

But, in modern society, the domain of politics stands out as the Kingdom of Conventional Wisdom. The prominent exception are pollsters and other "quants" like Nate Silver's team at fivethirtyeight.com. But the vast majority of practitioners, analysts, researchers, and commentators trade in almost nothing but conventional wisdom. Everyone in the social system either echoes and reinforces the conventional wisdom, or tries to modify it by adding their own "wisdom" (i.e. rules of thumb, causal explanations or possibilities, etc.), usually backed only by a few stories, conversations, or a clever idea in the middle of the night.

Anyone running for or serving in elective office, especially at a Federal/national level, has to confront the prevailing conventional wisdom, whether favorable or unfavorable, because everyone believes that the conventional wisdom expresses the Laws of Politics at the present time.

Examples

The most obvious present example is Donald J. Trump's nomination as Republican Party candidate for the President of the United States. The Trump candidacy has been (wrongly) called a Black Swan event. There have been a ton of articles and blog posts on why Trump was never going to be the nominee, never going to have a chance of winning, or, at least that his chances were extremely small. There have also been a ton of articles that have attempted to explain how people who espoused conventional wisdom were so wrong. The tone of these reflective articles has been especially somber during periods when Trump's poll numbers were surging and it looked like he might actually win (the nomination or the general election).

Let's not fall into the trap of believing that only politics is susceptible to Orange Swans. Here are a few "conventional wisdoms" in other domains that fell to Orange Swans (i.e. seemingly impossible events that nonetheless came to pass or were observed):

World War I will be "over by Christmas" (Britain) -- see here and here
Earth is the center of the Universe -- geocentrism
Arrival of "abnormal" astronomical visitors like comets are unpredictable -- see here.

I'm leaving out extreme events that are due primarily to random/stochastic processes or would otherwise better be describe as a different colored Swan.

Why Orange Swans are Sometimes Extreme

There is nothing in the deterministic generating process that makes Orange Swans extreme. They might be experienced as extreme relative to the strength in belief they are impossible and, by implication, that the whole system of conventional wisdom might be wrong. The successful prediction of Halley's comet did not have a revolutionary effect because, by that time, there was considerable acceptance of new conventional wisdom about astronomy and what was or was not predictable.

The Orange Swan event will also be experienced as extreme if it threatens the social status of the arbiters and beneficiaries of conventional wisdom, for whom the Orange Swan might be imagined as "the end of the world".

Why Orange Swans are Surprising

When conventional wisdom includes strong rules about what is impossible, and that "impossible" event comes to pass, then it feels extremely surprising, and even unpredictable in principle. Not every system of conventional wisdom is susceptible, because not all include strong rules about impossibility.

Going into the 2016 elections, one expression of conventional wisdom about who can and can't win the nomination was presented in the book The Party Decides:

"...for the past several decades, The Party Decides shows, unelected insiders in both major parties have effectively selected candidates long before citizens reached the ballot box."

This version of conventional wisdom said: Trump was not, and could never be, acceptable to these "unelected insiders", and therefore would not be successful in getting the Republican nomination.

The events that unfolded were surprise after surprise for those holding this and other conventional wisdoms. (In their defense, Trump's recent decline in the polls and likely election defeat are seen as justifying the conventional wisdom that a candidate like Trump -- in general and especially in particular -- could not win a Presidential election.)

Why Orange Swans Can Be Rationalized in Retrospect

The causal relationships leading to the Orange Swan are rarely mysterious in retrospect, in part because randomness and uncertainty rarely play more than a minor role. Sometimes a new conventional wisdom takes over, replacing the old. Other times, the conventional wisdom reasserts itself, writing off the Orange Swan as an aberration. Occasionally, there is a general loss in confidence in any strong conventional wisdom.

How to Cope with Orange Swans

From an analysts point of view, here's how you can cope with Orange Swans:

Take inventory the conventional wisdom, especially what is deemed as "impossible".
Use scenario analysis methods to generate possible futures that appear to be "impossible" through the lens of conventional wisdom, but have plausible causal chains.
For critical problems, use formal or semi-formal methods of evidence analysis and inference, such as Analysis of Competing Hypotheses. This will help you avoid the pitfall of minimizing evidence that might support hypotheses in favor of Orange Swan events.

Public Statement to the Commission on Enhancing National Cybersecurity, 6-21-2016

2016-06-21T14:31:00.002-07:00

[Submitted in writing at this meeting. An informal 5 min. version was presented during the public comment period. This statement is my own and does not represent the views or interests of my employer.]

Summary

Cyber security desperately needs institutional innovation, especially involving incentives and metrics. Nearly every report since 2003 has included recommendations to do more R&D on incentives and metrics, but progress has been slow and inadequate.

Why?

Because we have the wrong model for research and development (R&D) on institutions.

My primary recommendation is that the Commission’s report should promote new R&D models for institutional innovation. We can learn from examples in other fields, including sustainability, public health, financial services, and energy.

What are Institutions and Institutional Innovation?

Institutions are norms, rules, and social structures that enable society to function. Examples include marriage, consumer credit reporting and scoring, and emissions credit markets.

Cyber security[1] has institutions today, but many are inadequate, dysfunctional, or missing. Examples:

overlapping “checklists + audits”;
professional certifications;
post-breach protection for consumers (e.g. credit monitoring);
lists of “best practices” that have never been tested or validated as “best” and therefore are no better than folklore.

There is plenty of talk about “standards”, “information sharing”, “public-private partnerships”, and “trusted third parties”, but these remain mostly talking points and not realities.

Institutional innovation is a set of processes that either change existing institutions in fundamental ways or create new institutions. Sometimes this happens with concerted effort by “institutional entrepreneurs”, and other times it happens through indirect and emergent mechanisms, including chance and “happy accidents”.

Institutional innovation takes a long time – typically ten to fifty years.

Institutional innovation works different from technological innovation, which we do well. In contrast, we have poor understanding of institutional innovation, especially on how to accelerate it or achieve specific goals.

Finally, institutions and institutional innovation should not be confused with “policy”. Changes to government policy may be an element of institutional innovation, but they do not encompass the main elements – people, processes, technology, organizations, and culture.

The Need: New Models of Innovation

Through my studies, I have come to believe that institutional innovation is much more complicated [2] than technological innovation. It is almost never a linear process from theory to practice with clearly defined stages.

There is no single best model for institutional innovation. There needs to be creativity in “who leads”, “who follows”, and “when”. The normal roles of government, academics, industry, and civil society organizations may be reversed or otherwise radically redrawn.

Techniques are different, too. It can be orchestrated as a “messy” design process [3]. Fruitful institutional innovation in cyber security might involve some of these:

“Skunk Works”
Rapid prototyping and pilot tests
Proof of Concept demonstrations
Bricolage[4] and exaptation[5]
Simulations or table-top exercises
Multi-stakeholder engagement processes
Competitions and contests
Crowd-sourced innovation (e.g. “hackathons” and open source software development)

What all of these have in common is that they produce something that can be tested and can support learning. They are more than talking and consensus meetings.

There are several academic fields that can contribute defining and analyzing new innovation models, including Institutional Sociology, Institutional Economics, Sociology of Innovation, Design Thinking, and the Science of Science Policy.

Role Models

To identify and test alternative innovation models, we can learn from institutional innovation successes and failures in other fields, including:

Common resource management (sustainability)
Epidemiology data collection and analysis (public health)
Crash and disaster investigation and reporting (safety)
Micro-lending and peer-to-peer lending (financial services)
Emissions credit markets and carbon offsets (energy)
Open software development (technology)
Disaster recovery and response[6] (homeland security)

In fact, there would be great benefit if there were a joint R&D initiative for institutional innovation that could apply to these other fields as well as cyber security. Furthermore, there would be benefit making this an international effort, not just limited to the United States.

Endnotes

[1] "Cyber security" includes information security, digital privacy, digital identity, digital information property, digital civil rights, and digital homeland & national defense.

[2] For case studies and theory, see: Padgett, J. F., & Powell, W. W. (2012). The Emergence of Organizations and Markets. Princeton, NJ: Princeton University Press.

[3] Ostrom, E. (2009). Understanding Institutional Diversity. Princeton, NJ: Princeton University Press.

[4] “something constructed or created from a diverse range of available things.”

[5] “a trait that has been co-opted for a use other than the one for which natural selection has built it.”

[6] See: Auerswald, P. E., Branscomb, L. M., Porte, T. M. L., & Michel-Kerjan, E. O. (2006). Seeds of Disaster, Roots of Response: How Private Action Can Reduce Public Vulnerability. Cambridge University Press.

Entry in #BSidesLV Logo Contest

2016-04-28T20:17:00.001-07:00

Here's my entry in the BSides Las Vegas logo contest. The crowd-chosen slogan is "Popping calc.exe since 2009".

click to enlarge

#Tay Twist: @Tayandyou Twitter Account Was Hijacked ...By Bungling Microsoft Test Engineers (Mar. 30)

2016-03-30T01:19:00.001-07:00

[Update 5:35am From CNBC http://www.cnbc.com/2016/03/30/tay-microsofts-ai-program-is-back-online.html:

Microsoft's artificial intelligence (AI) program, Tay, reappeared on Twitter on Wednesday after being deactivated last week for posting offensive messages. However, the program once again went wrong and Tay's account was set to private after it began repeating the same message over and over to other Twitter users. According to a Microsoft, the account was reactivated by accident during testing.
"Tay remains offline while we make adjustments," a spokesperson for the company told CNBC via email. "As part of testing, she was inadvertently activated on Twitter for a brief period of time." (emphasis added)

I'm puzzled by this explanation but I'll go back through the evidence to see which explanation is best supported.]

[Update 6:35am It now looks like the "account hack" was really a bungled test session by someone at Microsoft Research -- effectively a "self-hack".

Important: This episode was not "Tay being Tay".]

The @Tayandyou Twitter chatbot has been silent since last Thursday when Microsoft shut it down. Shortly after midnight today, Pacific time, the @Tayandyou Twitter account woke up and started blasting tweets at very high volume. All of these tweets included other Twitter handles in them, maybe from previous tweets, maybe from followers.

But it became immediately apparent that something was different and wrong. These tweets didn't look anything like the ones before, in style, structure, or sentience. From the tweet conversations and from the sequence of events, I believe that the @Tayandyou account was hacked today (March 30), and was active for 15 minutes, sending over 4,200 tweets.

[Update 4:30am
The online media has started posting articles, but they all treat this as more "Tay runs amok". Only The Verge has updated their story. If you read an article that doesn't at least consider that Tay's Twitter account was hacked, could you please add a comment with link to this post? Thanks.]

Here's an example of Hijacked-Tay's tweets (credit to @daviottenheimer):

(click to enlarge)

Notice that @TestAccountInt1 is in all these tweets. That account is very new, and probably tied to this takeover:

Notice the second to last tweet: "straight up drivin the hot mess express CHOO CHOOOOO". This is a very distinct expression, and if this was the real Tay chatbot, there would most likely be another tweet exactly like this somewhere in Twitter history. But I could only find two:

[Update 4:20am -- Corrected:]
The first is a retweet of AI-Tay (Mar 23). The second tweet may or may not be a historical tweet (i.e. pre-Tay), but there is no date. I tried to find it through search inside Twitter and I couldn't find it.

Hijacked-Tay's most frequent tweet, by far, was "You are too fast, please take a rest...". Here is a very small sample:

At about 12:15am, I flagged the account as "hacked" on the account page for @Tayandyou. Just a minute later, @Tayandyou tweets were set to "protected", so I couldn't see them any more.

Other folks (including @rogue_analyst) could still see @Tayandyou, but said that the tweet activity had ceased. A few minutes after that, no one could see tweets from @Tayandyou. Either Twitter took down the account or the hackers did themselves, having achieved their laughs. Here is a sample of how human users reacted:

During this takeover, Hijacked-Tay sent out over 4,200 tweets in about 15 minutes. Current tweets = "100K", while it was "95.8K" when it was shut down.

Notice that @Tayandyou's location is "the internets". Notice that the location for @TestAccountInt1, above, is "Da Internet". This is suggestive evidence that @TestAccountInt1 was created with @Tayandyou in mind.

Let's see if this episode gets any press attention, and how they treat it. I wonder if they will attribute it to Microsoft and more misbehavior because of AI poisoning?

[Update 2:30am]

Looks like someone is cleaning up the account. Tweet count was reduced steadily from 100K back down to 95K, which is 800 less than when Tay was shut down last week. Also all the pictures that were uploaded (mostly just text saying "You are too fast, please take a rest...") were also deleted.

Yes, there has been some press coverage from online media. No investigation and they don't mention hijack. But at least they present some information:

These articles treat this restart as if it were the real AI-Tay chatbot, seeing all these tweets as just another example of antisocial behavior that it "learned" last week. Several articles are just copied/repeated from other sources (a.k.a. "recycled beer foam").

They do include some interesting new tweet content from Hijacked-Tay, including this:

Another very distinct phrase. "Kush" is slang for marijuana. The line comes from a rap song by Gucci Mane called "Ordinary Gangsta". It's not a copy/paste of historical tweets, however. It's not even copy/paste from the song. I believe this was typed by a human, one of the hackers who took over the account.

Here is some video of the tweet stream, with sound track added. Not very informative.

[Update 3:30am]

There's a threat on 4chan.com/pol/ here. Mostly they are just reacting to the new activity. No evidence that anyone there is involved in the hack. In fact, none of the posters seem to be aware that it was hacked, and instead they believe this was AI-Tay.

There is some new information, including images posted by Hijacked-Tay. This was apparently the last image posted by Hijacked-Tay.

The binary decodes into ascii text:

Looks like the work of a hax0r to me, for the lulz.

Media Coverage of #TayFail Was "All Foam, No Beer"

2016-03-29T17:17:00.000-07:00

One of the most surprising things I've discovered in the course of investigating and reporting on Microsoft's Tay chatbot is how the rest of the media (traditional and online) have covered it, and how the digital media works in general.

None of the articles in major media included any investigation or research. None. Let that sink in.

All foam, no beer.

With only three exceptions, none of the minor media (including blogs) included any investigation or research either. Here are the three standouts:

"Microsoft’s Tay is an Example of Bad Design (or Why Interaction Design Matters, and so does QA-ing)" -- e.g. inadequate black list
"TayAndYou - toxic before human contact" -- e.g. inadequate filtering of training corpus, incl. tweet corpus.
"Why did Microsoft’s chatbot Tay fail, and what does it mean for Artificial Intelligence studies?" -- e.g. poor marketing communications and real-time management

There were about 500 articles on the Tay and aftermath, give or take. There were serious articles in New York Times, The Guardian, Washington Post, and others. At least 25 articles featured "what went wrong" or "lessons learned" angle. A small number of these included quotes from AI experts, who offered their opinions but did not, themselves, do any investigation or research into Tay.

Looking at the human side, I'd guess that there were about 200 to 400 humans involved in writing, editing, contributing to, and publish these articles. (Yes, probably fewer humans than number of different articles because of syndication or similar.) How could all of those humans, many who are considered professionals and journalists, decide to write and publish articles without credible evidence? I can think of five reasons.

First, they all seemed to believe the tweets were self-evident and thus there was nothing to investigate. Tay was AI because Microsoft said it was, and look at all these anti-social tweets! Case closed!

Second, this story easily fits into one of the frames and narratives about AI in the real world: it's potentially dangerous and could easily "go rogue".

Third, the Copy/Paste Ethic in our culture, where many people believe that it is perfectly reasonable and morally proper to write an article or paper by copying and pasting content from other digital content. Ask any high school teacher or college professor about this, and be prepared for either a rant or tears of frustration.

Fourth, economic incentives that have been crushing media organizations for years have pushed them to push out content as fast as possible with as little cost as possible. Even many articles that looked original were mostly quoting and referring to other articles.

Fifth and finally, some of these articles fit a the definition of a "hot take". A "hot take" is "a piece of opinion journalism hastily written in a scolding tone... [where] there’s a “just telling it like it is” attitude, even if, according to the best available data, it is not like that at all." In the case of Tay, the hot take articles focused on ranting and stone throwing, either at AI, at Internet trolls and troll culture, or both.

In the primary articles the net result was all foam, no beer. For the secondary and tertiary sources that mimicked and repeated from the primary articles, they were no better than recycled beer foam. Let that image sink in.

Recycled beer foam. From: The Monitors by Keith Laumer, p 1

Microsoft's Tay Has No AI

2016-03-27T16:19:00.001-07:00

(This is the third of three posts about Tay. Previous posts: "Poor Software QA..." and "...Smoking Gun...")

While nearly all the press about Microsoft's Twitter chatbot Tay (@Tayandyou) is about artificial intelligence (AI) and how AI can be poisoned by trolling users, there is a more disturbing possibility:

There is no AI (worthy of the name) in Tay. (probably)

I say "probably" because the evidence is strong but not conclusive and the Microsoft Research team has not publicly revealed their architecture or methods. But I'm willing to bet on it.

Evidence comes from three places. First is from observing a small non-random sample of Tay tweet and direct message sessions (posted by various users). Second is circumstantial, from composition of the team behind Tay. Third piece of evidence is from a person who claims to have worked at Microsoft Research on Tay until June 2015. He/she made two comments to my first post, but unfortunately deleted the second comment which had lots of details.

(click to enlarge)

AI Worthy of the Name

What is and is not "artificial intelligence" (AI) is a complex topic that is hotly debated. I won't resolve the debate here, but instead only explain my point of view.

To be worthy of AI, it is not enough that a system "look intelligent". The first famous chat bot was ELIZA, written between 1964 and 1966. "ELIZA was implemented using simple pattern matching techniques, but was taken seriously by several of its users, even after Weizenbaum [the author] explained to them how it worked." It looked intelligent, but really wasn't.

To be worthy of AI, the system's behavior needs to be a direct consequence of computational emulation of human intelligence at some level of abstraction. For Tay to be AI, its behavior should be based on some combination of NLP and common sense knowledge/reasoning, including understanding of conversations.

At the very least, any "learning" in Tay should be due to improved understanding, i.e. better perception, conception, affect, anticipation, and similar -- however these are implemented computationally or abstracted.

Does all of "machine learning" (ML) fit within this definition of AI? No. Some does, some doesn't. In nearly all use cases of ML, what we care about are outputs. We don't care if it is human-like or not. In some cases, ML does resemble human reasoning, but in other cases it does not.

[Update 9:45pm] Are modern search engines built on AI? I don't know anyone who makes that argument. Instead, the technology in search engines to mine the "intelligence" inherent in networks of links (the original Pagerank of Google) and various other properties of content and search behavior.

1) No Evidence of NLP or Common Sense Knowledge

Natural Language Processing (NLP) is a set of techniques that attempt computer understanding of natural language. (See the Stanford NLP site.) NLP involves sentence understanding, question answering, sentiment inference, and more.

Common sense knowledge is understanding of our everyday existence and the implications. Simple examples of such knowledge: "All trees are plants" but also "Some plants are not trees", and so on. It turns out that common sense knowledge is both vast and hard to compile in machine usable form. (See Cyc project here and recent news here). For what its worth, most NLP systems to not also try to include common sense knowledge and reasoning.

I won't go through a lot of examples, leaving that as an exercise for readers. Here's just one tweet sequence between Tay and a troll:

Looking at the above tweet stream, I can't see any evidence that Tay is performing any of this. Each of Tay's replies has some relation to the human tweet, but only in a vague general way. I can't see any specific text in Tay's replies that reflect any understanding of the human tweet. Furthermore, there is no evidence that Tay understands this sequence of tweets as a conversation. Each tweet+reply is atomic.

[Update 9:45pm] Here is the enlarge Tay reply-tweet from above. Tay found copied it verbatim from its Twitter corpus:

Also, I see no evidence that Tay has common sense understanding of any concepts, either those of humans or in Tay's utterances.

How could anyone believe that Tay had NLP capabilities or common sense? Part of the trick is the cryptic nature of tweets, being limited to 140 characters. Another is the social norms of many twitter users, especially younger users, who write cryptically, elliptically, and don't follow standard grammar and spelling. Most twitter users get used to inferring meaning from cryptic tweets, whether or not that meaning was being intention on the part of the tweet author.

2) The Bing Team Was Involved in Tay

Bing is Microsoft's web search engine, competing with Google and others. From tay.ai web page: "Tay is an artificial intelligent chat bot developed by Microsoft's Technology and Research and Bing teams to experiment with and conduct research on conversational understanding." (emphasis added) Why would the search engine team be involved in social chat bot research? I don't know. I'm not aware of other social bot research where search engine people, tools, or methods are being used.

One of the odd behaviors noted was that Tay was reusing, verbatim, tweets from recent history (i.e. not through direct interaction with users, a.k.a. "learning"). This example comes from a post by Steve Merity. Here's the human tweet and Tay's reply:

Where did Tay get this? Everything after "..." is copied from this tweet from its corpus:

It's easy to imagine this as a search result given "Ted Cruz" and "Zodiac Killer" as search text.

3) The Inside Story From Former MSR Engineer

In the comment shown above, you can see this anonymous commenter say that the technology behind Tay was basically search engine. This aligns with the fact that the Bing team was a significant participant.

In the deleted tweet, this commenter added important details (he/she left MSR in June 2015):

He/she did not see any AMIL bot (ALICE or other) in Tay, but instead such a bot might have been used to expand the corpus or train the search engine.
The basic design was this (crudely sketched):

The input text from human was converted into a Regular Expression (i.e. text with wild cards, etc.)
The regular expression is entered into the search engine, which searches over a social media corpus (presumably including Tweet history)
The search results are filtered through Python code ("a 100 or so" if-then-else statements) to pick the result(s) to use, and piece them together into Tay's reply tweet.

There is no understanding or reasoning about conversational context.
No NLP or semantic reasoning. No neural nets, deep learning, or other ML outside of what search engines are capable of.

Conclusions

From these three sources of evidence, I infer two things. First, I believe that an AIML-capable library (probably ALICE) was added after June 2015 in order to implement rule-based behavior they couldn't otherwise easily implement through the pipeline listed above. Second, I believe that the only learning in Tay was a) adding to the corpus through real-time interaction, and b) tuning the search engine though the normal adaptive mechanisms.

Microsoft #TAYFAIL Smoking Gun: ALICE Open Source AI Library and AIML

2016-03-26T15:24:00.004-07:00

[Update 3/27/16: see also the next post: Microsoft's Tay has no AI"]

As follow up to my previous post on Microsoft's Tay Twitter chatbot (@Tayandyou), I found evidence of where the "repeat after me" hidden feature came from. Credit goes to SSHX for this lead in his comment:

"This was a feature of AIML bots as well, that were popular in 'chatrooms' way back in the late 90's. You could ask questions with AIML tags and the bots would automatically start spewing source into the room and flooding it. Proud to say I did get banned from a lot of places."

A quick web search revealed great evidence. First, some context.

AIML is acronym for "Artificial Intelligence Markup Language", which "is an XML-compliant language that's easy to learn, and makes it possible for you to begin customizing an Alicebot or creating one from scratch within minutes." ALICE is acronym for "Artificial Linguistic Internet Computer Entity". ALICE is free natural language artificial intelligence chat robot.

Evidence

This Github page has a set of AIML statements staring with "R". (This is a fork of "9/26/2001 ALICE", so there are probably some differences between Base ALICE today.) Here are two statements matching "REPEAT AFTER ME" and "REPEAT THIS".

Snippet of AIML statements with "REPEAT AFTER ME" AND "REPEAT THIS"
(click to enlarge)

As it happens, there is an interactive web page with Base ALICE here. (Try it out yourself.) Here is what happened when I entered "repeat after me" and also "repeat this...":

In Base ALICE, the template response to "repeat after me" is "...". In other words, NOP ("no operation"). This is different from the AIML statement, above, which is ".....Seriously....Lets have a conversation and not play word games.....". Looks like someone just deleted the text following three periods.

But the template response to "repeat this X" is "X" (in quotes), which is consistent with the AIML statement, above.

Conclusion

From this evidence, I infer that Microsoft's Tay chatbot is using the open-sourced ALICE library (or similar AIML library) to implement rule-based behavior. Though they did implement some rules to thwart trolls (e.g. gamergate), they left in other rules from previous versions of ALICE (either Base ALICE or some forked versions).

My assertion about root cause stands: poor QA process on the ALICE rule set allowed the "repeat after me" feature to stay in, when it should have been removed or modified significantly.

Another inference is that "repeat after me" is probably not the only "hidden feature" in AIML rules that could have caused misbehavior. It was just the one that the trolls stumbled upon and exploited. Someone with access to Base ALICE rules and also variants could have exploited these other vulnerabilities.

Poor Software QA Is Root Cause of TAY-FAIL (Microsoft's AI Twitter Bot)

2016-03-25T16:51:00.002-07:00

[Update 3/26/16 3:40pm: Found the smoking gun. Read this new post. Also the recent post: "Microsoft's Tay has no AI"]

This happened:

"On Wednesday morning, the company unveiled Tay [@Tayandyou], a chat bot meant to mimic the verbal tics of a 19-year-old American girl, provided to the world at large via the messaging platforms Twitter, Kik and GroupMe. According to Microsoft, the aim was to 'conduct research on conversational understanding.' Company researchers programmed the bot to respond to messages in an 'entertaining' way, impersonating the audience it was created to target: 18- to 24-year-olds in the US. 'Microsoft’s AI fam from the internet that’s got zero chill,' Tay’s tagline read." (Wired)

Then it all went wrong, and Microsoft quickly pulled the plug:

"Hours into the chat bot’s launch, Tay was echoing Donald Trump’s stance on immigration, saying Hitler was right, and agreeing that 9/11 was probably an inside job. By the evening, Tay went offline, saying she was taking a break 'to absorb it all.' " (Wired)

Why did it go "terribly wrong"? Here are two articles that assert the problem is in the AI:

"It’s Your Fault Microsoft’s Teen AI Turned Into Such a Jerk" - Wired tl;dr: "this is just how this kind of AI works"
"Why Microsoft's 'Tay' AI bot went wrong...AI experts explain why it went terribly wrong" - TechRepublic tl;dr: "The system is designed to learn from its users, so it will become a reflection of their behavior".

The "blame AI" argument is: if you troll an AI bot hard enough and long enough, it will learn to be racist and vulgar. ([Update] For an example, see this section, at the end of this post)

I claim: the explanations that blame AI are wrong, at least in the specific case of tay.ai.

[Update 3/26/16 12:40pm]
Here are some good commentaries and analysis that address additional problems, beyond "respond to me" exploit that is my focus:

"Microsoft’s Tay is an Example of Bad Design (or Why Interaction Design Matters, and so does QA-ing)" -- e.g. inadequate black list
"TayAndYou - toxic before human contact" -- e.g. inadequate filtering of training corpus, incl. tweet corpus.
"Why did Microsoft’s chatbot Tay fail, and what does it mean for Artificial Intelligence studies?" -- e.g. poor marketing communications and real-time management

(WARNING: Foul, profane, and offensive language in images below)

Poor Software QA (Process) is Root Cause

[Update 3/26/16 4:00pm -- The software QA process involves detecting and preventing defects, both in the development process itself and in the lifecycle as software moves from development to production. It usually involves people with official QA roles and titles, but not always. In modern software development, everyone participates and has some responsibility for software QA.]

Sleuthing by @daviottenheimer led to discovery of evidence that twitter users were exploiting a hidden feature of Tay: "repeat after me". He found evidence in tweets and replies. I later found evidence in the on-line board 4chan.org/pol/. /pol/ is a chat and photo sharing board that appeals to people who are "anti-normie" to an extreme. They like "popping bubbles" (my words), i.e. trolling people, especially public people, who are mainstream, normal, proper, and/or politically correct. It is a free public board with no membership required and no real-name requirement.

The first /pol/ thread "Tay - New AI from Microsoft" started Wednesday, March 23, at 11:00am. (Here). This thread and subsequent threads show that the /pol/ community was enthusiastically trolling Tay, essentially at random and without any evidence of knowledge of the inner workings of Tay. Contrary to the experts in the two articles above, Tay was not poisoned by direct trolling. Here is an example:

(click to enlarge)

While many of the tweets and replies by Tay led to many laughs by the trolls, Tay was not poisoned in the first few hours.

Then, at about 16:50, @MacreadyKurt stumbled upon the undocumented command: "repeat after me".

(Click to enlarge)

At 16:55, @BASED_AN0N posted instructions on 4chan.org/pol/ (trip code "yp45OVHP"), along with proof of concept (POC).

(Click to enlarge)

Within minutes, this instruction was repeated on 4chan. Many more twitter trolls following @Tayandyou saw the exploit in their timeline, and the real poisoning began [but see Update].

[Original post]
This constitutes poisoning because Tay was not simply repeating the text a single time. Instead, the text was added into it's Natural Language Processing (NLP) system so that these words and phrases became reusable and remixable in future discourse. And, since Tay's NLP was designed to continuously learn and "improve" in real-time ("on-line learning"), the more the trolls conversed with Tay using these foul words and phrases, the more Tay reinforced them and used them.

[Update 5:25pm PDT]
I now think that the previous paragraph is wrong. After some searching, I haven't found evidence that the seeded/inserted text was used later by Tay. Instead, it appears that Tay ONLY repeated the text after the "repeat after me" prompt. Then, trolls would retweet and/or screen grab and tweet the photos or post on 4chan. Two examples are here and here.

[Update 6:45pm PDT] This article in Business Insider says that "In some — but not all — instances, people managed to have Tay say offensive comments by asking them to repeat them." Some of the images in the article give the impression that "repeat after me" did not immediately precede the worst Tay tweets. However they appear to be using pictures posted on social media by trolls, not pictures they got from Twitter threads. Therefore we can't vouch for the "...not all.." statement.

[Update 7:00pm PDT] Here is one example from Business Insider of "repeat after me" not appearing before the offending Tay tweet. I captured this image from Google cache of Twitter thread.

(Click to enlarge)

OK. This isn't a case of "repeat after me". But also isn't as foul/profane as the other examples. Instead, it looks like a fairly typical NLP generated sentence drawing on a large corpus. Here the NLP is linking Hitler, totalitarianism, and atheism, but putting them inappropriately in the context of Ricky Gervais. If the question was "Is Ricky Gervais a Christian?" Tay might have replied "Ricky Gervais learn religion from Jesus, the inventor of Christianity" using the same sentence structure. This sort of mistaken semantic construction is fairly common in generative AI, but if the sentences/phrases are short enough, then human readers tend to overlook them or interpolate some reasonable meaning (much like adults do with very young children).

Q: Was "Repeat after me" a Result or Consequence AI? A: No.

It is just too imperative. It's not chatty. Instead, I'd guess that this was a rule-based design feature, probably left over from the early stages of development where software engineers were the only people interacting with the Tay bot. Very simply, the "repeat after me" allows the developer-user to manually seed Tay's NLP system and then immediately see what happens with further interaction. [Update 8:19pm] Or it may be an early feature put in before the full NLP system was working.

Put another way: the sort of AI you need to make chat work does not also work well to act on imperatives. Just compare robot AI (where natural language AI is sometimes used) to conversational/social AI and you'll see that they don't share common functionality, and often have completely different architectures.

Also, there were quite a few rule-based behaviors that overrode and/or by-passed the NLP AI. Microsoft called it "a lot of filtering". One example is anti-trolling rule for the topic "Gamergate".

(Click to enlarge)

Nearly all social/interactive AI outside of academic research has some "cheats" or "kludges" that are hard-coded by developers to control behavior in a way that would be hard/complicated/costly to do with the AI engine itself. The "repeat after me" command was just one, probably to aid development and testing.

Q: Why Was It There? A: Bad QA.

The Official Microsoft Blog post, titled "Learning from Tay's introduction", does NOT acknowledge that the root cause was exploit of a hidden feature. Instead, they describe the root cause as a "critical oversight for this specific attack". In other words, they claim they didn't do enough troll testing.

Instead, I believe that it is more likely that the root cause is poor software QA, which is different than "penetration testing" as you would do to test if your system was vulnerable to trolling. If "repeat after me" was, in fact, a rule-driven behavior explicitly put in by a developer, the the QA failure was not detecting it and not making sure it was removed. The Microsoft blog post does not describe their QA process, and it may be that they do not have any engineers dedicated to software QA. After all, this is a project of Microsoft Research, not one of the product divisions/groups.

Caveats

I don't have access to the Tay code, test results, processes and procedures, or organization charts. My claims above are extrapolations from the evidence, plus builds on my own experience in software development and in working with corporate software development teams. As such, I may be wrong and someone might be able to produce contrary evidence. I hope so.

[Update] Example of a Bot That DID Become "Casually Racist"

[Update 3/26/16 2:10pm]

In my commentary, above, I assert that the primary root cause of "Tay-Fail" was exploit of a hidden feature ("repeat after me" rule) that should have been removed during software QA process. To be clear, I am not asserting that it is not possible to poison a social bot through persistent trolling. It is. I just don't believe that was the primary root cause of the worst cases TAY-FAIL.

Here is a good example, told by a developer ("bitshepard") in a comment on Ycombinator Hacker News:

"I have a chat bot that went casually racist about a day or two after activating it. After looking through the logs, I found a particularly vitriolic person that was responsible for the source of this bot's newfound hatred of Asians. My bot didn't get fixated on one particular topic, it just spewed racism and vitriol for a while until it learned some more words. Rather than nuking from orbit immediately, I left it alone to see if it would get past the racism.

So far, it's been a few months since activating the bot. It's not nearly as casually racist as before, but from time to time still throws out something racist just for the lulz. It had a hard time learning context, because of its environment and the linguistic skills of the denizens, but it has gotten much better at when it interacts with people.

Occasionally, newcomers get misled into believing the bot is actually a living person with a mental illness, and not just a collection of random bits of code cobbled together."

Make Grammar Great Again

2016-02-18T13:50:00.000-08:00

(Click to enlarge)