Today, in 2016, here is where I think we are.

1. Operations main focus is now scalability

In the past, our primary purpose in life was to build and babysit production. Today operations teams focus on scale. For some it could be traffic related (number of concurrent sessions, number of users, size of the dataset). For others it could be ability to move between states safely and at high pace (for example, fintech where high stakes make consumer web approaches to operations too risky).

Automation is a necessary condition of scalability, and if you want everything to be automated, you must have devops as your goal.

Devops is not something you adopt. It's something you achieve.

— Dmitriy Samovskiy (@somic) September 10, 2014

2. Operations team is now a foundation of your platform team

Engineering teams exist in order to create product. With many teams in a company, there is often a need to have a dedicated team whose job is to ensure productivity growth - code reuse, implementation patterns, client libraries, core APIs. This is what I call a platform team.

Historically, this platform team would partner with opertations but today we are already at the point where operations is a part of the platform team. I have another post on this topic.

3. No more dedicated operations teams at early stage companies

This is NoOps and I realize that many readers will have strong objections here but I believe it’s a new reality. Look at any early stage company - there are rarely pure operations roles among co-founders or early employees. At that stage, it’s a luxury that only a few can afford - when you are iterating very quickly, you don’t worry yet about scaling your service for large crowds that may never come - you care about features and market fit. People may wear many hats, and some will still do a lot of operations but as an add-on, not as their main function.

A pure operations role in the form it used to exist at young tech companies is now extinct.

4. More and more enterprise workloads require webscale

This is an emerging trend, where traditional enterprises start selling products and services directly to consumers and quickly discover that their operations chops in webscale world are lacking.

This is especially obvious to me in part because I live outside of the Bay Area.

5. What got us to our current state?

I think 2 main factors - IaaS cloud (along with AWS unstoppable quest to standardize everything that is not code itself, by offering it as a service) and operations people’s widespread adoption of software engineering productivity techniques (in other words, operations people becoming better developers and moving up from scripts).

6. Where are we headed within next 6 years?

I am pretty confident that in another 6 years a lot more truly large scale applications and services will be successfully running without a dedicated operations team. Where all engineering teams have at least one customer-facing product service running in production, operations team no longer exists as a standalone part of the engineering organization (which does not mean that operations expertise or tech ops knowledge or best practices become not needed).

I think it’s the future.

There are several things that I think are important to understand in the context of this situation.

Side note and a disclaimer. I am not trying to convince you to make any investment decision one way or another - my goal is to merely illustrate some aspects of investment analysis thought process that I strongly feel you should be considering. Times article clearly tells me not everybody pays attention to these, which is unfortunate.

No matter what anybody says, the final decision is yours and yours alone - in tech ops we say “you own your availability,” in the investment world “you own your investment decisions” carries the same weight.

On private company valuations

I frequently come across comparisons of privately held company valuations with those of publicly traded companies. While I understand that sometimes people use such charts to add color to the point they are making in which case these comparisons are just illustractions roughly approximating reality, I suggest you be careful taking them literally.

Stock of a publicly traded company has many technical characteristics that make it significantly more transparent to a trader or investor than stock of a startup. There are SEC filings that are public information (interestingly, as an investor you don’t need to read them all - the fact that they are published and there is a legal framework that ensures the filings are very likely to be true, mean that information will be priced in the shares when you are going to buy), there is daily trading volume supported by liquidity, there are short sellers keeping stock price honest, often there are publicly traded derivatives - all these factors combined make it easier to understand and value (assign price to) public stock.

All of these things are either missing or very limited for private companies. In extreme case, I could negotiate purchase of a single share of a startup directly with management at a sky-high valuation. The price I will pay is a result of only 2 opinions about how much a company is worth now - mine and management’s, and valuation from last transaction could be used as basis for future stock sales.

Also, keep in mind overall stock structure and share classes. With publicly traded company, stock structure is simpler (fewer classes) and better documented - if a most sophisticated investor in the world and I own a share of AAPL, we hold absolutely the same thing with equal rights. (Yes, I am aware of dual-class share structures in public companies, etc).

In private companies, multiple share classes could exist with very different terms which significantly impact how much each share of each class is worth depending on a lot of variables (think liquidation preferences, for example).

The fewer classes company stock has, the easier it is to come up with market cap number using simple math. For a startup, calculating market cap is harder (sometimes significantly) - if a company issues preferred stock during last round of financing, it doesn’t mean you can automatically take that price and apply it to common stock.

Sum up: comparing valuations of publicly traded companies and private companies is often (or at least frequently) the same as comparing apples to oranges. If you want to know what you are doing when investing, you should analyze such comparisons with great deal of caution.

On investing hard cash in the startup where you work

Like with any investment decision, it’s impossible to give a general recommendation in this case without knowing company details, your personal financial situation and your appetite for risk. But nevertheless, the thought process you should follow when deciding whether to invest or not is universal.

Mathematically expected value of your equity grant most often starts out as strictly positive - there is a set of outcomes when it’s 0 and there is a set of outcomes when it could be worth something, so overall it’s greater than 0. In other words, you may not win but you will not lose. If you add to your position and pay cash for it, understand that this statement is no longer true - there is now a chance you can lose real money.

When investing in your startup, just like when you make any other investment, you should consider several aspects:

• expected return on the investment relative to other investments in your portfolio and other asset classes available to you (is the risk justified? is this the best use of your investment dollars on risk adjusted basis?)
• opporunity cost (what else could you do with the money you are about to invest if you decided not to go ahead with this investment?)
• total loss (how severely will you be impacted if you lost all of your investment?)
• diversification (if this investment goes through, what will it do to your portfolio diversification?)
• time horizon (how is the expected time horizon of this investment affects timings of your future financial goals?)
• liquidity risk (it may be very hard to convert your investment to cash when you need it - do you have enough cushion in your portfolio?)

Furthermore, contrary to popular belief that an investment decision is either “yes, invest” or “no, don’t invest,” in reality you can not decide whether to invest or not without knowing the price you will pay and terms of your investment. As a general rule, almost always there exists a price at which even the worst company in the world could be a reasonable investment.

When you are buying shares of a publicly trading company, you usually first determine whether you like the company and want to own its shares, then figure out a good price (or set of prices) at which you’d be willing to buy, and then execute your plan.

Unfortunately, coming up with the price you should pay for shares of your privately held startup is the hardest task of them all. It’s very possible that you are at a significant information disadvantage relative to sophisticated investors (which in case of tech startups are angels and VCs), founders and management. You have to find ways to compensate for it.

Sum up: Investing in anything (startups, stock market, real estate, bonds, gold, tulips, etc) without having a very clear idea why you are doing it and how various plausible outcomes could affect your finances, is a slippery slope. Not only are you deciding whether to invest, you also need to pay attention to price you will need to pay and terms associated with your investment.

On JOBS Act

This section is US specific. One of the parts of JOBS Act focuses on establishing a legal framework for startups to raise money from individuals who are not sophisticated investors.

I have been skeptical of this idea for a long time, and the Times article once again reaffirmed my skepticism. It’s a free country, and the fact that something is legal does not necessarily mean that you should do it.

I remain super convinced that it’s nearly impossible for an outside unsophisticated investor to overcome information disadvantage when investing in equity of a tech startup. And if I am a tag-along (attach my investment dollars to those of a sophisticated investor), I don’t see why founders would ever want to give me terms they are giving angels who are bringing their Rolodex, expertise and experience.

I do see how unsophisticated investors could loan money to startups (it would be a bond, which is senior to all equity investors - bond holders are paid first), but I am not sure whether it will work for founders (and existing angels).

Sum up: In my opinion, NY Times article is a very strong empirical argument why unsophisticated individual investors should not make equity investments in startups, even if it becomes legal under JOBS Act.

In order to be successful, a SaaS must deliver superior product to what I would otherwise be able to build myself, plus deliver it at a price that I’d be willing to pay. Since price of the service is a factor in sales, it’s better to price service at a reasonable level. This in turn drives a need for SaaS companies to pursue economies of scale - it’s much better for them to build solutions that can be sold to many customers vs. build a customized solution for every customer.

As a result, one of the things many SaaS companies would do is they would put every customer’s data into the same data store of some sort, and would ensure data confidentiality through their service.

And here lies the rub. By doing this, they end up with much bigger datasets than an original problem domain called for. Bigger datasets are harder to work with - they require more optimization, more security, more planning, more of everything. Sometimes the size of SaaS dataset even prevents them from building functionality that I would be able to easily build had I needed to build it only for myself, on my small dataset.

This was inspired by a realization that one of the SaaS services I am using, lacks a certain API which would be trivial for me to implement had I had all of the SaaS functionality under my own control but only for my data.

Could bigger not always be better in this context? Does this line of thought affect technical architecture of SaaS data stores? Should it?

On our way towards data-driven technical operations where runtime data are collected to be analyzed by machines instead of humans, we will inevitably start seeing statistics play bigger and more prominent role in our data analysis systems.

As someone who is very interested in statistics, I see the following five potential pitfalls that are waiting for us as the use of statistics becomes more widespread in techops. I will order them by decreasing severity (this, of course, is very subjective).

Applicability of current estimates to the future

Statistics is about estimating parameters of entire population (see some terminology) based on a sample of observations. It’s very easy to take one of your backend services, calculate 95th percentile of its response time during top 4 usage hours on each of the recent 8 Wednesdays, calculate some of this sample’s statistics and then say next Wednesday will be somewhat like this with such and such error.

The problem here is that we currently take for granted the fact that past 8 Wednesdays are a good predictor of what’s going to happen next Wednesday.

Identification of good time boundaries for relevant samples

Let’s say you are studying performance characteristics of your database cluster. You take data for the past 2 months and analyze it as a single sample and come up with some results. What you neglected to account for, however, is that at certain time during these 2 months you upgraded disks on your database servers to much faster ones. Or you swapped out one library with another that led to a significent throughput increase.

I see this left and right all over the place. If your sample includes observations that are not similar, your sample is not good enough. See this for more details.

Data exploration vs detection of actionable alerts

There are two distinct use cases for statistics in tech ops. On one hand, you have looking to learn something from the data. It could be hypothesis testing or it could be trying to detect a trend or it could be predicting the future.

Another use case however which is completely distinct is trying to alert off of statistical data in near real time.

These 2 use cases do overlap a bit but in large part they are separate and a tool meant for one may not be a good fit for the other.

Dissemination of insight gleaned from data

Most people who are paying attention to what’s going on in the world get bombarded with thousands of statistics per day. These numbers often come from all sorts of “experts” and talking heads, who throw them in just to sound more knowledgeable. Looking for example? Any statement that includes “on average” without explaining how the sample was obtained.

It’s important to us because sooner or later we will have to share insight we gain from our data anlysis with others and we have to be aware that our audience’s understanding of statistics could be subpar. This is especially important if a person with whom you share results of your analysis is going to make a decision based on his or her interpretation of the results.

Low bar set by enterprise buyers

A lot of software vendors whose tools include elements of statistical analysis sell to enterprise. Unfortunately, state of the art at enterprise is so low that when a sales person says “standard deviation,” they immediately get enterprse buyers’ attention.

As a result, tools include statistics for all sorts of things and misuse them. Case in point - claiming anything about “66% of your sample lies within one standard deviation from the mean” without explicitly proving normality.

Conclusion

Obtaining valid results by applying statistics in tech ops is not going to be easy but it’s an inevitable next phase that should be embraced as a challenge.

For more, please see this post.

I approach devops in enterprise as an end state. Aspirational end state that we are targeting to reach. Developers and operations sharing responsibility for the production environment, culture of collaboration, adoption of software development best practices in operations, heigtened awareness in development of run time and supportability issues - all these still stand. But the path to devops in the enterprise may end up being very different from that of a startup or a greenfield app.

(Side note: Devops is the same whether we are talking about a startup, a greenfield app or an enterprise - but paths to it differ. It's an important distinction.)

Devops is not easy in a typical enterprise. Discussing why this may be so is beyond the scope of this post. Instead, I want to focus on what to do about it.

I suspect that most enterprises could reach devops end state with fewer problems and sacrifices by introducing an intermediate step before devops. I couldn’t care less what it’s called but it can be described as platform/product alignment.

Take whatever org chart you have and identify all of technology. Draw a line not based on dev/ops but based on platform/product. Product is stuff you do for your customers, it implements your business logic. Platform is the rest, these teams focus on shared services and libraries as well as automating your runtime environments. Define what product delivers to platform (usually a set of binary artifacts like for example a WAR file to be deployed to a java container), sit back and enjoy watching your people's technical curiosity make things happen (or in other words - get out of the way of engineers).

The main trick here is that platform part of this alignment will consist of both developers and operations folks. Developers in these roles tend to be somewhat more inclined to adopt devops which helps this strategy a lot - instead of trying to convert everybody at once, you focus on those who are the easiest to convert to begin with.

One important note - under no circumstances should you draw any further lines within the platform team as it will kill the strategy. It’s extremely important to make sure your platform is one team, not two separate team reporting to a Senior Director of Platform. Everybody on the platform team is a Platform Engineer (with numerals like Platform Engineer IV if that’s how you roll), even if their current responsibilities are not the same. Remember that in ideal devops, a developer’s day is identical to operations folks’ day - writing code, planning code, peer reviewing code, deploying code, meetings (yes, I remember that we are still talking about the enterprise) and there is no need to have different titles for folks in dev and ops.

As platform part of your technology organization starts solving problems they are tasked to solve, they will inevitably stumble upon techniques and practices that constitute devops. Remember that devops is not an edict that was invented in a locked room and sent to the rest to follow - it’s a set of techniques and procedures that emerged in the trenches, from bottom up. When facing similar issues, smart people are more likely than not to effectively rediscover the principles that devops is all about.

And after platform reaches devops end state, I am confident that devops methodologies will gain critical mass in your organization sufficient to generate enough momentum to let your entire organization (product included) reach devops.

This idea is not rocket science. It’s simple and straightforward and I am sure there are a lot of organizations in the world who are aready doing it this way. It’s relatively painless and easy to implement, especially if you consider alternatives.

It’s good time to act now - please consider this reorg to set your team up for success in 2015.

Finally, please note that the title of this post says “a path,” not “the path.” I think it’s the easiest way to reach devops for a typical enterprise but it’s not the only one. The key is to realize that devops is not something you adopt - it's something you achieve.

Enterprise IT is an interconnected set of people, processes, services, hardware and software. While its main purpose in life is to support the business, you could argue a bulk of productivity gains we managed to achieve in recent decades as a society can be attributed to improving IT. So ultimately enterprise IT is a good thing.

Within an enterprise, IT has 2 different internal customers. Business itself is one, but there is another - it’s application software developers. We could debate significance of each but it’s not relevant here.

When you have something as complex as a typical enterprise IT organization, you need people to help manage it - that’s where technology project managers come in. They perform many important functions and it’s not my intention to question their value to organization. But project managers have a very specific mandate - essentially bring order to flow of work, both internal changes and change requests from customers.

What’s missing from project managers’ mandate is asking their customers “what do you want us to be able to do for you.” Because that’s a function of product managers. Unlike project managers who take something as a given and bring order to its surroundings, product managers actively shape what this something should look like in order to maximize its value.

Let me give you an example. A Rails developer can develop a web application in any field - healthcare, social, logistics management and so on. She doesn’t need specific knowledge about the domain she’s working with in order to be successful. It is product managers from whom this knowledge comes though - they are the ones who specify “requirements” and prioritize them.

Project managers and product managers are not interchangeable and these roles have totally different backgrounds. A PMP certification helps in project management but won’t do anything in product management. Knowing ins and outs of medical billing will help in product management at a medical billing company but won’t help much in project management.

Now that we’ve established the difference between project managers and product managers, let’s get back to our enterprise IT. Here is the main point of this post and I am going to highlight it.

Enterprise IT doesn't need product managers for their business customer because business requirements for IT are only very high level ("be able to run accounting", "have good uptime"). So there is no need for product managers here. But enterprise IT forgot that they have 2 customers. And application developers, because they are also technologists, have very specific detailed things they want enterprise IT to do for them but there is no one to shape enterprise IT from within to respond to these needs.

People managing enterprise IT as a product to internal application developers would be the ones working on the following problems:

• what characteristics of data store do developers need
• what access to production do you developers need to enable safe and effective troubleshooting and how enterprise IT can make it happen
• how network should be partitioned to support developers' needs regarding moving apps between environments

And this is exactly what IaaS cloud computing did - it built IT as a product offering for application developers, considering their needs and wants.

If you are an enterprise IT organization today, IaaS is your competition. Your application developer customers will choose you only if your product is better. Or if they are forced to by corporate information security. Make your own conclusions.

My personal position in this discussion is very simple. I am not saying Openstack should fully adopt AWS API, nor am I saying Openstack should not fully adopt AWS API. What I am saying is which API is presented doesn’t matter, as long as API exists and remains stable, with reasonable version management (so that developers can write against it easily). And I am not alone. Just give us a reasonable API, everything else doesn’t matter.

Ideological footing for the viewpoint that Openstack should just adopt AWS APIs can be traced to works of Simon Wardley. Simon was the first to introduce cloud computing community to several important concepts from science, he is a well established thought leader in information technology widely followed by both cloud pundits and practitioners.

As part of the recent debate, Simon wrote a post titled The 'Innovation' Battle, in which he says in part:

[...] innovation is not about the interface but should be above and below the interface.

[...] [others] believe that innovation of the interface is paramount. They believe that what matters is the interface, they hence strongly oppose adoption of AWS APIs.

In this post, I would like to expose a flaw in Simon’s logic and convince you that both camps in API debate are actually in line with Simon’s guiding principle that innovation of the interface doesn’t matter.

What is an interface?

First, let’s think of what an interface is, in most general terms. Interface is a set of behaviors and intended use cases that something provides and adheres to. When 2 things provide same or very similar interface, they become interchangeable. Over time, as the number of things providing same or similar interface grows and interface itself is refined (gains new desired behaviors based on feedback and observations, gets rid of behaviors that customers found unnecessary), it becomes a commodity.

Take a regular automobile. What is the interface of a car, in most general terms? It consists of several behaviors. One has to have a way to get situated in a seating position in front of a steering wheel and instrument dashboard. One has to have a way to get a car to move forward, back, turn and stop. One has to have a way to see what’s behind a car. You could go further and say that over time car interface gained expectations of ability to steer with hands and brake and accelerate with feet. Maybe ability to listen to music on your iPhone is a must now - interfaces do evolve.

These are all general things that one expects from a car and that allow most people to operate any car of roughly the same size without too much difficulty or a user manual. This is car’s interface.

What’s the interface of IaaS cloud? In most general terms, it’s ability to programmatically request a running OS image (we now call it an instance), uniquely identifiable and addressible, connected to a specified network, with ability to stop it.

Now on to a key point. Try to think through historical evolution of cloud - from the first version of Amazon EC2 to other IaaS clouds to newer versions to EC2. Has the generic IaaS interface changed in any significant way? Not really - no one is innovating in the interface any more because a lot of interesting work there has already been done. Even mighty Google who came out with their IaaS cloud last year, haven’t done any significant innovation in the interface (this doesn’t mean they haven’t innovated in other areas which are more important to customers).

Interfaces don’t expand simply because vendors want to sell more things - interfaces evolve only when customers are actually using new behaviors and functionality.

API as Implementation of the Interface

Even though all car manufacturers implement the same interface, their implementations are all different. Doors can open to the side or up. Door handles could be flush or not. Same controls could sometimes be knobs or buttons. Steering wheels’ diameter is different. Controls signage is different.

Have you thought about why car manufacturers haven’t converged to a single implementation of the interface? I have. Because customers want differentiation and customers don’t mind the differences in what looks to them like details.

Similarly, IaaS API is implementation of the general interface. Order of parameters, POST or GET or PUT for API calls, details how to sign your request, instance ids hex with an “i-” in front or numeric, public IP address as NAT or directly on network interface. The list could go on and on and you can name tons of these little differences if you had a chance to work with many clouds (not merely talk about many clouds but actually write code that uses many clouds).

These details don’t matter to most developers (and every developer I have personally spoken to), just like whether windshield wipers are turned on by pushing lever up or down doesn’t matter to most car drivers.

Conclusion

I think Simon is correct to point out that innovation should be happening outside of the interface. And it is. But his assumption that generic interface to an IaaS cloud and API are one thing is flawed - they are not, and cloud is not the only commodity where this is evident, as I showed above. API is an implementation of the interface. Nobody is innovating in generic interface anymore on a large scale - the feature set is pretty solid shaped by current capabilities of available technologies.

One can very well believe that Openstack would be better off adopting AWS APIs but it can’t be solely because of Simon’s “innovation of the interface” argument.

TL;DR - This post is not about ways and techniques how to run IT systems better or more reliably or deal with failure more efficiently; it's about mathematical models (or lack thereof) of what risk in IT systems actually is. I bet at least 80% of you will stop reading this post now. For the other 20% - I am happy you are curious.

Ideas that I describe in this post were in large part shaped by literature on risk management and modeling in capital markets and portfolio management.

Do you know what our biggest problem in IT is? It’s our complete and utter inability to measure risk of the systems and services that we build. Often, people don’t even understand what the risk is and confuse risk with other concepts.

Why do we need to measure risk? To do A/B testing among other things. When you implement two approaches to solving a problem and you need to decide which one to put in production, you’d better be able to compare risk of each approach and act accordingly. Also, deeper understanding of risk should help us analytically determine risk of bigger systems consisting of individual smaller systems, components and services whose risk we know. Ability to measure risk analytically will allow us to forecast systems’ risk before spending time and resources building them - a huge win.

Let’s say you have a system S. It can be an individual service in your stack or it could be entire web-facing system that consists of multiple frontend and backend services. Let A(t) be a function that somehow measures overall health of your system S at time t such that 0 ≤ A(t) ≤ 1. There is no universal rule what A(t) could be. For some systems, it could be percentage of incoming queries answered correctly within a predetermined time. For others, it could be percentage of available workers.

Now let’s pick 2 numbers f and r such that 0 ≤ f < r ≤ 1 (as you may have guessed, “f” stands for failure and “r” stands for recovery). We will say that system S experienced a failure if its A(t) falls down to or below f, and we will say that system S has recovered from failure when after hitting f, A(t) climbs back up to at least r.

In this formalization, time to recovery (TTR) is going to be T_r - T_f such that:

• A(T_f) ≤ f
• A(T_r) ≥ r
• T_f < T_r
• ∀t, T_f ≤ t < T_r : A(t) < r

If you average TTRs over a period of time, you get a well known measure called mean time to recovery (MTTR).

Similarly, time between failures (TBF) is going to be T_f2 - T_f1 such that:

• A(T_f2) ≤ f
• A(T_f1) ≤ f
• T_f1 < T_f2
• ∃! T_r , T_f1 < T_r < T_f2 : A(T_r) ≥ r
• ∀t, T_f1 < t < T_r : A(t) < r
• ∀t, T_r < t < T_f2 : A(t) > f

(last 3 conditions indicate there was exactly 1 recovery between failures at T_f1 and T_f2)

If you average TBFs over a period of time, you get mean time between failures (MTBF).

TBF is an indicator how prone S is to failure (could be bad system design, but also could be complex domain with a lot of external dependencies). TTR shows how good your tooling and processes around failure detection and remediation are. But by themselves neither one of them, nor their means, tell you anything about risk.

Instead, pick a duration for a period of time, sum all TTRs in each period and divide them by how long each period is. For example, let’s use monthly scale and in a month of January your TTRs were 3 hours, 5 hours and 1 hour. 3 + 5 + 1 = 9, divided by 24*31 (number of hours in January) - 1.21%.

If you repeat this arithmetic exercise for many months, you will end up with a series of numbers representing amount of failure in S per month. We will make an assumption here that the random variable of monthly amount of failure is normally distributed (this assumption could be proven wrong in the future but I am going with it by default because many processes in nature are normally distributed; our industry doesn’t publish enough datasets about operations metrics publicly so that confirming whether this is true is nearly impossible at this time - see below).

If you average these numbers, what you will end up is expected amount of failure per month (mean of normal distribution, corresponds to top of the bell curve). Is this risk? No.

Imagine a system which is down 5% of the time every single month for the past 5 years, no matter what you do, no matter how much business has grown over this period, how much more complex the system has become now and how much you improved your automation. Risk in this system is almost 0 - you are almost positive its amount of failure next month will once again be 5%.

Risk corresponds to standard deviation of monthly amounts of failure, not their mean (standard deviation is square root of variance). A system S₁ with smaller variance (for example, every month S₁ is in failure mode roughly same amount of time) has lower risk than a system S₂ with larger variance (for example, one month it is in failure 10% of the time, another month 0.1% of the time), even if expected amount of failure of S₁ is bigger than that of S₂.

How about an individual change (say something that you get asked about when you attend a change management meeting), either repetitive like pushing out a new release or performed only once? Similar to a system, a change has a success function with values between 0 (change totally didn’t work) to 1 (change worked as expected without unexpected side effects). Same logic about expected success rate and risk as standard deviation of outcomes will apply here as well - you will be modeling this change as if it were performed over and over again.

To sum up:

• TBF shows how prone your system is to failure
• TTR shows how good your troubleshooting is
• mean amount of failure per period shows expected amount of failure per period
• standard deviation of amount of failure per period is risk
• risk and expected amount of failure are two totally different measures

Last year Christopher Brown in his talk discussed devops as craft and as a science. In order to properly graduate from the former to the latter, our discipline needs to start sharing hard data about techops metrics so that reasearch can be done and new theories can be tested against real-life historical data. I am primarily looking at shops of techops excellence like Amazon, Etsy, Facebook, Github, Google, Netflix, Twitter and others.

The term “software defined” first appeared in the context of networking and applied to a new paradigm in switches and/or routers.

Traditionally these units are built as relatively monolithic hardware devices. Their logical structure however can be viewed as consisting of 2 parts: data plane and control plane.

Data plane is where your data packets between hosts on the network are traveling on their way from point A to point B. Algorithms here are relatively straightforward, uniform, sensitive to speed and they rarely if ever need to change drastically - in short, a perfect candidate to be implemented in hardware.

Control plane however does not share these characteristics. As networks become more sophisticated, algorithms here can be pretty complex (and complexity often leads to bugs that need to be fixed). Algorithms here are far from uniform because they are expected to support a wide range of use cases and deployment scenarios. And they could be very dynamic.

The whole point of SDN was to realize that control plane is not a good match for rigid hardware. The idea was to implement each plane on top of something that is ideally fit for its characteristics - hardware for data plane, software for control plane.

This explains “software.” Let’s explain “defined” now.

Even in the past, you could control your network devices programmatically and they had extensive capabilities to reconfigure themselves dynamically. While this was not a intended use case and usually was an afterthought, it still was not impossible. But no matter how you did it, you still ended up managing an inflexible control plane within hardware. What SDN did was eliminate control plan from hardware. In SDN, there is no control plane within hardware - it’s all outside of hardware, implemented in software.

In SDN, there must be a clear deliniation between data plane and control plane. Data plane does not make any decisions by itself. It only performs its work and feeds runtime data to control plane and accepts commands from the latter. There can be no commands from data plane to control plane.

For example, you hear a vendor introduce API and describe it as SDN. You start digging and find out that their API is just a facade to their old control plane that still lives in hardware intermingled with the data plane such that they essentially form one inseparable piece. Is this a true SDN? No.

To sum up, “software defined” is an excellent term (not a misnomer) and it actually stands for some important design principles - separation of data plane from control plane and control plane being implemented in pure software.

All “software defined” is automation and dynamic reconfigurability but not all automation and dynamic reconfigurability is “software defined,” no matter how much marketing departments may want you to believe otherwise.

"software defined X" is to X what a google self-driving car is to a car

— Dmitriy Samovskiy (@somic) January 2, 2013

We all know what API is and we all know lots of examples of successful services that were made possible by someone else’s API. We also could recall several examples when API backfired, at least in part. Take, for instance, Twitter itself and its relationship with their developer ecosystem, which probably could be best described as “rocky at times.”

But instead of looking at API from ecosystem’s perspective, let’s look at it from the point of view of a provider.

Imagine you lead a company that offers a way to consume your service’s functionality programmatically - i.e., through API or SDK.

It is indeed possible that your goal is to build an ecosystem around your main service. But, on the other hand, perhaps ecosystem is not in your plans. There are many services and use cases where it’s significantly easier to interact with a service programmatically. For example, when you are importing your HR data into a new system, you most likely want to automate this process, as opposed to manually enter each individual record one by one. In other words, maybe the reason you make API available is not to form an ecosystem but to allow your customers to automate their interactions with your service.

And here is the dilemma.

At the time API is introduced, a provider can’t credibly signal which way they want their API to be used - in other words, it can’t send a signal credible enough to indicate whether they want to foster an ecosystem or they only want to enable time-saving operations (history shows that executives’ interviews and blog posts are not credible enough).

Furthermore, while ecosystem-aspiring providers will never mind the use of its API for pure automation, automation-aspiring providers can’t control in advance whether their API is used by a developer to help kickstart an ecosystem, even against the will of the provider.

When a service puts up its API with the stated intention of growing an ecosystem, what they actually mean is “we want you developers to be a part of our ecosystem so that our audience grows and more people are using us (even if we share the spoils with you) but if you do something (and as of now, we have no idea what it might be) that we like and that we think we should have more control over, we will adjust the rules of our ecosystem as we see fit at that time.”

The dilemma is that a provider can’t outline its rules early enough to avoid having to change them in the future because it can’t know early enough what uses of its API it will like and what use cases it won’t want to tolerate.

In this context, you may want to revisit links from my blog post from about 2 years ago titled Ecosystems and Platforms.