I like the term “Technical Debt” because it is an easy metaphor for the average business owner to understand and put into real terms. A particular benefit of the debt metaphor is that it’s very handy for communicating to non-technical people. Just like financial debt, Technical Debt will incur interest payments, which come in the form of the extra effort that we have to do in future development because of the choices that we make now.

Technical debt isn’t always bad, just like how a business may borrow and incur debt to take advantage of a market opportunity, developers may also incur technical debt to hit an important deadline or get a particular feature to market faster than if they had “done it right the first time.” There may also be prudent debt within a system where team members recognize that it may not be worth paying down if the interest payments are sufficiently small, such as within portions of the system that are rarely updated or touched by development – we may not need to care about comment density, complexity, or refactoring if that sub-system is never going to receive feature updates. The tricky thing about technical debt is that unlike money, sometimes it’s difficult to effectively measure how it will impact your future velocity and, in some cases, may never need to be paid off in the future. Each type of technical debt must be weighed against the specific system and its lifecycle.

Technical Debt comes from various sources, some of which can be good and some bad, but the idea behind the technical debt metaphor is that there is a cost associated to taking short cuts, making mistakes, or deliberate choices and that the cost of not dealing with these issues will increase over time.
It’s no secret that I’m a big fan of sonarQube ( https://www.sonarqube.org ), an open source dashboard for managing code quality. It tries to calculate a technical debt (called a ‘code smell’) for a code base, using static code analysis findings like code coverage of automated tests, code complexity, duplication, violations of coding practices, comment density, and the following of basic coding standards.

And while it does of a good job of reporting the areas of technical debt that it can through code analysis, what do the numbers really mean to the business? And this is where get into the fuzziness of technical debt and how it impacts the long-term vitality of a project. When I think about the biggest cost of technical debt, it usually revolves around how the designs or code implemented today may slow down our ability to deliver future features, thus creating an opportunity cost for lost revenue.
Keeping this in mind, when measuring technical debt, it’s important and specific to each project to identify the impact that these different kinds of technical debt have. It is by evaluating each type of technical debt that has the potential to hurt and figuring out when there is too much of a certain type of technical debt that we can start to intelligently manage it.
When looking at a project and evaluating the different kinds of technical debt and how much they might cost you, this involves a fuzzier approach than just reviewing the sonarQube dashboard. Here’s some of the categories I like to group by when discussing different types of technical debt and the interest we may end up paying on them:

Different Types of Technical Debt

Architectural ($$$$)

If you’re building out a system where a key component or platform is fundamentally flawed so that it’s not scalable or reliable, this can be a huge problem that you may not even realize until real customers are running on your product. If you can’t scale out your architecture the way you need to because of core dependency problems or incorrect assumptions about how your customers will be using your system, you will have no choice but to rewrite or retool huge chunks of the system.
A good example of this would be the game Star Citizen and their choice to build out the game on the CryEngine platform (https://www.extremetech.com/gaming/237434-star-citizen-single-player-delayed-indefinitely)

Fragile Code ($$$)

In every large system, there’s always a couple of modules that seem to give developers the most problems. These are the sub-systems or components with code that is hard to understand and expensive and dangerous to change because it was poorly written to begin with or uses extremely outdated technology. Because these subsystems are so fragile, no developer wants to touch them and when they do it’s usually to go in and apply a very specific fix for their situation and then move on. Because these short-sighted fixes accumulate over time, the problem only gets worse. These fragile components need to be identified and evaluated for a complete rewrite to ‘bullet-proof’ them or they will continue to be an expensive debt on the project’s ledger.

Untestable or Undertested Code ($$$)

Writing Unit Tests takes time. It also requires that developers write their code so that it can be unit tested. A developer who is writing their code so that it can be unit tested tends to break their functionality up into small atomic components that make unit testing easy. If your system has monolithic functions that don’t automate well and you choose not to take the time to refactor them, you end up with tests that are brittle and slow and keep falling apart whenever you change the code. This causes your testing expenses to increase over time as additional options and features are added to the code base. Even worse is when brittle automated tests are ignored on failure because “it always fails anyways”. This can lead to an increase in manual and exploratory testing costs as well as additional costs in unplanned work when code is returned with a slew of bug reports that could have been avoided with proper automated testing in place.

No Automated Deployment ($$)

I’m including this under technical debt because we pay “interest” on this every time there is a release in terms of man hours and inherent risks. This is one of those hidden costs that nobody seems to think about until you actually sit down and review how it’s impacting not only your releases, but also your development cadence. Manual release processes are inherently error prone. As such, each release ends up being an all-hands on deck scenario “just in case”. These costs keep adding up over time. Not only with late nights, but also with taking time out of the development team’s normal cycle to prepare for a release and losing productivity during their present cycle. Is the cost of automating a deployment more expensive than scheduling one manual release? Probably. But automation pays huge dividends on each subsequent release of the product and probably has one of the best long-term ROI’s.

Black Box Code ($$)

This is the code that just works, written by some long-lost Jedi Code master who has since left the company or retired. We all know it works and we see it working within our systems, but nobody can explain why it works the way that it does. This is also a really tricky area because the business may decide that it’s OK to carry this technical debt on the project ledger because there are no plans to change any of the functionality that this Black Box is responsible for. And that’s fine. Until something changes. And it doesn’t work. This type of technical debt is like those mortgages with the huge balloon payment at the end. You can get away with not paying it down for years and then either bite the bullet and pay it off or if your product reaches end-of-life you may be able to retire the system without ever having to pay it.

Outdated Libraries ($-$$$$)

This can be a small amount of technical debt or it can be a huge amount of technical debt. That’s because this has to be evaluated on the risk that it presents to the business. This is especially important when outdated libraries contain newly discovered security flaws ( like the outdated struts library that was exploited during the Experian hack https://www.cyberscoop.com/equifax-breach-apache-struts-fbi-investigation or the Heartbleed vulnerability ( http://heartbleed.com/ ). An outdated library may be considered a small amount of technical debt until it isn’t, and then it becomes an all hands on deck remediation exercise to get your systems patched before being exploited.

Poor Error Handling and Instrumentation ($$-$$$)

If you don’t have proper error handling in your code, it’s hard to troubleshoot when something goes wrong. Even worse, you may not notice that certain sub-systems are erroring out unless you have a way of instrumenting those errors and performance issues. When the system isn’t working in the manner that it should be, if you don’t have instrumentation in place it’s difficult to pin-point the root cause of the issues unless you have built in various windows into the processes of your systems.

Copy-and-paste code ($$)

When code works, it works; and sonarQube does a pretty decent job of figuring out duplicate code blocks through it’s static code analysis. So, we may end up with many slightly different variations of code structures that developers have cut and pasted and then slightly modified over the iterations in order to get code into production. We always tell ourselves, “At some point I can go back and parameterize the code to consolidate and refactor the functions.” But, that time is rarely budgeted during the project’s iterations and the debt continues to pile up. Any changes to how the code works now requires the developer to remember where the multiple code locations are and make the same updates over and over again. If it’s only a few spots it may not be a big deal, but ignoring the problem causes the costs of making additional updates greater over the life of the project.

Inconsistent Programming Practices ($-$$)

Sometimes it’s easy to tell who wrote what portion of the system by reviewing the code. One developer may always seem to use one particular pattern vs another, or they create wrappers around certain modules in a very specific way that makes their usage different than how another developer instantiates that code, or have variables named a particular way. This practice may go unnoticed in small teams, but the more developers who are involved in updating a system the more complex this problem becomes and the harder it is to hand off to other developers. Code should be a developer-neutral as possible and one should only be able to tell who wrote a particular line of code by reviewing the check-in logs.

Backwards Compatibility ($-$$)

Usually this is a necessary debt that should be carried on a short-term basis. You’re going to want to maintain some sort of compatibility with the previous version. But what about the version before that one, or the one before that? The further you go to maintain backwards (or forward) compatibility of your systems the greater the cost to maintain and test all the compatibility scenarios that your system can handle.

Inefficient Design ($)

In today’s day and age, hardware is cheap. Sometimes you can get away with wasteful practices by throwing some hardware at the problem and it will go away for a while. This can lead to some lazy programming practices where inefficient memory usage or processing will not surface during initial rollouts. As you scale out your compute needs will grow and these problems will start to surface.

Magic Numbers ($)

In general, magic numbers are unique values with meanings or multiple occurrences that can preferably be replaced by named constants. Their use in code is generally low hanging fruit, meaning that they can easily be replaced, but they make it difficult for other coders not as familiar with a system to get up to speed on the how and why of a particular magic number’s use. Replacing these values with a named constant in your code allows a more descriptive identifier to be used within the code and thus an easier understanding of the code blocks as a whole.

Custom Functions for Built In features (0-$)

Every programmer is going to have a different level of experience with a particular framework. As they build out the functions for a system, they may end up creating particular functions that are already handled within the framework. Once that function is built (assuming it doesn’t have major bugs associated with it) it becomes a sunk cost. Sure, it’s inefficient, but as long as it’s working, then there’s not really any technical debt associated with the duplicate functionality.

Documentation (0-$)

Nobody reads documentation. Any documentation that is written is usually out of date by the time it’s published. So, is this really technical debt? Maybe not, it will depend on the complexity of the program and why the documentation is being produced. For small projects it may be easier for the developer to just read through the code and consider it to be ‘self-documenting’ (assuming we have good commenting practices). But for larger projects or systems that require regulatory scrutiny or are subject to an audit, the documentation may be considered a necessary evil that must be produced.

Summary

The technical debt metaphor is useful because it gives us a model that non-technical team members can use to evaluate the choices made throughout the lifecycle of a project. There is also a useful distinction between debt that must be paid down and debt that can be carried over time.

Prudent debt can be considered acceptable if the team recognizes that they are taking on that debt, and understand the trade-off of an earlier release versus the costs of paying it off. The important part of this evaluation process is that the team recognizes that they are in fact taking on these risks and weighing them against the efforts needed to remediate the issues further down the product lifecycle and plan for that eventual paying of the piper.

Even the best teams will have debt to deal with as a project progresses through its lifecycle – So it’s important that the team members recognize this and make a conscious choice of when to accept technical debt and when to take the time to remediate it.

The post Is There Such a Thing As Good Technical Debt? appeared first on Experiences of an Inland Empire Dad.

“Are feature flags better for risk mitigation, fast feedback, hypothesis-driven development or subscription tiers?”

Yes.

Feature flags can be used to enable many different behaviors within the final product, they can give product owners fine-grained control over the product’s behavior and user accessibility both throughout the development lifecycle as well as within the client facing product.  They are valuable technique to power more effective DevOps and drive innovation throughout the development and delivery process.

Common types of feature flags include:

Kill Switch

A feature flag is a condition in your code, an IF/THEN for two different options.
At its simplest, a feature flag can be used to flag a new or risky behavior.  The great thing about a kill switch is that they can be made available within the product separate from a deployment.

You can turn on the new feature at any time.  If the feature doesn’t behave as expected, it’s possible to shut it off quickly.  This allows for development on other features to continue without forcing a complete rollback of your production code.

With feature flagging, you’re mitigating risk by making every feature encapsulated and controlled so if a feature has problems in production, they can be turned off rather than having a deployment rolled-back.

Beta Feedback / Canary Release

For faster feedback, beyond an on/off switch, you can control at a very granular level who sees the new feature.  This allows you to expose a new feature to an advanced group of “friendly” customers who are willing to try out some new features you are developing.  These beta testers can review the feature and take it for a spin and give you great feedback that might not have been considered when building out the original use-cases for the product.

Hypothesis-Driven Development & A/B Testing

Hypothesis-Driven Development Feature flags are also good for long-term access level control.  They help prove out certain metrics by being able to measure the end-user behavior depending on whether or not a feature is enabled for a group of users and measuring their behavior.  For example, if you have a feature that only advanced users should access, you can use a feature to have a different experience for “Newbie” vs “Power users”.
You can use also feature flags to control localization, due to various regulations throughout the world, you may find that you need to enable features in one country and disable it in others in order to stay within regulatory compliance for that particular nation.

Subscription Plans and Permissions Toggles

You can bundle several flags together to form a subscription, for example you can have a bronze, silver, and gold tier and enable various sets of features that each subscription plan might have available to them based on what subscription is tied to their account.  This is a model that many of the “Freemium” apps out there subscribe to, where everyone is granted access to a base set of features, but other features are reserved only for customers who are in the paid groups.

Managing the technical debt of Feature Flags

Feature Flags have a tendency to multiply rapidly, particularly when first introduced.  Toggles need to be viewed as technical debt and they come with a carrying cost, so it’s important to also keep them from proliferating within the application.

In order to keep the number of feature flags manageable, a team must be proactive in removing feature flags that are no longer needed.  Once a particular canary feature has been turned on in production and has proven stable, it should be removed and expired.  Other techniques might include adding governance to your development rules by placing a limit on the number of feature flags a system is allowed to have.  Once the limit is reached in order to add another feature flag, you’ll need to review the current set and remove some existing flags in order to make room under the cap.

The post Feature Flags as a Continuous Delivery Release Tool appeared first on Experiences of an Inland Empire Dad.

“You can’t manage what you don’t measure.”

–Peter Drucker

In today’s world of technology, every company that runs their business on software is a technology company.  During my career as a developer, I’ve seen many bad practices in the software delivery process that were dismissed with the statement, “Well, we’re not a software company, were an X company”, where X would be whatever sector of business the company was doing business in.  Deployments for these companies were always slow and painful manual processes and as such we ended up performing infrequent releases because, well, they were slow and painful.  In today’s market, gaining a competitive edge over the rest of the market means being able to rapidly respond to the market and delivering value to the business through software.

We know that software development and delivery is an increasingly difficult process as the complexity of systems increases, and that managing and improving any process or system requires insights into that system.  Measurements and data are key to creating an effective software value stream.

When measuring devOps values, there are two approaches:

• Survey data
• System data

Neither system data alone nor survey data alone can measure the effectiveness of a modern software delivery pipeline.  It is a complementary approach where the survey data provides leading indicators to system data, or provides insights that system data might not be able to capture yet (or at all).  Because of this, it is critical for organizations to understand what they can and cannot measure with each approach, and what steps they need to take to gain visibility into their software delivery value streams.

There are several reasons why both system and survey data should be used to measure the value streams that define your software-delivery processes.

Survey Data

Survey-based metrics generally refer to data about systems and people (culture) that comes from responses.  When collected correctly, survey data can provide accurate insights into systems, processes, and culture.  Surveys are also particularly good at capturing holistic overviews of systems, because the answers that respondents provide synthesize data related to automation, processes, and culture.  They are great for getting a top-down view of your overall software delivery process.

System Data

System-based metrics generally refer to data that comes from the various systems of record that make up an end-to-end software delivery value stream.  For many development groups, they look for that data where it’s captured from a particular system of record, such as an agile tool like Team Foundation Server for version control, or Jira for bug tracking.  However, with many of these agile tools, while they are good at providing some very specific bottom-up metrics, it’s not enough to provide the kind of visibility and reports that can be tied back to the goals of the company.  For example, to measure time to market for a customer request, you may need data from a customer tracking system, the requirements system, the agile tool, and the deployment tool chain.  You would then need to be able to correlate the value of implementing that feature for the customer relative to the expense of developing it to understand if it was worth doing.

Data from systems can provide very granular data, allowing you to report on subsystems and components.  As system data matures in a company, eventually system-level data can to provide a relatively full view of your system, but this requires full instrumentation, plus correlation across measures and maturity in reporting and visualization techniques so that teams can understand system state.  It’s generally fairly easy to get a specific data-point out of a sub-system, but harder to correlate that data point to the overall health of the process.

Using Both

There are still some measures that are important to software delivery, such as cultural measures, that survey-based measures will pick up and system-based metrics may miss.  In addition, having both types of metrics provides opportunities for cross referencing your data: if your survey measures provide data that is drastically different from the data coming from your systems, this can highlight gaps in the system.

Survey data provides an alternate view of your system, allowing you to identify problems or errors when there are two opposing views.  Do not automatically discount your survey measures when this happens: there can often be cases where changes in configurations or system behavior alter the way that system data is collected, while survey measures remain true and it is only the delta in these two measures that calls attention to changes in the underlying system.

Get Baselines

The important thing is to get started with a baseline collection immediately, and for many companies that will mean collecting survey data while efforts to capture and correlate system data are in flight.  In the absence of complete system measurements, comprehensive surveys can provide a holistic view of your system relatively quickly (i.e., within several weeks).

Getting end-to-end system data can be a long journey as you first need to deploy a measurement solution across systems, and then make sure that cross-system integration is in place so that the data can be properly correlated.

While it is important to start as early as possible to get the benefits of system data, deploying survey data provides an almost immediate value and source of baseline information.  This is valuable both for base-lining current and future survey data, and for comparing survey with system data once they are in place and your instrumentation process matures.

Maturity of Data Collection

Over time, many of the metrics that you first collected through the survey that represented synthesized data for your process can be replaced by the system instrumentation and collection points within your tool chain and automated processes.

Summary

Data collection is a balancing act between survey data and system data.  Survey data is a great way to get started and provides leading indicators to system data.

As system data collection and telemetry matures in a company, it can provide a relatively full view of your system, but this requires full instrumentation, plus correlations across different system measures so that teams can understand each system’s holistic state.

The important thing is to get started with a base-lining and start collecting the data now and for many companies that will mean collecting survey data while efforts to capture and correlate system data are in flight.

The post DevOps Metrics – Are You Collecting the Data? appeared first on Experiences of an Inland Empire Dad.

When you look at the health of a project and ask yourself if you’ve got a healthy agile project, a good indicator to measure this is the build cadence.  You can draw a rough analogy that the number of builds is similar to the pulse of a project.  A regular “heartbeat” indicates a good, healthy project, but a sporadic “heartbeat” is going to uncover an anemic project and may indicate the need for some concern about the overall health of the project.

Slow and Sporadic Builds

If you find that your heartbeat is sporadic, slow or lethargic, it’s a good bet that at least part of your pipeline is not fully automated and that there are manual builds steps slowing you down.  It may also indicate that your developers are checking code in on a very infrequent basis.  If this is the case, it means that your project will be missing out on some of the basic build metrics and a lot of automated metrics surrounding build quality.

A slow heartbeat will in turn will lead to slower feedback cycles, you’ll see merge debt piling up, and often your deployments will become riskier, leading to deployments being moved to weekends or after-hours events in order to reduce deployment risk.

This might also indicate that you haven’t fully captured all the automation needed for your build and release pipelines and that there are unseen manual steps that will need to be accounted for each time you release a build.

Rapid, Consistent Builds

A regular build cadence indicates you have a healthy, strong project with a solid heartbeat.

If you have multiple builds per day, that’s a good indicator that you have a healthy project.  Developers are checking code in frequently and receiving good feedback from your various code scanning tools like sonarQube that will help guide them into producing quality code.

You are also able to rapidly run a CI build and run through many testing cycles and unit tests, increasing confidence in the quality of your code for the project.  Small changes to the source code can be quickly tested and defects can be isolated to these small changes to determine the root cause if there are any breaking changes.

Additional Metrics

Once you’ve got a handle on your project heartbeat, there are additional metrics that will become important as you take your project’s pulse:

Build Length

The build takes longer and longer, it may not be a bad thing, but if you are finding that a build takes 15 minutes to complete, it means each gated check-in turns into an enforced coffee-break for the developer performing the check-in.  This is turn could lead to fewer check-ins by the development team if it takes an inordinate amount of time to perform these check-ins.

If you find that the CI build it taking a longer, it may be time to look at breaking out the CI builds for your modules up into smaller, more specific CI builds per module and push some of the larger time-sucking builds to nightly builds and integration testing.

Build Fails

If a build is failing due to quality gates or unit tests, this may indicate that developers are committing code that is not yet ready to be merged, or that you are having consistent quality problems with a specific developer.

Number of Commits per Developer

While your project heartbeat might be strong, individual developers may end up with infrequent check-ins, leading to an accumulation of merge debt for when they do finally perform a check-in, so keep an eye not only on the overall heartbeat, but individual contributors as well.

Rogue Commits

Each commit should be tied back to a work item that adds business value (backlog, epic, feature).  Each commit should be tied back to a business value or else it becomes difficult to measure the flow of business value as the project progresses.

Summary

CI Build Cadence is a good rough indicator of your project’s overall health, but needs to be considered within the confines of the project’s overall progress.  Multiple builds per day triggered by various developer check-ins can indicate that you have a healthy project.  But long periods between code check-ins may indicate an anemic project that is accumulating technical merge debt leading to problems with integrations in the future.

The post DevOps Metrics (CI Build Cadence) appeared first on Experiences of an Inland Empire Dad.

Adopting DevOps seems like a great strategic decision within most management circles.  And yet, time and time again we can see enterprises that struggle with their DevOps adoptions.  This is fairly common because most people don’t seem to get that implementing DevOps is not something you can just go out and “buy” or even fully plan for.  Planning out the adoption is important because we want to have the right resources in the right place.  Tools are useful because they help to enable processes, but don’t confuse your tool with your process or your people.  Just because you’ve gone out and planned a strategy for DevOps adoption or bought an automation or collaboration tool, that doesn’t mean you’re doing DevOps, it just means you’re setting yourself up for success along the journey.

“The Plan is useless, but planning is essential”

–Dwight Eisenhower

We’ve been told at a strategic level that we want to move to DevOps, management has gone out and put together a great plan for the rollout, maybe they’ve even already implemented some “DevOps” tools in the enterprise.  But how do we ensure that we are able to execute on that plan from a tactical and operational level in order to make it successful?

This is where the concept of OODA loops (Observe, Orient, Decide, Act) come in.  The OODA concepts were developed by United States Air Force Colonel John Boyd as applied to the combat operations process.  It is surprisingly well suited for other situations where we are “in the trenches,” trying to become a change agent within our organizations.  If you don’t think being a change agent and waging a battle have a lot in common, you’re not paying attention.

Observe

“If we don’t communicate with the outside world to gain information for knowledge and understanding, we die out to become a non-discerning and uninteresting part of that world.”

–John Boyd

The first step is always to observe the situation.  By observing and taking into account new information we create an open system and gain the information that’s important to creating mental models.  That’s not to say our observations will ever be perfect, or even right.  There are pitfalls we need to be aware of throughout our observations:

• We often cannot see the complete information, or the information is imperfect in some way.
• We receive so much information that separating the information from the data becomes difficult.
• Our own experiences and opinions will influence how we perceive the information

When looking to be a change agent for DevOps transformation, start by finding out where you are, gather metrics that can be benchmarked against so that we know which way the needle is going as we start our DevOps journey. Not every technique is going to work at every company, so we need a way to figure out how to fail fast while promoting the things that are working.  We all have a mental model of how things “should” be, but that’s rarely an accurate depiction of reality.  Deciding on and gathering metrics will help you get a better understanding of your operational situation.

But don’t assume that having the data is enough.  What is more important is how you interpret that data to bring out the truly valuable information.  Even if you have perfect data, without judgement and understanding of the data it’s meaningless.  One of the challenges of being effective in observation is knowing what information to monitor and applying the correct filters for that data.

Orientation

This is where we build out our mental models and synthesize the data we’ve gathered into knowledge.  As we receive more information we are constantly breaking apart our old paradigms and rebuilding our mental models, creating new ones from the pieces in a continuous process.  Boyd called this “destructive deduction” and “creative induction”, using old fragments to form new mental models that more closely align with what is really happening around us.

Most people are not bad decision makers, they just fail to place the information they have at hand in the proper context.  It is the context of the information that we’ve synthesized that turns it into knowledge that will lead to good decisions.

Good orienting is the ability to constantly make new mental models on the fly and in the face of uncertainty and unpredictable change.  It is also a continual process, as soon as you create a new model it quickly becomes outdated as the environment around you changes.  So, you must practice and have a robust toolbox of mental models that you can cultivate and grow in order to quickly assimilate information into actionable knowledge.

A good anecdote that can demonstrate this is the old saying, “When all you’ve got is a hammer, everything is a nail.”  This is illustrated quite well in the paradigm shift between BlockBuster and NetFlix.  Blockbuster’s business model focused on hard-copy movie rentals and their brick and mortar stores as their primary business.  NetFlix when they first came on the scene were focused on DVD rentals by mail as their business plan.  NetFlix quickly saw the shift in consumers wanting to stream media via the internet and shifted their business model.  BlockBuster on the other hand were slow to move away from their tried and true business model.  Eventually Blockbuster tried to shift their business model, however by then it was too late.  Orienting and changing your mental models is a constant process of destruction and construction based on the environment around you and the data you have available to you.

You also need to have multiple mental models, because if you’ve only got one or two of them, by the very nature of human psychology you’ll warp your reality so that your think the data and information you’ve observed will fit your models, and everything once again becomes a nail for your hammer.

It’s not something that comes easy, as humans we tend to be creatures of habit.  But destroying and creating mental models is something that comes with practice and experience.  And it does become easier and eventually becomes something that doesn’t require deliberate thought, just something you do.

Always Be Orienting (ABO) should be a part of your daily mantra.  Make it a goal that every day you add to your mental model toolbox, examine your existing models and ask yourself if they still apply to the information you now have.

In addition, not all models are equal, some work and some don’t.  Ones that work in one specific situation don’t always work well in another.  This is especially true of DevOps transformations because we have a lot of uncertain variables in the equation, especially when it comes to people.

You can go out and read case studies of what has (and has not) worked for other companies on their own DevOps journey all over the internet.  And these are great ways to add to your toolbox so that you have additional models, concepts and strategies that are ready to immediately implement with a similar situation comes up.  Of course, since every situation is different, if those don’t work you’ll need to continue the process of orientation until you create a model that is better suited for your particular situation.

And that’s why you Always need to Be Orienting, orientation turns information into knowledge and it is knowledge that is the real predictor of making good decisions.

Decision

When we decide, we’re selecting the model that we believe most closely matches our current information about the situation. This is our best guess at selecting a course of action and hence why it’s so important that we have many different models to choose from in our Orienting phase.  The more we have the more we can increase the likelihood of having a model that closely resembles the Observations that we have on hand.

Making a decision is easy if you’ve got the right mental models in place because you’re going to be able to predict what happens in the future.  The ability to predict a future outcome can be the difference between success and failure.

Making better and faster choices than your opponent gives you a greater chance of successful outcomes and a decisive advantage in influencing those outcomes.  The better you become at understanding your data and turning it into knowledge that will help you predict the future outcomes, the better decisions you will be able to make.

As you gain experience, eventually you’ll have observations about a situation that match up with certain proven mental models, there is no need to create and destroy a series of mental models and decide on which one is most appropriate, you’ll already know.  Having this ability to quickly orient and act is something Boyd called “Implicit Outcome and Control.”  It’s what allows you to speed up your OODA loops and make quick and successful decisions.

Act

Nothing is of benefit is gained unless you close the loop by actually carrying out the decision and acting. Once the results of the action are observed, you start the loop again.

When competing against others, you gain advantage the faster and better you can cycle through the loop.

Everything is Connected

While traditionally depicted as a cycle or ring, because each decision does not happen in a vacuum, each loop is actually a set of interacting loops that are constantly interrelated, operating on and orbiting each other.  Some of the loops are going to be small, rapidly occurring iterations that occur multiple times each hour or day, while other loops have larger orbits, moving at much slower speeds that sometimes take weeks or months to see a measurable result from.

Even then, just like in orbits that involve celestial bodies, the orbits themselves will act upon other loops that are in close proximity, similar to how the gravity of planets or moons will affect other entities that are within their gravity well.  Again, nothing ever happens in a vacuum and just like with planetary orbits, a rogue asteroid or other cataclysmic event can come in and disrupt your entire system.

Orientation touches every aspect because it’s how we interpret a situation based on culture, experience, new information, analysis, synthesis and even heritage.  That’s one of the main proponents for including diversity in decision making systems, it adds more models and options to the decisions you can make that create a better chance over overall success when you act upon those decisions.

Summary

So that’s OODA loops (or orbits) at a high level, its just a system/framework that can be used to help us with tactical (and strategic) actions to enable a desired outcome (in this case, implementing a DevOps culture within our company).  Obviously, it can be applied to other things in life as this model in particular is adapted from a decision-making framework for combat situations but as seen here can be adapted for other decision-making processes.

There are a couple points that really need to be emphasized about why this system works:

• Having the right metrics in place in order to gather meaningful data that can be translated into information.
• Having diverse mental models that will allow us to choose the best fit for the data that you’ve gathered to turn that information into knowledge.
• You must be constantly adapting to evolving situations.

DevOps is a process that involved people, process and tools.  Because we inherently have a certain amount of uncertainty in that system, balancing the equation is going to be a juggling act that requires us to constantly review the system and make adjustments as we gain additional knowledge and see the outcomes of our actions.

We can plan our journey at the beginning and set ourselves up for success.  But we also can’t be afraid to throw out that plan as the situation changes and adapt to new data.

“Everybody has a plan until they get punched in the mouth”

–Mike Tyson

It’s how you react to adversity that defines you, not the adversity itself.

The post How to Address Tactical Blockers to Strategic DevOps Transformation appeared first on Experiences of an Inland Empire Dad.

What’s been great about this evolution is that we’re now seeing a true shift towards a collaboration suite that is capable of supporting both large and small businesses.  Even better is that Microsoft seems to have really put some effort into removing a lot of the complexity of SharePoint and more emphasis on real document and peer collaboration features.  Meaning you get more out of the box features and apps that enable the average end user to immediately start reaping the rewards of owning the system and that you can foster some real citizen developer cultures within your organization, no matter the size.

I think one of the major tipping points for the collaboration space was when Slack was launched, creating a de facto standard of the features and capabilities that the average end user is looking for in an enterprise social/networking tool.

Microsoft really took that to heart, and you can see that with their release of Teams over a year ago with how they enabled a lot of deep integration with their entire suite of tools.  Teams ended up being the glue that Microsoft was looking for.  This was really highlighted last summer when Satya Nadella announced the Microsoft 365 initiative at the Microsoft Inspire conference.  They envisioned a product family of Office 365, Windows 10, enterprise mobility, cloud storage and security all wrapped up with a nice bow on it.

Now we’re seeing these great features in both the Teams app as well as the Microsoft 365 packages and a it feels like they are pulling back from their previous swiss army knife approach to the collaboration space that was the SharePoint wheel.  Instead of getting complex and highly customizable software platforms that do “everything” but not always in the best or most intuitive way, we’re getting more purpose driven features delivered in holistic cloud-based tool that lower the entry point into the collaboration space for the end user whether they are a small business or a large enterprise.

So where does this leave SharePoint?  I think we’re going to see a fundamental shift this year where much of the concepts that have surrounded the old “team site” and community templates of SharePoint will be moved into the Teams platform for real-time peer-to-peer collaboration scenarios.  Honestly, I see a lot of features in the Teams product that I’ve always wanted in the SharePoint collaboration team sites, or the Team Foundation Server chat rooms, or clunky Exchange calendars & meetings, OneNote notebooks, etc.

I think that SharePoint as a platform and as the backbone to a lot of these technologies will still remain strong, but I think a lot of the front-end, user facing GUI features native to the SharePoint product are going to fall victim to digital transformation ideas, that your data will sit on a back end SharePoint system “somewhere” and that your end-users will use a bevy of apps to access that SharePoint backbone, with some of them never experiencing the native SharePoint UI.  But I think that can be a good thing and I’m looking forward to what 2018 is going to hold in store for us SharePoint geeks.

The post Digital Transformation with Microsoft Collaboration Tools appeared first on Experiences of an Inland Empire Dad.

However, DevOps is also a culture that builds upon itself, with each layer building on the foundation of what’s come before, so in this regard there are building blocks that you need to have in place before you can build the next layer.  I often refer to this as the “LEGO” approach.  You have a bunch of individual building blocks, and in some organizations, you’ll have areas that are more mature with several blocks already put together into some basic structures and processes and you need to form a strategy to put them all together into one cohesive DevOps structure.

Here’s some of the basic building blocks that I think every organization would need in order to build a DevOps culture.

Metrics: We have to be able to measure where we are if we want to figure out where we need to go.  Metrics will help us figure out where we are, what’s important to the organization, and how each step in the DevOps journey will help improve our process to add value for our end users.  You need to be able to measure and report on them in order to reinforce positive behaviors.  Find a couple of basic measurements and put that up on a dashboard somewhere for the entire team to see in order to help them focus on what we’re looking to improve.

Source Control: This is probably one of the most fundamental building blocks that a company must have in place before building out any other part of their DevOps journey.  They have to have their code in a place where it can be accessed and reviewed and see what changes have been made.  This is fundamental to being able to protect their code, perform check-ins, peer reviews, and get an understanding of just what exactly it is they are pushing through the delivery pipeline.

This is also where we start to build out some of our other metrics.  Things like code churn and active files become easy to identify, even some of the more ambiguous metrics like lines of code are at least available to us in order to wrap our arms around just how big a project is or how much technical debt might be lurking under the covers.

Work Item Tracking:  Use some form of work item tracking for Features, Backlog Items, Bugs, and Tasks.  I’m a big fan of Agile, so having a Product Backlog or Kanban Board visible to the entire group will allow them to focus on a goal.  Tracking Work Items clearly lays out who is responsible for what and adds a level of accountability.  Each iteration should have a clear definition of what “done” means for the team.

Continuous Integration: Having implemented a source control system that allows your code to be centralized means you now have a location where you can start building a Continuous Integration (CI) system that can retrieve the source code and perform Automated Builds, Unit Tests and Static Code Analysis (SCA), all of which can also help to contribute to some useful metrics on how well you’re doing.  Developers should be encouraged to check their code in early and often so that we get constant feedback every time the project is built, tested, and packaged into build artifacts on a separate build machine (avoid the “It Works On My Machine” Syndrome).

When building your CI blocks, this is a great place to gather additional metrics where you can see that there are consistent green builds, all your Unit Tests are passing, and if you’re using tools like sonarQube or Fortify it will help to produce additional metrics around code quality, standards and technical debt where team members can visibly see the quality of the code through reporting dashboards and gain confidence in their builds.

Automated Deployment (Single Environment FIRST): This is where we start seeing a lot of synergy, but also a lot of issues with building trust in the process.  Too often an over-eager engineer will try to automate the entire pipeline in one shot.  Instead, start small working on automating the deployment to just a single environment, avoid trying to “Boil the Ocean” and instead work on ironing out the kinks with a single environment deployment.  Build trust and confidence in the ability to deploy the code with no manual steps involved.  Do this in a “throw-away” environment where you can deploy and destroy over and over again.  Over time, other colleagues will see the amazing advantages to having and automated deployment and will invite you in to enable this for the other environments in your pipeline.  But get your automated deployments dialed in first in the single environment, bullet-proof it and build that trust with the rest of the team that this is a process they can trust and have confidence in.

Automated Testing: Once we’ve got a fully automated and repeatable process to deploy the system to a test environment, work on automating the testing of that system with integration and coded UI testing.  In the CI build, we run unit tests, but with a fully functional system we have the opportunity to run integration and load tests.  Having the ability to automatically reconstitute an entire environment and run more advanced testing is critical to increasing the team’s velocity and reducing downtime due to simple mistakes that can be caught through basic function testing.  However, be sure not to put all your trust in automation testing.  These tests are great for scripting out positive testing, like being able to log in or query for known information, but not every test can be accounted for and there’s nothing like having a human at the helm to do exploratory testing.

The great thing about this is that it also becomes an iterative process, each time you build out a test it can be scripted out and you can build out your suite of tests so each iteration is more comprehensive than the last.  I also want to point out that while Automated Testing is great for basic scenarios, but trying to maintain complex scenarios through scripts can sometimes be more trouble than it’s worth.  One of the worst things you can have is Automated Tests that continually fail and are ignored because it’s chalked up to “Oh, that always fails when the interface changes” because now you’re training your team to ignore certain tests.  Every scripted test should pass every single time before certifying the build.

Infrastructure As Code: Once we’ve got the deployment of the code and we can successfully deploy to a known set of servers or environments, the next step is to introduce the concept known as Infrastructure as Code (IaC) in the pipeline.  By having your code and infrastructure configurations both stored in source control, this will enable your team to stand up new environments by simply changing some parameters in source control.  Having parameterized scripts utilizing tools like Chef or Puppet, we can free up the manual process of logging onto VMWare portals to provision servers or setting up features like web servers that need to be configured before being able to successfully deploy to the environment, further reducing the chance of human error.

Monitoring: This is one of those areas that often gets overlooked.  Once the iteration is over and the latest code is pushed to production, we often forget that operating the system is just as important as building it.  But this is also one of the most critical pieces for business.  We must monitor our code and our systems as it is running in production to ensure we are delivering value to our end users.  This should not just be limited to the usual performance and exception monitoring either.  Monitoring should help answer the more interesting questions to the business, like how the end users are using the application, what features they use the most that might benefit from being expanded as well as features that may not be used at all and can be trimmed from the system in a future iteration. Someone once explained to me that feature curation like running a restaurant.  As you operate the restaurant you expand the menu and enhance certain meals but if nobody orders certain items then you take them off the menu.  No restaurant would ever try to have 400 items on the menu and do all the shopping and everything else that goes with having those menu items that nobody ever orders, so why would you maintain 400 features in your system that nobody ever uses?

Continuous Improvement: If something isn’t broken that doesn’t mean you can’t improve it.  As you move through the process of integrating different tools and processes into your delivery pipeline, think about how you can speed up the activities that take the longest or contain the most risk.  Review your workflow constraints and think about how you can reduce cycle times.  DevOps is not just about automating your pipeline. Sometimes you need to go back and rearchitect your existing system in a way that enables it to efficiently deploy in a way that does not impact your users.  A lot of systems aren’t built with that capability of handling sophisticated DevOps deployment strategies like Blue-Green deployments, canary builds, rings, and other strategies that enable zero downtime while creating low-risk releases.  But these are exactly the opportunities you’re going to want to target as your DevOps journey matures.

As many people like to say, “A journey of a thousand miles begins with the first step.”  The DevOps journey is no different, it starts with a few small steps and you start to move down the maturity road.  Not everyone starts in the same place, but it’s a road we all walk down together.  DevOps is about the journey and not the destination.

The post DevOps – Where should we start? appeared first on Experiences of an Inland Empire Dad.

Declutter What You Don’t Need

This should be obvious, but get rid of what you don’t need or no longer use.  For me, putting things in the garage for storage until later is like buying things on a credit card.  Eventually you need to pay it off.  Take the time before the new year to pay down your clutter debt by going through the piles and figuring out what needs to be stored, donated, or finally tossed.  For me, i find that this is a technique that works extremely well:

• Put out three bins labeled: keep, donate, and toss.
• Start going through the piles.  Spend 30 minutes doing this.
• After 30 minutes,  review what you’ve got in the bins
  • Keep: Now find a spot for what you want to hang on to
  • Donate: load the bin in the back of the car, get it out of the garage and to a place that can use it
  • Toss: If it’s beyond help, make sure you dispose of these items properly.
    • Recycle what you can
    • Properly dispose of those toxic items (i.e. paint, used motor oil, batteries, etc.)

I know there are various declutter programs and processes out there that say put it all in a big pile.  But lets face it, that’s a major chore and when you’ve got kids sometimes you’re going to keep putting it off over and over because it’s too big of a time commitment.  The garage isn’t going anywhere though (that’s what got us here in the first place!) so for me, 30 minutes of sorting and 30 minutes of dealing with the bins helps me to keep on top.  After all, everyone can find an hour during the weekend, right?  If i find myself with a couple more hours than I anticipated for the weekend, just rinse and repeat, but always keep it to 30 minute cycles.

Be Energy Efficient

This is an interesting one, but most people where I live tend to have attached garages to their homes.  The doors from the house to the garage wear over time, or maybe aren’t always as insulated as they could be, so take the time to check the insulation here and make sure it’s properly sealed and that if you have a self-closing hinge that it’s actually working properly.  Better yet, check the outside garage door as well. if your outer door isn’t insulated, consider replacing it as well with a properly insulated garage door in order to keep the extreme heat and cold from invading your garage space as well as your home.

Also review the lighting in the garage.  Are you using incandescent or CFLs?  Now might be a good time to swap them out for some LED lamps.  LEDs are usually twice as efficient and last four times as long as CFLs.  They also don’t contain toxic materials like mercury, which makes them safer to dispose of when they do need to be replaced.

Look to replace your traditional garage light switches with more modern motion sensing switches.  I can’t tell you the number of times I’ve walked into the garage to go grab something real quick, go back into the house to finish whatever I was working on, and forget to turn off the light until after

Rid Yourself of Toxic Chemicals

This is another big one with my family.  We’re really good about making sure our daily cleaning supplies are non-toxic.  But once it’s out in the garage, a lot of times we fall into the old trap of, “Out of sight, out of mind.”  This year during the annual greening of the garage, we’ll be taking a close look at a lot of the chemicals, glues, paints and other items that we’ve stored in there over the years to pull out what we no longer need, what’s expired, or what we can replace with something more organic.

Paints

Paints are another area to focus in on.  We’ve got paint cans in the garage from years ago.  Normally if you properly store latex paint it can last for up to 10 years.  The garage is not however the best place to store paint as it can be subject to some pretty extreme temperature fluctuations.  I’m guessing at this point it’s probably all gone bad.  so we’ll be disposing of the paints we’ve stored there for the last 6 years, “just in case we might need more of this paint some day.”  Instead we’ll be taking some pictures of the labels on the cans along with the color mixes that were used to create that specific shade and taping them up in the cabinet where we’ve been storing these paints.  So, just in case we do need to get this exact shade again, we’ll have the records easily at hand to be able to run down to the hardware store and have them mix up a fresh batch for us.

If you do want to hand onto the paints you’ve stored in the garage, just make sure that you….

• Do not use paint that is moldy.
• Do not use paint that has frozen (which can happen in colder climates when stored in the garage)
• Do not use latex paint produced before August 1990 as it may contain mercury.
• Probably not necessary to say this, but do not use paint produced before 1977 because it may contain lead.

So that’s the plan for this year’s annual greening and cleaning of the garage.

The post Annual Greening and Cleaning of the Garage appeared first on Experiences of an Inland Empire Dad.

Some people have referred to this as the “Velocity Gap”. The blocker that is our biggest constraint in delivery because we can only move as fast as our slowest member, and more often than not that ends up being the database implementation.  If the goal is that we deliver 10 times a day, 10 manual database deployments can create quite the bottleneck.

The post DevOps and Databases Velocity Gap appeared first on Experiences of an Inland Empire Dad.

The post Identifying Bottlenecks to devOps Transformations appeared first on Experiences of an Inland Empire Dad.