Alex Honor posted a helpful writeup on rundeck.org:

If you are new to Rundeck, watch Will Sterling give an introduction to what Rundeck can do and how he uses it to automate work at DataLogix.

Here are some notable quotes:

• “Multi-tentant command orchestration and process automation with WebGUI, CLI, and RESTful API.”
• “Target nodes with rich metadata. Never use hostnames again.”
• “Process automation via multi-step jobs…Options allow users to pick one or more values.”
• “Rundeck makes everything in the GUI available through the API.”

Besides showing off the basics, Will opened up Eclipse to step through ruby code that talks to Puppet to communicate node information to Rundeck. His code only includes nodes he can ping and have a certain class.

Will also showed off how he uses resty as nice shell based way to access the Rundeck API.

The post Will Sterling presentation on Rundeck at the April CLUE Meeting (Video) appeared first on dev2ops.

Following up from yesterday’s post.

The post Why we end up with complex tools appeared first on dev2ops.

“We have to grasp not only the Know-How but also ‘Know Why…’”, – Shigeo Shingo (Toyota)

From time to time, the business comes to us asking for help to reach a desired outcome. The outcome might be to launch a new product, add new features to the web site, anticipate a bunch of new customers, open a new market, or shorten lead time for customer requests.

So, we technicians put on our thinking caps and get excited because we are going to get to make something.
“What’s the best solution?”, we ask ourselves. “It better cover all contingincies for changing scale or future concerns”, say the far thinkers. “It’s a big project with lots of moving parts”, exclaims the project manager.

What often happens? We technicians get mired in how we solve the problem, using all the tricks up our sleeves, relishing in our know how. We know how that goes. We say: “Meta-object protocols will make our solution infinitely extendable, even dynamically! This inheritance hiearchy design coupled with good component composition will ensure the software architectured is correctly multi layered.” Blank stare and reply from the business manager: “What?” We say: “Oh it’s superior technology”

Admit it. We’ve all been there saying or listening to this kind of stuff.

What stands between us and our goal is the complexity of the solution.

“The most dangerous kind of waste is the waste we do not recognize.” – Shigeo Shingo (Toyota)

Complexity is a killer. We already live in a dynamic, fast paced world and the business is always changing. Our work world is complex so why make our solutions complex? Complex solutions are fragile, hard to roll out and adopt, time consuming to fix and extend. Complex designs just make life harder. Does the business outcome depend on any of this complexity in the solution? If not, we are gold plating and are just creating a form of waste.

So, how do we avoid building the complexity? Here’s some good rules to live by:

1. Stay focused on the “Know Why”.

Be clear you understand the desired outcome the business expects. Are you sure about it? Do others agree with your interpretation. If not, you’ll be alone defending your choices or you might suffer living with your own decisions.

2. Don’t fall prey to your “Know How”.

Watch out for the inclination to create advanced/sophisticated/over-engineered designs and implementions. This just adds time, risk, and money to get the work done.

3. Be disciplined.

Building a new solution and migrating from the “old way” of doing things to the new way is by definition a transformation. The new way will require its own new “Know How” and will not be perfect/complete/usable the first time (or the 2nd, 3rd… time). If  moving to the new solution is painful, you have to stay disciplined and keep iterating to make the solution less painful.  Stay focussed on the outcome the business cares about and expects from you as your guiding principle.

4. Do the simplest thing.

Choosing products and tools that are inherently simple to use, require little Know How from everyone. Tool shininess gets the heart rate up because new is fun but this isn’t what the business cares about.

You know the old saying, the shortest path between two points is a straight line.

The post The key to simplicity isn’t “Know How” it’s “Know Why” appeared first on dev2ops.

Below is the interview in full. If you like this interview, I recommend signing up for the Discover Performance newsletter. They routinely have good articles on interesting and relevant topics and avoid injecting too much self-serving HP bias (a difficult task for enterprise funded content!).

The post IT stability and business innovation are not enemies appeared first on dev2ops.

The conventional wisdom on shell scripts is that… well… “shell scripts suck”. But why? Shell as a language is extremely powerful and useful but shell scripts can quickly become unwieldy when trying to use amongst a team or in long-lived operations. But what if you had a framework that solved the team-level problems and the lacking of standardization while letting you use the full power and familiarity of shell? Enter Rerun, a simple tool that turns your favorite shell scripts into modular automation that has standardized options handling, command line completion, documentation generation, and a built-in test framework. Suddenly shell scripts don’t suck so bad anymore.

Why am I so interested in Rerun? Because I’ve seen Rerun have a positive effect on a very real human problem: In most non-startup organizations, the DevOps divide is made worse by a mismatch of skills, tools, and technologies.

It’s common for the Ops team to have used a tool like Puppet to automate server config and image building. But when it comes to app deployment and config, the various app teams don’t have the Puppet skills or motivation to follow suit. So each app team picks their own tooling or glue language. Of course, this just confuses Ops and makes their lives more difficult. Sometimes there will be a centralized release team (often now awkwardly rebranded as the “DevOps Team”) that will attempt to pick their own solution. But, neither Dev nor Ops ends up bring happy with the choice and the “DevOps Team” is now the bottleneck in the middle. Lots of noble DevOps intentions die in scenarios like this.

The effect of Rerun is that everyone can now come to the table on equal footing and use shell as their lingua franca. They can learn to collaborate using a simple framework for the “glue” that holds things together (of course, Ops still builds server images using a config management tool and Dev still builds their apps the way they want to). The built-in documentation generation and test automation framework makes handoffs easier. Everyone knows the simple command and options interfaces, but can also read each others code if need be (it’s just shell scripts, after all). Once you get everyone engaged and contributing to bridging the DevOps Gap, you can collaboratively start to look to other newer, specialized solutions.

I have to get Rerun’s creator, Alex Honor, to do a full post on Rerun. In the meantime you might find these videos interesting:

Video 1: Chuck Scott gives a tour of how he uses Rerun to turn his “keeper scripts” into reusable, standardized, test-driven automation

Video 2: Group discussion with Anthony Shortland, Lee Thompson, Chuck Scott that looks at an example of a DevOps toolchain automated with Rerun

The post Rerun: Making shell scripts even more useful (and a bit cool, again) appeared first on dev2ops.

(*Why simple and self-contained? Many reasons… the company-wide adoption of full config management solution is proceeding at uneven pace, the need to use a lowest-common denominator language so you can have simple handoffs, you want to avoid “religious” tool wars, you need a very small footprint, you need it to be totally portable, …. and the list goes on)

Here is where you can find the Jenkins Rerun module:

https://github.com/rerun-modules/jenkins

The post Automating the full management lifecycle of Jenkins using Rerun appeared first on dev2ops.

Where does this flow often bog down? Handoffs. Whether the handoffs are within a team (e.g. Dev to Dev) or between teams (e.g. Dev to Ops), there is always the need to pass work from one stage of the lifecycle to the next.

http://www.flickr.com/photos/seven13avenue/2791099838/in/photostream/

At DTO Solutions, our clients are often already aware that they have flow problems when they ask us to for help. When we use techniques like Value-Stream Mapping to learn how the work flows, handoff problems are prominent forms of waste that jump off of the page. The diagram below uses pie charts to highlight the relative time lost due to difficult handoffs during the product life cycle.

What are common reasons for difficult handoffs?

• Conversations, email, multitudinous wikis, spreadsheets, and trouble ticket systems are used to describe, in human language, how to process work. Words are open to interpretation and documents often lag behind current operating procedure. Just imagine being the person planning or performing the work and traversing the information across these various tools.
• Software product artifacts differ between stages of the process. Sometimes software resides in a directory on a file share and other times it’s a TAR file. The software handoff may contain the same bits, but must be handled or converted by the downstream stage of the software delivery process.
• Work can be considered “done” yet be unfinished or in a non-working state.The lack of a test or means to verify the work was done correctly often leads to products not ready for the next person down the line. This can leave the person in the downstream stage with what is essentially scrap that has to be rejected or redone.
• Ad hoc procedures or loose scripting often lead to different approaches and implementations for what should be standard operating procedure. This can lead to silo-specific utilities with different levels of quality and testing.

Handoff problems affect organizations, both big and small. Obviously, one answer to solving handoff problems is to minimize them. But if you are in an organization larger than just a handful of people, that just isn’t a realistic option. To decrease time to market and enable fast feedback, you are going to have to roll up your sleeves to solve the handoff problems.

Where are good places to start making handoffs smoother?

Here are a few of the top fixes that we find important for solving handoff problems at their source:

• Consistent packaging
  • The most direct way to simplify software handoffs between Dev and Ops is using a common system package format like RPM or Debian. Using a system package format also aligns application deployment and system provisioning practices.

• Encapsulated procedures
  • Rather than loose scripts or team-specific ones, choose a framework that enables modular automation. Using a modular approach results in a shared tool box of utilities and captured process.

• Converting information flows into artifact flows
  • Rather than rely on human read text as the product for the downstream process to handle, formalize it as an automation product and build on the idea of encapsulated procedures.

• Procedure verification tests
  • Verification testing should not be dominated by manual checks described in text documents. Building on the idea of converting information into artifacts, implement verification using a test automation framework. Most apps have some level of testing to verify functionality. Build a testing framework to verify an operation (eg, software deployment) procedure was successful by executing an automated test.

In subsequent posts, we’ll address each one of these fixes.

The post Improving Flow: Fix the Handoffs to Remove Your Worst Bottlenecks appeared first on dev2ops.

Lots of lip service has been paid to the importance of culture (“It all starts with culture”, “DevOps is a cultural and professional movement”, “Culture over tools”, etc..). But just as fast as everyone agrees that culture is supreme, the conversation turns straight to tools, tools, and more tools.

Recently, John Willis, my fellow dev2ops.org contributor and DevOps Cookbook co-author, let this tweet fly:

I am officially pulling the “C” out of CAMS… No one really gives a shit about it anyway.It’s always about the tools… #devopsIsDead

— botchagalupe (@botchagalupe) August 28, 2012

John has been as big of a culture warrior as anyone — constantly fighting to elevate the importance of and the discussion around DevOps Culture. He later said that this tweet was part exasperation and part challenge.

It was obvious to John that the difference between high performing and low performing companies was their DevOps culture, not the tools. But rather than be satisfied by the default explanation of DevOps Culture maturity being either that a company “gets it” or “doesn’t get it”, John was challenging the community to dive deeper into the issue.

During the week of Velocity London and DevOps Days Rome, there were finally some presentations that answered that call and were all about the culture. I did a presentation on defining DevOps Culture and what high performing companies do to reinforce it (based on the work of DTO Solutions). Michael Rembetsy and Patrick McDonnell gave a great peek behind the scenes of Etsy’s transformation to a company with a fast moving and high performing culture. Mark Burgess (CFengine) gave an interesting talk on the importance of, and science behind, relationships.

(slides were updated after the presentation)

(when you watch Mark’s video you will understand why there are no slides posted here!)

Update: John Willis knocks it out of the park talking about the importance of culture and the classic influence of Deming on this recent episode of the Food Fight Show.

The post Defining and Improving DevOps Culture (Videos) appeared first on dev2ops.

It’s no easy task to manage a DevOps transformation initiative. Not only are you addressing people, process, and tooling issues that span multiple silos, you are also balancing current business commitments, resource constraints, and good old fashioned human dynamics.

Where do you start? What are the traps you’ll need to avoid? What are the best practices to adopt? How do you work across roadblocks like functional silos and entrenched culture? And ultimately, how do you avoid this becoming another failed transformation project that could put you or your business at risk?

DTO Solutions repeatedly addresses these issues across it’s diverse client base. Either as coaches, architects, or change agents, stepping up to the responsibility for making DevOps transformations successful is at the heart of DTO’s business.

I don’t normally use this form to directly promote the services of DTO Solutions. But I believe that this is a special case as it is a unique offering that I’ve seen make a positive impact for our clients.

DTO Solution’s has taken it’s best practices and methodology for managing DevOps transformations and turned it into a 2-day workshop for managers. What was previously only available to client’s within broader engagements is now available as a stand alone public class.

You can read about the details and register here:
http://www.devopstransformation.com

The post DevOps Transformation Workshop for Technical Managers appeared first on dev2ops.

This inward emphasis is understandable considering that DevOps is a movement started by technologists and is a relatively easy sell to other technologists — breaking down silos, smoothing handoffs, baking in quality, rapid feedback, automating the hell out of everything — all messages that stand on their own merits.

But if we stop our DevOps evangelism at the boundaries of the IT organization, we are also missing the most valuable return on our DevOps investments. Why? Because the lessons and principles of DevOps can unlock something that is increasingly rare for today’s companies. DevOps can turn your IT Operations into a sustainable competitive advantage for your company. This is the DevOps message needs to be spread throughout the business end of your company, right up to the CEO and the Board of Directors.

To understand how DevOps can transform IT Operations into a competitive advantage, you first have to look at the current context within which businesses are operating.

Business reality #1: Technology alone cannot provide a competitive advantage

“Technology’s potential for differentiating one company from the pack – it’s strategic potential – inexorably declines as it becomes accessible and affordable to all”
Nicholas G. Carr
From ‘IT Doesn’t Matter’

The idea of technology being less and less of a competitive differentiator isn’t a new one. But by its very nature the big shift to online business has only hastened this decline in potential for differentiation.

Your customers are now coming to you via standard interfaces (browser or mobile app store) that connect to you via a standard protocol (http).

So by definition, your business has aligned itself behind a business channel where the point of transaction with your customers is on a level playing field with any of your competitors.

Business reality #2: Good ideas can and will be copied quickly

Recent high-profile product battles have reinforced the fact that any idea worth copying can and will be copied. Of course, copying ideas has always been a fact of business life. However, a distinct byproduct of our new online world is the increasingly rapid speed at which online ideas can be copied. Having the “better idea” just doesn’t give you much of a lead time. The barriers have become so low that some people are even basing entire business models on this idea (and I’m sure will soon see other people copying that idea as well).

So where can you find competitive advantage?

For the sake of this discussion, let’s put aside branding (“Enjoy Coca-Cola™”) and business model strategy (from viral effects to product bundling). These aren’t the focus of this blog and the long term impact for online businesses needs to be discounted because of the effects of business reality #2 described above.

What potential areas for competition are we left with?

1. Scale - removing any limits of data, users, transactions, etc. on the business
2. Lowest Cost - being viewed by your customers as the lowest cost option in the marketplace
3. Sustained Innovation - continuous pace of finding new ways to please customers and improve product market fit

I think the choice might be obvious by now, but let’s look at each one of these individually

1. Scale

Scalability, while essential, is not an area in which you can gain a strategic competitive advantage. Scaling of online services is essentially a solved problem. With enough money and enough smart folks (yes, there is a definite art to it) you can scale just about anything. And if you can do it, your competitors can do it too. Cross this off the list.

2. Lowest Cost

Being the low cost provider can look seductively attractive from the perspective of winning an individual sales deal. But when you consider the broader ramifications, life as the low cost provider starts to look a lot more difficult.

First, there can only be one “lowest cost” provider. This automatically sets you up for a high-stakes, winner takes all scenario. Second, hanging your hat on being the lowest cost provider can quickly turn into a race to the bottom, especially in a digital market where the marginal cost of each unit of your product/services is very low. At the end of the day you can’t control how low your competitors will go and ultimately you can’t get lower than free. Lowest cost might be a valid part of your overall business strategy, but it’s limited in its ability to provide a lasting competitive differentiation. Cross this off the list

3. Sustained Innovation

That leaves you with sustained innovation. Unfortunately “sustained innovation” also sounds the most difficult, doesn’t it? Consider the “innovation” part first — do you really have what it takes to be better than your competitors at finding new ways to improve product market fit and delight your customers? Then factor in the “sustained” part — do you really have what it takes to do that continuously?

But difficult or not, it’s what you are left with. So let’s start breaking it down and figuring out how to turn it into a core competency.

What is innovation?

“Innovation is the specific instrument of entrepreneurship. The act that endows resources with a new capacity to create wealth.”
                                        – Peter Drucker

There are lots of competing ideas on what exactly innovation is or isn’t. Some say the successful application of any new idea counts. Some say it has to be disruptive. Some conflate invention and innovation while others draw a clear distinction between the idea (invention) and innovation (the application).

But not matter which flavor of definition you subscribe to, there is a basic unit of activity supporting innovation. In the context of an online business, that basic unit of innovation activity is taking a business idea and transforming it into a feature that is providing the business feedback in a customer facing environment.

Competition based on innovation all comes down to this unit, or lifecycle. If you can get to a successful result quicker and more reliably than your competitors, a competitive advantage has been achieved.

Innovation is really a numbers game

Of course, everyone wants to believe that the result they will see each time they go through the innovation cycle is unqualified success. But in reality, the outcome is rarely so rosy. We envisage adoring praise from customers and piles of cash, but that rarely materializes.

How bad is it really? The Doblin Group did a global study and found out that only 4% of all innovation efforts meet or exceed targets for return on investment. Booz Allen found “no significant statistical relationships between R&D spending and the primary measures of financial or corporate success”. Even from within “innovation factories” like venture capital portfolios you hear numbers like 80% of companies either exit with minimal returns for investors or fail outright.

So despite our hardwired bias to think otherwise, the vast majority of things we try are just not going to work. From a macro perspective, innovation is a numbers game and the numbers are stacked heavily against us.

To gain an advantage you need to change the numbers in your favor

It’s easy to be mesmerized by the hero’s myth of innovation. This is where the hero musters up everything they can for that one spectacular shot at success. When looking to recreate this story, that one spectacular shot always appears to be a sure thing when it’s in mind of a wannabe hero. In reality, that one spectacular shot is really a high stakes gamble that rarely pays out.

Conventional wisdom on how to succeed at innovation is usually along the lines of “think harder and better”. Hire smarter people. Hold more meetings. Get more input. Hold your ideas close to the vest and keep tinkering with it for as long as you can while building up for a big reveal to the marketplace.

But even if you can hire the smartest people in the industry and ensure that you are always working from perfect and timely information, “think harder and better” approach still has you involved in a high stakes game where the odds are still stacked against you.

Rather than betting it all on those few high stakes gambles, there is a better — albeit less glamorous –way to win when competing in a numbers game. Give yourself more attempts at finding the right product market fit than your competitors can. Using a sports metaphor, take as many “shots on goal” as you can before the clock runs out.

Keep in mind that these shouldn’t be blind shots. You learn a little from each attempt and move quickly to get the next shot off. In the end you’ve tipped the raw odds in your favor and (hopefully) found success before time ran out on the market opportunity (or in a startup’s case, before the cash runs out).

In the example above, we have Company A employing the classic high stakes strategy and Company B playing the increased shots on goal strategy. Company B has gained the ability to safely move through it’s lifecycle 4 times faster than Company A. This gives Company B a distinct advantage. Assuming all other things are equal (same people to hire, same cash to spend, etc.) this means that Company B has 4 times the number of opportunities to find the right product market fit. Increase that number to 8, 16, 32, or more and you can see how this turns into a significant advantage.

The business community already recognizes the opportunity

The idea of finding competitive advantage through continuous innovation is already anestablished idea in the business community. The Lean Startup movement is built on this concept and makes the case quite eloquently.

“The amount of time a company can count on holding on to market leadership to exploit its earlier innovations is shrinking, and this creates an imperative for even the most entrenched companies to invest in innovation. In fact, I believe a company’s only sustainable path to long-term economic growth is to build an “innovation factory” that uses Lean Startup techniques to create disruptive innovations on a continuous basis.”
 -Eric Reis
The Lean Startup

It’s not just the Lean Startup niche that has gotten onboard with this idea. Across the business landscape you’ll find this message hitting home.

“The world is changing very fast. Big will not beat small anymore. It will be the fast beating the slow.”
                             -Rupert Murdoch
Chairman, News Corp

“An organization’s ability to learn, and translate that learning into action rapidly, is the ultimate competitive advantage.”
-Jack Welsh
Former CEO, General Electric

“We are already at the stage where almost all of the top managers are very well aware that if they do not lead their company on a process of ongoing improvement it is just a matter of time until you don’t have a company. This was not the case 20 years ago. I still remember very much that I had to fight 20 years ago on this concept. Now it’s common. Everybody knows it.”
                             -Eliyahu M. Goldratt
from ‘Beyond The Goal’ Lecture Series (2005)

While the desire from business leaders to innovate quicker has become clear, what hasn’t been as clear is how IT, now that it’s the new “factory floor”, will provide the capabilities necessary to enable this rapid learning that leads to rapid innovation.

This is where DevOps comes in.

DevOps is an essential tool for turning IT into a competitive advantage 

At it’s most fundamental level, DevOps is about breaking down silos and removing bottlenecks and risks that screw up an organization’s Development to Operations delivery lifecycle. The goal is to enable change to flow quickly and reliably from specification through to running features in a customer-facing environment.

It’s no coincidence that the common illustration for this Development to Operations lifecycle looks a lot like the innovation lifecycle that was previously discussed. By using DevOps ideas to remove the bottlenecks, inefficiencies, and risks from the Development to Operations lifecycle you are directly improving the speed and quality of the innovation lifecycle.

If the mandate of your business is to find competitive advantage through sustained innovation, IT’s mandate must be to provide the business with a fast moving, low friction, high quality lifecycle that enables that innovation advantage. How will you know when you have it moving fast enough? When the Development to Operations lifecycle is no longer the rate limiting step of the company.

Your goal must be to get to the point where you can tell the business “as fast you can tell us what you want and developers can code it, the changes will be live in a customer facing environment giving you direct feedback. And we can can do this with better quality and reliability than you are getting today.”

When you’ve reached that point with your operational capabilities, you will have successfully turned your IT operations into a strategic weapon that gives your company a sustainable advantage over it’s competitors.

One final note… Ignore this call to action at your own peril. Like all competitive advantages, once it has been discovered by the market you can either use it against your competitors or it will be used against you.

The post Use DevOps to Turn IT into a Strategic Weapon appeared first on dev2ops.