Data Center Automation

BMC currently offers two levels of support, Continuous and Premier. Continuous offers business hours support for severity 2, 3, and 4 and a 1 hour SLA 24 hours/day, 7 days per week for severity 1 issues. Premier offers a 1 hour business SLA for severity 2, 3, and 4 and a 1 hour SLA 24 hours/day, 7 days per week for severity 1 issues. But the difference is not just in the SLA and that is what I want to discuss.

When you purchase Continuous Support, your cases are handled by our global support team, and are routed to the location whose team who is working your business hours.  So, to put it bluntly, your issue will land in a pool of engineers and one of them will take your issue and follow it through to completion. There are so many times that I am in an escalation with one of our customers where they say to me, “I only want Jimmy to answer my issues, I get him a lot and I respect his technical ability.  Why don’t you know my environment? Everyone at BMC should know what we have installed and how it is implemented. Why don’t you know my use cases?”

Well, I know why I made this choice. For me, It is because of the level of attention I will receive when things do not go right for me. For the most part, this is why you purchase maintenance, to ensure BMC will be there when things go wrong.  If you buy the Ford, and your car breaks down, you will bring it to the dealer and your spouse might have to drive along with you so you can make it home. Then, you will have to decide who can work from home the next few days since your car will be out of service. However, if you buy the BMW, in my case, the dealer comes to your house drops off a free rental car and drives the car to the dealer and drops it back off to your house when it is fixed. (I have a sweet deal going, I know!)

Here is my situation. I have an expectation that this is what happens when my car is broken. I believe that no amount of negotiating at the Ford dealer would get me service like that. So, yes, I chose to pay more for my car to ensure I will receive the service I expect which I now rely on. I am effectively making my purchasing decision based on the support I will get. I am in effect chosing Premier Support.

Another great reason to purchase premier support is for the proactive nature of the offering; again, parallel to my choice to purchase a BMW. When my car needs its yearly service and oil change, etc. I don’t need to call for an appointment, they call me. With premier support you will have an engineer who is focused on your environment, who can recommend service packs, when he feels you will benefit from the fixes in that pack. He (or she of course!) can discuss the pros and cons of the install and help you plan for the implementation. This is something you would not get from Continuous support, because as much as you would want this, of course you see the benefit, it is just not part of the offering. Just as with the Ford and the BMW, you both have access to the service center as all maintenance customers have access to support and related service packs, however, with Continuous, you will have to call BMC, explain your environment and discuss whether or not the service pack will benefit you. With Premier, the conversation is more about when you should implement the service pack and how to plan for that.

My conclusion is, Premier Support is not for everyone. However, if you find yourself in a position where you want a single point of contact to understand your environment and be an expert in the solutions you own and always looking out for your best interest you should consider purchasing Premier Support. If you equate your time into dollars, I’m sure you could convince your management that it is money well spent.

I will end with a shameless advertisement. If you are interested in purchasing Premier Support or for more information on Premier Support, please contact Mary Morgan at mary_morgan@bmc.com.

Often one of the first questions a customer asks me about our application suite is "We are a Linux/Solaris/Windows shop, will your applicaitons run on that?".  To which I want to respond "Who cares?". Of course I don't and I tell them yes we support X.

Sadly we still have to ask this question because there are applications that will only run on certain OSes.  But most applications, and most business applications will run in an application server that is cross platform (eg .NET, J2EE), so the OS really doesn't matter.  And those that don't are usually ported to support multple operating systems.  So we can argue about the merits of Windows over Solaris over Linux but at the end of the day, unless you are doing something very specific that only one OS supports, you could provide your application from almost any operating system.  And not even a full operating system - Microsoft has it's 'server core' version of Windows and Linux vendors have their 'Just Enough OS' or 'Appliance OS' variants that provide enough operating system to run your application but none of the services you'll never use.

The OS is just another part of the infrastructure service you provide.  The OS now is the application server stack - J2EE, .NET, PHP.  It's trivial to manage the OS with an automation tool (eg BBSA) to push configs, security lockdowns and patches and that is even simpler with the JeOSes.  Until the OS becomes something akin to firmware that needs infrequent updates all of those mundane activities must be done.

To manage that new OS stack (the application server) there are similar management tools to manage configs, security and patches - such as BARA (for application servers) and BBDA (for databases).  These tools let you achieve the same automation as BBSA lets you achieve for servers.  You can also use these tools to push our your applicaitons and code updates.

To really get to this level of nirvana there's likely a lot of policy changes that need to take place. 

• The one-off servers must be gone. 
• Letting developers decide to stick with Java 1.ReallyOld.Version isn't acceptable. 
• You need to create classes of servers and cluster them together. 
• Standardize your builds. 
• No one gets access to the OS except the infrastructure team and then only when the automation tools fail.
• No one gets access to the application server layer except through the automation tools and perhaps any application-specific management tools needed.
• The personal relationships admins seem to develop with the servers they manage need to be dissolved.  The server is a throw-away, replaceable, rebuildable entity.  Make your servers names read like a bunch of UUIDs instead of 'Bart' and 'Homer' to re-enforce that.

You are building a 'cloud' of .NET or J2EE service.  It's a 'private cloud' or 'utility computing'.  Call it what you want.  You're pushing applications into the application 'cloud' you've just created and charging for use of the resources (whether internal chargebacks or otherwise).  When an OS/application instance fails there should be capacity already there to take over while the hardware or VM is replaced and added back into the pool. 

Cloud right now seems to be about throwing out a bunch of bare OS instances.  While that's all well and good and there are definately still challenges to resolve there, the fun part is going to be doing to the appserver what we've done to the OS.  For the most part it's a step beyond what is currently out there because we still think about the OS as the most granular and measurable unit.

In my first post, we talked about the first layer of the automation stack. We discussed how Embedded Automation (e.g Server or Database Automation), provides the domain knowledge to accurately provision and manage the devices and applications in the data center. Enterprise Job Scheduling provides the business-driven automation for scheduling tasks and tackling massive workloads like invoice processing or an end-to-end supply chain process. These are the building blocks upon which successful automation platforms are built.

As we move into the second layer of the automation stack, we find the integrated process automation that is the focus of BMC's Business Service Management (BSM) strategy. At the bottom we find the beginnings of BSM with the IT Service Management (ITSM) Process Automation. It was really the need to implement truly automated IT Infrastructure Library (ITIL) processes like Change Management or Incident Management that drove the first real people-to-people process automation in IT. The real power BMC's Remedy ITSM platform is not just the end-to-end processes, but also the process-to-process integration through the Configuration Management Database/System (CMDB/CMS). Incident Management is a much more powerful process if the ticket is automatically updated with the latest changes to the relevant assets and recent problems encountered. It was the disconnect of ITSM process automation from the managed assets that drove progress IT Orchestration.

IT Orchestration combines two automation disciplines that are often confused with each; Run Book Automation (RBA) and IT Operations Process Automation (both represented by BMC Atrium Orchestrator). While they go by many different names, the distinction is relatively easy to make. The original mission of most RBA tools was to take the almost simple, but highly repetitive, tasks that service desk engineers would spend so much time on and automate them. These tasks were often outline, step-by-tedious-step, in a Run Book. And example of RBA would simply pinging any server tied to a new incident ticket, and putting the results in the ticket. Another example would be constructing an end-of-shift report for a service desk manager by extracting incident and problem status information and summarizing it. The bottom line was that RBA tools needed to be highly flexible and also be able to connect easily with different IT tools.

Once RBA started interacting with ITSM process automation, IT Operations Process Automation was born. It was one thing to automatically generate incident tickets from a failure event. RBA was already adept at integrating with ITSM and acting on the behalf of ticket generators like event management systems. It is quite another to generate a change ticket for the fix and then request the embedded automation tool to actually execute the change. This kind of integration was an inevitable demand out of the more mature ITIL process automation domain. The achilles heel of any change management system was always the ability to actually verify that changes were made according to change policies, and what the change request specified was truly enacted. Now I could clearly associate a change with the actual action taken, and even roll-back the change, if necessary. This also empowers the Service Catalog to be clearly attached to a real automated process, like provisioning a server or a database. This is the genesis of Cloud Lifecycle Management, but we will get to that in another blog post.

The converse, and often overlooked, problem was ensuring that all changes were properly documented. Now the RBA tools really came into their own. A server administrator can now automatically create a Request for Change (RFC) from BMC Server Automation, and follow change policies, all without spending hours writing up the steps manual in Remedy Change Management. The perfect example of this is Closed-Loop compliance. When my embedded automation tools scans the system against a policy, and finds an issue, it can automatically attach the remediation for the security finding to a change request.

So, we come to the end of another walk through automation. I would appreciate your thoughts are your comments. How have you seen Orchestration used, or used it yourself? Where do you see the future of orchestration?

Next time we explore Dynamic Workload Management...

When it comes to the best way for provisioning, there have always been two schools of thoughts:  image based approach and scripted approach.  Both camps tout the various advantages of their method while citing shortcomings of the opposing school. 

The truth is they are both right.  Image based approach and scripted approach are like the Ying and Yang of provisioning.  What is the weakness for one is the strength for the other.

Imaged based provisioning offers:

• Speed – cloning an image is usually several times faster than a scripted install
• Portability – images can be easily transferred as a single entity between different locations and organizations
• Simplicity – cloning an image is much easier set up and execute than building out complex logic of a scripted install

Scripted provisioning offers:

• Flexibility –  scripts can contain sophisticated logic and can accommodate the level of customization required to build out complex applications necessary to achieve full stack provisioning
• Heterogeneity – scripts can be made to operate on any platform and hypervisor flavor, so it can support large mixed environments
• Manageability – scripts are light-weight and can take different inputs to achieve various desired results, there is no risk of image sprawl associated with having to keep an image for each variation required

Since neither approach is perfect and both offer situational advantages, the contest between them has no clear winner.  Users often have to choose between one and the other.

Wouldn’t it be nice to be able to have your cake and eat it too?

BMC says: “YES!”

With the layered approach provided by BMC’s Cloud Lifecycle Management solution, both imaged based provisioning and its scripted counterpart are used together to offer users the advantages of each while removing their weaknesses.  For example:  if the requirement is to simply build a group of virtual machines with standard operation systems to create a basic computing environment, cloning would be used for its speed and simplicity.  However, if bare-metal servers need to be built as hosts to the virtualization platform, a fully scripted install would provide the flexibility required.  Finally, for full stack provisioning, the combination of imaging and scripted install can be used to clone the base OS quickly and layer on the application stack on top using scripts and packages.

Having the option to choose one approach versus the other, or a combination of both, allows the users to enjoy all the benefits while not having to suffer any of the shortcomings. 

Life is good when you can have your cake and eat it too!

Blah…blah…blah…  everyone is telling you about the benefits of automation to your company, but automation has a personal impact on you as well.  IT must evolve to in order to improve services and to meet business demands.  Automation is a part of that evolution, but data centers do not run on their own.  You are still needed to how automation will be deployed and what automation will do in the delivery of services to your customers.

We are all familiar with mechanical robots that do repetitive physical tasks over and over under programmatic control.  These robots do physical tasks far faster with greater precision and fewer errors than we can accomplish on our own.  Notice these are tasks that are performed repeatedly.  The tasks must be well known and measurable – standardized in a way that does not require craftsmanship in every task to determine how to successfully complete the task every time under varying conditions.  If you were ever in a position to observe the first generation of robots being applied the first time to a new set of tasks, you would see the robots mimicking the movements and order of work previously done manually.  For example, some of the first printers were typewriters with keys actuated by electronic codes – in a way, more like early display terminals.  As these robots matured, creative new ways of accomplishing the work are defined – a new breed of craftsmanship.  For robots to successfully meet business demand, someone must determine what a robot can do, what the robot should do and then tell the robot how to accomplish the tasks.

Using robots to automate physical tasks certainly has personal impacts on all the people in the engineering and operations chain of production.   I am sure you have seen, heard or even know the repercussions robots have had on manufacturing industries such as automobiles and electronic equipment.  As companies employed robots to remain competitive in tight markets, to survive and even thrive, much of the craftsmanship moved from the mundane, simple, repetitive tasks to assuring robots are set up successfully to accomplish these tasks faster and more consistently. 

Many people who applied their craftsmanship in production have added to their skills to apply their craftsmanship to putting products into production.  Sometimes this transition is not obvious, but none the less crucial to the success of robotic task automation.  This transition is as essential evolution of the craftsmans’ guilds with the progression of technology – taking advantage of the current generation of craftsmanship to assure success of the transition and preparing for the entry of the next generation of craftsmen.  Craftsmanship still survives in production to do the tasks that robots cannot address adequately – complexities or variations outside the scope of robot capabilities – serving as a step in the learning curve of apprentices learning the craft and ultimately becoming qualified to instruct robots on how to execute new tasks.

There are many parallels in other industries where process control automation was applied for greater precision and speed than humans could accomplish with similar results and impacts.  Continuous chemical processing plants are one example.  The telephone network is another not so obvious example.  Starting with relay logic, the actions of operators connecting customer telephones was automated, but still physically comprehensible through the visible wires and relay contacts.  When computers were applied to telephone switching, the same logic was there, but the user visibility for changes and diagnosing troubles was different.  While many technicians worried that their careers were over, their knowledge was still applicable and valuable to telephone companies.  The telephone companies got business growth and flexibility to meet demand changes in an instant that were not previously possible.  The telephone companies still needed all the technicians – it just took some time for everyone to adjust to the changes required for success of the transition.

It was no large step from industrial process automation to business process automation and IT has been doing this well for decades.  IT customers and business managers have become accustomed to the performance of automated processes and expect the same from IT processes.  Data centers must go through the automation transition to support business and IT success, including dealing with the personal impacts of the necessary changes.  Managers making the decisions regarding deployment of automation within a data center need the skills of system administrators and operators to achieve effective automation of tasks and processes.  These system administrators and operators need to transition from accomplishing tasks manually in production to defining how the tasks should be accomplished by automation, identifying best practices and tasks, like documenting changes, that do not get done lack of time or distraction from acting on the backlog of requests for changes or resolving issues on the systems themselves.  A career path to Automation Specialist for system administrators and operators has proven to ease the personal hurdles of many IT departments transitioning to data center automation.  Managers can set clear guidelines and system administrators and operations get tangible visibility to the requirements for advancement.

What has been your experience?  What hurdles have you encountered in the transition to data center automation to whatever extent you have accomplished?  Bring your comments to the conversation and share your successes in overcoming hurdles with the community.