Diving Deep into Sublime Seas

Shadow IT – if you can’t stop ‘em, join ‘em

2013-05-08T14:22:00.001-06:00

There is a silent revolution brewing in the enterprise, and it is not cloud computing any more. Cloud computing, while evolving from blue-sky thinking to mainstream phenom in a few years, has resulted in several ripple effects across IT.

One that is particularly disturbing for IT managers is Shadow IT, or the introduction of technology within a company from the bottom-up. This refers to situations where new products and solutions are not brought in via a centralized evaluation and purchase process with oversight from the powers that be, but from the people in the trenches.

And not necessarily from select people in the trenches – the enterprise architects or software engineers who can perhaps be relied on to conduct their due diligence prior to acquiring new technology, but anyone in the company, regardless of their title and pay-scale can be the initiator.

One doesn’t have to look far to understand way. Business moves fast, and end-users want to keep up. Leveraging many of the SaaS-based tools and products out there allows them to do exactly that. Astute IT leaders are caught between a rock and a hard place - they don’t want IT being viewed as the anchor that slows innovation and growth, but they also don't want to see business being hurt on their watch due to lapses in security, compliance or performance.

As one starts digging deeper, the nature of this problem becomes more apparent. Products traditionally deployed within enterprises have been vetted by IT and set up for proper security and manageability, but they can have gaps and tend to be harder to use. A few years ago, employees had no choice but to reach for the nearest set of manuals, call Help Desk and log a ticket, or spend hours and days attending webinars and training sessions before they were able to use those products. Now, the prevailing feeling is “if a product requires me to read a manual, it’s friggin broken!”

Fueling that feeling, there's an ecosystem comprising hundreds of vendors offering various tools and solution alternatives in the cloud. Individual, group and department-level users are tempted to sign up, evaluate and/or use these software without IT's involvement. These are tools that are sometimes free, or may cost a few hundred dollars per month and easily fit into the budget of many managers. At times, even non-managerial employees happily sign up for some of these services using their personal credit card, given the immense productivity gains and ease-of-use they bring to the table.

By the time IT becomes aware, that software may already be well entrenched in the organization regardless of its quirks (or ahem... security holes) that IT is now forced to deal with.

These scenarios have placed IT in the crosshairs to take control of the situation, scan workstation logs and network ports for rogue usage, and enforce requisite standards for user privacy, data confidentiality, scalability and manageability. But punitive measures don’t exactly endear IT to their business counterparts - a recent survey by InformationWeek finds less than half of non-IT personnel regard IT to be integral to their business. Nor are such measures 100% effective.

The reality is that technology purchasing is undergoing unprecedented decentralization and that is here to stay. Attempting to control or put an outright stop to these practices is akin to swimming upstream and will eventually fail. Leading analyst firms estimate as much as 35% of enterprise technology spend will fall outside the IT department's budget within the next two years.

So what is a CIO to do?

I recently watched an Andreessen Horowitz video The Renaissance of Enterprise Computing where Gary Reiner, former CIO of General Electric says “if I was a CIO now, my job would be much more about procuring solutions than managing people.”

Gary’s statement points to the elephant in the room. IT has to participate in the process and guide it, rather than attempting to confront and control it.

But that may seem easier said than done. IT is its own victim due to the segregated collaboration tools and linear approval mechanisms such as ticketing systems and intranet portals implemented in the past. These intrinsically suffer from excessive communication latency and workflows that are hard-pressed to meet the urgency associated with seemingly mundane technology requests from business. End users are no longer content to send an email or log a ticket and wait for IT to act on it. They instead turn to the latest SaaS provider their spouse or neighbor was talking about, and are off and running in minutes and hours rather than days and weeks.

Newly launched SelectHub is designed to address this very issue. SelectHub enables both end-users and IT, along with other key stakeholders (Procurement, Finance, Enterprise Architecture and budget owners/managers) to be in lockstep throughout the evaluation process. Any of the above stakeholders can initiate an evaluation project within SelectHub with just a few clicks, invite other key personnel, assign them specific roles and each party can proceed to specify the scope, requirements and success criteria that matter most to them as well as engage in peer-level conversations regarding each area’s necessity and impact. Project participants would have visibility to all or part of the project, based on their roles.

Regardless of whether a software project conforms to a centralized top-down purchasing model, or occurs at a grassroots level, SelectHub serves as the tactical link between IT and business to accelerate evaluation and rollout of the right solution while enhancing IT participation and governance. Instead of treating each other as policy violators or innovation stoppers, both parties can pool their expertise and achieve measurable progress while eliminating altogether the fragmented communication and back-and-forth delays associated with emails and tickets.

Shadow IT inevitably results in loss of control and compromised security. With SelectHub, IT can take affirmative action by initiating or participating in, and influencing the evaluation process to orchestrate a reasonably satisfactory outcome for all stakeholders.

Introducing SelectHub

2013-05-06T10:49:00.000-06:00

I wanted to take a few minutes to share an exciting new venture with you: SelectHub.

SelectHub is a platform for companies to evaluate and procure software products in a collaborative manner. It helps users become aware of solution options, access research that may not be readily available in the public domain and conduct their evaluation in a streamlined, inclusive and thorough manner. They can choose to communicate with vendors and coordinate RFIs and proof-of-concepts via the SelectHub platform sharing as much or as little of the project details as they choose. When they do, it helps vendors understand buyer expectations more clearly, cater to inbound queries and requests in near real-time, and advance the solution sale process based on precise metrics for opportunity qualification and revenue predictability. SelectHub enables maximum efficiency on both sides.

The offering is a culmination of my years of frustration navigating complex and time-consuming software evaluation, purchase and implementation processes at various organizations, and working with other IT leaders and end-users that shared that frustration to craft a solution.

Early user feedback about SelectHub has been overwhelmingly positive! My team and I are grateful for the support we have received, and trust it will pave the way for SelectHub to become the gold standard in process management for enterprise technology purchases.

You can check it out at www.selecthub.com. There’s a one minute video that introduces the offering, and the product rationale is further explained at www.selecthub.com/why.

If you think SelectHub could be beneficial to any of your colleagues, please take a moment to share this video with them. Or if you are participating in a product evaluation yourself, sign up at SelectHub.com, create a project and add/invite other stakeholders.

SelectHub is free to use. The only thing we ask for in return is feedback about your experience. That will help our team tremendously as we continue to improve the service in both functionality and form.

Thanks for reading. I look forward to seeing you on SelectHub.com.

Preparing the IT Organization for the Internal Cloud

2010-07-05T13:24:00.000-06:00

A recent article on Cloud Computing quoting IBM’s Ric Telford caught my attention. He boldly predicts that the term Cloud Computing will become obsolete over the next 5 years while the underlying methodology will become standard practice for IT. While this level of IT maturation truly seems a few years out, I find internal or private clouds to be increasingly prevalent in certain pockets of industries such as financial services, energy and managed services. On the other hand, IT departments in many other industries such as manufacturing, retail and telecom continue to struggle to enable such capabilities. The struggles have not emanated necessarily from lack of desire, budget or technology leadership, but from cultural challenges such as the following:

(a) Perceived shift in power dynamics. Internal cloud and self-service delivery models make it unclear regarding where does traditional IT Operations stop and end-users (service consumers) start driving the delivery process. Many IT Operations personnel fear that end-users will need to be granted administrator privileges to enable self-service and it will be difficult to control end-user activity. End-users will not adhere to well-defined IT standards and the requisite training and oversight will dramatically increase Operations’ workload.

(b) The lines between IT silos become blurry. When an end-user group asks for an IT service, those services are traditionally broken down into specific requests in the form of tickets and assigned to specific silos such as Sys Admin, Storage Admin, DBAs, etc. The requests are then individually delivered, often with lots of coordination back and forth with the final service being received and validated by the end-user team. While a cloud-based delivery model aims to improve such out-dated modes of functioning, it is also susceptible to severe internal bottlenecks and confusion regarding what is the right unit of delivery and who needs to own it: the application admins at the top of the tier, or the sys admins at the bottom. On the other hand, average service recipients do not care about the individual silos that make up their request; they speak in the "language" of the application whereas traditional IT Ops speaks in the language of servers, storage, database and other infrastructure elements. Thus the lack of a common denominator for requesting and delivering self-service capabilities in many organizations arrests momentum pertaining to the deployment of internal cloud capabilities.

In other words, the true barrier to enabling a private cloud is not technology, but cultural apprehensions that old world IT bridges by throwing more bodies and silos at the problem – repetitive meetings and water cooler conversations within different IT personnel and application teams provide a framework for delivery. It’s a lot of wasted effort, but the old world process delivers the services (albeit it takes days, weeks or months!)

CIOs that are ultimately successful in having IT groups embrace internal cloud and self-service models solve these bottlenecks in a way that’s amenable to the overall organization culture. Some CIOs do it via baby steps by retaining existing IT structure and policies and offering specific infrastructure components such as servers and databases in self-service mode to project teams. In other words, they attempt to transform IT by offering self-service capabilities on a silo by silo basis. However that is akin to a restaurant delivering specific pre-cooked ingredients to customers and expecting them to compile those ingredients into the requisite meal. Obviously such “self-service” (if it can still be labeled that) IT delivery models require the end-user to be fairly sophisticated in asking for the right resources and being able to assemble them all together themselves. This also assumes that the end-user is disciplined enough to ask for these ingredients in the right quantity (i.e., they are not wasteful or overly cautious). These assumptions rarely hold true causing the success of such internal cloud strategies to be stunted at best.

A better approach might be to take the time upfront to define a common denominator (across IT silos) and corresponding units for self-service – via a macro-level reference architecture. Take any candidate end-user request or task that can be delivered in self-service mode: e.g., application provisioning, code releases, creating data copies or backups, restoring snapshots, creating users, resetting passwords, resolving incidents, etc. All of these require a universal definition of what’s to be delivered to the end-user. All underlying nuances such as IT silos, their lines of control and associated complexities need to be abstracted from the end-user. The more comprehensively IT is hidden behind the curtains, the better is the private cloud implementation.

Based on my work with multiple IT organizations across different industries, I find that the most pragmatic unit for self-service comes from the top-most layer of the IT stack – the application. However utilizing application-specific units is easier said than done since in most large organizations (the kind that can really benefit from a private cloud), IT operations doesn’t speak the forementioned “application language”. Even if they do, there are several applications in use and most applications have multiple parts and pieces (i.e., they are all “composite applications”). In order to make application-specific units more palatable to all IT silos, the underlying descriptors of the units have to reference physical infrastructure attributes and map it to their corresponding logical (app-specific) definitions within the reference architecture.

Most applications that have been developed over the last decade are 3-tiered. They have a web server tier, an application server tier and a database tier, along with associated sub-components such as a load-balancer tied to a farm of web servers. Hence this popular 3-tier “application template" acts as a robust common denominator for IT. By building self-service capabilities around such a template, most applications within the enterprise are addressed. Of course, there are certain applications that employ just one or two tiers: e.g., server-based batch applications, client/server reporting applications and so on. As long as the tiers used by these disparate applications are a sub-set of the chosen application template, they are adequately covered. On the same lines, some applications employ a use of unique middleware components such as a transaction processing monitor, a message bus, queues, SMTP services, etc. Obviously such applications would fall outside the 3-tier template and would need to be dealt with separately. However with a bit of deeper investigation, I usually find that such applications are vastly outnumbered by the standard application stack that employs some or all of the 3 tiers named above. Hence such exceptions do not make an appreciable difference (in most cases) to the notion of using a single master application template in the reference architecture.

By applying the 80-20 rule, one can keep the application template relatively straightforward in the initial iterations of the private cloud implementation and focus on those applications and tasks to be delivered in self-service mode that will have the biggest impact. I have seen this approach work well even in larger organizations with multiple customers or lines of businesses (LOBs) because the underlying definition (of the makeup) of the application remains consistent across these disparate LOBs.

Once the master application template is defined, each unique application type can be assigned a profile that describes the tiers it employs. For instance, AppProfile1 (or simply, Profile1) can refer to applications that always have a web server, application server and a database instance, whereas Profile2 can refer to applications that utilize a 2-tier model (just an app server and a database), and so on. Using concepts of polymorphism, all these profiles should refer to the master application template such that any changes to the base templates will be immediately reflected across all profiles. During future iterations, additional templates and profiles can be defined to refer to applications that may have additional tiers such as a message bus. These definitions can be laid out on paper (and later implemented using an automation tool), or ideally, set up within an automation platform such as Stratavia’s Data Palette from the very beginning.

Regardless of the application profiles (the fewer the better in the initial phase), the deployment needs to adhere to one key point: keep the individual components for each tier hidden from the end-user while giving IT teams full control over those components. By “components”, I’m referring to the following:

(i) infrastructure layers: i.e., servers (virtual or physical), storage (SAN, NAS, etc., depending on IT standards and budget sizes), networking capacity, etc.; and

(ii) software layers: the web server, application server and database instance. Compared to physical infrastructure like servers and storage, each of these software layers often encompass a “best of breed” configuration across a broader variety of vendors, proprietary platforms and open source frameworks (such as Apache Tomcat, Microsoft IIS, WebLogic, Websphere, Oracle DBMS, SQL Server, SAP Sybase, VMware Springsource, etc.), and hence tend to be much more complex and time consuming for provisioning, configuration and ongoing management. Hence the overall success of the deployment calls for limiting the number of application templates and profiles during its initial phase.

Each application profile should have a description of what quantities of which specific components are needed by each tier - for initial provisioning as well as subsequent (incremental) provisioning. For instance, a ‘web server’ tier in the Sales Force Automation (SFA) suite may comprise a virtual machine with X amount of CPU, Y amount of disk space and a standard network configuration running Apache Tomcat. When an authorized end-user (e.g., a QA Manager) asks for the SFA application to be delivered to a new QA environment, the above web server unit gets delivered along with the corresponding application server and database instance – all of them pre-configured in the right units and ready for the end-user to access! However when an authorized end-user asks for more web servers to boost performance in the production SFA environment, what gets delivered is N units of only the web server tier, with specific pre-defined configuration to tie it to the corresponding (previously existing) application server and/or database. All of these mappings may be held in a CMDB (ideally!) or in some kind of operations management database and referenced at run-time by the automation workflows that are responsible for service delivery.

The application templates and profiles with quantitative resource descriptions allow end users to receive services in an application-friendly language, while allowing IT Ops to provision and control individual pieces of server, network, disk space, database, etc., using their existing IT standards and vocabulary. All underlying components blend together into a single pre-defined unit, enabling the well-oiled execution of a self-service delivery model within the private cloud implementation.

Using application provisioning as an example, the below graphic (click on it for an enlarged version) depicts how various application profiles comprising multiple software and infrastructure layers can be defined.

Regarding the other previously stated bottleneck (i.e., the perceived shift in power dynamics), while IT Ops personnel are behind the curtain, it doesn’t mean they are any less influential. Using automation products such as Data Palette, they are able to exert control both in the initial stages of service definition (along with the Operations Engineering or Architecture groups), as well as during the ongoing service delivery process by monitoring and tuning the underlying automation workflows (again, in conjunction with Operations Engineering) as depicted in the graphic below. In other words, they define and control what automation gets deployed and what is the exact sequence of steps for delivering a particular component of an application in the environments they manage. Specific product capabilities such as multi-tenancy, granular privileges and role based access control allows them to grant end-users the ability to see, request and perform specific activities in self-service mode without the need to also grant any administrative privileges they could potentially misuse.

As the IT Ops personnel continually assess areas of inefficiency, they can refine the workflows and roll out improvements with little to no impact to end users. Such efforts, while changing the job description of what IT Ops does on a day-to-day basis, help to deliver a better end-user experience helping companies leapfrog from the IT of yesteryears to the “industrialization of IT” that IBM’s Telford refers to.

What’s the “Right Way” to Automate the Application Stack?

2010-02-23T13:10:00.000-07:00

Cloud computing is forcing IT organizations to rethink automation. Early adopters started out with the delivery of servers, storage and network connectivity in self-service mode via an internal cloud. While this helped reduce service delays, real service level improvements weren’t forthcoming. Application owners and end users were still experiencing significant elapsed times between service request and delivery. It was becoming obvious to the IT thought-leaders that merely provisioning and offering up “ping, power and pipe” quickly wasn’t going to have a meaningful impact. That just shifted the bottleneck up the stack into the application layers. The new focus of the cloud revolves around rapid deployment and mass management of applications. Without the ability to provision and manipulate discrete application services, the value of the cloud is stunted.

As the recent acquisition of Phurnace indicates, the traditional data center tools vendors have started figuring it out and are attempting to offer solutions that deliver automation aimed at the application layers. The advent of these vendors and their mainstay strength in server provisioning is bringing forth multiple interesting approaches to application automation. Approaches vary from the composite policy-based (but hands-off) VMware vApps strategy to the automation of selective app admin tasks for specific app components such as Java code deployment (e.g., Phurnace) to broader automation platforms with out-of-the-box modular content to address the entire administrative lifecycle of various app components (e.g., Stratavia). The key question that emerges for customers is “what’s the right way to automate applications?” Conventional wisdom says if your only weapon is a hammer, every problem looks like a nail. Armed with robust server virtualization and provisioning toolsets, some of the larger vendors are approaching the application stack with (no surprises!) a focus on provisioning. However at the risk of sounding clichéd, I have to say that automating application administration is a very different paradigm.

You see, at the server layer and below (including storage and networking), significant admin time is taken up in provisioning. Post-provisioning activities such as patching and configuration management are often handled via provisioning – i.e., by reimaging the OS with an updated (patched/reconfigured) image. So all in all, provisioning is the key administrative function at these lower layers.

However as you go above the server, you encounter discrete application layers such as webservers, application servers and databases – to name the most popular components. Each of these dictate a different operations lifecycle that is not dominated by provisioning and configuration management. In fact, provisioning barely takes up 15-20% of the typical App Admins’ time. The remaining 80% time is spent on other post-provisioning activities such as maintenance, incident response and service requests (the graphic below provides examples of these task categories).

But then the traditional vendors ask, why do these App Admins do all these things in the first place? Why can’t they, like Sys Admins and Network Admins, handle these other activities via Provisioning/Re-provisioning? For instance, instead of applying a new patch or performing a maintenance operation or doing a code release, why can’t the App Admin just provision a new image that has the desired changes? That will eliminate the need for these other post-provisioning activities and free up App Admin time.

This type of argument does not work because it goes against the grain of real-world application management. Instances of an application component frequently develop a unique fingerprint over the course of their use. This fingerprint is based on several factors including security requirements, performance adjustments, and the experience and skill-level (a.k.a. best practices) of the Admins managing them. To make things worse, these fingerprints can be dynamic in nature. The same application may look different at different times of the day. For instance, a database may serve as a transactional data-store during regular business hours and may be configured specifically to facilitate smaller read/write I/O operations, whereas at night time, the same database may be converted into a batch database with different buffer sizes, log file locations, etc. to facilitate bulk writes. Bland categorization and re-imaging of the application server (say, to apply a new application patch) by the Sys Admins causes much of this dynamic application fingerprint to be lost - creating in turn a lot more work for the App Admins (to restore the fingerprint as much as they can – assuming they themselves remember all the changes and can get it right the first time!)

The other challenge is that a single server frequently hosts multiple application types and instances. This is commonly encountered even in large-scale production environments. Each of these instances may have their own maintenance window and need to be patched/ reconfigured/upgraded and managed individually based on instance-specific standards and dependencies. Reimaging (the entire server) doesn’t afford granular control of individual instances. Now one may argue that this was more prevalent in the olden days when servers were expensive and with virtualization being commoditized now, each application instance can reside on its own server image thereby eliminating this problem. But reality goes deeper than that. It’s not just server resource optimization that called for multiple application types and instances to reside on the same server; cohabitation was also tied to performance, security and other considerations. For example, in the case of a performance sensitive application that utilizes federated databases, a DBA may elect to keep some of the associated databases on the same physical OS to minimize context switching and network latency incurred due to physical separation of the databases onto different servers. Regardless of whether the servers and underlying network adaptors are physical or virtual, the location of the underlying databases can make a big difference in terms of response time for a data-join operation being in sub-seconds versus minutes. Thus without proper understanding of the various application types, and related design considerations (such as transaction types, application access methods, data volume, affinity, partitioning, etc.) and best practices, choosing the wrong automation method can result in degraded service levels.

Attempting to offer application management in the cloud with just an application image provisioning model is akin to showing up to a gun fight with a rock. Proponents of this model can claim that rapid provisioning and policy-based reimaging of the relevant application components is the new way of application management in the cloud. While this approach may work for a handful of admin functions, it does not offer granular control or a pragmatic framework for most mandatory post-provisioning tasks (represented in the graph above) and hence will be discarded by savvy App Admins. Only solution providers that have a true application management DNA (with a deep understanding of task patterns associated with various application components) and offering automated application management capabilities out-of-the-box can win legitimate mind-share in the near term and sustainable market share in the long run.

Reference Architecture for Delivering IT as a Service

2009-11-12T16:30:00.000-07:00

I’m back now after a 4-month “hiatus” that comprised multiple customer engagements – productive activity that keeps me (and my rants here) relevant. A key area I have been working on is using Stratavia’s Data Palette to help my customers deliver IT as a service within their organizations – or to be more precise, helping them deliver applications as a service. From an “end-user” perspective such as project managers, application team leads and QA managers (i.e., the recipients of these services), individual components of the infrastructure (plain servers or raw storage) don’t matter; it’s about having fully baked apps appropriately packaged and delivered - including the webserver, the application server and the database layers.

The obvious reason CIOs are looking to upgrade their IT delivery capabilities is around improving business efficiency and agility, while of course reducing costs. But a less frequently cited, but equally vital reason is to keep up with the competition! For instance, every financial services firm out there has already built or is building an internal cloud. In fact, larger organizations across all industry verticals are taking the next step to attain scalability via newer delivery models such as self-service and cloud computing. But to truly gain tangible benefit from such scalable models is a challenge. And currently, application administration is the weakest link in the chain!

The whole premise behind cloud computing of being able to rapidly mass-deploy applications in the cloud frequently comes to a screeching halt due to the way IT currently operates. Most IT administration teams are not just geared up to provision and manage scores of complex heterogeneous application tiers in an agile manner – unless more and more manpower is added and even that doesn’t scale after a certain point. Sure there is help from the conventional systems management tools vendors like HP, BMC, IBM, EMC, VMware and Cisco. Automation products from these venerable vendors are able to help organizations reduce server, network and storage provisioning time from multiple weeks to a few hours. But then the bottleneck just shifts upstream into the application layers: specifically the middleware and the database tiers. End-to-end automation of these tiers is a pre-requisite to large-scale application deployments on the cloud.

Conventional server provisioning and runbook automation products do not have application-level smarts nor native (application-specific) automation functionality to be of help here. Apart from some very basic application binary installs, they cannot be used to automate the complex activities across the application operations lifecycle, depicted alongside - unless one ends up writing and maintaining millions of lines of custom script-code (job security, anyone?).

Stratavia provides IT organizations with a way to break this logjam at the database and application tiers. Stratavia does this via its Data Palette automation platform, along with a portfolio of automation modules, called DCA Apps that can be plugged into the underlying platform. The DCA Apps include solutions for the entire operations lifecycle of the database and middleware tiers represented in the graphic above. The solution allows companies to obtain the following benefits (while complementing prior investments such as HP/Opsware and BMC/BladeLogic ), thus truly enabling IT to be delivered as a service.

- Streamlining and improving IT operations

Standardize IT processes across heterogeneous platforms and assets

Reduce delay between service request and delivery; Improve service level metrics such as “first-time-right” and “on-time-delivery”

Establish & control delivery quality across multi-tiered skill sets; Enable non-SMEs to carry out complex operations

Improve service delivery with “self service” capability in key areas such as application build provisioning, code releases and migrations

Remove compliance & support risks due to variety of version, patch & configuration requirements

- Increasing efficiencies

Reduce IT Admin time spent on mundane activities

Increase Asset to Admin ratio

The reference architecture Stratavia enables to accomplish these objectives is as follows (click on the schematic to view an enlarged version):
The right side of the schematic above shows the major tiers that make up the entire application stack. The middle portion shows the role of the Data Palette automation fabric in both orchestrating and performing the administrative activities across the entire operations lifecycle. (The Data Palette platform includes the orchestration capabilities, while the DCA Apps perform the administration activities.) These lifecycle activities include provisioning and patching, configuration and compliance management, recurring maintenance (e.g., log pruning, backups, healthchecks, index rebuilds, table reorgs, partition shuffles, etc.), incident response (false positive alert suppression and white noise reduction, problem diagnosis and root cause analysis, auto-resolving known errors, etc.) and frequent service requests (e.g., code releases, database refreshes and cloning, upgrades, adding/modifying user accounts, adding space, restoring an application snapshot, failover, etc.) Data Palette also provides out-of-the-box integration adaptors to be able to auto-cut tickets, update a CMDB, and interface with various systems management toolsets in order to adhere to standard ITIL processes while carrying out these activities (not dissimilar to a DBA or App Admin who performs this work manually).

The fabric also helps in abstracting the backend component-level complexities from the end-users.

On the left side and the top, the schematic illustrates 4 classes of users:

Non-Technical End Users: This class of users refers to the application and business end-users. These users are typically are not too IT operations-savvy (nor should they have to be!) and conventionally request resources via a Help Desk / ticketing system such as BMC Remedy, HP Service Manager or Service-Now.com. Once the ticket is created, it is assigned to the appropriate technicians and may traverse multiple IT operations groups before the request is fulfilled. Data Palette enables self-service capabilities in this scenario by presenting a Service Catalog front-end to these users. Frequently, Data Palette’s native adaptors are used to integrate with existing ticketing systems so that these end-users do not have to be exposed to the Data Palette console (and not have to learn yet another tool or interface!). The Service Catalog is established within the system they are already familiar with, wherein they can put in their request along with relevant details such as service name, required duration, billing code, etc. Once the request is saved, it can be auto-routed to a manager for approval. The ticket creation or approval action triggers an automated workflow (within a Data Palette DCA App) that will provision the service and make it available to the end-user while updating/closing out the ticket once the service is brought online. The service is usually multi-layers and can comprise multiple sub-workflows that will provision a database instance, install an Apache webserver, WebLogic app server, create user accounts in the database and so on. The minutiae are abstracted from the end-user.

IT Operators – This category of users refers to Tier 1 personnel such as Help Desk operators, NOC personnel and even outsourced/offshore administrators in some cases. These users tend to be the preliminary points of contact for alerts from different monitoring tools or problem calls from end users. Data Palette empowers these IT operators to be able to carry out automated incident triage and even auto-remediation of recurring incidents thereby reducing the need to escalate to IT Operations Administrators (Tier 2 personnel). IT Operators are not SMEs, but have a greater degree of awareness of the IT environment and hence can have direct access to the Data Palette console (i.e., bypass the previously mentioned Service Catalog) along with the ability to execute specific workflows in certain environments - managed via Data Palette’s multi-tenancy and role-based access control.

IT Operations Administrators – These are the Tier 2 SMEs – the DBAs and the Application Admins that have privileges within Data Palette to deploy automation services, along with the ability to set relevant Policies and metadata to properly influence Data Palette’s automation behavior in the environments they manage.

IT Operations Engineers – These are Tier 3 SMEs - the IT Operations Engineers (also referred to as Applications or Database Systems Engineers or Architects) that have the ability to define automation services by configuring the Data Palette DCA Apps and the corresponding workflows, along with any site-specific pre and post Steps. They decide which automation service should be available to which user-type across the enterprise, what parameters should be entered (balancing ease-of-use against flexibility and control), which toolsets to integrate with, which metadata to leverage, and so on. Accordingly, their role within Data Palette is a super-set of the prior users allowing them to read, write (update) and execute automation workflows and corresponding service definitions.

Finally, the bottom portion of the reference architecture shows Data Palette integrating with existing enterprise monitoring and popular configuration and compliance audit toolsets such as MS SCOM, HP OVO, Patrol, Tivoli, Tripwire, EMC Ionix Configuration Manager and Guardium via its integration adaptors for these product sets. These products (and others like them) are frequently already deployed by enterprises for performance monitoring and scanning OS, database and application configurations and can be set up to invoke a Data Palette remediation workflow (via Data Palette’s web service APIs) to address drifts and SLA violations. Data Palette’s configuration repair and incident resolution workflows can fix the violations in online mode, or schedule the repair during the appropriate maintenance window based on pre-defined policies. Administrators can set up on an environment-by-environment basis, which violations should be immediately repaired, versus which ones need to be scheduled, versus which ones can be safely ignored.

Enterprise features of the Data Palette platform such as multi-tenancy, RBAC (role-based access control), single sign-on, LDAP integration and Smart Groups (wherein multiple intra-cloud assets can be addressed and manipulated as a single entity) along with self-configuring, out-of-the-box automation content for the database and application tiers makes the above architecture and corresponding value imminently attainable. (As a point of reference, a Proof of Concept takes 3 days to 2 weeks depending on the scope; a broader Pilot including integrations with existing toolsets can be implemented in 2 to 4 weeks.)

Email me at vdevraj at stratavia dot com if you would like a detailed whitepaper on this solution architecture.

Why aren't databases getting migrated to VMware?

2009-07-20T13:02:00.001-06:00

During a recent customer CTO focus group meeting, a key topic discussed was the perceived unwillingness (or should one say, inability) of many larger organizations to move their databases onto virtual servers.

The majority of databases, especially production systems continue to run on physical servers. Given the emphasis in today's economy on data center consolidation and the need to enable newer IT delivery models such as self-service and private cloud initiatives, migrating databases from expensive underutilized physical servers to shared virtual environments should be a no-brainer. However IT managers find themselves swimming upstream as soon as they broach the topic of database migrations with their application users or for that matter, their DBAs.

Migrating databases to virtual environments is difficult for several reasons - especially given their complexity and mission-critical nature, the extended time required to perform those migrations gracefully and the general lack of tolerance from application users to corresponding maintenance/outage windows. A database has hooks deep into the underlying operating system as well as the overlaying application stack. Even relatively “minor” migrations wherein both the source and target platforms and versions are exactly the same can impact database performance and stability if all the associated factors are not duly considered. Let's explore some of these factors.

A database migration in this context is defined as moving a database from a physical to a virtual environment. Whenever possible, the source and target environments retain the same operating system (OS) attributes. But frequently, those may need to be changed as well. For instance, when migrating an Oracle or DB2 database running on an IBM pSeries server with AIX 5.x to a virtual environment on VMware, the underlying operating system type has to change to a Linux flavor on the x86 platform. Accordingly, the following are the two main options for a database migration:
• The same OS on both the physical (source) and virtual (target) environments
• A change in the OS type or version in the target virtual environment.

The latter use case is obviously more complex than the former. Regardless, with both use cases, there are several factors, decision points and associated actions that need to be taken into consideration. The sequence of actions in the schematic below (Figure 1) illustrates a few of the core issues associated with such a migration for Oracle, SQL Server and DB2. (Click on the picture below to view a larger image.)

Especially as Rows 2 and 4 in Figure 1 reveal, there are specific actions that need to be taken both on the source (physical) and target (virtual) environments. Furthermore, all of these actions need to be taken in line with corporate standards and best practices including checking for change control approvals, carrying out the work within pre-defined maintenance windows, and rolling back specific actions upon failure or other environment conditions such as the maintenance window being exceeded.

All too often, IT managers and generic IT administrators (read, non-DBAs) mistakenly believe that the scope of database migrations is limited to provisioning a new target database server, copying the data contents from the source to the target and finally, re-pointing the applications to the new target environment. In this context, one needs to differentiate between a database server and a database instance. In the case of the former, one can use existing provisioning tools to set up the requisite platform and version build, patch-level, kernel parameters, file-systems and other operating system-related aspects (equating to Row 3 in Figure 1 above) and utilize standard run book automation (RBA) tools to interact with change control and ticketing mechanisms (equating to Rows 1 and 5 in Figure 1), however these provisioning and RBA tools fall short when it comes to the setup and management of much of the database internals (as shown in Rows 2 and 4 in Figure 1) – tasks that make up the core of the migration process. For instance, tools such as BMC BladeLogic, HP Opsware SAS and VMware vCenter Server can rapidly provision target database server environments, but lack the database instance and application specific context and the corresponding automation content to discern and establish crucial post-server-provisioning aspects. Conventional RBA tools such as BMC BladeLogic Orchestration Manager, HP Operations Orchestrator and VMware vCenter Orchestrator have the capability to interact with change control and ticketing systems to perform the peripheral tasks in Rows 1 and 5 in Figure 1, but lack knowledge of database internals hence requiring all the steps indicated in Rows 2 and 4 to be developed from scratch – a process that can take multiple years.

Similarly, Virtual server migration tools as VMware vCenter Converter (a.k.a. VMware P2V) and Novell PlateSpin PowerConvert that migrate a server in its entirety also don't do justice to the task at hand because a database server may comprise multiple databases/instances supporting several different applications, and the DBA may need to selectively migrate a few databases/instances at a time (based on application user and change management approvals) rather than the entire database server. P2V-like tools don't take these intra-database structures and other application-level dependencies into account. Also, these tools only deal with x86 servers running Windows and Linux. Non-x86 hardware platforms and other operating systems such as AIX and Solaris are woefully ignored.

A database instance has its own unique set of requirements and configuration options that need to be defined and managed. Take for example, an Oracle database environment. A typical mid-sized to large company (the kind that will benefit most from automation) may have dozens, even hundreds of database instances across different versions (e.g., 9i, 10g, 11g) and configurations (standalone, RAC, Data Guard, etc.). New databases have to be set up for multiple environments (production, QA, stage, etc.) and applications (OLTP, reporting, warehouse, batch, etc.). Based on the OS platform, database version, configuration, usage factors and size, different data transfer methods have to be selectively applied during a migration (e.g., RMAN duplicate, Transportable Tablespaces, Data Pump, Export/Import, etc.). Once the data is extracted, it needs to be imported into the target environment using the appropriate method(s) and reconfigured for user and application access. Structures such as stored procedures, triggers and other objects need to be transferred as well using appropriate mechanisms. External stored procedures need to be recompiled and relinked on the target. Depending on the source and target operating systems, issues such as endian formats, byte sizes and character types need to be considered. Shared libraries have to be installed and clustering related parameters have to be defined. Backup methods need to be reconfigured and rescheduled. Agents need to be reinstalled. Maintenance and other scheduled jobs have to be set up. Database links need to be re-established in the case of federated or distributed databases. Thus, several configuration, security and performance related factors need to be evaluated and appropriately addressed.

Many of these considerations have prevented DBAs from viewing automation as a reliable method to perform migrations. And performing them manually is not a realistic option since they can take up several thousands of DBA hours and carry a significant risk of performance degradation or downtime if human errors creep in. These complex issues prevent most databases from being migrated to virtual environments, in spite of the obvious cost advantages associated with virtualization. There is adequate fear prevalent amongst IT managers to keep their hands off databases and leave them running on mammoth underutilized physical servers.

That’s where database automation products such as Stratavia’s Data Palette can help. Specifically, Data Palette provides value over conventional server provisioning and RBA technologies as well as P2V-type tools in two key ways:
- due to its comprehensive database automation content, it can handle complex migration use cases out-of-the-box

- due to its ability to collect, persist and embed metadata within its automation workflows (to provide environmental awareness to those workflows), it can handle heterogeneity and ongoing changes in platform builds, versions, application attributes and usage profiles in a much more scalable manner. It allows a consistent set of migration workflows to be used across disparate environments from a central console, making the automation easy to deploy, maintain and run.

Migrating databases from physical servers to virtual machines becomes straightforward with Data Palette due to the ability of its workflow engine to span both physical and virtual environments (within a single workflow) - allowing the entire spectrum of activities related to database migration (as illustrated in Figure 1) to be automated end-to-end with the press of a button.

This automation content coupled with Data Palette’s role-based access control and Service Catalog type user interface allows senior DBAs as well as junior or offshore database operations personnel and even non-DBAs for that matter (e.g., trusted, but non-database-savvy IT personnel such as systems administrators and application project leads) to perform these migrations in self-service mode without requiring them to have knowledge of underlying database processes, and without local administrative access on the source and target servers - in a manner that meets all DBA and change control approvals.

Post-migration, a key Data Palette feature called Smart Groups™ allows the database instances to retain all of their prior monitoring, maintenance and other scheduled activity in a seamless manner without requiring any manual intervention. These advanced management capabilities and out-of-the-box automation content allows complex database physical to virtual migration projects to be completed in days, rather than weeks and months.

Implementing a Simple Internal Database or Application Cloud – Part II

2009-05-15T10:13:00.000-06:00

Based on the overview of private clouds in my prior blog, here’s the 5-step recipe for launching your implementation:

Identify the list of applications that you want to deploy on the cloud;

Document and publish precise end-user requirements and service levels to set the right expectations – for example, time in hours to deliver a new database server;

Identify underlying hardware and software components – both new and legacy – that will make up the cloud infrastructure;

Select the underlying cloud management layer for enabling existing processes, and connecting to existing tools and reporting dashboards;

Decide if you wish to tie in access to public clouds for cloud bursting, cloud covering or simply, backup/archival purposes.

Identify the list of applications that you want to deploy on the cloud
The primary reason for building a private cloud is control – not only in terms of owning and maintaining proprietary data within a corporate firewall (mitigating any ownership, compliance or security concerns), but also deciding what application and database assets to enable within the cloud infrastructure. Public clouds typically give you the option of an x86 server running either Windows or Linux. On the database front, it’s usually either Oracle or SQL Server (or the somewhat inconsequent MySQL). But what if your core applications are built to use DB2 UDB, Sybase or Informix? What if you rely on older versions of Oracle or SQL Server? What if your internal standards are built around AIX, Solaris or HP/UX system(s)? What if your applications need specific platform builds? Having a private cloud gives you full control over all of these areas.

The criteria for selecting applications and databases to reside on the cloud should be governed by a single criterion – popularity; i.e., which applications are likely to proliferate the most in the next 2 years, across development, test and production . List down your top 5 or 10 applications and their infrastructure – regardless of operating system and database type(s) and version(s).

Document precise requirements and SLAs for your cloud
My prior blog entry talked about broad requirements (such as self-service capabilities, real-time management and asset reuse), but break those down into detailed requirements that your private cloud needs to meet. Share those with your architecture / ops engineering peers, as well as target users (application teams, developer/QA leads, etc.) and gather input. For instance, a current cloud deployment that I’m working with is working to meet the following manifesto, arrived at after a series of meticulous workshops attended by both IT stakeholders and cloud users:

In the above example, BMC Remedy was chosen as the front-end for driving self-service requests largely because the users were already familiar with using that application for incident and change management. In its place, you can utilize any other ticketing system (e.g., HP/Peregrine or EMC Infra) to present a friendly service catalog to end users. Up-and-coming vendor Service-now extends the notion of a service catalog to include a full-blown shopping cart and corresponding billing. Also, depending on which cloud management software you utilize, you may have additional (built-in) options for presenting a custom front-end to your users – whether they are IT-savvy developers, junior admin personnel located off-shore or actual application end-users.

Identify underlying cloud components
Once you have your list of applications and corresponding requirements laid out, you can begin to start the process of defining which hardware and software components you are going to use. For instance, can you standardize on the x86 platform for servers with VMware as the virtualization layer? Or do you have a significant investment in IBM AIX, HP or Sun hardware? Each platform tends to have its own virtualization layer (e.g., AIX LPARs, Solaris Containers, etc.), all of which can be utilized within the cloud. Similarly, for the storage layer, can you get away with just one vendor offering – say, NetApp filers, or do you need to accommodate multiple storage options such as EMC and Hitachi? Again, the powerful thing about private clouds is – you get to choose! During a recent cloud deployment, the customer required us to utilize EMC SANs for production application deployments, and NetApp for development and QA.

Also, based on application use profiles and corresponding availability and performance SLAs, you may need to include clustering or facilities for standby databases (e.g., Oracle Data Guard or SQL log shipping) and/or replication (e.g., GoldenGate, Sybase Replication Server).

Now as you read this, you are probably saying to yourself – “Hey wait a minute… I thought this was supposed to be a recipe for a ‘simple cloud’. By the time I have identified all the requirements, applications and underlying components (especially with my huge legacy footprint), the cloud will become anything but simple! It may take years and gobs of money to implement anything remotely close to this…” Did I read your mind accurately? Alright, let’s address this question below.

Select the right cloud management layer
Based on all the above items, the scope of the cloud and underlying implementation logistics can become rather daunting – making the notion of a “simple cloud” seem unachievable. However here’s where cloud management layers comes to the rescue. A good cloud management layer keeps things simple via three basic functions:

Abstraction;
Integration; and
Out-of-the-box automation content

Cloud management software ties in existing services, tools and processes to orchestrate and automate specific cloud management functions end-to-end. Stratavia’s Data Palette is an example of intelligent cloud management software. Data Palette is able to accommodate diverse tasks and processes – such as asset provisioning, patching, data refreshes and migrations, resource metering, maintenance and decommissioning – due to significant out-of-the-box content. (Stratavia references them as Solution Packs, which are basically discrete products that can be plugged into the underlying Data Palette platform.) All such content is externally referenceable via its Web Service API for easy integration with existing tools – making it easy to integrate with 3rd party service catalogs such as Service-Now or Remedy.

Data Palette does not post restrictions on server, operating system and infrastructure components within the cloud. Its database Solution Packs support various flavors and versions of Oracle, SQL Server, DB2, Sybase and Informix running on UNIX (Solaris, HP/UX and AIX), Linux and Windows. Storage components such as NetApp are managed via the Zephyr API (ZAPI) and OnTapi interfaces.

However in addition to out-of-the-box integration and automation capabilities, the primary reason for keeping complexity at bay is due to the use of an abstraction layer. Data Palette uses a metadata repository that is populated via native auto-discovery (or via integration to pre-deployed CMDBs and monitoring tools) to gather a set of configuration and performance metadata that identifies the current state of the infrastructure, along with centrally defined administrative policies. This central metadata repository makes it possible for the right automated procedures to be executed on the right kind of infrastructure – avoiding mistakes (typically associated with static workflows from classic run book automation products)) such as executing an Oracle9i specific data refresh method on a recently upgraded Oracle10g database – without the cloud administrator or user having to track and reconcile such infrastructural changes and manually adjust/tweak the automation workflows. Such metadata-driven automation keeps the automation workflows dynamic, allowing the automation to scale seamlessly to hundreds or thousands of heterogeneous servers, databases and applications.

Metadata collections can also be extended to specific (custom) application configurations and behavior. Data Palette allows Rule Sets to be applied to incoming collections to identify and respond to maintenance events and service level violations in real-time making the cloud autonomic (in terms of self-configuring and self-healing attributes), with detailed resource metering and administrative dashboards.

Data Palette’s abstraction capabilities also extends to the user interface wherein specific groups of cloud administrators and users are maintained via multi-tenancy (called Organizations), Smart Groups™ (dynamic groupings of assets and resources), and role based access control.

Optionally tie in access to public clouds for Cloud Bursting, Cloud Covering or backup/archival purposes
Now that you are familiar with the majority of the ingredients for a private cloud rollout, the last item worth considering is whether to extend the cloud management layer’s set of integrations to tie into a public cloud provider – such as GoGrid, Flexiscale or Amazon EC2. Based on specific application profiles, you may be able to use a public cloud for Cloud Bursting (i.e., selectively leveraging a public cloud during peak usage) and/or Cloud Covering (i.e., automating failover to a public cloud). If you are not comfortable with the notion of a full service public cloud, you can consider a public sub-cloud (for specific application silos e.g., “storage only”) such as Nirvanix or EMC Atmos for storing backups in semi-online mode (disk space offered by many of the vendors in this space is relatively cheap – typically 20 cents per GB per month). Most public cloud providers offer an extensive API set that the internal cloud management layer can easily tap into (e.g., check out wiki.gogrid.com). In fact, depending on your internal cloud ingredients, you can take the notion of Cloud Covering to the next level and swap applications running on the internal cloud to the external cloud and back (kind of an inter-cloud VMotion operation, for those of you who are familiar with VMware’s handy VMotion feature). All it takes is an active account (with a credit card) with one of these providers to ensure that your internal cloud has a pre-set path for dynamic growth when required – a nice insurance policy to have for any production application – assuming the application’s infrastructure and security requirements are compatible with that public cloud.

Implementing a Simple Internal Database or Application Cloud - Part I

2009-01-28T16:25:00.000-07:00

A “simple cloud”? That comes across as an oxymoron of sorts since there’s nothing seemingly simple about cloud computing architectures. And further, what do DBAs and app admins have to do with the cloud, you ask? Well, cloud computing offers some exciting new opportunities for both Operations DBAs and Application DBAs – models that are relatively easy to implement, and bring immense value to IT end-users and customers.

The typical large data center environment has already embraced a variety of virtualization technologies at the server and storage levels. Add-on technologies offering automation and abstraction via service oriented architecture (SOA) are now allowing them to extend these capabilities up the stack – towards private database and application sub-clouds. These developments seem more pronounced in the banking, financial services and managed services sectors. However while working on Data Palette automation projects at Stratavia, every once-in-a-while I do come across IT leaders, architects and operations engineering DBAs in other industries as well, that are beginning to envision how specific facets of private cloud architectures can enable them to service their users and customers more effectively (while also compensating for the workload for some of their colleagues that have exited their companies due to the ongoing economic turmoil). I wanted to specifically share here some of the progress in database and application administration with regard to cloud computing.

So, for those database and application admins that haven’t had a lot of exposure to cloud computing (which BTW, is a common situation since most IT admins and operations DBAs are dealing with boatloads of “real-world hands-on work” rather than participating in the next evolution of database deployments), let’s take a moment to understand what it is and its relative benefits. An “application” in this context, refers to any enterprise-level app - both 3rd party (say, SAP or Oracle eBusiness Suite) as well as home-grown N-Tier apps that have a fairly large footprint. Those are the kind of applications that get maximum benefit from the cloud. Hence I use the word “data center asset” or simply “asset” to refer to any type of database or application. However at times, I do resort to specific database terminology and examples, which can be extrapolated to other application and middleware types as well.

Essentially a cloud architecture refers to a collection of data center assets (say, database instances, or just schemas to allow more granularity) that are dynamically provisioned and managed throughout their lifecycle – based on pre-defined service levels. This lifecycle covers multiple areas starting with deployment planning (e.g., capacity, configuration standards, etc.), provisioning (installation, configuration, patching and upgrades) and maintenance (space management, logfile management, etc.) extending all the way to incident and problem management (fire-fighting, responding to brown-outs and black-outs), and service request management (e.g., data refreshes, app cloning, SQL/DDL release management, and so on). All of these facets are managed centrally such that the entire asset pool can be viewed and controlled as one large asset (effectively virtualizing that asset type into a “cloud”).

Here’s a picture representing a fully baked database cloud implementation (if the picture is blurry, click on it to open up a clearer version):

As I had mentioned in a prior blog entry, there are multiple components that have come together to enable a cloud architecture. But more on that later. Let’s look at database/application specific attributes of a cloud (you could read it as a list of requirements for a database cloud).

Self-service capabilities: Database instances or schemas need to be capable of rapidly being provisioned based on user specifications by administrators, or by the users themselves (in selective situations – areas where the administrators feel comfortable giving control to the users directly). This provisioning can be done on existing or new servers (the term “OS images” is more appropriate given that most of the “servers” would be virtual machines rather than real bare metal) with appropriate configuration, security and compliance levels. Schema changes or SQL/DDL releases can be rolled out in a scheduled manner, or on-demand. The bulk of these releases, along with other service requests (such as refreshes, cloning, etc.) should be capable of being carried out by project teams directly– with the right credentials (think, role-based access control).

Real-time infrastructure: I'm borrowing a term from Gartner (specifically, distinguished analyst Donna Scott's vocabulary) to describe this requirement. Basically, the assets need to be maintained in real-time per specific deployment policies (such as development environment versus QA or Stage), tablespaces and datafiles created per specific naming / size conventions and filesystem/mount-point affinity (accommodating specific SAN or NAS devices, different LUN properties and RAID levels for reporting/batch databases versus OLTP environments), data backed up at the requisite frequency per the right backup plan (full, incremental, etc.), resource usage metered, failover/DR occurring as needed, and finally, archived and de-provisioned based on either a specific time-frame (specified at the time of provisioning) or on-demand -- after the user or administrator indicates that the environment is no longer required (or after a specific period of inactivity). All of this needs to be subject to administrative/manual oversight and controls (think, dashboards and reports, as well as ability to interact with or override automated workflow behavior).

Asset type abstraction and reuse: One should be able to mix-and-match these asset types. For instance, one can rollout an Oracle-only database farm or a SQL Server-only estate. Alternatively, one can also span multiple database and application platforms allowing the enterprise to better leverage their existing (heterogeneous) assets. Thus, the average resource consumer (i.e., the cloud customer) shouldn’t have to be concerned about what asset types or sub-types are included therein – unless they want to override default decision mechanisms. The intra-cloud standard operating procedures take those physical nuances into account, thereby effectively virtualizing the asset type.

The benefit of a database cloud includes empowering users to carry out diverse activities in self-service mode in a secure, role-based manner, which in turn, enhances service levels. Activities such as having a database provisioned or a test environment refreshed can often take multiple hours and days. Those can be reduced to a fraction of their normal time – reducing latency especially in situations where there needs to be hand-offs and task turn-over across multiple IT teams. In addition, the resource-metering and self-managing capabilities of the cloud allow better resource utilization and avoids resource waste, improving performance levels, and reducing outages and removing other sources of unpredictability from the equation.

A cloud, while viewed as bleeding edge by some organizations is being viewed by larger organizations as being critical – especially in the current economic situation. Rather than treating each individual database or application instance as a distinct asset and managing it per its individual requirements, a cloud model allows virtual asset consolidation, thereby allowing many assets to be treated as one and promoting unprecedented economies of scale in resource administration. So as companies continue to scale out data and assets, but cannot afford to correspondingly scale up administrative personnel , the cloud helps them achieve non-linear growth.

Hopefully the attributes and benefits of a database or application cloud (and the tremendous underlying business case) become apparent here. My next blog entry (or two) will focus on the requisite components and the underlying implementation methods to make this model a reality.

Protecting Your IT Operations from Failing IT Services Firms

2009-01-09T10:57:00.000-07:00

The recent news about India-based IT outsourcing major Satyam and its top management’s admissions of accounting fraud bring forth shocking and desperate memories of an earlier time – when multiple US conglomerates such as Enron, Arthur Andersen, Tyco, etc. fell under similar circumstances, bringing down with them the careers and aspirations of thousands of employees, customers and investors. Ironically Satyam (the name means “truth” in the mother language, Sanskrit), whose management have been duping investors for several years now (by their own admission) had received the Recognition of Commitment award from the US-based Institute of Internal Auditors in 2006, and was featured in Thomas Friedman’s best-seller “The World is Flat”. Indeed, how the mighty have fallen…

As one empathizes with those affected, the key question that comes to mind is, how do we prevent another Satyam? However that line of questioning seems rather idealistic. The key question should probably be, how can IT outsourcing customers protect themselves from these kinds of fallouts? Given how flat the world is, an outsourcing vendor’s (especially one as ubiquitous as Satyam in this market) fall from grace has reverberations throughout the global IT economy - directly in the form of failed projects, and indirectly in the form of lost credibility for customer CIOs who rely on these outsourcing partners for their critical day-to-day functioning.

Having said that, here are some key precautionary measures (in an evolving order) companies can take to protect themselves and their IT operations beyond standard sane efforts such as using multiple IT partners, use of structured processes and centralized documentation/knowledge-bases.
· Move from time & material (T&M) arrangements to fixed-priced contracts
· Move from static knowledge-bases to automated standard operating procedures (SOPs)
· Own the IP associated with process automation

Let’s look at how each of these afford higher protection in situations such as the above:
· Moving from T&M arrangements to fixed price contracts - T&M contracts rarely provide incentive to the IT outsourcing vendor to bring in efficiencies and innovation. The more hours that are billed, the more revenue they make – so,where’s the motivation to reduce the manual labor? On the other hand, T&M labor makes customers vulnerable to loss of institutional knowledge and gives them little to no leverage when negotiating rates or contract renewals because switching out a vendor (especially one that holds much of the “tribal knowledge”) is easier said than done.

With fixed price contracts, the onus on ensuring quality and timely delivery is on the IT services vendor (to do so profitably requires use of as little labor as possible) and subsequently, one finds more structure (such as better documentation and process definition) and higher use of innovation and automation. All of this works in the favor of the customer and in the case of a contractor or the vendor no longer being available, makes it easier for a replacement to hit the ground running.

· Moving from static knowledge-bases to automated SOPs – It is no longer enough to have standard operating procedures documented within Sharepoint-type portals. It is crucial to automate these static run books and documented SOPs via data center automation technologies, especially newer run book automation product sets (a.k.a. IT process automation platforms) that allow definition and utilization of embedded knowledge within the process workflows. These technologies allow contractors to move static process documentation to workflows that use this environmental knowledge to actually perform the work. Thus, the current process knowledge no longer merely resides in peoples’ heads, but gets moved to a central software platform thereby mitigating loss of key contractor personnel/vendors.

· Owning the IP associated with such process automation platforms – Frequently, companies that are using outsourced services ask “why should I invest in automation software? I have already outsourced our IT work to company XYZ. They should be buying and using such software. Ultimately, we have no control over how they perform the work anyway…” The Satyam situation is a classic example of why it behooves end-customers to actually purchase and own IP related to process automation software, rather than deferring it to the IT services partner. By having process IP defined within a software platform that the customer owns, it makes it conceivable to switch contractors and/or IT services firms. If the IT services firm owns the technology deployment, the corresponding IP walks out the door with the vendor preventing the customer from getting the benefit of the embedded process knowledge.

It is advisable for the customer to have some level of control and oversight over how the work is carried out by the vendor. It is fairly commonplace for the customer to insist on use of specific tools and processes such as ticketing systems, change control mechanisms, monitoring tools and so on. The process automation engine shouldn’t be treated any different. The bottomline is, whoever has the process IP carries the biggest stick during contract renewals. If owning the technology is not feasible for the customer, at least make sure that the embedded knowledge is in a format wherein it can be retrievable and reused by the next IT services partner that replaces the current one.

Zen and the Art of Automation Continuance

2009-01-02T13:31:00.000-07:00

The new year is a good time to start thinking about automation continuance. Most of us initiate automation projects with a focus on a handful of areas – such as provisioning servers or databases, automating the patch process, and so on. While this kind of focus is helpful in ensuring a successful outcome for that specific project, it also has the effect of reducing overall ROI for the company – because once the project is complete, people move on to other day-to-day work patterns (relying on their usual manual methods), instead of continuing to identify, streamline and automate other repetitive and complex activities.

Just recently I was asked by a customer (a senior manager at a Fortune 500 company that has been using data center automation technologies, including HP/Opsware and Stratavia's Data Palette for almost a year) "how do I influence my DBAs to truly change their behavior? I thought they had tasted blood with automation, but they keep falling back to reactive work. How do I move my team closer to spending majority of their time on proactive work items such as architecture, performance planning, providing service level dashboards, etc.?” Sure enough, their DBA team started out impressively automating over half-a-dozen tasks such as database installs, startup/shutdown processes, cloning, tablespace management, etc., however during the course of the year, their overall reactive workload seems to have relapsed.

Indeed, it can seem an art to keep IT admins motivated towards continuing automation.

A good friend of mine in the Oracle DBA community, Gaja Krishna Vaidyanatha coined the phrase “compulsive tuning disorder” to describe DBA behavior that involves spending a lot of time tweaking parameters, statistics and such in the database almost to the point of negative returns. A dirty little secret in the DBA world is that this affliction frequently extends to areas beyond performance tuning and can be referred to as “compulsive repetitive work disorder”. Most DBAs I work with are aware of their malady, but do not know how to break the cycle. They see repetitive work as something that regularly falls on their plate and they have no option but to carry out. Some of those activities may be partially automated, but overall, the nature of their work doesn’t change. In fact, they don’t know how to change, nor are they incented or empowered to figure it out.

Given this scenario, it’s almost unreasonable for managers to expect DBAs to change just because a new technology has been introduced in the organization. It almost requires a different management model, laden with heaps of work redefinition, coaching, oversight, re-training and to cement the behavior modification, a different compensation model. In other words, the surest way to bring about change in people is to change the way they are paid. Leaving DBAs to their own devices and expecting change is not being fair to them. Many DBAs are unsure how any work pattern changes will impact their users’ experience with the databases, and whether that change will cost them their jobs even. It’s just too easy to fall back to their familiar ways of reactive work patterns. After all, the typical long hours of reactive work shows one as a hardworking individual, providing a sense of being needed and fosters notions of job security.

In these tough economic times however, sheer hardwork doesn’t necessarily translate to job security. Managers are seeking individuals that can come up with smart and innovative ways for non-linear growth. In other words, they are looking to do more with the same team - without killing off that team with super long hours, or having critical work items slip through the cracks.

Automation is the biggest enabler of non-linear growth. With the arrival of the new year, it is a good time to be talking about models that advocate changes to work patterns and corresponding compensation structures. Hopefully you can use the suggestions below to guide and motivate your team to get out of the mundane rut and continue down the path of more automation (assuming of course, that you have invested in broader application/database automation platforms such as Data Palette that are capable of accommodating your path).

1. Establish a DBA workbook with weights assigned to different types of activity. For instance, “mundane activity” may get a weight of say 30, whereas “strategic work” (whatever that may be for your environment) may be assigned a weight of 70. Now break down both work categories into specific activities that that are required in your environment. Make streamlining and automating repetitive task patterns an intrinsic part of the strategic work category. Check your ticketing system to identify accurate and granular work items. Poll your entire DBA team to fill in any gaps (especially if you don’t have usable ticketing data). As a starting point, here’s a DBA workbook template that I had outlined in a prior blog.

2. Introduce a variable compensation portion to the DBAs’ total compensation package (if you don’t already have one) and link that to the DBA workbook - specifically to the corresponding weights. Obviously, this will require you to verify whether DBAs are indeed living up to the activity in the workbook by having a method to evaluate that. Make sure that there are activity IDs and cause codes for each work pattern (whether it’s an incident, service request or whatever). Get maniacal about having DBAs create a ticket for everything they do and properly categorize that activity. Also integrate your automation platform with your ticketing system so you can also measure what kind of mundane activity are being carried out in a lights-out manner. For instance, many Stratavia customers establish ITIL-based run books for repetitive DBA activities within Data Palette. As part of these automated run-books, tickets get auto-created/auto-updated/auto-closed. That in turn will ensure that automated activities, as well as manual activities get properly logged and relevant data is available for end-of month (or quarterly or even annual) reconciliation of work goals and results – prior to paying out the bonuses.

If possible, pay out the bonuses at least quarterly. Getting them (or not!) will be a frequent reminder to the team regarding work expected of them versus the work they actually do. If there are situations that truly require the DBAs to stick to mundane work patterns, identify them and get the DBAs to streamline, standardize and automate them in the near future so they no longer pose a distraction from preferred work patterns.

Many companies already have bonus plans for their DBAs and other IT admins. However they link those plans to areas such as company sales, profits or EBITDA levels. Get away from that! Those areas are not “real” to DBAs. IT admins rarely have direct control on company revenue or spending levels. Such linking, while safer for the company of course (i.e., no revenue/profits, no bonuses), does not serve it well in the long run. It does not influence employee behavior other than telling them to do “whatever it takes” to keep business users happy and the cash register ringing, which in turn promotes reactive work patterns. There is no motivation or time for IT admins to step back and think strategically. But changing bonuses and variable compensation criteria to areas that IT admins can explicitly control – such as sticking to a specific workbook with more onus on strategic behavior – brings about the positive change all managers can revel in, and in turn, better profits for the company.

Happy 2009!

PS => I do have a more formal white-paper on this subject titled “5 Steps to Changing DBA Behavior”. If you are interested, drop me a note at “dbafeedback at stratavia dot com”. Cheers!

Quantifying DBA Workload and Measuring Automation ROI

2008-09-07T22:50:00.000-06:00

I mentioned in a prior blog entry that I would share some insight on objectively measuring DBA workload to determine how many DBAs are needed in a given environment. Recently, I received a comment to that posting (which I’m publishing below verbatim) the response to which promoted me to cover the above topic as well and make good on my word.

Here’s the reader’s comment:
Venkat,
I was the above anonymous poster. I have used Kintana in the past for doing automation of E-Business support tasks. It was very good. The hard part was to put the ROI for the investment.

My only concern about these analysts who write these reports is that they are not MECE (Mutually Exclusive, Collectively exhaustive). They then circulate it to the CIO's who use it to benchmark their staff with out all the facts.

So in your estimate, out of the 40 hours a DBA works in a week (hahaha), how many hours can the RBA save?

The reason I ask is that repetitive tasks take only 10-20% of the DBA's time. Most of the time is spent working on new projects, providing development assistance, identify issues in poorly performing systems and so on. I know this because I have been doing this for the past 14 years.

Also, from the perspective of being proactive versus reactive, let us take two common scenario's. Disk Failure and a craxy workload hijacking the system. The users would know it about the same time you know it too. How would a RBA help there?

Thanks
Mahesh
========

Here’s my response:

Mahesh,

Thanks for your questions. I’m glad you liked the Kintana technology. In fact if you found that (somewhat antiquated) tool useful, chances are, you will absolutely fall in love with some of newer run book automation (RBA) technologies, specifically database automation products that comply with RBA 2.0 norms like Data Palette. Defining a business case prior to purchasing/deploying new technology is key. Similarly, measuring the ROI gained (say, on a quarterly basis) is equally relevant. Fortunately, both of these can be boiled down to a science with some upfront work. I’m providing a few tips below on how to accomplish this, as I simultaneously address your questions.

Step 1] Identify the # of DBAs in your organization – both onshore and offshore. Multiple that number by the blended average DBA cost. Take for instance, a team of 8 DBAs – all in the US. Assuming the average loaded cost per DBA is $120K/year, we are talking about $960K being spent per year on the DBAs.

Step 2] Understand the average work pattern of the DBAs. At first blush it may seem that only 10-20% of a DBA’s workload is repeatable. My experience reveals otherwise. Some DBAs may say that “most of their time is spent on new projects, providing development assistance, identify issues in poorly performing systems and so on.” But ask yourself, what does that really mean? These are all broad categories. If you get more granular, you will find repeatable tasks patterns in each of them. For instance, “working on new projects” may involve provisioning new dev/test databases, refreshing schemas with production data, etc. These are repeatable, right? Similarly, “identifying issues in poorly performing systems” may involve a consistent triage/root cause analysis pattern (especially many senior DBAs tend to have a methodology for dealing with this in their respective environments) and can be boiled down to a series of repeatable steps.

It’s amazing how many of these activities can be streamlined and automated, if the underlying task pattern is identified and mapped out on a whiteboard. Also, I find that rather than asking DBAs “what do you do…”, ticketing systems often reveal a better picture. I recently mined 3 months worth of tickets from a Remedy system for an organization with about 14 DBAs (working across Oracle and SQL Server) and the following picture emerged (all percentages are rounded up):

- New DB Builds: 210 hours (works out to approx. 3% of overall DBA time)
- Database Refreshes/Cloning: 490 hours (7%)
- Applying Quarterly Patches: 420 hours (6%)
- SQL Server Upgrades (from v2000 to v2005): 280 hours (4%)
- Dealing with failed jobs: 140 hours (2%)
- Space management: 245 hours (3.5%)
- SOX related database audits and remediation: 280 hours( 4%)
- … (remaining data truncated for brevity…)

Now you get the picture… When you begin to add up the percentages, it should total 100% (otherwise you have gaps in your data; interview the DBAs to fill those gaps.)

Step 3] Once I have this data, I pinpoint the top 3 activities - not isolated issues like dealing with disk failure, but the routine tasks that the DBAs need to do multiple times each week, or perhaps each day – like the morning healthcheck, dealing with application faults such as blocked locks, transaction log cleanup, and so on.

Also, as part of this process, if I see that the top items’ description pertains to a category such as “working on new projects…”, I break down the category into a list of tangible tasks such as “provisioning a new database”, “compliance-related scanning and hardening”, etc.

Step 4] Now I’m ready to place each of those top activities under the microscope and begin to estimate how much of it follows a specific pattern. Note that 100% of any task may not be repeatable, but that doesn’t mean it cannot be streamlined across environments and automated - even a 20-30 percent gain per activity has huge value!

Once you add up the efficiency numbers, a good RBA product should allow you to gain anywhere from 20 to 40 percent overall efficiency gains - about $200K to $400K of higher productivity in the case of our example with 8 DBAs - which means, they can take on more databases without additional head-count or, support the existing databases in a more comprehensive manner (step 5 below). These numbers should be treated as the cornerstone for a solid business case, and for measuring value post-implementation – task by task, process by process.

(Note: The data from Step 2 can also be used to determine how many DBAs you actually need in your environment to handle the day-to-day workload. If you only have the activities listed and not the corresponding DBA time in your ticketing system, no worries… Have one of your mid-level DBAs (someone not too senior, not too junior) assign his/her educated guess to each activity in terms of number of hours it would take him/her to carry out each of those tasks. Multiple that by the number of times each task is listed in the ticketing system and derive a weekly or monthly total. Multiple that by 52 or 12 to determine the total # of DBA hours expended for those activities per year. Divide that by 2,000 (avg. # of hours/year/DBA) and you have the # indicating the requisite # of DBAs needed in your environment. Use a large sample-set from the ticketing system (say, a year) to avoid short-term discrepancies. If you don’t have a proper ticketing system, no problem – ask your DBA colleagues to track what they are working on within a spreadsheet for a full week or month. That should give you a starting point to objectively analyze their workload and build the case for automation technology or adding more head-count, or both!)

Step 5] Now sit down with your senior DBAs (your though-leaders) and identify all the tasks/activities that they would like to do more of, to stay ahead of the curve, avoid some of those frequent incidents and make performance more predictable and scalable - activities such as more capacity planning, proactive maintenance and healthchecks, more architecture/design work, working more closely with Development to avoid some of those resource-intensive SQL statements, use features/capabilities in newer DBMS versions, audit backups and DR sites more thoroughly, defining standard DBA workbooks, etc.) Also specify how will that help the business – in terms of reduced performance glitches and service interruptions, fewer war-room sessions and higher uptime, better statutory compliance and so on. The value-add from increased productivity should become part of the business case. One of my articles talks more about where the additional time (gained via automation) can be fruitfully expended to increase operational excellence.

My point here is, there’s no such thing in IT as a “one time activity”. When you go granular and start looking at end-to-end processes, you see a lot of commonalities. And then all you have to do is summarize the data, crunch the numbers, and boom - you get the true ROI potential nailed! Sounds simple, huh? It truly is.

Last but not least, regarding your two examples: “disk failure” and “a crazy workload hijacking the system” – those necessarily may not be the best examples to start when you begin to build an automation efficiency model. You need to go with the 80-20 rule - start with the 20% of the task patterns that take up 80% of your time. You refer to your use cases as “common scenarios”, but I’m sure you don’t have the failed disk problem occurring too frequently. If these above issues do happen frequently in your environment (assuming in the short term) and you have no control over them, other than reacting in a certain way, then as Step 3 suggests, let’s drill into how you react to them. That’s the process you can streamline and automate.

Let me use the “crazy workload” example to expound further. Say I’m the DBA working the early Monday morning shift and I get a call (or a ticket) from Help Desk stating that a user is complaining about “slow performance”. So I (more or less) carry out the following steps:

1. Identify which application is it (billing, web, SAP, Oracle Financials, data warehouse, etc.)
2. Identify all the tiers associated with it (web server, app server, clustered DB nodes, etc.)
3. Evaluate what kind of DB is the app using (say, a 2-node Oracle 10.2 RAC)
4. Run a healthcheck on the DB server (check on CPU levels, free memory, swapping/paging, network traffic, disk space, top process list, etc.) to see if anything is amiss
5. Run a healthcheck on the DB and all the instances (sessions, SQL statements, wait events, alert-log errors, etc.)
6. If everything looks alright from steps 4 and 5, I update the ticket to state the database looks fine and reassign the ticket to another team (sys admin, SAN admin, web admin team, or even back to the Help Desk for further analysis of the remaining tiers in the application stack).
7. If I see a process consuming copious amounts of I/O or CPU on a DB server, I check to see if it’s a DB-related process or some ad-hoc process a sys admin has kicked off (say, an ad-hoc backup in the middle of the day!). If it’s a database process, I check and see what that session is doing inside the database – running a SQL statement or waiting on a resource, etc. Based on what I see, I may take additional steps such as run an OS or DB trace on it – until I eliminate a bunch of suspects and narrow down the root cause. Once I ascertain symptoms and the cause, I may kill the offending session to alleviate the issue and get things back to normal - if it’s a known issue (and I have pre-approval to kill it). If I can’t resolve it then and there, I may gather the relevant stats, update the ticket and reassign it to the group that has the authority to deal with it.

As the above example shows, many of the steps above (specifically, 1 to 7) can be modeled as a “standard operating procedure” and automated. If the issue identified is a known problem, you can build a rule in the RBA product (assuming the product supports RBA 2.0 norms) to pinpoint the problem signature and link it to a workflow that will apply the pre-defined fix, along with updating/closing the ticket. If the problem is not a known issue, the workflow can just carry out steps 1 to 7, update the ticket with relevant details there and assign it to the right person or team. Now I don’t need to do these steps manually every time I get a call stating “there seems to be a performance problem in the database…” and more importantly, if it’s truly a database problem, I can now deal with the problem even before the end user experiences it and calls the Help Desk.

In certain other situations, when Help Desk gets a phone call about performance issues, they can execute the same triage workflow and either have a ticket created/assigned automatically or solve the issue at their level if appropriate. This kind of remediation avoids the need for further escalation of the issue and in many cases, avoids incorrect escalations from the Help Desk (how many times have you been paged for a performance problem that’s not caused by the database?). If the problem cannot be automatically remediated by the DBA (e.g., failed disk), the workflow can open a ticket and assign it to the Sys Admin or Storage team.

This kind of scenario not only empowers the Help Desk and lets them be more effective, but also reduces the workload for Tier 2/3 admin staff. Last but not least, it reduces a significant amount of false positive alerts that the DBAs have to deal with. In one recent example, the automation deployment team I was working with helped a customer’s DBA team go from over 2,000 replication-related alerts a month (80% of them were false positives, but needed to be looked at and triaged anyway…) to just over 400. I don’t know about you, but to me, that’s gold!

One final thing: this may sound somewhat Zen, but do look at an automation project as an ongoing journey. By automating 2 or 3 processes, you may not necessarily get all the value you can. Start with your top 2 or 3 processes, but once those are automated, audit the results, measure the value and then move on to the next top 2 or 3 activities. Continue this cycle until the law of diminishing returns kicks in (usually that involves 4-5 cycles) and I guarantee your higher-ups and your end-users alike will love the results. (More on that in this whitepaper.)

Clouds, Private Clouds and Data Center Automation

2008-09-03T07:58:00.000-06:00

As part of Pacific Crest’s Mosaic Expert team, I had the opportunity to attend their annual Technology Leadership Forum in Vail last month. I participated in half-a-dozen panels and was fortunate to meet with several contributors in the technology research and investment arena. Three things seemed to rank high on everyone’s agenda: cloud computing and its twin enablers - virtualization and data center automation. The cloud juggernaut is making everyone want a piece of the action – investors want to invest in the next big cloud (pun intended!), researchers want to learn about it and CIOs would like to know when and how to best leverage it.

Interestingly, even “old-world” hosting vendors like Savvis and Rackspace are repurposing their capabilities to become cloud computing providers. In a similar vein InformationWeek recently reported some of the telecom behemoths like AT&T and Verizon with excess data center capacity have jumped into the fray with Synaptic Hosting and Computing as a Service - their respective cloud offerings. And to add to the mix, terms such as private clouds are floating around to refer to organizations that are applying SOA concepts to data center management making server, storage and application resources available as a service for users, project teams and other IT customers to leverage (complete with resource metering and billing) – all behind the corporate firewall.

As already stated in numerous publications, there are obvious concerns around data security, compliance, performance and uptime predictability. But the real question seems to be: what makes an effective cloud provider?

Google’s Dave Girourad was a keynote presenter at Pacific Crest and he touched upon some of the challenges facing Google as they opened up their Google Apps offering in the cloud. In spite of pouring hundreds of millions of dollars on cloud infrastructure, they are still grappling with stability concerns. It appears that size of the company and type of cloud (public or private) is less relevant, and more relevant is the technology components and corresponding administrative capabilities behind the cloud architecture.

Take another example: Amazon. They are one of the earliest entrants to cloud clouding and have the broadest portfolio of services in this space. Their AWS (Amazon Web Services) offering includes storage, queuing, database and a payment gateway in addition to core computing resources. Similar to Google, they have invested millions of dollars, yet are prone to outages.

In my opinion, while concerns over privacy, compliance and data security are legitimate and will always remain, the immediate issue is around scalability and predictability of performance and uptime. Clouds are being touted as a good way for smaller businesses and startups to gain resources, as well as for businesses with cyclical resource needs (e.g., retail) to gain incremental resources at short notice. I believe the current crop of larger cloud computing providers such as Amazon, Microsoft and Google can do a way better job with compliance and data security than the average startup/small business. (Sure, users and CIOs need to weigh their individual risk versus upside prior to using a particular cloud provider.) However for those businesses that rely on the cloud for their bread-and-butter operations whether cyclical or around-the-year, uptime and performance considerations are crucial. If the service is not up, they don’t have a business.

Providing predictable uptime and performance always boils down to a handful of areas. If provisioned and managed correctly, cloud computing has the potential to be used as the basis for real-time business (rather than being relegated to the status of backup/DR infrastructure.) But the key question that CIOs need to ask their vendors is: what is behind the so-called cloud architecture? How stable is that technology? How many moving parts does it have? Can the vendor provide component-level SLA and visibility? As providers like AT&T and Verizon enter the fray, they can learn a lot from Amazon and Google’s recent snafus and leverage technologies that can simplify the environment enabling it to operate in lights-out mode – making the difference behind a reliable cloud offering and one that’s prone to failures.

The challenge however, as Om Malik points out on his GigaOm blog, is that much of cloud computing infrastructure is fragile because providers are still using technologies built for a much less strenuous web. Data centers are still being managed with a significant amount of manual labor. “Standards” merely imply processes documented across reams of paper and plugged into Sharepoint-type portals. No doubt, people are trained to use these standards. But documentation and training doesn’t always account for those operators being plain forgetful, or even sick, on vacation or leaving the company and being replaced (temporarily or permanently) with other people who may not have the same operating context within the environment. Analyst studies frequently refer to the fact that over 80% of outages are due to human errors.

The problem is, many providers while issuing weekly press releases proclaiming their new cloud capabilities, haven’t really transitioned their data center management from manual to automated. They may have embraced virtualization technologies like VMware and Hyper-V, but they are still grappling with the same old methods combined with some very hard-working and talented people. Virtualization makes deployment fast and easy, but it also significantly increases the workload for the team that’s managing that new asset behind the scenes. Because virtual components are so much easier to deploy, it results in server and application sprawl and demands for work activities such as maintenance, compliance, security, incident management and service request management go through the roof. Companies (including the well-funded cloud providers) do not have the luxury of indefinitely adding head-count, nor is throwing more bodies at the problem always a good idea. They need to examine each layer in the IT stack and evaluate it for cloud readiness. They need to leverage the right technology to manage that asset throughout its lifecycle in lights-out mode – right from provisioning to upgrades and migrations, and everything in between.

That’s where data center automation comes in. Data center automation technologies have been around now for almost as long as virtualization and are proven to have the kind of maturity required for reliable lights-out automation. Data center automation products from companies such as HP (on the server, storage and network levels) and Stratavia (on the server, database and application levels) make a compelling case for marrying both physical and virtual assets behind the cloud with automation to enable dynamic provisioning and post-provisioning life-cycle management with reduced errors and stress on human operators.

Data center automation is a vital component of cloud computing enablement. Unfortunately, service providers (internal or external) that make the leap from antiquated assets to virtualization to the cloud without proper planning and deployment of automation technologies tend to provide patchy services giving a bad name to the cloud model. Think about it… Why can some providers offer dynamic provisioning and real-time error/incident remediation in the cloud, while others can’t? How can some providers be agile in getting assets online and keeping them healthy, while others falter (or don’t even talk about it)? Why do some providers do a great job with offering server cycles or storage space in the cloud, but a lousy job with databases and applications? The difference is, well-designed and well-implemented data center automation - at every layer across the infrastructure stack.

So, what's your “Database to DBA” Ratio?

2008-07-09T16:22:00.000-06:00

The “Database to DBA” ratio is a popular metric for measuring DBA efficiency in companies. (Similarly, in the case of other IT admins, the corresponding “managed asset to admin” ratio (such as, "Servers to SA" ratio in the case of systems administrators) seems to be of interest.) What does such a metric really mean? Ever so often, I come across IT Managers bragging that they have a ratio of “50 DB instances to 1 DBA” or “80 DBs to 1 DBA”... -- Is that supposed to be good? And conversely, is a lower ratio such as “5 to 1” necessarily bad? Compared to what? In response, I get back vague assertions such as “well, the average in the database industry seems to be “20 to 1”. Yeah? Sez who??

Even if such a universal metric existed in the database or general IT arena, would it have any validity? A single DBA may be responsible for a hundred databases. But maybe 99 of those databases are generally “quiet” and never require much attention. Other than the daily backups, they pretty much run by themselves. But the remaining database could be a monster and may consume copious amounts of the DBA's time. In such a case, what is the true database to DBA ratio? Is it really 100 to 1 or is it merely 1 to 1? Given such scenarios, what is the true effectiveness of a DBA?

The reality is, a unidimensional *and* subjective ratio, based on so-called industry best practices, never reveals the entire picture. A better method (albeit also subjective) to evaluate and improve DBA effectiveness would be to establish the current productivity level ("PL") as a baseline, initiate ways to enhance it and carry out comparisons on an ongoing basis against this baseline. Cross-industry comparisons seldom make sense, however the PL from other high-performing IT groups in similar companies/industries may serve as a decent benchmark.

Let's take a moment to understand the key factors that should shape the PL. In this regard, an excellent paper titled “Ten Factors Affect DBA Staffing Requirements” written by two Gartner analysts, Ed Holub and Ray Paquet comes to mind. Based somewhat on that paper, I’m listing below a few key areas that typically influence your PL:

1. Rate of change (in the environment as indicated by new rollouts, app/DDL changes, etc.)
2. Service level requirements
3. Scope of DBA services (do the DBAs have specific workbooks, or are the responsibilities informal)
4. # of databases under management
5. Database sizes
6. Data growth rate
7. Staff skills levels
8. Process maturity (are there well-defined standard operating procedures for common areas such as database installation, configuration, compliance, security, maintenance and health-checks)
9. Tools standardization
10. Automation levels

In my mind, these factors are most indicative of the overall complexity of a given environment. Now let’s figure out this PL model together. Assign a score between 1 (low) to 10 (high) in each of the above areas as it pertains to *your* environment. Go on, take an educated guess.

Areas 1 to 6 form what I call the Environmental Complexity Score. Areas 7 to 10 form the Delivery Maturity Score. Now lay out an X-Y Line graph with the former plotted on the Y-axis and the latter plotted on the X-axis.

Your PL depends on where you land. If you picture the X-Y chart as comprising 4 quadrants (left top, left bottom, right top and right bottom), the left top is "Bad", the left bottom is "Mediocre", the right top is "Good" and the right bottom is "Excellent".

Bad indicates that your environment complexity is relatively high, but the corresponding delivery maturity is low. Mediocre indicates that your delivery maturity is low, but since the environment complexity is also relatively low, it may not be a huge issue. Such environments probably don't see issues crop up frequently and there is no compelling need to improve delivery maturity. Good indicates that your environmental complexity is high, but so is your delivery maturity. Excellent indicates that your delivery maturity is high even with the environment complexity being low. That means you are truly geared to maintain service levels even if the environment gets more complex in the future.

Another thing that Excellent may denote is that your delivery maturity helps keep environment complexity low. For instance, higher delivery maturity may enable your team to be more proactive/business-driven, actively implement server/db consolidation initiatives to keep server or database counts low. Or the team may be able to actively implement robust data archival and pruning mechanisms to keep overall database sizes constant even in the face of high data growth rates.

So, now you have a Productivity Level that provides a simplistic, yet comprehensive indication of your team's productivity, as opposed to the age-old "databases to DBA" measure. Also, by actively addressing the areas that make up Delivery Maturity, you have the opportunity to enhance your PL.

But this PL is still subjective. If you would like to have a more objective index around your team's productivity and more accurately answer the question "how many DBAs do I need today?", there is also a way to accomplish that. But more on that in a future blog.

Six Ways to Tell if an RBA Tool Has Version 2.0 Capabilities

2008-05-01T23:14:00.000-06:00

Here’s an update to my last blog about RBA version 2.0. I promised a reader I would provide a set of criteria for determining whether a given RBA platform offers version 2.0 capabilities, i.e., the ability to create and deploy more dynamic and intelligent workflows that can evolve as the underlying environment changes, as well as accommodate the higher flexibility required for advanced IT process automation. In this piece, I expound mostly on the former requirement since my prior blog talked about the latter in quite some detail.

If you don’t have time to read the entire blog post, here’s the exec summary version of the six criteria:
1. Native metadata auto-discovery capabilities for target environmental state assessment
2. Policy-driven automation
3. Metadata Injection
4. Rule Sets for 360-degree event visibility and correlation
5. Root cause analysis and remediation
6. Analytics, trending and modeling.

Now, let’s look at each of these in more detail:

1. Native metadata auto-discovery capabilities for target environmental state assessment. In this context, the term “metadata” refers to data describing the state of the target environment where automation workflows are deployed. Once this data is collected (in a centralized metadata repository owned by the RBA engine), it effectively abstracts target environmental nuances and can be leveraged by the automation workflows deployed there to branch out based on changes encountered. This effectively reduces the amount of deployment-specific hard-coding that needs to go into each workflow. This metadata can also be shared with other tools that may need access to such data to drive their own functionality. Most first-generation RBA tools do not have a metadata repository of their own. They rely on an existing CMDB to pull relevant metadata or in the absence of a CMDB, the workflows attempt to query such data at runtime.

As I have mentioned in my earlier post, the existing metadata in typical CMDBs only scratches the surface for the kind of metadata required to drive intelligent automation. For instance, in the case of a database troubleshooting workflow, the kind of information you may need for automated remediation can range from database configuration data to session information to locking issues. While a CMDB may have the former, you would be hard-pressed to get the latter. Now, RBA 1.0 vendors will tell you that their workflows can collect all such metadata data at runtime, however some of these collections may require administrative/root privileges. Allowing workflows to run with admin privileges (esp for tasks not requiring such access) can be dangerous (read, potential security policy or compliance violation). Alternatively, allowing them to collect metadata during runtime makes for some very bulky workflows that can quickly deplete system resources on the target environments especially in the case of workflows that are run frequently. Not ideal.

The ideal situation is to leverage built-in metadata collection capabilities within the RBA platform to identify and collect the requisite metadata in the RBA repository (start with the bare minimum and then add additional pieces of metadata, as required by the incremental workflows you roll out). If there is a CMDB in place, the RBA platform should be able to leverage that for configuration-related metadata and then fill in the gaps natively.

Also, the RBA platform needs to have an “open collection” capability. This means, the product may come with specific metadata collection capabilities out-of-the-box. However if the users/customers wish to deploy workflows that need additional metadata which is not available out of the box, they should be able to define custom collection routines.

2. Policy-Driven Automation. Unfortunately, not all environmental metadata can be auto-discovered or auto-collected. For instance, attributes such as whether an environment is Development, Test or Production or what its maintenance window is, etc. are useful pieces of information to have since the behavior of an automation workflow may have to change based on these attributes. However given the difficulty in discovering these, it may be easier to specify them as polices that can be referenced by workflows. Second generation RBA tools have a centralized policy layer within the metadata repository wherein specific variables and their values can be specified by users/customers. If the value changes (say, maintenance window start time changes from Saturday 9 p.m. Mountain Time to Sunday 6 a.m. Mountain Time, then it only needs to be updated in one area (the policy screen) and all the downstream workflows that rely on it get the updated value.

3. Metadata Injection. Here’s where things get really interesting. Once you gather relevant metadata (either auto-discovered or via policies), there needs to be a way for disparate automation workflows to leverage all that data during runtime. The term “metadata injection” refers to the method by which such metadata is made available to the automation workflows - especially to make runtime decisions and branch into pre-defined sub-processes or steps.

As an example of how this works, the Expert Engine within Stratavia’s Data Palette parses the code within the workflows steps for any metadata variable references and then substitutes (injects) those with the most current metadata. Workflows also have access to metadata history (for comparisons and trending; more on that below).

Here’s a quick picture (courtesy of Stratavia Corp) that shows differences in how workflows are deployed in an RBA version 1 tool versus RBA 2.0. Note how the metadata repository works as an abstraction layer in the latter case, not only abstracting environmental nuances but also simplifying workflow deployment.

4. Rule Sets for 360-degree event visibility and correlation. This capability allows relevant infrastructure and application events to be made visible to the RBA engine – events from the network, server, hypervisor (if the server is a virtual machine), storage, database and application need to come together for a comprehensive “360-degree view”. This allows the RBA engine to leverage Rule Sets for correlating and comparing values - especially for automated troubleshooting, incident management and triage.

Further, an RBA 2.0 platform should be capable of handling not just simple boolean logic and comparisons within its rules engine, but also advanced Rule Sets for correlations involving time series, event summaries, rate of change and other complex conditions.

5. Root cause analysis and remediation. This is one of my favorites! Accordingly to varied analyst studies, IT admin teams spend as much as 80% of problem management time trying to identify root cause and 20% in fixing the issue. As many of you know, root cause analysis is not only time consuming, but also places significant stress on team members and business stakeholders (as people are under the gun for quickly finding and fixing the underlying issue). After numerous “war-room” sessions and finger-pointing episodes, the root cause (sometimes) emerges and gets addressed.

A key goal of second-generation RBA products is to go beyond root cause identification to automation of the entire root cause analysis process. This is done by gathering the relevant statistics in the metadata repository to obtain a 360-degree view (mentioned earlier), analyzing them via Rule Sets (also referred above) and then providing metrics to identify the smoking gun. If the problem is caused by a Known Error, the relevant remediation workflow can be launched.

Many first-generation RBA tools assume this kind of root cause analysis is best left to a performance monitoring tool that the customer may already have deployed. So what happens if the customer doesn’t have one deployed? What happens if a tool capable of such analysis is deployed, but not all the teams are using that tool? Usually, each IT team has its own point solution(s) that looks at the problem in a silo’d manner, which commonly leads to finger-pointing. I have rarely seen a single performance analysis tool that is universally adopted by all IT admin teams within a company and everyone working off the same set of metrics to identify and deal with root cause. If IT teams have such a tough time identifying root cause manually, should companies just forget about trying to automate that process? Maybe not… With RBA 2.0, regardless of what events disparate monitoring tools may present, such data can be aggregated (either natively and/or from those existing monitoring tools) and evaluated via a Rule Set to identify recurring problem signatures and promptly dealt with via a corresponding workflow.

All that most monitoring systems do is present a problem or potential problem (yellow/red lights), send out an alert and/or run a script. Combining root cause analysis capabilities and automation workflows within the RBA platform helps improve service levels and frequently reduces alert floods (caused frequently by the monitoring tools), unnecessary tickets and incorrect escalations.

Improving service levels – what a unique concept! Hey wait a minute, isn’t that what automation is really supposed to do? And yet, it amazes me how many companies don’t want to go there and just continue dealing with incidents and problems manually. RBA 2.0 begins to weaken that resistance.

6. Analytics, trending and modeling. These kind of capabilities are nice to have and if leveraged from an RBA context, can be really powerful. Once relevant statistics and KPIs are available within the metadata repository, Rule Sets and workflows should be able to access history and summary data (pre-computed based on predictive algorithms and models) to be able to understand trends and patterns to deal with issues before they become incidents.

These can be simplistic models (no PhD required to deploy them), and yet avoid many performance glitches, process failures and downtime. For instance, if disk space is expected to run out in the next N days, it may make sense to automatically open a ticket in Remedy. But if space is expected to run out in N hours, it may make sense to actually provision incremental storage on a pre-allocated device or to perform preventative space maintenance (e.g., backup and delete old log files to free up space, etc.), in addition to opening/closing of the Remedy ticket. A good way to understand these events and take timely action is to compare current activity with historical trends and link the outcome to a remediation workflow.

Run Book Automation Gets Smarter

2008-04-29T23:09:00.000-06:00

Over a year ago, I had written about different data center automation options, including Run Book Automation (RBA) or as some call it, IT Process Automation. Since then there’s been lots of activity in that space. Many IT organizations that had merely been curious about this kind of technology in earlier months have now begun to earmark specific budgets to evaluate and deploy these tools. I’m beginning to see more and more RFPs in this area. Even though some of the vendors in this space like iConclude and RealOps have been acquired and seem to have lost some of the core talent and drive behind the technology, this area is continuing to see a tremendous amount of innovation driven primarily by other startups. In fact, it almost seems that a new version of run book automation has evolved. Analysts are referring to the enhancements via glowing adjectives such as intelligent process automation, Run Book Automation version 2.0 and Decision Automation based on (the complexity of) the use case being automated (such as, disaster recovery automation in the latter case).

There appear to be primarily two catalysts for the emergence of RBA 2.0:
1. RBA 1.0 was too simplistic (and limiting). RBA by itself wasn’t meant to introduce any new automation functionality. It was merely designed to string along existing tools and scripts in the right sequence to automate specific low-level and mundane IT processes. Some prominent examples are automation of Help Desk (a.k.a. “Tier 1”) work patterns such as trouble ticket enrichment and basic alert-flow triage and response. Given that the primary user-base for this kind of technology were junior IT operators, the assumption was that they typically wouldn’t know much coding or scripting and would need generic out-of-the-box functions to automate things such as opening and closing tickets in popular ticketing systems, rebooting servers, changing job schedules, and so on. Also frequently these products wouldn’t even expose the source code within these steps for any kind of customization – all to keep complexity at bay. Depending on what you intended to accomplish, you end up buying different “integration packs” that allowed you to interact with specific toolsets that were already deployed within your environment and chain together the requisite steps.

2. RBA 1.0 was too static in nature. When you defined and rolled out a workflow, it made specific assumptions about the platform, version and state of the environment and the underlying toolsets that it connected to. If any of those underlying components changed, then all too often the workflow would cease to function in the manner expected, producing unreliable results and diluting the value of automation. Some of the RBA 1.0 products queried the state of the environment at runtime to avoid this problem, but that resulted in bulky workflow steps that consumed higher CPU, memory and I/O resources on the target environments (the impact was especially evident in the case of frequently run workflows).

Companies that had deployed RBA Version 1 began to run into these limitations as they attempted to move up the IT food chain - to Tier 2 and Tier 3 teams and their increasingly complex activities. Most Tier 2/3 areas, such as database administration, systems administration or application support called for these two deficiencies to be overcome.

Consequently RBA 2.0 showed up with specific enhancements to address the above two areas. (I'm going to lead with the solution for problem #2 since that's a wee more challenging.)
- More intelligent and dynamic workflows – Dynamic workflows evolve in a pre-defined (read, pre-approved) way as the environment undergoes changes. This is accomplished in RBA 2.0 via the introduction of a metadata repository between the automation workflows and the target environment. This metadata layer captures the current state of the target environment (along with historical data for comparison and trend analysis purposes) and injects this information into the workflows just prior to or during runtime (this process is referred to as “metadata injection”).

This centralized metadata repository gets populated via relevant collections – either natively or via integration with a CMDB if one exists in the environment. (Note: Even if a CMDB exists, native collections are still relevant since the metadata required for advanced process automation goes way deeper than the metadata found in a conventional CMDB. For instance, in the case of database automation, the type of collections include not only DBMS platform, version, patch level and configuration settings, but also functional aspects such as sessions logged into the database, wait events experienced, locks held, etc. – all of which may be required to either trigger or drive real-time automation behavior.)
Environmental attributes that cannot be auto-discovered can be specified within the metadata repository via a central policy engine.

Metadata injection is one of the key differentiators within RBA 2.0 to bring about requisite changes in automation behavior. This allows workflows to acquire a degree of dynamism (in a pre-approved way) - without getting bloated with all kinds of ad-hoc/runtime checks or worse, becoming stale (and having to be maintained/updated manually).

- Higher flexibility to accommodate higher complexity – RBA version 2 often exposes the source-code beneath the steps for editing and allows new steps to be added in any scripting language the user may prefer. It is not uncommon to see two disparate Tier 2 teams each with its own scripting language preference (if you don’t believe me, just look towards any Sys Admin team that manages both UNIX and Windows boxes or any DBA team that manages both Oracle and SQL Server… These preferences all too often have a tendency to get religious.) The ability to view code and modify the underlying workflow steps, along with support for disparate scripting languages (within the same workflow) allows admin teams to inspect out-of-the-box functionality and make relevant modifications to the existing workflow templates to fit their more advanced requirements.

Talking of products offering RBA version 2, Stratavia’s Data Palette is leading the charge in this area via its central metadata repository and decision automation capabilities. (The product just picked up another award today for making the top software innovators list at the Interop/Software 2008 event.)

Deploying RBA is a strategic decision for many organizations. Expect to have to live with your choice for quite some time. Before you place your bet on a specific solution, take a broad look at representative IT processes (if required, across multiple tiers/teams) you expect to automate today as well as, in the next 24 months, and ensure you are investing in a platform that comes closest to supporting your organization’s ambitions.

Data Palette Gets Better and BladeLogic Joins the BMC Camp

2008-03-18T10:07:00.001-06:00

It’s just Tuesday, and already it’s been a hectic week for data center automation with Stratavia announcing Data Palette 5.0 and BMC acquiring BladeLogic for about $800M.

Let me start with what's more relevant to customers and end-users: Data Palette 5.0 is something the Stratavia engineers have been working on for almost 2 years now (in conjunction with version 4). It is a game-changer in that it is the first data center automation (DCA) platform to combine native provisioning capabilities with run book automation and database automation to address DCA needs across multiple domains – the sys admins (with server provisioning and hardening requirements), the Tier 1/Help Desk (with generic run book automation requirements) and the DBAs/App admins (with more advanced database/application automation requirements).

The traditional DCA products like BladeLogic and Opsware have essentially focused on provisioning. In the case of Opsware, they started out with server provisioning and then expanded into network and storage provisioning by acquiring Rendition and Creekpath respectively. However these vendors quickly figured out that provisioning without process automation forms an incomplete offering. After all, most admins would like to do “something” with newly provisioned hardware – like harden the server, install specific applications on it, create user accounts and so on – all of which require process automation capabilities. To address that, Opsware last year acquired run book automation vendor, iConclude. BladeLogic addressed that problem via an OEM with run book automation vendor, RealOps (relabeled as Orchestration Manager) and when the latter got acquired by BMC, established an OEM arrangement with another vendor Opalis.

Sounds nice, huh? Well not so much for customers, because as anyone who has tried to integrate multiple products together knows, leveraging disparate tools in a seamless manner is not as easy as it sounds. There are always different GUIs, configuration styles and architectural components to reckon with, in addition to competing terminology, metadata and redundant functionality. Such integration challenges have been more pronounced in the case of BladeLogic especially, since with an OEM, you only have so much control over another vendor’s offering… But I guess BladeLogic has an excellent bunch of sales guys that have been able to sell into large organizations in spite of these challenges. Either that, or customers didn’t know they had an alternative.

With Data Palette 5.0, this hurdle towards a coherent integration of provisioning and run book automation capabilities is a thing of the past. Data Palette’s new console provides both set of capabilities that tie into each other via a common architecture that includes a shared console, policies, rules, metadata and consistent terminology. Now you can define templates for provisioning Windows, Linux and UNIX servers, along with corresponding CIS policies and apply them via a generic run book/workflow. You can use the same console to discover existing servers and leverage an automated run book to scan for compliance and configuration policy violations, report on them and optionally, remediate them. No more having to go from one product screen to the other and having to re-enter rules and preferences.

The other improvement is the thin client for both defining and deploying workflows. Both administration functions (such as defining workflows) as well as user functions (executing the workflows) can be accomplished via the web GUI, built primarily with Ajax and Flash. The new GUI implements the concept of a “digital whiteboard” with easy drag-and-drop of reusable workflows steps to illustrate a custom process. Each of the workflow steps are chained together via an input/output mapping that allows users to create new workflows without the need for scripting or having to study the code within each step. However if you are a scripter, the code is available for viewing/edits and you can add additional steps to the library in any scripting language with built-in version control (in fact, you can mix and match steps in different languages – all within the same workflow!!!).

Data Palette 5.0 also brings in the concept of custom policies. I realize different vendors seem to gravitate towards a different definition for the word “policy”. Within Data Palette, policies are a collection of common attributes that you can use to treat multiple “like” servers (or other data center assets) as one. For example, let’s say you have 40 Windows 2003 machines and 10 VMware Windows virtual machines that you need to apply PCI rules to. You can create a PCI policy outlining the relevant Windows-specific rules and attributes and then apply that singular policy to all 50 machines. Corresponding automation workflows can then utilize the policy attributes to log onto those servers, do the appropriate checks and report on all of them in a consistent manner. Finally, Data Palette’s original forte in database administration automation and 5.0’s innovation in terms of functionality and ease of use makes it stand head and shoulders above anyone else in terms of out-of-the-box database and application related content and capabilities.

BTW, in case you are somewhat wondering how Data Palette differentiates from BladeLogic and Opsware (both in terms of current capability as well as future direction), here’s a quick picture to represent each vendor’s competencies.

Both BladeLogic and Stratavia share some customers (BladeLogic Operations Manager used for server provisioning and configuration management, and Stratavia Data Palette utilized for database and application automation). They do have a good team that knows what they are doing. I would hate to see a good team and good technology disrupted. And looks like BladeLogic had no dearth of suitors. In closing, I hope BladeLogic gets its fair share of mindshare at BMC.

Taking a Stab at a Shared Industry Definition for a "DBA Workbook"

2008-03-10T12:00:00.000-06:00

Whenever I come across shops where business users, application support groups or infrastructure teams complain about the DBAs not meeting their expectations, the first question I ask is “OK, so what’s your DBA really supposed to do?”. And I’m amazed by the range of answers I get. It seems like there is quite a bit of vagueness regarding the DBA role. Just recently I blogged about measuring IT admin productivity, but in the case of DBAs, it appears there is not even a consistent set of expectations regarding what to expect from them, let alone measure their productivity.

Alright first things first! Has anyone come across a good DBA Workbook? I’m not even talking about a cookbook with step-by-step instructions on how to do different tasks, but just a preliminary workbook that documents exactly what the heck they are supposed to do.

So, what’s goes into an average DBA’s work-day? As I work at multiple companies, I see DBAs doing things all across the gamut - from ensuring backups are proper to tuning SQL statements to even writing stored procedures! It seems DBAs in many environments either gravitate towards what their predecessors used to do (and try to meet expectations that were set way before their time) or end up catering to whatever the loudest squeaky voice tends to ask. Sometimes I see DBAs being grouped into teams of specialists – like Operations DBAs (aka Systems DBAs), Application DBAs, Development DBAs and Engineering DBAs. I’m myself partial to having the DBAs segregated into not more than 3 groups: say, the Operations DBA team, a Development DBAteam and optionally, an Application DBA team (especially if the environment has large complex ERP/CRM type implementations). I don’t see the need for additional silos such as Engineering DBAs or Build DBAs; instead I view them as more of an area of focus for personnel within existing teams. For instance, each DBA team needs to have Tier 3 personnel that are responsible for Engineering-type DBA work like defining standardized database configurations. Anything more than 3 core teams tends to prevent economies of scale and can create workload imbalance in my experience (at any point in time, you may find one DBA group to be super-busy while another is relatively sitting idle or working on low priority activities). Similarly, creating too many silos can also lead to islands of institutional knowledge wherein none of the DBAs have a comprehensive understanding of the entire IT stack – from the database to the OS and storage, and all the way back to the application. A siloed lop-sided view can prevent effective root cause analysis during problem management.

It is also not uncommon to find sub-groups within the 3 main DBA groups based on DBMS platform (Oracle team versus SQL Server team, etc.) and OS platforms (UNIX DBAs versus Windows DBAs). The former sub-category tends to be more prevalent than the latter. However I’m increasing finding DBA managers interested in building cross-platform awareness. As leading DBMS platforms begin to display similar capabilities, databases are increasingly getting commoditized. Also with different 3rd party applications requiring multiple DBMS platforms as well as changing business requirements (say, the occasional M&A) casts more pressure on a DBA team to manage heterogeneous platforms. Furthermore, decision automation platforms like Data Palette help create an “abstraction layer” across physical DBMSs allowing DBAs skilled in one platform to be productive on others. So in the future, we may be seeing even less of a need for platform-based sub-categories.

Alright, so here’s a stab at a preliminary DBA workbook. Please feel free to adopt it as you see fit (since clearly there is no such thing as a “single size fits all” here.) This workbook is not supposed to be a precise guide for what DBAs in your specific organization should be working on, rather it’s meant to serve as a starting point template for you to customize per your business requirements. But do bear in mind that if you find the need to reinvent it completely, then perhaps you don’t really need a DBA, it’s some other role you are seeking to fill. So in other words, consider a 10-20% deviation from this definition as reasonable.

All you DBAs and DBA Managers out there – I would love to get your input on what you feel is right or missing here. Hopefully this can develop into a common definition of what should constitute a legitimate scope of work for DBAs. Such a definition sets expectations both within and outside the DBA team, and fosters higher collaboration. It should help managers analyze gaps in DBA scope of work in their own organizations, as well as identify gaps in skill-sets for future training and hiring. And the best part is that such a workbook can be linked to an SLA (that outlines the service levels the DBAs are responsible for achieving), and last but not least, be the pre-cursor for a more detailed cookbook comprising specific run-books for recurring (and some non-recurring yet complex) operations. The goal is to reduce surprises for other groups and avoid DBAs having to hear users and customers say things like “oh, I expected our DBAs to do more. In my previous company, they did so much more…”.

Also, if you ever venture into a DBA outsouring/co-sourcing arrangement with a 3rd party vendor, this can serve to delineate what you expect the vendor's DBAs to do versus your own team.

Operations/Systems DBA Workbook
- Database backups
- Database recovery
- Implementing robust change control and configuration management policies
- Capacity planning & system-level architecture
- OS/Storage/DB configuration and optimization
- Database security checks and audits, including compliance-related reporting & controls
- General database health checks
- Database monitoring
- Working with DBMS support (Oracle Support, Microsoft Support, etc.)
- Defining and maintaining quantitative SLAs and helping enhance current service levels
- Physical data model/architecture, DDL generation and maintenance
- DB server migrations and decommissioning
- Database creation and configuration
- Applying patches
- Upgrading the databases from version X to version X+Y
- Maintenance (proactive, reactive and automated); both scheduled and unscheduled
- Database refreshes
- Log-file review
- Trouble-shooting and repairs
- Reporting on state of databases
- Database tuning (proactive and reactive)
- Maintaining specialized environments such as log shipping, replication, cluster-based instances, etc.
- SOP (Std. Operating Procedure) definition & automation
- DBMS and related tools license management
- Documentation related to all Systems DBA areas
- Management of database-related tools (monitoring systems, tool repositories, etc.)
- Automated measurement of quantitative SLAs & any deviations thereof
- Development & utilization of detailed triage processes
- Detailed user workload analysis & segregation
- Infrastructure advise and planning
- Disaster recovery/failover services

Application DBA Workbook
- Setting up application/reporting environment
- Loading flatfile data into the database (script setup, automation & troubleshooting)
- Database cloning
- Application back-end process design and management
- Carry out application-specific trouble-shooting and reactive repairs
- Application configuration and setup
- New application implementation and upgrades
- Creating and implementing application security policy recommendations, including compliance-related reporting & controls
- Data management and manipulation. For example, ensuring data cleanliness, validating scrubbing rules, addressing batch job failures due to bad data (referential integrity violations, etc.)
- Perform application workflow analysis, characterization and segregation to reduce workflow and data-related failures
- Define triage and escalation procedures for each application failure such that appropriate parties can take ownership of specific issues during problem situations
- Liaison with 3rd party application vendors (e.g., Oracle/PeopleSoft Support) for support and trouble-shooting
- Work with in-house Development DBA and Development team in longer-term proactive application repair and re-architecture
- Application DBA related documentation and cross-training
- Assist QA in building and implementing test plans to validate and test application bug fixes, patches and upgrades
- Evaluate recurring/common application failures and define application “Repair SOPs” so that similar application failures get handled in a consistent manner.
- Strengthen application architecture for better performance and scalability and reduced failures - Working with the Systems DBAs on application/database integration (example: integrating two databases, etc.)
- Working with the Systems DBAs on configuring the database to optimally accommodate each application (example: providing consultative input on tasks such as data sizing, capacity planning, custom setup of memory structures and disk layouts, etc.)

Development DBA Workbook
- Specialized database SQL tuning
- Documentation related to all pre-production DBA areas and cross-training
- Participate in application and database design reviews
- New development projects implementation
- Logical data modeling and architecture
- Logical to physical design conversion
- SQL tuning
- Developing and deploying DDL/DML coding standards
- Coding (or recoding) of specific database modules for enhanced database/application performance
- Database code (DDL/DML) version control
- Scripting and testing schema builds and new rollouts
- DBMS software installation, configuration and management for development database environments as per standards set in Production (by Systems DBAs)
- Data cloning within development environments
- Training Developers on writing optimal SQL code
- General liaison between Systems DBAs and Developers
- General liaison between Systems DBAs and Application DBAs
- Change-control guidelines and provisions for new project implementations
- Assist QA in building and implementing test plans to validate and test pre-production DB bug fixes, patches and upgrades; as well as assist in stress/volume testing.

How Do You Measure IT Admin Productivity?

2008-02-21T09:56:00.000-07:00

A key part of my job is to carry out operational maturity assessments for different organizations. This helps me understand where a given IT admin team (usually DBAs, sys admins or app support teams) currently is (chaotic, reactive, proactive, etc.), where the business needs them to be, and how to bridge the gap in the shortest amount of time.

During the course of these assessments, I run into two kinds of IT administrators – the folks that view the extent of automation I (or their managers) advocate as a somewhat scary proposition and state that they wouldn’t want to jeopardize their environment by automating beyond a certain (rather basic) level. They claim to take complete personal responsibility for their environment and do what it takes to minimize risk in the day-to-day functioning of the company, by manually performing tasks that are considered critical to keeping the lights on. Let’s call this group the reluctant automators.

Then there are the folks that are eager to push the envelope to see how far and how fast their environments can embrace automation. Let’s refer to this group as the eager automators.

The latter group seems to view their jobs as something beyond keeping the lights on; they view their role as that of an enabler - wherein they allow business groups to focus on the organization’s core competency. Take for instance, a brokerage firm whose primary business function is to trade stocks profitably for their customers. This creates a primary layer of business users - the traders that trade the stock, and multiple peripheral teams that support the primary users and enable them to be successful. In most organizations where technology is not the core offering, IT admins tend to be a part of the peripheral layer.

Reluctant automators often fret about formal interactions with other groups and are usually disassociated from the primary business users. They view meetings as a time-sink; something that takes them away from their “real work” – i.e., doing things on their computers – things like moving data, keeping the configuration current, viewing log files, etc. – all relevant things, no doubt. They frequently view the keyboard activities as the “brainy stuff” and a meeting tends to be an activity where they can actually space out… (you know what I’m talking about, you have run into those folks with the vacant stares in meetings…)

The eager automators on the other hand, often state that repeatable things (the same ol’ stuff like moving data around, viewing log file output and updating configuration) leave them brain-dead and hence they can’t wait to automate them off their day-to-day workload (or at least semi-automate them such that they can be reliably offloaded to less senior people!) They manage the mundane by exceptions and use the freed up time to instead work on defining crisper user requirements and service levels, measuring deviations in performance and managing them. These kinds of activities require them to spend a lot of their time in meetings – with users, project teams and infrastructure peer groups. Even though some of these meetings are badly run (and are indeed boring), they have learnt to embrace these as opportunities to get better aligned with the business, understand growth trends , share insight and influence behavior.

So here’s the paradox that I (and I’m sure, countless IT managers) frequently run into. Given these two IT admin profiles, who is more productive? There is often a tendency in corporate America to mistake hard-work for higher productivity. But here, I don’t necessarily mean more hard-working in the sense of someone working 80 hours a week. Rather, who gets more stuff done? (Assuming there is a standard and reasonable expectation of “stuff” or job output/quality, in the first place.)

Both groups comprise ethical, hard-working people who usually put in way more than 40 hours per week to earn their salaries. However both camps are equally adamant that their outlook is the right one for the company.

I have my own thoughts on the matter (will post them in a future blog), but which school of thought do *you* subscribe to? And why is that appropriate? Unfortunately, there is no universally accepted HR manual that helps us decide whether one notion is better than the other.

And more pertinently, regardless of immanent viewpoints regarding individual employee performance, how can a manager quantitatively measure something as seemingly subjective as productivity? Good processes such as ITIL seem to help some by allowing work activities better documented by exposing # of service requests, # of incidents handled, etc. as well as time put in. However this information is often too uni-dimensional and does not share the complete story. For instance, it does not always take into account factors such as which incidents were truly avoidable and which service requests were redundant or occurred due to incomplete work done in the first place! And often, in many environments, IT admins get away with incomplete ticketing records. (So not all activities are logged into the ticketing system, and the information that gets logged is not always consistent or complete.)

Conventional IT management does not appear to provide an easy answer to these questions… how do you deal with it?

3 Predictions for MySQL Post Sun Acquisition

2008-01-22T08:10:00.000-07:00

The $1B acquisition of MySQL by Sun didn’t seem to get its fair share of publicity with the gargantuan $8.5B same-day acquisition of BEA by Oracle. Nevertheless it was a monster payout for MySQL, which finally has the opportunity come into its own. With Sun’s stewardship and products like Stratavia’s Data Palette providing automation functionality for MySQL, it finally has the opportunity to overcome its two primary weaknesses: (a) lack of proficient support tools and the resultant perceived lack of stability and scalability and (b) lack of personnel with adequate proficiency in managing enterprise class installations. These flaws have frequently relegated MySQL to a third-tier DBMS platform behind the usual suspects (Oracle, SQL Server, DB2), even legacy platforms (Sybase, Informix) and alas, nimbler open-source and pseudo open-source platforms (PostgreSQL and EnterpriseDB). The digerati have it pinned as being suited for merely small shops and/or small LAMP apps, in spite of some robust enterprise-class functionality in the newer releases.

With the acquisition, I predict 3 things to happen to MySQL in the next 12-24 months:
1. Enhanced native support (finally!) – More native tools for MySQL, especially better monitoring and ad-hoc GUI tools, which will greatly enhance DBA productivity and allow DBAs from other platforms to take on a more favorable disposition towards MySQL. Regardless, as in other DBMS platforms, there will always be gap between DBA demand and supply and MySQL-savvy automation products such as Data Palette will continue to be seen as a viable option to fill part of this gap.
2. More partner activity - With Sun’s partner-base being substantially larger than MySQL, more ISVs are likely to pay attention to the latter and begin to establish a tertiary tools eco-system – including better migration, upgrade and patching tools, data load/unload utilities and more robust maintenance and backup capabilities.
3. Higher innovation – With Sun’s reputation for innovation especially in areas open source, MySQL should come into its own with functionality matching and hopefully, surpassing the mainstream DBMS platforms (a smaller legacy footprint usually enables faster innovation…). Even table-stakes capabilities such as an enhanced SQL optimizer, partitioning and index-types for handling large data-sets without crashes/corruption would go a long way in building credibility for MySQL as a platform that can be relied on for mission-critical enterprise class deployments. These would also make it alluring for ISVs that are looking for more cost-effective options for embedding databases within their products. (I would think the latter would be an especially strong target market for MySQL, especially once the restrictive licensing issues are resolved…)

I still don’t expect a lot of automation capabilities for MySQL right out the chute. Funnily enough, I have been presciently upbeat about MySQL and had championed including out of the box support for MySQL within Data Palette a couple of years ago. While customers have appreciated that capability, ironically enough it was never a make or break deal for us (even with customers having some serious MySQL presence); just icing on the cake. However with the acquisition, I’m predicting MySQL will find more mainstream adopters for both itself and its supporting ecosystem making that early investment in MySQL automation capabilities worthwhile for Stratavia.

Go MySQL! And go Sun for stepping up to the plate! Pretty soon it’s going to be time to make it real… perhaps starting with fixing that open-source licensing model…?

Who is to blame for Oracle patch application failures... Is that even a real question?

2008-01-17T13:22:00.000-07:00

I just read a revealing blog entry by Jaikumar Vijayan about how two-thirds of DBAs miss timely and accurate application of Oracle’s quarterly critical patch updates and subsequent objections from some quarters regarding how this is showing DBAs in a bad light! Wow, I gotta say I sure don't agree with the latter view-point!

Honestly, I don’t think the article quite alludes that DBAs are to blame for this. However that is indeed my assertion. I have been in databases for over 16 years and as a Systems DBA, I consider myself and others like me to be the primary custodian of my company’s (and clients’) most important asset - data. To imply that somehow lack of corporate security standards in this area or lack of visible threats such as Slammer is sufficient grounds for ignoring patches is hare-brained in my humble opinion. I consider it my job to come up to speed on the latest patch updates, document the decision to apply the patch (or not!), educate and work with the security auditors (internal and external) if any, and ensure the databases I’m responsible for are totally secure. Not doing that is akin to saying, “oh, it’s okay for me not to test my database backups unless some corporate mandate forces me to do so.” Database patches, like auditing backups, forms one of the most basic job functions of a DBA. Claiming ignorance or being over-worked in other areas doesn’t count as an excuse, especially after your customers’ credit card data has been stolen by a hacker!

A comment to the above story by Slavich Markovich (one of the people quoted there), says that DBAs are not lazy and goes on to state that they just have too many things they are working on. Since when has a DBA not been “over-worked”? Way too often, that’s been part of the DBA’s job description – even though it doesn’t have to be that way. I know many fine DBAs that regardless of their work-load or corporate politics or prevailing state of ignorance, don’t let themselves be stopped from applying the relevant CPUs. They research the patchset, educate their peers, security groups and auditors, evangelize the benefits to application managers, arrange to test it in non-production and coordinate the entire process with change control committees to ensure its success.

Markovich lists several items in his comment that supposedly deter DBAs from adhering to a regular patch cycle. I’m taking each of his points one by one (in red font below) and responding to it:
1. The need to test all applications using the database is a heavy burden => yeah? so is coordinating and testing backups, deal with it!

2. Oracle supports only the latest patchsets => Oracle’s patch-set frequency and support policies tend to vary from version to version. Rather than making bland, generic statements, log onto metalink, do the relevant research, see what patch-set applies to which databases in your environment and go, make it happen!

3. The lack of application vendor certification of the CPUs => sure, certain CPU patches sometimes impact application functionality, but the majority don’t. Regardless, that’s what testing is for – to ensure your application functionality is not negatively impacted by a patch-set. If the testing shows no adverse impact, do a limited deployment and then move to a full deployment of that patch-set. CPUs are released by Oracle almost every quarter, so don’t expect all 3rd party vendors to update their certification matrix quite so rapidly. And BTW, most application problems that are suspected to have been caused by an Oracle CPU can be traced back to an error in the patch application process (why bother reading all the patch-set release notes, right?)

4. The simple fact that it takes a huge amount of work to manually shutdown the database and apply the patch in an organization running hundreds if not thousands of instances => if I had a nickel for every time I heard this… dude, ever read of run book automation for databases? If you are still stuck dealing with manual task methods and scripts, you only have yourself to blame. (Do yourself a favor, check out the Data Palette automation platform. )

5. For production critical databases you have to consider maintenance windows which might come once a year => ever heard of rolling upgrades/patches? If your environment isn’t clustered or lacks a standby environment, work with the application and change control committees to negotiate more reasonable maintenance windows or even better, build a business case for a Data Guard implementation. Remember, to be a successful DBA in today’s complex IT environment, you can’t just be a command-line expert, you need to possess good communication skills, not to mention salesmanship. Use those skills to jockey for reliable service levels – which includes, a well-patched and secure database environment. Don’t just attempt to live within the constraints imposed on you.

6. The lack of understanding by some IT security personnel of the severity of the problem simply does not generate enough pressure in the organization => quit blaming others, you are responsible for those databases, not some arcane person with a fancy title that has “security” somewhere in there.

7. All in all, I know of companies that analyze and deploy CPUs as soon as three months after release but those companies are very few and usually have budgets in the millions for such things… => also, quit generalizing. There are many DBAs that work for small, private companies with miniscule budgets that take their Oracle CPUs very seriously and vice versa.

The truth is, a secure, stable and scalable database environment has very little to do with the size of the budget and everything to do with astute DBAs that think outside the box and take charge of their environment.

Gartner Show Highlights Need for More Intelligence in Data Center Automation

2007-12-28T08:42:00.000-07:00

Gartner’s recent Data Center Conference was a blast! Between technical sessions, booth duty and a joint presentation on Data Center Economics with a client (Xcel Energy), I managed to sneak in a round of golf at Bali Hai (first time I ever played with a caddie…). While attendance at the event (3 to 4K maybe) was certainly not the highest in recent events, the conference sufficiently put the spotlight on 3 key areas: virtualization, automation and green data centers (in that order). Having been seen as the red-headed stepchild for the longest time (as compared to virtualization), it was good to see automation finally beginning to garner its fair share of attention (and IT budgets!).

There were dozens of tools and technologies with an automation theme. Not surprisingly, automation is also basking in the glory of the ripple effects from virtualization and green data centers. With virtualization boosting the number of “server environments”, each of which need to be individually monitored and managed, automation is being increasingly regarded as the other side of the virtualization coin. Similarly, with the awareness for green data centers increasing, I’m beginning to see innovation even in areas as obscure as automating the hibernation/shutdown of machines during periods of non-use to reduce energy consumption.

So given the plethora of tools and products, is every automation capability that is needed out there already invented and available? Not so. One of the biggest things lacking is “intelligence” in automation. Take for example run book automation toolsets. While many of them offer a nice GUI for administrators to define and orchestrate ITIL processes, they inherently introduce a degree of rigidness as well in the process. For instance, if there are 3 run books or process workflows that deal with server maintenance, the Maintenance Window is often hard-coded within each of the workflows. If the Window changes, all 3 of the workflows have to be individually updated. That creates a significant maintenance overhead. Similarly, if a workflow includes a step to open a ticket in BMC Remedy, the ticketing system’s details such as product type and version are often hard-coded. If the customer upgrades the Remedy ticketing system or migrates from Remedy to HP Peregrine, the workflow doesn’t function anymore! An intelligent process workflow engine would avoid these traps.

An intelligent automation engine is often characterized by the following attributes:

Centralized policy-driven automation – Policies allow human input to be recorded in a single (central) location and made available to multiple downstream automation routines. The Maintenance Window example above is a great candidate for such policy-driven automation. Besides service level policies (such as the Maintenance Window), areas such as configuration management, compliance, and security are well suited for being cast as policies.

Metadata injection & auto-discovery – Metadata is data describing the state of the environment. It is important for automation routines to have access to state data and especially be notified when there is a state change. For example, there is no point in starting the midnight “backup_to_tape” process as a 4-way stream when 2 of the 4 tape drives have had a sudden failure or are offline. The automation engine needs to be aware of what is available so it can launch dependent processes to optimally leverage existing resources. Such state data can be auto-discovered either natively, or via an existing CMDB, if applicable.

Event correlation and root cause analysis – Ability to acknowledge and correlate events from multiple sources and being able to leverage this information to identify problem patterns and root cause(s) would make automated error resolution more precise.

Rules processing – Being able to process complex event-driven rule-sets (not just simple boolean logic) allows triggering of automation in the right environments at the right times.

Analytics and modeling – Being able to apply dynamic thresholding as well as, analytical and mathematical models to metadata to discern future resource behavior is key for averting performance hiccups and application downtime.

During the Gartner show, I walked by many of the booths looking at what these supposedly “bleeding edge” vendors offered. Suffice it to say, I wasn’t dazzled. Rather than providing just asinine glue (which unfortunately, is what most of the current crop of process automation and orchestration tools are reduced to) to piece together multiple scripts and 3rd party tool interfaces and refer to it as “automation”, the customers I work with are increasingly interested in seeing offerings that leverage the above capabilities. Other than Data Palette, I don’t know of any products that do so. But then, you already knew that!

Data Palette 4.0 is out! Why should you care?

2007-10-17T23:32:00.000-06:00

On October 1, Stratavia released version 4 of Data Palette. I have been looking forward to this event for the following reasons:

1. Version 4 extends Data Palette’s automation capabilities to areas outside the database - to cover the entire IT life-cycle. This extension centers around three core areas:

provisioning, including server, storage and application provisioning;
service request and change management automation (e.g., operating system patches, application rollouts, system upgrades, user account maintenance, data migration, etc.); and
alert remediation and problem avoidance.

2. It expands Data Palette’s predictive analytics to storage management and capacity planning.

3. Further, it allows using these predictions, combined with root cause data and event correlating rule sets to enable decision automation for fast-changing heterogeneous environments.

I realize this release of Data Palette casts a rather wide net in the data center automation realm. However this is not mere bravado from Stratavia and it is certainly not mere coolness (“coolness” being a bunch of widgets that make for good market-speak, but don’t quite find their way into customer environments). If anything, it reflects the fact that data center automation is just not doable without a broad spectrum of capabilities. For an automation strategy to succeed, it simply needs to cast a comprehensive net, connecting all the diverse areas of functionality via a shared back-bone (the latter being especially key).

Let’s look at a couple of contrasts to understand this better. Take Opsware for example. They started out with basic server provisioning and then realized there’s more to automation. So they gobbled up a few more vendors (Rendition Networks for network provisioning, Creekpath Systems for storage provisioning and iConclude for run book automation) that couldn’t quite hack it on their own. Then before Opsware could expand any further, they ended up being bought by HP.

Another example is BMC – a relatively late entrant to the IT automation space. After wallowing in enterprise monitoring and incident management tool-sets for years, BMC made headlines a few months ago when it acquired run book automation vendor RealOps. It followed that up with the recent purchase of network automation provider Emprisa. I’m sure we will continue to see more purchases from HP, BMC and others as they continue to fill out the automation offerings in their portfolio (and try to stick them all together with bubble-gum, prayers and wads of marketing dollars!)

Buying a bunch of siloed (read, disjointed) tools does not an automation solution make! Why? Because there is no shared intelligence across all these tools. There is no singular policy engine commonly referenced by all of them. There are no central data collection, event detection and correlation abilities. When you define policies or rules in one tool, you often need to repeat the same definition in different ways in the other tools, significantly increasing maintenance overhead. When a policy or rule changes, one needs to go to N different places (N being all the locations where they are defined separately) – assuming it is even feasible to recall exactly all the areas that need to be updated. If you have more than a couple of dozen automation routines in place, good luck trying to reconcile what these things do, what areas they touch and what policies are duplicated across them.

In the case of Data Palette, this issue is addressed by its central Expert Engine architecture. This engine allows shared use of policy and metadata definitions. They get defined once and can be referenced across multiple automation routines and event rule sets. In other words, all the different areas of functionality and touch-points speak the same language.

Stratavia has dubbed its 4.0 release as the industry’s “most intelligent” data center automation platform. Obviously intelligence can be a subjective term. I actually see it as the most comprehensive data center automation platform of its kind (look Ma, no bubble-gum…).

Taking a Stab at a Shared IT Industry Definition of "Data Center Automation"

2007-09-02T03:15:00.000-06:00

The problem with certain grandiose terms such as "IT automation" and "data center automation" is that they have no shared definition across vendors in the IT industry. The only thing that's common is their repeated reference by multiple sources, all in different contexts and scope. They become part of the hype vernacular generated by different vendors and their marketing machines and eventually a word’s true meaning becomes irrelevant. In such a state, everyone thinks they know what it is, but no one really does and alas, it becomes so ubiquitous that people don't even bother challenging each others' assumptions regarding its scope.

The term “automation” is rapidly free-falling into just such a state in the IT industry. So here's me taking a stab at level-setting the meaning and scope for automation that (in my humble opinion) is capable of bringing both customers and vendors to the same playing field. Even if not, if it generates some cross-vendor discussion and allows people to challenge each other's perception of what data center automation ought to mean, my purpose would be served.

In a prior blog entry, I refer to 15 specific levels of requirements that need to be addressed for any IT automation solution to be effective. So ladies and gents, here’s that requirements stack. If these requirements are satisfied, you would have reached a state of automation nirvana and somewhere along the way, you would have imbibed (and likely, surpassed) the isolated automation capabilities touted by most IT tools vendors.

The accompanying picture shows Levels 1 to 8 as a pre-requisite to automation, and works itself all the way up to level 15 (autonomics) via a model that fosters shared intelligence across disparate capabilities and functions. It is important to get the context right with each of these levels before assuming one has attained them. And until one has got a particular level right, it is often futile to attempt to go to the next level. Short-cuts have an uncanny way of short-circuiting the process. That's why you find so many vendors and organizations with fragmented notions of these requirements just not cutting it in real-world automation deployments. Specifically because such approaches lack a sound methodology to build the proper foundation.

Nuff said. Let’s look at the individual levels now and how they build on one another.

Level 1 pertains to achieving 360-degree monitoring. This capability breaks through the typical siloed monitoring that exists today in many environments and allows problems to be viewed across multiple tiers, applications and service stacks in a cohesive manner –preferably, in the same call sequence utilized by the end-user application. Current monitoring deployments often remind me of the old fable of the seven blind men and the elephant (where each blind man would feel a different part of the elephant and perceive the animal to resemble a familiar item. For instance, one would touch the elephant’s tail and spread the notion that the animal looks like a rope, whereas another person would feel the elephant’s foot and try to convince everyone that the animal resembled a pillar). Lack of a comprehensive and consistent view of the same issue by multiple individuals and groups cause more delays in solving problems than the typical IT manager would admit.

Levels 2 and 3 utilize 360-degree monitoring for proper problem diagnosis, triage and alerting . Rather than leaving the preliminary diagnosis regarding nature and origin of a problem to human hands and eyes (say, a Tier 1 or Help Desk team), the monitoring software should be able to examine the problem end-to-end, carry out sufficient root cause analysis, narrow down the scope based on the analysis and send a ticket to the right silo/individual. Such precision alerting reduces the need for manual decision-making (and chances of error) regarding which team to assign a ticket to and positively impacts metrics such as first-time-right and on-time-delivery.

Level 4 pertains to ad-hoc tasks that administrators often do, such as adding a user or changing the configuration parameters of an application. There are lots of popular tools and point solutions for Levels 1 to 4 ranging from monitoring tools to ad-hoc task GUIs. However their functionality really ends there (or they try to jump all the way from Level 4 to Level 9 (Automation), skipping the steps in between and causing the resultant automation to be of rather limited use.)

Level 5 questions the premise of an “ad-hoc task”. Wisdom from the trenches often tells us that there is no such thing a one-time / ad-hoc task. Everything ends up being repeatable. For instance, when creating a user on one database, one may bring up her favorite ad-hoc task GUI, click here, click there, type in a bunch of command attributes and then hit the Execute button. That works for creating one user. However when the exact same user needs to be rolled out on 20 servers, it involves a lot of pointing, clicking and typing and leaves the environment vulnerable to human errors. Suddenly the ad-hoc GUI ceases to be effective.

Requirement levels 5 to 9 address this by calling for standard operating procedures (SOPs) that can be applied across multiple environments from a central location. Level 6 requires diverse sub-environments to be categorized such that different activities pertaining to them (including all service requests and incident remediation efforts) can all have standard task recipes. This refers to rolling up disparate physical environments into fewer logical components based on policies and usage attributes including service level requirements.

Task recipes, once defined, need to be maintained in a central knowledgebase and need to be in a format wherein they can easily serve as a blue-print for any subsequent automation. Further, the task recipes need to be directly linked to automation routines/workflows in a 1:1 manner such that one can reach the workflows from the SOP and vice versa. In other words, there needs to be a shared intelligence between the SOP and the automation routine. Keeping the two separate (for instance, keeping the task recipes on a SharePoint portal and keeping the automation routines within a scheduling or workflow tool, or worse, as a set of scripts distributed locally on the target servers) without any hard-wired connection and tracking between the two mediums will make it easy for task recipes and automation routines to get out of sync. Such automation tends to be uncontrolled; each user of such automation is left to his/her own devices to leverage it as optimally as possible, and eventually, its utility becomes questionable at best with each user (administrator) customizing the automation routines to their particular environment and their individual preferences. Hitherto noble notions such as shared intelligence and centralized control across task recipes and automation routines (and eventually, consistency in quality of work) go out the window!

With shared intelligence comes the ability to track and enforce standard task recipes across different personnel and environments. Level 10 states this exact requirement - the ability to maintain a centralized audit trail describing when an automation routine ran, on what server, who ran it (or what event triggered it), what were the run-results and so on – for ongoing assessment of SOP quality, pruning the task recipes and corresponding automation code, and finally, to ensure enterprise-wide adherence to best practices.

Level 11 or virtualization allows automation to be applied in a easier manner across the different environment categories defined in level 6. It does so via a “hypervisor” that abstracts and masks the different nuances across multiple categories by having a SOP call wrapper applying a task across different environment types (ideally, done via an expert system). Within that single SOP call, auto-discovery capabilities identify the current state of a target server or application, evoke a decision tree to determine how to best perform a task and then finally, call the appropriate SOP (or sub-SOP, as the case may be) to carry out the task in the pre-defined and pre-approved manner that’s most suited to that environment. In other words, level 11, much like storage virtualization that occurs within a SAN (think EMC Invista, not VMWare), calls for multiple disparate environments to be viewed as a single environment (or at the very least, a smaller subset of environments) and dealt with in an easier manner.

Levels 12 to 15 allow increasingly sophisticated levels of analysis and intelligence to be applied to a target environment, including event correlation, root cause analysis, and predictive analytics to be able to discern problems as they occur or ideally, before they occur and link those back to one or more SOPs which can be automatically triggered to avert an outage, performance degradation or policy violation and retain status quo, thereby making the target environment more autonomic.

Granted that many companies and administrators may not have the appetite to go all the way up to level 15 for most administrative tasks. But regardless, I would hope that this model provides a clear(er) roadmap for automation than just different groups and individuals scrapping together a bunch of proprietary scripts and Word documents, and keeping relevant information on how to exactly execute them within their heads or worse, each group relying on a bunch of disjointed tools, vendors and promises, resulting in a fragmented automation strategy and chaotic (read, unmeasurable) results.

Maybe unmeasurable results are not such a bad thing, especially if you are a large and well-established software vendor, who has little value to deliver, but thrives on FUD to keep the market confused.

If you are an IT manager or administrator and have a favorite software tools vendor, ask them how their automation strategy stacks up against these 15 levels. Drop me a note if you get an answer back.

Huge Disparity Between Generic IT Automation Trends and Database Automation Initiatives

2007-08-09T20:38:00.000-06:00

Let me start out with an apology for having been somewhat of an irregular blogger over the last few months. The good thing is, I haven’t been slacking off (at least not the entire time). Stratavia’s sustained growth has kept me on my toes. We just closed our Series B round and brought in several more customers. I have been working to set up an internal automation task force (ATF) within some of the larger customers to target areas and activities that would yield the most efficiencies via standardization and automation. This ATF effort has been received well as it is helping larger organizations gain value more rapidly from Data Palette and their overall automation efforts (in weeks rather than months). Also, I was fortunate enough to be able to present two papers at Collaborate 2007 in Vegas (If you attended one of those sessions – thank you!) and helped with Stratavia’s sponsorship of the recent Gartner Symposium in San Francisco.

At the symposium, I was rather surprised to see the space reserved for “data center operations” bustling with several new vendors and their automation offerings. Having focused on database automation for a while and being only too familiar with the typical DBA shop’s lukewarmness in embracing process automation, I thought the overall IT automation space was still nascent. Boy, was I mistaken!

As I compare generic IT automation trends and the prevailing action (or lack thereof) in the database automation arena, I’m literally filled with despair for the latter. At the Collaborate 2007 event, the sessions that drew the most attendees seemed to be either on Oracle RAC (real application clusters) or performance tuning – which tells me, the typical DBA is still enamored with learning how to configure clustered databases and/or turning performance knobs. She is in unchartered territory when it comes to process and decision automation – in fact, most DBAs still seem to be unfamiliar with the term run book automation (a sad statistic… especially in this day and age). Their knowledge of automation seem largely limited to “lights out” monitoring and patching – the kind provided by toys… oops! I mean tools like Oracle’s Enterprise Manager/Grid Control and Microsoft MOM. (Hello wake up, it’s not the 1990s anymore!)

On the other hand, there’s been a ton of interesting events in the generic IT automation space, as evidenced with the recent $1.65 Billion purchase of Opsware by HP and the acquisition of RealOps by BMC. The only downside to these transactions is that some of these products or rather, the capabilities in these product portfolios will likely die a premature death as the considerably larger acquirers lumber along, without knowing how to properly position and market these new-found competencies. (I just can’t see the larger vendors spending the requisite time and effort to evangelize and market these offerings effectively. Maybe I’m wrong and with these 800-pound entities entering this space, they will bring more legitimacy and avoid the need for an “evangelical sale” altogether. That would be a nice position for the IT automation industry to be in.) Hopefully the data management and database administration spaces will catch up with the rest of the IT automation industry soon.

Anyway, generic IT automation is not without its share of problems. The biggest one I see is that many CIOs continue to equate IT automation to just provisioning and patching. Heck, there’s so much more and by limiting the scope of automation, companies are not gaining the full value of automation. When you limit your scope, you inherently begin to seek and attract point solutions. No doubt, you need to start somewhere, and provisioning is as good an area as any to get started, but try not to be too narrowly focused and also, don’t just sit there basking in the results of a provisioning deployment for months and years. Get your Phase 1 scoped and deployed, measure results, make the appropriate tweaks and then move on to the next phase and the next set of “high-bar” tasks. And most importantly, make sure the automation platform you select has the capability to accommodate these all of these high-bar activities, without the need to acquire more tools.

The advent of run book automation (which BTW, is a nice descriptive name for process automation coined by Gartner) has really taken overall IT automation to the next level. Readers of my prior blog entries have heard me rave about decision automation. I personally see decision automation as being the next evolution of run book automation. It is akin to “run book automation on steroids” – the latter referring to capabilities such as auto-discovery, metadata injection, root cause analysis and predictive analytics – things that are rather essential for truly automating decisions. Decision automation takes run book automation beyond vanilla process orchestration or being merely the glue for carrying out ITIL-based activities in a specific sequence by incorporating a model for extracting and leveraging collective intelligence across processes.

Over the last year, Stratavia has been marrying run book automation with decision automation capabilities, especially in and around the database. That is proving to be a real game changer for many customers because a huge gap has existed in this area. Interestingly, many of Stratavia’s customers tend to be companies that already have provisioning and run book automation tools such as Opsware and RealOps, but have used them in limited/isolated areas such as server patching, software releases, and so on. By augmenting run book automation with root cause analysis capabilities (which is beyond the typical provisioning and run book automation tools’ competency), companies can better relate to recurring and disruptive problem patterns and reduce the number of incidents by converting reactive alerts to proactive maintenance processes. That effects typical SLA metrics like on-time-delivery and first-time-right, and makes the business more agile via speedier project deployments and enhanced time-to-market. The CIOs of these organizations are not content with merely rolling out an enterprise monitoring and ticketing solution, but are relentless in wanting to see themselves better aligned with the Service-Oriented or Value-Driven levels of Gartner’s operational maturity model and the advantages it brings their customers and employees.

Unfortunately this is not a universal trend yet. Many companies have yet to look at IT automation in a holistic way. The right way to go about it is not to jump directly into automation and address a point area or task and then basking in your laurels. I respectfully suggest the right way to automate comprises 15 distinct steps. And these steps have to be implemented in exactly the right order. Companies and products that try to short-cut the process (for example, by going directly to automation without first building standard and centralize operating procedures) end up not realizing the value they aimed for. (What is this “15-step” business, you ask? Well, more on that in a future blog entry...)

So what are the vendor/product choices for companies looking to implement IT automation the “right way”? I would urge you to consider Stratavia’s Data Palette without the slightest hesitation. A recent article in TheDeal (available at www.thedeal.com; the site requires a subscription to view content) has Gartner’s David Williams describing Stratavia as the only company he knows of, that brings run book automation capability to databases. (The exact statement in the article is “Stratavia makes run book automation software for an IT database, or software that enables a user to define a set of processes and integrate the necessary components to execute it automatically, according to Gartner Inc. analyst David Williams. The six-year-old company is one of the only companies Williams said he knows of to focus specifically on the underlying database.”; If you are interested in the entire article, drop me a line.) I don’t know about you, but to me, that’s a revealing statement from a well-respected analyst. And BTW, that’s what some of my customers are telling me as well - after evaluating other monitoring tools and database automation point solutions such as the Oracle Enterprise Managers and Grid Controls of the world, they are choosing to move ahead with a Data Palette deployment.

Do you know of any other process automation or decision automation products for databases? If so, please point me towards them; I would love to learn more and write about them.

An “ideal database health check” – what does that really mean?

2007-04-08T19:54:00.000-06:00

A good friend of mine, a senior Oracle DBA, recently looked at the automated database health check report we provide our customers and pointed out an interesting fact. He said "out of the 40 or so metrics you guys look at, at least 10 are outdated!" When I asked him to expound further, he responded “well, your report is still looking at cache hit ratios and such. The prevailing sentiment in the DBA community (especially the Oracle DBA community) is that hit ratios are useless. What truly matters are the wait stats and trace output.”

Now, does it really? I queried about a dozen of my other DBA friends on what they thought and unfortunately, it was a mixed message. Hours of Google searches showed a disappointing trend: the term “health check” is somewhat abused and there are no two similar health check utilities out there. Every DBA group that uses one seems to kinda make one up on the fly. I wish there was a standard health check that one could rely on.

Anyway, my limited research makes me wonder – why isn’t there an easily available standard health check? And more importantly, what really constitutes an “ideal” health check? Here are my two cents on the topic:

An ideal health check should provide an overview of a database’s stability across three major areas for consideration:

Availability
Performance
Scalability

Now, one could argue that all 3 of the above areas just point to one more area: performance. However from a practical standpoint, I would like to treat these 3 areas as distinct. For instance, in my mind, the well-being of a hot standby maintained via a Data Guard physical configuration is required for ensuring high availability, not necessarily for performance; whereas, ensuring sufficient buffer caches have been configured helps facilitate optimal performance moreso than availability. Also, in certain cases, performance and availability related actions may be at odds with one another. For instance, one may configure a redo-log switch every 10 minutes to ensure the primary database and hot standby are within 10-15 minutes of each other (helps availability), whereas the same action may result in higher I/O and negatively impact performance (especially if the system is already I/O saturated).

So, going back to the “ideal healthcheck”, one should ensure that there are no existing or upcoming issues that are impeding or are likely to impede any of these three areas. Below is a “top 40” list of things that would help ensure that is indeed the case. Comments are placed in-line, next to the statistic name/type, in italics.

Category: Availability
1. Database space (Should consider both database space and OS space; would be ideal to also consider growth trends.)
2. Archive log space
3. Dump area space (bdump, cdump, adump, udump, etc.)
4. Archive logs being generated as per SLA (Note: This requires the health check utility to have an understanding of the SLA pertaining to standby database upkeep.)
5. Archive logs being propagated as per SLA
6. Snapshot/Materialized view status
7. Status of DBMS Jobs
8. Replication collisions
9. Backup status
10. Online redo logs multiplexed (On different mount-points.)
11. Control file multiplexed (On different mount-points.)
12. Misc errors (potential bugs) in alert.log

Category: Performance
13. Disparate segment types (tables, indexes, etc.) in same tablespace
14. SYSTEM tablespace being granted as default or temp tablespace
15. Temporary tablespace not being a true temp tablespace
16. Deadlock related errors in alert.log
17. Non-symmetric segments or non-equi sized extents in tablespace (For dictionary managed tablespaces.)
18. Invalid objects
19. Any event that incurred a wait over X seconds (“X” to be defined by user during healthcheck report execution. Default value could be 5 seconds. Obviously, for this value to be available, some kind of stats recording mechanism needs to be in place. In our case, Data Palette is used to collect these stats so the health check report can query the Data Palette repository for wait events and corresponding durations.)
20. Hit ratios: DB buffer cache, redo log, SQL area, dictionary/row cache, etc. (While there is a mixed opinion on whether these are useful or not, I like to include them for DBAs that do rely on them to identify whether any memory shortage exists in the database instance and adjust the related resource(s) accordingly. While I do have an opinion on this matter, my goal is not to argue whether this stat is useful or not, instead, it’s to provide them to people that need them (and there are quite a few folks that still value hit ratios.)
21. I/O / disk busy (I/O stats, at the OS and database levels.)
22. CPU load average or queue size
23. RAM usage
24. Swap space usage
25. Network bandwidth usage (Input errors, output errors, queue size, collisions, etc.)
26. Multi-threaded settings (Servers, dispatchers, circuits, etc.)
27. RAC related statistics (False pings, cache fusion and interconnect traffic, etc. – based on the Oracle version.)

Category: Scalability (Note: For ensuring there are no scalability related issues, the health check generating mechanism ideally should be able to relate to current resource consumption trends and apply predictive algorithms to discern whether there will be contention or shortfall. In the absence of such predictive capabilities, a basic health check routine can still use thresholds to determine whether a resource is close to being depleted.)

28. Sessions
29. Processes
30. Multi-threaded resources (dispatchers, servers, circuits, etc.)
31. Disk Space
32. Memory structures (locks, latches, semaphores, etc.)
33. I/O
34. CPU
35. RAM
36. Swap space
37. Network bandwidth
38. RAC related statistics (False pings, cache fusion and interconnect traffic, etc. – based on the Oracle version.)
39. Understanding system resources consumed by non-DB processes running on the same server/domain (3rd party applications such as ETL jobs, webservers, app servers, etc.)
40. Understanding system resources consumed by DB-related processes running outside their normal scheduled window (Applications such as backup processes, archive log propagation, monitoring (OEM) agents, etc. This requires the health check utility to know which processes are related to the database and their normal execution time/frequency.)

The health check utility can show the above areas in red, yellow or green depending on whether any statistics needs urgent attention (red), needs review (yellow) or is healthy (green). Accordingly, the health of the database can be numerically quantified based on where each of the statistics show up (a statistic being in green status earns it 1 point, a yellow earns it 0.5 and a red earns it 0).

The above list can be expanded to accommodate additional configurations such as a custom standby setup or database audit requirements, and statistics pertaining to more complex environments such as RAC and Advanced Replication. Even entire categories can be added as appropriate. For example, “Security” would be a good category to add with statistics pertaining to user related information including account lock outs, and password change status, tablespace quotas, audit usage, virtual private database configuration, and so on.

Database and instance level health checks need to be shown separate, regardless of RAC. Things that need to show up at the database level would be statistics such as database space, etc., whereas statistics such as memory settings and hit ratios would be in the Instance health check report.

Lastly, it would be nice if the health check could include a recommendation on how to resolve a statistic that’s showing up in yellow or red, or even better, offer a link to an automated standard operating procedure that would help fix the situation.

I have meant this to be an open list. So any thoughts from the community on further augmenting (or better yet, standardizing) this list would be appreciated.