There is a common misconception that Remedy ARS is itself slow since so many administrators and customers have experienced this problem. Further, when these problems are discussed with other administrators it reinforces these misconceptions.

However, this is false. Remedy is not inherently slow. I have seen as many environments that have been performing well as those with performance problems. This begs the question what is different between these environments? What is causing all these performance problems which are perpetuating this myth?

Answering these questions in depth would require a detailed analysis of each specific environment to determine the basic limiting factors, bottlenecks, and analyzing the use and data volumes. Yet, in the majority of cases I have seen- the underlying culprit has been the database.

When a Remedy ARS Administrator starts to analyze a performance issue, the tendency is to work from the top down. This is a perfectly acceptable method to troubleshoot a performance issue. It will first rule out performance issues on the client.

Next they check the mid-tier and finally they will check the ARS application server. If these components are causing performance problems it is usually easy to spot. High memory usage, high CPU use, a large number of memory hard faults are all easily checked and provide a clear indicator of a performance problem. It is possible to have performance issues in the middle tiers beyond these types of issues. However, these limitations are generally not reached until there are very substantial loads from hundreds of concurrent users submitting tens of thousands of tickets each month.

At this point, analysis usually stops. One of two things will occur next. Either the administrator blames the problem on the ITSM applications or on the underlying Action Request System simply being inherently slow. Alternatively, they will open a ticket with support because ITSM and ARS are having problems.

In reality, what is usually occurring is latency at the database layer. For many Remedy Administrators the database is simply a black box where ARS stores all of its data. This is not the case, as the database is the single most important component in determining performance for any ARS application, including BMC Remedy ITSM.

Remedy ARS performs many queries for each action. Looking up a single record of data is a minimum of two database queries and can potentially be many more depending on the type of action being performed. In particular ARS is built on metadata for storing the structure of ARS applications. This metadata structure results in many small database queries occurring very frequently. When these frequent small queries are combined in with larger core data queries- even a minor delay or latency in receiving a response from the database will have a significant impact on the overall performance.

Analysis of the database performance and removing the bottleneck is a complex task and requires a set of skills that most Remedy administrators do not possess. This is why it is extremely important for Remedy administrators to work closely with their database administrators.

The following tools are available to help improve the database performance:

• Monitoring the database
• Maintaining indexes
• Sizing the database servers and storage
• Increasing database storage I/O

When these considerations are overlooked or neglected, the database will continue to degrade and age. Storage arrays will fill up, data files and indexes will become fragmented, or usage simply grows beyond what the database is able to satisfactorily manage.  Any one of these problems can significantly impact the database performance hereby causing performance problems with Remedy ARS.

Yes- you are hearing it again- planning is KEY.

I can almost guarantee that the greatest challenge you will encounter will be fostering new behaviours. As they say… “Old habits die hard.” My experiences with four large public sector enterprises tell me that this is especially true of the ITIL Release & Deployment Management process.

If you’re bigger-than-a-breadbox, and any organization besides a heavily funded start-up, you’re already in the thick of it. Your team of specialists beaver away, coding changes to your front-end applications while others are playing catch-up with middleware vendor releases and even more are chasing operating system patch strategies.

On a small scale, things just seem to work out. The “specialists” are a handful of Jack and Jill’s of all trades. However, this model has remarkable inefficiencies when it’s scaled up to a large enterprise. It doesn’t matter how good the teams are. The well oiled machine will almost always slow down or enter the exceptionally unfortunate state of “pants on fire… all… the… time.”

Management eventually asks “What is your improvement plan?”

Thankfully, there’s an entire industry dedicated to making documentation for your “plan” – aka an “ITIL Release and Deployment process improvement strategy”. Not having to re-invent the wheel is a good thing, but no process is an island. Realizing business value from R&DM will take a lot more than just enabling the “release management” feature on your favourite ITSM software solution.

And just like that… we’re back to the “changing behaviours” issue.

The experienced, well-trained and hard working people in your server-admin, e-business platform, and application development groups are very likely already heavily involved in “planning, packaging, testing, updating records, and deploying” (aka R&DM).

What you likely don’t have is:

1. Good communication between those groups,
2. Accepted schedules that provide windows of opportunity for o/s, middleware and application updates that don’t step on one another,
3. Reporting… of any kind.
4. Documented procedures that everyone can fall back on when things get confusing.

Based on my experiences, I STRONGLY recommend that you dedicate a resource for release management to shepherd the flock and take the time required to check off points 1, 2, 3 & 4 above.

Your specialists generally don’t need to know “how to do release management”- they are already doing it on a daily basis. What your dedicated resource should help with is changing behaviours and bringing your teams together so R&DM is a coordinated process. Do this and you can avoid a lot of wasted effort and cancelled releases.

So yeah, build up an R&DM process. There are great flow charts describing the whole process ready for you to copy and tell your teams to follow. This is an important first step.

When you’re sick of hearing how it’s “just not possible in our environment” consider taking the time to attend to the process of building new behaviours through effective organizational change management.

• People- Culturally is your organization ready for such a program?  Is management ready to embrace the shift from operational reporting and oversight of IT to services focused management and reporting?
• Process- And what about the process side of implementing ITSM? Is your organization unique thus requiring process modeling and re-engineering?

These and many other questions are important when starting on a path to IT operational excellence.  However, is there another way?

Technology

From a technology stand point, we can see that there are over 35 products that are validated and verified for ITIL v3 compliance [Pink Verify].  Many of these ITSM tools have been around for a dozen or more years and through processes such as customer feedback and industry best practices they have evolved their offerings to represent the very best in ITSM process compliance. Maybe these tools offer a way for some organizations to cut to the core of ITSM improvement and see more tangible rapid results?

Implementing an ITSM program with a tools first approach means that you are setting out to adapt the out-of-the-box best practices within your organization.   And while at first this seems like a potentially dangerous path, organizations have been doing this for years.  Monitoring tools, desktop suites, operating systems, and many more software packages are pre-set with processes and procedures built in.  So maybe there is room to consider the implementation of ITSM programs that are lead by the technology.

In my experience, organizations who implement ITSM tools such as BMC ITSM Remedy with an “out-of-the-box” mind-set  as opposed to tool-customization tend to yield greater returns and shorter ROI.  Starting with the point of view of “let’s try the tool the way it was intended to work” and train our users to use the tool, organizations not only shorten their time to deployment, but also often find that the ITIL and industry best practices which are baked into the tool are more than sufficient for their needs.  This in-turn yields a greater impact on the longevity of both the program and ITSM success.

When considering improving the state of your ITSM program, try looking to the industry leaders in ITSM tool technology.  Their business is making sure their tools are easy to use, follow best practices, and support core ITSM processes and procedures in hundreds of organizational situations such as yours.  It only makes sense to leverage their hard work and investments for your organization’s gain.

I have worked on many reporting projects and the reports I generally see are all standard, straightforward operational reports. The common trend is to use reports which simply look at counts of incident, changes, or problems apply some basic queries and filters and add some basic analysis, averages and percentages. These are your “classic” number of open incidents, queue reports, volume reports, and possibly getting into a little more depth with aging reports.

These reports are valuable to the day to day operations of a service desk, change advisory board, support group leads and IT managers. Providing this type of view of current and recent activity is important to ensuring that incidents are being resolved and changes are being correctly approved and evaluated. It is also useful for finding patterns of incidents to determine when a new problem investigation is required.

However, ITSM provides the capability to get so much more data at your fingertips. Making use of the service catalogue to track incidents against business service offerings, track service level agreement compliance and associated charge backs are a few. Tracking the costs associated with maintaining legacy systems can also help you determine at what point it becomes cost effective to initiate a replacement project.  These are just a few examples of how IT can make use of the in-depth data that ITSM is capable of tracking.

Even with advanced tools like BMC Analytics for BSM, the reports I have seen have not advanced much despite the power, access to data, and the ease of creating custom reports. Instead, the tendency when upgrading is to simply assume that the previously existing reports provide all the metrics and data that is required moving forward. So, in every case where I have implemented BMC Analytics, I have been asked to re-develop their existing reports, providing the same ol’ charts and metrics. While these existing reports are likely still valuable, proper analysis is needed to determine if the report is still applicable to today. Does the old report even make sense if it was developed for a different tool which did not follow the same processes used in ITSM?

Clients always agree with me when I tell them how useful the advanced reports could be.  They see the grand picture and think the tools are amazing.  Yet, despite this the answer is always “that is something we will look at in the future” or “it is not something that they need right now”. What they request are the reports that their managers, and support leads are accustomed to seeing.

This inertia is the largest challenge. It is important to realize that the best time to make changes is when implementing ITSM, or upgrading to a new version. Since changes are already being made it is easier to change the way the managers and support leads look at reports now than in a year when they have become more accustomed to using the same old reports on the new version of the system.  I highly recommend when upgrading or implementing to really dive into the existing reports you have and take the time to look into possible advanced reporting possibilities.  The data reporting possibilities are endless and it can transform your data into true insight.

If you are a large IT organization, effective alignment to ITSM standards will make it possible to support your internal investment decisions with real data.  In this case I’m talking about ITIL, or COBIT or any of the relevant ISO standards.

If you define your services (no small task), and build the agreements that enable those services – based on a standards model – you’ll be miles ahead of the pack in finding new ways to gain efficiencies or achieve greater levels of effectiveness. Your IT organization can navigate the rushing-waters of day-to-day business and also make the jump to building greater value by evaluating your IT Service CPRs (costs, performance and risk metrics) or comparing the CPR between services that are competing for scarce resources in your company.

There’s plenty of fodder out there to build a slide deck with “cost per incident” data that speaks to the benefits of effectively managing an incident life cycle or establishing the governance over IT changes that will effectively mitigate change-related risk. However, unless you have already “drunk the ITIL kool-aid” remaining focused on the incident-management model is not enough to demonstrate the value of aligning with a best-practice reference model.

We call it SERVICE management for a reason. While ballooning support costs and inefficient incident life cycles are significant challenges for a large IT organization, the focus needs to be on defining services and letting those definitions drive the value-proposition behind a better alignment to standards.

There are two reasons for this:

1. If you fully understand a service, you can identify the tech assets that support it. Those assets become the Configuration Items (CIs) destined to be included in a Configuration Management database (versus just including all assets) Transition processes (i.e. Change, Release) should only be concerned with CI impact. Incidents should be related to CI’s and therefore, services. If you don’t define your services… you can’t relate an incident (change, or problem) to a specific service.
2. If you document a service… you can quickly calculate:
• The Costs required to maintain that service,
• A consolidated Performance metric, by aggregating the performance of all associated CI’s,
• And you can “Risk model” all the gates in the service.

For example, if we define “documenting a service” as producing a complete, annotated work-flow diagram, isolating all stage-gates and stakeholders, we can then establish operating level agreements (aka: MOUs) with the owners of each gate, and then we will have the information we need to build the “service capability” portion of a service level agreement.

• With this information we can produce reasonable estimates of all associated costs (Note: the biggest bit will be the associated salary proportions).
• Use a standard measure to gauge performance (i.e. time. Read: how long does a transaction take to complete the work flow?).
• Now look towards any decent risk-register model (google “risk register”… there’s a wealth of examples out there) to document the associated risks and issues.

Now armed with Cost, Performance and Risk data… you are well positioned to compare a service to itself over time, or to compare one service to another.

Imagine being able to present Cost, Performance and Risk data on all the services that IT offers the organization. This is a decision-support gold mine!

1. Strength and stability for the future.

In my early years, (while still living at home) I was taught, as many of us were, to carve out a small portion of our pay cheques and put it into a savings account for a ‘rainy day’.  These small savings would slowly accumulate over time, compound and grow and eventually provide you with financial strength and security.  The days of the savings accounts have certainly gone, however the principal of saving a little each day still applies, and applies well to seeing the value of the CMDB. The little bits that are saved each day adds up by having the right information at our fingertips.  It compounds and grows and eventually it changes the IT organization’s operating culture, providing strength and security for it’s decisions, actions and credibility.  The CMDB is like that savings account, a little bit over time adds up.

2. Your focus needs more focus.

Many projects have split personalities, trying to be multiple things to different stakeholders.  Like a many headed hydra, it’s very difficult to keep your project driving in a straight line and consequently very difficult to demonstrate progress.  Not to mention when focusing on one head, you don’t know if another is about to wrap around and bite your behind.  Keep the CMDB project focused on the goals and objectives by continually relating the daily tasks and activities to those goals.  If you cannot directly relate an activity on the project to a goal or objective- it should be questioned, and likely removed. The most successful projects have had laser sharp focus and direction.

3. Simple steps can sound the loudest.

I clearly recall my first visit to the urban-musical STOMP.  I was memorized with the raw power of the production of sound from so many ‘regular’ items.  The show starts with a single simple rhythm, brump, brump, brump, brump… and builds adding more simple sounds until you can hardly sit still in your seat.  Starting simple with clear sound an entire symphony is built and created.  It is this concept that has significant impact on projects that are hard-starts such as the CMDB.  By taking small steps that demonstrate the potential success of the CMDB and slowly adding and building on those steps, an entire orchestrated movement can be created that launches the project.  Small steps can be very loud so don’t be afraid to take small ones.  They form the rhythm of the project and success is built on those never ending small steps.

Final thoughts

Thomas Wolfe said, you can’t go home again, and I think he might have said the same thing about creating a CMDB.  Once you’ve started on your way to a CMDB, the world that you knew before changes and there is no looking back.  By making simple steps with clear focused direction you’ll create small savings that demonstrate progress and create a successful CMDB.  Thought it might have been better said, you’ll never want to go home.

Yet, many of the ITSM administrators I have worked with keep having the same issue. “I am trying to modify a field, but all the options are disabled. I changed to base development mode and I was able to make my change.”

By customizing the field in base development mode and not as an overlay negates all the benefits that overlays provide. I have found that the problem is always a missing view overlay. When a field overlay is created a view overlay and a form overlay are also created automatically. And in most cases nothing else is required to complete the customization.

However, if a different view is selected, then no additional view overlay is created. Also if more than a single view is being customized then all the other views must also have overlays. These additional overlays must be created manually from the form menu.

Understanding this is simple if there are other changes required on a different view, or if the change has to be localized to another language. What is not clear is that there are many other field options which also require changes to a fields view properties.

A perfect example is adding a new selection attribute to a selection field. Because the selection attribute also defines the alias value on a per view basis, the function to add a new attribute will be disabled until all views have an overlay.

By simply checking that each view on a form has an overlay can avoid most issues working with form overlays or resolve them in a couple of clicks. This will save hours of frustration and a call to support to find why that option is disabled. I am all for avoiding that critical support ticket to find out why installing the latest ITSM version has wiped out your customization on production server.

When I engage with a new or existing client to explain a complete Change Management model I am met with a blank look and the words, “Yeah, we’re not ready for that.” Or “that wont work here” followed by an endless stream of examples crafted to demonstrate how things are “different here.” Then I spend a few minutes biting my tongue, resisting temptation to remind clients that they are not as different or as special as they think.

Here’s the thing: The ITIL books talk about organizations that exist with… nothing.   ITIL, version 1 through to version 3, have painted their picture of ITSM on a blank canvas or a “green field”. In that fictional world, we are told to:

1) Identify our inventory and relate it to services (Configuration Management),
2) Build a governance model to protect those services (Change Management), and
3) Establish gates to control the state of those services (Release Management).

The reality, as demonstrated by typical private and public sector clients, is that some kind of Release Process exists, and it’s erroneously called Change Management. They have a “Release Team” that they call “The CAB”.  There is often little evidence of any effort to establish a dialogue with representation from the business side of their house in support of their theoretical “IT Change Management.”

Now add to this picture and endless stream of vendors coming to their door to sell them a CMDB. They behave as if they have one in their brief case… and it comes in two colors. This is the stuff of great frustration and, more importantly, great distraction.  I propose to my clients that the CMDB can live on a spreadsheet while they get sorted on Change, Release and possibly Event management.  If (and that’s a big “if”) Transition Management processes begin delivering a measurable return on investment then perhaps it’s time to think about paying for a CMDB. Step one would be to tie the spreadsheet of data on configuration items to a module supporting their Operational Processes (Service Desk: incident management).

Don’t get me wrong… a CMDB is a wonderful thing, but it’s expensive, complicated and absolutely without value until there is an effective set of processes in place supporting Transition and Operations Management. That’s a much larger subject – worthy of it’s own blog post – but for now, lets at least agree that the CMDB doesn’t come first (unless you live in Imagination Land with a completely “green field” landscape in IT services).

Getting back to Change Management… and the whole “You’re calling release management – Change Management”. This has got to stop. LOL.

Look… the really important issue is this: Organizations are developing and implementing these processes without ever spending any time or effort on understanding what services they are actually providing. If you haven’t documented a service, you can’t associate IT assets to it (the service) and you don’t have a value proposition to justify exactly what is to be governed by an IT Change Management paradigm.

All you’re really doing with your undocumented services and a so-called “CAB” meeting is considering your preparedness to introduce a change to the live environment… aka: Release Management.

Typical clients turn to ITSM consulting for support and guidance when a pain-point has broken through the façade of ITSM practices. It would be nice to get pulled into the game before choices are made, but such is life … and the lack of a green field.

Reaching for IT Change Management success is absolutely not simply calling a meeting a “CAB” and making everybody fill out paperwork when they want to make adjustments to the production environment. This will lead to endless conversations about whether or not changing someone’s email profile constitutes a “change”.

No matter where you are on the “Change Continuum”, no matter what “amount” of Change Management your company has an appetite for, it is paramount that a “full-on” picture of Transition Management be presented. It should show the alignment of the process to various pain points in your organization, and – most importantly – an alignment to the benefits or process rewards. Now you can use that model to identify your organization’s “process appetite” and decide just what you’re going to engage in…