What’s The Problem?

Events can be misleading. Consider an example where several servers are behind a switch. We’ll further assume that we are monitoring the availability of the switch and the servers. When the switch goes down, what happens? A ton of notification is sent alerting everyone that all the servers are down, which is effectively true, but isn’t really the problem. Of course eventually the switch down alert comes in with all the server down messages. This is a simple example, where most good engineers will immediately diagnose the problem when they read the switch down alert, but a lot of messages were sent to notify you of the true problem. I always cringe when I know my boss is getting flooded with email that the sky is falling. Now, what if we use some logic in our notification that only sends out server down messages when the switch is OK, and suppresses all the server down messages when the switch goes down? That would be useful. Even better, let’s configure the switch down message to inform recipients with the list of servers that are unavailable due to the switch being down.

The switch example is easy to understand. Think about how useful correlating much more complex events can be, especially when critical information is included in the notification. Fixing a problem is a lot easier if the problem email or text message includes a concise description of the multiple conditions, what the root cause is, how to fix it, and who to communicate with for help. I’ve even worked with customers to include links to their own online SOP or Help Desk documentation. In my experience with problem email alerts, less is always better, as long as you always get all the notification you need. Correlation of events is the only way to simultaneously reduce the email count and dramatically improve the quality of information in alerts.

Logically Speaking

A Correlated Event is going to have multiple conditions. Sometimes all conditions must be true. We also want to be able to recognize when some conditions are true, while others are definitely not true. We may even want to specify that some conditions must be true, while others may be true, and some others must not be true (a really complex event…). We’re really only using three Booleans, AND, OR, and NOT, where we group the logically similar conditions. The logical order of listing the conditions should be:

1. Conditions that Must Be True
2. Conditions that May Be True
3. Conditions that Must Not Be True

The number of conditions can vary, but most complex problems can be recognized based on between 2 and 10 conditions. More may be needed when deducing problems along an extensive network path or across multiple application servers, for example.

I find it useful to include some synchronization and the awareness of persistence when correlating events. First, all conditions might not be based on measurements with the same interval. For example, disk statistics are typically collected hourly, whereas availability is tested every 1 to 5 minutes. Many other conditions will have intervals between these two extremes. I may want to specify that all conditions must happen with a specified interval. It’s also really valuable to be able to specify that X number of events must have happened in the interval. Picture network latency that gets flaky sometimes, but if it persists for X amount of time then it’s a problem.

Intelligent Notification

Once we recognize a complex problem based on the presence or lack of specific conditions we’re in a position to provide effective notification that will maximize the probability that a problem is fixed as quickly as possible. You’re setting up yourself and those around you for success. Here’s how I recommend configuring Correlated Event notification:

1. Describe the condition set
   1. what it means
   2. what’s the root cause
2. Describe the procedure to fix the problem
   1. Links to documentation
3. List the interested parties to contact for help
   1. ISP Contacts
   2. Network Admins
   3. Server Admins
   4. Application Support

How To Do It

You can build a BAT file, VB, Shell, or Perl script to use a CASE test using the Booleans described above, but you’ll have to build an interface to the database of events. You can even use a well crafted query to select for the conditions of interest. If you use Longitude, then you can just use the Correlated Event actions to define the multiple conditions, interval, and persistence, build your notification. Please email me if you have questions about using Correlated Events.

Good and Bad SLAs

Let’s start by poking fun at what will be the worst example of an SLA you’ve ever heard of or that I’ve been a party to implementing. I should point out this happened long before I became part of the Heroix team. I was brought in to design and implement a monitoring and reporting regime that supported the SLA between a web hosting provider and a Wall Street firm seeking its first web presence. Considering the big-time clients and huge capital expenditure (100 servers in two data centers), I was expecting a challenging assignment with a highly sophisticated and complex set of monitoring requirements. When I received my copy of the SLA, a single paragraph appendix to a large contract, it had one condition:

• No server shall experience greater than 30% average CPU usage during any rolling hour

You could have knocked me over with a feather. After disbelief, hilarity, and confusion, came concern and agitation. I actually suggested that we provide much more, which would have been included in any basic monitoring regime, and was rebuffed. What’s obviously wrong with this SLA is that the measure of success has no direct connection with actual service delivery to consumers. It was, however, very easy to measure. So the primary rule in creating effective SLAs is:

1. Measure things that directly impact service delivery or user experience

Our first rule provides the guiding principle in answering the questions, “What should I measure and why should I measure it?” Of course, the WHY part of the answer should always be “Because it directly impacts service. Some good examples WHAT to measure would be:

• Availability of systems and applications
• Success of sessions and transactions
  • Web Pages, DB Queries, etc.
• Response Times where applicable
• Loss of resources critical to service delivery
  • Disk or DB space, Connection or Session limits

When selecting SLA measures it’s important to choose things that you have control over and that can be measured objectively, even if the statistic is as simple 0 for True and 1 for False, as in the case of whether a required TCP port is accepting connections. Either it is (0) or it isn’t (1). A valuable planning exercise is to picture the data or transaction path, and reserve slots in your SLA for appropriate tests of each potential break point in a service. Using a typical web application example, a consumer connects to a web server, which creates a session on a back end application server, which in turn queries a DB server, ultimately sending a response back to the consumer. In our model the break points are the web, application, and DB servers, plus the network connecting them. By constructing a map of break points to monitor, you place yourself in a position to go beyond simply reporting a service failure by localizing where the service is breaking.

Although I’m sure you get why it’s important to localize the point of failure for a service. It is worth examining the answer. Recall that one of our principle goals is to achieve accountability. That doesn’t just apply to apportioning blame afterwards. It means knowing who owns the component that has failed, and should immediately be given the lead to find a solution. A process of discovery always happens as soon as a service failure is detected. In my experience, quickly identifying who in a group of varied specialists responsible for different technologies should own a problem can be unnecessarily time consuming, if you know what I mean… This is especially true if an SLA is poorly designed and the data are ambiguous as to the cause of the failure. In a well designed SLA with data from each break point and each team member seeing the same picture, it’s usually immediately clear who “owns” the problem. You actually facilitate taking ownership of the problem and effecting a solution.

How to Report SLA Data

Designing and implementing the best SLA will be for naught if you fail to build accessible views of the data that can easily be assimilated into a concept of operations. In other words, you have to build that intuitive picture of your transaction path that everyone’s going to share, and put it somewhere everyone can see it quickly and easily. You may have noted that we are discussing using SLA data in the present tense, as in live presentations. Don’t be confused if you expected an SLA to be some tabular historical report to be compared to contract terms and conditions. An effective SLA is all of these things. What’s the point of identifying what can impact service delivery if we don’t use it as an intensive monitor of the health of our critical application? So let’s use the same data to create live presentations of the application’s current state, while also generating historical reports of compliance with key standards.

There are some types of data that do not lend themselves to live reporting. For example, log data that’s collected nightly. Any type of daily or weekly aggregated data should be relegated to historical reporting. That can include both daily detail reports and long term summaries. Any data that is measured at least hourly can be represented effectively in live presentations or dashboards. Remember we want live data to be fresh (last 5-60 minutes) and not have to wait long periods for one component to refresh the picture again. For historical reporting, we’re really using the same data, just querying for longer periods, like weeks, months, and years (ok, daily if your feeling nervous or obsessive…).

Putting It All Together

A well designed SLA can be a critical tool for managers and technicians. Hopefully the process of automating it in live SLA dashboards and historical reports will actually reduce the workload on administrators. It should dramatically reduce the time normally spent in discovery when service problems arise. The SLA will provide accountability, timely assistance, and a unified picture. It will enable service providers to report proactively how well they are providing service.

How do we identify our application pool processes?

For IIS6 and below, begin setting up your application pool monitoring by ensuring we can run the IISAPP.VBS script on the monitored system. You either have to wrap the call to the script in the command that executes CScript, or you have to set CScript as the default scripting language. For example,

• Call CScript in front of the script name: cscript iisapp.vbs //NOLOGO
• Set CScript as the default: CScript //H:cscript //NOLOGO //S

Now we can run the iisapp.vbs script to identify the application pool processes. The parameters for the script are /a, ApplicationPoolName and /p, PID, like this:

W3wp.exe PID: 2232 AppPoolID: DefaultAppPool
W3wp.exe PID: 1160 AppPoolID: MyAppPool

NB: If you have more than one process for each pool, you probably have Web Garden (multiple worker processes) enabled, and should seriously evaluate if this provides any benefit.

So if MyApp is the pool I care about I know its PID is 1160, and we can move on to step two.

For IIS7 on Windows Server 2008 life is much easier. You can view the application pools in the tree view of IISMgr. From the command line, issue the following command:

%windir%\system32\inetsrv\appcmd list apppool

From the IISMgr interface, you can right-click on each application pool and set up ping and recovery parameters, process model details, security, and more.

TIP: You want to create separate application pools in IIS for each application. This prevents one application’s pool problem from taking down all applications. Plus, it makes it easier to do detailed workload characterization on each application.

How do we get statistics for our IIS6 application pool processes?

Here’s where we use Sysinternals’ PsList.exe to get information on IIS6 application pool processes. There are many parameters, or arguments, but the important ones for us are:

• -X; Shows process, memory, and thread information.
• PID; For example, PsList.exe -x 1160; we don’t actually include “PID” in the command, so it’s implied that the number following the command is a PID.

The results of the command above look like this:

C:\Windows\system32>pslist -x 1160

pslist v1.28 - Sysinternals PsList
Copyright - 2000-2004 Mark Russinovich
Sysinternals

Process and thread information for STUDIO:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time
w3wp 1160 8 18 276 12020 0:00:00.483 1:23:57.316
VM WS Priv Priv Pk Faults NonP Page
115204 20500 12020 12892 7141 41 189
Tid Pri Cswtch State User Time Kernel Time Elapsed Time
1164 10 66 Wait:Executive 0:00:00.000 0:00:00.015 1:23:57.316
1176 10 64 Wait:UserReq 0:00:00.000 0:00:00.000 1:23:57.303
1188 10 6063 Wait:UserReq 0:00:00.000 0:00:00.046 1:23:57.293

At the top of the output are the statistics for the w3wp process. Below follows a summary for each thread, including its Context Switches, Current State, User CPU Time, Kernel CPU Time, and Total Elapsed Time. This is all the information we’ll need.

For more Information

Diagnosing Application Pool crashes and hangs

Managing IIS7 Application Pools

What types of data will we want?

So here is how you build a good picture of workload and performance on .NET platforms. We will want our analysis to include both trend analysis and workload characterization. Trend analysis is simply looking at how the peaks are growing and estimating when resources will be exhausted. Workload Characterization includes examining the contribution of component workload processes during peak periods. Peak periods are important because they clearly show the relationship between capacity required to assure service and current resource consumption. First, define what needs to be included in the data collection to support the desired analysis, but it generally includes:

• Overall resource consumption on the platform
• Contributions of constituent workload processes (.NET apps) to global resource consumption

How much data will we need?

Next, we need to collect the data for our analysis. Usually, periods of data need to include sufficient samples and time in order to derive trends with high confidence. If you look at data over two weeks your trends will not have enough historical basis to be valid more than a few days into the future. A trend based on data collected over 12 months provides much higher confidence that a projection is valid several months out, or sufficient time to order new hardware. For trend analysis, it’s not so much the granularity of the samples that is important as that the data set includes all the normal variations seen in real workload processing plus the actual growth of resource consumption over time. In fact, an important best practice for maintaining the data set is to start summarizing trivial period measurements into fewer averaged samples. For example, summarize minutes into hours and hours into days. Without getting into all the statistical calculations specifically, it is sufficient for our analysis to reduce year-long analysis to days, weeks, and months, instead of minutes, hours, and days. We should strive to keep just enough data to meet our needs, without retaining data that is no longer statistically useful. In other words, we can get rid of a lot of fine granularity data (and storage space) that could be summarized into larger samples. For workload and performance analysis we do need data with finer granularity, but it need not cover long time periods, so long as it includes peak periods. So a sound strategy for data maintenance is to keep fine granularity data for a relatively short period, and start to summarize after 30 to 90 days.

What data should we collect?

Alright, now for what you want to collect for .NET platforms:

• Baseline Windows performance statistics
  • Win32_PerfRawData_PerfOS_System (Proc & Thread Count, Q-Len)
  • Win32_PerfRawData_PerfOS_Processor (% Processor Time)
  • Win32_PerfRawData_PerfOS_Memory (Free Mem & Page Faults)
  • Win32_PerfRawData_PerfDisk_LogicalDisk (Space, IO Ops and Q-Len)
• IIS performance Statistics (This will vary based on the type of applications)
  • Win32_PerfRawData_ASP_ActiveServerPages (Sessions, Requests, and Error)
  • Win32_PerfRawData_InetInfo_InternetInformationServicesGlobal (Cache Info)
  • Win32_PerfRawData_W3SVC_WebService (Total Connections and Throughput)
  • Win32_PerfRawData_NETFramework_NETCLRMemory (Heap and GC info)
• .NET Process Statistics
  • Win32_PerfRawData_W3SVC_WebService (Connections, Throughput , and Requests by application)
  • Win32_PerfRawData_ASPNET_2050727_ASPNETv2050727 (Requests and Sessions)
  • Win32_PerfRawData_ASPNET_2050727_ASPNETAppsv2050727 (Requests, Cache, Errors)

There are numerous .NET WMI Classes that can be easily viewed in the WBEMTEST interface, so if you have a very specialized application, it’s likely you will find the specific class that tracks what your application is doing.

Turning Data Into Useful Information

While collecting the data with WMI is straight forward, the same is not always true of interpreting the data. The win32_PerfRawData classes are just that, RAW. The statistics need to be ‘cooked’ (calculated) according to their data type. If you’d like to avoid this calculation, an easy way around it is to use the TypePerf command to get the data from PerfMon instead of WQL queries to WMI. WMI is a lot more efficient than launching numerous command processes running TypePerf, though. Once collected and cooked, store the data in a relational database; I use MySQL compliant MaxDB.

You will want to generate two types of representations of the data, Long Term Trends and Peak Profiles. Weekly or monthly periods of peak usage are great for profiling, depending on your business cycle processing. I like to build two graphs for my reports with both a trend over time and weekly or monthly profile. Here are sample queries (MySQL) for sorting the data from the ASPNETAppsv2050727 WMI Class:

Monthly Profile

select dayofmonth(aspr.sampletime) as SampleTime,
aspr.instance as ApplicationName,
round(avg(aspr.reqexectime),0) as AvgReqExecTime,
round(avg(aspr.reqwaittime),0) as AvgReqWaitTime,
round(avg(aspr.reqexecuting),0) as AvgReqExecuting,
round(avg(aspr.reqbytesintotal),0) as AvgReqBytesInTotal,
round(avg(aspr.reqbytesouttotal),0) as AvgReqBytesOutTotal
from aspdotnetv2_appsreq aspr
where ($TIME_CONDITION)
group by dayofmonth(aspr.sampletime), aspr.instance

Weekly Profile

select dayname(aspr.sampletime) as SampleTime,
aspr.instance as ApplicationName,
round(avg(aspr.reqexectime),0) as AvgReqExecTime,
round(avg(aspr.reqwaittime),0) as AvgReqWaitTime,
round(avg(aspr.reqexecuting),0) as AvgReqExecuting,
round(avg(aspr.reqbytesintotal),0) as AvgReqBytesInTotal,
round(avg(aspr.reqbytesouttotal),0) as AvgReqBytesOutTotal
from aspdotnetv2_appsreq aspr
where ($TIME_CONDITION)
group by dayname(aspr.sampletime), aspr.instance

To query for usage over time for trending, we simply eliminate the time functions in the statement, DayOfMonth and DayName. Also remove the aggregation of the variables so that

round(avg(aspr.reqexectime),0) as AvgReqExecTime

becomes much simpler as

aspr.reqexectime as ReqExecTime

Now that we have all the pieces to build our analysis reports. What are the burning questions that managers want answered in the reports?

• How well is the current platform supporting the load?
  • “Monitoring Service Levels & Performance Evaluation”
• How long will the current platform support Peak Usage?
  • “Capacity Planning“
• What does each application contribute to overall load?
  • “Workload Characterization“
• Should the applications be co-located?
  • “Workload Balancing”

With our data we can show the total load on the .NET platform. We can even go further to characterize the load that each .NET application contributes to the global activity. By examining the weekly and monthly profiles that show the relative activity of the applications we can characterize the workload sufficiently to perform workload balancing and understand peak usage constraints better. Extend the timeline and view the trend to forecast the point at which the current platform will find resources saturated. Now, this can be a bit time consuming to set up, so as usual I built a few custom .NET and Web Services solutions and reports for Longitude; if you’d like copies, just send me an email.

Once again our solution is provided by some cool free stuff, the Windows eventquery.vbs script. Microsoft provides this script on Windows 2003 Server in the system32 directory. The script lets you query the Event Logs much faster than reading through them. It takes several arguments that you can see by running the command, “eventquery /?” from a CMD Prompt. Here are the important ones for our solution:

• S - Server to query
• FO - Format; we use the argument CSV for comma-separated values
• L - Log to query (Security, System, Application, etc.)
• FI - Filter; we filter on DATETIME and ID
  • Other Useful Filters: User, Computer, Source, Type (i.e. Errors)
• V - Verbose

In my testing, searching for particular events using the eventquery vbscript is orders of magnitude faster than using WMI, especially on the biggest security logs. Furthermore, the method doesn’t demand the large IO bandwidth of solutions that download the entire logs. I built a BAT file to take a few arguments and execute the vbscript so that it only gets events in the last 5 minutes. The time threshold is calculated by another vbscript that I borrowed from my colleague Susan.

Here’s what’s in the BAT file:

set Server=%1
set File=%2
set EvtID=%3

FOR /F “tokens=1 delims=;” %%i IN (’cscript //nologo //b E:\scripts\time_threshold.vbs’) DO set TimeThreshold=%%i

EVENTQUERY.vbs /S %Server% /V /FO CSV /L %File% /FI “DATETIME gt %TimeThreshold% AND Id eq %EvtID%”

Here’s the time_threshold Vb script:

Dim MonthStr
Dim DayStr
Dim HrStr
Dim MinStr
Dim SecStr
MonthStr = DatePart(”m”,DateAdd(”n”,-5,Now))
if Len(MonthStr) = 1 then MonthStr= “0″ & MonthStr
DayStr = DatePart(”d”,DateAdd(”n”,-5,Now))
if Len(DayStr) = 1 then DayStr= “0″ & DayStr
HrStr = Hour(DateAdd(”n”,-5,Now))
if HrStr > 12 then HrStr= (HrStr - 12)
if Len(HrStr) = 1 then HrStr= “0″ & HrStr
MinStr = Minute(DateAdd(”n”,-5,Now))
if Len(MinStr) = 1 then MinStr= “0″ & MinStr
SecStr = Second(DateAdd(”n”,-5,Now))
if Len(SecStr) = 1 then SecStr= “0″ & SecStr
wscript.stdout.write(MonthStr & “/” & DayStr & “/” & Right(DatePart(”yyyy”,Date),2) & “,” & HrStr & “:” & MinStr & “:” & SecStr & Right(DateAdd(”n”,-5,Now),2))

This collection method gave us a way to get time-critical information from huge Windows Security Event Logs with very short interval tests every few minutes, instead of every 15 minutes or longer. Now the admin knows before the user calls that somebody is locked out or that a server has rebooted.

TIP - if you want to run .VBS scripts from a BAT file set the default script language using “CSCRIPT //H:CSCRIPT //S”.

As usual, I created a Longitude solution that uses this method. It’s called EventLogQuery. Please send me an email if you’d like a copy.

IIS Log Monitoring is a snap to set up. First, not all the useful statistics named above are captured by default. Selecting what you want to be logged is easily configured in the Extended Logging Properties. The controls are under the Web Site properties on Win 2003 Server and under Site\Logging on Win 2008 Server. You can select the fields that you want to keep track of. So all you have to do is click on the check box beside Time Taken and Sent Bytes to start logging that data. The best part is that you don’t even have to stop or restart IIS. IIS starts using the new logging format as soon as you save the changes.

Why you might care about some info:

• Server data - This is useful for understanding load in a clustered or load-balanced environment, e.g. which server or IP served which pages,
• HTTP Status (scStatus) - Server Client Status is the numeric status of the operation, for example, a Get of a page may yield a status of 200 (OK) or 404 (Page Not Found),
• Bytes Sent and Received - from the server perspective; this is best converted to a larger number such as KBytes or even MBytes for IO intensive sites,
• Time Taken - The length of time the action (e.g. a GET) took in milliseconds; I usually convert this to seconds, and set a threshold to capture slow pages,
• Referrer - The site the user last visited or provided a link to the current page; this is not always populated depending on how the user got the page,
• Port - only needed if you serve both HTTP and HTTPS, and want to differentiate between the two connection types.

Why you might not care about other info:

• User Name - this is normally not captured for HTTP connections, but could be useful for HTTPS connections,
• URI Query - only used for dynamic pages, and often not used, but is available,
• Win32 Status - non-intuitive data that doesn’t add to the Server’s HTTP status,
• Protocol Status - sub-status error code; I’ve never used it,
• User Agent - this is the browser the user employed to view your site, and is only useful when debugging compatibility issues,
• Cookie - this is a very specialized need rarely used, and people always know when they need to use it,
• csHost - Client host - will not usually be captured, and is largely irrelevant to aggregated views of web service.

You can parse the IIS logs with a FOR Loop in a Windows BAT file that counts the unique page hits, plus accumulates the Time Taken data to calculate Max, Min, and Avg times for each page. Or, you can send me an email, and I’ll give the Longitude solution I recently created for IIS web traffic monitoring. Longitude has a built in File Parser that I used for the collector. Even better, I configured the collector to set a pointer so that it only reads new entries in logs that can be really huge. I also created a few reports to go with the collector: Page Hits Summary, Slow Pages, and Page Referral Analysis, which I’ll be happy to pass along…

• Run an event report (application events as well as SLA events) to get a quick look at what happened over the past week or two. Then dig in with your favorite Event Monitor dashboard.
• Use the Event Monitor’s pulldown filters to focus in on events based on severity, time of occurrence, or application; work on the most urgent events, and once they’re cleared up you can change the filter and move on to the less critical issues.
• Use the Event Monitor’s drill-down features to learn more about unfamiliar events and their underlying root causes.
• If an event is being triggered too easily, you can change performance thresholds right in the Event Monitor, and Longitude will show you average, minimum and maximum workload values to help you make the right adjustments.
• Remember that you can enable actions right from the Event Monitor, and you can even suspend events if you don’t want to keep receiving notifications while you work on a problem.

Regardless of whether you use Longitude, another commercial network monitoring software product, or your own home grown scripts, try to do a little monitoring tune-up like this on a regular basis - monthly is probably sufficient - to keep your application performance monitoring practices up to date. It’s a little like cleaning out your Inbox or your coat closet at home - doing it as you go can be a real time saver in the long run.

First, there are the “we’ll do that the next time we have to reboot the server,” tasks. It’s helpful to have a set of tasks like this around for when there is downtime–while you are waiting for the phone call from vendor support or while the entire office is offline due to something beyond your control such as a localized communications outage. “Since no one can access the internet for the next couple hours…we will be performing some systems maintenance tasks…” These kinds of tasks should be well-contained and unrelated to the crisis at hand–i.e. don’t start things you can’t finish.

But the big opportunity presented by a crisis is the chance to try a new business process–after the immediate crisis is covered. The “Why?” is fresh in people’s minds. There will be a receptive audience asking, “how can we prevent a problem like this from happening again?” The IT department–usually not highly visible to the rest of the business–now has the attention of everyone, for good or bad. Have something good to propose and be ready to use the crisis to support your arguments for change.

In one company for whom I worked, a series of service disruptions provided the motivation to finally achieve management support for a platform upgrade. We had been running many websites on many servers with divergent code bases and rotating pagers through three people who had become accustomed to waking every night at 3am to restart IIS or maybe even reboot a server. We tried many things to work within the confines of the old systems–and we communicated what we were doing all along so management knew we were doing everything we could. But finally, a major outage in the middle of the day–as the company was trying to find customers and investors to avoid going out of business–provided the impetus to agree on a total development freeze to allow 3 months for platform migration.

It is a delicate balance. Nobody wants to hear an IT department complaining that they can’t do anything and that they need a crisis to act. Or worse, to present your ideas for solving problems and have a reaction like, “Well that’s great. Why haven’t you already done this?? Oh, sorry, we still don’t have any money.” So it is important, when you play the “now is the time to act!” card to have established a credible basis of competence…the systems ARE being held together with duct tape and aggressively monitored with the best application and performance monitoring systems you can find–and you can keep doing that…but even if you optimize your response time so that you can fix things in 10 minutes every time, users are going to notice and service levels will be affected.

When you encounter a situation that has you thinking, “I can’t believe they do this!” get over the frustration and plan for the day when the barriers to change can be blown away by crisis. Tinker on the development server and have a solution well-baked so you can roll it out: “to address performance and reliability problems, we are launching a new system…please be patient while we work out any new problems.” Crisis is not the time for experimentation: “We’ll try upgrading the software and see if that fixes it.” But it can be a time to step up and lead an organization out of the wilderness…to recognize…it really was broke, and now we need to fix it.

And the same applies to IT organizations and how they manage their mission critical applications. The typical IT department is always busy implementing new projects - rolling out new applications, building disaster recovery plans, and the like. Too often, regular, proactive monitoring falls by the wayside or simply falls behind the organization’s changing business practices and becomes irrelevant. Now that budget pressures have put many of these projects on hold, this is the perfect opportunity to step back and assess your company’s application performance needs.

Listen to our latest application performance monitoring podcast to learn more.

If you’re interested in learning more about application performance monitoring and how Longitude can help you improve service levels, we invite you to visit our web site at www.heroix.com and request a Longitude free trial.

I’m curious what the experience of IT professionals is with respect to monitoring, blocking, and otherwise regulating employee access to social media. It is fairly trivial to block certain websites or applications at the network level, but such actions will lead employees to find a way around…

You can certainly use network monitoring software like Heroix Longitude to evaluate whether there are bandwidth issues from employee usage. But usually the problems are not just about bandwidth–there is the “lost time” argument which presumes workers who deviate from their task list are “stealing time” from the company and there is the “idiot factor”–the employee who updates his or her Facebook status with confidential or proprietary company information. These are not really IT issues…but ultimately, where culture, company norms, and common sense fail to achieve the desired results, management will come knocking on the door to IT for answers to questions and solutions to problems.

What’s your experience? Has it changed recently? Do you block it all or have you been asked to? Have you performed surveillance on employees to see how much time they were spending on Facebook? How do you know they don’t just have it open in a window all day vs using it constantly? Almost 2 years ago, I friended my CEO on Facebook and at the time, that was kind of radical and risky. Now, the issue is more that you don’t know what to do when your CEO starts following you and you don’t want it! Or maybe I just live in an early adopter bubble.