1. Beef up the Physical CPUs - While a virtual machine can achieve the same SQL throughput as a physical server, it comes at a cost.  Increased CPU usage.  You will also want to plan for moderate increases in CPU related to SQL backup and restore operations. The same goes for network intensive workloads.  I would start by adding 20% to each SQL virtual machine to accommodate for these factors. 

2. Avoid Dynamic VHDs – Use pass-through disks where IO performance is critical or fixed VHD where slightly less IO performance is acceptable.  Keep in mind that fixed VHDs are easier to move around and support, so they are a good option unless you absolutely need the additional performance. 

3. Install Integration Components – If you are not installing SQL Server 2008 on a Windows Server 2008 guest, you will want to install Integration Components (On Server 2008 by default).  Doing so will provide better IO performance, reduce memory overhead, and allow the use of synthetic network cards versus emulated.  All of which are desired for SQL.  

4. Properly size memory – Keep from having to take databases offline with a reboot by putting extra thought into memory requirements upfront.  Also be mindful of the 64GB per VM limit within Hyper-V R2.

5. Look at licensing closely – There are separate licensing statements from Microsoft that relate to virtualization.  For example, if you license SQL on a per processor basis (Standard,Workgroup, or Enterprise), you need to have a license for each virtual processor.   Carefully look at the clauses for both SQL Server 2008 and the Microsoft guest operating system.  Depending on your consolidation strategy, it may make sense to switch licensing models.  Check SQL Server 2008 Licensing Frequently Asked Questions as a starting point.   Also look at any third party SQL backup products you may use. 

6. HBA tuning – With server virtualization becoming more prevalent in the data center, HBA vendors are starting to produce best practice documents for HBA settings.  Some get very specific for the array, operating system, and virtualization platform.  The two areas you want to focus on are HBA Queue Depth Settings and HBA drivers. 

7. Use Sysprep – If you are deploying a stand-alone instance of SQL Server 2008 R2 to your Hyper-V Infrastructure, using SysPrep can save you time for future deployments.  This functionality is just recently supported in SQL 2008 R2 or later; however deploying via SysPrep to clustered SQL instances is not supported.   Using SysPrep gives you the benefits of a faster and more standardized SQL Server 2008 deployment.  It can also serve as part of your disaster recovery strategy to restore SQL Servers.  Read How to: Install SQL Server 2008 R2 Using SysPrep from Microsoft to learn how to set this up.

Have you come across something that should be added to the list?  Was this helpful?  Let me know by leaving a comment.

1.  Unplug cluster NICS – If you are using any type of host based replication software to migrate data from existing shared storage to new storage, you will want to unplug the cluster NICS.  They are the network connections used for cluster updates and “heartbeat” communications.   I found this to be the case even with all Exchange Server and MSCS cluster services in a stopped state. 

2.  Update disk signatures - After you complete the data migraton to your new SAN or shared storage and try to fire up your application configured within the Microsoft Cluster Server (MSCS), it probably will not start.  If you see Event ID 1034 in the event log, it means that the cluster cannot find the disk signatures that it was expecting.  Makes sense because the disks are new.  So, just follow the instructions in Microsoft KB 243195 article and you will be up in no time. 

My experience was based on Exchange 2007, but this issue would occur on any MSCS clustered application that is moved to new shared storage. 

Please feel free to share these tips with anyone you think they may benefit!  If you’ve had a similar experience or liked this, let me know by leaving a comment.

I see a lot of content that appears to favor either Hyper-V R2 or vSphere for EVERY use case.  I think they are both good products so I wanted to point out some of the pros and cons of each one.

Hyper-V R2 makes sense if …

MS only please – You’re a Microsoft shop and don’t have someone who knows Linux, or has time to learn a little to get up to speed on VMware vSphere.  The fact that VMware supports 55 different OSs ranging from Windows, OS/2, Solaris, and NetWare is not important because you’re ALL or primarily Microsoft.

We’re fairly simple – Let’s pretend all your servers are basic like file, print, IIS, and Active Directory.  Their all fairly underutilized and you don’t see too much changing; they appear to be easy virtualization wins.  You don’t have enough servers to be concerned about high consolidation density; maybe something like 30 servers that easily fit onto 3 hosts with a 10:1 ratio.  Not being able to hot add VM’s virtual memory and CPU is not a requirement. 

You already own SCVMM and SCOM – While Hyper-V R2 is FREE it requires System Center Virtual Machine Manager (SCVMM) and Operations Manager (SCOM) to take advantage of the advanced features like resource scheduling with PRO.  Really, unless you use a third party product you need SCVMM to do physical to virtual migrations (P2V).  

VMware vSphere makes sense if …

We’re going all in – Your goal is to achieve 100% virtualization (or really close).  Maybe you have 50 or more servers and there are plenty of heavy hitters in the mix.  Features like 8-way virtual processors,  DRS with Resource Pools, or Hot add of virtual memory and CPU would benefit you.  You could envision setting up tiered storage, and using VMware Storage VMotion, to migrate virtual disks during peak processing or maintenance sessions without any application downtime. 

Automated Disaster Recovery– After you get virtualization in place you want to use VMware vCenter Site Recovery Manager to take the complexity out of DR.  If you have really short RTO and RPO objectives, you will want to stick with VMware to keep you out of the woods. 

We’re charging back – Let’s say you want to charge back the use of you applications and\or servers to the business.  Vmware has an off the shelf product VMware vCenter Chargeback, to model, measure and assign costs associated with virtual machines. 

We’re not into general purpose – With vSphere you’re getting a product that was built from the ground-up for virtualization.  You’ll get updates and enhancements more often since you’re not having to wait on the slower release cycle Microsoft follows.  You’ll also only have to apply security updates that are hypervisor related and not spanning across the general purpose operating system. 

We’re security conscious – Below are two 3 benefits VMware has in this area…

• VMware vShield Zones – vShield Zones enhances the privacy and confidentiality of virtual machines and their data. VMware vShield Zones enables customers to monitor and enforce network access to sensitive areas of the virtual datacenter, including the DMZ, or to servers with sensitive data that is subject to regulations such as PCI or SOX compliance. Administrators can securely bridge, firewall or isolate traffic across logical trust or organizational boundaries within existing vCenter Server deployments..
• VMware VMsafe – VMware VMsafe is a new technology that leverages the properties of vSphere to protect virtual machines in ways previously not possible with physical machines. VMware VMsafe provides an application program interface (API)-sharing program that allows partners to develop VMware-aware security products.
•  VMkernel Protection – Disk-integrity techniques protect the boot-up of the hypervisor using the Trusted Platform Module (TPM), a hardware device embedded in servers. To ensure the authenticity and integrity of dynamically loaded code, VMkernel modules are digitally signed and validated during load-time. These disk integrity mechanisms protect against malware, which might attempt to overwrite or modify VMkernel as it persists on disk. VMkernel also uses memory integrity techniques at load-time coupled with microprocessor capabilities to protect from common buffer-overflow attacks that are used to exploit running code.

Leave a comment and let me know if you’ve come across a situation where one platform prevails!

As I prepare for an offsite disaster recovery exercise, what I am really doing is going around charging dead batteries. If you have cars, kids, or maybe a boat; you know that a dead battery can ruin your day. I am talking about a different kind of batteries, what I like to call IT batteries. Let me explain.

    Related to IT disaster recovery,

     dead batteries are things like

  -Outdated technical documentation

  -A media kit with outdated software

  - Outdated contact lists or action steps

  -DR site or contract that does not meet current

    needs, or offer future flexibility

  - Key servers with failing or nonexistent backups

  So, what to do?

 One of my favorite inventions for making sure I avoid spoiled fun is the trickle charger.  In short, the trickle charger supplies a charge to the battery at  regular intervals to keep it fresh and dependable while it is stored. Sound like something you need for your DR Plan?  Here are some tips.

1. Use your task management software to schedule meetings or reminders such as “Refresh Media Kit”.

2. Created an automated process for pulling out required DR documentation and shipping it offsite or to the Cloud.

3. Before you buy any new hardware of software, ask yourself. Do I need this in DR?  Will this work in DR?

4. Test recovery of backups regularly, especially after OS or backup agent upgrades.  Maybe schedule an week called “Mini DR Week” for onsite tests.  Virtualization is great for this!    

5. If you have VMWare Infrastructure, consider using VMWare Site Recovery Manager to help orchestrate DR.

6. If you use a WIKI for documentation, consider creating a category labeled “Disaster Recovery”. Add applicable articles to this category, and also use the “PDF Book” feature to create a PDF to add to your DR plan. Review WIKI regularly to capture new articles that need to be added.

Do you have any tips for keeping your DR batteries charged?   Leave a comment and let me know your thoughts.

So Why SSD?

I was faced with the following challenges the led me to use the SSD drives.

• Heavy random SQL I/O workload exceeding 18,000 IOPS on one host
• Growing company that just made an acquisition
• Need to shrink hardware footprint
• Climate control system being stressed to the max during hot Texas summers

The SQL workload was previously allocated across 90 – 15k FC drives on a Clarion CX3-80.  The FC drives did a good job of handling the workload, but during certain peaks, I saw delays were seen that made me uncomfortable.  I anticipated needing 120 15K drives to meet the performance challenges over the next 3 years.

I really appreciated the fact the my EMC team was honest about the SSD drives delivering 2000 IOPS \ disk under a continuous random database workload.  By using 18 SSD drives in my design I was able to meet my challenge with plenty of room for performance growth.  This also allowed me to use only 6 Disk Array Enclosures (DAE’s) opposed to 14 if I went with all FC drives.

After Installing and configuring the SAN, I ran some IOMeter tests against it before going live.  I achieved 20,500 IOPS with the test setup below.

• 100% random 8K workload
• Dell PowerEdge R900 with 16 cores, and 128GB RAM
• 32 outstanding workers
• 1 hour test
• 5 ms or less average response time
• 1 LUN with 9 solid state drives

In production

Immediately after migrating data and hosts to the new SAN, I noticed a 4% power drop across a 55kV UPS.  In my case, that aspect alone with the reduction in cooling costs already made the SSD a great “value”.  I followed the EMC Whitepaper “Implementing EMC Clariion CX4 with Microsoft SQL Server 2008 databases” and everything has worked like it should.

DR Benefits

SSD drives seem to be targeted towards the specialized and heavy workloads, but I see a good case for using it at a secondary DR site such as Fibertown.  With a smaller footprint and less power required, maybe it could keep you from having to move to the next level of service.  By combining server virtualization with an SSD SAN, it seems like a very powerful package with a small footprint.   A bonus would be easier setup with less cabling.

If you have experience with SSD drives in your SAN, I would enjoy receiving a comment from you.

If you are repeating administrative tasks more than a couple times a week, please stop.  One of the most powerful ways to avoid errors and dramically save yourself time is to employ scripting.  Start by making a list of simple things that are less than enjoyable to do and just dive in. 

If you are a Windows admin, learning Powershell is essential.   Visit the Microsoft script center to see sample scripts and get an idea of what you can do.  There are countless blogs and books dedicated to Powershell once you are past the basics.  You can make some handy reports with Powershell that will give you a constant pulse of what is going on in your environment.

Am I going to work myself out of a job?  I have been trying to work myself out of a job for since I first learned scripting 8 years ago.  Instead, I kept getting promotions and nice raises.  If you were the decision maker faced with laying of people who would you want to keep?  The person who can do the job in 15 clicks or 1 click?  I always thought it would be fun to have a potential employer call one of my references where they learn that I did indeed get let go for “automating everything”.  Never happened.  

2. Hideout

It is really easy to hang out in your office with the door shut or hide from your users.  Trying an alternative approach is not as bad as you think.  I have had some of my best ideas come to me while I was listening you users in the break room.  Just have a normal conversations where you listen and care.

Another hidden benefit is learning more about the business.  Sometimes, hearing stories about how your business is impacting lives can be very inspiring.  Another benefit is being able to test the waters before presenting ideas to your management.  Try something similar to this, walk into a break room full or coworkers and ask something like this “I am not sure how to do it yet, but would it be useful to you if you could recover your own documents without having to call me? ”  You may sample several different groups for a week and find no one is interested in an idea you thought was great.  On the flip side you may find they love the idea, and you walk away with better ideas on how to implement the service.  Try it out.

3. Selfishness

You may have people around you who are hungry to learn and take on responsibility so why not let them.  Being selfish or thinking you are the only one who can do every task is just silly.  Challenge your junior team members to learn new things, and reward them with responsibility as they step up.   Sure, they may make mistakes so just be there to help them.  By doing this you’re a leader, and now you can focus improving your overall service.  

 Truth be told, I struggled with this for a long time in my career.

4. Free Tool Mindset

I get excited when I can add a new free tool into by bag that really works.  What seperates average admins from those at the top of their game?  I think admins with mindset that all their tools must be free are following a recipe for failure.  Dig deep and look at the true value you expect to get for each tool you use. 

As an example, consider the Powershell IDE you use to create your magical scripts.  If you plan on spending several hours each month developing and refining scripts this may be an area where spending a few dollars is a no brainer.

5. Poor Sales

Realizing that sales is part of your job responsibily is huge.  Just as you sell yourself and your value during an interview, you have to maintain this approach.  Simply put, your sales and communication skills can keep your management from pushing bad technology decisions on you. 

So you’ve got a great idea, now what?  Just like any good salesperson you need to do your research.  Learn everything about the value, and ROI for the technology.   Best of all, anticipate objections and have solid answers for every one of them.  Talking to reference customers and reading case studies can really help here.