Virtual Black Hole

Let’s Talk About VMware DPM

Steve — 2009-11-10T15:58:19Z

Today’s topic is going to be about VMware’s Distributed Power Management or otherwise known as VMware DPM. If you have not heard about VMware DPM before, I found an interesting video interview with Anne Holler who works at VMware as a Senior Staff Engineer. I found this interview at Vladan Seget’s ESX Virtualization site here and it is worth checking out. The video is pretty short, only about four minutes in length and gives a pretty good overview of how VMware positions DPM in conjunction with HA and DRS.

In a lot of the environments I have worked in over the years actually had most of the processing power happen at night during the backup process. Actually at one point I had a cluster that was oversubscribed to put it mildly. If you have ever driven a standard transmission then you know about the red line on the tachometer that you should not go over or the engine would blow up. Let’s just say my oversubscribed cluster was riding that line during the day and when the backup schedule kicked off I actually saw CPU% on the individual VMware ESX hosts themselves reach levels of 102%. Just like clockwork ever day when the backup’s kicked off the host would really max out. Pretty cool to watch but I digress. The point is I am not sure I would really think about using VMware DPM in my production environment. Now if I just built a brand new ten node cluster what was designed to cover the growth for x amount of time and I had just gotten started with putting virtual machines on the cluster then sure why not set that up? I would have plenty of resources available that I would feel pretty confident about that but in most places that I have worked or spent any time at there was not really that amount of spare resources and most customers were always trying to cram as much as they can on the clusters.

VMware DPM was actually introduced in Virtual Center 2.5 and VMware ESX 3.x as an “experimental” feature and only had one way of ‘waking’ a server and that was with a magic wake on lan packet. That worked great until you rebooted Virtual Center or restarted the Virtual Center server service for any reason. After the reboot Virtual Center would see the host as down and was not able to bring it back to life. That was one reason why the technology was experimental at the time. Now in vSPhere, VMware DPM actually has three different wake methods it can use to bring the host back up. Along with using wake on lan DPM will also use IPMI wake and ILO wake. One other really cool thing that was added to VMware DPM was the ability to schedule when an ESX host would wake back up. So if you know you were going to have a heavy load at 9am you could schedule the host to wake up at 8am to be pretty for the start of the business day.

So to recap, I am not sure that VMware DPM would be something I would setup and use on a regular basis. I have also worked in certain environments that the Change Control Board would literally lay an egg if they knew that hosts could and would shut down for any reason.

That is my take on this. So what’s your take on VMware DPM and are you using it currently in your environments? I would really like to what you all have to say and think on this? Inquiring minds want to know.

My Thoughts on the VCE Announcement

Steve — 2009-11-03T18:42:00Z

It seems the big news for the day is the Announcement of the VCE Coalition. Seems I have been reading blog posts about this all morning with the last post being Scott Lowe’s thoughts on the subject. I have been hearing that this has been in the works for a while now and yes it will be nice to be able to use one SKU to some sort of complete hardware package for running your virtual infrastructure. I do, however have to agree with Scott Lowe that VMware has the most risk from this adventure but I really have to ask the question. Where you really expecting anything different? Let’s take a look at the money angle. EMC owns VMware and CISCO has invested a lot of money into VMware also, at least in my mind, it was just a matter of time for something like this to happen. Well than again there was my favorite rumor that CISCO was going to buy VMware or even better yet EMC.

Another question to ask is who is really going to benefit the most from this? What will be the response from the other server OEM’s? A little healthy competition for things is supposed to be better for the customer right? Well I guess time will tell how this little adventure plays out and if this will mean more segregation between the server hardware providers. More and more blog post are getting added it seems like each minute with more information so, once the dust settles things should really start to get interesting.

Some Thoughts on Thin Provisioning

Steve — 2009-10-29T17:29:14Z

One of the enhancements or features in VMware’s latest release, vSphere, is the option to create virtual machines with thin provisioned disks. I found the skinny on Thin Provisioning with vSphere on the VMware Blog by Michael Adams.

vStorage Thin Provisioning optimizes storage costs through the most efficient use of storage in virtual environments. Storage requests more often than not are usually overestimated by users mostly to avoid having to go through the request/approval process. With vStorage Thin Provisioning, IT departments can now assure business users of storage space availability while deferring the actual costs of purchasing storage to when it is really needed. Full reporting and alerting on allocation and consumption ensure that virtual machines don’t really run out of storage, Storage VMotion and Volume Grow ensure that virtual machines can either migrate to datastores with additional storage or volumes can be increased in size when consumption approaches allocation.

Sounds pretty cool and can be a real cost saver. This was the topic of conversation on this week’s VMTN Community Roundtable. In case you missed it you, can hear the podcast from this link or you can also download from iTunes.

Storage providers have been able to provide thin provisioning for awhile now and this has worked really well. So now with vSphere we have a choice of where we want to handle thin provisioning. vSphere also makes it really easy to convert thick vmdk files to thin using Storage vMotion. So which one should you use? Well that is the real question. Now, truth be told, I have only set up thin provisioning in my lab environment. I am not sure if today I would try to make the selling point to switch from storage thin provisioning to VMware provisioning. I guess I am under the belief that it might be better to leave this type of provisioning and control with the storage group. I really have nothing to back that belief up one way or another it just seems to be the most logical choice. I do think that changing a process that is already in place might causes issues later. What happens if both VMware and the storage device both work to deliver thin provisioning at the same time? I do not have the equipment in my lab to test but I really have to wonder. Has anyone out there tried this?

Now on the flip side of the coin, if the storage device is not providing thin provisioning then it would make good sense to take advantage of this technology by letting VMware handle this for you. The next question would be, is thin provisioning for everything? Should it be used for some virtual machines and not others? For me, personally, I am not ready to push new technology into production without taking some baby steps first. I think thin disks will work great for VDI and can greatly increase deployment times of the desktops. I think that would be my first step and would not even consider the technology for mission critical application for at least a bit while my comfort and confidence level increases.

Aren’t these the steps that most companies follow when adapting new features or new technology? I have no doubt that thin provisioning in one way or another will become the de facto standard as we move forward in time. Now what say you? What are your thoughts on thin provision and how it should be handled?

Driving A Successful Virtualization Adaption

Steve — 2009-10-22T10:00:00Z

This week’s VMTN podcast was Title: #69 - The Virtualization Journey with Vittorio Viarengo was a discussion about the steps for driving a successful virtualization adaption. You can also download the podcast on iTunes. For years now I have had a tag line that I use. It’s my motto, if you will, and goes like this: “Virtualization is a journey, not a project.” Profound isn’t it? This is something I firmly believe in and have been preaching over the years. The idea is that there are steps that happen during this virtualization journey that happened at the different stages of your deployment. Without a doubt the first step is having the front line people, namely the administrators that will build, control and support the environment. Vittorio continued with the second step of getting the application owners on board as well as some trust in the infrastructure so that over time virtualization will become more accepted and would be the first thing the application owners would choose going forward. He went on to his third step of getting the C level manager or director on board and committed to virtualization moving forward to establish a virtual machine first policy in the infrastructure.

I fully believe in that concept and this falls right in line with my understanding of how things need to play out in a company to get the VM first policy in place but, I disagree with the order of the steps. I believe you need the C- level signoff as step two and the application owners step three. Let me explain my reasoning for this difference of opinion. Application owners that I have had the pleasure of working with over the years have had to take ownership of an application that they may be knowledgeable about or may not have any experience on that application at all and they were given the task to learn and deploy. These guys and gals will read and memorize the release notes and this is the guide that they will live by as well as the vendors System Engineer that is guiding the application owner on the install. I am happy to report that things have gotten better over the years as far as third party and OEM’s accepting and supporting their products in a virtual environment. In the olden days, when virtualization was in its early infancy, venders and OEM’s were not so quick to jump on the virtualization band wagon and would be quick to say that they would not support their product running in a virtual machine. Can you believe the nerve of these guys? So easy to say “we won’t support virtual machines”. Oh those where the days and although support has gotten better it is not fully there yet, across the board. This is the main reason but you also need to squash the company politics right from the get go. So getting the C level manager and/or Director on board to support the company’s virtualization journey and to make sure everyone in the organization is on the same page about the vision and direction the company is about to follow. There is actually another step for the C-level but I will come back to that in a second. Once you can get a policy in place then overall acceptance in the company will be much easier. This leaves some Managers in a position to push back on venders, if needed. Some phone calls might start sounding something like this; “You will support your product on a virtual machine because that is what we use. If you need help with some of the testing or anything we the company will be more than happy to help you with that but you will support us or we switch to a new product.”

Now the focus will shift to step three, the application owners and working with them to educate them on the power of virtualization as well as a little bit of training about what kind of resources they would really need to run this application. No you do not need a four way virtual machine with four gigabits of RAM to be your DHCP server. That is a different blog post altogether but understanding, communication and trust needs to be built with the application owners. It is their butt on the line when something happens to the application regardless if the failure was something they could not control. Once you build the trust the world is a better place for all.

I mentioned earlier that the C level manager and/or director really had two steps in this process and now we work towards the second step with management. Sometime during your deployment you will come up to what I call the “Magic Moment.” If one of my past assignments this “magic moment” was during an offsite disaster recovery test. I was able to prove to all a big reason why virtualization was the next best thing since the computer. I got to help design a rock solid disaster recovery plan and with living in the middle of hurricane central, this was a very good place to shine. Once I was given the green light that the LUNS were presented to the offsite virtual infrastructure I was able to recover the infrastructure in a matter of minutes and a lot of people really took notice. Once we got back from the test absolute full virtualization acceptance across the board happened. What a magic moment to share.

These are the steps that I believe happen on a company’s journey into virtualization and steps to the cloud. It is never a project but rather a life changing journey moving forward into tomorrow.

Extending A Volume On A VMDK

Steve — 2009-10-20T15:45:35Z

I was combing through the VMTN community forums and came across a couple of posts on the best way to extend or expand a vmdk file. This topic seems to come up on a regular basis, so I thought it might be a good idea to put together a blog on this topic.

The main tool that I have used to complete this task is the Microsoft utility called Diskpart. For most cases this utility will do the job just fine but it does have some limitations. Inside of the vSphere client it is very easy to expand the vmdk file to the new size that you need. Let’s take a quick look at the requirements to be able to extend a volume with Diskpart.

The volume must be formatted with the NTFS file system.
For Basic volumes, the unallocated space for the extension must be the next contiguous space on the same disk.
For Dynamic Volumes, the unallocated space can be any empty area on any Dynamic disk on the system.
Only the extension of data volumes is supported. System or boot volumes may be blocked from being extended, and you may receive the following error:

Diskpart failed to extend the volume. Please make sure the volume is valid for extending

You cannot extend the partition if the system page file is located on the partition. Move the page file to a partition that you do not want to extend.

Even though you cannot extend the system or boot volume you can easy use a helper virtual machine to attach the disk you want to extend. You should then have no problems performing the extend with the helper virtual machine. Got to love virtualization!!! I have found it just easier to use the helper virtual machine for any and all extends that I need to perform.

This is all pretty straight forward which brings me to my next question. What do you use when diskpart does not work or is there a 3^rd party tool that just really rocks? Before moving on to 3^rd party it is worth mentioning another VMware product that can perform this talk for you also. You can use vConverter to change the disk size during the conversion. This leaves vConverter as a very viable option to perform this task although it is one of the slowest options.

There is a tool call GParted from Sourceforge that is worth mentioning. They have the ability to boot from a cd-rom or usb to extend volumes. You can download this product straight from Sourceforge’s web site http://gparted.sourceforge.net/download.php

I found out from the forum that Dell has their own utility called ExtPart to be able to extend a volume. More information on this tool can be found at http://support.dell.com/support/downloads/download.aspx?c=us&cs=19&l=en&s=dhs&releaseid=R64398&formatcnt=2&fileid=83929

So who has any others tools they use and would like to share with the class today? It would be great to have a reference of all the options and tools available to us.

Organizing vCenter

Steve — 2009-10-15T19:29:23Z

I was recently asked what I thought was the best way to organize things in vCenter? I personally have used several different methods to organize things in different ways. I started to wonder how other admins were organizing their own environments and if there was any common methods of madness to this. First let’s take a look at the different ways I have organized things and then I will propose the same question to you and see what kinds of responses I can get from you, in the audience.

The different methods that I have used to organize my environment have been mainly with the use of resource groups and folders. Each of these options is located in different views; resource groups are located in the host and clusters view and folders are located in the virtual machine and template.

Starting with resource groups, I have built different groups in a few different ways in different clusters. Creating resource groups based on importance, such as high, med and low has been the most basic configuration that has been taught by VMware in the classes and as examples when talking about resource groups. Now resource groups setting only take effect when contention happens on the host. If there is no contention then these groups give you the ability to group together virtual machines based on operating system. I have had a “Windows” resource group as well as “Linux” resource group to separate the different groups and or teams and the virtual machines that each maintained as well as having “high” and “low” groups for the most important and least important or in other words, the things like the development virtual machines. I have also seen people create resource groups based on the type of application but vSphere now gives us that ability to use vApps to accomplish that kind of configuration.

So nothing too exciting with the resource groups, just a pretty straight forward configuration. Now, I also organized differently in the folders view. I would actually create different folders based on the application running on the virtual machine. One example would be to have all the exchange servers in one folder. In one case I actually created folders based on the virtual machine owner so I would easily know who is responsible for any virtual machine when issues arise. Moving forward in time I really started to use custom columns in vCenter for application owner and all information to be able to get hold of that person. As time went on it I created more and more custom columns to turn the main virtual machine page into my main source of information about all the virtual machines. In the end these columns would have all the information that the server database had listed and ended up replacing the server database for any information about the virtual machines.

Now we have gotten to the part of audience participation. How do you organize your vCenter? Inquiring minds what to know!

Fail your VCP4.0 and get a free re-take

Steve — 2009-10-09T13:47:02Z

I was reading some blogs and come across this notable piece of information posted by Tom Howarth over at the PlanetVM and since I just passed the VCP4 exam thought this would be worth mentioning again. The original post can be found here.

Upgrade to VCP4!
VMware is allowing participating candidates who fail a VMware Certified Professional on vSphere 4 exam (exam code - VCP410) to have a free re-take. Visit www.PearsonVUE.com/VMware/Upgrade for more details. Click Here to enroll in the 2nd Shot/Upgrade Program.

To read the full blurb click here

Using vWire To Search for Datastore Block Size Misconfiguration

Steve — 2009-10-07T20:19:29Z

Let me paint a picture for you on something that I have seen pop up more than once. You send a request to the SAN team to get some storage added to your virtual infrastructure. The SAN team replies back that your LUNS are ready and available. You rescan your adapters and find the new 1TB LUN that the SAN team just provided. You follow the wizard and use the default settings to create your Datastore and format VMFS. So life is good, you have another terabyte of space and that should tie you over for at least a little while. Time goes by and you have created a few virtual machines on the new LUN and then you get the request for a virtual machine that will have a 300GB data drive. You start to create the virtual machine and things come to a screeching halt. An error has occurred because you used the default setting when you created the datastore. The LUN was formatted with a 1MB block size and the largest file size that can be on this LUN is 256GB. To really make thing worse, you cannot change the block size without destroying the LUN and starting over from scratch. You do not have enough room to storage vMotion, the virtual machines, to different LUNs. That puts the virtual machine request on hold until you get more space available to work with.

Oh, if there was only a way to get some kind of idiot check or warning; “Are you sure you want to do that?” There is no warning for this built in to vCenter but using vWire’s searching capabilities we can search out the datastore’s block size and have the ability to send out an alert when a LUN is detected with non standard block size. Create the LUN and then get an email before you get too far along loading up the new LUN with virtual machines.

Let’s take a look at the choices and sizes of the block size:

1MB block size - 256GB maximum file size
2MB block size - 512GB maximum file size
4MB block size - 1024GB maximum file size
8MB block size - 2048GB maximum file size

The first thing to make note of is that the VMware SDK reports disk space in bytes and therefore a little conversion will be needed. I found and like to use this site to quickly make the conversion for me.

Let’s take a look at what the search looks like for this event. I convert 256GB to bytes and get a value of 274877906944. So, for this search I am looking for any datastore’s that are greater than 274877906944 or 256GB and block size equals 1MB. Below is an example of the actual search string.

SELECT DatastoreEntity_5097 FROM DatastoreEntity AS DatastoreEntity_5097

WHERE ( DatastoreEntity_5097.summary.capacity > 274877906944 )

AND ( DatastoreEntity_5097.info.vmfs.blockSizeMb = 1 )

That handles the 1MB block size so let’s take a look what the search would look like for datastore’s that was larger than 256GB and smaller than 512GB and was supposed to have a block size of 2MB.

SELECT DatastoreEntity_1691 FROM DatastoreEntity AS DatastoreEntity_1691

WHERE ( DatastoreEntity_1691.summary.capacity > 274877906944 )

AND ( DatastoreEntity_1691.summary.capacity < 549755813888 )

AND ( ( DatastoreEntity_1691.info.vmfs.blockSizeMb is null ) OR ( DatastoreEntity_1691.info.vmfs.blockSizeMb <> 2 ) )

We know from the last example that 274877906944 bytes equals 256GB and another conversion of 512GB to 549755813888 so our search will look for datastore’s that are bigger then 274877906944 and less than 549755813888 and also have a block size that is greater than or less than 2MB.

We can follow the 2MB example and change the byte size and block size to get the search needed for LUNS that are 512GB - 1TB in size.

SELECT DatastoreEntity_1691 FROM DatastoreEntity AS DatastoreEntity_1691

WHERE ( DatastoreEntity_1691.summary.capacity > 549755813888 )

AND ( DatastoreEntity_1691.summary.capacity < 1099511627776 )

AND ( ( DatastoreEntity_1691.info.vmfs.blockSizeMb is null ) OR ( DatastoreEntity_1691.info.vmfs.blockSizeMb <> 4 ) )

Lastly, let’s look at the example for datastore’s that are larger than 1TB and have a block size that is greater than or less than 8MB.

SELECT DatastoreEntity_1691 FROM DatastoreEntity AS DatastoreEntity_1691

WHERE ( DatastoreEntity_1691.summary.capacity > 1099511627776 )

AND ( ( DatastoreEntity_1691.info.vmfs.blockSizeMb is null ) OR ( DatastoreEntity_1691.info.vmfs.blockSizeMb <> 8 ) )

Pretty straight forward, but yet very powerful to get information and an alert that you need when you need it. This is just one example of how you can use vWire to help bring you situational awareness of your environment.

Automated Fix for a DRS Issue

Steve — 2009-08-06T20:23:11Z

Every once in a while something goes wrong in my lab and I get to spend some quality time working out what happened and how to fix it. Well after doing updates the other night I did encounter an error that I had not seen before and wanted to share. The actual error message was this…

“Unable to apply DRS resource settings on host (Reason: A general system error occurred: Invalid fault). This can significantly reduce the effectiveness of DRS”

In my experience over the years, I have learned that to address HA and DRS issue the first place to start with is the vCenter agent or other words the vmware-vpxa agent (mgmt-vmware) that runs on the host and this is the agent that enables communication between the host and vCenter. For that reason it makes it the most logical place to start when troubleshooting and a restart of this service tends to fix up a lot of errors and in this case the restarting of the service resolved the error.

So now I knew the error message to look for as well as what to do to resolve the issue. Perfect for what vWire was designed to do. I needed to create an alert in vWire so I know when this event happens.

So I am going to take the exact error message from vCenter which looked like this…

Unable to apply DRS resource setting on host server in clusterb

I need to add wild cards “.*” for the actual name of the host and cluster so when all said and done the alert looked like this.

Next comes the PowerShell script to restart the service. The restart-VMHostService is the cmdlet that I need to complete the task. So using straight PowerShell the code would look like this.

$vpxa = Get-VMHostService -VMHost $h | where {$_.Key -eq ‘ vmware-vpxa”}

Restart-VMHostService $vpxa -Confirm

With that command we need to make a couple changes so it will work with vWire. I created a foreach loop in case the script is run on more than one host and also changed the way it gets the hostname information so the script will take the piped information from vWire.

I was unable to repeat this issue in my lab but if it happens again I know that vWire will alert and resolve the issue for me before I even finish reading the alert email on that event. Could an administrator ask for anything more?

VMware Developer Day and VMworld 2009

Steve — 2009-07-29T18:44:35Z

If you are involved in virtualization at all then you know that VMworld is coming up August 31 - September 3 at the Moscone Center in San Francisco. There is a change in the format this year that is well worth mentioning. VMware is having Technology Exchange Developer Day on August 31, which is an event that is dedicated to ALL developers building solutions on VMware platforms.

If you are wondering if it is worth attending this event while your add VMworld this year let me express five reasons why this event would be worthwhile.

Reason 1: Content Designed Specifically for Developers

Technology Exchange Developer Day focuses on the needs of developers building solutions for VMware platforms.

This 1-day event will cover the latest on developer focused VMware APIs, best practices for integration with VMware products, and demos and technology previews of developer focused products. The agenda is available at www.vmwaretechnologyexchange.com.

Reason 2: (For TAP Program Members only) Get Product Roadmap Updates

Members of our TAP Program will get exclusive updates on VMware product roadmaps to assist them with their product planning activities.

Reason 3: Receive Hands-on Training from VMware Experts

Develop your skills and technical expertise by participating in interactive demo and technology previews. A complete list will be posted to the web site in the next two weeks.

Session Info:

http://blogs.vmware.com/developer/2009/07/vmware-technology-exchange-developer-day-sessions-schedule.html

Reason 4: Interact with VMware Engineers

Get your questions answered directly from our engineers who design these developer focused products. There will be multiple opportunities to meet and interact with these engineers during the sessions, through demos or in small-group sessions.

Reason 5: Get Complementary Software

All registered attendees will get a complementary 1-year developer license to vSphere Standard product.

Registration for the event is $249 per attendee and to keep with any updates on the event you can follow the VMware Developer Day team on twitter @VMDeveloperDay.

I am planning on attending Developer Day so hopefully I will get to meet and see you all at this event. If you are unable to make Developer Day but are still going to VMworld I want to let you know that I will be presenting again this year. My session is VM2724 - How to Prevent Headaches in your Virtual Environment and I have the opportunity and privilege this to year to co-present with Gene Kim, the CTO from Tripwire. I am really looking forward to this presentation and the VMworld event in general. Tripwire / vWire will have a big presence this year in the expo hall, so make sure you stop by booth 1414 and introduce yourself or just stop by and say hi.

See you all there!!!