PowerCLInic

Mount NFS datastore using reference host - DNS or IP

2012-02-21T22:18:00.004-05:00

Most people use host profile to configure new host servers to look like your existing host servers. However, this method may not apply to all due to license limitation, HW configuration differences, or any other reasons.

ICT-FREAK.NL blog had great post on using script to reference a existing host to mount NFS datastores on new ESX host server. I needed to do exactly that so I used the script from the blog to mount NFS datastore. It worked great but the problem was that the script does not distinguish between IP mounted datstore and DNS mounted datastore. My reference host server was using DNS to mount NFS datastore and the script was mounting the datastore using IP on my new host server.

Unfortunately, hosts in the cluster will treat DNS mounted datastore and IP mounted datastore as different datastore even if you are pointing to the same volume on NFS datatstore system. I decided to make some modification to the script to make it bit simpler and mount NFS datastore using exact configuration from the reference host.

You can validate the NFS IP or DNS mount from the SSH session or powercli command.
esxcfg-nas -l

Download script from HERE

vCheck6.0 is Released - PowerCLI man does it again!

2012-02-11T17:33:00.000-05:00

Virtu-AL.NET released vCheck 6.0 on 2/8/2012. vCheck 5.0 was the most popular PowerCLI script of all time and 6.0 brings more improvements and same impresseive VMware health checks. Biggest difference in this version is that all checks are written as "plugins" in the form of PowerCLI scripts and they live in seperate "Plugin" folder. What does this mean? It means you can easily create your own checks and easily plug in to the master health check. Be sure to check out Al's blog.

http://networkedblogs.com/tIQmk

Use GUI interface to create error free CSV file for automated VM build

2012-02-10T21:42:00.002-05:00

There are many automated VM build scripts that uses CSV file. I've been doing many of these automated deployments at work. Have you ever had failed automation because you mispelled templates, host name, datastores, clusters, vlan, and etc? I have. It is huge time sink and cleanup can be painful too. I did a little tinkering with Primalforms to create GUI interface that pulls information directly from the vSphere and writes on to the CSV file. There is no possible way to make spelling error other than the VM name it self. No more 2nd guessing on your CSV file integrity.

Once the CSV file is created, 2nd script builds the VM off the CSV and generates audit file after it completes the build. Also, it is huge waste of datastore space when you keep different versions of templates just because you have different CPU, RAM, or disk sizes. The script will resize CPU, RAM, Disk1 and Disk2 (or create new), puts it in to a folder or default folder, and puts the VM in to resource pool if you choose to do so. Enough talking and more screenshots.

Download the CSV Build GUI script HERE
Download the CSV Automated VM build script HERE

At the end of the build, it generates audit for VMs that were built by the script. I canceled the VM build that was on the screenshot... Screenshot below is from another test build that I ran.

VMware Home Lab - Save money with virtual Vyatta router and virtual ZFS storage system

2012-02-09T00:47:00.005-05:00

I built VMware home lab like any other VMware gurus that loves to tinker stuff at home It helps me to to learn new products and test my Powershell scripts without destroying production environment at work. :)

Below are my lab equipment hardware list:

ESXi1
Dell T110 II Power Edge server
E31220 3.1Ghz Quad Xeon processor
16GB of RAM
Five Gigabit Ethernet, including on-board NIC
240GB SATA HD 7200rpm

ESXi2
Dell T110 II Power Edge server
E31220 3.1Ghz Quad Xeon processor
16GB of RAM
Five Gigabit Ethernet, including on-board NIC
240GB SATA HD 7200rpm

ESXi3
Dell T110 II Power Edge server
E31220 3.1Ghz Quad Xeon processor
16GB of RAM
Five Gigabit Ethernet, including on-board NIC
Perc H200 Raid controller (3)1TB 7200rpm drives
240GB SATA HD 7200rpm

HP Procurve 1810G-24 Gigabit managed switch

What is so great about Dell T110 II? It is on VMware HCL. nuff said.

Notice I don't have Router or SAN storage unit. I used Vyatta virtual router for all my routing needs and virtual storage system for my storage. Vyatta router is powerful virtual router that you can download as virtual appliance or install it on Linux for FREE. At home, I have "router on the stick" configuration.
For my shared storage, I went with Solaris VM configured with ZFS. I didn't want to spend more money on hardware storage system so I bought H200 Perc raid card with three 1TB drives and installed it on my ESXi host server. ESXi server does not see the raid card because it is not supported. But, your VMs will see the card with VMDirectPath. Solaris VM uses VMDirectPath mode to access Perc H200 raid card. Perc H200 raid card is connected to three 1TB hard drive. With ZFS, I have NFS, SMB, and ISCSI storage running. It also supports storage snapshots, de-dupes and replication with fantastic performance. If you have more of these ZFS VMs and raid cards, you can setup SRM in your home lab. How cool is that? It has all the enterprise features for FREE. Just remember to check your CPU before you invest your money in virtual storage. VMDirectPath only works wtih Xeon CPUs. I am not sure about AMD processors...

You can learn more about ZFS configuration from the link below. Everyone is raving about it.
http://hardforum.com/showthread.php?t=1573272

Vyatta Router

Virtual Storage System

My Lab

PLINK PLINK! - SSH commands over Powershell

2012-02-08T14:16:00.001-05:00

Have you ever wanted to automate SSH commands using powershell? PLINK is the way to do it. On Powershell script, you can use plink.exe command to pass commands to SSH session. This opens the door to many possibilities. Most IT server manufacturers use SSH sessions to manage the hardware via SSH command line.
I am familiar with HP server devices. On HP servers, you can use SSH command line to connect to iLO and mount CD images via HTTP. You can use this technique to automate your ESX host server firmware update or even build ESX host server from customized CD ISO.

Example:
Scenario: I am a server administrator that wants to update firmware on 12 ESX host servers in the cluster. I want to fully automate the update process with Powershell/PowerCLI script. I do not want any VM downtime. I do not want endless mouse clicks...

Requirements:

HP licensed iLO and IP address
Fully automated DRS cluster
PLINK.EXE ver 0.60_q1.129
Webserver with ISO image configured with correct MIME type

Solution:

New HP firmware update DVDs are automated by default. You can take advantage it to automate the process

Get hosts in the cluster and use ForEach loop to put the server in maintenance mode one at a time.
Use the plink command in the script to mount the firmware update ISO image on the ESX host.
Reboot ESX host and Firmware DVD will kick in to automatically boot from the ISO. It will update the firmware automatically.
When ESX host comes back from disconnected status, take it out of maintenance mode and put next server in to maintenance mode.
Rinse and repeat and you will have hands free firmware update automated script.

So, how do you use the PLINK command to connect the ILO CD?

VMTools Update - Many ways to cut it.

2012-02-08T00:17:00.000-05:00

Updating VMtools can be pain the a$$. We have 6000+ VMs and it is challenging to keep all the VMTools updated. There are many ways to update it: Script it, do it manually, use Update Manager, or use "Check and upgrade vmtools during power cycling" settings in VM option. Thankfully, we have maintenance window for many of our VMs. I like to take advantage of these windows reboot windows to update the vmtools. Note that this option only works on VMs with older vmtools installed. It will not install VMtools on VMs that does not have VMtools running.

There are so many VMs out there and I can't be selecting this options one by one. That's where PowerCLI script comes to the rescue.

Script below will add the option:

Script below will remove the option:

Get-Snapshot - Are we there yet?

2012-02-07T23:53:00.001-05:00

At my work, we are in the middle of performing datastore upgrade. I am using storage vMotion to migrate the VMs live. For storage vMotion to work you need to make sure CD is not attached and there are no snapshots on the VM. On my storage vmotion script, I initially used Get-Snapshot to query the snapshots on the VM. It took very, very long time to query the information and it was too slow for my taste. Thankfully, snapshot info is available in API and I got the snapshot information quicker after I modified the script.

Get-member is my favorite command for any query. It shows you additonal extensions from most values from Get-**** commands. If you drill down in extensiondata, you will find wealth of information about the value.

$vm = get-vm vm_name
$vm | Get-member

And finally, command below will give information if the VM I am looking at has snapshot.

$vm.extensiondata.snapshot

You can loop it using ForEach command to scan through the VMs. It will be ALOT faster than using Get-SnapShot command.

Host Profile update made easy with PowerCLI GUI

2012-02-04T19:01:00.000-05:00

You don't usually associate Powershell with GUI. However, powershell is extremely .NET friendly. You can build pretty impressive Windows GUI interface with it. But, it is not easy writing the GUI code line by line in Powershell. That is why I like to use Sapien Primary Forms. You can build GUI inteface using the graphic utility and it will write the powershell script for you. How awesome is that?

Script below is one of the GUI interface script that I wrote. It checks host profile in the cluster and applies the host profile on single ESX host at a time. It does it by putting one non-compliant host in to maintenance mode by evacuating the VMs using DRS and vMotion. Once the host server is place in to maintenance mode, it applies assigned host profile and takes the server out of the maintenance mode once it is completed. I hope to post more of my GUI scripts in the future.

You can grab my script from here.

Automated ESX Virtual Machine VLAN test

2012-02-04T12:51:00.000-05:00

At work we moved away from standalone servers to HP blade servers with two 10Gbe pass-thru configuration. I recently built 5 ESXi4.1 hosts in to our growing number of ESXi clusters. Before putting the server in to production, we let it bake in the sun for about one to two weeks. After baking is over, we test VLAN and fault tolerance on NICs. After countless clicks to Vmotion a VM_HLPER VM through ESXi hosts, I decided to make script for it. Below is the script for my automated VLAN test using VM_HELPER machine. VM_HELPER is a small VM that helps administrators run tests like this. VM_HELPER should have access to all VM VLANs with IP address.
Be sure that ESX servers are not being used in production. The script will isolate each vmnic from vSwitch0.

NFS mapping via IP or DNS?

2012-02-02T14:55:00.000-05:00

In the place that I work, we use NFS for are datastore mounts. We use DNS to mount our datastores (I disagree with this approach). I was building new servers on a new physical plateform with new network configuration. Because of this reason, I didn't want to use existing host profile to configure the server.

Since datastore mounts will stay the same, I wanted to copy datastore mount from existing ESX host servers to new ESX servers that I built. I was being lazy and searched for the PowerCLI script. Google took me to ic-freak.nl PowerCLI script that will copy datastore mounts from reference hosts to the new hosts. The problem with the script was that it will only mount the datastore using IP although reference host had DNS based mounts.

vSphere will not think IP mounted and DNS mounted datastores are same even if they point to same datastore and folder. vMotion will fail if you try to vmotion a VM between them. You must either mount all datastores via IP or DNS, not both.

Below is the script without using Get-View statement. Without get-view, scripts are little bit faster using .extensiondata.

   
 $viserver = Read-Host "Enter VI server"  
 $refhost = Read-Host "Enter Reference host server"  
 $cluster = Read-Host "Enter Cluster name that has new hosts"  
   
 Connect-VIServer $viserver  
 $REFHOST = Get-VMHost $refhost  
 $NEWHosts = Get-Cluster $cluster | Get-VMHost | sort name  
   
 foreach($nfs in (Get-VMhost $REFHOST | Get-Datastore | Where {$_.type -eq "NFS"} )){  
   [string]$remotePath = $nfs.extensiondata.info.nas.remotepath  
   [string]$remoteHost = $nfs.extensiondata.info.nas.remotehost  
   [string]$shareName = $nfs.Name  
        
      Foreach ($NEWHost in $NEWHosts) {  
        $NEWHOST | Get-Datastore | Where {$_.Name -eq $shareName -and $_.type -eq "NFS"} -ErrorAction SilentlyContinue  
        If ((Get-VMHost $NEWHOST | Get-Datastore | Where {$_.Name -eq $shareName -and $_.type -eq "NFS"} -ErrorAction SilentlyContinue )-eq $null){  
          Write-Host "NFS mount $shareName doesn't exist on $($NEWHOST)" -fore Red  
          New-Datastore -Nfs -VMHost $NEWHost -Name $Sharename -Path $remotePath -NfsHost $remoteHost  | Out-Null  
        }  
      }       
 }

2011-10-16T18:01:00.000-04:00

We were having performance issues wtih Netapp filer in our production environment. We suspected that there may be problem with disk alignment (check out yellow brick's post regarding disk alignment) and VMs with high disk I/O. I was asked to provide VM I/O report for the VMs that are on the datastore. Unlucky for me, our production environment is on ESX 4.0 and this version does not have VM I/O vSphere API. VM I/O is availble from ESX(i)4.1 and up. Since Tivoli was our choice for server monitoring, I had to turn to Tivoli IT guys for help. They were able to produce the I/O report for me. So, IF we were on 4.1, how would I pull the information for VM disk I/O?

Lets take a look what is available for Virtual Machine monitoring from vSphere API.

http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/index.html

Counter is type of stat you will be using to look up disk I/O. Levels is the monitoring level that you can define on vsphere. Level "2" is the minimal requirement for this counter to work. Finally, this entity is for for virtual machine only.

If vSphere logging level is set to less than 1, these stat types are not available. See examples below.

Get-StatType cmdlet will give you type of stats that are available for the entity.

You should use any vm name that is up and running in the environment.

Get-StatType -entity (get-vm vmnamehere)

Level 1 logging does not provide much logging and as you can see from the screenshot, virtual disk I/O is missing from the avaiable monitoring list.

vSphere level 2 logging will provide more performance stats at the expense of more SQL disk usage.

Get-StatType -entity (get-vm vmnamehere)

Virtual disk montoring is avaiable for the VM along with alot more monitoring types. Other monitoring types is cut-off from the screenshot because list was too long.

So, How do we apply new found stats for virtual machines? We use Get-Type cmdlet to get the job done.
Example below will pull the stat data from 10/11/2011 to today's date.

>Get-Type -entity (get-vm vmnamehere) -stats virtualdisk.read.average -start 10/11/2011 -finish (get-date)

First blog post

2011-10-15T12:12:00.000-04:00

I don't know who is reading my first post. But if you are reading this, you've just won $10,000. Okay, I'm kidding. On this blog, I will be focusing on VMware technologies and scripting with PowerCLI. Couple of months ago, I didn't know jack about Powershell and PowerCLI. Now, I can script with confidence and picked up few tricks and tips along the way. Virtual, LucD, Reuben, and PowerCLI bloggers around the world inspired me to push further in to the world of scripting and automation. I hope to share my knowledge and be helpful to you to get what you need in the world of virtualization.