Summary of what happened in the past

I had vCloud Director 1.5 and SQL database for it in my home lab which contain two “VMware ESXi 5.0″ and “VMware vCenter 5.0″ respectively.
As it wasn’t functioning well for some reason, I removed both vCloud Director and it’s database and re-installed new vCloud Director 1.5 on the same infrastructure.

And where the problems begin

As I removed old vCloud Director without removing it’s vDC and disconnecting my hosts, it’s agent remain on these two hosts and as I tried to create nee vDC, I get an error on installing new agent.

How to solve it

I searched the internet and in first result saw a great/familiar* line “esxcli software vib remove -n vcloud-agent” which posted by “Chris Colotti” on his site.

I lunched the putty and checked the agent state on both hosts by using below command:

esxcli software vib list | grep vcloud

Then I tried to uninstall the agents on 1st host by running following command:

esxcli software vib remove -n=vcloud-agent

I get an error and removing process failed because it wasn’t in the maintenance mode, so I prepared all things and put this host in the maintenance mode and rerun the command again:

esxcli software vib remove -n=vcloud-agent

Because putting 1st host in the maintenance mode toke a long time I thought there should be a way to preventing it, and the result was lied in the result of this command:

esxcli software vib remove --help

Below screen showing interesting parameter in the “esxcli software vib remove” command

I tried previous command while adding “–maintenance-mode” parameter and it worked perfectly.

esxcli software vib remove -n="vcloud-agent" --maintenance-mode

Note #1:
Running command in the 1st line would be terminated by showing an error while second line command would be finish successfully while it intend to do the same task as first line. adding double quote (“) in these case may prevent some problems.

esxcli software vib remove -n=vcloud-agent --maintenance-mode
esxcli software vib remove -n="vcloud-agent" --maintenance-mode

Note #2:
It was familiar because I used similar command for adding/installing some VIBs from HP.

After couple of weeks, I made a “CA Server” on one of my windows server in my domain and start generating some certificates for most of my servers and services including “VMware View Connection Server 5.0″.
For security reason and as I had internal “CA Server” in my home lab, I changed the policy for SSL on “VMware View” to “Reject the unverifiable connection” until today which I decided to use my iPad again.

While I tried to connect the connection server, I received a warning about its certificate as it changed during importing the new certificate, so I removed the server from list and tried to add it back again.

But following warning shown up

When I checked the server certificate detail it was all right.

But as the iPad didn’t trust my new “CA Server”, it wasn’t accept its relevant certificate as well; so I had to add it as trusted server to my iPad.
For doing so, I browsed my “CA Server” web portal and installed its certificate.

After adding “CA Server” as “trusted root”, I tried to reconnect to the server and this time my server and it’s certificate worked fine

Second, I got my interesting seminar presentation and experience about server and desktop virtualization.
This seminar held in “Simorgh Hotel” with about 150 attendees, containing four presentations and one workshop; I was presenter for third and fourth presentations as well as responsible person for technical part of workshop, from preparing the whole virtual infrastructure, software and devices to the presenting them in the workshop.
Here, let me thanks my friends and team mates “Doomaan, Mehdi and Arbi” for their help and special thanks to my friend and team-mate, “Davoud” for his valuable helps, support and cooperation.
The interesting part in this presentation was that the “projector” was failed just 3 minutes after third presentation start and it made a lot of pressure on the support and management team to fix the problem asap.
As a lesson learnt, I’d check and get the confirmation of availability of second “projector” for the presentation. 

I’ve configured and sent one “VMware Document” for testing purpose to my iPad and it’s done successfully; this document was in “.mobi” format which is compatible with “Kindle” devices and software.

Today, I looked for a different way to open the PDF documents on the iPad/iPhone devices and preferred to use “Kindle App” on my devices.
It ended using built-in capability of this software and iTunes for passing this one as well, it’s so simple.

1. Open the iTunes application
2. Connect the target device (iPad, iPhone or even iPod Touch) and select it on the left panel
3. Select the “App” tab on the right panel so all application would show up

4. Move to the down of the page and “File Sharing” section would be appearing.
5. Select the “Kindle” from “Apps” list.

6. Press the “Add” button below the “Kindle Documents” and here we go, sending documents to the “Kindle App”

Tip #1:
File with “.mobi” extension which uploaded using iTunes in this way will appear in the “Book” section but if same file send to the kindle using email address and Amazon Service will be appeared in the “Doc” section.

Tip #2:
File with “.pdf” extension which uploaded using iTunes in this way will appear in the “Doc” section.

Hope you find it useful

◄ Back

Next ►

Picture 1 of 5

Public URL: https://www.evernote.com/pub/kasraeian/VMvS5CM-FC

Anyway, getting to the main point, in these situation admins may wants to terminate currently  halted VM process and restarting it as soon as possible.
I did this before on VMware ESX 3.5 and ESXi 4.x but this time lets check the process on VMware ESXi 5.0.

First of all get access to the host and for this series of action Local TSM or Remote TSM make no difference; There are two command under “esxcli vm process” which are going to be used for this purpose as shown in the below image:

First of all let’s get all info we need by running below command:

esxcli vm process list

This command will list all VMs in running state and some of their corresponding information like “World ID, Config File and Display Name”, after finding the target VM note it’s “World ID” somewhere for further use.
Now it’s time for doing the main action, but before that just check the “esxcli vm process kill” option in the below image:

As it shown in the above image, there are two options of “type” and “world-id” for “esxcli vm process kill” command.
“-t” or “–type” will indicate the type of kill operation and it could be “soft, hard, force” while “soft” is recommended to use first.
“-w” or “–world-id” is related the running VM and it will be use for selecting and killing process.
Now that we have  ”world id” of VM from previous step, we can use it for killing its process by following command:

esxcli vm process kill  -t soft -w XXXX

After running the command it will end with no prompt or message, just like below:

And for being sure, you can run the “esxcli vm process list” command again and looking for that VM.

Note:
If VMware HA is configured on the cluster which containing this VM, it will automatically restart this VM on the other host from the same cluster.
It can be a way for testing VMware HA configuration (in lab, education, …) to see if it’s ready to protect VMs. 

P.S: based on kill command and list command, you can see which VM made this post possible. 

I knew the result but as I tested that before but I just I tried to rename the new named shared storage to what it was before, but it generated some error about the duplicated name.
I had to unmounts those different named shared storage and mount them back, so first step should be unmounts current ISO files from VMs.
I used “RVTools” which is written by “Rob de Veij” for checking if any VM have mounted ISO files and there was more than 10 VMs.

As I’m trying to learn VMware PowerCLI and PowerShell generally, I decided to do it with some writing so I opened “PowerCLI4.1.1-Print-At-Home-Office.pdf”, looked for CD Drive related cmdlet and came across “Get-CDDrive” and “Set-CDDrive”.
I tried multiple times and finally managed to use it as below to disconnecting all ISO files from all VMs at the same time.

Get-VM * | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$true

* in the above code will select all VMs and this will impact the performance as it will check every single VM, even if they don’t have any ISO mounted; I’ll try to fix this problem.

After disconnecting al ISO files from VMs, I ran the RVTools again and as I expected the list of attached media was empty.
I tried to remove NFS shared folder from first host (ESXi50-1) and it’s finished successfully, I moved to the last one and tried same thing by selecting the datastore (NFS) and using “unmount” option.

After accepting all consequences , I received an error which indicating that this process is terminated because of some usage, related screenshot is posted below.

I ran RVTools again and there was no VM with any ISO image connected. I searched internet for any similar problem and find this thread on “VMware Community”.
Based on “Rich123” recommendation, I used MAP on the vSphere Client for checking the current state and selected only VMs, Hosts and datastores for viewing option, amazing map created as it’s shown in the below image.

Based on MAP view, I had two VMs with active connection to the NFS storage so opened their “Virtual Machine Setting” window, but here things get even more interesting; just check the below images:

Now, here is the question, if there is no ISO file and the VM placed on the iSCSI datastore, what causes the connection?
…
Well, few days back, when I was working on the VMware View 5.0, I made a snapshot from “Parent-XP” for using with VMware View Composer as well as making snapshot from “VMware ThinApp” for making safe and clear rollback point in order to use with “VMware ThinApp Suite”.

When I remembered these events, I removed both snapshots from VM and watched the MAP and “Recent Task” pane for changes.

After successfully removing both snapshots, I checked the possibility of removing NFS and it’s just done so easy.

For adding NFS share to all hosts at the same time with identical name, I used PowerCLI again and it’s related post can be found here.

Summary:
Disconnecting ISO files from VMs might not be enough for unmounting datastore from hosts (NFS in my case) if any snapshot had connection to these ISO files on that specific datastore.
If you get “The resource ‘XYZ’ is in use.” during unmounting datastore, checking the snapshot might be helpful.

Code #001 (Disconnecting CD Drives):
Last night, while I was looked for making some changes on my hosts in my home lab, I ran “RVTools” for checking VMs if they mounted any ISO images, so I can disconnect them.
I searched for a while and mixed some scripts with hope of making this possible and finally get it work with bellow script.
In order of disconnecting all CD Drives on all VMs in a data center, first of all, connect to the proper VMware vCenter with “Connect VIServer” command, like:

Connect-VIServer -Server vc.kasraeian.com -User USER_NAME -Password PASSWORD

Then run below code for disconnecting all CD Drives:

Get-VM * | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$true

Code #002 (Mounting NFS Share on all hosts):
For some reason I faced the problem with NFS share in my home lab so I decided to remove it and add it back to my hosts (two VMware ESXi 5.0).
If you want to use a GUI method using VMware vSphere Client which seems to be easier, you would end up getting some error because of duplicate name of shared storage on different hosts in the datacenter.
So as the VM may want to be migrated (manually by vMotion or Automatically by DRS), you don’t want to mount the same ISO files each time.
For this purpose it’s possible to use VMware vSphere PowerCLI by connecting to the VMware vCenter by bellow command:

Connect-VIServer -Server vc.kasraeian.com -User USER_NAME -Password PASSWORD

After connecting to the central management (VMware vCenter), it’s possible to run following commands and adding specific NFS share with the same name to all hosts:

$THosts = Get-VMHost
New-Datastore -Nfs -Name NFS_NAME -NfsHost HOST_ADDRESS -Path NFS_PATH -VMHost $THosts

I planned some upgrade on one of these hosts so tried to migrate VMs to the other host and placing it in the “Maintenance Mode”, just after my first try, I get an error and migration failed when the process was just about 9 percent;  the screenshot of the error is posted below.

I searched for “Migration to host <> failed with error Already disconnected (195887150).” and found “KB1030267“.
In this KB, it’s recommended to checking network health using three steps, so I did it while I was sure that I enabled “Jumbo Frames” in the past and the firewall was checked in the time of first installation.

I looked into the network configuration (Distributed Switch) for gathering some basic information about current state and IP settings.

While I was looked into the page, I pressed blue “i” sign (information button) next to the “vmk2″ for checking jumbo frames setting and after that the uplink information.

As it’s shown in the above screen shots, the “MTU (Maximum Transmission Unit)” in the VMKernel and dvSwitch is different; I opened the “dvSwitch Settings” window and changed the MTU to 9000.

Waited some times for the changes applied to the both hosts and then tried again for migrating same VM and it finished successfully.

This KB get the right “Resolution” but let me add some notes to it (mainly as reminder) in order to make it more clear; if the “Jumbo Frames” sets on the VMKernel for vMotion in one host, all other hosts should have same MTU as well as connected vSwitches/dvSwitches. Connected vSwitches/dvSwitches should have same or grater MTU so they would let the data exit the host in the first step.