VMwise

What Happens When You Pull Boot Luns from an Active 4.1 ESXi server?

KjB — Mon, 26 Mar 2012 19:17:36 +0000

A very good question. Not one that I would recommend attempting to find out in production, but which I had the opportunity to experience firsthand. If you’ve read some of my more recent posts, you’d find that we’ve been going through some SAN migration tasks, and we’ve been moving a lot of storage from one location to another. Last week, we had a particular project to evacuate a few LUNs that were on an array which was needed by another project. So, we do our typical storage vMotion tasks and move all of the virtual machines off. Then, we go through the tedious process of deleting the datastores, creating rules to mask the LUNs that we just cleared out, and then running the rules to mask out all of the now-evacuated LUNs. A very tedious process which has been shortened, albeit only somewhat from the command line, in vSphere 5 with the advent of the unmount and detach methods now available. After performing the tasks, we are about as ready as we are going to be and have given the go-ahead for the LUNs to be reclaimed. It is my fault for not also double-checking that these aren’t one of [...]

vCenter Custom Event Trigger Alarm for Permission Change

KjB — Mon, 27 Feb 2012 19:43:25 +0000

After our last audit, there were a couple of areas that were found to be lacking, in terms of proper identification and monitoring. One such deficit, was the lack of a way to properly monitor when permissions are changed in our environment. As our environment has grown, we’ve had to give custom permission sets to several teams outside of the typical vSphere admin group. These permissions, are of course, added at the lowest possible level, to give rights on as few objects as possible. So I started to find a way to alert when these changes took place. After all, they are recorded in the Events tab, so there should be a way to report on it. I started looking through the custom alarms, and couldn’t find one that worked the way I wanted it to. There is a datacenter based alarm, but that alarm did not report on permissions changed at a lower level, such as on a resource pool, or a host, or even a vm itself. After stumbling around trying to find an answer, I came upon a document created by Horst Mundt, back in 2010 that outlined precisely what I was looking for, a way to [...]

ESXi and Exalogic

KjB — Fri, 17 Feb 2012 19:03:28 +0000

We recently had an influx of several Oracle Exa* machines, and various testing has been taking place with these systems. While I won’t comment on OEL, and OVM, at least not yet, I did get a chance to play with a couple of compute nodes to see how well they would run ESXi. A quick search on the Oracle website showed that the compute nodes themselves are actually the Sun x4170 M2. The Sun X4170 is on the VMware HCL, so Step 1 is a go. The other distinguishing component is the addition of a QDR infiniband card, this one seems to be supported, step 2 check. Next thing is to just attempt an install. Step 3. Check. The install went through without a hitch. The drivers for the IB card, however, is not part of the image, and needs to be downloaded separately. So, download the driver here for 4.1 specifically, but drivers do exist for 4.0 and 5.0 as well. Once downloaded, you can use either esxupdate or vihostupdate.pl to install from the zip file. esxupdate –server–bulletinupdate vihostupdate.pl –server–bundle <MLX-OFED VIB> –install After the install of the driver, and a quick reboot, you will see one new 10 [...]

How to Identify ESXi Boot LUN

KjB — Tue, 07 Feb 2012 14:44:45 +0000

I recently posted about my misadventures into SAN array migrations and RDM’s here and here. Glad to say that part is at least completed. Now, it’s time to move off of the array all together, and get my boot LUNs from somewhere else. First thing’s first though, and I need to make sure I know which LUN I am actually booting from. This would typically be easy enough, as we use 6 GB LUNs for our ESXi installs. But, what if you end up with more than one, such as in the case of a migration where a new 6 GB LUN was presented and you need to boot to the other. Still, fairly simple enough, but let’s go the paranoid route of figuring out which LUN we are booting from, just in case. Not always the most obvious method, but it works in 4.1 and in 5, albeit with slightly different syntax. Log into ESXi, ssh or console. Run ‘ls -l /’ to get the UUID’s of the bootbank and altbootbank: ~ # ls -l / lrwxrwxrwx 1 root root 49 Oct 27 17:55 altbootbank -> /vmfs/volumes/bebbef72-6cbc41fa-b169-68d3824c6d51 drwxr-xr-x 1 root root 512 Sep 17 01:11 bin lrwxrwxrwx 1 root [...]

SvMotion with RDM with Windows Dynamic Disks

KjB — Mon, 30 Jan 2012 14:31:51 +0000

I recently had to move several virtual machines off a migrating array, and posted about my success with RDM’s to thick VMDK here. I also found that a few of the virtual machines I had to move included Windows dynamic disks, specifically, spanned volumes. Seeing as the largest volumes previously were 2 TB in size, in a few cases where larger volumes were necessary, windows spanned volumes were used to overcome the limitation and present a larger drive/disk to windows. Seeing as I hadn’t read anything specific about the success / failure of this type of storage migration, I thought I’d post my own experience. SvMotion works as expected, as I posted previously. It also worked with the dynamic volume. Not sure why I was concerned, disk is disk, I suppose, but I was being cautious nonetheless, and the disks migrated just fine. It’s good to see that layering complexity on top of complexity doesn’t always break the base functionality, as I’ve seen elsewhere. -KjB

SvMotion RDM to VMDK

KjB — Tue, 24 Jan 2012 14:28:24 +0000

We recently had need to evacuate one of our busy SAN arrays. For the most part, this was easy enough. Thanks to the storage vMotion (SvMotion) capability, it was easy enough to move the virtual machines without much difficulty using the normal migrate processes while the vm’s were up and running. The trouble comes with Raw Device Mappings (RDM’s). We have a general policy that when a large disk is needed, it is added as an RDM, as opposed to taking up an entire large datastore for this purpose. This has worked very well for us in the past, but in this case, it caused a bit of an issue. First, we needed to migrate this RDM, and since our SAN-based replication/migration tools did not cross vendor boundaries, this was not as simple as pushing a button. We needed to do the change with as little disruption as possible, and to avoid downtime as a general rule. Now, going back to the SvMotion, if you leave the defaults in tact, the RDM is moved to a new datastore as an RDM. You can, however, change this behavior by taking a couple extra steps. First, select migrate as you would normally, [...]

When Disks Fail, Sort Of!

KjB — Tue, 17 Jan 2012 20:45:37 +0000

Another interesting one for folks. This is the first time I’ve seen this, as most of my storage comes off of my SAN. We started getting alerts for virtual machines, and after pulling up the vm console, noticed it was black. Tried to reset the vm, but the reset task just sat ‘In Progress’ for without progressing any further. Logged onto the host and ran ‘vim-cmd vmsvc/getallvms’ to get a listing of all the current vm’s, and I got an error ‘Failed to login: SSL Exception: The SSL handshake timed out ‘. That sounds like a hostd issue, so attempted to restart that on my ESXi 4.1 box, /sbin/services.sh restart. All looked ok, until the script stopped at the ‘USB Arbitrator’ section. Uh-oh. Stopping there typically points to a disk/storage issue. Started going through the logs, and sure enough, there are errors trying to hit a certain path, as well as some new entries: vmkernel: 0:00:04:30.418 cpu5:4101)<6>megasas_service_aen[8]: aen received vmkernel: 0:00:04:30.418 cpu1:4409)<6>megasas_hotplug_work[8]: event code 0×0071 vmkernel: 0:00:04:30.461 cpu1:4409)<6>megasas_hotplug_work[8]: aen registered vmkernel: 0:00:05:28.470 cpu10:9067)VSCSI: 2245: handle 8196(vscsi0:2):Reset request on FSS handle 3474254 (2 outstanding commands) vmkernel: 0:00:05:28.470 cpu3:4215)VSCSI: 2519: handle 8196(vscsi0:2):Reset [Retries: 0/0] vmkernel: 0:00:05:28.470 cpu3:4215)megasas: ABORT sn 73067 cmd=0×28 retries=0 [...]

DC VMUG Jan 17th @ Nats park

KjB — Fri, 06 Jan 2012 15:41:00 +0000

Finally getting back into the swing of things with the new year. So, for my first post of 2012, a plug for the local VMware Users Group Meeting (DCVMUG). The event is @ Nationals Park on January 17th, 2012. Should be a good event, with presentations from Veeam and Tintri, and of course a forum for your questions at the end. Come if you can, should be interesting. -KjB

View Event Database configuration error

KjB — Fri, 18 Nov 2011 16:04:41 +0000

I’ve been hunting this problem down for almost a week now, and was finally able to resolve the issue. Having spoken to others, not sure why we were affected by this, but I suppose that is life, and if nothing else, I got to dig into the weeds with the View Event DB configuration. We’ve apparently been having this problem for some time, but it only recently was brought to my attention. We have been trying to add an Event DB to our view configuration, which comes in quite handy when it’s working correctly. Problem is, every time we’ve tried to add it, we’ve gotten an error: An error occurred while attempting to configure the database. Double check the database parameters and ensure that the database is not down, restarting, or otherwise unavailable. So, right off, tried to make sure the my db was good. Made sure db was up and running. Used SQL Server Studio to verify connectivity, check. Used ODBC to check connectivity, check. All was well with the db itself. So, I went back into the view server and tried again, making sure all the configuration was correct. In case you don’t know, the config screen is [...]

View desktops orphaned, disconnected, invalid, and console unreachable

KjB — Thu, 17 Nov 2011 20:50:31 +0000

Started troubleshooting a few issues with our View 4.6 environment and generally cleaning things up, and ran into an issue that I found rather annoying to fix. We started receiving provisioning issues from the connection brokers. Provisioning desktops would fail, desktops would fail to power on. And the ones that were on were not always shown as running. So, I used my handy client to connect to vCenter, and noticed issues issues with tasks failing claiming the host was in a disconnected state. Checking the host, showed the host perfectly happy. Tried to connect to a few vm consoles on the host, would fail stating an MKS failure citing a login failure. Hmm, that was strange, logging into the host showed no errors with regards to login problems in any of the local system logs. Since this was not going well, I figured I’d try the usual suspects, and restart the management agents. So, I logged into the troubled host and ran ‘/sbin/services.sh restart’. That produced another volley of errors, showing problems with .lock file issues and unable to lock /etc/vmware/esx.conf. That was a bit disconcerting. Ran stop on the service instead to check things out, /sbin/services.sh stop’. Ran it [...]