A lot of my day-to-day work involves building and tearing down various configurations containing ESXi-hosts. For this, I use my laptop; a Dell Precision M4600. This means that the ESXi-hosts are virtual machines. I use vESXi templates and linked clones inside VMware Workstation to speed things up and save on disk space, instead of rebuilding each host anew.

I use the concept of templates and linked clones for my Windows virtual machines as well. With Windows, we get the advantage of sysprep to generalize the Guest OS, but with ESXi, such a tool doesn’t exist. We need some workarounds to make sure the Guest OS installation is generalized so each linked clone has its own unique identity.

A friend and co-worker (thanks Sjors!) pointed out that the ‘Reset System Configuration’ option in the DCUI might offer the required sysprep-like functionality. This option executes ‘/sbin/firmwareConfig.sh’, which basically removes the state (state.tgz and boot.cfg files in /bootbank) of an ESXi-host and reboots the machine. By executing this script before we shutdown the virtual machine and marking it as a template, we follow the same basic workflow as compared to sysprep (where you run sysprep and choose generalized OOBE).

Build the Virtual Machine

Make sure the virtual machine is created in the correct fashion, for either VMware Workstation 8 or vSphere 5 as the host platform. The document ‘Running Nested VMs‘ on VMTN explains it pretty well. Eric Gray does an even better job explaining how to nest ESXi on a physical ESXi 5 host.

For Workstation, the process is pretty easy. Create a VM, Guest OS type ‘VMware ESXi 5′. Make sure you enable the virtualized hardware-assisted virtualization in the VM’s settings (‘Virtualize Intel VT-x/EPT or AMD-V/RVI’).

Install ESXi

Installing ESXi isn’t a hard thing to do. Just randomly hit [enter] and [f11], and you’ll have ESXi installed. I used vSphere Update Manager to apply the latest firmware to the host (to save time on deployment later on).

Generalize ESXi

I actually took two separate actions to generalize ESXi:

• From the DCUI, I deselected vmnic0 as a network adapter to be used for the management network and removed the DNS suffix. I then proceeded to restart the management network.
• I then executed ‘Reset System Configuration’. Power the virtual machine off as soon as it reboots, though!

The script behind ‘Reset System Configuration’ does re-generate SSL thumbprints (certificates) and SSH fingerprints. Along with new MAC adresses (generated when you deploy linked from the template) and a different name and network identity (which you configure the newly deployed host with), each linked clone you deploy based on the template is unique.

Caveat: VMFS UUID

I used a small SCSI-disk of 5 GB, since I rarely use local storage for virtual machines. But when I used a 40 GB disk (I think the New Virtual Machine Wizard defaults to this size), the ESXi installer automatically creates a local VMFS-datastore called datastore1. The script behind ‘Reset System Configuration’ does not change or erase the VMFS-datastore. It doesn’t touch the VMFS-datastore at all. When deploying multiple vESXi VMs based on a template that contains a VMFS-datastore, you probably will encounter all kind of funky stuff as the VMFS UUID is not re-generated. Please make sure your template vESXi does not contain a local datastore!

Mark as template

The last step in creating the template (on VMware Workstation) is to mark the virtual machine as a template (‘Enable Template Mode (to be used for cloning)’) and create a snapshot (I named it ‘Reset System Configuration’).

Do you know this page on vmware.com all too well?

Not being able to access the bits and bytes you need to evaluate a product and install it in your lab is just the worst. Us geeks can’t handle this kind of rejection, VMware!

Please bring back the VMTN Subscription! Josh Atwell has posted a perfectly viable proposal on how this could work. Please show your support here.

VMware Labs presents its latest fling: VDSPowerCli.
PowerShell is a scripting language Microsoft developed to help administrators manage the Windows environment. Third parties can write their own snap-ins (dynamic linked libraries) to implement new commands, which are called cmdlets. With VDSPowerCli, users can use the cmdlets provided by PowerCLI to manage vSphere Distributed Switch(VDS).

Features

VDSPowerCli gives you the ability to manage:

• VMware vSphere Distributed Switch
• Distributed Port Group
• Distributed Port

Download and try out VDSPowerCli!

I am digging deeper into HP P4000 (LeftHand) SANs again. This has been a consistent side-project of mine for over two years now. I started using the VSA as a simple, fast and reliable appliance for a shared storage and SAN workshop I’ve delivered at my employers training center, and did a real life implementation of a production environment a while back.

I noticed that when designing a P4000 solution with a number of factors like Multi-Site, stretched HA/DRS clustering, MPIO combined with vSphere, you need to be extra carefull and really kick the tyres of the set-up in a lab to get to know all the gory details. Today, I tested a couple of those factors in the design.

Routed iSCSI

The first pitfall is an obvious one. VMware’s software iSCSI Initiator cannot do routed iSCSI. Multiple paths cannot be routed across those subnets by the ESXi 5 initiator. This effectively means that you can only do a P4000 Multi-Site SAN with a single Virtual IP.

With VMware ESX, the preferred configuration is a single subnet and with ESX servers assigned to sites in SAN/iQ. (source)

This actually has an advantage or two, too. Failover between sites is transparent; the ESXi-hosts do not notice site failure. Assuming your hosts didn’t go down as well during the failure, obviously.

Multipath I/O with vSphere

One of the things that caught my eye is a quirk with round-robin MPIO in a vSphere environment.

A P4000 stripes data in a volume across storage nodes in a site in a RAID0-fashion: each storage node hosts an equal part of the volume’s data. I tested this today by creating a simple three node cluster, creating a volume and writing data to that volume. Before the test, each of the three nodes had 1,37 GB provisioned. After I had written some data, all three nodes had 1,65 GB provisioned. This proves data is written in equal sizes chunks to each of the nodes.

When using MPIO in a vSphere environment, each volume only has one Gateway Connection:

But when using the P4000 DSM for MPIO on a Windows machine, each volume has a Gateway Connection to each of the storage nodes (and an additional ‘control’ session). If the host is assigned to a ‘site’ in the CMC, it’ll only connect to storage nodes in the local site.

This difference in how the connections between a host and the storage nodes for each volume is made dictates how I/O a given volume flows to the SAN:

With vSphere, all I/O goes through the node hosting the Gateway Connection. ‘s1v3′ (third VSA in the first site) is hosting the Gateway Connection and sends back-end I/O to other (local) nodes to fetch or write data. This makes a single storage node the bottleneck for I/O for a specific volume; and makes balancing which storage node hosts the Gateway Connection for a given volume very important: you’d want every storage node to hosts an equally spread number of Gateway Connections. I would recommend creating one or more volumes for each storage node. In a six-node storage cluster, create six (or twelve) volumes.

When using VIP and load balancing, one iSCSI session acts as the gateway session. All I/O goes through this iSCSI session. (source)

While with Windows and the P4000 DSM for MPIO, I/O flows from the host to each storage node. In this case, there’s no problem with a single storage node hosting a Gateway Connection like with vSphere.

In the graphs, you can clearly see a different I/O pattern for a vSphere (ESXi 5) host and a Windows machine using the HP P4000 DSM for MPIO. There is a down-side to using the DSM: the number of iSCSI sessions from a host is much higher, possibly up to the maximum number of iSCSI sessions supported by the OS and in fact limiting the number of servers or volumes.

Note: The difference in speed is irrelevant, but probably caused by the fact that I’m running six VSA’s and a Windows VM on a single laptop. I wanted to show how ESXi only utilizes a single path while Windows uses all paths; the goal of these measurements where not to compare speed (MB/s).

Which node will host the Gateway Connection for a volume?

With a Windows server with the P4000 DSM for MPIO, this is superfluous. It simply doesn’t matter which host will host the Gateway Connection for a host. But for an ESXi-host, it does.
This brings me to one of the worst bits of the HP P4000 SAN (and especially its documentation!): with VMware vSphere, there’s no real way of influencing SAN/iQ’s decision which storage node will host the Gateway Connection for a volume. In fact, creating ‘sites’ in the Centralized Management Console (CMC) and placing both ESXi-hosts and storage nodes in a specific site will limit SAN/iQ’s choice of storage node to host the Gateway Connection: only storage nodes local to the ESXi-host first connecting to the volume can host the Gateway Connection.

This is a very bad thing. Using two scenario’s, I’ll explain why:

Site failure

Imagine all the Gateway Connections roles balanced evenly over all the storage nodes. Your environment is functioning perfectly. Suddenly, the primary site fails. Gateway Connections for all volumes that were bound to a storage node in this site are transparently failed over to the secondary site. The VMs that went offline during the power outage (since the ESXi-hosts went down as well) will be restarted using VMware HA. Everything works as expected, and your design has saved the day.
That is, until you restore the primary site. Since there’s no way for the administrator to manually failback the Gateway Connections to the primary site to restore the evenly balanced pre-failure situation, the volumes are stuck on the secondary site. In my experience, there is simply no way to recover from this, other than messing about with reboots of the storage nodes hosting the Gateway Connections. Obviously, that’s not ideal.  Only when the storage node hosting the Gateway Connection goes offline, the Gateway Connection is moved.

There’s no other manual way a Gateway Connection can be moved, as far as I know. The ‘RebalanceVIP’ doesn’t rebalance Gateway Connections, although a lot of people claim rebalancing should happen automatically (in SAN/iQ 9 and up, anyway), I haven’t seen this work for a vSphere environment. Stopping managers on the storage nodes might actually do the trick, but I have not seen consistent failover of Gateway Connections. Lastly, you could run more managers (and designate your primary site as ‘primary’). This might make SAN/iQ prefer to host a Gateway Connection on a storage node on the primary site. If you have more information on how designating a site as primary will affect Gateway Connections, please reach out to me. Leave a comment below or ping me on Twitter.

First boot

The other scenario is just as bad. Imagine you are booting your environment for the first time, or you are recovering from a total shutdown (all storage nodes went offline). The first ESXi-host to connect to a volume will bind the Gateway Connection to a storage node in the local site. And since an ESXi-host usually accesses all datastores, all volumes are bound to a storage node in the local site, leaving no volumes left for the ESXi-hosts on the other site to connect to locally. Also, this limits the performance capacity of the SAN: only storage nodes on a single site will handle active I/O. The storage nodes on the other site will only do Network RAID-10 I/O replicated to them.

To avoid this problem you can do a little magic trick. Before you boot up the ESXi-hosts, remove them from both sites in the P4000 CMC. This will ensure an even spread of Gateway Connections between storage nodes on both sites, because even though the first booted ESXi-hosts connects to all volumes, SAN/iQ will balance Gateway Connections evenly across all storage nodes, regardless of their site affinity. Afterwards you can restore site affinity for the ESXi-hosts, but make sure to doublecheck the Gateway Connections first.

Site-affinity for Virtual Machines

So, we’ve now established that only a single storage node will handle I/O for a given volume. That means that all virtual machines’ I/O to that volume is handled by that storage node. If you’re running a Multi-Site SAN and a stretched vSphere HA/DRS cluster, you might run into trouble: an ESXi-host on site 1 can use a storage node on site 2 for virtual machine I/O. And that I/O gets replicated (via the Network RAID-10 synchronous replication in a Multi-Site SAN) back to a storage node in site 1. That means double the I/O is flowing over the intersite link, and potentially leading to high latency.

The solution is rather simple and  just a tiny bit elegant:

DRS Groups and Rules

By grouping all the virtual machines in a single volume into a DRS Virtual Machine Group, grouping the ESXi-hosts on a site and creating a vm-to-host DRS ‘should’ rule, you can enforce virtual machines to run locally. There’s one big catch though: since we cannot control which storage node hosts the Gateway Connection for a volume, so we have to manually find out which storage node (and thus which site) is hosting the Gateway Connection for a volume. We cannot guarantee that this mapping of a group of virtual machines to a site will match real life. I’m sure @PowerCLIMan can create a script that runs as a scheduled task inside vCenter to correct for this, though.

How to prevent all this Gateway Connection stuff

Simple. HP needs to create a VMware PSA multipathing plugin (MPP) for the P4000. As seen with the DSM for MPIO on Windows, the P4000 is working fine with a special plugin from HP. We just need this plugin on VMware.

On the other hand, you could do the following:

Leave the storage nodes assigned to sites; this removes the ‘random’ Network RAID-10 issue where a volume might be synchronously replicated within a single site, leaving the other site without a replica. Instead of placing the ESXi-hosts in sites, just leave them unassigned. This way, SAN/iQ does not know where the ESXi-hosts reside, and will happily allow all hosts to connect to all storage nodes. It’s still a limitation of one storage node handling all active I/O, and you’ll increase the amount of I/O travelling between sites, because you now will almost surely have ESXi-hosts on site 1 connecting to storage nodes on site 2 randomly. This could increase latency, too, because an I/O between a host in site 1 travels to the storage node in site 2. This storage node replicates this I/O back to a peer storage node in site 1.

I’ve migrated my RSS feed over to FeedBurner.
FeedBurner should automatically redirect the current feed to the new feed.

It shouldn’t matter which feed URL you’re using, but if you’re experiencing any problems, please use the new feed URL and tell me.

You just don’t talk about the family. Respect the omerta, Monster VM.

My first try to publish the video wasn’t all that successful; but I’m using the iPad and only just discovered iMovie. That makes editting the videoclip much easier. I’ve now cut out the unneeded bits and uploaded the marketing video and actual product demonstration again.

Please visit the VMware booth at VMworld 2011 Copenhagen to see a demonstration of the device itself (running the MVP) and the management software (Horizon Mobile). It’s well worth your time!

I’ll be attending VMworld 2011 Europe in addition to the conference in the United States a while back. The Europe event is the first self-funded (well, mostly) trip to VMworld; that’s just because I think VMworld is awesome. I’ve got the luxury that I have spread attending break-out sessions between both editions, so my list of recommended sessions is huge.

Recommended Break-out Sessions

Below are the sessions that I absolutely recommend:

Attended in Las Vegas

• EUC1945 What’s New in VMware Workstation
• LAS4000 VMworld Labs Hardware Architecture
• LAS4002 VMworld Labs Using Auto Deploy with Stateless ESXi
• VSP1682 VMware vSphere Clustering Q&A
• VSP1708 VCDX Panel Defense Preparation
• VSP1926 Getting Started with VMware vSphere Design
• VSP1956 The VMware ESXi Quiz Show
• VSP3067 Mythbusters Goes Virtual

Will attend in Copenhagen

• BCO2395 Stretch Clustering – Now and Beyond
• BCO2479 Understanding VMware vSphere Stretched Clusters, Disaster Recovery and Planned Workload Mobility
• BCO2874 vSphere High Availability 5.0 and SMP Fault Tolerance – Technical Overview and Roadmap
• EUC2956 VMware Mobile Virtualization Platform: Solving Enterprise Mobility Challenges
• VSP1708 VCDX Panel Defense Preparation
• VSP1823 VMware Storage Distributed Resource Scheduler
• VSP2227 VMware vCenter Database Architecture, Performance and Troubleshooting
• VSP3116 VMware vSphere 5.0 Resource Management Deep Dive
• VSP3205 Technology Preview: VMware vStorage APIs for VM and Application Granular Data

Will grab audio/video/presentation from VMworld.com

• BCA2817 Designing vSphere Platforms for Maximum Tier 1 Application Performance
• BCO3334 Site Recovery Manager Future Product Directions: Cloud-Based Disaster Recovery
• BCO3420 Avoiding the 16 Biggest High-Availability and Distributed Resource Scheduler
• CIM1940 10 Best Free Tools for VMware vSphere Management in 2011
• CIM2452 VMware vCenter Operations Technical Deepdive
• CIM2680 Managing a VMware Environment: What To Do In The First 180 Days
• LAS4003 VMworld Labs Core Infrastructure Architecture
• LAS4004 VMworld Labs Cloud and Virtualization Software Architecture
• SPO3962 Enabling Business Continuity with VMware Metro vMotion Using Cloud-optimized
• VSP1425 Ask the Expert vBloggers
• VSP2384 Distributed Datacenters with Multiple vCenter Deployments: Best Practices
• VSP3255 VMware Storage vMotion Deep Dive and Best Practices
• VSP3866 Performance Best Practices and Troubleshooting
• VSP3867 Top 10 Virtualization Support Issues
• VSP3868 VMware vStorage Best Practices

Recommended Hands-on Labs

My favourite hands-on labs are:

• HOL04 Reducing Unplanned Downtime in Your Virtual Environment
• HOL05 Datacenter Migration and Disaster Recovery Protection for Your Virtual Environment
• HOL10 Advanced Troubleshooting and Performance Tuning for Your Virtual Environment

Other cool places

VMworld has a lot to offer. There are a few hidden gems, that I feel you must attend during your stay:

• VMware Company Store
• Alumni Lounge (located opposite Hands-on Labs); you even get an Alumni gift!
• Community Lounge. It’s the perfect spot to meet-up, interview, network, blog, tweet, stream/record video, take photos, podcast or just rest your feet.
• VMware User Group Booth in Solutions Exchange; Stop by the VMUG booth to learn more about membership and how you can get the VMUG Advantage (exclusive to VMUG members).

Cool Solutions Exchange Exhibitors

• Dell
• Drobo, Inc.
• FalconStor Software
• Nexenta Systems
• Nimble Storage
• OVH
• QNAP Corporation
• Safari Books Online
• Samsung Electronics, Co. Ltd.
• StarWind Software
• Tintri
• Veeam Software
• Zerto

Guides

I have already checked out the various guides available. In the Program Guide, I learned that, like last year, attendees will receive a travel pass that gives you free passage on the Metro and train system in Copenhagen during the event. Also, be aware of the limited schedule of the (free) airport shuttle: it won’t run after 8pm on Sunday.

Be sure to read through these guides

• Program Guide
• Pocket Guide
• Social Media Guide
• VMworld Partner Guide
• Gatherings & Tweetups

Workstation 8 promises to be a major release.

Besides upgrading the virtual hardware to level 8 (in line with vSphere 5), which brings 3D and vSMP optimizations, as well as expanded support for nested hardware virtualization, 64GB vRAM and HD Audio, the GUI has had a major upgrade. We finally have a ‘summary’ window for a VM, displaying the status of the VM like we’re used to in vSphere. The navigation pane has been replaced with a much cleaner, faster and searchable inventory view, there are live thumbnails everywhere (literally!), the power controls have been simplified, they’ve added a fully functional toolbar when in fullscreen mode (which makes the fullscreen mode less cripled and on par with the normal mode).

Also, the only reason anyone ever used VMware Server was the headless mode. This has now been re-introduced in Workstation with ‘VM Sharing’. Just drag a VM to the  sharing wizard, and the VM will start/stop with the host, regardless of logged in users. You don’t even need to run Workstation. From a remote Workstation console, you can access this VM. Using hostd’s permissions and roles, Workstation has implemented a granular and familiar way to control access to the shared VMs.

Probably the coolest gimmick in Workstation 8 is the connectivity to a vSphere or vCenter host. You can manage VMs running on vSphere directly from the Workstation Console. While you’re at it, just migrate those Workstation VMs to vSphere; you can do that, too. Or, alternatively, create a new VM on vSphere from within Workstation.

The ‘Teams’ concept has been replaced with folders (where the team functionality like network segments and bandwith throttling are now available to all VM’s).

I want to share a little video shot at the ESXi Quiz Show. @JTroyer is a great quizmaster, and the teams really made an effort to make the quiz show the funniest breakout session at #VMworld.

I’m liveblogging from the Tuesday technical keynote. Steve Herrod is explaining the VMware product portfolio. He’s concentrating on data and applications instead of devices, VM’s and other technical stuff on the back-end. With the slogan ‘simplify, manage, connect’, he’s now focssing on the second phase in VMware’s strategy: desktop. He breaks it down into services; desktop, app, data services with a unified service broker to connect users from any device.

With the help of View, desktops are broken down and simplified. With ThinApp, the apps are extracted and published. With ThinApp Factory, the process of extracting apps from Windows is automated, so publishing the apps in a catalog is hugely simplified. ThinApp Factory and Horizon are integrated for a full blown user experience.

Project Octopus is a Dropbox-like data service. It’s an enterprise oriented cloud file sharing service with features like policies, public/private cloud integration and more.

There’s a demo showing about the user’s perspective of using View, ThinApp, Horizon and Octopus. It’s showing a user’s first experience, including the Mobile Virtualization Platform. It’s ver cool that MVP is included in the keynote for the first time, it’s been lingering for years… The ‘work’ phone VM is called ‘the magical phone’. That’s just how many users might experience this MVP feature..

With the help of Horizon Mobile, the apps that were installed on the desktop are pushed to the mobile work phone. With partnerships with both LG and Samsung, the MVP will really take off in the upcoming months. I cannot wait to get my hands on this!

Now Octopus is showcased on the iPad. For the first time, VMware AppBlast is showed to the public. How sweet! Very subtle, VMware is showing AutoCad from within a View desktop.

Now shifting to the back-end infrastructure supporting all of this awesomeness, with a little siderack to the vSphere Cient for the iPad, now with vMotion support. Literally just swipe a VM to the destination host

After highlighting a couple of ‘firsts’ accomplished by vSphere 5′ Steve is talking about accessible innovation with VMware Go, the vSphere Storage Appliance. Bruce is doing a demo on these two products. First, a Windows host is provisioned with ESXi with the help of VMware Go. Both products are targeted towards the smaller shops. For the larger shops, there’s AutoDeploy to provision a group of hosts with ESXi in a repeatable, compliant manner.

Now shifting to some of the features of vSphere: storage pooling, VM placing on storage pools and tiers, storage DRS, storage and networking I/O Control. Next up of IP Adress Management, and specifically the problem with IP-addresses: besides being a unique identifier, it’s bound to a location. With VXLAN, VM IP-addresses can roam amongst different physical locations. Basically, it’s MAC over IP over UDP. With VXLAN, one of the biggest challenges with Site Recovery Manager has been resolved. How cool!

Last subject for this technical sessions is management, subdivided in three sections: monitor, correlate, remediate. In a sneak peak, VMware is disclosing it’s strategy on their manaement solutions. With vServices, applications inside VM’s are discovered automatically ang agentless and shown in the vSphere client. Also, dependencies between apps and VM’s are disovered and monitored.

VMware has announced new and improved offerings in their vCloud initiatives.

VMware vCloud Global Connect

Global Connect offers a global network of VMware vCloud Datacenters consisting of vCloud Datacenter certified VSPP Service Providers. Basically, different providers set up a business agreement / partnership to deliver a standardized global cloud service. All of the vCloud Datacenter program members have standardized on a strict base architecture and SLA to ensure global compatibility. With a single contract, a customer is able to migrate his workload to a different provider. The contracted provider is responsible for the SLA, regardless of the provider actually hosting the workload.

VMware vCloud Portal

vcloud.vmware.com is a gateway to various providers. In this portal, you can find a suitable (be it local or providing the right SLA) vCloud Provider using a map (and dig up detailed information about the provider) and test drive their offering. Initially, VirtaCore Systems, Colt, iland Cloud Infrastructure, Colt and BlueLock will be available from within this portal.

The vCloud is expanding

The vCloud Datacenter is expanding: Colt is expanding massively in Europe and Dell joins the vCloud Datacenter program. The number of vCloud Powered providers has risen to over 40. In total, there are 5600+ partners in the Service Provider program.

vCloud Connector

The vCloud Connector is a piece of software interconnecting different vCloud Providers. With the Connector, you can view, copy and deploy VMs and Templates across your own vSphere environment and the private and public vClouds out there. You can also do some basic interaction with your VMs. At this point, you can perform power operations, access the console and launch the vCloud Director GUI for the more advanced stuff.

The new version of vCloud Connector, 1.5, has had a big overhaul. Transfers now use parallel TCP/IP connections for faster transfers, checkpoints (or resume points) for added reliability and can be point-to-point to avoid staging issues. Also, the architecture has been changed to an agent based (and/or virtual appliance) type situation.

The GUI hasn’t seen much change, although there are multiple points of access now: vSphere Client, Web Access and integration with the vCloud Portal. Because of this integration, you can now transfer your VM’s from vCloud to vCloud from within the portal. Migration, however, is still ‘cold’: your workload will suffer some downtime. Migration between different providers will probably be charged, although there’s no standardization of cost because there’s no standard way the vCloud interconnect.

After a couple of weeks of well-spent time sailing and tinkering about the house (otherwise known as a holiday), I’m preparing for VMworld and finishing up some posts. I promised you the final part in the three-part series on the Dell Management Plug-in for VMware vCenter.
In this post I’ll show you how to get a new server into DMPVV’s line of sight and use the Deployment Wizard to configure the host.

Deployment Wizard

The Deployment Wizard gathers the necessary information (server list, Deployment Template, host name/networking information, and Connection Profiles) to build a scheduled job that provisions server hardware and deploys new hosts into a vCenter. Unfortunately, deployment of ESXi to an SD-card or USB-media is not yet supported. I do believe Dell is considering this feature for an upcoming release. Before initiating a deployment task, ensure the system has completed CSIOR and is not in the process of rebooting.

The wizard itself isn’t difficult to work with; getting a new host to play nice with the DMPVV, however, is a bit trickier:

Wonder how you make new physical servers available to the appliance?

There are a couple of requirements that you need to fulfill before a host will show up in the DMPVV interface.

Auto Discovery

The auto-discovery feature of a host system is not enabled by default; you can select it to be enabled at the time of purchase. Enabling it manually is done with these steps:

• During the POST boot, press CTRL-E to enter the iDRAC setup.
• Set the Auto Discovery flag to ‘enabled’ (it’s under ‘User Configuration’).
• Disable the ‘root’ and other iDRAC administrator accounts (if the deployment credentials in the appliance match those of an existing user, then the password of the existing user will be changed to match the appliance credentials).
• Enter the DMPVV’s IP-address in the ‘Provisioning Server’ field.

For more information on how to set up a network to support auto-discovery, see the Dell Auto-Discovery Network Setup Specification.

Collect System Inventory on Restart (CSIOR)

To enable CSIOR locally on a reference server:

• Power on the system, and during POST press <F10> to launch USC.
• Select Hardware Configuration – Part Replacement Configuration.
• Enable the Collect System Inventory on Restart setting, and exit USC.
• Cold reboot the system.

The EqualLogic Multipathing Extension Module for VMware vSphere is a Path Selection Plugin (PSP) in the VMware Pluggable Storage Architecture (PSA). It also includes a connection manager (EHCM) to manage iSCSI sessions to the EqualLogic array.

The EqualLogic Multipathing Extension Module for VMware® vSphere enhances the native multipathing capabilities of VMware vSphere 4.1 when used with an EqualLogic PS Series SAN.

• Automatic connection management
• Automatic load balancing across multiple active paths
• Increased bandwidth
• Reduced network latency

So, it provides a more knowledgable way (because it communicates with the array and knows about the tiered volumes) to do load balancing, which improves I/O performance. This means that (by using the EHCM in a CIM provider), the PSP routes I/O to the most optimal path.
 

Requirements

You should be running PS series firmware 4.0.7 or later (I recommend version 5.1.0) and have VMware vSphere 4.1. Also, make sure the network between the host and storage array is non-routed; ESXi MPIO cannot handle a routed  network. You should be able to put the host in maintenance mode and reboot it after installation for the MEM to kick in.

Software or Dependent Hardware iSCSI?

Software iSCSI is fully supported.

The EQL MEM supports the Broadcom NetXtreme II with iSCSI offload, otherwise known as dependent hardware iSCSI. There’s (still) no Jumbo Frames though, and the maximum number of sessions created from a single adapter is 64.

Because some HBAs do not support iSCSI session management, there’s no performance benefit associated with the MEM in that situation.

Configuring the networking stack

When you’re using software iSCSI, you’ll need to configure networking first.

• Create a separate vSwitch for iSCSI traffic with Jumbo Frames enabled
• Decide on the number of VMkernel ports. Assume a (n+1):n relationship between VMkernel ports and vmnics. If you want to dedicate two physical adapters to iSCSI, use three VMkernel ports. For four physical adapters, use five. I’ll explain why in a bit.
• Create VMkernel ports (with Jumbo Frames enabled)
• Configure the 1:1 relationship between vmk# and vmnic# (remove the extra uplinks) for the second and third (and any additional VMkernel ports). Don’t do this for the first VMkernel port.
• Enable the software iSCSI Initiator
• Bind the VMkernel ports to the iSCSI adapter (with esxcli swiscsi nic add –nic vmk# –adapter vmhba##) for the second and third (and any additional VMkernel ports). Don’t do this for the first VMkernel port.

Why do we want  ’n+1′ for for the VMkernel port? Why not just settle on a 1:1 relationship? The problem is, if the physical switch to which the first VMkernel port in the iSCSI subnet is connected fails, iSCSI traffic can be affected, even though you’re using multiple physical switches and have the iSCSI stack set up accordingly. This occurs because the first VMkernel port in the iSCSI subnet is used for the default route. The EqualLogic MEM uses ICMP ping to test for connectivity between host and SAN member. If the switch is down, ICMP pings cannot be routed back through the first VMkernel port and the MEM would not be able to rebuild the iSCSI session.

In the ‘Known Issues and Limitations’ section of the User Guide, Dell actually lists this problem:

Failure On One Physical Network Port Can Prevent iSCSI Session Rebalancing
In some cases, a network failure on a single physical NIC can affect kernel traffic on other NICs. This occurs if the physical NIC with the network failure is the only uplink for the VMKernel port that is used as the default route for the subnet. This affects several types of kernel network traffic, including ICMP pings which the EqualLogic MEM uses to test for connectivity on the SAN. The result is that the iSCSI session management functionality in the plugin will fail to rebuild the iSCSI sessions to respond to failures of SAN changes.

The solution is to configure the first VMkernel port to be connected to all physical uplinks (and thus all physical switches) for the iSCSI vSwitch. Because you bind multiple physical uplinks to the VMkernel port, you cannot bind it to the software iSCSI Initiator and it won’t do any iSCSI I/O. It’ll be a VMkernel port for ICMP ping only. iSCSI traffic will flow through the second and third (or even fourth and fifth if you’re using four physical uplinks) VMkernel ports. I think it was Duco Jaspars who came up with the solution for this issue a couple of months ago.

Installation

Installation is quite easy as EqualLogic has been so kind to include a ZIP-file which can be used with either Update Manager, the vMA or the vSphere CLI. I highly recommend using Update Manager to install the PSP, as this is by far the easiest way.

EqualLogic has also given us setup.pl, an installation and configuration script. There’s three parts to this script, ‘installation’, ‘configuration’ and ‘set parameters’. I’ll ignore the ‘installation’ part as we’re using Update Manager. The second part would save some time configuring the vSwitch, VMkernel ports, uplinks and software iSCSI Initiator, but as we’re using a slightly different setup (the first VMkernel port), this won’t help us much either.

Editing ehcmd.conf

That leaves us with the third part, ‘set parameters’. By default, the EHCM creates two sessions to a volume slice (portion of a volume on a single member). In configurations with four vmnics (four VMkernel ports and a fifth for ICMP), you’ll want to bump this number up to four sessions to take full advantage of the four physical links. Use setup.pl –setparam for this:

setup.pl --setparam --name="membersessions" --value="4" --server="esxi"

While you’re at it, increase the total number of sessions to a volume to 12:

setup.pl --setparam --name="volumesessions" --value="12" --server="esxi"

Instead of setup.pl, you can manually edit /etc/cim/dell/ehcmd.conf and make the same adjustments.

Resources

• EqualLogic MEM
• User Guide
• Fix List

In part two of this three-part series, I’m going to dive deeper into the main components of the DMPVV, which are the various profiles and templates. Also, I’m going to show you an example on how to manage an individual server by updating the firmware.

Profiles and Templates

Profiles and templates are the beating heart of the DMPVV tool. They contain the various settings for deploying and provisioning a physical server.

Connection Profile

A Connection Profile (CP) is a profile containing credentials for the various methods of connecting to a host and is associated to a root, datacenter, cluster or host object in the vCenter inventory. The iDRAC credentials are used to connect to the iDRAC controller in a physical server. By default, this controller uses a ‘root’ account. There’s an option to check the certificate, but if you’re using the default self-signed certificate, leave it disabled. The second credential in the CP is for the OpenManage Server Administrator Agent (OMSAAgent for short). These credentials are in fact the ESX(i)-root credentials.

Hardware Profile

A Hardware Profile (HP) contains settings from the physical hardware. This profile contains all BIOS, iDRAC and RAID-settings. The settings are gathered from a reference server and can be customized to your liking. When this profile is applied to a physical server, the hardware is configured automatically using the settings contained in this profile.

Hypervisor Profile

A Hypervisor Profile is composed of two items: installation media and vCenter destination cluster. These settings are used to automatically install the hypervisor during deployment.

Deployment Template

A Deployment is any combination of a Hardware and Hypervisor profile.

Dell Server Management

The Dell Server Management components are used for individual server-related management.

Accessing ‘Dell Server Management’

There are two ways to access the individual server-related management components. The first one is rather limited in its use, but what can you expect from a context menu item. The second one is conveniently placed in the tabs bar at the top of a server object in the vCenter inventory.

There are actually a few interesting items in this list:

Firmware Update Wizard

Updating firmware and BIOS-versions on Dell PowerEdge servers has never been easier. It takes an incredibly good tool to make me say that, as I am usually forced do some head-desk exercises while trying to update firmware/BIOS-version on Dell PE servers. I couldn’t agree more with @DuncanYB‘s post, ‘Dell Firmware updates‘. Even if the tool did nothing else than updating BIOS and firmware like this, I would highly recommend it to any admin out there with vSphere and Dell hosts.

Concluding this post

In this post, I’ve screenshotted my way through the different profiles and templates and used the Dell Server Manager portion of the DMPVV to upgrade firmware. In the third and last episode, I’ll show you how to ‘import’ a crisp new host into the DMPVV so we can actually deploy Hardware and Hypervisor profiles onto the host using the Deployment Wizard.

A while back, I investigated how to optimize backups in an EqualLogic storage environment. I wanted to make use of all the (EqualLogic) features available to me.

I created a list of ‘must haves’ for this design:

• Zero-cost, zero-impact on production environment while creating backups
• Maintain data consistency and integrity
• Be able to replicate backups to a secondary location (prevent using tapes as a off-site backup and archival tool)
• Separation of back-up creation and processing: production environment should have a very short back-up time frame, all the processing and logic happens ‘behind the curtains’
• Keep the backup server ‘light’ by using the SAN as a backup-to-disk target.

Utilizing SAN snapshots

I wanted to see if I could use the snapshot technologies available to me to fulfill the zero-cost, zero-impact item on the list. A snapshot is made without impact or cost, is fast to complete and is a static copy of the production data for my backup server to work with. On the downside, a snapshot isn’t application-consistent out of the box.

We need some kind of interaction between the array and the application to make the snapshot (and the data contained within) completely consistent. Luckily, the EqualLogic array has some cool Microsoft VSS integration. The device acts as a hardware VSS Provider. If you’re using applications that are VSS-aware (NTFS, Exchange, SQL all include a VSS Writer), you can leverage this integration. Just install the EqualLogic Host Integration Toolkit (Microsoft Edition) or HIT/ME for short, which includes the Auto-Snapshot Manager on the application server. The ASM/ME acts as a VSS Requestor and creates these snapshots (while maintaining data consistency using the various VSS components). Information about the snapshot is stored in XML and transferred from the array to the backup server using the ‘VSS-Control’ volume. The backup server picks up on this and mounts the volume based on the information inside the XML. This snapshot is then used as a source for backups rather than the original volume.

There is, how could there not be, a caveat. The major version of Windows should be consistent across backup- and application servers. Nowadays, this means Windows Server 2008 or 2008 R2. Please don’t mix 2003 and 2008 versions. Don’t blame EqualLogic for this, it’s a limitation due to Microsoft changing the XML format in Windows Server 2008, lacking backward compatibility.

Off-host backups

Next, I wanted to alleviate the application server. Although it now has a fast way to create a static copy of the application data (contained in the snapshot), the server still is actively involved in mounting the snapshot and moving data from the snapshot to the backup media or backup server. I want the back-up server to coördinate and execute all backup related tasks (including mounting the snapshot and copying the data). The only thing left to do for the application server is to prepare for the snapshot. The backup server then mounts the snapshot locally and copies the data to the backup target. When you’re using 3rd party backup software, this software will act as the VSS Requestor. The HIT/ME will do the VSS Provider-related tasks (in conjunction with the array hardware).

This requires off-host capabilities on the backup server, meaning that the backup software supports the VSS hardware provider included in the EqualLogic array and enables VSS transportable snapshots. Symantec Backup Exec, CommVault Galaxy, CA BrightStor ARCserver and Legato NetWorker all support this. For the off-host functionality to actually do its work, you’ll need to make sure that the SAN-snapshot is visible on the backup server. This usually means the backup server is connected to the iSCSI network. Also make sure both the application and backup servers can access the VSS control volume on the array. This is done while installing the HIT/ME. Create persistent, snapshot-only connections to the volumes to be backed up for the backup server (use a ACL entry for the volume inside the EqualLogic management tool for this). For the record: giving the backup server access to the original volume as well as the snapshot could lead to data corruption. Be warned.

An example

For example, Symantec Backup Exec has an option called ‘Advanced Disk-based Backup Option’ or ADBO to enable off-host backups. You won’t need the ‘Advanced Open File Option’ if using the ADBO. If you want to backup Exchange or SQL using VSS and the ADBO, you still need the agents for Exchange and SQL, but they need to be installed on the backup server (along with the Remote Agent for Windows and the EqualLogic HIT/ME). They’re not needed on the application servers themselves. You should however install the Backup Exec Remote Agent for Windows and the EqualLogic HIT/ME on the application servers.

To create transportable snapshots, specify that you want to use the ADBO  by selecting ‘Use offhost backup to move backup processing from the remote computer to media server’ and select the ‘Hardware – Use technology provided by hardware manufacturer’ snapshot provider. Be sure that you deselect ‘Use Advanced Open File Option’.

Full Granularity

Cool thing is, you can backup an entire volume, Exchange Information Store (or Storage Group) at the database level or SQL at the instance or database level, while still being able to do granular restores. Just enable ‘Use Backup Exec Granular Recovery Technology’ for each backup job.

Off-site backups

By leveraging the data replication features of the EqualLogic array, we can replicate the backup media target volume on the array to a secondary array in a different location. We can extend this functionality by adding a backup media server on the secondary location to do backup-to-tape operations. These two methods of adding resiliency to your backup strategy are configured using standard recommendations from both EqualLogic (for replication) and Symantec (for backup-to-tape). For simplicity sake, I won’t explain these in this post.

VMware vSphere

Things get a bit complicated if you’re doing all this with virtualized application servers. One of the requirements for the HIT/ME to function is that the volumes are mounted through the Microsoft iSCSI Software Initiator. This means that data volumes you’re backing up aren’t VMDK’s or RDM’s, but iSCSI volumes straight from the EqualLogic array. This might and probably will complicate your storage and virtualization platform designs, so you’ll have to make the trade-off between ‘simple’ and ‘awesome features’.

If you decided that off-host backups are worth the extra complexity, you might think: “Hey, I’m now using the ASM/ME, I could use the ASM/VE, too!“. ASM/VE is the Auto-Snapshot Manager for VMware vSphere, obviously. It creates virtual machine snapshots for all virtual machines on a given iSCSI volume before snapshotting the volume itself. Unfortunately, there’s not much you can do with those snapshots-in-snapshots from a backup perspective. If you’re after off-host backups for individual virtual machines, just use the VMware vStorage APIs for Data Protection plugin for your backup software, like the Backup Exec Agent for VMware Virtual Infrastructures and use ‘SAN transport mode’. Through the vADP and the VMware Tools, the virtual machine itself is somewhat VSS-aware (and thus consistent), too. Again, install the Remote Windows Agents for Granular Recovery stuff.

If you are using the ASM/VE, making a Smart Copy presents you with an option to “include PS series volumes accessed by guest iSCSI initiator.” I’ll quote Pete Koehler for his opinion on this matter: “I do not use this option for a few very good reasons, and rely completely on ASM/ME for properly capturing guest attached volumes”. Thanks Pete, couldn’t agree more.

Multipath I/O for performance

If you’re desperate for more performance, add MPIO. Your application servers (connected to the EqualLogic through the Microsoft iSCSI Software Initiator) can use MPIO. The backup server is no different. Use the specific DSM for MPIO from EqualLogic (installed alongside with the HIT/ME). Even if you use the VMware vADP to backup virtual machines, you can use MPIO. Beat that, HP LeftHand P4000.

In this three-part series I’m going to dive into the Dell Management Plug-In for VMware vCenter. In the last couple of months, I have used the Dell Management Plug-In for VMware vCenter (DMPVV) in a PoC/pilot environment. I wanted to share my experiences with you.

Today, we’re starting with an introduction to the tool and how to install/configure the appliance. Along the way, I will show how I use this tool to manage my physical environment.

Introduction

First off, what is the DMPVV?

The Dell Management Plug-in for VMware vCenter is designed to streamline the management processes in your datacenter environment by allowing you to use VMware vCenter to manage your entire infrastructure- both physical and virtual. From firmware updates to bare metal deployment, the Dell Management Plug-In for VMware vCenter will expand and enrich your datacenter management experience with Dell PowerEdge servers.

So, this tool provides a way to monitor, manage and provision physical servers from within vCenter, integrating it not only by using the same interface (the vSphere Client), but also extracting information from vCenter and correlating it with information from the iDRAC and Lifecycle controllers. Because of this integration, you can update a new host with current BIOS and other firmware versions, configure hardware components (like the BIOS and iDRAC) and install a hypervisor. Last but not least, you can clone configuration from one server to another (using, among other technologies, VMware’s Host Profiles). Once installed and configured, you can get an overview from each server, displaying stuff like asset and warranty information, health status and much more.

One of the coolest features I want to highlight is the PXE-less provisioning of the hypervisor to a physical server. This uses a combination of the Lifecycle Controller and iDRAC to deploy an installation ISO to the server. And since it is really tightly integrated with the VMware stack, the host is added to vCenter and configured using Host Profiles automatically, resulting in a true zero-touch deployment of a server. How cool!

Recently, Dell released version 1.0.1.72, which includes a trial version. See what’s new here, or discuss about the DMPVV here. Real release notes are found in the documentation section. Any questions might be answered in the FAQ.

Let’s take a look at the architecture view (taken from the Quick Install Guide):

Basically, the DMPVV is a virtual appliance registered as an extension against vCenter and a Plug-in for the vSphere Client. The appliance in fact does all the work, communicating with the physical servers, providing provisioning services, acting as a staging area for firmware/BIOS downloads from the Dell Online Repository, etc.

In a little more detail (taken from here):

Pricing

Licensing is per-server. Just figure out what number of servers you want to manage and choose a licensing package. One warning: 9G and 10G servers don’t consume a license; they’re missing the Lifecycle Controller and can’t take full advantage of the DMPVV’s features.

• 3 servers, $299 (about $100 per server); use this if you have a vSphere Essentials (vCenter Foundation) license.
• 10 servers, $799 (about $80 per server)
• 50 servers, $1799 (about $36 per server)
• 1000 servers, $2999 (about $3 per server)

To my knowledge, Dell doesn’t let you use multiple licenses at the same time. Make sure the license package is for a greater number of servers than you want to manage. For instance, if I have 20 servers, I can’t use two 10-pack licenses (which would be $201 cheaper than the 50-pack). I need to choose the 50-pack.

Prerequisites

• Before you dive into the appliance, make sure you have the latest version of the OpenManage Agent (6.5 at the time of writing) installed on your hosts. If you don’t, install it using the VIB and Update Manager. Configure the OMSA Agent to send SNMP Traps. Here’s how to do that. If you’re stuck (the DMPVV telling you that OMSA isn’t installed) set the ESXi advanced option CIMoemProviderEnabled and reboot.
• Make sure you have a DHCP-server inside the network you’re provisioning servers in.
• Administrative accounts on the vCenter Server, ESXi-hosts and iDRAC interfaces.
• Adobe Flash 10.0 on the client system.
• Internet Explorer 7 or 8 (without any proxy configured).
• Have a OpenManage OMSA Web Server component installed somewhere. I installed it on my physical back-up server.

Deploy the Virtual Appliance

Download the virtual appliance (trial version, valid for one vCenter and one host) and extract the bits somewhere safe. Now, open the vSphere Client and go for ‘Deploy OVF Template’. The wizard asks a couple of questions, but I’m not going over those bit by bit. Just make sure to answer these questions accordingly to fit the appliance into your virtual infrastructure. I chose a thin provisioned disk and attached the appliance to my Management VLAN. Power it on after deployment and switch to the VM’s console. Log in using ‘admin’ and set a password. While you’re in there, set the IP- and DNS-settings and configure the Time Zone. Make sure to tick ‘System Clock uses UTC’.

Admin Portal

Open your browser and go to https:///DellAdminPortal/index.html. Log in using the password you’ve just set.

First up is registering the appliance with a vCenter Server. For simplicity sake, please make sure to register using the vCenter hostname. While you’re at it, upload a license file (.bin) if you have one and check for updates (‘Update Virtual Appliance’ in the Appliance Management category). Time synchronization is important as ever in this appliance, so make sure to configure NTP settings.

The last order of business in the Admin Portal is to configure a back-up schedule. You’ll need to have a Windows Share and corresponding credentials and set a schedule. In version 1.0.1.72, I found a bug where the “Time for Backup” field was set back four hours: I set the time to 20:00, but after a refresh of the page it showed 16:00. This seems to be a cosmetic bug only, as the backup was indeed done at 20:00 sharp.

Dell Management Center Configuration Wizard

Inside the vSphere client, you’ll now find a ‘Dell Management Center’ button. You’ll be welcomed by a Configuration Wizard once you venture into the unknown.

As this wizard is helping you along there are some things to be aware of:

• You’ll need both the iDRAC and the ESXi root credentials to successfully create a connection profile.
• Be sure to tick the ‘Enable Alarms for Dell Hosts’ box, this will tightly integrate hardware monitoring into vCenter, enabling stuff like putting a host in maintenance mode if a critical alarm is received.
• If you are running a proxy inside you network, I strongly recommend disabling it for the DMPVV or making it completely transparent (i.e. without settings on the server/client).
• You’ll need a Staging Folder in order for the Firmware Repository to work (as long as you’re using the default firmware update repository). I’ve created a simple file share (\\vc01\dmpvv) for this purpose. I store my DMPVV backups here as well. Please make sure you use IP-addresses to point to the share (\\10.10.10.10 instead of \\vc01).
• When asked for the OpenManage Server Administrator Web Server URL, I used “https://bck01:1311/servlet/Login?omacmd=getlogin&page=Login&managedws=true#”. This is the link for ‘Manage Remote Node’ on a Web Server-enabled installation of OMSA on a physical Windows server. You could install the OMSA Web Server on your virtual vCenter Server, too, if you don’t have a physical server with the OMSA Web Server.

Miscellaneous Configuration Remarks

• Don’t worry if the OMSA Web Server URL looks truncated. It’s a known bug; your URL is safe and sound.
• I enabled the ProActive Systems Management feature. I’m not quite sure what it does; but enabling it only shows a login page in the Hosts view.
• I did enable the Warranty Expiration Notification. Especially for larger environments, receiving a notification well before warranty expires can really ease the process of renewal.
• Make sure that both the inventory and warranty data retrieval tasks have been executed manually once. This ensures consistent operation of the DMPVV after configuration. You can fire off both tasks from the Job Queue window.

Concluding this post

In this post, I’ve explained what the tool is for and how it is installed and configured. In an upcoming post, I’ll show you how to create the various profiles and templates, how to check up on an individual server and how to upgrade the firmware on a server. Stay tuned!

I just received an e-mail from VMware informing me I made vExpert 2011! This is a great surprise, as I did not expect to make the cut this year. Having missed the boat in 2010, I feel esspecially honoured this year. I would like to thank my fellow bloggers, VMware employees and the VMware community at large for giving me the opportunity to share and contribute.
Of course, special thanks go out to John Troyer and his social media team for recognizing my effort this year.

Back in the day, I used StarWind iSCSI Target a lot in as an instructor in training sessions. I let students configure a simple iSCSI Target to get some storage up-and-running for them to attach to the vSphere cluster they were building, so they could do all the funky VMware-stuff, like vMotion, HA and DRS. StarWind was an ideal solution, because, well, it was free and Windows-based. The effort it took to get the product in working order was minimal enough not to distract the students from focussing on what’s important.
 

So I was bummed out to see the free version of StarWind disappear. With an equally strong positive reaction, I noticed that the free version has somehow made it back into the wild. As of May 16th, StarWind Free iSCSI SAN has been made available again. It still installs on top of Windows, provides full snapshot and backup capabilities, it even does some dedup and caching magic. Best of all, no license is restricting you from using all this in a production environment.

Download it right here.