Rightfully so, some readers and other bloggers pointed out that PVSCSI isn’t supported for boot disks (Rich Brambley put up a really good post, for example). Rich, among others, suggested moving virtual machines back to a “two disk model,” with a boot disk and a separate data disk; this would allow for the greater performance of the PVSCSI controller on the data disk. This seemed to be a reasonable workaround. I don’t recall hearing about any significant issues with VMXNET3. Using the newer network driver seemed to be a good move all the way around.

Unfortunately, there is another drawback to both of these devices. Rich caught this drawback in his article, but relegated it to a small mention at the very end of the article that even I overlooked at first (emphasis mine):

There are some other factors to consider as well. For example, vSphere Fault Tolerance cannot be enabled on a VM using PVSCSI.

That’s right—you cannot use VMware Fault Tolerance (FT) on a virtual machine that is using the PVSCSI device. However, this restriction doesn’t just apply to the PVSCSI device; it also applies to VMXNET3! VMware FT cannot be enabled on a virtual machine using either the VMXNET3 or PVSCSI devices; vCenter Server will simply report an error that the network interface or disk controller isn’t supported for VMware FT.

In my opinion, this is a significant enough limitation that I felt it warrants its own post. If you are planning on using VMware FT in your environment, be sure not to configure any virtual machines to use VMXNET3 or PVSCSI if they might need to be protected with VMware FT. In this case, you’ll have to choose from either maximum performance or maximum protection—you don’t get both.

UPDATE: Rich Brambley shared links to two resources that describe the incompatibility between VMware FT and PVSCSI and VMXNET3:

VMware Communities: Unable to configure FT with error “Unsupported virtual machine configuration for Fault Tolerance. Device ‘Network adapter 1′ is not supported”
VMware Fault Tolerance Requirements and Limitations

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Another Reason Not to Use PVSCSI or VMXNET3

• vSphere Virtual Machine Upgrade Process
• Enabling Enhanced VMXNet
• Get Used to vSphere
• The Return of Virtualization Short Takes
• Hyper9 VMM Released

I’ve discussed this topic before, but I felt like it was a topic that needed to be revisited again. Storage admins need to know how their choices in storage technologies may or may not impact virtualization efforts, and this particular choice—leveraging pointer-based snapshots or deduplication—is particularly important.

FlexClones Versus Deduplication with VMware Infrastructure

A number of times over the last few months, I’ve run into situations where NetApp’s FlexClone technology was being heavily pitched to customers interested in deploying, or expanding their deployment of, VMware Infrastructure.

In case you aren’t familiar with the use of NetApp FlexClones in conjunction with VMware Infrastructure, have a look at these earlier articles of mine:

How to Provision VMs Using NetApp FlexClones
NetApp FlexClones with VMware, Part 1
NetApp FlexClones with VMware, Part 2
LUN Clones vs. FlexClones

Now, after you’ve read all those articles (you did read them, didn’t you?), it should be fairly clear that using FlexClones can be very advantageous. However, those advantages come with some tradeoffs as well, most notably in the complete and total lack of integration with VMware Infrastructure itself.

This lack of integration means that users can’t use VirtualCenter templates, because the cloning is taking place at the storage array instead of within VMware Infrastructure. This also means that customers can’t apply customization specifications during the cloning process, so users will need to create their own Sysprep answer files and manually Sysprep the VMs before invoking the FlexClone process. Users are required to create scripts and tools to do simple things like using the VM name for the guest OS name during cloning. (Author’s note: many of these issues have been addressed by NetApp’s Rapid Cloning Utility (RCU), which provides some integration into VirtualCenter.)

Deduplication, on the other hand, works seamlessly with VMware Infrastructure. This is primarily because the details of the deduplication are completely hidden; it all occurs “inside the box.” Nothing needs to be configured within VirtualCenter; no VMs need to be modified. The NetApp storage system handles the details of the deduplication process itself, and VMware Infrastructure just consumes the storage.

Looking at these two technologies in that light, one might ask: why use FlexClones at all? If deduplication works seamlessly with VMware Infrastructure and FlexClones don’t, then why bother? To be honest, there are some instances where FlexClones make sense—even with the lack of integration. Consider some of the examples listed below.

• In instances where a user needs to deploy lots of VMs in a very rapid fashion, FlexClones are much better. If time-to-deployment is the #1 driving factor, then FlexClones are the way to go. This could be particularly applicable and useful in VDI situations, as long as the broker doesn’t mandate handling provisioning itself (like VDM does).
• In environments where provisioning and re-provisioning occurs on a frequent, regular basis, then FlexClones make sense. Even though large numbers of VMs aren’t being provisioned, the time saved on frequent re-provisioning via FlexClones will not be insignificant.
• In situtations where there isn’t sufficient storage for the VMs before they are deduplicated, FlexClones may be a better option. Deduplication is post-process, meaning that storage will be needed for the full datasets until deduplication runs. In situations where that isn’t an option, then FlexClones can provide the same end benefit.

Personally, I’m of the opinion that unless an organization meets one of these criteria, then that organization should look to deduplication instead of FlexClones. Of course, that’s just my personal opinion, and I’m open to hear what others have to say about the matter. NetApp gurus, feel free to weigh in.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Republished: FlexClones or Deduplication?

• FlexClones Versus Deduplication with VMware Infrastructure
• NetApp FlexClones with VMware, Part 1
• 2031: Enhancements to NetApp Cloning Technology
• Storage Short Take #3
• Using NetApp Deduplication with Block Storage

1. First, all comments should be courteous. There’s no reason to personally attack another reader or author—simply state your position, why that is your position, the facts you feel support your position, etc. Leave the personal attacks somewhere else.
2. Second, all commenters should provide full disclosure. This helps avoid even the appearance of wrongdoing. Where a vendor’s products helps to address readers’ needs, I don’t mind a vendor mentioning their products. That vendor just needs to be sure to provide full disclosure. If you have a business relationship with an organization, disclose that. Be transparent and provide full disclosure.

Recently, I’ve had one commenter leave a series of comments on the site that blatantly and bluntly promote his employer’s products. Unfortunately, this commenter has failed to provide full disclosure. For that reason, I’ve been simply deleting this commenter’s comments. And I’m going to continue to delete this commenter’s blatant, outright comment spam as long as he/she refuses to provide full disclosure. Other readers deserve the right to know why a commenter is pushing a particular product or feature!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

A Comment Policy Reminder

• Train Signal, Welcome!
• Question to my Readers
• Storage Short Take #1
• Blogging Frequency May be Down
• The Site’s First Sponsor

In this article, I’m going to tackle what will probably be a sensitive topic for some readers: VMware over NFS. All across the Internet, I run into article after article after article that sings the praises of NFS for VMware. Consider some of the following examples:

• VMware over NFS - Why?
• Why VMware over NetApp NFS
• VMware & NFS

That first link looks to be mostly a reprint of this blog post by Nick Triantos. Now, Nick is a solid storage engineer; there is no question in my mind that he knows Fibre Channel, iSCSI, and NFS inside out. Nick is certainly someone who is more than qualified to speak to the validity of using NFS for VMware storage. But…

I am going to have to disagree with some of the statements that are being propagated about NFS for VMware storage. Is NFS for VMware environments a valid choice? Yes, absolutely. However, there are some myths about NFS for VMware storage that need to be addressed.

1. Myth #1: All VMDKs are thin provisioned by default with NFS, and that saves significant amounts of storage space. That’s true—to a certain point. What I pointed out back in March of 2008, though, was that these VMDKs are only thin provisioned at the beginning. What does that mean? Perform a Storage VMotion operation to move those VMDKs from one NFS datastore to a different NFS datastore, and the VMDK will inflate to become a thick provisioned file. Clone another VM from the VM with the thin provisioned disks, and you’ll find that the cloned VM has thick VMDKs. That’s right—the only way to get those thin provisioned VMDKs is to create all your VMs from scratch. Is that what you really want to do? (Note: VMware vSphere now supports thin provisioned VMDKs on all storage platforms, and corrects the issues with thin provisioned VMDKs inflating due to a Storage VMotion or cloning operation, so this point is somewhat dated.)
2. Myth #2: NFS uses Ethernet as the transport, so I can just add more network connections to scale the bandwidth. Well, not exactly. Yes, it is possible to add Ethernet links and get more bandwidth. However, you’ll have to deal with a whole list of issues: link aggregation/802.3ad, physical switch redundancy (which is further complicated when you want to use link aggregation/802.3ad), multiple IP addresses on the NFS server(s), multiple VMkernel ports on the VMware ESX servers, and multiple IP subnets. Let’s just say that scaling NFS bandwidth with VMware ESX isn’t as straightforward as it may seem. This article I wrote back in July of 2008 may help shed some light on the particulars that are involved when it comes to ESX and NIC utilization.
3. Myth #3: Performance over NFS is better than Fibre Channel or iSCSI. Based on this technical report by NetApp—no doubt one of the biggest proponents of NFS for VMware storage—NFS performance trails Fibre Channel, although by less than 10%. So, performance is comparable in almost all cases, and the difference is small enough not to be noticeable. The numbers do not, however, indicate that NFS is better than Fibre Channel. You can read my views on this storage protocol comparison at my site. By the way, also check the comments; you’ll see that the results in the technical report were independently verified by VMware as well. Based on this information, someone could certainly say that NFS performance is perfectly reasonable, but one could not say that NFS performance is better than Fibre Channel.

Now, one might look at this article and say, “Scott, you hate NFS!” No, actually, I like using NFS for VMware Infrastructure implementations, and here’s why:

• Provisioning is a breeze. It’s dead simple to add NFS datastores.
• You can easily (depending upon the storage platform) increase or decrease the size of NFS datastores. Try decreasing the size of a VMFS datastore and see what happens!
• You don’t need to deal with the complexity of a Fibre Channel fabric, switches, WWNs, zones, ISLs, and all that. Now, there is some complexity involved (see Myth #2 above), but it’s generally easier than Fibre Channel. Unless you’re a Fibre Channel expert, of course…

So there are some tangible benefits to using NFS for VMware Infrastructure. But let’s be real about this, and not try to gloss over technical details. While NFS has some real advantages, it also has some real disadvantages, and organizations choosing a storage protocol need to understand both sides of the coin.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Republished: Dispelling Some VMware over NFS Myths

• Only Thin Provisioned in the Beginning
• Proving VMware Over NFS
• Articles in Progress
• Storage Protocol Performance Whitepaper from NetApp
• Keeping Thin VMDKs Using NetApp SnapRestore

I haven’t hid the fact that I’m not really a fan of FCoE (see here or here), but I was starting to warm to the technology and thought that I was beginning to see some benefits to deploying FCoE. Namely, the fact that FCoE is inherently very compatible with “traditional” FCP, allowing organizations to leverage their existing FCP installation while transitioning to FCoE. Some hands-on time I’d recently spent with a Cisco Nexus 5000 switch showed me just how closely aligned the two technologies are and how (relatively) easy it was to extend an FC fabric using FCoE. OK, I think I get this.

Then, a few days ago, I read this article on FCoE divergence. Given that The Register can sometimes be quite sensationalist (and that’s putting it mildly), I contacted a colleague of mine whose input and knowledge I trust. He informed me that FCoE was currently limited in that FCoE is not multi-hop enabled—meaning, you can’t connect FCoE initiators on one switch to FCoE targets on another switch. (Apparently, this shortcoming is due to be corrected shortly.)

Whoa! That’s a limitation of which I was not aware. And with that limitation in mind, knowing that FCoE will—for the time being at least—be limited to convergence at the edge, I have to ask: why deploy FCoE at all? What real and specific benefits does an organization seek to gain by deploying FCoE as opposed to just deploying FC? Is the edge convergence really that worthwhile and valuable?

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Thinking Out Loud: Why Deploy FCoE?

• Is Unified Fabric an Inevitability?
• A Collection of Viewpoints on Cisco UCS
• FCoE versus MR-IOV…huh?
• Continuing the FCoE Discussion
• Blades Won’t Die, But They Will Change

I do agree with Steve that there is no innate relationship that means running VMware on blades is somehow “automagically” better:

It is amazing how frequently we hear IT managers talk about deploying blade servers as an integral component of their new virtual infrastructures - as if there were an obvious synergy between VMware and blade server architectures.

Absolutely! Blades are an option, just like rack mounted servers, and it’s up to the customer to choose (or us as consultants to recommend) the form factor that best meets the business needs. It might be blade servers, or it might be rack mounted servers. It just depends. So, on this one point, I agree with Steve.

Yet, at the same time, I also disagree with this point that Steve makes in his article:

Blade servers have always been an impediment to an optimal virtual infrastructure because they introduce limitations in efficiently utilizing power and cooling resources, budget, flexibility, manageability, bios and firmware updates, performance and troubleshooting.

Here is where Steve and I start to disagree. In fact, this specific article was something of the catalyst for a series of posts, written by colleague and friend Aaron Delp, detailing how blade servers and virtualization work well together:

Blades and Virtualization Aren’t Mutually Exclusive: Part One, HP Power Sizing
Blades and Virtualization Aren’t Mutually Exclusive: Part Two, IBM Power Sizing
Blades and Virtualization Aren’t Mutually Exclusive: Part Three, IBM Traditional Expansion Options
Blades and Virtualization Aren’t Mutually Exclusive: Part Four, HP Traditional Expansion Options

While this series of articles doesn’t squarely address all of the arguments against blades and virtualization, the series does make it clear that blades can produce power savings vs. rack mounted servers, and that blades do offer enough expansion options to accommodate the majority of virtualization deployments.

I also disagree with Steve about the value of the unified fabric, especially considering that right now unified fabric can exist only at the edge of the network and not at the core. That being the case, I find it hard to say that unified fabric is going to kill blade servers. So, again, I have to disagree with Steve’s position.

However, Steve’s not entirely wrong—virtualization, FCoE and 10Gb Ethernet, and yes even unified fabric will change how blade servers are designed and deployed. Cisco’s Unified Computing System (UCS) is one example of how blade servers are going to adapt to these agents of change, and I believe we’ll see more examples from other leading vendors in the coming months and years. But will blades die away entirely? No, I don’t think so.

Think I’m crazy? Think I’m out of my mind? Feel free to speak up in the comments—courteous comments are always welcome.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Blades Won’t Die, But They Will Change

• Alas, BladeVault is No More
• Blades and Virtualization Aren’t Mutually Exclusive: Part Three, IBM Traditional Expansion Options
• Blades and Virtualization Aren’t Mutually Exclusive: Part Two, IBM Power Sizing
• Blades and Virtualization Aren’t Mutually Exclusive: Part One, HP Power Sizing
• Blades and Virtualization Aren’t Mutually Exclusive: Part Four, HP Traditional Expansion Options

I was made aware of a serious (in my opinion) bug with VDR during a call with VMware support that I haven’t seen discussed anywhere. This is an internally known issue that causes snapshots to build up on VM’s that are members of VDR backup jobs.

During the backup process a new snapshot is created and VDR updates the snapshot descriptor file (vm_name-000001.vmdk) to mark the snapshot as un-removable. The bug is introduced when the backup process completes, it fails to mark the snapshot as removable causing them to remain.

The tricky part of the problem is that the snapshots are not visible through the vSphere Client, nor are they listed in apps like ‘RVTools’ that use the VMware CLI to gather data. They could potentially be listed in the new datastore views but I didn’t think to look there before I resolved it in my environment. I ran across them by logging into the service console and running the following command to list all the delta files on the datastores attached to the server.

find /vmfs/volumes/ -name \*delta\*

In my environment I noticed numerous VM’s with multi-gigabyte delta files that I couldn’t account for via snapshots listed in the GUI. Here is the solution I was given by VMware. Via the service console, browse to the location of the VMDK files for the affected VM. Run this command to identify the descriptors that need to be corrected, replacing ‘virtual_machine_name’ with the actual name of the VM.

grep –I ddb.dele *virtual_machine_name*-000???.vmdk

This command will quickly identify the delta files that are marked as non-deletable. The workaround is to edit the affected VMDK descriptor files and change “ddb.deletable” from “false” to “true”. You will probably also need to edit the root VMDK file and change this field as well, otherwise you may be left with one open snapshot. Note that due to a change in how ESX 4 performs file locking, you will probably need to SSH into the host that is currently running the VM to edit these files. Once you have edited all the files, create a new snapshot for the VM either via the GUI or command line. Then issue the “Delete All” snapshots command to force ESX to combine all the files and close all the visible and hidden snapshots.

As soon as more information is available, I’ll post it here. If any other readers have more information to share, please speak up in the comments.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Snapshot Issue with VMware Data Recovery

• Recovering Data Inside VMs Using NetApp Snapshots
• VMworld 2007 Top Support Issues Session
• Restoring VCB Full Backups with VMware Converter
• VM File-Level Recovery with NetApp Snapshots
• Keeping Thin VMDKs Using NetApp SnapRestore

I have no problem with other sites syndicating my content as long as proper attribution of the original author and original site is provided. Do me a favor: visit some of the URLs below (I’m not going to hyperlink them and give the site a traffic boost) and tell me how any of the examples I’ve listed below provide proper attribution of the original author and the original site:

http://networkvirtualization.com/content/unified-fabric-inevitability
http://networkvirtualization.com/content/vmware-io-queues-micro-bursting-and-multipathing
http://networkvirtualization.com/content/tap-vsphere-pvscsi-performance-separate-vm-boot-and-data-drives

Let’s see…content from my site, Chad Sakac’s site, and Rich Brambley’s site, all syndicated on their site without any clear attribution back to the original post—except for a very small link near the bottom of the article. If you hadn’t been looking for that link, or if I hadn’t told you that the articles above were written by me, Chad, and Rich, respectively, would you have known? And those are just the authors I recognized! How many more are there that I don’t recognize?

To whomever is running NetworkVirtualization.com: if you are going to syndicate content, you need to provide proper attribution. Otherwise, taking someone else’s content and allowing people to believe that it’s yours is called plagiarism, and it’s wrong.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

I Love Having My Content Stolen

• Blog Aggregation Taken Too Far
• Stolen Content
• Welcome, Aaron!
• One of the “Top Virtualization Blogs” of 2008
• HTTP Proxies and Command-Line Utilities

• Interested in more information on how it is, exactly, that Cisco is going to provide so much memory in their UCS blades and rack mount servers to make them ideal virtualization hosts? This article from CommsDesign and this “Catalina” article by Rodos Haywood both provide some great information on how Cisco is working around the Intel reference architecture limitations introduced with the Xeon 5500 and Quick Path Interconnect (QPI).
• This article provides a handy reference on how to unregister the Nexus 1000V vCenter Server plug-in. I wish I’d known this information several weeks ago…
• Need to view some configuration files on an ESX host? Just browse to http://<IP address of ESX server>/host and you’re all set. I learned of this handy little trick via Virtual Foundry.
• And speaking of handy little tips, here’s one Eric Sloof shared regarding the vCenter Ops Dashboard. Again, just browse over to http://<IP address of vCenter Server>/vod/index.html to view the vCenter Ops Dashboard.
• Adam Leventhal describes using the latest version of VirtualBox—which now includes OVF support and host-only networking—to run the Sun Storage 7000 Simulator. This is pretty cool stuff. I hope Oracle doesn’t kill it like Virtual Iron…
• I just mentioned Virtual Foundry a bit ago, but forgot to mention this great post on hardening the VMX file. Good stuff.
• For others who are, like myself, pursuing the elusive VMware Certified Design Expert (VCDX) certification, Duncan’s recent post describing the VCDX design defense is a great resource. Thanks, Duncan!
• The VMware networking team addresses some questions around using VMware for virtualized DMZs, and how to protect against Layer 2 attacks.
• Want to do manual linked clones in VMware Fusion? Here’s how.
• Via Matt Hensley, I found this VIOPS document on configuring a VMware vCenter Data Recovery dedupe store.
• This article has more information on installing ESXi 4.0 to a flash drive, a process I have yet to try. (Instructions for burning ESXi 3.5 to a flash drive can be found here.) Anyone else done it yet? I’d be interested in how it went.
• If you have any questions about SAN multipathing, Brent Ozar’s two part series on the topic may help straighten things out (here’s Part 1 and Part 2). I’m not sure that I agree with Brent’s statement regarding the ability of desktop-class SATA drives to saturate 4Gbps Fibre Channel, but I’m no storage expert so I could very well be wrong.
• VMware SE and friend Aaron Sweemer provides a handy script that can help fix Service Console networking when performing automated builds of VMware ESX.

That wraps it up for this edition of Virtualization Short Takes. Feel free to share thoughts, questions, or corrections in the comments, and thanks for reading!

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

Virtualization Short Take #27

• Virtualization Short Take #7
• TA2259 Ask the Experts at VMworld 2009
• Virtualization Short Take #16
• Next-Gen Stuff: Verifying the SHA-1 Fingerprint
• Next-Gen Stuff: Enabling CDP in ESX/ESXi

• VMware vSphere 4.0 (ESX 4.0 and ESXi 4.0)
• VMware ESX 3.5 Update 4
• VMware ESXi 3.5 Update 4
• VMware ESX 3.5 Update 3
• VMware ESXi 3.5 Update 3
• VMware ESX 3.5 Update 2

According to my contacts within VMware—and many of you have probably heard the same—the company is seeking to achieve SVVP certification for every ESX and ESXi release from 3.5 Update 2 onward for the maximum supported configuration of both CPUs and RAM on both Intel and AMD platforms. That includes both 32-bit and 64-bit versions of Windows Server.

You can view the full list of SVVP-certified platforms here.

This article was originally posted on blog.scottlowe.org. Visit the site for more information on virtualization, servers, storage, and other enterprise technologies.

SVVP Certifications for VMware vSphere

• VMware Expands SVVP Certifications
• VMware ESX 3.5 U2 Validated via SVVP
• A Quick Note to VMware
• Thinking on Microsoft’s Licensing Changes
• VMware ESX SVVP Validation Apparently Limited