It’s been about 9 months since I published my own set of Visio shapes for VMware vSphere and Horizon View.  The VMware Visio stencils have been quite popular, with the pages that host them seeing more than 37,000 page views since I first published them.  I’ve since updated both stencils with new shapes and diagrams as I’ve worked on different projects at Clearpath.  Most of the changes are in the Horizon View Visio Stencil, with new shapes and diagrams.  I included some hand-drawn shapes for Horizon View for some fun whiteboard like diagrams that I’ll be using in some posts very soon – the image to the left is a quick peek at what the hand drawn ones look like.  I also added shapes for Persona Management and ThinApp repositories, a Transfer Server shape, and a few client access device shapes.

Both updated stencils are linked below for download:

VMtoday VMware Horizon View Visio Stencil

VMtoday VMware vSphere Icons Visio Stencil

These are in Visio 2013 .vssx format.  If you need another format, leave a comment below and I’ll see what I can do.

If you want to link to these, link to my permalink page for VMware Visio Stensils so folks see the latest and greatest version.

EMC showed off ViPR (previously known as Project Bourne) at EMC World 2013. ViPR is EMC’s solution for Software Defined Storage (SDS), helping to round out the EMC² family’s (EMC, VMware, Pivotal, RSA) Software Defined Data Center (SDDC) vision. Dave provided our introductory coverage of ViPR here. Since being unveild at EMC World, I’ve heard some confusion in social media circles over what exactly ViPR is – along the lines of ‘ask 10 people what ViPR is and get 10 different answers’. To try to clear up some confusion, here is a brief overview of ViPR… [Continue Reading on Clearpath's Blog]

Interested in learning more about the joint EMC-VMware Pivotal startup?  Yesterday at EMC World 2013, Pivotal CEO Paul Maritz took the stage for a keynote to talk a bit about what Pivotal is, why it matters, and how the combination of EMC and VMware assets (think Greenplum, vFabric, GemFire, SQLFire) have already produced products for customers’ most demanding big data / fast data challenges.  I’ve got a quick write-up published on Clearpath’s Blog, with some pictures on how the combined products from both EMC and VMware fit into the Pivotal initiative.  Continue reading…

I’ve said it before, and I’ll say it again – it’s awesome working for an awesome company that supports the greater EMC, VMware and Cisco communities.  The inaugural class of EMC Elect are finding the custom shirts that Clearpath had made for them at EMC World.  The shirts are just one of our ways of saying thanks to the EMC Elect for their contributions to the EMC community.  If you are one of the EMC Elect, make sure you stop by the Bloggers area at EMC World to pick up your custom shirt!

The Clearpath team is attending EMC’s EMCworld event in Las Vegas this week.  EMC is announcing new products and solutions, and we’ll be covering those announcements in a series of posts on the Clearpath Blog.   If you’re not already following Clearpath, feel free to subscribe to our RSS feed or fill out the form for email updates on our site.  The first update of the week is from Ted Evans, our Baltimore area pre-sales technical consultant.  Ted writes about EMC’s DataDomain OS upgrade (DDOS) to version 5.3.  5.3 includes a host of improvements that will improve backup performance using DataDomain Boost and increase the flexibility of your DataDomain backup appliances by consolidating backup and archive functions on the same box.

Check out the feature list for DDOS 5.3 in Ted’s post (http://blog.clearpathsg.com/blog/bid/289811/The-Latest-Software-Update-for-Data-Domain) and stay tuned to Clearpath’s blog for updates throughout the week!

VMware has released updates to several products over the past couple weeks.  Many of the updates take the version 5.1 products announced at VMworld 2012 to Update 1 (i.e. the first major service pack).  The updates, spanning vSphere (ESXi and vCenter), vCloud Director, vCenter Operations, vCenter Orchestrator, vCenter Site Recovery Manager (SRM), vSphere Data Protection, and the vSphere Storage Appliance, include bug fixes, security fixes, and functionality changes.  Here’s what you need to know about the updates, including a few upgrade bugs that you need to know about before you pull the upgrade trigger.

Continue reading on the Clearpath Solutions Group blog….

I’ve had several folks ask me recently about how to support very large NTFS volumes on vSphere virtualized Windows servers.  The current limitation for a VMDK in vSphere 5.1 is 2TB minus 512B.  FWIW, a Hyper-V virtual disk in Windows Server 2012 can be up to 64TB.  Those asking the question want to support NTFS volumes greater than 2TB for a variety of purposes – Exchange databases, SQL databases, and file shares.  Windows (depending on the version and edition) can theoretically support NTFS volumes up to 256TB (depending on cluster size and assuming GPT), with files up to 256TB in size (see http://en.wikipedia.org/wiki/NTFS for more).  The way that they wanted to solve the limitation is to present multiple 2TB VMDKs to a Windows VM, and then use Windows Logical Disk Manager (LDM) to convert the VMDKs to dynamic disks (from the Windows default Basic Disk), then concatenate or span multiple disk partitions into one large NTFS volume.  Talk about a monster VM….   The question to me, then, became this: Is using spanned dynamic disks on multiple VMDKs a good idea?  Here are some of my thought on the question.

First, there are no right or wrong answers.  How you choose to support big data/large disk requirements will be a mix of preference, manageability, performance, recoverability, and fault domain considerations.  These considerations will be at a few different levels – storage array, VMware, Guest OS, guest Application, and backup systems.  A large spanned Windows volume can offer some simplified management – you might not have to worry as much about running out of space, or junior engineers having to think about where to place data in the guest OS.  I tend to avoid using LVM/Spanned Windows Dynamic Disks within VM’s when possible for a variety of reasons – here are some of my considerations (for a variety of systems – Exchange, SQL, file servers, etc.):

Performance

• Some applications, such as Microsoft SQL, can benefit from having more, smaller disks with multiple files in a database filegroup.  Having different files on different disks, on different vSCSI controllers can increase SQL’s ability to do asynchronous parallel IO.  Microsoft’s recommendation for SQL (http://technet.microsoft.com/library/Cc966534) is to have between .25 to 1 data files per filegroup per core, with each file a on different drive/LUN.  So a 8 vCPU SQL server would have between 2 and 8 .mdf/.ndf files on an equal number of drives.  This lends itself to more, smaller VMDK’s that are not striped or spanned by Windows.  This requires a bit of design work within the database, optimizing your table and index structures to span multiple files in a file group.
• Smaller, purpose built files/volumes/LUNS can be placed on the right storage tier with the best caching mechanism (e.g. SQL log volumes placed on RAID1/0 with more write cache availability).
• A single volume may have a limited queue depth.  You’ll probably increase queuing capabilities as you scale out the number of VMDK’s, and Windows will be able to drive more IO as additional disk channels are opened up.
• A greater number of virtual disks spread over different VMFS datastores may increase the number of paths used to service the workload.  This may allow for increased storage bandwidth, more in-path cache, and more storage processor efficiency.

Manageability

• By using Dynamic Disk striping, spanning or software RAID within the guest, you are introducing an extra layer of complexity that you will need to keep in mind while performing operations on the VM/VMDK.  A storage operation on an array, LUN, VMFS datastore, or VMDK within your guest-striped volume could take the whole volume down.
• Having smaller, purpose-built VMDK’s allows you to move specific parts of your workload to a physical storage tier that best suits it.  Putting everything into one monolithic volume doesn’t allow this level of granularity.  For example, I might create a smaller Exchange mailbox database and put executives mailboxes in it.  I would then place the mailbox database in a VMDK on a VMFS on a LUN on a high tier or replicated disk (great use case for VASA Profile Driven Storage BTW).  The interns mailbox datastore would be placed on the lowest tier of non-replicated storage.  This configuration would also lend itself to more targeted and efficient backup schemes.
• This Microsoft TechNet article for Exchange 2013 storage architecture (http://technet.microsoft.com/en-us/library/ee832792.aspx) suggests that using GPT Basic Disks is a best practice, although Dynamic disks are supported.  Conversely, you could deduce that using spanned dynamic disks is not best practice. The TechNet article also recommends keeping your Exchange mailbox databases (MDB) under 200GB, so there’s no need for a VMDK over 2TB is you’re following best practices.

Fault Domains

• The spanned dynamic disk configuration adds an additional layer of complexity and introduces another thing that can fail in the environment.  I’m a big fan of reducing complexity and design elements.
• Splitting your workload out onto different databases, Windows Volumes, VMDK’s, VMFS, LUNs creates smaller fault domains.  A failure of any one of these components would take the entire system out if you placed everything into a single large guest-striped volume.  If you split everything out, a failure in one of the components would be less likely to affect a large population of users or the entire functionality of an application.  Take the Exchange example again – a failure of one LUN/VMFS/VMDK/NTFS/Exchange MDB would not down email for your entire user base if you are following best practices of keeping MDBs under 200GB and distributing MDBs on different logical or even physical layers.

Backup and Disaster Recovery

• Backing up and restoring smaller files, filegroups, or virtual disks (depending on which levels you do backup – within app, guest-OS agent based, or VM based) is faster.  This leads to more efficient backups, shorter backup windows, less downtime on restores, etc.
• Think about that Executive mailbox database – do you really want to tell your CEO that he has to wait while mailboxes for the interns and former employees are restored in your massive LUN/VMFS/VMDK/NTFS/MDB?  No – you want to be able to target that MDB for rapid recovery and more frequent backups (and why is it not in a DAG???).

Volume Maintenance and Management

• Consider volume maintenance tasks like CHKDSK.  The larger the volume, the longer CHKDSK will run to find and correct errors or corruption in the file system.  I’d much rather get a CHKDSK done faster, especially if the CHKDSK needs to force the volume offline!
• If/When we get >2TB VMDK support, think about the time necessary to do a storage vMotion of a massive virtual disk.  And I hope you don’t experience some sort of failure in-flight, or you’ll be starting again from scratch.

Personal Preference / Experience

• I’ve been burned by striped/spanned Windows dynamic disks in the past.  Those experiences led to long, sleepless weeks.  It was sometime back around Windows 2000 or 2003, and I’m sure the technology is more stable now.  But… once bitten, twice shy.

Design Architecture Decisions & Final Thoughts

• Design your apps/data today to fit comfortably within that 2TB size.  You can grow a VMDK up to 2TB (for now – bigger VMDK’s are on the horizon), and dynamically grow the volume within the guest to fill that space if needed (or just thin provision up front and be done with it).  As your data grows in the future, so too will vSphere’s capabilities when it comes to VMDK size.  Don’t handicap yourself today for a perceived future need.
• The same principles are true for VMFS sizing.  Just because you can make a 64TB VMFS datastore does not mean that you should.  Consider a few smaller datastores, with their backing LUNs spread over storage array controllers and storage paths.  Use Storage DRS and storage profiles to manage placement of VMDKs on the best datastore based on capacity and performance.
• Even on VMFS5, where VAAI ATS has eliminated SCSI reservations, there are limits to the total size of VMDK files a single host can address, based on VMFS Heap Size.  Without recently released updates, the maximum VMFS Heap Size on vSphere 5.0 was 256MB, allowing only 25-30TB of open VMDK files per host.  With the patch, we can address 64TB of open files.  If we’re talking about very large volumes within a guest OS, it is possible that you could be bumping into this limit.  RDMs don’t count against the VMFS Heap Size limits.
• There may be times where a single large guest volume is needed.  In those cases you have to weigh the tradeoffs.  RDMs are often a good choice for > 2TB volumes.  Virtual Mode RDM’s can be converted to VMDK with an online storage vMotion once vSphere supports VMDK’s with sizes greater than 2TB, but Virtual Mode RDM’s also have a 2TB limit.  So that leaves you with physical mode RDMs and the limitations that they present.  Another option would be to present a LUN via iSCSI to the guest OS, and use the Windows iSCSI initiator to mount the LUN.
• For file shares, consider DFS.  You could create several smaller file servers and/or shares, and present them in a single DFS namespace.  DFS would make future server upgrades and migrations very simple.
• You might also consider using NTFS mount points within the guest for your large volume requirements (http://technet.microsoft.com/en-us/library/cc753321.aspx).  Mount points could be the best of both worlds as it provides single drive letter access to multiple independent disks with the risk of spanned/stripped software RAID (although they may introduce another layer of complexity).
• NAS storage like EMC Isilon Scale-out NAS or VNX Unified Arrays and serve up to 16TB file systems over SMB 3.0.  Placing the data on a storage array may offer you additional options for array-level snapshots, replication and deduplication, without the overhead of a Windows server.
• The customers who asked these questions were all still working with Windows Server 2008 R2.  Windows Server 2012 is starting to change how I’m approaching this question.  For example, CHKDSK has been rewritten to support much larger volumes.  SMB 3.0 and Scale-Out File Server offer some pretty neat ways of dealing with bigger data sets.  Data Deduplication is built into NTFS in Server 2012.   Resilient File System (ReFS), with support of volumes with 16k clusters of up to 2^78 bytes (75,557,863,725,914,300,000,000), provides for greater than petabyte scale data sets. Storage Spaces allows for stupid simple expansion of storage space on JBOD without the complexity of traditional storage arrays; combined with Cluster Shared Volumes (CSV) you get clustered, failover storage for anything from simple file shares to SQL Server 2012 Parallel Data Warehouse for big data analytics.
• Find your happy place.  I subscribe to the ‘everything in moderation’ maxim for my life, and this extends into my technical architectures.  Just as I avoid a single ‘beefy’ server for a vSphere ESXi host (and two hosts does not a vSphere cluster make either), I also avoid having a ton of tiny servers.  I shoot for a moderate number of sensibly apportioned hosts.  The same is true for the number of disks/datastores I design.  One big volume and you’ve got all your eggs in one basket.  Conversely, if you split your workload up to the Nth degree, you will increase complexity and the number of managed elements in your environment.  Find a sensible middle ground that supports your performance and manageability objectives, while not introducing undue complexity.
• I’m happy to hear your thoughts on this topic – leave a comment below.

I just published an article on the Clearpath Blog that looks at how to size a network for PCoIP traffic in support of a VMware Horizon View deployment.  I also had a look at other network issues that could affect your PCoIP deployment, like latency and jitter thresholds.  Here’s a preview:

PC-over-IP is a purpose-built desktop remoting protocol designed to adapt to network conditions to deliver the best desktop experience possible.  PCoIP renders desktop video, pixel by pixel, to the endpoint (thin client, zero client, Windows client, etc.), while receiving keyboard, mouse, and other peripheral device data (audio, USB) on the datacenter-hosted desktop.  The UDP-based protocol uses a combination of codecs (graphics, text, icons, photographs, video, PDF), combined with compression and intelligent encoding, to deliver a lossless virtual desktop experience.

Planning for PCoIP traffic on the WAN and LAN are critical steps in developing a design architecture for VMware View deployment.  A Well designed environment ensures that the PCoIP protocol can deliver the expected experience to your end users.  Latency, jitter, and available bandwidth are…  [Continue reading on the Clearpath blog to learn about specific bandwidth requirements, latency and jitter thresholds, and how to calculate bandwidth for a VMware Horizon View solution]

vSphere Storage APIs for Storage Awareness (VASA) are one of several VMware vSphere Storage APIs.  VASA, new in vSphere 5.0, provides vCenter with a way of interrogating storage array LUNs and associated datastores to gain visibility into the underlying hardware and configuration of the storage layer.  Storage capabilities, such as RAID level, thin or thick LUN provisioning, replication state, caching mechanisms, and auto-tiering are presented through VASA to vCenter (a unidirectional read operation by vCenter against the array).  With VASA, vCenter can identify which datastores possess certain capabilities.  By associating a VM – or specific virtual disks within a VM – to storage profiles, we can begin to take advantage of VMware’s Profile Driven Storage capabilities.  With VASA helping to guide VM placement, IT can deliver a higher quality of service to match SLA’s.

A few examples of how using VASA can help IT guarantee SLAs are:

• A user-defined storage profile defined for ‘High Speed Sequential Write’ could be associated with a VMDK used for database logging.  This same profile would be assigned to VMFS datastores based on RAID10, with ample write cache.
• VM’s running critical applications could be associated with a storage profile for ‘Synchronous Replication’.  Datastores protected by a SAN-based replication package (such as EMC SRDF or EMC RecoverPoint) would be assigned this profile to guarantee replication of VM’s on the datastore.  VMware SRM would then be used to guarantee crash and application consistency, and automated failover/back capabilities.
• Test/Dev VM’s could be associated with a storage profile for lower tiered disk without a flash based caching mechanism (i.e. EMC FAST Cache) to keep low priority machines from consuming expensive disk and cache.
• A cloud provider configures multiple tiers of storage in a gold/silver/bronze fashion and assigns appropriate storage profiles to the datastores.  Customers choose which tier they want (based on cost vs. performance) and have VM’s automatically provisioned on the correct storage tier.  This can be done in vCenter or in vCloud Director!

VASA-enabled profile driven storage can be combined with vSphere Storage DRS for automated capacity and performance (IOPS) load balancing of like-datastores.  Greater degrees of automation decrease risk while improving SLA’s.  Taken one step further, VMware’s forthcoming vVols technology will basically create a bidirectional VASA capability, where a VM can tell the underlying storage what performance, features, and capabilities it requires and the storage array will automatically create a VMDK on itself to match the demands from the VM.

EMC VNX fully supports the current version of VASA in vSphere 5.1.  To give you an idea of what data can be seen through VASA, here are the storage capabilities exposed to one of the vCenters running against a VNX 5500 (OE 32) in the Clearpath lab:

Enabling VASA against an EMC VNX is very simple.  As of OE (Flare) for Block 5.32, the VASA provider is embedded in the storage processors.   The same is true for OE 7.1 for File and Unified systems, where the VASA provider is embedded in the Control Station.  If your VNX runs versions prior to these I recommend upgrading (feel free to give us a call at Clearpath to discuss your upgrade process and solution compatibilities).  If you can’t upgrade just yet, you’ll need to do a few extra steps of installing the EMC Solutions Enabler to broker access between vCenter and the Storage Processors for VASA.  Here’s the step-by-step procedure for configuring VASA against a VNX running OE 5.32 or later:

From your vCenter Server vSphere Client switch to the Storage Providers node under Administration:

In the Web Client, switch to vCenter -> Storage -> ‘Manage’ tab -> Storage Providers:

Click Add to add a new Vendor Provider:

Enter the details for your array.  For VNX Block, this will be the Name/IP of one of your Storage Processors (SPA or SPB).  You should register only one SP per array for VASA – using both SPA and SPB on the same array is not supported.  I suggest adding a dedicated user account for VASA on the VNX to prevent account lockouts.  The URL should be added in this format: https://<Name or IP of SP>/vasa/services/vasaService:

If you get an error: “A problem was encountered while registering the provider”, try using the vSphere Web Client or log into the vCenter Server and launch the C# vSphere Client there to complete the addition of the Vendor Provider.

You may be prompted to accept certificates when adding the Vendor Provider.  These are certificates from the VNX.  Click Yes to accept them.

Give vCenter some time to sync with the VASA Provider on the VNX.  After a few moments you can review your array datastores to see discovered VASA information:

You can now Manage Storage Capabilities, Enable VM Storage Profiles, and Create VM Storage Profiles.  To do this, switch to the VM Storage Profiles tab under the Home | Management screen on the vSphere Client:

From here you can take several actions:

1. First, Manage Storage Capabilities to verify that all of the characteristics you want to expose have been discovered through VASA.  You can also define your own Capabilties
2. Next, create VM Storage Profiles (as pictured here) to associate one or multiple storage capabilities with your profile.  For example, you might select “Solid State Storage, FAST Cache, Synchronous Replication” for a Platinum tier VM Storage Profile.  For example, you could create a User-Defined Storage Capability for ‘Busted up old SAN’ and associate it with your old Dell iSCSI array, and a User-Defined Storage Capability for “New Hotness in a EMC VNX wrapper”.  You would assign these User-Defined profiles to their respective datastores.  You could then mix and match capabilities present on both arrays with the busted/hotness capability to have VM’s provisioned to the right LUN type on the correct array.
3. Finally, Enable VM Storage Profiles.  Storage Profiles are enabled at the Cluster or Host level, and require Enterprise Plus vSphere Licensing.

With profiles defined in vCenter, you can now edit the settings of individual VM’s and assign your desired VM Storage Profile to the entire VM, or selectively apply profiles to individual VMDK’s depending on their needs.

If you assign VM Storage Profiles to a VM that you later clone or convert to a template, those VM Storage Profiles stay with the VM so any new VM’s created/cloned inherit the same settings.

You’re done.  Your vCenter is now using VASA to communicate with your VNX to enable Profile Driven Storage – a basic building block in the Software Defined Datacenter!

If you have problems, wonder about creating a user account for VASA on the VNX, or want to see other best practices, I recommend EMC Primus 297057: “Login to the VASA Provider on the VNX Control Station fails with “A problem was encountered while registering the provider.”"

A few points of clarification after the good Twitter discussion I just had with Dan Fallon, Jim Millard, and Matt Davis in response to my posting of the new VMware Horizon View and VMware Horizon Workspace / Suite licensing.  In the details on licensing for the new products (full details are on the Clearpath blog), I mentioned that Horizon View licensing changes to per named user from the previous per concurrent user licensing when you upgrade to the new Horizon Suite (a combination of Horizon View, Horizon Mirage, and Horizon Workspace) from View Premier bundle licensing.  A few things to keep in mind about this:

• When you purchase this upgrade (50% off promotion starting this week btw), your View Premier bundle licensing is exchanged for Horizon Suite licensing and your View Premier per concurrent user licensing is exchanged for per named user licenses.
• When you upgrade from View Premier bundle licensing to Horizon Suite, you get two named users of Horizon Suite for every one concurrent license of View Premier exchanged.  The 2fer can work out in favor of many View customers.
• Mirage was per-named-user before VMware acquired Wanova.  Horizon Application Manager (the predecessor to Horizon Workspace) was also per named user.  This swap brings View in line with licensing terms of the other end user computing offerings in the suite when licensed as part of the Horizon Suite.
• If you want to keep using View Premier only and not upgrade to the Horizon Suite licensing, you can keep using the per concurrent user licensing.  You are not being forced to change licensing terms like Citrix did to XenDesktop customers a few years back.

Details of the licensing changes can be found in this VMware Horizon Licensing and Upgrading Whitepaper: http://www.vmware.com/files/pdf/view/VMware-View-Pricing-Licensing-and-Upgrading-white-paper.pdf.  This whitepaper also covers the announced End of Availability for View Enterprise edition, Horizon Application Manager, and ThinApp.  I have more info on the ThinApp EOA here: http://vmtoday.com/2013/02/the-reports-of-my-death-have-been-greatly-exaggerated-vmware-thinapp/.

VMware Horizon Suite and all of the components that make up the suite were made Generally Available (GA) on Monday (March 4, 2013).  VMware Horizon View 5.2, VMware Horizon Mirage 4.0, and VMware Horizon Workspace 1.0 versions are available as stand-alone products or all bundled into the Horizon Suite.  This is an exciting set of releases from VMware.  I wrote a summary of the releases on the Clearpath blog last week if you don’t yet know what is included in the upgrades and new products.

With the release of the new versions and Suite, VMware introduced some new licensing schemes.  VMware has also moved several products to End of Availability (EOA) status, meaning that you’ll no longer be able to buy them.  EOA products include VMware View Enterprise Edition, VMware ThinApp, and Horizon Application Manager.  If you own these products, there are upgrade paths to non-EOA products.  There are also some great promotions going on to entice customers to upgrade to the new Horizon Suite bundles.  I just published an article on the Clearpath blog that details the licensing changes, upgrade paths and promotions – check it out here: http://blog.clearpathsg.com/blog/bid/274033/VMware-Horizon-Licensing-Details-and-Promotions.

Also a quick reminder that the free vCloud Suite upgrade (from vSphere Enterprise Plus to vCloud Suite Standard w/ vCloud Director and vCloud Networking and Security) expires on March 15 (it was scheduled to end in December 2012 but was extended).  Check out the details here: Free vCloud Suite Promotion.

I’ve had questions from customers and have seen some musings on social media about a VMware End of Availability Notice (EOA) for VMware ThinApp.  I thought I might set the record straight here.  The EOA notice is posted on the VMware ThinApp How To Buy page, with a PDF FAQ document here: http://www.vmware.com/files/pdf/thinapp/VMware-ThinApp-EOA-FAQ.pdf.  From the How to Buy page, we read that ThinApp will no longer be available as of December 15, 2013, with no orders for ThinApp being accepted after that date.  This is crappy wording by VMware.

ThinApp is not being discontinued!  The current stand-alone licensing scheme for ThinApp client licenses and the ThinApp Suite (including ThinApp client licenses, ThinApp packager, and VMware Workstation) are being discontinued.  ThinApp as a technology is alive and well, and is more important than ever with the launch of VMware View 5.2, VMware Horizon Workspace, and VMware Mirage – products all capable of delivering ThinApp virtualized software into the end user computing environments that they support.  For those who have heard me speak on VMware’s end user computing solutions at VMUG’s, VMware Knowledge Series, or Clearpath events know that ThinApp is an important element in the abstraction or decoupling of the apps that our users care about, and the Windows OS as the tool that we still need to run those apps.  Decoupling enterprise apps from traditional workstations is a huge part of enabling bring your own device (BYOD) and the shift from IT servicing devices to IT delivering services to our users.  ThinApp isn’t going anywhere.

What is changing is how you buy ThinApp.  As of December 15, 2013, you will no longer be able to buy ThinApp as a stand-alone solution.  It will only be available for purchase through the acquisition of VMware Horizon View (ThinApp is already included in VMware View Premier bundles), VMware Horizon Workspace, or VMware Horizon Mirage licensing.  All three of these solutions are included in the VMware Horizon Suite bundled licensing.  If you have stand-alone ThinApp today, you can continue to use it and get support until the published end of support date.  After December 15, you will not be able to buy stand-alone licensing for ThinApp – if you want stand-alone licensing or need to renew a VMware support agreement, get a quote and buy it now.

Horizon View, Horizon Workspace, and Horizon Mirage all have the ability to push ThinApp packages to the endpoints that they manage, so it makes sense to bundle ThinApp with these solutions.  I do have mixed feelings on dropping stand-alone licensing for ThinApp as I’ve had several instances where the problem required only ThinApp.  I’ve also grown accustomed to having ThinApp packages of my common apps sitting in a DropBox share so they are always available no matter where I work.  But as I think about it, that’s the point.  Users shouldn’t be maintaining their own apps – IT should be delivering them as a service, keeping apps patched, fast, easily accessible.  And that’s just what the Horizon suite of products is about – all apps and user data delivered where, when and how our users are demanding them.  I guess we really are on the horizon of something really cool in end user computing, so long as IT is willing to rise to the challenge of new demands and new technologies that strike a balance between end-user freedom and IT control.

It’s that time of year again, where virtualization bloggers everywhere are courting your vote in the vSphere-Land 2013 Top VMware and Virtualization Blogs poll.  The voting asks the virtualization community to select the blogs/bloggers that they feel are the most influential, helpful and otherwise awesome based on longevity, quality, length of post and frequency.  While I could tell you about how I strive to make each of my posts of the highest quality, great in length (I’ve even had complaints of being too long-winded), filled with original content aimed at educating my reader (and having a bit of fun) and providing useful things like a collection of VMware themed desktop wallpapers and a VMware View Performance Analysis of Logs tool or a load balancer virtual appliance to use for test View deployments, or free VMware vSphere and Visio shapes and stencils, and have been doing so since 2008, that wouldn’t be entertaining for anybody.  So instead, I’ll ask you to consider voting for VMtoday and all of your other favorite vBlogs MadLibs style:

People all over <place> My Moms Livingroom are letting their <plural noun> stomachs be heard by voting in the 2013 vSphere Land Top VMware and Virtualization Blogs <noun> headlamp by voting for VMtoday and other <adjective> chunky <plural noun> bourbons.  Folks think VMtoday is great because of its <adverb> sorrowfully <adjective> decadent posts on topics around VMware, EMC, <Proper Noun> Las Vegas, and DC VMware User Group news.  Headlines <verb> bemoan VMtoday’s articles as being <adverb> scantily <adjective> amusing.  Some might even suggest that VMtoday <verb> squats in the top <number> 457,296 of all VMware bloggers in the <place> parking garage.  While there are some that are certainly better, VMtoday deserves at least a <noun> lawnmower if you have found it to be <adjective>fizzy, but only after you have voted for 9 other best bloggers.  Remember, only those who <verb> climb have their voices heard, so vote today!

That was fun – haven’ done a MadLib in years!  In all seriousness, take a few minutes on this Friday afternoon (or whenever you have the misfortune of reading my nonsense above) and show your support for the many awesome (much more awesome than me) vBloggers who pour a ton of personal effort into freely sharing their knowledge with you.

Vote here: http://vsphere-land.com/news/voting-now-open-for-the-2013-top-vmware-virtualization-blogs.html by clicking the vote button.

I did a bit of troubleshooting today for a customer who was experiencing very slow logon times to VMware View desktops running Windows XP.  I suspect the problem is a fairly common one so I thought I might share my troubleshooting methodology and the solution that got the login time back to normal.  Following a rigid methodology may be overkill for many troubleshooting situations.  If you strongly suspect a root cause to a problem, check the solution before digging in to analytic troubleshooting.  A little bit of Googling may eventually get you to an answer for a particular problem, but having a firm troubleshooting process will help in all situations.

I’m going to lay out my troubleshooting methodology for you, with some VMware View specific examples.  If you’re not interested in the lesson, scroll to the bottom for the probable causes and solution to my particular issue.  If you want to learn a bit about a tried and true methodology for problem solving, read on!

My troubleshooting approach is borrowed from the Kepner-Tregoe process for Analytic Trouble Shooting as written about in their book The New Rational Manager.  The Keppner-Tregoe methodology dates back to the 1950′s and has been used worldwide by corporate, government and other institutions to solve problems and make sound decisions.  The Keppner-Tregoe Analytic Trouble Shooting method was used by NASA to help land Apollo 13, and has been identified by ITIL/ITSM as a recommended problem solving technique.

Trouble Statement

The first step in the method is to define the trouble statement.  That is, what exactly is the problem we are trying to solve?  The better your trouble statement, the quicker you can zero in on what or where the problem may be.  It may seem simplistic or silly, but a trouble statement verbally stated or written makes sure everyone involved in troubleshooting is actually troubleshooting the same issue, not chasing down tangents, unrelated symptoms, etc.  In this case, the opening trouble statement from the customer was pretty simple: “Domain account logons to VMware View desktops is slow and/or or doesn’t complete.”

Specify

As this was a new customer to me, the opening trouble statement pretty much covered the extent of my knowledge of their particular environment.  I have a decent bit of working knowledge on VMware View that can carry me through most troubleshooting, but a more specific understanding of the problem limits the depth of memory (and overworking of already tired neurons) I need to get to the solution.  We get more specifics by asking the right questions.  The specifying questions you ask can be generalized across most any analytic trouble shooting effort (IT, mechanical, relationships, etc.).  The specifying questions attempt to observe the problem (defect) from all dimensions to define a more exact trouble statement that you will use to begin to hone in on a root cause.  Specifying questions, in and of themselves, do not attempt to identify the root cause.  The questions attempt to answer the IS and the IS NOT of the following dimensions:

• WHAT: What is/is not the object, person or unit with the defect?  What is/is not the defect on the object?
• WHERE: Where is/is not the object with the defect observed?  Where is/is not the defect on the object?
• WHEN: When is/is not the object with the defect first observed?  When is/is not the defect observed in the cycle of the object?  What is/is not the pattern of the when?
• EXTENT: How much of the object is/is not affected?  How many objects have/do not have the defect?  Who many defects on the object?  What is the trend?

The IS NOT in these specifying questions deals always, in all four dimensions, with a closely related object or defect which could be affected, but is not related to the problem at hand.

Some examples of specifying questions that could be used in troubleshooting the slow logon times for View desktops are (not all will apply to your particular situation, just some seeds to start you along):

What:

Q. What is the object/process with the defect?

A. The logon.  Put your jumping to conclusions mat away – we don’t know much beyond the stated problem of ‘the logon is slow or doesn’t complete’.

Q. What is the defect on the object?

A. Logons are a fairly complex operation, requiring interaction with external components (Active Directory, GPO, etc.).  At this point, we’ve only defined the object with the defect as a generic ‘logon’. Is the defect on the desktop itself, or a logon script, or a group policy, or maybe something I haven’t even heard of? I don’t know…  Attempt to define this as much as possible.  Do Windows Event logs have an entry of interest that suggests something like “the group policy processing engine experienced a failure’ or ‘domain controller cannot be reached’?  In these cases, the group policy processing engine or the connection to the domain controller is a better answer to the ‘what is the object with the defect’ question.

Q. What is not the object with the defect?

A. Can we eliminate a what?  Maybe the Windows Event Log reports that group policy processing is successful.  We can probably state that the IS NOT as the problem IS NOT a failure of the local group policy processing engine, or the failure IS NOT a failure to communicate with a domain controller to obtain group policies.

Q. Is the defect on local logons as well as domain logons?

Where:

Q. Which View desktop(s) or desktop pools is the problem observed in?  What View desktops or pools is the problem not seen in?

Q. Which OU are the desktops in?

Q. Does the problem exist on physical and virtual desktops, or just virtual?

Q. Which users are impacted (users in OU A but not in OU B)?

Q. Over PCoIP, or on the vSphere console?

When:

Q. When did the problem start?

A. Yesterday; 10:57pm; After desktops have been running for 12 hours;

Q. When does the problem not happen?

A. Saturday.  (Is anyone working on Saturday to actually validate that it isn’t happening then too?)

Q. When in the cycle does the problem occur?

A. When I refresh the pool.

Q. When in the cycle does it not occur?

A. At logoff. After recompose with snapshot X.

Extent:

Q. Are all desktops in the pool affected?

Q. What is the extent of the problem on the desktop?

A. Can’t log in, ever.  Or, login completes, but desktop continues to be terribly slow.

Q. What is not the trend?

A. The trend is not that the problem is extending to newly provisioned desktops, just on already provisioned desktops.

Working Tool Questions

So now we know specifics around what our problem is.  Now we put our thinking man (or woman) tools to use with ‘working tool questions’.  These questions assist us in interpreting our specifications as we week a root cause.

Difference:

Q. What is different, odd, unusual, peculiar or distinct about the IS compared to the IS NOT?

A. The problem occurs on Windows XP View desktops, but not on Windows 7 View desktops.  The distinction is the OS version.

A. The problem occurs on VM’s in this OU, but not that OU.  What is different about the OU’s?  What are the differences in the GPO’s linked to the OU?

Change:

Q. What has changed in, about, or around this difference (deals with WHAT and WHEN)?

A. A new GPO enabling View Persona Management was linked to the OU for the pool with the issue.

A. A Windows update was applied by automatic updates.  The update was removed after a refresh of the pool, until the next automatic Windows Update installation.

Possible Cause:

Q. How could this change possibly cause the trouble?

A. GPO for View Persona Management changes the logon process, as the roamed profile needs to be accessed over the network instead of locally on desktop.  If the Persona Profile repository was unavailable, logons may be slow until a temporary profile was created, or could fail.

A. If Windows Update for KB123456 was applied to the desktop, then the Kerberos encryption method would be updating causing authentication problems with the domain.

Most Probable Cause:

Q. IF _________ is the cause, how does that explain the IS and the IS NOT facts?

A. If Persona GPO was linked to OU for View desktops in the affected pool, and NOT linked to OU for non affected pools, the defect would be seen only on the pool with the linked GPO and NOT on the unlinked pool.

Verify:

Q. Does the cause check out in real life?  Verify it.

A. Unlink the GPO and do a gpupdate.  Does the problem still exist?

• If no, then problem exists with GPO or functionality called by GPO. Congratulations, but don’t pop the top on a cold one just yet (ok, maybe just one – it’s been a long day)….  Even if the problem is resolved with your verification, you’re not done with your work.  In our case, the GPO might bring Persona Management functionality that is required by the design goals of the project.  Repeat the process to identify what/where/when/extent about the GPO is causing problems (or use your new found knowledge to craft a more specific Google query).
• If yes, does the verification step add to our trouble statement, provide additional answers to our specifying questions (do we have a new IS or IS NOT), or answer any of our working tool questions (did we find another change in, about, or around the IS or IS NOT)?

Think Beyond the Fix

Once you’ve identified and corrected what you believe to be the root cause of the problem in your trouble statement, you still have some work to do.  Finding the root cause using sound analytic troubleshooting methodology makes for a good engineer.  Thinking beyond the fix makes for a good leader and manager.

Extend the Fix:

Q. Are there similar unity needing the same fix?

A. Yes, the same GPO is linked to another OU for a View pool that we have not yet tested.

Extend the Cause:

Q. Did the cause do other damage?

A. Yes. An incorrect setting in a GPO caused corruption in the View Persona profile.  Rebuild profile to repair corruption.

Stairstep the Cause:

Q. What caused the cause?

A. Poor change management process.  –> Use the Advanced Group Policy Management Console in the Microsoft Desktop Optimization Pack for GPO revision tracking and change approvals.

A. A bug in the View Persona Management components causes a deadlock.  –> Update to latest version of View.

A. Ignorance.  –> RTFM.

If you are still reading, congratulations!  You’ve completed my Analytic Problem Solving 101 course.  I hope the methodology will prove useful to you.  Before you go, I thought I would share some specifics of this particular issue.  In this particular case, I ran through the methodology partly in my own mind, and partly out loud with the customer in about 5 minutes.  Testing the causes took a bit longer.

I quickly narrowed down the probable causes to be:

1. Permissions on the Persona repository.  Set it per VMware’s guidance in the View Administration Guide, which points to a Microsoft TechNet Article for Security Recommendations for Roaming Profile Shared Folders (last updated in 2003), which is incorrect per VMware KB 2008377.  Experience has shown me that many deployments do not get permissions right because of the old-fashioned DO loop of documentation on this.  If permissions are not configured correctly the profile may not be created correctly, causing slow/failed logons.  While permissions were not to my liking, profiles were being created.  An IS NOT.
2. Persona Management Bug: VMware KB 2011823 describes a problem where Logging in to the View desktop takes a long time when persona management is enabled when Persona and a antivirus/endpoint protection product conflict.  The problem was resolved in View 5.1.2 which was in use in this deployment.  Another IS NOT.
3. Performance of CIFS server where Persona repository resides.  I’ve seen in the past where slow disk performance on the DFS server, caused by a thin provisioned vmdk not growing fast enough as new profiles were created, led to hanging at logon.  The file server seems ok here.  Another IS NOT.
4. Persona Configuration: Persona had been configured with all of the folder redirection options configured.  The IS for this was XP workstations, the IS NOT was Windows 7.  Ah ha – there is a known issue with redirecting Desktop and Start Menu folders using Persona on Windows XP (see VMware KB 2019937 – Desktop login takes a long time when using Persona Management in VMware View Manager 5.0.x / 5.1.x). This is the most probable cause.  To verify, we removed redirection for Desktop and Start Menu and logons were fast.  We applied the ‘fix’ to other desktop pools/OU’s GPOs.  Stair stepping revealed a concern that Desktop and Start Menu items would not be protected by backups on the file server.  I explained that Desktop and Start Menu items would still be protected by Persona roaming profiles, just not redirected to an alternate folder location on the file server.

If you’re stuck on troubleshooting, drop a comment below or check out VMware’s KB that details the steps for Troubleshooting VMware View Persona Management here: http://kb.vmware.com/kb/2008457.

I’ve written in the past about how high CPU Ready values can cause performance problems in VMware vSphere environments.  For those who don’t know, CPU Ready is a measure of the amount of time that a guest VM is ready to run, but the VMware ESXi CPU Scheduler on the host is not able to immediately allocate cycles to the guest because it is busy doing work for other VM’s.  CPU Ready values are exposed through ESXTOP and in the vSphere Client.

I’m often called into customer environments to do performance troubleshooting, and CPU Ready is one of the first performance measurements I check my first few minutes in the environment (I also look at memory balloon driver metrics, disk latency, CPU utilization and memory utilization of both hosts and guest VM’s).  Unfortunately, I’m often called in after the excrement has made physical contact with a hydro-electric powered oscillating air current distribution device, and the customer is demanding a quick fix.  Checking a few basic metrics in the vSphere Client is often enough to put me on the trail of the problem.

Note that the summation value is shown on hosts, guest VM’s and guest vCPU’s in the vSphere client.  The different counters have slightly different meanings.  Host CPU Ready might be a bit higher than an individual guest VM’s CPU Ready counter, for example.  Host CPU ready is a good value to look at if all the VM’s are suffering performance issues.  If just a single or a few VM’s are suffering performance issues, look at the guest VM CPU Ready value.  The guest VM CPU Ready value is a summation of the CPU Ready of each vCPU on the guest.

As a rule of thumb, a Real-Time CPU Ready value of 10% or greater on a vCPU indicates declining performance for server workloads (I usually go with a bit lower value for VMware View virtual desktops (VDI) as users are much more likely to perceive CPU Ready on desktops that they are actively using than on a server they are connected to through a client-server setup).  Theoretically, on VM’s with multiple vCPU’s the guest VM counter is safe to go beyond 10% so long as the per-vCPU counter is under 10%.  For 2 vCPU VM’s the whole VM CPU Ready value can hit 20%, for a 4 vCPU 40%, etc. before we hit that 10% rule of thumb (Because the ESX CPU Scheduler has to co-schedule all vCPU’s on a VM, bigger VM’s are more prone to CPU Ready on hosts with CPU contention.  This probably offsets the theoretical vCPU percentages).

The problem, however, is that the vSphere Client shows CPU Ready as a Summation of Milliseconds of CPU Ready for the Sampling Period.  Summation of milliseconds is not always an easy value to wrap your head around as the impact of the number changes depending on the VM configuration, the charting period (View) / sampling interval.  In some cases a summation value of 2000 can indicate problems, and in other views 1,000,000 may be ok.

In the vSphere client, the chart/graph’s are shown with an update interval.  The summation values are for the entire interval.  For the ‘Realtime’ interval, we’re really looking at 20 second time slices.  On the Past Day view, the interval is 5 minutes (300 seconds).  Past week is 30 minutes, past month is 2 hours, and past year is 1 day.

A little math is needed to convert the summation of milliseconds value to a percentage value – an easier number to understand and compare.  I covered how to convert the summation value to a percent here: High CPU Ready, Poor Performance.  VMware one-up’d me ( ) by publishing a KB article a couple years ago that presented the same formula for converting summation in the vSphere Client to a percentage.  The formula goes like this:

As somebody who struggles with numbers, I don’t want a formula, I want easy.  To save me and my customers from my slow touch-point math, I made this quick set of reference tables to determine at a glance if the CPU Ready summation value I saw in the vSphere client was something to worry about.  I have tables for 1vCPU, 2vCPU, and 4vCPU VM’s.

Note – if you use ESXTOP you see CPU Ready (%RDY) as a percentage, in realtime – no conversion necessary.  If you want to capture ESXTOP realtime CPU ready and then analyze it later, use ESXTOP Batch Mode then analyze in Excel or Windows Perfmon.  ESXTOP counters are explained here: Interpreting ESXTOP Statistics, and here: ESXTOP Performance Counters.

The color coding may seem a bit odd and arbitrary, so here are some real-world numbers to help clarify: If you hit 10 seconds of CPU Ready out of every 20 seconds?  Big deal.  40,000 seconds out of a day’s 86400 seconds?  You’ve got problems. These are easy examples – you’re waiting for CPU cycles ~50% of the time.  But say you hit 90% CPU ready for 10 seconds out of a whole month? No big deal – a blip on the radar (it wouldn’t even show up in the vCenter Statistics Roll-ups).

High CPU Ready for a short period may not be a huge problem, but the numbers can be deceiving.  For example, let’s say you are looking at the Past Year view with a default sampling interval of 86400 seconds (1 day) for a single vCPU VM.  Now let’s say that you see an average summation value of 2,000,000 in the ‘Past Year’ table in vCenter.  That’s 2.315% by our formula.  Sounds low, right?  Not so quick.  As we deal with longer periods of time, the percentages shift a bit from our 10% rule of thumb.

2.315% of our 86400 second time slice is 2,000 seconds, or 33 minutes per day.  This is an average, so there were higher days and there were lower days, but on average, we waited more than a half an hour per day for CPU scheduling for our poor little VM.  Bump the scale up, we waited 12,167 minutes per year –> 203 hours per year –> 8.45 days per year for CPU scheduling.  Let’s say you have an SLA to deliver 5 nines of reliable, high performance for this workload. 8 ½ days is about 97.65% uptime – a long way off your 99.999% SLA.

I have reflected this in my charts by marking the increasingly longer intervals with diminishing warning values (yellow and red) – not because it indicates a problem at a specific point in time, but because it could indicate a systemic problem in the environment.  This is where some critical analysis comes in.  Drill into the smaller time intervals (past day) – is that 33 minutes taking place all at once or spread over the day; overnight, during a backup window, when you are exempted from the business hour SLA or is it happening for longer periods at critical times during the workday?  Hope you packed a lunch, cause you’ve got some troubleshooting ahead of you!

I hope the explanations of CPU Ready and the CPU Ready cheat sheet tables are helpful to you.  Questions, corrections, or additions – leave a comment below!

Over on the Clearpath blog I posted an article on the newly released EMC VNXe Operating Environment (OE) code update.  The update includes support for data-at-rest and data-in-flight (via SMB 3.0) encryption, SRM support, promotion of snapshots to read-write for VMware NFS datastores, and some new options for RAID group configurations.  Here’s a peek at the post:

EMC released VNXe Operating Environment MR4 (version 2.4.20932) on January 7^th.  The upgrade brings a few significant enhancements to the VNXe platform.  Before I cover the new features, let’s have a quick look at the VNXe series for those not familiar with it.

The EMC VNXe Series, a part of EMC’s VNX family, is an affordable unified storage platform designed for smaller businesses.  The unified platform provides both file and block services – iSCSI on block, and CIFS and NFS on file, with a nice base feature set including File Dedupe, Block Compression, Thin Provisioning, and Snapshots, all managed through a very easy to navigate, wizard-driven Unisphere management package.  Additional software packs are available for the VNXe that enable remote replication, file-level retention, and application-aware replication.  Host connectivity includes 1GbE and 10GbE.

The VNXe series includes the VNXe 3150 model and the VNXe 3300. VNXe 3150 utilizes quad core CPUs and offers up to 25 2.5″ drives in the 2U rack mount form factor or up to 12 3.5″ drives in the 2U rack mount form factor.  Both models support the SAS, Near-line SAS, and EFD drive technologies.  The VNXe 3100 supports up to 100 drives on a dual Storage Processor (SP) configuration, up to 192TB of raw capacity. On VNXe 3300 systems, you can have up to 150 drives with a maximum raw capacity of 240TB.

The VNXe is a good platform for smaller VMware environments, including remote/branch offices.  VNXe is also great for Microsoft Exchange, Hyper-V, and SMB file share – all easily carved, configured, and presented to hosts using Unisphere wizards.  The VNXe is also included in smaller configurations of the EMC VSPEX for VMware View 5.1 virtual desktops.

With the 2.4.20932 (MR4) release of the VNXe OE code, the VNXe gains several new features.  Let’s look at these new features and what they mean for VNXe customers.

First is support for Self-Encrypting Drives (SEDs).  A VNXe purchased with SEDs provides Data at Rest encryption of all data on the array.  Encryption is transparent and automatic…. [Continue Reading on Clearpath's Blog]

I’ve been doing a several VMware View installs for customers this month.  During one of the engagements I ran into a strange issue where the Quick Launch icons on the Windows Taskbar of provisioned Linked Clones were blank.  Here’s what it looked like:

Clicking on the icons resulted in a warning: “Can’t open this item.  It might have been moved, renamed, or deleted.  Do you want to remove this item?”  No, Windows; I don’t want you to remove it, I just want you to fix it, thank you very much.

I did a bit of digging and found another user on the VMware Communities forum had the same issue but no resolution.  A little more looking pointed me at this post by Chris Kliewer.  Chris identified the problem and provides a solution.  Here’s what’s going on:

I used Sysprep to copy the user profile that I customized / optimized by specifying the CopyProfile paramater.  When the customized profile is copied to the Default profile a couple folders go missing .  To fix the problem edit your template after Syspreping by adding the following folders – “User Pinned” and “Taskbar” as follows:

•  C:\Users\Default\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\
• C:\Users\Default\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\User Pinned\Taskbar

Shut down.  Snapshot.  Recompose.  Done.

It looks like this problem can affect local profiles, as well as profiles managed by View Persona Managment and Liquidware Labs ProfileUnity.

If you want to automate or customize the pinned icons in the Quick Launch taskbar, see the script provided here: http://theitbros.com/copy-taskbar-icons-windows-7-sysprep-v2/.

EMC announced today the inaugural 2013 class of EMC Elect.  The EMC Elect program, announced in November, recognizes EMC community including customers, partners, EMC employees, analysts and independent voices for going above and beyond to share their knowledge, collaborate, communicate (on social media, the EMC Community Network, at events like EMC World, etc.), and for being generally awesome in their EMC area of expertise.

A panel of highly respected foundering members of the EMC Elect were chosen for their extensive participation with EMC as a brand. Their leadership, commitment and positive attitude set the cadence of the community. These 10 members reviewed the 270 nominations to choose their fellow EMC Elect.  As a founding member, I can say that the review process was rigorous considering each nominee’s contributions to the community as a whole.

I invite you to join me in congratulating these folks - the full list of EMC Elect 2013 members is here: https://community.emc.com/community/connect/emc_elect?view=blog).  I know there are many people deserving of the recognition who were not nominated this year.  I encourage you to think about who in your own local EMC community is influential and most helpful and nominate them for next year’s class of EMC Elect!

Congrats to the new EMC Elect!

My awesome wife and terrific in-laws got me one of those nifty new Yoga 13 IdeaPad’s by Lenovo (Lenovo Yoga 13 IdeaPad Ultrabook 13.3″ Touch-Screen Convertible Laptop) for my birthday/Christmas (one of the benefits of having your birthday 3 days before Christmas is “bigger” gifts). I’ll be using the Yoga at home and at work as we do some BYOD (Bring Your Own Desktop) against the VMware View environment at Clearpath Solutions Group.  The form factor of the Yoga is great – lightweight to be used as a tablet, but big enough to be a full time laptop.  I also dig Windows 8 – I use the traditional Windows desktop at the office, but switch over to the Modern UI for couch surfing in the evening.  I did install Start8 from StarDock to give me back the Start button / menu.  But what I don’t like is the limited storage and low RAM (4GB).

The Yoga shipped with a 128GB SSD, but Lenovo screwed up their partitioning, leaving only 50GB or so for user apps and files.  Not enough for me.  Fortunately, Lenovo released a patch to fix the partitioning, giving me a 100GB C: partition.  Download the patch here: http://support.lenovo.com/en_US/downloads/default.page# – it’s under the General Information section after you select your model and OS.  The patch failed to run with the crappy McAfee antivirus that came preinstalled – I uninstalled it since it was just a trial version and had already caused issues with installing Windows Updates, Hyper-V features, etc.

I usually run several virtual machines in VMware Workstation on my laptop, but the 4GB or RAM that came installed don’t leave much room.  Between the RAM and storage, it was time for an upgrade.  The Yoga 13 has an open mSATA slot for a second hard drive and the single 4GB DIMM can be swapped out for a 8GB DIMM.  I’m the DIY type, subscribing to the Self Repair Manifesto and the Ron Swanson Pyramid of Greatness, so I rolled up my sleeves and got right to work.

Here’s what you need to know to upgrade your Yoga.  This could very well be an unsupported, warranty voiding upgrade, so use caution!

First, the hardware.  I went with Crucial for my added memory and bought it through Amazon: Crucial Single 8GB DDR3 1600 MT/s (PC3-12800) Notebook Module, CT102464BF160B for about $45.  I also bought a Crucial 256GB mSATA hard drive from Amazon for about $200 Crucial m4 256GB mSATA Internal Solid State Drive CT256M4SSD3.  I’ve had very good luck with Crucial components in the past and the reviews for the mSATA drive prove it to be a speedy SSD.

Next, you’ll need some tools – order them with the drive and memory so you are ready to upgrade as soon as the friendly FedEx or UPS man show up:

• Double-sided tape: Scotch® Removable DoubleSided Tape 3/4 inch x 400 inches Dispenser (667)
• Nylon pry tools – I used this set for my upgrade: InterfuseTM Metal & Nylon Spudger Repair Opening Pry Tool Kit
• Small phillips screwdriver and T-5 Torx screwdriver – I got this nice little set of precision screwdrivers that had the right size phillips and Torx: General Tool 63518 Precision Ultratech Screwdriver Set, 18-Piece
• An anti static wrist strap if you want to protect the sensitive electronics in your new Yoga from static shock: Belkin Anti-Static Wrist Band with Adjustable Grounding

Update: Feb 2, 2013: There is a firmware update available for the Crucial mSATA drive I linked to above.  Go here to download: http://www.crucial.com/help/ssd/index.aspx?source=web.

The Lenovo Yoga does not have a CD/DVD/BluRay drive or a ethernet network port.  If you want these for your Yoga, I recommend the following:

• CD/DVD/BluRay Reader & Writer: Samsung SE-506BB/TSBD 6X USB2.0 External Slim Blu-ray Writer Drive (Black)
• Plugable USB 2.0 to 10/100/1000 Gigabit Ethernet LAN Wired Network Adapter for Windows, Mac, Chromebook, Linux, and Specific Android Tablets (ASIX AX88178 Chipset) – Make sure you get the updated drivers for Windows 8 for this USB-to-Network adapter from http://www.asix.com.tw/download.php?sub=searchresult&PItemID=84&download=driver.

You might also want to pick up a spare AC power adapter for your Lenovo Yoga (Lenovo Part Number: 0C19868; Model: PA-1650-37LC; LC PN: 36200235).

As far as the process, I’ll give you some tips that I picked up while upgrading my Yoga and point you to some resources to help you along.  Anti-static strap up and get your upgrade on by following these steps!

1. Remove the keyboard.  Fold the laptop into tablet position.  Then, use the flat end of the black nylon pry bar (assuming you bought the set I linked to above) to gently pull up on the top of the keyboard.  Put the pry bar between the flat plastic that surrounds the keys and the sloping bezel around the keyboard (i.e. – don’t try to pry the sloping part off with the keyboard).  Work your way around both sides with the pry bar, gently lifting until the plastic clips on the sides release.  When you get to the bottom corners put the pry bar down and use two hands (one on the ESC key and one on the Delete key) to pull the keyboard about 1/8″ toward the monitor hinges.  This should pull it loose from the housing.  If you are just upgrading memory, the DIMM is exposed.  Remove the old, put the new one in.  Skip to steps 6 & 8 to reinstall the keyboard. Next, remove the keyboard cable.   Follow the cable down to where it is connected to the motherboard.  There is a small black bar over the cable.  This lifts upward on a hinge – gently raise it towards the memory DIMM until the cable is free.
2. Remove the top and bottom covers.  Use your phillips bit to remove the 5 exposed screws on metal plate under the keyboard.  Keep track of which screws go into which holes – they are different in size.  Flip the laptop over and close the screen over the keyboard. Remove the 10 T-5 Torx screws from the bottom plate – these are all the same.  Now, fold the monitor all the way back into tablet mode.  Using the flat black pry bar, begin to separate the black top from the silver bottom.  Start on the outside of the hinges and work your way towards the palm rests.  The front corner of the palm wrests are held in by magnets, so you might have to pull lightly.  When the black cover comes free, don’t yank on it – there’s cables attached that you don’t want to dislodge.
3. Install the mSATA drive.  At this point, you should be able to see the open mSATA slot under the top corner where the ESC key is.  Insert the drive (it only goes in one way) and secure it with a single screw (this should have come with the mSATA drive – use the one with smaller threads).  If you don’t have the screw, you could use one of the shorter phillips screws you removed from the plate under the keyboard.
4. Now, snap the black top cover back into the silver bottom plate.  Reinstall the Torx screws on the bottom.  Reinstall the phillips screws in the plate under the keyboard.  Don’t over-tighten – just snug them a touch.
5. Place small squares of double-sided tape in each of the four corners of the keyboard, another in the top center, another under the space bar.
6. Look at the bottom of the keyboard.  See those small metal clips that protrude?  Remember them – if your keyboard feels very springy after reinstalling it, you probably didn’t set the  clips into their slots in the plate under the keyboard.
7. Insert the keyboard ribbon cable into the slot as far as it can go, then press the black locking bar back over the cable to secure it.
8. Lay the keyboard as flat as you can on the opening where it belongs.  Align the bottom tabs (under the space bar).  Press down gently to engage those clips I told you about in step 6, while sliding the keyboard downward towards the space bar side until it falls into the bezel.  Gently snap the sides and then the top of the keyboard back in.
9. Fire up the laptop and test it out.

Assuming you did everything right, you should see 8GB of RAM in the System Information window.  From the Modern UI, type System Information to see the System Information app.  You should see your installed RAM value on the screen that opens.

Provisioning the new hard drive space is a bit more work.  You’ll need to initialize and format the drive.  Here’s how you do that:

From the Modern UI, type ‘Format’ – switch to the System search on the right.  You’ll see an option to ‘Create and format hard disk partition’.

An Initialize Disk window will open when Windows detects the new hard drive.  Keep the defaults and click OK.

The new drive will now appear as an unformatted device in the Disk Management window.

Right-click the unformatted space and choose New Simple Volume.

The New Simple Volume Wizard launches.  Click Next on the Welcome Screen.

Keep the default size to use all of the space on the new drive for a new partition.

Assign a drive letter.

Keep NTFS as the file system, with the default Allocation unit size.  Add a label – I use DATA, because, well duh – that’s what it will hold.  Tick the box for Perform a quick format.

Finish!

Now when you launch My Computer you will see a new drive with the label and drive letter you assigned.  Use your space wisely!

I put my SkyDrive folder on the new drive as I am shifting most of my personal document storage to SkyDrive.  If you want to do the same, download the SkyDrive client from here: http://windows.microsoft.com/en-US/skydrive/download and install.

During the setup wizard, switch your SkyDrive folder location to the D: drive.

Now one more change.  Windows 8 creates some default Libraries with locations for you to save, search and interact with your files.  I’m going to add SkyDrive folders to each library (Documents, Pictures, etc.) so SkyDrive becomes my default location for my documents.  To do this, open My Computer (Explorer) and right-click –> Properties on the library you want to change.

Click ‘Add’, browse to your new SkyDrive folder, and select the appropriate sub-folder.  Then select the folder you added and click ‘Set save location’ to make the new folder your default save location for the Library.

With your libraries redirecting your My Documents and other folders to SkyDrive, you’ll have a cloud-based copy of all of your files automatically.

Hope this has been helpful!  If you need help or a visual on doing the hardware upgrade, I recommend you check out this video on YouTube: http://www.youtube.com/watch?v=Z5uCooHm4kw&list=UUmni3hsnYA0XuulA_V7xmtg&index=1.  The guy who made it (Eben Howard) did a good job of capturing the feel of the upgrade – how much pressure to exert, snap tab locations, etc.  His blog post on the process is here: http://squidpony.com/not-games/hardware/lenovo-yoga-13-memory-upgrade/.

It’s been a long time since I published an article in my Storage Basics series – the series has been some of the top content on my site to date.  I sat for a beta version of the VMware Certified Advanced Professional – Desktop Design (VCAP-DTD) test today and was reminded by the test of a post I had in waiting about an alternate way to calculate IOPS (hint hint – memorize these formulas).  I have several more articles sitting in draft form, but a new job, crazy kids, home improvement projects, and a wife with chronic cerebral spinal fluid (CSF) leaks (it sounds bad – and it is – but look on the bright side: not many guys can get away with saying that their wife’s brain leaks!) all take time.  But enough excuses and back to storage….

I wrote in Storage Basics – Part II: IOPS on how to calculate theoretical IOPS for a given disk type and speed, and then followed it up in Storage Basics – Part III: RAID with some information on  how different RAID configurations impact IOPS.  I.E. More disks equals more IOPS, but with a write penalty assigned for different RAID types.  I then moved into a discussion of interconnects in Storage Basics – Part IV: Interface, but didn’t do a good job there of tying the interconnect speed to performance and IOPS.  Finally, in Storage Basics – Part VI: Storage Workload Characterization, I hinted briefly on how I/O request size can impact storage performance.  Let’s dig into this topic a little deeper!

The Original IOPS Forumula

Calculating IOPS at the disk/RAID level is fun and all (and I suggest you brush up on it if you are sitting for the VCAP exams), but does not paint a full picture of the storage subsystem.  The basic formula to get a single disk IOPS (I covered this in Part II) looks like this:

For simplicity sake, let’s just assume that this formula yields 180 IOPS for a standard 15k RPM SAS disk (the industry standard number of IOPS for this speed and type of disk)

The formula for RAID IOS is in Part III – here’s a sample for RAID5, where:

p = IOPS required, f = write IO penalty factor, r = % Read, w = % Write

Let’s put it to use: let’s say I have a workload that I’ve measured as needing 4000 IOPS from the OS/application perspective, of which 70% are write (not unusual in VMware View Linked Clones): here’s how you figure out how many disks you need from an IOPS perspective for those linked clones:

*

*12,400 Back-end IOPS needed on my spindles in a RAID5 array

Then, convert the number of back-end IOPS needed to a number of disks needed given our RAID5 assuming the 15k SAS disk:

That is, I need 69 disks (rounded up) required on the back-end to meet our 4000 front-end IOPS, ignoreing things like best practices when putting a bunch of disks in a RAID group/storage pool (4+1 RAID5 sets) and hotspares (1 in 30 for SAS)

A quick run at the math suggests that you should probably not be using RAID5 for a write-heavy workload (run the math yourself with a write penalty of 2 for RAID0+1 to  see how many fewer disks you would need).  Also, this particular example leaves out any calculation for cache impact, including EMC FAST Cache or the View Storage Accelerator (CBRC).  If you have those technologies in the mix, then a very rough rule of thumb would be to size your back-end IOPS based on your front-end IOPS calculation (4000 in the example above, instead of 12,400) – I say very rough because this really depends on your cache size and cache hit ratio, and the rule pertains a bit more to read-heavy workloads than write heavy.  If you have CBRC in the mix, you *might* be able to greatly reduce and maybe eliminate the Read IOPS from the equation.  But before you go taking my word as gospel, test, test, and test some more using your particular workloads.

The Alternative IOPS Formula

But this is all back-end spindle stuff and me having fun with .  Hopefully your SAN admin already knows this, so go ahead and give her your front-end IOPS count and read:write ratio and you’ll get some LUN’s presented.  But you, the VMware admin, might be responsible for server hardware, including buying the interconnect card (HBA, iSCSI NIC).  How do IOPS inform your selection?  This is where the alternate formula for calculating IOPS comes in.  Here’s the forumla:

Warning: I had initally written this based on simple Gb to mB conversion – totally ignoring encoding, baud line rates, overhead, duplex settings, etc.  This skewed the number of IOPS that the formula yielded upwards of what is really possible.  Thanks to Stephen Foskett and Phil Jaenke for keeping me honest on these numbers and some fun Twitter back-and-forth.  Real throughput numbers for different devices can be found here: https://en.wikipedia.org/wiki/List_of_device_bandwidths#Storage.  While I’m using ‘real’ line rates now, keep in mind that this is all still theoretical.  Your real world experience will differ, perhaps even differ drastically, depending on your unique environment.

Thanks again Stephen Foskett for this formula for converting FC to rates:

For 1, 2, 4 and 8Gb FC with 8b/10b encoding:

;

then convert Mb to MB:

Solved for 8Gb FC with a rate of 8.5MBaud (8.5GHz) we get:

;

For 16Gb FC, encoding is changed to 64b/66b encoding. Otherwise the math is similar: 1700MBps

I’ve updated the numbers below to reflect the real-world numbers above as of 1/2/2013.

So let’s say you have a sequential write throughput of 850MBps (i.e. 8Gb FC), and block sizes of 64KB (default block size for Windows XP/2003).  The theoretical IOPS in this situation is as follows:

Theoretical Maximum IOPS

Now, let’s swap that 8Gb FC HBA for a 10Gb Ethernet storage protocol (iSCSI, NFS, or FCoE – we’re just talking theory here, so pick your poison):

Theoretical Maximum IOPS

Let’s say you are rocking 16Gb FC HBA’s with the latest version of vSphere – 5.1 (5.1 introduced full support for 16Gb FC HBA’s:

Theoretical Maximum IOPS

Bottom line here: more bandwidth = more throughput = more IOPS.  But, your back-end spindles must be able to support this level of IO.  16Gb FC ain’t pixie dust that makes latency on spinning disks go away.

Now, keep vSphere 5.1 and your 16Gb FC HBA’s but switch to Windows 7 View Desktops or Windows Server 2008 R2, which both try to use 1MB block sizes (on aggregated write I/O).  In reality, you probably won’t see a whole lot of 1MB block sizes in your VMware workloads as Windows memory pages are still 4k in size, and thus files are loaded into RAM in 4k blocks, and ESXi may randomize and split up IO further – more in Jim Moyle’s document on Windows 7 IOPS for VDI: A Deep Dive.  You can capture and view your block sizes, and plot out the data in ESXi using vScsiStats.  But for the sake of science, let’s work the numbers:

IOPS

Whoa – what just happened there?  Bigger block sizes gave us lower IOPS potential?  Yep, and that’s a good thing.  We’re moving the same or more data through the pipe, but doing it with a lower IOPS requirement from our back-end spindles.  And at the same time, we’re realizing greater CPU efficiency on our hosts (10-15% reduction in CPU on your ESXi hosts when you switch from 8Gb FC to 16Gb FC according to VMware’s Storage I/O Performance on vSphere 5.1 over 16 Gigabit Fibre Channel whitepaper).

Now, let’s flip the equation around (digging deep to remember my 7th grade algebra here) and assume our 4000 IOPS requirement from our earlier example.  Also assume that we’re dealing with a 4KB I/O size typical of many Windows workloads.  Let’s calculate how much storage bandwidth you need to satisfy the requirement:

  Solve for X.

or .12Gbps

In this example, you’ll be using roughly 12% of your 1Gbps iSCSI or NFS.  A pair of 1Gbps NIC’s for storage traffic and you are set (as long as your storage admin carves up enough spindles and/or cache to support that 4000 IOPS).

So there you have it.  A couple ways to calculate IOPS requirements to figure out both back-end spindles for a given workload, as well as interconnect speeds for the workload.  Put both numbers together and you’ll be well on your way to correctly sizing your next storage implementation.  Questions, comments, or class clown comments?  Drop a line below!

Extra credit:

• Play with the numbers to see how different block sizes impact decisions around storage protocols/interconnects.  I.E. – SQL with 8KB IO, backup applications with 256KB IO, etc.
• Think about Storage Alignment – how does a misaligned partition cause this math to get funky?  Hint: 2 blocks being returned for each IO when misaligned.
• For a little more reading on IO size and IOPS, I recommend the following:
  • Improve VDI Performance with IO Length Trending – read down through the comments (Chad Sakac has a good one here: http://myvirtualcloud.net/?p=988&cpage=1#comment-3620).
  • Large I/O Block Size Operations Show High Latency on Windows 2008 – good info on ESXi breaking down large guest disk IO into smaller block sizes.
  • VMware View Storage Considerations: Covers a bunch of storage related items, including block size and the IOPS formula I introduced here.

If you want to see all of the articles in my Storage Basics series, visit this link: http://vmtoday.com/category/storage/storage-basics/.