Alright a little forward I admit but I wanted you to feel my point…

Ok. What option am I talking about? Namely the, “Consolidate snapshots between bookmarks”. Now a couple of things:

• The option has its purpose <to someone no doubt>. What it means is that the ONLY snapshots that will be consolidated are the ones between user defined bookmarks. ONLY…
• The option is unchecked by default. So its not EMC’s fault if you check the option and consolidation doesn’t work as intended. Know your options <myself included>, Although to me this should say Only Consolidate Snapshots between bookmarks as that is the producible outcome within the GUI.
• IMO the reference in the admin guide (page 51) is a bit, shall we say, poo-poo laden…

  Consolidate snapshots between bookmarks. Default = disabled. The system will snap the start and end times of the consolidation to a bookmark with the time stamp nearest to the times specified when daily and weekly consolidations are configured—within a ten percent range.

  NOTE: This setting has no effect on the Maintain all snapshots for
  period.

Where is this option located? See below,  but under the replica volume within your CG > policy configuration tab> protection settings…

  <——-Select the secondary copy under your CG                                                  

             What a consolidated snapshot looks like ———->

Seems so sweet, so natural, to check this box, you know? It might as well be laced with white fluffy bunnies. One would think you would want to consolidate snapshots between bookmarks but oh no checking this option will cease system automated snapshot consolidation. As noted it only ceases automation, you can still consolidate snapshots outside of user defined bookmarks manually via the CLI.

For your reference the Primus number is emc216467 below…

Well, well, well, what do we have here. A new point release from our old pal gill. Looks to be feature rich and full of tasty morsels. You’ve heard it from me before <until I was blue in the face> but again if you aren’t using RecoverPoint for your local and remote replication needs then perhaps you would have more fun kicking yourself in the chest. Hey to each his own..

Anyways lets take a look at some key improvements shall we…source as follows here.

• Synchronous replication support-Not sure I fully get this statement, “Data can now be replicated synchronously over Fiber Channel”. Yeah, yeah, OK, I get it but wasn’t this a feature in 3.1? I assumed that all replication over FC albeit local or remote (stretched CDP) was synchronous.  Continuing on with the next statement, the fact that you now have the option to always replicate in sync mode or always replicate in async mode or dynamically float between the two (which was the default option) is a welcome change. Now forcing sync replication was possible in 3.1 but hardly intuitive. It was a force regulation,  snapshot granularity/write lag tweak <hut, hut, hike> from what I remember.
• Hardware Management GUI wizards- You cant go wrong with GUI’s in my opinion. It simplifies common tasks providing workflow to multi step processes. Personally, I don’t feel I have to assert my smarts, my manhood, by truck in’ it up at the command line when a task can easily be accomplished via the GUI. I hear you, CLI has its place (i.e. scripting mundane tasks, larger env’s maybe) just don’t follow the herd on this unless you enjoy inflicting pain to yourself.  But hey, you know me I’m into simplicity as a virtue, doing what is right and efficient for our customers and ultimately making it reproducible and comprehendible for others to grasp. I don’t have to manage these environments but I have a vested interest in seeing our customers succeed. Remember its not about you : ) Anyways here are the additions..
  • RecoverPoint Installation Wizard-this was available in 3.1 (or something like it) but not part of the RP management application.
  • Add New RPAs Wizard – HIP
  • RPA Replacement Wizard -HIP
  • Upgrade tool-Assists in providing NDU’s of 3.1 code to 3.2 code – HOO-
  • New RP/SE Installer Wizard -RAY
• New Management Application GUI wizards
  • Getting Started Wizard-Guides user through RP licensing, system reporting, alert settings for new installs.
  • New Consistency Group Wizard
  • Add Replication Set Wizard
  • Journal Volumes Wizard
• More Granular System Monitoring
  • Increased awareness around looming limits, i.e. licensing, RPOs, specific policies, etc.
• Stat tools-Optimized, Unified, and Amalgamized
  • Granularity of stats available at the per minute, hour, and day (per year available for this time frame)
  • Also available a tool that binds these stats into a graphical representation within Excel.
• vCenter Server Monitoring
  • This by far is one of the most magnetic features worth mentioning, integration between vCenter and the RecoverPoint Management. Yes you now have view-only insight into the replication status of each LUN within vCenter via RP GUI. In addition RP can even generate alerts informing you when a VM is no longer protected. Nice to see that VMware partnership leveraged a bit more..
• LDAP support
  • I hear the need for this a lot, it was only a matter of time before it made its appearance and the time is now. Complete integration with MS-AD.
  • With that said the Security-Admin role as been added to allow configuration of LDAP, change users, roles, and security levels. Sadly the “admin” account has been stripped of said permissions and has taken on custodial duties at a local Dairy Queen in Poughkeepsie NY, a real EGO killer if you ask me…
• Support for CLARiiON LUNs greater than 2TB and attachment of up to 2048 LUNs-CLARiiON Splitter Only
• Online help
  • Not sure if you ever noticed but prior to 3.2 there was no online help within the RP Management GUI, peachy addition if I must say.
• ESRS support
  • Now Available-Remote preemptive support, monitoring and auditing via secure means by EMC Support for your RPA’s.
• Improved SAN Diagnostics
  • Of note here, during the initial installation it is no longer possible to continue forth if there are any errors during the SAN Diagnostic phase.
• AND FINALLY VMWARE SRM TOLERENCE OF CLR CONFIGURATIONS WITHIN RECOVERPOINT IS NOW SUPPORTED. Note: Updated compatible SRA is not available from VMware’s site as of 6-26-2009.

Well that’s about it for now. This list is certainly not definitive as it only focuses on the new features. So I encourage you visit the source for more detail especially in the way of what this release fixes. Happy hunting and bravo EMC, thanks for making a great product even better!

Continuing on, here is a quick synopsis of the following webcast, http://www.vmware.com/a/webcasts/details/261, title above.

• So what’s new you ask? Storage efficiency as it relates to the following…
  • New iSCSI software initiator
  • PVSCSI adapter
  • VMDirectPath IO
  • Virtual disk thin provisioning
• Improved management
  • vCenter storage capabilities
  • Dynamic expansion of VMFS volumes
  • Enhanced storage vMotion
• vStorage Thin Provisioning
  • Consumes only what is used (over commitment)
  • Fully supported on block storage
  • VM sees full logical disk size at all times
  • full reporting and alerting on allocation and consumption
    • improves storage utilization
    • eliminates over provisioning
  • Via enhanced Storage vMotion gives you the ability to migrate thick to thin disks
  • Potential negative? Metadata updates needed more frequently with thin disks
  • Fragmentation a problem? incremental size increase is based on block size of VMFS volume. Lower block sizes less of an issue.
  • TP Option available at: VM creation, clone to a template, clone a VM, migrate a VM (Storage Motion)
    • Reporting and alerting are key, Rate of disk growth (writes specifically) important to note for thin disk. Ultimately this can affect performance.
    • Eager zeroed thick disk required for FT, thick lazy zeroed default for VM creation

  • Additional Datastore Management built around the following:

    • Added Datastore views, you can now manage Datastores independent of the hosts.

    • Much need usage reports on a per DS level

    • The ability to set alarms/alerts on capacity, and group DS’s as foldered objects. Not only that you         can set permissions on who can allocate to certain DS’s. Bitchin’..

  • Dynamic expansion-VMFS volume grow

    • Current options to alleviate oversubscription

      • increase the Datastore size or VMFS Volume (add extend, span, grow the VMFS volume)

      • Storage vMotion

      • Cold migrate of VM to another Datastore

      • Delete old and unused VM’s from Datastore

    • VMFS volume grow allows you to expand an extend so that it fills the available adjacent capacity there fore improving VM availability

      • Can grow as many times as needed up to max volume size of the VMFS volume

      • Must grow LUN backing for the VMFS Datastore first. So LUN expansion and then the VMFS volume grow

      • Volume LUN size is still maxed at 2TB. This stems from SCSI 2 compliancy with in the vmkernel.

      • Storage VMotion now supports NFS/iSCSI and FC and fully GUI integration. local experimental

  • Enhanced Storage vMotion

• Storage VMotion now supports NFS/iSCSI and FC and is fully GUI integration. Local disk migrations are experimental.
• Changes within the mechanisms behind this function
• Snapshot in 3.x to do sVMotion, in vSphere however, enhanced Storage VMotion flows as follows.
• Copy VM home to new location
• Start changed block tracking
• Pre-copy disk to destination (multiple iterations)
• Copy all remaining disk blocks Fast suspend/resume
• VM to start running on new home and disks
• Delete original VM and disks
• Furthermore in vSphere, there is support for moving VMDK’s from thick to thin formats and migrating RDM’s to VMDK’s
• Software iSCSI initiator
• No longer requires Service Console connection for initiating communication with iSCSI target.
• Additional performance tuning parameters under advanced within the initiator properties
• CPU cost improvement—read +10 to 25% improvement, write 20 to 50% improvement
• Jumbo frames and 10G support for the TCP/IP stack
• vStorage API for multipathing
• Pluggable storage architecture, that gives the storage vendors the ability to write MP software with insertion into the vmkernel
• Para-virtualization SCSI adapters
• SAS Para-Virt PCIe storage adapter
• Hardware spec written by VMware
• Provides functionality similar to bus logic, lsilogic and lsilogic SAS
• Supports MSI-X, PME, MSI capabilities in the device
• Drivers available for windows server 2003, 2008, and RHEL 5
• Key benefits—
• Lower overhead and higher CPU efficiency in I/O processing
• Higher throughput and low latency
• Better performance under high I/O conditions
• Better VM scalability (more VMs, vCPUs per host)
• Caveats-does not support boot disks with ESX 4
• VMdirectpath I/O (Experimental)
• Feature allows you to map a single HBA to a single VM. Prevents sharing of the HBA by more than a single VM.
• Allows VM’s to directly access the underlying hardware devices
• vMotion, hardware independence and sharing of physical I/O devices not supported to the that VM using VMdirectpath I/O
• Experimental support for the following i/O devices:
          -Qlogic QLA25xx 8G FC
          -Emulex LPe12000 8G FC 
          -LSI 3442e-R and 3801e (1068 chip based) 3 Gb SAS adapters

If you’re like me there is no shortage of links, webcasts, youtube videos, etc. available to us that explain the upgrade considerations for vSphere. But who has the time to sift through all that? Well I don’t, but I did so you don’t have to. Here is a quick rundown below of what to consider when designing future engagements around upgrades…

Summary below taken from  here, the How to Upgrade from VMware Infrastructure 3 to VMware vSphere 4 webcast.  I plan on going through the others listed here and will comment accordingly…

**************************************************************************

New Performance Enhancements in VMware vSphere 4

http://www.vmware.com/a/webcasts/details/249

Improve Storage Efficiency and Management with VMware vSphere 4

http://www.vmware.com/a/webcasts/details/261

VMware vSphere 4 QuickStart Series Part 1: Install and Configure ESXi

http://www.vmware.com/a/webcasts/details/260

VMware vSphere 4 QuickStart Series Part 2: VM Management with VMware vCenter Server

http://www.vmware.com/a/webcasts/details/259

VMware vSphere 4 QuickStart Series Part 3: Cluster Setup, Availability and Load Balancing

http://www.vmware.com/a/webcasts/details/258

VMware vSphere 4 QuickStart Series Part 4: Monitoring, Availability, Back Up and Next Steps

http://www.vmware.com/a/webcasts/details/257

**************************************************************************

****Migration Checklist if needed****

http://www.vmware.com/files/pdf/vsphere-migration-prerequisites-checklist.pdf

****All VMware Products that aren’t supported with vSphere yet****

http://partnerweb.vmware.com/comp_guide/docs/vSphere_Comp_Matrix.pdf

Phased Approach for Customer Upgrades

Phase 1- vCenter upgrade to vCenter 4.0

· Upgrade compared to net new.

• Net new-performance and configuration (clusters, RP, folders and roles/permissions) data would be lost can it be easily recreated? Does the customer want to retain performance metrics? Involved ACLS littered through VC that need to be recreated?
• Upgrade-preserves configuration and performance metrics but loose the opportunity to refresh the environment. Time available to upgrade is also a consideration, is VC a tier 1 app?
• DB considerations-SQL 2k/oracle 9i no longer supported
• Read permission to root system disk for network service account-Not really sure if this is any different than the VC 2.X or why it needs this.

Phase 2- ESX upgrade

• VUM-VMware Update Manager. For customers with vCenter. Does the customer need to reconfigure partitioning, not an option with VUM. Quickest method to upgrade.
• Host update utility (host needs to be in maintenance mode). For customers without vCenter. Does the customer need to reconfigure portioning, not an option with HUU. Next quickest method to upgrade.
• Or clean install. For customers with either. Only method that allows you to reconfigure your mount partitions.
• vSphere REQUIRES 64-Bit hardware, Check HCL

Phase 3- VM upgrade

Note: Order is paramount here, don’t deviate

• VMware tools upgrade first. Required. Reboot needed, plan accordingly.
• Virtual hardware upgrade second. Required. Reboot needed, plan accordingly.
• Virtual hardware version 7 (ESX4) not supported on previous versions. You can select version 4 hardware for backwards compatibility

******Time for upgrade. Take this for what it is worth, but here are some estimates for said process******

•                 Management- ~2 hours per vCenter instance
•                 ESX- ~1hour per host
•                 VM-~15 minutes per VM

Other info worth noting…..

• No upgrade to VMFS needed
• No upgrade from boot CD, only host update utility or VUM
• Upgrade virtual hardware manually or automatically through VUM. Remember VMware tools first then virtual hardware

Note: Check HCL’s for systems, I/O and storage array compatibility, guest OS, software ,etc. These items are continually updated. Just because a particular item was supported in 3.X does not mean it is supported in 4.X

Licensing in vSphere-Relax the days of Old are behind us. 4steps to complete.

General licensing can be done after phase1. If running in evaluation mode you have 60 days to complete.

Process for licensing down to 4 basic steps. No portal activation step or license server needed. No FlexLM. Although if you are running 3.X hosts you will need to maintain your license server until you are fully upgraded to 4.

1. Receive license keys in email
2. Enter license keys into VC UI
3. Assign license key to ESX hosts
4. Product is now activated.

Note: Also a more accurate view of entitlements will be available in this release, Phew what a welcome change. 3.X entitlement views were spacey, convoluted and generally cryptic. 

Console OS/Service Console Considerations

The Console OS, as you know, is now encapsulated in a VMDK file stored on a VMFS datastore. Ok, what design considerations need to be discussed you ask?

• Size requirements- recommended 10G
• Placement-Local VMFS or Shared Storage? Here is an interesting caveat, if you decide to use shared storage then the LUN allocated has to be masked and zoned to only that single host. Well that last option sounds overly complicated, does it to you? Let’s keep it simple, Local VMFS  for the COS….Oh and you can’t storage vMotion the COS once it’s installed. So be decisively sure that you want the COS on local VMFS : ).
• If upgrading you will need to verify that any agents that existed in 3.X are supported with vSphere. And to that end you will need to install a supported version.

Upgrade Paths

Most of our customers if not all are at ESX 3.X and VC 2.5. So there isn’t much in the way as far as direct path upgrades. See below…

• Direct Path Upgrade from VC 2.X to VC 4
• Direct Path Upgrade from ESXi 3.5 to 4.0
• Direct Path Upgrade from ESX 3.X to 4.0

Old and New coolness with vSphere and what you need to know

Why do we upgrade? Increased code stability, general supportability and yes, new coolness (features). Here are some limitations with the new features in vSphere and what you need to consider when designing (not all encompassing but a good start)

VCenter Linked mode-Design Considerations

                1. VC instances need to be part of a domain

                2. DNS- rule of four (short, long, reverse, and domain lookup)

                3. NTP services- less than 5 minutes apart from other VC instance

                4. VC server cannot be a DC or terminal server

Storage vMotion-Design Considerations

                1. Make sure VMs don’t have snapshots

                2. Persistent or RDMs supported

                3. Max of 4 simultaneous vMotion or Storage vMotion accessing a single datastore

Thin provisioning-Design Considerations

                1. Thin disk and VMware FT? Not happening

                2. More disk needed to inflate thin disks?

Host profile-Design Considerations

1. Identify golden master for baseline. Not much here, self explanatory.

vNetwork Distributed Switches-Design Considerations

                1. Configure physical switches accordingly for PVLAN’s

                2. No more than 16 vDS per vCenter Server

                3. Existing standard vSW’s require multiple physical NICs in order to have zero downtime to migrate to vDS

HA-Design Considerations

                1. Same as 3.x requirements

                2. VM monitoring requires latest version of VMware tools

Fault Tolerance-Design Considerations

A lot stacked against FT right now, it no doubt has a niche use case due to its limitations. Customer needs to full disclosure as to where it fits in their environment.

                1. Supported CPU’s is a must, check HCL’s. Start here.

                2. Enable hardware virtualization in BIOS

                3. Turn off power management in BIOS, performance implications if not

                4. Disable hyper-threading in BIOS, performance implications if not

                5. Physical mode RDM’s not supported, virtual mode are

                6. Storage vMotion not supported

                7. NPIV not supported

                8. VMDK must be thick-eagerzeroed (thin will be converted)

                9. Gigabit NIC needed for FT logging (10G can be used)

                10. VMs in HA enabled cluster required

                11. DRS cannot be used for protected VM (manual VMotion OK)

                12. Primary and secondary hosts must be on same build number

                13. SMP VM’s not supported

                14. Hot add of devices not supported

                15. Snapshots not supported

                16. VM hardware must be at Version 7

                17. No more than 4 to 8 FT enabled VM primaries or secondary’s on a single host

                18. Para virtualization guest OS not supported

                19. Remove 3rd party clustering software before protecting with FT

Distributed Power Management-Design Considerations

                1. Ensure vMotion is working correctly-no kidding..

                2. Ensure Wake On LAN NICs are supported and set to auto negotiate

                3. Test each host prior to go live on cluster

The goal is simple..copy data from one set of LUNs to another. Data on source LUNs must remain online to minimize downtime.

All source data must sync to destination LUN to maintain consistency between the pairing, while source is online and available. Furthermore, the tool used must be byte for byte as opposed to block based, stripe alignment (64K or 128K) needs to stick on destination LUN. So SAN Copy is not an option

And most critical of all, the disk signatures for each clustered disk resource must be transferred from source to destination LUN.

Enter Open Migrator..a host based tool from EMC, cluster aware, that does just that. Migrates source to destination, byte for byte, allowing one to stripe align the destination LUN prior to migration. Copy occurs online while source is servicing what it services.

Two reboots, one to install OM and attach OM to each source and destination LUN, a filter level driver captures writes to the source and writes to both the source and destination. And one reboot to perform the swap once the data migration completes.

Now OM has been around for years, so its nothing new to some. But to me it is, my world has been filled with SAN Copy and EMCopy.

BTW, I hate MS clusters (my demons). They have their place I suppose, but I am hoping VMware’s Fault Tolerance eradicates their footprint from this luscious planet……Umm anyway…

Where were we, oh yes..sounds peachy huh? And honestly I was impressed after my first cluster migration, but it seems my trusting nature was a bit premature as the second cluster turned out to be a train wreck.

Here is a quick rundown of the headaches that occurred…

• LUNs were In Sync and Verified prior to the final reboot. Passive node was shutdown.
• Reboot occurred on active node. LUNs came up, destination LUNs (now source), came up mounted with the correct drive letterings and signatures
• The old LUNs come up unmounted signatures swapped. Ok cool, so what happened?
• Well the destination LUNs were in a state of RAWness, inaccessible, no filesystem present. WTH?
• Now the old signatures were tied to these new LUNs of which there was no data. So as you would guess the cluster service was fornicated.
• Ok sir, what did you do to rollback? Why rollback? I had no data, I HAD NO DATA ON THE NEW SOURCE LUNS. I felt like I had been railed in the face by a freight train, not good. So rollback was our only option.
• Now to rollback I had to  swap the signatures back to the original LUNs for the cluster service to start. Delightful, funny enough that was the only part of the migration that worked seamlessly.
• And here we go….
  • We first needed the old signatures documented and paired with the appropriate disk id. We accomplished this through searching the system event log on the active or passive node. I believe the event ID we filtered for was 1304.
  • Once determined, we used  a utility called dumpcfg from the Windows 2000 Resource Kit, to change the signatures. Here is the command…
    • dumpcfg –s <replace with signature> <replace with disk id>
  • BTW here is the path to your current siggy’s——HKLM/System/CurrentControlSet/Services/Clusdisk/Parameters/Signatures
  • Of course, prior to this we disabled the cluster disk within Device Manager/Show Hidden Devices/Non-Plug and Play Devices, and rebooted.
  • Once this command was run we rebooted and enabled the cluster disk and cluster service and rebooted one last time.
  • Once compete the cluster service started with out a hitch, passive node was brought online, failover and failback was rocking.

I will kiss and make up with Open Migrator over time, no worries, it just wont be my tool of choice for clusters…MSCS-1 Me-0.