Following on from my iSCSI problems the other weekend one of my cheap vSphere hosts died and wouldn’t respond following a reboot.

After some investigation and disassembling it would power-up and spin the fans but switch off after about 15 seconds, as everything is integrated onto the motherboard it wasn’t looking too repairable, one of the clips holding on this heatsink had broken; which I assume leads to overheating at boot time and had cooked the chip underneath.

Luckily it was still under warranty from HP; so a quick call to HP support in the UK (well the number is in the UK anyway )  and a call was logged for a replacement motherboard and PSU (just incase); they did offer to send an engineer but I said I was ok fitting it myself and they shipped the parts next-day via courier.

Just incase you ever need to do the same here is a step by step for replacing the motherboard.

1st step is to remove the case – which is just a case of undoing the thumbscrew on the back and sliding the cover off.

2nd step, remove the front bezel by pressing the tabs below – you will need to move the hard drive to remove the board

3rd step – remove the front of the drive cage by undoing the screw and removing the panel.

4th step lift the latch and slid the drive out of the way (you don’t need to totally remove it)

5th step, undo all the cables to the motherboard – try to remember where they came from

Be careful with the case temperature sensor – it’s quite tricky to get to without tweezers unless you remove the CPU 1st.

6th step – remove the CPU and fan/heatsink assembly by undoing the screws below – you’ll need a long torx or small flat head screwdriver to get at the screws

7th step – once undone, the motherboard screws (marked MHnn) can be removed and the motherboard lifted out.

8th step – heatsink/fan assembly is stuck to the CPU itself with heat paste and it should come off with a gentle pull.

The re-assembly is just the same process in reverse – the replacement motherboard kit should have a syringe of heat paste to re-apply to the heatsink/CPU.

New motherboard fitted and server has been returned to service.

I encountered this situation in my home lab recently – to be honest I’m not exactly sure of the cause yet, but I think it was because of some excessive I/O from the large number of virtualized vSphere hosts and FT instances I have been using mixed with some scheduled storage vMotion – over the weekend all of my virtual machines seem to have died and crashed or become unresponsive.

Firstly, to be clear this is a lab setup; using a cheap/home PC type SATA disk and equipment not your typical production cluster so it’s already working pretty hard (and doing quite well, most of the time too)

The hosts could ping the Openfiler via he vmkernel interface using vmkping so I knew there wasn’t an IP/VLAN problem but access to the LUNs was very slow, or intermittent – directory listings would be very slow, time out and eventually became non-responsive.

I couldn’t power off or restart VMs via the VI client, and starting them was very slow/unresponsive and eventually failed, I tried rebooting the vSphere 4 hosts, as well as the OpenFiler PC that runs the storage but that didn’t resolve the problem either.

At some point during this troubleshooting the 1TB iSCSI LUN I store my VMs on disappeared totally from the vSphere hosts and no amount of rescanning HBA’s would bring it back.

The Path/LUN was visible down the iSCSI HBA but from the storage tab of the VI client

Visible down the iSCSI path..

But the VMFS volume it contains is missing from the list of data stores

This is a command line representation of the same thing from the /vmfs/devices/disks directory.

OpenFiler and it’s LVM tools didn’t seem to report any disk/iSCSI problems and my thoughts turned to some kind of logical VMFS corruption, which reminded me of that long standing but never completed task to install some kind of VMFS backup utility!

At this point I powered down all of the ESX hosts, except one to eliminate any complications and set about researching VMFS repair/recovery tools.

I checked the VMKernel log file (/var/log/vmkernel) and found the following

[root@ml110-2 /]# tail /var/log/vmkernel

Oct 26 17:31:56 ml110-2 vmkernel: 0:00:06:48.323 cpu0:4096)VMNIX: VmkDev: 2249: Added SCSI device vml0:3:0 (t10.F405E46494C454009653D4361323D294E41744D217146765)

Oct 26 17:31:57 ml110-2 vmkernel: 0:00:06:49.244 cpu1:4097)NMP: nmp_CompleteCommandForPath: Command 0×12 (0×410004168500) to NMP device "mpx.vmhba0:C0:T0:L0" failed on physical path "vmhba0:C0:T0:L0" H:0×0 D:0×2 P:0×0 Valid sense data: 0×5 0×24 0×0.

Oct 26 17:31:57 ml110-2 vmkernel: 0:00:06:49.244 cpu1:4097)ScsiDeviceIO: 747: Command 0×12 to device "mpx.vmhba0:C0:T0:L0" failed H:0×0 D:0×2 P:0×0 Valid sense data: 0×5 0×24 0×0.

Oct 26 17:32:00 ml110-2 vmkernel: 0:00:06:51.750 cpu0:4103)ScsiCore: 1179: Sync CR at 64

Oct 26 17:32:01 ml110-2 vmkernel: 0:00:06:52.702 cpu0:4103)ScsiCore: 1179: Sync CR at 48

Oct 26 17:32:02 ml110-2 vmkernel: 0:00:06:53.702 cpu0:4103)ScsiCore: 1179: Sync CR at 32

Oct 26 17:32:03 ml110-2 vmkernel: 0:00:06:54.690 cpu0:4103)ScsiCore: 1179: Sync CR at 16

Oct 26 17:32:04 ml110-2 vmkernel: 0:00:06:55.700 cpu0:4103)WARNING: ScsiDeviceIO: 1374: I/O failed due to too many reservation conflicts. t10.F405E46494C454009653D4361323D294E41744D217146765 (920 0 3)

Oct 26 17:32:04 ml110-2 vmkernel: 0:00:06:55.700 cpu0:4103)ScsiDeviceIO: 2348: Could not execute READ CAPACITY for Device "t10.F405E46494C454009653D4361323D294E41744D217146765" from Plugin "NMP" due to SCSI reservation. Using default values.

Oct 26 17:32:04 ml110-2 vmkernel: 0:00:06:55.881 cpu1:4103)FSS: 3647: No FS driver claimed device ‘4a531c32-1d468864-4515-0019bbcbc9ac’: Not supported

Due to too many SCSI reservation conflicts, so hopefully it wasn’t looking like corruption but a locked-out disk – a quick Google turned up this KB article – which reminded me that SATA disks can only do so much

Multiple reboots of hosts and the OpenFiler hadn’t cleared this situation – so I had to use vmkfstools to reset the locks and get my LUN back, these are the steps I took..

You need to find the disk ID to pass to the vmkfstools –L targetreset command, to do this from the command line look under /vmfs/devices/disks (top screenshot below)

You should be able to identify which one you want by matching up the disk identifier.

Then pass this identifier to the vmkfstools command as follows (your own disk identifier will be different) – hint: use cut & paste or tab-completion to put the disk identifier in.

vmkfstools-L targetreset  /vmfs/devices/disks/t10.F405E46494C4540096(…)

You will then need to rescan the relevant HBA using the esxcfg-rescan command (in this instance the LUN is presented down the iSCSI HBA – which is vmhba34 in vSphere)

esxcfg-rescan vmhba34

(you can also do this part via the vSphere client)

if you now look under /vmfs/volumes the VMFS volume should be back online, or do a refresh in the vSphere client storage pane.

All was now resolved and virtual machines started to change from (inaccessible) in the VM inventory back to the correct VM names.

One other complication was that my DC, DNS, SQL and vCenter server are all VMs on this platform and residing on that same LUN. So you can imagine the havoc that causes when none of them can run because the storage has disappeared; in this case it’s worth remembering that you can point the vSphere client directly at an ESX node, not just vCenter and start/stop VMs from there – to do this just put the hostname or IP address when you logon rather than the vCenter address (and remember the root password for your boxes!) – if you had DRS enabled it does mean you’ll have to go hunting for where the VM was running when it died.

In conclusion I guess there was gradual degradation of access as all the hosts fought with a single SATA disk and increased I/O traffic until the point all my troubleshooting/restarting of VMs overwhelmed what it could do. I might need to reconsider how many VMs I run from a single SATA disk as I’m probably pushing it too far – remember kids this is a lab/home setup; not production, so I can get away with it

In my case it was an inconvenience that it took the volume offline and prevented further access, I can only assume this mechanism is in-place to prevent disk activity being dropped/lost which would result in corruption of the VMFS or individual VMs.

With the mention of I/O DRS in upcoming versions of vSphere that could be an interesting way of pre-emotively avoiding this situation if it does automated storage vMotion to less busy LUNs rather than just vMotion between hosts on the basis of IOPs.

You may recall I previously posted on problems installing VMware Workstation 6.5 on Windows 7, this problem seems to have been resolved with the upcoming VMware Workstation 7 which adds support for Windows 7 as a guest and as a host.

You can download the Workstation 7 RC build here Release Build here and see the full features list, I can confirm it installed perfectly on my Windows 7 Ultimate x64 machine.

(Screenshot from RC build, see above link for RTM build)

Some new features include official support for Windows 7 *with Aero support!* (shown below)

And best of all – it now provides "official support for ESX as a guest VM under Workstation (my previous posts on workarounds for Workstation 6.5  here)

As an aside I’m running Windows 7 on a machine with a 64-bit SSD hard drive, I’m hoping to make use of the linked clone functionality to save disk space as I often run VM’s which are built from a common base OS template (see this post here for more info on how I’ve managed linked images in the past)– performance so far has been great both for host and guest as I/O doesn’t get as bogged down as it does with traditional spindle based disks.

**UPDATE: ah, the perils of the scheduled post – as this article went live the final RTM build of Workstation 7 has been released, I’ve updated the links in this post**

Now the noise around VMworld has calmed down I thought I would let you know that the vast amount of excellent technical content that was presented at the event itself is available to stream online or as an MP3 (audio only) or slide download (audio only).

As you’d expect, the catch is it’s not free to you unless you attended VMworld in-person. However, you can purchase a VMworld subscription which costs $699 USD per annum and gives you full access to stream and download content from the event, and all previous events back to 2004 – so if travel and time out of the office is not an option for you – how about you (or your employer) pay for a subscription to the content itself – which is obviously cheaper than attending in person.

I have pasted a full list of all sessions from VMworld 2009 US below – please don’t ask me to post the sessions online, this is explicitly forbidden as you’d expect – if you want the content I’m afraid you’ll have to pay – click the graphic below (but it’s excellent value IMHO

Note: you’ll need a vmworld.com account to view the session details linked below (it’s free and can be done here)

Super Sessions

SS4880	NetApp: Clear up the Cloud – Key Infrastructure Requirements and Real-World Implementations
SS5000	Dell: How to Get Ahead in the Cloud With Your Feet Planted Firmly on the Ground
SS5001	VMware: Extending Your IT Beyond the Datacenter: The vCloud Initiative
SS5081	Wyse: Desktop Virtualization / Cloud Computing: We Did It – Here’s How and What we Learned
SS5082	Cisco and VMware: Delivering Innovation for Virtualization
SS5120	IBM: What You Need to Know to Virtualize Today’s Data Center
SS5121	Intel: Technology transformations central to the evolution of flexible computing
SS5140	EMC: Infrastructure Architectures Purpose Built for the Virtual Datacenter
SS5160	HP: Stop Virtualizing Servers, Start Virtualizing Infrastructure
SS5220	Symantec: Complete the Promise of Virtualization
SS5240	VMware, Cisco and EMC: Engineering Developments Enabling the Virtual Datacenter
SS5241	VMware: Question and Answer Session with Paul
SS5440	VMware: Enabling Better Business Outcomes with Policy-Driven Service Level Management

Business Continuity & Disaster Recovery

BC1500	vCenter Site Recovery Manager "Up and Running" – Best Practices & Avoiding the Pitfalls
BC2082	VMware vCenter Site Recovery Manager Performance and Best Practice (online only)*
BC2142	Data Recovery – Install / Configure and troubleshooting
BC2253	Pain-free VMware Agentless Backup AND Recovery – VCB & Beyond (online only)*
BC2260	Automated Disaster Recovery for Branch Offices using SRM and vSphere 4
BC2412	Managing SAP Virtualization, DR and Upgrades with EMC Ionix Technology (online only)*
BC2425	Using CDP for Cost-Effective Application Recovery in vSphere Environments
BC2520	911 Call Center Leverages VMware To Ensure Survivability
BC2537	VMware Site Recovery Manager with Enterprise Replication – A Customer Testimonial
BC2541	Re-architecting Backup and Recovery for Virtual Environments: Best Practices
BC2565	An Optimized Approach to Workload Availability in Virtualized Environments
BC2704	Site Recovery Manager, a real user experience
BC2760	Protecting the corporate infrastructure: Leveraging Site Recovery Manager for multi-site DR protection.
BC2961	VMware Fault Tolerance Architecture and Performance
BC3042	VMware SRM Enhanced Testing (online only)*
BC3083	Mission Critical: Virtualization and Robust DR Architectures for Vital Systems (online only)*
BC3086	A practical guide for recovering key applications using VMware Site Recovery Manager
BC3189	How Storage Enhances Business Continuance: From Backup to Fault Tolerance
BC3197	High Availability – Internals and Best Practices
BC3209	Creating the Fastest Possible Backups Using VMware Consolidated Backup -A Design Blueprint
BC3210	Preview of Best Practices for Recovery with Site Recovery Manager and NFS
BC3223	Planning for Optimized and Cost Effective Storage Utilizing Deduplication and Virtualization Technologies
BC3301	DR Architecture Design Workshop with SRM
BC3369	VMware Fault Tolerance Real-World Use Cases
BC3370	VMware Fault Tolerance – Overview and Best Practices
BC3384	Using IP based replication as an Enabler for Server Virtualization and Storage Repurposing
BC3386	DR Solutions and Services with Sungard and VMware
BC3387	Advanced Data Protection for Microsoft SQL and Exchange Server in a VMware Environment
BC3396	How VMware uses Site Recovery Manager for its own disaster recovery
BC3421	SRM Architecture & Features: The Road Ahead
BC3425	VMware Availability Solutions and Futures
BC3602	VMware Fault Tolerance – vSphere Workflows and API Considerations (online only)*
BC3780	How Customers are Using VMware Site Recovery Manager for Automated Disaster Recovery
BC4440	Increase consolidation ratios in vSphere environments while simplifying and improving backup and recovery
BC4580	Why you should be using Virtual Backup Appliances for Business Continuity & Disaster Recovery?
BC4840	Exploring vStorage API’s and how Veeam and Dell partner for Success
BC5380	How to bring V-Cloud & Cloud Computing to Backup & Recovery

Business Workshops

BW4740	vSphere – Evangelizing the Value Proposition
BW4741	VMware View – Evangelizing the Value Proposition
BW4742	Realizing the Return on Investment In VMware
BW4743	Planning Your Virtualization Journey

Desktop Virtualization

DV1392	Lead Practices for Integrating VMware View with Active Directory
DV1406	Integrating VMware View with your VPN (online only)*
DV1460	Leveraging Vmware View and Cisco WAAS in a ROBO deployment
DV1666	VMware View Reference Architecture
DV1667	Norton Healthcare Desktop
DV1788	The 4 C’s of Desktop Virtualization for Healthcare: Costs, Clients, Continuity, and Compliance
DV1790	Understanding TCO & ROI for VMware View (next gen VDI)
DV1961	VMware Workstation and Player Technology Preview
DV2000	VMware View Client Virtualization Platform and its Use Cases
DV2181	Leveraging SRM with VMware View – Lessons Learned
DV2223	Printing Considerations in a View Architecture
DV2243	The View Test – Running a POC
DV2363	Next Generation Client Virtualization – Technology Deep Dive
DV2380	Getting the Most from View Composer – Tips, Tricks, and Best Practices
DV2387	Scripting within VMware ThinApp
DV2439	Breaking Down Desktop Virtualization Alternatives
DV2461	Virtualization on mobile phones? Why do I need that?
DV2478	Application Troubleshooting in VMware ThinApp
DV2484	Server and Storage Sizing for VMware View
DV2485	Choosing the Right Virtual Desktop Infrastructure Platform
DV2507	VMware View – Upcoming Attractions
DV2546	Case Study: Making the Grade with VDI and K12 Education
DV2626	VMware View Security Architecture
DV2672	Cerner Millenium deployed in a VMware View environment
DV2697	VMware View – Remote Display Experiance
DV2705	ThinApp Packaging & Deployment
DV2714	Optimizing VDI Storage with VMware View: Strategies for Success
DV2734	VMware ThinApp â?? Integration with VMware View (online only)*
DV2782	Application and Desktop Virtualization
DV2785	ThinApp Reference Architecture
DV2787	VMware ThinApp – A Review of Best Practices and Design Considerations (online only)*
DV2801	Integrating VMware View into your environment
DV2861	The Journey to a Dependable VDI Solution
DV2941	Operationalizing Desktop Virtualization
DV2943	VDI to the edge, deploying virtual desktops to remote and branch offices
DV2981	What’s new with ThinApp
DV3081	From Strategy to Reality – VDI lessons learned at Embarq
DV3200	Creating a VMware View Design for the Enterprise
DV3243	ViewPlanner: An Automated Measurement Infrastructure for Large-Scale VDI Deployments
DV3258	From NT4.0 to Virtualized Desktops and Applications
DV3260	Comprehensive Performance Analysis of Remote Display Protocols in VDI Environments
DV3266	Efficient VDI Design – The Math behind VDI
DV3292	The Effective use of VMware ThinApp in Higher Education
DV3300	Switching to the Mac with VMware Fusion: Lessons Learned & Deployment Tips
DV3458	Better Software Development and Testing using VMware Workstation
DV3498	VMware View Performance and Scalability on Cisco UCS
DV3532	Virtual Desktop Security
DV3567	Don’t throw that PC! How to convert old PCs to Thin Clients using a thin Linux OS and VMware View
DV3660	VMware Fusion – Present and Future
DV3746	How to Optimize VMware View ROI with Thin Computing
DV4243	How to make BYOPC (Bring Your Own PC) a Reality?
DV4381	An Insider’s View of Mobile Phone Virtualization
DV4521	From Servers to Desktops Virtualization in a "Next Gen Data Center" for Telecommunications Companies
DV5080	Desktop Virtualization / Cloud Computing: Using the Vendor Ecosystem to Drive Maximum Success

Enterprise Applications

EA1396	SQL, Exchange and SharePoint Deployments on EMC Celerra Unified Storage (online only)*
EA1480	Best Practices for Java and J2EE Applications on VMware ESX
EA1640	SAP Scaling with VMware vSphere and IBM x3850 M2
EA1820	Virtualizing Critical Healthcare Applications
EA1941	Accelerate the Enterprise Java Application Lifecycle with SpringSource
EA2342	Oracle Database Virtualization Strategies
EA2436	Efficient and High Performance Virtualization of Oracle Database Environments using vSphere
EA2442	Software Licensing in the Virtual Enterprise: Current Problems and Future Trends
EA2526	SharePoint vSphere Best Practices
EA2583	HPC/Grid Computing and Virtualization
EA2631	Virtualizing Exchange 2007 on vSphere 4 – Technical Considerations and Customer Success Story
EA2649	Virtualizing IBM Lotus Domino and Sametime: Planning to Successful Deployment at Whirlpool
EA2652	IBM DB2 on VMware: The Most Cost-Efficient and Dynamic Database Platform
EA3104	Enterprise Application Performance and Scalability on vSphere
EA3196	Virtualizing Blackberry Enterprise Servers (BES) on VMware vSphere 4
EA3216	Best Practice for Virtualizing Active Directory using vSphere (online only)*
EA3234	Virtualizing SQL Server in a VMware vSphere environment
EA3241	Beyond Infrastructure as a Service: Developer and Runtime Services with VMware and our Partners
EA3347	Analyzing Application Stack Performance using AppSpeed
EA3376	Virtualizing Citrix XenApp using vSphere
EA3389	US Navy Marine Corps virtualizes 700,000+ Microsoft Exchange users to Achieve Mission Critical Reliability
EA3408	MySQL on VMware: Performance and Deployment Best Practices
EA3448	SAP on VMware: Advanced Networking for High Performance
EA3481	Virtualization of Analytic Databases
EA3582	Developing and Deploying Virtualized Real Time Communications within the Enterprise
EA3579	Oracle Enterprise Workloads on VMware How-To (online only)*
EA3605	Virtualizing Tier 1 Applications: The Value of the vSphere Internal Cloud as a Better Platform for Apps
EA3606	Database Consolidation on vSphere (SQL and Oracle)
EA3820	The Spheres – Using WebSphere with vSphere
EA3940	Cerner Millennium Scalability when deployed on VMware vSphere and Nehalem
EA4403	VMware IT: Exchange 2007 at VMware
EA4404	Oracle E-Business Suite: Virtually Powering VMware
EA4760	Building Cloud Ready Applications

Technology and Architecture

TA1394	vSphere 4 Advanced Storage Log Analysis
TA1440	Networking in ESX: VM DirectPath Dynamic – the road to Direct VM to Hardware (online only)*
TA1523	The Path to vSphere Unleashed
TA1541	Cool little things marketing did not tell you about vSphere 4.0
TA1670	Overview of VMware vSphere
TA1962	How and Why we Upgraded Herning Kommune’s Production Environment to vSphere 4.0 at GA
TA2021	Linux Virtual Machines: Templates, P2V and Other Tips and Tricks
TA2103	Virtual Networking with vSphere 4.0 – What’s New
TA2105	Virtual Networking Concepts and Best Practices
TA2222	ROBO – ESX implementation at Kroger Store Systems
TA2231	Securing Virtual Environments – Before and After VMsafe
TA2254	vStorage – Storage Integration for Cloud OS
TA2259	Ask the Experts – Virtualization Design
TA2262	vSphere Enterprise Stability – It’s all in the Design
TA2266	Designing the Next Generation Data Center with Unified Computing & VN-Link
TA2381	Warehouse In A Box
TA2384	Deploying Cisco Nexus 1000V in a VMware vSphere Environment
TA2400	Hypervisor Competitive Differences: What the Vendors Aren’t Telling You
TA2405	Nielsen Company Leverages VMsafe Virtual Firewalls from Altor Networks for Unprecedented Security
TA2467	Best Practices to Increase Availability and Throughput for the Future of VMware
TA2509	Storage Best Practices for Scaling Virtualization Deployments
TA2525	VMware vSphere 4 Networking Deep Dive
TA2543	VMware’s Secure Software Development Lifecycle
TA2544	A Comprehensive Look at the Security and Compliance of vSphere 4
TA2623	Enhanced Storage VMotion in vSphere 4
TA2627	Understanding "Host" and "Guest" Memory Usage and Other Memory Management Concepts
TA2646	Creative Solutions: How Florida Hospital virtualized AIX and Mastered SAN Replication for DR
TA2650	Take PowerCLI to the Next Level
TA2682	Achieving 10+ Gbps File Transfer Throughput Using Virtualization – End-User Case Study
TA2689	Controlling the Storage Impact of Virtual Server Sprawl
TA2699	Measuring Virtualization Platforms and Trends Using VMmark (online only)*
TA2713	Safe At Any Speed with VMware DRS & DPM
TA2731	Tips for Planning and Upgrading to vSphere 4
TA2732	Enterprise Approach to Deploying vSphere
TA2942	Performance Best Practices
TA2955	Sizing Tools for Storage in vSphere and VMware View Environments
TA2963	ESXtop for Advanced Users
TA3045	Implementing VMware vSphere 4.0 with HP Virtual Connect Flex-10 technology
TA3105	Long Distance VMotion
TA3195	Stateless ESXi: Scalable Rollout and Management of Virtualized Hosts
TA3212	Virtualizing Resource Intensive Applications, a Case Study with SAP
TA3220	VMware vStorage VMFS-3: Architectural Advances since ESX 3.0
TA3261	Using VMware to Conduct Digital Forensics and Criminal Investigations (online only)*
TA3264	iSCSI Scalability and Storage Performance Enhancements in vSphere 4.0
TA3286	Applications in the Cloud: Getting off the ground
TA3302	Security Considerations When Building Virtual Infrastructures Across Security Zones
TA3324	Performance Troubleshooting in Virtual Infrastructure
TA3326	Building an Internal Cloud-the Journey and the Details
TA3353	Building the TVE a Collaboration Story
TA3402	How Dell Uses Virtualization: Paving the Path to Large-scale Distributed Computing
TA3406	What is New for Storage in vSphere 4.0
TA3438	Top 10 Performance Features of VMware vSphere 4
TA3461	Tech Preview: IO DRS – Providing Performance Isolation to VMs in Shared Storage Environments
TA3488	All Hypervisors Are Not Created Equal – The Unique Advantages of VMware ESX (online only)*
TA3521	vNetwork Monitoring, Troubleshooting, Security, and Management (with Live Demo)
TA3555	Getting to over 90% Virtualization
TA3557	Using NFS Storage Infrastructure for vSphere
TA3576	Early vSphere Deployment Stories
TA3603	Getting The Most Out Of VMotion: EVC, Performance Tuning, and Troubleshooting
TA3880	Head-To-Head Comparison: VMware vSphere and ESX vs. Hyper-V and XenServer
TA3882	The Cloud- What is it and why should I care
TA3901	Security and the Cloud
TA4020	Beyond the Data Center – Virtualizing your remote and distributed IT workloads
TA4060	The Path to COS-less ESX: Migrating Server Operations from ESX to ESXi
TA4100	Internal Clouds: Customer perspective and implementations
TA4101	Buying the Cloud: Customer perspective and considerations on what to send to an external cloud
TA4102	Unveiling New Cloud Technologies
TA4103	Engineering the Cloud-The Future of Cloud
TA4160	What is new in the vSphere SDKs and APIs
TA4242	Securing the Virtual Data Center in Enterprises and Clouds
TA4341	Virtual Network Performance
TA4720	Virtualization and Cloud Computing with AMD Opteron™ Processor-based Platforms
TA4820	What Keeps Clouds Up?
TA4860	Deploying the Foundation for a Virtualized Dynamic Data Center
TA4881	Designing Dynamic Data Centers with NetApp and VMware
TA4901	IBM System x Delivers Innovation for Virtualization Solutions for Today and Tomorrow
TA4902	IBM’s Cloud Computing Solutions
TA4940	Navigating the Cloud: IT Management Challenges and Opportunities
TA4941	I/O Clouds in the Virtualized Data Center
TA4960	Virtualizing Email Security â?? The St Lawrence College Story (online only)*
TA5161	How Expedia uses Storage VMotion to Allow Datacenter to Take Flight
TA5360	Enabling the Virtual Data Center with Symmetrix V-Max
TA5400	The Soft Underbelly of Bare Metal – Real World Security Lessons from the Datacenter to the Cloud
TA5460	Optimize Your Storage: A Blueprint for Storage Resource Management in VMware Environments
TA5461	Securing the Cloud
TA5540	The Evolution of Open Source Virtualization
TA5600	Virtualization: How Cisco IT Leverages Operational Excellence to Drive Innovation

Virtualization 101

V11721	Best Practices for Successful VI Design
V12226	Building a High Availability and Disaster Recovery Solution with VMware
V12644	Designing a Virtualization Infrastructure for the Small Environment
V12789	VMware vCenter Converter 101 (online only)*
V13100	The VMware Competitive Advantage – A Comparison of Server Virtualization Offerings
V13226	Getting Started with Virtualization Using VMware ESXi
V13227	Introduction to Virtualization
V13229	Introduction to VMware vSphere
V13395	Getting to Yes! Keys to Launching a Successful Data Center Virtualization Program
V13478	Executing Enterprise Virtualization – Continuing Case Study with USMC
V13496	VMware vSphere and VI Best Practices – Tips and Tricks (online only)*
V13760	Overcoming the Hidden Challenges of Virtualization

Virtualization Management

VM1401	Using vCenter Lab Manager and View for the Education Cloud
VM1461	Avoiding the hurdles in scaling Lab Manager
VM1682	vCenter Database Architecture: Technical Deep Dive
VM1700	vApps and advanced VM templates in vSphere 4
VM1724	Technical Preview: CapacityIQ
VM1882	VMware Update Manager 4 Performance and Best Practices
VM1900	Accelerating Large Scale Migration – Best Practices, Tools, and Processes to Address Challenges
VM1960	Datacenter Consolidation and Migration with VMware Site Recovery Manager
VM2100	vSphere Orchestration for SAP’s Demo Business
VM2120	Assessment methodology for LCM customization with Orchestrator
VM2164	Virtualization and Compliance: The Auditor’s Perspective
VM2166	VMware’s New Licensing Model & Compliance Best Practices
VM2201	Turfwars! : The Future Staffing of Virtualized Environments
VM2241	Managing vSphere with VMware PowerCLI
VM2265	Achieving Measurable Advantages Through Better Virtual Systems Management
VM2280	vCenter Mobile Access – Managing Virtual Infrastructure From Your Phone
VM2408	Tech Preview: VMware vCenter ConfigControl
VM2409	Virtual Center: Troubleshooting Unleashed
VM2440	Deploying vSphere Mgmt Automation: Dependency Mapping, Root Cause, Configuration & Compliance
VM2472	Introduction to VMware vCenter Chargeback
VM2506	Automating the Virtual Data Center
VM2511	Automate Routine Actions and Enforce process in Virtual Environments (online only)*
VM2630	Using Unnoc to Perform Monitoring of vCenter, ESX Servers, and Virtual Guests
VM2643	Infrastructure Performance Panel
VM2648	Managing Compliance in Virtual Environments
VM2657	Best Practices for deploying VMware vCenter Lifecycle Manager.
VM2674	VMware vCenter Server Heartbeat Best Practices
VM2684	What’s New in vCenter Server 4
VM2700	Lab Manager Best Practices
VM2706	Improve Cloud Interoperability Using Virtualization Management Standards
VM2711	VI Performance Optimization: From the Datacenter to the Desktop (online only)*
VM2717	What’s New In Lab Manager 4?
VM2724	How to Prevent Headaches in your Virtual Environment
VM2764	From Virtualization, Automation to From Virtualization, Automation to Cloud Managemen
VM2766	Integrated and Application Centric Deployment Approach to Achieve Wider Virtualization Adoption
VM2847	Upgrading to vSphere – Things to consider
VM3021	VMware’s Tooling of VI Operations Best Practices
VM3041	Integrating Virtualization with Capacity Management
VM3085	Top 10 Reasons vCenter is the Best Platform for Virtualization Management
VM3103	How VMware Reduces Cost-per-Application and OpEx Costs
VM3120	How does VMware fit in with your management processes? VMware Integration with Big 4 + Microsoft
VM3142	Large Scale Virtualization
VM3217	vCenter Databases: Setup, Management and Best Practice (online only)*
VM3235	Introducing VMware vCenter Product Family: Managing Service Levels Across Dynamic IT Infrastructure
VM3237	VC Linked Mode in vSphere 4.0 (online only)*
VM3263	Sustainable Change Management Processes in a Virtual World
VM3325	vSphere and ESXi Log Files 101 & 102 (online only)*
VM3352	Operational Savings Achieved through Virtualization
VM3360	Assuring Service Levels for P2V migrations with AppSpeed
VM3404	Alarms for vCenter 4.0 (online only)*
VM3405	A Spotlight on vSphere Storage Management: Today and Tomorrow
VM3414	Study reveals production best practices for virtualization (online only)*
VM3427	VMware Administration for The Average Administrator
VM3433	Technical Deep Dive: VMware Host Profiles
VM3463	Monitoring Hardware Health with vCenter 4
VM3470	Seven Steps To Understanding And Mitigating Virtualization Security Risks
VM3483	Managing Application Performance with vCenter AppSpeed
VM3528	Does Virtualization Change The Way We Secure IT Environments?
VM3566	Best Practices for Managing and Monitoring Storage in vSphere
VM3580	Capacity Planning and RoI – Before, During & After the Project
VM3607	Before Chargeback, I Need To Know What It Costs
VM3609	Assessment Best Practices for VMware Partners
VM3620	Incident Handling in a Virtualized Data Center
VM3881	Business Objects SAP Virtual Infrastructure Lab Mgr Deployment and Migrating Between Network Ranges
VM3903	Practical Virtual Systems Management – Climbing The Hierarchy of Needs
VM4000	Operationalizing Software Update Management in a Growing Virtual Environment
VM4061	How to Generate Reports out of the vCenter Database
VM4120	Introduction to VMware vCenter Lifecycle Manager
VM4261	Ensuring Database Performance using VMware AppSpeed
VM4320	Vision for virtualization-aware IT management
VM4380	Tech Preview: vCenter Server
VM4520	Analyze to Automate – The Evolution of Vizioncore and How We Truly Extend VMware’s vSphere
VM4800	The "Next Generation Data Center" for Telecommunication Companies
VM4900	Building a Service-Driven Data Center: Making an Internal Cloud A Reality
VM5002	How to Gain Sponsorship and Successfully Enable Virtualization Transformation with VMware
VM5040	Amplify the Economic Benefits of Virtualization
VM5361	An Introduction to VMware vCenter Orchestrator
VM5401	Conquering Costs and Complexity in a Virtualized Environment: Research and Case Studies
VM5420	Using Lab Manager in a Regulated Healthcare Environment
VM5521	Unified Computing System – Infrastructure Management and Provisioning
VM5560	Data Center Optimization through Virtualization: Compute More, Consume Less

Instructor-Led Labs (PDFs only)

LAB01	VMware vSphere 4 – New Features, Best of, Advanced Topics
LAB02	VMware vCenter Site Recovery Manager – Advanced Operations
LAB03	VMware View Advanced Config & Troubleshooting
LAB04	VMware vSphere 4 – Performance Optimization & Troubleshooting
LAB05	VMware vSphere 4 – Security Hardening & Best Practices (vShield Zones)
LAB06	VMware vCenter Server Heartbeat
LAB07	VMware vCenter Lab Manager 4
LAB08	VMware vCenter CapacityIQ
LAB09	VMware vCenter AppSpeed
LAB10	VMware vCenter Lifecycle Manager
LAB11	VMware vCenter Chargeback
LAB12	Scripting VMware Infrastructure (PowerShell/Perl Toolkits)

Self-Paced Labs (PDFs only)

SPL13	VMware® Infrastructure 3 to VMware vSphere™ 4 Upgrade
SPL14	VMware vSphere™ 4 Base Install
SPL15	VMware View Base Install & Config
SPL16	VMware vCenter Site Recovery Manager (SRM) Basic Install & Config
SPL17	VMware vCenter Converter
SPL18	VMware vCenter Data Recovery
SPL19	VMware® ThinApp™
SPL20	VMware® Fusion™
SPL21	VMware View Experience
SPL22	VMware vCenter Orchestrator
SPL23	VMware vSphere™ 4 Virtual Networking Fundamentals

I wasn’t able to to go VMworld US in person this year because my wife and I were expecting a baby at that time, but luckily I received a VMworld subscription from VMware as a benefit of the vExpert programme; had I not I would have probably shelled out of my own pocket for one.

Justifying the spend

Whilst we seem to be slowly emerging from the economic apocalypse of the last 18 months it’s still very hard to get sign-off to attend such events in person and too many org’s treat VMworld/Tech-Ed as marketing type events – unless you are a vendor with a stand this couldn’t be further from the truth – these conferences are primarily technical training boot-camp camps, with some networking and general trade show features thrown in. However, they are what you make of them – the onus is on you to hunt down the sessions/track or people you are interested in – nobody drives your schedule but you – non self-starters need not apply.

I use the following analogy – which applies equally to Microsoft Tech-Ed and VMworld (..and I’m sure Oracle World, Apple World, etc.)

A typical 5-day technical training course on an individual product (Exchange, ESX, Windows 2008) in the UK will cost in the region of £1,500-3,000 GBP and those 5 days will be slow-paced (9.30 –> 4.30pm affairs). The course content and material has to cater to the lowest common denominator delegate, for a geek/experienced tech this can make for frustratingly slow progress and means you only cover a very narrow technical focus, or broad high-level overview – you can’t easily dive in and out of the bits that are relevant to you with a traditionally delivered course and even the best instructor in the world can’t dedicate that much time to you in a classroom environment.

So compare that training course is £2-3000 + travel + accommodation + time out of the office to VMworld (for example..), even at the most expensive register on-site on the day prices

• VMWorld Full Conference Pass* 1,260 EUR (£1,176 GBP at current exchange rate) (Tech-Ed 5 days c.£2000 full price* ticket)
• Travel (airfare from most of continental Europe, economy/flexible flight) c.£400**
• Hotel (normal business hotel, 4 nights) £900**

+Access to on-demand streamed and downloaded content following the conference (access allowed until the next VMworld) included

+Lunch/breakfast usually included

+Networking opportunities, access to product teams and managers included

+trade show with relevant vendors/suppliers included

+bag and pen included (ok, I’m struggling with that one! )

note:

*Early registration attracts a large discount on the full conference pass – look for “early bird” tickets which can knock a significant percentage off the full price

**If you are prepared to “slum” it with budget airlines and hotels this is significantly cheaper.

Prices for reference:

VMworld Europe 2009 prices

Tech-Ed Europe 2009 Prices

With Microsoft Tech-Ed they usually give a complimentary Technet Direct subscription – which is worth hundreds of pounds on it’s own and gives you multiple copies of almost every Microsoft product for your own use.

So if you look at it pragmatically – VMworld/Tech-Ed give you the flexibility to tailor your content to what is important to you; as well as the ability to take all the information away with you to review online post-conference (even for the sessions  you didn’t make in person)

With a training course you walk away with a nice certificate, some spiral bound manuals and if you are lucky – a pen

And they both come out to roughly the same price.

I’m not saying this is for everyone – you need to be a self-starter to make the most of these conferences, and if you do a limited scope day-job and that is all you are interested in doing, traditional training courses are probably your best bet but for those of us that work as consultants or want to broaden our horizons – go for it!

My write-ups of previous VMWorld and Tech-Ed events can be found at the following links:

Tech-Ed EMEA 2008

http://vinf.net/2008/10/31/off-to-microsoft-teched-emea-2008/

http://vinf.net/2008/11/04/teched-emea-2008-it-pro-day-1/

http://vinf.net/2008/11/04/teched-emea-2008-it-pro-day-2/

http://vinf.net/2008/11/06/teched-emea-2008-it-pro-day-3/

http://vinf.net/2008/11/06/teched-emea-2008-it-pro-day-4/

http://vinf.net/2008/11/07/teched-emea-2008-it-pro-day-5/

VMworld Europe 2009

http://vinf.net/2009/02/23/vmworld-partner-day-keynote/

http://vinf.net/2009/02/24/vmworld-partner-day-wrap-up/

http://vinf.net/2009/02/24/vmworld-europe-day-1-keynote/

http://vinf.net/2009/02/24/dc02-best-practices-for-lab-manager-vmworld-europe-2009/

http://vinf.net/2009/02/24/dc14-overview-of-2009-vmware-datacenter-products-vmworld-europe-2009/

http://vinf.net/2009/02/24/vmworld-europe-day-1-wrap-up/

http://vinf.net/2009/02/24/vmware-client-hypervisor-cvp-grid-application-thoughts/

http://vinf.net/2009/02/25/vmworld-europe-day-2-keynote/

http://vinf.net/2009/02/25/how-vmware-it-use-vmware-internally/

http://vinf.net/2009/02/25/vmworld-europe-day-2-wrap-up-a-good-day-despite-the-curious-lack-of-forks/

http://vinf.net/2009/02/26/hands-on-lab-01-vsphere-features-overview/

http://vinf.net/2009/02/26/hands-on-lab-12-cisco-nexus-1000v-switch/

Judging by the several hundreds of hits that this post on my blog gets every day since January I would say there is a *serious* demand from the general internet community to use laptops with the Intel 855 family video chipset with Windows 7, and it’s not even been released to the general public until now!

Whilst this may be an “older” chipset; in reality those laptops aren’t beyond serviceable life – and if anything the performance increases in Win7 will make them more usable and extend their life a bit further.

Nobody realistically expects fantastic 3D/Aero graphics from this combo, but a driver that supports the panel’s native resolution would be more than adequate for browsing/word processing etc. no doubt these ex-corporate laptops are being cycled around family members/students for some time to come rather than forcing people to squint at standard VGA.

There are ways to get the Vista driver bodged into Win7 (see this post and it’s comments), but it’s far from ideal or stable

Please Intel/Microsoft – surely you must be able to produce a basic/compatible driver by today! otherwise I’m sure some of the Windows launch parties could be something of a disappointment!

if it goes by way of evidence – this is the number of hits my original blog post about this problem has had in the last 10 months, it will be interesting to see what it does from launch day onwards!

I must be a bit behind the times, but I have just stumbled across the beta version of a blackberry app that let’s you publish and manage your blog directly from your phone, there is also an iPhone version

Very cool, you can now all look forward to accidental posts from my pocket and I can look forward to RSI from my blackberry Pearl

If you are interested you can download it OTA from httphere

Incidentally it also supports the builtin camera, and this post was typed on on a blackberry 8120 pearl.

Can’t see a way of viewing stats yet, which would be a nice addition.

Following on from my last post on problems entering maintenance mode with FT-enabled VMs, I seem to have found another one – if you have the rather excellent AppSpeed product deployed on an ESX cluster and you want to put a host into maintenance mode it gets stuck at 2% as it can’t move the AppSpeed probe VM onto an alternative host

If you try to manually vMotion the problematic probe off to another host in the cluster you get the following error

If you shutdown or suspend the AppSpeed probe VM then the switch to maintenance mode continues as expected.

This would make sense as it plugs directly into a dedicated vSwitch on that host to monitor network traffic so vMotioning it off wouldn’t be of any use – assuming the other nodes in the cluster are also running AppSpeed probes.

However it would be great if there was a more automated way to handle this? guess it’s tricky as on one hand its great that AppSpeed doesn’t rely on any ESX-host agents and is essentially self-contained with probes running as VM appliances but on the other hand the probe doesn’t know the guest is being put into maintenance mode so should be shut down/suspended rather than vMotioned to an alternative host.

There is integration with the vCenter server via a plug-in so maybe in future versions that could trap a maintenance mode event and initiate (or suggest) shutting down the AppSpeed probes.

I have a 2 node vSphere cluster running on a pair of ML115g5 servers (cheap ESX nodes, FT compatible) and I was trying to put one into maintenance mode so I could update its host profile, however it got stuck at 2% entering maintenance mode, it appeared to vMotion off the VMs it was running as expected but never passed the 2% mark.

After some investigation I noticed there were a pair of virtual machines still running on this host with FT enabled – the secondary was running on the other server ML115-1 (i.e not the one I wanted to switch to maintenance mode)

I was unable to use vMotion so that the primary and secondary VMs were temporarily running on the same ESX host (and that doesn’t make much sense anyway)

That makes sense, the client doesn’t let you deliberately do something to that host that would break the FT protection as there would be no node to run the secondary copy. incidentally this is good UI design – you have to opt-in to break something – so you just have to temporarily disable FT and should be able to proceed.

If I had a 3rd node in this cluster there wouldn’t be a problem as it would vMotion the secondary (or primary) to an alternative node automatically (shown below is how to do this manually)

However in my case all of the options to disable/turn-off FT were greyed out and you would appear to be stuck and unable to progress.

the fix is pretty simple and you just need to cancel the maintenance mode job by right-clicking in the recent tasks pane and choosing cancel, which then re-enables the menu options and allows you to proceed. Then turn-off (not disable – that doesn’t work) fault tolerance for the problematic virtual machines

The virtual machine now doesn’t have FT turned on, if you just disable FT it doesn’t resolve this problem as it leaves the secondary VM in-situ, you need to turn it off.

So, moral of the story is – if you’re stuck at 2% look for virtual machines that can’t be vMotioned off the host – if you want to use FT – a 3rd node would be a good idea to keep the VM FT’d during individual host maintenance; this is a lab environment rather than an enterprise grade production system but you could envision some 2-node clusters for some SMB users – worth bearing in mind if you work in that space.

If you’re in the UK and are interested in Active Directory – Brian Desmond (an MVP for AD) will be giving a talk on Active Directory design on 29th October in London.

Details here on the ADUG website here and – registration is free.

I’ve been working with Active Directory for a long time but it’s always refreshing to have some Q&A with industry peers, so if you can make it it promises to be an interesting evening.

Not sure if it will be recorded/streamed for people that can’t make it – if not can I make that suggestion to the ADUG team- I can bring some recording equipment if you need it.

My home lab has a pair of HP ML110 servers with 8Gb of RAM running vSphere 4 (more info here) it’s configured in a cluster with iSCSI storage running from an old HP D530 PC with a 1Tb hard disk running OpenFiler. it performs pretty well and meets most of my needs, I thought I’d do a quick couple of screenshots of the average performance I have seen on it over the last 3 months.

it’s running a constant load of about 17 mostly Windows virtual machines and a varying load of test environments which are suspended to disk – think the most I have ever had running on the 2-node cluster at one time was about 45 VMs and performance was ok – trying to use VUM to patch all those VMs at the same time killed things though, as all the VMs are running from a single 1Tb SATA disk over OpenFiler.

This is a list of all the VMs, you can create your own html list as follows, or you can also save it as a CSV to import into Excel to manipulate.

the following screenshots show the last 3 months of performance stats from vCenter as the number of VMs has increased and decreased as I’ve provisioned and removed VMs for testing.

Overall CPU usage for the cluster

vMotion and VM Reconfiguration activities

Cluster Memory consumption

The new overview page feature can show you a quick summary of virtual machine performance

Drilling down into the performance tab on each host gives more information on specific performance like disk and network

You can also produce a stacked graph showing guest CPU usage of each VM on a host

or identify which VMs have the busiest virtual disks

You can also view a stacked (per VM) graph showing on a per-host basis how much physical RAM the guests are consuming, relative to each other over time.