Krystaltek

My Own Private 'Cloud' - Home Lab

2012-01-10T17:40:00.001+11:00

Last year back at vForum I traded lab stories with Alastair Cooke (@demitassenz) over a beer or two. He motioned that I should explain what I’ve compiled and how it helps my work. He beat me to the line by about 3 months with his post http://www.demitasse.co.nz/wordpress2/2011/12/whats-in-my-vcp5-study-lab/, but better late than never.

Truth be known I use various lab’s, My home lab and mobile lab’s I have on my laptops. The last few years I have collected a few bits and pieces which seem to give me a decent start. There are quite a few other bits and pieces I’d like but I also like my marriage just the way it is!!

When I moved house about a year ago I decided to do it all tidy like. I hated having computers on the ground in my office when there was other space. I decided to purchase a nice 27RU server rack all enclosed, with sides, doors, power rail, shelves, fans etc. It lives in the garage . It cost me $400 and was well worth it.

From the top down, I use a Cisco SLM2024 24x 1gb network switch http://www.cisco.com/en/US/products/ps9997/index.html. It supports pretty much everything I need in a layer 2 switch. I would have preferred other layer 3 or IOS models but the price was about $250 or something from memory and I couldn’t pass it up.

I have a QNAP 410-TS for my NAS stacked with 4 x 1tb SATA drives. Aside from general SMB sharing, Time Machine backups, Twonky Media Server for XBMC, I use it to serve 2 main NFS mounts for my ‘production’ vSphere 5 lab. It also presents a few smaller iSCSI LUN’s to vSphere for some VMFS based stuff, when required.

I have 2 x Quad Core AMD X4 machines. Each of these machines has 16gb RAM , 4 x 1gb Intel nics and boots vSphere 5 from USB. In all honesty the second host rarely gets powered on. It was setup for DPM but often I use one of the handful of USB keys I have which are different vSphere versions. When I need to do some previous version (or later) testing for an extended period of time I use this box. I have many other VM’s living on the NAS which I can register on this host. The cost of these machines were around $450 each from memory and most of that was RAM. They are ridiculously cheap and a hell of a lot easier than a rackmount server.

On the primary host I run a Windows based vCenter VM which is the lifeblood of the lab (OK I installed some other stuff on it too). I have a collection of other bits and pieces I run:

Windows 20008 R2 Domain Controller
vMA 4, 4.1, 5
View 5 and Win7 guest
vCD 1.5
vCloud Connector
vShield
vDR 2
vCenter Operations
vSphere Replication Management Server
vCenter Mobile Access
A mixture of Windows 2000, 2003, Centos, Redhat guests which are always required for testing stuff.
A virtual ESX host for pretty much every ESX and ESXi version back to 3.5
A few other vendor VM’s (NetApp, Splunk etc)

The final addition to the lab came just before Christmas. I now have an EMC VNXe 3300 SAN which allows me to present storage to the ESX environment via iSCSI. I am most interested in the integration bettwen vSphere/vCenter and the array. I have quite had the time to really put it through its paces yet but it’s installed, cabled and we have little lights working.

NOTE: The second host is out for repairs right now

My password is complex, therefore secure.

2012-01-08T20:12:00.000+11:00

For the beginning of the year I thought I would take some time to update all my passwords and given that I am also starting a new job I think my timing was pretty good.

I have used a series of passwords for a few years now and despite my attempts to make each one unique I also am required to be somewhat human at times and I found myself re-using them, especially when creating various accounts on the fly. Fortunately, I have generally used something like password safe or keepass to save those passwords, It’s my get out of jail free card when my memory fails.

Password security is something I and everyone these days should treat seriously. When it came to creating some new passwords I started considering what most people consider ‘complex and secure’. Certainly most password generators will confuse you with extremely difficult to remember characters from the various character sets. This I find problematic but also a bit of a mis-nomer. Most of the time when people hear complex they think difficult, not really true.

There are a few ways baddies try to crack passwords but the majority of the time bruteforce attacks are the norm. These attacks walk up and down the character sets for each character in the password until they find matches. With sophisticated tools this becomes somewhat trivial for them. In fact the latest bruteforce standard Class F allows for 1 billion iterations per second.

In an effort to explain this a bit further let’s examine what would happen when you choose a 6 character password with different characters.. say abCd4! The tool simply needs to start at the first character and walk up and down the a-z, A-Z, 0-9, !-= and other printable character sets, incrementing one at a time and it will nail you password in a matter of hours. Unless you change your password every few hours you’ll be in trouble. Yes increasing the password character limit to 8 or so will increase the time to crack but only marginally. Hands up those who actually use a password of 14+ characters.. tap..tap..tap.tap

The alternative I use and which have been around for a while now is rather than a ‘password’ using a ‘passphrase’. Why is a passphrase more secure? Well here’s the skinny. Let’s assume I choose a complex password and passphrase like the following:

Password: W3r+Kwsn!

Passphrase: we are the knights who say ni

Right now I’d like to introduce you to a cool tool developed by my friend and former securityfocus.com columnist, Microsoft Principal Security Architect, Timothy ‘Thor’ Mullen has developed http://www.hammerofgod.com/passwordcheck.aspx

Tim’s tool ‘Password Strength Checker’ illustrates the difference between the two using mathematic calculations to determine the time taken to crack the password based on the number of iterations required. Let’s enter the first password, assuming brute force class F and see how long the password takes to crack.. ‘640 days’ hmm not bad right. Well let’s compare that to the passphrase.. 23,317,943,213,061,100,000,000,000,000,000,000,000,000 years’ wowsers!!

So what’s the deal? Do not assume complex characters add an equivalent amount of security to your passwords. I don’t know about you but I’d find it a hell of a lot easier remembering the latter and probably easier to type to boot. The other great thing about passphrases is that they can be anything that makes sense to you, close to your heart or that just sticks in your head. Like ‘For those about to rock! We salute you’ or whatever you like.

For fun, guess how long it takes to crack vmw@re as a password..under 60 secs :)..so change dem default passwords.

P.S If you want greater detail on this and other common security principles I’d highly recommend Tim’s book ‘Thor’s Microsoft Security Bible’ http://www.amazon.com/Thors-Microsoft-Security-Bible-Collection/dp/1597495727

P.P.S Those aren't my paswords!

NetApp VSC 4.0 Beta

2011-12-22T09:55:00.001+11:00

NetApp has released an external beta of Virtual Storage Console version 4.0

Some major features to watch for are:

Platform support

Data ONTAP

7.3.4 + support for 7-mode
8.1.0+ support for cluster-mode
ESX & ESXi 4.0/4.1/5.0 support

Cluster-mode

Discovery of cluster-mode objects (Vservers, cluster nodes, LIFs and clusters)
Local path affinity in NFS cluster-mode environments
NFS and VMFS datastore provisioning, resizing, deletion and deduplication management (7-mode feature parity)

7-mode only

Online, GUI-based guest OS alignment (VMFS datastores only)
NFS datastore support will come with ONTAP 8.1.1

7-mode and cluster-mode

NFS VAAI plug-in installation
Data ONTAP 8.1.0 operating in cluster-mode
Data ONTAP 8.1.1 operating in 7-mode
Consolidated storage controller credentials management

Please note this version is NOT intended for production use.

For more information please read and ask questions on the discussion board

https://communities.netapp.com/groups/vsc-40-external-beta

NetApp Storage Best Practices for VMware vSphere - Technical Report (Updated)

2011-12-09T11:37:00.003+11:00

NetAppp has released version 3.0 of Technical Report 3749 (NetApp Storage Best Practices for VMware vSphere)

This version includes information for vSphere 5 around storage and networking design.

The document can be found here. http://media.netapp.com/documents/tr-3749.pdf

If you are a NetApp and VMware customer, you cannot afford to be without this document.

Well done to the team who put it together.

Pagefile separation with VMware Site Recovery Manager

2011-12-09T11:23:00.000+11:00

Yesterday at the Melbourne VMUG we ran a user roundtbale with myself, Mike Laverick and Peter Marfatia from VMware with participation from i'd suggest 30+ users.

The topic was one close to Mike's heart and the discussion was lively and informative.

I was prompted to provide the NetApp Technical Report that details the process of Pagefile separation onto non-replicated datastores and how that process might look. I couldn't for the life of me remeber the TR number and promised to look it up.

The technical report can be found here

http://media.netapp.com/documents/tr-3671.pdf

Specifically APPENDIX A: Vm's with non-replicated transient data

* NOTE: I am not promoting or preventing anyone from such a design. If you reallly want to have that dicsussion, feel free to contact me. I have been involved in many discussions wearing different hats on this topic.

New Role: VMware or Bust

2011-11-29T17:21:00.000+11:00

I guess now that the cat is out the bag maybe it’s time for me to come clean..

For the last 2 years I have been both a permanent and contract employee at Empired Ltd. A services based company with offices in Perth and Melbourne. My role at least for a year or so was to help customers bring Infrastructure projects to fruition. My main role was as a virtualisation SME however I dabbled in most things I could lend my hand to when required.

For the last 13 months of that gig I was contracted out to the NetApp PS team in Melbourne working on a large state government project, CenITex, for those who are familiar.

During my time at NetApp I worked with some of the smartest technical people you could ever hope to meet. The times were challenging for various reasons I wont elaborate, but I learnt so much from the experience. Enterprise storage can be a beast and it was really a cool piece of work to be involved in.

I made some great mates on that gig and even got to work a little closely with some of the NetApp stalwarts particularly in the virtualisation space.

I thank all those guys who helped me and those I dealt with both locally and also in the global space as well. I really felt part of the NetApp family, even though I was a contractor as are most of the PS team are in Melbourne. I also thank the NetApp in general for the opportunity to spend time in the inner sanctum of such a great company. There is a reason they are consistently voted among the best if not the best company to work for not only in Australia but worldwide.

The next step for me is one I am hugely excited about. Many years ago and a Sys Admin I inherited a VMware environment in its infancy. I was pretty hooked on the technology and it exploded from there. With each role I have specialised more and more in the technology and am just loving professional life right now. I consider myself very fortunate to have been granted the opportunity to fulfil one of my bucket list goals.

Starting in 2012 I am joining the team at VMware Melbourne, Australia in a SE role. I am super excited about joining some colleagues I have known for quite a while and working with some new fresh faces. I am also very excited and somewhat humbled to be joining the global VMware empire.

I will see out the rest of the year at NetApp and enjoy the Christmas break especially this year with my family and prepare myself for the new gig. I know my passion will enable me to throw everything I have into the role and I cant wait to get started.

Finally, Thank you to all who directly or in-directly became my referees and to those who supported me throughout this process and most of all for the opportunity. Let’s have some fun with this..

VMware Auto Deploy and NetApp Virtual Storage Console

2011-11-24T21:38:00.001+11:00

This subject came up a few days ago and I figured it was worthy of a line or two.

NetApp and other vendors have tools or plugins that are integrated into vCenter in order to apply vendor configurations to ESX hosts. Specifically the one I deal with currently is NetApp Virtual Storage Console. VSC has many other features other than host config. It also provides cloning and provisioning tasks as well as backup and restore capabilities. You can find the release notes here if you are interested

https://now.netapp.com/NOW/knowledge/docs/virtual_storage_console/relvsc211/html/software/rnote/frameset.html

I recently saw a presentation that explained that VSC 2.1.1 was in fact functional and supported with ESXi 5 hosts that were deployed with Auto Deploy. In fact in a sense that statement is true enough however let’s examine the true aspect of how such a tool works in reality.

As you know Auto Deploy is used to deploy a stateless image to an ESXi host into RAM for the duration of the boot lifetime. Auto Deploy also uses Host Profiles for post deployment configuration and is an integral part of the overall host config management of such an ESXi environment. The key take-away from the previous sentence was ‘boot lifetime’. As you may know the configuration settings of a host get lost during a reboot. This presents a problem when trying to manage settings on a host and also when configure settings that require reboots to stick. Case in point, VSC.

Let’s focus on the first AD host in the VSC console (192.168.100.213). This is one of my test AD hosts. We can see that the NFS settings are in an alert state.

I can choose to ‘show details’ which will give me a list of the defaults and the required setting changes. To apply the setting change I choose ‘set recommended values’.

NOTE: The only NFS settings that require a reboot are Net.TcpipHeapSize and Net.TcpipHeapMax. In addition, The HBA timeout settings wil require a reboot, but MPIO settings will not.

After applying the correct NFS settings as per VSC we notice that the status is now pending reboot.

When the host reboots it rebuilds based on the image profile AD is using and essentially does a full re-deploy. At this stage I am not using Host Profiles for this purpose. Let’s open the VSC console again in the vSphere client. Notice anything different? That’s right, nothing.

The host has lost any pending changes and in fact all previous config after the reboot. This poses a problem for tools such as VSC in my opinion, but what does it mean.

To me you have a choice to make, If you are going to use Auto Deploy then it’s a given that you will use Host Profiles or Pre-Built Images. The beauty of Host Profiles is you have a single point of management again and you can ensure that all hosts carry the same settings everytime once the host has joined vCenter.

It is possible in this VSC case would be to use the third option which is ‘skip host’. Skip host allows you prevent VSC from checking these hosts for config management. In this instance you could use this option for all AD hosts in your vSphere 5 environment and continue to use Host Profiles. However that is not as simple as it seems. If you reboot those hosts at all, patches etc. you will need to repeat the process again, and again.

The message I have been seeing from vendors to customers is that you should if not ‘must’ use our tools to manage your config in order for us to support you, rightly or wrongly. I think that mindset has to change, the lines are blurring and it becomes harder to achieve your goal with a single glass of pain these days in my opinion.

Given all that. I would still recommend NetApp customers use VSC. VSC does plenty more than manage configuration settings across hosts as I mentioned earlier. Ia would just ensure that customers give careful consideration to what it is they are trying to achieve and how best to reach those goals. The rest is up to you!

VMware Update Manager in a Stateless World

2011-11-05T22:15:00.002+11:00

Auto Deploy is a new addition to vSphere 5 that allows the rapid provisioning of ESXi servers to physical hardware storing the ESXi image in memory for a truly stateless server deployment. It uses a PXE boot mechanism and PowerCLI to create image profiles to be deployed as the ESXi build state. The post configuration is handled by Host Profiles which you attach to the Image.

At the Sydney vForum a few weeks back we saw a demo which showed the rapid deployment of 10 hosts achieved in a matter of minutes. I have done this in my lab and find it to be powerful and fast. (Granted there was a slight hiccup with the demo, but it was nothing to do with the product. Turns out you have to power on the hosts first).

I know of a few customers who are already looking to use AutoDeploy as the method for ESXi stateless install. That’s neither here nor there for the purposes of this, but I do wonder what implications that decision has for their patch management strategies.

I dicussed this topic with a few people at vForum and we all came to the conclusion that the use of VMware Update Manager would not make sense. In the event that a patch or newer build was released it would make more sense to roll the update into a new Auto Deploy image and stage the redeployment of your hosts. Essentially this would require a reboot and a reinstall. You might manage this even cluster wide with a PowerCLI script. To patch ESXi hosts a reboot is generally required in any case. It would mean that your end-to-end deployment process would have to work flawlessly and be tested and re-tested frequently but I see that as just part of good operational procedure. I see less overhead in re-deploying hosts then in patching them to be honest.

So what's left for Update Manager? Of course we could still use it for VMware Tools upgrades and the like but what if Update Manager could become the tool with which you created Auto Deploy images. Update Manager already has the capability to deal with and create baselines so as a future extension I’d love to be able to manage the creation and deployment of images to hosts via the Update Manager interaface. To me it make sense to use only one interface to do this. Taking one step further, possibly the Update Manager UI could be incorporate the creation of Host Profiles in a similar fashion. If that was the case, would we still call it Update Manager? I’d assume it would go through a rename if it were to encompass the host installation and configuration features but I have spoken to a few colleagues on this and most of we agree that it would make sense. Let's wait and see how these tools develop.

If you have started using Auto Deploy as yet or are planning to do so I’d love to know what your plans are.

Melbourne VMUG - Registration Open - December 8 2011

2011-10-25T20:11:00.003+11:00

The end of the year is fast approaching, need I remind anyone? After wrapping up a successful vForum in Sydney last week I thought that would probably see the year out for VMware events.

Enter the December Melbourne VMUG. This VMUG will be unlike any other we have experienced in Melbourne. The event will be a full day event and we are incredibly lucky to have such distinguished international guests as Scott Lowe, Scott Drummonds from EMC and Mike Laverick from rtfm-ed.co.uk

The day will involve technical sessions, partner exhibits, networking, free breakfast and lunch and much more.

The event will be held A the Melbourne Park Function Centre from 8:00am-5:30pm? It is a free event however it does require registration.

I'll be there, I expect to see most my local colleagues there (at least if not without a very good excuse). I would encourage you to bring along your colleagues, friends, bosses, wives (ok maybe not) and make it a really good day. There is going to be some great content and maybe even a few surprises on the day.

For more details and registration visit this link
http://www.myvmug.org/e/in/eid=99

Some of the final details are still being worked out and will be posted either here and/or on the event site in coming days.

Finally, if you would like to be involved or have any suggestions please feel free to contact me @g_mulholland or Craig Waters @cswaters1 or @mvmug. We welcome any and all support.

Melbourne VMUG - This Week

2011-09-27T10:20:00.001+10:00

The Melbourne Quarterly VMware User Group meeting is being held this week. The VMUG’s are a great opportunity to come together with your fellow VMware users to discuss virtualization trends, best practices, and the latest technology!

The event will be held at Trinity Grammar School (Kew) this Thursday Sept 29 from 2pm to 6pm

Meeting Highlights

VMware vSphere 5 - Nathan Wheat, Senior vCloud Specialist, VMware
10 Questions in 10 Minutes
Round Table Breakout Discussion
Food, Drink, and Networking

For more information and registration please see the link below:

http://www.myvmug.org/e/in/eid=155&source=5

Hope you can make it!

How vSphere 5 Challenges Storage Design

2011-09-20T20:06:00.000+10:00

vSphere 5 revealed around 150 new features many of which are still to be understood. The new storage features however, got plenty of publicity. Storage DRS or SDRS, Profile Driven Storage and VASA, Datastore Clustering , VMFS5, additional VAAI capability and Storage IO Control improvements were the main ones.

Many of these new storage features now introduce new design questions when we look at vSphere storage. The integration of such features in a design piece is now paramount to a working model. It gave me a chance to think about how I would redo things in some recent environments I’ve encountered and how I might in the future. I found that the way I provision new virtual machines would be impacted by my Datastore and backend LUN/Volume layout. Specifically, when dealing with different types or classes of data that may be present in a VMware environment. In an ideal world we would have as few as possible paths to spin-up new workloads. Although this was somewhat possible in previous version but with some of the new vSphere 5 features this I greatly enhanced with Clusters and Profile Driven Storage and SDRS particularly.

First some background on a couple of these new features which I’ll talk about in decision making later.

SDRS – Storage DRS provides a load balancing and virtual machine placement mechanism much like its cousin DRS did in previous versions for virtual machines across hosts. Virtual machines can now be migrated across Datastores within a Datastore cluster. SDRS basically prevents Datastore imbalances by monitoring space conditions and latency, not to be confused with SIOC which is a fairness control.

Profile Driven Storage – This new feature in vSphere 5 integrates with storage capabilities to allow you to effectively tag a Datastore with specific characteristics. By doing this it greatly simplifies the provisioning process and you'll see why later. Administrators will then have the ability to provision a new virtual machine and attach a storage profile to any virtual disks. The benefit is realized further when the Datstore/Storage design encompasses and end to end storage profile. By that I mean when an Administrator provisions a new virtual machine he can select the silver profile for his operating system and gold for his data (as an example). The storage profile definition can be your own and I must admit I don’t like the gold, silver, bronze approach. In my eyes it tells me nothing about the profile I am attaching to my virtual disks. Unless recorded somewhere else these labels mean nothing, they might as well be Larry, Moe and Curly. I would prefer them to be something like disk-tier_replication-type i.e. sas_semisync, sata_norrep or perhaps you could mirror the cluster naming convention. To me this gives me a clear indication of what service I am selecting. Storage Vendors will also be able to integrate with the vStorage APIs for Storage Awareness (VASA) to leverage information profiling.

The value of both of these new features become apparent particularly when provisioning virtual machines. Again, my goal was to define the ‘as few as possible’ touch provisioning model. What these technologies do allow is for me to deploy virtual machines without the care or need of targeted placement and at the same time ensuring that the virtual disk format matches my storage design, performance and capacity management.

I had a think about the way I could achieve this and came up with 3 ways.

Design #1

In this design we have 3 Datastore Clusters with each containing Datastore attached to a single storage profile. Cluster 1 is dedicated as tier 1. The definition of tier 1 can be of your own determination but I like to think of it as not only a performance characteristic but also as determined by my RPO/RTO requirements. This allows me to place such workloads on high performance disk (SSD or SAS as the case may be) and also guarantee a certain level of backup or protection or defined SLA. I am going to leave transport protocol out of this discussion for the moment. I am a big NFS fan although the multipathing that fibre brings and some of the new features in VMFS5 are certainly reasons to consider your position. If going with VMFS, ensure you keep block size or VMFS versions separated in their own cluster for Storage vMotion and other reasons.

The beauty of this model is that we are using the cluster as the management entity as it should be, by that I mean 1 profile per cluster, 1 datastore class per cluster. This greatly simplifies the provisioning process as the placement of VMDk’s is now dynamic based on SDRS algorithm’s but also guaranteed to comply with the required profile, or tier for the want of another word.

SDRS Integration – At this point SDRS will have the capability to move VM’s within the cluster without any restriction. With technologies such as deduplication and snapshot backups the movement of virtual machine can have an impact. Deduplication to me is not a huge issue, sure you lose the dedupe savings/ratio on the volume, but you’ll probably gain it back when the process runs its next pass. Snapshots is a bigger issue, you’ll probably want to re-baseline a snapshot copy after SDRS has run. The good news here is that SDRS can indeed be scheduled. If you were able to line all your ducks up you could run the SRDS process after hours, prior to dedupe is set to run etc. Although it is possible to run multiple NFS mounts inside a volume container which would mean that dedupe savings would be retained (as dedupe is on a volume basis) the question then becomes one of sizing and RTO/RPO. Specifically with NetApp any volume based replication such as Volume SnapMirror would mean that the entire volume (containing 1 or more Datastores) would be your recovery point and hence your time to recover becomes greater. With 16tb volumes however it would be technically possible but I wouldn’t recommend it, like I said deduplication is not a massive problem in my eyes here. Always consider the overhead of manually schedules with anything, particularly SDRS and as such you may determine which method works well for you as you determine any operational overhead required. In my opinion, it is certainly manageable to run SDRS on your production virtual disk containers.

I/O load balancing with SDRS could be used in this case as we would expect that each vmdk on these Datastores is a specific class of data and thus has certain performance characteristics, meaning an imbalance in reported I/O latency would suggest a Datastore/volume wide issue and probably be less likely to report false positives by higher I/O profile vmdk’s. I find most peoples natural reaction to I/O balancing is that it will cause more than usual migrations or at least recommendations and whilst that may be true it will also become important where the space utilization on all Datastores in the cluster is similar. In this case SDRS uses the I/O balance metric in addition to the space metric to determine the best migration. Again, there is nothing forcing you to generate recommendations only or schedule SDRS if you are worried about the Datastore rate of change, however as mentioned you’ll have to accept the overhead.

Design #2

This design implements a single Datastore cluster whereby all customer Datastores are housed in the one cluster pod. The benefit of this design is that the cluster may be easier to manage however there are a number of drawbacks. This design would only work well for smaller implementations keeping in mind that there is a maximum of 32 Datastores per cluster. This model is also not recommended as it creates additional layers of management where each Datastore must be attached to the storage profile of choice.

SDRS Integration – This is where things get sticky. When SDRS looks to move virtual machine disks around it does so at the cluster level. What this means that in environments where transient and non-transient data is separated SDRS will be unable to manage the placement of these virtual disks correctly. Whilst affinity or more correctly anti-affinity rules may be employed to help at this time it may become cumbersome to create a rule for each virtual machine to keep vdmk’s apart, although it may be technically possible (depending on number of Datastores and layout etc). In the case where a Datastore is dedicated for vswp space it is not considered in calculations for SDRS rebalancing, meaning that it should not be a consideration for SDRS integration.

I/O Balancing although could be still used although while mixing tiers of storage/data into the one cluster it means that SDRS migrations may violate your design from a performance and RTO/RPO perspective.

In this design there may be no effect on backup and recovery as most replication is done at a Datastore/Volume level. The fact that all Datastores reside in one cluster should not make any difference in this case, other than retaining deduplication ratios.

This design does not offer flexibility especially for larger scale deployments that are looking to use features such as SDRS and define replication targets for backup and recovery. I would not recommend anyone to use such a configuration, although probably technically possible without SDRS etc.

Design #3

This design is a hybrid between the first two. It couples the profile of tiers 1 and 2 into a single cluster for non-transient data types and employs a second cluster for tier 3, transient data types i.e. vswp, logs. I have seen this setup on paper and whilst it separates the transient Datastores into a separate cluster it does not differentiate between other data classes in the first cluster.

SDRS Integration – Cluster 1 would be fully capable of using SDRS however where there is a delineation between disk tier or storage profile as we have in this case I would suggest being careful allowing virtual disks to be migrated to other Datastores that do not match the end to end profile. You may be able to use affinity/anti-affinity rules here again to separate such but I would also suggest as before that it would be hard to manage. The second cluster is for purely transient data and as such all Datastores therein will be ignored by the SDRS algorithm. I wouldn’t turn it on at all for this cluster.

Creating the Storage Profile Catalog

Create the Storage Profile and assign Storage Capabilities. The Storage Capability in this case must be unique to the Profile for it to be meaningful. If you assign the same capability to Gold and Silver and then assign that capability to you Datastores in different clusters your provisioning rules may be violated later on.

Assign a user defined Storage Capability to each Datastore in the cluster. NOTE: A Datastore can have only 1 user-defined and 1 system-defined Storage Capability. A cluster cannot be assigned a Storage Capability however where all Datastores in the cluster contain the same Storage Capability the cluster will inherit this capability.

Provision a new virtual machine. Notice when selecting the Gold Profile only the NFS_Cluster matches the compliance check.

Once the virtual machine provisioning has completed you can verify the compliance in the Storage Profiles window

In the event I had chosen an incompatible Datastore or cluster I would see the following report. (Notice the Datastore difference in these two shots.

Wrapping up…..

I purposely added Designs 2 and 3 to this article to show, that although technically possible to create such designs they produce some potentially problematic outcomes. The ideal scenario for what my environment was to look like was Design 1. I believe this design allows me to create 3 specific points of management, each with a particular data profile, RPO/RTO and or performance tier. In each entity SDRS can be leveraged without having an impact on the storage layout and if you recall my initial premise, incorporates Profile Driven Storage to allow ‘few’ touch provision/placement. You can see how mixing and matching Storage Capabilities and Profiles within a cluster could get complex. Generally speaking it makes little sense and will only make life hard for yourself.

On a side note I demonstrated the virtual machine provisioning from design 1 for some people yesterday and they were blown away by the ease and flexibility of it. As usual, get the design pieces right and it’s magic.

vForum Sydney - Community Party

2011-09-20T16:52:00.000+10:00

For those of you going to vForum in Syndey next month Alastiar Cooke (@DemitasseNZ) is organising a community party on the Tuesday evening before vForum Sydney starts. The event will be free to all who wish to attend however we ask that you register your interest at the followin signup page.

http://vmdownunderground.eventbrite.com/

In the spirit of the VMUnderground party and vBeers initiative we hope you will be able to come along and enjoy a few ales with your peers and colleagues before, what should be a great conference commences the following daay. movement. The event will be held at the Pumphouse and wont be one to miss.

Hope to see you there.

NetApp Virtual Storage Console 2.1.1 – vSphere 5 Ready

2011-09-09T19:50:00.003+10:00

NetApp Virtual Storage Console version 2.1.1 is now available. I thought I’d take a minute to note some of the new key features that caught my eye.

vSphere 5 Support

VSC now fully supports vSphere and vCenter 5

Mbralign

As many who have worked with NetApp and the VSC beforehand we know that there was always a limitation of our tools with respect to ESXi. Given that vSphere 5 is now only available in ESXi flavour and that has been VMware’s roadmap for a while now we now have a version of the mbralign tool that works with ESXi (version 4 and 5). This is a good step forward in my estimation as we had always previously designed a physical ESX host into the every environment to handle alignment.

Better Host Integration

In 2.1.1 we now have the ability to nominate hosts for Monitoring and Configuration to bypass or ‘skip’ as it were. This is a handy addition as at times there were valid reasons you may not want configuration settings being read or pushed to such hosts. Previously this capability was only available to controllers. In addition, a new ‘Status’ column now allows for some better health reporting in the GUI with respect to why an alert for a host has occurred.

Better Cloning Features

The cloning function of VSC now allows for greater customisation during the cloning process. In fact, it allows the ability to change memory and cpu configuration but probably most importantly prevents clones being misaligned. There is also now the ability to do space reclamation after deletion inside the guest for NTFS file systems on NFS, however there are some caveats around this.

Datastore Replication

VSC now incorporates a remote replication feature which uses snapmirror (asynchronous) to clone NFS datastores from a source vCenter to another vCenter site. The incorporation with snapmirror is an efficient way of doing replication due to the nature of incremental updates after the initial baseline synchronisation. NOTE: The target and destination vCenters must be the same version

There are other improvements such as integration with controllers running Cluster mode but I’d encourage you to read the offical documentation for any further information

VMware vSphere Provisioning and Cloning Administration Guide http://now.netapp.com/NOW/knowledge/docs/virtual_storage_console/relvsc211/pdfs/cloning.pdf

VMware vSphere Installation and Administration Guide http://now.netapp.com/knowledge/docs/virtual_storage_console/relvsc211/pdfs/install.pdf

VMware vSphere Backup and Recovery Administration Guide http://now.netapp.com/knowledge/docs/virtual_storage_console/relvsc211/pdfs/backup.pdf

vSphere 5 Auto Deploy PowerCLI Help

2011-08-26T21:21:00.000+10:00

I noticed some people had this issue so far and I ran into it. Some folks were doing an Auto Deploy configuration and ran into problems when it comes to the PowerCLI section.
I realise some folks will not be as familiar with PowerCLI as I or others are so perhaps something simple like this can save some frustration.

Firstly if you have not used the Auto Deploy feature yet perhaps a read of either Duncan or Gabrie's artciles would help (see below links)

The problem arises when trying to run PowerCLI commands. In some situations the commands do not work and will give the 'not a valid cmdlet' error.

This issue can occur when installing PowerCLI on a 64bit machine. The installer drops a 64bit and 32bit version of the PowerCLI app onto the host. Some commands are not available via the 64bit version.

In actual fact in my case I did not need to add any PSSnapin's in the 32bit version of the app. I believe that this was the case as I was using PowerCLI on the machine i had also installed Auto Deloy and vCenter. If do require using these cmdlets in the 64bit version or on another host with the 32bit version you will need to manually add the snapins by using the command:

Add-PSSnapin VMware.DeployAutomation and

Add-PSSnapin VMware.ImageBuilder

To verify they were added correctly you can execute the Get-PSSnapin command and you should see the following:

Name : VMware.DeployAutomation
PSVersion : 2.0
Description : Cmdlets for Rule-Based-Deployment

Name        : VMware.ImageBuilder
PSVersion   : 2.0
Description : This Windows PowerShell snap-in contains VMware ESXi Image Builder cmdlets used   to generate custom images.

Now you are free to use the commands at will as shown by the output below

Auto Deploy Links
http:///www.yellow-bricks.com/2011/08/25/using-vsphere-5-auto-deploy-in-your-home-lab/
http://www.gabesvirtualworld.com/vsphere-5-how-to-run-esxi-stateless-with-vsphere-auto-deploy/
http://pubs.vmware.com/vsphere-50/index.jsp?topic=/com.vmware.vsphere.install.doc_50/GUID-71F8AE6C-FF4A-419B-93B7-1D318D4CB771.html

Greg

Number 5 is ALIVE!!!

2011-08-25T15:05:00.001+10:00

This afternoon VMware made available its much anticipated release of vSphere 5. Although the NDA has been lifted for a while it' still a relief to see the final GA product available for download.

Below are some handy links:

Direct ESXi download link - http://www.vmware.com/download/download.do?downloadGroup=ESXI50
Direct vCenter download link - http://www.vmware.com/downloads/download.do?downloadGroup=VC50
Documentation Center - http://pubs.vmware.com/vsphere-50/index.jsp
Documentation Links - http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

You can also find your way through the main vmware.com pages

If you havent read about some of the new features or why you should be excited the vSphere Blog put together a list of the new features. They are quite comprehensive, check it out

http://blogs.vmware.com/vsphere/2011/08/vsphere-50-features.html

Enjoy

Configure iSCSI port binding in VMware vSphere 5.0

2011-07-25T17:39:00.001+10:00

One of the noted improvements I found in vSphere 5 was the ability to bind iSCSI ports to VMkernel interfaces via the GUI.

Previously in vSphere 4.1 the administrator had to achieve this task via the CLI using commands such as esxcli swiscsi nic add –n vmk0 –d vmhba34.

The first thing you’ll notice when viewing the properties of the Software iSCSI Properties is the new Network Configuration tab.

This tab allows you to perform the binding task. It is important to notice that you are only able to bind the iSCSI adapter to a compliant network adapter/port group. If there are no such configurations you will see the following warning message.

In order to do this you should create 1 or 2 separate VMkernel port groups each with a separate active nic and bind both of these to the iSCSI adapter. Note that any other adapters on the vSwitch should be set to unused. I am leaving the Policy Exceptions out of scope for this article.

FF 5 mins
Now that I have created my 2 vMkernel interfaces I can bind them to the adapater. Hit the Properties of the iSCSI adapater again and you now should be able to add adapters.

First added already

Select the second port group and add.

Finally both my adapters are bound to the iSCSI interface.

After closing the window you will be asked to rescan the adapters and that’s all there is to it. Easy huh!

Datastore unable to enter SDRS Maintenance mode

2011-07-21T19:25:00.001+10:00

I spent some time in the last night playing around with some of the new storage features of vSphere 5. What I saw really impressed me although I came across an issue which annoyed me at first, but with a little persistence and some more caffeine I was able to find a relatively simple answer.

I setup a test to try out the new Datastore Maintenance feature. I simply right clicked the Datastore and went to put it into SRDS Maintenance mode. What I encountered was the SDRS Maintenance omde status remained at 1%.

Now there are generally 2 reasons why this would be a problem. 1. SDRS is disabled on a disk in the VM or SDRS affinity rules are preventing migrations.

As for SDRS being disabled for a disk there are could be many reasons such as a virtual disk part of the same VM resides in a separate Datastore cluster, the disk is independent, when the virtual disk is a template etc. It is important to ascertain why this might be the case and rectify the issue.

The second problem and the one I found and I suspect most people will want to check is when using SDRS affinity and anti-affinity rules. In my case I had an anti-affinity rule preventing vmdks from a virtual machine from residing on the same Datastore in the cluster. This rule was preventing SDRS from recommending and performing the migration to evacuate the Datastore.

The SDRS window shows the fault that was logged at the time.

I happened to find a way around this Storage DRS advanced option IgnoreAffinityRulesForMaintenance to 1. By default the setting is 0. After changing to 1 I was able to perform the task again and the rest was successful.

Voila! One empty Datastore. One very cool feature!

Introducing Virtual NUMA in vSphere 5

2011-07-21T19:03:00.000+10:00

There has been a lot of talk already about new features announced in vSphere 5 since the cloud launch last week. After being involved in the beta and doing some of my own discovery/testing in my home lab I stumbled across something that I thought was cool if not interesting.

Virtual NUMA is now part of vSphere 5 and although it won’t get the attention of some of the highlight features like Storage DRS I thought it may deserve an introduction at the least.

Virtual NUMA is basically the ability to extend the NUMA architecture to the guest operating system of a virtual machine. It works only with virtual machines with hardware version 8 and is enabled by default for such virtual machines that have 8+ allocated vCPU’s.

So why is it important? In much the same way that NUMA allows performance benefits for operating systems and applications on a physical host, be it ESX or other, Virtual NUMA now allows some of your most ‘Monster’ virtual machines to also benefit from the underlying NUMA architecture in the host. Stay tuned for some further information on this in the future. I know Frank Denneman is working on some more information on performance and impact on NUMA latency.

Configuring guest CPU entitlements

In order to configure Virtual NUMA settings you will need to set the number of virtual sockets and number of cores per socket setting for the virtual machine.

Keep in mind that the total number of cores for the virtual machine needs to be less than or equal to than total number of sockets x total number of cores per socket in the hosts. In fact in my experience it will not let you configure more than the hosts can handle, but I’d like to see this on some ‘real’ hardware myself.

At the guest level if the cores per socket is greater than 1 then the Virtual NUMA node size is equal to the socket size. If however, the number of cores per socket is 1 then the Virtual NUMA node size match the sizing on the host.

Advanced NUMA Parameters

You can add these advanced options to the virtual machine configuration file.

Option	Description
cpuid.coresPerSocket	Determines the number of virtual cores per virtual CPU socket. Also determines the size of virtual NUMA nodes if a virtual machine has a virtual NUMA topology. You can set this option if you know the exact virtual NUMA topology for each physical host.
numa.vcpu.maxPerVirtualNode	If cpuid.coresPerSocket is too restrictive as a power of two, you can set numa.vcpu.maxPerVirtualNode directly. In this case, do not setcpuid.coresPerSocket.
numa.autosize	When you set this option, the physical host is matched every time you power on the virtual machine.
numa.autosize.once	An exception to this behavior is if your virtual machine template is created from a virtual machine that has been powered on with an ESXi 5.0 host, the template will have the same virtual topology.
numa.vcpu.min	Minimum number of virtual CPUs in a virtual machine that are required in order to generate a virtual NUMA topology.
numa.vcpu.maxPerMachineNode	Maximum number of virtual CPUs that belong to the same virtual machine that can be scheduled on a NUMA node at the same time. Use this attribute to ensure maximum bandwidth, by forcing different NUMA clients on differentNUMA nodes.
numa.vcpu.maxPerClient	Maximum number of virtual CPUs in a NUMA client. A client is a group ofvirtual CPUs that are NUMA-managed as a single entity. By default, eachvirtual NUMA node is a NUMA client, but if a virtual NUMA node is larger than a physical NUMA node, a single virtual NUMA node can be backed by multipleNUMA clients.
numa.nodeAffinity	Constrains the set of NUMA nodes on which a virtual machine's virtual CPU and memory can be scheduled.

NUMA Node Affinity

It is possible to also associate the virtual CPU and Memory of a virtual machine within specific NUMA nodes. Sounds cool right, but there is a big caveat. The ESXi CPU Scheduler has the ability to balance all virtual machine compute resources across NUMA nodes to ensure that 1. Performance is optimised but also 2. A degree of fairness is applied to each virtual machine on the host. By constraining which NUMA nodes can be utilised for specific virtual machines you potentially thwart that ability. The way I look at it, don’t do it unless you have a valid reason and understand the implications.

The process itself is straight forward. The easiest way is to power off the virtual machine and add a row to the Configuration Parameters on the virtual machine. The value’s I have set of 0,1 will limit the virtual machine to NUMA nodes 0 and 1.

So there you have it. I think I know of some specific cases in the past whereby this may have allowed me to better manage virtual machines that have disproportionate large memory consumption, but time will tell how useful a feature it truly becomes.

If you have used it or plan to, particularly on your ‘Monster VM’s’ (love that term) then I’d love to hear about it.

3TB SATA Disk Drives with NetApp DS4243

2011-07-20T17:38:00.001+10:00

NetApp has now released 3tb SATA disk drives for its DS4243. These 3tb disks require ONTAP 8.0.2 or later. You can read some more info here http://media.netapp.com/documents/ds-3096.pdf

I did some rough back of the napkin math. For a secondary controller I deploy 4 x 42RU racks per 6280 pair at the moment. 7 shelves in the two racks with the controllers and 9 shelves in the empty racks. Each Shelf can hold 24 disks. My math looks like this:

24 disks x 3tb = 72tb per shelf.

72tb x 9 shevles = 648tb per rack

72tb x 7 shelves = 504tb per rack

648tb x 2 (9 shelf racks) = 1296tb + 504tb x 2 (7 shelf racks) = 1008tb

That’s a raw total of 2,304tb !wowsers!

NOTE: The 6280’s and 6240’s can support nearly double the amount of disks in this system (1,440). The 6210’s support 1200 disks http://www.netapp.com/us/products/storage-systems/fas6200/fas6200-tech-specs.html

If you expanded to 6 racks with 3m SAS cables think of the data you could store. Even going to fewer 47RU racks would make it interesting.

Pretty scary this world we live in nowadays isn’t it.

Need a ticket to VMworld 2011?

2011-07-20T09:32:00.000+10:00

My fellow vExpert Greg Stuart in conjunction with Symantec, Veeam and Xsigo is giving away a ticket to VMworld 2011 in Las Vegas.

The ticket will include a conference pass, air fare (to the value of $500US) and 5 nights at Harrahs Las Vegas.

Entry:

To enter this contest, post one comment/reply including the following information:

· Your favorite thing about VMware

· What you look forward to most at VMworld 2011

· Who you hope meet at VMworld 2011

In your comment, please include the following:

· First and last name

· Email address to contact you if you win

· Twitter handle (if you have one)

The winner will be randomly drawn Sunday July 24th, notified via email and announced Monday morning July 25th on vDestination.com.

Rules:

1. The contest is open to anyone in the world, however, the air fare is only covered up to $500 US dollars.

2. You must be able to attend VMworld 2011 in Las Vegas, Nevada (conference pass is only valid for the US conference). I don’t want this pass to go to waste.

3. Only one valid entry per person.

4. Contest entry must be received by Sunday July 24th 11pm EST, no entries will be considered after 11pm EST.

For more information and to enter, see this link to Greg's blog http://vdestination.com/2011/07/15/win-a-free-pass-to-vmworld-2011-from-vdestination/

VMware vSphere 5 Clustering Deepdive Book - Available

2011-07-13T05:41:00.001+10:00

The second book in the Deepdive series from Duncan Epping and Frank Denneman is now available. For months these guys have been working tirelessly on the detail on this book. I know there were many late nights and early mornings spent in dialogue with Engineers understanding the new technologies, writing, creating associated diagrams, formatting text and plenty more.

This version of the book deals specifically with vSphere 5. The book comes in at around 350 pages (100+ more than the first) and is available in both Black/White and Colour versions from Amazon and Createspace. There will also be a kindle version of the book available.

For anyone out there who hasn’t seen the vSphere 4 clustering deepdive that Duncan and Frank I recommend you add it to your library. Of 28 odd customer reviews on Amazon, 25 of them were rated 5 star. It is clearly the bible for vSphere clustering technologies. You can find it here http://amzn.to/oSgTDg

The vSphere 5 version promises to be even better and even deeper. Plenty of deepdive material and accompanying graphics to illustrate some awesome new features. If you really want to know how this stuff works you simply have to buy this book, you will not be disappointed.

Here is this chapter list for the new book. I’m sure you will agree there are some very interesting topics.

Part I vSphere High Availability
Chapter 01: Introduction to vSphere High Availability
Chapter 02: Components on High Availability
Chapter 03: Fundamental Concepts
Chapter 04: Restarting Virtual Machines
Chapter 05: Adding Resiliency to HA (Network Redundancy)
Chapter 06: Admission Control
Chapter 07: VM and Application Monitoring
Chapter 08: Advanced Options
Chapter 09: Summarizing

Part II vSphere Distributed Resource Scheduler
Chapter 10: Introduction to vSphere DRS
Chapter 11: vMotion and EVC
Chapter 12: DRS Resource Entitlement
Chapter 13: Resource Pools and Controls
Chapter 14: Calculating DRS Recommendations
Chapter 15: Guiding DRS Recommendations
Chapter 16: Introduction to DPM
Chapter 17: Calculating DPM recommendations
Chapter 18: Guiding DPM Recommendations
Chapter 19: Summarizing

Part III vSphere Storage DRS
Chapter 20: Introduction to vSphere Storage DRS
Chapter 21: Datastore Clusters
Chapter 22: Datastore Cluster Settings
Chapter 23: Storage vMotion and SIOC
Chapter 24: Calculating SDRS Recommendations
Chapter 25: Guiding SDRS Recommendations
Chapter 26: Summarizing

Part IV Integration
Chapter 27: Integration

Happy Reading!

Well done Duncan and Frank

Getting VMware NMP Right

2011-07-12T19:48:00.001+10:00

Recently I have seen some interesting configurations dealing with mixed storage environments. That in itself is not terribly amazing, however I am seeing some interesting configurations that generally require a fair bit of work to fix.

The main problem that I see occuring is when change on the storage array or ESX host goes un-documented. In addition to that it is important to understand what happens at an NMP level when the host gets rebooted.

I have witnessed NMP configurations that are continuously not best practise. Ultimately that’s fine, it’s your decision but in most cases it’s born out of poor operational process, poor management of the environment or just not understanding the technology. I must stress (with my NetApp hat on) that it is important to talk to your storage vendor about this. Each vendor has a nominated best practise for each and all type of array they sell, what’s more vendors such as NetApp and EMC etc. have tools that allow you to manage these settings with greater automation on the hosts.

NMP Components

As you can see below the VMware NMP is broken down into 2 sub sections. The SATP and PSP. By now you should know what they are and mean so I won’t re-hash. What I would re-iterate though is how the configuration is of importance. The SATP is configured automatically on the host at boot time. In fact, the host sends a SCSI-3 INQRY command to the storage array to determine what type of array it is connected to. Once the return command has been received by the host NMP is able to configure the SATP accordingly. The PSP is then set based on the resulting SATP setting. There are a number of permutations this can bring. See the FC Configuration Guide for more info on this.

What I frequently see is that for an ALUA array the SATP is set to MRU, which seems to be the default, most vendors would however tell you that manually changing the PSP to Round Robin is the way to go, purely because it is capable of distributing IO down each path, whereas MRU and Fixed will not. How you choose to do this is your business. I use NetApp VSC but have written some PowerCLI scripts to do it as well for when VSC or other tools are not installed.

The other issue I have noted is when host reboots are few and far between or simply not scheduled. I have run across some environments where despite having an ALUA capable array NMP is set to a mixture of policies for a subset of Datastores on the host. I have also previously seen such configuration cause issues when path failovers or head/cluster failovers occur, it aint pretty! Most of this configuration comes when reboots of hosts are not done. In order to change the SATP of the host a reboot has to take place, if not the policy will stay enforced until such time as the reboot happens. I should point out though that a PSP change does not require a reboot of the host.

My overall point is if you are using FCP or ISCSi in an ESX environment, it is very important to ensure that you keep consistency across the environment where possible in my opinion. Storage best practises exist for a reason and should be implemented. It is also a good idea to use scripts to check the configuration on a timely basis if not using a management tool provided by the storage vendor.

Below are some recommendations I have sourced from some of the main storage vendors and a link to their best practise guide if I can locate it.

NetApp

Fixed non ALUA
RR with ALUA
http://media.netapp.com/documents/tr-3749.pdf

EMC (Not Symmetrix)

MRU non ALUA (A/P)
RR with ALUA
Powerpath/VE (Third-Party MPP)
http://www.emc.com/collateral/software/white-papers/h6340-powerpath-ve-for-vmware-vsphere-wp.pdf

HP EVA

MRU or RR With ALUA
http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA1-2185ENW.pdf

HDS

NetApp Operations Manager DR Options - Virtual or Physical?

2011-07-05T19:54:00.001+10:00

I was compelled to address the requirement for protecting NetApp Data Fabric Manager (Operations, Provisioning, Protection Manager). I would have assumed that the default position of most would have been to quote Olivia Newton John ‘let’s get physical’. I was pretty right, so I decided to explore the theory.

With anything of this nature, the first and foremost point of order was to identify the business requirements for the protection of DFM. I approach this by trying to ascertain what services and processes we are in fact trying to protect by this solution. Generally speaking if you can’t answer that then what I am about to tell you will be of little use.

In this scenario if DFM were to experience an outage of any kind of outage we would lose the ability to provision storage via Provisioning Manager and thus relationships that are configured with Protection Manager would not be configured during this time. The second feature that would not be available is Protection Manager driven snapshots would not be initiated and thirdly obviously reporting etc. would be down.

Let’s examine those issues for one minute before we move on. Firstly although our chosen method of storage provisioning is via Provisioning Manager we would still have direct access to the storage controllers and would still be able to perform such tasks. There is definitely some magic to the process when using Provisioning Manager which you don’t generally get directly from the filer but retro-fitting the volumes etc. in Provisioning Manager later on is not terribly difficult. I’d also question how much storage provisioning would be going on when you’ve just experienced this type of failure. Secondly, and probably more importantly, if your snapshot policies is set to generate snapshots on an hourly basis for instance then it is clear the we cannot have an RTO greater than 1hour. Furthermore, I am not aware of any compliance regulations about long term backups. Backups fail all the time, if you miss one, initiate another. SnapVaults can fail regularly for one reason or another but the subsequent vault works fine, no harm no foul!

Let’s leave the application corruption side of the argument to one side. We’ll utilize snapshots for recovering from this process.

So let’s look at some possible scenarios:

1. 1 Single VM – HA protected

In this scenario we would be protecting the guest VM from a Host based outage with the use of VMware HA. Although, this is not a 0 downtime scenario it may in fact be sufficient in many situations. What it does not protect against is a Datastore/LUN or loss of storage access, nor does it protect from a major site loss.

2 VM’s – Clustered

Clustering the guest using MSCS shares the same characteristics as scenario 1 however it will provide application uptime in the event both guests are on different hosts and they both don’t fail

1 VM – SRM

A single VM protected with SRM would protect against either site loss or storage loss, however it can be overkill for an environment that does not already have SRM implemented

4. 1 Host – Physical

A single physical host provides little if no protection against a host, site or storage failure

2 Physical Hosts – locally clustered

This scenario provides redundancy similar to a virtual MSCS cluster

6. 2 Phsyical Hosts – geo clustered

This scenario provides storage and site loss however once again if the storage or entire DC goes down in the primary site then managing the failed site via ops manager is somewhat pointless. It can though manage other storage controllers that are configured/discovered by DFM

So for one moment lets step back to the original requirements. The requirement was to reduce the time which we are unable to provision new storage, take snapshots and report using DFM. So let’s agree to put a strike through number 4 straight away. I’m going to go further and striker out number 5 too. I don’t see the point in locking myself down to physical hardware where there is not a good reason to. This scenario doesn’t provide me any better failover than number 2. SRM is not an option now or anytime in the future so let’s take out number 3.

So we are left with scenario’s 1,2 and 6. The differentiating points between these as I see it are:

- 2 potentially gives greater application level uptime then 1

- 6 gives the ability to manager storage controllers from other datacenters at a time when one is offline. The big issue I have with this is that it can be quite costly to setup in that you would require a secondary clustered host and access to the same database and LUNs etc. so to my mind you may get more benefit by leveraging SRM at this point. This scenario is going to have significant financial impact so for all but special cases I am going to take it off the table, but it’s there for the future as an option.

Now the difference again between 1 and 2 was application uptime. Can my environment sustain a certain amount of downtime? I am going to define a random number of 15 minutes of downtime as my RTO here just for sake of the argument. In this case I believe the first scenario is the cheapest, simplest and most cost effective to achieve the goals I have outlined.

The final option I have not introduced as yet is a more manual one but could be utilised. A secondary VM or even a physical server in the remote site with a stale connection to the DFM database LUN replicated via snapmirror from the primary site to the secondary site. In the event of a failure the services would be started and DFM connected to the mirrored LUN. This would give you the ability to manage the secondary site storage (not the primary in the event of a site failure). You would have to reverse the process doing a snapmirror resync etc. when the primary site came back online, so even though it sounds like a relatively low cost solution the operational effort could outweigh the benefit. Case in point, how long would you expect this process to take for someone to do? 10 mins?, 15 mins?, 30 mins?

I am often reminded about the misconceptions of a virtualisation and the points of failure it introduces. In many cases they are just that ‘misconceptions’ and generally speaking I find that it stems from a lack of understanding in the technology. I have thrown together a list of reasons why I think a VM is a better approach, in this scenario particularly in addition to what I have explained above:

If the ESX host fails the VM gets restarted after 13 seconds when the host is declared dead (This is a feature of VMware HA). If RTO is less than a few seconds then it’s not an option sure, but in reality its rare.
VMware HA runs whether the vCenter service or guest is running. vCenter is only a required at HA configuration time. The config is copied to all primary nodes (1^st 5 hosts in the cluster). vCenter is not a POF.

A VM is a far more flexible arrangement, it can be moved between hosts/storage, upgraded, backed up and snapped easier.

In the event of a large scale failure you would simply log on to the ESX host where the VM resides and restart it. You can disable DRS for this single VM so you always know where it resides.

Recovery of a physical host takes far greater time. It requires access to the datacenter possibly. Even in a clustered environment, there is only 1 active host so you are exposed to greater risk for that duration.

If local storage issues were to affect a physical server (ESX Host or VM) only the ESX type would be able to self heal using HA (unless using a physical cluster in which case it wouldn’t be a problem).

An outage on SAN storage or Network is an external influence and will affect either implementation. In fact a VMware environment does if not should, be configured for greater levels of redundancy than a physical deployment.

Every time a change or upgrade is required in a clustered environment the changes need to be mirrored across both nodes.

2 Physical boxes is a hell of a lot of extra cabling and rack space then a VM

I’m sure there are others, but you get the idea.

So what’s the solution. In my opinion, with the detail I have provided, I believe I would use a single VM in a HA cluster for small host failures and small outages. In reality I see more datacenter facilities changes or other third party outages than ESX Host or storage outages, so I am not overly worried about an ESX host dying on me, even so, I understand the risk. I would also incorporate the cold standby VM in the second primary site using a snapmirrored replicated LUN. This would provide me total recoverability if and only if, the event that the entire primary site went offline (Never seen it happen, yet!). What’s beautiful in this instance, I have not cost the business a cent with this solution. Everything in my solution is already in place and happening.

In conclusion it is imperative to have clearly defined goals. Keep in mind that technology is not the only driving factor in architectural decisions in an organization. Sometimes the best technical decisions will be swept under the carpet in favor by a more practical, cost effective means. If you have outlined the design in line with the goals then the business case should be evident to all and if it is really that important to the business the money will be placed where ones false teeth are otherwise inserted.

vExpert 2011 - Sharing The Love

2011-07-03T08:54:00.000+10:00

The news came through late Friday that VMware and @jtroyer have awarded me the vExpert award for 2011.

The VMware vExpert Award is given to individuals who have significantly contributed to the community of VMware users over the past year. vExperts are book authors, bloggers, VMUG leaders, tool builders, and other IT professionals who share their knowledge and passion with others. These vExperts have gone above and beyond their day jobs to share their technical expertise and communicate the value of VMware and virtualization to their colleagues and community.

I was very humbled by the award as I joined some very illustrious company of my peers in this field. People such as Jason Boche, William Lam, Gabrie van Zanten, Luc Dekens, Bas Raayman, Cody Bunch, Vaughn Stewart and many others.

The award is not rate technical prowess but more the willingness passion of the indivual to share his or her experiences with the wider community. The simple reason I chose to do this over the 2 or so years is that the VMware community is like no other tech community i've been a part of. The knowledge and wealth sharing as well as professional friendship across continents has to be seen to be believed. I have certainly learned a great deal some these experts as the community and product has grown and I thought it was time to give some back. All in all I was thrilled to make a contribution and even more thrilled that that contribution has been recognised offcially this year.

To this date I have not seen any other Aussies recieve the award. Hopefully on monday there will be some more such my friend and now EMC employee Andre Leibovici. If anybody knows of any please let me know, i'd like to make contract with them.

To my fellow vExperts, Congratulations! I think John has some cool stuff lined up for us, we'll have to wait and see..

Greg

http://www.vmware.com/communities/vexpert/
http://www.vmware.com/

Virtual Storage Tiering – The NetApp Way

2011-06-01T21:40:00.001+10:00

Storage presents many and varied challenges. One such challenge that the industry has been dealing with recently is that if Auto-Tiering. The concept is relatively simple and helps customers address the needs of ever changing data and performance needs. Customers are now recognising that performance benefits can be enhanced with the ability to understand what data is being served at any given time. Auto-tiering is the process of identifying hot blocks of data that reside on your storage arrays and serve them out of a layer of super-fast disk or even better flash type cache. Enough on the history lesson, let’s fast forward and introduce NetApp’s Virtual Storage Tier.

Virtual Storage Tier is designed to identify what we like to call ‘hot blocks’ of data that is being read from disk. The goal of such is to be able to respond to changing I/O demands on a storage system in conjunction with providing efficient (there’s that word again) use of existing capacity. It is important to note that Virtual Storage Tier which is an extension to NetApp’s read cache (Flash Cache) which can exist in every controller regardless of size or cost. It is another one of NetApp’s defining technologies and one which sets them apart from the competition in my opinion. If you think about it every system caches, we are always pulling information from disk into system memory, but only NetApp allows that system memory to flow over into cache thus extending the ability for workloads to be served from cache with lower latency and improved I/O at a fraction of the cost of disk. NetApp cache can deliver hundreds of thousands of IOPS from a small amount of dumb, inexpensive disk which compared to other methods such as many expensive, slower disks is a massive benefit.

NetApp’s idea was that such a technology should be completely dynamic and self- managing which we will explore a bit later on. Keeping in line with NetApp’s Unified Architecture, such a service had to be completely scalable across all SAN and NAS protocols which is the NetApp mantra. Further to that, NetApp realised that there may be different disk types or disk tiers if you like on any controller or pair of and a tiering service should be independent of disk type.

NetApp Virtual Storage Tier differs from some typical auto-tiering methods which will only promote data blocks to disk once they have been read many times. In that case once the hot data has been identified a lift and shift takes place where the blocks are re-read from disk and placed into cache. Virtual Storage Tier works differently by tagging any read block for promotion to cache when it is first read from disk. It never actually moves blocks between disk tiers. There is a complex set of algorithms that determine whether data blocks are promoted based on their propensity to be read frequently. Configuration of the Virtual Storage Tier is as simple as enabling the feature and can be configured per volume to ensure that specific workloads are classified as high or low cache candidates although this is not required for Virtual Storage Tier to work, but provides flexibility for your environment.

The benefits of such a dynamic, real time solutions means that once hot blocks are identified they are pretty much already in cache and do not have to be moved in and moved back when they become stale, thus saving controller cycles. In traditional auto-tiering the process runs as a scheduled task when the controllers are quiet or as a background process to help mimise the impact on the controllers and the data being served. Virtual Storage Tier works at the 4k block level in conjunction with everything WAFL does so the efficiency in which blocks are identified and promoted is quite granular, accurate and timely which is very important.

But wait there’s more. Quite often I see and hear of customers loading up on expensive SAS aggregates in order to provide better tier 1 performance. Not only do these shelves take up allot of space in a rack but they also don’t give you bang for buck as far as storage capacity and in my opinion I think more and more we don’t get the performance level we often expect from them. Lately power and physical space have become a huge issue for customers trying to scale their datacenters. Sure virtualisation is helping all that, but data growth isn’t slowing. Until there is a cheap 2tb SSD on the market that replaces my tier 3 / 4 disk I think we’ll still have a problem. I saw some figures recently on power utilization in the datacenter kw/tb measured between different vendors and across tiers and let’s just say the NetApp numbers were outstanding however I cannot divulge the details sadly for reasons of privacy. With Virtual Storage Tier we can increase performance whilst decreasing our backend spindle count and hence saving power. NetApp internally have some numbers around this however it matters not, it’s obvious anyway.

NetApp’s message is all about storage efficiency and their goal is to always integrate their processes and technology with the other features. Virtual Storage Tier is no different in this case. It benefits from NetApp technologies such as de-duplication. For instance think about how 5 blocks on disk deduped would reside in cache, not as 5 blocks but as 1 as the blocks have already been deduped, so every time a read request for any of those 5 blocks hits the cache only a single cached blocked would be served, now that’s efficient when you think about the amount of blocks in a volume. I’ll let you ponder the ramifications of that one. This one of the reasons NetApp can place entire workloads inside a 16TB cache.

NetApp have tested many different workload types with this technology and have seen some significant improvements in performance. There are many real world scenarios that can benefit from this type of implementation. Of course there are some situations where data that is not cache friendly. In that instance shifting the data to a different disk tier is generally the most widely accepted solution. With Data ONTAP 8.01 we can move such data around transparently with DataMotion. The same release of Data ONTAP also allows the capability to utilise shelves of SSD disks in the rare event that caching is not an option however the cost can significantly increase at this point. In many cases SAS disks can be a suitable and more cost effective alternative.

Hopefully now you have an understanding of how Virtual Storage Tiering plays out and how it could be used to help you or your customers be storage efficient.