blog.aarondelp.com

Cisco UCS QoS vs. HP Flex-10 vNICs in VMware

noreply@blogger.com (Aaron Delp) — Mon, 15 Mar 2010 20:33:00 +0000

This post will be more conceptual than technical. I recently was asked how Cisco's UCS & HP's Flex-10 network design approaches affect vSphere designs. Even though the industry is moving towards a unified 10GB fabric, there are different ways to move data through this big "pipe" and still ensure/prioritize delivery. As you would guess, Cisco and HP approach this problem very differently. Cisco takes a network centric approach to the problem and HP takes a server centric approach to the problem.

HP's Flex-10

HP Flex-10 takes a 10GB connection and carves it up into multiple virtual NICs. The size of the "pipes" can be turned up and down to match the amount of bandwidth needed for the NIC. Think of it as placing smaller pipes in the big 10GB pipe. This approach is great for vSphere admins because the virtual switches in vSphere can be configured to look just like they did with a bunch of 1GB links into the server. The transition to this technology is seamless for the vSphere administrator. I'll borrow a diagram from Barry's awesome article on Flex-10. If you haven't read it, please do!

What is the down side to this method?

The down side to this approach is by placing multiple pipes within the larger pipes, you have now placed a CEILING on how much data can pass through that particular pipe. Let's say you present a 1GB vNIC to vMotion and during a vMotion it would be to your advantage to have access to more bandwidth. Too bad, 1GB is all you will ever get.

Cisco UCS's QoS

Cisco UCS uses a method known as Quality of Service (QoS). Most of us "server guys (and gals)" have no idea what this is. Here is how I have come to understand it. If this is wrong, please correct me. Network traffic is given a priority and this priority kicks in WHEN THERE IS CONTENTION on the network. So, instead of smaller pipes inside a large pipe, you have more of a priority system in place to guarantee certain levels of service. Think of this as a FLOOR model. You can have as much as you want as long as everyone else gets their minimums (they get their quality/guarantee of service). If something needs to spike and there is room, it can spike and then return to normal. Here is a diagram of our Cisco UCS with traditional switches. This isn't 1000v but you get the idea. As you can see, two big 10GB pipes into the virtual switches instead of smaller pipes into multiple virtual switches.

As the vSphere administrator, this looks very different from my old multiple 1GB links into my multiple virtual switches!

What is the down side to this method?

At this time, QoS for Cisco UCS appears complex to configure and represents a shift in thinking for the vSphere administrator.

How is the QoS implemented for Cisco UCS and VMware?

That is a very good question. I can't seem to find any documentation on how to actually do this yet. I'm sure there is a Cisco internal doc somewhere but I haven't found anything public that lays out the hardware that is needed (do I need 1000v or Palo for this, can I use a CNA and the standard switches?) nor have I found a "cook book" that documents how to properly make QoS happen in a vSphere environment. I'm sure this will happen in time and if you have a link, please leave a comment!

Which is better?

It depends on your point of view and the comfort level of your team. I can easily see advantages to both approaches. One is easier to implement, the other appears to be a more elegant (but complex) solution. Cisco has once again brought a disruptive technology to the table that can't be ignored. What are your thoughts?

Virtualization Podcast Directory

noreply@blogger.com (Aaron Delp) — Mon, 15 Mar 2010 17:57:00 +0000

I have been a Podcast junkie for years. I've subscribed and unsubscribed to countless feeds over the years. Up until recently, there has only a handful of podcasts with a virtualization focus. I have noticed a nice uptick in this number recently. So, in the interest of spreading the love to everyone out there, here is a list of the virtualization podcasts I am subscribed too. As always, I'm behind on my episodes but I hope to catch up soon! If you know of any others or if you have any additions or corrections, please let me know!

VMware Communities Roundtable - Weekly VMware podcast hosted by the honorable John Troyer
APAC Virtualization Roundtable - For all those folks on the "other side of the world" that aren't able to make the VMware Communities Roundtable, hosted by Andre Leibovici
Infosmack - Hosted by Greg Knierieman and Marc Farley
Virtualization Security Roundtable - Hosted by Edward Haletky (Texiwill)
Virtumaia Podcast - Hosted by Rich Brambley and Marc Farley
Mike Laverick's Chinwag - Hosted by Mike Laverick

Looks like Marc is the Leo Laporte of the Virtualization Podcast world!

VMware Lab Manager Install Notes and LDAP Import

noreply@blogger.com (Aaron Delp) — Thu, 04 Mar 2010 22:01:00 +0000

Setting up Lab Manager can be a little complex. It isn't as straight forward as some of the other VMware products so I wanted to provide some tips and tricks to get it all up and running.

Things you will need prior to Lab Manager Installation

In vCenter, create the datastores, virtual switches, and Resource Pools that you will need. The Lab Manager (LM) install will detect them at install and this will make configuration MUCH easier
Create all groups and users that you will need in either Active Directory or LDAP
If you will be using ip pools, define a block of static ip's ahead of time!
The Lab Manager server is currently a Windows 2003 based server. It can be virtual and on the same ESX hosts that it will be controlling. If you do this, DON'T name it lab-manager. If you do, you will get an error during installation because the install tries to create a folder in vCenter called lab-manger. You will have to rename your virtual machine to proceed. Also, you will need to change the speed of the vmxnet3 NIC per Jason's article here.
Make sure both forward and reverse DNS lookup work between the vCenter server, LM server, and all vSphere servers
The LM Server requires IIS 6.0 and .NET 2.0 to be installed. IIS MUST be installed before .NET 2.0
DON'T put the LM Server into the AD Domain. VMware recommends against this even if you are importing users and groups from AD into the LM Server. I asked why at Partner Exchange and I was told because it isn't needed and changes to AD could mess up the LM server.

LDAP/AD Integration

Integration with Active Directory or LDAP is the key to Lab Manager. Lab Manager allows you to create single users on the box but NOT groups. This makes security and configuration VERY difficult. At the same time, the LDAP integration leaves a little to be desired in the implementation. Here's how to do it:

From the Lab Manager Interface, On the left hand side choose Settings and click the LDAP tab:

Once that is complete, you are ready to import groups. Click the Import Groups Button.

Here's the magic. Because the group and users have already been created in Active Directory, you can choose the group and assign it to the users role (the default role is read only so be sure to change it). All users in this group are now Lab Manager Users

A few interesting notes about this import process. If you look at the group once it is imported and no one in the group has logged in yet, the group appears blank! This threw me for a little bit. I expected it to populate with the users at creation time. Instead the list populates at each USER's FIRST LOGIN! My group has three users in it total. As users log in, they will populate the group and also appear on the Lab Manager's Users list. Here are a few screenshots as I logged in my test users.

Lab Manager Group with only first test user logged in:

Lab Manager Users Pane with two users created from login:

Look for more articles as I get everything set up!

IBM's New eX5 Server Announcments

noreply@blogger.com (Aaron Delp) — Tue, 02 Mar 2010 13:02:00 +0000

I wanted to tell everyone about the new server lines IBM announced today. I attended IBM Business Partner training on this a few months ago and the products are impressive. I was under NDA until today to speak about anything. I can talk about the IBM products specifically but I'm still not able to talk about the Intel Nehalem EX chipset. I will have in depth posts of the EX chipset when it is officially released. Also, I am writing this from notes taken a few months ago so a few things might be slightly off. If you see a mistake, please let me know and I will correct it!

As always, Kevin does an awesome job of laying out the products (and has some great pictures) so head over to his site for an introduction. Hot off the press, Kevin has another article on just the X5 blade here.

Here's the basics:

The servers contain Intel's yet to be announced Nehalem EX chipset. I can't discuss the details on that since I'm still under NDA. I will present what has been pre-announced by Intel.
The Intel Nehalem EX (Intel 75XX) was designed by Intel to be the 4 socket follow up to the previous generation, the Intel 74XX. This was the IBM 3850M2 and the HP DL 580 servers.
(My opinion here, don't fuss at me Intel and IBM) Intel intended the Nehalem EX to be a 4 socket architecture. IBM modified the architecture in cooperation with Intel for 2 socket servers.
IBM has released the following servers based on Nehalem EX:

2 socket rack server called the x3690 X5. It can hold two Intel 75XX processors and 32 memory slots
2 socket blade server called the X5 blade. This was a pre-announce so I can't talk much about it yet. One thing that will be cool about the blade is it will be "lego based". By this I mean you can buy one and snap on another for a 4 socket blade
4 socket rack server called the x3850 X5 and the x3950 X5. This will stack like the previous generation of 3850's and 3950's. Each 3850/3950 will hold four Intel 75XX processors and 64 slots of memory
Additional memory can be bolted on to any of the models above using an IBM exclusive attachment called the MAX5. This will be a 1U (for the rack servers) with 32 memory slots or 1 blade width attachment that will give you an additional 24 memory slots. It attaches directly into the Intel QPI (Quick Path Interconnect) bus for easy, low latency memory expansion of the models
If I remember correctly both the 3690 and the 3850/3950 will have 1 GB on board network ports but an Emulex card can be added to the systems to replace the 1GB with 10GB on board

What do we know about the Intel EX chipset and why do we care?

I will point you to links here and here. As I stated before, I'll have in depth analysis of the chipset when it is announced. The why we care part is actually really cool. There are some great advancements in the technology but there are also many things to make your life easier at time of purchase as well.

In conclusion, I'm very excited about the 2 socket offerings. They appear to be very innovative and exciting. I wasn't given access to any other vendor's early release information so I'm not even sure if anybody else will offering 2 socket servers based on Nehalem EX. Interesting times indeed...

Configuring Multiple Palo Adapters in the Cisco UCS B250 Blade

noreply@blogger.com (Aaron Delp) — Tue, 23 Feb 2010 21:43:00 +0000

I found out something interesting today that I wanted to share. Version 1.1.1 of the Cisco UCS Manager code only supports one Palo card on the B250 at this time. Here is the line from the release notes and the link to the release notes.

Cisco UCS M81KR Virtual Interface Card. In a UCS B250 M1 Extended Memory Blade Server, the UCS M81KR Virtual Interface Card must go in slot 0 only and can not be mixed with other adapters.

This by itself isn't that big of a deal because I can't see many people wanting more than one Palo card at this time. The problem is that the Netformix configuration tool allows you to configure the B250 with 2xPalo cards with no errors. I have no idea if you can take the Netformix configuration and turn it into an order at this time (there might be an error check somewhere else in the system that would kick this out) but somehow I doubt it. Just in case, I wanted to let everyone know.

How Cisco UCS Deals with Split Brains

noreply@blogger.com (Aaron Delp) — Mon, 22 Feb 2010 12:01:00 +0000

This will be a short post this morning. I wanted to pass along how Cisco UCS deals with a split brain scenario. I'll start by explaining how you would get into a split brain scenario. In normal operations, one of the 6100's is the active brain and the other is the stand-by. A split brain in UCS would happen when both of the cluster interconnects betweenthe 6100 Fabric Interconnects are severed (the L1 and L2 ports). The active brain still thinks he is active and the stand-by no longer sees the active so he tries to take over. You now have a potential power struggle because both brains think they are in charge.

Luckily the Cisco UCS folks are way ahead of this scenario. They added logic to the Serial EPROM (SEEPROM) in the UCS chassis to resolve the situation. The odd number of chassis that are added to a UCS Domain act as judges during split brains. For example with four chassis, three are acting as judges. A marker is added to the SEEPROM on these chassis to make them quorum resources. To clarify this a little further bit more, if there is an odd number of chassis, all of them will be used. If there is an even number of chassis, it will drop the last one (n-1) so the number of quorum chassis will always be odd.

When the split brain is detected, both 6100's will immediately demote themselves and then claim as many of the quorum resources as possible. Whoever claims the most quorum chassis wins and promotes himself back to the active manager. The scenario would look something like the following. Pretty slick!

Buying an HS22V for VMware? READ THIS!

noreply@blogger.com (Aaron Delp) — Mon, 15 Feb 2010 05:14:00 +0000

I have had some interest from our customers in the new IBM HS22V Blade Server. There is a great overview of the details of the new blade over at Kevin's site here. I did find out one thing very interesting that I wanted to share. The HS22V is different from previous models because it will only take up to two 1.8 inch SSD drives. No hard drives here! That's a great advancement except for one thing; the list price of ONE of the drives is currently over $1600!!! This means over $3200 (list prices!) to load an operating system if you want a raid-1 set. That is a pretty high price. Here's a screenshot of the IBM configuration tool with the SSD drive.

But, if you are running VMware ESXi you have another option. Hidden in the other section (not the storage section) is an option for ESXi version 3.5 or 4.0. The best thing is it is only $75 list!!

This cost difference brings about an interesting choice for ESX based organizations vs. ESXi. How much are you willing to pay for that Service Console?

VMware PEX: Lab Manager Design and Implementation

noreply@blogger.com (Aaron Delp) — Thu, 11 Feb 2010 21:18:00 +0000

VMware PEX: Lab Manager Design and Implementation (TECHMGT0922). This is written as I sit in the session so it could be messy.

Architecture

Lab Manager Server (Windows based)
vCenter Server
One or more ESX 3.5 or 4.0 servers, ESXi 4 servers
TCP port 902/903 for virtual machine console access
Brower client to Lab Manager (LM) server is tcp443
TCP 5212 from LM Server to ESX servers and TCP 443 from lm to vCetner
Read the manual for all the requirments (installer checks for them and spits out errors)
Install a service account on lm server - page 14-15 of user guide as details on permissions needed
Pre-crreate resource pools if you want to use them so that it will pick them up at install time
Pre-create all virtual switches, distributed switches, etc.
Don't join it to the domain so nothing from AD would get pushed to it as a member server

Storage Design

LM uses Linked Clones to save disk space
Make sure the I/O can support your environment (just because you have the space doesn't mean you have enough I/O!)
LUN locking and limit of 8 nodes if using vmfs (NFS doesn't have this limit)
Understand the concept of disk chains and how they work (this isn't well documented)

Network Design

When Setting up LM server, create the default Physical network
Design Considerations for Networking: Physical Network vs. Virtual, Fenced vs. Non-Fenced - Will IP's be from an IP Pool, DHCP, or Static?
Physical network is nothing more than a connection out to a physical network
Virtual network - a network that is may or may not be connected to a physical network (could be on a different ip, vlan, etc) - A virtual network can be connected to a physical network if needed upon deployment
Fencing - The ability to create a fence around a configuration (group of vm's) so they are isolated from the rest of the network. If the fenced configuration needs to get out, it will do so through a NAT router (small Linux vm). In this case it would have an internal ip inside the fence and an external ip address outside the fence using the NAT router. This is great for machines and applications that all have the same ip. This way there will not be ip conflicts on the network.
Host spanning - fencing isolation was limited to one host in version 3, the ability to cross servers with vMotion is called host spanning. Host Spanning needs the Distributed Switch (and Enterprise Plus license to get Distributed Switch)
IP Pools - Takes a lot of ip addresses - if using fencing remember the the NAT router needs ip's as well

Fencing Considerations

Fancing can't use DHCP (DHCP can't cross the fencing to provide the address)
IP Static Pool must be used
At least one virtual machine needs to conntect to physical network (otherwise virtual network with no outside connection)
Fence policy is traffic In and Out, All Blocked, or Out Only. There is no In Only policy.

Be careful with outside communications if using fencing (many machines with same name but different ip's all hitting an outside source!)

Domain COntroller - member servers can be in a configuration with the same name as others as long as the machine pasword with AD doesn't expire (30 days by default). Otherwise, put a clone of the DC in the configuration and run it private to the configuration group
Domain Controller Clone - be careful - a cloned dc will come up with a .169 address because it detects one with the same ip address already out there. Best way to do this is clone the DC and completely isolate it from the production network.
SQL server - if outside the fence, what happens when multiple configurations hit it?? Maybe different instances of the same database on the same server - adds a little bit of a manual intervention to the process
Can create a workstation that is inlcuded in the configuration for the user to use as a workstation "in the fence"

Good Article: VMware KB1000023 - How to Backup the VLM Database

VMware PEX: Site Recovery Manger "Up and Running"

noreply@blogger.com (Aaron Delp) — Thu, 11 Feb 2010 16:59:00 +0000

This VMware Partner Exchange Session (PEX) was Site Recovery Manger "Up and Running" (TECHBC0321). I'm writing as I go so this might be a little messy.

This session will focus on the problems typically encountered during SRM implementations.

Required Components
2x vCenter servers
2x SRM Servers
Replication Product from the Storage Vendor
SRA (Storage Array Replication) from the Storage Vendor

Install Workflow
1. vCenter at each site
2. SRM Server (seperate server)
3. SRM needs a DB instance
4. SRA (Often the most complex and causes the most problems) - Install on SRM server

SRA's Function

Setup - Query for replicated luns, match luns to vm inventory
Failover - Automates promotion of LUNS at remote site, and
Testing - LUN Snapshot creation

What can go wrong during SRA install?

Not all SRA's are create equal. Each one is different and have different levels of effort put into the development Some require additional framework (Java JRE for example) Always read all release notes and the install guide prior to the install attempt
Always download a fresh SRA FROM THE VMWARE SITE NOT THE VENDOR SITE, many vendors change versions on a frequent basis
Whatever you do on one site, do it on the other site
When configuring SRA at the protected site, it may fail if not all components are installed at the recovery site (not configured, just installed)
What if no datastores appear but the SRA seems to be installed OK? This is because the datastore doesn't have any vm's on it
Always verify you have all the needed license features on BOTH storage systems to fully support replication in BOTH directions

Design Considerations

Disparate networks (re-ip of servers) - Most Common
Stretch vlans (no re-ip of servers) - Less Common
DNS services
Active Directory services - Could be dedicated for testing and failover or same production AD
Considered Applications with Hard Coded IP's
Remember Default Gateway and Subnet Mask
When performing a recovery, the less changes the better (DOC-1491 in VMware Communities
SRM Supports RDM's but it isn't recommended
If using multiple virtual hard disks, make sure both of them are replicated (or exist) at both locations
SRM does not support replicating virtual machines with snapshots
Need port 80 https tunnel between sites for site pairing (it is encrypted but travels on port 80 instead of 443 to make security easier
150 protection groups / 1000 protected vm's
A protection group can hold consist of datastores if a virtual machine spans datastores

VMware PEX: Reliable vCenter Database - Operations, Management and Troubleshooting

noreply@blogger.com (Aaron Delp) — Wed, 10 Feb 2010 22:31:00 +0000

I was finally able to attend a VMware Partner Exchange (PEX) session that I was able to discuss. This session was Reliable vCenter Database - Operations, Management and Troubleshooting (TECHBC0330). This is being written as I'm in the session so it might be a little messy.

vCenter at startup takes data from the Windows registry, the vpxd.cfg parameter file, and the vCenter database (VCDB).
The executable name of the service is vpxd
vpxd -p and -P are important because they are used to reset the password
Almost everything you do in vCenter requires interaction with the database. For example:

to start a vm - reads location in the db and send commands command

If vCenter fails - VMotion and DRS will fail but hosts and vm's will continue to run
vCenter won't start with corrupt or inaccesable database but it will run with an empty database
HA will be able to execute commands but won't have any "eyes" to see how to execute them

What workload that is running on the vCenter Server will determine how long your environment can be down. For example a farm containing mainly servers that aren't moving around and won't be restarted could go hours/days without a vCenter Server. On the other hand in environments with View, SRM, etc. the lack of a vCenter server will be noticed very quickly because machines won't be able to powered on.

Sizing and Location

You have the option of a physical or virtual machine and you can co-locate the VCDB or put the VCDB on a separate server.
Recommendations

vCenter and VCDB - virtual and co-location only to 40hosts or 400 vm's
if physical - must reatrt db's together, one could take down the other
The speaker recommended seperate virtual vCenter and db servers with an anti-affinity rule to disable both vm's from being on the same hosts. (I'm not sure how I feel about that in the case of power failures)
If the database server is virtual, you can take a db backup by cloning the vm

Protecting the vCenter and VCDB's

Can be protected many ways (too much information to fast to list) but methods included physical rebuild, VMware HA, vCenter Heartbeat, MSCS, and FT

Registry Components

The DSN Information is held in the registry: HKLM\Software\VMwareInc.\VMware Virtual Center\DB
Four objects of value under that:

1 = DSN Name
2 = Login ID
3 = password (encrypted)
4 = driver being used

Database Structure Components

The VCDB is mainly performance information (over 80% of the database typically)
The other information is the configuration of accounts and security information for vCenter
All db tables have the prefix VPX_ - It is NOT recommended to use the tables directly!

Why VMWare's VMmark Scores Have Become Useless

noreply@blogger.com (Aaron Delp) — Sun, 07 Feb 2010 14:00:00 +0000

Well, it was bound to happen. Every time an industry benchmark standard comes out, the manufacturers eventually figure out ways to "cook the books". I've seen a LOT of FUD flying around from both HP and Cisco lately about VMmark scores and I have been asked a lot of questions about both platforms. After a taking close look at the scores, I'm ready to throw in the towel.

Before I go further, take a look at the VMmark, 8 cores scores posted here. You will see the Cisco B200 Blade is on top right now (of the major vendors, I don't count Fujistu, sorry Fujistu) with 25.06 and the HP is next with the BL490 24.54. A couple of points:

What is the different between really really fast and really really really fast??

What is the difference between 25.06 and 24.54. Maybe 1-2%? Honestly, not much if they both meet your needs and you won't be pushing them to their limits. I'm sorry but that is within a margin of error and/or the test could be reconfigured by everybody to meet the score. At the end of the day both of them will meet your needs very well and the title of "fastest blade" means nothing!

Both Cisco and HP sell "big memory solutions" but they are no where to be seen!

Take a look at the memory in the details for both of them. Both the HP 490 and B200 use 96GB of memory. Where is the B250 with the larger memory footprint? Where is the BL490 with either 144GB or 192GB of memory? You will also notice that the HP BL490 memory is running at 1333Mhz and the B200 is running at 1066 Mhz. Since there is no big jump in performance numbers the VMmark score isn't memory bandwidth bound or HP would have had an advantage. I suspect (although I don't have proof) that the VMmark score is now CPU bound and any memory above 96GB doesn't help the scores. I further think (again, no proof) that the test isn't pushing the maximum memory bandwidth because there is no change from 1333 Mhz to 1066 Mhz. It would be interesting to see if the drop to 800Mhz by HP would be noticed in the scores.

Cisco is using an EMC SAN with SSD's on the back end!

Take a look at the EMC Storage section on the Cisco benchmark. They are using an EMC CX-240 with SSD drives! There is NOTHING wrong with this, SSD's are coming down in prices but they provide a clear, known advantage to the IOP's numbers that could easily be the sole reason for the 1%-2% increase. I'm willing to bet that if HP used the same storage configuration, they would produce similar scores.

Why didn't Cisco use the Palo card?

Cisco is using the Q-Logic CNA for the tests. Why didn't they use the Palo card? I suspect because it isn't "technically" released yet but that is the benchmark everyone wants to know about.

What am I trying to say here?

What I'm saying is that both HP and Cisco make great products and they will go to great lengths to make the other look bad. They are so close to each other from a VMmark score perspective that any clear difference can't be shown with the current test. Don't make a purchase based on a score!

HP Blades Offer a 16GB DIMM, With a Catch

noreply@blogger.com (Aaron Delp) — Sat, 06 Feb 2010 19:21:00 +0000

I found out something interesting on Friday, HP is offering a 16GB DIMM in their blades! My first thought was wow, that sucker is gonna be expensive (and it is!). But, after that I started to dig deeper as I always do, I found out something that is slightly disturbing. The 16GB DIMM is actually a quad rank DIMM and not a dual rank DIMM and it is only 1066Mhz speed. Many of you are saying... So what?

Well, this actually can make a difference from a design perspective. Take a look at the Memory tables in the BL460 and BL490 QuickSpecs and you will see what I mean.

HP BL460 G6 QuickSpecs
HP BL490 G6 QuickSpecs

Most of the DIMMs sold by the major vendors are dual rank and 1333Mhz Speed. Let me explain the concept of ranks first. According to the Intel Nehalem architecture, you can only have 8 ranks per memory channel. Each memory channel consisted of either 2 or 3 DIMMs per channel. I have more information on memory layout in this article I wrote on Scott Lowe's site awhile back. I never really discussed ranks because it was a limit you didn't hit. You were only using either 4 ranks (2xdual rank dimms on the BL460) or 6 ranks (3xdual ranks on the BL490). Quad rank DIMMs blows the math out of the water. You can only put 2 on a memory bus to generate 8 ranks. This means the BL490 no longer brings extra memory capacity to the table. Both the BL460 and BL490 top out at 192GB with the 16GB DIMMs.

Memory speed is the next issue. If you are running a memory bandwidth intensive application, you will expect about a 7% boost in performance by keeping the memory speed at 1333MHZ instead of dropping down to 1066Mhz. Because the maximum speed of the 16 GB DIMM is 1066Mhz you will never reach a 1333Mhz speed. Furthermore, populating both slots in the memory channel (the max of 12 DIMMs) drops the speed from 1066 Mhz to 800 Mhz. The performance drop from 1333 Mhz to 800 Mhz is over 30%!! This leads to an interesting trade off of memory capacity vs bandwidth speed.

While I applaud HP for thinking outside the box and bringing a 16GB DIMM to market, don't assume it is the same DIMM as the others. Remember, "One of these is not like the others...."

My First Race (with Dougnuts)!!

noreply@blogger.com (Aaron Delp) — Sat, 06 Feb 2010 17:54:00 +0000

Today I ran my first race! I am very excited but this race also requires an explanation and some pictures. A few years ago some NC State University college students started a challenge. You had to race from the Bell Tower on campus to the local Krispy Kreme (about 2.2miles), eat a dozen doughnuts, keep them down, and run back, all in an hour.

I am training for my first 10k coming up at the end of March in Chrleston, SC so I thought this would be a good warm up for me. The event is local, it was the right distance, and it is a little crazy and different so I was in. I registered just in time because registration was cut off the day after I signed up at 6,000 runners. Krispy Kreme's from all over the state made over 72,000 doughnuts and trucked them in for the race. I signed up for the casual group meaning that I didn't have to eat the full dozen. I ended up eating three. I ran the race in 1:14 but that really doesn't matter because you stop half way through the race and hang out, eat doughnuts, drink water, and just enjoy the atmosphere. Some highlights of the race to prove it is a little different:

Mother and Daughter with KK T-shirts and the words "Family Bondin Southern Style" written on them
The quote "3 hours sleep, 9 beers, 12 doughnuts, I'm starting to feel bad"
We had the "privilege" off seeing two challengers "lose" their dozen doughnuts
Got to see Fat Bastard, Elvis, Superman, Wonder Woman (it was a buy), Elmo, Cookie Monster and many people running in just shorts or underwear in 30-40 degree weather
The famous quote "I'm never eating another doughnut again" more than once

Here are some pics:

Cisco UCS - How Many FEX Uplinks Do I Need?

noreply@blogger.com (Aaron Delp) — Thu, 04 Feb 2010 05:23:00 +0000

As a Data Center Architect, I have a constant need to know not just HOW to do things, but WHY to do things. As I dig deeper into the Cisco UCS system I find the concept of FEX (Fabric Extenders) very facsinating. The number of FEX uplinks may not seem like much, but a couple of cables can have a very significant impact on the design of a UCS system. If need a refresher on how a UCS system is set up, please see my first article for more information.

What is a Fabric Extended (FEX)?

In very simple terms, the FEX serves as the "pipe" between the blades in a UCS chassis and the 6100 Fabric Interconnects (FI's). Each FEX has a maximum of four 10GB connections. Think of it this way, you can "choose" your bandwidth back to the FI's in one, two, or four 10GB increments (the three uplink option isn't supported). If you plug in one, you get 10GB bandwidth spread over 8 blades. Need more bandwidth? Plug in the second to get 20GB. If you really need the maximum then you can go for all four connections for a total of 40GB per FEX. Remember, each UCS chassis contains two FEX's and each FEX is connected to one (and only one - don't cross connect them!) 6100. If you plug in all eight connections, you achieve a maximum of 80GB to the chassis or 10GB per blade. If you are interested in how the traffic flows from the blades to the FEX port (referred to as pinning), here is a great link from Rodney Haywood that details this relationship. Here is a picture of a FEX close up:

Here is how you would cable them (don't cross connect them!!)

Why does this matter?

FEX uplinks directly affect three different areas:

The bandwidth from the 6100's to the chassis
The number of chassis supported per pair of 6100's
The number of vNICs supported by the upcoming UCS Palo blade card.

Let's tackle each of them one by one

1. Maximum Bandwidth per Chassis based on FEX Uplinks -> (FEX's * Uplinks *10)

To calculate the amount of bandwidth available to a chassis: (FEX's * Uplinks *10). So, If I have two FEX's, each with 2 uplinks, I have 40GB at my disposal (2*2*10).

2. The number of chassis supported per 6100 is inversely proportional to the number of FEX uplinks -> ((total ports - uplinks) / FEX Uplinks per chassis)

Every time you use more than one FEX uplink, you actually reduce the number of chassis you can plug into the system. Let me use a simple example.

Let's say you have a 20 port Cisco 6120 with the 6x10GB module in it for a total of 26 ports. To make the math simple, you decide to dedicate six 10GB links for northbound (out of the chassis) traffic. You can support up to 20 chassis by using a single FEX uplink per chassis. What if you need a second FEX uplink? The number of chassis goes down to 10 because you need two per chassis but you only have 20 ports to physically plug into. If you need need 4 FEX uplinks, then you can only support 5 chassis per 6120.

To calculate the maximum number of chassis, this is the formula to use: (total ports - uplinks) / FEX Uplinks per chassis. To use the example above, a 6120 with 4 uplinks yields 5 chassis ((26-6) / 4 = 5)

UPDATE: As pointed out by UnixPlayer, what about uplinks out of the chassis?? Great question and I admit it was late when I wrote this. I have now updated this to include uplinks. You of course need some uplinks or this will happen to you!

3. The number of vNICs and vHBAs supported on a Palo card is proportional to the number of FEX uplinks -> ((15*FEX uplinks)-2)

As pointed out by Kevin on his article, the number of uplinks determines the number of vNICs the Palo card can present. The thoeretical maximum of the card is 128 but only 58 can be used currently. This is for both vHBA's and vNICs.

As you can see, there are some interesting design choices to be made based on bandwidth, scalability, and virtual I/O. I really like the customization ability of the UCS system to tailor to the requirements of the customer but it is also very important to understand the relationships presented above when designing the system.

Cisco UCS Information for "Server People"

noreply@blogger.com (Aaron Delp) — Tue, 02 Feb 2010 14:10:00 +0000

I've been working with the UCS equipment as time allows for the last few weeks. I've also had the privilege to visit Cisco TAC for UCS here in Raleigh, NC to pick their brains a bit. Here is a quick bullet list of some features that I found interesting from my server based perspective.

The amount of 10GB over subscription from the UCS chassis to the 6100's (Fabric Interconnects) is proportional to the number of uplinks. There are four connections maximum per FEX, per chassis. One uplink will provide an 8:1 ratio, two uplinks a 4:1 ratio, and lastly four uplinks for a 2:1 ratio. Three uplinks is not supported. (My next article will be an detailed article on the FEX's)
This may seem obvious to the Cisco folks but it wasn't to me. The 6100's are "backwards". They are designed to be mounted in the back of the rack so all cabling is towards the rear of the UCS chassis. Cooling is "front to back" on the 6100's to match the UCS chassis.
You can "Mix and Match" adapters cards on each blade because the uplink is a common 10GB fabric. This means if you only need a few Palo cards in a chassis and maybe CNA's on the rest, you can do that. Service Profiles won't be compatible but you do have that flexibility
Only Cisco memory is supported on UCS blades. No 3rd Party memory
The UCS Chassis needs 2 power supplies. It ships with zero. 3 power supplies provide N+1 redundancy and four power supplies provides N+N. As more power supplies are added, the load is distributed evenly across each power supply
The UCS Chassis has 8 fans but needs 4 to operate so it is N+N redundant
When a Chassis is plugged in, multiple blades are powered up in serial fashion to prevent an in-rush current spike that could blow the circuits. This has been a problem with other blades customers of mine in the past.
The 6100's are active/active for 10GB data but are active/passive for management of the chassis. At any given time one 6100 is active and constantly passing information over the L1/L2 connections to keep the passive management module up to date
The FEX connections on the back of a UCS chassis CAN'T be cross connected to the 6100's. I'll have more information on this in the next article
The UCS Manager allows up to 4 KVM connections at one time. I'm still checking if this is 4 per UCS Manager, 4 per chassis, or 4 per blade (If you know, please leave a comment!)
The maximum number of vNICs the Palo card can present is 56 and is dependent on the number of FEX links from the chassis to the 6100's. I'm still getting details on this information and I will post this in the near future
The 6100 Fabric Interconnects are licensed per port. The 6120 comes with 8 ports licensed and the 6140 comes with 16 ports licensed. Additional ports must be purchased individually, kind of like an FC switch. This applies to both Northbound and Southbound traffic (I REALLY don't like this!!!)
Smart Net needs to be purchased on the 6100's, each chassis, the blades, and the expansion modules in the 6100's. Smart Net lasts for one year so if you want three year coverage, you need to purchase quantity 3 of Smart Net item for each. This is VERY different from HP and IBM servers.
The first three chassis in a UCS domain (managed by the same 6100's) communicate via an SEPROM to verify and prevent a split brain scenario in the event of the 6100's losing communication
The UCS Manager includes the ability to e-mail alerts and all "call home" to Cisco, much like a NetApp storage system

Cisco UCS vs IBM and HP - Where are the Brains?

noreply@blogger.com (Aaron Delp) — Mon, 25 Jan 2010 22:47:00 +0000

UPDATE: Thank you to everyone for the great comments! Please look for the updated sections that I have highlighted below. I have learned a lot from everyone and I will continue to update this as more information rolls in. I welcome any and all comments. Thank you!

As many of you know, my company recently acquired some very nice lab gear for customer demonstrations and proof of concept work. Many of my peers already know the UCS systems inside and out but I really need hands on to "get it".

As I learn the UCS system I will share my experiences here. My perspective is to share what is different (good and bad) about UCS compared to the IBM and HP Blade products. Before anyone asks, I will only be covering IBM and HP. If you have additional experiences, please share them in the comments. I also have no intention of picking sides. At the end of the day I sell and support all of the above systems and I can get the job done with all of them. They all have their own unique strengths and weaknesses that I intend to highlight.

In case you aren't familiar with what UCS is, I suggest you take a look at Colin's post over on his blog. He does a great job putting all the pieces together. Plus, I'm going to steal a few of his graphics. (thanks Colin!!)

A UCS system consists of one or more chassis and a pair of Cisco 6120 switches that provide both the 10GB bandwidth to the blades as well as the management of the system. The last part of that statement is the key to understanding how UCS is currently different from the competition. I define management in this example as the control of the blade hardware state. This includes identification, power on, power off, remote control, remote media, and the virtual I/O assignments for MAC and WWPN's.

By moving the management from the chassis level to the switch level, the solution can now take advantage of a multi-chassis environment. Here's a simple modification of Colin's diagram to illustrate this point.

(UPDATED!) What are the limitations to the Cisco UCS model?
Someone asked in the comments how this scales. Honestly that was a great question. I'm still learning Cisco and I was wrapped up in making it work. Let's take a look at that. Currently you can have up to 8 chassis per pair of UCS Managers (Cisco 6100's). That number will increase in the upcoming weeks and eventually the limit will top out at 40. But, the more realistic limitation is either 10 or 20 depending on the number of FEX uplinks from the chassis to the 6100's unless you are using double wide blades. If you don't understand what that means right now, don't sweat it. I'll be posting about that shortly.

(UPDATED) What if you need to manage more than the chassis limitations today?
If you need to go above the limit, then you have two options. The first option is to purchase another pair of 6100's to create another UCS System and they will be independent of each other. The second option is provided by BMC software. This will allow you to manage more chassis and the solution also provides additional enhancements. I admit I know little to nothing about the product so I'll just post the link from the comments and you can take a look. The brain mapping for that would like this.

How do you get into the brains?
Each 6120 has an ip address and both 6120's are linked together to create a clustered ip address. The clustered ip is the preferred way to access the software. The clustering is handled over dual 1GB links labeled L1 and L2 on each switch. They are connected together like this:

Cisco uses a program to manage this environment called creatively enough, Cisco UCS Manager or UCSM. To access UCSM, point a browser at the clustered ip address. Once authenticated, you will be prompted to download a 20MB java package (yes it is java, yuck!). Here is a pic of ours with both chassis powered up.

Notice that both chassis are in the same "pane of glass". This allows for management of all the blades from one interface and the movement of server profiles (covered later) from one chassis to another within the same management tool.

How does this compare to IBM?

IBM is a two part answer.

IBM Part One - Single Chassis Interface in AMM

IBM uses a module in each BladeCenter chassis called the Advanced Management Module (AMM). There can be up to two AMM's in each chassis. If there are two AMM's, one is active and the other is passive. They share the configuration and a single ip address on the network. In the case of failure of the primary, the passive module becomes active and communication resumes on the original ip address. The AMM will control power state, identification, virtual media and remote control out of the box. Virtual I/O (both WWPN and MAC) is an additional purchased license in the AMM. The product is called the Blade Open Fabric Manager (BOFM). I don't know if BOFM supports 10GB but I know it supports 1GB ethernet and 2/4GB FC. This is what it would look like with brains in place:

As you can see, each chassis is managed individually. In my experience, this is the most common configuration I have seen.

IBM Part Two - Multiple Chassis Management with IBM Director

IBM does have a free management product called IBM Director that can pull all this together into a single pane of glass. The blade administration tasks are built into the interface and virtualized I/O is handled through the Advanced BladeCenter Open Fabric Manager. Advanced BOFM is a Director plug-in and is a fee based product. Logically it would look something like this:

The downside to this solution is you now have another server in your environment to manage. In my experience Director is a little flaky at times but I also haven't tried the newest version which is a redesign to address many of the issues.

How does this compare to HP?

HP is a two part answer as well. I haven't implemented HP's Virtual Connect over multiple chassis so I will ask that if you know this answer and can throw some links my way, please do and I will update this section.

(UPDATED!) HP Part One - Single Chassis Interface in Onboard Administrator (OA)

HP's approach is very similar to IBM. HP's management modules are called the Onboard Administrator and there can be a maximum of two in each chassis. HP is different from IBM because each module requires an ip address. At any given time one ip address is active and one ip address is passive. If you access the passive module on the network, it will tell you that you are on the passive module and instruct you connect to the active module. Like the IBM AMM, the OA will control all basic functions such as power state, identification, virtual media, and remote control. Like IBM, HP has a separate product for virtual I/O called Virtual Connect. Unlike the IBM and Cisco products, HP's Virtual Connect is implemented at the I/O module level. The only way to achieve virtual I/O is to purchase the HP I/O modules. HP's brain mapping is a little different than IBM because you can connect up to four chassis into one interface. Since you probably won't be able to power more than four chassis in a rack, think of it as consolidation at the rack level.

(UPDATED!) HP Part Two - Multiple Chassis Interface in HP Insight Tools

After you get to four chassis, HP Insight Tools need to be brought in to fulfill the needs. Based on the comments below it appears that two products will fit the bill. To manage the chassis and blade functions you will need Insight Dynamics VSE Server Suite and to manage the virtual I/O you will need the Virtual Connect Enterprise Manager product. Both the Insight Dynamics VSE Server adn the Virtual Connect Enterprise Manager is fee based.

Summary

(If you made it this far, I'm impressed!) Cisco's approach feels very "up to date". I really like the idea of not having to add another server (and additional fees for virtualized I/O) to the environment for management of the products. By moving all of the management centrally to the switches you are better able to see the environment and implement a multi-chassis/multi-rack solution. IBM and HP offer a similar solution that has grown over time but the roots of the interface are in single chassis/rack management. But, at the end of the day both IBM and HP offer a centralized management solution.

Thoughts? Concerns? Please leave a comment!

NetApp CIFs Tricks

noreply@blogger.com (Aaron Delp) — Mon, 25 Jan 2010 19:45:00 +0000

Yes, the Cisco UCS blog posts will start up "soon". Still putting the finishing touches on the first port. In the meantime...

I recently had to perform the following during a CIFs (Windows File Sharing) installation from a NetApp storage controller. The chances of me remembering this again aren't very good so I wanted to post it here for later. We had two issues that caused us some grief.

Issue #1 - For whatever reason when looking for a domain controller it wasn't "attaching" to the local domain controller. The system would ask for a list of domain controllers but then try to communicate with remote AD servers, some of which were behind firewalls. NetApp is nice enough to allow us to "pin" the storage to a preferred list of domain controllers to correct this behavior. From the command line, use the following commands:

cifs domaininfo - lists which domain controllers the NetApp is communicating with. The preferred list is a list you specify, the favored list is the list AD thinks are closet to you, and then the rest are listed.
cifs prefdc - This command allows you to populate a list of the domain controllers you want to communicate with first. More than one can be entered in the command seperated by spaces in the format: cifs prefdc add (domain) (dc1) (dc2) (etc...)
cifs resetdc - After a dc is added you need to reset the connection
cifs prefdc print - Shows the list

Issue #2 - The site admin wasn't a domain admin. This leads to many permission related issues because by default when a NetApp is added to AD only the local NetApp admin (created during CIFS setup) and the Domain Admins are in the machine administrators group. We needed to add the site admin into the Administrators group on the NetApp. This was achieved using the useradmin command. Here is the syntax: useradmin domainuser add (username) -g Administrators

After these two steps were complete, we were able to proceed.

That's A Lot of Hardware!

noreply@blogger.com (Aaron Delp) — Tue, 19 Jan 2010 15:21:00 +0000

Just a quick post today. As some saw on Twitter yesterday I will be getting my hands on some pretty impressive hardware. My company has decided to move our customer demo lab to our office and all the gear arrived yesterday. Here's a few pictures for now but we will be setting all of this up over the next few weeks. I will be posting some impressions and tips as I go. With my HP and IBM Blade background I am hoping to write a good bit on the UCS experience. In addition to the EMC NS-120, I am hoping to integrate our existing EMC NS-960 for some experience with that hardware as well. Should be interesting!!

Picture #1 - 2x Cisco UCS Chassis each with 4 blades, 2x Cisco 6120 Nexus, 1x Cisco Nexus 5020

Picture #2 - A LOT of NetApp disk shelves (NetApp controller not pictured)

Picture #3 - EMC NS-120 still in the box (but not for long!)

Installing NetApp VSC According to Best Practices

noreply@blogger.com (Aaron Delp) — Mon, 18 Jan 2010 13:53:00 +0000

If you haven't checked out NetApp's Virtual Service Console, you should. I did an article on it after NetApp Insight which is available here.

Vaughn recently posted on setting up VSC access to the NetApp using RBAC (Role Based Access Control) permissions. This procedure is not currently in the VSC manual.

Quick tangent: Creating RBAC for every product appears to be an ongoing trend within NetApp. Documentation exists for RBAC installation on SMVI (it's in the manual), VSC (link above), Snap Drive in a virtual machine, and I think there is a RCU writeup around but I can't find it right now. This is great from a security perspective but gets a little tedious if you are loading multiple products on the same NetApp controller, and double the pain if it is an HA unit! (HINT to NetApp, figure out a way to consolidate this please!!)

Let's say you were an early adopter to VSC and installed it per the manual. You probably used root as the user id and you never enabled SSL on the filer. If this is the case, you are sending the root password in clear text (Yikes!). Based on Vaughn's article we can easily go back and fix this.

Configure and Enable SSH on each NetApp Controller if not already enabled

From the command line you can use the secureadmin setup ssl and secureadmin status command as shown below. This can also be configured from FilerView -> Secure Admin

Create the role, group, and user on each NetApp controller. Enter each line from the command line

useradmin role add vsc-role -a login-http-admin,api-aggr-list-info,api-cf-get-partner,api-cf-status,api-disk-list-info,api-ems-autosupport-log,api-fcp-adapter-list-info,api-fcp-get-cfmode,api-license-list-info,api-lun-get-vdisk-attributes,api-lun-list-info,api-lun-map-list-info,api-nfs-exportfs-list-rules,api-qtree-list,api-snmp-get,api-snmp-get-next,api-system-get-info,api-system-get-version,api-volume-autosize-get,api-volume-list-info,api-volume-options-list-info
useradmin group add vsc-group -r vsc-role
useradmin user add vsc-user -g vsc-group

From the vSphere Client, go to the NetApp tab, Repeat the following for each controller

Right Click on the controller and click Modify Credentials

Enter the newly created vsc-user id and password, check Use SSL and click OK

Congratulations, you have just configured your vCenter Server to communicate with the NetApp systems in safe and secure way!

#vmtip Archive From Twitter

noreply@blogger.com (Aaron Delp) — Fri, 15 Jan 2010 04:26:00 +0000

For a few weeks now I have been posting VMware and storage related tips to Twitter. I have been using the hash tag #vmtip for each of them. I keep an archive of them in Evernote so I can remember what I have done but it isn't organized. This is simply an attempt to better organize them into categories. This won't be updated every day, but I will try to keep it somewhat up to date.

Last Update: January 14th, 2010

VMware Related Tips

#VMware tip: vSphereU1 increases the max# of vm's per host to 160 for up to 8 hosts (was 100), still only 40 vm's per hosts if >8 #vmtip
#VMware ESXi local boot only supported option today. ESXi Boot from SAN and PXE are both experimental right now (via @DuncanYB) #vmtip
Need to get data from a #VMware Workstation or ESX(i) vmdk? VMware Disk Mount Utility. It saved me this week! http://bit.ly/6rtY8e #vmtip

VMware vCenter Related Tips

Prior to loading #VMware vCenter, make sure you set the final machine name, static ip address, and domain membership! #vmtip

Virtual Machine Alignment Tips

#VMware tip: Windows 2008 vm's do not need alignment if created fresh. If it was upgraded from W2k03, it will be misaligned. #vmtip

VMware Lab Manager Related Tips

#VMware Lab Manager Tip: If using VMFS, there is a maximum of 8 hosts per datastore due to disk chains. There is no limit for NFS. #vmtip
#VMWare Lab Manager Tip: Lab Manager disk chains can not span volumes due to the linked clone technology #vmtip

NetApp Related Tips

#VMware on #NetApp tip: VSC will tweak ESXi installs for NetApp. Great since no NetApp Host Utilities Kit for ESXi #vmtip
#NetApp on #VMWare vmdk alignment tip: Windows Dynamic Disks, Linux LVM's and Citrix Servers can not be aligned with mbralgin #vmtip

NetApp SMVI Related Tips

When doing #VMware SRM on #NetApp and using SMVI, you CAN'T take a VMware snapshot as part of the backup! #vmtip
This appears to be undocumented: #NetApp SMVI backup of Windows virtual machine on IDE disk are not eligible for single file restore #vmtip

NetApp SnapDrive Related Tips

#VMware on #NetApp: When installing Snap Drive, check the install account is an admin on BOTH the server & filer before install! #vmtip
#NetApp SnapDrive 6.2 for Windows requires .NET 3.5 SP1 and 3 MS hotfixes (with reboot) BEFORE installation of SnapDrive. #vmtip

Creating VMware NFS Datastores on NetApp in 3 Easy Steps

noreply@blogger.com (Aaron Delp) — Wed, 13 Jan 2010 22:06:00 +0000

I am often asked by customers how to set up a VMware NFS datastore on NetApp storage. The first time I received the question, I pointed them to the NetApp vSphere TR. It turns out the information isn't currently in the document. I spoke to Vaughn about it and this was an over site that will be corrected. In the meantime, here is how I create NFS shares in 3 easy steps. I'm also taking screenshots from my lab for the first time, let me know what you think of the screenshot format vs. just a bullet list.

Step One - Create the volume on the NetApp system

Log into FilerView, Open Volumes, Click Add -> Click Next

Accept the default value of Flexible Volume and Click Next

Create a name for the volume, set the language type, and click Next

Choose which aggregate to create the volume in and click Next

Set the size of the Volume, please notice the pull down defaults to MB NOT GB!, I typically don't set a Snap Reserve but if you don't understand the implications of this, just use the default of 20%. Click Next

Click Commit

Click Volumes -> Manage to and you will see the newly created volume

If your NetApp system has been configured for CIFs, you will need to make a slight change to the Q-Tree type of the volume. Click Volumes -> QTrees -> Manage. If the QTree type is UNIX, skip to Step Two. If the QTree type is NTFS, proceed.

Click on the volume link (sim3_vmware_01 in this example) to get the following screen

Change the QTree type to UNIX and click Apply

Step Two - Create the NFS Export (Share the Volume)

Click NFS -> Manage Exports -> Click on the Permissions for the newly created volume

Make sure Read-Write Access, Root Access, and Security are all checked. Click Next

Click Next at the Export Path Screen but write down this path, you will need it later!

At the Read-Write Access Screen, uncheck the All Hosts box and enter the ip addresses of all the VMkernel ports for the vSphere server(s). NOTE: This is not the Service Console IP address, it is the VMkernel ip address that vSphere will use to "talk" NFS to the storage

Repeat this process for the Root Access Screen and click Next

Click Next at the Security Menu accepting the defaults

Click Commit. You are now finished configuring the NetApp System!

Step Three - Create the share in vCenter

From the vCenter Client, Click on a vSphere server and click the configuration tab. Click Storage, Click Add Storage

Choose Network File System and Click Next

Enter the IP address of the NetApp Storage, the path to the export that you wrote down from step 2, and give your datastore a name as it will appear in vCenter

You should now see your storage

UPDATE: I was hoping to stay away from the command line for this article. This was really designed for users that are just beginning to get their feet wet with NFS. But, as Mike pointed out, there is one command that should be run on each volume and this can only be achieved from the command line. It is outlined on page 37 of the 1.0 version of the TR. The command is: vol options (volume-name) no_atime_update on where volume name is the name of the volume (sim3_vmware_01 in my example). Thank you for pointing that out Mike!

A few final notes. Once all of this is complete I usually test read/write access by pulling up the datastore browser and creating a folder in the datastore and then deleting it. Also, if the datastore will be protected by NetApp's Snap Manager for Virtual Infrastructure then I will disable snapshots. This is all detailed in Vaughn's vSphere TR.

VMware Disk Mount Utility to the Rescue!

noreply@blogger.com (Aaron Delp) — Wed, 13 Jan 2010 12:46:00 +0000

I had a bit of a scare over the holidays. I usually keep two copies of my data at all times. One copy is on my laptop and the second copy is on an external hard drive at the house. Well, what happens if you are installing a new OS on your laptop (one copy gone) and as you are copying back all of your data, the external hard drive starts clicking and throwing up errors (two copies gone). Yikes!

I tend to "sip my own champagne" (I don't "eat my own dog food", too crude of a reference) so I run my corporate workstation in a virtual machine with VMware Player. All of my data was in one big 32GB vmdk file on the external hard disk. I cracked the case on the USB disk and mounted it on my home PC. The drive was recognized! I tried to copy the vmdk off to the c:\ so I could transfer it to the laptop. The transfer was VERY slow. It was going to take most of the night and the next day to copy but I needed my data NOW!

VMware Disk Mount Utility to the rescue! In case you aren't familiar with the product, you can download it here and the manual is here. It allows you to mount vmdk's from either Windows or Linux. There are some limitations as specified in the document but you can mount VMware Workstation as well as ESX virtual machines.

Using the tool I was able to mount the vmdk and just copy out the data I needed for work the next day. Over the weekend I was able to recover all the rest of my data. It took 3 days to copy 100GB worth of vm's and there were a few casualties. My ESXi machine and a couple of XP builds were corrupt and I will have to recreate them. I was lucky I got the data back but a big thanks to VMware for the tool!

Heading to VMware Partner Exchange?

noreply@blogger.com (Aaron Delp) — Tue, 12 Jan 2010 16:51:00 +0000

I will be attending the VMware Partner Exchange for the first time this year! I'm very excited and I hope to post some useful information so stay tuned.

I've decided to do a little social experiment via Twitter, we'll see how this works. If you are interested in getting together for drinks or dinner one night, please let me know via a reply on twitter and I'll add you to the being followed section of the list. Here's the catch, you will need to follow the list as well. Here is a link to follow the list. As we get closer to the conference I'll send out information to the list or we can just group argue over where to go. After the conference, I will remove the list.

Not sure when this will happen or how formal it will be, just seeing if there is interest (it's an experiment remember!)

UPDATE: To be clear, this is a "bring the person, not the company" event. Anyone is welcome but please don't plan on pushing/selling anything to this crowd. This is meant to be a social event only.

Cisco 4001i Nexus Switch for IBM BladeCenter in Depth

noreply@blogger.com (Aaron Delp) — Tue, 12 Jan 2010 12:37:00 +0000

I have been talking to a few customers recently about the Cisco Nexus 4001i Switch for the IBM BladeCenter. The product looks very nice and I have recently discovered some additional information that I wanted to share.

If you are unfamiliar with the product, head over to Kevin's site and take a look at this link and this link. He has some very good links from Cisco about the switch. In addition, I found a link to the IBM RedPaper on the switch here. For those of you that don't like links, here's the summary: It is a 20 port (14 down to blades, 6 uplinks), non-blocking 10GB FCoE capable switch utilizing the Nexus OS. The FCoE functionality is optional. To enable FCoE you need to purchase the FC Enablement Kit (IBM Part Number 49Y9983).

Here are some additional notes:

The 4001i is NOT an FC Forwarder, it is a FIP Snooping switch
You may have already noticed but this switch DOES NOT have FC ports. To talk FC you will need to go out of the 4001i into a Nexus 5k and break out the FC there
If using the Emulex Virtual Fabric Blade Adapter there is no vNIC functionality
The switch only supports Cisco SFP's/SFP+ cables and it doesn't ship with any. You will need to purchase them separately to go with the switch. They are NOT resold through IBM. Since there are 6 uplinks, you will need a maximum of 6 SFP's (or copper SFP+ cables) per switch.
The switch will support a 1GB connection down to the IBM Blade 2port/4port 1GB Adapter. I questioned why you would need this but on second thought I really like this. This way you can provide additional 1GB connections to a server that may not need 10GB without the purchase of additional 1GB switches. The fact that you can "share" a 1GB and 10GB CFFh slot is VERY nice!

I'll post more on the switch as I dig deeper!

NetApp Insight: SMVI 2.0 In Depth

noreply@blogger.com (Aaron Delp) — Tue, 01 Dec 2009 15:27:00 +0000

This is a little (alright, it's a lot) late but I still have a few Insight sessions to write up. Jack McLeod gave a great presentation on SMVI 2.0 and best practices. The presentation was very technical in nature and contained great information.

In case you aren't familiar, NetApp has a product called SMVI. SMVI stands for Snap Manger for Virtual Infrastructure and this is the second version of the product. SMVI is a software product that is loaded on a Windows server (the vCenter server as a best practice) and interacts with the vSphere server(s) to perform seamless snapshots of virtual machines in a state that can be recovered easily.

How is SMVI different from just taking a NetApp SnapShot?

The difference is SMVI will communicate with vSphere to take a VMware Snapshot that ensures the virtual machine is in quiet state. If you are using SnapMirror, SMVI has the ability to kick off the replication to a remote site from within the application as soon as the backup is complete. In addition, SMVI contains the ability to perform single file restore and individual vmdk restores. More information on the product can be found in the NetApp SMVI TR-3737. This is currently for version 1.0 but will (hopefully) be updated for 2.0 shortly.

What are the major enhancements to SMVI 2.0 over the first version?

There are a few major enhancements to SMVI 2.0. The full list can be found in the SMVI 2.0 Release Notes.

Single File & VMDK Restore - You can now restore individual files within a vmdk (Windows vm's only) as well as individual vmdk's from a SnapShot instead of restoring the entire volume
The ability to optionally include Independent Disks - In version 1.0, Independent Disks were ignored by SMVI
Trigger pre/post/failure scripts from the backup job
NetApp SnapShots created with SMVI now have a more consistent naming scheme
For virtual machines that span multiple datastores, SMVI now includes the ability to exclude certain datastores
Application consistency for VSS aware applications (provided by the VMware Snapshot)
VMware snapshots on a volume are now run in "groups" instead of sequentially trying to snapshot every vm on the volume one at a time. This parameter can be modified.
The ability to NOT take a VMware Snapshot as part of the process. This is currently necessary for integration with VMware SRM. More on that in an upcoming SRM on NetApp article
Cleaner Interface with Wizards for better user experience
FlexClone license is needed for restore capability on NFS, SnapRestore license is needed for VMFS and NFS

SMVI 2.0 Recommended Best Practices

When installing SMVI and adding the storage, create an account on the storage subsystem for smvi use. DO NOT USE ROOT.
Install SMVI on the vCenter Server. If you have more than one vCenter server, install SMVI on each server. Only one SMVI server per vCenter server is supported
Enable SSL communication on all Storage Systems to encrypt communications
Install the SMVI Repository and config files on shared storage for protection and easier recovery
Whenever possible, take datastore level snapshots to reduce the total number of snapshots
Be careful with aggressive DRS settings. A vMotion during a virtual machine backup can lead to backup "implications"

SMVI 2.0 Limitations

Virtual Machines may reside on different datastores (NFS & VMFS), but all datastores must be on the same controller
SMVI does not support traditional volumes, NetApp FlexVols only
Virtual Machines must be restored to their original datastore
You can't restore a datastore that has been removed from vCenter
Only Volume SnapMirror is supported, Q-Tree SnapMirror is not supported
The SnapMirror relationship must already be created and the baseline transfer complete
SMVI will only initiate single path SnapMirror configurations
Restoration for virtual machines that span three or more VMFS datastores is not supported
SMVI can't take VMware snapshots of virtual machines that have iSCSI software LUNs with the Microsoft iSCSI initiator or NPIV RDM LUNs
Single File Restore is only available on Windows virtual machines currently
VMware SRM is not compatible with SMVI using VMware Snapshots. This feature MUST be disabled for SRM to function