So, today’s article is on VLAN separation, the problems it solves, and the problems it sometimes creates. Not all networks are cut from the same cloth. Some are simple and some are complex. Some are physical and some are virtual. Some are clean while others are quite messy. The one thing that they all have in common is that Cisco UCS works with all of them.

A Look Back

In UCS, we have a topology that we call “disjoint Layer 2″ (DJL2) which simply means that a there are networks upstream from UCS that are separated from one another and cannot all be accessed by the same UCS uplink port (or port channel). For instance, you might have upstream VLANs 10, 20, and 30 on UCS uplink port 1 and VLANs 40, 50, and 60 on UCS uplink port 2. Prior to UCS 2.0, this configuration was not supported (in End Host Mode (EHM)). The main reason is that prior to 2.0, when VLANs were created, they were instantly available on ALL defined uplink ports and you could not assign certain VLANs to certain uplink ports. In addition to this, UCS uses the concept of a “designated receiver” (DR) port that is the single port (or port channel) chosen by UCSM to receive all multicast and broadcast traffic for all VLANs defined on the Fabric Interconnect (FI). To make this clear, UCS receives all multicast/broadcast traffic on this port only and drops broadcast/multicast traffic received on all other ports. Unless you have DJL2, this method works really well. If you do have DJL2, this would lead to a problem if you defined the above VLAN configuration and plugged it into pre-2.0 UCS (in EHM). In this situation, UCS would choose a designated receiver port for ALL VLANs (10-60) and assign it to one of the available uplinks. Let’s say the system chose port 1 (VLANs 10, 20, and 30) for the DR. In that situation, those networks (10, 20, 30) would work correctly, but VLANs 40, 50, and 60 (plugged into port 2) would not receive any broadcast and multicast traffic at all. The FI will learn the MAC addresses of the destinations on port 2 for 40, 50 and 60, but necessary protocols like ARP, PXE, DHCP (just to name a few) would be broken for these networks. In case you’re wondering, pin groups do not solve this problem so don’t waste your time. Instead, you need UCS 2.0+ and DJL2 which allows specific VLANs to be pinned to specific uplink ports. In addition, you now have a DR port for each defined VLAN as opposed to globally for the each FI. If you want to know more about the DR port, how it works, and how you can see which ports are the current DR on your own domain, please see the Cisco whitepaper entitled “Deploy Layer 2 Disjoint Networks Upstream in End Host Mode” located here: http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns944/white_paper_c11-692008.html

The Rules

You’ve probable figure out that if this were super easy, I wouldn’t be writing about it. Well, yes and no. It’s easy to turn on the DJL2 feature, but there are some lesser known rules around making it work. There is no “enable DJL2″ button and you won’t find it by that name in UCSM. You simply enable it when you assign specific VLANs to specific uplink ports. It’s then automatically on. But many people make a mistake here. Staying with the above example, you want port 1 to carry VLANs 10-30 and port 2 to carry VLANs 40-60. When you first enter VLAN manager, you will see VLANs 10-60 defined and carried on ports 1 and 2. You might think to just take VLANs 40-60 and assign them to port 2. Well, that does remove 40-60 off of port 1, but it would also leave 10-30 on port 2 (along with 40-60). So you must isolate VLANs to their respective uplink ports. Furthermore, if you define a new VLAN, you need to go into VLAN manager and pin it to the port(s) you intend and remove it from the ports it should not be on. The main thing to remember here is that the original UCS rules on VLAN creation have not changed. That is, a created VLAN is always available on all uplink ports. That still happens even when you have DJL2 setup because UCS manager has no idea where to put that new VLAN unless you tell it – so it follows the original rule. I recommend looking at your VLAN config in NXOS (show VLAN) before you run it in production. This will verify that the changes you wanted to make are truly that changes you made in the GUI.

ENM Source Pinning Failed

So now we have DJL2 setup properly on our uplink. Let’s look at the server side as it is often an area of confusion. It’s probably also the way most of you found this blog entry because you googled for the term “ENM Source Pinning Failed”. Let me explain why. When you create vNICs on a service profile using the config we had above (10-30 on port 1 and 40-60 on port 2), you are not able to trunk/tag VLANs from BOTH port 1 and port 2 to the same vNIC. For example, you can have have a single vNIC with VLANs 10, 20, and 30 and another vNIC with VLANs 40, 50, and 60 and both vNICs can be on the same server. But you CANNOT have a single vNIC with VLANs 10 and 40. If you do, the vNIC will go into an error state and will lose link until one of the VLANs is removed. The picture below might help – keep in mind that this diagram is very simplistic and that you can also get an ENM source pin failure with just a single FI:

The above illustration shows a configuration where the user wants to have VLANs 10-50 reach a single server, but this will not work in a DJL2 configuration and will result in ENM Source Pin Failure. Instead, the illustration below would achieve the desired result of VLANs 10-50 reaching the same server, but do not violate the DJL2 rules and would work fine.

Hopefully this helped explain DJL2 a little better and maybe alleviate the ENM Source Pinning error you might be getting.

Thanks for stopping by.

-Jeff

So, about 2 years ago I was with a customer who had opted to purchase UCS over their incumbent HP hardware for their private cloud build. As a first step, we upgraded the firmware on the UCS system. What I did not know at the time was that the mgmt0 cable plugged into the “B” Fabric Interconnect (FI) was showing link, but was not on the right vlan (or wasn’t passing traffic). When it came time in the upgrade to failover the management instance of UCSM to the “B side”, we lost access completely to UCS manager. This and other seemingly related events (but were actually totally unrelated in hindsight) led me to believe that UCSM had failed in some manner and started me down a multi-hour troubleshooting session that I really wished had never happened. I opened an enhancement request to allow UCSM to detect this situation in the future and move UCSM back to the originating FI if it is unable to find the default gateway. Had I known this trick that I am about to tell you concerning the UCS shells, I might have been smart enough to get out of my situation much fast. The sad thing is I actually did know this – it was just knowledge from so early on in my UCS learning curve that I didn’t fully absorb the importance of it. So, now is your chance to start absorbing…

If you have spent any time around UCS (and if you are reading this, you probably have), you know that there is a command-line interface in addition to the provided GUI. The actual “UCS” command line is the starting point “shell” that you are automatically in when you ssh to the UCSM Virtual IP (VIP). We’ll refer to this as the root shell for the purposes of this document. Although root is the main shell, there are many sub-shells available to you in UCS that accomplish various tasks. This post will focus on accessing two specific sub-shells, local-mgmt and NXOS. This article assumes you have knowledge of what each of these shells is for and will not discuss the details of these sub-shells, but will give you an understanding of how to navigate the root shell to gain access to these other sub-shells.

It helps if you think of the shells in hierarchical manner (such as the graphic above). As I mentioned, there are additional sub-shells beyond what are listed above, but NXOS and local-mgmt are by far the most-used, and they are unique in how you can access them. Because the root shell sits above the sub-shells of both fabrics, it allows you to access either sub-shell of either fabric (assuming you are connected to the UCSM VIP and not an individual FI). For instance:

Notice that I started out on Fabric B because that was the controlling instance (FI) of UCSM (you can flip the controlling instance back and forth without data plane disruption – a post for another day). While on Fabric B, I typed connect local-mgmt A. The UCSM root shell then connected me to the local-mgmt sub-shell on fabric A. Had I typed just connect local-mgmt (omitting the “A”), it would default to the fabric that the VIP is currently on (in this case, B). From the root shell, you can do the same type of connection to the NXOS sub-shell on either fabric as well. You cannot jump from a sub-shell to any other sub-shell. You must “exit” back to the root shell to enter any sub-shell.

Back to my bad day story…had I remembered this trick, how would I have avoided the issue? Well, I could always access the A Fabric Interconnect. From there, I could have run connect local-mgmt B and
accessed UCSM which was running just fine on Fabric Interconnect B, and flipped UCSM back to Fabric Interconnect A using local mgmt commands. The success in doing that would have instantly led me to the mgmt0 connection on the B fabric. Things like this are much easier to spot the second time around though – and I saw it again at a customer in production who had a faulty connection to FI-B. In that instance, fixing it was really easy (and they thought I was really smart – no, I didn’t tell them the truth).

That’s pretty much all there is to it. If you want to play around with the various other shells, you can type connect ? at the root shell and it will return all the possible devices you can connect to.

P.S. Ironically, the same day I wrote this article, I got a call from a co-worker who “could not connect back to UCSM after the primary FI rebooted during a firmware upgrade”. We used this trick (which he thought was way cool) and then discovered later that he had a flaky Ethernet cable in mgmt0 in the (formerly) subordinate FI. If you’re curious about why the enhancement I referenced above didn’t help here, it’s because the enhancement (mgmt0 interface monitoring) is enabled by default on all NEW installations but left at the previous setting on any UPGRADES (because change is a bad thing). I believe that change went into the 2.0 release.

Thanks for your time.

-Jeff

In another blog post I covered how to restore a failed Fabric Interconnect (FI). It gives you some insight into a complete and total rebuild of an FI in a worst-case scenario. While that would accomplish the “factory defaults” you desire, it’s a painful way to get there. Thankfully, the “erase” process is pretty easy. There is no way to do this in the GUI so grab your favorite ssh client and connect to either FI. Once connected, type the following:

FI-A # Connect local-mgmt

FI-A (local-mgmt)# erase config

That’s it! You’ll need to confirm the command before it executes, but that will start the process. You then need to repeat the process on the other FI. There is a way you could erase both of them by connecting to the VIP and not directly to the FI, but I’m going to cover that feature in another post because it’s pretty cool all by itself.

-Jeff

So, everyone would agree that a helping hand is nice to have now and then. Like the time I thought it would be a good idea to skateboard while holding onto my brother’s car as he drove down the street. It was his helping hand reaching down to pick me up off the road (bleeding) and sitting me in the car that I won’t forget (I still have that scar on my hip). It was brother helping brother – an understanding that when one is down, the other will help get him on his feet (hopefully before mom sees so that we could get our story coordinated as to how it happened). In the UCS world, the brothers in this scenario are the Fabric Interconnects (I’m not sure who the mother is).

There are times when a Fabric Interconnect might encounter a software failure – for whatever reason, and land at the “loader” prompt. It’s rare, but it can happen. The loader prompt is a lonely place and it’s not pleasant. The good news is that if you still have a single FI working, you can use it to resurrect the broken FI. First off, if you ever find yourself staring at the loader prompt, stop cursing and just try to unplug it and plug it back in. Don’t worry if “dir” shows no files – just try it. I’ve seen it work and the FI comes right back on the next boot. If that doesn’t work, you have some work to do…

The loader is just that – a loader. It’s “loads” an OS – like a bootstrap. You need 3 files to permanently get out of the loader – kickstart, system, and the UCSM image. Luckily all of these live on your remaining FI. The bad news is you can’t get to them without bringing it down as well. So, if you’re in production and can’t get afford to bring down the entire UCS pod, you should stop reading and call TAC. They can get you the 3 magic files you need and can get it all running without bringing anything additional offline. But if you’re in a situation where you can afford to take down the remaining FI, you can fix this problem yourself.

To make this work, you will need:

• Non-functional FI
• Functional FI
• FTP Server
• TFTP Server

Your basic recovery will include:

1. Disconnect the L1/L2 cables between the FI’s to avoid messing up the cluster data they share
2. Boot FI-A to loader
3. Force FI-B to loader
4. Boot kickstart on FI-B
5. Assign IP address to FI-B
6. FTP kickstart, image, and ucsm images from FI-B to an FTP server
7. Reboot FI-B back to its normal state
8. Get kickstart image onto TFTP server (unless FTP/TFTP are the same server)
9. Boot kickstart image on FI-A via TFTP server
10. FTP kickstart, system, and ucsm images down to FI-A
11. Copy ucsm image file to the root
12. Load system image on FI-A
13. “Activate” the firmware on FI-A
14. Connect L1/L2 cables back and rejoin the cluster

Reboot the “good” FI (known in this document now as FI-B), and begin pressing CTRL+R to interrupt the boot process. You will find FI-B now stops at the loader prompt too. Now type

boot /installables/switch/ <tab>

which will show you all files in this folder. You are looking for the obvious kickstart file and you want the latest one. To make the display easier to read, I would type this:

boot /installables/switch/ucs-6100-k9-kickstart <tab>

Backspace is not an option so if you make an error, use the arrow keys and the “delete” key to fix typos.

Select the latest image, hit enter, and FI-B now beings to boot the kickstart image. Give it a few minutes and you should find it stops at the “boot” prompt. This prompt is not as lonely as the loader prompt, but it’s still not a fun place to be (at least you can backspace now). You actually will have much more functionality then you did with loader, but won’t need it for this exercise. At this point you need to assign and IP address to FI-B so that you can FTP the kickstart image to an FTP server. The commands will look like this:

#Config t

#int mgmt 0

#ip address X.X.X.X <mask>

#no shut

Wait 10-15 seconds

#<ctrl+z to return the shell to the top level>

# copy bootflash:installables/switch/ucs-6100-k9-kickstart <tab>

Select the latest version and copy it to the FTP server.

DO NOT USE THE FILES IN THE ROOT OF BOOTFLASH AT ANY TIME DURING THIS PROCESS. Nothing catastrophic will happen, but the FI will not boot in the end.

The shell will prompt you for ftp server address and credentials and it should look something like this:

You need to allow about 10-15 additional seconds after you “no shut” the interface for the IP to become active and useable.

You now need to copy the system and UCSM images as well as you will need them soon enough. The other two files will look something like:

installables/switch/ucs-manager-k9.2.1.0.418.bin

installables/switch/ucs-6100-k9-system.5.0.3.N2.2.10.418.bin

again – your versions will be different

Once you are returned to the boot prompt, and all 3 files are copied, the kickstart file is on the FTP server. You should boot FI-B back into production. You now need to get the kickstart file available via TFTP using whatever process you do to make that happen. One word of caution here – TFTP blows. It runs on UDP, it’s slow, and it has no error checking. If your first attempt at booting fails, try a different TFTP server program (trust me on this – I had bruises on my head from banging it on the wall). Once the file is available via TFTP, return to FI-A which is at the loader prompt. You will now boot that kickstart image via TFTP using these commands:

Incidentally, you cannot ping this address from an outside station. Just FYI

Then it begins loading the image. It should take just a few seconds to actually start booting. FI-A will not land at the boot prompt like FI-B did earlier. You need to rebuild the filesystem on FI-A, so type:

#init system

This will take a few minutes. When it’s done, you can now use FTP to copy the 3 files down to FI-A. Use this command to retrieve each file:

#Copy ftp: bootflash:

The shell will prompt you for everything it needs to copy the files.

After all 3 are copied, one very important command needs to be run now and it won’t make sense, but you must do this. You need to run this command:

Copy bootflash:/ucs-manager-k9.2.1.0.418.bin bootflash:/nuova-sim-mgmt-nsg.0.1.0.001.bin

The nuova-sim-mgmt-nsg.0.1.0.001.bin is an exact name that is needed here.

Now that you have all 3 files local on the FI, you would be able to recover much quicker if the FI were to lose power. At this moment, if that happened, you would be returned to the loader prompt, but you would be able to boot via bootflash instead of TFTP. Anyway, you are now at the boot prompt and need to finish booting. Type load bootflash://ucs-6100-k9-system.5.0.3.N2.2.10.418.bin. This will start loading the system image and when it’s done loading, it will look for the UCSM image that you also copied and the FI should come up. It will walk you through the setup menu and since the L1/l2 cables are not connected, I would go ahead and set it up as standalone – we will join it to the cluster soon. Once you are logged into the FI, you need to activate the current firmware to set the startup variables. The easiest way to do this is in the GUI. Just go into Firmware Management and select “Activate Firmware” and select the FI. You will likely see that no version is in the startup column. Regardless, you need to activate the version that is already there. If it doesn’t let you, exit Firmware Management and navigate to the Fabric Interconnect on the left-side Tree menu and activate the firmware from there using the “force” option. This will fix up the ucsm image file that we copied to the root as well (turns it into a symbolic link).

That’s about it. You should be OK to erase the config on FI-A (#connect local-mgmt), hook up the L1/l2 cables and rejoin the cluster on the next reboot. I really hope you don’t need to ever use this… I mainly wrote this blog for myself because in the lab we do a lot of crazy stuff and I often forget a step here and there. So I wanted it all written down to refer back to and I’ve wanted to get this one done for quite some time.

Thanks for reading…

-Jeff

So, have you heard the news? The world’s fastest growing x86 server company is joining forces with the world’s fastest application acceleration company? While the partnership on the blade front is somewhat recent news, Cisco has long been supporting Fusion-io accelerator products in our C-Series servers (and will continue to do so). In June 2012, Fusion-io and Cisco inked a deal that would extend our partnership to cover UCS B-Series Blade servers as well. It’s not simply changing the form-factor and connector; it’s also extending UCS Manager to include integrated support for the cards for inventory and management purposes. To read more on the partnership, look here: http://www.fusionio.com/blog/coming-to-cisco-ucs-blade-servers-soon-iomemory/

The Hardware: Initially, Cisco will be incorporating the 365GB and the 785GB Fusion ioDrive2 starting with the M3 line of blade servers. Both products offer persistent NAND flash-hardware combined with custom software that offers extreme low-latency, high performance, and a proven track record of reliability.

When I first heard of Fusion-io, I had the same though that I’m sure many of you have right now. It basically goes something like this “so they make some really fast SSD cards, so what?” But after an intense 1.5 days of training and interviews at Fusion-io headquarters in Salt Lake City, I have come to learn it’s quite a bit more than that. Fusion ioDrives are based on MLC NAND flash memory. That last sentence is loaded with terms begging for explanation, so I won’t leave you hanging…

Geek Speak: SLC vs MLC:
Flash memory stores data in an array of cells. There are two main types in use today: SLC (single level cell), and MLC (multi-level cell). Both have their pros and cons, but MLC is king when it comes to density (because more data is stored in each cell) and the cost per bit of MLC is lower. Traditional thinking is that SLC is king when it comes to reliability, but technology improvements have now driven MLC reliability to be on par with SLC.

NAND vs NOR: The invention of flash memory grew from traditional EEPROM technology. Traditional EEPROM would not be a good fit for many of the uses we use flash for today. Here’s why: Imagine if you had a USB flash drive that contained a large unneeded file. If you delete the file, you can fit your new data onto the drive, but you were not allowed to erase a single file on the drive unless you were willing to delete ALL files on the drive and re-write the ones you want. Big limitation. Both NOR & NAND (Not OR & Not AND if you’re wondering) overcome this and could be used for ever-day application use (cameras, phones, PDAs, tablets, media players, etc), but NAND is by far the most popular due to its better density and lower cost. The most important difference between NOR and NAND is how they handle reads: NOR is true Random Access when it comes to reads and NAND requires page (block) reads which is not the most efficient. But cost often wins in the enterprise space.

The Fusion ioDrive2s in Cisco UCS servers are not implemented as a traditional storage device. The interface to access the storage is not SAS, SATA, or FC. Each of these interfaces would introduce latency inherent in each respective technology. Because the ioDrive2 requires direct access to the server’s PCIe bus, it puts the ioDrive2 as close to the CPU as possible. There is no doubt that an SSD drive from any electronics store is much faster than spinning disk. SSD is implemented as a traditional SATA device which makes it extremely user-friendly, but introduces a great deal of latency due to SATA itself. The following graphic showing the IO path illustrates this problem:

When that same SSD device is put into an array on the storage network, the problem is compounded due to the HBA, storage switch, and raid controller in the array. The following graphic illustrates the problem of this added latency:

In contrast, direct access to the PCI results in extreme low-latency. Consider the following graphic to see the difference:

If I could summarize the importance of this technology, it would come down to this on paragraph: x86 History has shown that the key to making devices faster lies in their proximity to the CPU. In the server world, Distance=Latency. Consider CPU cache which used to be made of SRAM and housed on the motherboard. Beginning with the 486 CPU, the L1 cache was integrated in the CPU die itself. Fast forward a decade or so and we now have 3 levels of cache directly on the CPU die. Another example is in memory architectures. We used to house all system memory in one location behind a single memory controller (Northbridge) and all CPUs would compete for access. AMD entered the server market and introduced a new design that brought banks of memory closer to each CPU and gave each CPU an individual memory controller (NUMA). Intel integrated the memory controller inside the CPU die starting with Nehalem. Lastly, take PCIe in Intel’s E5 line of processors. For the first time, the function handling PCIe itself was integrated into the CPU die as well. This gives many devices in the system a boost in performance. All of these examples illustrate that there is an obvious speed advantage when devices are moved closer to the CPU. So it’s quite paradoxical that we keep storage far away (CPU->HBA->storage switch->raid controller->processor on SSD ->flash controller on SSD). Unless the problem you are solving is storage density (multi-terabyte), there are better ways to crack this nut. These new ioDrive2s eliminate much of this latency and you instead have the CPU talking directly to the flash controller (all ioDrives have a controller). This is incredibly efficient and blazingly fast.

While architectures such as this do not eliminate the need for a SAN, they certainly have the potential to reduce the storage requirements of the array if designed correctly. This fact alone makes Cisco unique in this I/O accelerator space as every other major server OEM offering an I/O accelerator product would delight more in selling you a large storage array, especially if they make it. Cisco is not in the SAN array market and wants to help you build a solution that is truly “best of breed.” If that means a large storage array (or not), we’ll help you architect it.

Is It Worth It? The idea of faster storage access extends to universal application use. It doesn’t matter if you’re run a database, a virtual server farm, VDI, Big Data, or a private cloud of some sort. If you were able to access your data faster – a LOT faster – would you do it? Is it cheap? No. It’s quite frankly not the cheapest option you’ll buy for your server. But it’s quite unlike any option you’ve ever seen and the cost needs to be weighed against other options for increasing access to your storage. Traditionally this is done by adding spindles (DAS or SAN). It is not economically feasible for you to add enough spindles to equal this performance.

We’re supporting Windows 2008 R2/2012, RHEL and SLES, ESX. For the actual versions of each, please see the Cisco UCS Interoperability tool: http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

I found an interesting blog that compares and Intel enterprise-class SSD with a Fusion ioDrive2 (the same card to be used in UCS blade servers). Coincidentally, the blog authors (Percona) happened to be using a Cisco UCS C250 to compare the results of the two drives, but Cisco had nothing to do with the testing or the results.

Intel SSD 520: http://www.mysqlperformanceblog.com/2012/05/01/testing-intel-ssd-520/

Fusion-io: http://www.ssdperformanceblog.com/2012/05/testing-fusion-io-iodrive2-duo/#more-230

The comparison at its most basic level breaks down to the following:

As you can see the Fusion ioDrive2 more than doubles the SSD performance across the board. Based on data from both companies, I put together following table. Please note that I used statistics for the lower capacity 365GB Fusion ioDrive because the largest Intel SSD is 480GB

Obviously the Intel drive is not lacking in performance and does quite well (I wish the SSD in my laptop was this fast). But again, it’s no comparison to the numbers that the Fusion ioDrive2 put up. Like me, you have probably used an SSD in some form or another and experienced the tremendous speed increase it brings. Imagine a drive that more than doubles that performance…

We’ll be shipping the Fusion ioDrive2 for Cisco UCS blades REALLY soon (it’s real – I saw it with my own eyes. I tried to stick it in my bag but they counted them before and after show-n-tell! That’s an actual picture of the card earlier in the article). In the meantime, your Cisco UCS sales team should be able to assist you with any detailed questions, but as always, your comments and questions are welcome here.

Thanks for reading

-Jeff

So, one of the huge advantages of Cisco UCS is its approach to “statelessness”. If you are not familiar with this concept, just know that anything that ties an operating system, hypervisor, or application to a specific piece of hardware is considered “stateful” and is not desirable in datacenter servers. Using this methodology, Cisco has made the upgrade path extremely easy for a customer to upgrade from one server model to the next without having to re-install anything. To be more specific, I upgraded various operating systems and hypervisors that were running in a service profile assigned to a B200 M2 and moved the profile to a brand new architecture of a B200 M3. The UCS portion of this migration is really (really) easy – you simply associate the profile from an M2 and assign it to an M3. The OS or hypervisor takes care of the rest. This article will cover the details of how this migration worked and what steps I took to make sure it was a success. Disclaimer: everything you’re about to read is totally unsupported by Cisco TAC. As a company, we have not tested nor certified this process. I am simply reporting here what I, myself, have tested and seen work. So don’t call Cisco if this doesn’t work. Feel free to leave a comment and I’ll look into it when I can find time.

I tested of few different installations of various hypervisors and OS’s throughout this process and hope to test more as I have time. These are my results thus far:

NOTE: All migration testing was done booting from XIO Emprise 5000 FC SAN using a Cisco VIC CNA.

Environment:

B200 M2
Palo (2.0.2q)

B200 M3

VIC1240 (2.0.2q) – A 1280 should work identically as they are based on the same ASIC

UCS Manager

2.0.2R
I installed each of the below onto the M2 and then migrated it to the M3. I was mainly looking for the instance to still boot after migrating. I did not test every aspect of the installation once it was on the M3. For instance, if you added or removed vNICs or vHBAs and caused the PCI bus to re-enumerate, you may have some cleanup to do. I was simply verifying that you could actually get to the point of being able to clean up at all.

1. VMware ESXi 4.1 Build 348481 migrated without issue. I was running Fnic version 1.1.0.113.2-4vmw (according to vmkload_mod –s fnic)
2. VMware ESXi 5.0 Build 441354 migrated without issue. I was running Fnic version 1.2.0.3-1vmw (according to vmkload_mod –s fnic)
3. Windows Server 2008 R2 migrated without issue in one test I ran, but had the dreaded STOP 0x7B in separate (different) attempt. It depends on some factors that we’ll cover in a minute. If you just keeping score, mark one more down for UCS, provided you installed Windows 2008 using Cisco provided Palo drivers version 2.0 or later. If you installed using a version of the drivers prior to 2.0, you will likely see the STOP 0x7B trap screen. The good news is you can still get it to work, but it requires some manual input on your part. This is the technical part so you can skip this paragraph if you are not in this situation.

There are certain types of devices that Windows needs to install drivers for prior to the time that the normal plug-n-play process is available. Disk Drive controllers would fall into this category (as do the various bridges that sometimes lead to disk controllers). Because Windows is bypassing the plug-n-play manager, it directly starts these drivers. The portion of the registry that stores all of this is known as the Critical Device Database (CDDB). You can locate it at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase. When Windows is starting up, it loads the drivers in the CDDB. When it’s done, it should have access to the boot volume because the boot volume’s controller driver would be in the CDDB. But what if it isn’t? Well, in that case you get STOP 0x7B. Windows is trying to tell you that it has loaded all drivers it has and that the boot volume is still not accessible.

On a normal computer, this is quite unpleasant because you can’t always get back into Windows to fix it. If it happens due to a BIOS upgrade, it’s likely that the bridge was updated and you need a new driver prior to the BIOS upgrade. Check out this blog for more detailed info on the CDDB if you are interested. Luckily, with UCS, you can roll the profile back to an M2 with Palo and boot it right back up. Once you are back into Windows, you can fix this problem pretty easily. Here is the problem and solution:

When you install the OS from scratch and feed it the 2.0 or higher drivers during installation on the M2 with Palo, it builds the following tree in the CDDB:

However, if you installed with an earlier version of the drivers, the CDDB tree is not built the same and it does not create the entries for the 12×0/mlom and it looks like this:

This behavior is expected of course because the VIC 1240/1280 did not exist at that time you used those older drivers. However, the new Palo (M81KR) drivers contain all the CDDB tree info for both old and new VIC cards. To fix the problem, all you have to do is:

1. Boot windows on the M2.
2. Select (highlight) HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase\PCI#VEN_1137&DEV_0045&SUBSYS_00481137
3. Click File->Export and export this key to a file.
4. Then open the reg file in notepad.
5. Change the string 00481137 to 00841137 (you are just swapping the “4″ and the “8″.
6. Save the changes and close the file.
7. Double click the reg file to import it into the registry.
8. Shut down windows and move the profile to the M3.

I would not suggest editing the string in the registry directly as this would overwrite the only working string you have. Last Known Good would help here, and will also help if you mess anything up while doing the above steps. Just remember, Last Known Good is only valid until you successfully login.

I hope this article is helpful to you. Let me know if anything isn’t clear.

Side note: If you want more information on statelessness and how it benefits you, see this blog article. But briefly, take a NIC, for example, that typically has a MAC address burned in at the factory. If you install an application and it licenses itself to the MAC address you will find it painful to replace the NIC if it fails because the MAC address will change. Cisco UCS makes this simple by creating a portable wrapper for the application (and OS) to run in that contains the MAC address (called the Service Profile). The Service Profile contains a lot more than the MAC address (over 100 items of server identity can be stored in it) and because it’s portable, I can move it from physical server 1 to physical server 2 and the application sees no changes. Think of it as virtual machines technology for the physical side of the datacenter. This capability is not unique to Cisco (HP, Dell, and IBM all have some aspect of it), but only UCS offers you complete control over every desired aspect of server identity.

-Jeff

So, there is a little known feature in Cisco UCS that allows one to monitor traffic on the control plane without the hassle of actually hooking up an Ethernet analyzer. The control plane is physically the “mgmt0″ port on the Fabric Interconnect (FI) and it is used for managing the FIs themselves and for attaching to KVM sessions on the blades. Cisco UCS makes the capture process really easy and in this article I’ll show you how to do it. Remember, this process only captures control plane traffic – not data traffic (UCS has a similar function called Traffic Monitor for that). So, how is this useful? Well, there’s always the chance that something unexpected could happen in UCS Manager as a result of malformed packets entering the Fabric Interconnect’s mgmt0 port. TAC would need to see what the data looks like in order to determine the cause. But the more likely scenario here is that you are using the Cisco UCS XML API and would like to inspect the management traffic being sent to and from the Fabric Interconnect from either UCS Manager or some other external manager controlling UCS. This is an extremely useful tip to aid in the XML API learning process. If you are not familiar with the API of Cisco UCS, it’s an extremely powerful engine provided to customers, system integrators, independent software vendors (ISV). You can find additional information about the API, download it, and learn how to work with it on the following site:

http://developer.cisco.com/web/unifiedcomputing/start

To get started, open an ssh connection to the FI cluster address.

As you can see, the built-in protocol analyzer is called “ethanalyzer” and it’s based on Wireshark open source code. It both captures packets and displays them to the screen (by default). In the example above, it is simply capturing and displaying the last 10 packets, but it’s very flexible to capture more. Remember when I said that this is for control plane traffic only? Well that’s true, but let me add a caveat to that statement by saying that some management traffic enters the switch through the data ports (LACP, DCBX, FC, and FCoE). Ethanalyzer is able to capture these packets as well and does so using different Ethernet interfaces than mgmt0. To make this easy to understand, think of this in terms of a host that has 3 interfaces (there are physically more, but only 3 are exposed to ethanalyzer).

eth0=interface mgmt0
eth3=internal interface listening for low priority control packets on the data plane ports (IGMP, CDP, and mgmt-only TCP/UDP/IP/ARP)
eth4=internal interface listening for high priority control packets on the data plane ports (STP, LACP, DCBX, FC, and FCoE)

I mention this not because I think you will use eth3 and eth4, but if you run the command, you are going to see high and low options and I’m sure that it will make you curious. So, now to make ethanalyzer capture what you are looking for is pretty easy. Here’s an example of me capturing the first 100 frames that FI sees on mgmt0 after I execute the command:

Pretty cool. But this is still just spitting the output to the screen which is of limited value. It would be more useful to look at the captured packets in an actual protocol analyzer like Wireshark. With ethanalyzer, you can do just that:

Then you can copy the file to your workstation like this:

And the CLI will prompt you for all of the server info prior to the copy. If you play around with the command a bit, you’ll see there are additional options for filters, etc. My suggestion would be to capture all the packets with no filters and then use filters inside of Wireshark to inspect the data you are looking for. Be sure and take a look at the “autostop” option – it will allow you to stop capturing based on time and/or log file size. Finally, if you are looking for detailed information about the analyzer, including how to capture management traffic on data plane ports, please refer to the whitepaper “Ethanalyzer: Cisco NX-OS Software Built-In Packet Capture Utility” located here:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps11541/white_paper_c11-673817_ps9670_Products_White_Paper.html

Thanks for stopping by!

-Jeff

So, in my last article, I discussed Appliance Ports and how to set them up. But there was a hidden gem in there that I felt deserves its own article because it’s just that cool. If you’ve ever setup the MTU on servers because you want to use an iSCSI array, you’ve suffered through how exactly to get the OS to recognize the new MTU size. As I pointed out in my last article, this process this involves a registry hack, ifconfig, esxcfg-vswitch, or setting the MTU manually within the Windows adapter properties. It’s worth the time to investigate because many applications perform better when the conversation doesn’t have to be fragmented into many small chunks.

If you’re using UCS, and if you are reading this you most likely are, and you are using one of the Cisco VIC cards (including Palo/M81KR, VIC-1240, and VIC-1280), you’re in for a real treat. Due to Cisco’s strong integration with the OS/Hypervisor, the VIC driver will inform the installed OS/Hypervisor of the MTU size designated on vNIC itself in the service profile. It’s simple, elegant and makes a lot of sense. Every other vendor is going to opposite direction – they are making a change in the driver which in turn changes the setting on the hardware. The fault here is that the hardware may not always be the same. If the blade server in use today is upgraded to the newer version, the NIC might change (HP’s BL460 G7 has a different NIC vendor than the HP BL460 Gen8 for example). This means that the registry hack you did for the old server will not work on the new server. It also means a huge headache for server admins to know which tweak to make for which OS, server, and NIC combinations. UCS simplifies all of this. The end goal is to alter the MTU size on the NIC – so why not just start with the NIC?

Just for reference, I thought it might be handy to re-publish the ways to verify the correct MTU size for your OS or Hypervisor. The designated MTU can be verified as follows:

Verifying the MTU Size

Windows:

netsh interface ipv4 show subinterfaces

Linux:

ifconfig

ESX:

esxcfg-vswitch –l (that’s an L)

Testing the MTU Size

Windows:

ping –f –l 8000 <storage appliance ip address>

If you get replies, it’s working. If you get “Packet needs to be fragmented but DF set”, windows is trying to tell you that the packet is too large to pass through and you specifically told ping not to fragment the packet.

Linux:

ping –M do –s 8000 <storage appliance ip address>

ESX:

vmkping –s 8000 <storage appliance ip address>

I use 8000 to keep it common between Windows, Linux, and ESX. The largest true packet size is 8972 which sends a 9000 byte packet when you add in IP and ICMP header info of 28K. Some OS’s accept a parameter of 9000 and others max at 8972, but 8000 works on all of them and demonstrates that the MTU is most likely working as you expect it to.

As always, thanks for stopping by and I hope this article provides some value to you.

So, I recently had a customer that wanted to enable “Jumbo Frames” to a UCS server that had the Cisco Virtual Interface Card (VIC) installed in it (also applies to Palo/M81KR, VIC-1240, VIC-1280). You might also know this process as “maximizing the MTU”. In this particular situation, the customer had an iSCSI appliance directly connected to the fabric interconnects (Whiptail in this case, which is not officially supported by Cisco as of this writing, but this process will be the same for any iSCSI appliance – supported or unsupported). It’s not the first time this has come up so I thought I’d write it down so that everyone can benefit (including me when I forget how I did all of this). This article will be helpful if you’re using any NAS storage such as NFS, CIFS/SMB, or iSCSI.

In Cisco UCS, we support certain storage appliances directly connected to the fabric interconnect via an Appliance Port (see supported arrays here: http://bit.ly/teL5Pb). The Appliance Port type was introduced in the 1.4(1) release of UCS Manager. Prior to this release, you had to put UCS in “switch mode” to attach an appliance directly to it. As a side note, appliance ports share some characteristics with server ports: They learn MAC addresses and they have an uplink (border) port assigned (static or dynamic) to them. Because they learn MAC addresses like any normal switch port does, as soon as a device is “heard” its MAC is considered “learned” and the switch will no longer need to flood packets destined for that MAC to all ports. This is good to know because when configuring an Appliance port, you are given the opportunity to add an “Ethernet Target Endpoint” – which is the MAC address of the connected appliance. This is an optional field and is not required for most appliances (as they broadcast their MAC address when connected), but if you have an appliance that doesn’t or if you have connectivity issues, you should enter the MAC in this field (See Figure 1 below). It should be noted that appliance ports do not support connected switches – at all. The port will shut down when it detects a switch on the far side.

So, let’s dig right in and setup an appliance port…step by step.

1. QoS Configuration
   1. This step is only needed if the appliance is trying to use a protocol or file system that benefits from a large MTU (Jumbo Frame) like iSCSI or NFS, for example. If your storage does not, you can move on to Step 2 because the MTU is usually the default Ethernet size of 1500 bytes.
   2. Go to the LAN tab, expand Policies, and look for the QoS Policies item.
   3. Right click “QoS Policies” and choose Create QoS Policy.
   4. Give it a name.
   5. Select an unused Priority from Bronze, Silver, Gold, or Platinum. I will use Bronze (most likely none of these are in use on your system, but make sure).
   6. Click OK to save the changes to the new policy.
   7. While still on the LAN tab, select “QoS System Class” under the “LAN Cloud” item.
   8. You will see a dialog similar to Figure 1.

   Figure 1
   

   

   1. Change the MTU from normal to 9000 (again, I used Bronze).
   2. Check the Enabled box for Bronze.
   3. Click the Save Changes button.
   4. Set the MTU on the storage appliance to 9000

I should mention here that some storage appliances will allow an MTU of 9216, and if so, you can choose that so long as you make the Priority in UCS 9216 as well. We won’t get into it in this article, but if the appliance is not directly attached and is instead somewhere north of the fabric interconnects, you would need to match the MTU priority in UCS to the same size as the switch port connected to the fabric interconnect. But that scenario would not include Appliance Ports as they require the array to be directly connected.

If the appliance is plugged into both fabric interconnects (and it should be unless this is a lab like mine), repeat Step 2 on the opposite fabric interconnect. My suggestion is that you get one side working at a time.

1. Appliance Port Configuration
   1. On the Equipment tab, select the fabric interconnect where the appliance is plugged in to.
   2. Right-click the correct port from the unconfigured Ethernet ports and select “Configure as Appliance Port” (See Figure 2).
   3. If you’re using iSCSI storage AND you want to maximize the MTU, choose the same Priority class from the drop down menu that you configured in Step 1 above. We used Bronze in my example. Otherwise, just continue.
   4. Don’t use Pin Groups unless you’re very familiar with how they work and why you’re doing it.
   5. There should not be any need for a Network Control Policy here in this example.
   6. Select the correct port speed based on the speed of your storage appliance.
   7. Decide if the port is a trunk or access port.

      Note: I should mention that if this is a trunk port connected to the appliance, the VLANs you input here are not stored in the standard VLAN database that the fabric interconnect uses for server and uplink traffic. You can see this by looking on the LAN tab and you will notice VLANs for the Appliance cloud as well as the LAN cloud – these entries are separate from one another. So, if you are trying to put the appliance on an existing VLAN already configured on your servers, you will need to create an identical appliance VLAN for the appliance port using the same VLAN ID you use for the server’s vNIC (alternatively, you could create this VLAN ahead of time on the LAN tab under Appliances).

   8. Click OK to save the changes.

       

   Optional: As explained above, if you know the MAC address of the appliance, input in the dialog box (this simply pre-populates this address into the mac-address table in the FI). As explained above, most appliances will broadcast their MAC when they are connected, but it will not hurt to enter it here. My suggestion is to leave the endpoint blank and re-visit it if you cannot get it to work.

Figure 2

At this point, assuming you have configured the uplink and appliance VLANs correctly and that you have enabled the chosen storage appliance VLAN within your northbound infrastructure, you should be able to ping the appliance IP address from a workstation outside the UCS system. If you cannot, check the VLAN configuration for both the fabric interconnect LAN Cloud and the Appliances Cloud (both on the LAN tab) as well as the MTU size on the appliance port and the appliance itself. If your intention is just to make the storage available within the UCS system, it may not be available to outside systems at all because the VLANs are not accessible outside.

It’s now time to setup the VIC-enabled servers to access the storage. The plan is to add a vNIC on the server that is in the same VLAN as the storage. If using iSCSI, you need to match the MTU size of the vNIC to the size already configured in Step 1. The steps are as follows:

1. Server Configuration (this is disruptive and will reboot the server). See Figure 3
   1. Locate the desired Service Profile on the Servers tab.
   2. Right-click the vNICs item and choose “Create vNIC”
   3. Name the vNIC (i.e. “eth2″)
   4. Fill in the dialog with all required information such as MAC pool, correct Fabric, and the VLAN that matches the appliance port created in step 2 above.
   5. Be sure and select the appropriate VLAN that matches the appliance
   6. If using iSCSI and maximizing MTU, change the MTU of the vNIC to 9000 (vNIC MTU maximum is 9000, not 9216)
   7. If using iSCSI and maximizing MTU, select the correct QoS Policy you defined in Step 1
   8. Click OK to save the changes.

Figure 3

TIP: If this procedure were being done on an HP, IBM, or Dell server, you would have the additional step of going into the OS to set the MTU to match (9000). Depending on the OS/Hypervisor, this involves a registry hack, ifconfig, esxcfg-vswitch, or setting the MTU manually within the Windows adapter properties. This would be required on every server that plan to use the iSCSI appliance. With UCS and the Cisco VIC, it involves none of these because Cisco has strong integration with the OS/Hypervisor and the VIC driver will inform the installed OS/Hypervisor of the new MTU size automatically. Whatever MTU size you designate on vNIC itself will be used by the OS/Hypervisor. So, in this case, once you create the storage vNIC and reboot the server, the MTU size will already be at the correct value of 9000. How cool is that? Regardless of what OS you install, you don’t have to worry about finding the right command to set the MTU!

The designated MTU can be verified as follows:

Windows:

netsh interface ipv4 show subinterfaces

Linux:

ifconfig

ESX:

esxcfg-vswitch –l

If you would like to test the end-to-end MTU, there is an easy process for that as well:

Windows:

ping –f –l 8000 <storage appliance ip address>

If you get replies, it’s working. If you get “Packet needs to be fragmented but DF set”, windows is trying to tell you that the packet is too large to pass through and you specifically told ping not to fragment the packet.

Linux:

ping –M do –s 8000 <storage appliance ip address>

ESX:

vmkping –s 8000 <storage appliance ip address>

I use 8000 to keep it common between Windows, Linux, and ESX. The largest true packet size is 8972 which sends a 9000 byte packet when you add in IP and ICMP header info of 28K. Some OS’s accept a parameter of 9000 and others max at 8972, but 8000 works on all of them and demonstrates that the MTU is most likely working as you expect it to.

Note: This article is written to the lowest common denominator. It does not involve LAN Pin Groups or vNIC Templates. Those are topics that I hope to write articles on in the future, but are covered well in our product documentation.

As always, thanks for stopping by.

-Jeff

P.S. If you’re using a different CNA than the Cisco VIC and would like specific direction on setting the MTU for it (inside the OS/Hypervisor), drop me a note below and let me know which one. I’ll do my best to dig up the instructions.

So, first let me define some terms….the Cisco VIC is also called “Palo” – a codename that sort of stuck (much the chagrin of the marketing team). Palo’s official name is M81KR – now do you see why “Palo” sort of stuck ? We have some new VIC cards as well – the VIC-1240 and VIC-1280 and Sean McGee (@mseanmcgee) talks more about the VIC-1280 here. The VIC-1240 is a built-in option on the M3 blades. Now that we settled that, where is Part 1 of this article? Well, my good friend Ryan Hughes (@angryjesters) got the ball rolling on this. He took it upon himself to write an excellent article explaining how to access the obscure-but-useful command called LUNLIST. So if you are looking for Part 1 to this article, I’m not the author of it. I learned some things reading Ryan’s article, which is not all the surprising since I’m rarely with Ryan when I don’t learn something. You should check out his site if you have not seen the article already, but briefly, LUNLIST is a command that shows you what the Cisco VIC HBA can actually “see” on the fabric – much like a typical HBA BIOS would…but way cooler.

Why am I writing part 2? Well, in the comments to Ryan’s article, a responder noted that during the HBA POST process, the VIC itself will show you if zoning and LUN masking are correct and that LUNLIST may not be needed. While that comment is partly true, LUNLIST is definitely needed and is a great help in troubleshooting. There are prerequisites that must be met for the VIC to show success during POST, and when POST does not show you what you expect, you don’t always know where to start. Is the problem in the profile, the Fabric Interconnect, the upstream FC switch, or the array itself? It’s this kind of thing that makes server administrators irritated with boot from SAN to begin with. There is too much of the setup that is out of their control – and requires a lot of joint troubleshooting with the SAN team. Cisco UCS certainly makes this a lot easier, and I wrote an article back in late 2010 that outlines the basics of a boot from san scenario in UCS. Check it out if you are not familiar with the process, but I believe there is always room for improvement, and this area is no different. So with Cisco UCS 2.0 we introduced LUNLIST (and we’re not close to being done in this area by the way). UCS has a cousin command called LUNMAP that has been around a long time, but LUNLIST is the steroid-using one of the two and when I am troubleshooting, I solely look to LUNLIST. Let’s see why…

As Ryan pointed out, LUNLIST only works prior to the OS HBA driver loading. Once the driver loads, the VIC boot BIOS is no longer in control and will not return valid data. This means that it’s more difficult to use LUNLIST to determine if your configuration is “looser” than it should be by having excess LUNs allowed to the wrong host(s). One reason I like LUNLIST compared with legacy HBA BIOS tools is that I do not have to open a KVM to the server in question and I do not have to catch the server at just the right second during POST. I can just let all the servers attempt to boot, and from one CLI, quickly and easily look at any number of HBA’s in any number of servers. Pretty cool stuff. Another reason I like LUNLIST better is that in a single output, it can tell me if my problem is in the Boot Policy, the zoning config, or the LUN masking. Let’s take a look at some output to show you what I mean.

To get to the command, you need to gain access to the UCS CLI and run the following:

connect: connects to the VICs management processor

attach-fls: attaches to the fabric login service of the adapter

Once you run lunlist, you see output similar to the below. This one is from a server where the end-to-end configuration was all done correctly and the server could boot from SAN or attempt an installation to do so:

Now let’s break it apart and describe what you are seeing:

So, you now may be starting to see the usefulness of this command. But perhaps it will make more sense if you look at the output of a non-working configuration….

1. Incorrect LUN masking:

   Here is the LUNLIST output from a server that is having an issue with incorrect LUN masking. The host has not been allowed access to the LUN. The same problem would likely result if the host is not setup in the array at all, or if it was created on the array but someone mis-typed the host’s WWPN. Zoning is correct because the Nameserver Query Response succeeds (line 11) and returns a WWPN target that matches the WWPN target in the boot policy (line 5). The HBA successfully logged into the fabric and was able to see that a LUN of ID 0×00 is visible (line 9). But when the LUN is queried for additional information, it fails with “access failure” (line 7).

   

    

2. Incorrect Zoning:

   In this example, the host is not zoned correctly. It is either in a zone by itself, not zoned at all. This is an easier one to troubleshoot because the host cannot see a LUN nor can it see any available WWPN targets. Look at lines 8 and 9 and notice that there is no response returned for either of these queries. Note that the PLOGI is unsuccessful (fc_id in line 5 is 0×000000) because the host was unable to successfully establish a session with the target.

   
   

3. Incorrect SAN Boot Target in the boot policy:

   In this example, you can clearly see that the WWPN configured in the boot policy (line 5) does not match the available target found on the fabric (line 10). In this situation, the PLOGI (line 5) is once again unsuccessful because a session cannot be established between the host and the target.

   

    

4. Incorrect LUN ID in the boot policy

   In this example, someone entered the incorrect LUN ID into the boot policy for the server (line 7) and it does not match the LUN ID found on the fabric (line 9).

   

    

5. Lastly, I want to show what it looks like when a properly configured host has multiple LUNs presented. I simulated additional targets in the out below, and I wish I could show you actual multiple targets too, but my lab just isn’t that big . However, if you would like to donate a larger array, I’d be happy to include it in my future examples

   

6. One last example is what happens when you run LUNLIST and the OS is up and running with the driver for the VIC loaded.you will get this:

That’s pretty much all there is to it. Hopefully this will be useful to you when you need to troubleshoot a UCS blade that’s not booting from SAN. Remember, you need UCS 2.x (or higher) for the command to work and you can only use the command prior to the OS loading. As always, please let me know your feedback and thanks for stopping by.

-Jeff

Article Update:

If you have more than 2 vHBAs, you may need to know which are which. There is an additional command in the adapter shell you can run called “vnicpci” which will list all interfaces along with their associated server interfaces.

Update #2

If you are using rack servers (UCS C-series), the syntax changes slightly to target the specific server (because there is no chassis). To target rack server 5, for example, you would type:

“connect adapter 0/5/1″ (chassis 0, server 5, adapter 1). Technically speaking, you could just use “connect adapter 5/1″ since rack servers do not require a chassis #, but to keep the syntax as close as possible to blades, I add the “0″ in place of the chassis.