I want to take a second to call out a very special resource that I encourage you to take the time and watch. It’s almost an hour long, but it’s worth it. Hearing about the history behind all of the protocols we now use and probably largely take for granted, in addition to the math behind it all, is a great way of getting perspective.

An Ethernet-based LAN is commonly referred to as a “broadcast” network. Any CCNA can tell you that a broadcast domain is the area in which a broadcast sent by a single node reaches all other nodes. This is a data-link layer concept, but easily most widely attributed to Ethernet. First off, if a unicast destination address of an Ethernet frame is not known to a switch, it will be flooded out all other ports by default. This is Ethernet’s way of giving the Ethernet frame the best chance at reaching it’s destination. Also, any frames addressed to FF:FF:FF:FF:FF:FF are forwarded out all ports.

Ethernet has one way of preventing loops, and that’s split horizon – meaning that broadcast frames entering a switchport are not re-sent out the same port that it came in on. However, if a loop is formed using more than 2 switches, (as is common) then this rule would not apply. As Radia pointed out, Ethernet was not originally meant to be forwarded – it was originally meant to provide link layer connectivity between two points. The Ethernet header has no hop count like a Layer 3 protocol does, so loops can get out of control with only slightly complicated topologies. So, Spanning Tree was born to serve as the other loop-prevention mechanism. Find the less preferred path(s) through the switched LAN and effectively disable them, which is the most barbaric way of creating a loop free topology.

The general consensus is – spanning tree sucks. Obviously Radia invented it for a reason – STP is actually very valuable, seeing as without it, our networks would loop out of control. However, therein lies “the beef” – we hate STP because it is necessary. Ethernet is inherently a broadcast-oriented protocol, and the very native functionality of switching is what’s responsible for causing loops. Using STP means that we have to kill valuable bandwidth that we as customers have purchased. Why can’t we use that port that’s been turned off?

Meanwhile, Layer 3 and the protocols used to build Layer 3 topologies (i.e. routing protocols) function ENTIRELY differently! The forwarding decision is different, loop prevention is handled much differently (more than just the addition of a hop count).

Thinking about all of this, I had an epiphany:

Classical Ethernet is to a distance vector routing protocol as TRILL is to a link-state routing protocol.

As an Ethernet switch, you learn a MAC address on a port by looking at the source address on received frames. You know that traffic destined for that address needs to go out that port, but that’s it. You don’t have a holistic view of the network. An Ethernet switch’s perspective is completely locally significant.

This egress link may be the best way out, it may not be. I’ll tell you one thing – if STP decides that link needs to be blocked, it won’t matter anyways.

At the end of the day, this isn’t that great. Ethernet isn’t used for L3 connectivity for just this reason – suboptimal routing choices are made because of this limited perspective, and we’re not using IP to make these forwarding decisions because IP works on another level entirely – namely, to move packets from one Ethernet to another, or to other mediums. Distance Vector routing protocols in L3 function very similarly – they use metric math to figure out a loop-free topology, because they only know the small amount of information about each prefix that a neighboring router will tell it. There is no holistic viewpoint. This is the reason why distance vector is commonly referred to as “routing by rumor”.

So…we need to find a way to move frames through an Ethernet by using similar forwarding mechanisms to IP, but still look like an Ethernet. Enter TRILL, stage right.

TRILL takes the best parts of link-state routing protocols (neighbor relationships, full knowledge of topology for better forwarding decisions) and applies it to Layer 2. Now, we have bridges that have holistic views of the network, and are able to intelligently deliver frames along every path. We no longer need to kill a link to deliver a loop-free topology.

Fun Fact: IS-IS was chosen over OSPF for a few reasons, but a notable one is the fact that OSPF is pretty much purpose built for IP/IPv6. IS-IS works directly on top of Layer 2 so there’s no need to mess with IP addresses in the configuration of TRILL.

Since TRILL operates by encapsulating the original Ethernet frame and “routing” it at Layer 2 intelligently, we don’t need to rewrite our host networking stacks. Even unknown unicast addresses don’t cause problems, because the TRILL LSDB has enough information to get the frame to where it needs to go. (we just care about getting to the last or egress rbridge). On top of a TRILL header, we have a completely new Ethernet frame on top of it all, where the addressing is done on a next-hop basis (sound a little like IP routing now?).

From Cisco

Conclusion

I didn’t go into detail on FabricPath specifically in this post, but I will mention this article by Cisco on TRILL which is actually co-written by Radia Perlman. It’s a pretty good explanation of the reasons behind TRILL and how it works on the inside.

At the end of the day, TRILL is a step towards the original intent that we should have had for our LANs but because of the history were never able to achieve. Think of all the things we’ve learned about IPv6 now that it’s approaching higher adoption rates. It’s impossible to anticipate every little problem that a protocol may produce, but bandaids like STP and TRILL are necessary to continue to evolve the network, and overcome the scalability challenges that are being presented every single day.

As a reminder, here are the valid combinations (these are still accurate to my knowledge, but may change in a few weeks if any new tech is announced at Cisco Live) of FEX and blade VIC:

I was reminded by a TAC engineer of a command I’d heard of back when I first learned UCS but had totally forgotten.

Through the (heavily underestimated in my opinion) command line of the UCS Fabric Interconnects, you can log directly into the CLI of the FEX/IOM themselves, and run commands that give an ascii-based visual representation of the current connectivity between IOM and blades in a chassis.

F340-31-16-UCS-2-B# connect iom 1
fex-1# show platform software redwood sts
Board Status Overview:
 legend:
        ' '= no-connect
        X  = Failed
        -  = Disabled
        :  = Dn
        |  = Up
        ^  = SFP+ present
        v  = Blade Present
------------------------------

        +---+----+----+----+
        |[$]| [$]| [$]| [$]|
        +---+----+----+----+
          |    |    |    |
        +-+----+----+----+-+
        | 0    1    2    3 |
        | I    I    I    I |
        | N    N    N    N |
        |                  |
        |      ASIC 0      |
        |                  |
        | H H H H H H H H  |
        | I I I I I I I I  |
        | 0 1 2 3 4 5 6 7  |
        +-+-+-+-+-+-+-+-+--+
          - | | | | : | |
         +-+-+-+-+-+-+-+-+
         |-|v|v|v|v|v|v|v|
         +-+-+-+-+-+-+-+-+
Blade:    8 7 6 5 4 3 2 1

Wild, right? Keep in mind that unless you have a 2104 IOM installed, this command won’t work, as “redwood” is the code name for that generation of FEX. If you want to do the same for the 220X series, use the keyword “woodside”:

fex-1# show platform software woodside sts
Board Status Overview:
 legend:
        '  '= no-connect
        X   = Failed
        -   = Disabled
        :   = Dn
        |   = Up
        [$] = SFP present
        [ ] = SFP not present
        [X] = SFP validation failed
------------------------------

(FINAL POSITION TBD)     Uplink #:        1  2  3  4  5  6  7  8
                      Link status:        |  |  |  |  :  :  :  :
                                        +-+--+--+--+--+--+--+--+-+
                              SFP:       [$][$][$][$][ ][ ][ ][ ]
                                        +-+--+--+--+--+--+--+--+-+
                                        | N  N  N  N  N  N  N  N |
                                        | I  I  I  I  I  I  I  I |
                                        | 0  1  2  3  4  5  6  7 |
                                        |                        |
                                        |        NI (0-7)        |
                                        +------------+-----------+
                                                     |
             +-------------------------+-------------+-------------+---------------------------+
             |                         |                           |                           |
+------------+-----------+ +-----------+------------+ +------------+-----------+ +-------------+----------+
|        HI (0-7)        | |        HI (8-15)       | |       HI (16-23)       | |        HI (24-31)      |
|                        | |                        | |                        | |                        |
| H  H  H  H  H  H  H  H | | H  H  H  H  H  H  H  H | | H  H  H  H  H  H  H  H | | H  H  H  H  H  H  H  H |
| I  I  I  I  I  I  I  I | | I  I  I  I  I  I  I  I | | I  I  I  I  I  I  I  I | | I  I  I  I  I  I  I  I |
| 0  1  2  3  4  5  6  7 | | 8  9  1  1  1  1  1  1 | | 1  1  1  1  2  2  2  2 | | 2  2  2  2  2  2  3  3 |
|                        | |       0  1  2  3  4  5 | | 6  7  8  9  0  1  2  3 | | 4  5  6  7  8  9  0  1 |
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+
 [ ][ ][ ][ ][ ][ ][ ][ ]   [ ][ ][ ][ ][ ][ ][ ][ ]   [ ][ ][ ][ ][ ][ ][ ][ ]   [ ][ ][ ][ ][ ][ ][ ][ ]
+-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+ +-+--+--+--+--+--+--+--+-+
  -  -  -  -  |  |  |  |     -  -  -  :  |  |  |  |     -  -  -  :  -  -  -  :     |  |  |  |  -  |  -  |
  3  3  3  2  2  2  2  2     2  2  2  2  2  1  1  1     1  1  1  1  1  1  1  9     8  7  6  5  4  3  2  1
  2  1  0  9  8  7  6  5     4  3  2  1  0  9  8  7     6  5  4  3  2  1  0
  \__\__/__/  \__\__/__/     \__\__/__/  \__\__/__/     \__\__/__/  \__\__/__/     \__\__/__/  \__\__/__/  
    blade8      blade7         blade6      blade5         blade4      blade3         blade2      blade1

This is generated in real-time when you run the command. As you can see, you can not only view the status of the physical ports where the FEX is connected to the FI, but also the backplane ports that typically are very misunderstood. The legend is there, you can see the connectivity provided to each blade.

I highly encourage you to visit this subreddit, the TAC engineer on that thread did a great job of explaining exactly the reason for all of this connectivity.

For those that are not familiar with the idea of distributed switches in general, I’ll overview the concept briefly. Let’s assume you’ve got a virtual deployment that has quite a few hosts. Each host has one or more virtual switches, and each host maintains the control and forwarding functions locally inside the software switch of the hypervisor.

Eventually you tire of administering the individual hosts’ vSwitch configuration to make changes, so you decide to figure out out to do it centrally. Those that lean more towards a virtualization skillset tend to opt for the VMware Distributed Switch. Strictly speaking, and especially for the purposes of this post, the VDS is merely one type of distributed switch. The general idea is that the control and configuration functions of a vSwitch are abstracted, leaving only the software needed to actually move packets around (data plane). This allows for a centralized point of management and control.

This is a logical diagram of course – the vast majority of deployments (for either VDS or Nexus 1000v) results in the control plane actually residing on a virtual machine in the environment being controlled. Does this sound dangerous? It is -unless you know what you’re doing and remember not to trip over your own virtual cables.

Nexus 1000v

I mentioned that the VDS is really one implementation; in this diagram, the function of “control” is fulfilled by vCenter. Let’s refer back to our scenario, and say that you’re either a Cisco-savvy virtualization administrator, or that someone else is, and you’d rather they manage the network switching functions in the hypervisor. This is a good use case for the Nexus 1000v, where the data plane is fulfilled by a Cisco Virtual Ethernet Module (VEM) and the control plane is fulfilled by the Virtual Supervisor Module (VSM). However, the architecture is the same – abstracted control plane, distributed forwarding plane.

Want to know what it’s like when this whole architecture is creating using Open Source Software? Check out Open vSwitch – wicked cool stuff.

The Nexus 1000v VSM is run either as a virtual machine inside the environment, or inside a separate piece of gear called the Nexus 1100 appliance. (It’s basically a server with a locked-down hypervisor on it. By the way – it is not a friend of mine.) The benefits of the VDS in general are realized here, with some added benefits like the familiar NX-OS CLI for those that want it, plus a myriad of features that the VMware VDS still has yet to implement – most notably in my opinion: CoS tagging for virtual port groups.

The Nexus 1000v works by communicating directly with the virtual line cards inside each host using a Control VLAN, as well as direct communication with vCenter, for things like syncing configuration of port groups, etc.

VM-FEX

VM-FEX is actually not too different from the Nexus 1000v in terms of architecture. Both solutions utilize a Cisco-created software module for distributed forwarding functionality (VEM). However, the control plane is found in a different location than with the Nexus 1000v, and the reasons why you would use VM-FEX are quite different, in my opinion.

First off, if you haven’t seen the below video by UCSGuru, take a moment to watch – it will be worth your while, and he does a far better explanation of VM-FEX (and FEX in general) than I could do here.

As you can see, the purpose of VM-FEX is twofold: first off, we want the control plane to reside in some kind of central location, most likely the top of rack or end of row switch. This means fewer points of management while maintaining scalability. The other purpose is to push (or extend – see?) the fabric ever closer to our workloads/virtual machines so that this simplicity is realized within the hypervisor.

The Nexus 7000, 6000, and 5000 series switches, and the UCS Fabric Interconnects (remember that “VM” tab that no one ever touches?) are all able to serve as the control interface. This means that we can manage the remote “linecards” that represent each host through a central point, same as the Nexus 1000v.

That’s not the only difference between VM-FEX and the Nexus 1000v – in my opinion, if it just came down to centralized management, the Nexus 1000v is a much better way to go, but we’ll get to that. VM-FEX goes a step further, allowing virtual machines to plug directly into the fabric being presented (Either a Nexus switch or Fabric Interconnect). Just like all FEX technology from Cisco, this is loosely based on 802.1br (Cisco’s flavor is VN-TAG). As you saw in the video, a big reasons to run VM-FEX is indeed the centralized management component – it, like the 1000v, creates a VDS in vSphere, and allows you to administer the available network port groups from a centralized point. However, with VM-FEX being a hardware-based solution, you get to plug each virtual machine directly into the fabric (The limit with the M81KR is 116 virtual adapters per host).

Cisco says that the 1000v is designed for configuration simplicity – offering a comfortable NX-OS CLI in the virtual space, designed for non-critical virtual machines only. Their recommendation for all “critical”, gotta-be-up-or-bust systems, you deploy VM-FEX.

By the way – VM-FEX can now also be deployed with Windows Server 2012 and Hyper-V. Since the Nexus 1000v and VM-FEX both use the Cisco VEM software module to allow the control plane to interact directly with the hypervisor, both products can now be used with Hyper-V .

Pros and Cons

Hardware Requirements – A Cisco VIC (i.e. 1240, 1280 or m81KR “Palo”) is required to perform VM-FEX in hardware. There are ways to perform VM-FEX using software, when cards from QLogic or Emulex are present, but this defeats the purpose of VM-FEX in the first place, so the point is moot. Bottom line, if you really want to take advantage of this, you need a Cisco VIC, which means you need to be running Cisco UCS. The 1000v is a software solution, so the compatibility requirements are with the hypervisor and management software (like vCenter), not with the underlying hardware.

Limitations and Maximums – If you choose to use VM-FEX in VMDirectPath mode, bypassing the hypervisor entirely, keep in mind that you’re still limited to the number of devices that ESXi can support per host for this function. The vSphere 5.1 documentation points out that you’re limited to running only 8 VMDirectPath PCIe devices per host, and 16 per virtual machine. The 1000v has limitations but they are not very scary. Something like up to 64 hosts/VEMs, 1024 vEthernet interfaces per port profile, 32,000 MAC addresses per VEM, you get the point. And it’s not a hardware solution, so scaling it may not be better, but it is simpler.

vMotion Support – Prior to vSphere 5, deploying VM-FEX in VMDirectPath mode meant preventing vMotion of all involved virtual machines, as there was no way to communicate this move to the fabric. Check out this good overview by Ivan, the trick used here is pretty cool, and it solved a pretty big problem with VM-FEX. Virtual machines connected to the Nexus 1000v have and will be able to vMotion, as the 1000v is merely a replacement for the already software-based virtual switch per host.

Conclusion

I won’t lie – prior to conducting my research, I wasn’t optimistic about VM-FEX – I barely ever hear about it being deployed, and there’s still the glaring truth that should be acknowledged, which is that you have to have a pretty big Cisco investment to even think about it (Ivan calls VM-FEX “the ultimate lock-in”). However – for shops that don’t mind buying Cisco for a while, VM-FEX is not all that bad.

If I were to argue against the deployment of VM-FEX at this point, it would be for purely architectural and operational reasons, and those always depend on many factors. For instance, the Cisco 1000v is a software hypervisor switch, and east-west traffic has the potential to stay local to the host. Any FEX technology means that traffic MUST travel back north to the fabric device before being switched, so even two virtual machines on the same host must hairpin traffic up there in order to talk. The part that “depends” is the fact that this N-S traffic flow might not matter that much. It’s likely you won’t be running into bandwidth constraints, and the latency caused by such a hop is pretty negligible. Again – it depends on your applications.

The other big thing to consider here is administration. If you’re deploying VM-FEX in UCSM, that’s another pane of glass to touch – most shops will get into UCSM a fraction of the amount of time they spend in the switch CLIs or vSphere client. VM-FEX from a straight Nexus switch isn’t that bad, but it still means that someone familiar with a Nexus CLI (usually not a virtualization admin) will have to administer the virtual realm’s networking. Sometimes it’s a sharp learning curve, sometimes it’s not. Again, it depends.

Frankly, I don’t believe VM-FEX should be thought of as merely an alternative to the 1000v. While you can’t really run the two simultaneously on a host, I believe that if it was a question of management simplicity, the N1KV is more than an acceptable choice. Use VM-FEX if you not only want this centralized management but also the other features VM-FEX offers.

For instance, we haven’t each started talking about the performance gains VM-FEX claims to provide. (Read this slideshow – in 2010 Cisco claimed 12-15% CPU performance improvement for standard mode and 30% improvement to I/O performance when using VMDirectPath) The arguments in this space have been going on for some time – some say that the performance is worse with VM-FEX because it means more north-south traffic. Others claim it’s faster because it’s done in hardware as opposed to software (VDS, 1000v), making the extra hop north to the fabric switch insignificant. My findings on this are best saved for another day.

Other Links

http://www.cisco.com/en/US/solutions/collateral/ns224/ns945/ns1134/qa_c67-693220_ns1124_Networking_Solutions_Q_and_A.html

http://www.cisco.com/en/US/prod/collateral/modules/ps10277/ps10331/white_paper_c11-618838_ns1124_Networking_Solutions_White_Paper.html

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/whitepaper_c11-620065_ps10277_Products_White_Paper.html

    
$cmd = "<configConfMos inHierarchical='true'> 
      <inConfigs>
          <pair key='org-root/org-" + $orgName + "/ls-" + $serviceProfileName + "' >    
              <lsServer
                  agentPolicyName=''
                  biosProfileName=''
                  bootPolicyName='" + $bootPolicyName + "'
                  descr='' 
                  dn='org-root/org-" + $orgName + "/ls-" + $serviceProfileName + "' 
                  dynamicConPolicyName=''
                  extIPState='none'
                  hostFwPolicyName=''
                  identPoolName='" + $UUID_POOL_NAME + "'
                	localDiskPolicyName='default'
                	maintPolicyName='default'
                	mgmtAccessPolicyName=''
                	mgmtFwPolicyName=''
                	name='" + $serviceProfileName + "'
                	powerPolicyName='default'
                	scrubPolicyName=''
                	srcTemplName=''
                	statsPolicyName='default'
                	status='created'
                	type='initial-template'
                	usrLbl=''
                	uuid='0'
                	vconProfileName=''>
                	<vnicEther
                		adaptorProfileName='VMWare'
                		addr='derived'
                		adminVcon='any'
                		identPoolName=''
                		mtu='1500'
                		name='" + $VNIC_A_NAME + "'
                		nwCtrlPolicyName=''
                		nwTemplName='" + $VNIC_TEMPLATE_A_NAME + "'
                		order='3'
                		pinToGroupName=''
                		qosPolicyName=''
                		rn='ether-" + $VNIC_A_NAME + "'
                		statsPolicyName='default'
                		status='created'
                		switchId='" + $switchId + "'>
                		</vnicEther>
                		<vnicEther
                			adaptorProfileName='VMWare'
                			addr='derived'
                			adminVcon='any'
                			identPoolName=''
                			mtu='1500'
                			name='" + $VNIC_B_NAME + "'
                			nwCtrlPolicyName=''
                			nwTemplName='" + $VNIC_TEMPLATE_B_NAME + "'
                			order='4'
                			pinToGroupName=''
                			qosPolicyName=''
                			rn='ether-" + $VNIC_B_NAME + "'
                			statsPolicyName='default'
                			status='created'
                			switchId='" + $switchId + "'>
                		</vnicEther>
                		<vnicFcNode
                			addr='pool-derived'
                			identPoolName='" + $WWNN_POOL_NAME + "'
                			rn='fc-node' >
                		</vnicFcNode>
                		<vnicFc
                			adaptorProfileName='VMWare'
                			addr='derived'
                			adminVcon='any'
                			identPoolName=''
                			maxDataFieldSize='2048'
                			name='" + $VHBA_A_NAME + "'
                			nwTemplName='" + $VHBA_TEMPLATE_A_NAME + "'
                			order='1'
                			persBind='disabled'
                			persBindClear='no'
                			pinToGroupName=''
                			qosPolicyName=''
                			rn='fc-" + $VHBA_A_NAME + "'
                			statsPolicyName='default'
                			status='created'
                			switchId='" + $switchId + "'>
                		</vnicFc>
                		<vnicFc
                			adaptorProfileName='VMWare'
                			addr='derived'
                			adminVcon='any'
                			identPoolName=''
                			maxDataFieldSize='2048'
                			name='" + $VHBA_B_NAME + "'
                			nwTemplName='" + $VHBA_TEMPLATE_B_NAME + "'
                			order='2'
                			persBind='disabled'
                			persBindClear='no'
                			pinToGroupName=''
                			qosPolicyName=''
                			rn='fc-" + $VHBA_B_NAME+ "'
                			statsPolicyName='default'
                			status='created'
                			switchId='" + $switchId + "'>
                		</vnicFc>
                		<lsRequirement
                			name='" + $SERVER_POOL_NAME + "'
                			qualifier=''
                			restrictMigration='no'
                			rn='pn-req' >
                		</lsRequirement>
                		<lsPower
                			rn='power'
                			state='down' >
                		</lsPower>
                	</lsServer>
              </pair>
          </inConfigs>
      </configConfMos>"

Invoke-UcsXml -XmlQuery $cmd

If most of your script is composed of normal cmdlets, this looks pretty absurd – so if you can avoid calling direct XML, you should. Same thing if you’re trying to create your own PowerTool-esque library (say, with Python instead of bleh PowerShell), you would take these XML calls and hide them away in modular functions so that you can call them with a single command and a few arguments.

Most other templates (like vNIC and vHBA) have a dedicated cmdlet for creating those constructs. For instance:

Add-UcsVhbaTemplate -Org $organization -Name "BARE-NONP-B" -Descr "Non-Prod Baremetal Fabric B" -IdentPoolName "WWPN-BARE-NONP-B" -SwitchId A -TemplType "updating-template"

As I tab through the cmdlets that start with “Add-UcsServiceProfile” I noticed there was no “Add-UcsServiceProfileTemplate” as one would expect.

Poking around in the release notes for v1.0 of the PowerTool library, I noticed that one of the new features was the ability to filter Service Profiles based on type (aimed at being able to get all the Service Profiles in the system:

# Get all Service Profile Templates.

Get-UcsServiceProfile -Filter 'Type -clike *-template' | select Ucs,Dn,Name

So I ran a Get-Help for this cmdlet:

-Type <string>
    Specifies if service profile or service profile template needs to be created. Valid values are: initial-templat
    e, instance, updating-template

    Required?                    true
    Position?                    named
    Default value
    Accept pipeline input?
    Accept wildcard characters?

So….at least in this version of PowerTool, the same cmdlet is used to create SPs and SPTs, just need to set this “flag” to one of those three options.

For those keeping track, Dave Meyer was on the panel at the time representing Cisco but is now CTO and Chief Scientist with Brocade and getting to do some really cool stuff with OpenDaylight.

They’re at it again – a post on SDN News indicates that in September, a similar symposium will be held, but will encompass the entire Software Defined Datacenter, not just networking. Couple that with the fact that vendors have had quite a bit of time to put together SD* strategies (and even products for some), this should make for an excellent event. Per Tech Field Day MO, the day’s event will be live streamed over the web.

More details to come, I’m sure, but the page is definitely worth keeping an eye on. Check it out!
http://www.sdncentral.com/events/sddc-techfieldday-2013/

I’m close with someone who owns a small business that provides consulting services on products like SAP, where a customer demands a SME on how the software works, and Layer 7 stuff like data migrations from dev to prod. Neither party gives two craps how the infrastructure works. Neither party cares about the plumbing. They may have a few VMs in Rackspace, but they don’t know what Openstack is, nor should they. They just care about apps, because those are what they do, and that’s what brings home the bacon. As a fellow infrastructure person, I want to reiterate that it’s okay to not be this way; there’s plenty of room for people that care about the infrastructure  - that care about the technical differences betwen vBlock and Flexpod. However, at the end of the day, the infrastructure’s main purpose is to run applications, and if it can’t do that efficiently, or change with flexibility, then there’s a problem.

Hanging on to the old way of doing things is what’s going to keep perpetuating this “me vs. you” mentality, which is a dated, unproductive, and costly mentality to have. Right now the apps people are pretty pissed off that every change they need to make requires a request to the “evil infrastructure team”, which will undoubtedly cause some irate emails, work stoppages, and all manner of technical discussions that go nowhere because neither team knows how to communicate with each other.

When you see a blog post saying grand, strange statements like “the network of tomorrow will be configured in Eclipse”, you might get a little squeemish. I know I did at first. Keep in mind that there will always be a physical network. Sure, a lot of functions are moving into the hypervisor, but something will have to connect the hypervisors together.

The key here is to remember that the applications people are waiting for the network to catch up to demand. Not scalability or capacity so much, but flexibility (which is actually a superset of scalability and capacity anways). Flexibility in being able to provision a certain slice of infrastructure within minutes, seamlessly. Flexibility in being able to make a policy change across a massive global infrastructure immediately, and seamlessly. The application people don’t care about the plumbing. They don’t care about changing the oil in a car, just that the car is able to get them from point A to point B, or to point C if needed. The point is that the infrastructure needs to be smarter, and more agile, so that the apps people don’t have to worry about coming to us for a change request. It’s time to get out of the way – it’s time to stop this “us against them” mentality.

Stepping out of the philosophy and into reality, this is the driving factor behind wanting to do everything in software. It’s the reason why commodity hardware driven by open source software (or at the very least open standardized protocols) is so attractive to the bleeding edge technologists right now. These are all building blocks of the infrastructure of tomorrow, because they give us the tools to define exactly what our network needs to do, and change on a whim, when the applications demand it.

Where does this leave the average infrastructure-VAR out there? Most VARs out there that deploy network, compute, storage or even virtual infrastructure are composed mainly of really strong infrastructure folks that are usually very specialized in one particular area. There is a large part of this industry that has never written a script much less knows anything about full-on application development (yes there is a growing group that do). When the infrastructure of tomorrow demands that the infra people are able to take software development methodologies and apply them to network engineering, these folks will need to adapt, or lose their edge. Articles like this one  are way off the mark by making statements like all network engineers will lose their jobs, which of course, is preposterous. However it is true that the times are changing, and those that don’t change with it will encounter some issues. How that change presents itself – or more specifically, how quickly – has yet to be seen, but with the rapid pace of development of cool new toys like OpenStack, OpenFlow, OpenDaylight, and Open vSwitch (see a trend there?), I would say it’s going to happen sooner rather than later.

The winds of change are blowing – be ready.

So, let’s work backwards from the point where our packet leaves *some* interface on a router, which would be considered purely an act of the forwarding plane. In order to get to that point, we need to populate the RIB with some entries.

I established an eBGP neighbor relationship and advertised the 123.123.123.0/24 network to R2:

R2#show ip bgp
BGP table version is 2, local router ID is 10.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 123.123.123.0/24 10.1.1.2                 0             0 321 i

R2#show ip route 123.123.123.0
Routing entry for 123.123.123.0/24
  Known via "bgp 123", distance 20, metric 0
  Tag 321, type external
  Last update from 10.1.1.2 00:00:14 ago
  Routing Descriptor Blocks:
  * 10.1.1.2, from 10.1.1.2, 00:00:14 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 321

Ultimately, the role of a routing protocol, or the RIB, is to maintain topology information, to allow each router to make decisions about which path is best. That sentence is essentially the loose definition of the control plane. Routing protocols simply help the routers exchange this topology information.

Since the routing protocol doesn’t directly control the outgoing interface for a given router, the decision process is simply the identification of the best route, and the next-hop address is produced as a result of that selection. The next-hop address for a given route is usually in the same subnet as one of the router’s interfaces.**

In the case of BGP routes, it is possible that the next-hop address is not a locally significant address, if an eBGP neighbor is not using the “next-hop-self” command. In this case, another routing method like an IGP would be used to get packets to where they need to go. This post will analyse the case where a directly connected next-hop address has been provided.

 Once the next-hop address is known, then the forwarding plane takes over. In the case of Cisco routers, the FIB is powered by CEF, and is responsible for matching a particular flow with a next-hop adjacency to be forwarded to.

R2#show ip cef 123.123.123.0
123.123.123.0/24, version 10, epoch 0, cached adjacency 10.1.1.2
0 packets, 0 bytes
  via 10.1.1.2, 0 dependencies, recursive
    next hop 10.1.1.2, FastEthernet0/0 via 10.1.1.2/32
    valid cached adjacency

With an entry in the FIB, we can see that the outgoing interface for this particular route (learned through BGP as we saw before) is Fa0/0. All packets destined for this subnet will be forwarded out this interface, until a reconvergence of the control plane dictates otherwise.

1. Part 1 – A first look at the CSR 1000v from Cisco
2. Part 1.9 – An examinations of using FHRPs in a virtual environment
3. Part 2 – A comparison of virtual routing redundancy options

Seeing as these were all pretty technical configuration-oriented posts, I wanted to take a step back and think about some of the reasons why one would want to perform routing in a virtual environment. Clearly, the main focus is Data Center, where you get the most bang for your buck. While it’s true that companies like Vyatta can be offered in a physical form factor, the vast majority of those looking to perform x86-based routing will do so in the context of a hypervisor, where the virtual router can enjoy the reliability and mobility that comes with today’s virtual and cloud infrastructures.

Architectural Considerations

First – as I’ve said before – the concept of performing virtual routing is done on a per-tenant basis. Each tenant gets their own routing instance (or pair of instances if you’re doing VRRP, etc). That instance can announce a prefix using an IGP or BGP, and voila – you have the ability to make the vast majority of the connectivity northbound of the hosts L3.

The big idea here is that north-south traffic that for so long has been hairpinned at the physical core switch no longer has to do so. The only traffic that needs to flow north-south is nonrouted traffic between hosts, or traffic leaving the hosts and going somewhere completely different, such as out of the data center. Traffic between tenants or between VLANs on the same tenant can be localized as much as possible.

Cloud providers can either integrate this into their product offerings, meaning that the customer provisioning process includes the provisioning of a device like this, and the connectivity offerings into the customer’s cloud are tightly integrated with the feature sets of the router provided. Users of Openstack could allow Quantum to interact with a virtual router so that the customer is not aware that the routing is being done virtually, and this extra virtual machine does not count against their purchased space.

Connectivity

As depicted in the above diagram, VPN can now be terminated directly into the virtual environment. Without it, we normally would terminate VPN connections to a large router or firewall near the DC edge, and interpret these to VLANs (and then likely some kind of overlay segment) to get into the tenant’s environment.

Vyatta is known to support IPSEC/SSL secure VPNs now. The CSR 1000v does have a few more options, ready to support MPLS and DMVPN (the latter of the two arguably is proprietary).

On Cisco’s FAQ page on the CSR 1000v, they specifically call out the idea of using the CSR as a customer edge device to terminate an MPLS VPN. This produces more control over this connectivity, and allows for some additional per-tenant flexibility. On the other hand, IPSEC and SSL based VPNs are much more widely accepted VPN solutions, and it’s hard to say if this will get any immediate traction.

I also want to remind folks that OTV is among the features included in the CSR. I don’t want to get into a feature set comparison in this post (especially since OTV is proprietary), but it warranted  a mention. I think we’re going to see some interesting OTV topologies if we can now do it directly out of the hypervisor.

Security / SMT

I’d like to differentiate Virtual Routing from Virtual Security. VMware’s vCloud Networking and Security (new version of vShield), Juniper’s vSG, Cisco’s VSG and now ASA 1000v – all completely different from virtual routing products like the CSR 1000v and Vyatta. It should be stated that Vyatta takes a more “swiss army knife” approach, providing firewall and VPN functionality using more of a L3 approach. Cisco would say that the CSR 1000v provides these features, which is true, but they’d also point out that the ASA 1000v is what would be proposed as more of a secure multi-tenant solution.

Now, the routing is recommended from both companies to be performed per-tenant, and the idea that VLANs (or VXLANs if you’re bigger) can be used to segment traffic is not a foreign concept to most.

SDN

Ah, yes – that acronym again. SDN is a powerful use case for this concept but let me be clear – the fact that these solutions run as virtual devices does not make it SDN, and all leading products in the virtual routing space do not currently offer control plane abstraction at present. Yes, they have APIs, so do other solutions. I’m looking for centralized control, and my point in bringing up SDN at all is that x86 based solutions have shown to provide a faster point of entry for SDN. (rephrase)

Cisco has yet to really push the big red SDN button with the CSR (not that I haven’t heard it thrown around a little bit) but it’s clear that Brocade intends to wield the great sword that is Vyatta into the SDN space. Because of the fact that Brocade has yet to announce what that really means, it has contributed to the notion that Vyatta itself is SDN, which of course is false.

I think right now Brocade is weighing it’s next move very carefully with Vyatta. They’re involved with OpenDaylight and have shown no hesitation in support of OpenFlow. Let us not forget that just prior to the acquisition, Vyatta was cooking up something in it’s evil lab called vPlane - still not a shipping product yet, but it’s possible that Brocade is incubating this idea just a little more. Who knows what we’ll see, but I would expect something from Brocade soon, probably in tight integration with OpenDaylight. Again, performing networking and firewall functions in x86 is not SDN, just more conducive to SDN.

More To Come

I won’t get into features or performance in this post. I have just one more post to come in this series concerning an apples to apples comparison of the products that are out there, and I hope that this series continues to spark the conversation.

No doubt – those that have been doing x86 networking for a while will have no problem with this idea. Vyatta is not new like the CSR is, so there are shops out there that have been doing this, both in the hypervisor and in bare-metal applications. I’m reminded by this wave of activity of all the college kids that spent the last few years spinning up linux-based firewalls for their home or small office use for some time (that work really really well), and those geeks are now out in the industry (I’ll raise my own hand on this one).

Should ASIC-based networking go away? Of course not – there are different benefits to that approach. For now, we can agree that this concept is certainly gaining momentum because of the rapid adoption of cloud technologies, and the identification of virtual routing as a realistic need in the near future.

Because of my role, I had the luxury of learning a lot of different areas, and learning them fast. Unfortunately, goals like the CCIE (a fire that has been raging ever since that day in Washington, DC when I passed my CCNP TSHOOT exam) have had to take a backseat because of the fact that focusing on one area was just not possible.

Last week, I started a new position at a technology services company based out of Texas that, among other things, specializes in Data Center offerings from Cisco and EMC. This was ultimately a pretty tough decision to make, but I believe it will allow me to do a few things a little better.

My new role will allow me to focus specifically on data center networking. This means all the way down to the hypervisor, and all the way up through the core and into the WAN. Among many other things, the last two years have had a lot of virtualization and storage experience. While I will definitely be keeping those skillsets as current as possible, this focus will permit me the ability to more efficiently reach two immediate goals. I made a decision to pursue R/S until I accomplish the CCIE. I still strongly believe that having a Route/Switch skillset serves as the best foundation for nearly all other technical learning, and right now, R/S is the lowest hanging of all the fruit. I will move directly back into the DC track after finishing the CCIE R/S, making my second immediate goal the CCIE DC. I work with technology from both tracks so much, it just makes a lot of sense.

I will also be utilizing my existing knowledge to help promote a programmable way of doing data center networking. Whether or not this literally means a certain three-letter acronym that I’m sure we’re all pretty tired of hearing about, or a simple script to make life easier for a few people, I believe that this skillset is a big part of tomorrow’s network engineering staff.

In addition, I am planning on dedicating more consistent time to writing – I’ve learned to really truly love it over the past few years, and for some strange reason, you guys keep on reading. I consider myself honored to simply be listed in the same blogroll as some of these other folks, and I remind myself every day that the reason I do it is to pay forward all of the advice and help I’ve received from those who came before me.

I will not be attending Cisco Live this year. In addition to the simple fact that the move is too close to the date, my short term goals will require some pretty severe focus. Undoubtedly, I will tune in online, and I have already been lucky enough to participate in some pretty sweet NDA presentations concerning material I’m sure we’ll all be hearing about at Live this year. So, suffice it to say that I won’t be putting my head in the sand this year, but I do need to make sure my goals have everything they need to succeed.

I’m constantly reminded that the Unified Skillset is not a goal, or an end-state, but a constant, always changing, never ending journey. I was given some great advice not too long ago – “If you’re not getting better, you’re getting worse”. In some ways, my day-to-day will change, but in a big way, this is still pretty much business as usual. I still love technology, I still strive to be better than I was yesterday, and I still try hard to fill the room with people smarter than me.

Thanks to all who have made the past two years what it is for me.

-Matt

When I first heard the term quite a few years ago, I found that fast food soda machines are a perfect analogy of this idea.

For each type of soda being served in a restaurant, a completely separate dispenser assembly must be made. A new nozzle, a new lever, a new plastic housing. While is true that many of these dispenser assemblies are similar, especially in purpose, they are not the same. We have dedicated a slot for each type of soda. This is the unconverged network.

Move to that fateful day at Taco Bell when I went to grab a cold glass of Baja Blast and – voila!

Pictured: FCoE

 The idea of using these dedicated nozzles was not good enough for Pepsi, and they moved towards a strategy of building – more complicated (and probably expensive) nozzles, yes, but far fewer of them. Simply provide an easy mechanism so that each soda flavor can utilize the same infrastructure, and you have yourself a soda machine that does not compromise on it’s ability to deliver soda, yet it does so in a more efficient way.

Moving out of analogy and into the real world; one of the first big examples of this in the technology realm (at least as long as I’ve been around) is the consolidation of hundreds, or even thousands of dedicated voice circuits at branch offices:

onto the WAN, where they’re carried via IP (VoIP):

No longer is there a need for dedicated connectivity at each remote site – a costly endeavour that’s been in place for so long we’ve forgotten what it was to spend money on anything else. Nonetheless, as WAN providers started to offer more features that gave the network and telephone admins the warm and fuzzies that the WAN could handle the convergence of VoIP, which is arguably one of the most sensitive traffic types out there, the convergence happened, and VoIP is enjoying widespread adoption in such a scenario, where before it was only accepted on the easily controlled environment that is the Campus LAN.

Focusing specifically on Data Center now, the same has happened but even more recently with storage networks. While there is still (and maybe always will be) a use case for the dedicated storage network, carrying nothing but transport for block storage:

…it’s starting to make a whole lot of sense to utilize the great advances made in technology like FCoE (and yes, even NFS and iSCSI) – no doubt given new life thanks to 10/40/100GbE – to create a single fabric that does not care whether the traffic is a write request for a disk somewhere in the fluffy clouds, or your ePayment for that latest eBay purchase.

Networks that are intelligent enough and sized well enough to handle the needs of today AND tomorrow is clearly where these last few years have created for us to operate.

For those that have been reading me long enough know this isn’t the first time I’ve brought up this topic, though my daydreams probably focus more on the “people” side of things. Network convergence is old news – I didn’t write this post to inform the world that this is around the corner, clearly it’s happening. I did write it to point out how it originally started clicking for me back then, as well as serve for a gentle reminder that the technology is moving in this direction, and those pursuing the Unified Skillset will no doubt be the network rockstars of tomorrow.

Symptoms

Since this post was inspired by an experience of mine, I will briefly explain the problem symptoms that surfaced as a result of incorrect settings that will be explored later in the post. A customer was having problems getting vSphere HA to converge properly, and was also having intermittent connectivity between vCenter and the ESXi hosts.

It was pretty bad – the vSphere client was laggy, vCenter’s resource utilization was pretty high, and I was getting strange messages like:

Vsphere has detected that this host is in a different network partition that the master to which vcenter server is connected, or the Vsphere HA Agent on the host is alive and has management network connectivity but the management network has been partitioned. This state is reported by a vSphere HA master agent that is in a partition other than the one containing the host.

Duncan Epping has a great article on Host Isolation, specifically with regards to this error message as well.

It seemed like vCenter was able to get to the hosts, but the client kept refreshing like it was losing connectivity several times a second. I SSH’d into each of the hosts and discovered that they could not ping each other over the management network, and they definitely should have been able to.

 Redundancy Design Options

The myriad of blog posts concerning vSphere Standard vSwitch redundancy typically will show a topology like this:

As mentioned before, you’ll have no trouble finding posts that explain the basics of how each vSwitch NIC Teaming policy works. If you prefer to hear it straight from the horse’s mouth, try VMware’s Virtual Networking Concepts whitepaper. If you prefer a more personal approach, this article by Ken Cline is very popular with respect to this topic.

I’ll assume you’ve at least read those two links by now, so as a quick summary, the available load-balancing policies on a vSwitch are:

1.  Route based on the originating virtual port ID – this selects an uplink when a virtual device (virtual machine or vmkernel) attaches to the vSwitch. Traffic leaving this vNIC destined outside the host will always leave on that pre-determined pNIC, unless that pNIC were to fail.
2. Route based on IP Hash – this is nothing new to anyone familiar with 802.3ad; the idea is that a given packet has a hash made of it’s source and destination IP addresses, and the resulting math will determine which link is used to egress from the host. Traffic from the same IP address within the host to the same IP address outside the host will always leave on the same pNIC, but other traffic flows may fall on another NIC, even if the virtual source is the same.
3. Route based on source MAC hash – this is another hash-based selection mechanism, but since it’s only source based, and will correspond to an actual vNIC, this produces much the same behavior as the very first policy.
4. Use explicit failover order – not used very often. Suffice it to say it’s really not load balancing at all, it’s more of a hardcore deterministic method of placing traffic on the pNIC that you want. Should that pNIC fail, the next will be used.

Now, with respect to the problem I was experiencing in the first section; I had seen this behavior before, and knew that in the network topologies I’d worked with (etherchannel to the hosts in at least some way), the first policy where traffic is routed based on the originating virtual port ID would produce problems. To sum up that story, the customer was indeed using port channels from a Cisco 3750 stack, and after reading a fairly helpful KB article from VMware (ESXi Host Requirements for Link Aggregation), I had the documentation I needed to confirm with the customer that the appropriate policy for this topology was “route based on IP hash”.

Now – many of you know that my roots are not in virtualization, but on the network side, and I refused to leave it at that. That KB article did mention that IP hash method was required if the upstream switching fabric was running link aggregation.

Keep in mind that switch virtualization features like Cisco’s VSS or StackWise are included in this – those technologies are aimed at providing link aggregation without any specific host dependencies other than standardized 802.3ad

What that KB article failed to explain, however, is why this requirement was put into place. It’s clear that the first policy in our list does not work with link aggregation – the experience mentioned at the beginning made that very clear. What wasn’t clear was the technical reason for this. I wanted to know exactly why, when this policy is selected, pings between hosts management vmkernels did not succeed.

This deep-dive resulted in an exploration of exactly what’s happening within the vSwitch. When the “virtual port ID” policy is selected, each virtual NIC is more or less statically pinned to a given pNIC.

Policy 1 – Route Based on Originating Virtual Port ID (Click for Details)

This allows us to have deterministic traffic flows, but just as important, it allows our upstream switching infrastructure to learn a given vNIC MAC address on a single physical switchport, rather than multiple.

Notice in the above diagram that no multi-chassis etherchannel is being attempted. The hosts are simply single homed to two completely separate upstream switches. Sure, those switches provide L2 connectivity for the whole thing using some kind of link between them, but that’s it. Any connection out of these hosts must go to one host or the other.

However, in today’s DC, this topology is becoming increasingly uncommon. Aside from the obvious single point of failure, this requires separate administration of each switch, and running link aggregation is a very common way to provide link redundancy. So, we want to use link aggregation, but how does this impact our virtual environment?

In smaller DCs, this is a fairly common topology:

ESXi Host Networking Redundancy with 802.3ad and Cisco StackWise

Don’t use Catalyst 3750s? The same applies with the 6500′s VSS feature. Same end-result, just a different method. vPC is a different beast, since vPC requires LACP to function properly and as mentioned, LACP is not supported on the standard vSwitch (it should be, though).

As you can see in the above diagram, we’re utilizing etherchannels between the switch stack (logically a single switch so that’s okay) and each ESXi Host. However, our load balancing policy is still set to the default of “route based on originating virtual port”. Remember, this policy pins traffic from a virtual port to a physical port, and traffic from that virtual port can ONLY ever leave the host on the port it was assigned.

Here’s where I stumbled across some additional vSwitch behavior that most resources on the web (actually all that I could find) fail to mention. Not only does this policy restrict the pNIC that a certain virtual entity can use to send traffic out of the host, but it also means that if traffic COMING IN to the host were to be received on another pNIC it is blocked by the vSwitch.

Maybe a visualization will help:

Note that on the left host, the traffic for the management vmkernel port is leaving on VMNIC0 (shown in green). This means that all traffic leaving that vmkernel destined somewhere outside the host will always leave on that NIC. However, the vmkernel for the other host is pinned to VMNIC1 (shown in red). Since other traffic destined for these vmkernels is being sent in to the host on a pNIC other than what it is pinned to, then that traffic is dropped. As I mentioned, most articles speak in depth about traffic leaving the host, but not about traffic being received by the host while in this mode.

The question remains – why is traffic entering what is clearly a non-working NIC for this traffic type? The answer lies in how 802.3ad works. Keep in mind that the recommendation is to NOT use this “virtual port ID” policy when 802.3ad is present, and this is why. Whenever you’re using this policy in vSphere, it’s aimed at allowing the switches to learn a given MAC on a single port, and none others. However, if the switches are configured for a port channel, MACs are not learned on physical interfaces, but on the logical port channel interfaces they’re a member of.

SwitchA#show mac address-table address 6c20.56be.a6c0
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  1    6c20.56be.a6c0    DYNAMIC     Po10

When a frame enters the switch destined for the MAC address shown, the interface listed is a port channel interface. Because of this, the switch defers to whatever load balancing mechanism is configured on that switch – a common one is based off of source and destination IP address, just like the second vSwitch policy. Because of this, traffic heading towards the host may enter on a completely different link. The diagram shown above is more of a worst case scenario – it’s likely that certain traffic flows will work, and others will not. This was the case in my problem at the beginning of this post.

This is why you configure the second policy – “route based on IP hash”, rather than the default “virtual port ID” policy. The “ip hash” policy is the vSphere equivalent of standards-based 802.3ad. It knows the switch may send traffic on whatever link it chooses, so this policy allows such behavior.

Which was ALREADY terrifying enough – but then my mind went here:

Have a great weekend.

1. In my first post, I explained what the different types of QoS policies were in the context of Cisco’s MQC
2. In my second post, I went through the actual configuration on specific platforms like the Cisco Nexus and Unified Compute platforms, as well as a brief mention of vSphere’s participation, but less on the QoS aspects and more on MTU.
3. I also made a QoS-related post that explored why it’s important to have a proper QoS configuration, especially in a converged infrastructure like so many data centers are becoming.

In today’s converged environments, it’s no longer acceptable to simply group all traffic on a single interface into a QoS policy, if the end device is not performing marking of it’s own. While there are many ways to classify traffic, the most efficient way is to use L2 markings, otherwise known as Class of Service (CoS). Since this is in the Ethernet header, performing classification does not require deep inspection of the packet.

Take most VoIP phones out there on the market, Cisco included. The majority allow you to plug that phone into the wall jack, and on that phone, there’s another port that allows you to plug in your PC. The phone essentially serves the purpose of an extremely small switch, allowing both your PC as well as the phone itself to get network connectivity. Therefore, the phone will send both types of traffic upstream. However, it’s important to give the traffic that originated from the phone priority, while traffic that was simply passed through the phone from the PC should get less priority. This is done on the switch level, but the switch can easily figure out which frames are which if they’re marked appropriately.

With the dawn of virtualization, servers are now no different. We have so many applications converging into the data center, we need to make sure that our QoS trust boundary is close enough to the “packet generators” so that we’re classifying priority traffic specifically. There are two ways to do this.

1. Use a server with a lot of NICs. Configure the hypervisor so that the VMs with high priority can only use some subset of physical links, and put all the other VMS on other links. Classifying the traffic is pretty simple in this case – merely treat traffic that arrives on that set of switchports differently than the other switchports. This is essentially physical classification.
2. Classify traffic within the hypervisor, based on the originating virtual switchport (or port group). Use this classification to tag relevant traffic with a CoS value so that the upstream switch can immediately identify the proper class to place the traffic in

Most of the time, if QoS is absolutely needed, and no hypervisor solution is available, then number 1 works….okay. With virtual networking solutions like Cisco UCS where vSphere NICs are no more physical than the virtual machines riding on top, then yeah it’s probably no big deal to rely on the separation of NICs to classify traffic. However, method number 2 isn’t that far-fetched.

By far, the most widely used solution on top of vSphere that allows us to use the second method is the Cisco Nexus 1000v. This product is not for everyone – I think the biggest users of the 1000v are the network people, and some organizations have not been built so that those two silos can interact well. Sad, but true.

This isn’t a sales pitch for the 1000v, but the fact is that right now it’s one of the only solutions that allow you to do this. Classification is fairly easy – just like anything else in MQC, you must define a policy-map in order to do anything:

n1000v(config)# policy-map mark-silver
n1000v(config-pmap)# class class-silver
n1000v(config-pmap-c-qos)# set cos 3

Pretty simple, right? Now we apply it to a vethernet port-profile (where VMs plug in):

n1000v(config)# port-profile type vethernet SILVER_VMS
n1000v(config-if)# service-policy input mark-silver

Traffic that egresses hosts on this port profile will now have a CoS tag of 3 in their Ethernet header, allowing you to easily classify traffic out of a single NIC if you wanted to.

The 1000v is actually capable of some pretty advanced QoS features, considering what it is. Check out the QoS configuration guide for info on  policing, classification, marking, and more.