THIS WILL BE ONE OF THE LAST RELEASES OF THE MIBS FOR NET-SNMP

During the OpenBSD 5.1 development cycle, I committed the CARP MIB to the base OpenBSD snmpd. The kernel sensor MIB has been in the base snmpd for a few releases now. That leaves the pf MIB which was committed to 5.1-current some weeks ago and will be present in the 5.2 release.

So, you’ve got a few options.

1. Want to still run Net-SNMP with the extra MIBS? Go to the SNMP MIBs page and follow the directions. No change from previous versions. However, make plans to migrate away from Net-SNMP for OpenBSD 5.2.
2. Only use the CARP or kernel sensors MIB? Use the base snmpd(8). There’s no configuration necessary, just run the daemon. The MIB files are in /usr/share/snmp/mibs/ (The pf MIB file is present there, but the implementation is not part of snmpd(8) in 5.1). You should also read my guide on what’s changed between the Net-SNMP and snmpd(8) implementations of the MIBs.
3. Want to use the base snmpd(8) but still have a requirement for Net-SNMP? See my blog post on using both together.

• Who is this user? A guest? An internal user? A member of the Finance department?
• What device is the user bringing onto the network? A corporate PC? A Mac? A mobile device?
• When are they connecting? Are they connecting to the secure network during regular business hours or at 02:00 in the morning?

These questions can all be answered easily within ISE and are all standard policy conditions that are relatively easy to implement. In the post below I’m going to focus on the How? — How is the user or device connecting to the network? Asked another way, the question is Wired? or Wireless?

Why Does Wired or Wireless Matter?

TL;DR answer: Because if you can differentiate between the two types of users, you can apply different policies to each group. (And you almost always want to apply different policies to wired vs wireless users).

Longer answer: The most basic ISE policy would typically assign the user to a specific VLAN. This is really the most fundamental policy action. If corporate user, then put in VLAN X. If guest user, then put in VLAN Y. Policy complexity goes up from there which causes an even greater divergence of wired and wireless policy. Now, if you evaluate your wireless users against that same policy, does VLAN X and Y even exist on the wireless network? Probably not. Your WiFi network is probably either a more enterprise-y deployment with distributed access points (APs) and central controllers that have their own VLANs for bridging user traffic or you’ve got standalone APs doing the AAA requests themselves in which case they will ignore the VLAN that ISE sends and will dump every user into the same VLAN (which violates policy).

It’s important to apply unique ISE policies to wired and wireless users because it’s typical to have different business/security/technical policies for them.

RADIUS Attributes

Underneath the covers ISE uses the RADIUS protocol to perform authentication, authorization, and accounting (AAA) functions. Between a client (the switch, access point or wireless controller where the user is connected) and the server (ISE) RADIUS passes attribute/value pairs (AVPs). These AVPs allow client and server to pass relevant information back and forth. For instance, the client informs the server of the username, password and MAC address of the end user trying to get onto the network. The server in turn replies with AVPs of its own which can include things like whether to allow or deny the access, the IP address the client should take and how long they’re allowed to stay on the network.

In addition to the basic AVPs, there’s a lot of AVPs exchanged that carry metadata about the session. This metadata is what’s used to answer the Wired? and Wireless? questions.

Inspecting RADIUS Attributes

Ok, so the RADIUS AVPs are being exchanged, how do we inspect them? This is the job of a “condition” in ISE.

I equate policies in ISE to an if-then-elseif-then ladder. The individual rules in the policy equate to a single “if” or “elseif” rule. The ISE conditions then are the expressions that are evaluated in the “if” or “elseif” rule. In fact, the policy editor screen even expresses the rules in an if-then fashion:

Cisco ISE If-Then-ElseIf-Then Ladder

As you can see in the screenshot, the conditions Cisco-IP-Phone, isFinanceUser, and isCorporateUser are consumed by the policy rules but their actual definition is not visible when looking at the policy. The conditions are defined separately from the policy which allows a condition to be treated as a building block and reused in multiple rules. My example policy below will use this feature when testing if a client has initiated an 802.1X connection.

Condition Categories

Conditions are defined in the Policy, Policy Elements, Conditions menu in ISE. There are six categories of conditions but I’ll only be looking at the Authorization (authz) conditions. Authz conditions come in two flavors: simple and compound. Simple conditions evaluate exactly (1) expression while compound conditions can evaluate multiple expressions with each either logically ANDed or ORed together.

Building the Conditions

ISE does most of the heavy lifting for us in the conditions department. It comes with policies for matching wired and wireless users already. However what I’ve found useful is to create conditions to match specific wireless networks as identified by their SSID. This allows specific policies on a per SSID basis.

Inside Policy, Policy Elements, Conditions, Authorization, Compound Conditions, add a new policy. Name the policy and give it a description. Then choose to “Create New Condition”.

Creating a New Condition

Select the expression under “Radius” called “Called-Station-ID”. For a WiFi user, this AVP is sent to ISE as the MAC address of the access point and the SSID concatenated together with a colon (:) between them. Since the MAC address is irrelevant to our policy, we’ll use a “Matches” operator instead of “Equals” to just inspect the SSID part. Do not use “Equals” with a value of the SSID. It won’t work. “Match” conditions take regular expressions as a value. To match the SSID “Corp” use:

.*:Corp$

This says to match any number and type of character (.*), a colon (:), and the word “Corp” which must appear at the very end of the AVP value ($ means “at the end”).

SSID_Corp Condition

That’s it for conditions. We’ll pair this with the existing one called “Wireless_802.1X”.

Updating the Policy

Now that we have the building blocks of our policy rules created we can update the policy. Written in plain language, our mock policy is this:

• Users in the finance department who connect to a wired port using 802.1X must be put in their own VLAN
• All other corporate users connecting on a wired port using 802.1X must be put in the general user VLAN
• Any user who connects to the “Corp” WiFi network using 802.1X must only connect between the hours of 08:00 and 18:00
• Everyone else is denied access

Even though I’m mentioning the policy results (the VLANs and time restrictions) above, I’m not going to show how to create them.

Translating our plain language policy into ISE means we’ll have to modify our existing finance and corporate user rules, add a rule for the Corp WiFi network, and modify the default rule from its stock “allow all” posture to “deny all”.

Whereas the simplest policy rules — and even the stock rules that come with ISE — only use a single condition, we’ll need to use multiple conditions which are logically ANDed together to get our desired results. For example, in our first rule we’ll need to look for members of the finance department AND users who are connecting on a wired 802.1X port. By adding conditions from the library, this rule can be built up to match our business policy.

Add Condition to the Rule

Notice the choice of ANDing or ORing the conditions together.

The AND/OR Selector

There it is. The first rule is complete. Now update the wired corporate user rule and create the WiFi rule and the policy should look like so:

The Policy

By using the built-in conditions to test for wired and wireless connectivity and creating a custom condition to look for a specific SSID, unique policy results can now be applied to users based on answers to the questions Wired? and Wireless?

• Layer 2 plug and play characteristics
• Layer 2 adjacency between devices
• Layer 3 routing and path selection
• Layer 3 scalability
• Layer 3 fast convergence
• Layer 3 Time To Live field to drop looping packets
• Layer 3 failure domain isolation

An article on FabricPath could go into a lot of detail and be many pages long but I’m going to concentrate on five facts that I found particularly interesting as I’ve learned more about FabricPath.

#1 – FabricPath is not a network topology

When I first started learning about FabricPath, I believed that it came with a requirement that your network topology conform to certain rules. While I now know that is not true, there is a common topology that is discussed when talking about network fabrics. It’s called the spine+leaf topology.

This is similar to a traditional collapsed core design with a few differences.

• When we’re talking about a fabric, all links in the network are forwarding. So unlike a traditional network that is running Spanning Tree Protocol, each switch has multiple active paths to every other switch.
• Because all of the links are forwarding, there are real benefits to scaling the network horizontally. Consider if the example topology above only showed (2) spine switches instead of (3). That would give each leaf switch (2) active paths to reach other parts of the network. By adding a third spine switch, not only is the bandwidth scaled but so is the resiliency of the network. The network can lose any spine switch and only drop 1/3rd of its bandwidth. In a traditional network that runs Spanning Tree Protocol, there is no benefit to scaling horizontally like this because STP will only allow (1) link to be forwarding at a time. The investment in an extra switch, transceivers, cables, etc, is just sitting idle waiting for a failure before it can start forwarding packets.

So while the spine+leaf topology is commonly used when discussing FabricPath, it is not a requirement. In fact, even having full-mesh connectivity between spine and leaf nodes as shown in the drawing is not a requirement. You could connect each spine to every other leaf. You could connect spines to other spines or a leaf to a leaf.

According to Cisco, there is a lot of interest from customers about using FabricPath for connecting sites together (ie, as a data center interconnect or for connecting buildings in a campus). An example of that might be a ring topology that connects each of the sites.

The drawing shows FabricPath being used between the switches that connect to the fiber ring. This is obviously a very different topology than spine+leaf and yet perfectly reasonable as far as FabricPath is concerned.

FabricPath is a method for encapsulating Layer 2 traffic across the network. It does not define or require a specific network topology. The rule of thumb is: if the topology makes sense for regular old IP routing, then it makes sense for FabricPath.

#2 – FabricPath introduces its own unique data plane

In order to achieve the benefits that FabricPath brings over Classical Ethernet, some significant changes needed to be implemented in the data plane of the network. Among these changes include:

• The introduction of a Time To Live field in the frame header which is decremented at each FabricPath hop
• A unique addressing scheme consisting of a 12-bit switch ID which is used to switch frames through the fabric
• A Reverse Path Forwarding check is done on each frame as it enters a FabricPath port (another loop prevention mechanism)
• A new frame header format with these new fields

In order for the hardware platform to switch FabricPath frames without any slowdown, new ASICs are required in the network. On the Nexus 7000, these ASICs are present on the F series I/O modules. It’s important to understand that not only do the FabricPath core ports need to be on an F series module but so do the Classic Ethernet edge ports which carry traffic belonging to FabricPath VLANs. This last requirement may impact certain existing environments where downstream devices are connected on M1 or M2 I/O modules.

FabricPath is also supported on the Nexus 5500 running NX-OS 5.1(3)N1(1) or higher. Cisco’s documentation isn’t exactly clear how FabricPath is implemented on the 5500 series but I’ve been told 55xx boxes do it in hardware (the original 50xx boxes do not support FabricPath).

#3 – FabricPath does not unconditionally learn every MAC in the network

One of the key issues with scaling modern data centers is that the number of MAC addresses each switch needs to learn is growing all the time. The explosion in growth is due mostly to the increase in virtualization. Consider a top-of-rack, 48-port Classical Ethernet switch that connects to 48 servers. That’s 48 MAC addresses that this switch and all the other switches in the network need to learn to send frames to those servers. Now consider that those 48 servers are really VMware vSphere hosts and that each host has 20 virtual machines (an average number, probably low for some environments). That’s 960 MAC addresses. Quite an increase. Now multiply that out by however many additional ToR switches are also servicing vSphere hosts. All of a sudden your switches’ TCAM doesn’t look so big any more.

Since FabricPath continues the Layer 2 adjacency that Classical Ethernet has, it must also rely on MAC address learning to make forwarding decisions. The difference, however, is that FabricPath does not unconditionally learn the MAC addresses it sees on the wire. Instead it does “conversational learning” which means that for MACs that are reachable through the fabric, a FabricPath switch will only learn that MAC if it’s actively conversing with a MAC that is already present in the MAC forwarding table.

Consider Switch 2 in this example. Host A is reachable through the fabric while B and C are reachable via Classic Ethernet ports. The MACs of B and C are learned on Switch 2 using Classic Ethernet rules which is to say that they are learned as soon as they each send frames into the network. The MAC for A is only learned at Switch 2 if A is sending a unicast packet to B or C and their MAC is already in Switch 2′s forwarding table. If A sends a broadcast frame into the network (such as when A is sending an ARP ‘who-has’ request looking for B’s MAC), Switch 2 will not learn A’s MAC (because the frame from A was not addressed to B, it was a broadcast). Also if A sends a unicast frame for Host D, a host that Switch 2 knows nothing about, Switch 2 will not learn A’s MAC (destination MAC must be in the forwarding table to learn the source MAC).

The conversational learning mechanism ensures that switches only learn relevant MACs and not every MAC in the entire domain thus easing the pressure on the finite amount of TCAM in the switch

#4 – FabricPath ports do not have IP addresses

One area where FabricPath gets confusing is when it’s referred to as “routing MAC addresses” or “Layer 2 over Layer 3″. It’s easy to hear terms like “routing” and “Layer 3″ and associate that with the most common Layer 3 protocol on the planet — IP — and assume that IP must play a role in the FabricPath data plane. However, as outlined in #2 above, FabricPath employs its own unique data plane and has been engineered to take on the best characteristics of Ethernet at Layer 2 and IP at Layer 3 without actually using either of those protocols. Below is a capture of a FabricPath frame showing that neither Ethernet nor IP are in play.

Instead of using IP addresses, an address — called the “switch ID” — is automatically assigned to every switch on the fabric. This ID is used as the source and destination address for FabricPath frames destined to and sourced from the switch. Other fields such as the TTL can also be seen in the capture.

#5 – FabricPath employs Equal Cost Multipath packet forwarding

In Classic Ethernet networks that utilize Spanning Tree Protocol, it’s no secret that the bandwidth that’s been cabled up in the network is not used efficiently. STP’s only purpose in life is to make sure that redundant links in the network are not used during steady-state operation. That’s a poor ROI on the cost to put in those links and from a scaling/capacity perspective, it’s equally as poor since the network is limited to whatever the capacity is of that one link and cannot employ multiple parallel links. (Ok, you technically can using etherchannel but you understand the point I’m trying to make)

Since FabricPath doesn’t use STP in the fabric and because the fabric ports are routed interfaces and therefore have loop prevention mechanisms built-in, all of the fabric interfaces will be in a forwarding state capable of sending and receiving packets. Since all interfaces are forwarding it’s possible that there are equal cost paths to a particular destination switch ID. FabricPath switches can employ Equal Cost Multipathing (ECMP) to utilize all equal cost paths.

Here S100 has (3) equal cost paths to S300: A path to each of S10, S20, and S30 via the orange links and then from each of those switches to S300 via the purple links.

Much like a regular etherchannel or a CEF multipathing situation, FabricPath ECMP utilizes a hashing algorithm to determine which link a particular traffic flow should be put on. By default the inputs to the hash are:

• Source and destination Layer 3 address
• Source and destination Layer 4 ports (if present)
• 802.1Q VLAN tag

These values are all taken from the original, encapsulated Ethernet frame.

An interesting value-add that FabricPath does is to use the switch’s own MAC address as a key for shifting the hashed bits. This shifting prevents polarization of the traffic as it passes through the fabric (ie, prevents every switch from choosing “link #1″ all the way through the network due to their hash outputs all being exactly the same). The benefit of this is only realized if there’s more than (2) hops between source and destination FabricPath switch.

So there you have it. Are you currently using or planning a FabricPath deployment? Please share your thoughts in the comments below.

For some background, I’ve been using the LG Optimus Pad running Android 3.0 for the last year or so and an HTC Desire running Android 2.3 for about two years. Prior to that I had an HTC Dream running Android 1.5 and then 1.6. I recently started using a 3rd-gen iPad with WiFi & 4G. My perspective on all this is from a long-time Android user who is stepping into the Apple camp for the first time. I use my tablet for productivity, studying, and some social networking. I’m not a gamer.

Apps

My first impression of the Apple App Store is how polished it appears. Every app has multiple screenshots and very detailed descriptions of what it does and what was changed in the latest version of the app. I’m not sure if this is a policy or just a convention adopted by app developers, but it works. In the Google Play store (formerly the Android Market) it can be hit and miss as far as what information the developer shares about their apps. I know I’ve seen lots of comments from users asking the developer to describe what they’ve changed in their latest version.

Just a gut feel here, but the App Store also doesn’t seem to have as many “junk” apps in it. I suppose this is a direct result of Apple’s tight control over what gets published in the store. As a result, my impression while browsing through the store was that I didn’t have to be quite as careful about what I installed given the knowledge that Apple had tested and vetted each app. Of course most people are familiar with the Google Play store and their reputation for allowing most anything into their store. This has both pros and cons but in this case, knowing that a second set of eyes was scrutinizing an app before I installed it made the Apple App Store feel a bit safer.

One thing that really surprised me was how different the same app can be between Android and iOS. Evernote is the best example I have. Not only does the app have a different interface, but it functions differently too. Browsing through notebooks on iOS is easy and intuitive and best of all, editing a note works exactly the way you expect it to. Highlighting text and applying formatting changes works without errors as does creating and modifying bulletted lists. I’ve had difficulty doing all of these things in the Android app and had to compromise by using much less formatting in my notes than I would’ve liked.

How I had to compose notes on Android

How I'm able to compose notes on iOS

As a result of the quality of apps like Evernote on iOS, I expect my use of these tools and my resulting productivity to rise appreciably.

My last observation on the apps is how much more I’m spending in the App Store than in Google Play. Many more apps are available for free in the Play store (eg, Angry Birds) and on top of that, the paid apps in the App Store generally seem to cost more than paid apps in the Play store. It’s not uncommon to see an app in the App Store for $9.99 or more whereas that would be considered an “expensive” app in the Play store. That said, taking an average of the 24 most popular paid apps currently listed in each store results in an average cost of $2.69 for the Play store and $2.78 for the App Store. Perhaps my impression is skewed because I’m purchasing a bunch of iOS apps all at once vs spread over a few months on the Android tablet.

Hardware & Software

Ok, the hot-button issue first: software. Android software updates can be very frustrating because of how the ecosystem works. First Google releases a new version into the wild. The hardware manufacturers then have to take it and do their modifications, testing and certification on their various devices. The wireless carriers then get involved (in the case of a mobile phone or a 3G-enabled tablet) and go through their own rounds of testing and certification. Finally after what normally takes many months, it’s in the hands of the end user. Case in point, Android 4.0 (Ice Cream Sandwich) was released in October of 2011. Now five months on, devices like my LG Optimus Pad or the Samsung Galaxy Tab 10 still don’t have an official upgrade. By contrast, when a new version of iOS is released it doesn’t take long at all to get into the hands of the end user. With over-the-air upgrades now possible, the update can be put onto the device from almost anywhere.

On the hardware front, the thing that I noticed most of all with Android devices is the lack of choice when it comes to 3G-enabled tablets. This is partially due to the wireless carriers in my area only carrying a single Android tablet each and partially due to the lack of choice across the market. There are way too few 3G-enabled tablets coming to market in a time when mobile data growth and adoption is through the roof. This blows my mind and is my single biggest sticking point on the whole Android ecosystem. Apple, in my opinion, got it right by giving consumers the choice of 3G from the beginning.

Operation

One feature I love in Android is how apps can share directly with one another.

Sharing a URL on with another app on Android

For example, when viewing an article in an RSS reader, the URL can be shared with a second app. This can either cause the app to open so that further work can be done by the user or the app can process the information that has been “shared” with it in the background (for example, that’s how the Read It Later app does it when it saves the URL to your reading list without you having to leave the RSS reader). The benefit of this is that once Read It Later is installed and linked with your RIL account, any other app on the device can take advantage of that linkage. Conversely on iOS, the reader itself would need to link with your RIL account. This means that for every app that wants to send something to RIL it has to be configured with your RIL username and password.

One feature I’m finding I really like in iOS is how backgrounded apps stay exactly where you left them for as long as you choose to leave that backgrounded app running. In Android — and I notice this most often with web browsers — the backgrounded app will eventually be terminated. This causes the application state to be lost which means whatever tabs or pages were open are lost. In iOS I can leave that browser open all night and come right back to the exact spot I left it the night before. The list of backgrounded apps on iOS also contains every single app that is running. On Android — at least with 3.0 and 2.3 — that list is limited in size and only shows the very most recently used apps.

Disaster Recovery

Disaster recovery isn’t really a term associated with consumer tech, but since I’ve come to rely so heavily on the tablet and have now loaded it with all kinds of information, it is something I’ve been thinking about. On my Android devices I was left with two options out of the box. Option #1 was to find an app in the Play store which was capable of backing up my device. Since I’m not comfortable storing my backups in the cloud, I want something that would save the backups locally and then allow me to copy them off to my PC. I never actually researched an app that would do this though (which I’ve always kind of marveled at since as an IT professional I understand the risk of not having those backups). Option #2 was to use the native Android “backup my device” feature. Again, I haven’t used this either since I’m not willing to store my backups in the cloud but some quick research indicates that this feature isn’t totally reliable and that not all apps are compatible which really defeats the purpose.

Apple seems to have done a better job in iOS. First there’s the backup that iTunes grabs every time the iPad is tethered to my PC. Next, to make things even easier, there’s now WiFi syncing with iTunes which means I don’t even have to tether to get that backup done. And if that wasn’t easy enough there’s also an option to backup to iCloud which means a backup can be taken anywhere as long as there’s Internet connectivity.

And of course, the flip side to taking a backup is restoration. Again, Apple seems to have gotten this right. The first time the iPad is turned on it asks if this is a new device or if it should attempt to restore a backup from iTunes or iCloud. Apple thought about the restoration right up front. Android? I honestly don’t know. And that’s problem enough. I never had confidence in the “backup my device” feature because I couldn’t find any concrete information about how to do a restore.

My Thoughts So Far

Walking out of the big box store, iPad in hand, my bank account was over $800 lighter. The iPad, the smart cover, the extra power adapter and the Zagg screen protector all add up. I don’t spend that kind of money lightly and I wondered as I drove off if this was going to be a costly mistake. Well, about an hour after opening the box, I was hooked. This thing is so easy to use. And it hit me: that’s exactly what I wanted. I didn’t want a “project” gadget that I would have to hack together and figure out workarounds for. I needed a tool that would keep me productive and that would get out of my way while working on it. The iPad is it.

My Android experience has been long and taken me through multiple devices. The Google Play store has a lot of great apps but I’ve found that even best of breed ones like Evernote are buggy and not totally polished. This lead to me finding workarounds and making compromises which made the tablet obtrusive when using it. Nowhere was this more apparent than when browsing the web. I have four browsers installed on my Android tablet (stock Browser, Dolphin for Pad V1.0 Beta, Opera Mobile and OverSkreen) and none of them are fully functional. Each of them suffers in at least one of these areas: speed, stability, Flash playback, and interface design. Without a doubt though Android does one thing better than Apple’s iOS ever will: integrate with Google products. Enter your Google username and password once during device setup and you’re hooked into your Gmail, contacts, calendar, RSS feeds, Google+ and so on. Being a big user of Google’s apps, this was a killer feature for me and one that I miss on the iPad.

Conclusion

I draw no conclusions here, good, bad or ugly. My intent is not to put one platform ahead of the other, but to give insight into the experience and opinion of a long-time Android user who recently made the plunge into the Apple world. To each their own.

Apple’s Bonjour protocol has been getting some attention lately and not because the new iPad was recently released, but more because of the rate of adoption of iDevices in the corporate world.

Bonjour is Apple’s implementation of networking “zeroconf”: a group of protocols and methods for connecting devices to a network with “zero configuration” on the part of the user or the network administrator (Ever seen a 169.254.x.x IP address? That’s one part of zeroconf). The part of Bonjour that is getting attention is the service discovery (SD) piece which allows a consumer (ie, an iPad) to discover providers (ie, printers, Apple TVs, etc) automatically. SD relies on link-local multicast messages to facilitate the discovery. This works great in a flat, Layer 2-only home network (which is where most iDevices have lived until recently) but breaks down in an enterprise network made up of multiple Layer 2 domains. If the consumer device is not in the same Layer 2 domain as the provider device, it will not be able to discover the provider.

The first article I read this week about Bonjour was from IT World Canada: Apple’s Bonjour protocol tamed for enterprise Wi-Fi. They go on to talk about Aruba Networks’ recent announcement of support for the Bonjour protocol (specifically mDNS used for SD) in their wireless LAN controller software.

[Aruba Networks] announced Thursday an update to its WLAN controller software that lets multicast domain name services, like Apple’s Bonjour protocol, work simply and securely but without creating a drag on the network. The new feature is called AirGroup, a reference to Apple’s nomenclature for capabilities like the mobile printing service AirPrint and AirPlay.

The article also mentions Aerohive’s announcement of a controller-less access point that acts as an SD gateway between Layer 2 domains. More information on Aerohive’s announcement is available in their blog post titled Breaking Subnet Boundaries with Bonjour: Simplifying Apple TV and AirPlay in the enterprise.

In designing this feature, we wanted to preserve the plug-and-play nature of Bonjour. We wanted our customers to be able to flip a switch and see services throughout the network. We specifically did NOT want our customers to have to learn a lot about the Bonjour protocol design, how to route and flood multicast frames, and think carefully about how to replicate multicast traffic between VLANs or smoosh devices together on to one Bonjour super-VLAN.

Lastly, the Packet Pushers recently did a podcast with Aerohive where they talk about their upcoming Bonjour gateway product: Show 94 – Aerohive and Bonjour Gateway. It’s worth a listen.

Instantly familiar with Unix servers – In the larger, more mature environments there’s typically Unix systems that exist to support network operations. From trusted hosts/jump boxes to syslog and RADIUS/TACACS servers, Unix is typically the platform of choice. When coming into an environment like this, where others would face a learning curve in dealing with these systems, network engineers with Unix knowledge are able to start using them right away with little effort. Additionally, being familiar with daemons such as syslogd, tftpd, radiusd, etc, makes it really easy to start using these tools in support of the network. Whereas other network engineers likely stop learning about the Unix systems once they’ve learned how to login and do essential tasks only, a Unix background allows the engineer to swiftly navigate through the system and leverage the tools and services that are there to do a better job of maintaining the network.

Leg up in understanding Unix-based network operating systems – From Arista’s EOS to Juniper’s Junos, lots of network operating systems can trace their roots back to open source Unix. Sure they all have their own CLI shell that mostly hides the hard, Unix-y center, but underneath it’s still there. Being able to understand process management, memory management, file system mount points, heck, even the root user and what it’s capable of gives a network engineer an immediate advantage on these OSes even though they might not have seen them before.

Scripting and automation – Part of Unix culture is being able to create scripts and bits of code to help administer a system or automate a repetitive task. Those ideals are really valuable in the networking world too. Having the ability to create scripts which help manage network devices makes a network engineer more efficient and all but eliminates the likelihood of human error. Knowledge of PERL, shell scripts and even PHP enables the creation of tools for doing bulk config changes, mass backups, and automated reporting. This enables an engineer to concentrate on high-value tasks such as building out the network rather than tedious, low-value (but still important!) tasks.

Hands-on experience with ports, packets, and protocols – Considering that Unix is a very network-centric operating system, it’s very hard to learn Unix and not pick up network knowledge at the same time. Learning what a TCP/UDP port is, understanding how sockets are opened and closed, even something very simple like memorizing the TCP/UDP port number of common services (Pick a random port number and ask a Unix person what it’s used for. I bet they know the answer) are all byproducts of working on Unix systems.

Most Unix systems have at least one network sniffer installed (tcpdump, Wireshark, etc) which makes them easy to play with and learn about. They also give you experience troubleshooting bits on the wire and help to visualize how multilayer network models (like the OSI model) work in practice. Sniffer experience also helps drive home the theory of how certain protocols work such as TCP with its 3-way handshake; nothing breeds understanding like seeing it in action, or better yet, troubleshooting it. Having quick access to a sniffer on a Unix system makes it really effortless to pick it up and start using.

Most Unix systems also come with some sort of firewall software. Cutting your teeth on these firewalls can gain you valuable experience and knowledge in understanding traffic flows, writing firewall policies, and further understanding of the TCP 3-way handshake, windowing, and sequence numbers. That knowledge will help you in both managing enterprise firewall systems and talking with security/firewall teams in a language that they understand.

Lastly, since Unix boxes are able to run just about every network service under the sun — ftp, smtp, dns, dhcp, tftp, www, ntp, snmp, various databases, and so on — it’s convenient (and dare I say fun??) to play with these tools and gain experience and knowledge of these services and application layer protocols. This experience is invaluable when it comes to troubleshooting issues in a production environment because you’re able to understand how the service works right from the lowest network layers all the way up through the stack to the application layer. This understanding combined with firewall experience can be a killer combination when it comes to provisioning and troubleshooting.

Professional advancement – So far I’ve talked only about the technical and knowledge benefits of being a network engineer and Unix geek but there’s a whole other aspect: career advancement. Unix experience allows you to set yourself apart from others by solving problems they can not, bringing value to your team and department by way of writing scripts and tools, and demonstrating deep understanding of ports, packets, and protocols. By standing out from the crowd you make youself more valuable to your employer which increases your job security and also your chances of promotions, raises, etc. WIN!

Summary – Although I didn’t realize any of these things at the time they were happening, I can see it now looking back. Unix has made me a better network engineer and I believe all network engineers would benefit from a moderate level of Unix experience. These two seemingly unrelated areas of skill and knowledge are actually more related than they first appear and can be mutually beneficial to those that claim both as part of their overall skill set.

SR-IOV requires support in the BIOS as well as in the operating system instance or hypervisor that is running on the hardware. Until very recently, I had been under the impression that SR-IOV was handled solely in hardware and did not require any software support; unfortunately, I was mistaken. Software support in the operating system instance or hypervisor is definitely required.

Like Scott, I didn’t realize there was a software dependency. Here’s the kicker though:

I do not have a timeframe for SR-IOV support in VMware vSphere or Microsoft Hyper-V.

Ouch. I immediately wondered how the Virtual Interface Cards (VICs) in Cisco UCS were able to successfully present multiple vNICs and vHBAs to the hypervisor. Was Cisco pulling a fast one? Was this only possible if the UCS node was running an OS like Windows or Redhat on bare metal and not when running vSphere or Hyper-V?

To the Google! Actually, first to bradhedlund.com. Brad (@bradhedlund) is well known in the blogosphere and Twitterverse as a Cisco UCS authority. He plainly points out [Cisco UCS criticism and FUD: Answered, answer to question #6] that the UCS VIC does not employ SR-IOV but uses alternate, standards-based PCIe methods of presenting virtual hardware to the hypervisor.

To further drive the point home, the datasheet for the first generation “Palo” adapter [UCS M81KR Virtual Interface Card] and the gen 2 1240/1280 cards [UCS Virtual Interface Card 1240 Data Sheet] both state that IO virtualization is achieved without the use of SR-IOV.

So there we go. Question asked and answered. I don’t quite understand what the “alternate methods” are, but I know that the VIC adapters will work just fine with vSphere and Hyper-V.

On a side note, the thing that led me down this rabbit hole was Scott’s recent post on SR-IOV support coming in the next version of Hyper-V [SR-IOV Support in the Next Version of Hyper-V].

So after getting my hands dirty in PERL and TCL/expect, I found that somewhere between v2.3.1 and 2.3.8, the logic that governs the login process to a Cisco device was changed. RANCID tries to determine if at first login, it’s being dropped into privileged mode or regular exec mode. In v2.3.8 it does this by hunting for two characters in the output coming off the switch during login:

• The greater-than sign “>”, meaning it’s exec model, and…
• The octothorpe! “#”, meaning it’s privileged mode

Well, turns out, on the switches that weren’t being collected any more we use the octothorpe in the login banner. Things like “This switch is in cabinet #1234″. Really innocuous stuff. However, RANCID sees this character and believes it’s seeing the prompt for privileged mode. As a result, it doesn’t bother issuing the “enable” command. This causes problems later on when it tries to run commands like “show run”.

As per the RANCID FAQ, their take is that you need to avoid using “>” and “#” in the banners. This is annoying, especially for a character as common as “#” and also pretty challenging to drive home in a team of multiple network engineers.

My solution was to amend the regular expression that RANCID uses to find the prompt. It now looks for the “>” or “#” characters (as well as the string ” (enable)” in case you’re on an CatOS box) followed by optional whitespace and then a line return.

--- bin/clogin.orig     Tue Mar 20 10:00:35 2012
 +++ bin/clogin  Tue Mar 20 10:22:01 2012
 @@ -750,7 +750,11 @@
 }

      # Default prompt.
 -    set prompt "(>|#| \\(enable\\))"
 +    # This has been modified from the stock regexp by adding the "\\s*$". This
 +    # works around our use of the '#' character in the login banner at $WORK.
 +    # Note that this would fail if we had the '#' character at the end of a
 +    # line in the banner, but for now, we don't.   -jknight 2012.03.20
 +    set prompt "(>|#| \\(enable\\))\\s*$"

      # look for noenable option in .cloginrc
      if { [find noenable $router] == "1" } {

For us, anchoring the regexp to the end of the line worked because we don’t use the octothorpe at the end of line, just in the middle of sentences and maybe at the beginning of lines. If you have a banner like this:

#####################
# THIS IS MY SWITCH #
#####################

This won’t work and you’ll still have issues.

Here’s how to run both at the same time and leverage snmpd for the OpenBSD-related MIBs and the Net-SNMP daemon for its ability to retrieve data from scripts and extend itself using loadable modules and smux sub-agents.

One of the unique features of the Net-SNMP daemon is its ability to proxy SNMP requests and forward them to another agent. By using this feature, queries for OpenBSD-related OIDs can be proxied through to the OpenBSD snmpd and Net-SNMP can handle the rest.

The settings for Net-SNMP are configured in /etc/snmp/snmpd.conf and should look like this:

proxy -v2c -c public 127.0.0.1:161 .1.3.6.1.4.1.30155
agentaddress 172.16.0.3:161


The proxy setting takes the same basic arguments as snmpwalk(1), snmpset(1), etc. You should explicitly set the SNMP version and community string as these are not passed through from the original SNMP query. This community string is used when querying the OpenBSD snmpd so make sure it matches with how snmpd has been configured (see below). The IP and port are where the OpenBSD snmpd is listening and the OID string is where all the OpenBSD-related MIBs are rooted at. Net-SNMP will proxy any requests for children of this OID to the OpenBSD snmpd.

The agent must be manually bound to a specific IP (as opposed to the default of “all IPs on the machine”) using the agentaddress setting because of the second listener on 127.0.0.1. You can’t have one listener on a specific IP and the other on “all IPs” for the same port.

I know what you’re thinking: put snmpd on a port other than 161. As of this writing, you cannot configure the listening port in the OpenBSD snmpd, only the listening IP address.

The configuration for OpenBSD’s snmpd is done in /etc/snmpd.conf and should look like this:

listen on 127.0.0.1
read-only community public

This is enough to allow Net-SNMP to query snmpd. Other options may be present in your snmpd.conf but these two are the only ones related to the coexistence with Net-SNMP.

Note: If you send traps from the OpenBSD snmpd, you can still set “listen on” to 127.0.0.1. Trap PDUs will be sourced from the outgoing interface IP address.

With the settings above, Net-SNMP will still handle queries for standard MIBs such as HOST-RESOURCES-MIB. These too can be proxied through to OpenBSD snmpd by using the OIDs below.

Please post a comment if you are one of the people who needs to keep the Net-SNMP daemon around. It’d be great to build up the features in the OpenBSD snmpd so we can one day eliminate the need for Net-SNMP.

Now that OPENBSD-CARP-MIB and OPENBSD-PF-MIB have been added to the base snmpd in OpenBSD (CARP-MIB will be in 5.1-release, PF-MIB in 5.2, and the SENSOR MIB has been there since 4.5), I wanted to document the differences between these MIBs and the corresponding implementation of the MIBs that I wrote for Net-SNMP.

Both implementations provide the same set of OIDs and allow the same data to be retrieved. Whatever you were querying via Net-SNMP is available via snmpd.

What has changed is the base OID where the CARP and PF MIBs are rooted at as well as the name of certain OIDs.

Difference #1 – New base OpenBSD OID

The Net-SNMP implementation used a private/reserved enterprise OID of 64512 (ie, .1.3.6.1.4.1.64512) as the base OID for all the OPENBSD-* MIBs. In the snmpd implementation, the OpenBSD enterprise number 30155 is used as the root for OPENBSD-* MIBs.

Fix: Change all occurrences of 64512 to 30155 in all query strings

NOTE: All the OIDs after the enterprise OID should remain unchanged. Eg, a query string of .1.3.6.1.4.1.64512.a.b.c.d should be modified to .1.3.6.1.4.1.30155.a.b.c.d with a.b.c.d remaining unchanged.

Difference #2 – New CARP OID

The only exception to the rule that the a.b.c.d in .1.3.6.1.4.1.64512.a.b.c.d should remain unchanged is with the CARP MIB. Due to an overlap in OID assignments, the CARP MIB had its base OID changed from 3 to 6 in the snmpd implementation. When fixing query strings that refer to the CARP MIB, .1.3.6.1.4.1.64512.3.b.c.d should be changed to .1.3.6.1.4.1.30155.6.b.c.d

Fix: If your query string uses the OID name, no changes are necessary. Just ensure your NMS and SNMP tools are loading the MIB definition (OPENBSD-CARP-MIB.txt) from /usr/share/snmp/mibs/. If your query string uses numeric OIDs, change .1.3.6.1.4.1.64512.3.X to .1.3.6.1.4.1.30155.6.X

Difference #3 – New OID names in PF-MIB

In order to avoid naming conflicts between OIDs (not just within the OPENBSD-* MIBs, but with 3rd-party MIBs as well) some of the OID names in the PF-MIB were modified in the snmpd implementation to make them more unique and to identify them as belonging to PF-MIB. For example, “running” has been renamed to “pfRunning”, “memory” to “pfCntMemory”, and “tcpFirst” to “pfTimeoutTcpFirst”.

Fix: If you use OID names in your query strings, replace any Net-SNMP names in the table below with the corresponding snmpd name.

This table shows the old Net-SNMP name and the corresponding new snmpd name. Note that some of the old names are duplicates (such as “fragment”, “removal”, “count”, etc) so be careful if you’re doing a search & replace.

info

pfInfo

running

pfRunning

runtime

pfRuntime

debug

pfDebug

hostid

pfHostid

counters

pfCounters

match

pfCntMatch

badOffset

pfCntBadOffset

fragment

pfCntFragment

short

pfCntShort

normalize

pfCntNormalize

memory

pfCntMemory

timestamp

pfCntTimestamp

congestion

pfCntCongestion

ip-option

pfCntIpOption

proto-cksum

pfCntProtoCksum

stateTable

pfStateTable

state-mismatch

pfCntStateMismatch

state-insert

pfCntStateInsert

state-limit

pfCntStateLimit

src-limit

pfCntSrcLimit

synproxy

pfCntSynproxy

count

pfStateCount

searches

pfStateSearches

inserts

pfStateInserts

removals

pfStateRemovals

loginterface

pfLogInterface

name

pfLogIfName

ipBytesIn

pfLogIfIpBytesIn

ipBytesOut

pfLogIfIpBytesOut

ipPktsInPass

pfLogIfIpPktsInPass

ipPktsInDrop

pfLogIfIpPktsInDrop

ipPktsOutPass

pfLogIfIpPktsOutPass

ipPktsOutDrop

pfLogIfIpPktsOutDrop

ip6BytesIn

pfLogIfIp6BytesIn

ip6BytesOut

pfLogIfIp6BytesOut

ip6PktsInPass

pfLogIfIp6PktsInPass

ip6PktsInDrop

pfLogIfIp6PktsInDrop

ip6PktsOutPass

pfLogIfIp6PktsOutPass

ip6PktsOutDrop

pfLogIfIp6PktsOutDrop

sourceTracking

pfSrcTracking

count

pfSrcTrackCount

searches

pfSrcTrackSearches

inserts

pfSrcTrackInserts

removals

pfSrcTrackRemovals

limits

pfLimits

states

pfLimitStates

sourceNodes

pfLimitSourceNodes

fragments

pfLimitFragments

timeouts

pfTimeouts

tcpFirst

pfTimeoutTcpFirst

tcpOpening

pfTimeoutTcpOpening

tcpEstablished

pfTimeoutTcpEstablished

tcpClosing

pfTimeoutTcpClosing

tcpFinWait

pfTimeoutTcpFinWait

tcpClosed

pfTimeoutTcpClosed

udpFirst

pfTimeoutUdpFirst

udpSingle

pfTimeoutUdpSingle

udpMultiple

pfTimeoutUdpMultiple

icmpFirst

pfTimeoutIcmpFirst

icmpError

pfTimeoutIcmpError

otherFirst

pfTimeoutOtherFirst

otherSingle

pfTimeoutOtherSingle

otherMultiple

pfTimeoutOtherMultiple

fragment

pfTimeoutFragment

interval

pfTimeoutInterval

adaptiveStart

pfTimeoutAdaptiveStart

adaptiveEnd

pfTimeoutAdaptiveEnd

sourceTrack

pfTimeoutSrcTrack

interfaces

pfInterfaces

ifTable

pfIfTable

ifEntry

pfIfEntry

ifIndex

pfIfIndex

ifDescr

pfIfDescr

ifType

pfIfType

ifRefs

pfIfRefs

ifRules

pfIfRules

ifIn4PassPkts

pfIfIn4PassPkts

ifIn4PassBytes

pfIfIn4PassBytes

ifIn4BlockPkts

pfIfIn4BlockPkts

ifIn4BlockBytes

pfIfIn4BlockBytes

ifOut4PassPkts

pfIfOut4PassPkts

ifOut4PassBytes

pfIfOut4PassBytes

ifOut4BlockPkts

pfIfOut4BlockPkts

ifOut4BlockBytes

pfIfOut4BlockBytes

ifIn6PassPkts

pfIfIn6PassPkts

ifIn6PassBytes

pfIfIn6PassBytes

ifIn6BlockPkts

pfIfIn6BlockPkts

ifIn6BlockBytes

pfIfIn6BlockBytes

ifOut6PassPkts

pfIfOut6PassPkts

ifOut6PassBytes

pfIfOut6PassBytes

ifOut6BlockPkts

pfIfOut6BlockPkts

ifOut6BlockBytes

pfIfOut6BlockBytes

tables

pfTables

tblTable

pfTblTable

tblEntry

pfTblEntry

tblIndex

pfTblIndex

tblName

pfTblName

tblAddresses

pfTblAddresses

tblAnchorRefs

pfTblAnchorRefs

tblRuleRefs

pfTblRuleRefs

tblEvalsMatch

pfTblEvalsMatch

tblEvalsNoMatch

pfTblEvalsNoMatch

tblInPassPkts

pfTblInPassPkts

tblInPassBytes

pfTblInPassBytes

tblInBlockPkts

pfTblInBlockPkts

tblInBlockBytes

pfTblInBlockBytes

tblInXPassPkts

pfTblInXPassPkts

tblInXPassBytes

pfTblInXPassBytes

tblOutPassPkts

pfTblOutPassPkts

tblOutPassBytes

pfTblOutPassBytes

tblOutBlockPkts

pfTblOutBlockPkts

tblOutBlockBytes

pfTblOutBlockBytes

tblOutXPassPkts

pfTblOutXPassPkts

tblOutXPassBytes

pfTblOutXPassBytes

tblStatsCleared

pfTblStatsCleared

tblAddrTable

pfTblAddrTable

tblAddrEntry

pfTblAddrEntry

tblAddrTblIndex

pfTblAddrTblIndex

tblAddrNet

pfTblAddrNet

tblAddrMask

pfTblAddrMask

tblAddrCleared

pfTblAddrCleared

tblAddrInBlockPkts

pfTblAddrInBlockPkts

tblAddrInBlockBytes

pfTblAddrInBlockBytes

tblAddrInPassPkts

pfTblAddrInPassPkts

tblAddrInPassBytes

pfTblAddrInPassBytes

tblAddrOutBlockPkts

pfTblAddrOutBlockPkts

tblAddrOutBlockBytes

pfTblAddrOutBlockBytes

tblAddrOutPassPkts

pfTblAddrOutPassPkts

tblAddrOutPassBytes

pfTblAddrOutPassBytes

labels

pfLabels

lblTable

pfLabelTable

lblEntry

pfLabelEntry

lblIndex

pfLabelIndex

lblName

pfLabelName

lblEvals

pfLabelEvals

lblPkts

pfLabelPkts

lblBytes

pfLabelBytes

lblInPkts

pfLabelInPkts

lblInBytes

pfLabelInBytes

lblOutPkts

pfLabelOutPkts

lblOutBytes

pfLabelOutBytes