Fear not. It’s not as bad as it sounds. I’ll explain why.

First off, yes, the docs are correct, you do have to enable the http and https services on your access switches. CWA does not work if those commands are not present. I found the admin guide was weak in explaining the need for the http server when doing CWA. It’s clear that if you’re doing local web auth, where the switch is serving up the portal pages, that you need the http server turned on but it’s not totally clear you need that for CWA. The Switch Configuration Required to Support Cisco ISE Functions document does a bit better job.

So that’s the bad news. Here’s the good news: the users don’t actually establish an IP layer connection to the http service on the switch.

CWA works by redirecting the user’s browser to the web auth portal running on the ISE Policy Service Node when the user first tries to hit the web. In order for this redirection to happen transparently, the switch needs some help from the http server service. Think of enabling the http server kind of like enabling a feature that allows the switch to do Layer 7 inspection of tcp/80 and tcp/443 packets.

Ok, so the guest users don’t hit the http server but the service is still running which means your switches now have two extra tcp ports open on the management plane that they otherwise wouldn’t. Well, we can’t close the ports, but we can put up an ACL on the management interface which blocks incoming connections to port 80 and 443.

ip access-list extended SWITCH_MGMT
deny tcp any any eq 80
deny tcp any any eq 443
permit ip any any

If you don’t care about the ports being open and only want to prevent management of the switch via the web UI, you can turn off the management web pages without turning off the entire http service using these commands:

ip http secure-active-session-modules none
ip http active-session-modules none

So there you go. In this case, you can have your cake and eat it too.

This is the third post in my series on virtual routing. Other articles in the series are

• An Introduction to Layer 3 Traffic Isolation
• Configuring VRFs on IOS and Junos

Shared Services Objectives

The shared services module has 3 main objectives:

1. Provide shared network services to multiple VRFs by way of common infrastructure
2. Permit select traffic exchange between VRFs and shared services resources
3. Prevent traffic exchange between VRFs

The last one isn’t a reason for setting up a shared services module (ie, you don’t setup shared services in order to try and keep traffic from passing between VRFs) but instead is a very important design objective. Since the shared services module has connections into multiple VRFs and is exchanging routes with them, due care is needed to prevent the shared services area from becoming a transit path between VRFs.

Now, you might think that since you’re connecting your VRFs together via a common area that you’re increasing your risk of traffic crossing between VRFs (ie, guests being able to access your corporate network resources). The natural reaction is to stuff some firewalls in the mix. Filter everything in and out of the VRFs! Firewalls will keep us safe! Keep in mind though that each VRF only forwards traffic toward a destination if it has a route for that destination. As long as routes are being filtered properly in the shared services area, VRF A will not contain routes for networks in VRF B (and vice-versa). If a user sends a packet into VRF A with a destination address belonging to a network in VRF B, the worst case is that the packet will follow the default route in VRF A (which probably leads towards the Internet exit point in that VRF) and best case it will just be dropped because there is no matching route. Either way, even without a firewall, traffic does not cross from A to B.

Remember: Traffic will not be directed towards (or through) the shared services area unless there is a matching route present in a VRF’s routing table that points towards the shared services area.

Having said that, we can’t throw the firewall out the window. Objective #2 is to provide selective traffic exchange between VRFs and the shared services and that’s where the firewalls come in. They will be used to permit only the traffic that corresponds to the specific services that are being provided. For example, if one of the shared services is DNS, only allowing port 53 traffic into the shared services area.

Cheating

The basic topology for this shared services module is shown here:

The consolidated router in this topology is actually a Juniper SRX firewall running Junos and will be performing route exchange, route filtering, and firewall duties all in the same physical box and in a single firewall instance.

The consolidated router is configured with one routing instance for each VRF in the network. Each instance is of type “virtual-router” and is used to exchange routes with the VRFs in the network as well as containerize those routes.

By importing explicit prefixes between each routing instance and the global, aka “master” routing instance, route filtering is achieved. The OSPF process running within each routing instance is then directed to pick up these imported prefixes and advertise them to its neighbors (which are OSPF processes running in the VRFs on the edge routers).

OSPF has no ability to filter outgoing route updates. The edge routers could be configured with distribute lists to prevent incoming route updates from entering their routing tables however this would not prevent the prefixes from being present in the OSPF link state database. More importantly, this method creates two points of management for route filtering vs. a single point on the consolidated router.

BGP may be a better protocol to use between the consolidated router and the edge routers as it would more easily allow filtering of sent and received prefixes. It would probably remove the need for routing instances at all (thus making the design more in line with Cisco’s validated design of having a single routing table in the shared services area). However I’m basing this post on a real life deployment which actually used OSPF.

 By employing routing instances on the consolidated router it’s possible to:

• Accurately filter routes in and out of the shared services area
• Extend and reuse the concepts of virtualized routing tables which are already being used in the core of the network (and therefore something we’re already familiar with)
• Avoid the need to manage route filtering in more than one location; everything is done on the consolidated router

On top of this, a firewall policy will be deployed that permits only specific traffic based on typical 5-tuple information. So even though a VRF has full routes for the shared services area, the firewall policy will control exactly what traffic is allowed into (and out of) the area.

The Configuration

For brevity, I’m only going to show the configuration related to the blue VRF/routing instance. I’m also not going to show the firewall security policy as this post is meant to focus on the routing configuration.

First, configure the instance.

routing-instances {
    blue {
        instance-type virtual-router;
        interface ge-2/0/1.0;
        interface ge-3/0/1.0;
        protocols {
            ospf {
                area 0.0.0.0 {
                    interface ge-2/0/1.0;
                    interface ge-3/0/1.0;
                }
            }
        }
    }
}

This gives us a basic virtual-router instance with two interfaces bound to it and OSPF running within it. The edge routers should be configured similarly with with their respective interfaces bound to the blue VRF and OSPF enabled.

The two interfaces ge-2/0/1.0 and ge-3/0/1.0 face the edge routers and are configured as regular point-to-point /30 interfaces.

interfaces {
    ge-2/0/1 {
        description "Connects to Edge Router #1";
        unit 0 {
            family inet {
                address 10.2.0.1/30;
            }
        }
    }
}

At this point the blue routing instance should contain all of the routes from the blue VRF.

The interface facing the shared services area is a regular interface with no associated routing instance; it’s part of the default instance (named “master” on the SRX platform).

interfaces {
    ge-2/0/0 {
        description "Connects to shared services switch";
        unit 0 {
            family inet {
                address 10.10.0.1/24;
            }
        }
    }
}

At this point the “master” routing instance only contains the “direct” route for the /24 network configured on ge-2/0/0.0.

Now, to import routes into “master” from “blue”, configure an appropriate policy statement that selects routes in “blue” and apply that statement as an import policy on the “master” routing instance.

policy-options {
    policy-statement routes_from_vrfs_to_master {
        term from_blue {
            from {
                instance blue;
            }
            then accept;
        }
        term default_deny {
            then reject;
        }
    }
}
routing-options {
     instance-import routes_from_vrfs_to_master;
}

This config will suck all the routes from the “blue” routing instance into the “master” instance. To be a little more granular, a prefix-list could be used in the “from” stanza of the policy-statement to only permit specific prefixes but that would limit reachability to only a portion of the blue network. The default_deny term is important; you’ll want to make sure that’s the last term in the policy-statement.

Now to do the reverse and import routes into blue from master. This is where the filtering becomes relevant.

policy-options {
    prefix-list ss_prefixes {
        10.10.0.1/24;
    }
}

This prefix-list specifies the network(s) that belong to the shared services area and is used to filter the list of routes imported into blue.

policy-options {
    policy-statement routes_from_ss_to_vrfs {
        term 1 {
            from {
                instance master;
                prefix-list ss_prefixes;
            }
            then accept;
        }
        term default_deny {
            then reject;
        }
    }
}

The above chunk selects prefixes in the master table that match the prefix-list.

routing-instances {
    blue {
        routing-options {
            instance-import routes_from_ss_to_vrfs;
         }
        protocols {
            ospf {
                export routes_from_ss_to_vrfs;
            }
        }
    }
}

And this chunk applies the policy in order to:

• Import the routes into blue from master
• Export the routes out of blue to OSPF so that the prefixes are advertised to the edge routers

At this point traffic should flow bidirectionally between shared services and the “blue” VRF while at the same time preventing traffic flow between “blue” and any other VRF.

The four sessions were:

• Journey to the Cloud
• Cisco UCS
• Data Center Networking
• Powering the Cloud

Journey to the Cloud

Presented by Ronnie Scott (Cisco)

According to Cisco’s own measurements:

• 77% of IT time spent keeping lights on (opex spend)
• 23% spent delivering new capabilities (capex spend)

Drivers for adopting cloud:

• Reduce IT cost
• Simplify IT operations
• Improve pace of delivery
• Better align IT resources to business needs

Many keywords used when describing “cloud”: ubiquitous, convenient, on-demand, shared pool, rapidly provisioned, minimal management.

• This is taken from the NIST definition of cloud (A NIST definition of cloud computing (PDF))
• If you’re only running big servers with vSphere, you’re not operating a cloud. You’re operating a virtualized environment.
• Virtualization is a stepping stone to cloud
• To be a cloud, the environment must:
• Be provisioned on-demand using self service
• Have broad network access
• Have pooled resources
• Be able to rapidly expand
• Be measurable

Virtualization is a stepping stone towards a cloud architecture. However the level of virtualization must be broad and complete. Everything must be virtualized:

• Server/compute
• Network
• Storage

To run a real cloud, enterprises need to move to a capacity planning model and away from a per project spend/resource allocation model. Set high watermarks on the pool of resources and buy more/plan for more when the HWM is hit.

• New applications/services being introduced as part of a project should consume resources from the existing pool of compute/storage/network. No (or minimal) need to purchase equipment that will only run this new app/service.
• IT budgeting and spending models need to change. With a cloud architecture, capital outlay is necessary when capacity planning says so, not when project plan for a new app says so.

Cloud delivery models:

• SaaS/Application – Aimed at end users; provides applications at scale
• PaaS/Platform – Aimed at developers; provides execution platform at scale
• IaaS/Infrastructure – Aimed at system administrators; provides infrastructure at scale

IaaS/PaaS – Targeted at running applications with variable resource demand (VDI, DR, application test/dev).

SaaS – Delivers functional apps with low per user/transaction cost (Email, UC, web hosting)

Cisco has a new cloud model: virtual private cloud

• Cloud services that simulate private cloud in public cloud infrastructure

Cisco sees a model for the future where cloud environments utilize a brokerage service with intelligence to move workloads between public/private/community clouds. Applications would be tagged with certain criteria that specifies the level of service it requires. The broker would use this information to dynamically place the workload into a certain cloud environment based on configured policy.

Cisco’s cloud strategy:

• Build the base
• Core, “crown jewel” apps run in-house on enterprise private cloud
• Rent the spike
• Workload spikes satisfied with public/community clouds

Elements of a private cloud

• Self service portal
• Users order services
• Service delivery automation
• Delivery of those ordered services is automated
• Resource management
• Resource allocation to the ordered services is managed
• Operations management
• Integration of the service with operations tools, help desk tools, SLA monitoring, etc
• Life cycle management
• Cradle to grave: the ordered services need to be deprovisioned at some point in their lifetime too.

Architecture for private cloud

• Resources (compute, storage, bandwidth)
• Resource management (vCenter, UCS Manager UCSM, BMC, etc)
• Service management (vCloud Director, BMC)
• Customer access, self provisioning tools (BMC, Cisco Intelligent Automation for Cloud CIAC)

The service management layer needs to talk to the resource management layer in order to provision the physical resources from the pool. The elements in the resource management layer need APIs to make this possible. Cisco products like UCS and the Nexus line all have rich, complete APIs which make this automation possible.

Steps for moving to private cloud:

• Consolidate
• Virtualize
• Automate
• Self service provisioning

Cisco UCS

Presented by Willem van Schaik (Cisco)

Cisco UCS

• Fastest growing business ever at Cisco
• $1bn dollar business
• 2nd in blade server market in North America
• Cisco Reaches 10,000 UCS Customers

Nexus 5000 virtual chassis

• Nexus 5k acts like a supervisor
• Nexus 2k fabric extender (FEX) acts like remote line card
• Nexus 5k + 2k model was extended to Cisco UCS
• FEX is only network piece installed in the chassis
• Management of the system is done via the Fabric Interconnect (fancy Nexus 5000s running UCS Manager software on top of NX-OS)

10Gbe benefits

• More bandwidth is nice, but not the biggest win
• Low latency is nice, but not the biggest win
• Biggest win is consolidating multiple networks and traffic onto one cable
• Builds on “virtualize everything” concept (virtualizing the network)
• Wire once, use many
• Or as Cisco says, wire for bandwidth not connectivity

Generic rackmount servers vs. generic blade servers:

• Most rack server components (NICs ports, storage ports, etc) duplicated 1:1 on a blade
• Same number of ports and connections to manage in a 16x blade enclosure as with 16 rackmount servers
• Additionally, every blade enclosure is (at best) its own point of management (worst case, every single blade is its own point of management)
• No opex savings with blades

With UCS:

• All points of management moved to single spot: top of rack in the Fabric Interconnect
• No mgmt point in back of the blade enclosure (only thing in the back is a dumb fabric extender/FEX)
• UCS chassis are like a storage disk shelf. Plug and use, nothing to configure, no management points on the shelf/chassis
• One point of management for N chassis/blades

UCS Manager (UCSM)

• Management software for entire UCS environment
• Runs on Fabric Interconnects
• RBAC (Storage, server, network teams all use same console)
• Full API for integrating with other tools. UCSM GUI written on top of the XML API
• Other tools can use the API to do the same tasks as UCSM (eg, software from CA)
• API allows management from any platform. Apps exist for iPhone/iPad, Android and Playbook
• Central firmware mgmt for all components
• Fabric Interconnect, Converged Network Adapters, FEX, etc
• Firmware can be staged (redundant firmware banks) in hours and applied on a scheduled or off hours basis

Service profile

• Wraps all the server parameters and policies into a template
• Lives in UCSM not on the blade
• Creates the personality of the server

The new 6200 series Fabric Interconnect

• Unified ports
• Port can be configured for data center bridging or fiber channel
• 6248 based on Nexus 5548
• Expect to see 6296 based on Nexus 5596

UCS 2.0 increases bandwidth to blades

• 8 port FEX (up from 4 ports in first version)
• Port channel can now be created between the FEX and the upstream Fabric Interconnect
• No more pinning a blade to a specific upstream port on the FEX
• 1 blade can use up to 40Gb of bandwidth out of the chassis (limited by the bandwidth from the FEX to the blade; see the bullet point right below)
• From FEX down to blade, increased lanes on the highway from 1x to 4x 10Gb lanes
• Also in a port channel
• 40Gb per port per blade

VM-FEX: virtual NICs on the Palo card become remote ports on the Fabric Interconnect

• FEX idea cascades from Fabric Interconnect to FEX and from FEX to Virtual Interface Card (Palo) on the blade
• 802.1Qbh/VN-Tag (pre-standard) used to identify traffic to and from individual VMs
• Fabric Interconnect creates a logical interface for each VM (based on the VN-Tag). Other end of that logical interface is the virtual NIC created on the Palo card.
• All switching done in the FI, ALWAYS. Even traffic between two VMs on the same vSphere host must flow up to the FI and then back down
• Allows policy, control and monitoring of all traffic
• VMDirectPath mode gives best performance
• Switching all done in ASICs
• In this mode, Vmotion is supported under ESXi 5

Data Center Networking

Presented by Ronnie Scott (Cisco)

Fiber Channel over Ethernet is standardized. Period. Competitors are spreading FUD if they say otherwise.

Fiber channel encoding is inefficient

• FC uses 8b/10b encoding
• 4Gb FC != 4Gb throughput
• 10Gb Ethernet uses more efficient encoding scheme. 4Gb FCoE is more efficient on the wire than native 4G FC
• 16G FC will have more than double the real throughput of 8G FC because it adopts the same encoding used with 10GbE.
• 3-for-2: The FCoE Bandwidth Bonus (blogs.cisco.com)

Spanning Tree vs. Virtual Port Channel

• 50% of network bandwidth is wasted in an STP network
• STP is absolutely necessary through to eliminate loops
• VPC allows all links to be forwarding
• Two upstream switches “collude and lie” to southbound switch (VPC == “lies and deceit protocol”)

FabricPath

• New method of transporting frames in the data center
• No STP
• All links active
• Layer 2 adjacency between end devices over a routed, Layer 3 fabric
• Spine and leaf design
• High bisectional bandwidth
• FP based on TRILL which was standardized before anyone had an implementation
• TRILL has some things that need changing to add things that weren’t in the standard
•  and should be (according to Cisco)
• Likely to see a TRILL v2 at some point
• Work already underway on next version of TRILL
• FabricPath enabled with one command per interface. No configuration needed.
• Don’t be surprised if FabricPath is the default interface mode in the data center one day

Overlay Transport Virtualization – OTV

• Layer 2 adjacency between end devices in different data centers across a Layer 3 data center interconnect
• Manages traffic tromboning of FHRP VIP (FHRP becomes active in both DCs)
• Prevents STP BPDUs from crossing the DC interconnect
• Enables Vmotion between data centers
• Vmotion between data centers has distance/latency limits!
• Kudos to Ronnie for actually bringing this up. Most discussions on OTV throw around the idea of long distance vmotion as a cure-all to any DR problems you might have.
  

Location/ID Separation Protocol – LISP

• Decouple location from device identification
• IP addresses today do both: they define the network (location) of a device and its identification (how do I talk to that device)
• By decoupling the two, a device can have its location changed without changing its ID (ie, without changing how clients identify and connect to the device)
• Allows you to move workloads around in the cloud and make sure those services and devices can be reached no matter where they are
• No load balancing, no DNS changes, no client side changes. Works all in the network.
• LISP – Finding the Optimized Path for your Workload (blogs.cisco.com)

Powering the Cloud

Presented by Willem van Schaik (Cisco)

From above: one of the elements of cloud is automation. In order to build automation:

• You must understand all processes needed to provision a server/app today
• The provisioning involves people from all teams (network, server, storage, application support). Everyone must be involved in setting the automation rules and workflows.

Cisco tools:

• newScale catalog: self service portal
• Cisco Tidal orchestrator: automation piece

Cisco bought newScale and Tidal to get best of breed self service and orchestration tools.

Orchestration must be full life cycle. You need to deprovision the stuff too at some point! Don’t forget.

Amazon cloud is cheap when compared to in house infrastructure that is not being fully utilized (as is the case the majority of the time). Your private cloud is cheaper than Amazon if you’re running at a very high level of utilization. If running at high utilization though, how do you handle spikes in load? Burst your spikes into the public cloud (ties back to “rent the peak” above). This requires rules though for which workloads can/should be moved into public cloud.

Using Cisco’s own measurements, they flipped the 77% opex/33% capex ratio to 40% opex (keeping the lights on) and 60% capex (introducing new services) by moving to a full cloud architecture.

Buying a Vblock is like buying a car, there’s limited options to choose from on the order sheet. You have to pick from the options the manufacturer has made available.

Flexpod is a set of reference designs (for Exchange, for Oracle, etc). You order all the parts yourself, which means you can customize the order, and you integrate the pieces on your own. Flexpod also contains benchmark information to help you with capacity planning.

Final thoughts: This stuff is not easy. Start small. Seek help from Cisco on the really hairy parts. If you can realize full cloud, the benefits are big in cost and time savings.

Below are the steps for properly resetting an admin password and for changing the security policy so the lockout doesn’t happen again.

Unlock the Admin

The unlock process is really a password recovery and works a lot like password recovery on an IOS device. You need console access to the appliance and the ISE software DVD/ISO. A reboot is required.

ISE systems can be installed on dedicated server hardware or as virtual appliances under VMware vSphere. The box in my lab was a virtual appliance so these steps are going to reflect console access and rebooting of a VM.

#1 – Reboot from ISE DVD/ISO

To get to the recovery console, the appliance needs to be booted from the ISE installation media. I had the ISO image handy so I used that. Now under vSphere, when the VM reboots, any media that was attached prior to the reboot is disconnected. The trick is to have the console window for the VM open in vSphere Client and hit the <F2> key when you see the VMware BIOS screen. With the machine sitting in the BIOS, it gives you time to reattach the ISE ISO to the DVD drive before the OS starts to load up.

Connect to ISO image on local disk

Also while in the BIOS, adjust the boot device order so it hits the CD-ROM drive before the hard drive.

CD-ROM before Hard Drive

If you’re doing a recovery on a physical appliance, you’ll probably still want to check your boot device order and also set it to boot from CD/DVD drive first.

Save your BIOS changes and boot the machine.

#2 – Reset Admin CLI Password

When the machine boots from the ISE DVD it will display a number of boot options.

ISE Boot Menu

If the appliance is a VM or is a physical appliance with a keyboard/mouse attached, choose #3. If the appliance is accessed via a serial console, choose #4.

The recovery menu now appears and asks which admin account to recover.

ISE Password Recovery Screen

Choose the account and enter a new password. This password will be used to log in on the appliance’s console. It does not work on the web UI.

Reboot the appliance now, making sure to eject/disconnect the DVD/ISO image so that it boots normally.

#3 – Reset the ISE GUI Admin Password

With the appliance booted normally, log in on the console using the password that was set in step #2. Remember: the console admin account is different than the web UI admin account. They have the same username but can have different passwords. Use the command “application reset-passwd ise admin” to set a new web UI admin password.

Reset ISE Web UI Password

The screenshot above shows other options that can be used with the “application” command.

The web UI should now be accessible using the password that was just set.

Change the Password Lockout Policy

The default password policy says that admin accounts will be locked out if their passwords are not changed once every 45 days.

ISE Admin Lockout Policy

This can be adjusted in Administration, System, Admin Access. Expand the Settings folder and highlight Password Policy.

ISE Password Policy Screen

Did I Need to Reset the CLI Admin or Am I Just Forgetful?

I confess, I’m not 100% sure that I needed to reset the CLI admin password. None of the passwords in my password safe were working on the CLI so it was either expired or I forgot to store the CLI password in the safe. If your web UI password doesn’t work, try starting from step #3 to see if you can avoid rebooting the appliance. Best case it works, worst case you start from step #1 and reset all the passwords.

Can anyone comment on whether the password policy configured in the web UI also applies to the console admin user?

Both of these plugins are great and work really well on their own.

However, when both plugins are in use and TMAC submits a comment, GASP inspects the comment to see if the checkbox has been marked, finds that it hasn’t been, and silently rejects the comment. (Aside: the exception to this is if you are a logged-in user and you initiate a manual TMAC check, any new tweets will successfully pass through GASP).

Since GASP is just checking the HTTP POST variables to see if the box was marked, we can have TMAC tweak the POST variables in its favor when submiting a new comment. Ideally GASP would have filters/hooks throughout the code which would allow us to do this all within GASP but unfortunately there are no filters present.

I’m modifying the POST variables in my theme’s functions.php like so:

function tmac_gasp_workaround($posts)
{
    $_POST['gasp_checkbox'] = 'tmac_workaround';

    /* return $posts untouched */
    return $posts;
}
add_filter('tmac_mentions_check_posts', 'tmac_gasp_workaround');

The $posts argument is what the tmac_mentions_check_posts filter passes in. We return it unchanged. By creating the gasp_checkbox POST variable and setting it to an arbitrary value, GASP will now believe that a user marked the checkbox and the Twitter comments will be successfully posted.

I don’t believe this fix opens any sort of window for a spammer to get through GASP’s checks. The function above is only called during TMAC’s scheduled check for new tweets and that check is spawned by an asynchronous HTTP POST request that WordPress initiates to itself as users are browsing the site. The POST variable that’s being influenced above belongs to that async HTTP request and not any HTTP requests initiated by users (or spambots) so users must still mark the box for their comments to be posted.

This post is the second in my series on virtual routing. The others are:

• An Introduction to Layer 3 Traffic Isolation
• VRFs and Shared Services Cheating with Junos

IOS

IOS is rather straightforward when it comes to configuring a VRF. Give the VRF a name, bind some interfaces to it, install some routes, and you’ve got a basic setup ready to go.

switch(config)# ip vrf guest_inet
switch(config-vrf)# description This is the VRF for guest Internet

The “ip vrf” command creates the VRF. This is the minimum configuration required when using VRF lite.

switch(config-vrf)# interface vlan200
switch(config-if)# ip vrf forwarding guest_inet
switch(config-if)# int tunnel5
switch(config-if)# ip vrf forwarding guest_inet

Interfaces are bound to a VRF using the “ip vrf forwarding” command. With the command applied, all packets traversing the interface will be forwarded according to the routes present in the named VRF’s routing table.

The commands above would be suitable for the middle switch in this little network:

We still don’t have two-way traffic flow though. The network where the guest users are sitting will be present in the guest_inet routing table because the connected route for the vlan200 interface was installed as soon as the interface was bound to the VRF. However there is no route present which directs traffic through the tunnel interface towards the Internet. Although it’s possible to run dynamic routing protocols in a VRF, that’s a little more complicated than I plan to get in this post. Instead, I’ll just show how to add a static default route.

switch(config)# ip route 0.0.0.0 0.0.0.0 10.10.10.201 vrf guest_inet

The command is just a default route pointing to the far end of the tunnel with the addition of the “vrf” keyword at the end. This route is now present in the guest_inet routing table.

To do some reachability tests inside the VRF, use the ping and traceroute commands also with the “vrf” keyword.

switch> ping 10.10.10.201 vrf guest_inet

switch> traceroute 10.10.10.201 vrf guest_inet

One of the easiest ways to trip yourself up is to accidentally do pings and traceroutes from the global table. If you’re not careful you can end up wasting a lot of time troubleshooting why your ping isn’t working. Also remember when looking at the routing table to specify the VRF.

switch> show ip route vrf guest_inet

Lastly, to see a list of configured VRFs along with their interfaces, use the “show ip vrf” command.

switch> show ip vrf interface
 Interface         IP-Address          VRF           Protocol
 Tu5               10.10.10.200        guest_inet    up
 Vl200             10.200.200.1        guest_inet    up

Junos

Junos has the concept of routing instances which are used to setup separate instances of OSPF, ISIS, BGP, etc, as well as static routes. Interfaces are bound to routing instances and each instance has at least one routing table and a single forwarding table. The first contrast with IOS is that Junos has different types of routing instances. There are types for Layer 2 VPNs, for Layer 3 VPNs, and for basic virtual routing. The “VRF” instance type is actually for use with MPLS Layer 3 VPNs. For setting up VRF lite, the “virtual-router” type is the most appropriate.

routing-instances {
  guest_inet {
    instance-type virtual-router;
    interface ge-2/0/0.200;
    interface ge-2/0/2.0;
  }
}

This bit of the configuration creates the guest_inet routing instance and binds two interfaces to it. There’s no additional configuration needed on the actual interfaces themselves.

When this routing instance is created, there is a corresponding routing table created named guest_inet.inet.0. This table will receive the “direct” routes from the two interfaces that were added to the routing instance.

root@router> show route table guest_inet.inet.0

guest_inet.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.200.200.0/24    *[Direct/0] 00:01:55
                    > via ge-2/0/1.200
10.10.10.200/30    *[Direct/0] 00:01:55
                    > via ge-2/0/2.0

Like the IOS example above, we need a default route to send traffic towards the Internet. Protocols such as OSPF, BGP, and ISIS are configured in a routing instance at the [routing-instances guest_inet protocols] level and static routes are configured at [routing-instances guest_inet routing-options].

root@router> show configuration routing-instances guest_inet routing-options
static {
  route 0.0.0.0/0 next-hop 10.10.10.201;
}

When doing reachability tests, use the “routing-instance” keyword to specify which instance to use.

root@router> ping www.google.com routing-instance guest_inet

root@router> traceroute www.google.com routing-instance guest_inet

To see the status of a routing instance use “show route instance”

root@router> show route instance guest_inet detail
guest_inet:
  Router ID: 10.200.200.1
  Type: virtual-router State: Active
  Interfaces:
    ge-2/0/1.200
    ge-2/0/2.0
  Tables:
    guest_inet.inet.0 : 3 routes (3 active, 0 holddown, 0 hidden)

Further Reading

Cisco:

Path Isolation Deploying VRF-Lite and GRE  – This particular section of the Path Isolation Design Guide gives a nice example of using GRE with VRF lite. It includes the commands needed to make VRF lite work in an example network. It even goes much farther and talks about virtualizing routing protocols and then gets into using full MPLS.

Juniper:

Routing Instances Overview  – An overview of what routing instances are and the different types.

Configuring Routing Instances  – A really quick introduction on how to configure routing instances. Sadly it only shows syntax for dynamic routing and doesn’t include the command syntax for static routes.

We’ve been talking a lot at work recently about virtualizing our big workloads. One of the first points of discussion is usually storage performance and someone inevitably brings up RDMs. This VMware blog post reminds us that a) RDM and VMDK performance is pretty much on par these days and b) that the storage method should be chosen based on a combination of business and technical requirements, not on assumptions or outdated information. Thanks to Paul Gifford (@paulgifford) for the link.

Virtualized Exchange Storage: VMDK or RDM or…?
http://blogs.vmware.com/apps/2011/11/virtualized-exchange-storage-vmdk-or-rdm-or.html

In early versions of ESX the virtualization overhead associated with deploying virtual disks (VMDK files) was much higher than it is today and why it was considered a best practice to place Exchange data files on physical mode raw-device mappings (RDM). As ESX and vSphere have evolved the performance difference between RDMs and virtual disks has become almost nonexistent.

Earlier this month Riverbed announced version 7 of its RiOS software which runs its Steelhead WAN acceleration appliances. There’s a couple of new features which are interesting, including support for IPv6 traffic, UDP traffic, and video optimization. The release describes video optimization as “application layer multicasting [which] allows a single video stream to serve a large number of viewers in a particular location.” With the accelerated adoption of video in the enterprise this sounds pretty good but the article does not mention what types of video codecs and formats it works with.

Riverbed RiOS 7
http://www.datacenterknowledge.com/archives/2011/12/06/riverbed-accelerates-rios-wan-optimization

New features and optimizations included in the version 7.0 release include native support for HTTP video, User Datagram Protocol (UDP), and IPv6, as well as an extension to its existing optimizations for virtual desktop infrastructure (VDI).

Ivan Pepelnjak (@ioshints) has a great writeup about the history of MPLS in IOS and Junos. It goes a long way in describing some of the differences in the two implementations and, more importantly, describes why the two are different. A good read for anyone who works with the MPLS features on these platforms.

Junos Day One: MPLS Behind The Scenes
http://blog.ioshints.info/2011/12/junos-day-one-mpls-behind-scenes.html

The fundamental reason for widely different MPLS implementation is the first use case: Cisco IOS started with tag switching targeted at IP-over-ATM transport, where every IP prefix needs an end-to-end LSP; Junos started with layer-3 MPLS/VPNs, where you need LSPs only toward BGP next hops.

Traffic Isolation Using VLANs

VLANs work great in a Layer 2 switched network, but what happens when you need to maintain this traffic separation across a Layer 3 boundary such as a router or firewall? Typically, if you have two VLANs that each terminate on a router and the router has an IP address in each VLAN, the devices in those VLANs are free to talk to each other by passing traffic through the router. That traffic isolation that was gained by putting the devices in different VLANs is now lost. In fact, because some network engineers are only familiar with traffic isolation at Layer 2 and not at Layer 3, the overall network design will often be compromised to allow for VLANs to span end-to-end in the network so that traffic separation can be maintained. This of course necessitates bridging everywhere in the network which can lead to serious issues.

There is a way to maintain traffic isolation across Layer 3 devices. It’s called Virtual Routing and Forwarding. Virtual Routing and Forwarding, aka VRF, allows the routing table in a Layer 3 switch or router to be virtualized. Each virtualized table contains its own unique set of forwarding entries. Traffic that enters a router will be forwarded using the routing table associated with the same VRF that the ingress interface is associated with and will be sent out an egress interface associated with the same VRF. Much like VLANs, VRFs ensure logical isolation of traffic as it crosses a common physical network infrastructure.

There are three general concepts behind VRFs:

1. Access Control
2. Path Isolation
3. Shared Services

The next sections look at each of these in turn.

Access Control

Access control refers to how end devices are identified and segmented at the network edge (aka, access layer). Users need to be segmented before they input traffic into the network so that the network knows which virtual network to associate their traffic with. Access control must take into consideration both wired and wireless network access methods.

Two of the most common methods for segmenting wired end devices is by static VLAN assignment and 802.1X Network Access Control. Static VLAN assignment is where a VLAN is configured on an edge port and that VLAN does not change regardless of who or what is plugged in.

switch(config-if)# switchport mode access
switch(config-if)# switchport access vlan 101

This method is simple to implement but is costly to maintain. Every time a new device is plugged into the port the VLAN might have to be changed. If the port is located in someone’s office then this might not happen very often, but if the port is in a meeting room it can be a nightmare if you have a mixture of employees and guests plugging in. Additionally, if the port is left on the employee VLAN and a guest plugs in, they are now on the employee network and can access the same network resources as employees. This is an obvious security risk. Although it’s hard to maintain, static VLAN assignment is easy and inexpensive to implement and requires no additional equipment, tools, or training.

The more advanced alternative to static VLAN assignments is to employ 802.1X on all edge ports. The 802.1X standard is a method of authenticating end devices to the network. Based on the results of the authentication and the policies in place, the network can automatically assign a VLAN to a port (among other things). Presumably, devices being used by employees would successfully authenticate and be placed on the employee VLAN. Devices owned by guests would fail the authentication and be placed into the guest VLAN. Implementing 802.1X however, is rather complex. It requires additional equipment to perform the authentication and run the policy engine that determines things like VLAN assignment. The upside though is that it removes the burden of manually tweaking VLAN assignments and ensures that the end devices are always placed into the correct VLAN, eliminating the risk of guests getting on the employee VLAN.

Access Control on Edge Ports

On the wireless side, end devices can be segmented by way of separate SSIDs for different groups of users. An SSID could be created for employees, guests, and contractors each of which is bound to its own VLAN on the uplink side of the wireless controller. Mechanisms such as 802.1X can also be employed on wireless connections to bind an end device to a certain VLAN after it’s associated with the wireless network.

Wireless Access Control

In the end, all of these solutions segregate end devices by placing them in the appropriate VLAN. VLAN assignment is the first step in segregating end device traffic. Traffic entering the network on that VLAN will eventually hit a Layer 3 device where it will be forwarded based on the routing table that is part of the VRF to which the VLAN is bound.

Traffic Hitting a VRF

Path Isolation

Path isolation refers to the method used within the core of the network to keep each VRF’s traffic isolated. As stated earlier, once traffic hits a Layer 3 device, it will normally be forwarded between interfaces which may allow traffic to route between VLANs. Each of the path isolation methods below keeps traffic inside its assigned VRF as it travels between Layer 3 devices.

Routing Between VLANs/Subnets

The hop-by-hop method creates switched virtual interfaces (SVIs) on top of 802.1q tags between each Layer 3 device in the network. For each pair of connected devices, there is (1) SVI created per device, per VRF. Unlike a Layer 2 network where VLAN tags are bridged end-to-end, each tag is used only on one interconnect and each device acts as a Layer 3 hop in the traffic path. When everything is fully provisioned, you end up with a path through the network that is strung together by SVIs. This can be really cumbersome to manage since for every VRF you have to configure multiple interfaces from edge to edge and manage all that extra IP addressing too. If there are multiple potential paths through the network from edge to edge, then the SVI string needs to be provisioned on the alternate paths as well. The upside is that SVIs are relatively easy to understand and are very well supported on all types of hardware and software versions. Because of the large management overhead though, this method should only be used on very small networks.

Hop-by-Hop SVI Interfaces

A more scalable alternative to hop-by-hop is to encapsulate each VRF’s traffic inside a tunnel. Since a tunnel can be provisioned directly between two edge routers, nothing needs to be touched in the core of the network. In fact, the VRFs don’t even need to be provisioned in the core of the network (assuming there are no edge devices connected to the core). This simplifies the provisioning of paths through the network and eliminates the risk of a mistake being made on a core router during provisioning. If provisioned correctly, a tunnel also provides built-in path redundancy (unlike hop-by-hop which you have to manually account for). Assume an active tunnel is following this path through the network: A->B->C and that router B fails. As long as there is an alternate path through the network between the two tunnel endpoint addresses on A and C, the tunnel will re-route and the VRF traffic inside the tunnel will continue to flow. One caveat with tunneling is that not all devices perform tunneling in hardware and some don’t even support protocols such as GRE at all. You tunnel endpoints need to support GRE and depending on your traffic load, should perform GRE functions in silicon.

Edge-to-Edge Tunnels

The final and most scalable method is full on MPLS. MPLS dynamically creates paths from edge router to edge router for transporting VRF traffic through the network. Although MPLS is the most scalable it is also the most complex. An MPLS network requires BGP and LDP (Label Distribution Protocol) both of which must be well understood by the network provisioning and operations teams. MPLS also has the strictest hardware requirements of any of the path separation methods. Network devices must be capable of running BGP and LDP, must have enough memory on board to handle all the entries in the routing and label forwarding tables, and should be capable of label switching in silicon for best performance.

The hop-by-hop and tunnel methods are what’s known as “VRF lite” — you’re using VRFs but without full-blown MPLS to tie everything together. VRF lite is often found in enterprise networks where the number of VRFs in play are still manageable using these manual path isolation methods.

Shared Services

Shared services are things like DNS, DHCP, and Internet access that are typically common to all VRFs. Rather than running a set of DNS servers and a set of DHCP servers for each virtual network, you stand up one set of servers that can service everyone. Internet access is the same. Running multiple Internet services is costly and time consuming so it’s usually shared among all VRFs.

Shared services are typically located in their own little module that hangs off the edge of the network. This module is one of the trickiest parts of a VRF-enabled network because it’s really easy to accidentally allow traffic to leak between VRFs if proper care is not taken. Since the servers and Internet edge devices that sit in the shared services module need to talk to end devices in all the VRFs, this module needs to contain routes for all of the VRFs. It would be really easy to accidentally allow routes from VRF A to be advertised through the shared services module into VRF B (and vice-versa) thus allowing devices in A and B to freely communicate.

Shared Services

Another big challenge with shared services is the fact that VRFs can have overlapping IP address space. It becomes increasingly difficult to provide services like DNS and DHCP on a single server for overlapping IP networks. In this case it may be necessary to actually have multiple servers that serve a subset of VRFs or even just an individual VRF. The “shared” in “shared services” now refers more to the shared infrastructure which connects these servers to the rest of the network rather than the servers themselves being shared.

Final Word

This is the first post in what I hope to turn into a short series on Virtual Routing and Forwarding. Future posts will discuss how to configure VRFs, practical applications for VRFs in an enterprise network, and go into more detail on shared services.

I was able to reproduce both of these issues on OpenBSD/i386 5.0.

UPDATE 2011-11-29: Net-SNMP 5.7.1 + fixes has been pushed to the OpenBSD 5.0-stable ports collection as well as -current ports. OpenBSD users should sync their ports tree and rebuild ports/net/net-snmp to alleviate the issues below.

#1 – Values returned by hrStorageUsed are incorrect

The first is with respect to the values returned via hrStorageUsed in the HOST-RESOURCES-MIB. The numbers are way off. For example, prior to the upgrade a disk that reported hrStorageUsed of 1873784 units (3.8GB) now reports 23920757 units (49GB). The Net-SNMP git repo shows there’s been some changes since v5.6.1 in the code that gathers disk stats. Looks like a bug was introduced somewhere.

Solution: Use Net-SNMP v5.7.1. The hrStorageUsed values in v5.7.1 are consistent with the values returned from v5.6.x. OpenBSD users, the net/net-snmp port in CVS HEAD is already updated to 5.7.1. Sync up and go.

#2 – Segfault when querying hrStorageTable

The second issue is a segfault in snmpd when querying the hrStorageTable. The segfault is happening on the memcpy() call(s) in netsnmp_fsys_arch_load(). The segfault is inconsistent. Sometimes you can get a handful of good queries and responses before it happens. Other times it happens on the first query. I don’t know why the segfault is happening, but I know that replacing memcpy() with strncpy() eliminates it. This is pretty hackish but I’m not skilled enough to debug this any further.

Solution: Apply the patch here. On OpenBSD you should be able to save the patch into ports/net/net-snmp/patches/ and rebuild the port. You will need this on Net-SNMP v5.7 and 5.7.1.

Am I the only one?

Are other Net-SNMP users seeing these issues? Does anyone even use Net-SNMP anymore (particularly on OpenBSD)? Please leave a comment and let me know.

I’m crossing my fingers that I won’t have to write another patch file the next time I upgrade Net-SNMP.

The first is an older article from networkworld.com that reminds us all that VTP clients are also capable of updating VLANs on the network, not just servers.

When I first heard that a VTP client can update a VTP server under the right conditions, I was frankly a non-believer. No way. I’d seen evidence to the contrary in several documents at cisco.com and in Cisco courses – but all the evidence was written, without my doing any experiments. So, I spent some time experimenting a few years ago, and found that it’s true – clients can overwrite VTP server’s VLAN databases.

Full article is here http://www.networkworld.com/community/node/19931.

The second article comes from etherealmind.com and is one of the only positive articles I’ve ever read about VTP. Greg’s take is that VTP is not inherently bad but instead the way network engineers deploy it is the reason it’s capable of causing so much damage.

A lot of people regard Cisco’s Virtual Trunking Protocol(VTP) as nothing but trouble. Frankly it’s hard to find many people who will implement it on their network and most people have war stories about full site outages caused by VTP and switch installs. I find this baffling – it’s a great technology that dramatically reduces time, configuration errors, and improves troubleshooting – features that we should all embrace and use wherever we can. In this post, I want to suggest a different design method for effectively using VTP in your network.

Full post is here http://etherealmind.com/vtp-design-fate-sharing-failure-domains/.