Cisco IOS hints and tricks

Goodbye, Blogger comments

2009-07-14T10:43:00.000+02:00

I received numerous complaints about the inability to write comments to my blog in the last few weeks. I had comment-related problems for months, but the situation was getting out of control: initially only the Internet Explorer users had (manageable) problems, then it spread to some Firefox users, later it became impossible to submit comments at all and the last marvel in this buggy saga was the “automatic” conversion from Publish request to preview screen. I had enough, Google has yet again proven that you usually get what you pay for.

The best International NetRider comes from Slovenia

2009-07-14T06:52:00.003+02:00

On July 1^st, Cisco Systems ran the annual International NetRiders competition and Andraž Piletič (whom you might remember from his NIL Monitor posts published in our blog) was the top score winner.

Congratulations, Andraž!

I was absolutely delighted to hear the news (the best part was that almost nobody knew Andraž took part in this event). I always knew Andraž was a fantastic engineer and his achievement also shows that our student internship program rocks.

Question everything

2009-07-13T06:42:00.004+02:00

In one of our discussions, Stretch provided an excellent graph illustrating that the ISP competition seems to reduce prices almost linearly and asked me in a later comment to justify the inverse relation between subscription charges and consumer choice.

You might consider this debate to be purely between Stretch and myself, but it’s an interesting example of what you might need to do in daily your job. If you want to be a great networking engineer, you have to be prepared to question everything, including common wisdoms, “well-known truths”, “common practices” and facts that look too good to be true. Ready? Let’s go …

Start with the source. The graph came from ars technica article US broadband report: more popular, more expensive, which used the data from Pew Internet’s Home Broadband Adoption 2009 report. I had a “somewhat” biased opinion (I’m playing the devil’s advocate here), but this clearly falls into “looks too good to be true” category. Anyhow, if you want to understand what’s being reported, you have to go as far back to the original data as possible; in our case, read the Pew Internet’s report, not the summary provided by ars technica (although they did a good job).

The obvious conclusion. Looking at the graph, the message is simple: whenever there’s competition, the prices go down. Are you happy with this explanation? Do you question its validity? Do you understand what the underlying unknown variables might be?

Find the alternatives. Whenever something looks too good, consider the alternatives. How about this: it’s easy to have low prices in urban environment with very high population density (Manhattan, London, Paris, Vienna or Singapore). It’s also obvious that there’s a lot of competition there (everyone tries to cherry-pick these customers). The rural places have low population density, therefore higher prices (assuming the prices somewhat reflect the actual cost of providing services) and almost no competition. The low population density would thus automatically correlate higher prices (assuming they are related to higher costs) with low competition. Are there any other alternative explanations? Give them in the comments.

Evaluate the alternatives. Do you find the second scenario viable? I’m not claiming it’s correct, I’m trying to nudge you to think. Can you find enough information in the graph, the article or the report to decide which explanation is valid? Document you findings in the comments.

Why does this matter? Why should you as a network engineer invest your time in this process? You’re constantly bombarded with “facts” nicely laid out in media articles, design guides, performance charts or whitepapers. Do you accept the “fact” that a router should belong to at most three OSPF areas and that an area should have at most 50 routers … or are you trying to understand what the actual limitations are? Are you happy to be mediocre and use simple recipes (without knowing where they come from) or will you try to become innovative by understanding the in-depth details and using them to your advantage? Are you willing to digest whatever media throws at you or do you want to form your own opinion? The choice is yours.

Looking for additional information on Netflix video streaming

2009-07-12T08:37:00.000+02:00

I'm looking for details on how Netflix streams videos over the Internet. I've found their description of encoding and bit rates, but was not able to find lower-layer details (I can only assume they use UDP, but I would like to verify that with someone who's actually using the service).

I would also appreciate any information on whether they work with Service Providers (for example, using local direct peering) to ensure the upstream Internet connections are not clogged with streamed video.

Drawing the diagrams

2009-07-10T06:36:00.000+02:00

Every so often, someone asks me what tools I use to draw the diagrams. Years ago I was perfectly happy with Visio, but since Microsoft bought it, it became so bloated that I’ve been forced to drop it (it would take minutes to start on my laptop) and revert back to PowerPoint.

Cisco provides great icon libraries (including the visionary “space router” icon shown on the right) in Visio and PowerPoint format and I’m lucky enough to have an older version where the colors of the devices are not light blue but a darker shade of blue/green/gray. Drawing connections between the devices is obviously easier in Visio than in PowerPoint, but if you keep the diagrams simple, you can work around the limitations.

PPPoE testbed

2009-07-09T07:24:00.006+02:00

Following the ADSL QoS discussions, I decided to test “a few” things in the lab. I didn’t want to build a huge lab with DSL modems and DSLAM and decided to emulate an end-to-end DSL network with routers.

Step 1: Create a PPPoE session between a client (SOHO router) and a server (NAS). I’d never configured it before, so I’ve visited uncle Google first. It gave me tons of useless hits (no wonder) and a few somewhat useful ones. All of them used the old VPDN-centric PPPoE syntax. It seems I’ve stumbled across another “niche spot”: PPPoE testbed implemented with recent IOS configuration syntax (bba-group).

Read more in the “PPPoE Testbed” article in the CT3 wiki.

When the “passive-interface” command is missing from OSPF

2009-07-08T07:08:00.002+02:00

A few days ago a member of the cisco-nsp mailing list asked an interesting question: “the passive-interface command is not available in a VRF OSPF process. What can I do?”

It turned out he stumbled across CSCeb86068, which is already fixed in a later software release for his platform.

The passive-interface command tells the routing process to ignore packets received from the specified interface. In case of OSPF, the relevant packets are the hello packets, as an OSPF router will not exchange routing updates without an established adjacency. You can get the same results by deploying an inbound access list on the interface (which is the functionally equivalent workaround for this bug), although this method generates more configuration overhead than the OSPF-specific solution.

Trigger EEM applet with SNMP

2009-07-07T07:11:00.007+02:00

Anderson sent me an interestion question:

I'd like to use the snmpset command to get my router to execute an EEM script. Are there OIDs that are associated with EEM scripts that could help me achieve this?

Although EEM has associated MIB, it has a single read-write variable: the size of the history table. It's thus not possible to use EEM MIB to trigger EEM events. However, EEM 2.4 added support for SNMP notification events, which you can use to trigger EEM applets based on incoming SNMP traps/informs.

You can therefore use the event snmp-notification command on a router and the snmptrap command on a Linux host to remotely trigger EEM applets.

Read more in the Trigger EEM applets with SNMP Informs article in the CT3 wiki.

This article is part of You've asked for it series.

Followup: All-I-can-eat

2009-07-06T06:50:00.003+02:00

The “All-I-can-eat-mentality” article has triggered (as expected) numerous responses. Some of them provided useful data, links to more information or informative perspectives – many thanks to those readers. A few others were unfortunately following the “I-am-right” line without considering facts. Most of the readers from the Service Provider community decided to stay anonymous (when you read all the comments, it becomes obvious they made a wise decision) or respond off-line.

Whatever your position in this issue, I would like to ask you to keep your comments focused on the topic. Although you were all infinitely more polite than the usual forum/blog crowd and provided some really good arguments, writing angry replies does not help. What’s happening with Internet is (like it or not) our common problem … or you could take the blue pill and continue bashing the other side.

I particularly liked the summary of our discussion posted on Slashdot (where someone included the link to my blog):

Whoa, whoa, whoa, that article seems to be promoting a balanced viewpoint that denies a) that telcos are totally evil and b) that we should all be allowed to have as much bandwidth as we want and not have to pay for it. We'll have none of that nonsense on /.

It’s also painfully obvious that the current handling of ISP service offerings is often dismal. For example:

Publicly accessible service definitions don’t exist. What most large ISPs offer on their web sites is the X Mbit @ Y $/€/whatever per month. They “forget” to mention that the offered rate is PIR, with expected CIR being only a few percent of PIR.
When introducing the traffic caps, some providers started with ridiculously low and probably a bit overpriced figures.
A while ago some people decided to keep P2P traffic (which is the worst bandwidth hog of all) in line by resetting TCP sessions (a link to the recent state of affairs would be highly appreciated).

In the next few weeks, I’ll try to cover a few of the topics raised in the comments, including:

Why do we have to live with oversubscriptions?
Why are the Service Providers forced to use traffic management?
Is it possible to have a fair and consumer-friendly service definition?
Why will everyone have to invest in deep packet inspection (DPI)?

Another milestone reached :)

2009-07-05T09:22:00.000+02:00

According to FeedBurner statistics, my RSS feed had exactly 5000 subscribes on June 30^th.

Disclaimer: The Feedburner subscriber count is a notoriously unreliable measure, as it tries to estimate the actual number of subscribers for on-line services that cache the feeds (for example, Google Reader) … but the result is nonetheless quite nice.

What went wrong: end-to-end ATM

2009-07-03T07:03:00.000+02:00

Red Pineapple was kind enough to share his 15-year-old ATM slides. They include interesting claims like:

ATM has the potential to displace all existing internetworking technologies

One single network handles all traffic types: Bursty data and Time-sensitive continuous traffic (voice/video).

All these claims are still true if you just replace »ATM« with »IP«. So what went wrong with ATM (and why did the underdog IP win)? I can see the following major issues:

Not all interfaces are created equal

2009-07-02T21:53:00.000+02:00

Two days ago I’ve managed to write aGenuineStupidity™ (OK, maybe I cannot get a trademark on this concept): the MQC shaping actions cannot be attached to a Dialer interface; they have to be specified on the underlying physical interface (in case of PPPoE link, the outside Ethernet interface).

The reason for my stupidity (apart from the obvious one: writing without testing) is the difference between true logical interfaces and dialer templates. A tunnel interface or a VLAN interface is a true logical interface; it behaves like any other interface (with a few exceptions; for example, tunnel interface does not have an output queue) and you can use most QoS actions (including shaping) on it. A dialer interface is even more “conceptual”. It can never be operational on its own – as soon as the link is established, it’s bound to a physical (for example, BRI0:1) or virtual access interface (which is yet again bound to a physical interface) and the shaping is performed on the final physical interface.

This behavior (on top of being unexpectedly inconsistent) results in interesting quirks. For example, you have to shape PPPoE packets (based on their IP characteristics) on the physical Ethernet interface which usually doesn’t even have an IP address.

… and let’s hope that the late hour hasn’t resulted in another blunder.

Service Provider “Services and Solutions” bootcamp

2009-07-01T07:17:00.001+02:00

Last week I happened to be copied on an e-mail which included a PDF describing a “SP Services and Solutions” bootcamp I knew nothing about. As I was slowly reading the document, I’ve realized it’s a perfect course for Service Provider (or channel partner) engineers who need an overview of the various technologies used in modern (I hate to use the marketing phrase “next-generation”) SP networks and the ways they can be used to create customer-focused solutions.

My next question (to our training manager) was: “when is this going to be available, I want to blog about it” and the answer astonished me: “we’re doing the first customer teach this week”. Our SP gurus managed to develop a course they always wanted to teach almost undercover.

Read the rest of my thoughts about this fantastic course in Fragments

ADSL QoS basics

2009-06-30T07:25:00.005+02:00

Based on the ADSL reference model we’ve discussed last week, let’s try to figure out how you can influence the quality of service over your ADSL link (for example, you’d like to prioritize VoIP packets over web download). To understand the QoS issues, we need to analyze the congestion points; these are the points where a queue might form when the network is overloaded and where you can reorder the packets to give some applications a preferential treatment.

Remember: QoS is always a zero-sum game. If you prioritize some applications, you’re automatically penalizing all others.

The primary congestion point in the downstream path is the PPPoE virtual interface on the NAS router (marked with a red arrow in the diagram below), where the Service Provider usually performs traffic policing. It’s better from the SP perspective to police the traffic @ NAS than to send all the traffic to DSLAM where it would be dropped in the ATM hardware. Secondary congestion points might arise in the backhaul network (if the network is heavily oversubscribed) and in DSLAM (if the NAS policing does not match the QoS parameters of the ATM virtual circuit).

In the upstream direction, the congestion occurs on the DSL modem – the path between the CPE and the modem (Ethernet or Fast Ethernet) is much faster than the upstream ATM virtual circuit. Secondary congestions might occur in DSLAM or the backhaul network. NAS usually does not police inbound traffic, as it’s assumed the DSL access network already limits the user traffic to its contractual upstream speed.

Based on the congestion analysis, it’s obvious you cannot use queuing on the CPE (marked “2” in the diagrams) to influence the ADSL QoS as you don’t control a single congestion point. You have to use traffic shaping on the CPE to introduce artificial congestion points in which the queues will form. You can then use the usual queuing mechanisms to prioritize the application traffic.

The shaping configured on the PPPoE interface on the CPE router neatly removes the congestion on the DSL modem. The backhaul network is rarely congested in the upstream direction (unless your friendly neighbors are devoted fans of P2P protocols).

When configuring the upstream shaping rate, you just have to take in account the extra overhead introduced by the PPPoE framing, which is not yet present in packets shaped on the Dialer interface, and reduce the upstream shaping speed to a value slightly below your DSL upstream speed.

If your DSL configuration uses PPPoE Dialer interface, you have to shape the traffic on the Dialer interface, not the outside Ethernet interface. The outside Ethernet interface transmits PPPoE-encapsulated IP traffic on which you cannot use the IP-based queuing classifiers.outside Ethernet interface.

Assuming most of your traffic is TCP-based (or that all non-TCP traffic is prioritized), the shaping on the inside LAN interface will cause enough TCP delays to slow down the downstream TCP transmission. However, it’s harder to determine the correct shaping rate and optimize the shaping behavior when the high-priority traffic is not present; we’ll cover these issues in an upcoming post.

There is no local command authorization

2009-06-29T07:06:00.000+02:00

Shahid wrote me an e-mail asking about local command authorization. He would like to perform it within the AAA model, but while AAA local authorization works, it only allows you to specify user privilege level (and autocommand), not individual commands (like you can do on a TACACS+ server).

This article is part of You've asked for it series.

Help appreciated: touch-screen drawing

2009-06-28T15:16:00.002+02:00

I’m looking for a touch screen device that would work (well) with PowerPoint. I’d like to start drawing my diagrams with a pen, not with a mouse; I have a completely unfounded irrational belief that drawing with a pen might be faster and easier than using a mouse. Any (tested) ideas?

IOS HTTP vulnerability

2009-06-26T08:40:00.001+02:00

The Cisco Subnet RSS feed I’m receiving from Network World contained interesting information a few days ago: Cisco has reissued the HTTP security advisory from 2005. The 2005 bug was “trivial”: they forgot to quote the “<” character in the output HTML stream as “<” and you could thus insert HTML code into the router’s output by sending pings to the router and inspecting the buffers with show buffers assigned dump (I found the original proof-of-concept exploit on the Wayback Machine). However, I’ve checked the behavior on 12.4(15)T1 and all dangerous characters (“<” and quotes) were properly quoted. So, I’m left with two explanations.

Internet anarchy: I’ll advertise whatever I like

2009-06-25T06:41:00.001+02:00

We all know that the global BGP table is exploding (see the Active BGP entries graph) and that it will eventually reach a point where the router manufacturers will not be able to cope with it via constant memory/ASIC upgrades (Note: a layer-3 switch is just a fancy marketing name for a router). The engineering community is struggling with new protocol ideas (for example, LISP) that would reduce the burden on the core Internet routers, but did you know that we could reduce the overall BGP/FIB memory consumption by over 35% (rolling back the clock by two and a half years) if only the Internet Service Providers would get their act together.

Autocommands in AAA environment

2009-06-24T07:18:00.000+02:00

A reader who prefers to remain anonymous has reported an interesting observation: autocommands configured on local usernames do not work after configuring aaa new-model.

This article is part of You've asked for it series.

IS-IS is not running over CLNP

2009-06-23T07:06:00.001+02:00

A while ago I’ve received an interesting question from someone studying for the CCNP certification: “I know it’s not necessary to configure clns routing if I’m running IS-IS for IP only, but isn’t IS-IS running over CLNS?”

I’ve always “known” that IS-IS uses a separate layer-3 protocol, not CLNP (unlike IP routing protocols that always ride on top of IP), but I wanted to confirm it. I took a few traces, inspected them with Wireshark and tried to figure out what’s going on.

You might be confused by the mixture of CLNS and CLNP acronyms. From the OSI perspective, a protocol (CLNP) is providing a service (CLNS) to upper layers. When a router is configured with clns routing it forwards CLNP datagrams and does not provide a CLNS service to a transport protocol. The IOS configuration syntax is clearly misleading.

It turns out the whole OSI protocol suite uses the same layer-2 protocol ID (unlike IP protocol suite where IP and ARP use different layer-2 ethertypes) and the first byte (NLPID) in the layer-3 header to indicate the actual layer-3 protocol. I was not able to find any table of layer-3 OSI protocol types, so I had to experiment with Wireshark to figure out the values for CLNP, ES-IS and IS-IS (yes, these three are distinct L3 protocols).

You can find all the details (including the comparison of OSI and IP protocol stacks) in the IS-IS in OSI protocol stack article in the CT3 wiki.

This article is part of You've asked for it series.

ADSL reference diagram

2009-06-22T06:38:00.003+02:00

I’m getting lots of ADSL QoS questions lately, so it’s obviously time to cover this topic. Before going into the QoS details, I want to make sure my understanding of the implications of the baroque ADSL protocol stack is correct.

In the most complex case, a DSL service could have up to eight separate components (including the end-user’s workstation):

End-user workstation sends IP datagrams to the local (CPE) router.
CPE router runs PPPoE session with the NAS (Network Access Server) and sends Ethernet datagrams to the DSL modem.
DSL modem encapsulates Ethernet frames in RFC 1483 framing, slices them in ATM cells and sends them over the physical DSL link to DSLAM.
DSLAM performs physical level concentration and sends the ATM cells (one VC per subscriber) into the network.
The backhaul network (DSLAM to NAS) could be partly ATM based. The ATM cells could thus pass through several ATM switches.
Eventually the ATM cells have to be reassembled into PPPoE frames. In a worst-case scenario, an ATM-to-Ethernet switch would perform that function.
The backhaul network could be extended with Ethernet switches.
Finally, the bridged PPPoE frames arrive @ NAS which terminates the PPPoE session and emits the IP datagrams into the IP core network.

I sincerely hope no network is as complex as the above diagram. In most cases, the backhaul would be either completely ATM-based …

… or Ethernet based (when the DSLAM has Ethernet uplink interface):

The NAS could also be adjacent to DSLAM or even integrated in the same chassis.

Am I missing anything important? I know you could deploy numerous additional devices (for example, Cisco is promoting the Service Exchange Framework and Service Control Engine), but these devices would be placed deeper into the IP core.

ATM is like a duck

2009-06-21T11:49:00.000+02:00

It was (around) 1995, everyone was talking about ATM, but very few people knew what they were talking about. I was at Networkers (way before they became overcrowded Cisco Live events) and decided to attend the ATM Executive Summary session, which started with (approximately) this slide …

… and the following explanation:

As you know, a duck can swim, but it's not as fast as a fish, walk, but not run as a cheetah, and fly, but it's far from being an eagle. And ATM can carry voice, data and video.

Inter-VRF static routes

2009-06-19T07:09:00.003+02:00

Swapnendu was trying to implement inter-VRF route leaking in multi-VRF environment without using route targets. He decided to use inter-VRF static routes, but got concerned after reading the following paragraph from Cisco’s documentation:

You can not configure two static routes to advertise each prefix between the VRFs, because this method is not supported. Packets will not be routed by the router. To achieve route leaking between VRFs, you must use the import functionality of route-target and enable Border Gateway Protocol (BGP) on the router. No BGP neighbor is required

Internet Socialism: All-I-can-eat mentality

2009-06-18T07:18:00.001+02:00

Every few months, my good friend Jeremy finds a reason to write another post against bandwidth throttling and usage-based billing. Unfortunately, all the blog posts of this world will not change the basic fact (sometimes known as the first law of thermodynamics): there is no free lunch. Applied to this particular issue:

Recommendations for keepalive/hello timers

2009-06-17T07:09:00.008+02:00

The “GRE keepalives or EIGRP hellos” discussion has triggered another interesting question:

Is there a good rule-of-thumb for setting hold-down timers in respect to the bandwidth/delay of a given link? Perhaps something based off of the SRTT?

Routing protocol hello packets or GRE keepalive packets are small compared to the bandwidths we have today and common RTT values are measured in milliseconds while the timers' granularity is usually in seconds.