Nurd Land

Robot 2 - The Platform

2018-08-19T10:36:00.001+10:00

So I have received a new platform to build my robot on and I have recorded a short video showing the contents.

This platform is only plastic but being transparent makes it easier to see through to what you are doing. I’ve used one of these before and the only downside is the plastic isn’t that strong and can break with rough handling.

Here is the video.

Next video will show some assembly as I start to work out what else I need to build this.

Robot 1 Service Discovery

2018-08-15T16:36:00.000+10:00

Robot 1 - Service Discovery

Contrary to popular belief, a robot does not have to have human form to be considered a robot

I am starting a new project to build an autonomous vehicle of sorts but small scale. The aim is to challenge myself with many problems across multiple domains and to build something that in the end is to a degree autonomous.

I plan on building this around a Raspberry Pi 2 along with a cheap robot platform purchased form eBay.

My language of choice for the software will be python as that is well supported on the Pi.

To be transparent, I have actually attempted this before but now I'm going to go back and redo and document / video the whole process so others can benefit from my learning.

So the first problem that I know I will encounter based on past experience is how to log into my Pi over the network. I have a wireless USB card so it can join the network but it obviously gets a new address each time it starts.

I know there are many service discovery protocols out there so I plan on using one of those to get the Pi to announce itself.

I've found the following project on GitHub which looks like the ticket:

https://github.com/ZeWaren/python-upnp-ssdp-example

I'll work on writing some python based on this.

I'll also create a project in GitHub so I can share my code with people should they be interested.

Ryan

Puppet Apprentice

2014-05-05T15:22:00.000+10:00

Puppeteer

As I move my career towards the world of DevOps and automation / orchestration, I feel I need to learn the latest tools of trade for these fields. So, to that end, I'm installing the free version of Puppet Enterprise as a proof of concept. I've heard many people tell me how fantastic a tool Puppet is but haven't actually played with it.

So here I go.

For those wondering, there isn't technically any difference between the enterprise version and the normal puppet version but there are plenty of add ons that require the enterprise version (so I believe). The enterprise version will allow control of up to 10 nodes for free which is more than enough for a proof of concept.

I'm also hoping to build Puppet into a larger story about automation and orchestration.

Just to clarify, if you don't know the difference between automation and orchestration or think they mean the same thing here's the useful definition:
Automation is like musicians in an orchestra, looking the music and following the instructions, the orchestrator is the person up the front conducting, making sure everyone is on the same page and working together as a team.

In other words, the musicians are like the scripts or tools we use in lots of place to do useful work, like puppet agents or like simple shell scripts whilst the orchestrator would be something that can connect lots of systems and coordinate their efforts into something more useful than individual automations.

Hopefully that makes sense. There's a definite distinction between these two terms and it's useful to have an understanding of both for forming a bigger picture.

I'll post again when I've had time to play with my new puppet installation and have managed to make it do useful work.

Cheers.

Tail in Windows

2014-02-04T11:14:00.001+11:00

One of the most useful tools to solve a computer problem is log files. Log files can tell you lots of information about what is happening when a problem occurs. The problem with log files though is that they can often be enormous in size and this creates its own needle in the haystack problem.

The solution as any seasoned *nix admin will tell you is to use tail -f to monitor the log file in real time then attempt to trigger the problem again and see what pops up. If you are lucky the last few lines will relate to your problem and you'll be much wiser about how to solve it.

Now move that process to windows. I've been doing a lot of automation work in windows of late using powershell. Having to solve various problems under windows and lamented the lack of a useful tool like tail. Sure there are stand alone utilities that can perform that function but none seemed to be as simple as tail.

That is until I discovered powershell. Specifically Powershell version 3. Perhaps to appease people like me they have added a '-tail' option to the get-content commandlet. This command simple gets the content of a file. When you combine the tail option with the wait option, you can see anything that's added to the file in real time:

C:> get-content -path "somelogfile.txt" -wait -tail 10

The wait parameter tells get-content to keep waiting for new data to be added whilst the tail parameter says to show the last 10 lines of the file.

In order to get powershell version 3 onto you machine you will need to download an update.

Hacking Active Directory

2013-09-19T16:49:00.000+10:00

So here's the deal. I've been given the task of automating creating objects in active directory from outside the domain.

Being someone with a programming background I thought perhaps power-shell was the way to go. As it turns out, the ActiveDirectory powershell module is included when you made a Windows 2008 server into a domain controller by adding the feature 'Domain Services'.

Ok, so thats ok if you are running a script on the machine itself but that doesn't solve the problem of running from another machine.

Further research discovered Active Directory Web Services (ADWS) which export that powershell functionality over the network. These web services, whilst adhering to normal web services standards, are not meant to be consumed by a web application using WSDLs but by the ActiveDirectory powershell module.

This web service is installed by default in the same way the power shell module is and it listens on the port 9389. This gave me a nice simple test to see if it was running on a given domain controller.

A simple command you can run with the powershell command-lets is:

PS > Get-ADOrganizationalUnit -filter 'name -like "*"'

If you do not specify anything else, it will try to communicate with the ADWS running on the same domain as the machine that is executing the powershell. For a lot of use cases this is what you want and is quite the expected behaviour.

For me, I wanted to run this from a more distant machine. You can specify which machine to query using the server parameter. At this point you will also have to supply credentials for this new domain using the -credentials parameter.

How to create credentials for non-interactive scripts can be found here.

PS > Get-ADOrganizationalUnit -filter 'name -like "*"' -Credential $mycreds -server <servername>

Using this simple set of tools I can now manipulate active directory from outside the domain (given suitable credentials) and automate regular tasks like adding or deleting users or OU.

Hopefully this has been helpful to some.

CCIP It Isn't

2012-05-01T17:25:00.001+10:00

If you are still listening, then perhaps you've heard. Cisco is retiring the CCIP certification. This both good and bad for me.

Firstly, it's bad as now, instead of studying only two more exams to get CCIP, I now have to study 5 or 6 more (more on that later) to get there.

On the other than, it's much closer to the real world in terms of the type of routing you need to know (e.g. IS-IS is back in) which I think is a good thing.

But what this means now is that instead of going for a CCIP certification, I will instead be going for a CCNP SP (SP = Service Provider). In line with other Professional level certifications, there is now a corresponding Associate level certification (CCNA SP) that is a prerequisite for this.

So going back to Associate level, means my certification path is now:
CCNA SP => CCNP SP

The CCNA SP is now two exams and the CCNP SP is four exams for a total of 6 exams but there may be some overlap with the CCNP and thus a possible exam or two that I won't need to do.

There are of course now new topics like IS-IS to learn that weren't in CCIP (though they used to be under the old routing exam BSCI) if you took the ROUTE exam.

I'm actually looking forward to this as it seems much more aligned to work I'm doing now and we all know that's a huge help when trying to learn new things.

Here's to more certificates and the joy of learning.

CCIP It is.

2012-01-03T16:49:00.000+11:00

The treadmill continues. I'm embarking upon another Cisco certification. After my failed CCDP attempt due to lack of study materials, I'm aiming instead for a CCIP which seems much more interesting to me considering it's application for cloud (tm).

First up is QoS which I hope to finish by end of March this year. I'll update you as to how I go.

The IPv6 Conundrum

2011-11-23T15:21:00.001+11:00

I have been making use of IPv6 for over 10 years now and only recently has support for it become good enough that I can think about treating IPv6 like IPv4. For a long time IPv6 was the poorer cousin of IPv4, less features, less support, more bugs.

Nowadays, I think it's reasonable that most people would expect IPv6 to be production ready. I certainly do. I would expect IPv6 to be able to support all my network needs and in theory replace IPv4.

Yet even today I've managed to find an issue that has broken down this assumption. I'll walk you through it but it's reasonably obscure but still annoying nonetheless.

First a little background. Here at work recently upgraded one of our switch blocks to support gigabit speeds to the desktop PCs. We used Cisco 2960S series for the access layer and 3560X series for the distribution layer.

Something interesting about these models of switches is that they all have a dedicated management port. This is a Good Thing &tm; as it allows you to completely separate your management traffic from your production traffic. For those vague on security, this means it's that much hard to take control of the network. Done properly, and your management IP addresses don't need to be routable on your production network at all.

Of course, this sounded great to me. Given, that this post is about IPv6, you might be able to see where this is going, but stick with me.

The configuration for the 2960S is easy enough. The management port is the only port configured for routed mode and given that they aren't normally L3 switches, this makes the management port the only physical port with an IP address. Simply string all the management ports together onto a management switch and you're away.

So far so good, I have a management port on each access switch, assigned both a IPv4 and an IPv6 address. Each port hooks back to a management switch which is connected to the rest of the network via a router (ok, it should be a firewall I'll admit).

Then we come to the 3560X switches. Since these switches are operating as distribution devices they do routing. Now here's the complicated bit, in order to provide separation for the management port, it needs to be placed into a vrf instance (virtual routing and forwarding) else the management traffic will be routed as per normal along with the production traffic.

Not so hard you say? VRF Lite has been supported on these devices for a while now that's true. VRF lite works a treat, for IPv4. When you start talking about IPv6 inside a vrf, you need to move it up a whole notch. Currently that technology only exists for complicated solutions like 6VPE over an MPLS network.

Thus, whilst I can separate my management traffic for IPv4, IPv6 still isn't supported to the same level and thus I cannot have separated management traffic for these devices if I wish to use IPv6 for management.

Yes, I can hear you say, why use IPv6 for management? You know, the simple answer is because I should be able to. After all this time, there shouldn't be any operational differences between these two protocols.Yet there still are which created issues if one is to try to head down the IPv6 only path.

As a result, I'm still forced to treat IPv6 (as much as I love it) as a lessor protocol compared to IPv4. That's just the way it is.

CCDP ARCH Attempt 2

2011-09-26T21:42:00.002+10:00

Yep, It's treadmill time again. Sometimes I wonder, when it feels like this that I bother putting myself through it. Then I remember I really enjoy the feeling after getting that pass mark. But I don't' want to get ahead of myself so I'm just putting it out there that hopefully by lunchtime tomorrow (15hrs time) I'll know if I've passed my 2nd attempt at the ARCH exam. Missed by 4% last time (that's not giving anything away I hope) so I'm hoping the 3months since and the boat load of study have been enough to get me over the line.

ASA and OSPF

2011-08-03T16:05:00.001+10:00

I have recently updated my ASA cluster to 8.4.X in the hope that this might introduce some stability in my IPv6 support. Little did I know it would introduce other issues for IPv4 that I have only now resolved.

Firstly a little bit of background. I run a pair of Cisco ASA 5510 in active/standby mode. They run OSPF to learn about internal routes whilst having a default route pointing out to my policy routers.

The problem I had encountered since the upgrade (or perhaps shortly after) was that the ASA cluster was no longer sending messages to syslog. Upon investigating I could see an obvious route in the routing table, learned via OSPF yet the firewall would state that there was no route.

This was unusual behaviour and whilst I wanted to resolve it , I also needed syslog working. So I created a static host route for the syslog server so that I could get that working. In order to complete the process I had to disable redistribution of static routes within OSPF else all other syslog traffic would end up at the firewalls.

This of course then broke remote access VPN as an IPSec VPN would generate a static route on the ASA which would then need to be redistributed. So I lived with that for a while whilst I continued to investigate.

Today I needed to get the VPN working again so I renabled the static route redistribution which then caused the syslog routes to dissapear. I finally solved it when I looked at the configuration in CLI mode and discovered that OSPF had three network statements! Now normally this wouldn't be an issue but when I looked closely at the routing table I discovered something unusual. For each internal destination there were multiple routes in the routing table, only one of which had an interface associated with it.

I theorized that a bug in ASDM prevented any of the network statements from showing up and thus caused me to add superfluous network statements. In turn, these superfluous network statements caused invalid routes to be added to the routing table and prevent normal traffic from working.

I believe this is correct as upon removing the extra network statements and bouncing the OSPF process, the interface-less routes dissapeared and normal traffic resumed. I was able to remove the static routes and everything functioned as expected.

For reference, I'm running 8.4.2. I can only assume this is a bug and will be resolve in a future update. I'm just glad I've resolved it.

Whole Lotta IPv6

2011-06-02T17:07:00.000+10:00

Wow, I've just had the pleasure of receiving my allocation from APNIC of a /32 prefix of IPv6 address space. That's a lot of address space.

It's a hard thing to think about how big IPv6 address space is, so to put it into perspective, I'll equate it to the size of a single LAN. Now in IPv4 world, a single LAN is usually a /24 which gives 254 hosts on a network. That's a good number for most applications. When we move to IPv6, to support auto configuration, the recommended number of bits was set at 64 bits. Already that's mind boggling-ly huge but ignoring that for now, we can use a /64 as the basis for measuring the size of various IPv6 prefixes.

So a single IPv6 LAN is 1 /64 prefix. The recommended allocation for enterprise networks is a /48 which adds 16 more bits. So this means we have potentially 65,436 IPv6 LANs in a /48 prefix.

Can you see where I'm going with this?

My allocation is a /32 because we are a data center and we allocate address space to clients. So this is 16 more bits again from a /48. One way of looking at it is that there are 65,536 /48s in a /32 or 4,294,967,296 IPv6 LANs in a /32.

If you recognize that big number, you're doing well, it's the total number of IPv4 addresses possible. So my /32 IPv6 allocation has as many LANs as there are IPv4 addresses possible.

Wow. Each one of these IPv6 LANs is also stupendously big but lets not think about that!

To top it all off, this is just a very small piece of the IPv6 global pool. Is it sinking in how big it all is now? There's just so much more room in IPv6 land to spread your stuff out without worrying about conserving every last bit.

Now I just need to renumber everything!

ASA, Failover and IPv6 , Part 2

2011-05-23T10:43:00.001+10:00

After posting the previous post about my ASA cluster and IPv6 I began to have problems.

Initially the solution described (setting the next IPv6 hop to an anycast address) worked as expected and I could get IPv6 traffic through the firewall without trouble. But after a while, for some reason, the traffic would stop working.

My standard investigation process eventually led me to log onto the ASA and try to ping the next hop address which in this situation is an anycast address shared by both next hop routers.

Usually the ping to the anycast would fail whilst the ping to the individual IPv6 addresses would succeed after which point the anycast address would start working again.

This was annoying to say the least and I started to have doubts about the design using an anycast address.

I did know that I wasn't running the latest firmware for my ASA (only 8.2.X) but to upgrade required a memory increase for the new firmware. I simply had to live with it for the mean time until I could get an memory upgrade through.

Fast forward to the (almost) present. Memory upgrade has been completed and I've now got the latest (8.4.X) firmware on the ASA cluster. Before you ask, yes, my problem has now been resolved with the IPv6 routing through the firewall working consistingly for several weeks now. I haven't changed the design, the next hop out of the ASAs is still an anycast IPv6 address. Similarly, the internal next hop address is also an anycast address. Both directions work a treat and have been since the upgrade.

I can take away from this the fact that it is a valid design to use an anycast address as a next hop. Sure it's not quiet the same as a redundancy protocol but it works and that's all I care about. I presume there was a bug or issue with the older ASA firmware that prevented this from working properly.

Now I can move forward to World IPv6 Day testing.

ASA, Failover and IPv6

2011-04-19T19:56:00.000+10:00

Today I finally solved a long standing issue with IPv6 that has been bugging me.

Firstly a bit of background. I have long been an IPv6 advocate and when possible I have been enabling IPv6 in my infrastructure in the hope of one day having a production ready IPv6 network. The last piece of the puzzle was my firewall infrastructure. Last year I upgraded from a single PIX firewall to a pair of ASA firewalls in an active/standby failover cluster.

So far so good. I had already had IPv6 working under the PIX firewalls and soon had it working under the ASA. Until, that is, I enabled failover.

The zeroth problem that has yet to be resolved is the lack of dynamic routing support for IPv6 on the ASAs. There are ways around this but it means that the solution used for IPv4 cannot be used for IPv6 (e.g. the advertising of VPN /32 routes via OSPF) and I believe any differences in operation like this will make transition all the more difficult.

The first actual problem was that the code at the time (8.0 ) did not support failing over of the IPv6 addresses. What this meant was that when the firewalls failed over (often for apparently no reason) the IPv6 address configured on an interface would be come unavailable to the rest of the network. Since there was no dynamic routing this meant that static routes pointing at the ASA for IPv6 needed to be updated both inside and outside every time a fail over occured. This wasn't ideal but I lived with it whilst the fail overs were few and far between.

A subtle issue here is related to IPv6 auto-configuration. Initially I used EUI-64 on both internal and outside interfaces which meant that a fail over would actually change the address on the interface due to the different MAC addresses for each physical NIC. Sure, the obvious solution would to be configure a manual address on each interface. The problem was since the code didn't do correct fail over of the IPv6 addresses, each firewall would hear the others address and complain about duplicate IPv6 addresses and shutdown IPv6 on each interface. Totally useful!

Fast forward a bit and a new release of code now supports IPv6 fail over (8.2.4). This means the internal and external IPv6 addresses get moved across when there's a fail over event. This is great but for reasons unknown to me didn't actually solve my problem. My ASA cluster has two routers inside and two routers outside. After each fail over event, the active ASA could only contact one of the external routers which meant that the static IPv6 routing on the ASA (remember no dynamic routing) needed to be updated each and every time there was a fail over event. So despite supporting IPv6 fail over properly (i.e. each firewall no longer complained about seeing each other) my IPv6 solution was no better off.

Now, I no longer needed to change the static routes on the internal and external routers, but I did have to change the routes on the ASA itself.

I was getting frustrated with this state of IPv6 support given the imperative to embrace IPv6 in the Asia Pacific region. I imagined that it shouldn't really be this hard.

Fast forward to today.

From some dusty corner of my mind I recalled today, whilst trying to debug this issue, the anycast type of IPv6 address. I had it in my head that my networking kit didn't support anycast but it was worth a try.

Checking on the external router, I was able to enter:

ipv6 address xxxx:xxxx:xxxx:ffcb:ffff::1/64 anycast

on one of the external routers. Excellent. I repeated the same on the second router then checked for connectivity on the ASA. It worked! So I updated the routing on the ASA to point its static IPv6 default route to the anycast address now instead of the router specific one.

The next step was to repeat this for the internal routers. My internal 'routers' are L3 switches and I was expecting less complete IPv6 support from them but to my surprise I was able to repeat the above command on the internal L3 switches.

Following this, I repeated the ping test on the ASA and was thankful to get a response. So again, I updated the static route pointing to the internal IPv6 /48 network to go via the anycast IPv6 address.

Now, after all this wrangling, I have a stable IPv6 infrastructure. I can ssh to an IPv6 address on my external routers and presumable, it will all work after a fail over event. I can't see why it wouldn't. I'm now starting to see the usefulness of the anycast addreess type. For a long time (i.e. up until today) I had the impression that they were only good for application level stuff, e.g. dns queries.

I'm now keen to make more use of anycast within my network for say dns or ntp servers instead of using mechanisms such as multicast routing (which isn't yet supported on L3 switches for IPv6!).

Happy routing!

Passed CCNP

2011-01-27T13:42:00.000+11:00

Well finally after redoing my SWITCH exam then finally sitting TSHOOT on the 14th of Dec last year, I have completed all the requirements for my CCNP and am now certified.

I even have a nice little logo that I can include in signatures which is a little bit of icing.

Of course there is no rest and I've already started in on my next certification, CCDA though I'm going to be studying (I think) for the new 2.1 curriculum that has recently been released. This of course makes things harder as the reference material is still being written for this certification but hey, nothing wrong with a challenge!

After CCDA, on to CCDP which is also being updated.

After I get my CCDP its on to the CCIE.

Wish me luck!

The end of the global IPv4 address pool

2011-01-27T13:27:00.004+11:00

Yes, the end is nigh, rumor has it that 2nd of Feb 2011 will be the official announcement day for the end of the global IPv4 pool. Technical types will no doubt understand the implications of this.

What irritates me though, is the media who choose to display their complete ignorance in order to drum up business (I can only assume).

Here are a few pointers for any media types out there:

"IT'S the end of the web as we know it.
Since its inception, the internet..."

The 'web' is not the same thing as 'the Internet', you cannot use these terms interchangeably.

"Web developers have compensated for it by creating IPv6"

Web developers did not invent IPv6, the IETF did many years ago.

"At best, their user experience will be clunky and slow."

IPv6 will not break things or make your Internet experience worse. Buggy software will but that's harder to blame in a news story.

"The current generation of iPhones, for example, won't display anything with an IPv6 address correctly."

Apple iPhones can do IPv6 just fine (those that run iOS 4.X) and will have no issues accessing the IPv6 Internet when it becomes more available. The issue is carrier support for IPv6 which is still absent in most markets.

As a matter of fact, most recent devices will support IPv6, the issue has been the infrastructure. Sure there's no content and what there is has restrictive terms (e.g. Google over IPv6) that see only a few make use of it. Yet the biggest issue is the infrastructure upon which the Internet exists. There's no great incentive to move over to a new protocol considering the costs that could be incurred.

That being said, as part of a companies normal upgrade cycle, IPv6 will be included on newer kit purchased. Some companies will choose to take the initiative and enable IPv6, perhaps gradually; other companies will choose to pretend it is a security risk and disable it everywhere lest someone hacks them via IPv6.

My only hope is the huge media beat up about the exhaustion of IPv4 will get more average businesses aware of IPv6 so that will start asking their ISPs about it and thus create some level of demand!

Will you?

2% of IPv4 address space left!

2010-12-01T13:15:00.000+11:00

Yes, you heard me. IANA has only 2% of all IPv4 addresses left to hand out. In terms of /8 this is 7. Of these 7, five will be automatically handed out to the 5 regional registries meaning there are really only 2 /8s left up for grabs and rumor has it these will go early next year making and end to it all.

I just wonder how much media beat up it will get?

And the other burning question, are you IPv6 ready will soon be on everyones lips. I at least know my work is ready but still I haven't got native IPv6 yet and I've no idea about my home or mobile operators as to wether they will have IPv6 available anytime soon.

Even major content is still shy about IPv6, come on Google, it's time to get out from behind your Google over IPv6 program and offer native IPv6 to anyone who asks without some prior arrangement. It's not helping.

If you're worried about the .01% of people who might have an issue, make sure you make available lots of resources on how to fix their connectivity!!

Anyway, that's my IPv6 rant for today.

Hope you enjoyed!

In the 1/8 block.

2010-11-17T22:24:00.001+11:00

Some while ago I noticed that APNIC had been allocated the 1.0.0.0/8 block. This was a cause for concern as it has been one of the more well know bogon prefixes. These are address blocks that may be used within organizations instead of the rfc1918 address space, e.g. 10.0.0.0/8. I've spoken people who have done this and the usual reason is to avoid issues with others who might be using rfc1918 address space.

iPhone screenshot.

The result, many ISPs would filter these bogon routes lest they leak out onto the Internet.

Now, to cut a long story short, I've noticed my iPhone now has an IP from this range. Time will tell if there are any connectivity issues.

TSHOOT Books

2010-11-16T13:33:00.000+11:00

Yay, today I finally got my foundation learning guide for TSHOOT. I discovered recently that you can get two types of Ciscopress books, Exam certification guides which are all about passing the exam, and the Foundation Learning guides which are more like a course and give you much more depth.

I already had the certification guide so I thought I should really have the foundation learning guide (after failing SWITCH) so I ordered it from Amazon but as usual when you are waiting for something it takes forever to turn up.

So today, some 19 days later it finally arrived on my desk so now for some hard core reading as my exam is less than 1 month away!!

Wish me luck!

CCNP SWITCH Passed

2010-11-01T16:34:00.000+11:00

Yes, finally I managed to pass CCNP SWITCH exam.

Lesson to be learned, make sure you read all those tiny sections at the back of the book that mention exam updates.

It turns out that between publishing of the official certification guide and my sitting the exam the first time around, Cisco had updated the exam curriculum. As a result, the book author had written extra content as a PDF to be downloaded from the publishers website.

If I had know that before my first attempt, perhaps I would have passed, who knows. As it is I took advantage of several other aids to study for my second attempt and I guess it all worked as I passed.

Only one more exam for CCNP, that's TSHOOT booked for Dec 14th!

Wish me luck.

Fixed my Internet

2010-10-01T10:08:00.000+10:00

Last night I finally fixed my Internet! I have a Cisco 857W router for my ADSL connection (thank you eBay) but had problems with it. Every so often, it would simply stop passing traffic. The router wouldn't report any particular issues but wouldn't pass any traffic down the ADSL line. Everything else on the router appeared to be normal.

So, being the network engineer, I downloaded the latest IOS and updated my router. Not only did this not fix the problem but it introduced another one. It would appear that every new connection would loose the first few packets. A ping would loose the first packet, telnetting to port 80 on a website, would delay for several seconds before responding.

It was enough of a problem to prevent the ABC's iView on the PS3 to stop working which was annoying.

Normal web surfing would still work but felt a tad slower. Speed tests would show the expected bandwidth but have a horrendous ping response (> 1000ms).

So last night I downgraded slightly but went from an ED release (Early Deployment in Cisco speak) to an MD release (maintenance release) and found that all these problems have gone away.

I guess this says that there are problems with the latest ED release for my router. It remains to be seen if this MD release fixes the original problem that I was trying to fix.

Wish me luck!

CCNP Switch Fail

2010-09-30T10:57:00.001+10:00

I recently failed my CCNP SWITCH exam. Only by a few marks ( < 5%) but enough nonetheless. At first I was upset but now I've come to realize that I wasn't thinking the right way about a lot of the questions, particularly the simulations.

Previously, I had seen sims in Cisco exams which clearly stated what the end goal was in technical terms but this exam was more about giving a set of requirements and letting you figure out what was required. During the exam I wasn't thinking through the requirements at all, just my own set of technical goals, e.g. must get this ether channel up.

Having had time to think about, I now realize that this comes down to my lack of experience (despite doing Cisco networking for 8 years) of doing design work. I have never sat down at a planning meeting with a client to determine their requirements, not have I been involved in a peer review process in an engineering team.

My networking role is just me looking after a large network (> 800 virtual users) all by myself, with no other network engineers and no 'client'.

The end result is I know my network very well but I have never had to plan or design from scratch which showed up in the exam.

I want to be a good engineer but planning and design are hard to study for unless you've had the exposure.

Never fear, I will keep at it and I hope now I at least better understand what is required of these types of questions in the exam.

Enabling rapid-pvst

2010-09-13T10:41:00.001+10:00

This post is about my attempts to enable rapid-pvst (802.1w) on one of my switch blocks at work. My previous attempt had resulted in lots of loopback errors disabling up-links on the access switches.

My only thought at the time was that perhaps I had way too many end-to-end VLANs that took too long to converge. Now, having read up on rapid-pvst , I now believe it was probably the fact that during the process of enabling rapid-pvst, the default pvst has to be switched off and for a small period you are running without any spanning tree. If you have enough traffic then the probability is high that you could get a loop during this interval. It could still be a combination of effects though and since my original failure, I have been agressive with switchport trunk allow vlans ... to restrict the number of end-to-end VLANs and have been rewarded with success.

I now have managed to get rapid-pvst working on one of the switch blocks to which I originally had problems with. This time though, knowing more I took a cautious approach.

Step 1, Enable loopback errdisable recovery so that if for some reason the original problem reoccurred I wouldn't have to get console access to my access switches or reboot them. This can be done with errdisable recovery cause loopback
Step 2, Turn on consoles message so I can see any errors that might occur. (terminal monitor)
Step 3, Enabled rapid-pvst and wait spanning-tree mode rapid-pvst
Step 4, Save your work once switch is stable.

I noticed on each access layer switch, the management interface went down and came back up. Once I had completed all the access layer switches, I then did the core switch (only one) and once finished I had a stable switchblock running rapid-pvst. Here is the output of show span vlan 188:

VLAN0188

  Spanning tree enabled protocol rstp

  Root ID    Priority    24764

             Address     001b.8f97.2180

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    24764  (priority 24576 sys-id-ext 188)

             Address     001b.8f97.2180

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time 300

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi0/1               Desg FWD 4         128.1    P2p

Gi0/2               Desg FWD 4         128.2    P2p

Gi0/3               Desg FWD 4         128.3    P2p

Gi0/4               Desg FWD 4         128.4    P2p

Gi0/5               Desg FWD 4         128.5    P2p

Gi0/6               Desg FWD 4         128.6    P2p

Gi0/7               Desg FWD 4         128.7    P2p

Gi0/8               Desg FWD 4         128.8    P2p

Gi0/9               Desg FWD 4         128.9    P2p

Gi0/10              Desg FWD 4         128.10   P2p

Gi0/11              Desg FWD 4         128.11   P2p

Gi0/12              Desg FWD 4         128.12   P2p

Gi0/17              Desg FWD 19        128.17   P2p Edge

Gi0/27              Desg FWD 4         128.27   P2p Peer(STP)

Gi0/28              Desg FWD 4         128.28   P2p Peer(STP)

This switch block is one of the ones that I had the loopback issue with so I am happy to have gotten it working here. I will now rinse and repeat and see if I can get it working everywhere!

Are we there yet? Sick of waiting for IPv6?

2010-09-09T09:54:00.000+10:00

Just a quick one today.

I've been using IPv6 since the days of the 6bone and have been waiting ever since for commercial service. Initially I was only interested in IPv6 access at home but now that I look after my own commercial offering, I really want IPv6 available at work.

My provider (who also happens to be my parent company) has allowed me to join their trial of IPv6 via a tunnel from my Internet routers. This along with a /48 allocation of address space as at least allowed me to do a proof of concept and get all my infrastructure properly configure.

The big question is, with IPv4 running out within the year , why oh why am I still waiting for my service provider to get on board and provide IPv6 service?

I've now been told December this year but only if I sign up to a new Internet connection. I've yet to be told if they will be charging extra for IPv6 which would be counter productive. Given that my provider is my parent company, I have no choice but to wait, but what a long wait it's been.

The trouble with TCAM

2010-09-08T09:19:00.002+10:00

Let me start by saying that whilst I am not a CCIE, I understand that the commands presented here are supposed to be CCIE level when it comes to switches.

The background is that I look after a network which uses layer 3 switches for the core/distribution layer. These are mostly 3560s. When I first started implementing the 3560 switches, I read that they supported IPv6. Being an early adopter when it came to IPv6, I sought to enable IPv6.

It turns out that how you enable IPv6 on the 3560 family is by repartitioning the CAM/TCAM tables using the sdm prefer command. This command dictates how much space us used for various kinds of resources, such as layer 2 entries, L3 routes, multicast routes and the mix between them.

Now when I first enabled IPv6 on my 3560s I didn't really understand what a TCAM was or why it was critical to layer 3 operations so I ended up making a choice that for years has impacted the performance of the network.

The command I used at the time was:

# sdm prefer dual-ipv4-and-ipv6 vlan

I figured at the time that I had a few vlans and that would be the way to go. Here is the table showing the mix of resources you get when you choose this option:

The selected template optimizes the resources in

 the switch to support this level of features for

 8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  8K

  number of IPv4 IGMP groups + multicast routes:    1K

  number of IPv4 unicast routes:                    0

  number of IPv6 multicast groups:                  1K

  number of directly-connected IPv6 addresses:      0

  number of indirect IPv6 unicast routes:           0

  number of IPv4 policy based routing aces:         0

  number of IPv4/MAC qos aces:                      0.75K

  number of IPv4/MAC security aces:                 1K

  number of IPv6 policy based routing aces:         0

  number of IPv6 qos aces:                          0.5K

  number of IPv6 security aces:                     0.5K

Can you see something a bit strange here? This line is the issue:

number of IPv4 unicast routes:                    0

Since most of my network was still IPv4, this line allow no space in the TCAM for IPv4 unicast routes! That was most of my traffic. The net result was periodic spikes in CPU usage on the switches when significant traffic went through them. It wasn't until recently, when studying for my CCNP SWITCH exam that I realized that these switches actually do routing in hardware for most traffic as long as there is room in the TCAM.

So I had a configuration that specifically did not have any room in the TCAM so all IPv4 unicast routing on these switches was being done in software. Now the CPU in a 3560 isn't great but its probably sufficient for low level traffic and having a dedicated backup LAN meant that a lot of heavy traffic wasn't routed, yet periodically there was enough traffic to spike the CPU. The cpu would max out at over 80% which is enough to mean other services could suffer.

Before you start thinking that I was a bit negligent letting this issue carry on for 'years' let me state that I had tried to debug this according to the methods suggested by Cisco.

I started out with doing:

#show proc cpu | ex 0.00

This shows the cpu tables excluding anything that's not taking up any CPU. The output of this showed IP Input was the process taking up all the CPU. This is exactly what to expect if lots of traffic is getting punted to the CPU. The next step is to find out why. The command:

# show ip cef switching statistics

       Reason                          Drop       Punt  Punt2Host

RP LES TTL expired                        0          0          1

RP LES Features                           0       4881          0

RP LES Total                              0       4881          1

All    Total                              0       4881          1

This command shows what is causing the CPU punts to occur. TTL is obvious but features requires more detail:

# show ip cef switching statistics feature

IPv4 CEF input features:

       Feature                Drop    Consume       Punt  Punt2Host Gave route

       NAT Outside               0          0       4881          0          0

Total                            0          0       4881          0          0

IPv4 CEF output features:

       Feature                Drop    Consume       Punt  Punt2Host    New i/f

Total                            0          0          0          0          0

IPv4 CEF post-encap features:

       Feature                Drop    Consume       Punt  Punt2Host    New i/f

Total                            0          0          0          0          0

IPv4 CEF for us features:

       Feature                Drop    Consume       Punt  Punt2Host    New i/f

Total                            0          0          0          0          0

This command, in my case, showed huge amounts of NAT Outside Punts. At this point I was stumped. I searched repeatedly for anything that could trigger NAT and explain what was going on.

As you may have guessed by now, that output was a furfy with the problem had nothing to do with NAT.

From the above output of the sdm preferences, it is now obvious that my naive choice for the sdm prefences resulted in no space in the TCAM for IPv4 routes and thus all IPv4 routing was being done by the CPU using the IP Input process.

The solution? Simply change the sdm preferences to dual-ipv4-and-ipv6 default!
"desktop IPv4 and IPv6 default" template:

 The selected template optimizes the resources in

 the switch to support this level of features for

 8 routed interfaces and 1024 VLANs.

  number of unicast mac addresses:                  2K

  number of IPv4 IGMP groups + multicast routes:    1K

  number of IPv4 unicast routes:                    3K

    number of directly-connected IPv4 hosts:        2K

    number of indirect IPv4 routes:                 1K

  number of IPv6 multicast groups:                  1K

  number of directly-connected IPv6 addresses:      2K

  number of indirect IPv6 unicast routes:           1K

  number of IPv4 policy based routing aces:         0

  number of IPv4/MAC qos aces:                      0.75K

  number of IPv4/MAC security aces:                 1K

  number of IPv6 policy based routing aces:         0

  number of IPv6 qos aces:                          0.5K

  number of IPv6 security aces:                     0.5K

Now I have plenty of space for both IPv4 and IPv6 routes what I loose is policy based routing but hey, that's something I can live with. Since this change I haven't had a single CPU spike (> 2 days now).

I have also since learnt that you don't get taught about SDM preferences until you study routing and switching at the expert level.

That's what you get for being an early adopter!

IPv6 Caveat - Apache & XP

2010-09-06T22:02:00.000+10:00

I encountered an interesting problem with my Intranet today.

Certain pages were just hanging halfway through loading. Sure that doesn't sound very exciting but upon inspection of the source code I discovered it was a SOAP call that was hanging.

Trying a manual connection via telnet replicated the hang with an interesting point. It was trying to connect via IPv6. Now I didn't recall manually entering an IPv6 AAAA record into my primary DNS but it so happens that the target machine for this SOAP call is an XP box running the Windows version of the Apache web server and yes, I did recently enabled IPv6 on that machine.

So, diagnosis (yes short post today) is that whilst XP (SP3) supports IPv6 and Apache on other operating systems supports IPv6 it would appear that the two together do not support IPv6. That coupled with Active Directory automatic DNS updates and you have my problem.

The solution was to create a different URL for the SOAP service which only had an IPv4 record and update all the SOAP calls and WSDL files that described the SOAP service in the first place.

So, lesson to be learned, IPv6 support is still lacking despite being around for 15 years. I guess if I'd updated this box to Vista or Windows 7 it might be different but I'm not sure. Before you ask, no I can't install linux, my security vendor doesn't support linux, at least not for this product.

Oh well. Not everything is ready for IPv6 yet.