NetworkSherpa

Solarwinds Geek Speak

John Harrington — Wed, 06 Jul 2016 13:38:12 +0000

I’m sharing some articles I wrote on the Solarwinds Geek Speak blog. I recommend you start with the 80/20 rule below post below before reading through the rest.

A disclaimer: Solarwinds didn’t ask me to promote these posts – I’m sharing them because they’re posts I would have otherwise published on this blog.

Add a comment below if anything strikes a chord, or if you want to disagree, correct me, etc.

The post Solarwinds Geek Speak appeared first on NetworkSherpa.

Reserved Internal Address Ranges

John Harrington — Wed, 01 Jun 2016 06:00:39 +0000

I’ve added a new question to my own network integration checklist, specifically when integrating chassis-based or integrated solutions:

Does the system use any reserved internal address ranges?

Some chassis-based systems reserve private IP address ranges for inter-card communication. This is a perfectly fine setup as long as you, the network integrator, know what ranges are in use. However, you’ll have a frustrating case of ‘disappearing packets’, if you’re not aware that these ranges are in use.
I first saw this issue on an IXIA XM12 chassis a few years back. As I later discovered, each line card received a /24 from an RFC1918 address range. The supervisor used an IP address in each range to communicate with each line card. When I used a conflicting range in my testing the chassis would swallow my packets, and I was left scratching my head until I figured this out.
I thought this was a one-off, but I hit it again recently on an Ericsson Call Control Node. Same problem, but a little easier to detect this time. Nevertheless, I’ve been stung twice now on this issue, so I’ve added to my checklist and brought it to your attention.
If an appliance uses a reserved internal IP address range, the vendor has preallocated your address space to their node. This is a little cheeky in my view, but you still need to ask about this in the planning phase. If you discover the requirement early enough, you may be able to direct the address allocation to a non-conflicting range.

The post Reserved Internal Address Ranges appeared first on NetworkSherpa.

Strain relief

John Harrington — Mon, 30 May 2016 08:29:13 +0000

I’ve got a problem with sagging cables, and I’ve got a simple solution. Examine the side-by-side images below which show the same fiber connection between a switch and a firewall. The image on the left shows a sagging cable which crosses in front of the switch in the rack unit just below it.
As you may know, this cabling install is a violation of the 167th rule of networking:

Thou shalt contain your cables to your own rack unit and shalt not, under any circumstances, impede access to other rack units or blades.

The image on the right ticks the box for me. There’s no room for a dedicated 1RU horizontal cable manager, but there is room for a zero-RU strain relief bar (as seen below). The result is a relatively neat cabling job. It’s no work of art, but it’s functional.
A strain-relief bar is a cheap metal bar that you can bolt on when you rack-mount your switch. It allows you to velcro your fiber patches to the bar, taking the strain to help prevent breaks and preventing the dreaded cable droop. You should, of course, take care to ensure you don’t block access to any field-replaceable units, cards or ports on your network device.
The strain-relief bar is a cheap and easy solution to cable management problems when you can’t dedicate a rack-unit to cable horizontal cable management.

The post Strain relief appeared first on NetworkSherpa.

Effectiveness – Network Truths, Principles and Fallacies

John Harrington — Tue, 24 May 2016 21:56:56 +0000

I recently gave a 13-minute talk to the Irish Network Operators Group (INOG). In this talk I argue that you can become more effective, and a happier engineer by standing back and reflecting. The talk discusses how you work – with reference to some great truths, principles and fallacies.

I introduce The twelve networking truths and the 8 Fallacies of Distributed Computing. I then describe a handful of my own learnings and fancy terms like Chesterton’s Fence and the Gordian Knot.

Check out the video folks, I’d love your feedback.

I was quite nervous preparing for the talk, but I know myself well enough by now to recognise these small fears as opportunities. This is the same fear I faced when I started this blog.

This isn’t false modesty, I’ve done talks before in Amazon and survived, but I was still very nervous. Still, it took me hours of consideration, planning and preparation in addition to a fair bit of anxiety.

I got a real buzz from doing the talk and overcoming the fear. Upon reviewing the video, I know I’ve got a few things to work on. But, by facing a small fear, I’m now slighter better at public speaking and a more likely to do it again in the future.

If you live near Dublin I strongly recommend a trip to an iNOG meet-up.

The post Effectiveness – Network Truths, Principles and Fallacies appeared first on NetworkSherpa.

Clear Pricing for Network Services

John Harrington — Tue, 19 Apr 2016 21:36:24 +0000

I had to buy some switches recently and needed to gather a second quote from another vendor. I went to the Dell website and was pleasantly surprised to quickly find a clear price and a buy-now button for each device on their website.
Normally you’d need an account of the vendors portal to get this information, so it is refreshing to have straightforward access to clear hardware pricing. However it was the list of professional services options shown in the attached image that caught my eye.
You can choose options for ‘rack and cable’ basic deployment or more comprehensive ProDeploy solution. The out-of-hours cut-over and other on-site planning options are a lot more expensive, but I’d imagine those tasks require more polished and experienced engineers to be engaged, and a lot more time-risk. Lastly you have the more tightly defined remote consulting options, with a 1-hour or a higher-priced 1-case options of the course of a single year.
You could quibble with some of the prices but I was really surprised to see fixed pricing for professional services, presented in an easy-to-consume and transparent manner. I’ve always had an interest in business. Now that I’m more focused on network consulting I’m paying very close attention to how consulting effort is estimated and priced and delivered.
I love the idea of offering product-ized professional services. Of course there’s quite a risk with offering a fixed-price service. You need to put strong protections in place to avoid your engineers exceeding the effort you’ve priced into the offering. When it goes badly each engagement could become a loss-making opportunity.
On the other hand, offering a fixed-price for a well-defined task provides a clear incentive for process improvement. Any productivity gains you make from refining your deployment process goes straight to your profitability.
I’m going to dig deeper into this area. Do you have any experience with fixed-price or ad-hoc hourly network consulting engagements that you could share?

The post Clear Pricing for Network Services appeared first on NetworkSherpa.

Writing elsewhere on the net

John Harrington — Tue, 29 Mar 2016 21:43:06 +0000

Hi Folks,
I write for a few other publications, so I’ve made this handy page to link to external articles. I’ll update this page as new articles are released.

Human Infrastructure Magazine

Issue 23 – How To Unblock Your Project
Issue 27 – Email Stinks For Process Documentation

Network Computing

Demystifying The 10x Network Engineer
The Broken Window Theory of Network Configuration

Packet Pushers

All my posts on the PacketPushers Blog
Enjoy.

The post Writing elsewhere on the net appeared first on NetworkSherpa.

Include the why

John Harrington — Tue, 22 Mar 2016 21:54:08 +0000

I recently stumbled upon an interesting speech from 1984 by Charlie Munger of Bershire Hathaway fame. Charlie is Warren Buffet’s right-hand-man, and a straight talking genius in his own right. It’s a fairly long speech and Charlie has a few very interesting things to say, but one particular section on ‘explaining the why’ really struck home.
Here’s a brief quote:

….if you always tell people why, they’ll understand it better, they’ll consider it more important, and they’ll be more likely to comply. Even if they don’t understand your reason, they’ll be more likely to comply.
So there’s an iron rule that just as you want to start getting worldly wisdom by asking why, why, why, in communicating with other people about everything, you want to include why, why, why. Even if it’s obvious, it’s wise to stick in the why.

The ‘why’ is notably absent from most conversations in our high-tech sphere. I’ve wasted countless hours interpreting solutions to ill-defined or undefined problems. I’m guilty of writing many ‘why-less’ documents and emails also. Upon reflection, I can recognise the folly of not explaining the problem at hand before launching into the solution.
When I drop the reasoning and background and go straight for the solution, I find that I trigger much frustrating communication, organisational friction and ultimately lost time.
Perhaps I feel I’m being efficient by banging out a quick email which omits the ‘why’, but it’s certainly not effective. Perhaps I fall into the trap of assuming everyone is on the same page and the ‘why’ should be self-evident.
I’m with Charlie on this one, I think we could all provide more ‘whys’ in our email, network designs, change control documents, etc. I’m going to try and add more ‘why’ to my communications over the coming weeks with the goal of reducing friction and improving the effectiveness of my communication.
What do you think?

The post Include the why appeared first on NetworkSherpa.

Getting started with Network Packet Generators

John Harrington — Wed, 09 Mar 2016 22:36:24 +0000

A friend of mine has just ordered a shiny new packet generator for his network lab. I’ve spent some time working as a QA engineer in a network lab and wanted to share some advice.
You can purchase stateful and stateless packet generators from major vendors like Spirent, IXIA or Agilent. If you just need to test throughput, latency or loss, a stateless packet generator will do the trick. The test hardware will use an ASIC to produce line-rate 10G traffic or higher. The Cisco Enterprise Testing Book calls this a ‘bit-blaster’ which I love. In the wrong hands it can also be a ‘network-melter’.
You need a stateful packet generator if you want to test your routing protocols in conjunction with traffic load. A stateful packet generator such as Ixia’s IxNetwork, will use dedicated CPUs to form and maintain adjacencies, inject routing protocol packets, etc. You can use the stateful feature to inject prefixes which are then used as test targets by high-rate stateless traffic.
Licensing is a major source of pain when operating a stateful packet generators. There are often licenses required per protocol and even per-combination of protocols. For example, I had to buy a license for LACP and OSPF individually but there was also an additional license to use OSPF over a LACP LAG. You should thoroughly investigate feature licensing before you choose your vendor or cut that first purchase order.
There is a steep learning curve for packet generators, mainly because they are GUI configured and windows-based. I’m sure the GUI was user friendly at the start, but with the explosion in features and options the UX is pretty poor. These systems are now crying out for a structured CLI.
The packet-generators also behave differently from regular routers, in that they will blast packets out a TX interface regardless of path health and whether they have a route to the target IP address or not. This allows the packet generator to measure loss and convergence, where the loss and re-appearance of test frames is recorded by the RX port of the generator.
A big risk with load testing is that a 10Gig flow could escape the lab into your enterprise network. This can easily happen if you’re injecting a line rate traffic flow into a device-under-test (DUT) and you lose your route to the flow destination. The test traffic flow will often match the default route and leak into the lab-management network, potentially taking your enterprise network off the air. Not that I have ever, (repeatedly) taken a lab off air mind you! You can mitigate this by disconnecting the DUT mgmt ethernet interface, place it in a mgmt VRF or use more specific routes back to your enterprise network.
There is a step learning curve for these tools and you can easily lose hours of your time with nothing to show for it. I recommend the following steps sequentially to gain the most productive use of your time.

Start small – A single DUT with two ports connected to packet generator
You don’t have CDP. Toggle interfaces on DUT and tester to ensure you’re on the right port.
Ensure you have ARP resolution before you start your traffic flows.
Test a single flow at a very low traffic rate. Verify you’re receiving no packet loss before continuing.
You can then ramp up the line rate to test for errors, throughput etc.
Or.. you can keep the line rate low and add dynamic routing for a single DUT, adding the prefixes as flow targets.
When you’re happy with the performance of a single DUT, you can move on to a system-under-test (SUT).

The post Getting started with Network Packet Generators appeared first on NetworkSherpa.

5 ways to fail – WAN link acceptance

John Harrington — Thu, 03 Mar 2016 08:00:10 +0000

I’ve had an interesting few months doing WAN circuit turn-ups for a new Data Centre. I dealt with three major carriers, and each experience was worse than the next. I’m not sure why I held such high expectations but I was surprised by their hopeless inefficiency in delivering what should have been a standard product. In this post I’ll examine the problems I saw and their root causes.
In all three situations, 1Gbps Layer-2 ethernet circuit was ordered with a copper ethernet handoff from a rack-installed NID/NTU/whatever-you-call-it-yourself. Lets look at the five issues I hit whilst troubleshooting.

Link up at both ends – No CDP received

There was a lot of blaming the end-customer on this one. “Are you sure that CDP is enabled?”. There was a huge amount of frustration here. The carrier would send an email to confirm that ‘they had tested’, provide no actionable details of their troubleshooting, then close the ticket. This went on for days bouncing between the annoyingly named ‘deliver’ and ‘assure’ teams. The ‘deliver’ team felt they had delivered the circuit and the ‘assure’ team assured us that the circuit wasn’t live and they couldn’t help us.

The ‘deliver’ team felt they had delivered the circuit and the ‘assure’ team assured us that the circuit wasn’t live and they couldn’t help us.

We kicked up an almighty stink and got an excellent engineer on a conference call who was able to solve the problem within hours. The problem was that one end of their circuit had a switch between their MPLS PE and their NID, the other didn’t. The provisioned circuit assumed no switch in the path, so the carrier switch added a transport Vlan tag at ingress that the egress customer router didn’t understand. Similarly the carrier switch saw my Vlan tags as an unknown transport tag and rejected my frames.
Diagnosis: Circuit not provisioned to expect dot1q on one end of circuit.

Link stays down after flap

In this case the carrier handed off a circuit that was passing CDP frames. When I shut/un-shut the port on my switch the corresponding NID port stayed down until the carrier shut and un-shut the NID port. I’m suspicious of a faulty hardware diagnosis generally, but in this case a NID swap fixed it. I later found out that the carrier manually stages the configuration on the NID before shipping to site, so it may well have been a provisioning issue… we’ll never know.
Diagnosis: Faulty NID

1Gbps Link negotiating to 100Mbps

I also saw issues negotiating the correct speed/duplex settings. In this case I couldn’t get beyond 100/full when auto negotiating on a 1G port. After a lot of ‘try all the options’ troubleshooting, it turned out that two of four pairs in the ethernet cable were dead, allowed a max autoneg rate of 100Mbps. I can’t blame the carrier for everything; I should have caught this one sooner.
Diagnosis: Damaged Cat5e cable prevented 1Gbps, but working twisted pairs allowed auto-negotiation to 100Mbps full.

CDP frames passing, but I couldn’t ping

This one stumped me for a while. I was trunking two vlans across the circuit, and trying to ping from SVI to SVI on 3750X’s. No dice. Once I converted my port layer-3 port I could ping without issues.
After checking my SVI and VLAN configuration was okay, I had a hunch and re-added the VLAN/SVI configuration again but configured one VLAN as a native vlan. Praise the god of ping…. it worked for the native VLAN.
Diagnosis: The carrier hadn’t configured the link to accept dot1q tagged frames from the customer.

Failing Backups

Two days later I received reports of network backups failing but I couldn’t find an obvious network issue. But as soon as I heard a report from another use who could only fetch small files, I immediately knew what the issue was. A quick ping with an MTU of 1500 showed full loss, but a ping of 1496 bytes worked just fine.
Diagnosis : MTU issues caused by carrier not raising MTU to cater for VLAN tag.

Sherpa Summary

These problems can’t be new nor unique to me. I’d love to see more folks publish their failures so that we could avoid the situation where we all solve the same set of problems in parallel. While I wait for nirvana, I hope these examples of failure help speed you through the circuit acceptance process and on to more meaningful work.

The post 5 ways to fail – WAN link acceptance appeared first on NetworkSherpa.

Does your Wave2 AP need NBase-T?

John Harrington — Tue, 01 Mar 2016 08:00:07 +0000

Cisco recently launched the 2800 and 3800 series 802.11ac wave-2 access points. The 3800 Datasheet quotes a theoretical maximum throughput of 5.2Gbps when operating in Dual 5GHz radio mode (2 x 2.6Gbps). If you ran two cables to your AP you could use the second ethernet port to create a 2 x 1Gbps LAG. However there is still some debate about whether 2Gbps of throughput is sufficient for a single-radio Wave2 AP.
Some companies may not be willing to invest the time and expense to swap out their copper for fiber or run yet more copper to their APs. The NBase-T standard 802.3bz provides an alternative approach, promising speeds of 2.5Gbps or 5Gbps over Cat5e cabling over 100 Meter runs.

Peter Jones from Cisco is the chair of the NBase-T alliance and presented to us in Tech field day on the new 802.3bz standard and the technology behind it. Cisco terminology for NBase-T-like functionality is ‘MultiGigabit Ethernet’. Currently the Cisco Catalyst 2k, 3K, and 4K switching line have specific models or line cards which support a number of combined UPoE/MultiGig ports. The reason for new hardware is that new digital signal processors (DSPs) are required to achieve the 2.5Gbps and 5Gbps rates.

This isn’t the first time Cisco have used a hardware solution to extract greater throughput from existing cabling plant. The standards-based LRM transceiver allows you to run 10Gbps over existing OM2 multimode fiber. The proprietary Cisco BiDi transceiver allows you to run 40Gbps of a standard duplex pair of OM3 or OM4 multimode fiber. In both of those solutions the transceiver got smarter – however with NBase-T we need new switches or line cards.

What’s the business case?

You can start small by adding a MultiGig 3850 model such as a WS-C3850-24XU to an existing 3850 stack or add a MultiGig line card to your 4500. However I recommend that you step back and look at the broader picture before you jump into this new technology.
Will your AP throughput require NBase-T? – If you choose a dual-radio Wave2 AP you double your demand, but do you need dual-radio AP? I’m not a wireless guy, but I suggest you perform due diligence to avoid overbuilding your infrastructure. In short, do a trial if you can and create a business case based on real-world expected throughput.
What is the pay-back period? – When will we see APs with 10G fiber connections? Will you get enough value from your Multigigabit switch ports and APs before fiber-terminated APs are launched. Talk to multiple vendors about their roadmaps.
Standards – The standard is close to ratification is not yet fully ratified. Cisco is shipping MuliGigabit support rather than NBase-T support. This may be a small risk, but it’s a risk you’ll need to quantify.
InterOperability – If you do commit to NBase-T, does your second choice AP vendor inter-work with your current switch?

Sherpa Summary

Cisco did a great job of presenting an integrated solution for Wave2 technology. I think NBase-T is really cool technology and I’ll wager that 802.3bz will eventually be widely deployed. But I recommend that you fully examine the requirements and constraints before jumping into any new technology.

The post Does your Wave2 AP need NBase-T? appeared first on NetworkSherpa.