I’m sitting at my desk right now (9pm Sydney time), waiting for my 3am change window that I need to be onsite at the data centre for. Now I am supposed to be working on a presentation I am giving on Friday entitled “Automation, Orchestration, and SDN”. I’ve been given about 20 minutes to cover these topics, so I’m spending quite a bit of time trying to get the right mix of content without geeking out too far for the intended audience (C-levels, project managers and lead engineers).

While trying to get the mix just right, my mind keeps going back to a comment I made last week on the “APAC Virtualization Roundtable” podcast (Episode 74 “SDN & Virtualisation from a Network Engineers perspective”)…

“Increase the awesome, remove the mundane!”

While initially sounding facetious, this comment is kind of where my brain has been going of late when trying to work out where I should focus my energies. While I’m thinking of ways to apply this idea to many aspects of my life, I would like to present them in the context of the presentation I am about to give (and the same context to which I spoke about in the podcast).

So I present to you my, modified, task prioritisation filter…

Remove the mundane

It’s a simple statement of fact that I get bored easily. It’s part of the reason why I blog so rarely… it’s hard for me to concentrate long enough to string sentences together once my work day is over! Another statement of truth is that I am lazy! I don’t like doing things over and over again. I believe that every network engineer must truly understand how the parts of the network interoperate together – be it Spanning Tree, OSPF, MPLS or firewalling. To do this, we need to get our hands dirty and “string tin together”. I also believe that there are only so many times we should do this by hand. Once you “know” it, automate it! By doing this you are not only speeding up the time it will take you should you need to do this again, but you are also effectively “documenting” the process in such a way that it records what has been done and allows other engineers to benefit immediately.

Now the realities of my day job mean that I cannot just refuse to do certain tasks because they are boring, but I can start to work smarter not harder. I really need to start automating the regular tasks I do in such a way that when I am required to do them, they can be done with a few taps on the keyboard or a few clicks on a website. Once I have automated these functions it then becomes easier to offload that to somebody else (i.e. make it their problem!).

This is why I’ve spent time over the past few months learning to code in Ruby (Ruby just works the way my brain works, I am sure Python is wonderful too!). This is also what tools like Puppet and Chef allow us to do. By moving our repetitive workloads into automated scripts, and starting to adopt aspects of the “Infrastructure as Code” mentality, we can start to make the boring and mundane tasks something we can spend less time doing.

I’m going to work on a series of posts about “Removing the Mundane” over the next couple of months, as I investigate and experiment with how automation and orchestration can improve the daily workflows for both myself and my customers.

I also honestly believe that “SDN” (such a loaded term!), will go a long way to helping us automate traditionally labour intensive tasks. Whether we are talking methods to directly program ASICS, or building overlay networks using protocols such as VXLAN, NVGRE or MPLS, the ability to define a service end to end and apply that as an atomic action across the network can not be underestimated.

Increase the Awesome

So now that we have freed up all of this time for ourselves, what should we do?

I am not sure if I am just old and at that stage of my career, or if I am just falling for industry buzzwords, but over the past 12 months my viewpoint of my role as a network engineer has changed. Despite what we might think, our customers are not buying stretched VLANs, VPNs, or TCP port 80 and 443 opened on the firewall. What they are “buying” from us is an application that provides a service. The skills I bring to the table are ability to deliver that service from where its hosted (whether in the cloud or our own data centres) to where it is consumed by the user, while also maintaining quality of service, privacy and responsiveness to ensure the best user experience.

The definition of this service include:

• All of the database, application, and front-end servers that the app depends
• Any storage requirements for the servers
• Any load balancers, and firewalls along the path that provide for scale
• Any layer 2 and 3 networking requirements required to allow the above components to operate
• … and probably many more I cannot think about right now!

If we can start to define the end to end requirements of the services we need to deliver and have them deployed orchestrated in such a way that they do exactly what they need to do, when they need to do them without our intervention we can begin to focus on actually delivering better applications and services to our end users. This is what I mean by “Increasing the awesome!”.

Once we take our focus away from the repetitive and mundane we can focus on delivering better services. The “server guys” learned this during the virtualisation craze, and put it to the ultimate test with “cloud networking”. Once the mundane task of spinning up servers and deploying “the standard stack” of applications was removed and servers could be deployed in minutes (sometimes seconds!) a whole new world of possibilities were opened up. It was no longer a pipe dream to spin up and down web servers on demand to meet demand (and reduce cost). Another benefit was the ability to focus on horizontal scalability by taking advantage of load balancers, Message Queues and scalable distributed databases because it became easier to deploy additional servers to a task instead of just larger ones!

Do I know what “awesome” is for the networking industry? Well other than beers paid for by vendors… no, but I certainly know that while we’re all focusing on mundane tasks, we are not focusing on delivering the awesome!

Deliver your Space Jam

This last one was not actually a part of my original comment, but its inline with what I have been thinking about. I am not sure if you have seen this “Pep Talk” by Kid President or not, but I urge you all to watch it.

I actually have a weekly reminder in my calendar for 2pm on Wednesday (Hump Day) with that link to “inspire me”. In line with “Increasing the Awesome”, you need to find something you are passionate about and work towards delivering that. Right now, I am working through a couple of things that I believe will make the “world a better place” while still utilising my professionals skills. Satisfaction in life comes from using what you do well to achieve a worthy goal.

What’s your Space Jam?

Now how will you work to deliver it?

Mop and Bucket

Our industry might be changing, even if it is unclear in which direction, but we need to focus on the best way to deal with these changes. We need to stop being road blocks, and start working with other technology teams, as well as the business as a whole, to make each others lives easier so that we can all focus on delivering more awesome.

(And to make sure this isn’t all just airy fairy rubbish… more awesome delivered the right way also means more customers thus more revenue – and happier ones at that!)

Disclaimer: I’m not going to claim these thoughts as anything original by me, I know for a fact I stole the “Increase the awesome” line from somebody who presented at a conference I watched on youtube, but for the life of me I cannot remember who or where. Having said that, this doesn’t change the fact that the message is true!

Finally… it was bound to happen. My three year journey is complete.

It was about this time last year that I posted about my second JNCIE-ENT lab attempt, and sadly it didn’t go the way I wanted it to!  Due to work commitments I was not going to be prepared to sit the 2012 Q3 round of lab offerings, so I resolved to sit the December / January round.

I picked up my studies again and worked on my weak areas noted during my first two attempts. I paid particular attention to areas of multicast and switch security as these two topics were areas of weakness for me last time. I was lucky enough to work on a project at the end of last year that included nearly 1000 ports of 802.1x with dynamic VLAN allocation, so that proved to be an excellent “lab environment” for me.

After a need to reschedule for March, as the January exams were cancelled in Sydney, I knew I was going to do everything in my control to pass this time around. I didn’t want to face the thought of making attempt #4!

Long story short this time around I felt much more prepared and actually finished with more than enough time to go back through all of the questions to confirm the points I believed I had earned. I cannot say enough about good time management efforts during the lab. Set a maximum amount of time per question (or per point) and really make yourself stick to it. Unless you know you need this question to continue on to later questions, just leave it as it is (preferably in a non-broken state), and move on to other questions. Earn more points and come back later.

I finished up the exam reasonably confident, but I also knew I felt confident after attempt #2. I caught a train home and prepared myself for the 15 business day turn around for the exam results. Luckily I didn’t have to way this long to get my results, and after a couple of days I was advised that I had passed!

I am now JNCIE-ENT #368… and Im very happy / relieved

In previous posts Ive covered tips and tricks etc, so I would really like to dedicate this post to saying thanks to people.

All my thanks

• First I would like to thank Liz Burns and everyone at the JNCP team. Thanks for all the work you guys have put in to making these exams and the encouragement you provided me over the past three years has been must appreciated.
• All of my current and former co-workers. Thanks for the opportunities I was given to work on real life network environments and exposure to quirky designs. Also there is no end of thanks for putting up with me on all those days I had my cranky pants on
• All of my current and former customers. Without your need for new networks, and not to mention some of the crazy requests you have thrown at me over the years, I wouldn’t be able to get my head around some of the tricks of the lab.
• Thanks to my friends at Juniper, most notably Francois Prowse, Ashton Bothman and Doug Hanks. Again thanks for the encouragement and the kick in the butt I needed not to give up. Many people from within Juniper were invaluable in answering many of the problems I ran into while studying.
• To everyone on #juniper over on Freenode. Much appreciation for your advice… “hrmpf“
• To Chris Jones and the guys at Proteus Networks. I was lucky enough to be performing “Technical Editing” duties on the new Proteus JNCIE-ENT Preparation Workbook (which was obviously pre-release at the time). As part of this I labbed almost every scenario from the workbooks to ensure they were correct and the descriptions matched. I cannot recommend this guide enough to people studying for the JNCIE-ENT, and I plan to do a separate review in the near future.
• To Burkey and to Nick. I hope you guys both know what your support meant. #FHP
• To “The Wolfpack”. ARRRROOOOOOOOOOOOO! You know who you are, thank you for allowing me to “de-stress” and express myself in some pretty non-conventional ways
• Last but not least, to my wife Belinda. Thanks for all the love (and feeding) during this journey. You suffered more of the cranky pants than anyone else, and yet you still encouraged me to keep going. Love you heaps

Earlier today Ethan Banks wrote a really good blog posts about “Thoughts on Working as a Consultant for a VAR“. I found his point of view quite interesting and I will say I can understand his points. I can also say that I would rather be a consultant than a full time engineer at a customer site. As a little bit of background I have spent most of my career working as a consultant. I did do a two year stint as network operations manager for a wireless ISP which itself was quite fast paced, but other than that Ive work as a consultant in one form or another.

Maybe I have ADD, maybe I just need to focus, but I have found that constantly having different projects going allows me to satisfy these tendencies. I feel I work better with more than one thing to occupy my time. I see friends who work for enterprise customers who spend their days submitting change requests that third party support companies fulfil, or spend months writing detailed design guides for projects that inevitably get canceled and all that time is spent without getting to touch the things they got into this industry for. Which brings me to my first point…

Toys, Toys Toys!

Im 32 years old, and I still love toys. Sometimes I think the only reason why Im in this industry is the ability to play with expensive and complex toys. And I get paid to do it! I enjoy the challenge of learning a new piece of equipment and deploying it into a customers network. I enjoy learning all the new opportunities that a new software update or a router line card can offer to my upcoming network designs (my sales guys also enjoy being able to sell those things to existing customers… bloody sales drones!).

I have worked at my current employer for 10 months now and in that time I have worked on many projects, not the least of which has included rolling out the second Juniper QFabric deployment in Australia (and one of the first in the world), delivering security solutions based on high end data centre SRX devices including IDP, SSL Inspection and Proxying, a relatively large Juniper MAG deployment across multiple sites with failover, and numerous network audits across many verticals including finance, utilities, enterprise and service provider networks.

The other aspect to toys is the ability to get your hands on all the right equipment to learn and play with *before* you need to deploy it. This really comes into its own when you have…

The right employer

Ethan’s posts hinted at being burned by his previous employer. I know I can attest to that over the past 15 years Ive been in the industry. Sometimes its the customers, sometimes its the employer. The right employer will understand your career aspirations and will endeavour to provide you with all the resources you need to be successful during both your time with them as well as your career beyond them. The right employer understands how often engineers change jobs in this industry and that what goes around comes around.

Your employer needs to understand that certification is beneficial to him, but more importantly that you as the engineer need to understand the content of the exams you are passing. Perpetuating the degradation of industry certifications by promoting brain dumps or becoming a “puppy factory” for certifications churning out sub-par engineers devalues both the certification as well as your ability to sell your services to customers.

The right employer will cover your expenses for gaining certification, and provide you with the resources, both equipment and time, to actually complete these career goals.

If you’re lucky your VAR will give you access to both pre-sales and post-sales implementation work giving you a wider view point of the network life cycle as well as building up skills that end customer engineers will never get to practice. And you will do that many times a week!

Honest about the downsides

• You’re supposed to know everything: I agree with Ethan that this can be stressful, but it doesn’t take long to pick up the habit of either being one google search ahead of the others, or as I often say “Not knowing the answer, but knowing where to find them”. If you work for a consulting firm that doesn’t require your customers to buy some form of vendor TAC support then you are further against the wall and you/they are not operating in your customers best interest. *Always* have the ability to escalate.
• You’re a commodity: Yes you are a commodity, but you also are (should be) “in demand”. If you are doing your job well, and your employer knows how best to leverage your skill set appropriately (not always in your control) then this works in everyones favour. If you are not doing your job well then you shouldn’t be a consultant, and if your employer doesn’t know how to effectively sell your skills you should be looking elsewhere. You will get abused, ignored and eventually get bored and quit or made redundant. Scary truth, but its better to move on and find something you are happier doing.
• You’ll sacrifice your body if you’re not careful: Well I’m not even going to pretend to deny this one. I am living proof that ignoring the health aspects of your life will creep up on you, but Im not convinced this is specifically related to VAR lifestyle. Any high stress, fast paced career can do this to you. You just need to keep these things in check. On a side note to this one, I’ve taken up a Paleo/Primal lifestyle for the past couple of months and Ive found wonderful improvements to my health without major sacrifices (in both time and diet), but thats a whole other blog post!

Mop and Bucket

Yes working for a VAR can be hard work. It can be fast paced and involve a lot jumping between contexts, but if you are like me you just may find that the very draw card to this career option.

Sometimes I dream about working on a single network and finishing all those “little things” that I wish I could implement for all of the networks I build. Then I spend two weeks onsite at a customer and all I want to do is get out of there.

If you are willing to invest 12 months of your life to “give it a go”, you will learn far more in that time, and touch a wider array of equipment than you could hope to touch in 3 technology refresh cycles at an end customer.

And this is what keeps me coming back to work tomorrow!

The Setup

So there I was, setting up my “command centre” for the 15 hour flight from Sydney to San Jose via San Francisco. I had my Macbook and my iPad fully charged, as well as ye olde print book ready just in case. I idly flipped through the inflight entertainment guide to see what movie would be playing on the main cabin display.

“Midnight in Paris” ? What’s this rubbish? Oh… Woody Allen… right… yeah… that’s just what I need

Thankfully I usually fall asleep the minute the plane pulls away from the terminal and wake up just as we’re coming into land. Should be able to sleep most of the way, and maybe read a few pages of my book between naps. This usually gets me in trouble with my wife, but I was flying solo on this trip so I was free to do as I pleased.

As fate would have it, my Macbook battery was at about 50% just as the movie started so I decided to “save some for a rainy day”.

Let me put on my headphones and see what this movie is like. It’s a Woody Allen flick (in Paris no less) so Im bound to be asleep in 20 minutes.

Oh it’s about “Art” is it? Lets make that 15 minutes!

Why is Rachel McAdam’s character such a bitch? That’s not like her.

Hrrmm… I never realised how much Owen Wilson acting the way he always does was so much like Woody Allen before.

And just like that, I was sucked into watching this movie I had no desire to see!

The Wonder

So the premise of the movie is that Owen Wilson’s character, Gil, is a writer on vacation in Paris. His romantic notions of life in Paris during the 1920′s, during the time of Picasso, Hemingway and F. Scott Fitzgerald, are at odd’s with his wife and her family. While he wants to revel in the beauty and atmosphere, they are more interested in keeping up appearances.

During a stroll on his own looking for inspiration, and after a few drinks, he finds himself lost as the clock strikes midnight. At this time an old car pulls up along the road and the occupants invite him to join them. During this trip he is transported back to the 1920′s where he meets his heroes Hemingway, the Fitzgeralds, Gertrude Stein, and a whole host of other literary and artistic greats. Over the course of the movie he spends his days with his family and his nights with his heroes. During his time with his heroes he is able to approach each of them for advice about his book, his personal situation and life itself.

This same opportunity was offered to me back in 2011, and that was why I was on this plane. I had been invited to attend Tech Field Day’s Network Field Day 2 at the end of October of that year. I had actually received the invitation on night of my birthday in early September, and here I was with the opportunity to rub shoulders with my own personal heroes in the networking industry. The likes of Ivan Pepelnjak from a blog I read daily (blog.ioshints.info), Greg Ferro and Ethan Banks from the Packet Pushers Podcast, not to mention Stephen Foskett the man behind the whole event, as well as several regular contributors to the Packet Pushers Community and friends from Twitter. Just like Gil, I had the opportunity to get feedback and advice from those whose advice I sincerely valued. The ability to sit down with both the other delegates as well as the representatives of each of the sponsors was truly insightful.

Coming from Sydney, the “scene” in the whole Bay Area is entirely different to what I was used to. People dressed extremely casually here and there was a vibrant community buzzing with all sorts creative and cutting edge technologies. This was uniquely demonstrated during the Open Flow Symposium that was held on the day before the Network Field Day event began. Both vendors, implementors and attendees were all excited about the possibilities that were coming. Some of these ideas I was aware of before arriving in San Jose, but the depth and pace of these changes took me by surprise. Everybody was open about ideas – both what would, wouldn’t and couldn’t possibly work! Many names and faces I had been following online for the past 18 months attended this event. I was in awe of everything. I dont think I spoke to too many people that day as I was too stunned.

Over the next (densely packed) two days, we were taken from vendor site to vendor site and presented with their latest and greatest – and usually by their brightest. No questions seemed off limits and each of the vendors truly seemed to take on board the questions we raised and the “advice” we offered. These two days were completely information overload mode, and I dont think I was prepared to capture all of the useful information that was being presented to us. I have had to go back and watch the various videos several times to see what parts I had previously missed.

One thing that was pointed out to me by Stephen Foskett as we were driving through all these massive campus buildings was that “Over there is this massive building, costing millions of dollars. They do something in our industry, and I have no idea who they are. And this area is filled with these places. It’s exciting to see so many new companies and wonder ‘whats going on in there’ “.

I had an opportunity to spend intense nerdy times with the other delegates and sponsors during the jam-packed days, then follow on into the night just geeking out because the atmosphere was so charged. Everywhere I turned there were people doing “great things” that I wanted to be a part of.

Just like Gil, I too was caught in the position of “being where I wanted to be, and never wanting to leave”

The Reality

After 5 days away, I returned home to my normal life. Well, sort of normal. My wife was in the middle of a three month stint working in an outback Aboriginal Community School, and I wasn’t going to see her for another week or so. Then there’s all the things at work that needed tending to since I had been away.

I found my self longing to be back in San Jose with the group I had just spent so much time with. It didn’t matter that all the other delegates had returned to their own homes, that is where I wanted to be. And my reality was far removed.

“Nostalgia is denial – denial of the painful present… The name for this fallacy is ‘Golden Age Thinking’, the erroneous notion that a different time period is better than the time one is living in. It’s a flaw in the Romantic imagination of those people who find it difficult to live with the present.”

This quote above comes from a character in the movie who was a know-it-all A-hole, but he had a point. His witty remark to Gil’s character was both spiteful yet accurate, though his didn’t stop Gil from still pursuing his dream.

I knew that if I wanted to progress it into the areas in the industry that I wanted to be a part of, that I would have to make certain changes to my career path. I mapped out what I felt were a series of career goals and achievements that I would need to accomplish in order to make headway. Since this time I have changed jobs to focus on the particular projects and technologies I felt I needed, buckled down, put in a lot of study and research.

Also during this time I went into “Social Media Radio Silence”, as I was busy focusing on some of my end goals and needs. I was so busy focusing on the future, I was completely ignoring the present. I had lost my inspiration to write because I felt so overwhelmed by all the things I didn’t know. Instead of documenting my discoveries, I was actively avoiding the things I couldn’t answer.

Life got in the way and I was feeling discouraged.

The Re-Awakening

During Gil’s trips back and forth between time periods, he meets a young French girl from the 1920′s named Adriana. Even though she lived in the time period that Gil romanticised, she longed to live in an earlier period of Paris’ history – a time she felt was truly inspired. And fate (and of course the script) would have it, both Gil and Adriana found themselves in Adriana’s romanticised time – La Belle Epoque!

While Adriana is lost in her dream, they meet Paul Gauguin and Edgar Degas who themselves were discussing how “this generation is uninspired”. This is when Gil discovered the truth, and had to explain it to Adriana who didn’t want to leave:

“Adriana, if you stay here though, and this becomes your present then pretty soon you’ll start imagining another time was really your… You know, was really the golden time. Yeah, that’s what the present its. It’s a little unsatisfying because thats what life is – just a little unsatisfying”

And in with these words both Gil, and myself of re-watching the film, knew what needed to be done.

If I ever want to write something worthwhile, I have to get rid of my illusions that I’d be happier in the past”

Mop and Bucket

In early February, Stephen Foskett was out in Australia to keynote two VMUG gatherings in Sydney and Melbourne. During this time we spent the better part of two days catching up, discussing life, the industry, and careers. Whether he knows it or not, those conversations really helped solidify some of my goals and plans.

I’ve learnt that Australia is no farther away than I make it, and I can still be quite active and involved even if I am not “in the heart of things”. Positioning oneself to take advantage of opportunities that arise, and to connect with various people both within our industry, as well as the clients we have, is the best way to stay in front of the game and stay involved. This is evident in the fact that I was invited to be a Juniper Ambassador in October 2012.

I would like to think that I have more technical posts coming in the near future, because it certainly feels like a long time since my last one. I’m feeling inspired again, and getting ready to re-engage. And for that, I am truly sorry

PS. Please refer to my Disclosure Statement in reference to my participation in the Tech Field Day events, as well as the Juniper Ambassador program.

Im not sure how many of you follow my wife, @MrsJanitor, on Twitter but we met 7 years ago today.

When I asked about our plans for today she surprised me with some information. Ever the romantic I decided to publicly share my love, and thus I present you:

And to show just how much she “gets me” her only reply was:

Yes, my wife is a high school teacher and she carries her Juniper umbrella on playground duty!

Love you babe and life wouldn’t be the same without you!

PS. Belinda is the owner of all vendor swag that enters this house. So remember that when you are choosing what to hand out guys

Well, now that we’re into February, I guess I should probably do a post about the new website layout I worked on over the Christmas break. Hopefully if your viewing this from my website, as opposed to via an RSS feed, you notice my new theme as well as the amazing new logo provided by Stijn from Studio Stayne.

I approached Stijn in November about some design ideas I had for a logo for the blog, asking very non-creative questions with some pretty broad strokes ideas of what I was after. Within a couple of days he had presented me with a handful of options that were all unique, yet still managed to meet my design “spec”. Im not good at making decisions, but after some ideas back and forth about the things I liked I received several more workshopped ideas taking the best bits of each design. The fact that I am not very creative is probably obvious given the layout of my blog for the previous two years, but taking some advice from Stijn and incorporating his selection of colour scheme into the theme I had been testing out really helped me make the right decisions. I cannot guarantee that the content of my blog posts will get any better, but it certainly will look a lot better while you’re here

I know that a couple of my Twitter and Facebook friends have approached Studio Stayne for some new logos since I released these new designs, and I cannot recommend their services strongly enough. Make sure you take the time to check out the gallery of other artworks he has provided in the past, as some of it is truly amazing. Follow them over at Facebook on their page.

More technical content coming soon!

I have a bee in my bonnet. After my last post full of love and bromance, this one is full of hate and vitriol – and I don’t apologise! We have all seen many presentations on each vendors latest and greatest “fabric” technologies over the past 18 months. It doesn’t matter which vendor, whether the presenter is sales or tech, or even enterprise or service provider focused – at some point almost every one declares that their solution is “the end of spanning tree”. It gets worse when they actively advise that you do not run spanning tree in your environment.

And I don’t buy it

The Premature Death of Spanning Tree

Spanning Tree: noun – A pox on the house of networking

Everybody loves to hate on Spanning Tree. Haters gonna hate. While we’ve all been bitten by something horrible happening related to spanning tree, I have seen many more things go wrong because people *didnt* configure Spanning Tree properly.

Vendors knew how painful it was and went to great lengths to ensure that we didn’t need to do anything so that it would “just work”. Which is great… right up until the point that you run into a limitation on the STP PixieDust Mode. Often this comes in VLAN dense environments when you max out the total number of spanning tree instances that your devices will handle. Oh thats easy – lets run out Multiple Spanning Tree!

I can hear the gasps from many people now. If people hated Spanning Tree, then they have a full lynch mob ready with pitch forks and stakes at the mere mention of poor MSTP. And they hate it for a reason. MSTP makes you think about how spanning tree works again and all the PixieDust goes away. And networks become hard again.

I will let you in on a little secret:

I actually really like MSTP and I implement it every chance I can get.

Yes its a little harder and requires a little more forethought, but I would rather do that at the design time then have to overhaul my design later to meet some new need well after my network has reached “critical mass”. I have spent many hours rebuilding spanning tree designs because I needed more than 128 instances. Sadly in more than one case I needed to work out the best way to deal with a group of Catalyst 3750 in PVST+ with 1000 VLANs configured (and 900 VLANs with STP disabled).

And things get messy, and things get hard. So lets find a new solution.

The Magical Healing Powers of Woven Unicorn Hair Fabrics

Somewhere over the rainbow, far beyond the Dark Forest of Broccoli Despair, many magical elves have worked hard to deliver us a the perfect solution to the problems I listed above. Vendors have taken this creation and moulded into their own “Fabric Solutions”. Some created skinny jeans, others an uncomfortable sweater vest. Sadly most of the time they have just presented us with a sensible pair of slacks that the sales people try to sell as a three piece suit.

A sensible pair of slacks (Unicorn Hair or otherwise), is perfectly apt when used as intended, but if you drape them over your shoulders and call them a shirt then your wrong (or a hipster. In which case your cardigan is probably over the top of your shirt-slacks).

And so it is with data centre fabrics. I agree that most of these solutions will allow us to disable spanning tree on our core/fabric facing interfaces. We will get many of the benefits of multi-path layer 2 and some times efficiencies gained by avoiding the flooding of L2 addressing information. Turning off spanning tree into the fabric core makes sense. Im happy with that.

So what about all those edge interfaces?

Do we live in a world where end users never plug two ports together?

A client PC never bridges interfaces?

How about “Oh my VoIP phone has two network ports let me just…. BOOM!”

Maybe you have no requirement to integrate with other networking infrastructure, but end stations can still do bad things and thats usually when you don’t want them to.

The Indiscriminate Killing of Canaries

So how do we go about detecting these loops? Well over the past couple of decades we’ve presented ourselves with a whole cage full of canaries that can alert us to loops or other similar problems in the network. These are our early warning signals that “Something bad just happened…” and better yet “… so let me just fix that for you!”. And sadly, many of these have been built around the functionality that Spanning Tree provides.

Let’s take the BPDU Guard feature as an example. BPDU Guard is set on an access port or another port that you do not expect to see Spanning Tree Packets (BPDUs). If a BPDU is detected, the switch will usually log a message and send the port into a blocking mode. In the scenarios listed previously the offending port is now taken out of action and the loop is removed. If we have disabled Spanning Tree on all ports then the BPDU will never be sent or received and our little bridging loop will happily continue. Well at least until your switch is a bubbling blob on the bottom of your rack.

Another feature available on most switches is the BPDU Filter. With BPDU Filter enabled on a port the switch will pass all traffic on the port but silently drop the BPDU messages. Now I agree that their are certain times when this feature is useful, such as when interconnecting with a 3rd Party that you “know” can never form a loop with you and you do not want to either learn a STP root from them or go into block due to election issues.

Sadly, our good friends at VMWare love to advocate that we implement BPDU Filter on the ports facing our VMWare Hosts. Unfortunately I have been bitten by loops coming from inside a VMWare environment due to a Microsoft Guest Bridging two vNICs in separate VLANs. A BPDU from the came in from the Physical NIC on VLAN A out to the vNIC in that VLAN and back out through vNIC and VLAN B. Thankfully when this happened, my canary (BPDU Guard) signalled that there was a problem then promptly died in its cage and disabled the port to the VMWare Host. Yes this would have some undesirable effects on all the other guests on that host, but we were alerted to the problem and needed to fix it. In the scenario with BPDU Filter these alerts would have been filtered out and the loop would continue unnoticed.

So what other methods do we have to detect possible bridging loops that do not involve Spanning Tree to be operational? I have the following list as a start to some ideas, and I am looking for others that you might know of too:

• Broadcast Storms
  • Possible Mitigation: Storm Control
• Multiple Mac Addresses on a  Port
  • Possible Mitigation: Max MAC Address restrictions
• MAC Address Flapping
  • Possible Mitigation: MAC Flap Dampening
• High CPU Usage (in some cases)

Mop and Bucket

Yes, I’ve written his post at 2am, but its been something that I have been thinking about for the past 8 or 9 months.

I can see that Spanning Tree doesn’t have an indefinite future, but calling it dead today is premature. If you are looking at fabric technologies or worse still you dont have a new fangled fabric but hate spanning tree so bad that you have just turned it off, then ask yourself how you will detect loops in the edge networks and how you will mitigate them.

Take your canaries with you and let them do their job and don’t strangle them at the top of the mine shaft.

If you do you might just find that the Emperors new cloths are just a sensible pair of slacks!

Yesterday was possibly one of the proudest days in my professional life. My very good friend Anthony Burke sent me a text message just after 9am to tell me that he had passed his JNCIA-Junos exam. This is a close tie with when he earned his place as a delegate to Network Field Day 4 back in October.

So why am I so proud about this? Well Anthony and I have become great friends over the past 18 months all through the magic of social media. One day for no reason (as is the way of things) we just started talking. Tweets became skypes and in January, even though we had only ever met face to face once, my wife and I drove the 10 hour trip to Melbourne to attend his wedding. (Sitting at table answering the question of “So how do you know Anthony?” with “Oh. From the Internet” was a humorous experience)

So how is this different to any other random person on twitter? Well very early one I realised how eager Anthony was to learn more about network and to soak up knowledge from those willing to share it. I’m not entirely sure if he just caught me on the right day (Im pretty sure I commenced our first Skype with a rant about something he never saw coming because I needed to vent at somebody). Over the course of the coming months we maintained a Skype IM dialog open most of the day discussing various networking, technology and career issues.

I spent time helping him learn new topics and sharing any possible pointers and tidbits I had that might help him with what he was working on. In return he acted as a sounding board to bounce ideas off and offering up suggestions. Over time I noticed his advice back to me started to surprise me with how much he had already learned.

In my time as a senior engineer and various management roles I have been required to assist junior staff with both their technical and their career development. I always found this to be a difficult task because I was trying to balance what I wanted out of them as co-workers with what it is they wanted to do with their own careers. I had financial incentive to push them down the path that benefitted me more than it did them. Each time they moved away from “my chosen path” there was a feeling of disappointment or failure (on my part). Even when I worked with the staff to clarify where they wanted their careers to go, I felt they were trying give “the right answer” or at least the one they thought I wanted to hear.

I have since learnt that the most personally rewarding mentoring system for me is one where I get no personal benefit from the success or failure of the other participant. Helping them make the decisions that are right for them without those choices directly impacting my day to day work life has been quite a different experience. I am better able to frame my advice to give both the benefit of my own experiences coupled with knowing where they want to go professionally.

Mop and Bucket

In a recent email conversation, reference was made to the the gutting of the middle ranks of IT. We have a lot of junior staff and a lot of highly skilled engineers, but we are starting to see a thin mid-section (something my mirror cannot attest to). Whether this is due to the focus on specialisation vs generalist IT or not is uncertain to me, but I have seen evidence to suggest this accurate (or becoming so).

So how do we move junior engineers onto the path of becoming journeymen and future experts? I suggest that each of you should keep your eyes open to “up and comers”. The ones who “have it”. You will be surprised when and where you find them, but they are everywhere. Take the time to help them out when possible. Answer the questions that they ask. Take time to get to know their strengths, their weaknesses and their career aspirations. Put them in contact with your contacts and help build their network of resources. Everyone “knows a guy who knows that thing”. Pass it on. Our contacts are not just secret tools to make us look good.

Once they become a journeyman engineer, encourage them to also look for people to mentor and repeat the process. I have found this process to be very rewarding for me personally, and I plan to continue doing it. Having a mentor (or several) above you and below you can do a lot to encourage and maintain your own growth.

When I first started talking with Anthony, he had passed his CCNA and started studying for his CCNP. Since then he has continued on with increasing his Route, Switch and Security skills. He fell in love with the ASA (urgh) and despite much encouragement I couldnt talk him around to playing with Junos. In late October he finally got his hands on his own SRX110 and started using Junos for the first time. I just over a month he has learnt more than I learnt in my first 6 months playing with Juniper kit. He passed his JNCIA-Junos yesterday, and apparently preparing for his JNCIS-SEC.

Taking on new technologies and really understanding them so quickly? This is why I am proud.

If this sounds like a bit of a love letter to a friend then so be it

And Burkey…. #FHP

I don’t know about you, but I am regularly told “You worry too much” or “You don’t need to worry about that”. Sometimes its “What are the chances of that ever happening?”. These are things that Ive heard from many people over the years and the best I can come up with is:

“That’s what you pay me for! Im here to think of the worst case scenario and then mitigate against that.”.

This is usually followed by confused looks from those around me who do not seem to grasp what I am getting at here.

The way I see it, it is my job to constantly be thinking about worst case scenarios.

• “What happens if we lose this device/site/cable?”
• “What happens if we all backups are lost?”
• “What if…”

These sorts of questions are exactly the reason why I fell in love with Networking as a discipline within IT. The very fact that I have the ability to build redundant systems that take serious effort to bring down draws me deeper in (assuming Im given appropriate budget ),

Why do I build many of my networks like a Service Provider network? Because I have found that these basic design principles are usually the most robust. Configurations utilising OSPF to carry core routing information and BGP to provide end user routes stands up to some serious beating – and it is extensible too!

Why do I cry when I hear vendors pronouncing “With our new Wonder Fabric Technology you can now turn of Spanning Tree”? I cry because I feel that this is sending the wrong message. I have a whole other post coming on that topic, but please people don’t just turn off spanning tree. Are all your edge ports protected? Can you ensure that nobody will ever mis-cable? (And don’t even get me started on VMWare’s view about filtering BPDUs!)

Why do I prefer two stand alone systems providing redundant network services over a single HA unit? Devices redundant power, RE’s and line cards, but with a shared management plane are still susceptible to risk of incorrect configuration causing a service interruption. Switch Stacks, Virtual-Chassis, VSS and what ever other similar technology all suffer from this problem. I would rather a technology such as Multi-Chassis Link Aggregation, Virtual Port Channels, or even utilising VRRP/HSRP or anycasted Services to provide the desired network redundancy. Sometimes this is “harder”, but again – This is what you pay me for

Mop and bucket

While I know that I am pre-disposed to the negative and pessimistic tendencies and views, but am I the only one who feels that “Worst Case Scenario Thinking” is one of the prime reasons people pay us? My wife could easily plug in a couple of cables and “make intarwebz happen”, as is proven by the millions of home users CPE, but true network design and redundancy comes from thinking about the worst that can happen and how to mitigate against these risks.

Im curious as to the thoughts of those of you out there.

As many of you know, I had the honour and good fortune of being able to take part in beta testing the JNCIE-ENT lab in August of 2011. The day completely wiped me out and I was walking around in a daze for the remainder of that week. While I knew the technologies I was unprepared for the time-management skills required to pass this lab. Needless to say it was no surprise when I received my fail-mail advising me that I was unsuccessful.

Well after almost a year, I decided it was time to get back on the horse and try again for getting those digits. Here in Australia the Juniper lab exams are only offered every 3 months, and I felt I was unprepared to sit the exam when it was offered in January. I knew that a friend of mine, and now colleague, Cooper Lees was preparing to take his JNCIE-SEC exam during the May schedule so I decided to book my attempt for the same day.

I have spent a lot more time working with Junos and various Juniper hardware devices in the time since I took the beta exam in August, and I was now a lot faster on the CLI, and didn’t need to refer to any documentation during the exam (except verify some config when I thought I had completed each step correctly! Needless to say… I hadn’t).

I had the good fortune at both my previous job at eintellego and also my current position at ICT Networks to have a wide variety of Juniper equipment at my disposal to create a lab environment in which to hone my skills required to get through the exam. I made quite extensive use of these labs to prepare and its true that nothing beats hands on experience when it comes to the lab. The JNCIE-ENT exam is based on EX4200 switches and SRX240′s – so be prepared to configure and support anything on these devices (sometimes items on “the edge of the blueprint”!).

So the lead up to the lab date was “exciting” with Cooper and I giving each other some “friendly rivalry” to try and be ready for the exams we had in front of us. We swapped experiences and ideas back and forth. Even though we were sitting different exams, having somebody else work with you on problems during prep time was certainly a god send. I had this during my prep for the beta with Nick Ryce and Chris Jones, both of whom have now gained their JNCIE-ENT certification.

On lab day Cooper and I met around 8am for breakfast near our office (which is only a few blocks from Juniper’s Sydney Office). A hearty breakfast of Bacon, Eggs, Grilled Tomato and Coffee certainly helped calm my stomach. After some banter about how we just wanted to “get this thing started!”, we decided to head over to the lab location and wait. We got there a little early, but it gave us time to settle down. The foyer area outside the room that served as the lab location included a table tennis table, an xbox and a pinball machine. Cooper showed off his fine table tennis skills while the rest of us pretended like we weren’t stressed

While talking to the other candidates, we worked out that there were 3 attempting JNCIE-SEC, 3 attempting JNCIE-SP, and just myself attempting JNCIE-ENT (The “easy one” apparently ). All of the candidates had a wealth of experience behind them including front line engineers, instructors and consultants. This was great company to be a part of.

Well the next part of the day is all covered by NDA, but I can tell you that I was wiped out by the end of the day. I feel like I was much better prepared this time around, though I am not sure if I scored enough points to earn a pass. I have already gone over some of the “stumpers” from the exam and worked my way through various solutions.

Im reasonably sure I am going to have to give this lab another attempt, but there is not much I can do about that now. This is all in the hands of the Juniper Certifications Team now. There is a 21 day SLA on the turn around of lab results, so all I can do is wait and prepare to book my next attempt.

I would like to leave the following advice to anybody preparing to take the JNCIE lab in the future:

• Prepare for time management. There is a lot crammed into the exam
• Read the entire exam. There are a lot of steps throughout the paper that can be consolidated and completed at the same time. I suggest making a list of all questions that affect each device and trying devise a strategy on how to meet all requirements before diving in.
• Know all of your topics. The lab exam is laid out quite clearly by topic, and these topics align with the same major headings as the blueprint. You are required to successfully complete at least one task from each section to pass. This means that you cannot say “Oh, I am weak in CoS or Multicast” and think you can make up your points elsewhere. Not completing a task from each section is an instant fail. BE PREPARED AND KNOW YOUR STUFF!
• My workstation during the lab was a Windows notebook computer with an external keyboard and mouse. You are allowed to bring your own keyboard and mouse if you wish. Unfortunately this was not what was tripping me up. You see, I am a Mac user and I had to get used to the fact that Ctrl is used instead of the Command key – I spent a lot of time pressing ALT This will probably not be a problem for most people, but I will invest more time in lambing on a windows computer so I get the hand of where the keys are This may change in the future, but this was what I experienced.
• The lab is accessed via a VPN + Remote Desktop and a console server. All of this was up and running before I sat down at the lab machine. There was a little bit of lag for keystrokes, but nothing outside of what I am used to working on customer equipment in remote locations. Due to the time constraints in the lab though, you will want to be well versed in the Junos CLI short cuts (including Ctrl+W, Esc+b, Esc+f, Ctrl+A and Ctrl+E at the very least). These should help you move around the CLI without waiting for your console to catch up.
• Unlike other vendors, then JNCIE lab does include external machines that operate to show if your configuration is working. Be sure you know how to diagnose correct operation of your protocols and features from the blueprint  because it will be of great benefit with these devices behaving correctly.
• And last of all Juniper have made the sensible decision that IPv4 and IPv6 are equal. (Except of course where EX Licenses are concerned!). Know how to configure most of the tasks on each protocol – because you never know where you might get tested! Remember, this is the future of networking whether you like it or not, so get your IPv6 on!

As mentioned, the rest is just a waiting game now, and I promise I will post an update regardless of pass or fail! Until then, Im back to labbing some of the scenarios I came up against that I thought “should have worked!”.

POST: Well this blog post was supposed to be published over a week ago, unfortunately I have received my “Fail Mail” in the last couple of days, so I will be preparing to take this lab once again in August This isn’t the worst news Ive received over the past week, as one of my best friends died in an accident at the beginning of last week and the “Godparent Card” has been activated. This is why my post was delayed somewhat as I have been working through assisting his wife and 2 young boys (both under 3) to deal with everything that is happening.