Bit Pilot

Splunk makes two key acquisitions

2018-06-21T07:42:00.002-07:00

Now Splunk data and insights can be acted upon more quickly along with automated remediation and forensics.

https://techcrunch.com/2018/06/11/splunk-nabs-on-call-management-startup-victorops-for-120-m/
"With VictorOps, the company gets a system to alert the operations team when something from that muddle of data actually requires their attention."

https://seekingalpha.com/article/4151274-splunk-acquire-phantom-cyber-350-million
"to bolster its data integration, security analysis and automation capabilities as it continues to build out its machine learning offerings."

The enemy of great

2018-06-21T07:38:00.000-07:00

In the space between GOOD and PERFECT lies somewhere you occassionally want to be: GREAT. Very few things need to be perfect. For many things we settle for good, and probably should. We need a precious few to get GREAT.

No more
"Meaning: When deciding whether to commit to something, if I feel anything less than, “Wow! That would be amazing! Absolutely! Hell yeah!” - then my answer is no." - Derek Sivers

BUT MOST OF THE TIME:

"Better a diamond with a flaw than a pebble without." -Confucius via Entrepreneur

...but "Obsession with perfection can paralyze" - Psychology Today

THEREFORE:
“Don't put off until tomorrow what you can do today.” ― Benjamin Franklin

There are times when perfection is called for, of course, but allow me to suggest to you that most of the time, “good enough” will do. There’s a point where it takes more and more energy to achieve smaller and smaller gains. via LifeHack.org

Network Device API intro "Postman"

2015-08-20T15:07:00.003-07:00

Playing around with the basics on network device API calls again. Postman is a great tool to try things out interactively. Works great for me as a Chrome plugin.

Kudos to Matt Oswalt for a great post on Nexus 9000 NX-API

Finally, a little bit of super generic Python for getting 'show version' from a switch:

import requests

import json

url='http://YOURIP/ins'

switchuser='USERID'

switchpassword='PASSWORD'

myheaders={'content-type':'application/json-rpc'}

payload=[

{

"jsonrpc": "2.0",

"method": "cli",

"params": {

"cmd": "show version",

"version": 1

"id": 1

}

]

response = requests.post(url,data=json.dumps(payload), headers=myheaders,auth=(switchuser,switchpassword)).json()

DevOps for Network Engineers

2015-06-25T12:11:00.001-07:00

Network Engineers need to not only think about network programmability and automation. Take a higher look into reducing operations and moves/adds/changes into the smallest incremental steps first. Here are some great references to give you some thoughts on this.

What is DevOps in Simple English – Rackspace

Ivan Pepelnjak @ioshints describes how "Infrastructure as Code" makes sense

@dave_tucker gives a nice "NetOps to DevOps" plan

Bimodal IT – Lydia Leong of Gartner

BONUS: Familiarize yourself with revision control. @DavidJohnGee describes Git

Larger, quicker, mobile Networking as in IoT

2015-03-11T13:31:00.000-07:00

I was wondering how thousands of new, tiny, mobile devices (such as sensor networks and the Internet of Things) might be assisted by IPv6. Think about the obvious needs introduced:

1) Many Addresses - estimates range from 10's to 100's for every person

2) Low Memory - sensors will be small and somewhat cheap

3) Mobile - where are they going to be and will they be needed elsewhere tomorrow?

4) Low-touch - by sheer numbers managing one-by-one is an impossibility

The current Internet Protocol dominant on the Internet (IPv4) isn't so upstanding for this. The newer version (IPv6) is looking much more handy! Found this article from a European Research project looking into just that.

www.iot6.eu

Check out all their material

IoT Links Feb-2015

2015-03-03T05:30:00.000-08:00

This student paper gives a pretty decent lay of the land.
https://sites.google.com/a/cortland.edu/the-internet-of-things/summary

Cisco's 2011 whitepaper framed things up nicely and is still very relevant.
www.cisco.com/web/.../IoT_IBSG_0411FINAL.pdf

Ecobee, Lyric, and Nest...oh my!
http://smartthermostatguide.com/

Big Data, analytics, and finding value in Internet of things.
http://www.networkworld.com/article/2889975/careers/future-proof-your-it-career-8-tech-areas-that-will-still-be-hot-in-2020.html

Sensor technology will be all the difference for IoT
http://www.forbes.com/sites/tjmccue/2015/02/24/a-butterfly-wing-inspired-sensor-may-save-your-life/

Always wanted my cell phone mic right inside my mouth!
https://wtvox.com/2014/10/top-10-implantable-wearables-soon-body/

Regional leadership in IPv6

2012-10-04T10:23:00.001-07:00

Network World claims "The US is overtaking the world on IPv6" , but I'm not entirely convinced.

Seems like when the numbers can support the claim such as overall traffic, North America vs. all of Asia numbers are shown. When it's a bit closer, such as in number of addresses or vendor support the numbers are broken down by country to show the US is slightly ahead of China but not all of Asia.

It's nice to see that North America is no longer ignoring the protocol and the future of the Internet. Now let's see how we are doing in percentage of bandwidth and percentage of total users!

SSL and TLS - What do browsers use to encrypt?

2012-09-20T09:23:00.000-07:00

I'll spare you all the gory details you can read on wikipedia, but these are protocols used to encrypt data exchanged by browsers and web servers for keeping information private and unchanged. Secure Sockets Layer (SSL) has been through versions 1, 2, and 3. All by 1996 ! Transport Layer Security (TLS) was released as version 1.0 in 1999. (Also a !).

You may often see references to SSL3/TLS1.0 because "TLS 1.0 does include a means by which a TLS implementation can downgrade the connection to SSL 3.0, thus weakening security."

TLS1.1 was defined in 2006. TLS1.2 was defined in 2008. As of 2012, IE9 appears to be the only major browser to support it.

In his great BlackHat 2010 presentation, Ivan Ristic presents great statistics on server support for these protocols discovered during surveying:

nping, a ping for more protocols

2012-08-13T11:24:00.001-07:00

Nping is really handy when you suspect that ping or traceroute aren't giving you clear results. I really like it for a TCP Connect to a website:

$ nping --tcp-connect -c 3 --delay 2s www.google.com

Starting Nping 0.6.01 ( http://nmap.org/nping ) at 2012-08-13 12:09 MDT
SENT (0.0067s) Starting TCP Handshake > www.google.com:80 (74.125.225.176:80)
RECV (0.0209s) Handshake with www.google.com:80 (74.125.225.176:80) completed
SENT (2.0086s) Starting TCP Handshake > www.google.com:80 (74.125.225.176:80)
RECV (2.0250s) Handshake with www.google.com:80 (74.125.225.176:80) completed
SENT (4.0113s) Starting TCP Handshake > www.google.com:80 (74.125.225.176:80)
RECV (4.0244s) Handshake with www.google.com:80 (74.125.225.176:80) completed

Max rtt: 16.390ms | Min rtt: 13.108ms | Avg rtt: 14.507ms
TCP connection attempts: 3 | Successful connections: 3 | Failed: 0 (0.00%)
Tx time: 4.00581s | Tx bytes/s: 59.91 | Tx pkts/s: 0.75
Rx time: 4.01892s | Rx bytes/s: 29.86 | Rx pkts/s: 0.75
Nping done: 1 IP address pinged in 4.02 seconds

You can also scan MAC addresses on your local subnet:

$ sudo nping --arp-type ARP 172.16.16.1-10 --count 1

SENT (4.8530s) ARP who has 172.16.16.1? Tell 172.16.16.11

RCVD (4.8546s) ARP reply 172.16.16.1 is at 1C:DF:DF:53:91:91

SENT (5.8538s) ARP who has 172.16.16.2? Tell 172.16.16.11

RCVD (5.8548s) ARP reply 172.16.16.2 is at 00:26:9F:37:33:33

SENT (6.8556s) ARP who has 172.16.16.3? Tell 172.16.16.11

RCVD (6.9364s) ARP reply 172.16.16.3 is at A4:EF:57:E3:EA:EA

SENT (7.8575s) ARP who has 172.16.16.4? Tell 172.16.16.11

SENT (8.8582s) ARP who has 172.16.16.5? Tell 172.16.16.11

SENT (9.8587s) ARP who has 172.16.16.6? Tell 172.16.16.11

SENT (10.8600s) ARP who has 172.16.16.7? Tell 172.16.16.11

SENT (11.8612s) ARP who has 172.16.16.8? Tell 172.16.16.11

SENT (12.8621s) ARP who has 172.16.16.9? Tell 172.16.16.11

RCVD (12.8656s) ARP reply 172.16.16.9 is at 00:21:CF:BD:1F:1F

SENT (13.8634s) ARP who has 172.16.16.10? Tell 172.16.16.11

Integrating routing with CDN

2012-07-13T08:15:00.000-07:00

A content delivery network (CDN) is a large distributed system of servers deployed in multiple data centers in the Internet.[citation needed] The goal of a CDN is to serve content to end-users with high availability and high performance.

Uniform Resource Identifier (URI) is a string of characters used to identify a name or a resource on the Internet. - en.wikipedia.org/wiki/URI

Border Gateway Protocol (BGP) is the protocol which is the core routing decisions on the Internet. It maintains a table of IP networks or 'prefixes' which designate network reach-ability - en.wikipedia.org/wiki/BGP

The multiprotocol extensions of BGP has been used to carry many types of information. I stumbled across a great progression on the idea from this NANOG presentation from the guys at Comcast.

Here's their packet capture screenshot. Notice the Address Family - URI.

IPv1 , v2, and v3

2012-06-22T11:25:00.001-07:00

I always hear "What happened to IP version 5?" But we've seen the answer many times. "Some experimental testing version".

While reading Charles Kozierok's excellent TCP/IP Guide, I found the history of why IPv4 was actually the first version of IP as in a Layer 3 on it's own. The addressing layer was part of the original Network Control Program and then Transmission Control Program combined into a single Layer3 and 4 functionality. TCP became Transmission Control Protocol by its 'fourth iteration'.

Why hasn't networking followed Moore's Law ?

2012-06-19T08:46:00.001-07:00

Andy Bechtolsheim's keynote at NANOG55 in Vancouver looks into what's been going on with networking versus Moore's Law:

Follow the link to his slides in PDF. There are some fantastic graphs on:

Server 10/40/100G Adoption Cycle
Total Datacenter Switch Revenue by Protocol & Speed

Today's custom switch silicon support 64 10G ports on a single chip but forecast to scale by 4X in the next 3 years. He wraps up the presentation looking into ways to alleviate the current high cost of optics slowing down 40G/100G adoption.

IPv6 Day

2012-06-06T09:55:00.001-07:00

Early today, major web sites enabled their DNS 'quad A' or AAAA entries.

These include google.com, bing.com, facebook.com, yahoo.com, youtube.com and aol.com.

I especially like Facebook's IPv6 address:

2A03:2880:2110:3F03:FACE:B00C::

It looks like traffic to Google over one backbone immediately doubled... Albeit up to almost 2% of total traffic.

I'm told that unlike last year, these updates are planned to be permanent. Sure is a step in the right direction!

A Brief History of SDN

2012-05-31T09:38:00.005-07:00

Software Defined Networking is a new and exciting development in networking. However, it's been a long time coming:

1985 IGRP planned to take into count metrics beyond hop count to include reliability and load

1991 OSPF specified calculating paths to deliver better throughput, lower latency, etc. John Moy's excellent book remains one of my very favorites.

1996 Cisco Policy Based Routing was introduced with IOS 11.0 and "provides a flexible mechanism for network administrators to customize the operation of the routing table and the flow of traffic within their networks."

2006 Cisco OER became PfR sometime around 2010. It allows the traffic forwarding decision to be based on delay, packet loss, and link loading. Arguably the closest precursor to SDN (at least from Cisco) in that it has a 'separation of control plane', 'a centralized controller' and possibly 'programmability by external applications.

Please send me other examples!

Slow TCP but no packet loss

2012-05-24T11:04:00.000-07:00

Many times I've heard the claim "out-of-order packets" wreak havoc with your application. Many interpretations conclude that must be because of poor application coding. Blaming others can flow both ways, it seems. My favorite comment in all threads about Duplicate ACKs and Fast Retransmit concludes with " how the network performance is improved by a server reboot. "

Back to our main concern here. When a receiver gets TCP datagrams 1,2,3 and 5 it will immediately send an ACK expecting 4 as it already did to confirm receipt of datagram 3. Ding. First 'duplicate ACK'.

Now, listen closely as I don't plan to get in this habit. The RFC for TCP Congestion Control is a moderately pain-free read and gives the authoritative fundamentals on how TCP should behave. Give it a look!

The key to dig out is somewhere around page 6. Duplicate ACKs cause sender to reduce window size. The whole mechanism and scenarios for SLOWING down transmission rates of the TCP flow are detailed.

MORAL: nothing has to be lost and TCP can still determine congestion has been encountered and slow down. Sometimes significantly. Also came across these great course slides.

Two missing but vital Cisco Nexus vPC commands

2012-05-17T09:44:00.003-07:00

So you come across a pair of Nexus 5548 switches and dive right into setting up a Virtual PortChannel following the configuration guides. You'll wind up with something like this:

Things go swimmingly! That is until you want to upgrade or have to power cycle one of the switches. Your server can't get anywhere nor be reached for quite a few seconds. Odd, you think. Surely frames can happily flow along the remaining active link. Isn't that what vPC is for? As one switch comes back online, you then get ANOTHER period of lost connection to your server.

The following two commands come to the rescue.

auto-recovery
delay restore [sec]

The first allows a switch to either takeover as primary or re-enable a vpc during loss of keepalive. A great description of the history behind this is posted by Chris Marget.

The second helps prevent the vpc disruption when a vpc member switch comes back online. The delay gives time to join the vpc domain and learn information before trying to forward frames.

conf t
vpc domain 1
auto-recovery
delay restore 120

Have at it! Some great Cisco operations guides are here and here.

TCP Throughput across the Internet

2012-05-08T13:04:00.002-07:00

After reading Brad Hedlund's excellent overview and digging into the great tools at SWITCH, I ran some quick numbers for what transfers might see across the Internet.

First, taking low packet loss of 0.01% and checking across various delays from 5 to 60ms, I get:

Then, using 60 ms as a rather normal round-trip time (RTT) across the Internet, I tried values for packet loss increasing to a very normal 0.1% :

Looks like it's pretty hard to fill that 25Mbps DSL. In the future, I will look at tuning options for improving that. Comments welcome!

Response Time Composition

2012-05-04T10:20:00.000-07:00

Waaaay back in 2006, I used a network monitoring appliance from Network Physics whom I'd lost touch with. It delivered this graph that quickly became my best friend:

One glance at the Response Time Composition and I could see what had changed from a baseline. I could also see where to dig in further as to the cause of reported 'slowness'. Of course it was never the network and quite often the server or sheer massiveness of application chattiness.

I'm pleased to say my old friend lives on and has been seriously enhanced by the folks at OPNET in their AppResponse Xpert tool. Even more super-cool is the SaaS edition that fires up right in your #cloud -running application via a sweet bit of JavaScript.

10Gig options within the rack

2012-04-27T13:46:00.000-07:00

This is a roundup of my preliminary look into the current state of 10 Gigabit Ethernet cabling within a rack. While 10GBASE-SR was the common compatibility choice, it's also one of the more expensive. I've seen it commonly supported in devices ranging from firewalls to storage.

While a bit dated now, Lisa Huff gives a great analysis of the copper lay of the land in 10GBASE-T vs 10GBASE-CR :

"Both SFP+ DAC and 10GBASE-T products will be needed in the long term – 10GBASE-T for inexpensive connections and 10GBASE-CR (SFP+ DAC) for lower latency and lower power consumption."

Howard Marks gives a great update on what's happening with the previous power-hog champion, originally weighing in at 8W PER PORT :
All I Want For Christmas Is 10Gbase-T - Network Computing

..which is partially in response to the original (and convincing) attack on 10GBASE-T by the illustrious Mr. Ferro

DIY Router or Switch Management port

2012-04-23T13:19:00.001-07:00

Say you've become very fond of that dedicated management port on your Cisco Nexus switch. Now how to connect your Catalyst 6500 into that out-of-band management network?

Local policy to the rescue! This works like a champ and is not to be confused with Policy-based Routing. (Although PBR does have a great ring to it!).

A local policy is triggered by packets involving the device itself. Not traffic being routed through. Here's a sample:

ip local policy route-map local-mgmt
!
ip access-list extended mgmt
permit icmp host 10.2.3.4 10.161.161.0 0.0.0.255
!
route-map local-mgmt permit 10
match ip address mgmt
set ip next-hop 10.2.161.1
!
end

Where 10.2.3.4 is a loopback or other management-related interface on your switch and 10.2.161.1 is the next hop router that gets you to your management networks. Of course you could just alternately make your OOB management a single, large, flat subnet. Connected routes for the win!

For more information, Petr Lapukhov has several other examples, his fourth coinciding with mine above.

Brief Netflix analysis

2012-04-18T14:01:00.001-07:00

Firing up Netflix in a browser, I decided to see what could be found. Wireshark was my main cohort in this exercise. Selecting 'Statistics...Conversations' and then sorting by Bytes gave me these:

Back in the main capture, I looked at what order these remote IP's were referred and then tried a reverse DNS lookup:

>nslookup 50.19.81.73
Name: ec2-50-19-81-73.compute-1.amazonaws.com
Address: 50.19.81.73

>nslookup 65.126.84.9
Name: a65-126-84-9.deploy.akamaitechnologies.com
Address: 65.126.84.9

>nslookup 65.126.84.11
Name: a65-126-84-11.deploy.akamaitechnologies.com
Address: 65.126.84.11

>nslookup 65.126.84.18
Name: a65-126-84-18.deploy.akamaitechnologies.com
Address: 65.126.84.18

So it seems the main 'logic' of the website such as account login, pulling up my favorites, and searching for movies is hosted on Amazon. My guess is that the movie poster images are hosted on Akamai. The HTTP content of that field confirms:

GET /en_us/boxshots/166/60020865.jpg HTTP/1.1\r\n

Host: cdn-5.nflximg.com\r\n

Keep-Alive: 115\r\n

Connection: keep-alive\r\n

Referer: http://movies.netflix.com/WiHome\r\n

Finally, I pulled out the TCP stream for the top flow based on Bytes.

A 'whois' on the IP reveals it is from Level3:

NetRange: 8.0.0.0 - 8.255.255.255

CIDR: 8.0.0.0/8
NetType: Direct Allocation
OrgName: Level 3 Communications, Inc.

The Wireshark decode shows an HTTP/1.1 stream (Content-Type: application/octet-stream\r\n , Server: Level-3 Origin Storage/1.5\r\n)

A little bit of packet capture tells us they use Amazon Web Services for the front-end and account logic, Akamai for static content, and Level3 Storage for the media streams. Some to-do items: check browser data for the application type that plays the media and look for TCP header data to learn more about flow control and media streaming.

North American IPv6 Summit

2012-04-16T10:02:00.000-07:00

Last week, Denver hosted the 2012 North American IPv6 Summit.

The keynote by John Curran of ARIN kicked things off nicely. My favorite points were:

A few years ago, NAT was sounding the death knell for IPv6 adoption.
Today, media providers are hindered by NAT. (Think tracking and targeting...) Follow the money people - IPv6 is gaining support.

Other interesting tidbits overheard:

Enable a client for IPv6 and see content come in that way as much as 10% of the time.
Enable a web server for IPv6 and only get <1% traffic over it.
Over 30% of DNS domains are IPv6-enabled largely due to efforts of Godaddy and other leading registrars.

I learned from Oracle's Paul Zawacky that 'dual stack' is becoming the transition method of choice. Owen Delong from Hurricane Electric advised to stick with a /64 per VLAN and stay away from "IPv4 thinking" when considering alternate subnet masks. Finally, NAT64 is a very interesting method for moving all hosts to a pure-IPv6 environment and maintaining v4 Internet access.

@scotthogg organized and summarized the event quite nicely. Thanks!