I just returned home after spending almost a week in London attendingCisco Live. Much can be said about the event and many has already summarized their experience, so the plan for this blog post is to make a short resumé of the sessions I attended to. Many were great, most were good but a few were less than good. I skip the latter here and focus on the best pieces.

TECSEC-3030 – Advanced Network Access Control with ISE

This was a techtorial, which means that the session covers a full 9-hour day of presentation. Relative to most people I have a lot if experience with Cisco ISE. I have attended a 5-day pre-ATP class for Cisco ISE and done a handfull of implementations. Nevertheless, this techtorial was really relevant for me since I got a lot of repetitions of theories and behind-the-scenes that is easy to forget about in the daily work. Also, since the speakers have so much in-depth knowledge of this new product is gives alot to hear what they say (and what the do not say). I am totally convinced that Cisco has raised the old 802.1x-horse to a new level by combining products like ISE and Anyconnect with the new concept of TrustSec to allow the right device access to the right parts of the network, not only defined by what user is using the device but also based on how it is connected (wired/wireless/vpn) and what kind of device is it (comporate computer/private Ipad/mobile phone). This rocks!

BRKNMS3134 – Advanced NetFlow

Netflow is a protocol to gather information about network traffic for further analyzis. Instead of analyzing the traffic-flow inline, netflow-enabled devices (routers) collects information about which devices that “talks” to who, amount of traffic and ports. This information is sent to Netflow collectors for analyzis.

I am not a netflow-guy. I have only tried it a few times. But this Live-session was really cool. With Flexible Netflow the sky is the limit when it comes to which kind of information to select and where to send it. Netflow v9 is the key!

BRKSEC-2071 – Securing DNS

The fact that there are vurnerabilities in the DNS-procotol is nothing new. And it has been known for a while now that DNSSEC is the solution to most security-related DNS-issues. The session contained a live demo of DNS-cache poisoning a´la Kaminsky and thereafter a complete walkthru of actions to prevent this from happening.

The speakor Stenthor Bjarnason really showed in-depth knowledge in the subject!

BRKSEC-3005 – Advanced IEEE 802.1x for wired networks

This session was an advanced session that discussed all aspects of 802.1x-implementations in wired networks. It went through the concept of authentication and authorization, radius-attributes for downloadable access-lists and vlan-changes and discussed aspects on how to handle non-802.1x-enabled devices with MAB. Further on there was a lot of information about how to troubleshoot dot1x and how to handle PKI in dot1x-implementations.  The speaker was awesome!

BRKIPM-2999 – LISP – A Next Generation Networking Architecture

I am not a thru router guy. At least not compared to my fellow r/s-friends who eat MPLS for breakfast. So attending a LISP session was really a step out of my comfort zone. But it was so cool! It is not easy to explain in a few centences what LISP is. In short you can say that LISP is a new way to rout ip packets, not only based on destination but on other parameters aswell. And since these parameters are stored in central units, you can say that LISP uses something similar to DNS to query how to route traffic. And in a sence you can also say that LISP is a way to tunnel traffic. This will be big!

BRKSEC-2046 – Deploying Security Group Tags

SGT is one of the building blocks that builds the foundation of Cisco TrustSec. In short: SGT is a way to tag packets ingress in access layer devices so that they can be filtered egress centrally. The reason to do this as a complement to a firewall is to gain speed. Also, in 802.1x-enabled networks the access-switch has a lot of knowledge about the traffic (type of device, authentication information…) which means that traffic can be tagged (and further filtered) not only based on ip-information but also based on username/device type/connection type/<insert almost anything here>.

BRKSEC-3033 – Advanced Anyconnect Deployment and Troubleshooting with ASA 5500

This was my favourite. It is always a pleasure to listen to Håkan Nohre, imho one of the greatest brains when it comes to Cisco ASA and Cisco-based security solutions. I know this subject by heart so I cannot say that there was a lot of news for me. But it it really cool to listen to how inspired and excited Håkan is when he talk about what he loves most.

Roundup

I must say that all sessions on Cisco Live keeps a very high quality when it comes to content and how it is presented. I have left out a few of the ones I attended here and the reason for not all sessions being perfect for me was not the content but more that I choose the wrong sessions. So Cisco Live, keep up the good work. And a final note: The PDF material provided after attending Cisco Live is by far the most comprehensive and good technical reference material one can find. Even if I might not attend IRL on Cisco Live next year I will definately pay for the virtual pass, making me able to access the presentations online!

I recently had a TAC-case regarding a Cisco ASA 5510-firewall with Anyconnect-clients which had issues with VPN-clients not being able to connect due to “no address available”. It turned out that the “show vpn-sessiondb anyconnect”-command showed 50+ anyconnect-sessions that were over one month old! Like this:

sh vpn-sessiondb anyconnect
Session Type: AnyConnect
Username : aaaaa Index : 110
Assigned IP : zx.zx.zx.zx Public IP : qw.qw.qw.qw
Protocol : Clientless DTLS-Tunnel
License : AnyConnect Essentials
Encryption : RC4 AES128 Hashing : SHA1
Bytes Tx : 40577016 Bytes Rx : 5480886
Group Policy : DfltGrpPolicy Tunnel Group : DefaultWEBVPNGroup
Login Time : 10:43:24 CEST Fri Dec 16 2011
Duration : 34d 23h:20m:15s
Inactivity : 32d 2h:00m:04s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none

Username : zzzzz Index : 152
Assigned IP : x.x.x.x Public IP : y.y.y.y
Protocol : AnyConnect-Parent DTLS-Tunnel
License : AnyConnect Essentials
Encryption : AES128 Hashing : none SHA1
Bytes Tx : 13671510 Bytes Rx : 8421169
Group Policy : DfltGrpPolicy Tunnel Group : DefaultWEBVPNGroup
Login Time : 04:39:57 CEST Tue Dec 20 2011
Duration : 31d 5h:23m:42s
Inactivity : 31d 4h:14m:45s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none
...
...
...

The strange thing about this was that there was indeed an idle-timeout configured for DfltGrpPolicy:

group-policy DfltGrpPolicy attributes
vpn-idle-timeout 60

The solution provided was to add ssl keepalives on the group-policy. And that had the desired effect. After adding the commands below, there were no more stale sessions:

group-policy DfltGrpPolicy attributes
webvpn
anyconnect ssl keepalive 300

Strange thing though. The idle-timeout should be enough to kill those sessions. I still havent got any explanation from TAC regarding why the ssl keepalive-command was needed. Anyone?

I have recently implemented a few Cisco Ironport WSA-solutions. When doing a follow-up after the implementation, the customer usually reacts with “Oh… WSA? We forgot about that. It probably works…”

But what difference does it make? If the customer forgets about their web proxy, what good does it do? Lets have a look at an implementation…

I asked one of our customers for permission to peek into their WSA for the purpose of this blog post. This customer has a few hundred users and is a fairly traditional type of user with mostly office users, each with a personal computer. This customer doesnt limit web browsing, except for filtering out access to known obviously bad web categories like child porn. Except for that, free access to the Web.

Fig1: General Statistics

The first thing to look at is an overview of web activity above. The average web traffic an business day is roughly one million is a working day consists of one million web requests. A web page contains several objects (images, scripts) where each object needs to be requested individually. In this implementation the clients generates 1 million transactions (requests) per day, or 20 million transactions per month.

But what is the content of the requested material? If we look at But WHAT users to surf? If you then look at the purity of operations as it starts to get interesting for real!

Fig2: Purity

Here you can see that just over 10,000 (10.6K) transactions have been stopped this month because of URL category! That is, such as child porn! There are objects (pages, images, etc.) that the user consciously or unconsciously sought but that the system has already been blocked at the access-trial because the source is known and undesirable.

One can also see that almost 3,000 (2.797) object has been blocked due to malware detection. Remember that the WSA scans all through traffic for known viruses, scripts, or other type of malware. The source category has been approved or unknown the WSA have downloaded content. But when checking the contents, they have discovered something unwanted. This little fella has thus stopped nearly 3,000 viruses in the past month!

Overall, 99.8% of web traffic this month has been “clean”. 0.2% may seem to be disappearing bit, but it is still almost 34 000 (33.8K) potentionella threat that was blocked already at the front door!

If you want more detailed information about the type of threat blocked, you can obviously get it also:

Fig3: Malware

With the help of the dynamic Sender Base system scored all websites on the internet. Based on a number of factors such as known virus outbreak or the credibility of a domain, each site a web reputation score from -10 to +10. WSA is configured to always block the sources with the lowest score and always allow the web site with the highest score. But how does this when in reality?

Fig4: Web Reputation

Here we can see that nearly 10,000 transactions in the last month blocked because of Web reputation.

The conclusion I draw every time I look at this type of reporting is that the WSA is blocking lots of web traffic in the covert, and it’s surprisingly rare that users react to the IT department because they can not browse to a specific site. It may be that the user deliberately tries to make stupid mistakes on the internet, but my experience and absolute conviction is that it almost always is something that happens unconsciously. A link to an email or on facebook that look “nice”, but takes the user to a  malware site in some obscure corner of the Internet.

Key figures for this particular device, a typical month “at work”:

• The number of transactions: 20.4 million pieces.
• The number of blocked transactions: 33 800 pcs.
• The number of blocked Malwares / viruses: 2797 pcs, or one every 3 minutes during business hours!
• Dare you not to check the content of your web traffic?

The problem

have you ever had an open TAC case with Cisco, just waiting for them to provide either a solution or some other kind of feedback, and all that happens is that the TAC engineer sends you an email telling you that they “have work in progress” or something else not-making-the-case-evolve?

If so, I guess you have seen that the moment the engineer sends you an email, you also get a case update email telling you that the case has changed status to “customer pending”.

And that is a bit evil. I am pretty sure that more often than not, the reason for the engineer to send that email to you is not to tell you something, but to to actually change the case status. I have a feeling that the engineers effeciency is measured in how long the case is “Cisco pending” and as soon as the case is put over to the customer side, it is “all cool”. just like throwing a burning ball between two perssons. Or like a chess-clock that measure the time spent on each side.

The solution

The best way to handle this is to get even with their own weapons. Last week I had a mail dialogue with TAC that looked like this:

TAC: we are working on the information You sent. we will get back to you tomorrow.
[case status: Customer pending]

Me: thank you very much, I appretiate it.
[case status: customer updated -> Cisco pending]

TAC: you are welcome. have a nice day.
[case status: Customer pending]

Me: you too…
[case status: customer updated -> Cisco pending]

TAC: thank you very much!
[case status: Customer pending]

Me: please do not answer this email, since it changes the status of the case to “Customer pending”, which does NOT reflect the current situation.
[case status: customer updated -> Cisco pending]

I won!!!

When purging and cleaning ancient posts I found this post where I wished everyone a Happy New 2011. And I felt that it was time for an update.

So, what happened during 2011 – did I become a Cisco CCIE Security? The short answer is: No.

In february 2011 my written CCIE Security exam expired. Shortly after that my CCNA/CCNP/CCSP/whatever certifications also was about to expire, and to prevent that from happen I passed the CCIE Security Written once more. So, that means that I have another 18 (like 12 from now) months to do another Lab attempt.

During 2011 there was no way that I could find enough time to study for the lab. Primary of course because of the general work load, but also was my schedule filled with cool projects. Not only have I continued my journey to teach (I have made  my own study material on which 2 different Cisco ASA-workshops were based), I have also done a lot of implementations of Cisco ACS5 and 802.1x, and lately a few ISE-implementations as well.

So, will I ever get that CCIE number? I dont know, but I will continue to try. I have recently purchased the “Ultimate CCIE Security Self Paced bundle” from INE as a complement to the material I already have from IPExpert. I find a few hours every now and then and try to focus to gain the speed/accuracy needed for the dreaded exam.

Stay tuned, I´ll be back.

/Jimmy

I am fan of RSS readers. I use Google Reader all the time to keep track of interresting blog and news sites. Actually, i rarely visit blog sites direct, just from my RSS reader. And I love it.

But there are a few really good blogs that are configured not to post the full blog posts in their RSS stream. And this sucks. Here is an example:

Screen dump of Router Freak blog from RSS Reader

What happens when I come to these entries is either:

1. I read the ingress of the blog post. Find it really interresting and click the header that links me away from my RSS reader to the actual site where I continue to read ‘the full story’.
2. I read the ingress. Find it (probably, because the feed is in my reader) somewhat readworthy but doesnt care about reading the full post because that will link me away from the reader.

What happens more and more often is #2 above. And that´s sad. Because I really like to read what good bloggers writes. But I wanna do it in my reader.

So please, configure your RSS feed to contain the text of the ENTIRE blog post, not just the first x bytes… If it is more interresting for you to have me seeing your ad-banners on your page (which I only do if i make a ‘real’ visit) than it is for you to have me read your content, sorry You´ve lost me as a reader.

Recently we tried to join an Cisco ISE instance to Active Directory without success. The problem seemed to be because of the length of the ISE host name. Even though the system supports host names up to 19 characters long, we couldn’t add the ISE to AD until we shortened the name to be maximum 14 characters.

Another one of those undocumented “features” that I wish I have read about before getting stuck. I wish this short post is indexed so that other people find out and gets a push in the right direction because of it.

Hello

I am currently working on a task (INE CCIE Security WB 1 Task 2.9) where I am supposed to configured an radius-based IOS auth-proxy. The task is this:

Configure Authentication PRoxy settings on R3 per the following requirements.

• US the radius server at 10.0.0.100 with the authentication key CISCO.
• The authentication proxy should apply to the users sessions initiated from VLAN23 towards VLAN13.
• Authentication users should be allowed to send ICMP packets and initate TCP sessions.
• Configure the ACS server with the user named PROXY and the password of CISCO1234.

In ACS I have added the R3 as AAA client (Cisco IOS Radius). I have also added the user PROXY with the following cisco av pair´s:


auth-proxy:priv-lvl=15
auth-proxy:proxyacl#1=permit icmp any any
auth-proxy:proxyacl#1=permit tcp any any

 
In R3 I have added the following config:


aaa new-model
aaa authen login CON none
line con 0
login authen CON
aaa authen login default group radius
aaa author auth-proxy default group radius
!
ip http server
ip http authen aaa
ip auth-proxy name AUTHPROXY http
!
ip access-l ext INBOUND
permit udp any any eq rip
permit tcp any host 136.1.23.3 eq www
deny ip any any log
!
int fa0/1.23
ip access-group INBOUND in
ip auth-proxy AUTHPROXY

 
This is what happens when I fire up a browser and http´s to the R3 interface:
 
(debug aaa authen, aaa author, auth-proxy and radius is on)
 

Rack1R3#
*Jan 3 01:15:40.229: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.229: SYN SEQ 984706124 LEN 0
*Jan 3 01:15:40.229: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
*Jan 3 01:15:40.237: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.237: ACK 4057202766 SEQ 984706125 LEN 0
*Jan 3 01:15:40.237: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
*Jan 3 01:15:40.241: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.241: PSH ACK 4057202766 SEQ 984706125 LEN 282
*Jan 3 01:15:40.241: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
Rack1R3#
*Jan 3 01:15:40.245: Router interested packet returning src 136.1.23.123, dst 136.1.23.3
*Jan 3 01:15:40.257: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.261: ACK 4057202967 SEQ 984706407 LEN 0
*Jan 3 01:15:40.261: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
Rack1R3#
Rack1R3#! I fired up IE, entered the url and it is now showing a login prmpt "level_15 or view_access"
Rack1R3#
Rack1R3#! I enter the credentials PROXY/CISCO1234 and hit enter...
Rack1R3#
Rack1R3#
*Jan 3 01:16:52.743: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.743: FIN ACK 4057202967 SEQ 984706407 LEN 0
*Jan 3 01:16:52.743: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
*Jan 3 01:16:52.748: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.748: SYN SEQ 1525595421 LEN 0
*Jan 3 01:16:52.748: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
*Jan 3 01:16:52.756: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.756: ACK 2275096303 SEQ 1525595422 LEN 0
*Jan 3 01:16:52.756: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
*Jan 3 01:16:52.756: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.760: PSH ACK 2275096303 SEQ 1525595422 LEN 325
*Jan 3 01:16:52.760: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
*Jan 3 01:16:52.764: Router interested packet returning src 136.1.23.123, dst 136.1.23.3
*Jan 3 01:16:52.772: AAA/BIND(00000006): Bind i/f
*Jan 3 01:16:52.772: AAA/AUTHEN/LOGIN (00000006): Pick method list 'default'
*Jan 3 01:16:52.776: RADIUS/ENCODE(00000006):Orig. component type = HTTP
*Jan 3 01:16:52.776: RADIUS/ENCODE(00000006): dropping service type, "radius-server attribute 6 on-for-login-auth" is off
*Jan 3 01:16:52.776: RADIUS(00000006): Config NAS IP: 0.0.0.0
*Jan 3 01:16:52.776: RADIUS/ENCODE(00000006): acct_session_id: 4
*Jan 3 01:16:52.776: RADIUS(00000006): sending
*Jan 3 01:16:52.776: RADIUS/ENCODE: Best Local IP-Address 10.0.0.3 for Radius-Server 10.0.0.100
*Jan 3 01:16:52.780: RADIUS(00000006): Send Access-Request to 10.0.0.100:1645 id 1645/4, len 71
*Jan 3 01:16:52.780: RADIUS: authenticator 63 22 AD D4 03 CA 91 6C - 71 F8 27 E9 70 12 2A 18
*Jan 3 01:16:52.780: RADIUS: User-Name [1] 7 "PROXY"
*Jan 3 01:16:52.784: RADIUS: User-Password [2] 18 *
*Jan 3 01:16:52.784: RADIUS: NAS-Port-Type [61] 6 Virtual [5]
*Jan 3 01:16:52.784: RADIUS: Calling-Station-Id [31] 14 "136.1.23.123"
*Jan 3 01:16:52.784: RADIUS: NAS-IP-Address [4] 6 10.0.0.3
*Jan 3 01:16:52.796: RADIUS: Received from id 1645/4 10.0.0.100:1645, Access-Accept, len 181
*Jan 3 01:16:52.796: RADIUS: authenticator 4E 80 7B 47 1A 03 96 83 - BA 01 FE 83 9E A6 BB A6
*Jan 3 01:16:52.800: RADIUS: Framed-IP-Address [8] 6 255.255.255.255
*Jan 3 01:16:52.800: RADIUS: Vendor, Cisco [26] 30
*Jan 3 01:16:52.800: RADIUS: Cisco AVpair [1] 24 "auth-proxy:priv-lvl=15"
*Jan 3 01:16:52.800: RADIUS: Vendor, Cisco [26] 49
*Jan 3 01:16:52.800: RADIUS: Cisco AVpair [1] 43 "auth-proxy:proxyacl#1=permit icmp any any"
*Jan 3 01:16:52.800: RADIUS: Vendor, Cisco [26] 48
*Jan 3 01:16:52.804: RADIUS: Cisco AVpair [1] 42 "auth-proxy:proxyacl#2=permit tcp any any"
*Jan 3 01:16:52.804: RADIUS: Class [25] 28
*Jan 3 01:16:52.804: RADIUS: 43 41 43 53 3A 30 2F 31 37 34 39 66 2F 61 30 30 [CACS:0/1749f/a00]
*Jan 3 01:16:52.804: RADIUS: 30 30 30 33 2F 50 52 4F 58 59 [0003/PROXY]
*Jan 3 01:16:52.808: RADIUS(00000006): Received from id 1645/4
*Jan 3 01:16:52.812: AAA/AUTHOR (00000006): Method list id=0 not configured. Skip author
*Jan 3 01:16:54.815: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:54.815: ACK 2275096504 SEQ 1525595747 LEN 0
*Jan 3 01:16:54.815: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
Rack1R3#
Rack1R3#! ... and the browser give me another login prompt...
Rack1R3#
Rack1R3#

 

See those lines in bold? What is happening here? They are not in the output from the solution guide. The “radius-server attribute 6 on for login-auth”-message can be tweaked away with a specific command but why should that be neccesary? And what about “AAA/AUTHOR Metod list id=0 not configured. Skip author”, that feels like a fatal error. But I do have “aaa authorization auth-proxy default group radius”-command.
 
Anyone?

I wonder if one can convert a Cisco Wireless Controller 2106 into an ASA 5505 or vice versa. It seems to be the same hardware. Anyone that knows if there is any burned-in differences, or is it just a matter of replacing the software?

I will try to swap the CF-card in an ASA5505 with one from an WLC and see what happens. Stay tuned.

ASA5505:

WLC2106:

I think Windows 7 behaves strange with AnyConnect and IPv6

I have recently been doing a lot of ipv6-configurations and as part of that I tried out the ipv6-support in the Cisco Anyconnect-client. While doing that I found out a lack of functionality when it comes to ipv6 in combination with Windows 7 and the Aynconnect-client.

Since I have no native v6-support from my ISP I have an ipv6-tunnel from sixxs.net, providing my with my own /48-prefix network. An internal linux-host on my home networks serves as an ipv6 default-gateway and my home ASA firewall has an ipv6 default-route pointing towards that machine.

I have been abroad for a few days and fooled around with the Anyconnect while wasting time at the hotel room, and what I found out is a bit strange. Windows simply doesnt care about the Aynconnect v6-address when it comes to DNS lookups.

The ASA firewall at home has been configured with an v6-address on the inside interface and a default-route as stated above. I have added an ipv6-pool in addition to the normal ipv4 vpn-pool configured in my DfltGrpPolicy and my VPN-clients gets an v6-address as well as an v4-address:

So I have a Windows7-client with ipv4-only configured on the nic, and dual-stack configured on the tunnel-interface. Look what happens when I try to resolve an hostname that only has an A-record (that is, v4):

The wireshark-capture prooves that only an A-record is resolved:

On the other hand, when I manually resolves an AAAA-record (v6) I get an instant lookup:

And the corresponding wireshark-capture:

Also, when I enter http://[2a00:1450:8001:63] in an browser I get the Google web-page.

So: My client has full connectivity with both v4-internet and v6-internet. Still, I cannot reach v6-internet in a decent way since windows doesnt resolve AAAA-records.

Shouldnt it do lookups of both AAAA and A-record as it would if I had dual stacks configured on the ordinary nick? Is this something wrong in Windows? Or in the Anyconnect-client? Or have I done something wrong?

Enlighten me!