I have recently implemented a few Cisco Ironport WSA-solutions. When doing a follow-up after the implementation, the customer usually reacts with “Oh… WSA? We forgot about that. It probably works…”

But what difference does it make? If the customer forgets about their web proxy, what good does it do? Lets have a look at an implementation…

I asked one of our customers for permission to peek into their WSA for the purpose of this blog post. This customer has a few hundred users and is a fairly traditional type of user with mostly office users, each with a personal computer. This customer doesnt limit web browsing, except for filtering out access to known obviously bad web categories like child porn. Except for that, free access to the Web.

Fig1: General Statistics

The first thing to look at is an overview of web activity above. The average web traffic an business day is roughly one million is a working day consists of one million web requests. A web page contains several objects (images, scripts) where each object needs to be requested individually. In this implementation the clients generates 1 million transactions (requests) per day, or 20 million transactions per month.

But what is the content of the requested material? If we look at But WHAT users to surf? If you then look at the purity of operations as it starts to get interesting for real!

Fig2: Purity

Here you can see that just over 10,000 (10.6K) transactions have been stopped this month because of URL category! That is, such as child porn! There are objects (pages, images, etc.) that the user consciously or unconsciously sought but that the system has already been blocked at the access-trial because the source is known and undesirable.

One can also see that almost 3,000 (2.797) object has been blocked due to malware detection. Remember that the WSA scans all through traffic for known viruses, scripts, or other type of malware. The source category has been approved or unknown the WSA have downloaded content. But when checking the contents, they have discovered something unwanted. This little fella has thus stopped nearly 3,000 viruses in the past month!

Overall, 99.8% of web traffic this month has been “clean”. 0.2% may seem to be disappearing bit, but it is still almost 34 000 (33.8K) potentionella threat that was blocked already at the front door!

If you want more detailed information about the type of threat blocked, you can obviously get it also:

Fig3: Malware

With the help of the dynamic Sender Base system scored all websites on the internet. Based on a number of factors such as known virus outbreak or the credibility of a domain, each site a web reputation score from -10 to +10. WSA is configured to always block the sources with the lowest score and always allow the web site with the highest score. But how does this when in reality?

Fig4: Web Reputation

Here we can see that nearly 10,000 transactions in the last month blocked because of Web reputation.

The conclusion I draw every time I look at this type of reporting is that the WSA is blocking lots of web traffic in the covert, and it’s surprisingly rare that users react to the IT department because they can not browse to a specific site. It may be that the user deliberately tries to make stupid mistakes on the internet, but my experience and absolute conviction is that it almost always is something that happens unconsciously. A link to an email or on facebook that look “nice”, but takes the user to a  malware site in some obscure corner of the Internet.

Key figures for this particular device, a typical month “at work”:

• The number of transactions: 20.4 million pieces.
• The number of blocked transactions: 33 800 pcs.
• The number of blocked Malwares / viruses: 2797 pcs, or one every 3 minutes during business hours!
• Dare you not to check the content of your web traffic?

The problem

have you ever had an open TAC case with Cisco, just waiting for them to provide either a solution or some other kind of feedback, and all that happens is that the TAC engineer sends you an email telling you that they “have work in progress” or something else not-making-the-case-evolve?

If so, I guess you have seen that the moment the engineer sends you an email, you also get a case update email telling you that the case has changed status to “customer pending”.

And that is a bit evil. I am pretty sure that more often than not, the reason for the engineer to send that email to you is not to tell you something, but to to actually change the case status. I have a feeling that the engineers effeciency is measured in how long the case is “Cisco pending” and as soon as the case is put over to the customer side, it is “all cool”. just like throwing a burning ball between two perssons. Or like a chess-clock that measure the time spent on each side.

The solution

The best way to handle this is to get even with their own weapons. Last week I had a mail dialogue with TAC that looked like this:

TAC: we are working on the information You sent. we will get back to you tomorrow.
[case status: Customer pending]

Me: thank you very much, I appretiate it.
[case status: customer updated -> Cisco pending]

TAC: you are welcome. have a nice day.
[case status: Customer pending]

Me: you too…
[case status: customer updated -> Cisco pending]

TAC: thank you very much!
[case status: Customer pending]

Me: please do not answer this email, since it changes the status of the case to “Customer pending”, which does NOT reflect the current situation.
[case status: customer updated -> Cisco pending]

I won!!!

When purging and cleaning ancient posts I found this post where I wished everyone a Happy New 2011. And I felt that it was time for an update.

So, what happened during 2011 – did I become a Cisco CCIE Security? The short answer is: No.

In february 2011 my written CCIE Security exam expired. Shortly after that my CCNA/CCNP/CCSP/whatever certifications also was about to expire, and to prevent that from happen I passed the CCIE Security Written once more. So, that means that I have another 18 (like 12 from now) months to do another Lab attempt.

During 2011 there was no way that I could find enough time to study for the lab. Primary of course because of the general work load, but also was my schedule filled with cool projects. Not only have I continued my journey to teach (I have made  my own study material on which 2 different Cisco ASA-workshops were based), I have also done a lot of implementations of Cisco ACS5 and 802.1x, and lately a few ISE-implementations as well.

So, will I ever get that CCIE number? I dont know, but I will continue to try. I have recently purchased the “Ultimate CCIE Security Self Paced bundle” from INE as a complement to the material I already have from IPExpert. I find a few hours every now and then and try to focus to gain the speed/accuracy needed for the dreaded exam.

Stay tuned, I´ll be back.

/Jimmy

I am fan of RSS readers. I use Google Reader all the time to keep track of interresting blog and news sites. Actually, i rarely visit blog sites direct, just from my RSS reader. And I love it.

But there are a few really good blogs that are configured not to post the full blog posts in their RSS stream. And this sucks. Here is an example:

Screen dump of Router Freak blog from RSS Reader

What happens when I come to these entries is either:

1. I read the ingress of the blog post. Find it really interresting and click the header that links me away from my RSS reader to the actual site where I continue to read ‘the full story’.
2. I read the ingress. Find it (probably, because the feed is in my reader) somewhat readworthy but doesnt care about reading the full post because that will link me away from the reader.

What happens more and more often is #2 above. And that´s sad. Because I really like to read what good bloggers writes. But I wanna do it in my reader.

So please, configure your RSS feed to contain the text of the ENTIRE blog post, not just the first x bytes… If it is more interresting for you to have me seeing your ad-banners on your page (which I only do if i make a ‘real’ visit) than it is for you to have me read your content, sorry You´ve lost me as a reader.

Recently we tried to join an Cisco ISE instance to Active Directory without success. The problem seemed to be because of the length of the ISE host name. Even though the system supports host names up to 19 characters long, we couldn’t add the ISE to AD until we shortened the name to be maximum 14 characters.

Another one of those undocumented “features” that I wish I have read about before getting stuck. I wish this short post is indexed so that other people find out and gets a push in the right direction because of it.

Hello

I am currently working on a task (INE CCIE Security WB 1 Task 2.9) where I am supposed to configured an radius-based IOS auth-proxy. The task is this:

Configure Authentication PRoxy settings on R3 per the following requirements.

• US the radius server at 10.0.0.100 with the authentication key CISCO.
• The authentication proxy should apply to the users sessions initiated from VLAN23 towards VLAN13.
• Authentication users should be allowed to send ICMP packets and initate TCP sessions.
• Configure the ACS server with the user named PROXY and the password of CISCO1234.

In ACS I have added the R3 as AAA client (Cisco IOS Radius). I have also added the user PROXY with the following cisco av pair´s:


auth-proxy:priv-lvl=15
auth-proxy:proxyacl#1=permit icmp any any
auth-proxy:proxyacl#1=permit tcp any any

 
In R3 I have added the following config:


aaa new-model
aaa authen login CON none
line con 0
login authen CON
aaa authen login default group radius
aaa author auth-proxy default group radius
!
ip http server
ip http authen aaa
ip auth-proxy name AUTHPROXY http
!
ip access-l ext INBOUND
permit udp any any eq rip
permit tcp any host 136.1.23.3 eq www
deny ip any any log
!
int fa0/1.23
ip access-group INBOUND in
ip auth-proxy AUTHPROXY

 
This is what happens when I fire up a browser and http´s to the R3 interface:
 
(debug aaa authen, aaa author, auth-proxy and radius is on)
 

Rack1R3#
*Jan 3 01:15:40.229: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.229: SYN SEQ 984706124 LEN 0
*Jan 3 01:15:40.229: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
*Jan 3 01:15:40.237: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.237: ACK 4057202766 SEQ 984706125 LEN 0
*Jan 3 01:15:40.237: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
*Jan 3 01:15:40.241: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.241: PSH ACK 4057202766 SEQ 984706125 LEN 282
*Jan 3 01:15:40.241: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
Rack1R3#
*Jan 3 01:15:40.245: Router interested packet returning src 136.1.23.123, dst 136.1.23.3
*Jan 3 01:15:40.257: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:15:40.261: ACK 4057202967 SEQ 984706407 LEN 0
*Jan 3 01:15:40.261: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
Rack1R3#
Rack1R3#! I fired up IE, entered the url and it is now showing a login prmpt "level_15 or view_access"
Rack1R3#
Rack1R3#! I enter the credentials PROXY/CISCO1234 and hit enter...
Rack1R3#
Rack1R3#
*Jan 3 01:16:52.743: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.743: FIN ACK 4057202967 SEQ 984706407 LEN 0
*Jan 3 01:16:52.743: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1103
*Jan 3 01:16:52.748: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.748: SYN SEQ 1525595421 LEN 0
*Jan 3 01:16:52.748: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
*Jan 3 01:16:52.756: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.756: ACK 2275096303 SEQ 1525595422 LEN 0
*Jan 3 01:16:52.756: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
*Jan 3 01:16:52.756: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:52.760: PSH ACK 2275096303 SEQ 1525595422 LEN 325
*Jan 3 01:16:52.760: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
*Jan 3 01:16:52.764: Router interested packet returning src 136.1.23.123, dst 136.1.23.3
*Jan 3 01:16:52.772: AAA/BIND(00000006): Bind i/f
*Jan 3 01:16:52.772: AAA/AUTHEN/LOGIN (00000006): Pick method list 'default'
*Jan 3 01:16:52.776: RADIUS/ENCODE(00000006):Orig. component type = HTTP
*Jan 3 01:16:52.776: RADIUS/ENCODE(00000006): dropping service type, "radius-server attribute 6 on-for-login-auth" is off
*Jan 3 01:16:52.776: RADIUS(00000006): Config NAS IP: 0.0.0.0
*Jan 3 01:16:52.776: RADIUS/ENCODE(00000006): acct_session_id: 4
*Jan 3 01:16:52.776: RADIUS(00000006): sending
*Jan 3 01:16:52.776: RADIUS/ENCODE: Best Local IP-Address 10.0.0.3 for Radius-Server 10.0.0.100
*Jan 3 01:16:52.780: RADIUS(00000006): Send Access-Request to 10.0.0.100:1645 id 1645/4, len 71
*Jan 3 01:16:52.780: RADIUS: authenticator 63 22 AD D4 03 CA 91 6C - 71 F8 27 E9 70 12 2A 18
*Jan 3 01:16:52.780: RADIUS: User-Name [1] 7 "PROXY"
*Jan 3 01:16:52.784: RADIUS: User-Password [2] 18 *
*Jan 3 01:16:52.784: RADIUS: NAS-Port-Type [61] 6 Virtual [5]
*Jan 3 01:16:52.784: RADIUS: Calling-Station-Id [31] 14 "136.1.23.123"
*Jan 3 01:16:52.784: RADIUS: NAS-IP-Address [4] 6 10.0.0.3
*Jan 3 01:16:52.796: RADIUS: Received from id 1645/4 10.0.0.100:1645, Access-Accept, len 181
*Jan 3 01:16:52.796: RADIUS: authenticator 4E 80 7B 47 1A 03 96 83 - BA 01 FE 83 9E A6 BB A6
*Jan 3 01:16:52.800: RADIUS: Framed-IP-Address [8] 6 255.255.255.255
*Jan 3 01:16:52.800: RADIUS: Vendor, Cisco [26] 30
*Jan 3 01:16:52.800: RADIUS: Cisco AVpair [1] 24 "auth-proxy:priv-lvl=15"
*Jan 3 01:16:52.800: RADIUS: Vendor, Cisco [26] 49
*Jan 3 01:16:52.800: RADIUS: Cisco AVpair [1] 43 "auth-proxy:proxyacl#1=permit icmp any any"
*Jan 3 01:16:52.800: RADIUS: Vendor, Cisco [26] 48
*Jan 3 01:16:52.804: RADIUS: Cisco AVpair [1] 42 "auth-proxy:proxyacl#2=permit tcp any any"
*Jan 3 01:16:52.804: RADIUS: Class [25] 28
*Jan 3 01:16:52.804: RADIUS: 43 41 43 53 3A 30 2F 31 37 34 39 66 2F 61 30 30 [CACS:0/1749f/a00]
*Jan 3 01:16:52.804: RADIUS: 30 30 30 33 2F 50 52 4F 58 59 [0003/PROXY]
*Jan 3 01:16:52.808: RADIUS(00000006): Received from id 1645/4
*Jan 3 01:16:52.812: AAA/AUTHOR (00000006): Method list id=0 not configured. Skip author
*Jan 3 01:16:54.815: AUTH-PROXY:proto_flag=4, dstport_index=4
*Jan 3 01:16:54.815: ACK 2275096504 SEQ 1525595747 LEN 0
*Jan 3 01:16:54.815: dst_addr 136.1.23.3 src_addr 136.1.23.123 dst_port 80 src_port 1104
Rack1R3#
Rack1R3#! ... and the browser give me another login prompt...
Rack1R3#
Rack1R3#

 

See those lines in bold? What is happening here? They are not in the output from the solution guide. The “radius-server attribute 6 on for login-auth”-message can be tweaked away with a specific command but why should that be neccesary? And what about “AAA/AUTHOR Metod list id=0 not configured. Skip author”, that feels like a fatal error. But I do have “aaa authorization auth-proxy default group radius”-command.
 
Anyone?

I wonder if one can convert a Cisco Wireless Controller 2106 into an ASA 5505 or vice versa. It seems to be the same hardware. Anyone that knows if there is any burned-in differences, or is it just a matter of replacing the software?

I will try to swap the CF-card in an ASA5505 with one from an WLC and see what happens. Stay tuned.

ASA5505:

WLC2106:

I think Windows 7 behaves strange with AnyConnect and IPv6

I have recently been doing a lot of ipv6-configurations and as part of that I tried out the ipv6-support in the Cisco Anyconnect-client. While doing that I found out a lack of functionality when it comes to ipv6 in combination with Windows 7 and the Aynconnect-client.

Since I have no native v6-support from my ISP I have an ipv6-tunnel from sixxs.net, providing my with my own /48-prefix network. An internal linux-host on my home networks serves as an ipv6 default-gateway and my home ASA firewall has an ipv6 default-route pointing towards that machine.

I have been abroad for a few days and fooled around with the Anyconnect while wasting time at the hotel room, and what I found out is a bit strange. Windows simply doesnt care about the Aynconnect v6-address when it comes to DNS lookups.

The ASA firewall at home has been configured with an v6-address on the inside interface and a default-route as stated above. I have added an ipv6-pool in addition to the normal ipv4 vpn-pool configured in my DfltGrpPolicy and my VPN-clients gets an v6-address as well as an v4-address:

So I have a Windows7-client with ipv4-only configured on the nic, and dual-stack configured on the tunnel-interface. Look what happens when I try to resolve an hostname that only has an A-record (that is, v4):

The wireshark-capture prooves that only an A-record is resolved:

On the other hand, when I manually resolves an AAAA-record (v6) I get an instant lookup:

And the corresponding wireshark-capture:

Also, when I enter http://[2a00:1450:8001:63] in an browser I get the Google web-page.

So: My client has full connectivity with both v4-internet and v6-internet. Still, I cannot reach v6-internet in a decent way since windows doesnt resolve AAAA-records.

Shouldnt it do lookups of both AAAA and A-record as it would if I had dual stacks configured on the ordinary nick? Is this something wrong in Windows? Or in the Anyconnect-client? Or have I done something wrong?

Enlighten me!

Newer versions of Cisco ASA requires more memory. Running anyconnect with multiple platform support requires more flash-memory than built in. There are memory upgrades available for purchase from cisco.com which I highly recommend. However, for lab-purposes any DDR memory and CompactFlash-card will do. Have a look in my lab gear.

First, an ASA5505. On the overview photo below you can see that it has one single DDR memory-slot (to the far lower right corner on the picture). I have tried both 512Mb-modules and 1Gb-modules and both worked fine. Even if it is not visible from outside there is also an CF-slot. Remove the cover and replace the current CF-module with a bigger. I have tried both 2Gb and 4Gb-modules with success.

Picure of ASA5505 internals. Note the CF-slot in the bottom part and the memory to the right.

Picture of upgraded memory module from an ASA5505

ASA5510 comes in different flavours depending on hardware revision. Older versions have 4 memory slots that needs to be filled with pairs of identical modules. In newer revisions there are only one single memory slot, and I guess (but I am not sure) that it support larger memory modules!

Picture of label on top of an Revision 01 ASA5510.

Picture of an ASA5510 Revision 01 filled with 2x512Mb. Note the disk1: CF-card accessible from outside and the internal disk0: CF-module just adjacent to in in the bottom of the picture.

Picture of the memory-modules I use in an ASA5510 Revision 01.

Picture of an ASA5510 Revision 03-label.

Picture of an Revision 03 ASA5510 with one single memory slot.

Picture of the memory module I use in an ASA5510 revision 03.

Again, remember that third party memory modules are not supported from Cisco. I strongly discourage using non-supported hardware in any production environment!

And one final note: When you replace the CF-module you will notice that your current startup-config as well as the activation-key are gone. To avoid this, take your old original CF-card and put it in your computer. Make sure that your computer shows “hidden files“. Copy all content from the old module (maybe via a folder on your computer if you can only insert one CF at a time) and paste it back to your brand new large CF. And voila, all licensing and config are visible to the ASA! Also. On 5510+ there are double CF-slots: one internal and one external. Replace the external and address it as disk1:, put all large files there and your startup-config as well as hidden files containing your licenses will be untouched on the internal CF-card, addressed as disk0:

To Håkan: This is the memory module I bought. J

 I while ago I got into a discussion with one of my customers regarding ipv6. He told me that one reason not to migrate to ipv6 was for security. 

- I dont want to tell the entire world what IP addresses I have on my servers. And when using ipv4 and NAT my internal ip addresses are hidden.

The discussion was interrupted and I didnt get any chance to finish it. 

When using private ipv4-addresses on your LAN i can assume that you have any of these addresses:

• 10.0.0.0/8
• 172.16.0.0/12
• 192.168.0.0/16

So, how many addresses do you have to choose from? Lets count (roughly!):

• 10.0.0.0/8, that is 256 * 256 * 256 addresses, 16 777 216 available addresses
• 172.16.0.0/12, that is 16 * 256 * 256 addresses, 1 048 576
• 192.168.0.0/16, that is 256 * 256 addresses, 65 536.

That gives us a total sum of 17 891 328 available addresses. That´s a lot, isnt it?

But what if you get yourself a nice little pool of ipv6-addresses? For various reasons we can be pretty sure that you will get a /48 network from your ISP. Then you will probably divide this into one or many /64-networks on your internal LAN. So, how many addresses are there available?

First of all, dividing that /48-range into /64-subnets will give you 65536 different available networks. Next, an ipv6-address is 128 bits long. With 64 bits for specifying the network part you will have 64 bits left for addressing each individual host on your internal network. And 64 bits gives us 18446744073709551616 unique combinations. So that is how many addresses you have available in each subnet when using ipv6.

So, if you see it as a security benefit to hide your sensitive servers addresses, which do you prefer? ipv4 or ipv6?

If a hacker would portscan your ipv6-range, how long will it take? Lets assume that he scans 100 addresses per second, then it will take him 5 849 424 173 years(*). And that should be compared to the 50 hours it will take to port scan all private ipv4-addresses mentioned above.

And besides. That attack would probably be performed from internet. How many public ipv4-addresses do you have? It will be enough to portscan them. 100 addresses per seconds, you do the math.

/Jimmy