“I cannot connect with my Cisco IPSec VPN-client when I am behind a firewall”

“I can connect my VPN-client but can´t get any traffic thru”

“I have changed the settings on the Transport-tab and now I don´t know which settings are correct”

Have you heard them all? I have, plenty of times! In fact, IPSec can be really messy to get to work, because when it was invented there was no such thing as NAT devices and IPSec doesn´t play very well with address translations. There are workarounds. Alot of them. And that´s why it is a bit messy.

Client settings

In the client there are settings under the transport tab tha changes how the client communicates with the head end. Unfortunately, these settings are not protected which means that the end use can (and will!) changes these settings. So, what does the different settings mean?

Enable Transparent Tunneling: No

This mode is the vanilla way of IPSec by the book. The tunnel is setup by using ISAKMP (udp/500) and the actual data is sent as ESP (ip/50).

Because of the way ESP works it doesn´t work well if the client is behind a firewall or other NAT device. If outbound ISAKMP is allowed, the client can connect and authenticate. But the ESP-data will never get thru and the user will experience that the tunnel is broken even if he was able to login.

Enable Transparent Tunneling over UDP

 This is the most common way to overcome the limitations of ESP. The tunnel setup is still being done over ISAKMP (udp/500) but the actual data is encapsulated in udp-packets (udp/4500).

For this to work, the head end must be configured with ‘crypto isakmp nat-traversal‘. This command will cause the head end to tell the client during tunnel setup to send data over udp/4500 instead of ESP. Without this configured in the head end, the client will experience the same thing as when Transparent tunneling is disabled (see above) because it will still use ESP for data-transfer unless told otherwise by the head end.

Enable Transparent Tunneling over TCP

This setting is rarely being used. It was invented in the Cisco VPN3000 concentrator and is also supported in pix/ASA. By tunneling traffic over a TCP/port both the tunnel setup and the actual data is sent over that port. That means that ISAKMP (udp/500) is not being used when doing IPSec over TCP. The default port (and most common) is tcp/10000 but any port will do good. But, the port must be specified in the head end with the ‘crypto isakmp ipsec-over-tcp port 10000′ command.

Answers

So, what are the answers for the end user questions on top of this post? I would say:

Q: “I cannot connect with my Cisco IPSec VPN-client when I am behind a firewall”

A: Make sure that the firewall administrator at the current location makes sures that the following ports are opened outbound:

• udp/500 (ISAKMP)
• udp/4500 (IPSec nat-traversal)
• udp/10000 (IPSec over TCP)

Q: “I can connect my VPN-client but can´t get any traffic thru”

A: Enable transport tunneling over UDP in the Transport-tab and try again. If you can still connect but not communicate, make sure that the firewall administrator (at the site to which you are trying to connect!) enables nat-traversal with the ‘crypto isakmp nat-traversal’-command.

Q: “I have changed the settings on the Transport-tab and now I don´t know which settings are correct”

A: Duh! How would I know? Ask your firewall administrator or IT helpdesk. If you still has a copy of the original .pcf-file, try to delete your current profile in the vpn-client and import the .pcf-file again.

My recommendations

Since there are a number of ways to configure the VPN client and the central firewall, which one should we use? Which one gives us least headache? I would say that you should choose from the below, in given order:

1. Don´t. Use AnyConnect instead of the old IPSec-client. There are zillions of reasons why, but they doesn´t fit into this blog post. Anyway. Migrate to AnyConnect if possible!
2. Use Transparent Tunneling over UDP. Make sure that the central firewall is configured with NAT-traversal as explained above.
3. Use Transparent Tunneling over TCP. But don´t. There is an extra overhead in encapsulating the end user traffic in yet another layer of TCP-sessions. UDP is better. But if you want to use TCP, use port 10000 because it is already entered by default in the vpn client.
4. Use the client without transparent tunneling. You use GRE and will never get the client to work from behind a firewall.

Yesterday at the RSA Conference Cisco released a new product named ASA CX. As usual when Cisco releases information about new products you have to dig alot to see thru all marketing material and find technical details. And so is defenately the case here also.

There are a few videos recentely uploaded to Youtube by Cisco that describes the product, and a few links in the marketing material cross-referencing eachother. But not much more than that. Yet. However, this is what I have found (and what I can guess by reading between the lines):

ASA CX is Micro Application Aware. This means that it should be able to filter traffic based on Layer7-information to for example block Facebook Chat, but allow Facebook Updates. Allow Skype, but block Bittorrent. And so on…

ASA CX also saids to be web reputation aware and to be able to block 0 day malwares. Together with Identity Based Firewalling (allow/deny traffic baesd on user/group-belongings rather than just ip addresses) and URL filtering it smells alot like they have put a Cisco Ironport WSA-box inside of the ASA.

Cisco ASA CX is by Cisco Prime Security Manager which is shipped with (within!) the ASA CX, which means no more ASDM!

What confuses me most is that even though there is information on Ciscos website that ASA CX comes as 2 modules (“CX SSP-10″ and “CX SSP-20″) there is also a new product line of ASA:s visible on the product comparison chart: 5512-X, 5515-X and so on… And with yet no information available on in which models of ASA you can put the CX SSP-modules, I still cant tell what´s needed to run ASA CX. Can I upgrade my existing ASA-firewall to CX with a module? And if so, which models can be upgraded? If not, what models of ASA CX appliances are available? Does an ASA5512-X contain an XS SSP-10? And so on….

A conclusion: It´s really thrilling that the next generation of ASA Firewalls can do this granular application inspections that hasn´t been possible yet. And together with functions available in the WSA, ASA CX can be a really potent threat to it competitors! ASA is no longer a packet filtering firewall!

Update!

According to a anonymous but normally highly trustworthy source (who prefer to call himself Deep Throat ), the CX will at first be a module available only for the high-end 5585-X ASA:s. At a next step the CX will be a software function available in the newly released 5505-X ASA-models. There will probably not be any CX-support in the legacy ASA:s.

Or – ASA Lan2Lan-VPN for dummies.

I often get questions related to Lan2Lan-tunnels in ASA. This post serves as a cheat-sheet for different software versions.

Pix v6.x

isakmp enable outside
isakmp policy 1 authentication pre-share
isakmp policy 1 encryption des
isakmp policy 1 hash md5
isakmp policy 1 group 1
isakmp policy 1 lifetime 86400

isakmp key cisco123 address 5.6.7.8 netmask 255.255.255.255

access-list 100 permit ip 10.0.X.0 255.255.255.0 10.0.Y.0 255.255.255.0
crypto ipsec transform-set MYTSET esp-des esp-md5-hmac

crypto map CMAP_OUTSIDE 10 ipsec-isakmp
crypto map CMAP_OUTSIDE match address 100
crypto map CMAP_OUTSIDE set peer 5.6.7.8
crypto map CMAP_OUTSIDE interface outside

access-list nonat_inside permit ip 10.0.X.0 255.255.255.0 10.0.Y.0 255.255.255.0
nat (inside) 0 access-list nonat_inside

sysopt connection permit-ipsec

Pix/ASA v7.0 – 8.2

isakmp enable outside
isakmp policy 1 authentication pre-share
isakmp policy 1 encryption des
isakmp policy 1 hash md5
isakmp policy 1 group 1
isakmp policy 1 lifetime 86400

tunnel-group 5.6.7.8 type ipsec-l2l
tunnel-group 5.6.7.8 ipsec-attributes
pre-shared-key cisco123

access-list VPN permit ip 10.0.X.0 255.255.255.0 10.0.Y.0 255.255.255.0

crypto ipsec transform-set MYTSET esp-des esp-md5-hmac

crypto map CMAP_OUTSIDE 10 ipsec-isakmp
crypto map CMAP_OUTSIDE 10 set transform-set MYTSET
crypto map CMAP_OUTSIDE 10 match address VPN
crypto map CMAP_OUTSIDE 10 set peer 5.6.7.8
crypto map CMAP_OUTSIDE interface outside

access-list nonat_inside permit ip 10.0.X.0 255.255.255.0 10.0.Y.0 255.255.255.0
nat (inside) 0 access-list nonat_inside

ASA v8.3+

crypto ikev1 enable outside
crypto ikev1 policy 1
authentication pre-share
encryption des
hash md5
group 1
lifetime 86400

tunnel-group 5.6.7.8 type ipsec-l2l
tunnel-group 5.6.7.8 ipsec-attributes
ikev1 pre-shared-key cisco123

access-list VPN permit ip 10.0.X.0 255.255.255.0 10.0.Y.0 255.255.255.0

crypto ipsec ikev1 transform-set MYTSET esp-des esp-md5-hmac

crypto map CMAP_OUTSIDE 10 ipsec-isakmp
crypto map CMAP_OUTSIDE 10 set ikev1 transform-set MYTSET
crypto map CMAP_OUTSIDE 10 match address VPN
crypto map CMAP_OUTSIDE 10 set peer 5.6.7.8
crypto map CMAP_OUTSIDE interface outside

object network MY-LAN
subnet 10.0.X.0 255.255.255.0
object network HIS-LAN
subnet 10.0.Y.0 255.255.255.0
nat (inside,outside) source static MY-LAN MY-LAN destination static HIS-LAN HIS-LAN

I am a huge fan of Cisco ISE and Trustsec. I have done a few live implementations as well as at home (anyone should run Trustsec at home! ). There will probably be a lot of ISE-related posts here in the near future.

Here I just want to reflect on how well the built-in profiler works in ISE (1.04). I have run the profiler for a few days now and have automatically gathered a complete list of devices in my home network. From here I can build my 802.1x authorization policies to give granular access to devices of a specific type, rather on plain user-based 802.1x.

For example: All NintendoWII-devices will automatically get Internet-only access. The HP-Device can be automatically moved to the Printer-Vlan (which does only have access to elsewhere on the jetdirect-ports) and the Microsoft Workstations should only get access to the core network if they are successfully authenticated via EAP-TLS. The sky is the limit…

I just returned home after spending almost a week in London attendingCisco Live. Much can be said about the event and many has already summarized their experience, so the plan for this blog post is to make a short resumé of the sessions I attended to. Many were great, most were good but a few were less than good. I skip the latter here and focus on the best pieces.

TECSEC-3030 – Advanced Network Access Control with ISE

This was a techtorial, which means that the session covers a full 9-hour day of presentation. Relative to most people I have a lot if experience with Cisco ISE. I have attended a 5-day pre-ATP class for Cisco ISE and done a handfull of implementations. Nevertheless, this techtorial was really relevant for me since I got a lot of repetitions of theories and behind-the-scenes that is easy to forget about in the daily work. Also, since the speakers have so much in-depth knowledge of this new product is gives alot to hear what they say (and what the do not say). I am totally convinced that Cisco has raised the old 802.1x-horse to a new level by combining products like ISE and Anyconnect with the new concept of TrustSec to allow the right device access to the right parts of the network, not only defined by what user is using the device but also based on how it is connected (wired/wireless/vpn) and what kind of device is it (comporate computer/private Ipad/mobile phone). This rocks!

BRKNMS3134 – Advanced NetFlow

Netflow is a protocol to gather information about network traffic for further analyzis. Instead of analyzing the traffic-flow inline, netflow-enabled devices (routers) collects information about which devices that “talks” to who, amount of traffic and ports. This information is sent to Netflow collectors for analyzis.

I am not a netflow-guy. I have only tried it a few times. But this Live-session was really cool. With Flexible Netflow the sky is the limit when it comes to which kind of information to select and where to send it. Netflow v9 is the key!

BRKSEC-2071 – Securing DNS

The fact that there are vurnerabilities in the DNS-procotol is nothing new. And it has been known for a while now that DNSSEC is the solution to most security-related DNS-issues. The session contained a live demo of DNS-cache poisoning a´la Kaminsky and thereafter a complete walkthru of actions to prevent this from happening.

The speakor Stenthor Bjarnason really showed in-depth knowledge in the subject!

BRKSEC-3005 – Advanced IEEE 802.1x for wired networks

This session was an advanced session that discussed all aspects of 802.1x-implementations in wired networks. It went through the concept of authentication and authorization, radius-attributes for downloadable access-lists and vlan-changes and discussed aspects on how to handle non-802.1x-enabled devices with MAB. Further on there was a lot of information about how to troubleshoot dot1x and how to handle PKI in dot1x-implementations.  The speaker was awesome!

BRKIPM-2999 – LISP – A Next Generation Networking Architecture

I am not a thru router guy. At least not compared to my fellow r/s-friends who eat MPLS for breakfast. So attending a LISP session was really a step out of my comfort zone. But it was so cool! It is not easy to explain in a few centences what LISP is. In short you can say that LISP is a new way to rout ip packets, not only based on destination but on other parameters aswell. And since these parameters are stored in central units, you can say that LISP uses something similar to DNS to query how to route traffic. And in a sence you can also say that LISP is a way to tunnel traffic. This will be big!

BRKSEC-2046 – Deploying Security Group Tags

SGT is one of the building blocks that builds the foundation of Cisco TrustSec. In short: SGT is a way to tag packets ingress in access layer devices so that they can be filtered egress centrally. The reason to do this as a complement to a firewall is to gain speed. Also, in 802.1x-enabled networks the access-switch has a lot of knowledge about the traffic (type of device, authentication information…) which means that traffic can be tagged (and further filtered) not only based on ip-information but also based on username/device type/connection type/<insert almost anything here>.

BRKSEC-3033 – Advanced Anyconnect Deployment and Troubleshooting with ASA 5500

This was my favourite. It is always a pleasure to listen to Håkan Nohre, imho one of the greatest brains when it comes to Cisco ASA and Cisco-based security solutions. I know this subject by heart so I cannot say that there was a lot of news for me. But it it really cool to listen to how inspired and excited Håkan is when he talk about what he loves most.

Roundup

I must say that all sessions on Cisco Live keeps a very high quality when it comes to content and how it is presented. I have left out a few of the ones I attended here and the reason for not all sessions being perfect for me was not the content but more that I choose the wrong sessions. So Cisco Live, keep up the good work. And a final note: The PDF material provided after attending Cisco Live is by far the most comprehensive and good technical reference material one can find. Even if I might not attend IRL on Cisco Live next year I will definately pay for the virtual pass, making me able to access the presentations online!

I recently had a TAC-case regarding a Cisco ASA 5510-firewall with Anyconnect-clients which had issues with VPN-clients not being able to connect due to “no address available”. It turned out that the “show vpn-sessiondb anyconnect”-command showed 50+ anyconnect-sessions that were over one month old! Like this:

sh vpn-sessiondb anyconnect
Session Type: AnyConnect
Username : aaaaa Index : 110
Assigned IP : zx.zx.zx.zx Public IP : qw.qw.qw.qw
Protocol : Clientless DTLS-Tunnel
License : AnyConnect Essentials
Encryption : RC4 AES128 Hashing : SHA1
Bytes Tx : 40577016 Bytes Rx : 5480886
Group Policy : DfltGrpPolicy Tunnel Group : DefaultWEBVPNGroup
Login Time : 10:43:24 CEST Fri Dec 16 2011
Duration : 34d 23h:20m:15s
Inactivity : 32d 2h:00m:04s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none

Username : zzzzz Index : 152
Assigned IP : x.x.x.x Public IP : y.y.y.y
Protocol : AnyConnect-Parent DTLS-Tunnel
License : AnyConnect Essentials
Encryption : AES128 Hashing : none SHA1
Bytes Tx : 13671510 Bytes Rx : 8421169
Group Policy : DfltGrpPolicy Tunnel Group : DefaultWEBVPNGroup
Login Time : 04:39:57 CEST Tue Dec 20 2011
Duration : 31d 5h:23m:42s
Inactivity : 31d 4h:14m:45s
NAC Result : Unknown
VLAN Mapping : N/A VLAN : none
...
...
...

The strange thing about this was that there was indeed an idle-timeout configured for DfltGrpPolicy:

group-policy DfltGrpPolicy attributes
vpn-idle-timeout 60

The solution provided was to add ssl keepalives on the group-policy. And that had the desired effect. After adding the commands below, there were no more stale sessions:

group-policy DfltGrpPolicy attributes
webvpn
anyconnect ssl keepalive 300

Strange thing though. The idle-timeout should be enough to kill those sessions. I still havent got any explanation from TAC regarding why the ssl keepalive-command was needed. Anyone?

I have recently implemented a few Cisco Ironport WSA-solutions. When doing a follow-up after the implementation, the customer usually reacts with “Oh… WSA? We forgot about that. It probably works…”

But what difference does it make? If the customer forgets about their web proxy, what good does it do? Lets have a look at an implementation…

I asked one of our customers for permission to peek into their WSA for the purpose of this blog post. This customer has a few hundred users and is a fairly traditional type of user with mostly office users, each with a personal computer. This customer doesnt limit web browsing, except for filtering out access to known obviously bad web categories like child porn. Except for that, free access to the Web.

Fig1: General Statistics

The first thing to look at is an overview of web activity above. The average web traffic an business day is roughly one million is a working day consists of one million web requests. A web page contains several objects (images, scripts) where each object needs to be requested individually. In this implementation the clients generates 1 million transactions (requests) per day, or 20 million transactions per month.

But what is the content of the requested material? If we look at But WHAT users to surf? If you then look at the purity of operations as it starts to get interesting for real!

Fig2: Purity

Here you can see that just over 10,000 (10.6K) transactions have been stopped this month because of URL category! That is, such as child porn! There are objects (pages, images, etc.) that the user consciously or unconsciously sought but that the system has already been blocked at the access-trial because the source is known and undesirable.

One can also see that almost 3,000 (2.797) object has been blocked due to malware detection. Remember that the WSA scans all through traffic for known viruses, scripts, or other type of malware. The source category has been approved or unknown the WSA have downloaded content. But when checking the contents, they have discovered something unwanted. This little fella has thus stopped nearly 3,000 viruses in the past month!

Overall, 99.8% of web traffic this month has been “clean”. 0.2% may seem to be disappearing bit, but it is still almost 34 000 (33.8K) potentionella threat that was blocked already at the front door!

If you want more detailed information about the type of threat blocked, you can obviously get it also:

Fig3: Malware

With the help of the dynamic Sender Base system scored all websites on the internet. Based on a number of factors such as known virus outbreak or the credibility of a domain, each site a web reputation score from -10 to +10. WSA is configured to always block the sources with the lowest score and always allow the web site with the highest score. But how does this when in reality?

Fig4: Web Reputation

Here we can see that nearly 10,000 transactions in the last month blocked because of Web reputation.

The conclusion I draw every time I look at this type of reporting is that the WSA is blocking lots of web traffic in the covert, and it’s surprisingly rare that users react to the IT department because they can not browse to a specific site. It may be that the user deliberately tries to make stupid mistakes on the internet, but my experience and absolute conviction is that it almost always is something that happens unconsciously. A link to an email or on facebook that look “nice”, but takes the user to a  malware site in some obscure corner of the Internet.

Key figures for this particular device, a typical month “at work”:

• The number of transactions: 20.4 million pieces.
• The number of blocked transactions: 33 800 pcs.
• The number of blocked Malwares / viruses: 2797 pcs, or one every 3 minutes during business hours!
• Dare you not to check the content of your web traffic?

The problem

have you ever had an open TAC case with Cisco, just waiting for them to provide either a solution or some other kind of feedback, and all that happens is that the TAC engineer sends you an email telling you that they “have work in progress” or something else not-making-the-case-evolve?

If so, I guess you have seen that the moment the engineer sends you an email, you also get a case update email telling you that the case has changed status to “customer pending”.

And that is a bit evil. I am pretty sure that more often than not, the reason for the engineer to send that email to you is not to tell you something, but to to actually change the case status. I have a feeling that the engineers effeciency is measured in how long the case is “Cisco pending” and as soon as the case is put over to the customer side, it is “all cool”. just like throwing a burning ball between two perssons. Or like a chess-clock that measure the time spent on each side.

The solution

The best way to handle this is to get even with their own weapons. Last week I had a mail dialogue with TAC that looked like this:

TAC: we are working on the information You sent. we will get back to you tomorrow.
[case status: Customer pending]

Me: thank you very much, I appretiate it.
[case status: customer updated -> Cisco pending]

TAC: you are welcome. have a nice day.
[case status: Customer pending]

Me: you too…
[case status: customer updated -> Cisco pending]

TAC: thank you very much!
[case status: Customer pending]

Me: please do not answer this email, since it changes the status of the case to “Customer pending”, which does NOT reflect the current situation.
[case status: customer updated -> Cisco pending]

I won!!!

When purging and cleaning ancient posts I found this post where I wished everyone a Happy New 2011. And I felt that it was time for an update.

So, what happened during 2011 – did I become a Cisco CCIE Security? The short answer is: No.

In february 2011 my written CCIE Security exam expired. Shortly after that my CCNA/CCNP/CCSP/whatever certifications also was about to expire, and to prevent that from happen I passed the CCIE Security Written once more. So, that means that I have another 18 (like 12 from now) months to do another Lab attempt.

During 2011 there was no way that I could find enough time to study for the lab. Primary of course because of the general work load, but also was my schedule filled with cool projects. Not only have I continued my journey to teach (I have made  my own study material on which 2 different Cisco ASA-workshops were based), I have also done a lot of implementations of Cisco ACS5 and 802.1x, and lately a few ISE-implementations as well.

So, will I ever get that CCIE number? I dont know, but I will continue to try. I have recently purchased the “Ultimate CCIE Security Self Paced bundle” from INE as a complement to the material I already have from IPExpert. I find a few hours every now and then and try to focus to gain the speed/accuracy needed for the dreaded exam.

Stay tuned, I´ll be back.

/Jimmy

I am fan of RSS readers. I use Google Reader all the time to keep track of interresting blog and news sites. Actually, i rarely visit blog sites direct, just from my RSS reader. And I love it.

But there are a few really good blogs that are configured not to post the full blog posts in their RSS stream. And this sucks. Here is an example:

Screen dump of Router Freak blog from RSS Reader

What happens when I come to these entries is either:

1. I read the ingress of the blog post. Find it really interresting and click the header that links me away from my RSS reader to the actual site where I continue to read ‘the full story’.
2. I read the ingress. Find it (probably, because the feed is in my reader) somewhat readworthy but doesnt care about reading the full post because that will link me away from the reader.

What happens more and more often is #2 above. And that´s sad. Because I really like to read what good bloggers writes. But I wanna do it in my reader.

So please, configure your RSS feed to contain the text of the ENTIRE blog post, not just the first x bytes… If it is more interresting for you to have me seeing your ad-banners on your page (which I only do if i make a ‘real’ visit) than it is for you to have me read your content, sorry You´ve lost me as a reader.