The steps for this are the easiest to do:

1. Install the Cisco IPSec Client on the laptop.
2. Enroll with the CA  to download the certificates.
3. Configure the client to connect to the remote server with the certificate.

This is the “Ez” part of EzVPN.  But if grasshopper isn’t careful about the configuration, all unhappiness will befall upon the network engineer.  First, download the client from Cisco’s website and install it.  This is a simple program to install, so I am not going to go through the grizzly details.  But a word of caution.  If you are doing testing in a lab, use a 32 bit version of XP.  There is a version of the client that will work on 64 bit Vista and Windows 7, but it is not worth trying to troubleshoot any issues with that particular client when doing a lab.  Just use XP and call it good.  64 bit XP will not work at all.

After the client is installed, fire it up.  You will need to get the certificates.  Go to Certificates and choose Enroll

Now enter the fields as shown.  Note the /cgi-bin/pkiclient.exe at the end of the enroll url.  This was not needed on the router, but it is needed on the VPN client.

The next bit is crucial.  The OU MUST have the name of the VPN group that is defined on the router.  Without it, the VPN connection will fail.

Click enroll!

Yay!  Now configure the client to connect to the EzVPN server:

Note that we use the certificate that we just downloaded.  Click on save, and the connection should now be in the main window of the VPN client.

Double click on the connection entry.  You will be prompted for the username and password.  We have a local username and password on the EzVPN server of ccie/ccie.

Note that we have the save password checkbox available.  In the EzVPN server configuration, we used the save-password option in the EzVPN client profile.  Enter the password of ccie, and we get authenticated.  After being authenticated, the little status bar in the lower left will change to “Securing Channel…”, and then the VPN will disappear into the taskbar.  There will be a little icon of a lock next to the clock in XP.  If it is closed, you are authenticated!

Time to verify.  Double click on the lock icon, and that will bring back the main window.  Click on Status, then click on Statistics.

Now we run a ping from a command window in XP to 192.168.1.1.  The results:

Note that we did four pings, and the result was four encrypted and decrypted packets.  The ping went through the tunnel!  But what if we ping the 10.10.30.3 interface on R3?

No increase in the encrypted and decrypted packets.  This is because the split tunnel was defined on the EzVPN router as going to the 192.168 networks.  10.10.30.3 was not defined as interesting, so that traffic goes out the interface unencrypted.  In the real world, this would be used to allow a remote user to use the Internet at home and only sending traffic that needs to go to the corporate network over the tunnel.  This split tunnel can be verified in the route details tab.

Note the routes to the 192.168 networks, but none listed for the 10 network.  If no split tunnel is defined, then the route details would come up as all zeros (0.0.0.0).

Verification is complete, as well as the full EzVPN lab.  Yay!

Good luck!

But I digress.

In my award-seeking earlier post, I discussed how to setup a CA server with an IOS router.  I then enrolled two additional routers with the CA server and they now have signed certificates.  Those very certificates can now be put to use by creating an EzVPN connection using those very certificates rather than pre-shared keys for authentication.  First, though, the obligatory lab diagram:

Here, R2 is going to connect to R3.  R2 is going to be the EzVPN client, and R3 is going to be the EzVPN server.  But what about that puffy little cloud?  That is a connection to a Windows XP box that is running the Cisco IPSec Client.  The IPSec client is going to enroll with the CA Server (R1) and connect to R3 as a remote connection.  A little cheating must be done, however.  R3 can directly connect to the XP machine, and because we have OSPF running, the XP machine will be able to connect to everything off the bat.  In order to test, a ping will be sent from the loopback of R1 to the IP address assigned to the VPN from the VPN pool.

So why use EzVPN?  On an IOS router, EzVPN allows a remote IPSec client, whether it is an ASA, router, or the Cisco IPSec VPN Client to connect “on the fly” to the EzVPN server.  This doesn’t make much sense with most site-to-site VPN connections with routers, where other L2L connections, such as VTI, DMVPN, or GET VPN can be used.  It also doesn’t scale well with a very large internal environment.  For users on the road, however, this allows a connection to the network from nearly anywhere.  Also, for home users that have a DHCP address assigned by their broadband ISP, they can use a small router or a small ASA to connect as a client when connectivity is needed to the main office.  Keep the following in mind: If the customer is a large company that has many remote users, then using an IOS router is not the best bet for remote users.  In such a case, and ASA device provides a much more feature-rich solution.  But, for the small office that has maybe a couple of workers that needs remote access, this could do.  And it must do for passing the CCIE:Security lab.  So we do.

There are multple ways to setup the EzVPN server.  The example here will focus on using a DVTI.  Why DVTI?  DVTI are virtual dynamic tunnels that are similar to VTI tunnels, so they offer the benefits of VTI, such as the sending of routing protocols, configuration of policy, and determining encrypted traffic through routing rather than by interface.  The biggest benefit is that it sets a static route in the table that can then be redistributed through any IGP, which allows the entire network to know the new connection “on the fly”.  This is done by configuring a Virtual Template that is then cloned for each new connection created by various EzVPN clients.

Here are the steps for creating the EzVPN server:

1. Turn on AAA.
2. Define the ISAKMP policy.
3. Create a local username/password.  A RADIUS or TACACS+ server can also be used, but this example will use a local username.
4. Set the crypto identity to use distinguished names (DN).  This is needed for certificates in our example.
5. Create the ISAKMP client configuration group.
6. Create the ISAKMP profile.
7. Create an IPSec transform set.
8. Create an IPSec profile.
9. Configure the Virtual-Template interface.
10. Configure an local pool of addresses for the VPN.
11. Configure an access-list for a split tunnel (optional).

And that’s it!  Uh…yeah…

In our example, R3 is the server, so the following is configured on R3.  I am going to put all the steps in at once:

aaa new-model
!
!
aaa authentication login VPN local
aaa authorization network VPN local
!
username ccie password 0 ccie
!
crypto isakmp policy 10
 encr aes
 group 2
crypto isakmp identity dn
!
crypto isakmp client configuration group VPN_GROUP
 key cisco
 dns 192.168.1.1
 domain kidvelvet.local
 pool VPN_POOL
 acl VPN_SPLIT_TUNNEL
 save-password
 split-dns kidvelvet.local
crypto isakmp profile VPN_PROFILE
 match identity group VPN_GROUP
 client authentication list VPN
 isakmp authorization list VPN
 client configuration address respond
 virtual-template 1
!
!
crypto ipsec transform-set STRONG esp-aes esp-sha-hmac
!
crypto ipsec profile IPSEC_PROFILE
 set transform-set STRONG
 set isakmp-profile VPN_PROFILE
!
interface Virtual-Template1 type tunnel
 ip unnumbered FastEthernet1/0
 tunnel source FastEthernet1/0
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile IPSEC_PROFILE
!
ip local pool VPN_POOL 172.31.255.1 172.31.255.254
!
ip access-list extended VPN_SPLIT_TUNNEL
 permit ip 192.168.0.0 0.0.255.255 any
 permit ip 192.168.1.1 0.0.0.0 any
 permit ip 192.168.2.2 0.0.0.0 any
 permit ip 192.168.3.3 0.0.0.0 any

Take notice at the Virtual-Template in the ISAKMP profile.  While the order shown is the order that is in the router configuration file, if you do not configure the Virtual Template before adding it to the ISAKMP profile, a rejection will occur, saying that it isn’t configured.  And while I am on the subject of configuring the Virtual-Template, make sure that the type tunnel is specified at the end.  I accidentally created a Virtual-Template without doing the type tunnel, and I was unable to remove the Virtual-Template.  The only recourse to remove the interface was to reboot the router.  Not good.  Time is valuable in the CCIE lab, and rebooting routers is not one of the best ways to spend it.

The save-password option is needed for the client to connect properly.  Otherwise, other IOS routers will not be able to use a configured password to connect.  Again…not good.

I also had to change the split tunnel to add specific routes.  Why?  OSPF is also advertising specific routes.  If the routes are less specific than the routing protocol, then the routing protocol will win the longest match first battle royale.  This means that the original traffic will not be encrypted across the tunnel, but the return traffic will be.  That’s not what we want.  In a production environment, we would simply add the VPN network as another network on the router and then advertise.  But playing around in a lab may not always lead to best practice.  I know the CCIE lab doesn’t always ask to follow best practice.

Finally, the key option is needed only for non-certificate configurations.  It is left over from some previous testing.  But if EzVPN is configured with pre-shared keys, then this line is necessary.

I made the VPN local pool unique to the network.  Each router interface uses the 10.10.<vlan>.<router number> IP scheme.  Each loopback uses a 192.168.<router number>.<router number> IP scheme.  So, the 172.16 block of private IP addresses will distinguish this traffic as being VPN traffic.  By using the split tunnel, I am only encrypting traffic going from the VPN to the loopbacks, but not to the interfaces.  This will be crucial in testing.

The EzVPN client for IOS is much simpler.  Here are the steps:

1. Create the ISAKMP policy.
2. Set the ISAKMP identity to DN.
3. Create the IPSEC client profile.
4. Create the virtual template.
5. Apply the client to an outside and inside interface.

MUCH easier!

So remember when I made those certificates in that past blog post?  Well, that certificate isn’t going to work.  The reason?  When connecting to an EzVPN server, the VPN client needs to somehow identify with the VPN group that it will attach.  A server can contain multiple tunnel groups, and each one will have a different group name.  When using a pre-shared key, this can be done by listing the group in the client and then listing the key after the group name.  However, a certificate is being used instead of a key.  In order to associate the certificate with a key, there needs to be something in the certificate that identifies the group name.  That would be the OU, or Organizational Unit.  You put the group name in the OU when associating with the trustpoint, and that will associate with the proper EzVPN group profile.

Because the certificate was already downloaded, we need to remove it.  This is done by removing the trustpoint.  A second trustpoint could be created, but let’s not muddle up the configuration on the router.  So to remove the trustpoint:

R2(config)#no crypto ca trustpoint kidvelvet
% Removing an enrolled trustpoint will destroy all certificates
 received from the related Certificate Authority.

Are you sure you want to do this? [yes/no]: yes
% Be sure to ask the CA administrator to revoke your certificates.

R2(config)#

And the trustpoint and certificates are now gone.  So let’s create a new trustpoint with the proper OU.  The group on R3 is called VPN_GROUP, so we will create a trustpoint that has the OU=VPN_GROUP attached to it.

R2(config)#crypto pki trustpoint kidvelvet
R2(ca-trustpoint)# enrollment url http://10.10.10.1:80
R2(ca-trustpoint)# subject-name OU=VPN_GROUP
R2(ca-trustpoint)# revocation-check none
R2(ca-trustpoint)#exit
R2(config)#

Now authenticate against R1 and enroll:

R2(config)#crypto ca authent kidvelvet
Certificate has the following attributes:
 Fingerprint MD5: 393B5E2E 740B2173 411BBCC2 BE9B9955
 Fingerprint SHA1: 17501051 B95BF739 39B7A116 8A12B7A3 EFC9B5E9 

% Do you accept this certificate? [yes/no]: yes
Trustpoint CA certificate accepted.
R2(config)#crypto ca enroll kidvelvet
%
% Start certificate enrollment ..
% Create a challenge password. You will need to verbally provide this
 password to the CA Administrator in order to revoke your certificate.
 For security reasons your password will not be saved in the configuration.
 Please make a note of it.

Password:
Re-enter password: 

% The subject name in the certificate will include: OU=VPN_GROUP
% The subject name in the certificate will include: R2.kidvelvet.local
% Include the router serial number in the subject name? [yes/no]: no
% Include an IP address in the subject name? [no]: no
Request certificate from CA? [yes/no]: yes
% Certificate request sent to Certificate Authority
% The 'show crypto ca certificate kidvelvet verbose' commandwill show the fingerprint.

R2(config)#
Feb 16 23:54:59.291: CRYPTO_PKI:  Certificate Request Fingerprint MD5:
 3A1A1955 4F193A09 1F136673 F48F1B4C
Feb 16 23:54:59.299: CRYPTO_PKI:  Certificate Request Fingerprint SHA1:
 F2725BDF 8C195B48 253B39CE 979A4FD1 DFEE69CC
R2(config)#
Feb 16 23:55:01.451: %PKI-6-CERTRET: Certificate received from Certificate Authority
R2(config)#

Note that the certificate was automatically enrolled.  I added the grant auto command to the pki server.  This is needed for when we enroll the IPSec Client on the XP box.

Now the rest of the configuration:

crypto isakmp policy 10
 encr aes
 group 2
crypto isakmp identity dn
!
!
!
!
!
crypto ipsec client ezvpn EZVPN
 connect auto
 mode client
 peer 10.10.30.3
 virtual-interface 1
 username ccie password ccie
 xauth userid mode local
!
interface FastEthernet0/0
 ip address 10.10.20.2 255.255.255.0
 speed 100
 full-duplex
 crypto ipsec client ezvpn EZVPN
!
interface FastEthernet0/1
 ip address 10.10.40.2 255.255.255.0
 ip dns view-group ezvpn-internal-viewlist
 duplex auto
 speed auto
 crypto ipsec client ezvpn EZVPN inside
!
interface Virtual-Template1 type tunnel
 ip unnumbered FastEthernet0/0
 tunnel mode ipsec ipv4

The client is now installed.  Briefly after installing the client, some output will appear that will show the client is active:

R2(config)#
Feb 17 00:12:27.177: %CRYPTO-6-EZVPN_CONNECTION_UP: (Client)  User=ccie
 Group=  Server_public_addr=10.10.30.3  Assigned_client_addr=172.31.255.64
R2(config)#
Feb 17 00:12:27.241: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to up
Feb 17 00:12:27.829: %LINK-3-UPDOWN: Interface Loopback10000, changed state to up
Feb 17 00:12:28.241: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2,
 changed state to up

This tells us that the client is up, it has been assigned the IP address of 172.31.255.64 from the pool, and a couple of new interfaces have appeared.  The Virtual-Access2 interface is the actual DVTI, and the loopback10000 interface is assigned the 172.31.255.64 address.  Hmm…sounds like things are working.  Verification is in order:

R2#sh crypto isakmp sa det
Codes: C - IKE configuration mode, D - Dead Peer Detection
 K - Keepalives, N - NAT-traversal
 X - IKE Extended Authentication
 psk - Preshared key, rsig - RSA signature
 renc - RSA encryption
IPv4 Crypto ISAKMP SA

C-id  Local           Remote          I-VRF    Status Encr Hash Auth DH Lifetime Cap.

1018  10.10.20.2      10.10.30.3               ACTIVE aes  sha  rsig 2  23:54:27 CX
 Engine-id:Conn-id =  SW:18

IPv6 Crypto ISAKMP SA

R2#

The key here is the ACTIVE state, with the aes encryption, sha hash, and rsig for authentication.  The rsig means that the certificate is being properly used.  Now the routes:

R2#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

 172.31.0.0/32 is subnetted, 1 subnets
C       172.31.255.64 is directly connected, Loopback10000
 10.0.0.0/24 is subnetted, 4 subnets
O       10.10.10.0 [110/20] via 10.10.20.3, 20:52:19, FastEthernet0/0
C       10.10.20.0 is directly connected, FastEthernet0/0
O       10.10.30.0 [110/20] via 10.10.20.3, 20:52:19, FastEthernet0/0
C       10.10.40.0 is directly connected, FastEthernet0/1
 192.168.22.0/32 is subnetted, 1 subnets
C       192.168.22.2 is directly connected, Loopback10
 192.168.1.0/32 is subnetted, 1 subnets
S       192.168.1.1 [1/0] via 0.0.0.0, Virtual-Access2
 192.168.2.0/32 is subnetted, 1 subnets
C       192.168.2.2 is directly connected, Loopback0
 192.168.3.0/32 is subnetted, 1 subnets
S       192.168.3.3 [1/0] via 0.0.0.0, Virtual-Access2
S    192.168.0.0/16 [1/0] via 0.0.0.0, Virtual-Access2
R2#

Routes have been added statically that use the Virtual-Access2 interface.  Nice!  After a ping test to 192.168.1.1, we see the following IPSec traffic:

R2#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/91/224 ms
R2#sh crypto ipsec sa

interface: Virtual-Access2
 Crypto map tag: Virtual-Access2-head-0, local addr 10.10.20.2

 protected vrf: (none)
 local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
 remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
 current_peer 10.10.30.3 port 500
 PERMIT, flags={origin_is_acl,}
 #pkts encaps: 7, #pkts encrypt: 7, #pkts digest: 7
 #pkts decaps: 61, #pkts decrypt: 61, #pkts verify: 61
 #pkts compressed: 0, #pkts decompressed: 0
 #pkts not compressed: 0, #pkts compr. failed: 0
 #pkts not decompressed: 0, #pkts decompress failed: 0
 #send errors 0, #recv errors 0

 local crypto endpt.: 10.10.20.2, remote crypto endpt.: 10.10.30.3
 path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
 current outbound spi: 0xA4DE3E5B(2766028379)

 inbound esp sas:
 spi: 0x4C64298A(1281632650)
 transform: esp-aes esp-sha-hmac ,
 in use settings ={Tunnel, }
 conn id: 103, flow_id: SW:103, crypto map: Virtual-Access2-head-0
 sa timing: remaining key lifetime (k/sec): (4555185/3075)
 IV size: 16 bytes
 replay detection support: Y
 Status: ACTIVE

 inbound ah sas:

 inbound pcp sas:

 outbound esp sas:
 spi: 0xA4DE3E5B(2766028379)
 transform: esp-aes esp-sha-hmac ,
 in use settings ={Tunnel, }
 conn id: 104, flow_id: SW:104, crypto map: Virtual-Access2-head-0
 sa timing: remaining key lifetime (k/sec): (4555192/3075)
 IV size: 16 bytes
 replay detection support: Y
 Status: ACTIVE

 outbound ah sas:

 outbound pcp sas:
R2#

Verification is complete.  Traffic is being encrypted and decrypted.  Now on to the puffy cloud…er, XP machine.  Ok, I think that is going to be saved for an additional post…

Good luck!

For this exercise, the layout will be used for a future lab involving the not-so EzVPN.  Instead of using pre-shared keys for the EzVPN, certificates are going to be used instead.  But to use those certificates will require a CA server and enrollment to the server.  Here is the current lab:

R1 is going to be the CA Server, while R2 and R3 are going to enroll in the CA server.  R2 will then connect to R3 using EzVPN (which can be saved for another post).

So what exactly happens with a CA server?  A CA server is a Certificate Authority, which means that it has the authority to sign certificates that are used by members of the domain (or any device that it grants a certificate).  When a device installs a CA certificate, that device says that it will trust any other certificate that was signed by that server.  Therefore, if there are five devices that are talking to each other, and each device has the same CA certificate installed, then they all know that certificates that were signed by the CA can be trusted.  If the certificate was not signed by the CA, it is not considered trusted.

The following steps will need to be taken:

1. Configure NTP on the server and clients.
2. Generate exportable keys on the CA server.
3. Generate a public/private key pair.
4. Create a database to store the certificate.
5. Configure the PKI server.
6. Start the PKI server, which generates the CA Certificate.
7. Generate non-exportable keys on each client.
8. Create a trustpoint on each client that points to the CA Server.
9. Authenticate against the CA server.
10. Enroll with the CA server.
11. Grant requests on the server for each enrollment request.

NTP MUST be configured and running on all clients and the PKI server.  The certificates are time based, and not getting this done first will result in some real nastiness.  For the purposes of this lab, NTP has already been configured.

On R1, we generate exportable keys:

R1(config)#crypto key gen rsa gen label kidvelvet exportable
The name for the keys will be: kidvelvet
Choose the size of the key modulus in the range of 360 to 2048 for your
 General Purpose Keys. Choosing a key modulus greater than 512 may take
 a few minutes.

How many bits in the modulus [512]: 1024
% Generating 1024 bit RSA keys, keys will be exportable...[OK]

R1(config)#
Feb 14 23:44:03.664: %SSH-5-ENABLED: SSH 1.99 has been enabled

If you use the default 512 modulus, SSH 1.5 will be enabled (which is version 1 only).  Version 1.99 is Cisco parlance for both versions 1 and 2.  Yeah, I know…

Now the keys are exported into a PEM format int nvram using either DES or 3DES encryption.  Please do not use DES.  You might as well be encrypting it with a Cap’n Crunch decoder ring.  Anyway…

R1(config)#crypto key export rsa kidvelvet pem url nvram: 3des cisco123
% Key name: kidvelvet
 Usage: General Purpose Key
Exporting public key...
Destination filename [kidvelvet.pub]?
Writing file to nvram:kidvelvet.pub
Exporting private key...
Destination filename [kidvelvet.prv]?
Writing file to nvram:kidvelvet.prv

Verify the keys:

R1(config)#do sh crypto key mypubkey rsa
% Key pair was generated at: 23:44:03 UTC Feb 14 2011
Key name: kidvelvet
 Storage Device: not specified
 Usage: General Purpose Key
 Key is exportable.
 Key Data:
 30819F30 0D06092A 864886F7 0D010101 05000381 8D003081 89028181 00A64D6E
 CBE95BF2 66AAEDA2 8E12A033 08B6A4A1 CE86E1E3 7698B146 04AF26FC 697CBAA1
 E28F0965 BA61FC01 21C9D743 45BF3C7A 8D11A5AE F6198027 41ED7477 104AE448
 8CA5B40A CB932829 FA9319B2 B0931950 F797654F 8267C257 06A6EAB1 A5F36BED
 BEF073AB DB4D11C5 1F7EE5E9 9227ED4B 5C7D072C ED568559 3D02B14D 11020301 0001
% Key pair was generated at: 23:44:05 UTC Feb 14 2011
Key name: kidvelvet.server
Temporary key
 Usage: Encryption Key
 Key is not exportable.
 Key Data:
 307C300D 06092A86 4886F70D 01010105 00036B00 30680261 00A1F93C FC636CC3
 646FB928 15398065 4D0725B1 40D1D7DD 9AADD9F3 09A55C62 CFB710B2 F910E65F
 3F193814 52412C49 A00CE841 4A303AD0 E5263B2C BFCE32BE 994ADB67 A4710C30
 0E61F5BF A4418CEB 94D118E9 049A5979 762179AF 3D6B28C8 6F020301 0001

So far, so good.  Turn on the HTTP server so that the clients can enroll:

R1(config)#ip http server

Now we are ready to create the PKI server.  This is where the certificate will be generated for the CA Server.  This certificate will be used at the client trustpoints to verify that the local certificates were signed by a trusted authority.

R1(config)#crypto pki server kidvelvet
R1(cs-server)#database url nvram:
% Server database url was changed. You need to move the
% existing database to the new location.
R1(cs-server)#data level min
R1(cs-server)#issuer-name CN=kidvelvet.kidvelvet.local L=Hillsboro C=US
R1(cs-server)#lifetime ca-ser
R1(cs-server)#lifetime ca-cer
R1(cs-server)#lifetime ca-certificate 365
R1(cs-server)#no shut
%Some server settings cannot be changed after CA certificate generation.
% Please enter a passphrase to protect the private key
% or type Return to exit
Password: 

Re-enter password:
% Exporting Certificate Server signing certificate and keys...

% Certificate Server enabled.
R1(cs-server)#
Feb 14 23:46:53.236: %PKI-6-CS_ENABLED: Certificate server now enabled.

Our CA Server is ready and open for business!

The clients are a bit easier to configure.  First, the keys must be generated.  These do not need to be exportable.

R3(config)#crypto key gen rsa
% Please define a domain-name first.

Oh, yeah…there is this thing that must be done before generating keys.  Define a host name and domain name.  At least Cisco said “please”…

R3(config)#hostname R3
R3(config)#ip domain-name kidvlevet.local
R3(config)#crypto key gen rsa
The name for the keys will be: R3.kidvlevet.local
Choose the size of the key modulus in the range of 360 to 2048 for your
 General Purpose Keys. Choosing a key modulus greater than 512 may take
 a few minutes.

How many bits in the modulus [512]: 1024
% Generating 1024 bit RSA keys, keys will be non-exportable...[OK]
R3(config)#
Feb 15 00:49:51.732: %SSH-5-ENABLED: SSH 1.99 has been enabled

Version 1.99 is up…keys generated…

The trustpoint is the next step.  There are some options here, but the main option is the enrollment url:

R3(config)#crypto ca trustpoint kidvelvet
R3(ca-trustpoint)#enroll url http://10.10.10.1:80
R3(ca-trustpoint)#rev none
R3(ca-trustpoint)#exit

Next, authenticate against the CA Server defined in the trustpoint to download the CA certificate:

R3(config)#crypto ca auth kidvelvet
Certificate has the following attributes:
 Fingerprint MD5: 393B5E2E 740B2173 411BBCC2 BE9B9955
 Fingerprint SHA1: 17501051 B95BF739 39B7A116 8A12B7A3 EFC9B5E9 

% Do you accept this certificate? [yes/no]: yes
Trustpoint CA certificate accepted.

Got the cert, now it is time to enroll and receive a cert for the client:

R3(config)#crypto ca enroll kidvelvet
%
% Start certificate enrollment ..
% Create a challenge password. You will need to verbally provide this
 password to the CA Administrator in order to revoke your certificate.
 For security reasons your password will not be saved in the configuration.
 Please make a note of it.

Password:
Re-enter password: 

% The subject name in the certificate will include: R3.kidvlevet.local
% Include the router serial number in the subject name? [yes/no]: no
% Include an IP address in the subject name? [no]: no
Request certificate from CA? [yes/no]: yes
% Certificate request sent to Certificate Authority
% The 'show crypto ca certificate kidvelvet verbose' commandwill show the fingerprint.

R3(config)#
Feb 15 00:51:01.504: CRYPTO_PKI:  Certificate Request Fingerprint MD5:
 FE071EED 0069A988 F75597AA E1C876D3
Feb 15 00:51:01.508: CRYPTO_PKI:  Certificate Request Fingerprint SHA1:
 FAE6C472 B328BDA1 64EF24CC A615B7B5 ACF8FB6C
R3(config)#

Verify the certificates:

R3(config)#do sh crypto ca cert
Certificate
 Status: Available
 Certificate Serial Number: 0x4
 Certificate Usage: General Purpose
 Issuer:
 cn=kidvelvet.kidvelvet.local L=Hillsboro C=US
 Subject:
 Name: R3.kidvlevet.local
 hostname=R3.kidvlevet.local
 Validity Date:
 start date: 00:51:24 UTC Feb 15 2011
 end   date: 23:46:52 UTC Feb 14 2012
 Associated Trustpoints: kidvelvet 

CA Certificate
 Status: Available
 Certificate Serial Number: 0x1
 Certificate Usage: Signature
 Issuer:
 cn=kidvelvet.kidvelvet.local L=Hillsboro C=US
 Subject:
 cn=kidvelvet.kidvelvet.local L=Hillsboro C=US
 Validity Date:
 start date: 23:46:52 UTC Feb 14 2011
 end   date: 23:46:52 UTC Feb 14 2012
 Associated Trustpoints: kidvelvet 

R3(config)#

We have both of the certificates.  However, I had to do a little magic to make it happen.  With the current configuration, the certificate will be placed in a “pending” state, waiting for the CA Server to grant the cert.  On the CA Server, the following will show a list of certificate requests:

R1#crypto pki server kidvelvet info requests
Enrollment Request Database:

Subordinate CA certificate requests:
ReqID  State      Fingerprint                      SubjectName
--------------------------------------------------------------

RA certificate requests:
ReqID  State      Fingerprint                      SubjectName
--------------------------------------------------------------

Router certificates requests:
ReqID  State      Fingerprint                      SubjectName
--------------------------------------------------------------
3      pending    FE071EED0069A988F75597AAE1C876D3 hostname=R3.kidvlevet.local
1      granted    C15C833ED28A14D331AE40146C24A1C0 hostname=R2.kidvelvet.local

Here, R3 is in a pending state.  R3 has sent the enrollment request, but the request needs to be granted:

R1#crypto pki server kidvelvet grant 3

The request ID of 3 is granted, and the resulting output:

R1#crypto pki server kidvelvet info requests
Enrollment Request Database:

Subordinate CA certificate requests:
ReqID  State      Fingerprint                      SubjectName
--------------------------------------------------------------

RA certificate requests:
ReqID  State      Fingerprint                      SubjectName
--------------------------------------------------------------

Router certificates requests:
ReqID  State      Fingerprint                      SubjectName
--------------------------------------------------------------
3      granted    FE071EED0069A988F75597AAE1C876D3 hostname=R3.kidvlevet.local
1      granted    C15C833ED28A14D331AE40146C24A1C0 hostname=R2.kidvelvet.local

Now that it is granted, the certificate shows up in the Available status rather than the Pending status.  A festivus miracle!  Our certs are now ready to use with any VPN technology that comes our way…including the dreaded EzVPN.

Good luck!

GET VPN uses the Group Domain of Interpretation (GDOI) specification that is defined in RFC 3547.  Unlike other site to site VPNs where the endpoints manage both phase I and phase II of the VPN, GDOI uses the concept of a Key Server (KS) to control data and control plane traffic between VPN endpoints.  The need for GDOI comes from having an internal network (such as MPLS) that requires encryption on the end points.  Traditional site to site VPNs do not scale very well, and DMVPN can take longer to rekey, causing some delay in setting up phase II traffic.  DMVPN uses a sliding window to prevent anti-replay attacks, creating the delay.  GDOI uses a time-based window that prevents attacks but speeds up the rekey process.  This is because all group members in a GDOI use the same security SA (common policy), so a sliding window will not even work.

GDOI uses the original IP header when tunneling traffic, so it works well in a private environment.  However, because private IP addresses are blocked on the Internet in accordance to RFC 2827, GDOI does not work well in a publicly routet environment, which is where DMVPN is most suited.  If the DMVPN is being used in a private environment (such as an environment that utilizes multiple frame relays without MPLS), then GDOI can be used in conjunction with DMVPN…but this is a pretty rare case.  That will be saved for a later post.

So how does GET VPN work?  First, there are key servers on the network that store both the phase I and phase II information.  There can be one key server on a network, but if the key server were to fail, so would the VPNs, which may create a Career Limiting Event (CLE).  In order to allow network administrators and engineers to keep their jobs, the concept of a KS COOP was developed.  This allows multiple servers to act as key servers on the network.  If one were to fail, the other would maintain the current SA and become responsible for rekey responsibilities.  This is done by exporting the key of the primary KS and importing it into the secondary KS.

The keys are exhanged between the KS and the Group Member (GM).  There are two types of keys.  First, there is the Key Encryption Key (KEK) used for the Re-Key SA, while the Traffic Encryption Key (TEK) is used for the encryption of the traffic using the Data Security SA.  The initial exchange is a pull from the GM to the KS, and the keys are exchanged.   This is called the Request Message.   During the rekey process, the KS pushes the rekey to the GM.  This is called the Push Message.  During the third exchange, the POP and CERT payloads are sent from the GM to the KS, which is called the ACK Message.  Finally, the KS verifies all the data and the HASH message and issues a Key Download Message.

That was a bit of a brief overview.  Check out Cisco’s website or the RFC for a more detailed (and less entertaining) description.  For now, a quick and dirty configuration is in order.  Our network diagram:

Here, R1 and R2 are going to be the group members (GM).  R3 and R4 are going to be the Key Servers (KS).  R3 and R4 are going to be in COOP mode.  All routers are able to route to each other on both the Ethernet and Loopback interfaces by setting up EIGRP and advertising all Ethernet and Loopback networks.

The tunnel between R1 and R2 is going to encrypt any traffic going between the Loopback interfaces.  While in a normal environment we would use the internal networks, this will do for our lab to get the desired results.  However, it will lead to a bit of a pitfall, which will be described later.

First, we should setup the KS routers.  R3 will be the primary KS, while R4 will be the secondary KS.  The steps for setting up the KS are similar to setting up a regular VPN, with some funky additions:

1. Define the ISAKMP policy.
2. Create the ISAKMP keys (pre-shared authentication only).
3. Define the interesting traffic with an ACL.
4. Generate an exportable key (needed for COOP).
5. Create the IPSec transform set.
6. Create the IPSec profile.
7. Create the GDOI group.

The two key points here are generating an exportable key and creating the GDOI group.  This is what differentiates this from other site to site VPNs.  The exportable keys are necessary to create the COOP.  Even if only one KS is going to be used, it is good practice to generate the exportable keys for future use.

R3:

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key cisco address 10.10.10.1
crypto isakmp key cisco address 10.10.20.2
crypto isakmp key cisco address 10.10.30.4
!
!
crypto ipsec transform-set STRONG esp-aes esp-sha-hmac
!
! Next we do the ACL before creating the GDOI groups
!
access-list 101 permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
!
! Next we create the exportable keys
crypto key gen rsa general-keys exportable label GDOI_KEYS
!
! Now the GDOI fun stuff!
!
crypto ipsec profile GDOI
 set security-association lifetime seconds 1800
 set transform-set STRONG
!
crypto gdoi group GDOI_GROUP
 identity number 1
 server local
 rekey retransmit 10 number 2
 rekey authentication mypubkey rsa GDOI_KEYS
 rekey transport unicast
 sa ipsec 1
 profile GDOI
 match address ipv4 101
 replay time window-size 5
 address ipv4 10.10.30.3
 redundancy
 local priority 100
 peer address ipv4 10.10.30.4

If you notice the ACL, we are using what is called a Symmetric ACL.  The defined traffic on both sides of the ACL are identical.  This allows us to encrypt all traffic on the loopback interfaces.  We do NOT want to add the 10.10.0.0/16 network, as that would also try to encrypt traffic between the key servers and the group members.  This will lead to pain.  It also shows the value of using GET VPN in a private environment.  We could use the loopbacks as the endpoints and the source of the key exchanges, while the traffic from all the private networks can talk to each other with a minimal ACL entry.

R4 is the same, but the last part of the GDOI profile is a bit different, along with the ISAKMP keys.  Before getting started, it would be a fine idea to export and import the keys from R3.  First, use the export command to expose the keys to the terminal:

R3(config)#crypto key export rsa GDOI_KEYS pem terminal 3des CISCO1234
% Key name: GDOI_KEYS
 Usage: General Purpose Key
 Key data:
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDAeSciKuAgSdzbR+nNeTpSrGfF
6B7wf8Mk+ZS10iBpxWhFOykksg+ffl6GqSLIqifAplFO0mUQUXmowt6U+90lUkzZ
Uk60n/Z3/6LjyMEMYxTZxzoZYOAw6YBMNGgE45UeWBw40tg/pjmWd8DKgw1uE6hz
0U2x2xPjhbdQmOVk2wIDAQAB
-----END PUBLIC KEY-----
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,1C89433617040F00

48hBYLj3eURGsLZE6jMTDTm1VBGjFQhqSyNElBBH2j6AUqt+Fi5aZXwQ81CX28/u
FWdVRFeQA+auicyR3axE/WbpmWxM8fXBL9aJcT0esEebfaSLVA28hfb1BxNngVvq
5F4GzT32IwYxrRbadeyegx00ndVzQp7vWyOdlbb5I7cRdIALi1Ikk6tOXbnbFehG
bpt9WPWD77YR2UJeKJQmvLfxnBq1wqlVqwLIXGDCk3e2SG0ow6o2n+ekca6qCJ9F
r+Zhyc4XM2zBtVK2yd71jVFyWss5G3Xc5b8mBOupdH6CBAcv7cwOpT2Lv/NnktwX
1FkDicHWJcvDCSf/iEbBKKWHrihu2raZoOJPXgqtgy5l0+jNW8Qt8xbGPaUipnVx
gFePzwrJkVDmuSxdZ4m1cElkxCA5wo0BejHwU+88RW7zEMFV54iAWt0Yjt8NN/PP
CDrqtBCiXJHAUqd1NII6PGRhLee+DokGsWK9GSFKGV/Rf8CYcbCiGSnxNgmx8z7E
Hp/wOMdHN0PG4RqWaCJpPFF9HOAQ5Vt5/YaVZJwe42JQ1zLLHIuW6qepEj4N8J6F
m2+OIth6aojeyJH+xpl4cygUnUT7TitVKmeuBISaW5s5wOVxr/1Lr50c8INgR6uX
56A2b1STmMUMkDshqcuRKBV9Je8kFOEdgPU/Btsp5uX9GC/jPbHp5WvCwlzAAbn7
MrW4T2CAyH5iMq6864pPEDiUMsZALqE6qRcV3x9qLnbpiEdUswT933LnEAmgmCLu
C1FlXZFWpDu6zRZqz1/aNBbrJy1qbmFF0Nqv2ynAbo8iLsY6Ig0MCQ==
-----END RSA PRIVATE KEY-----

R3(config)#

The first line, oddly enough, is done from configuration mode.  The last word in the line is the passphrase for the key for export.  Make sure you use the same word at the end of the import line.  Now to R4 for the import:

R4(config)#crypto key import rsa GDOI_KEYS pem terminal CISCO1234
% Enter PEM-formatted public General Purpose key or certificate.
% End with a blank line or "quit" on a line by itself.
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDAeSciKuAgSdzbR+nNeTpSrGfF
6B7wf8Mk+ZS10iBpxWhFOykksg+ffl6GqSLIqifAplFO0mUQUXmowt6U+90lUkzZ
Uk60n/Z3/6LjyMEMYxTZxzoZYOAw6YBMNGgE45UeWBw40tg/pjmWd8DKgw1uE6hz
0U2x2xPjhbdQmOVk2wIDAQAB
-----END PUBLIC KEY-----
quit
% Enter PEM-formatted encrypted private General Purpose key.
% End with "quit" on a line by itself.
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,1C89433617040F00

48hBYLj3eURGsLZE6jMTDTm1VBGjFQhqSyNElBBH2j6AUqt+Fi5aZXwQ81CX28/u
FWdVRFeQA+auicyR3axE/WbpmWxM8fXBL9aJcT0esEebfaSLVA28hfb1BxNngVvq
5F4GzT32IwYxrRbadeyegx00ndVzQp7vWyOdlbb5I7cRdIALi1Ikk6tOXbnbFehG
bpt9WPWD77YR2UJeKJQmvLfxnBq1wqlVqwLIXGDCk3e2SG0ow6o2n+ekca6qCJ9F
r+Zhyc4XM2zBtVK2yd71jVFyWss5G3Xc5b8mBOupdH6CBAcv7cwOpT2Lv/NnktwX
1FkDicHWJcvDCSf/iEbBKKWHrihu2raZoOJPXgqtgy5l0+jNW8Qt8xbGPaUipnVx
gFePzwrJkVDmuSxdZ4m1cElkxCA5wo0BejHwU+88RW7zEMFV54iAWt0Yjt8NN/PP
CDrqtBCiXJHAUqd1NII6PGRhLee+DokGsWK9GSFKGV/Rf8CYcbCiGSnxNgmx8z7E
Hp/wOMdHN0PG4RqWaCJpPFF9HOAQ5Vt5/YaVZJwe42JQ1zLLHIuW6qepEj4N8J6F
m2+OIth6aojeyJH+xpl4cygUnUT7TitVKmeuBISaW5s5wOVxr/1Lr50c8INgR6uX
56A2b1STmMUMkDshqcuRKBV9Je8kFOEdgPU/Btsp5uX9GC/jPbHp5WvCwlzAAbn7
MrW4T2CAyH5iMq6864pPEDiUMsZALqE6qRcV3x9qLnbpiEdUswT933LnEAmgmCLu
C1FlXZFWpDu6zRZqz1/aNBbrJy1qbmFF0Nqv2ynAbo8iLsY6Ig0MCQ==
-----END RSA PRIVATE KEY-----
quit
% Key pair import succeeded.

R4(config)#

Note how this was imported.  The first key (public) was added, then the word “quit” was typed.  Then instructions came up asking for the private key.  The key was pasted, followed by the word “quit”.  Then, we are done.  Yay!  The keys are now imported successfully, as the boys in Berkeley have just noted.

With keys properly imported, R4 can now be configured:

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key cisco address 10.10.10.1
crypto isakmp key cisco address 10.10.20.2
crypto isakmp key cisco address 10.10.30.3
!
!
crypto ipsec transform-set STRONG esp-aes esp-sha-hmac
!
! Next we do the ACL before creating the GDOI groups
!
access-list 101 permit ip 192.168.0.0 0.0.255.255 192.168.0.0 0.0.255.255
!
! Next we create the exportable keys
crypto key gen rsa general-keys exportable label GDOI_KEYS
!
! Now the GDOI fun stuff!
!
crypto ipsec profile GDOI
 set security-association lifetime seconds 1800
 set transform-set STRONG
!
crypto gdoi group GDOI_GROUP
 identity number 1
 server local
 rekey retransmit 10 number 2
 rekey authentication mypubkey rsa GDOI_KEYS
 rekey transport unicast
 sa ipsec 1
 profile GDOI
 match address ipv4 101
 replay time window-size 5
 address ipv4 10.10.30.4
 redundancy
 local priority 50
 peer address ipv4 10.10.30.3

If you look at the highlighted lines, you will see the ISAKMP key now has the other server in it.  The address in the profile is for R4, the peer address is R3, and the local priority is 50, which relegates it to the secondary server.

The group members are a bit easier to configure.  Here, we have the following steps:

1. Define the ISAKMP policy.
2. Define the ISAKMP keys to the KS.
3. Configure the GDOI group.
4. Configure the crypto map.
5. Apply the crypto map to an interface.

Because all the IPSec information is handled by the KS, the GMs simply need to make an IKE Phase I connection to the key servers.  After the four messages are exchanged, the tunnel is created between all GMs.

R1:

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key cisco address 10.10.30.3
crypto isakmp key cisco address 10.10.30.4
!
crypto gdoi group GDOI_GROUP
 identity number 1
 server address ipv4 10.10.30.3
 server address ipv4 10.10.30.4
!
crypto map GDOI_MAP local-address FastEthernet0/0
crypto map GDOI_MAP 10 gdoi
 set group GDOI_GROUP
!
interface FastEthernet0/0
 ip address 10.10.10.1 255.255.255.0
 speed 100
 full-duplex
 crypto map GDOI_MAP

Thats it!  R2 is configured the same way, except that fa0/0 will have an IP address of 10.10.20.2.  Otherwise, they are the same.  The KS COOP handles all the information about the group members, so the ISAKMP keys only need to have the servers added.  This creates an incredibly scalable solution in a multi-site enterprise environment.  Once the KS routers are configured, the template for each additional group member is a copy/paste solution.  And network people like copy/paste…it increases the ability to get things done with minimal effort.

No configuration is complete without fully verifying functionality.  First, verify that phase I is complete from the GM to the KS:

R1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.10.30.3      10.10.10.1      GDOI_IDLE         1015    0 ACTIVE
10.10.10.1      10.10.30.3      GDOI_REKEY        1016    0 ACTIVE

IPv6 Crypto ISAKMP SA

R1#

R1 looks good.  And R2:

R2#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.10.20.2      10.10.30.3      GDOI_REKEY        1012    0 ACTIVE
10.10.30.3      10.10.20.2      GDOI_IDLE         1011    0 ACTIVE

IPv6 Crypto ISAKMP SA

R2#

Nice.  Send some traffic from the loopback on R1 to the loopback on R2, then look at the IPSec SA:

R1#ping 192.168.2.2 sour 192.168.11.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.11.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 128/144/180 ms
R1#sh crypto ipsec sa 

interface: FastEthernet0/0
 Crypto map tag: GDOI_MAP, local addr 10.10.10.1

 protected vrf: (none)
 local  ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
 remote ident (addr/mask/prot/port): (192.168.0.0/255.255.0.0/0/0)
 current_peer  port 848
 PERMIT, flags={origin_is_acl,}
 #pkts encaps: 10, #pkts encrypt: 10, #pkts digest: 10
 #pkts decaps: 10, #pkts decrypt: 10, #pkts verify: 10
 #pkts compressed: 0, #pkts decompressed: 0
 #pkts not compressed: 0, #pkts compr. failed: 0
 #pkts not decompressed: 0, #pkts decompress failed: 0
 #send errors 0, #recv errors 0

 local crypto endpt.: 10.10.10.1, remote crypto endpt.:
 path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
 current outbound spi: 0x46AD6F53(1185771347)

 inbound esp sas:
 spi: 0x46AD6F53(1185771347)
 transform: esp-aes esp-sha-hmac ,
 in use settings ={Tunnel, }
 conn id: 305, flow_id: SW:305, crypto map: GDOI_MAP
 sa timing: remaining key lifetime (sec): (775)
 IV size: 16 bytes
 replay detection support: Y  replay window size: 5
 Status: ACTIVE

 inbound ah sas:

 inbound pcp sas:

 outbound esp sas:
 spi: 0x46AD6F53(1185771347)
 transform: esp-aes esp-sha-hmac ,
 in use settings ={Tunnel, }
 conn id: 306, flow_id: SW:306, crypto map: GDOI_MAP
 sa timing: remaining key lifetime (sec): (775)
 IV size: 16 bytes
 replay detection support: Y  replay window size: 5
 Status: ACTIVE

 outbound ah sas:

 outbound pcp sas:
R1#

Cool!  We see encrypted traffic and decrypted traffic.

Now verify that the KS COOP server is functioning:

R3#sh crypto gdoi ks coop
Crypto Gdoi Group Name :GDOI_GROUP
 Group handle: 2147483650, Local Key Server handle: 2147483650

 Local Address: 10.10.30.3
 Local Priority: 100      
 Local KS Role: Primary   , Local KS Status: Alive     
 Primary Timers:
 Primary Refresh Policy Time: 20
 Remaining Time: 9
 Antireplay Sequence Number: 2757

 Peer Sessions:
 Session 1:
 Server handle: 2147483655
 Peer Address: 10.10.30.4
 Peer Priority: 50              
 Peer KS Role: Secondary , Peer KS Status: Alive     
 Antireplay Sequence Number: 2751

 IKE status: Established
 Counters:
 Ann msgs sent: 2755
 Ann msgs sent with reply request: 2
 Ann msgs recv: 2749
 Ann msgs recv with reply request: 2
 Packet sent drops: 0
 Packet Recv drops: 1
 Total bytes sent: 1518072
 Total bytes recv: 2472067

R3#

Yep, we can see that R3 and R4 are in a primary/secondary relationship.  We can also see that both are alive.

In our little scenario, there are two pitfalls.  The first one I alluded to earlier.  The pitfall comes from the Symmetric ACL that is applied to all of the 192.168.0.0/16 network.  So what happens if a PING is sent from R1 to R3 using the loopback interface?

R1#ping 192.168.3.3 sour 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.3.3, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.1
.....
Success rate is 0 percent (0/5)
R1#

Bummer.  It fails.

Because the ACL defines that all traffic going from 192.168.0.0/16 to 192.168.0.0/16 is interesting traffic, that traffic must use IPSec to get across.  The KS is not a GM, so there is no IPSec tunnel between the KS and the GM, only an ISAKMP tunnel exists.  So, traffic from the loopback of R1 is being encrypted to the loopback of R3, but there is no IPSec SA associated with R3.  Hence, it fails.  One way around this is to use the loopbacks as the endpoints of the tunnel and the source points for the key servers.  Then apply the crypto map to the physical interface, but have all the information sourced from the loopback.  Then you could have a separate loopback network that is used only for GDOI.

The second pitfall is lab related.  All traffic is being routed through R3, which is a KS.  This is not best practice for design.  Each KS should server ONLY as a KS and not as a KS and a main router.  If R3 were to fail, then neither GM can get to R4 due to routing issues.  In a multi-site design, best practice would dictate that there would be a KS in different locations.  The GMs would then use the KS that is closest by hop count as the primary.

Finally, best practice also dictates that NTP should be used to synchronize time.  This is especially important in GDOI, as it uses a time scale for anti-replay protection.  Also, if certificates are used with a CA server, NTP MUST be configured.  In short, just configure it, will ya?

Good luck!

First, a diagram of our VTI tunnel:

We are using RIP v2 as a protocol to pass the fa0/0 networks and the loopback0 networks.  R1 is able to ping loopback0 on R2 with the R1 loopback0 as the source.  The loopback1 networks are not being forwarded through RIP.  EIGRP is going to be used with a VTI to send traffic for those two endpoints.

First things first: The VTI needs to be configured on both sides.  Here are the basic configurations:

R1:

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key cisco address 10.10.20.2
!
!
crypto ipsec transform-set STRONG esp-aes esp-sha-hmac
!
crypto ipsec profile VPN
 set transform-set STRONG
!
interface Tunnel0
 ip address 172.16.0.1 255.255.255.252
 tunnel source FastEthernet0/0
 tunnel destination 10.10.20.2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile VPN

R2:

crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key cisco address 10.10.10.1
!
!
crypto ipsec transform-set STRONG esp-aes esp-sha-hmac
!
crypto ipsec profile VPN
 set transform-set STRONG
!
interface Tunnel0
 ip address 172.16.0.1 255.255.255.252
 ip authentication mode eigrp 90 md5
 ip authentication key-chain eigrp 90 RIP
 tunnel source FastEthernet0/0
 tunnel destination 10.10.20.2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile VPN

While the tunnels have been built, no traffic is going through.  Without either a dynamic routing protocol or a static route, this is just a pretty tunnel.  Now traffic needs to go through the tunnel.  EIGRP will be used:

R1:

router eigrp 90
 passive-interface default
 no passive-interface Tunnel0
 network 172.16.0.0 0.0.0.3
 network 192.168.11.0
 no auto-summary

R2:

router eigrp 90
 passive-interface default
 no passive-interface Tunnel0
 network 172.16.0.0 0.0.0.3
 network 192.168.21.0
 no auto-summary

A couple of notes here.  The passive-interface default makes ALL interfaces on EIGRP 90 passive.  A passive interface does not send out updates.  By using the default and then adding the Tunnel0 interface, EIGRP traffic is now limited to being advertised only over the Tunnel0 interface.  The no auto-summary is needed to prevent summarization of the networks.

Our VTI, however, it not going to come up without a couple of changes to the firewall.  ACLs need to be added to the firewall to allow UDP 500 (isakmp) and ESP in order for everything to connect.  Add the following to the ASA:

access-list outside extended permit icmp any any
access-list outside extended permit esp any any
access-list outside extended permit udp any any eq isakmp
access-group outside in interface outside

The tunnel should be up, and the routes should be added.  On R1, the following output is seen:

R1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

 172.16.0.0/30 is subnetted, 1 subnets
C       172.16.0.0 is directly connected, Tunnel0
 192.168.11.0/32 is subnetted, 1 subnets
C       192.168.11.1 is directly connected, Loopback1
D    192.168.21.0/24 [90/297372416] via 172.16.0.2, 01:05:50, Tunnel0
 10.0.0.0/24 is subnetted, 2 subnets
C       10.10.10.0 is directly connected, FastEthernet0/0
R       10.10.20.0 [120/1] via 10.10.10.254, 00:00:03, FastEthernet0/0
 192.168.1.0/32 is subnetted, 1 subnets
C       192.168.1.1 is directly connected, Loopback0
R    192.168.2.0/24 [120/2] via 10.10.10.254, 00:00:03, FastEthernet0/0

Cool!  The routes are going through for both RIP and EIGRP.  The R shows RIP routes, and the D shows EIGRP routes.  With this up, the ISAKMP and IPSec SAs should show up as well:

R1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.10.10.1      10.10.20.2      QM_IDLE           1004    0 ACTIVE

R1#sh crypto ipsec sa

interface: Tunnel0
 Crypto map tag: Tunnel0-head-0, local addr 10.10.10.1

 protected vrf: (none)
 local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
 remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
 current_peer 10.10.20.2 port 500
 PERMIT, flags={origin_is_acl,}
 #pkts encaps: 934, #pkts encrypt: 934, #pkts digest: 934
 #pkts decaps: 943, #pkts decrypt: 943, #pkts verify: 943
 #pkts compressed: 0, #pkts decompressed: 0
 #pkts not compressed: 0, #pkts compr. failed: 0
 #pkts not decompressed: 0, #pkts decompress failed: 0
 #send errors 0, #recv errors 0

 local crypto endpt.: 10.10.10.1, remote crypto endpt.: 10.10.20.2
 path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
 current outbound spi: 0x8CD9D885(2363086981)

 inbound esp sas:
 spi: 0xB49A225C(3030000220)
 transform: esp-aes esp-sha-hmac ,
 in use settings ={Tunnel, }
 conn id: 17, flow_id: SW:17, crypto map: Tunnel0-head-0
 sa timing: remaining key lifetime (k/sec): (4434634/2753)
 IV size: 16 bytes
 replay detection support: Y
 Status: ACTIVE

 inbound ah sas:

 inbound pcp sas:

 outbound esp sas:
 spi: 0x8CD9D885(2363086981)
 transform: esp-aes esp-sha-hmac ,
 in use settings ={Tunnel, }
 conn id: 18, flow_id: SW:18, crypto map: Tunnel0-head-0
 sa timing: remaining key lifetime (k/sec): (4434634/2753)
 IV size: 16 bytes
 replay detection support: Y
 Status: ACTIVE

 outbound ah sas:

 outbound pcp sas:

The VTI is working, phase I and phase II are complete, and the routes show up in the routing table.  Now that the VTI is working, MD5 authentication can be added to EIGRP.  Authentication can be done in plain text, but given that this is the CCIE:Security exam, MD5 would be a better choice.  The following steps are needed to add authentication to EIGRP:

1. Create a key chain with a key string.
2. Add authentication to the interface.
3. Use MD5 as the mode of authentication on the interface.

First, the creation of the key chain.  RIP is already using a key chain, so that key chain will also be used for EIGRP:

R1:

key chain RIP
 key 1
 key-string cisco

R2:

key chain RIP
 key 1
 key-string cisco

The same key number should be used on both sides.  For RIP, authentication was added to the fa0/0 interface.  Because EIGRP is using the Tunnel0 interface, the EIGRP authentication lines should be added to that interface instead:

R1:

interface Tunnel0
 ip address 172.16.0.1 255.255.255.252
 ip authentication mode eigrp 90 md5
 ip authentication key-chain eigrp 90 RIP
 tunnel source FastEthernet0/0
 tunnel destination 10.10.20.2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile VPN

R2:

interface Tunnel0
 ip address 172.16.0.2 255.255.255.252
 ip authentication mode eigrp 90 md5
 ip authentication key-chain eigrp 90 RIP
 tunnel source FastEthernet0/0
 tunnel destination 10.10.10.1
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile VPN

Note the highlighted lines.  These lines were added to allow EIGRP to use MD5 authentication.  To verify that MD5 is indeed being used for authentication, the interface detail should reveal this:

R1#sh ip eigrp 90 interface detail
IP-EIGRP interfaces for process 90

 Xmit Queue   Mean   Pacing Time   Multicast    Pending
Interface        Peers  Un/Reliable  SRTT   Un/Reliable   Flow Timer   Routes
Tu0                1        0/0       229      71/2702      3654           0
 Hello interval is 5 sec
 Next xmit serial <none>
 Un/reliable mcasts: 0/0  Un/reliable ucasts: 1/26
 Mcast exceptions: 0  CR packets: 0  ACKs suppressed: 9
 Retransmissions sent: 13  Out-of-sequence rcvd: 1
 Authentication mode is md5,  key-chain is "RIP"
 Use unicast

Indeed, the authentication mode is MD5, and the key-chain is RIP.

A final note: The VTI tunnel is sending the traffic for EIGRP.  Because this traffic is going through a tunnel, there is no need to change the ASA to allow EIGRP traffic through it.  If this was not a tunnel, then the ASA would need the following line added to the ACL:

access-list outside permit eigrp any any

Remember, EIGRP is its own protocol.  This is not using TCP or UDP to pass traffic.

Good luck!

The first time I took the test, I went in with the idea that I probably wouldn’t pass the test the first time.  My fellow CCIEs in my workplace have noted on many an occasion that the first time you take the test it is simply to learn how to take the test.  They couldn’t be more right.  While there is the end of the bell curve that passes the first time through, I can see that taking the test really helps in developing a strategy to pass that cannot really be accomplished by simply studying the material.  Learning the environment, what to expect, and seeing what they really want in the testing situation can eat up valuable time.  And time really isn’t on your side during the exam.

Granted, the test can obviously be passed the first time.  But for many of us, the first time is the real eye-opener not only to the weaknesses in your study, but weaknesses in strategy.  The first time I took the test, not a lot caught me off-guard, but a few things did take me MUCH longer than they should have taken.  And that can be the difference between passing and failing.

There are many people out there who give out advice for passing the test.  Some advice is helpful, some is trivial, and some is downright silly.  Yes, I know that I need to know the blueprint.  Thanks, Captain Obvious.  I would have NEVER guessed that.  Yes, I know that having a good night’s sleep and a hearty breakfast are important before the exam.  It is also important before meeting a customer.  It is also important before taking a long drive.  It is also important before starting pretty much any day of your life.  So thanks, but not really all that helpful.

The real trick to this test is knowing enough material that you can do many of the tasks in a very quick amount of time.  This means doing the same thing over and over to the point where things that maybe took you an hour or two in the real world can get done in 15 to 20 minutes.  This is difficult for many engineers, as we have a tendency to want to double check and triple check as we do configurations for our customers and companies.  No such luxury here, and this isn’t a production environment.

Ok, so there is a taste of what I think needs to be done.  Here then are my suggestions to you, the CCIE candidate (and, more importantly, the CCIE:Security candidate), on how to get through the test with success.  Please note: I am taking my third attempt in April, so I am going to be employing these strategies for my next attempt.

Read The Exam

Go through the ENTIRE lab and read through all the tasks that need to be done.  What will become painfully obvious is that there will be a “core” section that needs to be completed to make a lot of the other pieces work properly.  If you don’t have the core working, you could be in some serious trouble.  Look at the core, get it done quickly, do your debugs and health checks, then complete the other parts of the exam.  This is especially true with the CCIE:Security lab.

Check All the Switches

My last exam really tripped me up because I failed to look at the switches on the exam.  Make sure that there aren’t any settings on any of the ports that may cause issues.  Are there ports turned off?  Are there ports that should be trunks that are access ports?  Is the routing correct?  Are there some strange ACLs floating around?  After reading through the lab, look at the switches and verify that everything is connected and configured properly.  This could save you HOURS on troubleshooting.

Practice on a Regular Keyboard and Mouse

This may sound strange, but your lab will have a normal keyboard and mouse.  No NDA violations here…you aren’t getting a laptop.  So why practice on a laptop?  If you only have a laptop, connect a keyboard and mouse to it to get the feel of using a desktop with a monitor.

Go Through the Support Site During Practice

The last test I took I think I got around 10 points simply by checking my work on the Cisco Support Site (what used to be the Universal CD).  You have access to this, and it is the same website you have at home with one difference – no search function.  So, get used to navigating quickly to various documents that you may use on the exam.  Look through the blueprint, and know where to get information quickly on the site through navigation.  Part of the Universal CD are configuration examples and guides.  These are PURE GOLD!  If you get stuck on a question, or you just want to check your work, being able to pull up a configuration example quickly will allow you to verify the work you have done.  Get good at navigating this site;  it will pay in precious points.

Take Breaks

While you do get a 40 minute break for lunch, it is early on in the lab.  The last part of the lab can get long, and sometimes our skills diminish as our blood sugar gets low.  While you cannot bring food into the lab, you can briefly leave the lab and go to the snack bar in the lounge.  Take advantage of this.  Clear the head, get a snack, and step away from the computer.  Sometimes that extra 10 minutes away can let you come back refreshed.  I actually had an “ah-ha!” moment during the break that allowed me to finish a question.

Use Flashcards for the OEQ Study

One of the downsides to the CCIE:Security is the 4 Open Ended Questions (OEQs) at the beginning of the lab.  You must answer four short questions before starting the exam.  I won’t go into the arguments for/against the OEQs…that is for other bloggers and forum boards.  But for now, you need to answer the questions.  I am using a flashcard program for my iPhone to do some quick studying of questions I have created based on the blueprint and data from the Cisco Press CCIE book.  It is a good idea to make sure that you understand the underlying technologies and different kinds of attacks when taking the OEQs.  Having quick access to self-made questions will go a long way in passing the OEQ.

Find the Easy Questions

You will find that some tasks are very easy and quick, while others take a large amount of time.  The main thing to remember is that you are not configuring a system for a customer; you are trying to pass the exam.  This means that you need to accumulate points.  If a task looks like it will take WAY to long (and it isn’t the core), then pass on it and do another task.  Better to get the points on simpler things than spend an hour to get the same amount of points from a complex task.  I use the paper that is provided to list the different questions, then I rate them by easy, medium, and hard.  I do the core, then do the easy, then do the medium, then do the hard.  Remember, you can still pass the exam without even attempting a question or two, as long as you get the rest correct.  Save those last hard points for last.

Learn the Debugs

When studying a technology, such as configuring a site-to-site VPN, run the debugs and look at what should be normal when it is working.  Then break things and see what the debugs look like.  Understanding what an error is and what debugs look like in both a functional and non-functional environment goes a long way in verifying that you have everything working properly.  This also applies to the show commands.  What are all the show commands that can be used for a technology to verify a correct setup?  Know these inside and out.  Knowing just the configuration isn’t going to cut it.

Stay Out of the Rabbit Hole

As in real life, it can be easy as engineers to keep battling a problem until it is solved.  This strategy is a recipe for doom if followed on the exam.  While those of us in the field would never go to a customer and say “Well, we got 80% of your network working.  Enjoy!”, it is the proper response to the lab.  You need 80% for your number, so keep that in mind when you run into a bit of an issue with a task.  A good way to get out of that habit is to not do that while doing practice labs.  If you are working on a practice lab and can’t get something working quickly, then move on to the next part of the lab or some other technology.  Don’t stick around.  Finish up some other pieces first, then come back to it.  This will help develop a study habit of not going so deep into something that your time isn’t used to get other study topics done.  If you do that during your study, you are more likely to do that during your test.  Remember, your CCIE number doesn’t come attached with your score.  Get your points and leave your pride at the door.

Do NOT Study the Night Before the Exam

Seriously.  Don’t.  Ever.  If you don’t know the material when you hit the hotel room, do you think an extra 4 hours that night is going to help?  Get the studying done ahead of time, fly in to town, relax, have a Fresca, and enjoy the evening.  I have yet to hear someone tell me “I am so glad I studied the night before, or I would have failed.”

Use Your Twelve Bucks

Yeah, this really doesn’t apply to the exam.  But one of the benefits of the exam is that you get a twelve dollar meal voucher to use at the cafeteria during lunch (at least in SJC).  Last time I got a stir fry, and apple, and a Snapple.  6 bucks and change.  Well, tack on 3 extra waters and I have used up nearly my entire twelve bones.  And while food isn’t allowed at the lab, drinks are.  I got three extra bottles of water, two of which I took back to the hotel.  Some of the other test takers grabbed a few bags of chips.  Thing is, you just spent 1400 bucks on a test.  If you fail, you ended up having a 1400 dollar lunch.  Might as well grab a few drinks or chips.

Well, I hope that these tips were a bit useful for your studies.  I plan to employ these techniques in April when I give another go at the exam.  Here’s to the third time being a charm.  In the meantime, I will try to keep posting my little mini labs that I will use for my own studies.  Hopefully, they will be useful to you, the gentle reader, as well.

Good luck!

In the following example, I demonstrate how to quickly get the Auth Proxy setup with an IOS router that authenticates against an ACS server. This is the most basic of examples, and it should be able to be accomplished within 15 minutes.  TACACS+ is used, but RADIUS can be used as well on the ACS.

The basic steps to setting up an Auth Proxy:

1. Turn on AAA on the router.
2. Create AAA authentication and authorization mechanisms.
3. Turn on the HTTP server.
4. Authenticate against the HTTP server using AAA.
5. Configure the Auth Proxy settings.
6. Configure the ACL to only permit necessary traffic.
7. Configure the ACS for Auth Proxy.

The following diagram will be used for the setup:

In this example, the XP machine is going to HTTP into R1.  However, R2 is going to intercept the request and make sure that the Windows XP user has the proper credentials to get into the network.   The ACS server is a Windows 2008 server running Cisco Secure ACS 4.2.1.  R2 is going to authenticate against this server before modifying the ACL that will permit authenticated traffic across.

First, we setup AAA on R2.  AAA needs to be turned on, authentication needs to be allowed for the default login to be with TACACS+, and authorization needs to be turned on for auth-proxy.  As a precaution, the login for the console will need to be turned off.  It is always a good idea in the lab to do this; otherwise, you could get locked out of the console, which provides a less than optimal situation.

aaa new-model
!
aaa authentication login default group tacacs+
aaa authentication login CONSOLE none
aaa authorization auth-proxy default group tacacs+
!
<snipped text>
line con 0
 exec-timeout 0 0
 logging synchronous
 login authentication CONSOLE
line aux 0
 login authentication CONSOLE

Now we must turn on the HTTP server on R2 and authenticate using AAA:

ip http server
ip http authentication aaa

The TACACS+ server needs to be setup.  We are going to use fa0/1 as the source interface for TACACS+.  This is important, as this will be the IP address that is used in the server itself.

ip tacacs source-interface FastEthernet0/1
tacacs-server host 10.10.10.100 key cisco

Now the actual IP Auth Proxy will need to be setup.  The IP Auth Proxy will have a name of AUTH_PROXY and the banner will say “Please Enter Your Username and Password:”.  Everything else will be left as default.

ip auth-proxy auth-proxy-banner http ^C
Please Enter Your Username and Password
^C
ip auth-proxy name AUTH_PROXY http

Note that telnet and FTP can be used instead of HTTP.  However, using the HTTP keyword will require the use of HTTP for authentication.

The IP access list will also need to be created.  We are going to deny everything through the router; however, in order to access the auth-proxy HTTP server, an ACL entry allowing that traffic will need to be added.  Also, EIGRP is being used in our example, so EIGRP traffic will also need to be allowed.

ip access-list extended DENY
 permit tcp any host 45.45.20.2 eq www
 permit eigrp any any
 deny   ip any any
 deny   icmp any any

Finally, the IP Auth Proxy and the access list are applied to the fa0/1 interface:

interface FastEthernet0/1
 ip address 10.10.20.2 255.255.255.0
 ip access-group DENY in
 ip auth-proxy AUTH_PROXY

The router configuration is now complete.  Now the ACS server will need to be configured.  The ACS will be configured with a username of cisco and a password of cisco.  One the user cisco is authenticated, the ACLs in the ACS will be applied, and all IP and ICMP traffic will be allowed through the authenticated host.

First, a user named cisco with a password of cisco should be added to the ACS:

Next, under Network Configuration, we add the R2 client and authenticate using TACACS+.  We put the secret key of cisco here.

After submitting and applying the changes,  go into the Interface Configuration and add the auth-proxy setting into the TACACS+ configuration.  This is not a configuration that is enabled by default in the group settings.

The usual submit/apply routine is then done.  Finally, the settings for the proxy ACLs are added to the configuration.  Note that the shell privilege level of 15 is added; otherwise, it would not be possible to add the ACLs into the interface settings of the router.

Submit, apply, done.  Yep, that is it.

First, the router should be tested against the ACS to make sure that the communication between the two is working properly.  This can be accomplished with the test aaa command:

R2#test aaa group tacacs+ cisco cisco legacy
Attempting authentication test to server-group tacacs+ using tacacs+
User was successfully authenticated.

Neato!  R2 is successfully authenticating the user cisco with the password of cisco against the AAA server.  If the auth-proxy is working, then an HTTP connection to R1 should invoke the auth proxy on R2.

The username of cisco and password of cisco is used.  The result?

And now the R1 login is presented:

A quick look at the R2 auth proxy cache:

R2#sh ip auth-proxy cache
Authentication Proxy Cache
 Client Name cisco, Client IP 10.10.20.100, Port 1105, timeout 60, Time Remaining 60,
 state ESTAB

The auth-proxy was successful, and the client name cisco from the IP address of 10.10.20.100 is now authenticated.  The ACLs have also changed:

R2#sh ip access-list
Extended IP access list DENY
 permit ip host 10.10.20.100 any (14 matches)
 permit icmp host 10.10.20.100 any
 10 permit tcp any host 45.45.20.2 eq www
 20 permit eigrp any any
 30 deny ip any any (1698 matches)
 40 deny icmp any any

There are now unnumbered entries in the ACL.  These entries match the proxyacl entries in the auth-proxy section of the ACS server.  Now that cisco has been authenticated from 10.10.20.100, that host is allowed through using either IP or ICMP.

The auth-proxy default timeout is 60 minutes.  The number can be lowered by using the inactivity-time command after assigning a name to the auth-proxy:

ip auth-proxy name AUTH_PROXY http inactivity-time 60

To remove an authenticated user from the auth-proxy list, use the clear command, along with the IP address of the user that needs to be inactivated.  To clear all of the users, use the * instead of the IP address:

R2#clear ip auth-proxy cache 10.10.20.100

This command will remove our current user, and now cisco will need to re-authenticate through the auth-proxy.

With a little bit of practice, this entire setup should be accomplished in about 15 minutes.  Do not forget that if the auth proxy needs to go through a firewall, then the firewall will need to allow the TACACS+ protocol (TCP 49), the HTTP protocol (TCP 80), and any other protocols that are in the proxy ACL list in the ACS.

Good luck!

A customer of mine is using one ISP, which happens to be a T1 connection.  Really cool if you live in Lickbucket, AK or insist that 1998 was the coolest year ever.  Fortunately for me and my sanity, my customer does not fit into either category.  The T1 connection is used for both Internet access and site-to-site VPN connections to a number of satellite offices.  The VPNs are utilized for direct dialing between IP phones.  Not only is this solution undersized for their needs, but recently the T1 connection failed, causing the whole main office to go down.

Right now, the  customer is using a pair of ASAs at the main site.  While the ASA doesn’t offer a BGP solution like Juniper (grrr…), the ASA does offer redundant links out to the Internet.  However, unlike BGP, the IP address schemes for each ISP will be disparate.  This is no big deal if you have only outbound connections, but what about VPNs?  If we go with an additional ISP, we then need to split the traffic so we can utilize both links, but we still need a failover solution in case either link dies.

One feature of the ASA is the ability to put multiple peers on the firewall.  With this option, the VPN connections at each of the satellite offices can point to both external IP addresses for the redundant ISP links, providing for failover for the VPN as well.

In order to test out the theories, I setup the following lab:

With this lab, a local ASA simulates two ISP connections to R2 and R3.  R4 is the ISP connection to the remote location, and the remote ASA connects to R4.  R1 and R5 loopbacks are going to simulate end hosts.  R4 has three loopbacks that is going to simulate Internet connections.  R2, R3, and R4 are running EIGRP, and R3 is using an offset list to prevent asymmetric routing.  BGP could have been used to create a more “realistic” Internet connection, but…well, screw that.  EIGRP is easy, and the routing protocol isn’t the point of the lab.

After setting up EIGRP, getting the routes setup correctly, and opening up the firewalls to allow ICMP through, the meat of the lab is now at hand.  The failover between ISPs is done by SLA monitoring.  The SLA monitor sends a ping to a downstream router of choice, and if the router is responding, then the routes associated with the SLA monitor stay in place.  However, if the downstream router does not ping, then a second static route that is given a high metric (in this example, 250), takes over in the routing table.  Here is what the configuration in the lab would look like for the default static routes:

route outside 0.0.0.0 0.0.0.0 192.168.1.2 1 track 1
route backup 0.0.0.0 0.0.0.0 192.168.2.2 254
!
sla monitor 123
 type echo protocol ipIcmpEcho 192.168.1.2 interface outside
 num-packets 3
 frequency 10
sla monitor schedule 123 life forever start-time now
!
track 1 rtr 123 reachability

Two default routes exist.  The first default route points to the first “ISP” and has a metric of 1, and there is a track of 1 placed after it.  The track 1 matches the Response Time Reporter (rtr) that is the last statement of the SLA monitoring configuration.  The second default route points to the second “ISP” and has a metric of 254.

SLA monitor is configured with the interface and the downstream router (or host) that will receive the ping.  In this example, we are pinging R2 from the outside interface on the ASA to verify connectivity.  In the real world, this is going to be one of the better options, as many ISPs don’t really like having intermediate routers or hosts pinged through their systems.  The SLA monitor schedule line starts the monitoring process.  The final line is the Response Time Reporter associated with tracking the ping and linking the ping to the primary route using the track command.

After configuration, the routing table looks like the following:

ASA# sh route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
 * - candidate default, U - per-user static route, o - ODR
 P - periodic downloaded static route

Gateway of last resort is 192.168.1.2 to network 0.0.0.0

S    1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
C    10.10.10.0 255.255.255.0 is directly connected, inside
C    192.168.1.0 255.255.255.0 is directly connected, outside
C    192.168.2.0 255.255.255.0 is directly connected, backup
S*   0.0.0.0 0.0.0.0 [1/0] via 192.168.1.2, outside

The default route is currently going through the outside interface.  However, if the link to 192.168.1.2 is severed by doing a no shut on the fa0/0 interface on R2, the following result is:

ASA# sh route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
 * - candidate default, U - per-user static route, o - ODR
 P - periodic downloaded static route

Gateway of last resort is 192.168.2.2 to network 0.0.0.0

S    1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
C    10.10.10.0 255.255.255.0 is directly connected, inside
C    192.168.1.0 255.255.255.0 is directly connected, outside
C    192.168.2.0 255.255.255.0 is directly connected, backup
S*   0.0.0.0 0.0.0.0 [254/0] via 192.168.2.2, backup

The default route has now changed to the backup interface, and the next hop is going to ISP2.  Notice the metric: it is now at 254 rather than 1, as it is using the floating static route that was created.  After doing a no shut on fa0/0 on R2, the routes go back to normal.

One down, one to go…

The failover on the ASA is working, but what about the VPNs? There are now two different ISPs providing two different IP addresses on the outside. So the ASA at the remote site needs to connect to the backup interface on the local ASA for the primary VPN, but when the primary connection fails, it will need to connect to the outside interface of the local ASA. One way to accomplish this is to create two peers on the ASA on a single crypto map. However, not only do two peers need to be added, but two different tunnel groups (tunnel groups require the IP address of the endpoint for the name on S2S VPN connections).

The local ASA looks like the following:

access-list VPN extended permit ip host 1.1.1.1 host 6.6.6.6
!
crypto ipsec transform-set STRONG esp-3des esp-sha-hmac
crypto map VPN 10 match address VPN
crypto map VPN 10 set peer 10.10.40.2
crypto map VPN 10 set transform-set STRONG
crypto map VPN interface outside
crypto map VPN interface backup
crypto isakmp enable outside
crypto isakmp enable backup
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
!
tunnel-group 10.10.40.2 type ipsec-l2l
tunnel-group 10.10.40.2 ipsec-attributes
 pre-shared-key cisco

This is a pretty standard crypto map solution.  The access list at the top is creating a tunnel from l0 on R1 to l0 on R6.  Everything else will go out the “Internet”.  The next hurdle is the routing.  Redundancy would require that if ISP2 went down, then the VPN would go through ISP1. We can accomplish that by doing the same thing that we did for the outside link, only we create another SLA monitor for the backup link and track that connection:

route backup 1.1.1.1 255.255.255.255 192.168.2.2 1 track 2
route backup 10.10.40.2 255.255.255.255 192.168.2.2 1 track 2
route outside 1.1.1.1 255.255.255.255 192.168.1.2 254
route outside 10.10.40.2 255.255.255.255 192.168.1.2 254
!
sla monitor 456
 type echo protocol ipIcmpEcho 192.168.2.2 interface backup
 num-packets 3
 frequency 10
sla monitor schedule 456 life forever start-time now
!
track 2 rtr 456 reachability

Two routes have been added that will go to the backup interface for the VPN.  The first route is the endpoint that will go through the VPN tunnel.  The second IP is the actual VPN termination point on the firewall.  By creating the routes in this manner, the VPN will be forced through the backup ISP unless it goes down.  If the backup ISP goes down, the VPN will go through the outside ISP until it comes back up.  This allows both connections to still be used rather than having one connection sitting idle yet costing the company money.

Routing is complete, so now the remote ASA needs to be configured to use the backup ISP as the primary and the outside ISP as the secondary for the VPN.

access-list VPN extended permit ip host 6.6.6.6 host 1.1.1.1
!
crypto ipsec transform-set STRONG esp-3des esp-sha-hmac
crypto map VPN 10 match address VPN
crypto map VPN 10 set peer 192.168.2.1 192.168.1.1
crypto map VPN 10 set transform-set STRONG
crypto map VPN interface outside
crypto isakmp enable outside
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
!
tunnel-group 192.168.2.1 type ipsec-l2l
tunnel-group 192.168.2.1 ipsec-attributes
 pre-shared-key cisco
tunnel-group 192.168.1.1 type ipsec=l2l
tunnel-group 192.168.1.1 ipsec-attributes
 pre-shared-key cisco

The configuration is pretty much the same, but there are some distinct differences.  First, the access-list for the VPN is reversed to match the outgoing traffic as with any VPN connection between two IPSec devices.  Second, the set peer statement has two IP addresses: The primary address followed by the backup address is the primary is unreachable.  Finally, there are two tunnel groups created.  This needs to be done because the tunnel group must match the IP address.  Because we have two potential tunnels, we need two tunnel groups.

So with everything in place, testing can now begin.  The following scenarios should be tested:

1. Current state: When we ping 6.6.6.6 from 1.1.1.1 on R1, the ping should be successful, and phase one should complete between 192.168.2.1 and 10.10.40.2.  This can be confirmed with a sh crypto isakmp on both ASAs.  A ping to 2.2.2.2, 3.3.3.3, or 4.4.4.4 on R4 from R1 (either 1.1.1.1 or 10.10.10.2) should also complete.  A traceroute to those IP address should show the route going through R2.
2. R2 goes down: Ping from 1.1.1.1 to 6.6.6.6 is successful.  Ping to R4 on any interface is successful, but the traceroute should show that the path now goes through R3. The crypto maps on both ASAs remain the same as in scenario 1.
3. R3 goes down: Ping from 1.1.1.1 to 6.6.6.6 is successful. P Ping to R4 on ay interface is successful, and the traceroute should go through R2 as in scenario 1.  However, the crypto maps on the ASAs should now show a connection between 10.10.40.2 and 192.168.1.1.  This can be confirmed with a sh crypto isakmp on both ASAs.

If these scenarios are successful, then the lab setup is confirmed.  First, scenario 1.  The routing table on the ASA:

ASA# sh route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
 * - candidate default, U - per-user static route, o - ODR
 P - periodic downloaded static route

Gateway of last resort is 192.168.1.2 to network 0.0.0.0

S    1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
S    6.6.6.6 255.255.255.255 [1/0] via 192.168.2.2, backup
C    10.10.10.0 255.255.255.0 is directly connected, inside
S    10.10.40.2 255.255.255.255 [1/0] via 192.168.2.2, backup
C    192.168.1.0 255.255.255.0 is directly connected, outside
C    192.168.2.0 255.255.255.0 is directly connected, backup
S*   0.0.0.0 0.0.0.0 [1/0] via 192.168.1.2, outside

So far, so good.  A traceroute to an “Internet” address (one of the loopbacks):

R1#traceroute 5.5.5.5

Type escape sequence to abort.
Tracing the route to 5.5.5.5

 1 192.168.1.2 28 msec 44 msec 16 msec
 2 10.10.20.1 8 msec *  28 msec

Good!  The traceroute is going through R2.  Is the VPN going through R3?  It should be terminating on the 192.168.2.1 address:

ASA2# sh crypto isakmp sa

 Active SA: 1
 Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 192.168.2.1
 Type    : L2L             Role    : responder
 Rekey   : no              State   : MM_ACTIVE

After pinging 6.6.6.6 from R1 with a 1.1.1.1 source, we see on ASA2 that the active peer is 192.168.2.1, so we are going through R3 and terminating on the backup interface of ASA1.  So for scenario 2, we shut the fa0/0 interface on R2 to verify that all traffic will go through R3.  After shutting the interface down, we look at the routing table on ASA1:

ASA# sh route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
 * - candidate default, U - per-user static route, o - ODR
 P - periodic downloaded static route

Gateway of last resort is 192.168.2.2 to network 0.0.0.0

S    1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
S    6.6.6.6 255.255.255.255 [1/0] via 192.168.2.2, backup
C    10.10.10.0 255.255.255.0 is directly connected, inside
S    10.10.40.2 255.255.255.255 [1/0] via 192.168.2.2, backup
C    192.168.1.0 255.255.255.0 is directly connected, outside
C    192.168.2.0 255.255.255.0 is directly connected, backup
S*   0.0.0.0 0.0.0.0 [254/0] via 192.168.2.2, backup

The default gateway has now changed to the backup interface, and the backup interface is being used for the VPN.  A traceroute to the “Internet”:

R1#traceroute 5.5.5.5

Type escape sequence to abort.
Tracing the route to 5.5.5.5

 1 192.168.2.2 20 msec 44 msec 8 msec
 2 10.10.30.1 8 msec *  28 msec

The traceroute is now going through R3 instead of R2.  The VPN:

ASA2# sh crypto isakmp sa

 Active SA: 1
 Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 192.168.2.1
 Type    : L2L             Role    : responder
 Rekey   : no              State   : MM_ACTIVE

The backup interface on the ASA is still terminating the VPN.  Scenario 3 is up next:

ASA# sh route

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
 * - candidate default, U - per-user static route, o - ODR
 P - periodic downloaded static route

Gateway of last resort is 192.168.1.2 to network 0.0.0.0

S    1.1.1.1 255.255.255.255 [1/0] via 10.10.10.2, inside
S    6.6.6.6 255.255.255.255 [250/0] via 192.168.1.2, outside
C    10.10.10.0 255.255.255.0 is directly connected, inside
S    10.10.40.2 255.255.255.255 [250/0] via 192.168.1.2, outside
C    192.168.1.0 255.255.255.0 is directly connected, outside
C    192.168.2.0 255.255.255.0 is directly connected, backup
S*   0.0.0.0 0.0.0.0 [1/0] via 192.168.1.2, outside

All of the routes are now going through the outside interface.  Traceroutes should now go through R2:

R1#traceroute 5.5.5.5

Type escape sequence to abort.
Tracing the route to 5.5.5.5

 1 192.168.1.2 24 msec 16 msec 12 msec
 2 10.10.20.1 40 msec *  28 msec

All traffic is going through R2 for the Internet.  The VPN:

ASA2# sh crypto isakmp sa

 Active SA: 1
 Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 192.168.1.1
 Type    : L2L             Role    : responder
 Rekey   : no              State   : MM_ACTIVE

Yep, the ASA is terminating on the outside interface rather than the backup.  All of our testing is complete.

Yay!

1. Create an isakmp policy
2. Create an isakmp key
3. Create an ipsec transform set
4. Create an access list defining the encrypted traffic
5. Create a crypto map
6. Apply the crypto map to the physical interface

Pretty straight forward.  So what would it look like?  Let’s assume that we have the following basic network:

If you were to create your IPSec tunnel using the above steps, it would look like the following:

crypto isakmp policy 10
encr 3des
authentication pre-share
crypto isakmp key cisco address 10.10.20.2
!
!
!
crypto ipsec transform-set STRONG esp-3des esp-sha-hmac
!
crypto map IPSEC 10 ipsec-isakmp
set peer 10.10.20.2
set transform-set STRONG
match address ENCRYPT
!
interface FastEthernet0/0
ip address 10.10.10.2 255.255.255.0
duplex auto
speed auto
crypto map IPSEC
!
ip access-list extended ENCRYPT
 permit ip host 1.1.1.1 host 2.2.2.2

This would require the ASA to allow both ESP and UDP 500 through it for the connection to be made.  The UDP 500 would be needed for phase I, while ESP would be needed for phase II.

One way to get around this would be to use a GRE style tunnel.  The GRE tunnel would effectively hide the tunnel; however, GRE would add 4 bytes to each packet, which increases overhead.  Unlike a crypto map that is applied to the physical interface, the GRE tunnel would only require that GRE was allowed through the outside of the firewall.

The VTI gets around this by creating a tunnel that protects ipv4 traffic without the extra overhead.  Like GRE, it would also need to be allowed through the firewall, but this can be done by allowing protocol 50 (ESP) through on the lower security interface.  In our example, e0/0 on the firewall is the outside interface, and e0/1 is the inside interface.  So, we would allow ESP through on the ACL allowing inbound traffic on the outside interface.

The steps to creating the VTI tunnel are not that much different, but the differences are significant:

1. Create the isakmp policy
2. Create the isakmp key
3. Create the ipsec transform set
4. Create the ipsec profile
5. Create the tunnel interface
6. Add the profile to the tunnel
7. Route encrypted traffic through the tunnel

Instead of creating a crypto map that is applied to the physical interface, an ipsec profile is applied to the tunnel. Let’s take a look:

crypto isakmp policy 10
 encr 3des
 authentication pre-share
crypto isakmp key cisco address 10.10.20.2
!
!
crypto ipsec transform-set STRONG esp-3des esp-sha-hmac
!
crypto ipsec profile VTI
 set transform-set STRONG
!
!
interface Tunnel0
 ip address 192.168.1.1 255.255.255.252
 tunnel source FastEthernet0/0
 tunnel destination 10.10.20.2
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile VTI
!
ip route 0.0.0.0 0.0.0.0 10.10.10.1
ip route 2.2.2.2 255.255.255.255 Tunnel0

In this example, a crypto isakmp policy is still used; however, the ipsec profile is used instead of a crypto map, and that ipsec profile is then added to the tunnel interface (the tunnel protection line). We then direct the traffic that we want to be encrypted by routing it through the tunnel interface rather than the physical interface. The routes then determine the encrypted traffic rather than an IP access list.

By using the VTI, your encrypted traffic can be easily found by doing a

sh ip route | i tunnel0

which will show all the traffic that is being routed through the tunnel. Because the traffic is routed through the tunnel, it will be encrypted. Because of the lack of an access-list to determine interesting traffic (making the interesting traffic line essentially permit ip any any), the encrypted traffic must be controlled by routing.

My blog will also do some focusing on Skepticism as well.  I seem to use both skills in tandem, so while it may seem a bit incongruous, in their own little way, the two seem to relate in my life.

On we go!