Networking Stacked Knowledge

Case Study: Cisco Phone 7942 unexpected auto-call/auto-dial

2013-01-29T14:00:00.000+08:00

Problem Description:
This is an actual ticket escalated to our team.

User reported that his Cisco IP Phone automatically dials a number once he picks up the handset of the phone.

HW/SW Versions:

Cisco IP Phone 7942
SCCP42.9-2-3S
CUCM 7.1 (7.1.5.33900-10)

Troubleshooting:

Same behavior is observed when pressing the speaker button;
Same behavior is observed when pressing the headset button;
Only one number is being called consistently;
IP Phone is registered correctly in CUCM;
Resetting the phone has stopped the auto-dialling.

Root Cause:
After some research, we found the bug documenting this behavior:

CSCtz29414 - 7942/7962 - Call History is Auto-Dialed When Pressing Speaker Button

Symptom:
When a 7942/62 phone is populated with call history records (e.g. received/placed/missed calls), the endpoint will auto-dial the number contained in its call history when the speaker button is pressed.

Conditions:
IP Phone Call History contains Received/Placed/Missed Phone number record prior to pressing speaker button to go off-hook. Phone will continue to auto-dial the number listed in Call History once error condition is experienced.

Workaround:
Reset IP Phone to clear error condition

1st Found-In:
9.2(3)

Fixed-In:
9.2(3)ES7
9.3(1)TH2.2
9.2(3)MN1.22

Workaround:
As per the bug above, reset the phone

Permanent Fix:
Upgrade the phone firmware.

Aptigen .NID File Format

2012-07-27T11:23:00.002+08:00

</Default>

</AptiTags>

<Title>Speed Dials Site A-Z</Title>

<URL>http://localhost/Aptigen/SITE_A-Z/file2.xml</URL>

</MenuItem>

<URL>http://localhost/Aptigen/SITE_A-Z/file3.xml</URL>

</MenuItem>

<URL>http://localhost/Aptigen/SITE_A-Z/file4.xml</URL>

</MenuItem>

</CiscoIPPhoneMenu>

</ScreenObject>

</Default>

</AptiTags>

</DirectoryEntry>

<Name>Aline Cara</Name>

</DirectoryEntry>

</CiscoIPPhoneDirectory>

</ScreenObject>

</Default>

</AptiTags>

</DirectoryEntry>

<Name>Magd Darren</Name>

</DirectoryEntry>

</CiscoIPPhoneDirectory>

</ScreenObject>

</Default>

</AptiTags>

<Name>Pierre John</Name>

</DirectoryEntry>

<Name>Raymond Edward</Name>

</DirectoryEntry>

</CiscoIPPhoneDirectory>

</ScreenObject>

</ScreenObjects>

<Server>localhost</Server>

<DefEmulatorManuf>Cisco</DefEmulatorManuf>

<CodeLanguage>VBNET</CodeLanguage>

<PreferredEditor>Notepad</PreferredEditor>

<PromptForEditor>False</PromptForEditor>

</ApplicationSettings>

</AptigenEnterprise>

</APTIGEN>

Wireless clients getting DHCP Server in a different subnet

2012-06-20T10:44:00.000+08:00

Issue:

Running "ipconfig /all" on the client shows that the wireless DHCP server is in a different subnet and/or a different DHCP server than expected;
Confirmed no network connections observed from wireless client;
Wireless network infrastructure consists of Lightweight Access Points controlled by Wireless LAN Controller (WLC);
Even if IP address is released and renewed, client is still getting the correct IP address but DHCP Server is still in the different subnet.

#SNIPPET START

Wireless LAN adapter Wireless Network Connection:

   Connection-specific DNS Suffix  . :
my.company.com

   Description . . . . . . . . . . . : Intel(R)
Centrino(R) Advanced-N 6200 AGN

   Physical Address. . . . . . . . . :
00-27-EE-EE-EE-EE

   DHCP Enabled. . . . . . . . . . . : Yes

   Autoconfiguration Enabled . . . . : Yes

   Link-local IPv6 Address . . . . . :
fe80::c010:5d39:eeee:eee%11(Preferred)

   IPv4 Address. . . . . . . . . . . :
10.100.30.228(Preferred)

   Subnet Mask . . . . . . . . . . . :
255.255.254.0

   Lease Obtained. . . . . . . . . . : Saturday,
June 09, 2012 6:56:11 PM

   Lease Expires . . . . . . . . . . : Sunday,
June 10, 2012 12:56:11 AM

   Default Gateway . . . . . . . . . : 10.100.30.1

   DHCP Server . . . . . . . . . . . : 192.168.1.1

   DHCPv6 IAID . . . . . . . . . . . : 218113808

   DHCPv6 Client DUID. . . . . . . . :
00-01-00-01-15-9D-76-61-00-EE-EE-EE-EE-EE

   DNS Servers . . . . . . . . . . . : 10.100.5.130

                                       10.100.5.132

                                       10.20.4.100

                                      
10.20.4.80

   Primary WINS Server . . . . . . . :
10.100.30.209

   Secondary WINS Server . . . . . . :
10.100.30.210

   NetBIOS over Tcpip. . . . . . . . : Enabled

#SNIPPET END

Resolution:

First thing to confirm is if there is any actual network issue; i.e., does the client have any network connectivity issues? In this example, it's already confirmed that there are no issues.
In this case, there is actually no network/configuration issue; the WLC is configured as a DHCP Relay (as opposed to instead of as a DHCP Server); hence, the client sees a virtual IP address of the WLC, instead of the actual DHCP server.
Here's the rough run-through of the process;

Wireless Client DHCP request --> AP --> WLC (dhcp relay) ---> DHCP Server
DHCP Server reply --> WLC --> AP --> Wireless Client

Reference:
Wireless LAN Controller (WLC) FAQ

Case Study: Juniper client disconnects randomly with nc.windows.app.23711

2012-04-30T16:01:00.000+08:00

Below is an actual issue seen several times in our working environment involving Juniper VPN client (NC Connect)

Issue:

System Log Message: The Network Connect session terminated. Do you want to reconnect? (nc.windows.app.23711).

- Juniper Networks > Network Connect 7.1.0

- MS Windows 2000, Windows XP, Windows Vista, Windows 7

Analysis:

This error indicates that some installed software or program in your laptop is trying to modify the network routes within your laptop. The Juniper VPN application see this as a threat and therefore throws this message and disconnects the VPN tunnel.

Historically, this is widely caused by the older version of the Bonjour service (versions earlier than 1.06) used by Apple. Recently though, we have discovered that some device management applications are also causing the same issue.

I have experienced this at home myself. I have a Canon printer management suite that auto-runs on startup of my laptop. It automatically scans my home network for my printer. When I connect to the company network via VPN, I lose connection to my home network, thus losing connection to my home printer. To look for the printer, one of the things that the Canon management suite does is modify the route table of my laptop, which is the "threatening" action detected by Juniper which then causes the VPN to be disconnected. Luckily though, the simple solution here is to turn off the Canon management suite before I connect via VPN.

Troubleshooting with debugs/Logs:

Below steps may help confirm if the above error is indeed generated by "unauthorized" modification of the computer's routing table. We've had multiple issues in which there is nothing indicated on the logs, but later on we determined the cause is the same.

Launch NC troubleshooting (Start > All Programs > Juniper Networks > Network Connect 7.1.0
On the Logs tab, select "Detailed Info"
On the Information tab, select "Show All" from the dropdown, and click "Copy to Log"
On the Diagnostics tab, click "Start Diagnostics, then after completing successfully, click "Copy to Log"
Connect via VPN.
Once the connection breaks and you get the error message, go back to NC Troubleshooting > Logs tab > click Explore Log Files
Search each of the log files for "unauth"

Workaround:

First thing to check is the Bonjour service version installed; versions earlier than 1.06 are known to cause this issue.

Next, check the computer for any device management applications/application suites. These refers to any device managed by this computer remotely over the network (wired or wireless), such as printers, scanners, fax, modems, and cameras. If so, turn them off (via Task Manager) and connect again to test.

Apple iOS Upgrade Not Working over Wireless

2012-03-20T21:39:00.002+08:00

Some iPhone and iPad users are trying to update to iOS 5 through wireless connection but fails.

Apple iOS OTA (over-the-air) update is not proxy-aware.

To check if using proxy, go to Settings > Wi-Fi > the name wifi network, and scroll down to HTTP Proxy.

Workaround: use other methods to update (such as using wired connection iTunes).

Laptop WiFi Connection fails

2012-03-20T17:57:00.000+08:00

Laptop cannot get any IP address on the wireless adapter; previously connected via both wired and wifi.
AlwaysResolved by toggling or turning off/on the wifi switch of the laptop.

Windows XP, Windows 7

Checking the AP and/or WLC shows authentication failure due similar to below:

Mar 12 06:54:04.731: %DOT11-4-MAXRETRIES: Packet to client ABCD.EFGH.IJKL reached max retries, removing the client
Mar 12 06:54:04.731: %DOT11-6-DISASSOC: Interface Dot11Radio0, Deauthenticating Station ABCD.EFGH.IJKL Reason: Previous authentication no longer valid
Mar 12 06:54:04.738: %DOT11-4-MAXRETRIES: Packet to client ABCD.EFGH.IJKL reached max retries, removing the client

This happens usually during the following situations:

laptops go back from standby/hibernate mode;
when laptop is using both wired and wifi connections then disconnects from wired; or
combination of above both situations.

Resolution:
Change 802.1X Authentication to use 'user authentitcation' only. Computer authentication uses only the cached information which maybe stale, inactive, and/or expired. Specifying user authentication forces the laptop to re-authenticate with the authentication servers -- essentially 'refreshing' the authentication information.

Note that this may not apply to every office network.

FileZilla FTP Client fails through BlueCoat Proxy

2011-12-05T14:34:00.000+08:00

Symptoms:

When using FileZilla FTP client through a bluecoat proxy, the FTP connection attempt fails with the below error message:

Status: Using proxy ftpproxy.mydomain.com

Status: Resolving address of ftpproxy.mydomain.com

Status: Connecting to 10.192.5.37:21...

Status: Connection established, waiting for welcome message...

Response: 220 Blue Coat FTP Service

Command: USER ftpuser@targetftpsite

Response: 331 Enter password.

Command: PASS ********

Response: 332 Enter proxy password.

Error: Login sequence fully executed yet not logged in. Aborting.

Error: Server might require an account. Try specifying an account using the Site Manager

Error: Critical error

Error: Could not connect to server

Versions:

FileZilla FTP Client v3.5.2
The LAN proxy server is a BlueCoat proxy server

Resolution/Workaround:

1. Go to FileZilla Client > Edit > Settings

2. In Connection > FTP > FTP Proxy

2.1. Click on Custom and do the following:

2.2. Under Custom, enter the following:

USER %u@%h

PASS %p

ACCT %a

2.3. Proxy host: <your_bluecoat_proxy_server>

3. Click OK.

Source:

http://forum.filezilla-project.org/viewtopic.php?p=19457#p19457

MPLS Label Distribution Parameters

2011-05-30T09:46:00.066+08:00

MPLS Label Distribution Parameter Options:

Label Space Options	Per-Interface	Per-Plaform
Label Distribution	Unsolicited downstream	Downstream-on-Demand
Label Allocation	Independent Control Mode	Ordered Control Mode
Label Retention	Liberal Label Retention	Conservative Label Retention

Label Space Options

Per-Platform Labels

LFIB: [Label | Action | Next Hop ]
one label assigned to a destination network and announced to all neighbors
locally unique, valid on all incoming interfaces
smaller LFIB, FIB; faster label exchange
less secure than per-interface labels; label-spoofing
default in Frame-Mode MPLS

Per-Interface Labels

LFIB:[IN intf | IN Label | NH intf | NH Label]
one label for each destination, each device, each interface
secure; prevents label-spoofing; labeled packets/ATM cells only accepted from interface where label is assigned
default in Cell-Mode MPLS

Label Distribution

Unsolicited Downstream

label advertised to all neighbor LSRs, regardless of whether upstream or downstream
Frame-Mode

Downstream-On-Demand

label advertised only upon request of upstream LSR
cell-mode

Label Allocation

Independent Control

LSR can assign a label for a prefix w/o outgoing/downstream label
for edge LSRs only (requires L3 capabilities)
faster label propagation
for unsolicited downstream / frame-mode

Ordered Control

local label allocated and propagated only
if exist(nexthop label) propagae/allocate label, else request label from nexthop
for downstream-on-demand/cell-mode

Label Retention

Liberal Label Retention

all received label stored in LIB, even if not from next-hop LSR
improves convergence speed; allows easy failover for link failures
frame-mode

Conservative Label Retention

only received labels from nexthop LSRs are stored; others are ignored
downstream-on-demand/cell-mode

Standard Parameter Sets for Cisco Platforms

Parameter	Routers, frame interfaces	Routers, ATM interfaces	ATM switches
Label Space	per-platform	per-interface	per-interface
Label Distribution	unsolicited downstream	downstream-on-demand	downstream-on-demand
Label Allocation	independent control	independent control	ordered control
Label Retention	liberal label retention	conservative OR liberal	conservative

BGP Route Dampening

2011-05-25T01:02:00.000+08:00

At a Glance:

designed to reduce router processing load caused by unstable routes;
prevents sustained routing oscillations without affecting well-behaved routes;
RFC 2439: BGP Route Flap Dampening;
minimizes BGP updates by suppressing unstable routes.

Route-Dampening Operation:

EBGP route flaps = 1000 penalty points

IBGP route flaps not dampened;
penalty not user-configurable;

penalty decays via exponential decay algorithm

if penalty > suppress limit, route is dampened

if penalty < reuse limit, dampened route is propagated

if penalty < 1/2 reuse limit, flap history forgotten

route is never dampened more than the maximum suppress time limit
unreachable route with flap history is in history state (still in BGP table to retain flap history)
penalty is applied to an individual path, not the prefix

Configuring Route Dampening:

router(config-router)# bgp dampening [half-life reuse suppress max-suppress-time] [route-map route-map-name]
half-life	time for penalty to halve (default 15 minutes)
reuse	penalty points when dampened route is reused (default 750)
suppress	penalty points when route is suppressed (default 2000)
max-suppress-time	maximum time a route is suppressed (default 1hour; maximum 255 minutes)

router(config-route-map)#
set dampening half-life reuse suppress max-suppress-time

- used for less aggressive dampening of routes towards root DNS servers
- dampening of smaller prefixes more aggressively
- selective dampening based on BGP neighbors and route-map match criteria

Other Commands:

clear ip bgp ip_addr flap-statistics [{regexp regexp} | {filter-list listname} | {ipaddr mask} ]

clear ip bgp dampening [ipaddr mask]

show ip bgp dampened-paths

show ip bgp flap-statistics [{regexp regexp}|{filter-list listname}|{ipaddr mask [longer-prefix]}]

debug ip bgp dampening

BGP: Regular Expressions for AS-PATH Filtering

2011-05-23T18:13:00.001+08:00

Symbol	Description
\|	logical OR
.	match any
[x..y]	match one in range
^	match beginning of string
$	match end of string
_	match any delimiter/white space
( )	group as a single atom
*	match 0 or more instances of previous atom
?	match 0 or 1 instance of previous atom
+	match 1 or more instances of previous atom
\	escape character; if followed by a number n, points to the nth atom

Example 1: Advertise routes with empty AS-PATH (internal routes)

router bgp 123
neighbor 5.6.7.8 remote-as 387
neighbor 5.6.7.8 filter-list 1 out
!
ip as-path access-list 1 permit ^$
!match "blank" atom at the "start" and "end" of string
!

Example 2: Accept only default routes, preferring primary route based on AS PATH:

	AS387 (primary ISP)
	/
AS123
	\
	AS462 (backup ISP)

router bgp 123
neighbor 1.2.3.4 remote-as 462
neighbor 1.2.3.4 route-map FILTER in
neighbor 5.6.7.8 remote-as 387
neighbor 5.6.7.8 route-map FILTER in
!
route-map FILTER permit 10
! default routes from primary ISP (AS 387) accepted are preferred (larger weight)
match ip prefix-list DEFAULT_ONLY
match as-path 10
set weight 150
!
route-map FILTER permit 20
! default routes from backup ISP are accepted, with lower preference than primary ISP routes
match ip prefix-list DEFAULT_ONLY
set weight 100
!
ip as-path access-list 10 permit _387$
ip prefix-list DEFAULT_ONLY seq 10 permit 0.0.0.0/0
!

Example 3: AS PATH Filtering with AS Path Prepending:

- customer in AS123 is performing AS-PATH pre-pending

AS123
10.0.0.1	\
	\
	AS462

router bgp 387
neighbor 10.0.0.1 remote-as 213
neighbor 10.0.0.1 filter-list 10 in
!
ip as-path access-list 10 permit ^123(_123)*$
! accepts "123", "123 123", or "123 123 123"

Example 4: AS PATH Filtering with AS Path Prepending, multiple customers:

- multiple customers performing AS PATH Prepending

Customer 1
	\
Customer 2 -	AS387
	/
Customer 3

router bgp 387
neighbor 10.0.0.1 remote-as 123
neighbor 10.0.0.1 filter-list 10 in

neighbor 20.0.0.1 remote-as 456

neighbor 20.0.0.1 filter-list 10 in

neighbor 30.0.0.1 remote-as 789

neighbor 30.0.0.1 filter-list 10 in

!
ip as-path access-list 10 permit ^([0..9]+)(_\1)*$
! accepts repeating instances of "123", "456", and "789"
! does not accept strings non-repeating strings (e.g. "123 123 100")
!

Notes:
atom 1 = at least one instance of a number at the beginning of the string;
atom 2 = 0 or more instances of a whitespace and atom 1 until the end of the string.

BGP Path Attributes and Route Selection

2011-05-04T12:02:00.000+08:00

BGP Path Attributes

Mandatory Well-Known
Origin (i, e, ?)
AS-Path (sequence of AS-Numbers to access network/IP)
Next-Hop (ip address)

Discretionary Well-Known
Local Preference (for routing policy)
Atomic Aggregate (flags route if aggregated)

Optional Non-Transitive
Multi-Exit Discriminator (MED) - multiple entry pts to one AS
Originator-ID - for route reflector environment
Cluster-List - for route reflector environment

Optional Transitive
Aggregator - IP address & AS of routers that aggregated routes
Community - for route tagging

BGP Route Selection Order:

Prefer highest weight (local to router)
Prefer highest local preference (global within AS)
Prefer routes that the router originated
Prefer shorter AS paths
Prefer lowest origin code (IGP < EGP < Incomplete)
Prefer lowest MED
Prefer external (EBGP) paths over internal (IBGP)

For IBGP paths, prefer nonreflected routes (no originator-ID) over reflected routes
For reflected routes, prefer shorter cluster-list

For IBGP paths, prefer path through closest IGP neighbor
For EBGP paths, prefer oldest (most stable) path
Prefer paths from router with the lower BGP router-ID

BGP: Multihomed Customer to Single ISP in Load-Sharing Setup with Static Routes

2011-05-04T11:22:00.001+08:00

Outbound Traffic (CE to PE):
- each customer router uses closest CE as exit point;
- CE routers must be collocated to have load-sharing;

Inbound/Return Traffic (PE to CE) (pre-IOS 12.2):
- true load-sharing is impossible to achieve with multiple PEs;
- per BGP route selection, only one route will be the best route (to the customer network)
- can be optimized by dividing the customer address space

Customer Network: 11.2.3.0 /24
Customer Network "division" assigned to PE1: 11.2.3.0/25
Customer Network "division" assigned to PE2: 11.2.3.128/25

Configurations:

PE1 Config:
!
ip route 11.2.3.0 255.255.255.128 serial 0 tag 1000
ip route 11.2.3.0 255.255.255.0 serial 0 tag 1000
!
router bgp 123
redistribute static route-map INTOBGP
...
!

PE2 Config:
!
ip route 11.2.3.128 255.255.255.128 serial 0/0 tag 1000
ip route 11.2.3.0 255.255.255.128 serial 0/0 tag 1000
!
router bgp 123
redistribute static route-map INTOBGP
...
!

BGP: Multihomed Customer to Single ISP in Primary-Backup Setup with Static Routes

2011-05-04T11:00:00.003+08:00

Scenario:
Multihomed customer connected to a single service provider on multiple permanent links;
Customer network using OSPF; provider network using BGP; CE-PE via static routing;
Floating static routes configured as backup on both CE and PE;

Floating Static Routes in BGP:
Once active, the floating static routes will be permanently installed in BGP; static routes are locally sourced which is preferred;

admin distance cannot be used in route-maps; hence, use communities
tag floating static routes; tags mapped to specific communities;
use route maps to modify the weight and/or local preference;
default weight: 32768

Sample ISP Route-Community Mapping:

ISP (AS 123) Service Offerings
Advertise CU Routes	Primary?	Route Tag	Community	LocalPref
No	Yes	1000	no-export 123:31000	100
No	No	1010	no-export 123:31000	50
Yes	Yes	1001	123:31000	100
Yes	No	1011	123:31000	50

Customers Configurations:

CE1 Config (Primary link):
ip route 0.0.0.0 0.0.0.0 Serial0
!
router ospf 1
default-information originate
!

CE2 Config (Backup link):
ip route 0.0.0.0 0.0.0.0 Serial0 250
!
router ospf 1
default-information originate
!

Provider Configurations:

PE1 Config (Primary link):
ip route 11.2.3.0 255.255.255.0 serial 0 tag 1000
!
router bgp 123
redistribute static route-map INTOBGP
!

PE2 Config (Backup link):
ip route 11.2.3.0 255.255.255.0 serial 0/0 tag 1010 250
!
router bgp 123
redistribute static route-map INTOBGP
!

Common Route-Map Configurations on both PE1 and PE2:

route-map INTOBGP permit 10
match tag 1000
set community no-export 123:31000
set local-preference 100
!
route-map INTOBGP permit 20
match tag 1001
set community 123:31000
set local-preference 100
!
route-map INTOBGP permit 30
match tag 1010
set community no-export 123:31000
set local-preference 50
set weight 0
!
route-map INTOBGP permit 40
match tag 1011
set community 123:31000
set local-preference 50
set weight 0
!

IOS: EIGRP Peering Flapping, Auth Failure - %DUAL-5-NBRCHANGE: IP-EIGRP: Auth failure

2010-11-04T14:52:00.000+08:00

This is an actual case we have raised recently with Cisco as we are having unexplained EIGRP flaps between two of our devices. It has been working for more than a year -- actually, it never had any issues when this was brought online last year.

Scenario:

EIGRP peering flaps between two devices, due to authentication failure. The output of show logging is flooded with the below syslogs repeatedly:

Nov 2 01:30:43.436 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is down: Auth failure

Nov 2 01:30:45.040 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is up: new adjacency

Nov 2 01:30:47.316 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is down: Auth failure

Nov 2 01:30:48.820 GMT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 2: Neighbor 10.10.10.2 (GigabitEthernet1/1) is up: new adjacency

Topology is straightforward:

Router1 Gi1/1 <-----> Gi2/2 Router2

Router1 Gi1/1 = 10.10.10.1/24
Router2 Gi2/2 = 10.10.10.2/24

MD5 Authentication is used and the same key string is configured on both devices

Router1#show key chain MYCHAIN

key-chain MYCHAIN
key 1 -- text "myCiscoChain"
accept lifetime (always valid) - (always valid) [valid now]
send lifetime (always valid) - (always valid) [valid now]
Router1#
Router1#
Router1#
Router1# show run | begin key chain
key-chain MYCHAIN
key 1
key string 5 098123456SA679
...
Router1# show run int Gi1/1
interface GigabitEthernet1/1
ip address 10.10.10.1 255.255.255.0
ip authentication mode eigrp 2 md5
ip authentication key-chain eigrp 2 MYCHAIN
...

Problem:
The issue was with a Level2/Severe bug with the IOS image running on one of the devices. Bug details below:

CSCdu73495 - All routes to network not seen because of invalid md5 authentication
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCdu73495

Enhanced Interior Gateway Routing Protocol (EIGRP) routes cannot be seen even when message digest algorithm 5 (MD5) is authenticated on all routers. This problem is intermittent and may occur when authentication is turned off and subsequently turned back on again. Sometimes, this problem occurs just after authentication is enabled.

Workaround: This problem is intermittent and may be resolved by disabling and reenabling authentication a second time. This problem may automatically be resolved after a few minutes.

EIGRP Authentication problems & flaps on unrelated links

This bug is a duplicate of CSCdu73495, which causes authentication-related breakage in establishing peers, which eventually clears up on it's own after an indeterminate time. It can be triggered by bouncing peers/interfaces. You will not encounter this issue if you disable EIGRP authentication. CSCdu73495 was resolved in later versions of 12.1E IOS.

EIGRP neighbour cant be established if use MD5 authentication

C2610 EIGRP neighbour could be established via md5 authentication first time. After shut/no shut c2610 ethernet interface, it can't established any more. Via serial interface works fine.

EIGRP MD5 Authentication Breaks Neighbor Adjacencies over LANE

In a LANE environment with 3 or more devices running EIGRP, when upgrading from 12.1(6)E4 to 12.1(10)E4 on 7500's, EIGRP neighbor relationships may not be formed between devices running 12.1(10)E4. This is verified by performing a on one of the devices running 12.1(10)E4. The workaround for this scenario is to wait an unpredictable amount of time for the neighbors to converge, or remove and re-add EIGRP authentication from the interfaces on the affected devices. Also, neighbors can be statically configured in order for EIGRP to use unicast, rather than multicast.

2921-EIGRP flap due to bad TLV received on serial interface

Symptom: EIGRP flaps observed due to retransmission retry limit exceeded. Bad TLV error messages are seen in the logs. Conditions: Issue seen when 2921 replaces the 2611 device with similar configs.

Workaround: None. Apart from 2921, customer is using 2611 that works fine.

Known Affected Versions (Not a comprehensive list):
12.1(9)M

12.1(26)M
15.0M
12.1(8b)E15
12.3(12e)M

Fixed-In (Not comprehensive list):

12.1(10.2)M
12.2(4.2)M
12.0(30)SZ4
12.0(32)S6b
12.0(32)S7
12.0(32)SY4
12.0(32.3)S
12.1(6)E11
12.1(10.5)E
12.1(10.5)EC
12.2(4.2)PI
12.2(4.2a)DA
12.2(5.1)S
12.2(6.4)B
12.2(6.4)PB
12.2(15)BW
12.2(15)BX
12.2(15)ZN
12.0(32.11.1)SY

Workarounds:

1. Disable then re-enable EIGRP authentication;
2. Instead of MD5, use clear text authentication; or
3. Disable EIGRP authentication.

Permanent Fix:
Upgrade IOS version.

Due to intermittence/unpredictability, either use clear text authentication or disable authentication outright if IOS upgrade is not possible immediately. However, bouncing (disable/re-enable) the authentication can serve as your quick fix.

IOS: show interface FastEthernet mod/port - Detailed

2010-10-26T12:11:00.000+08:00

The show interface output for physical interface

Router#sh interfaces FastEthernet 6/1

FastEthernet6/1 is up, line protocol is up (connected)
Hardware is C6k 100Mb 802.3, address is 0009.11f3.8848 (bia 0009.11f3.8848)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Full-duplex, 100Mb/s
input flow-control is off, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:14, output 00:00:36, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1117058 packets input, 78283238 bytes, 0 no buffer
Received 1117035 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
285811 packets output, 27449284 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Show interface output (physical interface) explained

up, line protocol up (connected) - the "up" is the physical layer (OSI layer 1) status of the link; the "line protocol up" is the data link layer (OSI layer 2) status of the link. Possible output are as follows:

up, line protocol up
up, line protocol down
down, line protocol down

Hardware - the interface hardware type, as well as the hardware/MAC address.

Description - the user-specified interface description as configured in the interface configuration mode.

MTU 1500 bytes - Maximum Transmission Unit.
BW - Bandwidth.
DLY - Delay (in microseconds).

reliability - reliability, as fraction of 255 (where 255/255 = 100% reliability), exponential average over 5 minutes.
txload - current output load, as fraction of 255 (where 255/255 = 100% saturation), exponential average over 5 minutes.
rxload - current input load, as fraction of 255 (where 255/255 = 100% saturation), exponential average over 5 minutes.

Encapsulation - current data link/layer 2 encapsulation of the interface.
loopback - defines if loopback (hardware or software) is enabled or disabled.

Full-duplex, 100Mb/s - current duplex and speed settings of the interface.

ARP Type - the Address Resolution Protocol type enabled.
ARP Timeout - the time in hh:mm:ss for each entry remains in ARP cache before being removed.

Last input 00:00:14, output 00:00:36 - the time in hh:mm:ss when the last packet was received (input) or transmitted (output) by the interface.

output hang - the time in hh:mm:ss when the interface was reset because of a transmission that took too long.

Last clearing of "show interface" counters - the time when the interface counters are last cleared via "clear counter" command.

Input queue: 0/2000/0/0 (size/max/drops/flushes) - the input queue counters and thresholds; the first number (size) is the current number of frames in the queue; the second number (max) is the maximum number of frames in the queue before it starts dropping; the third number (drops) is the number of frames dropped because the max was exceeded; the last number (flushes) is the number of low-priority frames dropped due to Selective Packet Discard (SPD) algorithm when CPU is overloaded.

Total output drops: 0 - total number of packets dropped because the output queue is full; high output drops may indicate mismatched bandwidth settings of this and the remote connecting interface.

Queueing strategy - either First-In/First-Out (fifo), priority-list, custom-list, and weighted-fair.

Output queue :0/40 (size/max) - The number of packets in the output queue. Size is the current number of frames in the queue. Max is the number of frames the queue can hold before it starts dropping frames.

5 minute input/output rate - The average input and output rate seen by the interface in the last five minutes. The interval can be changed via the "load-interval " interface command.

packets input, bytes - Total number of error-free packets received by the system. Total number of bytes, including data and MAC encapsulation, in the error free packets received by the system.

no buffer - The number of packets received and discarded because there is no buffer space. Can be caused by broadcast storms.

Received broadcasts - The number of broadcast/multicast packets received by the interface.

runts - The number of packets that are discarded because they are smaller than the minimum packet size.

giants - The number of packets that are discarded because they exceed the maximum packet size (MTU).

throttles - The number of times the receiver on the port was disabled, possibly due to buffer or processor overload.

input errors - Includes runts, giants, no buffer, CRC, frame, overrun, and ignored counts.

CRC - Number of packets where the CRC generated by the originating far-end device does not match the checksum calculated from the data received; usually indicates noise or transmission problems on the LAN interface or the LAN bus itself.

frame - Number of packets received incorrectly having a CRC error and a noninteger number of octets A(elignment errors).

overrun - Number of times the receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver's ability to handle the data.

ignored - Number of received packets ignored by the interface because the interface hardware ran low on internal buffers.

watchdog - Number of times watchdog receive timer expired. It happens when receiving a packet with length greater than 2048.

multicast - Number of multicast packets received.

pause input - Number of times the connected device requests for a traffic pause when its receive buffer is almost full. This counter is incremented for informational purposes, since the switch accepts the frame. The pause packets stop when the connected device is able to receive the traffic.

input packets with dribble condition detected - A dribble bit error indicates that a frame is slightly too long. This frame error counter is incremented for informational purposes, since the switch accepts the frame.

packets output, bytes - Total number of error-free packets transmitted by the system. Total number of bytes, including data and MAC encapsulation, in the error free packets transmitted by the system.

underruns - Number of times that the transmitter has been run faster than the switch can handle. This can occur in a high throughput situation where an interface is hit with a high volume of bursty traffic from many other interfaces all at once. Interface resets can occur along with the underruns.

output errors - Sum of all errors that prevented the final transmission of datagrams out of the interface.

collisions - Number of times a collision occurred before the interface transmitted a frame to the media successfully. Collisions are normal for interfaces configured as half duplex but must not be seen on full duplex interfaces. If collisions increase dramatically, this points to a highly utilized link or possibly a duplex mismatch with the attached device.

interface resets - Number of times the interface transitioned from up to down to up.

babbles - Number of times that the transmit jabber timer expired. A jabber is a frame longer than 1518 octets (which exclude framing bits, but include FCS octets), which does not end with an even number of octets (alignment error) or has a bad FCS error.

late collision - Number of times a late collision occured. A late collision occurs when two devices transmit at the same time, and neither side of the connection detects a collision. The reason for this occurrence is because the time to propagate the signal from one end of the network to another is longer than the time to put the entire packet on the network. The two devices that cause the late collision never see that the other is sending until after it puts the entire packet on the network. Late collisions are not detected by the transmitter until after the first 64 byte slot time. This is because they are only detected in transmissions of packets longer than 64 bytes.

deferred - Number of frames that have been transmitted successfully after they wait because the media was busy. This is usually seen in half duplex environments where the carrier is already in use when it tries to transmit a frame.

lost carrier - The number of times the carrier was lost in transmission. This is usually caused by a bad cable. Check the physical connection on both sides.

no carrier - Number of times the carrier was not present in the transmission. This is usually caused by a bad cable. Check the physical connection on both sides.

output buffer failures, output buffers swapped out - Number of failed buffers and the number of buffers swapped out. A port buffers the packets to the Tx buffer when the rate of traffic switched to the port is high and it cannot handle the amount of traffic. The port starts to drop the packets when the Tx buffer is full and thus increases the underruns and the output buffer failure counters. The increase in the output buffer failure counters can be a sign that the ports are run at an inferior speed and/or duplex, or there is too much traffic that goes through the port.

Reference: Troubleshooting Switch Port and Interface Problems
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015bfd6.shtml

CatOS: %SYS-3-PORT_OUT_DISCARD flood on disabled switchports

2010-10-26T04:57:00.000+08:00

Scenario
Multiple %SYS-3-PORT_OUT_DISCARD syslogs are generated for a switchport which is currently disabled/administratively shutdown.

2010 Jan 04 16:04:10 EST -05:00 %SYS-3-PORT_OUT_DISCARD:Port 4/47 detected 6029 output discard error(s) in last 30 minutes

MySwitch> sh port status 4/47
# = 802.1X Authenticated Port Name.

Port Name Status Vlan Duplex Speed Type
----- -------------------- ---------- ---------- ------ ----------- ------------
4/47 disabled 66 full 100 10/100/1000
MySwitch>

MySwitch> (enable) sh run 4
... [output omitted] ...
#module 4 : 48-port 10/100/1000BaseT Ethernet
set port disable 4/3-48

Explanation
This is due to an identified bug on the CatOS version.

CSCeg24345 - WS-X6748-GE-TX: Tx counters increment on not connected ports

WS-X6748-GE-TX: Tx counters increment on not connected ports On a WS-X6748-GE-TX module in a Catalyst 6500 running CatOS 8.2(2), a port that is not-connected may increment Tx counters as well as ifOutErrors, ifOutDiscards and txCRC

This bug impacts CatOS releases prior to 8.6 and occurs on WS-X6748-GE-TX blades. It is a cosmetic bug and is non-service impacting.

Versions:

1st Found-In: 8.2(2)

Fixed-In : 8.4(3.2), 8.4(4), 8.6(0.85)TAL

Note that although this is not service-impacting, it may wreck havoc on your monitoring system, as it will generate one syslog for each disabled port every thirty minutes.

IOS: %SYS-SP-3-CPUHOG: RFSS_server_action

2010-08-16T10:18:00.000+08:00

Scenario:
A Cat6K throws the following syslog messages:

Jul 18 01:48:12.362 EDT: %SYS-SP-3-CPUHOG: Task is running for (4000)msecs, more than (2000)msecs (0/0),process = RFSS_server_action.

-Traceback= 4045D2CC 4045F5F8 4045F504 4047F45C 4047ED38 4047F31C 40481F5C 40489F04 4048A3CC 4048AF5C 40485DE4 4048B1AC 404816A8 402E41D8 40451534 4029A764

Jul 18 01:48:14.366 EDT: %SYS-SP-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs (1/0),process = RFSS_server_action.

-Traceback= 4045D2A8 4045F5F8 4045F504 4047F45C 4047ED38 4047F31C 40481F5C 40489F04 4048A3CC 4048AF5C 40485DE4 4048B1AC 404816A8 402E41D8 40451534 4029A764

Jul 18 01:48:18.370 EDT: %SYS-SP-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs (2/1),process = RFSS_server_action.

-Traceback= 4045D2A8 4045F5F8 4045F504 4047F45C 4047ED38 4047F31C 40481F5C 40489F04 4048A3CC 4048AF5C 40485DE4 4048B1AC 4048B504 404817C0 402E440C 40451660

Description:

The traceback shown indicates a problem with writing into the flash disk. Running the privileged-mode command "dir disk1:" will cause your login session to apparently hang for a few minutes. After that the logs will be filled up with a new batch of the above %SYS-SP-3-CPUHOG syslogs and traceback messages.

------------------ show disk1: all ------------------
172683264 bytes available (83296256 bytes used)
******** ATA Flash Card Geometry/Format Info ********
ATA CARD GEOMETRY
Number of Heads: 16
Number of Cylinders 978
Sectors per Cylinder 32
Sector Size 512
Total Sectors 500736

ATA CARD FORMAT
Number of FAT Sectors 245
Sectors Per Cluster 8
Number of Clusters 62495
Number of Data Sectors 500596
Base Root Sector 598
Base FAT Sector 108
Base Data Sector 630 %

Error show disk1: (TF I/O failed in data-in phase)

Workaround/Resolution:

Reseat the compact flash card.
If error still occurs, reformat the flash card.
If error still occurs, replace the flash card.

FortiOS v3.00 MR5 - CPU Usage Too High

2010-07-05T20:37:00.001+08:00

Problem:

Fortigate 3600 running version 3.00 MR5 Patch 2 keeps sending high CPU trap SNMP traps to the SNMP trap servers. CPU utilization is confirmed to be high, based from the output of “get system performance status” or from the GUI. From “diag sys top”, confirmed that the “merged_daemons” process is using 99% of the total CPU, then shortly goes down to 14%.

Cause:
This is due to bug documented below:

0062617: race condition in flgd can cause merged_daemons to spin
The merged_daemons was constantly in the 'R' state and consuming 99% of CPU (when top is first started, the usage will display as 99% -- the usage will decrease to 14% while top is running).

Fix: Build: 0566

Workaround:
Restart merged_daemons as follows:

Enter diag sys top and take note of the PID of merged_daemons
Enter diagnose sys kill 11 [pid]

Note that merged_daemons may still climb back up to 99%.

Resolution/Workaround:
Upgrade to FortiOS MR6 or later.

IOS: %EARL_L3_ASIC-SP-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt

2010-01-04T16:25:00.002+08:00

Dec 18 09:54:43.989 JST: %EARL_L3_ASIC-SP-STDBY-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt
Dec 18 09:54:43.993 JST: %EARL_L3_ASIC-SP-3-INTR_WARN: EARL L3 ASIC: Non-fatal interrupt Packet Parser block interrupt

Description
These messages are indicating that the switch has received an invalid packet which contained a Layer 3 IP checksum error. These packets are normally being dropped silently within older IOS. In some IOS releases, the switch informs of this condition to warn users that there is (are) devices outside sending IP packets with checksum errors and/or with wrong length.

See CSCdz10360 (Need a CLI to be able to disable L3 error checking in HW) regarding this enhancement.

Workaround
These messages are purely informational. You may either:

SPAN all the Vlans and look at layer3 IP source address then remove the device generating invalid packets (unfortunately the switch doesn't track the IP address. The only way is to sniff every suspected Vlan to find out where those invalid packets are coming from).

Configure (this is a new config option added by means of CSCdz10360):
no mls verify ip checksum ---> to stop to check for packet checksum errors
no mls verify ip length ---> to stop to check for packet length errors
no mls verify ip length minimum ---> to eliminate check for IP packets that are minimum length.
no mls verify ip same-address ---> to stop checking for packet having equal source and destination IP address.

Do nothing as these are pure informational.

IOS: %ETHCNTR-3-LOOP_BACK_DETECTED : Keepalive packet loop-back detected on [chars]

2010-01-04T16:13:00.003+08:00

Scenario
The switch reports this error message, and the port is forced to linkdown:
%ETHCNTR-3-LOOP_BACK_DETECTED : Keepalive packet loop-back detected on [chars]

Oct 2 10:40:13: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on GigabitEthernet0/1
Oct 2 10:40:13: %PM-4-ERR_DISABLE: loopback error detected on Gi0/1, putting Gi0/1 in err-disable state

Description
The problem occurs because the keepalive packet is looped back to the port that sent the keepalive. Keepalives are sent on the Catalyst switches in order to prevent loops in the network. Keepalives are enabled by default on all interfaces. You see this problem on the device that detects and breaks the loop, but not on the device that causes the loop.

Workaround
Issue the no keepalive interface command in order to disable keepalives. A disablement of the keepalive prevents errdisablement of the interface, but it does not remove the loop.

Permanent Fix
In Cisco IOS Software Release 12.2(x)SE-based releases and later, keepalives are not sent on fiber and uplink interfaces by default. Upgrading the IOS version to this or later images should prevent the above issue in the first place.

Fortigate: Password recovery

2009-12-01T21:17:00.002+08:00

To reset the FortiGate unit password:

Connect the terminal to the FortiGate unit using the null modem cable.
Log on at the console with the user name "maintainer" and password "bcpb" followed immediately by the unit serial number. You must enter the alphabetic characters of the serial number in upper case.
Enter the following commands:
config system admin
edit admin
set password mypassword
end

Special Notes:

You must first power down the FortiGate unit, and power it up again.
Follow the above steps within one minute of the restart

If the maintainer login is initially unsuccessful, try the following two tips to ensure successful login:

You many not have the correct serial #. Copy the serial # displayed on the console during initial boot-up and paste it into a terminal editor window.
In the terminal editor window, finish composing the full password by adding "bcpb" before the serial # and then copy & paste the entire password into the console.

F5 BIGIP: Verify/Restart SNMP Daemon

2009-11-25T00:24:00.004+08:00

Just in case you need to check the status and/or restart the SNMP daemon of the bigip (i.e., because it has stopped responding to SNMP polling), enter the following commands via the CLI:

For BIGIP v4

Check the SNMP daemon status:
/etc/bigstart/status/S40snmpd status

The correct output should be:
Status snmpd: (pid xxxxx) is running
Status bigsnmpd: (pid yyyyy) is running
Status rlxsnmpd: is not running

If the result is different from above (i.e., bigsnmpd is not running), restart the SNMP daemon:
/etc/bigstart/status/S40snmpd restart

For BIGIP v9

Check the current status of the SNMP daemon:
bigstart status snmpd

Restart the SNMP daemon
bigstart restart snmpd

Verify status of the SNMP daemon:
bigstart status snmpd

[root@bigip:Active]~# bigstart status snmpd
snmpd run (pid 12707) 90 days, 1 start
[root@bigip:Active]~# bigstart restart snmpd
[root@bigip:Active]~# bigstart status snmpd
snmpd run (pid 4822) 6 seconds, 2 starts
[root@bigip:Active]~#

CatOS : SYS-2-MOD_TEMPSENSORFAIL flood from X6148A-GE-45AF

2009-11-11T01:24:00.004+08:00

CSCsl37513
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsl37513

SYS-2-MOD_TEMPSENSORFAIL:Module w/ X6148A-GE-45AF and CatOS

Symptom:
Numerous WS-X6148 linecards generate the following error:

%SYS-2-MOD_TEMPSENSORFAIL:Module # temperature sensors failed, please %powercycle the module

Conditions:
No production impact related to this message.

Workaround:
Powercycle module as requested by the error message.

set module power down module_number
set module power up module_number

Permanent Fix:
Upgrade IOS/CatOS to the below versions or later:
8.7(0.22)FW124
8.7(1.62)LAR
8.6(5.7)
8.7(0.22)BUB48
12.2(33.3.13)SXH
12.2(33)SXH4

IOS: IP SLA : SNMP : Router crashes and reloads if up for more than 497 days

2009-11-11T01:24:00.003+08:00

CSCsa57468
rttmon-mib does not return getnext value when queried via snmp

Symptom:
Concord poller crashes when polling a router that has been configured with IP SLA. Infact this DDTS will surface when doing snmp gets for the objects mentioned in the Conditions section below coming from any NMS (e.g. Concord, IPM, Spectrum, etc.)

Conditions:
The SNMP GETNEXT request is sent to the router for the following OIDs:

rttMonJitterStatsCompletions
rttMonStatsCaptureCompletions
rttMonStatsTotalsInitiations
rttMonStatsCaptureEntry (rttMonStatsCaptureCompletion etc.)
rttMonStatsCollectEntry
rttMonStatsTotalsEntry
rttMonJitterStatsEntry
rttMonHTTPStatsEntry.

The router does not return the next index of these OIDs, but the same index. This happens only when the router has been up and running for longer than 497 days.

Affected IOS Versions:

12.2(15)T
12.2SXH

Workaround:
This problem is only happening when polling the CISCO-RTTMON-MIB via snmp get. Use the IOS CLI to avoid it.

Permanent Fix:
Upgrade the IOS version.

Fixed in:

12.3(14.12)M
12.4(1.5)M
12.2(33)SRC
12.2(40)SE
12.2(44)SE
12.3(11)T6
12.3(11)YW
12.3(14)T2
12.4(1.8)T
12.4(1a)M
12.2(33)SXI
12.2(32.8.80)SR
12.2(32.8.11)XID112.9
12.2(33.1.7)SXH
12.2(33)SXH2
12.2(33)SB
12.2(32.8.99a)SR133
12.2(32.8.11)XJC153.1

IOS: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key

2009-11-08T12:15:00.002+08:00

Symptoms:
The device getting numerous %SSH-3-PRIVATEKEY syslogs, usually followed by a traceback such as the following:

Nov 7 02:40:49.542 GMT: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key for
-Process= "SSH Process", ipl= 0, pid= 148
-Traceback= 61D48360 61D44B24 61D462C4 6053BD88 6053BD6C
Nov 8 02:16:22.452 GMT: %SSH-3-PRIVATEKEY: Unable to retrieve RSA private key for
-Process= "SSH Process", ipl= 0, pid= 148
-Traceback= 61D48360 61D44B24 61D462C4 6053BD88 6053BD6C

Explanation:
Often seen if hostname or domain name of the router has been changed.

Workaround/Fix:

Remove existing RSA Key:
crypto key zeroize rsa
Gnerate RSA key with the following commands:

show crypto key mypubkey rsa
crypto key gen rsa general-keys label label
ip ssh rsa keypair-name label

where label = unique label/identifier