The Frontline Express

Password Recovery on Cisco Catalyst Switch - Part 3

2008-10-04T21:32:00.004+08:00

Password Recovery on Modular Catalyst Switches (Big Boxes) Running IOS

For this last part, we will then go to the Password Recovery on modular type Catalyst Switches running IOS. This typically applies to the Supervisors of the Catalyst Switches.

These are:

Cisco Catalyst 4000 (Route Switch Module)
Cisco Catalyst 4500 (Supervisors 2+, 3, 4, 5, and 6)
Cisco Catalyst 6500

- Hybrid Mode (IOS on MSFC) or commonly known as Hybrid IOS or MSFC IOS

- Native Mode (IOS on both Supervisor and MSFC) or commonly known as Native IOS

- Applies to all Supervisor models

We can actually further divide this topic into different parts as follows:

Password Recovery on 4500 running IOS
Password Recovery on Catalyst 6500 with Hybrid IOS (MSFC IOS)
Password Recovery on Catalyst 6500 with Native IOS

The main reason these 3 are on a single category is that they have a lot of similarities, and we just need to take note of the minor differences.

Password Recovery on Catalyst Switches running IOS is very similar on how you do Password Recovery on Cisco Routers. The same concept of bypassing the start-up configuration (by setting the Configuration Register to 0x2142 value) applies to Catalyst Switches running IOS.

Do a cold boot.
During boot-up, do a break sequence to go to ROMMON mode.
While in ROMMON, change confreg value to 0x2142.
Do a reset command, or manually boot a desired image.
Say NO to initial configuration dialog.
Type enable to enter Privilege EXEC mode.
Restore the start-up configuration to the running configuration.
Change or remove the old passwords.
Save the changes.

To discuss it in details, the first thing that we need to accomplish is to break into the ROMMON of the Catalyst Switch. Break sequence differs between platforms and the Terminal Emulator being used. From my experience, it is usually either Ctrl+Break for HyperTerminal or Alt+B for TeraTerm.

In Catalyst 4500 Switch, you just need to power cycle it from the power supply unit (PSU). Once it starts its boot sequence, you'll be notified on when you can do a break sequence. Furthermore, the break sequence for Cat4500 IOS Switch is Ctrl+C.

For Catalyst 6500's MSFC IOS, you need go back to the Supervisor side (type Ctrl+C,C,C), and issue the reset 15 or reset 16 command. The command to be used depends on where the Supervisor module is inserted. If Supervisor module is in slot 1, slot 5, or slot7, then use reset 15 (it means that the Supervisor is on the first slot). Otherwise, if Supervisor module is in slot 2, slot 6, or slot8, then use reset 16 (it means that the Supervisor is on the first slot). This command will make the MSFC side (which is referenced as either in slot 15 or 16) to reboot or power cycle.

Afterwards, go back to the MSFC side immediately with a switch console command. Press Enter once in order to see the boot sequence. As soon as you're on the MSFC side again, do the break sequence in order to enter the ROMMON mode.

For Catalyst 6500 running Native IOS, power cycle it just like Cat4500 IOS. But you need to wait for the Supervisor side to boot up properly. Wait for it to change console ownership from Supervisor/Switch Processor (SP) to MSFC/Router Processor (RP). This is usually indicated by a noticeable informational console message. Just wait for it to show the version of the ROMMON, before doing the break sequence.

Once in ROMMON mode, it will be then the same for the 3 different platforms. Just type confreg 0x2142, and press enter. This command will change the configuration register value to 0x2142, which simply means that we're going to bypass any saved configuration once the Catalyst Switch boots up properly.

Note: If you're using one of the latest IOS version for the Cat4500 IOS, you may need to just enter the confreg command, and you'll be asked a few questions that in the end will result to the same configuration register value of 0x2142.

After this, just type reset in order to reload the Catalyst Switch once more. Do not interrupt the boot up sequence with any break sequence. Just let it boot up normally.

Once booted up, just type No when asked if you want to enter the Initial Configuration Dialog. To verify, you can do a show run command to see if it is on default configuration. Compare it with the output of the show start command to see that the saved configuration wasn't really applied to the Catalyst Switch.

You can then proceed in typing enable to enter the Privileged EXEC mode, without any need to enter any passwords. Afterwards, restore the saved configuration to the running configuration by typing copy startup-config running-config command.

To change or remove the password configurations, issue config terminal command to go to the global configuration mode, and then type enable secret to change the secret password, or no enable secret to remove secret password. Also type enable password to change the regular password, or no enable password to remove regular password. You may also want to change or remove telnet (line vty) and console (line console) passwords just like in the Password Recovery for small boxes.

While still in global configuration mode, we need to revert the configuration register value back to 0x2102. This is in order for the Catalyst Switch to automatically load any saved configuration upon boot up on succeeding reloads. To do this, type config-register 0x2102.

Lastly, issue write memory or copy run start to save the configuration with new passwords. This will also save the new configuration register value. And once again, we're done!

I hope this series helps you with all your Password Recovery needs in Catalyst Switches. Free to leave comments for any feedback! =)

Password Recovery on Cisco Catalyst Switch - Part 2

2008-09-27T23:08:00.007+08:00

Password Recovery on Catalyst Switches Running CatOS

Although there hasn't been any new Cisco Catalyst Switch platforms that are running CatOS, there's still a large customer base for the existing models. Just to note some of those that are still around:

Cisco Catalyst 2948G
Cisco Catalyst 2980G
Cisco Catalyst 4000 (Supervisors 1 and 2)
Cisco Catalyst 5000 (the whole line is already or nearing EOL)
Cisco Catalyst 6500 - Hybrid Mode (CatOS on Supervisor and IOS on MSFC). Applies to all Supervisor models

In CatOS Password Recovery, it is best that you know the whole picture, or all of the steps, first before you even try it. Why? Well, there's only a 30-second window to perform the CatOS Password Recovery upon boot up of a Catalyst Switch running CatOS.

Do a cold boot.
Wait for the Catalyst Switch to fully boot up.
Press Enter key at least once.
Type enable, and then press Enter key.
Type set ena, and press Enter for 4 times.
Type set pass, and press Enter for 4 times.
Configure new passwords, or leave them as NULL (or none).

Now to discuss it further, the first thing that you want to do is to do a cold boot (power off and then on) the Catalyst Switch. This is needed for the Cat OS Password Recovery to work (it won't work on a reloaded, or warm boot, Catalyst Switch. Afterwards, you just need to wait for it to fully boot up again (which will take around 2-3 minutes).

After the boot up, take note that you will only have 30-second window to perform the rest of the steps. The reason behind this is that there's no password configuration applied to the Catalyst Switch running CatOS within the first 30 seconds after boot up. With that, we can go beyond the console login, and the enable (Privileged EXEC) mode, without a need to type any password. Furthermore, the password is actually set to NULL within this timeframe, or similar to typing nothing.

So, as soon as you see the prompt for the console password, press Enter once and you'll see the Switch prompt. Just type enable and press Enter again, and you'll be inside the Privileged EXEC mode. Afterwards, you can now change the console and enable passwords to what ever you want. But in order to save time, you can set it to NULL for the meantime.

Note: In case these steps didn't work as expected, it may only mean that you're already outside the 30-second window, and needs to start from scratch again.

To change the enable password, type set pass and press Enter. You'll then be asked for the old password (which will be NULL for now) so you just need to press Enter again. The next two prompts will ask you to type a new password and to retype/confirm the new passwords. Just press Enter for these since we just want it to be set to NULL in order to make the 30-second window.

To change the console password, just do the same steps for changing the enable password, but this time, use set ena command instead.

After this part, you're done with the CatOS Password Recovery. You just need to do the set pass and set ena commands again in order to change the console and enable passwords to whatever you want. This time, you can leisurely do it with out any time limit to consider. Easy eh? =)

Password Recovery on Cisco Catalyst Switch - Part 1

2008-09-06T16:52:00.002+08:00

Password Recovery on Fixed Switches (Small Boxes)

For starters, here's a list of what I consider as Fixed Catalyst Switches or Small Boxes:

Cisco Catalyst 2900XL / 3500XL
Cisco Catalyst 2950
Cisco Catalyst 3500
Cisco Catalyst 2940
Cisco Catalyst 2960
Cisco Catalyst 2970
Cisco Catalyst 3560
Cisco Catalyst 3750

Now, here's simplified view of the procedures of Password Recovery on Small Boxes:

Power off the Catalyst Switch.
Hold down the MODE button before powering up again.
Go to switch: prompt.
Initialize the flash memory.
Rename the configuration file.
Manually boot up the IOS image.
Skip initial configuration dialog.
Go to Privileged EXEC (or enable) mode.
Restore the old configuration file name.
Restore the configuration to the Catalyst Switch.
Change or remove configured passwords.
Save configuration changes.

Note: It is assumed that you are already connected to the switch thru console cable, and seeing console output from a terminal emulator application like HyperTerminal in Windows.

On a more detailed manner, the Catalyst switch can be powered off by unplugging the power cable. Then, hold down the MODE button before reconnecting the power cable to the Catalyst Switch again. You can release MODE button only when the STAT LED or the first port LED stops blinking or goes out. From the CLI (thru HyperTerminal), you shoud see switch: prompt.

In switch: prompt, initialize the internal flash memory of the by issuing flash_init command. Afterwards, issue load_helper command. You can then check the content of the flash memory by issuing dir flash: command, and look for the configuration file - config.text. Just type rename flash:config.text flash:config.old to change the name of the configuration file. This step is to ensure that the Catalyst Switch won't recognize any saved or startup configuration upon boot up.

In Fixed Switches, although there's still a start-up configuration file on the NVRAM, the actual configuration is saved on the flash memory with the default name, config.text. You can actually change the default name to something else thru the global configuration mode when the Catalyst Switch is properly booted up.

After the renaming, issue boot command to boot the first image in the flash memory. Or specify an image name by using boot flash:(image name) command instead. After boot up, the System Configuration Dialog will show up. Just type No when asked if you want to continue with the initial configuration dialog. Then, issue enable command. This is the actual point of bypassing the password. Wherein if the configuration is loaded (and there's a password applied), you cannot actually go to the Privileged EXEC or enabled mode.

You can then restore the original name of the configuration file by typing rename flash:config.old flash:config.text. Then, type copy flash:config.text system:running-config to restore the configuration to the running configuration of the Catalyst Switch. You can actually use a shortcut to do both steps with a singe command. You can use copy flash:config.old system:running-config instead.

Note: Be sure that you use copy flash:config.text system:running-config command (or copy startup-config running-config) and not the reverse, which is copy system:running-config flash:config.text (or copy running-config startup-config). If you do the reverse, you'll be overwriting the original configuration that leads to a bigger problem.

To change or remove the password configurations, issue configure terminal command to go to the global configuration mode, and then type enable secret to change the secret password, or no enable secret to remove secret password. Also type enable password to change the regular password, or no enable password to remove regular password. You can also change or remove telnet (line vty) and console (line console) passwords by using the following commands:

For telnet:

Switch(config)#line vty 0 15
Switch(config-line)#password
Switch(config-line)#login

For console:

Switch(config-line)#line con 0
Switch(config-line)#password
Switch(config-line)#login

Lastly, issue write memory or copy run start to save the configuration with new passwords. And we’re done! =)

Next time, we’ll do Password Recovery on Modular Switches Running CatOS.

Password Recovery on Cisco Catalyst Switch - Introduction

2008-09-05T16:49:00.000+08:00

If you are a LAN Switching engineer and mostly working on Cisco Catalyst Switches, one of the tasks that you should know is Password Recovery. This is something that is very easy to do, but very risky as well.

Easy? Well, Password Recovery is all procedural, and has the same concept in all Cisco Catalyst Switches. Yes, it is very risky too. Especially, if you get too confident, and forgot to do some verifications... or in my own words - "checkpoints". The worst case scenario on doing password recovery is if you overwrite the saved or start-up configuration file and there's no backup. Nice huh?! =)

For this topic, I've divided it into 3 parts:

Password Recovery on Fixed Switches (Small Boxes)
Password Recovery on Modular Switches Running CatOS
Password Recovery on Modular Switches Running IOS

The last part can actually be divided further into another 3 parts:

Password Recovery on 4500 running IOS
Password Recovery on Catalyst 6500 with Hybrid IOS (MSFC IOS)
Password Recovery on Catalyst 6500 with Native IOS

The reason these 3 are on a single category is that they have a lot of similarities, and we just need to take note of the minor differences.

Concept of Password Recovery

First of all, why do we need to do Password Recovery? Well, it's either you are too old to remember simple passwords, or you've used to many passwords in a short span of time. =P Kidding aside, there's a lot of situations wherein Password Recovery is needed. The best example is when a new network administrator took over a new network with poor documentation. Password Recovery also enables us to resume management of the Catalyst Switch without resetting to factory default (we don't want to lose the current configuration).

Now, as I said earlier, Password Recovery has the same concept in all Cisco Catalyst Switches (and even with Cisco Routers). In Password Recovery, you actually want to bypass the password set on a Catalyst Switch. Most of the time, we want to bypass both the console and enable-mode (Privilege EXEC) passwords. In order for us to bypass the password, we need to bypass the saved or start-up configuration. Why? Well, the console and enable-mode passwords are actually part of the start-up configuration.

Bypassing the start-up configuration is similar to resetting the Catalyst Switch back to factory default, BUT without losing or overwriting the current configuration. We will actually restore the start-up configuration before we can change or remove the passwords.

The steps on how to bypass the passwords or the start-up configurations are actually the main differences in doing Password Recovery on Fixed Switches, Modular Switch Running CatOS, and Modular Switches Running IOS.

Limitations / Considerations

The main limitation on doing Password Recovery on Cisco Catalyst Switches is that you cannot do it remotely (i.e. telnet). Meaning, you should be physically onsite where the Catalyst Switch is, in order to accomplish the Password Recovery procedures. You need to be directly connected to the Catalyst Switch thru console, and to do some manual work, like doing cold reboot.

I guess that's all for now, and wait for the next part. =)

Power Supply Redundancy in Cisco Catalyst 6500

2008-02-09T16:23:00.000+08:00

Case Study

I’ve been doing case reviews again, and I came across a case about Power Supply Redundancy in Cisco Catalyst 6500. The customer wanted to add another linecard (module) in their Catalyst 6500, but when the linecard was inserted, the Cat6500 showed a 'power denied' message. The Cat6500 is currently using 2 2500W power supplies in redundant mode, and the customer wanted to replace it with their spare 3000W power supplies. To minimize/prevent service interruption, the customer is asking us if he can inter-mix power supplies with different capacity. He's planning to change power supplies one at a time.

This should have been an easy case to answer. But then, it showed that even if I gained a lot of experience thru the years, I could still show some stupidity. And mind you, this one is a long-standing stupidity.

You see, the engineer who handled the case was quick and confident on his answer. He simply said yes that the Cat6500 would be able to support the redundancy of power supplies with unequal capacities. He even informed the customer that in a scenario where there are power supplies with unequal capacity, the Switch would operate using the limit of the lower capacity power supply to ensure complete power redundancy.

If I had this case, I would have given the same answer. And then, it hit us in our face when the customer replied. Apparently, the customer took our word for it and tried swapping one of the power supplies. We’re correct on the part that it will be just fine to inter-mix power supplies of unequal capacity. It’s on the second part that we were proven wrong when the customer showed us a ‘show power’ command output.

The command output showed that the Cat6500 Switch assumed the limit of the higher capacity. When I saw this part from the case, I got surprised and my first thought is that this could be a bug! I have known/accepted it for a long time that in power redundancy, a Switch will assume the lower limit. It’s what I learned when I was first taught about power redundancy in Cat6500.

Being curious, I searched for the updated document for Power Supply Redundancy. After getting into the correct link, I then realized my stupidity. It specifically noted that for Cat6500 running IOS (Native), the power supplies with unequal capacities will both come up, and the Switch’s total power (wattage) is equal to the output of the power supply with the higher capacity.

Redundant and Non-redundant Power Supply Configurations

The table below will summarize the possible power supply configuration scenarios on a Catalyst 6500 Switch.

Power Supplies	Redundancy	Outcome
Equal Power Output (Wattage)	Enabled	Since both will have equal power drawn, one power supply can support the whole chassis load, when the other power supply fails. Also, load sharing will be in effect wherein each power supply will provide half of the total power requirement of the chassis.
Unequal Power Output (Wattage)	Enabled	In CatOS (Hybrid), if the output difference between the power supplies is less than 10%, redundancy will still be enabled. But if the difference is greater than 10%, redundancy will be disabled wherein only the power supply with the higher output will be enabled. In Native IOS, both power supplies will be up, and the total power rating of the Switch will be equal to the output of the power supply with higher capacity.
Equal or Unequal Power Output (Wattage)	Disabled	The power available to the Switch will be equal to the combined power output of the power supplies, regardless if they have equal or unequal power outputs.

Final Note

Now, you may ask what will happen if the 'active' power supply with higher capacity fails. How will the redundant power supply (with lower capacity) support the Catalyst 6500 Switch? Well, if the current power requirement of the whole Switch (chassis, supervisor(s), fan tray, and linecards) is lower than the capacity of the remaining power supply, there won't be any problem. But if the remaining power supply's capacity is not enough, the switch will shutdown (power-deny) components in this order:

Power over Ethernet (PoE) devices. The Switch (or the supervisor - being the intelligent component) will shutdown PoE devices from high-numbered ports to low-numbered ports on modules from highest-numbered slots to the lowest-numbered slots (descending order).
Modules (Linecards). The Switch will shutdown modules from the highest-numbered slots to the lowest-numbered slots (descending order) until the total power requirement will be less than the capacity of the remaining power supply. Supervisor linecards (or supervisor engines) and Switch Fabric Modules (SFMs) will be exempted or bypassed.

Need to know more? Visit this link on Power Supply Redundancy in Catalyst 6500.

EtherChannel Load Balancing

2007-11-24T23:24:00.000+08:00

Whew! We just had a very busy week prior to the Thanksgiving weekend. I got loads to do, and currently stuck with case reviews. Part of my job is to do case reviews so that I can check if any of my engineers is stuck in a particular case. If I find one, I review it and try to figure out how I can best help my engineer to move forward with that case.

I haven’t done a lot yet when I reviewed a case from one of my newest engineers. He’s been stuck with a customer who wanted to ‘equally’ load-balance Switch traffic between two switch ports connected to another switch. The customer is currently having 100% utilization in one link, and only 10% in another link.

Try as he may, my engineer was having a hard time explaining to the customer that it is impossible to load-balance EtherChannel links in exact 50-50 split. I see a coaching opportunity here since EtherChannel is one of the fundamental topics that we train our new hire engineers on. To think that I even reinforced it with a special training session! =(

Enough of the angry outburst for now, and let me share what I know about EtherChannel Load Balancing in general.

EtherChannel Introduction

Before we delve into the EtherChannel Load Balancing topic, let’s have an introduction first on what is EtherChannel, and why do we need it on our Switches.

By definition, EtherChannel (EC) offers bandwidth scalability within a Local Area Network (LAN) by providing up to 800 Mbps, 8 Gbps, or 80 Gbps of aggregate bandwidth for a Fast EtherChannel (FEC), Gigabit EtherChannel (GEC), or 10 Gigabit EtherChannel (10GEC) connection, respectively.

Under normal (default) configuration, redundant connection between Switches will be dealt by Spanning Tree Protocol (STP). STP, which prevents ‘switching loops’, will block any redundant connection until there’s only a single connection or link between the Switches. You can just imagine that these blocked links are potential additional bandwidth, but are currently wasted since no traffic will be allowed to pass through them.

With EtherChannel, these redundant links will be aggregated (or ‘channelized’) in order to maximize bandwidth utilization. Switches will treat EC links as a single logical connection, thus it can function as either an access or a trunk ‘logical’ port. EC will also show up as a single port in STP, and therefore, the redundant ports under an EC link will have the same STP state.

Aside from better bandwidth utilization, EtherChannel will also offer ‘built-in redundancy’. This means that if any of the links aggregated by EC fails, the other links will stay up and pick up the traffic from the failed link. This results to negligible traffic loss whenever a link fails. Better yet, there won’t be any STP re-convergence that will take place since STP still treat these aggregated ports as a single logical link.

EtherChannel Load Balancing

To implement Load Balancing, EtherChannel translates the addresses in a packet (either the MAC address, the IP address, or the port number) from their binary form to a numerical value. This numerical value corresponds to one of the links in the EC in order to distribute packets in all the links in the EC.

EtherChannel Load Balancing (or frame distribution) uses a Cisco−proprietary hashing algorithm. This algorithm is deterministic that when you use the same addresses and session information, you always hash to the same port in the EC. This will prevent out−of−order packet delivery.

The hash algorithm calculates a numerical value between 0 and 7. Switch ports in the EtherChannel will then correspond to these numerical values. The port setup includes a mask which indicates which values the port accepts for transmission. If an EC uses the maximum number of switch ports in a single EtherChannel (8 ports), each port accepts only one value. But if you have less number of switch ports in the EC, each port accepts more than one value. The table below shows the number of numerical values that will correspond to switch ports (based on the number of switch ports on a given EC link):

No. of Ports in an EC link	Load Balancing
8	1:1:1:1:1:1:1:1
7	2:1:1:1:1:1:1
6	2:2:1:1:1:1
5	2:2:2:1:1
4	2:2:2:2
3	3:3:2
2	4:4

EtherChannel will only aggregate the bandwidth of up to eight compatible or similarly configured switch ports into a single logical link. And in newer Cisco Catalyst Switches, the Load Balancing policy can be based on MAC address (L2), on IP address (L3), or on a port number (L4). Aside from that, the policy can also be based on either the source address, destination address, or both.

Please note that you cannot specify or manually choose a link that a particular traffic flow uses. You can only influence the Load Balancing by choosing a ‘distribution method’ that will give the best distribution. If you’ll ask about the hash algorithm, this cannot be configured or changed to load balance the traffic among the links in an EtherChannel. In the end, there’s a need to experiment a bit in order to select the best distribution method for a specific EtherChannel.

Related Commands

Here are some of the commands that will with your EtherChannel Load Balancing needs:

For CatOS:

Check Load Balancing (Frame Distribution) Policy

show port channel info

Determine the port for use in the EC to forward traffic

show channel hash < {src_ip_addr | dest_ip_addr | src_mac_addr | dest_mac_addr | src_port | dest_port} [dest_ip_addr | dest_mac_addr | dest_port]>

For IOS:

Check Load Balancing (Frame Distribution) Policy

show etherchannel load−balance

Determine the port for use in the EC to forward traffic

test etherchannel load−balance interface number> <{ip | l4port | mac} [source_ip_add | source_mac_add | source_l4_port] [dest_ip_add | dest_mac_add | dest_l4_port]>

*For more information, you can visit this link from Cisco.com

Man vs (Networking) Machines

2007-11-17T11:16:00.000+08:00

In our Technical Support Team, we consider what we do everyday as a learning experience. There's always these moments wherein we already thought we have learned everything. But then again, a customer will point out a new angle to an old/known issue (which we have encountered countless times before), and we'll feel astonished we haven't thought of it that way.

Sometimes, I can't help but think that there's a war going on. You, or any network professionals, may even say that this is a crusade. Yup. A crusade for your organization's network to stay 'up and running'. Or else, your boss will surely fire you. =)

Have you noticed that no matter how much you or your team take care of your network, they seem to always give you a problem anytime they want? Have you shouted once, and delivered one of the classic lines from old war movies ("Don't die on me, g'dammit!")?

Well, we hear a lot of these 'war stories' everyday. You can even consider us as the reinforcement whenever the situation has worsen for the customer. And starting today, I'll give dispatches from the 'frontline', and hope everyone else learns from our experience.

Welcome to Frontline Express! If you want fresh updates, subscribe thru email or one of the available readers. Cheers!