Rivald's Blog

Cisco Catalyst 3850 Upgrade Issue - Internal Error

2018-11-27T20:23:00.000-05:00

While upgrading a pair of non-stacked 3850-48Ps, I encountered this error message:

[1]: % An internal error was encountered. Operation aborted.

I did have some space issues, so I deleted previous versions. Even that did not fix the issue. I tried installing it from tftp, to no avail.

Eventually, I ran the software install command with "new force"

software install file tftp://10.20.30.40/cat3k_caa-universalk9.SPA.03.06.09.E.152-2.E9.bin new force

Success. I rebooted both switches. One crashed on reboot, but rebooted cleanly afterward.

Cisco 3850 WS-C3850-48P-E Switch with Wrong License Level Lanbase

2018-11-25T13:13:00.000-05:00

I received a couple of refurbished 3850 PoE Enterprise switches. One switch was perfectly fine, with IP services enabled permanently.

The other:

Slot# License name Type Count Period left

----------------------------------------------------------

1 lanbase permanent N/A Lifetime

License Level on Reboot: lanbase

I issued a request for an RMA - however, the switch was labeled correctly on the case: WS-C3850-48P-E.

On a whim, I ran this command:

license right-to-use activate ipservices

I rebooted as instructed from the switch, and now I have:

Switch#show license right-to-use
Slot# License name Type Count Period left
----------------------------------------------------------
1 ipservices permanent N/A Lifetime
1 lanbase permanent N/A Lifetime

License Level on Reboot: ipservices

So, yay - no need for an RMA.

Upgrading pfsense 2.4.3_1 to 2.4.4 - Issue Replacing PHP 5.x with 7.x Due to FRR

2018-11-23T14:40:00.001-05:00

I ran into a problem upgrading pfsense 2.4.3_1 to 2.4.4 due to a dependency on the old PHP 5.6.x from the older package of FRR .2_2 (Free Range Router.) The problem was that PHP56 was marked as vital due to the FRR dependency, and it was not possible to update FRR to .2_4, as FRR .2_4 needed pfsense 2.4.4 (I could not decide whether this was a modern O. Henry story or a missing epilogue from Catch 22.)

The fix:

1. connect to the pfsense server with SSH
2. Open a shell
3. Run this command:

pkg set -v 0 php56

Your upgrade should work - I had to do a couple of update attempts before it worked.

SSH No Matching Cipher Found With SSH to Older Cisco Gear

2018-03-21T20:30:00.000-04:00

I've seen this error on recent SSH versions, including the OpenSSH used by OS X and others:

Unable to negotiate with 192.168.77.1 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

Several people suggest modifying your local ssh client config. However, I'd rather not use 3DES unless I have to. So, I created an alias and put it in .bash_profile:

alias ssho='ssh -c 3des-cbc'

after a quick . ~/.bash_profile

I can ssh with the alias:

ssho 192.168.77.1
password:

Of course, the best fix would be to update IOS on the Cisco gear.

Openstack Ansible Pike - Cannot Create Image in Horizon

2018-03-01T20:30:00.000-05:00

There is a bug with OpenStack Pike deployments using Ansible. When you attempt to create an image in Horizon, you will see the following error:

Followed by this entry in the images list after closing this:

You should still be able to create images via command line, but Horizon won't work.

There is a bug filed for this:

https://bugs.launchpad.net/openstack-ansible/+bug/1730722

Their fix is simple: connect to the horizon container from an infrastructure node:

lxc-attach -n `lxc-ls -1 | grep horizon | tail -1`

edit /etc/horizon/local_settings.py

and change:

HORIZON_IMAGES_UPLOAD_MODE = 'direct'

HORIZON_IMAGES_UPLOAD_MODE = 'remote'

After that, restart the horizon container

lxc-stop -n `lxc-ls -1 | grep horizon | tail -1`

lxc-start -n `lxc-ls -1 | grep horizon | tail -1`

And simply wait a few minutes. Image creation should work.

Powercenter Integration Service Won't Start: Error SF_34004 - Missing Environment Variables

2017-07-06T23:46:00.000-04:00

I recently had an issue where I could not get the Informatica PowerCenter Integration Service to start after an install. The service generated this error code: SF_34004. There is an Informatica KB article describing this problem and the appropriate fix:

https://kb.informatica.com/solution/18/Pages/120251.aspx

Basically, it's suggesting that the Integration Service was configured to connect to the PowerCenter repo using the schema credentials and not the domain administrator.

However, I found that this did not help. the repo service was running correctly and the integration service was using the correct credentials. I checked the user running PowerCenter, and realized the INFA_HOME, INFA_DOMAIN_FILE, and other variables were not set correctly. After setting these and restarting PowerCenter, I was able to enable the Integration Service.

Abnormally Long First Boot Times on Cisco 3750x's with 15.2.x code

2017-05-05T22:36:00.003-04:00

I recently upgraded some Cisco 3750x (WS-C3750X-48T-S to be precise) from 12.2.x code to 15.2 code (15.2(4)E4.)

The reboot took a rather long time due to microcode updates. It was about 25-30 minutes. The following messages were displayed:

ssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssp
Front-end Microcode IMG MGR: Preparing to program device[0]...25166 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrwssssssssssssssssssspssssssssssssssssssspp
Front-end Microcode IMG MGR: Preparing to program device[0]...86370 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrwssssssssssssssssssspssssssssssssssssssspw

% Front-end Microcode IMG MGR: HW image is upgraded. MCU reset causes the switch to reload

If you're making such an upgrade, be prepared for a long wait. It will likely be worse on a stack.

While the 3750x is not a speedy switch, management-wise, this was impressively slow. The regular boot time after the upgrade was about 5-6 minutes, so be patient. Switches in a stack will take even longer to boot.

Android Nougat and OTG Sound Blaster E3 Sound Issues

2017-02-28T21:39:00.000-05:00

I recently upgraded to Nougat and experienced sound issues using an OTG cable and my Sound Blaster E3 DAC. The sound was basically panned hard to the left, slower than normal, and there was some distortion.

I realized that ES File Explorer was activating each time I plugged the E3 in. The simple fix was to remove ES File Explorer. That being said, I'll probably experiment with its settings as it's a useful utility.

Raspberry Pi and 1-Wire Temperature/Environmental Sensors

2016-10-30T16:20:00.000-04:00

I've discussed 1-wire before. In this case, I needed a cheap temperature sensor to alert on environmental conditions in an server room.

I set up a Raspberry Pi with stock Raspian and install digitemp like so:

apt install digitemp

I used an iButtonlink usb master (here: http://www.ibuttonlink.com/products/linkusb and a t-sense sensor here:
http://www.ibuttonlink.com/products/t-sense )

The USB device shows up as /dev/ttyUSB0 as an FTDI serial port.

I can walk the 1-wire tree like so:

sudo digitemp_DS9097U -s /dev/ttyUSB0 -w
DigiTemp v3.6.0 Copyright 1996-2007 by Brian C. Lane
GNU General Public License v2.0 - http://www.digitemp.com
Turning off all DS2409 Couplers
..
Devices on the Main LAN
2149B214090000D1 : DS18B20 Temperature Sensor
01C3235419000000 : DS2401/DS1990A Serial Number iButton

That being said, you'll want to create a dot file to use this like so:

sudo digitemp_DS9097U -s /dev/ttyUSB0 -i
DigiTemp v3.6.0 Copyright 1996-2007 by Brian C. Lane
GNU General Public License v2.0 - http://www.digitemp.com
Turning off all DS2409 Couplers
..
Searching the 1-Wire LAN
2149B214090000D1 : DS18B20 Temperature Sensor
ROM #0 : 2149B214090000D1
Wrote .digitemprc

I have one temperature sensor at the moment. After creating the .digitemprc, I can query it:

sudo digitemp_DS9097U -s /dev/ttyUSB0 -t 0
DigiTemp v3.6.0 Copyright 1996-2007 by Brian C. Lane
GNU General Public License v2.0 - http://www.digitemp.com
Oct 30 20:16:45 Sensor 0 C: 21.00 F: 69.80

Setting Up RAID Volumes on Cisco UCS C series with Avago/LSI/MegaRAID Controllers

2016-10-06T22:43:00.000-04:00

On a C220M4, I discovered that the controller configures the drives as JBOD. Fixing this is easy, but not quite as straightforward as one would expect... especially considering that the controller has been a mainstay of servers for a very long time.

1. Go into the controller firmware (I had to do a control-M before I saw the control-R option):

As you can see, create virtual drives is not an option. Select "Make Unconfigured Good"

Select all the drives and click OK.

Now you have the option to create a volume.

Cisco UCS C220M4s and the Purple Screen of Death - Could not start pcpu 1: TSC sync timed out

2016-10-06T22:34:00.001-04:00

Recently, I had to install ESXi 5.5 on a Cisco rackmount server. VMware crashed with the purple screen of death.

Here's the display:

These are relatively new servers, so there was nothing especially helpful on the web. Other people had this problem, but I couldn't find any real solutions, other than a reference to C states on the CPU.

I experimented a bit and figured that CPU power savings might be the problem (I'd seen this before on Opteron servers.)

The fix is to set CPU performance to HPC (it defaults to Enterprise) in BIOS:

That's advanced -> CPU Performance -> HPC (from Enterprise)

This solved the problem.

Avocent ACS 6016/6008 Renaming Ports From CLI

2016-10-05T13:27:00.001-04:00

These console servers are fairly easy to setup, but the CLI is segmented to facilitate differing levels of access.

If you wish to rename a serial port (i.e., with hostname) from the CLI, you need to do the following

1. Figure out which port is being renamed (i.e., port 11)
2. log in with appropriate credentials via SSH (or direct serial)
3.

cd ports

4.

cd serial_ports

4b. You can check out the parameters here:

--:#- [serial_ports/physical] cli-> show

port: 11

enable_cisco_rj45_pin-out = yes

status = enabled

speed = 9600

parity = none

data_bits = 8

stop_bits = 1

flow_control = none

5. set port name

set port_name=my_new_name

i.e.,

--:#- [serial_ports/cas] cli-> set port_name=unused

8. save the config

save

That's it, basically.

AnyConnect 4.x on ASA 8.4.x: AnyConnect is not enabled on the VPN server

2016-08-30T00:58:00.001-04:00

I recently had to do a failover on a pair of ASA5520s. On failing over to the secondary, AnyConnect stopped working. I did not see a license error, but users who connected received this error:

AnyConnect is not enabled on the VPN server

There weren't any smoking guns in the logs. I did notice that the newest version of the config was missing the client image definitions in webvpn. They were in the flash drive, but not copied to the secondary. So, when the secondary took over, this portion of the config was dropped.

The solution was to re-add the images:

config t
webvpn
anyconnect image disk0:/anyconnect-macosx-i386-4.3.00748-k9.pkg 1

anyconnect image disk0:/anyconnect-win-4.3.00748-k9.pkg 2

Cisco ASA tftp Copies and Issues with atftpd

2016-03-08T22:13:00.002-05:00

while attempting to copy a new anyconnect pkg file from a CentOS based server running atftpd, I received the following error on a Cisco ASA 5505:

%Error reading tftp://mytftpserver/anyconnect-win-3.1.14018-k9.pkg (Undefined error code)

On the tftp server in /var/log/messages, I saw:

Mar 6 14:49:19 mytftpserver atftpd[15147]: Serving anyconnect-win-3.1.14018-k9.pkg to 192.168.47.5:32947

Mar 6 14:49:19 mytftpserver atftpd[15147]: Requested file to big, increase BLKSIZE

I had installed tftp a while ago... specifically atftp-server. After looking at the man page for in.tftpd, I concluded there was no good way to set BLKSIZE on this package. The fix was:

sudo yum erase atftp-server

sudo yum install tftp-server

I then edited the xinet.d/tftp file to enable it and change the target directory and restarted xinetd. After this, I was able to copy the pkg file over successfully.

Mellanox ConnectX-2 10GB Interface on FreeBSD 10.2

2015-12-28T14:30:00.000-05:00

Although Mellanox's FreeBSD driver for the ConnectX-2 is included in the kernel source, the kernel modules are not included with the generic kernel. To use this card, build a custom kernel.

1. update the FreeBSD source using freebsd-update

sudo freebsd-update fetch
sudo freebsd-update install

2. copy the generic kernel config to a custom kernel
cd /usr/src/sys/amd64/conf

sudo cp GENERIC MYKERNEL01

4. edit the config file (sudo vi MYKERNEL01.) Add these lines to the bottom:

#### Mellanox ConnectX-2 support
options OFED
options IPOIB_CM
device ipoib
device mlx4ib
device mlxen

5. Compile the kernel and install it

cd /usr/src
sudo make buildkernel KERNCONF=MYKERNEL01
sudo make installkernel KERNCONF=MYKERNEL01

6. reboot

sudo shutdown -r now

Your new interface should show up when the machine comes back:

mlxen0: flags=8802 metric 0 mtu 1500
options=d07bb
ether 00:02:c9:52:ad:23
nd6 options=29
media: Ethernet autoselect (autoselect )
status: active

You can configure however you'd like (IP, MTU, etc.)

If you run freebsd-update on a regular basis, you may want to append a kernel rebuild (step #5) to a script so you can rebuild the kernel automatically if there is a kernel source update.

2010 Macbook Pro and problems with Yosemite and/or El Capitan

2015-11-19T19:00:00.000-05:00

I have had terrible issues with a mid-2010 MBP 15" core i7 laptop. Doing a clean installation of either Yosemite or El Capitan, the installer fails toward the end or crashes on first boot. I have not tried a fresh Mavericks install. However, a restore from a Mavericks time machine backup works perfectly fine. Alternatively, a linux installation also worked well with no crashes.

Quite a few posts suggest it is a log board problem and that Apple would fix it. However, being impatient, I tried my own work around. I noticed that the problem didn't happen if I reduced the RAM from 8GB to 4GB (1 stick of DDR 3 instead of 2.)

A prevailing theory for this failure is that the Nvidia 330M card was crashing the system when it switched from the onboard to the 330M.

A couple of suggested work arounds:

1. disable automatic video switching in power preferences. This will likely consume battery at a faster rate

2. use gfxCardStatus and force it to either onboard or discrete

What I did was:

1. do a fresh install of El Capitan with only 1 stick of DDR 3 (4GB)

2. fully patched the OS

3. installed gfxcardstatus and forced it to onboard only.

I haven't had a crash since.

Windows Netstat and findstr - Same as find | grep on Linux/Unix

2015-05-27T14:00:00.000-04:00

Unix/Linux users are familiar with the following patterns of usage for finding listening ports:

looking for services listening on ports 443 or 444:
netstat -a | grep 44[34]

find a service listening on port 602, 612, 604, 614, 608, or 618:
netstat -an | grep 6[01][248]

To do the same on windows, use netstat and findstr:

find a window's service listening on port 602, 612, 604, 614, 608, or 618:
netstat -an | findstr /R 6[01][248]

Determining What Kind of SFP Is Installed in a Cisco Switch

2015-04-29T10:28:00.000-04:00

There are several round about ways of determining what kind of SFP is installed in a socket on a Cisco switch, but I think the easiest is this:

testswitch#show int gi3/1/2 capabilities
GigabitEthernet5/1/1
Model: WS-C3750X-48
Type: 1000BaseLX SFP
Speed: 1000
Duplex: full
Trunk encap. type: 802.1Q,ISL
Trunk mode: on,off,desirable,nonegotiate
Channel: yes
Broadcast suppression: percentage(0-100)
Flowcontrol: rx-(off,on,desired),tx-(none)
Fast Start: yes
QoS scheduling: rx-(not configurable on per port basis),
tx-(4q3t) (3t: Two configurable values and one fixed.)
CoS rewrite: yes
ToS rewrite: yes
UDLD: yes
Inline power: no
SPAN: source/destination
PortSecure: yes
Dot1x: yes

You can see the type and everything. You'll need to be enable mode, of course.

Acme/Oracle SBC Useful Commands

2015-03-24T20:03:00.000-04:00

To get the box-id (useful for licensing, etc.)

show version boot

To get the current amount of sessions

show sessions

To check the state of session agents

show sipd agents

To lookup a number in the local routing table
If you had created an LRT policy called inboundnums

show lrt route-entry 15555551212

To backup the config

backup name_of_backup

To verify the config

verify-config

Using 3rd SFP's on Cisco 3560s and 3750s

2015-02-28T19:19:00.000-05:00

It's well known that Cisco does not especially like to support third party hardware products within their switches. Perhaps one of the most common cases of this is the support for 3rd party SFPs on the Catalyst line.

During the day of GBICs, 3rd party interfaces usually worked. Enter the SFP. Here's a typical response from a 3750 when plugging in, say, a Finisar SFP:

Feb 27 00:01:47.952: %GBIC_SECURITY_CRYPT-4-VN_DATA_CRC_ERROR: GBIC in port Gi2/0/1 has bad crc
*Feb 27 00:01:47.952: %PM-4-ERR_DISABLE: gbic-invalid error detected on Gi2/0/1, putting Gi2/0/1 in err-disable state

and a show int status demonstrates this:

show int status | inc Gi
Gi2/0/1 err-disabled 1 auto auto unknown

As an unspported work around work around, you can tell the switch to ignore the fact that the SFP was not blessed by Cisco. Keep in mind that Cisco will likely tell you to put in Cisco branded SFPs and that they won't support you with 3rd party SFPs in place (i.e., this is at your own risk!)

from conf t:
service unsupported-transceiver
no errdisable detect cause gbic-invalid

You'll see this message

Warning: When Cisco determines that a fault or defect can be traced to
the use of third-party transceivers installed by a customer or reseller,
then, at Cisco's discretion, Cisco may withhold support under warranty or
a Cisco support program. In the course of providing support for a Cisco
networking product Cisco may require that the end user install Cisco
transceivers if Cisco determines that removing third-party parts will
assist Cisco in diagnosing the cause of a support issue.

If you then move the SFP to another port, you should see something like this in int staus:

Gi2/0/2 connected 1 a-full a-1000 unsupported

It works... don't forget to do a write mem.

RedHat Enterprise 7/CenOS 7 Firewall oneliners

2015-01-20T14:35:00.001-05:00

Assuming you're using the default zone of "public" (you may need to temporarily disable selinux (setenforce 0)):

1. To allow everyone to access port 8080/tcp:

firewall-cmd --zone=public --add-port=8080/tcp --permanent

2. Allow a server from the IPv4 address 10.20.30.40 to access this server on port 1234 over UDP:

firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="10.20.30.40/32" port port="1234" protocol="udp" accept' --permanent

RedHat/CentOS error: connection activation failed: connection 'x' is not available on the device y

2015-01-20T12:18:00.001-05:00

You'll see this error using nmcli if you try to bring up a connection in the nmcli that references a NIC that is disconnected/unplugged - either unplugged from a baremetal server or disconnected in VMware. Here are some examples of what it might look like:

connection activation failed: connection 'ethernet' is not available on the device ens32

connection activation failed: connection 'eth0' is not available on the device ens192

etc.

The fix is really as simple as connecting the cable or re-enabling the NIC in VMware and running the nmcli command again.

connection activation failed: connection '' is not available on the device

2015-01-20T12:14:00.000-05:00

Simple Clustering on CentOS 7/RHEL 7 for an haproxy Load Balancer

2015-01-14T15:52:00.000-05:00

Here are the steps to get a simple cluster going. We're not going to share storage, so the quorom isn't going to work. We'll disable stonith and quorom.

pre-req: add hosts file entries for all nodes on all nodes, or at least make sure DNS is working correctly. You might receive errors like:

Error: unable to get crm_config, is pacemaker running?

yum install pcs fence-agents-all -y
firewall-cmd --permanent --add-service=high-availability
# set the password for the hacluster user - it should probably be the same on all nodes
passwd hacluster
# disable haproxy, as the cluster will start it

systemctl disable haproxy

# enable the services
systemctl enable pcsd
systemctl enable corosync
systemctl enable pacemaker
systemctl start pcsd.service

### we're not going to have a stonith nor a quorom
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore

# check to see if it's alive
systemctl is-active pcsd.service
pcs cluster auth `hostname`
pcs cluster setup --start --name myclustername `hostname`
pcs cluster status
# add an IP as a resource
pcs resource create vip1 IPaddr2 ip=172.29.23.80 cidr_netmask=22 --group haproxy

to add an additional node called "mynode2"
authorize it on the master:

pcs cluster auth mynode2

(authenticate using "hacluster" user)

add it:

pcs cluster node add mynode2

You'll need to start the other node:

pcs cluster start mynode2

To see the status of the nodes:

pc status nodes

example:
pcs status nodes
Pacemaker Nodes:
Online: mynode1
Standby:
Offline: mynode2

Now we add haproxy. Since the haproxy service wouldn't be too useful without the IP address, we'll set up a colocation rule as well.

pcs resource create HAproxy --group haproxy systemd:haproxy op monitor interval=10s
pcs constraint colocation add HAproxy vip1
pcs constraint order vip1 then HAproxy

# optional - we want the cluster to "favor" mynode1.
# if mynode1 is restarted, for example, mynode2 will get the resources,
# until mynode1 is back and running
pcs constraint location vip11 prefers mynode1

We'll want to turn off haproxy in systemd as the cluster will start it:

systemctl disable haproxy

CentOS 7 error: Connection 'wired connection 1' is not available on the device ens32 at this time.

2015-01-14T11:37:00.000-05:00

This error is presented when attempting to bring up a network connection using nmcli. The connection name can be anything, of course.

There is a RedHat bug on this titled "RHEL 7 syslog shows failure related to network.service" bug #1079353.

There is no fix listed in the bug. I've seen this on a virtual machine, and the easiest work around was to delete the virtual machine's NIC, reboot, add a new NIC - reboot. After doing that, I deleted the new network connection using nmcli and applied the new device to the original connection profile I was attempting to use.