Slaptijack

Pro Tip: Keep Asterisk Configuration Files in Version Control

Scott Hebert — Sun, 30 Oct 2016 05:00:00 +0000

An Asterisk server has a very involved configuration system. One instance I manage has over 100 configuration files. As these configurations grow, it can be helpful to have a separate server running for testing and debugging problems. If you are pla...

TACACS Detected 'Invalid Argument'

Scott Hebert — Fri, 29 Jul 2016 13:00:00 +0000

As always, I've changed pertinent details for reasons.

I was working on an ASR the other day and received the follow error:

RP/0/RSP0/CPU0:ASR9K(config-tacacs-host)# commit
Fri Jul 29 12:55:46.243 PDT

% Failed to commit one or more configuration items during a pseudo-atomic
operation. All changes made have been reverted. Please issue 'show configuration
failed [inheritance]' from this session to view the errors
RP/0/RSP0/CPU0:ASR9K(config-tacacs-host)# show configuration failed
Fri Jul 29 12:55:55.421 PDT
!! SEMANTIC ERRORS: This configuration was rejected by
!! the system due to semantic errors. The individual
!! errors with each failed configuration command can be
!! found below.

tacacs-server host 10.0.0.2 port 49
!!% 'TACACS' detected the 'fatal' condition 'Invalid Argument'
!
end

The problem here is that the tacacs daemon thinks the configuration contains an invalid argument. It doesn't. So, restart tacacs:

RP/0/RSP0/CPU0:ASR9K# show proc | inc tacacs
Fri Jul 29 12:56:32.376 PDT
1142   1    2  108K  16 Sigwaitinfo 7399:06:34:0893    0:00:00:0109 tacacsd
1142   2    0  108K  10 Receive     7399:06:35:0099    0:00:00:0000 tacacsd
1142   3    2  108K  10 Nanosleep      0:00:05:0940    0:00:00:0057 tacacsd
1142   4    1  108K  10 Receive     7399:06:34:0957    0:00:00:0000 tacacsd
1142   5    1  108K  10 Receive        0:00:00:0664    0:00:41:0447 tacacsd
1142   6    1  108K  10 Receive     2057:20:44:0638    0:00:44:0805 tacacsd
1142   7    2  108K  10 Receive     1167:26:53:0781    0:01:02:0991 tacacsd
1142   8    3  108K  10 Receive     1167:26:51:0567    0:01:29:0541 tacacsd
1142   9    2  108K  10 Receive      403:35:55:0206    0:01:09:0700 tacacsd
RP/0/RSP0/CPU0:ASR9K# process restart tacacsd
Fri Jul 29 12:56:54.768 PDT
RP/0/RSP0/CPU0:ASR9K# show proc | inc tacacs
Fri Jul 29 12:56:58.455 PDT
1142   1    3   64K  16 Sigwaitinfo    0:00:03:0806    0:00:00:0069 tacacsd
1142   2    1   64K  10 Receive        0:00:03:0998    0:00:00:0000 tacacsd
1142   3    3   64K  10 Nanosleep      0:00:03:0977    0:00:00:0000 tacacsd
1142   4    1   64K  10 Receive        0:00:03:0867    0:00:00:0002 tacacsd
1142   5    3   64K  10 Receive        0:00:03:0818    0:00:00:0000 tacacsd
1142   6    2   64K  16 Receive        0:00:03:0818    0:00:00:0000 tacacsd
1142   7    1   64K  16 Receive        0:00:03:0818    0:00:00:0000 tacacsd
1142   8    3   64K  16 Receive        0:00:03:0818    0:00:00:0000 tacacsd
1142   9    3   64K  10 Receive        0:00:00:0673    0:00:00:0003 tacacsd

And try again:

RP/0/RSP0/CPU0:ASR9K# config t
Fri Jul 29 12:57:04.787 PDT
RP/0/RSP0/CPU0:ASR9K(config)# tacacs-server host 10.0.0.2 port 49
RP/0/RSP0/CPU0:ASR9K(config-tacacs-host)# commit
Fri Jul 29 12:57:20.627 PDT
RP/0/RSP0/CPU0:ASR9K(config-tacacs-host)#

Two Column for Loop in bash

Scott Hebert — Mon, 18 Apr 2016 13:00:00 +0000

I had this interesting question the other day. Someone had a file with two columns of data in it. They wanted to assign each column to a different variable and then take action using those two variables. Here's the example I wrote for them:

IFS=$'n';
for LINE in $(cat data_file); do
    VARA=$(echo ${LINE} | awk '{ print $1}')
    VARB=$(echo ${LINE} | awk '{ print $2 }')
    echo "VARA is ${VARA}"
    echo "VARB is ${VARB}"
done

The key here is to set the internal field separator ($IFS) to $'n' so that the for loop interates on lines rather than words. The it's simply a matter of splitting the column into individual variables. In this case, I chose to use awk since it's a simple procedure and speed is not really an issue. Long-term, I would probably re-write this using arrays.

OpenSSH: Using a Bastion Host

Scott Hebert — Sat, 26 Mar 2016 13:00:00 +0000

Quick and dirty OpenSSH configlet here. If you have a set of hosts or devices that require you to first jump through a bastion host, the following will allow you to run a single ssh command: Host * ProxyCommand ssh -A <bastion_host> nc %h %p ...

Interleave Two Lists of Variable Length (Python)

Scott Hebert — Fri, 18 Mar 2016 13:00:00 +0000

I was recently asked to take two lists and interleave them. Although I can not think of a scenario off the top of my head where this might be useful, it doesn't strike me as a completely silly thing to do. StackOverflow's top answer for this problem us...

Socket Timeouts in urllib2

Scott Hebert — Sat, 05 Mar 2016 00:30:00 +0000

One of my scripts that makes an API call has been failing silently lately. It appears that the connection is timing out, but I am not catching that particular error. So, I fixed that.

@@ -8,6 +8,7 @@
 import os
 import random
 import shelve
+import socket
 import sys
 import time
 import urllib2
@@ -225,11 +226,15 @@

     queries['eve-kill'] += 1
     try:
-        data = urllib2.urlopen(request)
+        data = urllib2.urlopen(request, timeout=60)
     except urllib2.HTTPError, e:
         print('url: {}'.format(url))
         print('error: {}'.format(e))
         sys.exit(1)
+    except socket.timeout, e:
+        print('url: {}'.format(url))
+        print('error: {}'.format(e))
+        sys.exit(1)

     j = json.load(data)

I set the timeout particular long at 60 seconds. Since I made the change, I do not think it has ever actually taken that long.

Also, I need to refactor this and add my own exception class since it seems I am doing the same thing on most of my exceptions.

Cache has broken packages, exiting

Scott Hebert — Tue, 23 Feb 2016 15:00:00 +0000

I've been getting the following error from cron.daily in my inbox lately: /etc/cron.daily/apt: Cache has broken packages, exiting That's an annoying email to get everyday. I decided I would apt-get clean and that would probably fix the problem: Th...

MySQL OOM'ed, But Pelican Lives

Scott Hebert — Mon, 15 Feb 2016 15:00:00 +0000

I use Pingdom's free service to monitor slaptijack.com. Apparently, late Friday night, oom-killer decided that the server needed more memory and took out the MySQL server. To make matters worse, I missed the alarm from Pingdom, and slaptijack.com was down for pretty much all of Saturday. The fact that oom-killer was invoked is annoying, but more on that later.

The beauty of using Pelican rather than Wordpress (or any other database-driven content engine) is one less point of failure for the site. Obviously, I don't have all of slaptijack.com converted to Pelican yet (and perhaps never will), but at least parts of the site were up and working despite MySQL being down. If nothing else, this incident is enough to convince me that moving to Pelican was a good idea.

OOM'ed

Honestly, this is just plain annoying. The slaptijack.com server is has 1.75 GB of RAM and runs about half a dozen web sites -- all of which are much, much smaller than slaptijack.com. The server really doesn't do anything other than run Apache and MySQL. In fairness, I haven't spent a lot of time tuning either application, but there really shouldn't be a need to.

I thought perhaps one of the sites was under attack. MySQL was OOM'ed at 23:44. I had a look to see if there was a big spike in hits. As the data below shows, there really wasn't anything interesting going on during the 23:00 hour.

$ grep 12/Feb */logs/*.1 | awk '{ print $4 }' 
>   | cut -d: -f2 | sort | uniq -c
    953 00
    556 01
    626 02
    981 03
   1364 04
   1928 05
   2799 06
   1391 07
   1586 08
   1300 09
   2074 10
   1173 11
   2220 12
    909 13
    982 14
    769 15
   1546 16
    757 17
    480 18
    592 19
    579 20
    625 21
    581 22
    747 23

Anyway, in the hopes of avoiding this in the future, I tweaked the Apache and MySQL server settings to reduce their memory footprint. I cut way back on the number of Apache processes running and reduced the number of MySQL connections allowed. According to mysqltuner, MySQL's max memory usage is now 165.1 MB. Of course, that doesn't take into consideration the footprint of just running MySQL server appears to be a bit over 300 MB:

$ ps auwx | grep [m]ysql
mysql     6780  0.1 27.6 1209700 480656 ?      Ssl  11:04   0:08 /usr/sbin/mysqld

And each Apache process is 70 MB to 100+ MB at the moment:

$ ps auwx | grep [a]pache
root      7717  0.0  1.5 386420 26744 ?        Ss   11:20   0:00 /usr/sbin/apache2 -k start
www-data  7721  0.0  4.3 391612 75532 ?        S    11:20   0:04 /usr/sbin/apache2 -k start
www-data  7723  0.0  4.1 390028 71968 ?        S    11:20   0:04 /usr/sbin/apache2 -k start
www-data  7724  0.0  5.2 406620 91332 ?        S    11:20   0:05 /usr/sbin/apache2 -k start
www-data  7731  0.0  4.7 389644 82796 ?        S    11:21   0:04 /usr/sbin/apache2 -k start
www-data  7732  0.1  5.8 406784 101736 ?       S    11:21   0:06 /usr/sbin/apache2 -k start
www-data  7733  0.0  4.1 390624 72436 ?        S    11:21   0:04 /usr/sbin/apache2 -k start
www-data  7806  0.0  6.2 414232 108988 ?       S    11:26   0:04 /usr/sbin/apache2 -k start
www-data  7854  0.0  4.1 390804 72556 ?        S    11:29   0:05 /usr/sbin/apache2 -k start
www-data  8679  0.0  5.1 408492 89404 ?        S    12:06   0:01 /usr/sbin/apache2 -k start
www-data  9277  0.1  4.1 391384 71948 ?        S    12:43   0:01 /usr/sbin/apache2 -k start

Removing a Single Line from known_hosts With sed

Scott Hebert — Mon, 08 Feb 2016 15:00:00 +0000

Ever so often, something changes on the network, and you find that your .ssh/known_hosts file has gotten out of date. Usually this happens after an upgrade or device change. You'll get the rather ominous warning that REMOTE HOST IDENTIFICATION HAS CHANGED!

If you are confident that someone isn't doing something nasty and the RSA key fingerprint on the other side has legitimately changed, you can safely remove the offending key and the new key will be added the next time you connect. Fortunately, this is easily done with a sed one-liner:

$ sed -i -e '185d' .ssh/known_hosts

In this case, '185' is the line number that was reported as containing the offending key.

Changes to OS X TCP Performance Tuning

Scott Hebert — Mon, 01 Feb 2016 15:00:00 +0000

While updating the TCP tuning parameters on one of my OS X 10.11 servers, I noticed that my existing OS X TCP Performance Tuning page had gotten out of date. The page was nearly eight years old, so it is no surprise that happened. I updated the page in...