0x10 - Security, sysadmin and stuff

When long tail meets viral

2014-03-27T09:45:00.000+00:00

Well, not really viral, just busier than expected.

Here is a screenshot showing all time traffic stats for this blog.

The absolute traffic volume of this blog isn't very exciting (nor very high). As you probably noticed, what is interesting is the recent spike in traffic. This isn't for a recent post that went modestly "viral", it is a 4 year old post. Out of the blue, for no apparent reason, the 10 useful little bash tricks post has become the most popular post on this blog by far. Huh.

Dirty, dirty devops

2013-11-01T17:24:00.002+00:00

No this isn't an opinionated rant about devops culture or anything. Its just a hacky knife plugin that lets me play with Sensu quickly.

I want to play with http://sensuapp.org/ as an alternative to Nagios and Zabbix etc. I've already got a nice git repo full of our cookbooks etc, but I don't want to go near them. I don't want to even risk messing up master etc.

Note: This assumes you have a chef workstation set up with knife available etc.

Anyway, I wrote this tiny little knife plugin that just calls a couple of bash commands:

require 'chef/knife'
# other require attributes, as needed

module Sandbox
class SandboxInit < Chef::Knife

deps do
end

banner "knife sandbox init (options)"

def run
puts "Creating chef repo as chef-repo"
system("git clone git://github.com/opscode/chef-repo.git")
puts "Creating .chef dir"
system("mkdir chef-repo/.chef")
puts "Creating knife config file"
system("echo 'current_dir = File.dirname(__FILE__)\ncookbook_path [\"\#{current_dir}/../cookbooks\"]' > chef-repo/.chef/knife.rb")
puts "Done. You can know cd into chef-repo, run knife cookbook site install BLAH and use vagrant etc"
end
end
end

See what I mean with hacky? Eww. Anyway, stick this file in ~/.chef/plugins/knife/ then you can use it as follows:

$ mkdir playing-around
$ cd playing-around
$ knife sandbox init
WARNING: No knife configuration file found
Creating chef repo as chef-repo
Cloning into 'chef-repo'...
remote: Counting objects: 223, done.
remote: Compressing objects: 100% (139/139), done.
remote: Total 223 (delta 80), reused 178 (delta 49)
Receiving objects: 100% (223/223), 38.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (80/80), done.
Checking connectivity... done
Creating .chef dir
Creating knife config file
Done. You can know cd into chef-repo, run knife cookbook site install BLAH and use vagrant etc
$ cd chef-repo

Now you can follow the instructions here https://github.com/sensu/sensu-chef

$ knife cookbook site install sensu
...
$ cd cookbooks/sensu/examples
$ bundle install
...
$ librarian-chef install
...
$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'opscode-ubuntu-13.04'...
[default] Matching MAC address for NAT networking...
[default] Clearing any previously set forwarded ports...
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] -- 8080 => 8080 (adapter 1)
[default] Running 'pre-boot' VM customizations...
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
...
[2013-11-01T17:16:35+00:00] INFO: Chef Run complete in 227.664146883 seconds
[2013-11-01T17:16:35+00:00] INFO: Running report handlers
[2013-11-01T17:16:35+00:00] INFO: Report handlers complete
[2013-11-01T17:12:47+00:00] INFO: Forking chef instance to converge...

We now have a machine that should have Sensu on it:

$ vagrant ssh
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64)

* Documentation: https://help.ubuntu.com/

94 packages can be updated.
55 updates are security updates.

New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Sun May 12 22:41:29 2013 from 10.0.2.2
vagrant@vagrant:~$ ps -ef | grep sens
sensu 6097 1 0 17:16 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-server -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-server.pid -l /var/log/sensu/sensu-server.log -L info
sensu 6154 1 0 17:16 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-api.pid -l /var/log/sensu/sensu-api.log -L info
sensu 6211 1 0 17:16 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-dashboard -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-dashboard.pid -l /var/log/sensu/sensu-dashboard.log -L info
sensu 6276 1 0 17:16 ? 00:00:00 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -b -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/sensu-client.log -L info
vagrant 6740 6651 0 17:21 pts/0 00:00:00 grep --color=auto sens

Woot. You can even browse the dashboard on http://localhost:8080 (username is admin and the password is secret).

When you're done, you can quickly clean up after yourself:

$ vagrant destroy
$ cd
$ rm -rf playing-around

How to ask about connectivity issues

2013-10-25T10:33:00.001+01:00

So you're trying to connect one system to another using this newfangled TCP/IP stuff the kids are using these days. But what's this? It doesn't work? You need to ask someone to look into it. You even know who. But what should you ask and what information should you provide so that they can investigate quickly and effectively?

As well as a bit of background to the problem (applications involved, environments etc), you should provide at least the following:

Has it ever worked?
Have you changed anything recently?
Source IP
Destination IP
Protocol (TCP or UDP etc) if it isn't obvious from the context
Port if it isn't obvious from the context

Common causes are:

Wrong information - are you connecting to the right host?
Routing
Firewalls
Network or system outages

Have you tried the following?

Doing a host, dig or nslookup if dealing with a domain
Pinging the remote host
Connecting using netcat or telnet
Traceroute

It is probably worth providing the output to the above commands if possible.

How do I report a vulnerability?

2013-08-16T00:05:00.001+01:00

Hanging out in the ##security IRC channel on Freenode you see a number of reoccurring topics. One of them is when someone says they've found a vulnerability, often in some website, and want suggestions on what to do next. So, I present to you:

http://howdoireportavuln.com

Edit: The site has had some major updates following feedback from Jericho. Feel free to leave feedback in the comments below.

A little python vs ruby

2013-05-16T17:10:00.002+01:00

Just a little comparison of python vs ruby which nicely sums up why I prefer ruby to python.

Coming from a perl background, parsing strings and manipulating data structures is the bread and butter of sysadmin scripting. Here we're going to split a string, add to the array, then join into a string again.

First, in python:

a = "a b c".split()
>>> a
['a', 'b', 'c']
>>> len(a)
3
>>> a.append("d")
>>> ",".join(a)
'a,b,c,d'

Ok, fair enough I guess. Now lets look at ruby:

a = "a b c".split
=> ["a", "b", "c"]
a.size
=> 3
a.push "d"
=> ["a", "b", "c", "d"]
a.join ","
=> "a,b,c,d"

Notice the difference? I'll try to summarise it in text.

For python, we split the string to get an array. We then run a global method to find the size of the array. We then call a method on array to append the new element. Finally we run a join method on a string and pass it the array.

For ruby, we split the string to get an array. We then run a method on the array to get the size (.length would also have worked). We then run a method on the array to add the element. We then run a method on the array to join to get our final string.

To me ruby is far more consistent. I'm not saying python is a bad language, I just find ruby to be particularly enjoyable to use.

Criticisms, suggestions and improvements welcome :)

Passphrase Generators Part II

2013-04-16T23:09:00.001+01:00

A while back I wrote a post about using bash as a passphrase generator (here). The idea being that it would be better to pick "randomly" from a list of common words than it would be to try to mentally choose four words at random. Humans are not very good a being random [1]. Derren Brown, if you're reading this, I'd love to see an episode where you plant a password into someone's head for something they believe is of utmost importance. This post addresses some security concerns about how the words are chosen at random.

Speaking of passphrases and passwords, I should probably clarify what I mean by the two terms. A password I take to mean a private string which more or less resembles a "word" in the traditional sense. Typical and poor examples would be "password", "Password1", "aksdfjslfjdf" etc. Something like "8yDcBPAllEFOLNNI89ZN4nq5" being a good example. A passphrase I take to mean a private string composed of multiple "words". For example "That's me in the corner, that's me in the spot light", "slgl sdf mcr asrj" are poor examples, and "Cupertino teenage fell Paradox" is a good example. The analogy being to word and phrase in spoken languages. So generally, a good passphrase will be better than a good password because of length, assuming you havn't done something silly like using the lyrics of your favorite song. The passphrase would still need to be random, but easier to remember.

Anyway, having spent some time chatting on IRC about the flaw of using % 10000, using Bash's $RANDOM variable and a few other things, I thought I would update the script to be more secure. The entropy lost from using % 10000 isn't particularly significant, but the way $RANDOM works means you could probably reduce the choice of possible words significantly if you knew when approximately the passphrase was being generated. Basically, $RANDOM seeds off gettimeofday's seconds, useconds and the bash PID. I haven't done the maths for exactly what the impact is, but I thought it would be worth this blog post.

I am actually going to provide two new scripts. The previous method used a dictionary file with 10000 common words downloaded from the internet, and embedded them in the Bash script. Now, I still think this is a good approach because 10000^4 still gives enough possible combinations and because there are only 10000 words, many words will be familiar to most people and should therefore be easier to remember. I will first present an alternative approach.

#!/bin/bash
words=${1:-4}

while true; do
clear
for i in $(seq $words); do
word=$(/usr/bin/shuf -n1 --random-source=/dev/urandom /usr/share/dict/words)
echo -n "$word "
done
echo ""
read -p "Accept (y/n) [n]: " answer

if [ "$answer" == "y" ]; then
exit 0
fi

done

You will probably notice two things. Firstly, it only takes one argument; the number of words to show, or 4 by default. The previous version of the script printed X number of words on Y lines so that you could choose a passphrase that suited you. However, there was always the temptation to create a new passphrase from a combination of what was shown on the screen. This is not ideal. You are better off rejecting a number of whole passphrases and then accepting one as-is, rather than combining several passphrases into one. The reason being that the process of combining passphrases is no longer random and there will probably be a bias towards a structure such as "adjective noun verb adjective". Or something like that anyway.

So, instead, we just keep looping until you accept the passphrase and clear the screen if you reject it (the default). By clearing the screen I hope to lessen the chance of you combining several passphrase - out of sight, out of mind as they say.

The second major change here is that instead of using 10000 common words downloaded from the Internet, we're using the OS's provided dictionary file. If your distribution doesn't have that file, see here http://en.wikipedia.org/wiki/Words_(Unix).

We use the shuf command X number of times to select X words for the passphrase. On one of my machines there are 98326 words in /usr/share/dict/words. I think I saw many more on my Ubuntu system. So you can probably expect some pretty obscure passphrases. You'll also note that we tell shuf to use /dev/urandom as the source of randomness as the built in source isn't secure.

Right, now onto the other option. In this case we're going to just modify the previous script and fix the source of randomness. We'll also introduce the while true loop and hide previous rejected passphrases.

#!/bin/bash

dict=$1

script=$2

chars="a-zA-Z0-9 "

size=$(cat $dict | wc -l)

echo "#!/bin/bash" > $script

( echo -n "dict=( "; i=0; cat $dict | while read word; do echo -n "$word " | tr -c -d "$chars"; if [ "$i" -eq "10" ]; then i=1; echo ""; fi; ((i++)); done; echo ")" ) >> $script

echo "size=$size" >> $script

cat << 'EOF' >> $script

words=${1:-4}

while true; do

clear

for i in $(seq $words); do

index=$(cat /dev/urandom | tr -d -c '0-9' | fold -w 4 | head -1 | sed 's/^0*//')

word=${dict[$index]}

echo -n "$word "

done

echo ""

read -p "Accept (y/n) [n]: " answer

if [ "$answer" == "y" ]; then

exit 0

done

EOF

chmod +x $script

Other than the loop change, the only difference is our source of randomness. Instead of using $RANDOM we read /dev/urandom directly. We strip out anything other than digits using the tr command and fold it to 4 characters (0000 to 9999). We then just take the first result and strip off any zeros from the front. We do this because bash doesn't treat 0080 as 80 and isn't happy using 0080 as an index. There is probably a better way of stripping of the zeros using bash's string manipulation, but this will do for now.

Running this script is the same as the previous version, you give it the dictionary and an output file name. E.g.

$ wget http://wortschatz.uni-leipzig.de/Papers/top10000en.txt

$ ./make.sh top10000en.txt ppgen.sh

Unfortunately the dictionary isn't provided over HTTPS, so it is possible that someone is going to MITM your download and change all the words to suit themselves.

Perhaps something like this might help:

$ wc -l top10000en.txt
10000 top10000en.txt

$ sort < top10000en.txt | uniq | wc -l
10000

That way at least you know there are 10000 unique words. Either that or find a new dictionary available over HTTPS (and let me know where you found it if possible).

Hopefully you'll find either of these approaches useful.

[1] http://www.ncbi.nlm.nih.gov/pubmed/17888582

Ultra Light Graphs (ULG)

2013-03-14T00:34:00.002+00:00

Ultra Light Graphs, or ULG for short, is a tool that allows you to create basic graphs using nothing more than a few lines of text. The text format is designed to be meaningful in itself, which makes ULG a sort of markup language for graphs. It isn't intended to replace something more comprehensive like graphml or dot, but it does allow you to quickly write something that is readable in your favourite text editor and can be turned into an image that you might want to insert into documentation etc.

Edit:

Installation can do be done using ruby gems:

$ gem install ulg

Using ULG

ULG is very easy to use. Just give it a file with the text in it and it will generate a png by default.

$ ulg -h
Usage: ulg.rb [-o OUTPUT] [-s FILE] [-d] FILE FILE...

Reads from stdin if no input FILE given

Other options:

-o OUTPUT Output format (png, dot, svg). Default png
-s FILE Save to file
-d Debug mode
-h This help

For example

$ ulg basic.ulg
Reading basic.ulg
Saving output to basic.ulg.png

Basic example

Right, lets say you want to draw a basic graph. You open up your favourite text editor such as vim. You want two nodes and an arrow, so you might intuitively type something like this:

internet ==> web dmz

Now, this is what ULG would make of just that one line:

Nice and simple.

Node shapes

Maybe you don't want two boxes. Perhaps you want one of them to be oval. That's easy too:

(internet) ==> [web dmz]

That would give you:

Notice also how we've also explicitly marked "web dmz" as a box.

Edge label

You've probably noticed that the arrow, or edge in graph-speak, doesn't have a label. We can give it one by just inserting some text somewhere inside the arrow:

(internet) ==http==> [web dmz]

This would give us:

Arrows

You can also control the shape of the arrow head, for example:

(internet) ==http==<> [web dmz]

Gives us:

Edge line

Of course, you might like your lines to be dashed instead of solid. No problem:

(internet) --http--> [web dmz]

Here we go:

Interesting example

Now, lets make things a little bit more interesting. I'll show you the text and image first, then I'll explain a few things afterwards:

option layout dot
option overlap false
option splines true

include backoffice.ulg

Internet ==http==> (web dmz|orange)

web dmz ==8080==> [app stack 1|blue]
web dmz ==8000==> [app stack 2|purple]

app stack 1 ==25==> internet
app stack 2 ==5432|red==> [database|green]

Where backoffice.ulg is

(back office) ==5432|red==> database
(back office) ==80==> (internet|red)

This would result in this rather pretty graph:

Quite a few things have happened here. Looking at the text, you will have noticed three lines starting with the word "option". These are basically just option names and values that get passed directly to Graphviz as global options.

We also include a ULG file called "backoffice.ulg", the contents of which is shown afterwards. You'll see that backoffice defines an "internet" node and the main text refers to an "Internet" and an "internet" node and that the graph only shows one "internet" node. This is because ULG ignores case and spaces when determining what the node is. So "Internet", "internet" and "inter net" would all be the same thing. This allows you to be a little be messy and inconsistent with your typing ;)

The "internet" node was labelled "internet" and not "Internet". This is because ULG defines styles based on the first occurrence of the node or edge. Because we included backoffice.ulg first, it defined the style for the internet node. This also means that once we have defined a node, such as the "(web dmz|orange)" we ignore any style characters and simply use the label. The next two occurrences were referred to as just "web dmz".

Another thing you will have noticed is the use of colour. This is achieved by simply adding "|colour" to the node or edge label. Any colours Graphviz understands should be fine.

Its probably also worth mentioning at this point that ULG doesn't care about whitespace between nodes and arrows, so you can use spaces or tabs - whatever is easiest to read. It does however require that an arrow is made up of at least two line characters (such as == or --)

Other things you can do

Finally, this example shows the different combinations of supported node and arrow styles.

# This line is ignored

# Include a file
include common-opts.ulg

# An option
option rankdir TB

# Graph
[box] = normal ==> (oval|blue)
oval =dot|red=o <diamond>
oval ..box....x box
box --diamond--<> diamond

another box === no arrow == {hexagon}

This gives us:

Hopefully this example speaks for itself.

Download

GitHub: https://github.com/zeroXten/ulg
Gem: https://rubygems.org/gems/ulg

OpSec Basics

2013-03-11T21:53:00.000+00:00

This is a simple little list of OpSec ideas that should be followed in most situations. Nothing ground breaking, just a gentle reminder :)

Take regular backups

Regularly backup systems and data using a combination of incremental and full backups.

Encrypt transmitted data whenever possible

Even on local networks. You may have perimeter firewalls and a switched network, but you have to assume someone is listening.

Minimize software to minimize vulnerability

Reduce the attack surface for remote access and local privilege escalation by only having essential software installed and services running.

Run different network services on separate systems

Not only does this simplify diagnosing operational issues, but means a compromise of one service doesn't directly affect other services. It also reduces the possibility of another service being used in the next stage of an attack.

Least privilege

This should apply at all levels, from system accounts to application logins. Don't use shared credentials and use systems that have granular enough permissions. Someone might not want to do something malicious, but if their system has been compromised the attacker might.

Patch your sh!#

Unpatched software is a liability. Every unpatched system is low-hanging fruit for an attacker.

Be informed

You should have a clear insight into the standard operational and opsec behaviour of your environment. This means having good operational monitoring and metrics, security systems such as HIDS and NIDS and a hardened centralized log collector for system logs and SIEM. Once you have that, take the time to review logs, events and set up alerts if possible.

Segment your network

Compromise is inevitable so the trick is to limit damage once it happens. Network segmentation stops the attack from moving around freely and thus adds complexity and delay to the next stages of their attack.

Use good crypto and don't roll your own

This applies to everything from choosing a password, to storing passwords and encrypting backups. Don't rely on home-made encryption or obfuscation tricks.

Have a response

It is a question of when you get hacked, not if. You should have an incident response plan well in advance of anything happening. Basics should include how to limit damage, how to collect information for forensics or the authorities and details of who to contact.

Default configuration

When you buy a new piece of kit, build a new system or install an application, or should remember to check and adjust any default configuration. Most things try to be easy, not secure, so leaving in default configuration can be dangerous. Default users are a classic example, but other more subtle configuration might give too much information away or might be permissive in terms of access or actions.

Sources:

http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf
Common sense

What I like about my 2002 Volvo S40 SE 2.0 Auto

2012-09-22T14:17:00.003+01:00

Ok, so, for a completely off-topic post about a car. I bought a 2002 Volvo S40 SE 2.0 Automatic a few weeks ago for family day trips etc and not commuting. So I wanted something cheap, comfortable and safe. This car seems to pretty much fit exactly, so much so that I thought I'd share some of the things I like about it. Perhaps you are in a similar situation and are considering buying a second-hand Volvo.

Comfort

For long day trips or weekends away this is quite important. Bouncing down the motorway in a old Fiesta isn't exactly comfortable. So, what does my car give me?

Climate control - Not just aircon, but proper climate control. Set it to AUTO, choose a temp and voila.
Automatic - Not to everyone's taste, but I'm kinda liking it. Makes traffic jams bearable.
Cruise control - Haven't used it yet, but nice idea for those long motorway journeys.
Heated front seats - Still too warm for them, but I think they'll come in handy in the winter :)
Adjustable dashboard light brightness - Nice little touch.
CD and tape player - CD player is a requirement, and the tape player is even handy for playing mp3s or Spotify from my phone.
2.0 Engine - Car is pretty responsive.
Heated door mirrors - Again, too warm to have used them yet, but will come in handy later.

Safety

The most important thing as a family man is of course safety.

Side markers - Little orange lights on the sides of the car at the front and back to help visibility of the car from the side. Required in North America but not in Europe, I didn't see a single other car last night with them whilst walking in town.
Headlight wipers - Not only do they look kinda cool, but sort of handy.
Driver and passenger airbags - The basics.
Side impact cushions - A nice extra little touch, adding to safety.
4 star Euro NCAP rating - "The S40 was awarded four stars for protection in frontal and side impacts, the only one of 13 family cars tested to achieve this result"

Family Friendly

The idea was to get a car that was big enough for day trips without being too big for the terrible on road parking where I live.

Parking voucher clip - Silly, but very cool. A little clip on the right of the windscreen to place parking vouchers etc. Handy, really handy and a nice little touch.
Boot size - 415 litres, big enough to put the pram base in without taking off the wheels.
Lots of random storage - Little storage spaces throughout the car.
Cup holders in central arm rest - Another nice little touch.
Full electric windows with driver-side control - All four door windows are electric.
Leather seats - Easy to clean and spill friendly, perfect for kids in the back.

Finally, and not in any particular category, awesome advice from the excellent community at volvoforums.org.uk.

Visualisation with Scratch

2012-08-19T22:03:00.001+01:00

Scratch allows you to create interactive stories, games, music and art using drag-and-drop programming. It is an excellent way to introduce basic programming concepts, such as variables and loops. But, it is also quite powerful, as it supports bi-directional communication to such things as Lego Mindstorms, and more generally, using TCP/IP. I decided to see what sort of data visualisation I could achieve with a few spare minutes and a bit of python code.

The first attempt ended up as a sort of "meme cloud". Here, you send Scratch an alert level and a message, and it will display a meme based on the level, with the message in a speech bubble.The levels are:

The most recent alert is shown with no transparency and with the speech bubble. As the alerts get older, they become increasingly transparent. Positions of the memes are random. It currently only supports 10 sprites, which means it will show up to 10 memes at any one time.

Here is a section of the code for one of the sprites:

The problem with this approach is that Scratch doesn't allow you to dynamically create sprites, which sort of limits the functionality for certain things. However, there is an alternative. A sprite has, amongst all sorts of things, a pen. This pen can have colour, thickness, shade etc. Moving the sprite while the pen is "down" basically just draws on the screen. So, although we can't dynamically create sprites, we can draw all sorts of things programmatically. This takes us to attempt number two.

The plan here was to draw some sort of time chart or graph. A python script sends Scratch a pen size and colour for every event, and Scratch draws this. As it receives events, the pen moves to the right to create a sort of time chart. If the pen gets to the end of a line, it drops down to a new line. If it gets to the bottom of the screen, it goes back to the starting position in to top left corner. This means you can see the details of a relatively large number of events on a single screen, in order of time.

The example video below just uses random colours and pen sizes, so you won't be able to see any patterns. However, with real data you should be able to see correlations such as spikes in the number of errors, or outages etc. Also, the time between events in the video is fixed, so it looks a little boring. The white square shows the current position.

Not bad for a bit of dragging-and-dropping, and some python. I'll leave it as an exercise for the reader to integrate it into something like Splunk or Nagios :)

You can download the meme cloud python here, and the meme cloud scratch file here. The line viz python can be downloaded here, the scratch file here.

Passphrase generator in bash

2012-01-18T10:04:00.000+00:00

[EDIT]

The script below has been improved upon and can be found here: http://blog.0x10.co.uk/2013/04/passphrase-generators-part-ii.html

Passwords are dead. Long live passphrases. Or something like that anyway. As perfectly summarized by XKCD here http://xkcd.com/936/, passwords are hard to remember and easy to guess. Passphrases on the other hand are easy to remember and hard to guess.
To help generate passphrases, I've put together a little bash script.
Firstly, download the dictionary containing 10000 most common English words. If you have a 4 word passphrase, thats 10^16 possible combinations.

$ wget http://wortschatz.uni-leipzig.de/Papers/top10000en.txt

Now save the following script:

 
#!/bin/bash

dict=$1
script=$2

chars="a-zA-Z0-9 "
size=$(cat $dict | wc -l)
echo "#!/bin/bash" > $script
( echo -n "dict=( "; i=0; cat $dict | while read word; do echo -n "$word " | tr -c -d "$chars"; if [ "$i" -eq "10" ]; then i=1; echo ""; fi; ((i++)); done; echo ")" ) >> $script

echo "size=$size" >> $script
echo 'count=${1:-10}
words=${2:-4}

for i in $(seq 1 $count); do
    first=1
    for j in $(seq 1 $words); do
        r=$RANDOM
        let "r %= $size"
        word=${dict[$r]}
        if [ "$first" == "0" ]; then
            echo -n " "
                fi
        echo -n "$word"
        first=0
    done
    echo ""
done
' >> $script
chmod +x $script

Using modulo on a random number? tut tut tut. Anyway, I've called this script make.sh.
Now run it, giving it the dictionary file and an output script name:

$ ./make.sh top10000en.txt ppgen.sh

What it basically does is embed an array of words into ppgen.sh plus the code necessary to show passphrases.
It will look something like:

#!/bin/bash
dict=( the of to and a in for is The that on 
said with be was by as are at from 
it has an have will or its he not 
were which this but can more his been would 
...
definite fragile rewards antiabortion respects careers backers seize inefficient 
conceptual densities EPS Me Sparc spirits experimentally shallow )
size=10000
count=${1:-10}
words=${2:-4}

for i in $(seq 1 $count); do
    first=1
    for j in $(seq 1 $words); do
        r=$RANDOM
        let "r %= $size"
        word=${dict[$r]}
        if [ "$first" == "0" ]; then
            echo -n " "
                fi
        echo -n "$word"
        first=0
    done
    echo ""
done

We can now run it:

$ ./ppgen.sh 
Does Brazil quotas message
amount losers sing Oregon
patient misleading rebels smoking
refuse subordinated clerk wrote
have handle Rockwell strong
Committee Insurance consumer else
tariffs Unit mouth loved
V singled graduate kidney
uranium Hungary scientists Mar
fined Hercules Kemp Indian

Or, if we wanted to create just one 5 word phrase:

$ ./ppgen.sh 1 5
Commodity dioxide Boris unit drop

Visualising rules from Firewall Builder - Part 1

2012-01-12T02:20:00.000+00:00

Firewall Builder (available here http://www.fwbuilder.org/) is a GUI that lets you configure a variety of firewall devices, for example iptables on a linux box or Cisco ASAs.

The idea is to take the configuration file from Firewall Builder and generate some sort of rule visualisation. The ruleset I'm using is basically just the built-in template called "fw template 3". I then saved the ruleset as testfw.fwb.

So, lets process the file:

user@host:~/fwbvis$ ./fwbvis.py testfw.fwb ignoreme testfw.dot
user@host:~/fwbvis$

Where the arguments are as follows:

fwbvis.py - the script (below)
testfw.fwb - the firewall config saved by Firewall Builder
ignoreme - to be ignored for now. It will become a configuration file for colours etc.
testfw.dot - the output file. We just generate a png at the moment, in this case called testfw.dot.png

Here is the result:

It isn't perfect, but it is a pretty good start. The colours are as follows:

red - reject
orange - deny
green - accept
blue - anything else

So, here is the script so far. It's pretty ugly btw...

#!/usr/bin/python

import sys
import pydot
from xml.etree.ElementTree import ElementTree

sFWFile = sys.argv[1];
sConfigFile = sys.argv[2];
sOutputFile = sys.argv[3];

try:
    ds = open(sFWFile)
except:
    print "Could not open file", sFWFile
    sys.exit(1)

try:
    tree = ElementTree()
    tree.parse(sFWFile)
except:
    print "Could not parse file", sFWFile
    sys.exit(1)

graph = pydot.Dot(graph_type='digraph')

prefix = ""
rules = []
objects = {}
for element in tree.iter():
    tag = element.tag
    if tag[0] == "{":
        uri,tag = tag[1:].split("}")
        if prefix == "":
            prefix = uri
    oid = element.get('id')
    if oid is None:
        next
    objects[oid] = {'tag': tag }
    for (key,value) in element.items():
        if key != 'id':
            objects[oid][key] = value

for rule in tree.findall('.//{%s}PolicyRule' % prefix):
    oid = rule.get('id')
    action = objects[oid]['action']
    color = "black"
    if action == "Reject":
        color = "red"
    elif action == "Deny":
        color = "orange"
    elif action == "Accept":
        color = "green"
    else:
        color = "blue"

    src = rule.find('.//{%s}Src' % prefix).find('.//{%s}ObjectRef' % prefix).get("ref")
    if src in objects:
        src = objects[src]['name']
    dst = rule.find('.//{%s}Dst' % prefix).find('.//{%s}ObjectRef' % prefix).get("ref")
    if dst in objects:
        dst = objects[dst]['name']
    srv = rule.find('.//{%s}Srv' % prefix).find('.//{%s}ServiceRef' % prefix).get("ref")
    if srv in objects:
        srv = objects[srv]['name']
    else:
        srv = "unknown"
    graph.add_edge(pydot.Edge(src, dst, label=srv, color=color))

#graph.write(sOutputFile)
graph.write_png(sOutputFile+".png")

So, there you have it. More to follow as things improve.

Some bash array functions

2012-01-01T00:00:00.001+00:00

Following on from the previous post, here are some functions for bash arrays. I'll add to this post later with more functions.
Test if a variable is an array:

# is_array - Tests whether variable name is an array
# Parameters:
#   name - name of variable to test
# Returns:
#   0 if an array
#   1 otherwise
is_array()
{
    # Only takes one argument
    if [ "${#}" -ne "1" ]; then
        return 1
    fi

    # Name of the variable to test
    local name="$1"; shift
    # Run the declare command against the variable
    local x=$(declare -p $name 2>/dev/null)
    # If it errored then the name probably isn't even
    # a variable
    if [ "$?" -ne "0" ]; then
        return 1
    else
        # It is at least declared, test if it is an array
        if [ "${x:8:2}" == "-a" ]; then
            return 0
        else
            return 1
        fi
    fi
}

Push values onto an array:

# array_push - Pushes one or more values onto an array
# Parameters:
#   name - name of the destination array
#   ...  - values to push onto array
# Returns:
#   Nothing
array_push()
{
    # Need two or more arguments
    if [ "${#}" -lt "2" ]; then
        return
    fi

    # Name of the destination array
    local name="$1"; shift
    # Number of elements we're going to push
    local numElements=${#}
    # Counter
    local i=0
    # Starting position of destination array is the
    # size of the array
    eval local start=\$\{\#$name\[\@\]\}
    # Iterate over the elements supplied, adding them
    # to the destination array
    while [ "$i" -lt "$numElements" ]; do
        pos=$((start+i))
        eval $name\[$pos\]=\"$1\";
        shift
        ((i++))
    done
}

Pop from an array:

# array_pop - pops a value from the end of the array
# Parameters:
#   name - name of array
#   dest - optional name of variable to put value in
# Returns:
#   Nothing
array_pop()
{
    # Need at least one argument
    if [ "${#}" -eq "0" ]; then
        return
    fi

    # Name of the array
    local name="$1"; shift
    # If we have a second argument, use it to write value
    if [ -n "$1" ]; then
        local dest="$1"; shift
    else
        local dest=""
    fi
    # Size of the array
    eval local numElements=\$\{\#$name\[\@\]\}
    # Index of element is one less than size
    local index=$((numElements-1))
    # Get the value
    eval local value=\"\$\{$name\[$index\]\}\"
    if [ -n "$dest" ]; then
        # Update the specified dest variable
        eval $dest=\"$value\"
    else
        # Simply echo the value
        echo "$value"
    fi
    # Unset from the array
    eval unset $name\[$index\]
}

One thing to note is the clash between variable names. E.g. if you wanted to return the value into the name variable, it would be bad to call that variable 'value'. local doesn't help here, best option would probably to use obscure variable names that are unlikely to clash.
Walk an array:

# array_walk - Walks through an array, calling the
#              callback function for each element
# Parameters:
#   name     - name of array to walk
#   callback - name of callback function
#   ...      - any parameters to be passed to callback
# Returns:
#   Nothing
array_walk()
{
    # Name of array to walk over
    local name="$1"; shift
    # Name of call back function
    local callback="$1"; shift
    # TODO - We should probably check whether callback
    # is actually a function

    # Number of elements in array
    eval local numElements=\$\{\#$name\[\@\]\}
    # Counter
    local i=0
    # Iterate over the elements in the array
    while [ "$i" -lt "$numElements" ]; do
        # Get the value
        eval value=$\{$name\[$i\]\}
        # Check whether it is an array
        is_array $value
        if [ "$?" -eq "0" ]; then
            # If so, recursively walk that array
            array_walk "$value" "$callback" "$@"
        else
            # Just a variable, so call the callback
            # function giving it the args and value
            $callback "$@" "$value"
        fi
        ((i++))
    done
}

Now, a little play with those functions:

declare -a aTest

echo First push
array_push aTest 'Hello' 'there' 'you' ':)'
echo new array is ${aTest[@]}
echo

declare -a aTest2
echo Second push
array_push aTest2 'How' 'are' 'you today?'
echo second array is ${aTest2[@]}
echo

echo Third push to add second array to the first
array_push aTest 'aTest2'
echo first array is ${aTest[@]}
echo

echo Defining destination array
declare -a aMerged
echo defining callback function
merge()
{
    local name="$1"
    local value="$2"
    # Simple push the value onto the destination
    array_push "$name" "$value"
}
echo Doing walk
array_walk aTest merge aMerged
echo merged array is ${aMerged[@]}
echo

echo Popping last element of merged array by reference
array_pop aMerged val
echo Value is $val
echo and again
array_pop aMerged val
echo Value is $val
echo merged array is now ${aMerged[@]}

Running it gives us:

$ ./arrays2.sh 
First push
new array is Hello there you :)

Second push
second array is How are you today?

Third push to add second array to the first
first array is Hello there you :) aTest2

Defining destination array
defining callback function
Doing walk
merged array is Hello there you :) How are you today?

Popping last element of merged array by reference
Value is you today?
and again
Value is are
merged array is now Hello there you :) How
$

So that sort of works. Next I should add an array_indexes() and array_values(). Oh, and happy new year :D
Edit:
Added array_pop function.

A different 2D array in bash

2011-12-29T11:51:00.000+00:00

I have recently been writing a pretend Object Oriented Programming layer on top of bash. Basically, it uses some global vars, function-within-functions and a ton of eval to create the illusion of OOP in bash.

Something like:

#!/usr/lib/oobash.sh

# Import the Car class
`import Car`

# Create a new "object"
new Car as mycar

# Set a variable
mycar.speed 100

# Call a function
mycar.drive

This will be made available here http://code.google.com/p/object-oriented-bash/ soon, but in the meantime I have needed to implement a 2D array in bash. A quick Google search shows some different ways of doing this. Firstly is to dynamically create arrays using eval. The other approach is to use a huge offset, e.g. 1000, then refer to array[i][j] as array[i*1000+j] etc. The latter example requires you to keep track of two indexes which is sort of annoying.

Edit:

I noticed someone has already done something similar at http://oobash.sourceforge.net/. I'l take a look at it some point and if it does the job I'll abandon my project.

So, in an attempt to find something different, I came up with the following. The idea is to store some metadata about how many elements we're adding.

An array with just one element might look like this:

[0] = 1
[1] = "Hello there"

The element at index 0 is the number of elements that follow. As the value is 1, the next element contains the value, which would basically be a scalar. Now, an array might look like this:

[0] = 2
[1] = "Hello"
[2] = "there"

Element 0 tells us to expect 2 values. Nice and simple huh? A more complete example:

[0] = 1
[1] = "Hello"
[2] = 3
[3] = "How"
[4] = "are"
[5] = "you?"

You get the idea. But, we have a problem. With arrays, one often wants to pop from the end of the array. In order to do that, we have to start at the end and work backwards. But the following example would break that:

[0] = 1
[1] = "Hi"
[2] = 3
[3] = "aaa"
[4] = 1
[5] = "bbb"

If we worked backwards from 5, we would get to element 4, which looks like an array count and conclude that (4,5) was a scalar pair. That isn't actually the case. But, there is a solution.

[0] = 1
[1] = "Hi"
[2] = 1
[3] = 3
[4] = "aaa"
[5] = 1
[6] = "bbb"
[7] = 3

At the cost of some space, we've added a count element at the end of each array. That way we can go to the last element, and know how far backwards to go.

Well, that's the theory. Now lets look at a push:

#!/bin/bash

# Function: array_push()
# Params:
#   name     - name of array receiving data
#   indexVar - name of variable to store index used
#   ...      - list of one or more values to be pushed
# Returns:
#   Nothing really
array_push()
{
    # Require at least 3 parameters
    if [ "${#}" -lt "3" ]; then
        return
    else
        # Set parameter variables
        local name="$1"; shift
        local indexVar="$1"; shift

        # Get the next index
        eval local index=\$\{\#$name\[\@\]\}

        # Make a note of how many elements we're pushing
        local count="${#}"

        # Set the index var to our index. This can then be used later 
        # to retrieve data from the array.
        eval $indexVar=$index

        # Record the count
        eval $name\[$index\]=$count

        # Moving up in the array
        ((index++))

        # We now iterate over the values, adding them to the array
        end=$((index+count))
        while [ "$index" -lt "$end" ]; do
            eval $name\[$index\]=\"$1\"
            shift
            ((index++))
        done

        # Finally, make another record of the count
        eval $name\[$index\]=$count
    fi
}

declare -a aTest

echo First push
array_push aTest iIndexA "Good day."

echo "index: $iIndexA"
echo new array is ${aTest[@]}
echo

echo Second push
array_push aTest iIndexB "To" "you" "sir!"

echo "index: $iIndexB"
echo new array is ${aTest[@]}

Running it gives us:

$ ./push.sh 
First push
index: 0
new array is 1 Good day. 1

Second push
index: 3
new array is 1 Good day. 1 3 To you sir! 3
$

Wonderful. We're not doing anything useful, but the data is there. It was at this point that I got bored of this approach. The problem is that doesn't extend to beyond two dimensions. The only way around this would be some sort of JSON like beginning/end tags instead of a count. E.g.

[0] = {
[1] = 'a'
[2] = }
[3] = {
[4] = 'b'
[5] = 'c'
[6] = {
[7] = 'd'
[8] = }
[9] = }

This is could get a bit slow to process. We'd have to do a lot of string comparisons, and we'd need to quote values to prevent confusion.

Alternatively, we could use an array as usual, but have our functions automatically test values to see if they're an array. That blog post shall follow shortly.

Fundamentals principles of better security

2011-10-26T18:34:00.001+01:00

Defence in depth

This is the daddy of all security advice. The idea is to not rely on any one particular technology, process or approach to security. By building up layers of defence you can tolerate the eventual compromise of one of the layers. For example, don't just run AV software and hope for the best, harden your system and keep it up to date. Then use technologies such as whitelisting, sandboxing and URL blacklisting to further reduce risks. Don't just rely on a perimeter firewall or NAT, firewall between networks and use IDSs.

Security is a process

Similarly to the defence in depth advice, don't ever expect to be completely secure. No product or solution will ever make you 100% hacker proof, regardless of what the marketing material states. Security is a process of identifying vulnerabilities, prioritizing and mitigating risks, making things a little bit better each time.

Security through obscurity

Sticking with obvious advice, don't rely on security through obscurity. This means relying on the secrecy of a design, implementation or process for security. For example, rather than relying on some obscure proprietary encoding to protect data, you should use well known and tested cryptography. It shouldn't matter that someone knows your data is protected with 256bit AES.

Crypto is hard

Speaking of cryptography, it is difficult. Very difficult. So unless your clever idea to protect data has just been rigorously tested by the crypto community, you probably shouldn't use it. Use a standard library instead, don't roll your own. You're going to get it wrong. Even implementing standard libraries can be tricky, so be careful.

KISS

The phrase Keep It Simple Stupid applies to most areas of life, including security and IT in general. Basically, complexity is the enemy of good security. Complex systems are difficult to understand, manage and maintain. A unnecessarily complex system will degrade over time which generally means holes and weaknesses developing. Bugs creep in and remain hidden. People become afraid of it, unwilling to investigate and fix issues.

Test, test, test

Speaking of bugs. Test the security of code during development by means of code review and static analysis etc. Test your applications before they go into production, eg through audit, fuzzing and standard QA testing. Test your infrastructure once it's in production by red-teaming and using external audits / pen testing.

Temporary is rarely

This might not sound like security advice, but it is. The classic being the "any/any" firewall rule. Probably got put there during development or testing. It fixed the problem, and probably has a comment like "TODO - Work around to get customer working before deadline. Must investigate and fix properly - A N Other, March 2005". Although unavoidable, at least don't pretend that you'll get around to fixing it later. Just accept that it is going to be yet another permanent hack, and perhaps invest another 10 minutes to make it suck slightly less.

Weakest Link

Aside from all that dodgy hacked code and long forgotten cronjobs, your systems are in pretty good shape. However, security is like a chain. It is only as strong as the weakest link. Perhaps your database server is uber-hardened. But if you've got a five year old cronjob running somewhere on an unpatched six year old server that connects to your database will full admin privileges, that hardening is meaningless.

Principle of least privilege

So, why would that cronjob that simply dumps data need to run as an admin? It was probably created when the database only had one user - the admin. Applying the principle of least privilege means that people and systems should only be authorised to do what is required for them to successfully do their jobs. Another example is developers having root access on production systems. Is that really necessary? Or even sysadmins having full access to the development source code repositories. Following this principle helps guard against accidental and malicious abuse of power.

You cannot create data retrospectively

But you can throw it away. This is an operational point that extends beyond security. You can do some very interesting and insightful things with operational metrics, giving you new visibility into your system and therefore greater understanding. From a security perspective, an IDS alarming might coincide with a jump in open file descriptors on a system, or a sudden jump in memory used. Perhaps the number of lines per minute in an application log file jumps or drops. By collecting, graphing and reviewing this data you could lower the time it takes to understand an attack. There is more to life than just network traffic. Not using certain data? Archive or delete it after 12 months. Simple.

Secure at the source

As a general principle, you want to secure as soon as possible. If data is being generated on a system, you wouldn't wait until that data has been transferred half way around the world before attempting to secure it. This can go as far as never letting unencrypted sensitive data even reach a harddisk.

Security is a user problem

On the one hand you have security, on the other you have usability. No matter how clever a security technology is, if it needs to be driven by a human then it will need to be usable. When designing a system, try to work security into it from the start so that it doesn't become a bolt-on that gets in the way of usability.

Security is everybody's responsibility

Another classic principle, security is the responsibility of everyone, from the receptionist to the CISO. Why? Well, weakest link for a start. The receptionist might not have root access on the production servers, but they are probably on the network and could be a pivot in further attacks. They would probably also make good sources of spear-phising emails to internal addresses. Educate people to take an interest in doing their best to maintain security. Reward them for reporting suspicious activity (e.g. a social engineering phone call).

Remote code execution == game over

Basically, if a box has been compromised to the extent that the perpetrator has run code on it, you have to assume you have lost control of that system. Rootkits are becoming more sophisticated by the month and you probably do not have the time to fully investigate exactly what happened to the system. Investigate it, understand what they succeeded at, what they failed at and perhaps even what their motivation was. Then rebuild the box and start from scratch. Also, don't rely on backups taken after a compromise.

You're going to get hacked

Almost finally. It's not a matter of if, but when. Work on the assumption that you are going to get hacked and look at how it's going to happen and how you can minimize the impact on the business. Make sure you have an incident response process.The idea is to stop an attack as soon as possible, reducing the impact whilst recording data for later forensic analysis (a hardened centralised syslog server might help).

Finally

This list isn't complete. I will add to it over time.

London Sky - Flickr pictures

2011-05-30T20:54:00.001+01:00

A slight tangent from the usual rare posts, but I've just uploaded some London skyline pics to Flickr.

The rest can be found here: Flickr

M4sl0w's hierarchy of needs (for geeks)

2011-05-01T12:41:00.003+01:00

What is Maslow's hierarchy of needs?

I suggest you look here or Google for it, but in summary it is a psychological theory to describe the levels of human needs and desires and how higher levels of needs can only be satisfied once all the levels below are satisfied.

Does this apply to geeks?

Well, even geeks have basic needs that must be met. I'll discuss them in turn below.

Caffeine

All geeks need caffeine in some form. There are a few crazy exceptions, but they're by far the minority. Popular sources of caffeine are coffee and soft drinks. Energy drinks are of course an excellent source of this essential nutrient and have helped many a geek through a late night coding frenzy.

Awesome workstation and shiny gadgets

So, you're getting your regular dose of free caffeine. This will no doubt be the first thing you do when you get to the office, if you haven't already picked a drink up on the way in. You sit down at you're desk and you're greeted with what you're going to be staring at for the next eight hours. Do you look at it and have flashbacks to the eighties or do you admire the technical marvel before you? As they say, "if you need something done, you may as well do it on something cool and expensive".

Fun and interesting colleagues

As a geek you probably dread social interaction. Or perhaps not. I think that cliché is somewhat dated. But you probably have to work with people around you. Perhaps even regularly. The goal here is to have fun at the same time, building up a sense of team and therefore a shared sense of purpose. That way you are more productive and get things done sooner leaving you with more time to sip your caffeine drink of choice. If you're lucky, you might even meet your colleagues after work in some form of social event.

Interesting work, a sense of control and doing things right.

Do it. Do it now. Do it right now. Do it right.

So, you're sipping your regular dose of caffeine, sitting back at your shiny workstation after a interesting and thought provoking conversation with a colleague. It is now time to get on with the day and do some work.

I think most people, and especially geeks, have a basic desire work hard, work efficiently and to do the right thing. As a geek, you are being paid not just to mash the keyboard from time to time, but to use your expertise to assess and prioritize what work needs to be done and how. A sense of control is very important here - if you and your colleagues know what needs to be done, know how it needs to be done, know that it can be done and that it is the right thing to do, but cannot do it for political reasons, then you're probably going to be somewhat demotivated (wow, that last 'sentence' was an abuse of the English language, but you get my point I hope).

It is the role of a good manager to align the goals and opinions of the geek to the business requirements.

And, if you're lucky, your work might even be particularly interesting. Everything is interesting if you think about it the right way :) What was the other one? There are no boring tasks, only boring people.

Play time and training

Now that you're being productive and the business is happy with your and your team's work, what else is there?

Geeks do what they do because they love it, not because it is a career path (by definition, otherwise they're just random people who happen to work in IT etc). It is a geek's natural instinct to experiment, play, create and break things. And to learn during the process - understand how things work, what makes things tick and where the boundaries lie.

So, what is the ultimate geek motivation? To play, experiment, test, break, fix, create and get paid for it. That is why Google has their 20% time. This means that you get to spend 20% of your time doing whatever you like, which can sometimes result in the birth of a new product or service. It gives geeks a chance to think "outside of the box" (Bingo!) and to try or learn new things.

Why bother? Isn't it a waste of time? Well, perhaps. But there is an asymmetry in the potential gains. Basically, there is a fixed loss - 20% of an employee's time. However, the potential gain has no boundaries. Perhaps someone comes up with something truly phenomenal which ends up being worth millions to the business. Or perhaps they create something useful that gets release to the community and makes the world a slightly better place.

And who knows, the company might even lower its staff turnover and save some money on recruitment.

Finally, a word on training. There are two types of training found during employment. Basic training required to do the job effectively and personal training which indirectly benefits the company but adds to the employee's sense of purpose and career progression. Both forms of training are motivating, with the latter being more so. Unfortunately, training is rarely a priority for companies and doesn't seem to exist much in the wild. Lucky for them geeks tend to take responsibility for their own training, but this does generally result in higher staff turnover - if someone has dedicated most of their free time and spare cash into training themselves, why wouldn't they move on to reap the rewards of a better income or more interesting work?

10 useful little bash tricks

2011-02-27T18:44:00.005+00:00

We're going to have a quick look at a collection of ten little bash tricks that you may or may not know. Or possibly forget. Or perhaps know, but don't tend to use often and so this might be a good reminder.

Please note, this stuff was tested on Bash 4.1.5. You can find further details and examples for all these tricks in the Bash manpage.

1::Using previous command

This trick is very simple. If you run a command and then have to repeat it (for example using sudo)
then instead of typing the whole command again you can refer to it using !!.

user@host:~$ aptitude install blah
[100%] Reading package lists^C
user@host:~$ sudo !!
sudo aptitude install blah
[sudo] password for user:
...
user@host:~$

As you can see, it even prints out the new full command.

2::Using previous argument(s)

Similar to the first trick, bash will allow you to easily reference the last argument of the previous command.

In this first example we're going to touch a few files, then decide to remove the last one.

user@host:~$ touch a b c
user@host:~$ rm !$
rm c
user@host:~$ ls
a  b
user@host:~$

As you can see, !$ only expanded to c. If you only had one argument, it would exand to that one argument.

We can also reference all the previous arguments using !*

user@host:~$ touch a b c
user@host:~$ rm !*
rm a b c
user@host:~$ ls
user@host:~$

3::Process Substitution

Right, now its time to move on to something a little different.

We want to compare the files in two directories to see which are in common, missing or changed etc. It would make sense to use the diff utility to help. We are also going to use md5sum to get a fingerprint of the files so we can see if they have changed. All we have to do is look through each directory (using find), do an md5sum of each file and compare the two lists.

Now, diff usually expects to operate on two files. We could redirect our results into files and then diff those files, but there is a smarter way. By using the <(list) for we can run our finds and diff will treat the results files.

diff <(cd boot-a; find . -exec md5sum {} \;) <(cd boot-b; find . -exec md5sum {} \;)

< 9c830b456ed37e0c0d63f2528fb43de5  ./initrd.img-2.6.35-23-generic
7,8d5
< f70fd0262a6f8e1e82028237e94657fc  ./config-2.6.32-25-generic
< c16c8e1705a54db8b96adce7de17710a  ./vmcoreinfo-2.6.32-25-generic
10d6
< b1f002028905e42f594c83603606ca0b  ./config-2.6.35-23-generic
12,16d7
< 668d54c704f22d85cb548f27a7e19d41  ./config-2.6.35-22-generic
< 882721477fcf705c02b60a4bb219b0c8  ./vmlinuz-2.6.32-25-generic
< 88472c27c3d18a832b5d85aea1edc541  ./abi-2.6.35-22-generic
< 496b4aed0005b07574e9dd3895089b23  ./abi-2.6.35-23-generic
< 6ee40238bdabb40a862c6f63262efa9d  ./System.map-2.6.32-25-generic
18d8
< 4e02de05c66cb0b322dc5c71f43882e9  ./System.map-2.6.35-23-generic
84c74
< ac9641014f0b460fc6eb5d2936bd869c  ./grub/menu.lst
---
> c5cce31e9e9eb884685137aac1ab8a8a  ./grub/menu.lst

As you can see, boot-a had quite a few files that boot-b was missing, and grub/menu.lst was different between the two directories.

4::Here document and redirect

Sometimes you need to put together a little config file, or perhaps a test HTML page. It is possible to do it quickly without having to resort to opening some sort of editor such as vi.

user@host:~$ cat << EOF > test.conf
> option1=blah
> option2=something else
> debug=true
> # end of config
> EOF
user@host:~$ cat test.conf
option1=blah
option2=something else
debug=true
# end of config
user@host:~$

What we did is specified a here document using << EOF. This keeps reading from the terminal until it encounters the string EOF. That is then redirected into the file, in this case test.conf.

5::List expansion when copying

This a very simple but extremely useful little trick that I'm sure you already know, but because of how often it is useful, I thought I'd put it here anyway.

Let us say you have a file you want to quickly backup. Instead of doing something like cp file file.backup we can use list expansion to shorten the command:

user@host:~$ ls
special.conf
user@host:~$ cp special.conf{,.backup}
user@host:~$ ls
special.conf  special.conf.backup
user@host:~$

6::Nested list expansion

Sticking with list expansion, it is worth remembering that you can have nested lists:

user@host:~$ touch {a{1..5},b{1,2,4},c{5..9}}
user@host:~$ ls
a1  a2  a3  a4  a5  b1  b2  b4  c5  c6  c7  c8  c9

7::Quick maths

Need to do a quick calculation? Already have a terminal open? Don't bother firing up a calculator, just do the calculation in bash:

user@host:~$ echo $((2**32))
4294967296
user@host:~$

See the Bash manpage for the supported operators.

8::Tilda expansion

Bash allows you to quickly reference the current $PWD and old one ($OLDPWD). This could be useful if you forgot to use pushd/popd:

user@host:~/a$ cd ../b
user@host:~/b$ ls ~+
inside_b
user@host:~/b$ ls ~-
inside_a
user@host:~/b$ echo $OLDPWD
/home/user/a
user@host:~/b$

So, ~+ expands to $PWD and ~- expands to $OLDPWD.

9::Setting default values

When you have a bash script that takes command line arguments you might want to set default values if no arguments are supplied by the user.

In the bash script below, we default arg1 to "a" if argument 1 is not supplied. We default arg2 to "b" and arg3 to "c". Of course, the order of the arguments matters.

#!/bin/bash

arg1=${1:-"a"}
arg2=${2:-"b"}
arg3=${3:-"c"}

echo $arg1 $arg2 $arg3

When we run the script without arguments, it uses the defaults. But when we give it arguments, it uses those values.

user@host:~/test$ bash test.sh
a b c
user@host:~/test$ bash test.sh x y z
x y z
user@host:~/test$

10::TCP and UDP

Finally, bash allows you to send TCP and UDP traffic directly using a special device of the format /dev/tcp/host/port (use udp instead of tcp if necessary).

Firstly, we set file descriptor 3 to use that device. We can do that with exec, as it suggests in the manual:

"Note that the exec builtin command can make redirections take effect in the current shell."

user@host:~$ exec 3<>/dev/tcp/localhost/80
user@host:~$ ls -la /proc/$$/fd/
total 0
dr-x------ 2 user user  0 2011-02-27 16:34 .
dr-xr-xr-x 7 user user  0 2011-02-27 16:34 ..
lr-x------ 1 user user 64 2011-02-27 16:34 0 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 1 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 2 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 255 -> /dev/pts/8
lrwx------ 1 user user 64 2011-02-27 16:34 3 -> socket:[156914]
user@host:~$

We quickly use $$ to see the open file descriptors for our process. As you can see, FD 3 has been opened for a socket connection, in this case a TCP connection to localhost on port 80.

We can now use redirect echo into FD 3. Note that we use the -e option so that bash interprets the escaped characters giving us new lines needed by the HTTP protocol. We then redirect into cat to read the output.

user@host:~$ echo -e "GET / HTTP/1.0\n\n" >&3
user@host:~$ cat <&3
HTTP/1.1 200 OK
Date: Sun, 27 Feb 2011 16:34:26 GMT
Server: Apache/2.2.16 (Ubuntu)
Last-Modified: Sun, 27 Feb 2011 16:25:27 GMT
ETag: "901-e-49d4602cddfad"
Accept-Ranges: bytes
Content-Length: 14
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

<h1>Hai!</h1>
user@host:~$

Cool huh? Not a telnet, netcat or wget in sight...

A simple Monte Carlo simulation with R

2011-02-05T00:30:00.018+00:00

I recently read How to Measure Anything: Finding the Value of Intangibles in Business by Douglas W. Hubbard. It's a fascinating and informative read on the problems and solutions of measuring "soft" variables typically found in business. Fluffy variables like productivity, quality, risk can be measured if you use the right techniques and work within the limitations of measurement and statistics.

There is an excellent example of using a Monte Carlo simulation (or method) to calculate the risk of leasing a new machine in a manufacturing process. You can find the example on pages 82 through to 86.

Given my hatred of spreadsheets and having recently started playing with R, I thought I would have a go at replicating the simulation using R.

This is what I wrote. Please note I'm still an R n00b so some things can be done better no doubt.

######################### Variables #######################

# Firstly set this to TRUE if we want to save our plot as a 
# PNG and if so, what file and dimensions
bDoPNG <- FALSE
sFile <- "htma.png"
iWidth <- 1024
iHeight <- 768

# The following values represent our 90% confidence interval 
# (CI) ranges for the various inputs to our simulation.

# We are 90% confident that the maintenance savins per unit 
# is between $10 and $20
vMaintenanceSavingsPerUnit <- c(10,20)

# We are 90% confident that the labour savings per unit 
# is between $-2 and $8
vLabourSavingsPerUnit <- c(-2,8)

# We are 90% confident that the raw material savings per unit 
# is between $3 and $9
vRawMaterialsSavingsPerUnit <- c(3,9)

# We are 90% confident that the production level per year 
# will be between 15K and 35K units
vProductionLevelPerYear <- c(15000,35000)

# The annual lease is $400K so we need to save this amount 
# just to break even for the investment
iAnnualLease <- 400000

# This is a quick cheat which basically means there are 
# 3.29 standard deviations in a 90% confidence interval
iStdDevCheat <- 3.29

# This is the number of simulations we are going to run
iNumberOfSims <- 100000

##################### Generate the basic data ###################

# A new data frame initiated to have iNumberOfSims rows in it
dData <- data.frame(seq(1,iNumberOfSims))

# We use the rnorm function to generate a distribution across 
# all the simulations for the maintenance savings. The mean is 
# literally just the mean of the range (e.g. (20-10)/2) and we 
# also give it the standard deviation of (20-10)/3.29.
dData$MainSavings <- rnorm(iNumberOfSims, 
mean(vMaintenanceSavingsPerUnit), 
diff(vMaintenanceSavingsPerUnit,1,1)/iStdDevCheat)

# Same again for the labour savings
dData$LabourSavings <- rnorm(iNumberOfSims, 
mean(vLabourSavingsPerUnit), 
diff(vLabourSavingsPerUnit,1,1)/iStdDevCheat)

# And the raw material savings
dData$RawMaterialsSavings <- rnorm(iNumberOfSims, 
mean(vRawMaterialsSavingsPerUnit), 
diff(vRawMaterialsSavingsPerUnit,1,1)/iStdDevCheat)

# And finally the production levels
dData$ProdLevel <- rnorm(iNumberOfSims, 
mean(vProductionLevelPerYear), 
diff(vProductionLevelPerYear,1,1)/iStdDevCheat)

# We can now create our total savings column based on the 
# inputs given. Because R is a vector language, the below 
# operation is applied to each row automatically.
dData$TotalSavings <- (dData$MainSavings + dData$LabourSavings +
dData$RawMaterialsSavings) * dData$ProdLevel

# Later on it will look better on the graphs if we deal
# with numbers in thousands so create a couple of shortcut variables
dData$TotalSavingsThousands <- dData$TotalSavings/1000
iAnnualLeaseThousands <- iAnnualLease/1000

# We now let R generate a histogram of our savings but without 
# actually plotting the results. We will end up with a series of 
# buckets (aka breaks) which will go on the X axis and the number 
# of simulations that fell within each bucket (on the Y axis)
hHist <- hist(dData$TotalSavingsThousands,plot=FALSE)

# We create a new data frame for the breaks and counts 
# excluding the last break
dHistData <- data.frame(
breaks=hHist$breaks[1:length(hHist$breaks)-1], 
count=hHist$counts)

# We can calculate the chance of the project making a loss as 
# the sum of counts where the breaks were less than 
# the annual lease (ie. $400K).
fPercentChanceOfLoss <- 100*sum(subset(dHistData,
breaks<iAnnualLeaseThousands,
select=count))/sum(dHistData$count)

# Calculate the median of the savings. That is 50% of
# the simulations had savings less than the median 
# and 50% had savings of more than the median.
fMedian <- median(dData$TotalSavingsThousands)

# We put that chance of loss in a sub title
sSubTitle <- sprintf("%02.2f%% chance of loss at $400K expense, 
median savings at $%02.0fK", fPercentChanceOfLoss, fMedian)

# Check whether we want to save our PNG
bDoPNG && is.null(png(sFile, width=iWidth, height=iHeight))

# Now draw the actual histogram, setting some labels but without 
# drawing the axis
hist(dData$TotalSavingsThousands,col="lightblue", main="Histogram of Savings",
xlab="Savings per Year ($000s in 100,000 increments)",
ylab="Senarios in Increment", axes=F)

# Add the sub title
mtext(sSubTitle)

# Draw the Y axis using default parameters
axis(2)

# Now draw the X axis explicitly setting values of the ticks/breaks
axis(1, at=hHist$breaks)

# That's it, turn off output if saving PNG
bDoPNG && is.null(dev.off())

And this is the pretty graph it produced. It should look similar to the one on page 86.

So yeah, go buy the book. Read it. Then have fun with R :)

Random spam

2010-07-19T14:58:00.003+01:00

Literally... This one got through gmail's spam filters recently... URL in it is

"hxxp://www.oerstedfiltrate.info"

which seems to just be spamming about working from home. Had a quick look through the JavaScript, didn't see anything too scary. Interesting approach though.


Delivered-To: MYEMAILADDRESS
Received: by 10.204.52.201 with SMTP id j9cs78472bkg;
        Mon, 19 Jul 2010 03:02:26 -0700 (PDT)
Received: by 10.42.1.140 with SMTP id 12mr1676540icg.27.1279533745005;
        Mon, 19 Jul 2010 03:02:25 -0700 (PDT)
Return-Path: 
Received: from col0-omc2-s16.col0.hotmail.com (col0-omc2-s16.col0.hotmail.com [65.55.34.90])
        by mx.google.com with ESMTP id 1si10103284ibz.26.2010.07.19.03.02.24;
        Mon, 19 Jul 2010 03:02:25 -0700 (PDT)
Received-SPF: pass (google.com: domain of roger_to@hotmail.com designates 65.55.34.90 as permitted sender) client-ip=65.55.34.90;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of roger_to@hotmail.com designates 65.55.34.90 as permitted sender) smtp.mail=roger_to@hotmail.com
Received: from COL118-W13 ([65.55.34.71]) by col0-omc2-s16.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
  Mon, 19 Jul 2010 03:01:42 -0700
Message-ID: 
Return-Path: roger_to@hotmail.com
Content-Type: multipart/alternative;
 boundary="_075a8098-55f9-4a99-8412-f3c29427bff7_"
X-Originating-IP: [61.247.52.224]
From: ROGER JOEL TORIBIO 
To: 
Subject: Re: leg
Date: Mon, 19 Jul 2010 10:01:42 +0000
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 19 Jul 2010 10:01:42.0497 (UTC) FILETIME=[63661D10:01CB2729]

--_075a8098-55f9-4a99-8412-f3c29427bff7_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable


~]##=3D=3B_ma__ke=2C%$$~_[mo__ney{._@~=3D=3Bfas__t--]=3B~=3B$interne__t]~@=
=3D?~$

mghmqnc7 lqij0l44 c62u2h47oi9 c8x30ng s212x4cu y9mja8bfhh0 a5dy7n2mv4 5fycq=
2eeh6p sqz7rt eipy94g
nz30d86cgvx 6m7p5v7hd0 98v3nyk 5ohjr jf3mia7km42b y71nk f6owy5dzh w7s3h2 5l=
xnlp3g3m
tu36zwq9k 5g4fcglnu ypbrhwgnm9e ie3s2w yywbw 8hvfk76thv3 zj0c9sifp csft04h =
7j0b6fuh647m egw3vcw60w3
f537u3 o7pf805s cxtnm339vm f7or2a183r8 9c4q70rfp 5sisqgdr4ib demuk81dnob yn=
lhlw47 p22fyvd
c5f1uyii 1y6gs3t13w zbawrjdbco79 ju86sw i1mty2678n0 k8ac2xjr0 cecdlbounyjk =
pp3o 0jxerx 3p0552fepkq
6uyg0x2v2xdr l929yeslo8 4csmmc2 dbu5o ntcjv rom7mlpdm0xc cask yl30c oq8n26
dxdld8iw2fl asbgaqse280 r8hi y5y3e9 zik23m ned6 2a3jp7lgntcx sh3x2e0t
           =20
_________________________________________________________________
Hotmail: Trusted email with Microsoft=92s powerful SPAM protection.
https://signup.live.com/signup.aspx?id=3D60969=

--_075a8098-55f9-4a99-8412-f3c29427bff7_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<style></style>
</head>
<body class=3D'hmmessage'>
~]##=3D=3B_ma__ke=2C%$$~_[mo__n=
ey{._@~=3D=3Bfas__t--]=3B~=3B$interne__t]~@=3D?~$

mghmqnc7 lqij0=
l44 c62u2h47oi9 c8x30ng s212x4cu y9mja8bfhh0 a5dy7n2mv4 5fycq2eeh6p sqz7rt =
eipy94g
nz30d86cgvx 6m7p5v7hd0 98v3nyk 5ohjr jf3mia7km42b y71nk f6owy5dz=
h w7s3h2 5lxnlp3g3m
tu36zwq9k 5g4fcglnu ypbrhwgnm9e ie3s2w yywbw 8hvfk76=
thv3 zj0c9sifp csft04h 7j0b6fuh647m egw3vcw60w3
f537u3 o7pf805s cxtnm339=
vm f7or2a183r8 9c4q70rfp 5sisqgdr4ib demuk81dnob ynlhlw47 p22fyvd
c5f1uy=
ii 1y6gs3t13w zbawrjdbco79 ju86sw i1mty2678n0 k8ac2xjr0 cecdlbounyjk pp3o 0=
jxerx 3p0552fepkq
6uyg0x2v2xdr l929yeslo8 4csmmc2 dbu5o ntcjv rom7mlpdm0=
xc cask yl30c oq8n26
dxdld8iw2fl asbgaqse280 r8hi y5y3e9 zik23m ned6 2a3=
jp7lgntcx sh3x2e0t
            
Hotmail: Trusted email with Microsoft=92s powerful =
SPAM protection.  target=3D'_new'>Sign up now.</body>
</html>=

--_075a8098-55f9-4a99-8412-f3c29427bff7_--

Happy birthday! Spam!!!!?

2010-06-30T23:23:00.004+01:00

So, it was my birthday fairly recently, thank you, and while checking through my Google Mail spam folder I noticed several emails wishing me a "Happy Birthday" but which seemed to come from sites/organizations/addresses etc that I didn't recognize.

This got me thinking.

Is there any spam floating around which tries to use birthdays as a social engineering vector? Obviously you have a one in 365.25 chance of that person having a birthday on the day they receive the spam email (plus or minus a few days and assuming of course uniform distribution (which isn't the case I'm sure, but I think spammers could probably time things quite well) and that the spam makes it through whatever filters). Even with those odds working against the spammer, is the potential increased click-through/purchase etc worth it? In fact, if this Birthday distribution is anything to go by then I should imagine that September offers better than 1/365 odds.

Has anyone seen this sort of "vector", does it exist?

Just a thought.

kthnxbye

BTW, I think those emails I saw were things I had signed up for with my +spam@gmail.com address.
If so, hopefully I don't have to worry about it for another year or so...

Different crc16 and the problem with hashes

2010-06-12T23:38:00.005+01:00

OK, so when I said "pretty slow", I think I actually meant that it is stupidly slow.

I'll get to exactly why in a second, but here is a slightly faster crc16 function (I "ported" it from some c code I found whilst googling, but unfortunately I cannot credit the original source as I cannot find it):


crc_table=(  0x0000 0x1189 0x2312 0x329b 0x4624 0x57ad 0x6536 0x74bf \
  0x8c48 0x9dc1 0xaf5a 0xbed3 0xca6c 0xdbe5 0xe97e 0xf8f7 \
  0x1081 0x0108 0x3393 0x221a 0x56a5 0x472c 0x75b7 0x643e \
  0x9cc9 0x8d40 0xbfdb 0xae52 0xdaed 0xcb64 0xf9ff 0xe876 \
  0x2102 0x308b 0x0210 0x1399 0x6726 0x76af 0x4434 0x55bd \
  0xad4a 0xbcc3 0x8e58 0x9fd1 0xeb6e 0xfae7 0xc87c 0xd9f5 \
  0x3183 0x200a 0x1291 0x0318 0x77a7 0x662e 0x54b5 0x453c \
  0xbdcb 0xac42 0x9ed9 0x8f50 0xfbef 0xea66 0xd8fd 0xc974 \
  0x4204 0x538d 0x6116 0x709f 0x0420 0x15a9 0x2732 0x36bb \
  0xce4c 0xdfc5 0xed5e 0xfcd7 0x8868 0x99e1 0xab7a 0xbaf3 \
  0x5285 0x430c 0x7197 0x601e 0x14a1 0x0528 0x37b3 0x263a \
  0xdecd 0xcf44 0xfddf 0xec56 0x98e9 0x8960 0xbbfb 0xaa72 \
  0x6306 0x728f 0x4014 0x519d 0x2522 0x34ab 0x0630 0x17b9 \
  0xef4e 0xfec7 0xcc5c 0xddd5 0xa96a 0xb8e3 0x8a78 0x9bf1 \
  0x7387 0x620e 0x5095 0x411c 0x35a3 0x242a 0x16b1 0x0738 \
  0xffcf 0xee46 0xdcdd 0xcd54 0xb9eb 0xa862 0x9af9 0x8b70 \
  0x8408 0x9581 0xa71a 0xb693 0xc22c 0xd3a5 0xe13e 0xf0b7 \
  0x0840 0x19c9 0x2b52 0x3adb 0x4e64 0x5fed 0x6d76 0x7cff \
  0x9489 0x8500 0xb79b 0xa612 0xd2ad 0xc324 0xf1bf 0xe036 \
  0x18c1 0x0948 0x3bd3 0x2a5a 0x5ee5 0x4f6c 0x7df7 0x6c7e \
  0xa50a 0xb483 0x8618 0x9791 0xe32e 0xf2a7 0xc03c 0xd1b5 \
  0x2942 0x38cb 0x0a50 0x1bd9 0x6f66 0x7eef 0x4c74 0x5dfd \
  0xb58b 0xa402 0x9699 0x8710 0xf3af 0xe226 0xd0bd 0xc134 \
  0x39c3 0x284a 0x1ad1 0x0b58 0x7fe7 0x6e6e 0x5cf5 0x4d7c \
  0xc60c 0xd785 0xe51e 0xf497 0x8028 0x91a1 0xa33a 0xb2b3 \
  0x4a44 0x5bcd 0x6956 0x78df 0x0c60 0x1de9 0x2f72 0x3efb \
  0xd68d 0xc704 0xf59f 0xe416 0x90a9 0x8120 0xb3bb 0xa232 \
  0x5ac5 0x4b4c 0x79d7 0x685e 0x1ce1 0x0d68 0x3ff3 0x2e7a \
  0xe70e 0xf687 0xc41c 0xd595 0xa12a 0xb0a3 0x8238 0x93b1 \
  0x6b46 0x7acf 0x4854 0x59dd 0x2d62 0x3ceb 0x0e70 0x1ff9 \
  0xf78f 0xe606 0xd49d 0xc514 0xb1ab 0xa022 0x92b9 0x8330 \
  0x7bc7 0x6a4e 0x58d5 0x495c 0x3de3 0x2c6a 0x1ef1 0x0f78 )

crc16()
{
    crc=0xFFFF
    for char in $(echo "$1" | fold -w 1); do
            byte=$(printf "%d" \'$char)
            (( crc= crc >> 8  ^ ${crc_table[($crc ^ $byte) & 0x00FF]} ))
    done
    echo $crc
}

In the above code the crc_table array would be a global variable.

Unfortunately, even though that crc16 function is somewhat faster, a BPA script of say 10 plugins each with 10 commands would still take just under 5 seconds to run. After thinking about it for five seconds, the reason is that that for each command (there are 10 x 10 = 100 of them) it needs to generate a crc16 about 5 times to store various bits of information. Then, once all the plugin data is loaded, there are relatively few actual reads of the hash which means that the script spends much much longer generating crc16 when inserting data, most of which is never going to be read and even if it is read it is only read a couple of times.

This is an extremely inefficient use of hashes, especially when written in something slow like bash.

So, what are the options?

Redesign the BPA.sh script to not load all the plugins (which sort of defeats the purpose)
Not use hashes in any form
Use other fake hash made from running two arrays in parallel (one for keys and one for values)
Use built-in associative array functionality of Bash 4.0 onwards

I think my cunning plan is to use a combination of the last two options. If the version of Bash is 4.0 or greater, then we can use the built-in functionality. If not, we degrade to the two array fake hashes.

I will be updating BPA.sh in the coming weeks then I guess...

Fun with bash: crc16, hashes and plugins

2010-05-22T16:14:00.008+01:00

Today I released simple plugin architecture for bash scripts.

To do what it does, it uses three interesting features of bash:

The source command to read in the plugins

Bash arrays turned into hashes using a crc16 function written in bash

Bash functions within functions to limit the scope of plugin commands

This post describes those three interesting features.

Bash Plugin Architecture

The project page is at: http://code.google.com/p/b-p-a/

It allows you to create plugins like:


user@host:~/bpa$ cat plugins/security/nmap.shp 
#!/bin/bash

registerPlugin "NMap" "nmap" "The mighty portscanner"

registerCommand 'SynScan' 'syn' 'TCP Syn scan of hosts'
registerCommand 'Ping' 'ping' 'Ping scan of hosts'

# Name of the plugin
NMap()
{
        # Name of the command.
        # Each command goes inside the Plugin() function
        SynScan()
        {
                # Code for this command
                hosts=$@
                nmap -sS -P0 $hosts
        }

        Ping()
        {
                hosts=$@
                nmap -v -sP $hosts 
        }
}

which can be used like


$ sudo security.sh --nmap --syn -- 192.168.0.1

For this to work, security.sh just needs to source the BPA.sh script and run a few commands (see the project page for details).

Sourcing plugins

This feature is probably the simplest, and part of the code of BPA.sh is below:


#
# Registers a new application.
#
# This must be the first thing called inside the application script.
#
# @param name
#   Name of the application.
#
# @param debug
#   (optional) If set to '1' debugging is enabled for the application. Defaults
#   to '0'.
#
# @param plugins
#   (optional) Sets the plugins directory for the application. Defaults to
#   './plugins' inside the directory of the application script.
#
# @return
#   Does not return a value
#
# Example:
# @code
#   new 'MyApp' 1 '/var/lib/plugins/myapp'
# 
new() {
        sAppName="$1"
        bDebug=${2:-"0"}
        sAppPluginsDir=${3:-"./plugins"}

        doDebug=$bDebug

        debug "New application $sAppName"
        for plugin in $(ls ${sAppPluginsDir}/*.shp); do
                debug "Sourcing $plugin"
                source "$plugin"
        done
}

Basically, when the application script security.sh sources the main script BPA.sh and calls the new function, looks inside a plugins directory for all files ending in .shp (shell plugin). For each file it finds it calls the source command.

Now, when you source a plugin, you run it's registerPlugin function and all of it's registerCommand functions which in turn uses the hash functions as described next.

Bash hashes

The registerPlugin function looks something like:


#
# Registers a plugin for the application.
#
# This should the first thing a plugin (.shp) script does.
#
# @param name
#   The name of the plugin.
#
# @param switch
#   The command-line switch used to access the plugin, using --switch.
#
# @param description
#   A description of the plugin. This is displayed when the --help switch
#   is used.
#
# @return
#   Does not return a value
#
# Example
# @code
#   registerPlugin 'MyPlugin' 'my' 'An example plugin'
#
registerPlugin()
{
        sPluginName=$1
        sPluginSwitch=$2
        sPluginDescription=$3

        debug "Registering $sPluginName"

        hash_set $sPluginName name "$sPluginName"
        hash_set $sPluginName description "$sPluginDescription"

        hash_set hPlugins "$sPluginSwitch" "$sPluginName"
}

The interesting thing in that function is the use of hash_set, which is:


#
# Creates hash with given name and sets key/value pair.
#
# @param name
#   Name of the hash.
#
# @param key
#   Hash key value.
#
# @param value
#   Hash value for key.
#
# @return
#   Does not return a value.
#
hash_set()
{
        name=$1
        key=$2
        value=$3
        lookup="${name}_keys"

        id=$(crc16 $key)
        eval $name\[\$id\]=\"\$value\"
        array_push $lookup $key
}

This function takes a name for the hash as its first argument, then a key followed by the value for that key. It then uses the crc16 function (shown below) to generate a numeric index for the actual bash array. It then uses the bash eval command to create a real array from the hash name that was supplied. Finally, it uses a simple array_push function to keep track of the keys in case we want to iterate over them.

Quite simple, and definitely overkill ;) It would be possible to just create two parallel arrays to track the key/value pairs, but this is more fun.

The crc16 function looks like this:


#
# Generates a crc16 value for the given string.
#
# @param string
#   String to be crc16'd.
#
# @return
#   Decimal crc16 value of string
#
crc16()
{
        # This could probably be more efficient...
        string=$1
        aString=($(echo $string | fold -w 1))
        cnt=${#aString[@]}
        crcPolyNom=0x8408
        crcPreset=0xFFFF
        crc=$crcPreset
        
        for ((x=0;x<$cnt;x++)); do
                char=${aString[$x]}
                byte=$(printf "%d" \'$char)
                for ((i=0;i<8;i++)); do
                        (( charLsb = $byte & 0x0001 ))
                        (( crcLsb = $crc & 0x0001 ))
                        (( doAdd = $charLsb ^ $crcLsb ))
        
                        (( byte = $byte >> 1 ))
                        (( crc = $crc >> 1))
        
                        if [ "$doAdd" -eq "1" ]; then
                                (( crc = $crc ^ $crcPolyNom ))
                        fi
                done
        done
        echo $crc
}

To get a value from the array, we just have to generate the crc16 for the key:


#
# Gets the value for key from the given hash.
#
# @param name
#   Name of the hash.
#
# @param key
#   The key value.
#
# @return
#   The value associated with the key
#
hash_get()
{
        name=$1
        key=$2

        id=$(crc16 $key)
        eval echo \$\{$name\[\$id\]\}
}

So yeah, this is pretty slow... but also pretty cool.

Functions within functions

You can declare a function within a function. For example:


NMap()
{
        # Name of the command.
        # Each command goes inside the Plugin() function
        SynScan()
        {
                # Code for this command
                hosts=$@
                nmap -sS -P0 $hosts
        }
}

In order to run the SynScan function, you must first run the NMap function. At first this might not seem very useful. But for a plugin system it is, as it allows multiple plugins to have the same commands, without any clashing.

BPA.sh calls the plugin function and command in the run function:


#
# Runs the actual application.
#
# @return
#   Does not return a value.
#
# Example:
# @code
#   run
#
run() {
        debug "Running $sAppName"
        parse $aAppArgs
        debug "Plugin: $sSelectedPlugin"
        debug "Command: $sSelectedCommand"
        $sSelectedPlugin
        $sSelectedCommand $aCmdArgs
}

So we could happily have another plugin, perhaps like:


OtherCoolPortScanner()
{
        # Name of the command.
        # Each command goes inside the Plugin() function
        SynScan()
        {
                # Code for this command
                hosts=$@
                othercoolportscanner -s $hosts
        }
}

...without worrying about mixups.

sp@m witt no spalling mistokes

2010-05-17T18:01:00.005+01:00

Just received the following email:


from Adobe 
reply-to newsletter@download-adobe-pdf.com
to me@gmail.com
date 17 May 2010 16:40
subject Download New Adobe PDF Reader For Windows
mailed-by mmmail.co.uk
 
hide details 16:40 (1 hour ago)
 

PDF Reader for Windows - Compatible with All Windows Platforms

Get the new and improved 2010 version.
The ultimate PDF software pack to open, read and share PDF files online.

Download New PDF reader here:

http://www.download-adobe-pdf.com/

New Features in PDF Reader:

- Search, open & share PDF files online
- 50% faster startup then previous versions
- Search & save online Internet content
- NEW! Improved user interface

Also receive the award winning OfficeSuite absolutely free.

New version available :

http://www.download-adobe-pdf.com/

Since the 90's, PDF has become the standard file format for document exchange. - Adobe

Thank you,

Adobe Reader Support



Click here to unsubscribe

Not only did this get through the usual spam filters Google so wonderfully provides, but it actually looks quite professional =/

A quick look at the whois output...


Domain name:             DOWNLOAD-ADOBE-PDF.COM
Name Server:             ns1.download-adobe-pdf.com 67.225.133.235
Name Server:             ns2.download-adobe-pdf.com 67.225.133.235
Creation Date:           2010.05.17

Status:                  DELEGATED

Registrant ID:           UALWS2Z-RU
Registrant Name:         Tommy Anderson
Registrant Organization: Tommy Anderson
Registrant Street1:      273 Parkway
Registrant City:         Miami
Registrant State:        FL
Registrant Postal Code:  33402
Registrant Country:      US

Administrative, Technical Contact
Contact ID:              UALWS2Z-RU
Contact Name:            Tommy Anderson
Contact Organization:    Tommy Anderson
Contact Street1:         273 Parkway
Contact City:            Miami
Contact State:           FL
Contact Postal Code:     33402
Contact Country:         US
Contact Phone:           +1 504 3442930
Contact E-mail:          tommy3849@hotmail.com

Registrar:               Regional Network Information Center, JSC dba RU-CENTER

Last updated on 2010.05.17 20:38:33 MSK/MSD

and suddenly it doesn't look so professional.

I haven't actually tried visiting the site yet, I don't have any systems I can happily pwn.

The unsubscribe link goes to:


http://www.mailingm.co.uk/3/unsubscribe.php?M=2723658&N=2435&=4&C=fbe0b28cf0f58710c5b4dea8a6e2ca68

A quick Google search:


Mailingm.co.uk  - Online Marketing Services Provider | Visit ...
This company specializes on marketing services through e-mail giving their clients the possibility of reaching a wider and international audience.
www.killerstartups.com › Site Reviews - Cached - Similar

In conclusion, this might just be some dodgy marketing for an Adobe reseller, but I don't think I'll take my chances.

A basic SSHD keylogger

2010-01-13T23:10:00.007+00:00


strace -p $(pgrep -n -u user sshd) 2>&1 | perl -ne '$_ =~ /^write\(\d+, "([^"]+)"\.\.\., 1\)/ && print time()." ".$1."\n";'

Speaks for itself really. Doesn't catch 'up' and 'down' keys, but does things like new line. CBA to investigate - it served its purpose.