Steve's Blog

Python streams

2017-02-23T10:49:00.000-05:00

I've become really comfortable with Java 8's streams over the past couple years. As a result, when I go back and do Python work, I get stream envy. I looked around and found a couple options, but none of them were really what I'm after... something quick and dirty that will just make my syntax a bit more readable.

So I threw this together and find it quite handy. I hope you find it helpful as well. If you save this to a module and name it 'streams.py', you can use it like the following:

from lazy_streams import stream

S = stream(range(250)) \
.filter(lambda x: (x%2 == 0))
print S.size()
print S.take(10).to_string()
print S.reverse().take(10).to_list()
S1 = S.map(lambda x: "Item %d" % x)
print S1.first_or_else('Nothing to see')
print stream(['Patty Cake', 'Jim Shoe', 'Justin Case']) \
.sort(lambda x: x.split(' ')[1]) \
.to_list()
print stream([[1, 2], 3, [4, 5, 6], 'seven']) \
.flatten() \
.to_list()

I've tested it on lists as big as 2.5 million items and the lazy evaluation seems to work really great!

Here's the gist:

Sharing a Python generator across multiple multiprocessing processes.

2017-02-17T16:06:00.001-05:00

Sometimes I need to do some work on a seemingly endless set of data. I often use generators to create the endless data. Say, for example I were trying to brute-force crack a zip file's password. I'd create a generator that methodically and continually creates new passwords for the cracking to to use in attempting to unzip the file.

Once I have my generator in place, I have a problem. What if I want to spread this brute-force attack across all the cores in my system to speed up this slow endless effort. One would think that multiprocessing will solve the problem.

Well, no. Generators aren't natively shared across processes in Python. They're copied to each process. This means that if you try to use the generator with multiprocessing, each process will get it's own copy of the generator and each process would get the same values from each copy.

To solve that, I devised a quick and dirty solution: Place the generator in it's own process and then have each worker process request the next value from it via inter-process communication using the multi-processing Pipe class.

To that end, here's an example. I post this here mainly to jog my memory next time I need to do this, but if you find it useful, great!

Non-freeze, single file distribution of your python project

2017-02-08T13:50:00.001-05:00

In my job, I often have to share/distribute Python code to others on my development team or to others in sibling development teams. These folks are technical and I can easily rely on them to have a reasonably contemporary version of Python already installed. Thus I don't really want to completely freeze my Python project (via cxFreeze, py2app, py2exe, etc.). Nor do I want to send my coworkers through the 'create a virtualenv, pip install -r, etc.).

I want a middle-ground. I want to just send them a file, knowing that it contains all the required dependencies and will just run, assuming they have Python in their path.

I've known for a while that if you zip up a directory full of Python files and so long as the root of the zip file contains a __main__.py file and you can directly run the zip file via python {filename.zip}.

What I didn't know is that you can concatenate a text file containing the Python shebang ('#!/usr/bin/env python') and a python zip file and the resulting file is runnable.

#!/usr/bin/env python

PK^C^D^T^@^B^@^H^@nLHJb±gL<94>^A^@^@Ò^B^@^@^K^@^\^@__main__.pyUT    ^@^C¯,<9b>Xö,<9b>Xux^K^@^A^D«<82><90>~^D^T^@^@^@u<92>ÑkÛ0^PÆßýWÜ²^GÙÐ$},^EÃÂ(<85>2Ú<90>ìq ^Tûâ<88>Z<92>9<9d>º<95>±ÿ}';ÎJ`Â^O<96>ôéû~ºÓ

ÈY©1^F<93>9£4@^Q&<81>+ü<8e>O^X<88>2#tCrÌ^N0©¸<83>ÛÑw°ÌÕÂ#®Ë¾^AaÀ^PÅ >«¼h@^P;c6<8d>"^Âý=^\AB!ej<91>  <9e>g¦ñ<95>Vgé<87>ÌqÏ^RC<8a>9<99>aq^O^O<9f>Cv^U^AþXQi<88>ÛÌ®°Üx<90>µó<98>^Gl­ ´Z^B<81>^E·¾&^B^C<83>ÕÚ§

,g^LÃ=kW^Y$r­â Æ]HÉ^UÝ5AÂ[V^FC$y<80>¢a<82>y^V^U{°(Ô<9b>|Ü^XW}¤h<8f>aØ}¦^G>o<96>êu<9c>e?k<8e>,àbºÔ/ýëh´¶^LÇp^A^Ye<8b>LÕBt<85>w£hi¿&(^HÛîä<96>tÿ.^@A>

dJuëX

I stumbled upon a tool called pex which will automate generating these shebang/zip files. While I found pex useful, it seems too much for my needs. So I started cobbling together my own shebang/zip files by hand as I needed them.

Ultimately, wanted to automate this some, so I ended up creating a simple bash script (which I named 'compile.sh') to speed things up. Here's that script. Hope you find this useful.

#!/bin/bash

#

# compile.sh

# Creates a self-runnable, single-file deployable for your python project.

# Your python project's main entry point must be in a file named __main__.py.

# It will embed in the deployable every dependency you've pip-installed in

# your virtualenv .env directory.

#

# The deployable still requires that python is installed on the target system.

if [[ ! -d .env/lib/python2.7/site-packages ]]; then

    echo "This script must be run from a virtualenv root."

    exit 1

fi

if [[ ! -f __main__.py ]]; then

    echo "This will only work if you have a __main__.py file."

    exit 1

fi

if [[ -z "$1" ]]; then

    echo "Usage: $(basename $0) {output_filename}"

    exit 1

fi

TMPDIR="$(mktemp -u /tmp/python.compile.XXXXXXX)"

CURDIR="$(pwd)"

TARGET="$1"

# Create payload zip with our code (remove extra junk)

mkdir -p "$TMPDIR/deps"

zip -9 "$TMPDIR/payload.zip" *

zip -d "$TMPDIR/payload.zip" compile.sh

zip -d "$TMPDIR/payload.zip" requirements.txt

zip -d "$TMPDIR/payload.zip" "$TARGET" > /dev/null

# Gather the virtualenv packages and clean them up

cp -R .env/lib/python2.7/site-packages/* "$TMPDIR/deps"

find "$TMPDIR/deps" -iname '*.pyc' -delete

find "$TMPDIR/deps" -ipath '*pip*' -delete

find "$TMPDIR/deps" -ipath '*easy_install*' -delete

find "$TMPDIR/deps" -ipath '*dist-info*' -delete

# Add the virtualenv packages to the payload

cd "$TMPDIR/deps"

zip -9 -r ../payload.zip *

cd "$CURDIR"

# Assemble the payload into a runable

echo '#!/usr/bin/env python' | cat - "$TMPDIR/payload.zip" > "$TARGET"

chmod +x "$TARGET"

# Cleanup

rm -r "$TMPDIR"

Python Decorators Examples

2015-10-07T11:33:00.001-04:00

I often go long stretches between times when I aggressively use python at clients. As a result, I get rusty on what I could consider somewhat advanced topics and have to spend time googling around to jog my memory. Reading my own code usually gets me back up and running quicker than reading others' code. Leaving this here as a breadcrumb to help me remember how to use python decorators.

If you find this helpful, awesome!

#!/usr/bin/env python
"""Decorator examples"""

def decorator_without_arguments(func):
def wrapped_function(*args, **kwargs):
"""Reverses the arguments before calling"""
new_args = reversed(args)
return func(*new_args, **kwargs)
wrapped_function.__name__ = "(%s -> %s)" % \
(decorator_without_arguments.__name__, func.__name__)
return wrapped_function

def decorator_with_arguments(arg1, arg2=None):
def actual_decorator(func):
def wrapped_function(*args, **kwargs):
"""Wraps results in explanation string"""
result = "(%s: %s) " % (arg1, ', '.join([str(x) for x in args]))
result += str(func(*args, **kwargs))
if arg2 is not None:
result += arg2
return result
wrapped_function.__name__ = func.__name__
return wrapped_function
return actual_decorator

def undecorated_subtract(number_1, number_2):
return number_1 - number_2

@decorator_without_arguments
def decorated_subtract(number_1, number_2):
return number_1 - number_2

def undecorated_add(number_1, number_2):
return number_1 + number_2

@decorator_with_arguments('ADD')
def decorated_add(number_1, number_2):
return number_1 + number_2

if __name__ == '__main__':
print undecorated_subtract(7, 3)
print decorated_subtract(7, 3)
print decorated_subtract
print undecorated_add(3, 7)
print decorated_add(3, 7)
print decorated_add

Embedding python in bash scripts

2014-07-23T13:17:00.003-04:00

As a software development consultant, I do a lot of bash scripting. I can do a lot of really creative things using nothing but bash and the binutils at my disposal, but sometimes I'll come across something that's just easier to do in a higher level scripting language. Enter python.

Conversely, there are a lot of things that are just easier and more straight forward to do in bash, so writing everything in straight python may be more work than it's worth.

Here's a quick posting to describe how you can embed some python code into your bash scripts and get the best of both worlds. Note: just as a heads, up, the examples in this posting are quite contrived.

Calling python from bash is easy. You simply use python's '-' argument and pipe in your python code. I typically wrap my python code in a bash function.

#!/bin/bash

function current_datetime {

python - <<END

import datetime

print datetime.datetime.now()

END

}

# Call it

current_datetime

# Call it and capture the output

DT=$(current_datetime)

echo Current date and time: $DT

You can also pass data into your embedded python script. I do that using environment variables:

#!/bin/bash

function line {

PYTHON_ARG="$1" python - <<END

import os

line_len = int(os.environ['PYTHON_ARG'])

print '-' * line_len

END

}

# Do it one way

line 80

echo 'Handy'

# Do it another way

echo $(line 80)

My usual use-case for doing this is if I'm extending someone else's bash scripts and have to 'go off the reservation' a bit. Sometimes I'm updating an existing 'legacy' script and need to look up some data... maybe do a REST call or something. Here's an example bash script that uses curl to call a REST service to get some weather data. Then is passes the raw JSON response to an embedded python script to interpret and format the results:

#!/bin/bash

function format_weather_data() {

PYTHON_ARG="$1" python - <<END

import os

import json

json_data = os.environ['PYTHON_ARG']

data =json.loads(json_data)

lookup = {

'200': 'thunderstorm with light rain',

'201': 'thunderstorm with rain',

'202': 'thunderstorm with heavy rain',

'210': 'light thunderstorm',

'211': 'thunderstorm',

'212': 'heavy thunderstorm',

'221': 'ragged thunderstorm',

'230': 'thunderstorm with light drizzle',

'231': 'thunderstorm with drizzle',

'232': 'thunderstorm with heavy drizzle',

'300': 'light intensity drizzle',

'301': 'drizzle',

'302': 'heavy intensity drizzle',

'310': 'light intensity drizzle rain',

'311': 'drizzle rain',

'312': 'heavy intensity drizzle rain',

'313': 'shower rain and drizzle',

'314': 'heavy shower rain and drizzle',

'321': 'shower drizzle',

'500': 'light rain',

'501': 'moderate rain',

'502': 'heavy intensity rain',

'503': 'very heavy rain',

'504': 'extreme rain',

'511': 'freezing rain',

'520': 'light intensity shower rain',

'521': 'shower rain',

'522': 'heavy intensity shower rain',

'531': 'ragged shower rain',

'600': 'light snow',

'601': 'snow',

'602': 'heavy snow',

'611': 'sleet',

'612': 'shower sleet',

'615': 'light rain and snow',

'616': 'rain and snow',

'620': 'light shower snow',

'621': 'shower snow',

'622': 'heavy shower snow',

'701': 'mist',

'711': 'smoke',

'721': 'haze',

'731': 'sand, dust whirls',

'741': 'fog',

'751': 'sand',

'761': 'dust',

'762': 'volcanic ash',

'771': 'squalls',

'781': 'tornado',

'800': 'clear sky',

'801': 'few clouds',

'802': 'scattered clouds',

'803': 'broken clouds',

'804': 'overcast clouds',

'900': 'tornado',

'901': 'tropical storm',

'902': 'hurricane',

'903': 'cold',

'904': 'hot',

'905': 'windy',

'906': 'hail',

'950': 'setting',

'951': 'calm',

'952': 'light breeze',

'953': 'gentle breeze',

'954': 'moderate breeze',

'955': 'fresh breeze',

'956': 'strong breeze',

'957': 'high wind, near gale',

'958': 'gale',

'959': 'severe gale',

'960': 'storm',

'961': 'violent storm',

'962': 'hurricane',

}

print "Current temperature: %g F" % data['main']['temp']

print "Today's high: %g F" % data['main']['temp_max']

print "Today's low: %g F" % data['main']['temp_min']

print "Wind speed: %g mi/hr" % data['wind']['speed']

weather_descs = [lookup.get(str(i['id']), '*error*') for i in data['weather']]

print "Weather: %s" % ', '.join(weather_descs)

END

}

WEATHER_URL="http://api.openweathermap.org/data/2.5/weather?q=Cincinnati,OH&units=imperial"

format_weather_data "$(curl -s $WEATHER_URL)"

Hope you find this information useful.

Formatting GUIDs with sed

2014-01-22T16:21:00.002-05:00

At my current client, I'm often given GUIDs for one reason or another. Often they are not in the format that I need them to be. Sometimes they have the dashes and I need them to be dash-free. Other times, the dashes have been removed and I need them back.

Here's some quick, down and dirty sed commands to swap them back and forth:

To remove the dashes:

$ echo 8EC60070-685F-41DB-C881-EACF9E74E4BD | sed 's/-//g'
8EC60070685F41DBC881EACF9E74E4BD
$

To put them back:

$ echo 8EC60070685F41DBC881EACF9E74E4BD | sed -rn 's/([0-9A-F]{8})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{12})/\1-\2-\3-\4-\5/p')
8EC60070-685F-41DB-C881-EACF9E74E4BD
$

It goes without saying you can embed this very sed command in a bash script and run it over many rows in a file.

Monitoring a web page for changes using bash

2013-03-14T14:26:00.000-04:00

There's this conference that I'd like to attend and I've heard that it's a hard-to-get-into type conference. When I go to their site it doesn't have any new info.

Rather than checking the site every day, I'd like to have it monitored and be alerted when something new DOES appear on it.

Now I know there are services like ChangeDetection.com that can monitor it for me, but I was wanting to cobble something together with the tools I already have. I'd also like to have the ability to customize what it consider "a change" at my disposal when/if I need it.

To that end, I threw together the following bash script. It monitors a URL and if it detects a change, it sends an email to my gmail account letting me know.

Hope you find it useful. BTW, I'm using a program called sendEmail to send the email notification. It's in apt if you're using a debian/ubuntu-like distribution.

#!/bin/bash

# monitor.sh - Monitors a web page for changes
# sends an email notification if the file change

USERNAME="me@gmail.com"

PASSWORD="itzasecret"

URL="http://thepage.com/that/I/want/to/monitor"

for (( ; ; )); do
mv new.html old.html 2> /dev/null
curl $URL -L --compressed -s > new.html
DIFF_OUTPUT="$(diff new.html old.html)"
if [ "0" != "${#DIFF_OUTPUT}" ]; then
sendEmail -f $USERNAME -s smtp.gmail.com:587 \
-xu $USERNAME -xp $PASSWORD -t $USERNAME \
-o tls=yes -u "Web page changed" \
-m "Visit it at $URL"
sleep 10
fi

done

Then from a bash prompt I run it with the following command:

nohup ./monitor.sh &

Using nohup and throwing it in the background allows me to log out and have the script continue to run.

Fun with Python's multiprocessing module

2013-02-04T22:38:00.000-05:00

At my current client, they asked me to write a Python script to aggregate data from a RESTful web service. Essentially, they have a web service end-point that takes a single customer ID as an argument and it returns some customer profile records as JSON.

They wanted my script to call the web service for each and every customer listed in a text file. The script will be run from a scheduled job (cron, windows scheduled task, etc.). Once productionalized, the text file will contain a large number of customer ID's (like a million or so). All the results need to be stored in a single text file for the run.

My first thought was to use Python's threading features, but I came with 2 problems with this:

Python's threading isn't "real" threading. A single Python process will run all threads on a single physical core regardless of how many cores the server has. The other cores will not be utilized.
Python's urllib and urllib2 packages aren't even thread safe.

Now I know there's a third-party thread-safe urllib3 package, but that doesn't solve the multi-core issue and I wasn't looking to install anything beyond Python 2.7's standard library on the server. After doing a little research, I came up with the idea of using Python's multiprocessing module.

The rest of this blog posting is a walkthrough of what I ended up building.

First things first - configuration

Since this is a script that will be run often and certain configuration settings will need to be updated from time to time by support staff, I decided to extract the configuration out to a separate configuration file. Initially, I had the settings stored in a .ini file, but I later thought why not use Python for settings. To that end, I have a settings.py file that holds a class that contains the settings:

#!/usr/bin/env python

"""

Settings file. See job.py for more info.

"""

# -----------------------------------------------------------

# Edit the values in this Settings class to control settings.

# -----------------------------------------------------------

class Settings:

"""Settings as a class. Why? Because I'm lazy."""

source_file = 'TestData.csv'

profile_data_file = 'results.txt'

log_file = 'log.txt'

failed_customer_file = 'failed.txt'

process_count = 10

url = 'http://division-1.internal-server.com/rest-service/endpoint/{0}'

There is one input file (source_file). This is the file that contains the list of customer ID's. Then there are 3 output files: profile_data_file, log_file and failed_customer_file. The profile_data_file file is where the script will put the customer detail records that it receives from the web service. The log_file is where the script will write logging messages for debugging problems after the fact. Finally, failed_customer_file is where the script will write customer ID's of customers that the script fails to retrieve from the web services.

process_count is a setting that will allow you to specify how many parallel processes the script should spawn.

url is the url of the web service end-point.

The main job script is called job.py. It imports the settings.py module and then creates a new class that inherits from the Settings class.

class Config(settings.Settings):

"""Configuration class"""

_runtime = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')

@staticmethod

def timestamp(filename):

"""Stamps a filename with the timestamp for this run"""

parts = os.path.splitext(filename)

return "%s-%s%s" % (parts[0], Config._runtime, parts[1])

This class has a static method called timestamp. In the main job, I will call the timestamp method to add the date/time of the run to the output filenames. So you might see something like:

outfile = open(Config.timestamp(Config.log_file), 'w')

This will open a file named 'log-20120204-112530.txt' for writing.

The Job class

The Job class is the main class that executes the job and spawns the worker processes. It has the following methods:

__init__ - constructor, set's up a multiprocessing Pool and 3 Queue's
run - Kicks off the job
get_customers - Reads the customer file
get_requests - Transforms the customers list into a list of processes requests
process_customer_queue - Deals with responses from the web service
process_log_queue - Deals with log messages from the worker processes
process_exceptions_queue - Deals with exceptions that occur in the worker processes
log - Writes log messages to the log file

Here's a copy of the Job class:

class Job:

"""Job that performs the run."""

def __init__(self):

self._manager = multiprocessing.Manager()

self._pool = multiprocessing.Pool(Config.process_count)

self._customer_queue = self._manager.Queue()

self._log_queue = self._manager.Queue()

self._exception_queue = self._manager.Queue()

def run(self):

"""Do it."""

start_time = datetime.datetime.now()

self.log('[Run started]')

customers = self.get_customers()

requests = self.get_requests(customers)

result = self._pool.map_async(get_customer, requests)

while not result.ready():

self.process_customer_queue()

self.process_log_queue()

self.process_exceptions_queue()

self._pool.close()

self._pool.join()

self.process_customer_queue()

self.process_log_queue()

self.process_exceptions_queue()

self.log('[Run finished]')

self.log('[Total runtime: %s]' % (datetime.datetime.now() - start_time))

def get_customers(self):

"""Read the source file."""

buf = [line.strip() for line in open(Config.source_file).readlines()[1:]]

return buf

def get_requests(self, customers):

"""Generate requests from customers."""

requests = [{ \

'customer': customer, \

'data': self._customer_queue, \

'log': self._log_queue, \

'exceptions': self._exception_queue, \

'url': Config.url, \

} for customer in customers]

return requests

def process_customer_queue(self):

"""Pull messages off the data queue."""

try:

message = self._customer_queue.get_nowait()

except Queue.Empty:

return

customer = message['customer']

details = message['server_response']['details']

outfile = open(Config.timestamp(Config.profile_data_file), 'a')

for record in details:

buf = "%s, %s, %s, %s\n" % (customer, record['id'], \

record['relevanceScore'], record['relevanceRank'])

outfile.write(buf)

outfile.close()

def process_log_queue(self):

"""Pull messages off the log queue."""

try:

message = self._log_queue.get_nowait()

except Queue.Empty:

return

self.log(message)

def process_exceptions_queue(self):

"""Pull messages off the exceptions queue."""

try:

message = self._exception_queue.get_nowait()

except Queue.Empty:

return

customer = message['customer']

exception = message['exception']

self.log("EXCEPTION GETTING %s! - %s" % \

(customer, str(exception)))

failed_file = open(Config.timestamp(Config.failed_customer_file), 'a')

buf = "%s\n" % customer

failed_file.write(buf)

failed_file.close()

def log(self, message):

"""Write message to the log file."""

logfile = open(Config.timestamp(Config.log_file), 'a')

timestamp = datetime.datetime.now().strftime('%Y/%m/%d %H:%M:%S')

logfile.write("%s - %s\n" % (timestamp, message))

logfile.close()

Some important things to highlight. The constructor creates three queues: customer_queue, log_queue, and exceptions_queue. These queues are used for interprocess communication. The constructor creates a Pool of worker processes. Each process will communicate back to the parent process via these queues. The customer_queue is responsible for holding the results of the web services calls. The log_queue is responsible for holding log messages. The exceptions_queue is responsible for holding exceptions that occur when worker processes have problems talking to the web service.

In the run method, the Job class will call the Pool class' map_async method. The map_async method splits the requests list into multiple lists and passes each list to a separate worker process for processing. Then the run method goes into a loop, waiting for the worker processes to all complete. While it's waiting, it will continually check the queues to see if any messages have been received.

The stand-alone function

The map_async method's first argument is the name of the function that each worker process should run the request list items with. In this case, the name of that function is get_customer. get_customer looks like this:

def get_customer(request):

"""Retrieve the customer details from the given request."""

customer = request['customer']

url = request['url']

data = request['data']

log = request['log']

exceptions = request['exceptions']

try:

log.put("Requesting details for customer %s" % customer)

req_url = url.replace('{0}', customer)

request = urllib2.Request(req_url)

response = urllib2.urlopen(request)

buf = response.read()

data_records = json.loads(buf)

coupons.put({'customer: customer, 'server_response': data_records})

log.put("Successfully retrieved %d details(s) for %s" % \

(len(data_records), customer))

except Exception, exc:

exceptions.put({'customer': customer, 'exception': exc})

This function is called for each request in the worker processes. It extracts queues from the request. It uses the put method on each queue to pass information back to the parent process.

Pulling it all together

The last thing in my script file is this:

if __name__ == '__main__':

# Let's do this!

Job().run()

It's important to embed the Job().run() call inside the 'if __main__' branching. This is because of how the multiprocessing internals work. When the Pool.map_async method is executed, Python will create a number of sub processes. When each process starts up, it's actually spawning a copy of the Python executable. When that process comes up, it imports the job module and then starts calling the get_customer function over and over with the requests. If the Job().run() were not embedded in a 'if __main__' branch, every worker process would spawn more Job classes. Nothing good would come of that!

Hope you find this helpful.

Create and track svn branches using git-svn

2013-01-29T11:41:00.002-05:00

My most recent client is utilizing svn for the source control needs. While I used to be a big svn guy, I've more recently been using git and have found the transition back to svn a little awkward.

Enter: git-svn. I love git-svn because I can exist in a svn environment without giving up all the things I love about git. One thing that continually escapes my memory is how to create a svn branch from the git CLI as well as how to associate a git branch with a svn branch.

This blog entry is my own cheat-sheet.

Say, I want to create a svn branch off of the trunk. Here's what I would do:

$ git checkout master

$ git svn rebase

$ git svn branch -m "Creating a feature branch" feature_1001

$ git svn dcommit

$ git checkout -b my_feature_1001 feature_1001

Another thing that isn't so intuitive is merging your changes back to trunk when you're done. Here's what I would do:

$ git checkout feature_1001
$ git rebase master
$ git checkout master
$ git merge feature_1001
$ git svn dcommit
$ git branch -D feature_1001

Then I have to use a svn client to actually delete the svn branch.

Also, if a svn branch is removed, you need to remove it by hand from git with

$ git branch -rd feature_1001

Reformatting for 80 characters in vim

2012-10-19T10:24:00.003-04:00

After reading The Pragmatic Programmer and Clean Code, I've pretty much settled on using text files (specifically in markdown format) for all my personal notes and documentation and editing them with vim. That being said, I like to keep my files limited to 80 columns wide. That way I can look at and review multiple documents simultaneously in splits and vertical split easily.

In my .vimrc I have the following setting to get the auto wrapping working for me:

set textwidth=80

That way as I type along, vim is keeping track of the file's column width and when I get to the 80th column it automatically moves me to the next line. Pretty handy.

Here's my problem though, if I change some text that I've already entered and reword it, my paragraph loses it's nicely formatted set column width. Here's an example but with the textwidth setting set at 40:

Before my edit change:

Lorem ipsum dolor sit amet, consectetur

adipiscing elit. Proin neque sapien,

facilisis eget tincidunt ut, porttitor

laoreet lacus. Class aptent taciti

sociosqu ad litora torquent per conubia

nostra, per inceptos himenaeos. Etiam

semper elementum congue.

After my edit change:

Lorem ipsum dolor sit amet, consectetur

adipiscing elit. Proin neque sapien,

facilisis eget tincidunt ut, porttitor

laoreet lacus. Class aptent taciti

sociosqu ad litora torquent per conubia

nostra, per inceptos himenaeos. I

forgot some text. Etiam

semper elementum congue.

Notice the problem: After editing the paragraph, the second to last line doesn't go to the 40th column before wrapping. This is a little annoying to me because I feel the need to reformat the paragraph to get it back to looking like the "before". It's not too big of a deal when I only have to reformat a line or two but if I've change something high up in a fairly long paragraph, reformatting dozens of lines can get tedious.

Here's a little tip that I stumbled across that can make that reformatting go a bit quicker:

1. Select the whole paragraph with V. and then h,j,k and/or l.

2. Then strip the new-line characters with J.

3. Finally reformat it with gq.

Hope you find this helpful!

CGI Script to display clone urls

2012-07-06T12:32:00.002-04:00

After some discussion, I was able to convince my client to convert from gitorious to gitolite. One nice feature that people like about gitorious is that the web interface provides an easy way to look up urls for cloning repositories.

In my mind, that's a pretty legitimate need. To that end, I threw together this bash script, which acts as a cgi and dropped it in the cgi-bin directory of the server that's running gitolite. Hope you find this helpful.

#!/bin/bash

REPOSITORY_DIR="/home/git/repositories/"

URL_PREFIX="git clone git@internal-build-server:"

cat <<DONE

Content-type: text/html

<html>

<head>

<title>Repository URLs</title>

</head>

<body>

<h1>

Repository URLs

</h1>

<p>

Repository URLs on this server follow a specific pattern. The pattern is

as follows:

</p>

git@internal-build-server:<i><font color="darkblue">{category}</font></i>/<i><font color="darkblue">{project}</font></i>

</center>

<p>

These URLs are both pull and push URLs. You do not need separate URLs

for pulling and pushing. Access control will be handled by a

server git update hook that is provided by gitolite.

</p>

<p>

In an effort to make life a little easier in locating your URLs, this script

enumerates URLs for the repositories located on this machine below.

</p>

DONE

CATEGORIES=$(find $REPOSITORY_DIR -type d -maxdepth 1 -mindepth 1 -not \

-iname '*.git' | sed -e "s|$REPOSITORY_DIR||g")

for CATEGORY in $CATEGORIES; do

echo "<h2>Category: $CATEGORY</h2>"

CAT_REPOSITORIES=$(find $REPOSITORY_DIR$CATEGORY -type d -iname \

'*.git' | sed -e "s|$REPOSITORY_DIR||g" -e 's/.git$//g')

for REPOSITORY in $CAT_REPOSITORIES; do

echo "$URL_PREFIX$REPOSITORY<br />"

done

ROOT_REPOSITORIES=$(find $REPOSITORY_DIR -type d -maxdepth 1 -mindepth 1 \

-iname '*.git' | sed -e "s|$REPOSITORY_DIR||g" -e 's/.git$//g')

echo "<h2>Uncategorized Repositories</h2>"

for REPOSITORY in $ROOT_REPOSITORIES; do

echo "$URL_PREFIX$REPOSITORY<br />"

done

cat <<DONE

<br />

</div>

</body>

</html>

DONE

Bash script for pulling/fetching multiple git clones

2012-06-25T12:11:00.001-04:00

In my current assignment, I'm acting as the main build guy for a number of projects that use git for source control. As such, I find it very useful to keep all my git clones up to date whether I'm actively developing in them or not. Additionally, I need to review the changes other developers are committing, so I'd like to get a summary of recent git activities.

Over time, I've put this little bash script together to help me with that. I've included the script in this posting so I can remember later what/why I did this. Disclaimer, I wrote this script and run this script in bash on Linux (not via git-bash in Windows). Also, I'm using Zenity for some nice UI look/feel.

1 #!/bin/bash

3 pushd ~/dev/repos > /dev/null

5 # The log file

6 PULL_LOG="$(mktemp)"

8 # Get a list of all the clones in this directory.

9 CLONES=$(find -maxdepth 2 -mindepth 2 -type d -name ".git" | sed -e 's|\./||' -e 's|/\.git||')

11 # Get a list of all the branches in clone/branch format

12 ALL_BRANCHES=$(for clone in $CLONES; do cd $clone; for branch in $(git branch -l | sed 's/\s\|\*//g'); do echo $clone/$branch; done; cd ..; done)

14 # Count the branches

15 BRANCH_COUNT=$(echo $ALL_BRANCHES | sed 's/ /\n/g' | wc -l)

17 # Start the log file

18 echo "Pull log for $(date)" >> $PULL_LOG

19 echo "--------------------------------------------------------------------------------" >> $PULL_LOG

21 # Function for pipping output to zenity progress dialog

22 function pull_clones() {

23 clone_counter=0

24 for clone in $CLONES; do

25 echo "Pulling branches for clone $clone" >> $PULL_LOG

26 echo "--------------------------------------------------------------------------------" >> $PULL_LOG

27 cd $clone

28 echo "# Fetching changes for clone $clone"

29 git fetch origin 2>> $PULL_LOG

30 for branch in $(git branch -l | sed 's/\s\|\*//g'); do

31 echo "# Merging branch $clone/$branch"

32 echo "Merging branch $branch" >> $PULL_LOG

33 git checkout $branch 2> /dev/null

34 git merge origin/$branch >> $PULL_LOG

35 echo | awk '{print count / total * 100}' count=$clone_counter total=$BRANCH_COUNT

36 let clone_counter=clone_counter+1

37 done

38 cd ..

39 echo >> $PULL_LOG

40 done

41 }

43 # Do it

44 pull_clones | zenity --progress --title='Pulling development clones' --width=512

45 zenity --text-info --filename=$PULL_LOG --title="Pull log" --width=500 --height=450

47 #Clean up

48 rm $PULL_LOG

50 popd > /dev/null

Patching tip using mocks in python unit tests

2012-06-06T10:47:00.003-04:00

I use the mock library by Michael Foord in my python unit tests and one problem always plagued me. Here's the problem and the solution.

Sometimes when I import a package/module in my code I use this pattern (let's call it pattern A):

"""file_module_pattern_a.py"""

import os

def get_files(path):

"""Return list of files"""

return os.listdir(path)

Other times, I use this pattern (let's call it pattern B):

"""file_module_pattern_b.py"""

from os import listdir

def get_files(path):

"""Return list of files"""

return listdir(path_variable)

Note the differente. In pattern A, I import the whole os package, while in pattern B, I only import the listdir function. Now in my unit tests, here's what I use for pattern A:

"""Unit tests for module file_module_pattern_a"""

from file_module_pattern_a import get_files

from unittest import TestCase

from mock import patch, sentinel

class StandloneTests(TestCase):

"""Test the standalone functions"""

@patch('os.listdir')

def test_get_files(self, mock_listdir):

"""Test the get_files function"""

test_result = get_files(sentinel.PATH)

mock_listdir.assert_called_once_with(sentinel.PATH)

self.assertEqual(test_result, mock_listdir.return_value)

This works great. The only problem is... if I use pattern B with this unit test, the mock_listdir never gets called. The unit test tries to use the REAL os.listdir function.

Here's the issue at hand. When I use pattern B, I'm actually adding the function to my module, not the global scope. As a result, the patch directive needs to reference my module, not os. Here's the correct unit test patch syntax:

"""Unit tests for module file_module_pattern_b"""

from file_module_pattern_b import get_files

from unittest import TestCase

from mock import patch, sentinel

class StandloneTests(TestCase):

"""Test the standalone functions"""

@patch('file_module_pattern_b.listdir')

def test_get_files(self, mock_listdir):

"""Test the get_files function"""

test_result = get_files(sentinel.PATH)

mock_listdir.assert_called_once_with(sentinel.PATH)

self.assertEqual(test_result, mock_listdir.return_value)

Using SSHFS from OSX

2012-06-04T11:43:00.000-04:00

SSHFS is a FUSE (Filesystem in Userspace) plugin that allows you to mount a drive/filesystem on your systems via SSH. This is a great way to transfer files securely over the public Internet. Best of all, it's free!

OSX supports FUSE through a program called OSXFUSE (or Fuse for OSX). You can download it at:

http://osxfuse.github.com/

Note, you'll have to download the plugins for OSXFUSE separately. The two most popular are: SSHFS and NTFS-3G (a plugin that allows you to mount NFTS volumes in read/write mode... which I will not be covering in this post).

Computers hosting sshfs directories don't need to have any special software installed on them. They simply need to be running sshd (the secure shell daemon) and not have file transfers disabled. Usually, sshd is configured by default to have file transfers enabled.

One thing that often confuses people coming from the Mac world is that SSHFS has no GUI. So how do you mount remote drives? Well, you'd normally do it from a terminal prompt.

In this blog entry, I will go over three ways to mount sshfs drives/filesystems using SSHFS:

Using terminal.app and some bash commands
Using Automator
By creating an Alfred.app extension

Get the software

Before we begin, first make sure you have the OSXFUSE software installed. Download it from the link listed above. You'll know you have it properly installed when you see it in your System Preferences:

You'll also want to ensure that you've installed the SSHFS plugin from the same site. To verify that you've installed it correctly, you'll need to run sshfs -h from a terminal window:

If you've got these two items installed, you're ready to go!

First way: From the bash prompt

This method is the most traditional way to mount a sshfs filesystem... and actually the other two methods will be invoking the same commands...we'll just be hiding them with the graphical interface.

First start a terminal window and create a directory to house the filesystem. I always put my filesystems in the /Volumes directory. You aren't required to put your directory there. You can put it anywhere you'd like. I just do so out of convention. Here's the command to do so:

steve@l00-1nsv01 $ mkdir /Volumes/ssh_fs_mount

I named my mount ssh_fs_mount. You can name yours anything you'd like.

Then you need to run the sshfs command to mount the remote computer's directory to your hosting directory. The command takes the following format:

sshfs username@hostname:/path/to/directory /local/directory

...where username is your username on the remote computer, hostname is the name of the remote computer contains the directory that you'd like to mount, /path/to/directory is the full path to the directory on the remote computer that you'd like to mount and /local/directory is the directory on your local computer that you'd like to house the mount on. Maybe an example will make it more clear.

I have a Linux computer that I do a lot of development on. It's called ubuntu64.local. My username on that computer it steve and my development folder on that computer is at /home/steve/dev. Here's the sshfs command that I would issue to mount my development folder from the Linux computer into the /Volumes/ssh_fs_mount folder on my Mac

sshfs steve@ubuntu64.local:/home/steve/dev /Volumes/ssh_fs_mount

Let's try it real quick:

A couple of items to note... First, I'm prompted for a password. This is prompting me for my password on the remote computer (ubuntu64.local in this instance). It's not prompting me for my Mac password. This is the remote computer verifying that I'm indeed steve@ubuntu64.local and not some other impostor. If I have key-based authentication working between my Mac account and my Linux account, I would not be prompted for the password. I normally do have key-based authentication between these two accounts, but I turned it off for this demonstration. If you're not using key-based authentication when ssh'ing between computers... you should. In fact, the other two methods in this blog entry (using Automator and the Alfred extension) assume you are using key-based authentication and aren't prompted for a password. You can find out more about setting-up key-based authentication at any of these links:

Google Search

Second, there's no other output from the command. No output means that sshfs was able to successfully complete the mount. If there had been a problem, sshfs would have complained with error messages. Once the mount is completed, the drive should appear on the desktop:

It should behave like a normal Mac drive. You can get info on it. You can browse it in Finder. You can even add/edit/remote files (assuming your account on the remote computer has the appropriate privileges).

When you're done and want to unmount the drive, simply right-click on it and select Eject.

Note, when the SSHSFS volume has been ejected, OSX automatically deletes the /Volumes/ssh_fs_mount directory. If you put your mounting directories in /Volumes, you'll always have to recreate them after ejecting.

Second Way: Create an Automator task

Using Automator, you can create a workflow to automate the steps you did in the first method. After all, that's what Automator is for.. automating repetitive things.

First, start Automator and create a new Application workflow:

Next, in the Text Library, drag the Ask For Text action to the workflow.

Check the Ignore this action's input and Require an answer checkboxes. Enter Enter a remote sshfs url and click OK in the question textbox.

Now, from the Utilities library, drag the Run Shell Script action to the workflow and drop it below the Ask for Text action.

Change the Pass input pulldown from to stdin to as arguments. Then past the following text into the textarea:

volume_name=/Volumes/sshfs_volume_$$

mkdir $volume_name

/usr/local/bin/sshfs $1 $volume_name

if [ "$?" == "0" ]; then

open $volume_name

else

echo "Unable to mount sshfs volume."

rmdir $volume_name

Save it to your desktop as SSHFS Workflow. It should now look like this:

Now, if you run it from your desktop, it will prompt you for a sshfs url. The format is the same from the bash method. I'll reuse my example:

steve@ubuntu64.local:/home/steve/dev

Again, like I mentioned before, this method assumes that you've already set up key-based authentication between your Mac account and your remote account.

If all goes well, the automator will automatically open the mounted volume. Just like the first method, when you're done, you simply right-click the volume and eject it via Finder.

Third Way: Create an Alfred Extension

Now if you're running Alfred, you're probably thinking, "I can just run the automator task from Alfred and be done with it." Yes. You could. But you can also create an extension. That way you can enter the sshfs url directly into the Alfred window and save yourself one last data-entry step.

To create the extension go to the Extensions tab of the Alfred preferences window. Click the + and select Shell Script.

In the Extension Name field, enter SSHFS and click Create.

In the Title field, enter SSHFS Mounter. In the Description field, enter Mount a sshfs volume. Make sure the Keyword checkbox is checked and the enter sshfs in the textbox next to it. Make sure the Silent checkbox is checked. Finally in the Command field, enter the following text:

volume_name=/Volumes/sshfs_volume_$$

mkdir $volume_name

/usr/local/bin/sshfs {query} $volume_name

if [ "$?" == "0" ]; then

open $volume_name

else

echo "Unable to mount sshfs volume."

rmdir $volume_name

If should look like this:

If you have Growl installed on your Mac, click the Advanced button and check the Display script output in Growl checkbox.

Save the extension and close the preferences window. Now from the the Alfred prompt, you can enter sshfs your_sshfs_url and hit enter, where your_sshfs_url is an actual sshfs url. Here's me mounting my ubuntu64.local system's tmp directory:

Just like the Automator method, the Alfred method presumes you are using key-based authentication between your Mac account and the remote computer's account. It should automatically open the folder you mounted. When you're done, simply right-click on the volume and eject it.

Run multiple python versions on your system

2012-05-30T10:53:00.000-04:00

I'm a software development consultant. I write python (as well as other languages) code for many clients and I don't get to dictate what their environment looks like. I've got clients running python as old as 2.4 while others are on the bleeding edge. Additionally, each client may have their own packages installed as well as differing lists of third party packages.

This post is a description of how I went about getting multiple versions of python installed in my ubuntu development machine and how I go about managing different package sets for different clients.

Get multiple pythons installed

Ubuntu typically only supports one python 2.x version and one 3.x version at a time. There's a popular ppa (personal package archive) called deadsnakes that contains older versions of python. You can find it:

https://launchpad.net/~fkrull/+archive/deadsnakes

To install it (per the instructions in the link above), you do the following:

steve@ubuntu64 ~ $ sudo add-apt-repository ppa:fkrull/deadsnakes

Then you need to update your cache:

steve@ubuntu64 ~ $ sudo apt-get update

Finally, simply install the other versions (I'm running on ubuntu 12.04 LTS, so I have python 2.7 already):

steve@ubuntu64 ~ $ sudo apt-get install python2.4 python2.5 python2.6

If you're following along, we now have python versions 2.4 through 2.7 installed the computer. If you run 'python', you'll see the default version is still 2.7.

steve@ubuntu64 ~ $ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Here's why: Each python version is stored in /usr/bin as python2.X where X is the version. There is a symbolic link named python that points to the version you want to be the default. Instead of typing python from the bash prompt, you could just as easily type python2.5:

steve@ubuntu64 ~ $ ls -l /usr/bin/python*
lrwxrwxrwx 1 root root 9 Apr 17 13:20 /usr/bin/python -> python2.7
lrwxrwxrwx 1 root root 9 Apr 17 13:20 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 1216520 May 21 12:13 /usr/bin/python2.4
-rwxr-xr-x 1 root root 1403624 May 3 00:17 /usr/bin/python2.5
-rwxr-xr-x 1 root root 2652056 May 12 08:43 /usr/bin/python2.6
-rwxr-xr-x 1 root root 2993560 Apr 20 19:37 /usr/bin/python2.7

steve@ubuntu64 ~ $ python2.5
Python 2.5.6 (r256:88840, May 3 2012, 04:16:14)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

This is a point of interest. I would not mess with the symbolic link. Ubuntu runs python for many internal maintenance scripts and those scripts are expecting the python version that shipped with ubuntu.

Use virtualenv to manage your python installations and package sets

So now that you have multiple versions of python on your system, how to manage them? How do you keep packages installed for one version separate from packages installed for another. What if you want to run one version of django for client X and a different version for client Y?

That's where virtualenv comes in. If your a ruby programmer, this is analogous to rvm. Virtualenv lets you manage the python versions and package installations separately for different projects or clients.

Installing virtualenv is simple

As always, you're just a single apt-get command away from having virtualenv ready to go:

steve@ubuntu64 ~ $ sudo apt-get install python-virtualenv

That it. Virtualenv is ready to go now.

Quick example

Say your starting a new project for a client. They are running python2.5 and want to use the mocks, nose and coverage packages for testing. Here a walkthrough of how to use virtualenv to manage the project.

First, let's create a directory for the project:

steve@ubuntu64 ~ $ mkdir -p ~/dev/project1
steve@ubuntu64 ~ $ cd ~/dev/project1

Next, run virtualenv to create the environment for the project:

steve@ubuntu64 ~/dev/project1 $ virtualenv -p /usr/bin/python2.5 .env
Running virtualenv with interpreter /usr/bin/python2.5
New python executable in .env/bin/python2.5
Also creating executable in .env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
steve@ubuntu64 ~/dev/project1 $

This command tells virtualenv to create a .env directory and to place a copy of the 2.5 version of python in it. This copy of the 2.5 python is brand-spankin' new. It doesn't have any packages (beyond the standard library) installed. You will need to install them yourself. Any packages that you install in this instance of python will not be available to main python installation or other virtualenv instances.

Before your can use this new copy, you need to activate it:

steve@ubuntu64 ~/dev/project1 $ source .env/bin/activate
(.env)steve@ubuntu64 ~/dev/project1 $

The activate script manipulates your path environment variable, placing the new python instance first in your path. This makes is so that when you run python, it will use the version from your instance:

(.env)steve@ubuntu64 ~/dev/project1 $ which python
/home/steve/dev/project1/.env/bin/python
(.env)steve@ubuntu64 ~/dev/project1 $ python
Python 2.5.6 (r256:88840, May 3 2012, 04:16:14)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Also, notice that your prompt starts with (.env). This tells you that you're running with the virtualenv instance activated. To install packages in your instance, use the pip command:

(.env)steve@ubuntu64 ~/dev/project1 $ pip install mock nose coverage
Downloading/unpacking mock
Downloading mock-0.8.0.tar.gz (749Kb): 749Kb downloaded
Running setup.py egg_info for package mock
warning: no files found matching '*.png' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.html' under directory 'docs'
warning: no files found matching '*.js' under directory 'docs'
Downloading/unpacking nose
Downloading nose-1.1.2.tar.gz (729Kb): 729Kb downloaded
In the tar file /tmp/pip-dH_WYa-unpack/nose-1.1.2.tar.gz the member nose-1.1.2/doc/doc_tests/test_selector_plugin/support/tests/mymodule/my_function$py.class is invalid: 'filename None not found'
In the tar file /tmp/pip-dH_WYa-unpack/nose-1.1.2.tar.gz the member nose-1.1.2/doc/doc_tests/test_restricted_plugin_options/restricted_plugin_options.rst.py3.patch is invalid: 'filename None not found'
Running setup.py egg_info for package nose
Downloading/unpacking coverage Downloading coverage-3.5.2.tar.gz (115Kb): 115Kb downloaded
Running setup.py egg_info for package coverage
no previously-included directories found matching 'test'Installing collected packages: mock, nose, coverage
Running setup.py install for mock
warning: no files found matching '*.png' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.html' under directory 'docs'
warning: no files found matching '*.js' under directory 'docs'
Running setup.py install for nose
Installing nosetests script to /home/steve/dev/project1/.env/bin
Installing nosetests-2.5 script to /home/steve/dev/project1/.env/bin
Running setup.py install for coverage
building 'coverage.tracer' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c coverage/tracer.c -o build/temp.linux-x86_64-2.5/coverage/tracer.o
coverage/tracer.c:3:20: fatal error: Python.h: No such file or directory
compilation terminated.
**
** Couldn't install with extension module, trying without it...
** SystemExit: error: command 'gcc' failed with exit status 1
**
no previously-included directories found matching 'test'
Installing coverage script to /home/steve/dev/project1/.env/bin
Successfully installed mock nose coverage
Cleaning up...
(.env)steve@ubuntu64 ~/dev/project1 $

To see that the packages have been installed, simply use them:

(.env)steve@ubuntu64 ~/dev/project1 $ python
Python 2.5.6 (r256:88840, May 3 2012, 04:16:14)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mock
>>> import nose
>>> import coverage
>>>

When you're done working on the project, deactivate it. You can always come back later and activate it again.

(.env)steve@ubuntu64 ~/dev/project1 $ deactivate
steve@ubuntu64 ~/dev/project1 $

Notice that when you deactivate the environment, your packages are no longer available:

steve@ubuntu64 ~/dev/project1 $ python
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mock
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named mock
>>> import nose
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named nose
>>> import coverage
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named coverage
>>>

You can create as many virtualenv environments as you like. I create one for each project that I work on.

Bonus material

To make life even easier, here's a couple additional things I do that you might find helpful!

First, once you've started to use virtualenv with some frequency, you start to get tired of downloading and installing the same packages over and over. Pip has the ability to cache your downloaded packages for reuse. To do that, you'll need to create a directory to store the download packages in:

steve@ubuntu64 ~ $ mkdir ~/.pip_download_cache

Then you'll need to set a variable to inform pip of the new directory. Add the following to your .bashrc file:

export PIP_DOWNLOAD_CACHE=/home/steve/.pip_download_cache

Now when you do a pip install, it will keep the downloaded files in the ~/.pip_download_cache directory. The next time you do a pip install of the same package, it will just use the copy from the directory instead of downloading it again.

Second, it can be tedious to always have to type 'source .env/bin/activate' every time you want to activate an environment. Since I always put my virtual environments in a .env directory I can count on the command to activate always being the same. So I create an alias for it. I added the the following to my ~/.bash_aliases file:

alias activate='source .env/bin/activate'

Now once I cd into the projects directory, I simply type activate to activate my virtual environment.

File encryption with vim

2012-05-25T10:05:00.002-04:00

You may not know it, but you can encrypt your files using vim... and it's pretty easy to do.

Turning encryption on for a file

If you're editing a file, to encrypt it, simply enter X at the : prompt.

Vim will then prompt you for a key. A key is the password for decrypting the file in the future. You'll have to enter it twice to ensure you typed it correctly.

Once you've entered your key, the next time you save/write, the file should be encrypted, see the status text at the bottom to be sure.

Once the file is encrypted, it will be unreadable to others.

Vim auto-detects whether files are encrypted. If you try to open an encrypted file, vim will prompt you for the key.

Vim uses whatever you enter for the key to decrypt the file. If you enter the correct things will look great. If you enter an invalid key, vim will present you with garbage to edit.

The encryption will stay on for the file until you tell vim you'd like to remove it.

Be mindful of your key. If you forget it, you will not be able to retrieve the contents of your file.

Turning off encryption for a file

Turning encryption off is equally as easy. Simply set the key setting to nothing. With the command:

:set key=

Be sure to write the file. Note that it is no longer encrypted (status text)

Wait! Before you go

As of vim 7.3, vim uses one of two encryption methods: zip and blowfish. Zip is the same encryption that PkZip uses and is somewhat weak (can be cracked). Conversely, blowfish is more contemporary and is much harder to crack. Zip, unfortunately, is the default encryption method in vim. I would strongly suggest that you use blowfish when encrypting your files. To do this is simple enough. Simply enter the command:

:set cm=blowfish

I put it in my .vimrc file so it's always set.

Some light string manipulation in bash

2012-05-24T11:00:00.001-04:00

This post is more for me than you. I had to do a little bash work this morning and thought I'd keep a record of some of the syntax and concepts that I used in case I have to do more in the future.

There are two files here: a function library and a script that consumes it. The function library is used to do different kinds of string justification (left justify, right justify and centering). Kind of like 'echo' on steroids. This could have easily been done in python, perl or ruby, but I wanted it to stay in bash this time to keep the calling scripts uniform.

Here's the sample file that consumes the library:

#!/bin/bash

. echo_helper

x='Steve is here!'

left_justify "$x" 40 .

right_justify "$x" 40 .

center_justify "$x" 40 .

echo $(pad_string 40 -)

x='Steve was here.'

left_justify "$x" 40 .

right_justify "$x" 40 .

echo $(center_justify "$x" 40 .)

And here's what the output should look like:

Steve is here!..........................

..........................Steve is here!

.............Steve is here!.............

----------------------------------------

Steve was here..........................

.........................Steve was here.

............Steve was here..............

Finally, here's the echo_helper library code:

#!/bin/bash

# Syntaxes to remember:
#
# String length of a variable: ${#varname}
# String offsetting of a variable: ${varname:offset} or ${varname:offset:length}

# Usage: pad_string LENGTH {PADDING_CHARACTER=' '}
function pad_string {
local line=''
local length=$1
local padding_character=' '
if [[ "$2" != "" ]]; then
padding_character=$2
fi
while((${#line} < $length)); do
line="$line$padding_character"
done
echo -n "$line"
}

# Usage: left_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}
function left_justify {
local message=$1
local length=$2
local padding_character=' '
if [[ "$3" != "" ]]; then
padding_character=$3
fi
echo "$message$(pad_string $length-${#message} $padding_character)"
}

# Usage: right_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}
function right_justify {
local message=$1
local length=$2
local padding_character=' '
if [[ "$3" != "" ]]; then
padding_character=$3
fi
echo "$(pad_string $length-${#message} $padding_character)$message"
}

# Usage: center_justify MESSAGE LENGTH {PADDING_CHARACTER=' '}
function center_justify {
local message=$1
local length=$2
local padding_character=' '
if [[ "$3" != "" ]]; then
padding_character=$3
fi
local half_length=$(expr $length / 2)
local half_message_length=$(expr ${#message} / 2)
local padding_length=$(expr $half_length - $half_message_length)
padding="$(pad_string $padding_length $padding_character)"
message="$padding$message$padding"
if [[ ${#message} > $length ]]; then # handle even/odd length issue
echo "${message:1}"
else
echo "$message"
fi
}

QuickLook from vim

2012-05-23T14:25:00.002-04:00

Here's another nifty trick I came up with. I'm probably not the first to think of it but I'll put it out there anyway.

Quicklook is a neat feature of OSX (versions since Leopard) that allows you to preview the contents of a file without having to open it. The way I've typically used it is to select a file in finder and hit the spacebar. OSX then opens a nicely formated window with the contents of my selected file displayed.

I keep a lot of my documentation, notes, etc in markdown format. Back in January, I attended CodeMash in Sandusky, Ohio and someone turned me onto a markdown rendering Quicklook plugin. It's a great little tool. I can keep writing my docs and notes in markdown, but when I Quicklook them, they are rendered for display nicely. If you have interest in this free plugin, you can download it here.

Anyway, I settled into this habit of editing a file in vim, saving it, then pressing spacebar on it from finder to preview it. It got me thinking, wouldn't it be great if I could just open Quicklook directly from vim?

After some quick googling around, I found a command-line tool in OSX that does just that. It's called qlmanage. From there it was easy enough to come up with a vim key mapping.

Put this (all one line) in your .vimrc file and you should be able to do Quicklooks of your files right from vim simply by hitting \v:

map <Leader>v :write<cr>:sil !/usr/bin/qlmanage -p % > /dev/null &<cr>:redraw!<cr>

It works from both MacVim as well as the terminal version. You don't have to use it solely with markdown files. In fact you can use it with any file that Quicklook understands (html for example).

Fun with gnu screen

2012-05-22T11:29:00.002-04:00

In my day job, I spend a fair amount of time working on numerous tasks on the command-line in Linux and OSX. Way back in my college days, I used a program called gnu screen (aka 'screen') but recently I've "refound" it. I have a new appreciation for screen and now use it every day.

What is screen?

Screen let's you have multiple shell sessions running simultaneously in the same terminal window. Here's some of the features that I personally find appealing:

With a little muscle memory, you can switch rapidly between the sessions.
It has the ability to split up your terminal window into multiple "window-splits" so you can see two or more shells sessions at the same time. The splits are even resizable.
You can 'detach' and 'reattach' to a screen session, allowing you to sign-out, leave and come back later, all the while keeping your terminal sessions active and running.
It's configurable and scriptable.
You can copy and paste between sessions with 'vi' like keystrokes.

How do you get screen?

Well, if you're running ubuntu like me, it's pretty darn easy. You're one command away from having it:

steve@ubuntu64 ~ $ sudo apt-get install screen

Pretty simple.

Getting started

Once you have screen installed, simply run it.

steve@ubuntu64 ~ $ screen

It will bring you to a welcome screen (I'll show you how to disable the welcome screen later).

At the bottom of the welcome screen, it will say

[Press Space or Return to end.]

Go ahead and do that. You'll be returned to a bash prompt. It's important to note that screen hasn't terminated, but rather it's spawned another shell prompt within itself and you're sitting at that new prompt. You can now begin using screen via keyboard command sequences. To prove it, press Ctrl-a and then press " (double quote). You should get a screen that looks like this:

This is the terminal session list. If you have more than one terminal session running, you'd seem them all listed here. Let's create another terminal session. Press Ctrl-a and then press c. This should take you back to a bash prompt. Your now in a new bash prompt. See see the updated list, press Ctrl-a and " again.

Now there's two bash sessions. You can select the one you'd like to see by using your arrow keys (or vi cursor movement keys) to highlight the one you want and then press enter/return.

When you want to quit out, simply exit out of all your bash prompts. Once the last bash session ends, screen will automatically terminate, leaving where you were when you initially started it.

My most used key sequences

As you've probably realized by now, screen uses Ctrl-a plus a key to drive it's functionality. Here's a list of key sequences that I use most often.

Ctrl-a Ctrl-a - Toggles between two sessions. It's kind of like the "last" button on your TV remote.

Ctrl-a c - Creates a new terminal session.

Ctrl-a A - Let's you set the title of a terminal session. The title is what appears in the session list.

Ctrl-a " - Lists the terminal sessions. You can then use arrow keys or vi keys to select which session you wan to go to.

Ctrl-a d - Detach screen so you can log off and come back later. Later when you sign back on, just type screen -r to reattach to your running screen session.

Ctrl-a : - Bring up screen's colon prompt (similar to vi). From the colon prompt you can issue commands directly to screen. (You'll probably use it most resizing window splits).

Ctrl-a [ - Start copy mode (for copy/paste).

Ctrl-a ] - Paste what you copied from copy mode.

Ctrl-a S - Split the screen horizontally.

Ctrl-a | (pipe) - Split the screen vertically.

Ctrl-a {tab} - Move focus from window split to window split

Ctrl-a X - Close the current window split.

Split example

Here a quick example of doing a couple window splits. First, start screen and a few terminal sessions:

steve@ubuntu64 ~ $ screen

{return}

Ctrl-a c

Ctrl-a "

Your terminal window should look like this:

Now press Ctrl-a S to split the screen. You should now have an upper split and a lower split. Press {return} now to select terminal session 2 in the upper split. In that terminal session run the top command.

Now press Ctrl-a {tab}. This will move the cursor down to the bottom split. Then press Ctrl-a " to bring up the session list.

Select session 0 and then issue the following command in the lower split: watch ls.

Next, let's split the lower window vertically. Press Ctrl-a | (pipe), Ctrl-a {tab}, Ctrl-a ". Select session 1.

As you can see, you can keep splitting, sub-splitting, etc. Remember, Ctrl-a {tab} moves the cursor to the next window split. That's how you move around.

When you're done, you can use Ctrl-a X to close the window splits. This does not quit/kill the process running in that split. The process stays in the session list until you exit it normally.

Scripting

Screen reads and processes the ~/.screenrc file at startup. Here's mine:

vbell off

startup_message off

defscrollback 1000

bindkey -k k1 colon

bindkey -k k2 screen

bindkey -k F2 windowlist -b

hardstatus off

caption always '%{Yb}[%=%t%=]'

altscreen on

shell -${SHELL}

You can go through the screen man page for what each line does. You can customize screen however you'd like.

There are two features that combined make startup processing especially interesting:

You can use the -c command line argument to give screen an alternate rc file to process on startup.
Using commands, you can do anything in the rc file that you could do interactively.

Using this notion, I create different screen rc files for different contexts. For example, when I'm working at a certain client, I would like to have screen set up a certain way, with specific terminal sessions setup. I create a bash aliases for each client.

In my ~/.bash_aliases file I have:

alias clientX='screen -c ~/screen_profiles/clientX'

alias clientY='screen -c ~/screen_profiles/clientY'

My ~/screen_profiles/clientX file looks like this:

altscreen on

vbell off

attrcolor b ".I"

termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'

defbce "on"

defscrollback 1500

bindkey -k k1 colon

bindkey -k k2 screen

bindkey -k F2 windowlist -b

screen -t "Home" bash

screen -t "Build" ssh my_username@build_server

screen -t "Dev2" ssh my_username@dev_server

screen -t "WebLogic" my_username@weblogic_server

screen -t "Todo/Notes" vi ~/notes/todo.markdown

hardstatus off

caption always '%{Yb}[%=%t%=]'

select 0

split -v

resize +20

focus

select 5

focus

This rc file does the following:

Does some normal setup that I do anyway (key bindings, colors, etc.)
Creates a terminal session titled, "Home" and runs bash in it.
Creates a terminal session titled, "Build" and ssh's me into the build server.
Creates a terminal session titled, "Dev2" and ssh's me into the dev 2 server.
Creates a terminal session titled, "WebLogic" and ssh's me into the weblogic server.
Creates a terminal session titled, "Todo/Notes" and edits my ~/notes/todo.markdown file using vi in it.
Splits the window vertically, resizing the left window to be 20 columns bigger.
Selects the "Home" session for the left window.
Selects the "Todo/Notes" session for the right window.
Puts the cursor focus on the left window.

So when I come into clientX's office for the day, I simply type clientX at the bash prompt and within a second, I have screen all set up with my splits and automatically logged into all the servers I commonly use. Pretty cool.

Gitolite... my new shiny penny

2012-05-17T10:41:00.003-04:00

At my current client engagement, I'm serving as the build-guy and all around code librarian. I'm managing 35 or so git repositories... keeping them tidy and organized.

Since the client seems to heavily utilize consultant help, they have a veritable revolving door of staff. Before my arrival they decided to purchase and use a tool called gitorious to aid in managing the repositories and access to them. It's pretty much like a private github. It has all kinds of features including web-based management of users/keys, browse-able git logs, etc. They have it running as an appliance on a virtual server here at the office.

It works pretty well and I've embraced it because it's the "officially-blessed-by-enterprise-architecture" tool.

It's not without it's drawbacks though:

In it's default configuration, EVERYONE/ANONYMOUS has READ access to ALL repositories.
Gitorious is heavily integrated with git's hooks, so if you want to add your own hooks to a repository, you're pretty much out of luck... unless you like tampering with their code - good luck with that.
If you don't want use their appliance and want to install it yourself, it's not all that straight forward.
The UI isn't really that intuitive. For example if you want to see a list of tags for a repository, you have to drill into a branch rather than seeing them at the repository level.
It costs money if you want to have your own on-site copy.

All that said, I started looking around for alternatives for managing medium to large numbers of git repositories (more than 5'ish) both for my own stuff as well as for future clients.

In looking around, I found a tool called gitolite. Gitolite seem to provide all the features that I want/need and has the benefits of being simple/easy to implement as well as being opensource/free.

Gitolite doesn't have a user interface. You mange it with... what else... git!

Installation is easy

If you're running on Ubuntu, you're only a command away from installation:

steve@ubuntu64 ~ $ sudo apt-get install gitolite

That get's app installed and ready to configure.

Configuration is easy too

Instead of housing gitolite on my personal account (which you could easily do... I just choose not to), I created an account for it called git.

steve@ubuntu64 ~ $ sudo adduser git

Adding user `git' ...

Adding new group `git' (1002) ...

Adding new user `git' (1002) with group `git' ...

Creating home directory `/home/git' ...

Copying files from `/etc/skel' ...

Enter new UNIX password:

Retype new UNIX password:

passwd: password updated successfully

Changing the user information for git

Enter the new value, or press ENTER for the default

Full Name []: Git Repository Home

Room Number []:

Work Phone []:

Home Phone []:

Other []:

Is the information correct? [Y/n] y

steve@ubuntu64 ~ $

After creating the account, you'll want to copy your public key to the git account's home directory for the one and only step of initial gitolite configuration.

steve@ubuntu64 ~ $ sudo cp .ssh/id_rsa.pub /home/git/steveb.pub

steve@ubuntu64 ~ $ sudo chown git.git /home/git/steveb.pub

steve@ubuntu64 ~ $

Now we're ready to do the initial configuration. You'll need to become the git account to do it.

steve@ubuntu64 ~ $ su - git

Password:

git@ubuntu64:~$

Then you'll want to issue the command gl-setup from the bash prompt, passing in the key you just copied.

git@ubuntu64:~$ gl-setup steveb.pub

It will prompt you to edit the default .gitolite.rc file, but don't change anything, it's fine as it is for now. You may have a need to change it in the future but I haven't so far. Just save and quit out of your editor without changing anything.

The default settings in the rc file (/home/git/.gitolite.rc) are fine for most

people but if you wish to make any changes, you can do so now.

hit enter...

creating gitolite-admin...

Initialized empty Git repository in /home/git/repositories/gitolite-admin.git/

creating testing...

Initialized empty Git repository in /home/git/repositories/testing.git/

[master (root-commit) 09aad52] start

2 files changed, 6 insertions(+)

create mode 100644 conf/gitolite.conf

create mode 100644 keydir/steveb.pub

git@ubuntu64:~$

That's it. Gitolite is all set up and ready to use!

git@ubuntu64:~$ exit

logout

steve@ubuntu64 ~ $

Using gitolite is a piece-o-cake

Managing your git repositories with gitolite is done entirely by manipulating a git repository called gitolite-admin. First let's use gitolite to create a new repository. Step one: clone the gitolite-admin repository from your system:

steve@ubuntu64 ~ $ cd dev

steve@ubuntu64 ~/dev $ git clone git@localhost:gitolite-admin.git

Cloning into 'gitolite-admin'...

remote: Counting objects: 6, done.

remote: Compressing objects: 100% (4/4), done.

remote: Total 6 (delta 0), reused 0 (delta 0)

Receiving objects: 100% (6/6), done.

steve@ubuntu64 ~/dev $

You'll notice that the gitolite-admin clone has a pretty simple directory structure.

steve@ubuntu64 ~/dev $ cd gitolite-admin

steve@ubuntu64 ~/dev/gitolite-admin(master) $ ls -la

total 20

drwxr-xr-x 5 steve steve 4096 May 17 09:23 .

drwxrwxr-x 13 steve steve 4096 May 17 09:23 ..

drwxrwxr-x 2 steve steve 4096 May 17 09:23 conf

drwxrwxr-x 8 steve steve 4096 May 17 09:24 .git

drwxrwxr-x 2 steve steve 4096 May 17 09:23 keydir

steve@ubuntu64 ~/dev/gitolite-admin(master) $

It has two directories: conf and keydir. The conf directory contains a file called gitolite.conf. You edit the gitolite.conf file to create repositories and manage permissions to those repositories.

steve@ubuntu64 ~/dev/gitolite-admin(master) $ ls -l conf

total 4

-rw-rw-r-- 1 steve steve 92 May 17 09:23 gitolite.conf

steve@ubuntu64 ~/dev/gitolite-admin(master) $

The keydir directory is where you should place public key files of the users you want to grant access to repositories. You'll notice that the public key you copied for your self is already there!

steve@ubuntu64 ~/dev/gitolite-admin(master) $ ls -l keydir

total 4

-rw-rw-r-- 1 steve steve 391 May 17 09:23 steveb.pub

steve@ubuntu64 ~/dev/gitolite-admin(master) $

Let's add that new repository. To do so, simply edit the conf/gitolite.conf file with your favorite text editor.

steve@ubuntu64 ~/dev/gitolite-admin(master) $ vi conf/gitolite.conf

Add the following lines to it:

repo gitolite-admin

RW+ = steveb

repo testing

RW+ = @all

repo my_cool_new_repository

RW+ = steveb

The two new lines in the file establish that you want to add a new repository named my_cool_new_repository and that you want to grant the user with the public key file steveb.pub full READ/WRITE/REFEDIT permissions.

Save the file. Commit it. Then push it back to the repository origin.

steve@ubuntu64 ~/dev/gitolite-admin(master) $ git add .

steve@ubuntu64 ~/dev/gitolite-admin(master) $ git commit -m"Added new repository my_cool_new_repository"

[master 393ef15] Added new repository my_cool_new_repository

1 file changed, 3 insertions(+)

steve@ubuntu64 ~/dev/gitolite-admin(master) $ git push

Counting objects: 7, done.

Delta compression using up to 2 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (4/4), 433 bytes, done.

Total 4 (delta 0), reused 0 (delta 0)

remote: creating my_cool_new_repository...

remote: Initialized empty Git repository in /home/git/repositories/my_cool_new_repository.git/

To git@localhost:gitolite-admin.git

09aad52..393ef15 master -> master

steve@ubuntu64 ~/dev/gitolite-admin(master) $

Gitolite sees your change to the conf/gitolite.conf file. It then creates a new blank, bare git repository called my_cool_new_repository.git. It also ensures that the steveb.pub file has the appropriate access to the git account (check out the git account's ~/.ssh/authorized_keys).

That's it. Your new repository is ready to clone!

steve@ubuntu64 ~/dev/gitolite-admin(master) $ cd ..

steve@ubuntu64 ~/dev $ git clone git@localhost:my_cool_new_repository.git

Cloning into 'my_cool_new_repository'...

warning: You appear to have cloned an empty repository.

steve@ubuntu64 ~/dev $ cd my_cool_new_repository/

steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ date > file1.txt

steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ git add .

steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ git commit -m"Added file1.txt"

[master (root-commit) 90e1635] Added file1.txt

1 file changed, 1 insertion(+)

create mode 100644 file1.txt

steve@ubuntu64 ~/dev/my_cool_new_repository(master) $ git push origin master

Counting objects: 3, done.

Writing objects: 100% (3/3), 257 bytes, done.

Total 3 (delta 0), reused 0 (delta 0)

To git@localhost:my_cool_new_repository.git

* [new branch] master -> master

steve@ubuntu64 ~/dev/my_cool_new_repository(master) $

Adding/removing user access is as simple as adding/removing key files from the keydir directory, updating the conf/gitolite.conf file and pushing.

The gitolife.conf file, while very simple, is incredibly powerful. It supports groups. Also, it will not only allow you to control access to git repositories, but will also allow you to assign permissions to branches and tags with regular expressions. Take a look at the documentation for much more info.

Use automator to create X11 shortcuts in OSX

2012-05-01T11:57:00.002-04:00

I have a Mac and yet I do a fair amount of work in X11apps on Linux systems. I really like the ssh's -Y flag that let's me run my Linux X11 apps and have their display output (and input) sent back to my Mac.

I just stumbled onto this in Automator and thought it might be worth sharing.

You can create Automator "Application" workflows that look/feel/behave like local OSX applications, but are actually running on remote Linux servers. In this posting, I will walk through creating an Automator workflow for Synaptic, an application package manager for apt based Linux distributions.

Requirements

You will need to ensure you have an environment that meets the following requirements in order to follow along:

A Mac running OSX (I'm using Lion. I've not tested this with other versions so if you're using another version, your milage may vary.)
You will need X11.app or XQuartz.app installed on your Mac. X11.app comes with OSX up to Lion. After Lion, you will need to install XQuartz.app yourself.
Access to a Linux system that is running SSH with X11 tunneling enabled (a default I believe) and gksudo and synaptic installed.
You need to have key-based authentication set up with your account so you are not prompted for a password when logging into the Linux machine via ssh.

gksudo is normally not needed. In this case we're running synaptic, a package manager for Linux, which needs elevated privileges in order to add/remove software packages on the Linux machine. If the X11 app you are wanting to run doesn't require elevated privileges you won't need gksudo.

Verify that you can log in without as password prompt

Before we create the Automator workflow, let's verify that you can log into the Linux system from your Mac without a password prompt. Start a terminal window and issue a command in the following format:

ssh username@linux_hostname

If you are prompted for a password you will still need to set yourself up for key-based authentication. Check out any of these links and then come back. You should be let in without a password as seen below:

Start a new Automator Workflow

Once you have confirmed that you can ssh from your Mac into your Linux system, you're ready to get started. Run Automator and start a new "Application' workflow.

Then we're going to add a single action to the workflow. In the Actions library locate the Utilities group and select it. Then drag the Run Shell Script action to your workflow.

Once you've done that, click on the Options link at the bottom of the action. It will reveal some options. Check the Ignore this action's input checkbox.

Finally, replace the cat command with the following ssh command:

ssh -Y username@linux_hostname gksudo -g synaptic &

Where username is your username and linux_hostname is your Linux host's hostname. Don't forget the & at the end. Then save your workflow as Synaptic. You can click the run button if you'd like to test it out.

Once you're satisfied with it, quit out of automator. Now you can run the workflow directly from OSX. Locate it in Finder and double-click on it.

That's it. You can create any number of workflows for individual Linux apps. I have one for LibreOffice, one for synaptic, one for Eclipse and one for gnome-system-monitor.

Optional - Set icon for your workflow

If you're like me, the generic Automator icon on my Synaptic workflow is a little annoying. You can set the icon to whatever you'd like. Here's one way to do it.

First, obtain a copy of the synaptic.png file from your Linux machine and place it on your Mac. I use the following command from a terminal:

scp steve@ubuntu32.local:/usr/share/pixmaps/synaptic.png ~/Desktop

This puts the synaptic.png file on my Mac's desktop. Then I double-click the synaptic.png file. It should open up in Preview. In preview, I select Edit->Select All, then Edit->Copy.

Next, I control-click on the Synaptic automator workflow and select Get Info. This brings up an informational dialog.

In the top-lefthand corner of the dialog, there is a small Automator icon. I click on it, causing it to glow blue-ish. While it's glowing, I press Command-V.

Finally I close the dialog. I also delete the synaptic.png file as I no longer need it. Now My synaptic Automator workflow has he official synaptic icon.

I hope you found this posting helpful.

Automated python unit testing, code coverage and code quality analysis with Jenkins - part 3

2012-04-27T13:03:00.000-04:00

This is the third and final posting in a series. Check here for the first posting and here for the second posting.

In this posting I will be explaining how to automate the unit testing, code coverage and code quality analysis that we did manually in the first two posts.

As the title of this series suggests, I use Jenkins for this automation (also known as continuous integration - ci). In keeping with the theme of using apt for everything, I will install Jenkins via apt. That being said, in my real job, I have downloaded a jenkins.war file from the Jenkins web site and I start it up from the command line.

Install and start Jenkins

Jenkins appears in the default apt that ships with Ubuntu, but it's a really old version... so old, in fact, that it won't run the plugins that we'll need in order to do the code quality and code coverage analysis. To get around this, refer to these instructions on the Jenkins web site to update your apt sources and get the latest version Jenkins via apt.

Here's a quick screen-shot of the instructions you'll need to follow:

Once you've got Jenkins installed, check to see if Jenkins is running. If it isn't start it.

sudo /etc/init.d/jenkins status
sudo /etc/init.d/jenkins start

Once Jenkins is up and running, let's pull it up in a browser. It will be listening on port 8080.

Install some needed Jenkins plugins

Jenkins has lots of plugins that will allow it to do lots of interesting things. We will be needing a few of them. To install plugins, click on the Manage Jenkins link from the list of links on the left-hand side of the page. Next click on the Manage Plugins link from the resulting page. Finally click on the Available tab from the Plugin Manager page.

This will show you a list of all the available plugins that Jenkins can use. Find and select (check the checkbox) for the following plugins:

Jenkins Cobertura Plugin
Jenkins GIT plugin
Jenkins Violations

After you've check the checkboxes, click the Download now and install after restart button at the bottom of the page.

Jenkins will install the selected plugins and then take a moment to restart. After Jenkins restarts we need to make a small configuration change in order for git to work properly. When Jenkins is done restarting, click the Manage Jenkins link from the left-hand list of links. The click Configure System from the resulting page. This will bring up a global Jenkins configuration page. Scroll down (past the Git section) to the Git Plugin section. Enter values in the Global Config user.name Value and Global Config user.email Value fields. Finally, click the Save button at the bottom of the page.

Our first job

When Jenkins comes back, click the New Job link from the left-hand list of links. This will bring you to a page where you can enter a job name and select a job type. Enter Project1 for the job name and select Build a free-style software project, then click the OK button.

This will bring you to the Configure page for your new job. This is where you tell Jenkins WHAT, WHEN and HOW to do. Let's get started!

1 - Where to get the source

Under the Source Code Management section, click the Git radio button. This will expand more options. For this project we'll be keeping things simple. Simply enter the path to your git repository. Mine's at /home/steve/dev/project1 (this presumes your git repo is on the same box that Jenkins is running on).

2 - When to run the job

Next go to the Build Triggers section and check the Poll SCM checkbox. In the resulting Schedule field, enter 5 *'s. (* * * * *) This makes Jenkins check the git repository for changes once a minute, every minute of every day. If it finds a change in the repository, it will "do a build". We'll be configuring "the build" to run the unit tests and code coverage and quality tests for us.

BTW, for larger Jenkins implementations (more than 10 or so Jenkins jobs), the Poll SCM option is a terrible idea. I can show you another way using git hooks if there is interest. For now, though, let's just use the Poll SCM option since this is just an example project.

3 - What to run

Next, let's configure the job to do what we want it to when it detects a change in the git repository. In the Build section, click the Add build step button and select Execute shell from the resulting popup. In the resulting Command field, enter the following text:

PYTHONPATH=''
nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive
python -m coverage xml --include=project1*
pylint -f parseable -d I0011,R0801 project1 | tee pylint.out

Let's review what's going on here. When the Jenkins job detects that a source code change has occurred in the git repository, it will clone a copy of the repository and then run the above script against the clone.

You'll notice that the nosetests command has become a little more complex. I've added a fair number of command-line arguments to it. This is to make sure that the code coverage runs over all the code, not just the code that's involved in the unit tests. It also ensures that nosetests write's it's output to a file that Jenkins can interpret.

When nosetests runs the code coverage, it generates a .coverage file. Jenkins can't read that. The third line of the script (python -m coverage xml...) converts the .coverage file to an xml format that Jenkins Cobertura plugin can read.

The last line (pylint...) runs pylint on the project and outputs it in a format that the Violations Jenkins plugin can read. I also have it disabling a couple warnings that I don't care to know about. (You can customize this all you want BTW).

4 - Interpret the results

Not only can Jenkins detect changes and run our tests. Given the correct plugins are installed (which we did at the beginning), it can interpret the results and display them in charts/trees/etc. For me this is the best part. All the other stuff, I could have done with some clever scripting and cron.

First stop is coverage. In the Post-build Actions section click the Public Cobertura Coverage Report checkbox. Then in the Cobertura xml report pattern field, enter coverage.xml.

This tells Jenkins to read a file named coverage.xml that will contain testing code coverage information in it. The line listed below (from the build script in step 3) is what creates this file:

python -m coverage xml --include=project1*

Next, click the Publish JUnit test result report checkbox and enter nosetests.xml in the Test report XMLs field.

This instructs Jenkins to interpret the a file named nosetests.xml that the nose command from creates:

nosetests --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=project1 --cover-inclusive

Finally, check the Report Violations checkbox and enter **/pylint.out in the pylint field.

This instructs Jenkins to interpret the pylint.out file that is generated by the pylint command from the build script:

pylint -f parseable -d I0011,R0801 project1 | tee pylint.out

At this point, you're done configuring the job. Click the Save button. Now you're ready to give it a go!

Run the job

Click the Build Now link on the left-hand list of links. This will manually kick off a run of the job. If no errors. You should see a screen that looks like the one below. If not refresh your page.

Not very interesting, is it? That's because we currently have 100% coverage and no test failures! Let's add some code (without tests) and see what happens.

Update the code

Add the following code to your ~/dev/project1/project1/authentication.py file. Then save and commit it.

def logout():
print 'You are now logged out.'

No go back to your Jenkins window and refresh the Project1 page. You may need to wait a minute. Remember, Jenkins only checks the git repository for changes once a minute. You should see some new info. The top graph is your code coverage. It indicates that you've dropped from 100% line coverage to 92%. This makes sense because we added function, but no test for it.

The second graph is your unit test pass/fail trend. It's all blue because all 3 of our tests are still passing. We'll break that next by adding a test for logout that fails. :-)

The third graph is your code quality report. It's indicating that we had 0 issues in the first build, but now have 1 issue in the second build.

You can get more info by clicking on the graphs. For example, let's drill into the code coverage graph.

You can browse even deeper. Click on the project1 link at the bottom.

Then click on the authentication.py link at the bottom.

As you can see by the red line at the bottom, the print statement at the bottom is never exercised by a unit test. Hmmmm. We'll need to fix that.

Add a failing test

Add this code to the ~/dev/project1/tests/authentication_tests.py file and then commit it to git:

def test_logout(self):

"""Test the logout function...badly."""

self.assertEqual(0, 1)

Wait a minute and then go back to the Jenkins Project1 page. It should now have a third build reporting the test failure.

That's it. Hope you found this helpful. Happy coding.

Automated python unit testing, code coverage and code quality analysis with Jenkins - part 2

2012-04-20T12:03:00.000-04:00

This is the second posting in a series. Check here for the first posting.

Where we gonna do this?

First let's create a directory and repository to house the source code. Enter the following commands:

mkdir -p ~/dev/project1
git init ~/dev/project1
cd ~/dev/project1

It should go something like this:

Next, I always add a .gitignore file to my repository so unrelated files aren't tracked in source control. Mine contains the following:

*.swp
*.pyc
tags
.coverage

Once I've created this file, I add it to the git repository with the following commands:

git add .gitignore
git commit -m"Initial commit. Added a .gitignore file."

It should look like this:

Let's get some source code in place

Before we get too ahead of ourselves, let's pause for some quick explanation. I don't like to mix my application code with my unit test code. I keep them separate. My application code will eventually need to be deployed somewhere, but the unit test code doesn't. To that end, I put the application code in one python package named after the project and the unit test code in another package called "tests". So let's create the two packages with the following commands:

mkdir project1
touch project1/__init__.py
mkdir tests
touch tests/__init__.py

This might not be a bad place to do a quick commit to source control:

git add .
git commit -m"Added two blank packages: project1 and tests"

Again, should look like this:

You could run the test runner, coverage and code analysis tools now, but the output wouldn't be very interesting. Instead, let's move on and add a failing test.

The first test

If you're a TDD type developer, this shouldn't come as a surprise. We'll write a test first. Then write some code in order to get the test to pass. In the tests directory, create a file named "authentication_tests.py" and place the following code in it:

#!/usr/bin/env python
"""Unit tests for the project1.authentcation module."""

from unittest import TestCase
from mock import patch
import project1.authentication as auth

class StandAloneTests(TestCase):
"""Test the stand-alone module functions."""

@patch('__builtin__.open')
def test_login(self, mock_open):
"""Test the login function."""
mock_open.return_value.read.return_value = \
"george|bosco\n"
self.assertTrue(auth.login('george', 'bosco'))

This is a test the will test a function named login that will reside project1.authentication module. If you run it now, it should fail because we haven't created an authentication module yet, much less a login function. To run the tests, simply type "nosetests" from the project root directory. Nose will scan your whole directory structure looking for unit tests to run.

Let's get the test to pass

Before we can move on to the topics of coverage and quality, we need to get the test to pass. To do this, create a file in the project1 directory named "authentication.py" and place the following code in it:

#!/usr/bin/env python

"""This module provides functions for authenticating users."""

def login(username, password):

try:

user_file = open('/etc/users.txt')

user_buf = user_file.read()

users = [line.split("|") for line in user_buf.split("\n")]

if [username, password] in users:

return True

else:

return False

except Exception, exc:

print "I can't authenticate you."

return False

Now when you run the test, everything should be hunky-dory.

Code Coverage

You're probably thinking things look pretty good right about now. Let's run the the code coverage tool to see where we stand. To run a coverage test, use this command: "nosetests --with-coverage --cover-package=project1". You'll find that we only have 67% coverage and that our test doesn't exercise lines 12-15 of our login method. Yikes.

You see... In our single test, we're only covering the happy path(tm). That is, we only check to see what the function does when a user enters a valid username/password. We don't check what happens when they enter an invalid username/password. Additionally, we don't check to see what the function does when it can't read the user file at all. We would need to add more tests for that. Edit the tests/authentication_tests.py file and make it look like this:

#!/usr/bin/env python

"""Unit tests for the project1.authentication module."""

from unittest import TestCase

from mock import patch

import project1.authentication as auth

class StandAloneTests(TestCase):

"""Test the stand-alone module functions."""

@patch('__builtin__.open')

def test_login_success(self, mock_open):

"""Test the login function when things go right."""

mock_open.return_value.read.return_value = \

"george|bosco"

self.assertTrue(auth.login('george', 'bosco'))

@patch('__builtin__.open')

def test_login_bad_creds(self, mock_open):

"""Test the login function when bad creds are passed."""

mock_open.return_value.read.return_value = \

"george|bosco"

self.assertFalse(auth.login('george', 'marbleRye'))

@patch('__builtin__.open')

def test_login_error(self, mock_open):

"""Test the login function when an error happens."""

mock_open.side_effect = IOError()

self.assertFalse(auth.login('george', 'bosco'))

Notice I've added two more tests: one to test invalid creds and one to test an exception occurring. Now we have 100% coverage.

Code Quality

So we now have tests that pass and 100% coverage. That's awesome! Everything's under control now. What did you say? "How about the quality of that code?" Let's check it out and see. To take a look at code quality use the following command: "pylint -r n project1". Ah. Well. We have a few things to clean up. Pylint is notoriously whiny. That said, the things it's complaining about in this case are legitimate if you're concerned about being a good python developer.

So let's get ourselves into compliance. First let's add a docstring to the project1 package. Edit project1/__init__.py and make it look like this:

#!/usr/bin/env python

"""

Project1 is Steve's example project for the blog.

It contains sample code here and there.

"""

That takes care of the first pylint complaint. Next let's fix up the project1/authentication.py file. Edit it and make it look like this:

#!/usr/bin/env python

"""This module provides functions for authenticating users."""

def login(username, password):

"""Log the user in."""

try:

user_file = open('/etc/users.txt')

user_buf = user_file.read()

users = [line.split("|") for line in user_buf.split("\n")]

return [username, password] in users

except IOError:

print "I can't authentication you."

return False

Now when you run pylint, you should have no complaints:

If you want a more detailed report from pylint, try using the command, "pylint -r y project1" instead. Used that way, it's very much more verbose.

Let's clean up

So here's what we've accomplished so far. We created a project and source code repository. We've created a sample piece of code and some tests for it. We've ensured that we have 100% code coverage and we've analysed our code quality and found it to be in good shape. Let's pause and make sure that all our work is committed to source control.

That's it for now. In the next post, I will walk through installing Jenkins and configuring it to monitor your source code repository. It will notice when you commit new code and automatically run your unit tests, code coverage and code quality analysis. It can be configured to notify you when things don't look so good. It can also display nice graphs over time of how your code is progressing. Stay tuned.

Automated python unit testing, code coverage and code quality analysis with Jenkins - part 1

2012-04-20T09:11:00.000-04:00

As a software development consultant, I do a fair amount of python development and even though python is a scripting language, I feel that it's still worthy of good coding practices, namely: unit testing, code coverage analysis and code quality analysis. In this series of blog postings I will describe my python development environment and how I engage the tools.

In this specific blog post, I will talk about the tools I use and how to get them installed/configured. In future postings I'll set up a sample code base with sample unit tests and run the tests manually. Finally in the last posting, I'll talk about using Jenkins to automate the testing.

Let's get started
I tend to do most of my python development on the Linux platform. I love python's cross platform features. While most of my client implementations actually run in Microsoft Windows, I still do my development in Linux.

For the purposes of this blog series, I will be using Ubuntu 11.10, the 64 bit version. This distribution/version isn't necessary, but that what I'm using. Python 2.7 comes with the default installation. If for whatever reason python isn't installed on your system, just type "sudo apt-get install python" from the bash prompt:

Source control

When we move on to automating the testing/analysis, you'll need to have some source control in place. I use git. Even if you aren't planning on doing the automation part, source control is still a good idea. Again, you can get it with a single command: "sudo apt-get install git"

Tools of the trade
There are a number of testing, quality and coverage tools out there for python. These are the ones I use:

Unit testing: unittest - Comes with python. It's part of the standard library.
Mocking: mock by Michael Foord - It's available in apt.
Test runner: nose - Also available in apt.
Code coverage: coverage - Also, also available in apt.
Code quality: pylint - Also, also, also available in apt.

So to get all these great tools you simply need to type one command: "sudo apt-get install python-mock python-nose python-coverage pylint"

That's it. If you're following along, you should have a fully ready python development environment. Tune in to the next posting where I'll create a repository, some sample code and sample unit tests. Then I'll manually run the testing, coverage and quality analysis tools against it.

Copy and paste from vim

2012-04-13T11:19:00.000-04:00

I've added the following to my .vimrc:

It allows me to highlight text in a file and then hit '\c' to copy the highlighted text to my X11 clipboard in Ubuntu.

It also allows me to just hit '\p' to paste from the X11 clipboard to the vim buffer.

My .vimrc in OSX has the following to provide the same functionality when I'm on the Mac: