Fortunately, this year’s PyCon had a great talk from Chris McDonough on how to get started with Python’s debugger, PDB. The friendly people at Stockholm’s Python meet-up suggested it.

I made notes while watching, to remove excuses from future me on launching pdb.

Where you want to have the debugger start, add the following code:

import pdb; pdb.set_trace()

This drops you into a PDB prompt. This is more helpful than putting in yet more
print statements.

Some helpful commands once you’re in:

• l - list code
• args - arguments to current function
• p <var> - print a var
• pp <var> - pretty print a var
• n - next
• s - step
• w - where (stack trace for current position)
• h - help

While in PDB, you can evaluate python code. The evaluated code won’t impact the running program.

For a bonus trick if you’re using Emacs, try out pdb track mode. Launch your python process to be debugged in an emacs shell (M-x shell). Stepping through the code in pdb will track with a source code buffer.

This worked out of the box with my emacs config. Your milage may vary.

I may actually start to enjoy debugging now!

Rspec has a great addition in autospec, which automatically re-runs specs that are changed or have the associated code changed. This model of working has a long history in the ruby community, with autotest.

I wanted a similar thing for python/nose, but hadn’t had much luck.

The solution is py.test, with some plugins added.

This was suggested to me at the Stockholm Python user group, and I am very grateful. My testing life is now much simpler.

To get started, try something like:

$ sudo pip install pytest pytest-xdist pytest-cov

The simple way to get started is to watch your tests folder:

py.test -f test/

Now as you change files, the relevant tests will automatically be re-run. Perfect to display on that second monitor.

As I was working on some legacy code, I was looking to improve the test coverage, so wanted to see how that was going with pytest-cov:

py.test -f --cov package.name test/

Adding the package.name for your main package means you won’t generate coverage for libraries you are using. Makes the output simpler.

Still not quite happy, it wasn’t showing me which lines I needed coverage on. No problem:

py.test -f --cov package.name --cov-report term-missing test/

For comparison, here is how to get similar results from nose, but without watch support:

nosetests test/
nosetests --with-coverage --cover-erase --cover-package=package.name test/

One hurdle I had with py.test was working with Twisted based test-cases. You need to ensure that you use a version of py.test later than 2.0. Debian squeeze does not package this by default, so use pip to install.

The annoyance is if you change the code so it doesn’t terminate. This still needs a context switch to kill off the tests and re-run. Keeps me focused on getting it right the first time.

pytest-xdist includes lots of really cool options, such as distribution of tests across multiple cores, versions, hosts and platforms.

I’ve since discovered this comprehensive summary of the options available. I’m happy I found this after selecting py.test, or I’d still be stuck in evaluation mode rather than writing code.

Update: check out sniffer if you want a more generic watcher framework.

There is some truth in that, as much of my day is spent typing. Writing code, taking notes for meetings, email, documentation, writing blogs posts.

It would make sense to become very good at typing.

The basics are often overlooked. You can get a lot of benefit from paying attention to them.

For typing, it comes down to three main areas:

1. Accuracy
2. Speed
3. Touch typing

Touch typing is perhaps the most important aspect of this. You need to be able to type without looking at the keyboard. If you get really good, you shouldn’t need to even look at the screen to confirm what you’re typing.

Accuracy and speed can be built together. Focus on accuracy to start with, and then stretch for speed. There is no point going faster if you can’t maintain a high accuracy.

I found two websites to be helpful in my quest for improved typing skills:

• Keyzen — a site focused on improving typing for programmers.
• 10 fast fingers — general typing improvement with natural language exercises.

Of course, the other thing to do to improve typing is to do lots of typing. If you are writing lots of code, email or documents, then focus on your typing while you are doing it.

It helps to have a good environment for typing, such as a comfortable keyboard, good posture, but you know this already. Add it to the list of things to focus on.

Also, make sure that you are using a program that can keep up with you as you type. I find MS Word to lag when in full WYSIWG mode, whereas emacs and vi always manage to keep up!

Update: You may also want to check out Typespeed.

In any field, if you expand your view to know all the ecosystem around you it’s beneficial.

—Davide Casali

If you are a developer, a product owner, or manage a development team, much of the input is going to be about product design. It is very useful to understand a bit about a good design process, so you can better ask questions and help refine the design based on technical constraints.

About Face, by Alan Cooper, is a great book to start with.

I highlighted much of it on my Kindle!

Here are some quotes:

Good design makes users more effective.

Customers, although they might be able to articulate the problems with an interaction, are not often capable of visualizing the solutions to those problems.

One of the most powerful tools designers bring to the table is empathy

One of the most dangerous practices in product development is isolating designers from the users because doing so eliminates empathic knowledge.

narrative is also one of our most powerful creative methods.

What would a helpful human do? What would a thoughtful, considerate interaction feel like?

All ideas are broken down into discrete sections, with lots of detail and examples on how to implement this with your customers.

Goal-Directed Design combines techniques of ethnography, stakeholder interviews, market research, detailed user models, scenario-based design, and a core set of interaction principles and patterns.

This process can be roughly divided into six phases: Research, Modeling, Requirements Definition, Framework Definition, Refinement, and Support

The book discusses the mindset required to be a good designer:

in anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away

Next time you find yourself crowing about what cool interaction you’ve designed, just remember that the ultimate user interface for most purposes is no interface at all.

The first part of the book focuses on processes to elicit a good design. The latter part of the book covers specific examples found in user interface design and attempts to encode common patterns.

I found the first section much more valuable to how I think about designing a new product. The latter parts I felt were obvious, although that is perhaps their importance.

Well worth a read.

Note: Bret’s Magic Ink is a useful counterpoint to About Face.

The core message of how to position a product is becoming more relevant as the marketplace continues to crowd.

In our overcommunicated society, the name of the game today is positioning. And only the better players are going to survive.

The book draws on a wide range of examples of different products, where a product could be a type of beer, the Catholic Church or your own career.

The key message is that your products position is determined by how it fits into the consumer’s mind. All the things that matter to you, to your company, are irrelevant if you can’t create a distinct position to separate you from everything else.

Experience has shown that a positioning exercise is a search for the obvious. Those are the easiest concepts to communicate because they make the most sense to the recipient of a message.

The authors are quite scathing about “me too” products that attempt to beat out a competitor by going head-to-head. Many examples are provided within the book.

The suicidal bent of companies that go head-on against established competition is hard to understand.

To repeat, the first rule of positioning is: To win the battle for the mind, you can’t compete head-on against a company that has a strong, established position. You can go around, under or over, but never head to head.

The leader owns the high ground. The No. 1 position in the prospect’s mind. The top rung of the product ladder.

The book covers a range of detail levels, and dips into more detail on key areas such as product naming:

As a guide, the five most common initial letters are S, C, P, A, and T. The five least common are X, Z, Y, Q, and K. One out of eight English words starts with an S. One out of 3000 starts with an X.

The authors are often quite blunt as to how they see things. This is a refreshing change from most business books:

Creative people often resist positioning thinking because they believe it restricts their creativity. And you know what? It does. Positioning thinking does restrict creativity.

I gained a lot of insight into how products have succeeded, or not, by reading this book. The examples are clear and will easily map to situations you are experiencing.

I recommend Positioning for anyone involved in creating something new. This is a book that I will re-read.

Emacs

The switch has been very positive. The learning curve has been relatively steep, as my expectations from a text editor are quite high.

Emacs strength (and weakness) is that it is incredibly extensible. Where I’m finding Emacs a win over Vim is that I don’t have to leave Emacs to get things done. With Vim, I tend to use more of a mix of terminal windows and the editor.

I began by using the starter kit to get going with Emacs configuration. This made it quicker to get moving, but added a lot of things I ended up not needing. After getting more comfortable with elisp, I started from scratch and rebuilt my emacs.d folder.

To jump in quickly, I also purchased the tutorial video from Peepcode. This certainly helped as emacs is a mental shift coming from Vim.

The big benefit I have found with Emacs is the extension packages. These can be installed from the ELPA repository, and include a range of different modes.

Some of my favourite modes include:

• paredit - essential for any lisp, it ensures your brackets match.
• deft - simplified note taking. (I it sync via Dropbox)
• magit - comprehensive Git workflow within Emacs
• markdown-mode - my current default for writing notes, although I’m leaning towards org-mode now.
• org-mode - highly capable note taking mode, with export options to everything. You can use it to write a book, create slides, or manage your todo list.

On the Mac, I’ve been using Aquamacs for most of my text editing, and used macports to install the command line client.

I have yet to try Emacs on Windows, as I haven’t been using Windows much at all. There is a release available here.

After cleaning up my .emacs.d configuration, I’ve now started using Emacs on Linux servers I use regularly. For temporary servers, I’ll fallback to Vim as Emacs is often not installed.

If you want to improve your emacs skills follow @emacs_knight.

Vim

To get started with Vim, it is worth reading The Vim Learning Curve is a Myth.

Over the years, the popular mythology around vim has become that it’s insanely difficult to learn; a task to be attempted by only those with the thickest of neck-beards. I’ve heard dozens of times from folks who are convinced it will take them months to reach proficiency.

These beliefs are false.

My feeling is that Vim is unrivalled for the simple task of text editing. Even after a day or two of learning, you will be faster.

Where things get a bit more complicated is when you start to realise that text editing isn’t the whole story for a text editor.

Platform support is superb. The first thing I do to a Windows machine is install Vim. It is rare for an application with a Unix heritage to be so comfortable on Windows.

Historically, I hadn’t been a fan of the graphical version of MacVim. This is something that is much better in recent releases. Load times are much improved.

Integration into external tools is where I feel Vim is lacking a bit. With the number of developers migrating from TextMate to Vim, this gap is being rapidly addressed. However, Vim isn’t as extensible as some other editors.

I really noticed this when trying out Clojure programming. If you are dealing with a REPL based environment, Vim has a way to go.

Closing Thoughts

If you aren’t using either of them, it’s not too late to start. You can’t make a bad choice.

If you are already using either Emacs or Vim, enjoy!

Both are great editors that allow you to be incredibly productive at working with text. Try learning a new feature each week. You use your text editor so often that a small improvement is a major payoff.

The tricky part was calculating dates in shell. This is what I ended up with:

#!/bin/sh

MONTHS_AGO=3
DATE_AGO_EPOCH=$((`date +%s` - $MONTHS_AGO * 31 * 24 * 3600))
OSTYPE=`uname`
FORMAT='+%Y-%m-%d'

if [ "Linux" = $OSTYPE ] ; then
  DATE_AGO_ISO=`date -d "1970-01-01 00:00 UTC + $DATE_AGO_EPOCH seconds" $FORMAT`
else
  DATE_AGO_ISO=`date -r $DATE_AGO_EPOCH $FORMAT`
fi

DB_CMD="db.items.find( { publish_date : { $lt : \"$DATE_AGO_ISO\" } } ).count()"
# DB_CMD="db.items.remove( { publish_date : { $lt : \"$DATE_AGO_ISO\" } } )"

echo $DB_CMD | mongo pz

This now provides a simple way of purging out old items that can be called from cron.

Previously at BackType, Nathan Marz has now built Storm into Twitter and then open sourced the platform.

Storm opens up a lot of possibilities, by bringing real-time distributed processing together in an elegant way. And it is built in Clojure!

Some useful links:

• Rationale
• Slides - detailed presentation on Storm by Nathan
• Starter project - a few basic demo use cases
• Mailing List
• Taking the Emmys by Storm - an hour long video of how to use Storm to aggregate data from Twitter in realtime to spot trends.

While playing around I wanted to try from Clojure (most examples are in Java). There is a great DSL for Clojure that makes using Storm super easy. I forked the storm-starter project to add examples based on a Gist from Nathan.

Nested Classes

Java allows for classes to be nested. However, Clojure doesn’t provide an obvious way of referencing them.

For example, Hadoop’s SequenceFile includes SequenceFile.Writer and SequenceFile.Reader.

To access these in Clojure, the period is replaced by a dollar sign.

For example:

(ns example
  (:import [org.apache.hadoop.io SequenceFile$Writer]))

...
  writer (SequenceFile$Writer. fs conf path
                               (class Text)
                               (class VectorWritable))
...

This thread on the mailing list has some more details.

Generics

From the Java code, generics add annotations about the type. However, at the JVM level, generics are nowhere to be seen. It turns out that generics are a compiler level feature only.

What this means for Clojure is that you can safely ignore them, provided that you honour the types that are included in the generic.

Thus the following Java:

List<NamedVector> apples = new ArrayList<NamedVector>();

Can be written in Clojure:

(def apples (ArrayList.))

The responsibility is then on the programmer to ensure the types put into the collection.

This thread on the mailing list has some background.

Primitive Arrays

I started out looking at how to construct a double[] to pass to DenseVector.

Turns out that there are a number of things to be wary of when constructing primitive arrays. Bradford Cross has written this up with examples in a blog post.

The considerations I took away were:

1. double-array function to migrate a Clojure data structure
2. make-array with Double/TYPE to pre-allocate the array
3. Type hints should be considered
4. Odd things are likely to be happening with type coercion, so check run time performance.

Summary of the tests:

1. Will your strategy beat the market?
2. Does your strategy tap a true source of advantage?
3. Is your strategy granular about where to compete?
4. Does your strategy put you ahead of trends?
5. Does your strategy rest on privileged insights?
6. Does your strategy embrace uncertainty?
7. Does your strategy balance commitment and flexibility?
8. Is your strategy contaminated by bias?
9. Is there conviction to act on your strategy?
10. Have you translated your strategy into an action plan?

Each test includes a more detailed description, and an example of a company or industry where it is particularly relevant. There are also links through to relevant articles or books.

Although aimed at corporate strategy, this is an interesting list of points to consider at a personal or project level as well.