The harmonization for the major scale is pretty simple; you stack notes vertically following every other note in the scale:

In modern music it’s fairly common to use non-traditional scales to generate new and interesting harmonies. For instance, in the example bellow I have a 7-note scale that is very different from the good ol’ major scale. Right after the scale we have harmonizations for every 3, 4, and 5 notes of the scale. Also notice that we break the thing about having the interval in which we pick notes be the same of the horizontal notes (that is, having chords in thirds, fourths, and so on). This leads to very interesting harmonies.

By the way, this is one of the reasons I got into programming in the first place; to generate material to use in my compositions (using Pascal back in the 90′s, if you must know).

So, in order to harmonize every scale we need to know how many scales exist and the answer will vary depending to whom you ask. In a way a scale is an ordered set of notes, so I used the 129 pitch class sets with cardinality greater than 5. Here is a complete list of all pitch class sets. Some people will find that using prime forms is too condensed and doesn’t take in account a bunch of scales, but it’ll do for the purpose of this post.

The code to generate the harmonizations is actually very simple (with the help of my in-progress python library). We harmonize the first note of a scale with harmonize_first_note and use it in a list comprehension to harmonize every note of the scale. The helper function rotate_set will generate every rotation of a list.

def harmonize_first_note(scale, interval, size):
    i = (interval - 1)
    indexes = [x % len(scale) for x in range(0, size * i, i)]
    return [scale[index] for index in indexes]
def harmonize_scale(scale, interval, size=3):
    scales = musiclib.rotate_set(sorted(scale))
    return [harmonize_first_note(scale, interval, size) for scale in scales]

Here’s how we use it: A scale is defined as a list of integers (we are using integer notation). Let’s put the major scale in a variable:

scale = [0, 2, 4, 5, 7, 9, 11]

The result of harmonize_first_note will be, as expected, just the first chord (C major) as a list of integers:

harmonize_first_note(scale, 3, 3)
>>> [0, 4, 7]

And the result of harmonize_scale will be a list with the harmonization of every chord in the scale. In the next example I’m using 4 as the chord size to generate tetrads:

harmonize_scale(scale, 3, 4)
>>> [[0, 4, 7, 11], [2, 5, 9, 0], [4, 7, 11, 2], [5, 9, 0, 4],
     [7, 11, 2, 5], [9, 0, 4, 7], [11, 2, 5, 9]]

Finally, lets generate tetrads separated by fourths instead of thirds:

harmonize_scale(scale, 4, 4)
>>> [[0, 5, 11, 4], [2, 7, 0, 5], [4, 9, 2, 7], [5, 11, 4, 9],
     [7, 0, 5, 11], [9, 2, 7, 0], [11, 4, 9, 2]]

The code to generate the LilyPond file is boring, so I won’t talk about it here. You can see it in the repository.

Generating harmonizations for every scale is interesting and all, but it start to get interesting when we can filter things. For instance, I may want to generate harmonizations for the scales that have more than 4 notes and have consecutive intervals greater than 1 semitone. This can be accomplished with filter_sets. This is how I’d express the previous example:

filter_sets(lambda intervals, size: size > 4 and 1 not in intervals)

The function filter_sets accepts an anonymous function with a list of the consecutive intervals in a set and the size of a set as parameters. If the condition in the body of the anonymous function is met the set is returned:

def filter_sets(condition):
    sets = {}
    for forte, pc_set in musiclib.PC_SETS.items():
        intervals = musiclib.intervals(pc_set)
        size = len(pc_set)
        if condition(intervals, size):
            sets[forte] = pc_set
    return sets

This is an example of how high-order functions can be used to abstract code. Also, the built-in function all is very handy to chain conditions. Let’s say I want to get scales that have more than 4 notes and have only 1 consecutive halftone and 1 consecutive whole tone:

filter_sets(lambda i, s: all([s > 4, i.count(1) == 1,
                              i.count(2) == 1]))

Since I want to filter scales by interval content, it makes sense to write a small function to help me. cond_interval_count abstracts the previous code:

def cond_interval_count(count_list):
    return lambda i, s: all([i.count(interval) == count for
                             count, interval in count_list])

So if I want the scales that have no consecutive halftones and only 1 consecutive whole tone I can write it as:

cond = cond_interval_count(((0, 1), (1, 2)))
filter_sets(cond)

There are a few things that can be improved:

• Use some pitch spelling algorithm, like (Meredith, 2003) so we won’t have chords like C, D#, G
• Have better functions for filtering. The function I’m using looks for consecutive intervals but it would be better to search for every interval in the Pitch Class Set.
• Handle octaves better in the LilyPond generation. Some notes would be better on a different octave:
  

This post only scratches the surface of all things we can do. For anything more complicated than that I’d use music21, a nice and professional Python library for computer-aided musicology. But it’s interesting too see how far we can go with a few dozen lines of pure Python code and the use of nice abstractions.

If you have an interest in this kind of thing, there are lots of Python packages for music programming and Notes from the Metalevel: An Introduction to Computer Composition is a nice book about the topic (but it uses Scheme, not Python for the code).

The advantage of reviewing code on paper instead of the monitor is that I can give my eyes a break and I can annotate more freely, connect things with arrows, draw boxes, etc. I don’t print my code everyday, after all this is not the 80′s, but I like to print the code when I feel it could benefit from some refactoring and re-structuring and my brain could benefit from same change in the medium and even location (I can grab the printouts and go to a park or cafe).

The best way I found to print code is by using the LaTaX listings package (it should be included in any modern LaTeX distribution). This package is intended to include code snippets in papers and books, but it works to generate code listings as well.

These are the options I have in my LaTeX document header:

\documentclass[10pt]{article}
\usepackage[T1]{fontenc}
\usepackage[scaled]{beramono}
\renewcommand*\familydefault{\ttdefault}
\usepackage{listings}

The beramono package is a version of Bitstream Vera Mono modified to work with TeX. Without it we’ll get the default LaTeX mono font:

But I like Bitstream Vera better (it looks even betten on paper):

Listings has support for Python out-of-the-box and has lots of options and features (check the manual to see what listings can do). These are the options I like to customize:

\lstset{
  language=Python,
  showstringspaces=false,
  formfeed=\newpage,
  tabsize=4,
  commentstyle=\itshape,
  basicstyle=\ttfamily,
  morekeywords={models, lambda, forms}
}

Options like tabsize and showstringspace are self-explanatory. The option formfeed=\newpage will replace the formfeed character by the LaTeX command \newpage, creating a new page at that point. Sometimes I put some line feed characters in my code so I can do things like narrowing to a page in Emacs. Emacs shows line feed characters as ^L, as you can see here:

The option morekeywords will add the keywords in the list of keywords recognized by Listings. In my example I added lambda, models and forms (I’ve been working a lot with Django lately), so these keywords will be pretty-printed.

Finally, I use the command \lstinputlisting to include a source code file in the LaTeX file (instead of having to type the whole thing in the LaTeX file). I like to separate each section by a horizontal line and add some space at the end. I define a command to abstract this:

\newcommand{\code}[2]{
  \hrulefill
  \subsection*{#1}
  \lstinputlisting{#2}
  \vspace{2em}
}

So in the end this is how I insert a Python file:

\code{Models}{../testapp/models.py}

And this is the result:

Pretty neat huh? I believe that reviewing my code on paper has save me hours of work and I find listings to be a very good option to print source code. I added a full LaTeX template here.

Do you print your source code? If so, what tool do you use? Let me know in the comments.

Definition

New Rich (NR) to be the owner, not the boss nor the emplayee. “To own the trains
and have someone else ensure they run on time.” (p. 23)

NR examples:

1. employee who gets 90% of work done in 10% of time and use the remaining time to do whatever he likes (e.g. road trips)
2. student who has an online video rental service and makes $5000 a month working 2 hours a week.

Being a NR is to challenge the status quo but is not to be stupid

1. Retirement is a worst-case scenario insurance: and it’s difficult to retire and mantain the same standard of living
2. Interest and energy are cyclical: the best is to have mini-retirements distributed throughout life. The author aims for 1 month overseas for each 2 months of work.
3. Less is not laziness: The point is to do less meaningless work and produce more meaningful result. 80/20 rule.
4. The timing is never right
5. Ask for forgiveness, not permission
6. Emphasize strengths, don’t fix weaknesses: ”Focus on better use of your best weapons instead of constant repair” (p. 34)
7. Things in excess become their opposite: Lifestyle design is not intended to increase idle time, but “the positive use of free time”. (p. 35)
8. Money alone is not the solution
9. Relative income is more important than absolute income: The ratio between money earned and time worked is what define how a person makes. NR make $5000 per hour. (They may earn 10,000 a month and only work 2 hours a month).
10. Distress is bad, eustress is good

Conquer fear by defining it. What’s the worst it can happen from a 1-10 scale?

1. Define your nightmare
2. What steps to repair te damage (even if temporarily)
3. What are the positive outcomes?
4. If you were fired today what would you do to get things under financial control?
5. What are you putting off out of fear? ”What we fear doing most is usually what we most need to do” (p. 46)
6. What is costing you to postpone action?
7. What are you waiting for?

Doing the unrealistic is easier than doing the realistic.

Planning the dreams

1. Create 2 timelines (6 and 12 months) and list 5 things you dream of having, being, and doing considering you couldn’t fail.
2. If you don’t know what to think in 1., consider what you’d do if you had $100 million in the bank and what make you excited to wake up in the morning.
3. Convert each “being” into a “doing”. For example:
   Great cook -> make Christmas dinner without help
   Fluent in Chinese -> have a five-minute conversation with a Chinese co-worker
4. Highlight (star) the 4 most exciting and important dreams from all columns.
5. Determine the cost of these dreams and calculate your TMI (target monthly income) for both timelines. Use the tools on the website.
6. Determine 3 steps for each of the 4 dreams in the 6-month timeline and take the first step now.

If someone has done it, ask for advice.

Elimination

Apply Pareto’s law and beware of Parkinson’s law.

1. If I had an heart attack and had to work 2 hours per day, what would I do?
2. If I had a 2nd heart attack and had to work 2 hours per *week*, what would I do?
3. What time-consuming activities can I remove?
4. Learn to ask “If this is the only thing I accomplish today, will I be satified with my day?”
5. Do not multitask.
6. Go on a low-information diet.
7. Check email twice a day (and latter once)

Automation

• Get a personal assistant
• Eliminate before you delegate
• Learn to delegate
• Each delegated task must be both time-consuming and well-defined
• Request an especific kind of assistant (e.g. “with excelent English”)
• Request an update after a few hours
• Break in small milestones
• Example: “Please reply and confirm what you will plan to do to complet this task”
• Get an assistent even if you don’t need one (to train how to command)
• Think “what tasks can I delegate do a VA?”

Income autopilot

He uses Prosoundeffects as an example.

The product can’t take more than $500 to test, has to lend to automation within 4 weeks and can’t require more than 1 day per week of management.

1. pick an affordable and reachable nich market
2. brainstorm products: explain the product in one sencence (e.g. ipod: “1000 songs in your pocket”)
3. price should be between $50–200: higher prices mean fewer units, less costumers, and lower-maintenance costomers.
4. resell a product
5. License a product
6. Create a product: Information is difficult to replicate (legally), e.g. training DVD for security systems.
7. how to become a top expert

• join 2 or 3 trade organizations
• read the 3 top-selling books
• give one free 1-to-3 hour seminar at a well known university (and/or companies)
• write 1 or 2 articles in a trade magazine
• join profnet

Testing the muse

micro test products with Google adsense

• find competition to determine why my product is better
• test advertisement with ebay auction (cancel auction on the last minute)
• set-up google campaign

1. Market selection
2. Product brainstorm
3. Micro-testing
4. Roll out and automation

Management by absence

1. 1. 0-50 total units of product shipped: do it all yourself
2. more than 10 units shipped per week
   add faq to website and answer questions
   find fulfillment services or “mailing services” or mfsanet.org
3. more than 20 units per week
   outsource
   set up merchant account
4. have fewer options

Liberation

How to negotiate to work remotely.

The first chapter is about building abstractions with functions. I think it’s remarkable that a book for beginners (pretty smart beginners, but still) introduces assignment only in the third chapter (on page 220). I really think this is the way to start a programming course. Probably all students know about mathematical functions and with functions we can talk about things like bound variables, scope, abstraction, composition, and recursion.

A powerful language needs to have the following things to allow the combination of simple ideas to form complex ideas:

• primitive expressions: the simplest entities in a language. Things like numbers and arithmetic operations and functions.
• means of combination: “by which compound elements are built from simpler ones”. Nesting combinations, such as square(2 * square(3 + 7)) are simple means of combination.
• means of abstraction: “by which compound elements can be named and manipulated as units”.

def is the simplest mean of abstraction. The following code creates a function and associates it with a name:

def square(x):
    return x * x

It’s important to make the distinction of the act of creating a function and naming it. We can create a function without a name (an anonymous function) with lambda:

lambda x: x * x

And we can assign it to a variable, giving it a name:

square2 = lambda x: x * x

And, in fact, we can see (with the help of the bytecode disassembler module) that Python will generate the same bytecode for both square and square2:

>>> import dis
>>> dis.dis(square)
  1           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                0 (x)
              6 BINARY_MULTIPLY
              7 RETURN_VALUE
>>> dis.dis(square2)
  1           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                0 (x)
              6 BINARY_MULTIPLY
              7 RETURN_VALUE

Having defined square, we can use it in combinations:

square(2 + 5)
square(square(7 + square (3)))

And, naturally, we can use square as a building block:

def sum_of_squares(x, y):
    return square(x) + square(y)
sum_of_squares(3, 4)
def f(a):
    return sum_of_squares(a + 1, a * 2)

The association between names and values, such as the name of a function, is saved in a place called the environment. Chapter 3 will talk about the environment in greater detail.

Applicative order and normal order

To evaluate combinations we follow a recursive rule (quoted verbatim):

1. Evaluate the subexpressions of the combination.
2. Apply the procedure that is the value of the leftmost subexpression (the operator) to the arguments that are the values of the other subexpressions (the operands).

The substitution model is a simple model to help us understand what happens during evaluation. To evaluate procedures we have the following rule:

To apply a compound procedure to arguments, evaluate the body of the procedure with each formal parameter replaced by the corresponding argument.

Keep in mind that “the purpose of the substitution is to help us think about procedure application, not to provide a description of how the interpreter really works” and SICP presents “a sequence of increasingly elaborate models of how interpreters work, culminating with a complete implementation of an interpreter and compiler in chapter 5″.

In the example below we can see two ways to evaluate the function f we defined previously. The function f is defined in terms of sum_of_squares, which is defined in terms of square, which is defined as the multiplication of a number by itself. In the evaluation method on the left, an expression such as 5+1 is evaluated and applied immediately. This method of evaluation is known as applicative order evaluation (a kind of strict evaluation, used in most programming languages, inclusive Python and Scheme). The evaluation method on the right only evaluates an expression when needed. It’ll fully expand all function calls first, and then evaluate what’s left. This is known as normal order evaluation. Compare the result of both methods in line 5:

# applicative order        normal order
f(5)                       f(5)
sum_of_squares(5+1, 5*2)   sum_of_squares(5+1, 5*2)
sum_of_squares(6, 10)      square(5+1) + square(5*2)
square(6) + square(10)     ((5+1) * (5+1)) + ((5*2) * (5*2))
(6 * 6) + (10 * 10)        (6 * 6) + (10 * 10)
36 + 100                   36 + 100
136                        136

Of course both methods yield the same answer, but this will not always be the case, as we’ll see latter in this post (exercise 1.5) and in this series.

Conditional expressions and predicates

The main point of this sub-section is to show how to write conditional expressions in Scheme by implementing a function named  abs to calculate the absolute value of a number. Since abs is a built-in function in Python,  I’ll use myabs (it’s pretty hard not to think about cheesy late-night infomercials with a function named like this ). The first implementation follows the mathematical definition and uses multiple predicates:

def myabs(x):
    if x > 0:
        return x
    elif x == 0:
        return x
    elif x < 0:
        return x

This is unnecessarily long and can be shortened as:

def myabs(x):
    if x < 0:
        return -x
    else:
        return x

And it can be even shorter by using the ternary operator (new in Python 2.5):

def myabs(x):
    return -x if x < 0 else x

This sub-section also shows logical operators such as and, or, and not. For instance, in Scheme the expression 5 < x < 10 would be written as (and (> x 5) (< x 10)) but the same thing in Python is just 5 < x < 10. Pretty cool, huh?

Exercise 1.4

This exercise asks the reader to describe the behavior of the following procedure. I’ll use the original Scheme code, then I’ll show the equivalent in Python.

(define (a-plus-abs-b a b)
  ((if (> b 0) + -) a b))

This code may look weird at first (and I’m not talking about the parenthesis ), but we can just apply the substitution model to understand how it works. Let’s copy the function’s body:

((if (> b 0) + -) a b)

If b is greater than 0, the conditional expression will return the + (addition) function, otherwise it’ll return the – (subtraction) function. In Scheme + and – are functions, just like sqrt. Let’s suppose that b is greater than 0 and the conditional expression will return +. We substitute + for the conditional expression:

(+ a b)

The resulting expression is just the sum of a and b. This kind of thing is possible because in Scheme functions are first-class; we can create them at runtime, pass them as arguments to other functions, and return them as values.

Functions are also first-class in Python, but + and – are not functions. We can access Python’s basic operators with the operator module. For instance, operator.add(2, 2) is equivalent to the expression 2 + 2. So, we can write the Scheme code above in Python as:

from operator import add, sub
def a_plus_abs_b(a, b):
    return (add if b > 0 else sub)(a, b)

Exercise 1.5

This exercise asks the reader to describe the behavior of the following code if the interpreter uses applicative order evaluation and normal order evaluation.

def p():
    return p()
def test(x, y):
    return 0 if x == 0 else y
test(0, p())

In applicative order evaluation, the interpreter will enter in an infinite loop, regardless of the value of test‘s first argument, because both operands will be evaluated before the function is called (line 7). With normal order evaluation, the interpreter will evaluate only what is necessary, so if the first argument of test is 0, it’ll return 0 because the second argument will not be evaluated.

Example: square roots by Newton’s method

One way to calculate square roots is by using Newton’s method of successive approximations. We start with a guess g for the square root of a number x and calculate a better guess by averaging g with , or:

The following procedure tests if the guess we have is good enough for the number x (the radicand) we want to compute the square root. If not it’ll keep trying to improve the guess until it’s good enough:

def sqrt_iter(guess, x):
    if is_good_enough(guess, x):
        return guess
    else:
        return sqrt_iter(improve(guess, x), x)

To improve a guess we average it by the number x divided by the guess:

def improve(guess, x):
    return average(guess, x/guess)

Average is easy enough to define:

def average(x, y):
    return (x + y)/2

Of course, we need to define is_good_enough. A basic test is to see if the square of the guess minus the original number x is smaller than some threshold (we use 0.001). This is not a good test for very small and large numbers (see exercise 1.7 in the book) but will do for a first try:

def is_good_enough(guess, x):
    return abs(square(guess) - x) < 0.001

Finally, we need to start at some point. We begin with 1.0 as a guess:

def sqrt(x):
    return sqrt_iter(1.0, x)

Procedures as black-box abstractions

One problem with our implementation of sqrt is that functions like is_good_enough, sqrt_iter, and improve are cluttering the global namespace. It’s very important to decompose a problem in sub-parts like we did, were each function does only one thing, but it’s also important to be able to group things that are not going to be used in other contexts (like improve and sqrt_iter). One solution is to nest the procedures in one block structure:

def sqrt(x):
    def is_good_enough(guess, x):
        return abs(square(guess) - x) < 0.001
    def improve(guess, x):
        return average(guess, x/guess)
    def sqrt_iter(guess, x):
        if is_good_enough(guess, x):
            return guess
        else:
            return sqrt_iter(improve(guess, x), x)
    return sqrt_iter(1.0, x)

Now the functions sqrt_iter, is_good_enough, and improve are internal to sqrt and are not exposed to other programmers.

But there’s something else. The variable x is bound in the scope of sqrt and since improve, is_good_enough, and sqrt_iter are in the scope of sqrt we don’t need to pass x explicitly as as argument to these functions. We can rewrite the code to make x a free variable inside these functions. In this case the interpreter will get the value of x from the enclosing scope. This is an example of lexical scoping:

def sqrt(x):
    def is_good_enough(guess):
        return abs(square(guess) - x) < 0.001
    def improve(guess):
        return average(guess, x/guess)
    def sqrt_iter(guess):
        if is_good_enough(guess):
            return guess
        else:
            return sqrt_iter(improve(guess))
    return sqrt_iter(1.0)

Summary

In the first section of SICP the authors introduce the notion of the environment and lexical scoping (both will be explored in more detail latter in the book), evaluation methods, and, above all, functional abstraction. I think this is a pretty sophisticated introduction for beginners and shows why SICP is considered one of the classics of computer science.

The notes for section 1.2 will come soon. Meanwhile, feel free to post suggestions and questions in the comments.

Our design of this introductory computer-science subject reflects two major concerns. First, we want to establish the idea that a computer language is not just a way of getting a computer to perform operations but rather that it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute. Second, we believe that the essential material to be addressed by a subject at this level is not the syntax of particular programming-language constructs, nor clever algorithms for computing particular functions efficiently, nor even the mathematical analysis of algorithms and the foundations of computing, but rather the techniques used to control the intellectual complexity of large software systems. Our goal is that students who complete this subject should have a good feel for the elements of style and the aesthetics of programming.

The emphasis in bold is mine to show a couple of my favorite quotes (my favorite quote of all times must be Alan Perlis’ “Syntactic sugar causes cancer of the semicolon”, but I digress).

The whole book is available for free and the book’s page at MIT has lots of extra material. Also, there are some highly recommended video lectures by the authors of the book.

SICP uses the Lisp dialect scheme, but since SICP concentrates on programming ideas rather than languages, there are some efforts to translate the code in the book to other programming languages. I think this is a good thing, because even if scheme is a beautiful language (it really is) and I think everyone should learn it, it’s good to see how the ideas presented in SICP can be translated to other languages.

I have read a good chunck of SICP and watched most video lectures before, but I didn’t do the exercises. Now, I want to read it again and do as many exercises as I can. Since I’ve been programming a lot in Python these days, I am curious to see how the ideas in SICP will translate to Python. So, my plan is to read the book and do the exercises using Python instead of Scheme. There are good notes and solutions for the exercises on the web and I may consult those as needed (giving fully credit, of course).

I’ll post my notes with the (Python) code. I’ll try to capture the essence of each section, instead of trying to post every single thing and exercise, but we’ll see how it works. I hope this will be a good resource for the Python community. Let me know in the comments what you think.

Why don’t you go ahead and read the post about section 1.1?

I designed the Python Quick Reference using LaTeX. I used common packages like color, multicol, and graphicx to have colored text, four columns, and to insert the python logo. I used beramono — a “version of Bitstream Vera Mono slightly enhanced for use with TeX” — and utopia to have better looking fonts. I used the tikz package to create the round boxes that separate each section:

The box creation is abstracted in the \header macro:

\newcommand{\header}[1]{
  \begin{tikzpicture}
    \node [fill=shade,rounded corners=5pt]
    {
      \parbox{.95\linewidth}{
        \large
        \textcolor{blue}{\sf \textbf{\raisebox{-15pt}{#1}}}
        \vspace*{1ex}
      }
    };
  \end{tikzpicture}
  \par
}

With this macro I can create each section with a command like \header{Built-in functions}. Simple and easy

Finally, I used the dirtree package to display the built-in exceptions hierarchy as a tree:

I like using something like LaTeX and dirtree because I can copy the exception hierarchy from the Python documentation and have LaTeX display it for me. If the list changes, it’s easier to update it than if I was using a graphical program. On the other hand I’m not 100% satisfied with the result, so I’m looking for another tree package for LaTeX. If you know one that might be good, let me know in the comments.

If you want something more substantial (and yet short), maybe the Python Pocket Reference is what you’re looking for. In addition to paper, it’s available in pdf, epub, and kindle (mobi) formats. (Disclaimer: I don’t have any affiliation with O’reilly and the above link it’s not an affiliate link.)

That’s it, a simple and good looking cheat sheet in minutes, courtesy of LaTeX. I may include a third page in the future with some useful functions from the standard library. There are also things that I didn’t figure out how to display in a short space, like data structures, iterators and generators. Please let me know in the comments if you have any ideas or suggestions.

UPDATE: Sarabander implemented some fixes and improvements (Thanks!). Go grab the new version!

In this post I’ll show how to configure emacs to write Python programs. We want to have not only basics things like syntax highlighting, but also code completion, easy access to Python’s documentation, ability to check for common mistakes, run unit tests, debugging, and a good interactive programming environment.

This setup is based on ipython and python-mode, but it’s also possible to use rope, ropemacs, and the auto complete mode as we can see here. I didn’t have much luck with ropemacs in the past but I’ll try it again in the future.

Tools installation and configuration

First we should install the tools we need.

• python-mode. There are two python modes; python.el and python-mode.el. Although python.el comes with GNU Emacs 22, I recommend you use python-mode.el since it has support for ipython. (There’s some talk about merging the two modes, but I don’t know it’s current status). Download and install python-mode.el and put the following in your .emacs:

  (require 'python-mode)
  (add-to-list 'auto-mode-alist '("\\.py\\'" . python-mode))

• ipython. Ipython is a powerful python shell with advanced features and nice code completion. Check ipython’s websitefor documentation and screencasts. If you’re using debian or ubuntu, install it with:

  sudo apt-get install ipython

  To configure ipython, edit ~/.ipython/ipy_user_conf.py and add your options. I like to use ipy_greedycompleter so it will complete things like "foo".TAB.

• ipython.el. It allows you to use python-mode with ipython. Download and install ipython.el and put the following in your .emacs:

  (require 'ipython)

• lambda-mode. This is only for aesthetics. When you install lambda-mode.el it will display a lambda character (λ) when you type lambda:
  The full name “lambda” will still be there, emacs is only displaying it differently. I like lambda-mode because it makes the code look shorter, neater, and cooler. Download and install lambda-mode.el and put the following in your .emacs:

  (require 'lambda-mode)
  (add-hook 'python-mode-hook #'lambda-mode 1)

  I had to set the variable lambda-symbol to the following, but YMMV:

  (setq lambda-symbol (string (make-char 'greek-iso8859-7 107)))

  There are other ways to have pretty lambdas and pretty symbols in emacs, check the Emacs Wiki page to see which one you prefer.

• anything. We will use anything for code completion. Download and install anything.el, anything-ipython.el, and anything-show-completion.el, and add the following to your .emacs:

  (require 'anything)
  (require 'anything-ipython)
  (when (require 'anything-show-completion nil t)
     (use-anything-show-completion 'anything-ipython-complete
                                   '(length initial-pattern)))

• Change comint keys. Comint is a minor-mode to deal with interpreter commands in an emacs buffer. It’s used by many modes, including ipython. Comint uses M-p and M-nto cycle backwards and forward through input history. I prefer to use the up and down arrow keys:

  (require 'comint)
  (define-key comint-mode-map (kbd "M-") 'comint-next-input)
  (define-key comint-mode-map (kbd "M-") 'comint-previous-input)
  (define-key comint-mode-map [down] 'comint-next-matching-input-from-input)
  (define-key comint-mode-map [up] 'comint-previous-matching-input-from-input)

• pylookup. We will use pylookupto search python’s documentation within emacs. First install the python documentation locally. If you use debian or ubuntu you can install it with one command:

  apt-get install python2.6-doc

  If the python documentation is not available as a package for your system, download it from the python website and unzip it somewhere in your computer. Then download pylookup and follow the instructions in the README file. Run pylookup.py to index the database by running:

  ./pylookup.py -u file:///usr/share/doc/python2.6-doc/html

  This command will generate a database file (pylookup.db). Naturally, you need to put the correct path for the python documentation. Next you need to add the following to your .emacs. Again, replace the values to match your system.

  (autoload 'pylookup-lookup "pylookup")
  (autoload 'pylookup-update "pylookup")
  (setq pylookup-program "~/.emacs.d/pylookup/pylookup.py")
  (setq pylookup-db-file "~/.emacs.d/pylookup/pylookup.db")
  (global-set-key "\C-ch" 'pylookup-lookup)

• autopair. A common feature in modern editors is to insert pairs of matching elements such as parenthesis, quotes, and braces. There are many ways to do this in emacs and installing autopair.el is probably the easiest one. You can configure it to work in all modes and exclude the modes you don’t to have it activated. For instance, I have it configured to work globally, except in the lisp-mode, where I use paredit:

(autoload 'autopair-global-mode "autopair" nil t)
(autopair-global-mode)
(add-hook 'lisp-mode-hook
          #'(lambda () (setq autopair-dont-activate t)))

I also have the following to in my .emacs to make autopair work with python single and triple quotes:

(add-hook 'python-mode-hook
          #'(lambda ()
              (push '(?' . ?')
                    (getf autopair-extra-pairs :code))
              (setq autopair-handle-action-fns
                    (list #'autopair-default-handle-action
                          #'autopair-python-triple-quote-action))))

• pep8 and pylint. These scripts are good to check your program for style errors and common mistakes. Install pylint and pep8:

  apt-get install pylint pep8

  and download and install python-pep8.el and python-pylint.el to integrate them with emacs and add the following to your .emacs:

  (require 'python-pep8)
  (require 'python-pylint)

• delete trailing space. When you type the return key in python-mode it’ll invoke py-newline-and-indent. As the manual says, it “deletes the whitespace before point, inserts a newline, and takes an educated guess as to how you want the new line indented.” This is great and I love this feature, but one unintended consequence is that you may end up with blank lines with trailing spaces. Python doesn’t care, but the pep8 tool will complain. Put the following in your .emacsto delete the trailing spaces when saving a file:

  (add-hook 'before-save-hook 'delete-trailing-whitespace)

• ipdb package. The ipdb package makes it easy to set breakpoints in the ipdb debugger. It’s easy to install it using easy_install(duh ):

  easy_install ipdb

• reimport package. reimport is a replacement for Python’s reload function that reloads all modules and sub-modules consistently. It’s useful for long-running programs and interactive development. Install it with easy_install:

  sudo easy_install reimport

• Templates. Yasnippet is a textmate-like templates for emacs. Download it here and pet the following in your .emacs:

  (require 'yasnippet-bundle)
  (yas/initialize)
  (yas/load-directory "~/.emacs.d/my-snippets/")

You can also install flymake to check the syntax on the fly. I find it distracting, so I don’t use it.

All right, these are the tools we need! Now we can go on and use these features.

Using python-mode

So now we’re ready to go. Open a python file, say test.py, type C-c ! (py-shell) to start the python interpreter. Go back to test.py (with C-x o) and type C-c C-c (py-execute-buffer) to send the whole buffer to the interpreter. In the following picture we can see how it works; it’s neat to be able to write code in one file and test it in the interpreter without the whole edit-compile-run-test cycle:

Code completion

Code completion is very useful and I find a must to be able to complete methods for both built-in and user-defined classes. For instance, if you type “os.path.” and M-tab emacs will show all methods available in os.path:

This completion method uses anything and you can use C-n and C-p to navigate through the completion items (when you’re done type return). This works for used-defined methods and functions too:

Access to documentation

Python has excellent documentation and it’s a huge bust in productivity to be able to access the documentation without leaving the editor. By using pylookup we can type C-c h to search any term in the python documentation. For instance, if we type “os.path.basename” and then C-c h to invoke the documentation, pylookup will open the documentation page for os.path.basename using the default browser configured in emacs (usually firefox):

However, many people prefer to use w3m inside emacs to browse technical documentation. With w3m we can read the documentation without leaving emacs, making it easy to copy code snippets and not needing to switch to an external program (to much switching can kill focus). Here’s the same documentation in the above picture but using w3m this time:

If you search for something with more than one possible answer, pylookup will ask you to choose among some possibilities, for instance, if I search for print I’ll get many options to choose from:

Code quality

The pep8 script checks if your code follow the PEP8 style guide. You can run it inside emacs by typing M-x pep8. The key M-x ` will jump to the place in the source code where the next error occurs:

You can run pylint inside emacs with M-x pylint, it behaves in a similar way to pep8:

Unit tests

It’s possible to run unit tests inside emacs if we use the compilation mode.  Type M-x compile and replace what will appear (most likely “make -k”) with your favorite python program to run unit tests. As with pep8 and pylint (they both use the compilation mode), you can type M-x ` to navigate to the next error. In the following screenshot we can see that test_square_of_100_is_1000 is not correct:

Debugging

There are many ways to debug python code. I like to use ipdb, ipython’s debugger, because it has advanced features like code completion and syntax highlighting. With the -pdb flag Ipython will go directly to the debugger when an exception is raised, instead of just printing a stack trace. (you may want to edit ~/.ipython/ipythonrc to make the change permanent).

To set breakpoints in the regular python debugger, you need to add the following lines to your code:

import pdb; pdb.set_trace()

With the ipdb package we can set breakpoints in a similar way if using ipdb:

import ipdb; ipdb.set_trace()

I like to make emacs highlight the lines that set breakpoints, so I can easily visualize where the breakpoints are and remember to remove them from the code after I’m done debugging. I use the following code to accomplish that:

(defun annotate-pdb ()
  (interactive)
  (highlight-lines-matching-regexp "import pdb")
  (highlight-lines-matching-regexp "pdb.set_trace()"))
(add-hook 'python-mode-hook 'annotate-pdb)

Notice in the following screenshot how the line that sets the breakpoint is highlighted and the program execution stopped at the breakpoint. I used the command n (next) to advance to the next statement:

I have a function to add breakpoints mapped to C-c C-t so I can set breakpoints easily:

(defun python-add-breakpoint ()
  (interactive)
  (py-newline-and-indent)
  (insert "import ipdb; ipdb.set_trace()")
  (highlight-lines-matching-regexp "^[ 	]*import ipdb; ipdb.set_trace()"))
(define-key py-mode-map (kbd "C-c C-t") 'python-add-breakpoint)

Templates

For textmate-like templates I use yasnippet (be sure to watch its demo). With it you can define templates easily and quickly to fill things for you, like a new-style class with documentation and an __init__ method. Python doesn’t have much boilerplate, but using yasnippet can help you to write code even faster.

Finding your way in the source code

Emacs allows us to navigate through source code by using tags. When you see a function call you can jump to its definition with M-. (find-tag) and jump back with  M-* (pop-tag-mark). Another useful command is tags-query-replace, to rename functions, methods, etc. To use it we need to generate a TAGS file. Emacs comes with the etags command but I recommend exuberant tags:

sudo apt-get install exuberant-ctags

Usually I put the following code in a Makefile to generate a TAGS file for a project:

ctags-exuberant -e -R --languages=python --exclude="__init__.py"

Dealing with multiple files

As we have seen, we can program in an interactive style in python-mode by sending the buffer to the python interpreter with C-c C-c. This works well for single file scripts and libraries, but no so well for more complex modules with sub-modules. For instance, if you open file4.py in the mainmodule bellow and execute it with C-c C-c it’ll fail if it depends on other submodules.

mainmodule
|--- __init__.py
|--- submodule1
     |--- __init__.py
     |--- file1.py
     |--- file2.py
|--- submodule2
     |--- __init__.py
     |--- file3.py
     |--- file4.py

We can define a master file using the py-master-file variable; python-mode will execute the file set in the variable instead of the current buffer’s file. This variable can be set as a file variable, but I prefer not to pollute every single freaking .py file in a module with a file variable, so I use a directory variable instead. Another advantage of using directory variables in this case is that each member of a team can set the py-master-file to reflect their file system layout. Create a file called .dir-locals.el in mainmodule‘s root with the following content:

((python-mode . ((py-master-file . "/path/to/interactivetest.py")
                 (tags-file-name . "/path/to/TAGS"))))

(Note that I also define the tags’ filename, so emacs’ll automatically load it when I read a python file located in this directory.)

To make this work I use a master file called interactivetest.py in the module’s root to re-import the module with the reimport package. This file is not really part of the module and I don’t even check it under version control. Now, every time we hit C-c C-c, regardless of what file we are changing, python-mode will execute interactivetest.py again, and update the whole module. For instance, this is what I have in the interactivetest.py for my aristoxenus library:

import reimport
import aristoxenus
reimport.reimport(aristoxenus)

But I may add a few things in the interactivetest.py file to make testing things interactively easier and to be able to save it between coding sessions:

from aristoxenus.parse import humdrum
foo = humdrum.parse_string("**kern\n4C")

Of course this is not a substitute for unit tests, but it’s nice to have foo available in the REPL so I can inspect it and quickly see what’s going on.

Conclusion

As you can see, we can have a nice programming environment for python in emacs. It may look like a lot of work, but in reality we only need to download and configure already existing packages. However, it would be nice to have a full-fledged development environment for python like slime.

I didn’t mention many things like the emacs code browser, the git interface, and the window session manager. They are not particularly specific to python, but they contribute to turn emacs into a powerful tool for development. Maybe I’ll mention then in another post.

I intend to make a screencast in the future showing these features, so stay tuned. Meanwhile, what do you use to develop python code?

PyCon

The videos from the 2009 and 2010 python conference are available at blip. I don’t like the browser interface in flash, though, so I prefer to browse the archives. A better alternative is to check the talk list at the PyCon website since it has the videos and more information such as slides, abstracts, and biographies.

These are a few talks I liked:

• Understanding the Python GIL
• PLY and PyParsing
• Optimizations And Micro-Optimizations In CPython
• Teaching compilers with python
• Writing Books using Python and Open Source Software
• New *and* Improved: Coming changes to unittest, the standard library test framework
• The Mighty Dictionary
• Unladen Swallow: fewer coconuts, faster Python
• Python Metaprogramming

Showmedo

Showmedo is a website with community produced screencasts for free and open-source software. It has a good amount of python videos such as a series about python 3, some demonstrations of the super cool ipython shell, and even videos for beginners.

Google’s Python Class

Google has a full series of lectures about python. If you are just starting to learn the language (or even starting to learn how to program), go check these out!

More advanced videos

Finally, these are a few more advanced videos and a must watch for any python programmer:

• Python 3000. A talk by Guido himself.
• Advanced Python or Understanding Python is a very nice talk by Thomas Wouters.
• Slightly Advanced Python: Some Python Internals by python über guru Alex Martelli

So, go watch those videos and don’t forget to recommend nice python videos to us.