Sanity, Inc. - Simple thinking for complex software

Save hours by reading mailing lists in Emacs over IMAP

Steve Purcell — Sun, 05 Sep 2010 09:54:05 +0000

Emacs' built-in newsreader Gnus is made for efficiently keeping up with high-volume lists; I show how to hook it into your IMAP account.

With features like configurable message threading, kill-files and per-thread and per-poster scoring, Gnus has the potential to save a lot of the time you would normally spend wading through high-volume mailing lists.

Nowadays most people read mailing lists via IMAP-enabled mail accounts, e.g. Gmail. Gnus supports IMAP directly, but it doesn't work well when the remote folders have thousands of messages, as is the case with many mailing lists. As a workaround, we can use offlineimap to maintain a local copy of our folders. As a side benefit, we can then read mail off-line.

We can use the excellent Dovecot to provide an IMAP interface to that local copy so that Gnus can access it; and we can do this without configuring Dovecot as a full IMAP server, which would need to run constantly and be configured for authentication etc.

On a Mac, you can install dovecot and offlineimap with Homebrew, MacPorts or, presumably, Fink.

In ~/.dovecotrc:

protocols = imap

mail_location = maildir:~/Library/Caches/OfflineImap

auth default {

}

And in ~/.offlineimaprc:

[general]

accounts = Mail

maxsyncaccounts = 1

ui = TTY.TTYUI

[Account Mail]

localrepository = Local

remoterepository = Remote

[Repository Local]

type = IMAP

preauthtunnel = dovecot -c ~/.dovecotrc --exec-mail imap

[Repository Remote]

type = IMAP

# or "type = Gmail", if applicable

remotehost = my.mail.server

remoteuser = myuser

remotepass = mypassword

ssl = yes

maxconnections = 1

realdelete = no

folderfilter = lambda foldername: re.search("^Lists\.", foldername)

Now, simply running "offlineimap" should sync your mailing lists into ~/Library/Caches/OfflineImap. (You'll need to do this regularly, e.g. before and after reading your lists. Julien Danjou recently wrote a nice elisp wrapper for offlineimap.)

All that remains is to tell Gnus how to access your local mail cache.

In .gnus:

(setq imap-shell-program "dovecot -c ~/.dovecotrc --exec-mail imap")

(setq gnus-select-method '(nnimap "Mail"

                                  (nnimap-stream shell)))

Now you can start Gnus with "M-x gnus", and then press "^" to enter the server buffer. Drill down into the "nnimap+Mail" server and use "u" to subscribe to each of the mailing lists. They should then appear in Gnus' main "*Group*" buffer.

That's enough to get started reading mail more efficiently, and as with all things Emacs, there's great potential for customising Gnus to suit your own preferences.

Swimming in code: recent hacks

Steve Purcell — Sun, 07 Feb 2010 13:06:21 +0000

Recently I've been on a hacking high, touching a lot of code, and in touch with a lot of cool programmers. In retrospect I'm surprised at the variety of stuff I've been doing, so I thought I'd write a short list.

Open source

Javascript

I added support for Twitter's "lists" to seaofclouds' tweet.js widget: see my clone on github. You can see it in action at the bottom of this page.

Clojure

Congomongo - Clojure's de-facto standard library for accessing MongoDB - had no support for MongoDB's file GridFS file storage scheme, so I added it.
When I've only had a few spare minutes, I've been filling in a few missing Clojure examples on Rosettacode. It's a great way to share knowledge, learn new libraries and algorithms, and solve puzzles.
I've contributed minor patches to Leiningen, the Clojure build tool.

Ruby

Accepted some patches for my darcs-to-git repository conversion script. (See my original article about this.)

Python

I've agreed to review (but not write) the unit-testing chapter of the PSF's planned Python book. Although I wrote the unittest module, I'm no longer an active Python user or member of the community.

Emacs lisp

My haml-mode patch for colorizing "filter:" blocks in haml files has been committed upstream, so now textile, markdown, javascript and ruby portions of these files are font-locked correctly.
Naturally, while I've been doing all this diverse hacking, I've been tweaking my emacs config quite heavily.

Misc

As I recently blogged, I customised emacs-handler and have played with full-screen support for Cocoa Emacs.

Closed-source

I have two main closed-source projects on my plate. The first, our site about celebrities and the charities they support, has seen me doing a bunch of Twitter API hacking recently.

The second project, currently Top Secret, is a Clojure data-oriented website, using MongoDB and fun stuff like geocoding and data mining.

Thoughts

The programming ecosphere is more diverse than ever, and there are more powerful tools available to programmers than ever before. It's thrilling to be involved with this, and to aid in the cross-pollination of tricks, skills and idioms between unrelated projects.

As a work-from-home entrepreneur, collaborating remotely on open (and closed) source projects with interesting people is a surprisingly good proxy for having real officemates.

Full-screen support for Cocoa Emacs on OS X

Steve Purcell — Sun, 31 Jan 2010 17:06:19 +0000

Prolific Japanese hacker Daisuke Murase (a.k.a. typester) has recently patched Cocoa Emacs to add a full-screen display mode. Here's how to add this must-have feature into your own local Emacs tree while you wait for it to get integrated into the official Emacs sources.

Let's start with the assumptions that using the latest dev version of Emacs is a Good Thing, and that full-screen display is a feature so wonderful (particularly on laptops) that it's worth some small effort to obtain.

I build my Emacs from a local copy of the official Emacs git repo, which is automatically updated from the master bzr tree. Just a "git pull", and I'm up to date.

I don't want to build from typester's repo because it hasn't been updated from upstream since December.

So I add typester's repo as an additional remote:

$ git remote add typester git://github.com/typester/emacs.git

$ git fetch typester

However, I find that his feature branch doesn't merge cleanly with the official clone, perhaps because he started with a different git clone of the emacs sources.

Looking at the history of typester's branch, the last upstream change was a3585f6c2a. So, to apply the changes he made on my current branch:

$ git diff a3585f6c2a typester/feature/fullscreen | patch -p1

And I see it applies without any conflicts! Great! (Here's a copy of the patch as it looks today.)

Now I can build as usual:

$ ./configure --with-ns

$ make && make install

$ mv nextstep/Emacs.app /Applications/

To toggle full-screen mode inside emacs:
M-x ns-toggle-fullscreen

Open "txmt:" URLs from Rails backtraces using Emacs on OS X

Steve Purcell — Sun, 31 Jan 2010 16:29:06 +0000

When Rails prints a backtrace in HTML, it's helpfully rendered as a "txmt:" link so that users can click open the corresponding location in TextMate on OS X. If you're an Emacs user, here's how to make those URLs open in Emacs instead.

First, get a copy of Daisuke Murase's emacs-handler project using either git (or by downloading a tarball).

Then apply the following patch:

diff --git a/Info.plist b/Info.plist

index 1880412..deb1fa4 100644

--- a/Info.plist

+++ b/Info.plist

@@ -8,6 +8,7 @@

                        <key>CFBundleURLSchemes</key>

                        <array>

                                <string>emacs</string>

+                               <string>txmt</string>

                        </array>

                        <key>CFBundleURLName</key>

                        <string>org.unknownplace.emacshandler</string>

Alternatively, you can just check out my clone of emacs-handler, which has the above patch built-in.

Finally, open the project in Xcode, and build it. Fire up the resulting EmacsHandler.app, and edit the app's preferences to point to your Emacs.

That should be all you need to do -- just keep EmacsHandler.app around somewhere on your Mac.

So you think you found a bug in a library?

Steve Purcell — Mon, 29 Jun 2009 11:18:09 +0000

From time to time I receive bug reports for the libraries I've written. Some reports describe genuine bugs (and I'll be the first to admit I'm not perfect), but far too many others demonstrate a missed step in the submitter's "bug assessment" mental process.

Specifically, ten years ago I wrote the unit testing module included in Python's standard library. The module was a fairly direct copy of JUnit's proven design at that time, and has since been used as the basis of (I'm sure) millions of unit tests in projects around the globe, including Python itself. It's fairly safe to say that the major bugs have been ironed out by this stage.

But still, from time to time, I receive emails smugly pointing out a glaring bug in the module, with indignant comments such as "I am trying to teach more people in my company to use unit testing, but after the[y] discover bugs like these they come to me discouraged."

Many such bug reports come complete with a short "demonstration" code snippet that shows a fundamental misunderstanding of how the library is to be used. I send polite responses, but I'd rather have simply been asked for a link to a good tutorial instead.

So here's a question I'd like to request that every programmer (whatever his opinion of his own ability) ask himself before reporting a bug in a library he's using:

"Given the number of people using this library, is it at all possible that I could be the first person to discover this bug, or is it more likely that I've misunderstood how to use the library?"

P.S. Why not subscribe to future articles?

Adding Array#to_proc to Ruby

Steve Purcell — Tue, 14 Apr 2009 19:37:30 +0000

Here's a neat Ruby trick for fans of Symbol#to_proc.

This magical method, introduced by both Rails and Ruby 1.9, lets you do things like this:

> %w(1 2 3 4 5).map &:to_i

=> [1, 2, 3, 4, 5]

And that proc can, of course, take multiple parameters:

 [1,2,3,4].zip([2,3,4,5])

=> [[1, 2], [2, 3], [3, 4], [4, 5]]> [1,2,3,4].zip([2,3,4,5]).map &:sum

=> [3, 5, 7, 9]

However, if you want to "close" a value into the proc, you're out of luck, so you have to define a block old-school-style:

> %w(kung ruby bar).map { |e| e + "-fu" }

=> ["kung-fu", "ruby-fu", "bar-fu"]

But check out the following little hack:

class Array

  def to_proc

    lambda { |target| target.send(*self) }

  end

end

Now look what you can do:

> %w(kung ruby bar).map &[:+, "-fu"]

=> ["kung-fu", "ruby-fu", "bar-fu"]

What do you think? Handy? Disgusting? Leave a comment to let me know!

Update: apparently others have had similar ideas, but with different semantics. So perhaps this trick is a little too opaque...

Liked this? Why not subscribe to my news feed to get more of the same in future.

All hail the innovators: 2 shining examples of lateral thinking

Steve Purcell — Wed, 08 Oct 2008 15:03:24 +0000

Wonderful things happen when smart people discard limitations that everyone else has been taking for granted, and two recent examples in the online world illustrate this beautifully.

Distributed version control

For years we assumed that working together on carefully versioned sets of files requires a central server to keep everyone on the same track. Then along came arch, darcs, mercurial, bazaar, git and others -- now it seems like a god-given right that every developer should be able to independently commit his source code changes and create branches, and the "central repository" has changed from a technical necessity to a useful convention.

SEOmoz's Linkscape

It seems like every notable search engine optimisation blog or site you'll find on the web offers home-spun tools for analysing link graphs, site values, click prices and suchlike. Many of these tools compile information desperately scraped from Yahoo's SiteExplorer and other sources, because the data that everyone really wants is locked up in Google's enormous index.

In a stroke of utter genius, the SEO consultancy and community SEOmoz took venture capital, and then several months later popped up with its own index of 30 billion web pages, opening the way for deep SEO analysis that has been impossible up to now.

Talk about thinking big.

Innovators, I salute you!

I find it thrilling to see these kinds of innovations, and it's a reminder to routinely step back from the problem in front of me, and see if I'm taking for granted a constraint that isn't really there.

Do you have any other examples? Leave a comment!

Skype power-user tip: edit sent messages

Steve Purcell — Fri, 25 Jul 2008 10:24:21 +0000

Ever quickly sent a skype message, only to immediately notice a typo? You can fix it using the following trick.

Imagine you just sent "The radio is borken".

To fix "borken", you simply enter this follow-message: "s/borken/broken/".

The sent message will be updated, and marked as "edited". Magic, eh?

This was tested between two Macs running Skype 2.7, so it may or may not work for you too. It's only possible to alter the last sent message you sent, and not earlier messages or those sent by another member of the chat.

UNIX users will note that the syntax echoes that of sed, awk, perl, vi and other tools, and the "s/old/new/" idiom is commonly used in geek-to-geek chats.

(Credit to my colleague and fellow Ruby on Rails consultant Stefan Kaes for noticing this.)

Rails tip: easily browse and search logs in colour with "less"

Steve Purcell — Tue, 17 Jun 2008 19:13:58 +0000

'less' is a great tool for browsing Rails log files, though you might not guess it if you try to view your "development.log" with it.

Take the colour back

Rails writes xterm control characters into log files to colour them. Usually 'less' escapes those characters, leading to an ugly display -- run 'less -R' to stop this escaping, allowing you to browse log files in glorious technicolour. (Set your $LESS environment variable to "-R" to make this the default.)

Follow the end of the log file

Shift-g will take you to the bottom of the log files (Use ctrl-c if less then pauses while calculating line numbers in a very large file).

Shift-F makes less work like 'tail -f' -- it watches the end of the file for new data.

Search for phrases or pesky backtraces

Use / to search forwards, and ? to search backwards.

To find the last Ruby stack trace in a log file, open the logfile in less, Shift-g to the end of the file, then search backwards using ?` (backtick) -- usually backticks are only found in Ruby stacktraces, at least if you're not using mysql.

On vim vs emacs

Steve Purcell — Sun, 15 Jun 2008 09:40:27 +0000

I often hear the proponents of vim and emacs discussing which is better, usually pitting emacs' extensibility against vi's archaic yet arguably effective and tendon-friendly scheme of editing modes.

But why choose?

I love the "vi way". I was a die-hard vim user until I realised that, with some customisation, emacs can be vim. After all, many advanced vim users end up working with multiple windows, and keeping their vims running for an extended period, which is exactly what emacs users do.

So for 8 or 9 years now, I've been a happy emacs user, working with vi key bindings all day long, while picking and choosing from emacs' vast library of useful extensions. I've got a highly customised emacs, learned some emacs-lisp, and yet I don't even know the emacs key bindings for mark/copy/paste. And when I'm working in a shell, I can (and do) drop into vim and still feel completely at home.

Curious vim users can get started by firing up emacs, hitting meta-x, and entering "viper-mode". That's built into every emacs you're likely to encounter. To get vim-specific features such as control-n/p expansion, window splitting, and rectangular selections, follow the instructions that come with vimpulse.