Check out this related post on the Tweag blog, plus the restclient and vterm packages for Emacs.

Introduction

A number of programming languages - Haskell, Elm, Ruby, JavaScript/TypeScript, Rust, SQL - are being used in the construction of our systems. This variety is common in industry, since large software systems are increasingly being built of several smaller programs (including browser applications) which collaborate via HTTP and REST APIs, and therefore need not be written in the same language.

Industry trends in programming languages

Programming languages and environments have evolved continuously as computing power has increased. Java and C# used more memory and processor cycles than C/C++, but came with virtual machines that could run code compiled anywhere, so they became popular. Interpreted languages such as Python, PHP, JavaScript and Ruby can be interpreted fast enough that their programs can be run instantly so their behaviour can be observed immediately.

Those changes have largely resulted from being able to run programs faster, but importantly, programmers now have enough power on their desktop computers to use tools that can inspect their code before it runs, perform sophisticated analysis of its structure and implications, and use that analysis to provide feedback to the programmer. In this way, developers can use software to help them to write high quality code!

The fundamental design of most programming languages - including Java, C#, JavaScript, Python, C, C++ - is unfortunately such that this “static analysis” of code can only provide limited help and assurances to the programmer. It is therefore common in the software industry to deliver software that exhibits many bugs when it is run, or software that is expensive and risky to change because it is difficult to know whether changes have broken something.

A separate category of languages were built to allow static analysis of program correctness at development time. The most mature of these languages, Haskell, has been continuously developed as an open standard since its first version in 1990, and is seeing industrial use by companies such as Facebook, Google, Microsoft, AT&T, the New York Times and various investment banks. In turn, features of those advanced languages have influenced newer hybrid languages such as Scala and Apple’s Swift. Elm is a Haskell-like language which offers the safety of Haskell but produces JavaScript code that can be run in a browser. Overall, these advanced languages are set to grow dramatically in use.

Choosing the best tool for the job

Mainstream programming languages offer some advantages because they are in more widespread use: expertise is more available online and in the labour market, and a wide range of code libraries and vendors support those languages. In our technology group, we use a combination of more mainstream languages and Haskell and Elm, following the principle of choosing the best tool for the job.

Some functionality of our system, such as managing users and access control, is a commodity for which widely-used packaged code exists for the popular Ruby language, so we use that. In contrast, the novel functionality with which our system provides its unique value to our customers is the ability to process and query huge quantities of geographic data, and to explore it via a sophisticated browser-based visualisation engine. It is for these novel functions we have chosen Haskell and Elm.

Haskell for expressing complex behaviour

Haskell has excellent libraries for integrating with databases and web services, and for building API services. Our systems pass all of their geographic data through an API server written in Haskell, and not only must this server be reliable and fast, it must also be possible to evolve its design rapidly as we discover new requirements without breaking existing code.

Even when we make dramatic changes, Haskell’s language design and its development tools tell us if those changes are inconsistent with the existing design, and they require that we eliminate those inconsistencies. Those same tools help us to be confident that we are correctly handling all the possible inputs and outputs, which we model with Haskell’s advanced datatype system. The tools force us to consider every possible failure case, so that our programs will handle them appropriately.

The net result of this has been that we simply do not experience unexpected errors in our Haskell programs when they are deployed, so we can deploy new releases confidently and spend our time building valuable features rather than investigating run-time failures.

Building beautiful, reliable web interfaces

Elm has brought comparable benefits to the development of our web application interface: with a complex set of interrelated controls and information on the screen, it’s important to maintain consistency of display, particularly when new data is being loaded periodically from multiple servers. It is generally extremely hard to write such applications reliably in JavaScript, but Elm lets us program and analyse our web application interface code correctly, and then it produces JavaScript which runs absolutely reliably in the browser. Our users almost never experience programmer errors in the interface, since the Elm language and tools ensure that we specify the code’s behaviour correctly before it even runs in the user’s browser.

Language choice and our team

Both the Haskell and Elm code-bases are worked upon by multiple developers simultaneously, and often experience significant change, but the associated language tools have consistently ensured that the team members have not accidentally broken each other’s work. Even as our code-bases have grown significantly, they have remained easy to work with.

The pool of developers available in the labour market with experience in these languages is relatively small, but very smart developers who value correctness of software have applied to us and been hired because we are using these interesting languages. Groups using Haskell in industry universally report that job applicants are plentiful, capable and motivated: the developers who are actively interested in learning progressive ideas and languages are exactly the ones who will succeed in a fast-moving environment building a complex product.

Some cross-training has been necessary for our team members with experience primarily in mainstream languages: it has taken the team a little time to learn the new languages and how to think about our code in the rigorous way they demand, which can be unfamiliar at first. Good software developers learn quickly, though, and our smart team of mixed industry experience levels has picked up and embraced both languages rapidly and enthusiastically. We are quickly becoming familiar with the tools, best practices and library ecosystem of each language, and our productivity is higher than we would have expected had we used only mainstream languages.

Industry support for the languages has increased significantly in the last decade, and is now very good. The Haskell and Elm software tools and libraries we rely on see regular releases and prompt bug-fixes, high-quality third party libraries are readily available for key parts of our applications, and online communities and reference information are readily available.

We are proud to be building complex software which consistently delights customers with its new features and reliability, and our language choices have contributed to our ability to do so.

The discussion covered my background, history with Emacs, various tips and tricks, and of course my involvement in MELPA.

Sacha has a table of contents for the video over on her site, and if you like this video you should check out the other 17 she’s recorded to date.

Thanks for having me, Sacha, and for all the work you do sharing information so enthusiastically with your many readers!

Drew Neil’s Peer To Peer video series captures the working practices of experts in various programming languages, providing viewers with tips, tricks and insights.

Each video features a host, who brings along a task and acts as a “customer”, and a guest, who has to solve the challenge in real time using their tools of choice.

For this video, in which I was the host, I proposed that Ollie tackle Dave Thomas’ Word Chains Kata – I use this non-trivial problem myself whenever I learn a new programming language, so I knew that it would keep Ollie on his toes.

Ollie’s own site is a treasure-trove of carefully-explained Haskell tips and tricks, so I wasn’t surprised to find myself enjoying his problem-solving approach and learning a lot.

To see what happened, check out the video:

After the session, and having picked up a couple of new tricks from Ollie, I even felt inspired to tackle the problem in Haskell once more myself.

Update: there’s also a great follow-up video featuring Ollie hosting John Wiegley

Open source

Javascript

• I added support for Twitter’s “lists” to seaofclouds’ tweet.js widget: see my clone on github. You can see it in action at the bottom of this page.

Clojure

• Congomongo - Clojure’s de-facto standard library for accessing MongoDB - had no support for MongoDB’s file GridFS file storage scheme, so I added it.
• When I’ve only had a few spare minutes, I’ve been filling in a few missing Clojure examples on Rosettacode. It’s a great way to share knowledge, learn new libraries and algorithms, and solve puzzles.
• I’ve contributed minor patches to Leiningen, the Clojure build tool.

Ruby

• Accepted some patches for my darcs-to-git repository conversion script. (See my original article about this.)

Python

• I’ve agreed to review (but not write) the unit-testing chapter of the PSF’s planned Python book. Although I wrote the unittest module, I’m no longer an active Python user or member of the community.

Emacs lisp

• My haml-mode patch for colorizing “filter:” blocks in haml files has been committed upstream, so now textile, markdown, javascript and ruby portions of these files are font-locked correctly.
• Naturally, while I’ve been doing all this diverse hacking, I’ve been tweaking my emacs config quite heavily.

Misc

• As I recently blogged, I customised emacs-handler and have played with full-screen support for Cocoa Emacs.

Closed-source

I have two main closed-source projects on my plate. The first, our site about celebrities and the charities they support, has seen me doing a bunch of Twitter API hacking recently. The second project, currently Top Secret, is a Clojure data-oriented website, using MongoDB and fun stuff like geocoding and data mining.

Thoughts

The programming ecosphere is more diverse than ever, and there are more powerful tools available to programmers than ever before. It’s thrilling to be involved with this, and to aid in the cross-pollination of tricks, skills and idioms between unrelated projects. As a work-from-home entrepreneur, collaborating remotely on open (and closed) source projects with interesting people is a surprisingly good proxy for having real officemates.

With features like configurable message threading, kill-files and per-thread and per-poster scoring, Gnus has the potential to save a lot of the time you would normally spend wading through high-volume mailing lists.

Nowadays most people read mailing lists via IMAP-enabled mail accounts, e.g. Gmail. Gnus supports IMAP directly, but it doesn’t work well when the remote folders have thousands of messages, as is the case with many mailing lists. As a workaround, we can use offlineimap to maintain a local copy of our folders. As a side benefit, we can then read mail off-line.

(Update: since this article was written, Gnus IMAP support has improved, so the following steps may or may not be strictly necessary.)

We can use the excellent Dovecot to provide an IMAP interface to that local copy so that Gnus can access it; and we can do this without configuring Dovecot as a full IMAP server, which would need to run constantly and be configured for authentication etc.

On a Mac, you can install dovecot and offlineimap with Homebrew, MacPorts or, presumably, Fink.

In ~/.dovecotrc:

protocols = imap
mail_location = maildir:~/Library/Caches/OfflineImap
auth default {
}

And in ~/.offlineimaprc:

[general]
accounts = Mail
maxsyncaccounts = 1
ui = TTY.TTYUI
[Account Mail]
localrepository = Local
remoterepository = Remote
[Repository Local]
type = IMAP
preauthtunnel = dovecot -c ~/.dovecotrc --exec-mail imap
[Repository Remote]
type = IMAP
# or "type = Gmail", if applicable
remotehost = my.mail.server
remoteuser = myuser
remotepass = mypassword
ssl = yes
maxconnections = 1
realdelete = no
folderfilter = lambda foldername: re.search("^Lists\.", foldername)

Now, simply running “offlineimap” should sync your mailing lists into ~/Library/Caches/OfflineImap. (You’ll need to do this regularly, e.g. before and after reading your lists. Julien Danjou recently wrote a nice elisp wrapper for offlineimap.)

All that remains is to tell Gnus how to access your local mail cache.

In .gnus:

(setq imap-shell-program "dovecot -c ~/.dovecotrc --exec-mail imap")
(setq gnus-select-method '(nnimap "Mail"
                                  (nnimap-stream shell)))

Now you can start Gnus with M-x gnus, and then press ^ to enter the server buffer. Drill down into the “nnimap+Mail” server and use u to subscribe to each of the mailing lists. They should then appear in Gnus’ main *Group* buffer.

That’s enough to get started reading mail more efficiently, and as with all things Emacs, there’s great potential for customising Gnus to suit your own preferences.

Further reading

• GNUS and Gmail setup for dummies - covers configuring Gnus to send mail over SMTP
• Sacha Chua’s guide to Gnus + Offlineimap + Gmail - the source for much of this article
• Gnus homepage

Let’s start with the assumptions that using the latest dev version of Emacs is a Good Thing, and that full-screen display is a feature so wonderful (particularly on laptops) that it’s worth some small effort to obtain.

I build my Emacs from a local copy of the official Emacs git repo, which is automatically updated from the master bzr tree. Just a “git pull”, and I’m up to date.

I don’t want to build from typester’s repo because it hasn’t been updated from upstream since December.

So I add typester’s repo as an additional remote:

$ git remote add typester git://github.com/typester/emacs.git
$ git fetch typester

However, I find that his feature branch doesn’t merge cleanly with the official clone, perhaps because he started with a different git clone of the emacs sources.

Looking at the history of typester’s branch, the last upstream change was a3585f6c2a. So, to apply the changes he made on my current branch:

$ git diff a3585f6c2a typester/feature/fullscreen | patch -p1

And I see it applies without any conflicts! Great! (Here’s a copy of the patch as it looks today.)

Now I can build as usual:

$ ./configure --with-ns
$ make && make install
$ mv nextstep/Emacs.app /Applications/

To toggle full-screen mode inside emacs:

M-x ns-toggle-fullscreen

First, get a copy of Daisuke Murase’s emacs-handler project using git or by downloading a tarball.

Then apply the following patch:

diff --git a/Info.plist b/Info.plist
index 1880412..deb1fa4 100644
--- a/Info.plist
+++ b/Info.plist
@@ -8,6 +8,7 @@
                        <key>CFBundleURLSchemes</key>
                        <array>
                                <string>emacs</string>
+                               <string>txmt</string>
                        </array>
                        <key>CFBundleURLName</key>
                        <string>org.unknownplace.emacshandler</string>

Alternatively, you can just check out my clone of emacs-handler, which has the above patch built-in.

Finally, open the project in Xcode, and build it. Fire up the resulting EmacsHandler.app, and edit the app’s preferences to point to your Emacs.

That should be all you need to do – just keep EmacsHandler.app around somewhere on your Mac.

Specifically, ten years ago I wrote the unit testing module included in Python’s standard library. The module was a fairly direct copy of JUnit’s proven design at that time, and has since been used as the basis of (I’m sure) millions of unit tests in projects around the globe, including Python itself. It’s fairly safe to say that the major bugs have been ironed out by this stage.

But still, from time to time, I receive emails smugly pointing out a glaring bug in the module, with indignant comments such as “I am trying to teach more people in my company to use unit testing, but after the[y] discover bugs like these they come to me discouraged.”

Many such bug reports come complete with a short “demonstration” code snippet that shows a fundamental misunderstanding of how the library is to be used. I send polite responses, but I’d rather have simply been asked for a link to a good tutorial instead.

So here’s a question I’d like to request that programmers (whatever his opinion of his own ability) ask themselves before reporting a bug in a library they are using:

“Given the number of people using this library, is it at all possible that I could be the first person to discover this bug, or is it more likely that I’ve misunderstood how to use the library?”

This magical method, introduced by both Rails and Ruby 1.9, lets you do things like this:

> %w(1 2 3 4 5).map &:to_i
=> [1, 2, 3, 4, 5]

And that proc can, of course, take multiple parameters:

[1,2,3,4].zip([2,3,4,5])
=> [[1, 2], [2, 3], [3, 4], [4, 5]]> [1,2,3,4].zip([2,3,4,5]).map &:sum
=> [3, 5, 7, 9]

However, if you want to “close” a value into the proc, you’re out of luck, so you have to define a block old-school-style:

> %w(kung ruby bar).map { |e| e + "-fu" }
=> ["kung-fu", "ruby-fu", "bar-fu"]

But check out the following little hack:

class Array
  def to_proc
    lambda { |target| target.send(*self) }
  end
end

Now look what you can do:

> %w(kung ruby bar).map &[:+, "-fu"]
=> ["kung-fu", "ruby-fu", "bar-fu"]

What do you think? Handy? Disgusting? Leave a comment to let me know!

Update: apparently others have had similar ideas, but with different semantics. So perhaps this trick is a little too opaque…