IMHO

The Most Amazing PostgreSQL Database

2012-10-07T00:00:00+02:00

For me PostgreSQL is the most amazing (open source) database around. Even though, there is much interest in stripped down NoSQL databases like key-value stores or “data structure servers”, PostgreSQL continues to innovate at the SQL frontier.

In this post, I’ll show a few of the newer, less known, features of PostgreSQL - far beyond standard SQL.

hstore

hstore is a key-value store for simple data types. Using hstore, we’re able to create a key-value store within columns of a table.

To enable the hstore extension, run create extension hstore' in the PostgreSQL prompt. After that the hstore data type is available for our table definitions.

Let’s create a simple table with a hstore column:

create table hstoretest ( id serial primary key, data hstore );

To insert a few rows, we use a special syntax:

insert into hstoretest (data) values ('key1 => 123, key2 => "text"'::hstore);

Query the table as usual:

select * from hstoretest;

 id |                           data
----+-----------------------------------------------------------
  1 | "key1"=>"123", "key2"=>"text"

The hstore extension provides a lot of operators and functions to work with hstore columns, for example, selecting all key2 values:

select data -> 'key2' as key2 from hstoretest;

 key2
------
 text
(1 row)

Some more examples can be found here.

JSON

A JSON data type was introduced in release 9.2. Currently this is nothing more than a validating data type, thus it checks if the string we put into that column is a valid JSON object.

Let’s create a new table to play around with this type:

create table jsontest ( id serial primary key, data json );

Now let’s insert an invalid row:

insert into jsontest (data) values ('{"title":wrong}');

ERROR:  invalid input syntax for type json
LINE 1: insert into jsontest (data) values ('{"title":wrong}');
                                            ^
DETAIL:  Token "wrong" is invalid.
CONTEXT:  JSON data, line 1: {"title":wrong...

And now with the correct JSON syntax:

insert into jsontest (data) values ('{"title":"right"}');

There isn’t really much more to the JSON data type besides the ability to return rows of non-JSON tables as JSON:

select row_to_json(hstoretest) from hstoretest;

                                       row_to_json
-----------------------------------------------------------------------------------------
 {"id":1,"data":"\"key1\"=>\"123\", \"key2\"=>\"text\""}
(1 row)

Nice if you’re used to work with JSON object (in Web applications for example).

PLv8

Working directly with JSON and JavaScript has been all the rage in many of the NoSQL databases. Using the PLv8 extension, we can use JavaScript (executed by Google’s awesome V8 engine) directly in PostgreSQL. Together with the JSON data type, this offers amazing new possibilities.

Currently PLv8 isn’t included in the standard distribution of PostgreSQL (9.2), but installing it isn’t very hard, the only dependencies are postgresql and the v8 engine. Some distributions already have v8 in their repositories (Archlinux).

Compiling and installing the extension is straight forward as soon as the dependencies are in place:

make && sudo make install

No we can enable the plv8 extension within our database (as we did with hstore):

create extension plv8;

A particular nice example of using JSON and PLv8 comes from Andrew Dunstan:

create or replace function jmember (j json, key text )
 RETURNS text
 LANGUAGE plv8
 IMMUTABLE
AS $function$
  var ej = JSON.parse(j);
  if (typeof ej != 'object')
        return NULL;
  return JSON.stringify(ej[key]);
$function$;

The jmember function allows us to parse and read the JSON string and returns the member identified by key:

select jmember(data, 'title') from jsontest;

     jmember
-----------------
 "right"
(1 row)

Andrew also shows how to build an index to speed up access times in his post.

k-Nearest Neighbors

In PostgreSQL 9.1, a nearest neighbor indexing was introduced. This allows us to perform orderings etc. by a distance metric.

For example, I’ve downloaded the ispell spelling dictionaries, and loaded them into a table words like this:

create table words (word varchar(50) primary key);
copy words from 'english.0';

This inserts roughly 50000 words into the table.

Since we’re working with text data, let’s introduce another extension pg_trgm, which builds tri-grams of strings (triples of three characters). Using theses tri-grams, we can compute a distance metric. Enable the extension like this:

create extension pg_trgm;

A tri-gram of hello would look like this:

select show_trgm('hello');
            show_trgm
---------------------------------
 {"  h"," he",ell,hel,llo,"lo "}
(1 row)

The distance metric is very simple, the more of these tri-grams match, the closer two strings are.

To take advantage of the nearest neighbor index, we have to build it:

create index word_trgm_idx on words using gist (word gist_trgm_ops);

Using the index we can query our table for a word, and return a list of most similar terms as well:

select word, word <-> 'hello' as distance from words order by word <-> 'hello' asc limit 10;
  word  | distance
--------+----------
 hello  |        0
 hellos |    0.375
 hell   | 0.428571
 hells  |      0.5
 heller | 0.555556
 hell's | 0.555556
 help   |    0.625
 helm   |    0.625
 held   |    0.625
 helps  | 0.666667
(10 rows)

The <-> operator comes from the pg_trgm extension, of course we could use simpler distances like numerical difference or geometric distance, but working with textual data is often perceived as particularly difficult (not so with PostgreSQL).

A REST API in Clojure

2012-10-04T00:00:00+02:00

Clojure is one of the most interesting new languages targeting the JVM. Initially only the JVM, in the meantime it is also available for JavaScript. Essentially, you can write Clojure and either execute it as Java program or JavaScript program, of course each flavor has its unique features as well.

Clojure is a Lisp, thus the syntax may be foreign, but it is really, really easy since there are very few syntactic variations. The language “Lisp” is very lean and usually easily learned.

In this post, we’re going to create a complete REST application from scratch. There are already some (very) good tutorials available, but some are not quite up to date (see Heroku’s Devcenter or Mark McGranaghan for good ones). Clojure itself is still a young language, Lisp of course has a lot of history.

Our application should allow creating, listing, fetching, updating, and deleting of documents.

A document looks like this (JSON encoded):

    {
      "id" : "some id"
      , "title" : "some title"
      , "text" : "some text"
    }

A GET call to /documents should return a list of these documents.
A POST call to /documents with a documents as body shall create a new document, assigning a new id (ignoreing the posted one).
A GET to /documents/[ID] should return the document with the given id, or 404 if the document does not exist.
A PUT to /documents/[ID] should update the document with the given id and replace title and text with those from the document in the uploaded body.
A DELETE to /documents/[ID] should delete the document with the given id and return 204 (NO CONTENT) in any case.

Creating the project scaffolding

We’re going to use Leiningen, the defacto build system and dependency manager for Clojure projects. Download and install it, then execute:

    lein new compojure clojure-rest

We’re creating a new Compojure project called clojure-rest. Compojure is the library that maps URLs to functions in our program. Compojure (and our project) builds on Ring is the basic Server API. To start the new project run:

    lein ring server

This starts the server on localhost:3000 and automatically restarts the server if any of the project files change. Thus, you can leave it running while we develop our application.

The new command generates two very important files for us:

project.clj is the project configuration. It states dependencies, the entry point etc. (read the whole documentation on Leiningen.org, and src/clojure_rest/handler.clj which contains a starting point for our application.

Project configuration (project.clj)

    (defproject clojure-rest "0.1.0-SNAPSHOT"
      :description "FIXME: write description"
      :url "http://example.com/FIXME"
      :dependencies [[org.clojure/clojure "1.4.0"]
                     [compojure "1.1.1"]]
      :plugins [[lein-ring "0.7.3"]]
      :ring {:handler clojure-rest.handler/app}
      :profiles
      {:dev {:dependencies [[ring-mock "0.1.3"]]}})

Update the file to look like this:

    (defproject clojure-rest "0.1.0-SNAPSHOT"
      :description "REST service for documents"
      :url "http://blog.interlinked.org"
      :dependencies [[org.clojure/clojure "1.4.0"]
                     [compojure "1.1.1"]
                     [ring/ring-json "0.1.2"]
                     [c3p0/c3p0 "0.9.1.2"]
                     [org.clojure/java.jdbc "0.2.3"]
                     [com.h2database/h2 "1.3.168"]
                     [cheshire "4.0.3"]]
      :plugins [[lein-ring "0.7.3"]]
      :ring {:handler clojure-rest.handler/app}
      :profiles
      {:dev {:dependencies [[ring-mock "0.1.3"]]}})

Besides the JSON parsing library Cheshire, we added the C3P0 Connection Pool, the H2 Database JDBC driver and Clojure’s java.jdbc contrib-library.

I also updated the :url and :description fields.

The request handler (handler.clj)

Next let’s have a look at the generated request handler src/clojure_rest/handler.clj:

    (ns clojure-rest.handler
      (:use compojure.core)
      (:require [compojure.handler :as handler]
                [compojure.route :as route]))

    (defroutes app-routes
      (GET "/" [] "Hello World")
      (route/not-found "Not Found"))

    (def app
      (handler/site app-routes))

The route GET "/" [] "Hello World" is responsible for our result we saw in the browser. It maps all GET requests to / without parameters to "Hello World". The (def app (handler/site app-routes)) part configures our application (registering the routes).

Our first step is to update the configuration. We’re going to work with JSON, so let’s include some Ring middlewares to setup response headers (wrap-json-response) and parse request bodies (wrap-json-body) for us. A middleware is just a wrapper around a handler, thus it can pre- and post-process the whole request/response cycle.

    (def app
      (-> (handler/api app-routes)
        (middleware/wrap-json-body)
        (middleware/wrap-json-response)))

We switched also from the handler/site template to handler/api which is more appropriate for REST APIs (documentation).

Next let’s define the routes for our application:

    (defroutes app-routes
      (context "/documents" [] (defroutes documents-routes
        (GET  "/" [] (get-all-documents))
        (POST "/" {body :body} (create-new-document body))
        (context "/:id" [id] (defroutes document-routes
          (GET    "/" [] (get-document id))
          (PUT    "/" {body :body} (update-document id body))
          (DELETE "/" [] (delete-document id))))))
      (route/not-found "Not Found"))

We define GET and POST for the context "/documents, and GET, PUT, DELETE for the context ":id" on top of that. :id is a placeholder and can then be injected into our parameter vector. The POST and PUT request have a special parameter body for the parsed body (this parameter is provided by the wrap-json-body middleware. For more on routes, take a look at Compojure’s documentation.

Before we define the functions to carry out the requests, let’s fix the imports and open a pool of database connections to work with.

The namespace declaration is used to define which namespaces shall be made available by Clojure.

    (ns clojure-rest.handler
      (:import com.mchange.v2.c3p0.ComboPooledDataSource)
      (:use compojure.core)
      (:use cheshire.core)
      (:use ring.util.response)
      (:require [compojure.handler :as handler]
                [ring.middleware.json :as middleware]
                [clojure.java.jdbc :as sql]
                [compojure.route :as route]))

We import C3P0’s ComboPooledDataSource, a plain Java class. Next, we fetch the functions defined in compojure.core, cheshire.core, and ring.util.response into our namespace, they can be used without qualifying. Finally we require some more libraries, this time with a qualifier to prevent name clashes or to support nicer separation. I’m not sure when to make the cut between :use and :require yet, so the cut is abitrary.

    (def db-config
      {:classname "org.h2.Driver"
       :subprotocol "h2"
       :subname "mem:documents"
       :user ""
       :password ""})

Note, we use a in-memory database. If you’d like to keep your database between restarts, you could use :subname "/tmp/documents" for example.

Next we open a pool of connections. C3P0 has no Clojure wrapper, so we deal with Java classes and objects directly (hence a bit more code).

    (defn pool
      [config]
      (let [cpds (doto (ComboPooledDataSource.)
                   (.setDriverClass (:classname config))
                   (.setJdbcUrl (str "jdbc:" (:subprotocol config) ":" (:subname config)))
                   (.setUser (:user config))
                   (.setPassword (:password config))
                   (.setMaxPoolSize 6)
                   (.setMinPoolSize 1)
                   (.setInitialPoolSize 1))]
        {:datasource cpds}))

    (def pooled-db (delay (pool db-config)))

    (defn db-connection [] @pooled-db)

Since we deal with a in-memory database, we need to create our table now.

    (sql/with-connection (db-connection)
      (sql/create-table :documents [:id "varchar(256)" "primary key"]
                                   [:title "varchar(1024)"]
                                   [:text :varchar]))

The intent should be easy to understand, for the details take a look at the java.jdbc documentation. We create a table documents with a :id, :title, and :text column. Note that the database column is called id, not :id.

The only thing missing are the functions to actually perform the actions requested by our clients.

To return a single document with a given id, we could come up with this:

    (defn get-document [id]
      (sql/with-connection (db-connection)
        (sql/with-query-results results
          ["select * from documents where id = ?" id]
          (cond
            (empty? results) {:status 404}
            :else (response (first results))))))

It reads like this: when called with an id, open a database connection, perform select * from documents where id = ? with the given id as parameter. If the result is empty, return 404, otherwise return the first (and only) document as response.

The response call will convert the document into JSON, this functionality is provided by wrap-json-response, which also sets the correct Content-Type etc.

Another nice one is the creation of new documents:

    (defn uuid [] (str (java.util.UUID/randomUUID)))

    (defn create-new-document [doc]
      (let [id (uuid)]
        (sql/with-connection (db-connection)
          (let [document (assoc doc "id" id)]
            (sql/insert-record :documents document)))
        (get-document id)))

Here we use Java’s UUID generator (without import, hence the full package name) to generate a new id for each document created. The second let statement is responsible to replace the user-provided id (if any) with our generated one. Remember that Clojure’s datastructures are immutable, so we need to use the document variable thereafter, instead of the doc which still contains the old (or no) id.

Returning the document is delegated to the get-document function.

The complete handler.clj

To round the post up, here is the whole program:

    (ns clojure-rest.handler
      (:import com.mchange.v2.c3p0.ComboPooledDataSource)
      (:use compojure.core)
      (:use cheshire.core)
      (:use ring.util.response)
      (:require [compojure.handler :as handler]
                [ring.middleware.json :as middleware]
                [clojure.java.jdbc :as sql]
                [compojure.route :as route]))

    (def db-config
      {:classname "org.h2.Driver"
       :subprotocol "h2"
       :subname "mem:documents"
       :user ""
       :password ""})

    (defn pool
      [config]
      (let [cpds (doto (ComboPooledDataSource.)
                   (.setDriverClass (:classname config))
                   (.setJdbcUrl (str "jdbc:" (:subprotocol config) ":" (:subname config)))
                   (.setUser (:user config))
                   (.setPassword (:password config))
                   (.setMaxPoolSize 1)
                   (.setMinPoolSize 1)
                   (.setInitialPoolSize 1))]
        {:datasource cpds}))

    (def pooled-db (delay (pool db-config)))

    (defn db-connection [] @pooled-db)

    (sql/with-connection (db-connection)
    ;  (sql/drop-table :documents) ; no need to do that for in-memory databases
      (sql/create-table :documents [:id "varchar(256)" "primary key"]
                                   [:title "varchar(1024)"]
                                   [:text :varchar]))

    (defn uuid [] (str (java.util.UUID/randomUUID)))

    (defn get-all-documents []
      (response
        (sql/with-connection (db-connection)
          (sql/with-query-results results
            ["select * from documents"]
            (into [] results)))))

    (defn get-document [id]
      (sql/with-connection (db-connection)
        (sql/with-query-results results
          ["select * from documents where id = ?" id]
          (cond
            (empty? results) {:status 404}
            :else (response (first results))))))

    (defn create-new-document [doc]
      (let [id (uuid)]
        (sql/with-connection (db-connection)
          (let [document (assoc doc "id" id)]
            (sql/insert-record :documents document)))
        (get-document id)))

    (defn update-document [id doc]
        (sql/with-connection (db-connection)
          (let [document (assoc doc "id" id)]
            (sql/update-values :documents ["id=?" id] document)))
        (get-document id))

    (defn delete-document [id]
      (sql/with-connection (db-connection)
        (sql/delete-rows :documents ["id=?" id]))
      {:status 204})

    (defroutes app-routes
      (context "/documents" [] (defroutes documents-routes
        (GET  "/" [] (get-all-documents))
        (POST "/" {body :body} (create-new-document body))
        (context "/:id" [id] (defroutes document-routes
          (GET    "/" [] (get-document id))
          (PUT    "/" {body :body} (update-document id body))
          (DELETE "/" [] (delete-document id))))))
      (route/not-found "Not Found"))

    (def app
        (-> (handler/api app-routes)
            (middleware/wrap-json-body)
            (middleware/wrap-json-response)))

Yeah, the whole program with connection pooling, JSON de/encoding in roughly 90 lines of (admittedly dense) code.

To sum it up: Clojure is fun, concise, and very powerful. Together with the excellent Java integration it ranks very high on my “languages I adore” list.

REST Framework Survey (Java, Haskell, Go, Node.js)

2012-09-23T00:00:00+02:00

Over the last few days I’ve experimented with various REST frameworks. The initial goal was to find The Language and framework to use for all future projects… . Of course there is no clear winner.

These were the contestants:

The client was written in Go because I wanted ~~it to be fast and easy to make concurrent requests~~ to try Go.

The server spec is to povide a way to insert, query, update, list and delete documents.

A POST to /documents should create a new document, generate a new UUID (v4) as ID and return the whole document.
A GET to /documents should return a list of all documents
A GET to /documents/[ID] should return the document with the given ID, or 404 if it is not found.
A PUT to /documents/[ID] should update the document with the given ID and return it.
A DELETE to /documents/[ID] should delete the document with the given ID.

The documents are encoded in JSON:

{
    "id": "some id",
    "title": "some title",
    "text": "some text"
}

The client may or may not send the id field, the server ignores it and either generates a new one, or uses the one from the URL.

The client’s steps:

Insert a document
Update that document
Delete that document

I used goroutines to make that concurrent and checked the consistency via a final GET /documents asserting an empty list.

Step 1

As database backend, I used an SQLite3 in-memory database connected via:

The implementation was easy with Java and Node.js. I’ve had my fair share of trouble with Haskell because there is no clear REST framework to use, at least I couldn’t find the obvious choice and went with Scotty because it’s simple and did what I needed, and my Haskell-fu has seen better days.

Here’s a inconclusive performance report¹ (how long it took, in seconds, so longer is worse) processing 10000 documents with the same client on each of the backends.

Node.js is a bit slower, Java and Haskell are pretty much on par.

Code

Here’s the code used to update (PUT) the new version of a document on the server. I’ve chosen the update case because it shows how to deal with path-parameters, as well as how to decode and encode JSON.

Node.js Code to wire PUT requests to the DB.

server.put('/documents/:docId', update); // define which function to call for that URL+method
...
function update(req, res, next) {
  var doc = req.body; // directly available as object through the BodyParser plugin
  doc.id = req.params.docId; // taken from the URL
  db.run("update documents set title = ?, text = ? where id = ?", doc.title, doc.text, doc.id);
  return read(req, res, next); // read the document or return 404
}

The Jersey annotations used to declare “updateDocument” as handler for PUT requests. JSON en/decoding is fully transparent (because of the @Produces annotation).

@PUT // handle PUT requests
@Path("/documents/{id}") // on that URL
@Consumes(MediaType.APPLICATION_JSON) // accepts only JSON
@Produces(MediaType.APPLICATION_JSON) // writes out JSON
// params can be references and injected, the body is automaticall decoded into the correct object
public Document updateDocument(@PathParam("id") String id, Document document) throws SQLException {
    // perform the database stuff (in a data access object as it is customary in Java)
    Document document = documentDao.update(id, document);
    if (document == null) {
        throw new NotFoundException(); // produces the 404 status
    }
    // the returned object is also automatically encoded
    // and all headers are set correctly
    return document;
}

The Haskell version is very concise, but with the various Monad-layerings a bit opaque (especially the DB code):

put "/documents/:id" $ do -- the URL this function is defined for
  id <- param "id" -- extract the parameter from the URL
  inputDocument <- jsonData -- parse the JSON body
  doc <- liftIO $ updateDocument conn id inputDocument -- write the stuff into the DB
  resultOr404 doc -- return the document or 404

-- for reference on how to deal with Maybe
resultOr404 :: Maybe Document -> ActionM ()
resultOr404 Nothing  = status status404 -- return 404 without a body
resultOr404 (Just a) = json a -- return JSON (also setting the content type)

Actually, the only server checking for the existance of the document is the Haskell variant. Simply because the type system enforces it!

Finally, here is the corresponding Go client code:

func updateDocument(id string) {
    doc := Document{id, "New Title", "New Text"} // set the new content
    jsonReader := getJsonReader(doc) // encode into JSON
    req, _ := http.NewRequest("PUT", base + "/" + id, jsonReader) // prepare the call
    req.Header.Add("Content-Type", jsonType) // set the correct content-type
    res, _ := client.Do(req) // execute the call
    if res != nil { // check for a response
        res.Body.Close() // close the response stream
    }
    // yeah, I ignore errors...
}

Step 2

After trying out the services with the in-memory database, I got curious and wanted to see how they’d perform using a PostgreSQL Database. So I switched the database layer to these:

The switch was pretty easy for Java and Haskell (a matter of exchanging the database driver and connect-string). For Node.js I had to rewrite more or less the whole app since there seems to be no standard interface for DB access.

The client does not need to change.

The performance of the REST services with Postgres instead of SQLite are shown below. This time they show the time needed to process 1000 documents. I wasn’t patient enough to wait for 10000 documents to finish – I took the average of three runs.

This time, Node blew Java away, Haskell was also significantly slower than Node. My guess is, that the JDBC and HDBC abstractions take their fair share of overhead, but Java’s extreme case might have another cause (I haven’t investigated here).

Conclusion

Writing v1

In general, writing the server in Java was quite easy (I’m used to that), Node.js was easy too (chaning was hard), and Haskell took me more time than the other two servers and the Go client together. I’d love to add Clojure and a Go server to the mix and see how they perform.

All three contestants work well and are reasonably fast. I never noticed Java’s problem with Postgres as shown here, but I rarely use JDBC directly these days.

Update to v2

The switch from SQLite to Postgres was pretty painful in Node.js. For Java and Haskell the switch was easy and fast.

Deployment

To deploy a Java webapp, you need a servlet container (I used Jetty) and deploy it within that. The war file usually includes all necessary dependencies. Every Node.js app starts its own server on its own port, so there isn’t much to it: install Node.js, install the required libraries with npm, Profit! Haskell produces a single binary which can be copied to the destination. Starting the binary, also starts the webserver and can thus be used directly or via a proxy. There are still some dynamic linked libraries which have to be present on the target machine.

The nice thing about Java is, that all necessary dependencies are bundled with the app itself. Haskell and its type system also check whether newer versions are still compatible, for Node.js it seems that our only option is to perform adequate testing.

Summary

This sums up my endeavour: I’ve had the most fun writing the Go client, but the Haskell services has the fewest possibilities for error. Haskell is really, really cool for writing REST services, and performs very good. Node.js is very young and provides very good productivity and performance to get something up and running, but (to me), maintaining Node.js code seems like something I wouldn’t want to do. Java is a compromise, nobody got every fired for using Java (except, maybe someone on the Android team).

What else to do with three similar REST services?↩

Programming Languages in Joy and Sorrow

2012-09-15T00:00:00+02:00

Most of us (programmers) know, and need to know, many programming languages. Some aren’t even perceived as programming languages any more (Shell-Scripts), some make us a living (Java, C# etc.), some are hard to replace (JavaScript), and some are just fun to play with (make your choice).

What makes programming languages differ is not syntax, syntax is nothing more than a mechanical translation, much like a cipher: back and forth without gaining or losing information.

General purpose language don’t even differ in what they’re able to express. All of them are “Turing Complete”¹ and hence are equally potent.

What makes a difference is how programming in these languages feel like. Even though Assembler and JavaScript are equally potent (in theory), they are two very different beasts.

What appeals to me are just a handful properties of a language:

simplicity,
expressiveness,
performance,
productivity.

Simplicity

If a programming language is complicated by itself, programs become brittle, programmers are distracted, there is no uniform structure etc. C++ is a wonderful and very expressive language, but it has so many ways of doing things, so many features to learn (and distract) that I would not care to use it again.

Simplicity was one of C’s big appeals, the language is very lean and has a small set of features. The language features were carefully chosen to be general enough to not be too restrictive.

Expressiveness

How much code do I need to write to get a job done? How good are the means of abstraction (how often do I need to repeat myself)?

Assembler doesn’t have much abstraction capabilities (mostly jumps), whereas functional languages offer powerful ways of separating, combining, and reusing code blocks.

A simple and expressive language is easy to understand, and has few, but powerful, means for abstraction.

Performance

Computers used to get faster every year for some time, currently we’ve reached a plateau and instead of scaling up, we’re scaling out by adding cores and machines.

Our programs don’t get faster by waiting for a better machine anymore. We need to actively take additional cores into consideration. The future (and probably the present) is distributed!

I think performance is still a major merit of today’s software. If your service can’t scale to “internet scale”², you’ll lose. If your competitor offers the same set of features, but twice as fast, you’ll lose.

So a modern programming does not need to be the fastest one on a single machine and a single core, but it should be reasonable fast and scale easily to threads and processes.

Productivity

Computers have become insanely fast, but programmer productivity stayed the same. Sure, we can’t call Intel for a brain-upgrade, but we can choose and provide the right set of abstractions to support us.

Besides the simplicity and expressiveness of a language, productivity involves the available libraries, the community and the culture of the community. Java, for example, has a very active Open Source culture, quite untypical for a business related language.

“Academic” (or, let’s call them “non-mainstream”) languages are often very beautiful and expressive, but their lack of practical libraries makes it hard to get up and running quickly. We’re in an age of quick-fixes and easy gains – what’s the point of choosing the newest language du jour if we can’t deliver faster (in the long run)? What’s the point of “engineering” if, in the end, we need even more time than by hacking³?

Productivity, for me, heavily affects the fun in programming and mostly subsumes simplicity and expressiveness.

Language assessment

In a business setting you rarely have a free choice of weapons, but for personal or pet projects, you can do that. Most people I know stick to the language from work because it is well known. Some are increasingly unsatisfied with their day-job language and set out for the search of their “own” language. I can’t tell you what the best language is. Everyone is different, like Yukihiro Matsumoto said here:

No language can be perfect for everyone. I tried to make Ruby perfect for me, but maybe it’s not perfect for you. The perfect language for Guido van Rossum is probably Python.

You are different from everyone else. Embrace the difference. Use your brain and make you own choice.

Of course, given the sheer amount of programming languages, making an educated guess is crucial. Taking only the 100 most popular languages into account is surely not too far off.

Go Philosophical

Languages are divided into a few categories concerning programming paradigm (object oriented, functional, procedural…), typing (strong, weak, static, dynamic) etc.

My personal preference is with statically typed functional languages because they provide very good abstractions and safety through the compiler.

Look for a language with good interop functionality.

We’ve invested a lot of time in building libraries, helpers, utilities etc. Of course starting from scratch is fun, but it’s rarely economical.

If you used to program in Java, start looking for JVM languages for example.

Read some code before diving in

Typing.io is a nice starting point for some languages, and of course, there is always github.

Do you feel comfortable with the layout, the language, does it feel natural? Do you even understand some of it?

Enough with the subjective assessment, show me something objective!

Here is an unbiased and objective survey of a few programming languages I’m interested in. It’s totally foolproof and you certainly should start a multimillion dollar company on it!

This chart shows the ratio of the number of search results for

"c programming" "sucks"

and

"c programming" "rocks"

as returned by Yahoo!.

That is, "c programming" "sucks" returns 20800 results, "c programming" "rocks" returns 222000 results, hence a ratio of 10.6731 in the chart. Thus, everything below 1 means, there are more “sucks” results than “rocks” results.

Since Haskell, Clojure, and Scala completely dominate the chart, here a version with the obvious winners removed:

The numbers were computed using Google Docs - Spreadsheets and its awesome ImportXML function:

=ImportXML("http://search.yahoo.com/search?p="&C4,"//span[@id='resultCount']")

Cell C4, referenced in the above code, contained the URL encoded search-string, for example %22c programming%22 %22rocks%22. Try it with prolog ;-).

Even some configuration files are considered Turing Complete, like sendmail’s.↩
Whatever internet scale is in your domain.↩
You’ve got to measure the whole lifetime of a project, it doesn’t help to get something working very quickly but with huge amounts of bugs, hard to change etc. I guess that’s a topic for a whole book on its own.↩

In the beginning was the Test...

2011-10-08T00:00:00+02:00

JUnit offers many features besides the standard assertTrue/assertEquals methods most programmers use. Let’s browse through the newer and more exotic features. They might come in handy at some time.

JUnit 4.9 Feature Roundup

Assuming you know something about unit testing and JUnit in particular, I won’t start at the very bottom, but talk a little about the features introduces during the last few versions:

Matchers
Assumptions
Categories
Theories
Rules

I hope there is something new in here for you. JUnits javadoc documentation is very good, but there is no single place describing these features. It’s not my goal to give a thorough treatment of them here, but it might be a good starting point.

Matchers

By including Hamcrest (core) into the default JUnit distribution, JUnit now allows the usage of assertThat leading to much easier to read tests and better error messages:

@Test
public void testUsingAssertThat() {
  assertThat(42, is(greaterThan(43))); // note, this will fail
}

JUnit includes only the Hamcrest core matchers, if you want/need more matchers, include hamcrest-all 1.1. Included matchers are documented here for Hamcrest and here for JUnit additions.

Output:

java.lang.AssertionError: 
Expected: is a value greater than <43>
     got: <42>

Assumptions

Assumptions allow tests to be ignored if the assumed condition isn’t met (instead of failling).

This test will be ignored if it is run on a Windows OS (for example):

@Test
public void testUsingAssumeThat() {
  assumeThat(File.separator, is("/"));
  ...
}

It is also possible to use assumptions in @Before or @BeforeClass methods.

Output (for example):

Test 'org.interlinked.junit.assumption.BasicTest.testUsingAssumeThat' ignored
org.junit.internal.AssumptionViolatedException: got: "\", expected: is "/"

Theories

With theories we can write parameterized tests. We define a few theories and some datapoints. JUnit will match the types of the datapoints and the theories.

Again, we have to use a special suite class Theories:

@RunWith(Theories.class)
public class TheoryTest {
  @DataPoint public static final String POINT1 = "POINT1";
  @DataPoint public static final String POINT2 = "POINT2";

  // mind the plural!
  // uses only the items of the array, never the whole array!
  @DataPoints public static final String[] POINTS = new String[] {"abc", "cde", "efg", "ghi"};

  @DataPoint public static final String[] POINTS_ARRAY = POINTS;

  @Theory
  public void testTheory(String param) {
      System.out.println("Got: " + param);
  }

  @Theory
  public void testTheoryWithTwoParams(String param1, String param2) {
      System.out.println("Got " + param1 + " and " + param2);
  }

  @Theory
  public void testArray(String[] array) { // gets called with POINTS_ARRAY, nothing else
      System.out.println("Got called...");
      assertThat(array.length, is(equalTo(POINTS_ARRAY.length)));
  }
}

Output:

Got POINT1 and POINT1
Got POINT1 and POINT2
Got POINT1 and abc
Got POINT1 and cde
Got POINT1 and efg
Got POINT1 and ghi
Got POINT2 and POINT1
Got POINT2 and POINT2
Got POINT2 and abc
Got POINT2 and cde
Got POINT2 and efg
Got POINT2 and ghi
Got abc and POINT1
Got abc and POINT2
Got abc and abc
Got abc and cde
Got abc and efg
Got abc and ghi
Got cde and POINT1
Got cde and POINT2
Got cde and abc
Got cde and cde
Got cde and efg
Got cde and ghi
Got efg and POINT1
Got efg and POINT2
Got efg and abc
Got efg and cde
Got efg and efg
Got efg and ghi
Got ghi and POINT1
Got ghi and POINT2
Got ghi and abc
Got ghi and cde
Got ghi and efg
Got ghi and ghi
Got called...

Rules

Finally, rules allow us to add behaviour to tests. They can be thought of some kind of AOP for JUnit. Using rules, we can often omit class hierarchies and still reuse functionality using delegation.

JUnit includes some rules to start with, but it is very easy to write our own rules.

public class RuleTest {
  @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
  @Rule public TestName testName = new TestName();
  private static boolean fileCreated = false;
  @Rule public LoggingRule loggingRuld = new LoggingRule();

  @Before
  public void printTestName() {
    System.out.println(testName.getMethodName());
  }

  @Test
  public void testCreatingAFile() throws IOException {
    File newFile = temporaryFolder.newFile("test1");
    assertThat(newFile.isFile(), is(true));
    fileCreated = true;
  }

  @Test
  public  void testCheckIfItExists() { // depends on testCreatingAFile...
    assumeTrue(fileCreated); // just to be sure ;)
    File file = new File(temporaryFolder.getRoot().getAbsolutePath() + "/test1");
    // the file should not exist (unless we use ClassRule for the TemporaryFolder, for example)
    assertThat(file.isFile(), is(false));
  }

  @Test(expected = NullPointerException.class)
  public void testThrowException() {
    throw new NullPointerException();
  }
}

Output:

Starting: testCreatingAFile
testCreatingAFile
Finished: testCreatingAFile
Starting: testCheckIfItExists
testCheckIfItExists
Finished: testCheckIfItExists
Starting: testThrowException
testThrowException
Finished: testThrowException

The TemporaryFolder and TestName rules are included in JUnit, the LoggingRule is a simple example:

public class LoggingRule extends TestWatcher {
  @Override
  protected void starting(Description description) {
    System.out.println("Starting: " + description.getMethodName());
  }

  @Override
  protected void finished(Description description) {
    System.out.println("Finished: " + description.getMethodName());
  }
}

Other rules included (see JUnit’s javadoc):

ErrorCollector: collect multiple errors in one test method
ExpectedException: make flexible assertions about thrown exceptions
ExternalResource: start and stop a server, for example
TemporaryFolder: create fresh files, and delete after test
TestName: remember the test name for use during the method
TestWatcher: add logic at events during method execution
Timeout: cause test to fail after a set time
Verifier: fail test if object state ends up incorrect

Unfortunately, rules seem to be local to the defining class, so you can’t put the into the suite class like @Before and @BeforeClass (which is really nice for opening external resources once for all tests).

Misc additions

Infinitest

For each change you make, Infinitest runs all the dependent tests. It’s continous testing for Eclipse and IDEA - free and open source (written by inproving works)!

ClasspathSuite

Most IDEs have their own ways for finding test classes to run, but usually I like to be IDE independent. Using the ClasspathSuite it is possible to have JUnit detect all test classes (or a subset of them) within the classpath (written by Johannes Link). There are efforts to include it into the standard distribution of JUnit.

Notebooks (dead wood)

2011-07-22T00:00:00+02:00

Moleskines were all the rage about six years ago. Whole blogs were, and still are, dedicated to them. Today, still many people carry these little black books.

I too loved these little books, but in the last few years I’ve changed some parameter of my journal every time I needed a new one. I changed the brand, the size, I used soft and hard covers, plain, square or ruled (although, they were always black).

It was a fun experiment, but now I’ve come to a conclusion of what suits my needs best: Leuchtturm 1917

Master (A4+) and
Medium (A5)

either dotted or plain.

The Master is large enough for keeping full A4 pages. I use it for plannings, sketches, technical stuff etc. Most of the time it sits on my desk and waits to be filled - usually one to two pages per day.

The Medium is just the right size for a journal. It fits easily in almost every bag, is large enough for notes, quotes, thoughts and so on. The smallest format, A6, is also quite nice. But for me, the Medium is simply the best choice.

The following sounds like an advertisement, but the tagline of the Leuchtturm 1917 company is “Details machen den Unterschied” (“Details make all the difference”), I have to include some of their noteworthy details:

Ink proof paper - earlier editions of the notebooks used a thinner paper (much like the paper used in Moleskine notebooks), but recent editions use a 80g/qm paper - perfectly suited for a fountain pen.
The “Dotted” variant is a nice compromise between plain and square, it’s unobtrusive and still provides enough guidance.
Numbers and Index - the pages are numbered and a small index is printed on the first few pages.
Detachable sheets - some companies offer notebooks with detachable sheets, sometimes you can rip out half of the notebook. Leuchtturm only has 8 sheets which is more than I ever ripped out of a notebook, so I consider this a plus.
Labelling stickers - quite nice for organisation-fanatics, during usage we have the slick plain look of the black notebook we all love, but for archiving purposes there are a few labelling stickers in each notebook package.

Even though we’ve access to email, note-taking applications, the Web etc. almost all the time, I still love the old classy feeling of a notebook and (in my case) a fountain pen.

I’ll end this post with a few links for your reading/tinkering pleasure.

How to Hide Google's "Google Plus Notification Count"

2011-07-19T10:00:00+02:00

To hide the Google Plus notifications on the upper right of Google’s services, install Adblock (Adblock Plus for Firefox or Adblock for Chrome) and add this rule to your ruleset:

##A#gbg1

The notification count is a link (A) with id gbg1, this rule hides that element. Let me know if it works for you, or not.

JavaScript

2011-07-19T00:00:00+02:00

JavaScript was developed 16 years ago by Brendan Eich. The majority of our Web experience relies heavily on JavaScript. Without it, we’d still have to reload a page for every click, we wouldn’t have the ability to interact in a sane way with the Web, and - I speculate - the Web wouldn’t be as omnipresent as it is now. Usability matters.

JavaScript is also one of the worst languages I’ve ever seen. It has many sharp edges and rough parts. Anyways, it also has very unusual, and hence, interesting parts like prototypical inheritance and functional aspects.

Even though, JavaScript is a standardized language, ECMAScript, there are still some undefined parts. The version that almost all Browsers support, ECMAScript3, was a turning point for the modern Web, ECMAScript4 was too bold and didn’t happen. And now ECMAScript5 is coming. ECMAScript5 still has some of the weird parts, but made a few clear cuts. Now the remaining inconsistencies between newer JavaScript implementations in browser shouldn’t be that bad. Of course, this doesn’t help much since there is always an IE6 around the corner.

The domain of JavaScript is the browser, I suppose that more than 99% of the worlds JavaScript code runs within a browser. But it is not limited to the browser. There are JavaScript implementations that can be embedded in other applications (e.g. Rhino to embed it into Java), or the awesome NodeJS project which allows JavaScript to be used as a general purpose language (with focus on network applications).

JavaScript seems to invade our lives wherever it can. Mac OS X Dashboard widgets, browser plugins, Smartphone apps, Webservers etc. there is always a way to use JavaScript!

With the advent of HTML5, there are even more possibilities like opening WebSockets etc.

Few languages are so clearly worth learning as JavaScript. - It’s an interesting language that doesn’t restrict how you use it. - It’s a language you get paid for developing in. - It’s still cool (who cares about Java anymore?). - It’s here to stay (for some time, though).

So, to sum it up, I believe investing in JavaScript pays off. Even if you’re an “enterprise developer” or something. JavaScript will get you, sooner or later, so get it first!

As Douglas Crockford said, JavaScript is a misunderstood language because few people take time to learn it properly. Take your time and get familiar with it. Two resources I’d recommend are: - Douglas Crockford - JavaScript the Good Parts (pre ECMAScript5) - David Flanagan - JavaScript - The Definitive Guide (covers ECMAScript5)

Don’t get too fixed on ECMAScript5, it can be easily “emulated” with ECMAScript3, and few browser fully support it by now.

Some cool stuff in, with or about JavaScript: - Booting the Linux Kernel in JavaScript - right in your Browser - CoffeeScript a scripting language that compiles to JavaScript - Protovis - one of many visualization libraries - JSLint - to prevent pitfalls - JavaScript - the lingua franca of the Web - ECMAScript 5.1 Specification - JavaScript Design Patterns - AES in JavaScript

How to dive into Legacy Code

2011-07-18T00:00:00+02:00

Diving into legacy code written some time ago can be a daunting task. It doesn’t even matter much if we’ve been writing it ourselves, or somebody else, code rots faster than we’d like to admit.

Currently faced with such a task I tried to do it in a systematic and repeatable manner.

My steps

First try to find the modules and their dependencies. I used IntelliJ IDEA for my current Java project. Since it also uses Maven, finding the dependencies was easy.

Create a graph of module interdependencies. Which modules depend on which? Find the “edges” of the system (modules that do not depend on other modules). I found them to be the best starting point for a more detailed analysis.

Find out what the purpose of each module is. Is it a layer in the system (like a DAO-module)? Is it a cross-cutting concern (model classes)?

The next step is to analyse each module by itself. For this step I recommend using doxygen. It can generate a very good documentation of the software at hand, even if no doxygen (or any other type of markup) was used, by analysing the dependencies, class hierarchies, call graphs of the program. doxygen supports many languages, chances are high yours will be too.

To get the most out of doxygen, I’ve used the following configuration file which enables many of the advanced analysis features (like call graphs etc): doxygen.config.

You have to edit the file and provide - at least - the input and output directories! After that, it’s simply a doxygen doxygen.config.

To generate this kind of documentation easily from Maven, here is a similar doxygen-maven-plugin configuration:

    <build>
        <plugins>
            <plugin>
                <groupId>com.soebes.maven.plugins.dmg</groupId>
                <artifactId>doxygen-maven-plugin</artifactId>
                <configuration>
                    <projectName>${project.artifactId}</projectName>
                    <projectNumber>${project.version}</projectNumber>
                    <optimizeOutputJava>true</optimizeOutputJava>
                    <extractAll>true</extractAll>
                    <extractStatic>true</extractStatic>
                    <recursive>true</recursive>
                    <exclude>.git</exclude>
                    <excludePatterns>*/test/*</excludePatterns>
                    <inlineSources>true</inlineSources>
                    <referencedByRelation>true</referencedByRelation>
                    <referencesRelation>true</referencesRelation>
                    <hideUndocRelations>false</hideUndocRelations>
                    <umlLook>true</umlLook>
                    <callGraph>true</callGraph>
                    <callerGraph>true</callerGraph>
                    <generateLatex>true</generateLatex>
                </configuration>
            </plugin>
        </plugins>
    </build>

The generateLatex option is nice if you wish to produce PDF files (for viewing on a Kindle for example).

With this plugin configured in you pom.xml, mvn doxygen:report is your workhorse.

If you’re unsure if the generated documentation is worth it, take a look at the doxygen documentation of JUnit 4.8.2 (zip file, 7.6MB).

Haskell Books and Tutorials

2008-10-24T00:00:00+02:00

Haskell is really a language very worth knowing. It does many things so different than most other languages which I really enjoy.

I’d like to use this post to mention a few really good resources for learning Haskell:

Learn you a Haskell

Learn you a Haskell is a very fun and entertaining tutorial for Haskell very much in the spirit of Why’s poignant guide to Ruby.

It isn’t finished yet, but it’s really good to start with. (Via Wadler’s Blog)

Real World Haskell

Real World Haskell is a upcoming book I really look forward to. It is freely available on its website, so check it out.

Wikibook

There is also a good collection of Haskell topics on Wikibooks – Programming/Haskell

Wikibook is more interesting if you already know a bit of Haskell and would like to understand it more in depth.

Others

I liked Yet another Haskell Tutorial very much, this aims at people with a bit of background, though.

Another more recent book on Haskell is Programming in Haskell which is nice, but also quite basic IMO.

Of course there is always the Gentle introduction to Haskell

Enjoy.

Git Overview

2008-08-24T00:00:00+02:00

My version control journey started with CVS, after that I looked at SVN, but never really used it. The shortcomings of centralized repositories were too obvious and with my increasing interest in Haskell I jumped on the distributed version control train with Darcs. I really, really liked it, but it had some nasty things too. After a while I was looking for something different and stumbled over Mercurial, again I was really happy with it but somehow my journey wasn’t over yet.

Recently I was watching two Git techtalks on YouTube:

The two talks are very different. The first one is more about why there’s place for yet another source code management with the rare opportunity to see Linus Torvalds giving a talk. The second one is about Git’s technicalities (commands etc.).

Basic Terms

Git distinguishes between the workspace, the index/staging area, the object store and remote repositories.

The workspace is where you do your actual work. It contains tracked files, untracked files and a special directory “.git”.

The index (or staging area) is used for preparing commits. You can add files to the next commit or even only parts of files for the next commit. All the changes you’d like to commit get into the index first. A commit takes everything in the index and persists it in the object store.

The object store isn’t really an issue here, and remote repositories are more or less clear what they are.

Quick Intro

For a more complete introduction see the Git tutorial (the cool stuff is in the next section).

Somewhere on the KDE Project I found a really nice Cheat-Sheet for Git.

To create a new repository just issue

git init

To start tracking files (or content) issue

git add <files>

To commit the added changes (note, only the state of the files when you issued
add will be recorded) issue

git commit

To push changes to a remote repository (ssh, filesystem etc.) issue

git push <remote url>

To clone a complete repository onto your local disk issue

git clone <remote url>

I like to do some general configuration for all my projects, something like this:


git config --global user.name "Your Name"
git config --global user.email your.mail@host.com
git config --global color.ui auto
git config --global color.interactive auto

This tells Git who you are and enables some nice color-features of Git.

To get the most out of Git learn one of these commands:

git help <command>
man git-<command>

So, git help config is equivalent to man git-config, similarily, git config... is equivalent to git-config.... The latter seems to be discouraged (since Git switches to a library approach or something), but is often used in tutorials or manuals.

That may be the most used commands with Git, the more interesting ones – and the ones for which you may want to switch to Git follow:

Commands

Adding/Removing changes to/from the Index

To add all changes in a file, simply use add, some examples:

add . adds all files in the current directory and its sub-directories to the index
add <file> adds a specific file to the index
add -u adds all changed files to the index (that is, it adds alls files Git knows about and that have changed)
add --patch lets you decide whether you want to include smaller changes within a file instead of all changes in the file(s) see The Thing About Git for a larger example. This is extremely useful if you accidentally made two unrelated changes to a file and you want to keep your commits clean (you do want to keep your commits clean, don’t you?!)
add -i is an interactive version of add with a text-based GUI combining all the above capabilities

If you accidentally added some changes to the index, simply use reset to clean it, or specify the file to remove from the index, like reset <file>. reset can do more than simply cleaning the index, you can use it to restore the state of the latest commit by using reset --hard. reset also takes a parameter telling it which commit to consider – instead of using the latest HEAD, use HEAD^ for the parent of the actual HEAD, or give it a SHA1 hash… . Take a look at this post for a better treatment of git reset and its possibilities.

Checking differences between commits/index/working directory

Sometimes you need to know what exactly the differences between two commits or the working directory and the index (or some commit) is.

This is actually fairly trivial with any revision control system, and of course also with git.

diff compares your working directory with the index
diff --cached compares your index with the latest commit (or any named commit you supply)
diff <commit> compares your working directory with the named commit (use HEAD for the latest commit)
diff <commit> <commit> compares (surprise) two named commits (for example, diff HEAD HEAD^ shows the differences from the last two commits)

To get a high-level view of changed files, Git offers the status command.

status shows a summary of changed files, files added to the index and files not tracked.
status -a shows what the index would look like if you issued add -u or a commit -a (that is, all changed files get commited or added to the index).
status -u {no, normal, all} using the -u parameter, Git’s behavior concerning untracked files can be changed (don’t show untracked files, show only files and directories, or show all untracked files recursively).

Another way to get an overview of your controlled content is by using ls-files:

ls-files shows all tracked files
ls-files -t shows files with some additional information (like cached, modified etc.)
ls-files --others shows untracked files

Somehow I find this more interesting than the status output, but this may come from my Mercurial experiences where hg status would output a very similar list as ls-files -t.

Most projects have some files which should not be tracked, for example binaries built from source, or backup-files from your editor… . Using .gitignore you can tell Git to always ignore these files (unless you order it to show you also ignored files, though).

There are several ways to tell Git what to ignore. The most common one is by creating a .gitignore file in the root of your project. This file could be added to the index and would be shared. The nice thing with this file is, that you can keep one in each sub-directory overriding the configuration of the parent directory. See gitignore for more.

The second way, local to your repository, is to edit the file .git/info/exclude to your needs. On Mac OS X, Git automatically adds .DS_Store files to that exclude file.

The third way is to specify a core.excludesfile configuration (for example git config --global core.excludesfile "/absolute/path/.gitglobalexclude") and create a global ignore file (I think an absolute path is needed, but a patch enabling relative paths was submitted).

The format of the files is simple:

!file never ignore this file (use !asdf.o if you excluded *.o, but you want to track asdf.o)
/file ignore the file in the root of the project
file ignore the file anywhere in the project
*.o ignore all files with an “.o” ending
dir/ ignore the directory “dir” but not the file “dir”

Commiting changes

So, let’s say we have a clean collection of changes in the index. The next step is to commit it.

The basic command is simple, just commit, but you can do more.

commit -a commit all changed files (adding every changed file to the index)
commit -m "commit message" supply the commit message at the command line
commit --amend edit the previous commit (that is, the new commit merges with the old one)

With Git it’s common to make branches for bigger experiments. Use a branch for example if you want to implement a new feature which requires many changes in your code-base, but it takes too long for a single commit.

So, to make a branch from the current HEAD, just issue checkout -b name. This creates a branch named “name” and switches to it. This is equivalent to branch name checkout name.

In this branch you can do whatever is needed to implement the new feature, but if necessary, you can switch back to the old branch (for maintenance or something). After you’ve completed your changes, just merge the branch with the new feature back to the original branch merge name. After that you can safely delete the “name” branch branch -d name.

The following picture shows two branches, Branch B is created during the working directory (red line) points to Branch A, we can switch between the branches with checkout and finally delete Branch B.

Of course there can be conflicts while merging the changes from the other branch into a branch which has evolved by itself. Either you resolve the conflicts all at once, or you keep your branch with the new feature more or less up to date with rebase. After each change to the master branch, you could issue a rebase master in the new branch to get the new changes of the master branch incorporated into your “name” branch.

Another very nice feature of rebase is to merge many commits into a single commit (or edit some of them). Suppose you made your changes within a new branch, now you’re ready but you’d like to have a single commit containing all changes of the branch. Just issue rebase -i master, and Git with alls commits leading from master to the current HEAD. Replace each “pick” with “squash” and Git merges all the commits into a single new one (you get to supply a new commit message). Look at git awsome-ness for a complete example.

The following diagram shows how commits are connected when a branch is made.

From the branch we can do a merge (first picture) or a rebase (second picture). Note that the merge automatically issues a commit if no conflicts arise!

Here a small illustration what you could do with a rebase command:

For smaller changes there’s also a special holding area for changes. Suppose you’re in the middle of a bigger change and you have to quickly fix something completely unrelated. Just issue stash and Git will put away all your current uncommited changes and present you with a clean working directory. Now make the quick-fix, commit and issue stash pop (equivalent to stash apply and stash drop) to continue working on your fancy new feature.

It’s also possible to stack stashes up, use stash list to see a list of stashes. If you come to the conclusion that your stashed changes deserve their own branch, issue stash branch foo to create a new branch “foo” and pop the latest stashed changes.

You never should temper with commits you already published to a remote repository (or other people)! If you find a mistake in a published commit use revert <commit> which records a special commit telling other people that something isn’t right with the specified commit.

Sharing or “backing up” your work

Sometimes you’re not the one who started the project, or you simply reinstalled or changed the computer. How do you get your repository back?

Nothing simpler than that: clone <url>. Unfortunately, clone cannot clone to a remote repository. I can’t clone my local project to my backup server – I have to either scp it to it or create a project there and pull the changes. After that I can use the standard command to publish or receive changes.

clone sets up all necessary configurations and you can simply issue pull to update your repository with remote changes or push to send your changes to the remote repository.

In a distributed environment it’s not uncommon that you might want to push or pull from different remote repositories. For that reason, Git allows you to configure shortcuts to other repositories. The main remote repository is called “origin”, and you usually push or pull the master branch. pull is a twofold operation, it first fetches the contents (using fetch) and then merges the new contents with your changes. Of course you could also use fetch, inspect the changes and merge afterwards.

So, how to you want clone the Git repository:

git clone git://git.kernel.org/pub/scm/git/git.git

After the clone, we can look at the remote sites and their respective URLs with remote -v. To look which branches are available at the remote site, issue branch -r.


me@host:git % git remote -v 
origin	git://git.kernel.org/pub/scm/git/git.git
me@host:git % git branch -r
  origin/HEAD
  origin/html
  origin/maint
  origin/man
  origin/master
  origin/next
  origin/pu
  origin/todo

Now let’s add another repository containing the Git project (note the git2 subdomain):

git remote add mirror git://git2.kernel.org/pub/scm/git/git.git

This tells Git to add the given URL with the shorthand “mirror” and to track all branches. If you’d like to track only some branches, for example only the master branch add -t master to the command.


me@host:git % git remote -v 
mirror	git://git2.kernel.org/pub/scm/git/git.git
origin	git://git.kernel.org/pub/scm/git/git.git
me@host:git % git branch -r 
  mirror/html
  mirror/maint
  mirror/man
  mirror/master
  mirror/next
  mirror/pu
  mirror/todo
  origin/HEAD
  origin/html
  origin/maint
  origin/man
  origin/master
  origin/next
  origin/pu
  origin/todo

You can either switch to a remote branch with checkout origin/todo or create a new branch based on some remote branch checkout -b myownbrach origin/todo, the latter also sets up “tracking” which means that push and pull automatically use the branch “todo” at the remote repository “origin”.

Another very cool feature of Git (and of course others) is the ability to “cherry-pick”. That is, we could take the patches (commits) interesting to us and incorporate them into our own branch. To accomplish that task, there’s cherry-pick at our disposal. Simply choose the commit you’d like to import from another branch and call cherry-pick <commit> at the branch you’d like to incorporate the patch (commit). Works like a charm.

How I use Git

Note that I just switched to Git, but since Mercurial was very similar I know most of the concepts to some degree.

I use version control for most of my files, be it configuration files in my home directory which get shared across my laptop, desktop and work machine or letters I write. Of course Git isn’t the best way to do everything, but it’s a better way than not syncing my files at all.

I have three “work-machines” which synchronise to a server accessible by all of my machines. For machine-specific files (in my home-dir for example) I made a special directory “.machinedependent” and created sub-folders for each machine. In this directories I put the configuration files I’d like to be version controlled even if the are only interesting for one machine. Then I hard-link files and soft-link directories and I’m done. Git handles this quite nicely.

All of my coding projects are handled by Git and stored (for backup purposes and accessibility) on the mentioned server. I like the fact that my code is distributed across several machine nowhere near each other. Git-home-history has some capabilities to produce encrypted archives of your work too, but I didn’t dig into that by now.

This, of course, didn’t touch more than the bare surface of Git. Look at

for more information.

Converting from Mercurial to Git

I downloaded hg2git and converted all my repositories to Git (you should be in an empty directory and it may be an advantage to use zsh):


for i in path/to/repos/*
do
  proj=$i:t
  mkdir $proj
  cd $proj
  git init
  hg2git.sh -r path/to/repos/$proj
  git checkout default
  git checkout -b master
  git branch -d default
  cd ..
done

Missing some things from Mercurial

There are actually two features I somewhat miss in Git. First I cannot clone my local repository to a remote server without hassles (login to the remote server and create an empty project for example, or just scp the whole project over… something like that).

Second, Mercurial had very nice short-cuts. For example I could issue hg st instead of hg status, Git also has aliases, but you have to configure them on your own.

git config --global alias.ci "commit -a" adds an alias “ci” to commit all updated files (instead of adding each changed file individually to the index).

Which one is the right one?

Well the question of “which version control software should I use” comes up sooner or later (for some people so late, they won’t even notice it). I cannot tell you which one is right, the choice between centralized source code management and a distributed one is the first step and gives different possibilities (distributed is the way to go, IMHO).

Anyway, you’ve got plenty to choose from.

For me only Darcs, Mercurial and (only recently) Git were interesting enough to dig deeper. I dumped Darcs because installing it was not so easy, I’m not root on all the machines I have access to and to compile a Haskell program into my home-dir was too much work for me. And of course I heard about some nasty bugs in it… .

Mercurial was awesome (take a look at the Mercurial TechTalk), but Git won me over for absolutely no reason (mabye because it was envisioned by Linus Torvalds, maybe because I like C way more than Python).

I think most of the time we are not the ones to choose the version control software, anyway.

Mercurial Version Control Status in the ZSH Command Line Prompt

2008-08-07T00:00:00+02:00

I sometimes forget to push or pull changes to or from a remote repository. To remedy the problem I wrote myself a little script to show me the status on the prompt.

The script is written for zsh and Mercurial, but can be easily adapted to other version control systems (I guess).

Everytime you change a directory the scripts looks up if there are uncommited changes, remote changes to be pulled or local patchsets to be pushed.

The prompt of a directory within a project with uncommited changes would look something like this:

  user@C2D:~/root-dir/subdir %            root-dir: U

Possible states:

U: Uncommited changes (use hg commit)
I: Incoming patchsets (use hg pull)
O: Outgoing patchsets (use hg push)

I have configured Mercurial to automatically update my repository after a pull, but some people might wish to add another status for “pending updates” or something like that.

The script uses “hg incoming -q” or “hg outgoing -q” to check for sets to be pulled or pushed, so you should configure the default paths in your .hg/hgrc file for pulling or pushing like this:

  [paths]
  default-push = ssh://yoursever.tld/Projects/PlanToTakeOverTheWorld/
  default = ssh://yourserver.tld/Projects/PlanToTakeOverTheWorld/

Unfortunately there are a few problems:

Accessing remote repositories over some network is costly. If you have a slow network connection you might want to temporarily switch off the remote status commands by using “export VCSNOREMOTE=true” please note that “export VCSNOREMOTE=false” has the same effect, in fact any string would switch off the remote commands. To enable them again use “export VCSNOREMOTE=”.

The second problem might be that you have to enter your credentials to access remote servers quite frequently. Use some sort of ssh-agent or something similar to avoid typing in your password all the time.

The third thing to remember is that updating the status with every prompt is useless, so I opted for updating after changing a directory (also if you type cd .) but at least after 60 seconds.

Installation: save anywhere on your disk and source it in your .zshrc file.

Download: vcs_status_prompt

Free and Open Source Math Programs

2008-02-24T00:00:00+01:00

In this post I’ll go through some of the most prominent math programs available with source code.

This is by no means “original” work, I just collected the headlines and links to various mathematical software projects out there. This started with an offer of my University (Mathematica for 13 Euro), but I don’t want to invest time in a tool I won’t have (free, or almost free) access to for the rest of my life.

The programs differ in their respective foci, the main areas are Computer Algebra System, Numerical Analysis, Statistics and general purpose desktop calculators.

My current choice is SAGE and R for my math and statistics needs. For simple calculations I usually use the calculator included with the OS or some interactive programming-shell (ghci or irb).

Hybrid

SAGE

SAGE is the new star on the open-source math sky. Essentially it is a CAS written in Python, but it provides interfaces to many math programs available. This includes commercial programs like Mathematica and Maple as well as free programs like Maxima, Octave or Singular.

From the homepage:

Each interface requires that the corresponding software is installed on your computer. SAGE includes GAP, PARI, Singular, and Maxima, but does not include Octave (very easy to install), MAGMA (non-free), Maple (non-free), or Mathematica (non-free).

SAGE runs as service on your server or desktop and you use your browser to interact with it (you can also use the command-line interface, though). It is very well documented and gives you access to the source code of the algorithms right within your browser.

Use SAGE for studying a huge range of mathematics, including algebra, calculus, elementary to very advanced number theory, cryptography, numerical computation, commutative algebra, group theory, combinatorics, graph theory, and exact linear algebra.

The goal of the SAGE project, which started 2005, is to provide the best mathematical software in the world.

SAGE is pretty large (on my Mac it uses 722MB right after the installation), but you can find even more (optional packages etc) at their homepage.

It’s also possible to interact with it via Texmacs or Emacs (again, available from their download page).

SAGE is available for Windows, Linux and Mac OS X.

Computer Algebra System

Maxima

Maxima is a system for the manipulation of symbolic and numerical expressions, including differentiation, integration, Taylor series, Laplace transforms, ordinary differential equations, systems of linear equations, polynomials, and sets, lists, vectors, matrices, and tensors. Maxima yields high precision numeric results by using exact fractions, arbitrary precision integers, and arbitrarily precision floating point numbers. Maxima can plot functions and data in two and three dimensions.

Maxima is very mature and stood the test of time. It started in 1982 and is “free” since 1998.

Maxima is included in SAGE.

Axiom

Axiom is a general purpose Computer Algebra system. It is useful for doing mathematics by computer and for research and development of mathematical algorithms. It defines a strongly typed, mathematically correct type hierarchy. It has a programming language and a built-in compiler.

Yacas

YACAS is an easy to use, general purpose Computer Algebra System, a program for symbolic manipulation of mathematical expressions. It uses its own programming language designed for symbolic as well as arbitrary-precision numerical computations. The system has a library of scripts that implement many of the symbolic algebra operations; new algorithms can be easily added to the library. YACAS comes with extensive documentation (hundreds of pages) covering the scripting language, the functionality that is already implemented in the system, and the algorithms we used.

Homepage Downlaods, Screenshots, Tutorial, Manual and Demo are available there.

PARI/GP

PARI/GP is a widely used computer algebra system designed for fast computations in number theory (factorizations, algebraic number theory, elliptic curves…), but also contains a large number of other useful functions to compute with mathematical entities such as matrices, polynomials, power series, algebraic numbers etc., and a lot of transcendental functions. PARI is also available as a C library to allow for faster computations.

PARI/GP is included in SAGE.

Singular

SINGULAR is a Computer Algebra System for polynomial computations with special emphasis on the needs of commutative algebra, algebraic geometry, and singularity theory.

Singular is included in SAGE.

GAP

GAP is a system for computational discrete algebra, with particular emphasis on Computational Group Theory. GAP provides a programming language, a library of thousands of functions implementing algebraic algorithms written in the GAP language as well as large data libraries of algebraic objects.

GAP is included in SAGE.

Macaulay2

Macaulay 2 is a software system devoted to supporting research in algebraic geometry and commutative algebra.

Numerical Analysis/Computation

Octave

GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Euler

Euler is a powerful interactive numerical laboratory. It quickly evaluates numerical functions, visualizes results, and allows to test and program numerical algorithms. The system can handle real, complex and interval numbers, vectors and matrices, and additionally a long data type for exact computation. Most routines are written in the high level Euler language and can be modified or extended by the user.

Scilab

Scilab is a scientific software package for numerical computations providing a powerful open computing environment for engineering and scientific applications.

Statistics

R

If you want to do some statistics there is no way around “R”. At least not in the free and open-source software world.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible.

I think R is a tool everyone should be familiar with, at least with the most basic operations. Statistics is such an important every-day tool you can immediately put to work as soon as you know something, anything, about it.

Homepage
Documentation
Tutorial
Screenshots
Books (currently 53 books!)

Command-Line/Desktop Calculator

Sometimes you just want to calculate very simple things, for this task a full fledged system isn’t really appropriate.

bc

bc is an arbitrary precision numeric processing language. Syntax is similar to C, but differs in many substantial areas. It supports interactive execution of statements. bc is a utility included in the POSIX P1003.2/D11 draft standard.

bc is usually installed on any Linux/Unix system or available via its package management.

“Other” command-line/desktop calculators

Essentially you can use any interactive shell, like Python’s python or Ruby’s irb as well as Haskell’s ghci or hugs.

GUI Calculators

Despite the included Desktop Calculators there are some very fine products out there, here some pearls I could find:

Mac OS X
- Magic Number Machine
Linux
- Qualculate!
- Calcoo
Windows
- Calcoo for Windows
- Any suggestions?

If nothing else, there is always Google Calc.

Emacs Basics

2008-02-24T00:00:00+01:00

It’s been a while since I wrote my Vim Introduction and Tutorial (exactly one year). A lot happened between now and then, I chose to get a better feeling about Emacs for example.

The reasons aren’t easily explained; The most prominent reason is the awesome AucTex-mode since I’m working heavily with LaTeX lately.

Anyways, learning Vim and Emacs is better than learning only one of them :-).

Vim started in 1991 and is based on Bill Joy’s Vi written in 1976/77. Emacs started as set of macros for the TECO editor 1976 and was rewritten in 1984 (GNU Emacs). So both editors are based on concepts invented more than 30 years ago and both editors are very unlikely to change any time soon.

I still like Vim’s way of doing things, for example the basic commands are very easy and very fast to reach.

At first I wanted to compile something similar to my Vim Tutorial, but I ended up publishing just a few notes and more or less a cheat-sheet for Emacs. If you like it more official look at the Emacs Guided Tour and its very fine documentation.

The best way to get up and running with Emacs is the included tutorial (start by typing C-h t).

I’m using Aquamacs on my Mac and GNU Emacs on Linux. Both use GNU Emacs 22 at its core, but especially Aquamacs provides some extensions to make it (almost) seamless on OS X.

Basic Concepts

You can’t learn Emacs and lean back, every mode Emacs provides offers its own key bindings and has its own specialities you have to know.

So, it doesn’t matter if you use Vim for day-to-day tasks and Emacs for special purpose text editing (like typing LaTeX, or interacting with some math software). Only the most basic commands are the same for all modes.

Two things to remember

You can cancel any operation either by pressing C-g or three times <Esc>.

To exit Emacs, type C-x C-c.

Major/Minor Modes

A Major Mode provides Key-Bindings, Indentation, Display, Syntax etc. There can be only one Major Mode active at a given time. How would you mix two different syntax-highlighting schemes?

A Minor Mode provides additional bindings and some other behavior. For example spell-correction, automatic line-breaks etc.

Emacs Lisp functions and key bindings

You can access all functions of Emacs with: M-x (that’s <Alt>-x or <Meta>-x or <Esc> x or <Option>-x depending on your setup, preferences and terminology).

C-x means pressing Control and x at the same time, <Esc> x means pressing Escape and x after another.

There is no way to bind all functions to some key-combo and it might be impossible to remember the bindings. So, if you want to replace some string with some other string type “M-x repl<Tab><Tab>” and Emacs gives you possible completions, “replace-string” sounds good.

Emacs’ Architecture

Emacs has a small Lisp core written in C. But the “meat” of Emacs is the expansion-library written in Emacs Lisp. Essentially everything you do and write to extend and configure Emacs is written in Emacs Lisp.

The Bookworm has an introduction to the Emacs introductions.

Undo/Redo

To undo almost every action you did, you can use either undo as command or one of the following key-bindings: C-/, C-_ or C-x u

Undo → C-/

To redo, you have to switch the “undo-direction” with C-g, after that you can use the usual undo-key of your choice.

Switch undo-direction → C-g

Basic Movement

Character movement
- Up → C-p
- Down → C-n
- Left → C-f
- Right → C-b

Page movement
- Page Down → C-v
- Page Up → M-v

Line movement
- Start of Line → C-a
- End of Line → C-e

Sentence movement
- Start of Sentence → M-a
- End of Sentence → M-e

Paragraph movement
- Start of Paragraph → M-{
- End of Paragraph → M-}

Other
- Start of File/Buffer → M-<
- End of File/Buffer → M->

Search and Search & Replace

Searching/Replacing

Incremental Search forward → C-s
Incremental Search backward → C-r
Incremental Search forward with regular expressions → search-forward-regexp
Incremental Search backward with regular expressions → search-backward-regexp
Replace some string → replace-string
Replace using regular expressions → replace-regexp

Text Surgery

Deleting/Killing
- Delete char forward → C-d
- Delete char backward → <Backspace>
- Kill word forward → M-d
- Kill word backward → M-<Backspace
- Kill to end of line → C-k
- Kill to end of sentence → M-k

Transpose (switch elements)
- Transpose chars → C-t
- Transpose words → M-t
- Transpose lines → C-x C-t

Case-operations
- Capitalize word → M-c
- Uppercase word → M-u
- Lowercase word → M-l
Text-completion → M-/ (repeat to cycle through possible completions)

New-Lines
- New line → <Return> or C-m (insert a CR before the cursor)
- Open new Line → C-o (insert a CR after the cursor)
- New line and indent → C-j

I like to format my text so that no lines are longer than 78 characters. Either I use the minor mode auto-fill-mode or I use M-q to re-justify (fill) the current paragraph (necessary if you edit a paragraph after it was auto-filled).

Cut/Copy/Paste

Basic commands

Maybe you noticed that I changed the terminology of deleting a character to killing a word. The difference is, that “killed” text is put into the Yank-Ring. From this ring, you can get your text back (pasting, yank). The size of your yank-ring depends on your setup.

Yank → C-y, use M-y to cycle through the various text-parts.

If you’ve marked some text, use C-w to cut the text, or M-w to copy the text into the yank-ring:

Cut → C-w
Copy → M-w

Regions

To mark some text, use the mouse, or set a mark with C-<Space> and move the cursor to the end point of the region. The text between the mark and the current cursor-position (also called point) is the region.

Set mark → C-<Space>

You can jump between point and mark using C-x C-x, this switches the mark and the point, so that your region remains intact.

Switch point and mark → C-x C-x

If you’ve enabled the transient-mark-mode Emacs highlights the region when you create it. As soon as you move the cursor, the region is still there, but the highlight is gone (verify with C-x C-x for example).

Rectangles

The mark and point also specify a rectangle over your text (upper right and lower left edge).

Kill the rectangle → C-x r k
Yank the rectangle → C-x r y
Insert space into the rectangle → C-x r o
Replace text of the rectangle → C-x r t
Clear rectangle (replace with space) → C-x r c

There is an interesting minor mode for additional rectangle-functionality called CUA.

Frames, Windows and Buffers

Emacs has the concept of Frames, Windows and Buffers. A Frame is a Window in the common sense, a Window is a part of the frame. For example if you split the screen of your Emacs frame, you got two windows within one frame.

The last concept is the Buffer, it’s simply a real or virtual file (virtual == not saved to disk).

Buffer commands:

Load a file (buffer) → C-x C-f
Show loaded buffers → C-x C-b
Switch to another buffer → C-x b
Close a buffer → C-x k
Save a buffer → C-x C-s

Frame commands:

Open new frame with file → C-x 5 f
Close this frame → C-x 5 0
Switch to other frame → C-x 5 o

Window commands:

Split window vertically → C-x 2
Split window horizontally → C-x 3
Close all other windows → C-x 1
Close this window → C-x 0

The Minibuffer (the last line of your Emacs screen) is a buffer
itself, but has some specialities I wont go into now.

Getting Help from Emacs

You could fill some very big books with Emacs-specific information, but Emacs calls itself a “self-documenting” editor. This means that the documentation reflects exactly how your Emacs is configured. If you change some key-bindings, the documentation reflects this changes and displays the right key-combo in the help files.

Some hints on how to get going using the help-system:

Show information on the current mode(s) → C-h m
Describe the command bound to the given key → C-h k
Search for a function → C-h a
Describe all key-bindings of the current mode(s) → C-h b

Further Information

Steve Yegge has some intersting articles on Emacs:

IBM has some nice Emacs tutorials too.

The EmacsWiki is another excellent source for Emacs related howtos, downloads and – to some extend – discussions.

Wish

I’d wish that Vim and Emacs would merge. That I could take advantage of Emacs’ awesome tool-support and Vim’s awesome way of editing small junks of text. I know there is Viper-mode and Vimpulse, but both are not exactly what I want. Viper-mode is plain Vi, and Vimpulse is early in development.

Is there a way to integrate a Vim window into Emacs without emulating and reimplementing the features?

For the time being, I’ll have to switch between the environments. Vim as my default-editor and Emacs for special purpose editing.

Addendum to "Time Machine for every Unix out there"

2007-12-06T00:00:00+01:00

My article about using rsync to mimic the behavior of Apple’s Time Machine generated a lot of traffic, and more important, a lot of feedback.

In this article I’ll summarize and try to clarify a few things.

First of all: I didn’t invent any of the things I’ve written in the article! All I did was to write it up in a manner I could understand it. The rsync developer(s) included the feature(s) for a specific purpose which might be exactly what we do now. But this was obvious, I hope.

Time Machine and Performance

Ars Technica has a very deep review of the new features of Leopard, including Time Machine.

Interestingly they do very much the same as I did in my script – each Backup gets its own folder and there is a link to the latest.

Some readers complained that Time Machine looks only at the changed files, where my script has to look at each file on the disk. For me, the difference is negligible. Apple introduced hard-links to directories, which saves a lot of time and a bit of space. Unfortunately this feature isn’t availabel on plain Unix, but it’s a neat trick!

Restoring the Files

Since we make full backups each time, you have your whole and complete directory structure in that Backup-directories.

Restoring a file is merely a simple copy command.

Or, if you want to restore the whole filesystem, use:

rsync -avz --delete host:/Backup/scurrent/ /target/

But keep in mind that —delete removes any file which is not in the backup!

Removing old Backups

Another huge advantage of full backups is: you can delete any backup you don’t need anymore.

Of course each backup-directory takes space, if only for the hard-links and directory structure. Cleaning up the backups is a breeze: just delete the directories you don’t need anymore. For example you could delete all directories older than 2 months or keep one week full, then only the monday backup. There are plenty of possibilities.

Other Tools

Many people suggested many interesting tools for doing backups. My argument against using such tools is that rsync is everywhere!

You find rsync in the smallest Linux distributions (like DSL), on PDAs like the Zaurus, on Windows using Cygwin and also on Nokia’s Internet Tablets.

One backup strategy for all your devices!

Space Requirements for the Links

Nothing in life is free, but the space used by the hardlinks for the backup is marginal. It is also dependent on the filesystem you use, ReiserFS seems to do a good job in packing more links in one cluster. But hey, these things are Backups you throw away space anyways!

Colon

There is one particular problem using a colon in the script: some operating systems don’t like them.

Here an updated version of the script without colons:


 #!/bin/sh

 date=`date "+%Y-%m-%dT%H_%M_%S"`
 rsync -azPE --link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date \
   && ssh $HOST "rm PATHTOBACKUP/current \
   && ln -s back-$date PATHTOBACKUP/current"

Another slight change in the script is, that all commands are connected with &&. This means the next command is only done when the previous command returned no error.

Of course, if anything messes up the “current” link, the script just makes another full backup without the links to the previous version.

Extended Attributes on OS X

One user suggested the addition of the -E parameter on Mac OS X. If you use the parameter on non-OS X machines it’s more or less redundant (-a contains executability), but on OS X some additional information is backed up.

Just use it.

Windows

Some users reported success with this strategy on Windows. I’m happy that we now have a unified backup strategy for all major operating systems without paying a single cent.

You have to use Cygwin to install rsync (there are binaries without the whole Cygwin on the net, here for example).

Measuring the Space Requirements

To lookup how much space your particular backups need, simply issue du -shc back-*. This prints each backup directory with its real memory usage:

  me@myserver:Backups % du -shc back-*                                                                                                               
  406M  back-2007-11-12T12:00:00
  6.8M  back-2007-11-13T20:28:56
  189M  back-2007-11-13T23:42:54
  2.6G  back-2007-11-14T08:15:43
  1.8M  back-2007-11-15T20:07:07
  7.9M  back-2007-11-18T23:57:09
  11M back-2007-11-19T21:53:34
  2.6M  back-2007-11-21T18:57:40
  2.1M  back-2007-11-21T19:11:02
  11M back-2007-11-22T01:19:38
  1.9M  back-2007-11-22T11:05:17
  19M back-2007-11-22T18:37:56
  7.4M  back-2007-11-22T22:45:04
  8.5M  back-2007-11-22T22:51:29
  9.2M  back-2007-11-24T12:45:12
  4.2M  back-2007-11-24T17:30:38
  11M back-2007-11-25T00:41:19
  1.9M  back-2007-11-27T20:41:40
  317M  back-2007-12-03T10:19:07
  12M back-2007-12-05T11:51:19
  3.6G  total
  me@myserver:Backups %

As you can see, the smallest backup (obviously without any changed file) requires 1.8MB.

Time Machine for every Unix out there

2007-11-13T00:00:00+01:00

rsync is one of the tools that have gradually infiltrated my day to day tool-box (aside Vim and Zsh).

Using rsync it’s very easy to mimic Mac OS X new feature called Time Machine. In this article I’ll show how to do it, but there is still a nice GUI missing – for those who like it shiny.

What Time Machine does

Time Machine makes a snapshot of your files every hour. The files are usually stored on a external hard drive connected to your Mac via USB or Firewire. Earlier Leopard versions (ADC preview versions) had the ability to make the backups to a remote drive (I’ve heard).

So if you lose a file, or did a devastating change to one of your files, simply go back in time until you find your file or a version that’s not corrupted.

Incrementally backing up all files every hour so that you can access them in reversed chronological order isn’t that hard with standard Unix utilities like rsync. The only missing thing is a nice GUI for which Apple is known to be quite good at.

Making full backups in no time every hour

You can use this method to make a backup every hour or every ten minutes if you like. There are many many features you can tune or configure to your own taste – excluding files that are larger than 1GB for example.

So, here the command to make the backup:

rsync -aP --link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP

Lets go through the parameters step by step.

-a means Archive and includes a bunch of parameters to recurse directories, copy symlinks as symlinks, preserve permissions, preserve modification times, preserve group, preserve owner, and preserve device files. You usually want that option for all your backups.
-P allows rsync to continue interrupted transfers and show a progress status for each file. This isn’t really necessary but I like it.
--link-dest this is a neat way to make full backups of your computers without losing much space. rsync links unchanged files to the previous backup (using hard-links, see below if you don’t know hard-links) and only claims space for changed files. This only works if you have a backup at hand, otherwise you have to make at least one backup beforehand.
PATHTO/$PREVIOUSBACKUP is the path to the previous backup for linking. Note: if you delete this directory, no other backup is harmed because rsync uses hard-links and the operating system (or filesystem) takes care of releasing space if no link points to that region anymore.
$SOURCE is the directory you’d like to backup.
$CURRENTBACKUP is the directory to which you’d like to make the backup. This should be a non-existing directory.

As said earlier, rsync has many many features. To exclude files over a certain size for example, use the option --max-size (unfortunately this is not available on the rsync version shipped with Mac OS X Leopard). The man page or the documentation can give you plenty of ideas in this direction.

So much for the theory of the most important command for our purpose. Here a simple script that makes an incremental backup every time you call it:

#!/bin/sh

date=`date "+%Y-%m-%dT%H:%M:%S"`
rsync -aP --link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$date
rm -f $HOME/Backups/current
ln -s back-$date $HOME/Backups/current

The script creates a file called “back” appended by the current date and time, for example back-2007-11-13T22:03:32 which contains the full backup. Then there is a symbolic link called “current” which points to the most recent directory. This directory-link is used for the --link-dest parameter.

You should look at the --exclude parameter (or better, --exclude-from= parameter) and learn how to exclude certain files or directories from the backup (you shouldn’t backup your backup for example).

The script above only works on the local machine because making links on a remote machine needs some extra work. But not much:

#!/bin/sh

date=`date "+%Y-%m-%dT%H:%M:%S"`
rsync -azP --link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date
ssh $HOST "rm -f PATHTOBACKUP/current && ln -s back-$date PATHTOBACKUP/current"

The -f parameter for the rm command is used to supress error messages if the current directory is not present, which would in turn prevent the link to be created.

To get that working you either use a public/private key authentication scheme or something else to avoid typing in your password. Another possibility is, of course, to mount the remote file-system on the local computer using the above script.

On my setup the script takes about 6 seconds to synchronize 46968 files and 29GB – this takes 20MB for the file structure (with no actual files to transfer of course). But afterwards, I have a complete backup of my system in a new directory.

On a much bigger setup (1.2 million files and 50GB of data) the backup takes about 30 minutes and takes about 3GB of space (just for links!), so it isn’t exactly free, but very convenient.

The space needed for the backup is determined by the shape of your directory structure. On the larger setup I have lots of Maildirs and a very deep directory structure so it takes much more space than my home-directory backup above. 3GB is quite a lot, but 20MB doesn’t hurt.

Advanced `rsync` parameters

Additional to the parameters described above, I usually employ a combination of these parameters in my backup:

--delete and --delete-excluded this tells rsync to remove files from my backups either if they are gone on my local machine, or if I decided to exclude them from my backup.
--exclude-from=FILE the file specified here is a simple list of directories of files (one per line) which should not be backed up. My Trash folder oder some .cache folders are candidates for this file.
-P is used to give more information on how far the backup is, and how many files are to be backed up. Additional it could resume an interrupted transfer (which doesn’t apply here because we create a blank backup each time we call the script).
-x this one is important because it prohibits rsync to go beyond the local filesystem. For example if you backup you Linux-root partition, you should not include the /proc directory because rsync will get stuck in it. -x excludes all mounted filesystems from the backup which is probably what you want in most cases.

Hard-Links

Each file in a directory is a link to the actual data on your hard-disk. The file-system keeps track of how many links to a area point, and only if the last link is deleted, the whole area gets deleted (in contrast to soft-links, these are pointers to the file-name, not the contents).

Here an illustration of two backups with three files each. File1 and File2 are the same in both backups, only File3 changed between Backup1 and Backup2. So in Backup2, File3 (changed) has to point to a different area than File3 in Backup1.

BTW, there is a nice project for Linux out there which provides the same functionality as Time Machine including a nice GUI which is also based on rsync and the procedure presented here.

The End

Credit: The initial idea for this approach came from Mike Rubel – rsync snapshots.

Also interesting if you have to cope with Windows: Optimal remote backups with rsync over Samba.

There are quite a few approaches out there which more or less do the same, but rsync is available on virtually every Unix out there (even the DSL with its 50MB footprint includes rsync). So using other tools might be more convenient, but I’ll stick with the omnipresent rsync.

rsync offers the possibility to store only the differences to the previous backup (using --compare-dest which should point to a full-backup instead of --link-dest). It then doesn’t make links to the unchanged files, it just leaves them out. This way you get an incremental backup without the “directory-overhead” of the --link-dest approach. But you have to be extremely cautious which one of older backups you delete because the newer backups just don’t contain some of these files (think of full-backups as checkpoints)! Using the --link-dest you can delete all backups but the last and you still got all the files, so I’m happy to pay 20MB per backup for this safety.

Full script

Here my full script with additional features:

#!/bin/sh

date=`date "+%Y-%m-%dT%H_%M_%S"`
HOME=/home/user/

rsync -azP \
  --delete \
  --delete-excluded \
  --exclude-from=$HOME/.rsync/exclude \
  --link-dest=../current \
  $HOME user@backupserver:Backups/incomplete_back-$date \
  && ssh user@backupserver \
  "mv Backups/incomplete_back-$date Backups/back-$date \
  && rm -f Backups/current \
  && ln -s back-$date Backups/current"

XML as human interface and DSLs

2007-10-30T00:00:00+01:00

While reading “The Definitive ANTLR Reference” by Terence Parr I encountered this quote. I’m a bit sceptic about XML beyond its use of data exchange, so the quote has completely won me over for the book.

Don’t be afraid to build a human-readable configuration file (I implore everyone to please stop using XML as a human interface!) or to build domain-specific languages to make yourself more efficient. Designing new languages and building translators for existing languages, when appropriate, is the hallmark of a sophisticated developer. — Terence Parr, The Definitive ANTLR Reference, Page 23.

Linux peripheral devices configuration

2007-09-01T00:00:00+02:00

This weekend I tried to get my new keyboard (Microsoft Ergonomic Keyboard 4000) and my Logitech MX1000 Laser mouse to work properly. The Keyboard has many extra-keys I didn’t bother to count, and the mouse has 12 buttons which can be very useful at times.

Almost accidently I solved a bugging performance problem with the Firefox browser. It was incredibly slow when opening Google Spreadsheets, well the whole system was incredibly slow while loading the spreadsheet… .

Configuring the Microsoft Ergonomic Keyboard 4000 and the MX1000 Laser mouse

Most of the information I used to get my new keyboard working (except of 6 keys) came from the Gentoo-Wiki and the Ubuntu-Forum.

What you need is the

Make sure that you get the right version of the patch for your Kernel (in case that 2.6.22 is outdated).

Extract the Kernel, patch it (patch -p1 < hid.patch) and compile it. I assume you’ve got some experience compiling a Kernel (just follow the instructions at the Ubuntu-Forum).

Now for the configuration, I didn’t follow the Gentoo-Wiki guide and didn’t hack my xkb configuration because I couldn’t see any advantages… it seems that the “evdev” XkbModel does the right thing (if you don’t set the Model to evdev, the keys are really messed up).

I used the Gnome Keyboard-Shortcuts program to assign functionality to the keys. The keys I couldn’t get to work are the special keys 1, 2, 3, 4, 5, Fav, and Zoom-In (Zoom-In is recognized by X, but is already assigned). Since I use GMail for mail I assigned the Mail-button to firefox "http://www.gmail.com".

The mouse is equally simple, follow the instructions here.

Here the /etc/X11/xorg.conf config (relevant parts):

Section "InputDevice"
    Identifier     "Keyboard Ergo. 4000 1"
    Driver         "evdev"
    Option         "Protocol" "evdev"
    Option         "Device" "/dev/input/event1"
    Option         "CoreKeyboard"
    Option         "XkbRules" "xorg"
    Option         "XkbModel" "evdev"
    Option         "XkbLayout" "us"
EndSection

Section "InputDevice"
    Identifier     "Keyboard Ergo. 4000 2"
    Driver         "evdev"
    Option         "Protocol" "evdev"
    Option         "Device" "/dev/input/event2"
    Option         "XkbRules" "xorg"
    Option         "XkbModel" "evdev"
    Option         "XkbLayout" "us"
EndSection

Section "InputDevice"
    Identifier     "MX1000"
    Driver         "evdev"
    Option         "CorePointer"
    Option         "Device" "/dev/input/event0"
    Option         "Buttons" "12"
    Option         "Resolution" "800"
EndSection

[...]

Section "ServerLayout"
    Identifier     "Default Layout"
    Screen         "Default Screen" 0 0
    InputDevice    "Keyboard Ergo. 4000 1" "CoreKeyboard"
    InputDevice    "Keyboard Ergo. 4000 2" "SendCoreEvents"
    InputDevice    "MX1000"
EndSection

The udev rules to fix the keyboard to event1 and event2 (/etc/udev/rules.d/10-nek4k.rules, the UID should not differ from yours):

KERNEL=="event*",SYSFS{modalias}=="usb:v045Ep00DBd0173dc00dsc00dp00ic03isc01ip01", MODE="0644", NAME="input/event1"
KERNEL=="event*",SYSFS{modalias}=="usb:v045Ep00DBd0173dc00dsc00dp00ic03isc00ip00", MODE="0644", NAME="input/event2"

The udev rules for the mouse (/etc/udev/rules.d/11-mx1000.rules):

KERNEL=="event*",SYSFS{modalias}=="usb:v046DpC50Ed2500dc00dsc00dp00ic03isc01ip02", MODE="0644", NAME="input/event0"

The ~/.xbindkeysrc configuration I use simply activates the “Forward” “Backward” functionality (thumb buttons):

# Backward and Forward buttons
"xvkbd -text "\[Alt_L]\[Left]""
  m:0x10 + b:8
"xvkbd -text "\[Alt_L]\[Right]""
  m:0x10 + b:9

This maps the mouse button 8 to Alt-Left, and button 9 to Alt-Right. Remember to start xbindkeys at the start of you session (for example add it to .gnomerc or even better to .xprofile).

To rearrange the mouse buttions how I like them, I use a Xmodmap file (/etc/X11/Xmodmap):

!MX1000
pointer = 1 10 3 4 5 7 6 8 9 2 11 12 14 13

This rearranges the second button (middle button on the scroll-wheel) to the thumb button, and reverses the tilt-buttons of the scroll-wheel. This could also be in .xmodmaprc.

Use xev to find out which number any button has (works for the keyboard too).

Performance issues with Firefox

I noticed severe performance issues with Firefox each time I opened a Google Spreadsheet. The interesting thing was that Xorg took 100% of the CPU, not Firefox itself. I resolved the issue more or less accidently: I installed the binary NVidia driver which gave the whole system a real performance boost.

Bootup Performance

The next thing I attacked was the boot-time. First I disabled many services I didn’t use. Usually I create a folder called disabled within the runlevel directories (/etc/rcS.d and /etc/rc2.d) and move the services I don’t use (or don’t know I use).

Using a coll utility called BootChart (stores it’s charts in /var/log/bootchart/) I found out that a big chunk of my startup time is taken by getting my IP by DHCP. I changed my configuration to a static IP, so I could reduce the startup time by 10 seconds.

My chart currently shows a lot of time (and I/O access) is coming from modprobe, I wonder if compiling the Kernel with all necessary drivers included into the Kernel would give me even less startup time… Maybe I’ll try that sometime too :-).

So, now my Ubuntu setup seems quite nice, unfortunately I edited a lot of files in the process, which are easily lost so this is more or less a tutorial for my future-me to get the stuff working again.

Challenge Response Spam Filter

2007-08-11T00:00:00+02:00

The challenge-response spam-filter (RFC3834) troubles me lately. If you don’t know it, here is how it works if both parties have a challenge-response spam-filter:

I write an email to someone@domain
The address gets whitelisted on my machine
The receiver doesn’t get my message
I get a message from the receivers mail-server to which I should reply
I reply to the automatic message
The receivers mail-server whitelists my address and delivers my initial mail

So far so good. Three questions pop up:

What if the spammer uses my whitelisted email address to send his spam?
What if only one of the two has a challenge-response spam-filter?
What if spammers start to automatically reply to those messages?

Concerning the faked email address: there are several methods to tackle this problem. One is to use SPF (RFC4408) records for the domain (explicitly list allowed hosts), the other is to whitelist an email address together with the sending host. I’m not sure whether this fully solves the problem… .

The second question is a bit more fuzzy. If Alice has a challenge-response spam-filter and Bob doesn’t, then we have two scenarios: Alice sends Bob a mail, or Bob sends Alice a mail:

A → B:

Alice sends Bob the mail
Bob’s address gets whitelisted
All mails are delivered

Good, here it seems that we don’t have a problem.

B → A (as happened to me):

Bob sends Alice a mail
Alice’s mail-server sends Bob a verification message
Bob’s spam-filter filters the verification message because the company selling
the spam-filter inserts advertising into the verification message
Bob never notices that the mail wasn’t delivered
Alice has to check her spam-folder to see whether someone didn’t receive the
verification message (or ignored it)

I’m very sceptical whether the challenge-response spam-filter is worth the trouble, partly because my spam-filter (GMail) filtered the verification message of my buddy, and partly because it’s very easy to automate the verification process. The argument of my buddy (who employs the challenge-response spam-filter) was that spammers use fake email addresses to send their mail and usually don’t receive any replies (I think fake email addresses are used by malware, which sends it’s junk using my address book → remember: automatically whitelisted since I’m likely to have had a conversation with these receivers before).

Currently I’m quite happy with automatic spam classification, they filter about 90 percent of my spam messages. The remaining messages are easily deleted by hand. A spam filter doesn’t need to recognize 100 percent of the spam messages as long it filters enough to keep the “mark as spam” actions sparse. A spam-filter which generates false-positives is much more troubling because one has to skim through (usually) thousands of spam messages and make sure there is no false-positive. Especially if you are a company relying on communication with customers, I wouldn’t risk loosing a single one for the comfort of not having to delete a few messages a day.

Vim Introduction and Tutorial

2007-02-23T00:00:00+01:00

I often tried to learn the great Emacs editor/IDE/operating system. The last time I tried it, I spent some time getting comfortable with it until I wanted to customize my .emacs file.

That was the point when I entered vi .emacs. As soon as I realized what I’ve done, I knew that Vim has won me over a long time ago.

So, here I am – using Vim as my editor¹ of choice.

Another big motivational boost came after I discovered that my preferred shell (ZSH) has an awesome vi-mode including the command mode (yes, you can switch between command and insert mode!).

Vim has a great deal of features, and learning them takes some time. Of course there are many online-tutorials and tips’n’tricks pages, but the help-files are very good too! There are overview-pages, summary pages and some comments at the commands.

I took the approach to start using some tutorial and let the help-system guide (type :help <command> to get help for the command) me through the rest. I like to try the commands in a test-file, and take a short note of important commands.

Another reason I like to use Vim is because it’s much more healthy than Emacs (using the default-keymappings). Healthy? Many commands are easily typed with a single keystroke – the virtue of a modal editor, instead of long command-chains with lots of modifier keys. Even if you have a natural keyboard, pressing Ctrl, Alt etc is certainly not natural at all.

Just remember: Vim’s basics are really very simple, but in combination the simple commands become very powerful.

Modes

You have 3 modes:

Command mode: all keystrokes are interpreted as commands
Insert mode: most keystrokes are inserted as text (leaving out those with
modifier keys)
Visual mode: helps to visually select some text, may be seen as a submode of
the the command mode.

To switch from the insert or visual mode to the command mode, type <Esc>.

To switch from the command mode to the insert mode type one of

i …switch to insert mode before the current position
a …switch to insert mode after the current position (append)
I …jump to the first non-blank character in the current line and switch
to the insert mode
A …jump to the last character of the current line and switch to the
insert mode

To switch from the command mode to the visual mode type one of

v …switch to the visual mode (character oriented)
V …switch to the visual mode (line oriented)
Ctrl-v …switch to the block-visual mode (select rectangles of text)

All commands that take a range (for example subtitution, delete, copy or indentation) work with the visual mode too.

Movement

The simplest movement commands are

h …move left
l …move right
k …move up
j …move down

Obviously these commands work only in the command mode, of course you can also use the cursor keys (in all three modes).

There are a lot of movement commands available in Vim, I’ll only cover a few, but if you need something special very often take a look at the help, I’m sure you’ll find something usable.

Vim distinguishes between screen-lines (those shown on the monitor) and real lines (those ended with a new-line).

So here the most important commands

0 …first column of the line
^ …first non-blank character of the line
w …jump to next word
W …jump to next word, ignore punctuation
e …jump to word-end
E …jump to word-end, ignore punctuation
b …jump to word-beginning
B …jump to word-beginning, ignore punctuation
ge …jump to previous word-ending
gE …jump to previous word-ending, ignore punctuation
g_ …jump to last non-blank character of the line
$ …jump to the last character of the line

If you remember just a few of them, you’ll get very quickly from A to B! Another important fact is, that these commands give the range for other commands.

Editing

Inserting text is pretty simple in Vim, just type i and start typing. But Vim offers quite sophisticated text-editing commands.

d …delete the characters from the cursor position up the position given by the next command (for example d$ deletes all character from the current cursor position up to the last column of the line).
c …change the character from the cursor position up to the position indicated by the next command.
x …delete the character under the cursor.
X …delete the character before the cursor (Backspace).
y …copy the characters from the current cursor position up to the position indicated by the next command.
p …paste previous deleted or yanked (copied) text after the current cursor position.
P …paste previous deleted or yanked (copied) text before the current cursor position.
r …replace the current character with the newly typed one.
s …substitute the text from the current cursor position up to the position given by the next command with the newly typed one.
. …repeat the last insertion or editing command (x,d,p…).

Doubling d, c or y operates on the whole line, for example yy copies the whole line.

Please note, many commands are much more powerful than I describe them here. For example you can specify a buffer into some text is yanked. Typing "ayy copies the current line into register a, pasting the contents of register a is done by "ap. Vim remembers the last few yanks and deletions in automatic registers, to show the contents of the registers type :registers, you can also use them to paste some older text.

Visual Block

Using the visual block-mode it’s possible to insert characters on each line of the selection easily.

Suppose you have selected a rectangle (using Ctrl-v), you can insert text in front of it by typing I (switch to insert mode) and inserting your text. As soon as you leave the insert mode, the text will be added to all the other selected lines. Use A to enter text after the selection.

Another useful feature is to substitute the whole block with a new text. For that matter select a block and type s, Vim enters the insert mode and you can type. After you leave the insert mode, Vim inserts the text in the remaining lines.

If you’d like to append some text at the end of some lines, use Ctrl-v$ and select the lines. The difference between the former variant is, that the $ explicitly says “end of line” whereas a selection with Ctrl-v operates on the columns, ignoring the text.

Using Ctrl-v:

 This is a testNEWLY INSERTED
 This is a     NEWLY INSERTED
 This is       NEWLY INSERTED

Using Ctrl-v$:

 This is a testNEWLY INSERTED
 This is aNEWLY INSERTED
 This isNEWLY INSERTED

Text-objects

Vim commands operate on text-objects these are characters, words, characters delimited by parentheses, sentences and so on.

For me the most important one is the inner word: iw. To select the current word, just type viw (v for selection mode, and iw for the inner word), similar for deletion: diw.

The difference between inner-word/block and a-word/block etc is that the inner variant selects only the contents like the characters of the word (no blank afterwards) or the contents of the parentheses but not the parentheses. The a-variant selects the parentheses or a blank after a word too.

iw …inner word
aw …a word
iW …inner WORD
aW …a WORD
is …inner sentence
as …a sentence
ip …inner paragraph
ap …a paragraph
i( or i) …inner block
a( or a) …a block
i< or i> …inner block
a< or i> …a block
i{ or i} …inner block
a{ or a} …a block
i" …inner block
a" …a block
i` …inner block
a` …a block

Here a quick visualisation of the commands the color and the [ ] mark the selected text:

Command	Text Object
`iw`	This is a [test] sentence.
`aw`	This is a [test ]sentence.
`iW`	This is a […test…] sentence.
`aW`	This is a […test… ]sentence.
`is`	…sentence. [This is a sentence.] This…
`as`	…sentence. [This is a sentence. ]This…
`ip`	End of previous paragraph. [This is a paragraph. It has two sentences.] The next.
`ap`	End of previous paragraph. [This is a paragraph. It has two sentences. ]The next.
`i(` or `i)`	1 * ([2 + 3])
`a(` or `a)`	1 * [(2 + 3)]
`i<` or `i>`	The <[tag]>
`a<` or `i>`	The [<tag>]
`i{` or `i}`	some {[ code block ]}
`a{` or `a}`	some [{ code block }]
`i"`	The "[best]"
`a"`	The[ “best”]
i`	The `[best]`
a`	The[ `best`]

Try them out and remember the ones you need regularly (in my case iw and i() they are the real time-savers!

Undo and Redo

Don’t be afraid to try the various commands, you can undo almost anything using u in the command mode – even undo is undoable using Ctrl-r.

Vim 7.0 introduced undo-branches, but I didn’t have time to dig deeper.

External commands

In Vim it’s easy to include the output of external commands or to filter the whole line or just a part through an external filter.

To issue an external command type :!command, the output will be shown and that’s it.

To filter the text through an external command type :!sort %.

To insert the output of the external command in the current file type :r!command (for example :r!which ls).

Search for “filter” for more information :h filter.

Searching and Replacing

Searching in Vim is very easy. Type / in the command mode and insert the term you search, and Vim will search the file (in forward direction) for the term. Use ? for the backward direction. Using n or N you can repeat the search in the same or opposite direction.

If the option “incsearch” is set, Vim immediately jumps to the matching text when you enter something. If “hlsearch” is set, it highlights all matches. To remove the highlight type :nohl.

Replacing something isn’t very hard too, but you should have a good understanding of regular expressions.

To substitute a regular expression with some other text, type :%s/old/new/gc this command takes the whole file %, and substitutes s the word "old@ with “new” and looks for more than one occurrence within one line g and asks if it really should replace the shown one c.

To replace some text only in a selected area, select the area, and type :s/old/new/g. This should look like :'<,'>s/old/new/g in the command line. You’ll understand '< and '> after the “Marks” section.

Completion

While you are typing, it’s pretty common to use the same word over and over again. Using Ctrl-p Vim searches the currently typed text backwards for a word starting with the same characters as already typed. Ctrl-x Ctrl-l completes the whole line.

If you’re not sure how to type some word and you’ve enabled spell-checking (:set spell), you can type Ctrl-x Ctrl-k to do a dictionary lookup for the already typed characters. Vim’s completion system has much improved during the last major update (Vim 7.0).

Note the completion commands work only in the insert mode, they have other meanings in the command mode!

Marks

You can set marks within your documents to jump quickly between different positions of a document or even many documents.

Vim automatically sets various marks like

{0-9} are the last 10 positions of closed files (0 the last, 1 the last but one)
< and > are the left and right position of marked texts
( and ) are the start or end of the current sentence
{ and } are the start or end of the current paragraph
[ and ] are the first or last character of the last yanked or changed text
. position of the last change
' or ` position before the last jump
" position before the last exit of the file (local to a file)
^ position of the last insert-stop

To set a manual mark, use m{a-zA-Z} (m followed by either a,b..z or A,B,..Z), and to jump to one of the marks (manual or automatic) you can choose between ' and `

' …sets the cursor to the first non-blank character in the marked line
` …sets the cursor to the exact position where the mark was set

There is a little difference between lower-case and upper-case characters:

{a-z} are local to a file
{A-Z} are stored and available over sessions (associated with a file)

You can use L for your work-log and T for your time-table for example, and quickly update the information there.

For example you can jump to the last known position of a file before it was closed by typing `" (it’s easy to configure Vim to do it automatically at start).

To get a list of all marks Vim knows about type :marks. To delete marks use :delmarks (:delmarks a b c removes marks a and b and c, to delete all marks use :delmarks!).

Tabs, Buffers and Windows

Vim 7.0 has introduced tabs. We all know and love tabs, so it’s not much to say here. (Just a note: tabs in Vim are a bit different than in other programs, you could also think of them as many Vim instances in a tabbed terminal window. The difference is, that each tab-page can have it’s own layout. For example I could split my screen of the first tab, and view the same file in one window at the second tab… . So Vim-tabs are a bit more powerful.)

To open many files in tabs via the command line use vim -p *.txt.

To switch between tabs use the mouse (in gVim) or type gt.

To create a new empty tab type :tabnew, or open a file in a new tab :tabe xyz.

Buffers and Windows are a bit harder to understand. A window is what you see when you open Vim, when you open the help system (by typing :help buffers), you’ve got two windows. So they are no actual windows, but view-ports that Vim offers.

You can open a window and split the current one horizontally using :sp or vertically using :vsp. This way Vim shows you the same buffer in two different windows. You can open a new file too, using :sp file or :vsp file. To switch between windows use the mouse or type Ctrl-w {hjkl} in the command mode.

A buffer is a file (most of the time), but isn’t necessarily visible. So there are usually more buffers than windows. To show a different buffer in the current window, you can switch them using :b NUMBER, where the buffer number can be looked up using :buffers. In the standard configuration Vim forces you to save the currently shown buffer before it allows you to switch to another buffer, so don’t be frustrated by it’s complains. (Type :set hidden to enable unsaved buffers, but be careful).

Here my notes from the help-file:

:b N switch to buffer N
:buffers show buffer list. Explanation:
- % current window
- # alternate buffer (switch using :e# or :b#)
- a active (loaded and visible)
- h hidden (loaded but not visible)
- + modified
:bd unload the buffer and remove it from bufferlist (don’t close Vim,
even on the last buffer)
:bun unload the buffer but stay in bufferlist
:sp #N split current window and edit buffer N
:w write the current buffer to disk
:e file load a file from disk
:q closes current window (and Vim if it’s the last one)
:new new empty window
:on close all windows but the active one (Ctrl-W o)
Ctrl-W {h,j,k,l} move between windows

Allow modified buffers to be hidden when the option ‘hidden’ is set. Buffers are automatically saved if the option ‘hidden’ is not set, but ‘autowrite’ is set.

Macros

Vim allows to replay some commands using . (a dot). For more than one command use macros.

You can start macro-recording using q and one of {0-9a-zA-Z}, so for example qq records the macro to buffer “q”. Hit q when you are finished recording.

Now you can replay the macro at any time using @q.

This is the end

I hope I could get you started for mastering one of the most sophisticated editors out there. The last thing I can do now is to include my configuration file. Use :help ... to explore Vim’s powers further and write a tutorial for the next apprentice.

Place the vimrc into your home-directory (~/.vimrc), but make sure you don’t have one already.

¹ Vim is an editor, no IDE or operating system. Don’t try to make an IDE out of it, if you like IDEs use one! Of course it’s possible to automate many tasks, like compiling and jumping to errors reported by the compiler, for that matter look at Vim’s plugins.

Haskell - Laziness

2007-01-29T00:00:00+01:00

Haskell is a non-strict, or lazy, language. This means it evaluates expressions only when it needs their results.

Laziness is one of the things that make Haskell special – really special. Lazy evaluation allows easy handling of infinite data-structures.

Lets look at a few infinite examples. The most simple one is the list of all integers

[1..].

That’s it¹. Another example – containing all odd integers looks like

[1,3..].

Another way to define a list of all odd integers is

filter odd [1..]

This is laziness at work. How is it possible to work with infinite lists? How can we filter an infinite list? The answer is simple: we do only what we have to do! The expression filter odd [1..] doesn’t do anything but to tell the compiler “if we need a value from that list, this is how it is computed”.

print $ take 10 $ filter odd [1..] computes the first 10 numbers of the list, and prints it to stdout (try to omit “take 10” from the expression).

The $ operator saves us from parentheses. The above example could easily be written as print (take 10 (filter odd [1..])). If you’re not very sure about precedences stick with the parentheses, I tend to write my code using parentheses and rewrite it using $ afterwards.

Fibonacci

A popular example for infinite lists is the Fibonacci row, it is easily defined as

  
    fib :: [Integer]
    fib = 1 : 2 : zipWith (+) fib (tail fib)

(zipWith takes two lists, and generates a new list by combining each pair of elements into a new one).

The definition of the list is recursive, another nice property of laziness – you couldn’t do that in a strict language! Please don’t be confused by the illustration – the three lists are the same list!

Obviously we wouldn’t use that list for real work using Fibonacci numbers – it would require too much memory…

If we’d like to generate all Fibonacci number the above definition is the best we can do, but, if we’d like to get a specific Fibonacci number, we’d rather take another definition:

  
    fib :: Int -> Integer
    fib n = fibHlpr n 1 2
      where
        fibHlpr 0 a _ = a
        fibHlpr 1 _ b = b
        fibHlpr n a b = fibHlpr (n-1) b (a+b)

This is a iterative process. Let’s compare the profiling of the two methods to compute the 100000st Fibonacci number:

Using the option +RTS -hc for profiling generates a graph of the memory usage on the heap (see the documentation) this reveals that both versions take about 600k of heap:

Using lists:

Using the iterative process:

Another profiling option reveals statistics for the garbage-collector (parameter -S):

Using the list:

  
604,159,196 bytes allocated in the heap
123,180,084 bytes copied during GC (scavenged)
 81,007,200 bytes copied during GC (not scavenged)
  4,234,128 bytes maximum residency (164 sample(s))

       1134 collections in generation 0 (  0.95s)
        164 collections in generation 1 (  0.57s)

         11 Mb total memory in use

  %GC time      50.1%  (48.6% elapsed)

  Alloc rate    399,083,404 bytes per MUT second

  Productivity  49.8% of total user, 47.1% of total elapsed

Using the iterative process:

  
602,528,044 bytes allocated in the heap
114,819,512 bytes copied during GC (scavenged)
 80,816,248 bytes copied during GC (not scavenged)
  4,017,816 bytes maximum residency (167 sample(s))

       1131 collections in generation 0 (  0.95s)
        167 collections in generation 1 (  0.58s)

         11 Mb total memory in use

  %GC time      50.2%  (48.6% elapsed)

  Alloc rate    397,003,108 bytes per MUT second

  Productivity  49.8% of total user, 47.0% of total elapsed

So, the new conclusion is, that both versions are nearly equivalent – something I haven’t seen coming.

This is another example of premature optimization, I thought that the infinite list is a problem, but obviously profiling proved me wrong, really wrong!

For computing specific Fibonacci numbers we should really use the formula shown in SICP, but precision gets an issue using non-integrals… .

Downside

Laziness has its downsides too, sometimes it requires quite a lot of memory. But, as I’ve mentioned in the first article, Haskell is incredibly flexible and we can easily force strict evaluation – look at this somewhat outdated article for examples, or at this entry at the Haskell Wiki.

Other performance tweaks can be found here.

foldl, foldr, and foldl’

Laziness’ problems are best illustrated using foldr, foldl, and fold’.

The fold operations “fold” a list to a single value, for example to generate the sum of a list, we could write:

foldr (+) 0 [1,2,3,4]

Until now, nothing happened, but if we evaluate the statement, the following list is generated:

foldr (+) 0 [1,2,3,4]
1 + foldr (+) 0 [2,3,4]
1 + (2 + foldr (+) 0 [3,4])
1 + (2 + (3 + foldr (+) 0 [4]))
1 + (2 + (3 + (4 + foldr (+) 0 [])))
1 + (2 + (3 + (4 + 0)))
1 + (2 + (3 + (4 + 0)))

And after that, all the additions are carried out.

foldl works the other way around:

foldl (+) 0 [1,2,3,4]

generates

foldl (+) 0 [1,2,3,4]
foldl (+) (0+1) [2,3,4]
foldl (+) ((0+1)+2) [3,4]
foldl (+) (((0+1)+2)+3) [4]
foldl (+) ((((0+1)+2)+3)+4) []
(((0 + 1) + 2) + 3) + 4

Again, after we build the list of additions we evaluate these, so the space requirement for foldr and foldl is O(n). That’s where foldl’ comes in, it is a strict version of foldl (it makes no sense to implement a strict version of foldr, try to figure out why).

foldl' (+) 0 [1,2,3,4]

generates no list of operations to carry out, but carries them out immediately. So The steps required by foldl’ are:

foldl' (+) 0 [1,2,3,4]
foldl' (+) 1 [2,3,4]
foldl' (+) 3 [3,4]
foldl' (+) 6 [4]
foldl' (+) 10 []
10

Use foldl' if you are sure that you have to carry out all operations, for example for sums or products.

¹ The notation [a..b] generates the list of all integers between a and b, obviously we can leave out the upper bound to generate infinite lists.

Update: There are some comments on Reddit explaining and clarifying some points. I had to update the article because I simply screwed up with the first version. Thanks for all suggestions (now I wish my blog could handle comments too).

Haskell Basics

2007-01-29T00:00:00+01:00

The article I wrote yesterday was just the beginning, today we’ll look at the next step in becoming Haskell experts.

Yesterday we’ve learned how to split up our program and how to compile, or run it. Today we’ll look at some basic features of Haskell.

Types

Haskell is strongly typed, meaning that the compiler knows the type of every object we have in our program. Yet we won’t see types too often in Haskell because it uses type inference, which means that the compiler deduces the type of your objects. We can help it by providing type information, or include it to be sure that our objects have the specified type.

Functions are often very generic and require not a specific type, but require that the type is the same at certain places. Let’s look at the type of the map function (we query the type via GHCi, the interactive shell of GHC, just start it with ghci):

  
    Prelude> :t map 
    map :: (a -> b) -> [a] -> [b]

So, map has the type: “first parameter is a function which takes type ‘a’ and returns type ‘b’, the second parameter is a list of objects with type ‘a’ and the returned object is of type list of objects of type ‘b’”. So, map takes function and a list, and returns another list. ‘a’ and ‘b’ aren’t further specified, and map really doesn’t care what is is.

Look at this example:

  
    map (\x -> x 40) [(+1), (*2), (/4)]

here the objects in the list are (partially applied) functions, and the function to map on the list is a lambda (an anonymous function), which applies the given argument to 40.

If we run the above code (again in ghci), we get:

  
    Prelude> map (\x -> x 40) [(+1), (*2), (/4)]
    [41.0,80.0,10.0]

Please note, that a list is always composed of objects of the same type, so in this case it’s the most general one. We can’t mix integers and characters in a list, but we can use tuples: [(42,'a'), (45,'b')].

Lambdas

The previous example used a lambda function, this is nothing special, just a function without a name. We use it if we define some small function which isn’t used more than once.

Let’s define a lambda which takes two integers and returns the product of the two:

  
    \x y -> x*y

We could also give it a name:

  
    product = \x y -> x*y

but then we would write it like

  
    product x y = x*y

Partially applied functions

This is where I felt the urge to learn Haskell: we can partially apply a function! We’ve already seen it above (for example *2).

Let’s define a function to increment an integer, inc:

  
    inc1 a = a + 1

we could also write the above function as

  
    inc2 = (+1)

The difference is the type, inc1 returns an integer, whereas inc2 returns a function! Of course both act the same, but there is a difference.

Lets look at another example:

  
    map (+) [1,2,4,5]

this returns a list of partially applied functions! It returns [(+1), (+2), (+4), (+5)].

One last example, filter can be used to, well, filter some list. Lets filter out all number larger than 4:

  
    filter (<=4) [1,2,3,4,5,6]

Again – we used a partially applied function! We can do quite a lot of amazing things with such easy and powerful abstractions, and that’s the reason why Haskell is known for it’s glue capabilities.

Combining functions

The last concept I’d like to show today is the combination operator: (.).

Using this operator, we create a new function which takes the arguments of the first, and returns the return-value of the second.

(f.g)(x) is the same as f(g(x)). So, why is this important? Well, as I’ve said in my previous post, Haskell is full of syntactic sugar, so using the combination operator we save some keystrokes.

Here a simple example which extends the previous filter example to include only those numbers whose double is smaller than 5 (smaller or equal to 4).

  
    filter ((<=4).(*2)) [1,2,3,4,5,6]

Another way would be to use a simple lambda function:

  
    filter (\x -> (2*x) <= 4) [1,2,3,4,5,6]

The combination operator makes code immensely readable, no useless arguments and parentheses which clutter up our programs, just plain functionality.

Look at this imitation of the rev command in Unix (taken from here):

  
    main = interact (unlines . map reverse . lines)

could it get any simpler?

Haskell

2007-01-28T00:00:00+01:00

Danger! If you are happy with your current knowledge of programming languages, don’t read on – Haskell might be responsible for some serious defects in your motivation.

I was always some kind of programming language geek. I loved learning and playing around with all kinds of programming languages¹. Currently I’m trying to learn Haskell. Even though I learned functional programming at my university (two semesters using Haskell), I didn’t really learn how to interact with the outside world. This is where things start to get messy, no matter how beautiful the language is.

Haskell has many good ideas and takes programming to the next step. Things like implicit concurrency, explicit concurrency, purely functional, laziness, polymorphism, and various evaluation schemes provide many ways to get our job done (I wish I could show you how to do that, in the meantime look at the tutorials at the end of the article or the Haskell Wiki).

For me, Haskell was one of the harder languages to learn. The pure functional core is easy to understand, but the crossover from the outside world is not as easy.

The reason, I think, is that Haskell provides quite a lot of syntactic sugar³ and it is incredible flexible. Another reason is that Monads seem to be quite hard to understand, and Haskell is an ancient word meaning “programming in monads”.

Haskell tries to encapsulate the dirty world in a special Monad², a show-stopper for many Haskell beginners (I’ve included some good tutorials on Monads at the end of this article to get you started).

Reading a Haskell program is a lot easier than writing one. The error messages from the compilers are very different, and due to type inference often hard to understand. It might be that someone who didn’t spent his last 15 years programming in an imperative paradigm is better suited.

Anyway, after you understand Haskell, your programs are much smaller and more clean than in any other language. Haskell is very expressive, and you tend to think more about your problem than how to formulate it in you language.

Currently my only concern is memory efficiency – looking at the great programming languages shootout Haskell is quite good, but in my current understanding it’s not straight forward to write memory efficient programs in Haskell. Anyway, we have the options and the possibility to improve since we won’t produce much code! Of course there are cases, where Haskell is not the first choice.

Startup

Currently I can’t teach anyone anything about Haskell programming – I’m still learning, but I can help you to get started.

Organization

When we start a Haskell program, the main function of the Main module gets called, so our entry-point (lets call it main.hs) should look like

  
    module Main
    where

     main = do
       ...

Larger programs should be split into modules, just like Java’s package system, Haskell modules are hierarchic. For example if we’ve got a module Interlinked.Demo, we’ll put it into the directory Interlinked and call the file Demo.hs. (This is not required for the Main module!!)

Demo.hs would look like:

  
    module Interlinked.Demo
    where

      callMe = 10

to include the module in another module, we’ll have to import it:

  
    module Main
    where

     import Interlinked.Demo

     main = do
       putStrLn $ show callMe

this imports all functions from the Demo.hs into the current namespace. If we have a name-clash (two functions with the same name), we have several possibilities to remedy the problem:

hide the unused function import Interlinked.Demo hiding (callMe)
Qualify all calls Interlinked.Demo.callMe
Qualify only one package import qualified Interlinked.Demo as D, and call it D.callMe

Compiling and Running

To run the program I’d recommend using runhaskell for development and compiling it to native code for the distribution or profiling.

To run our program we simply type runhaskell main.hs (if our Main module is in main.hs), or we can compile the code into a single executable using ghc: ghc --make main.hs, the --make option compiles all dependencies (if needed) and links everything.

To sum it up:

runhaskell main.hs for development
ghc --make main.hs for distribution and profiling

We could also fire up a interactive shell and evaluate the methods selectively using ghci main.hs.

Profiling

Profiling is quite important, especially in Haskell because it’s easy to make some stupid mistakes. To profile our program we have to compile it using ghc -prof -auto-all --make main.hs, this adds profiling code to our program (our code isn’t recompiled if we didn’t change our files, so we may need to delete old object files by ourselves).

To run our program with profiling enabled, we start it with ./main +RTS -p. This generates a file (main.prof) containing the profiling results.

See the profiling documentation for more information on profiling.

Glasgow Haskell Compiler

The Glasgow Haskell Compiler is one of the better known Haskell compilers, Haskell itself is just a specification and there are many implementations (like Hugs and the York Haskell Compiler). All of them implement Haskell 98, but in the meantime many extensions have been proposed and are available in some of the compilers (most of them in GHC).

More on Haskell

Since I didn’t tell you how to program in Haskell, I should point out some interesting tutorials:

On Haskell, Intuition and Expressive Power, a love-letter for Haskell.
Why Haskell another love-letter for Haskell.
Haskell Hacking is a multi-part tutorial showing some amazing tricks. Nevertheless, it is aimed at the beginner: Part 1, Part 2, Part 3, and Part 4
It is quite popular to implement Lisp (and Scheme) in Haskell: Write Yourself a Scheme in 48 hours and Writing a Lisp Interpreter in Haskell.
Hal Daumé III has written a very good introduction into Haskell: Yet Another Haskell Tutorial maybe the Dive Into Python for Haskell.

The API of Haskell is available at Haskell Hierarchical Libraries and on Zvon. There is also a nice (very nice!) search engine for the API: Hoogle.

Monads

The first two of the following tutorials give an easy to understand introduction (and a good metaphor) to Monads, and the third one (All about Monads) is the most complete one with quite a lot of background:

Other sources include the Haskell homepage and of course Google. If you like books more than online tutorials, check out The Haskell School of Expression, but it’s not meant for the beginner. The Haskell book for beginners seems to be Haskell. The Craft of Functional Programming. Recently (January 2007) a new book Programming in Haskell came up, this one is also targeted at the beginner.

¹ To date I’ve at least written a (more or less) simple program in Assembler, Bash, C, C++, C#, Cobol, GWA Basic, Haskell, Java, JavaScript, Lisp, Mosel, Pascal, PL/I, Prolog, Python, Ruby, Smalltalk, XQuery, XSLT, and various presentation languages like LaTeX and HTML, but there are so much more to learn. I can’t remember most of them, but as far as I can remember is Smalltalk the one I had most fun with. Probably Assembler ranges very high in the fun-scale too – I used to write little TSR programs and install them on my schools computers… no harm done. Pascal and C were the languages I learned programming with, but GWA Basic was the very first language I used (on my Atari 2600 ST) :-).

² Monads are used for a lot more than interacting with the world!

³ It’s certainly no syntactic minimalist like Lisp.

Update: I just found an interesting article with quite the same experiences like I had. Anyway, it’s good to see that it’s not completely my fault.

MacFUSE

2007-01-24T00:00:00+01:00

You probably heard about MacFUSE (announcement). MacFUSE is an OS X implementation of the popular FUSE project (well known to Linux enthusiasts).

I just found a tech-demo video showing some really interesting features and possibilities of FUSE (of course it’s not limited to MacFUSE).

SSHFS

The most popular filesystem is the SSHFS, it allows you to mount directories of remote computers via SSH. No more Samba or network sharing hassles, just login with your usual SSH-account and you’re done. Surely this is a great tool for people constantly working remote, but for the others?

SpotlightFS

Today Google announced (did I mention that MacFUSE comes from Google engineers?) SpotlightFS, which is also great, but again not very interesting for most people. You get real filesystem access to stored searches, instead of a special filetype containing your search.

What really kicks ass

Somewhere within the MacFUSE wiki was a link to a tech-demo of MacFUSE. It shows some great ideas of what can (and should) be done using this technology.

For example the PicasawebFS was a real eye-opener! Easy upload, download and organization of pictures and albums via your filesystem and live on Picasaweb (or Flickr).

Imagine your blog as filesystem – edit your entries with your favorite editor in Textile, Markdown, or whatever suits you!

The other (equally interesting) tech-demo filesystems are:

DocFS – another cool demo: it allows you to organize the documents written on Google Docs & Spreadsheets and open them locally (live Word-export). Check at least this one out!
ProcFS – a live view of the processes running on your computer (like /proc on Linux).
RSSFS – a filesystem view of a RSS feed, each entry is a link directly to the article.

See Macworld 2007 MacFUSE Tech-Demo for more.

Weblog Hoster/Software Comparison

2007-01-23T00:00:00+01:00

Choosing the right tool for a job is very important, but often it is very tedious and bothersome to collect all the data you need.

Once I was searching for a replacement of my previous Blog-hoster and ended up with Hobix (see here). Hobix is certainly not for everyone – and most likely not for me, but currently I’m stuck with it.

Anyways, if you are searching for a good Blog-software, or Blog-hoster check out WeblogMatrix.com, a fresh Matrix-Wiki from the makers of WikiMatrix.com and ForumMatrix.com. Maybe you can also add your own favorite system?

Via it’s developer.

Blue on red

2006-11-20T00:00:00+01:00

Del.icio.us is an amazing service¹, clean, fast, and usable – except for one thing, it’s color scheme.

A page which is bookmarked more often gets a darker background than other pages. Of course it’s a nice feature, but the problem is that the background is red and the font-color is light blue.

Some examples:

Red and blue are on both ends of the color spectrum, which makes it hard or even impossible for humans to focus on both colors simultaneously.

Additionally, blue light has to be much stronger than red light because humans have way less blue receptors then red and green.

So, here a is little Greasemonkey script to improve the colors, the result looks like this: .

¹ Even when I’m not posting my bookmarks to it, I still like to browse the popular pages or my old network.

How to read a paper

2006-11-19T00:00:00+01:00

Since I’m writing my master’s thesis, I have to read some scientific papers. In my previous years it wasn’t necessary to read those, so I’m not used to it. That’s a pity because they contain bleeding edge knowledge compressed in a self-contained form of a few pages.

I started reading them like technical books which didn’t work because the contents of books are usually well edited (by people who are concerned about the language, not the content) and comparatively easy to read. Reading a paper is quite different, they are very compact and have nearly no redundancy which makes them hard to read.

I’ve collected a few tips on how to read papers:

Read only one paper at a time. Papers differ in their notation and sometimes in their definitions (even papers from the same authors). Switching between papers (and notations as well as definitions) confuses and doesn’t help you to get the knowledge you need.
Don’t start in the middle of the paper. As said before, papers are highly compressed and have low redundancy, if you start in the middle you’ll miss the explanations about their notation and the definitions. Scientists usually make their papers self-contained, so they include everything you’ll need to know to understand the paper, use the information!
Read the whole paper before you start using the knowledge unless you’re absolutely sure that you don’t need the remaining chapters. But read the conclusion at the end of the paper, usually it contains an informal summary and open issues which may be of interest.
It’s a good to do a simple real-world example while reading a paper. It’s a method to slow you down and force you to understand every sentence of the paper.
Sometimes it takes hours only for a few pages of a paper, that’s not unusual, even for those who worked for years in the field.
Don’t get frustrated if you don’t understand some paragraph, read the rest of the chapter, maybe there are further explanations on the topic, or just get some sleep and let your sub-consciousness work out the details.

Update: Thomas Krennwallner pointed me to a guide called Efficient Reading of Papers in Science and Technology, it was written in 1989, but updated in 2000. Thanks, dude!

Brand new MacBook Pro

2006-11-08T00:00:00+01:00

Today I’ve got my brand new MacBook Pro, so I thought I share some observations for those who are tempted to buy one too.

(It seems that I’m not the only one who got a new MBP today. Have fun, Mat.)

Size

First of all I’m impressed by the size. It’s a 15.4’’ model with 1440×900 pixels. My previous 12’’ iBook had a resolution of 1024×768 pixels (that’s an improvement of 65 percent).

Performance

Before selling my PowerMac and iBook I made some performance tests. I used GeekBench which is available for all major platforms and comes in 32 and 64 bit flavors. Here the scores:

Mobile computer

iBook (G4 1.33GHz)
- 32 Bit: 60.9

New MacBook Pro (Core 2 Duo 2.16GHz)
- 64 Bit: 229.9
- 32 Bit: 214.1
- 32 Bit (Rosetta): 152.3

Desktop computer

PowerMac (G5 2×2GHz)
- 64 Bit: 158
- 32 Bit: 147.4
New PC (Core 2 Duo E6300 1.86GHz)
- 32 Bit (Windows): 217
- 32 Bit (Linux): 244.3

So, the upgrade from my iBook to my MacBook Pro increased the performance by 215 percent (32 Bit), by 150 percent (32 Bit – Rosetta), or by 277 percent (32 Bit → 64 Bit). The MacBook even beats the PowerMac! (Remember, this is only one benchmark, it doesn’t consider all components of a computer!)

The most interesting observation was the performance comparison of Windows and Linux – the same machine performs 12.5 percent better under Linux, but that’s another story.

Pictures

End notes

I hope I have better luck with my new mobile computer than one of my friends with his MacBook. His suffered from random shutdown problems (twice) and recently he lost his hard-drive (somehow…). Understandable that he’ll think twice before buying another Mac.

I switched from a Mac desktop to a PC desktop because upgrading a Mac is very expensive, but a mobile Mac is still my first choice. I prefer it not only because of it’s sleep mode, but also for it’s Unix underpinning and great user-experience.

Previous MBP problems

The MBP usually runs at about 50ºC (120ºF), but this goes up to 75ºC (170ºF) when compiling and I didn’t notice any whining sounds.

Using Google's Blogsearch

2006-10-20T00:00:00+02:00

I’m using hobix to power my blog. Unfortunately it requires some work to get the dynamic components up and running (search and comments).

So I decided to use Google’s Blogsearch which is pretty powerful.

To include a search box on your own site paste the following code into your HTML template and set the bl_url field to your own domain:


 <form method="get" action="http://blogsearch.google.com/blogsearch">
   <input name="as_q" size="15" maxlength="255" value="" type="text">
   <input name="sa" value="Google Search" type="submit">
   <input name="bl_url" value="blog.interlinked.org" type="hidden">
   <input name="ie" value="ISO-8859-15" type="hidden">
   <input name="oe" value="ISO-8859-15" type="hidden">
 </form>

The Blogsearch supports all standard operators from Google’s search, but also some blog-specific ones:

inblogtitle: search only within the title of the blog
inposttitle: search only within the title of the posting
inpostauthor: search only within the author field
blogurl: return only results starting with the given URL (that’s exactly what we use for our search-box)

The biggest advantage over Google’s Web-search is, that the Blogsearch almost instantly crawls your blog after you’ve pinged them, or a service monitored by Google.

I use two pinging-services:

http://pingomatic.com/
http://pingoat.com/index.php

Only a few seconds after the ping, my feed gets fetched from Google and my article is available via the Blogsearch.

GMail User Interface

2006-10-18T00:00:00+02:00

I really like Google’s user interfaces. Most of them are really good and their search interface changed the whole industry.

There’s currently only one thing I don’t get – the Delete button in GMail.

GMail introduced a Delete button some time in january.

Somehow they messed up their, at this point consistent, user interface. Compare the toolbars of the Inbox, Spam Folder, and Trash Folder:

The Inbox has the Delete button on the right, while the other views have the Delete button on the left. There’s a small difference between Delete at the Inbox and the Delete within Trash and Spam (Delete Forever), but is this a reason to re-align?

Why is it so annoying? I tend to remember the position of the buttons on the various interfaces I work with. Within an operating system all buttons are aligned according to a specific guideline. In different Web-applications it’s usually pretty messed up, but within a single Web-application it should be consistent.

This article was inspired by my clicking on Not Spam button the instead of Delete Forever button the tenth time.

I think it’s a good guideline to put the “most severe” action on the rightmost button, so here is a little Greasemonkey script to align the buttons for you (and me): gmail_delete_button.user.js.

There is a little quirk with this Greasemonkey script: After you clicked on one of these buttons GMail realigns the buttons. I’d be glad if someone had a suggestion…

Why Darcs

2006-09-10T00:00:00+02:00

One of my colleagues keeps asking my why I prefer Darcs over Subversion.

The question isn’t easy to answer. I think it’s because of personal preferences and not because of objective advantages. Anyway I’ll try to explain it in this article. There is also more objective and more extensive comparison of many version control systems available.

Please note, I’m no expert using Subversion, so please mail me if I’ve got something wrong.

Changes instead of versions

Darcs builds on top of patches. A patch is a set of changes to the repository, not only a single file.

Subversion assigns version numbers to files (or better repositories). [Subversion’s version numbering is much more natural than the one from CVS. It assigns the whole repository a version instead of each file a different one (like CVS). That’s a huge step forward, thanks Anders Engström for pointing that out!]

That’s an important difference, if not the most important one. When I work on a project I’m not interested in versions of a file, I’m interested in the changes my colleague made. Most of the time it doesn’t matter to me which files were changed, I just want the feature incorporated to my local copy.

Subversion doesn’t allow the so called cherry-picking, I can’t choose changes selectively. Darcs allows to check out changes selectively, I can check out feature X2 but not X1, even if X1 was made before X2.

Since Darcs doesn’t work on files, I can’t check out files directly. For example if I’d like to have a subdirectory of the repository, using Darcs I’d have to check the whole thing out (all patches).

A little example

Imagine a system administrator using Darcs for his/her configuration files. Think of a central server which holds all users, and some specialized servers holding only a subset of them. Think of a new user for one of the specialized servers (copied from the central server):

Using Darcs he or she can check out a patch called “new user ‘xyz’” and all files are updated.
Using Subversion he or she has to look through all changed files and copy the according lines to the according files. Or create the user by hand. Well, to sum it up, I don’t know how to do it in Subversion or CVS.

Darcs:

Subversion:

Of course this is a specialized example, but it illustrates the advantage of patches very well. Darcs is not black magic or something, it works usually, but has it’s limits too.

Subversion and CVS behave like Network File Systems because you either take the latest version or leave it. If you’d like to have a bug-fix, but not the new feature you’ll have to merge the bug-fix into your code. As soon as you’ve changed a prior version of a file you’ll have to merge it with the new version in the repository… .

Subversion is very well suited to keep two copies in sync, but even small differences are hard to get right.

Offline usage

If you’d like to use Subversion to keep track of fine grained changes you’ll have to check in very often¹ (for each little change). To do this you’ll need access to the server holding your repository. But what do you do if you’re on the train, in a hotel without internet, or in a small village outside the connected world?

Darcs allows me to work offline but still record my changes very very fine grained (my local repository is a full fledged repository). As soon as I have connectivity I can upload my changes to the server and all my peers can take advantage of the fruits of my work (and start cherry picking).

Interactive nature of Darcs

Most of Darcs’ commands have interactive components. This allows you to select changes for a patch even if you’ve done more. Compare that to Subversion or CVS, using those you can only commit the whole file.

The Darcs Wiki has a nice comparison of interactive Darcs commands to Subversion.

Test before commiting

Subversion and Darcs allow the testing of the code before any change to the repository happens. The difference is that Darcs tests the program on the local machine, and Subversion on the server. Darcs creates a clean copy of the code and runs the defined test suite on that code to make sure that the committed changes are working.

Ad Subversion: What do you do if you develop a program for the Mac on a Mac and your Subversion repository runs on Linux?

Conclusion

In the end it’s about you and your people. If you like Subversion and it’s way of doing things it’s fine. If you prefer CVS because it’s the most mature one, good!

If you need graphical tools to do your work, Darcs my be the worst option for you, it’s still command line oriented. At least I don’t know of a noteworthy GUI.

I like Darcs because it’s simple, and doesn’t clutter my projects with .svn oder .CVS directories. Most of the other features aren’t so hot if you’re the only one to use Darcs… . Anyway I’ve got more than a Network File System with version support :-).

Little hint: if you install Ian Macdonald’s bash completion you can expand commands and options with <Tab>. This, of course, isn’t tied to Darcs alone.

¹ I wonder why I’d like to check in small grained changes using Subversion because I don’t have the possibility to choose which one I’d like to build upon?

Rethink your Web-Interfaces

2006-09-09T00:00:00+02:00

Forbes recently published a list of Web-Applications for small businesses. Google won in several categories (Calendar, E-Mail, Information Managers, Spreadsheets), my question is why?

Some possible explanations come to mind, like

Google paid for the report,
the reviewer would like to work at Google,
the company likes the traffic Google sends them…

Anyway, I think more important is how Google designed their applications. Not that they have more features or more buttons to press, quite the opposite, they usually feature less possibilities!

Google engineers think about the interfaces for the Web and don’t just stupidly copy the interfaces from (bloated) Desktop applications.

Let’s compare Open Office’s (Neo Office’s) Spreadsheets, Zoho’s Spreadsheets and Google’s Spreadsheets toolbars:

OpenOffice:

Zoho:

Google:

As you can see, Zoho clearly modeled their application like a typical Desktop application. Google, on the other hand, threw out a lot of complexity and cleared the interface. They structured the menu to fit the current task (formatting, sorting or building formulas), which is natural in the Web.

Of course you could argue that one doesn’t need to re-learn how to use the spreadsheets in a Web-browser, but it’d require to re-learn how to use the browser!

Google removed the need of a Right-Click for the important features (only copy, paste etc. are included). It’s not very common to override the right-click within a Web browser, so customers might not expect it.

They’ve done the same for most of their applications, GMail offers few simple but very powerful metaphors (labels, search), the same with their calendar.

So, the lesson I’ve learned is to think about the different paradigms (this also goes under know your audience). A Web-GUI is different from a Desktop-GUI and vice-versa. Don’t just copy designs from one world to the other – it’s a lot of work and only a few people appreciate it in the end.

ICFP Programming Contest

2006-09-08T00:00:00+02:00

The ICFP Programming Contest is a little competition just before the International Conference on Functional Programming. Anyone can subscribe and attend. This year 364 teams took the challenge with their programming language of choice.

The contest is not tied to functional programming languages, you can choose whatever language you want. That’s the reason for this article. Which language has proven itself in this contest?

I’ll include the first, second and the Judge’s price here.

Year	First	Second	Judge
1998	Cilk	OCaml	J	http://www.ai.mit.edu/extra/icfp-contest/
1999	OCaml	Haskell	Haskell	http://www.cs.virginia.edu/~jks6b/icfp/
2000	OCaml	OCaml	GML	http://www.cs.cornell.edu/icfp/
2001	Haskell	Dylan	Erlang	http://cristal.inria.fr/ICFP2001/prog-contest/
2002	OCaml	C	Python	http://icfpcontest.cse.ogi.edu/
2003	C++	C++	Dylan	http://www.dtek.chalmers.se/groups/icfpcontest/
2004	Haskell	Haskell/C++	OCaml	http://www.cis.upenn.edu/~plclub/contest/
2005	Haskell	Dylan	Dylan	http://icfpc.plt-scheme.org/index.html

The results of the 2006 ICFP contest are not included since they weren’t published at the time of this writing.

To sum it up:

Language	First	Second	Judge	Sum
Haskell	3	2	1	6
OCaml	3	2	1	6
Dylan	0	2	2	4
C++++	1	2	0	3
Cilk	1	0	0	1
C	0	1	0	1
J	0	0	1	1
GML	0	0	1	1
Erlang	0	0	1	1
Python	0	0	1	1

Interesting, isn’t it? There are few well known programming languages in this table. Haskell and OCaml both won six times! No Java, no Ruby, no Perl program made it to the top, what’s up with those languages (Again, the contest is not limited to functional programming languages…)?

What are the possible interpretations of such a result?

Functional programming is pretty good (at least in Haskell and OCaml)
The tasks are “functional-friendly” (at least we know that there are differences and you should choose your tools to suit your task)
Only teams using obscure functional programming languages submit to ICFP (sorry, not true)
ICFP isn’t able to run programs written in other languages
Libraries don’t help you that much
Only smart people learn obscure programming languages

Do you think your language is better than the other ones? Prove it, and sign up for the ICFP 2007 :-).

The Programmer's Bill of Rights

2006-08-27T00:00:00+02:00

Jeff Atwood over at Coding Horror wrote The Programmer’s Bill of Rights I like the idea, but some points aren’t how I’d like to have them.

If we think of what’s really essential for doing our job, the list would be very short. A table, a chair, and a computer, that’s it. Unfortunately this won’t work for very long. We’d get back-aches from the cheap chair, headaches from the cheap monitor, and we’d be pretty frustrated working with the loud, slow and unfitting computer.

Please note that this is just a quick overview of things I think are important! So my Programmer’s Bill of Rights, or Wishlist looks like that:

Every programmer shall have a decent chair.
Jeff is absolutely right, we’re sitting most of the time. A bad back is a serious defect and hard or impossible to repair. The Aeron chair is a very fine choice and for this kind of chair well priced.
The table is usually not very interesting, it shouldn’t be too high or too low and leave some space for the legs, duh.
Every programmer shall have a monitor setting best suited for his or her task.
Jeff was on the right path here, but for me two monitors would not help very much. I’m currently using Eclipse, and in case you don’t know it, the window for the actual code is about a quarter of the whole available monitor space (subtract 1/4 on top, bottom, left, and right). So for me (and any other Eclipse user) a big monitor is much better than two monitors. On the other hand if I work with an text editor I’ll probably need the help system open all the time, in this case a second monitor is priceless.
Every programmer shall choose his or her own toolset.
Again I agree with Jeff. Whenever something is ours, we’ll handle it differently. If we chose your tools we’ll do anything to show our peers that we’ve chosen the right thing. We’ll find new and creative ways to use our tools and master them. No two persons are equal, so everyone has his or her own preferences. We need to be able to choose a keyboard⁶, a mouse⁷,and most important software we like to work with. Hell, the biggest flame-wars on the Internet are about Vim or Emacs, Windows or Unix. Let us choose!
It seems pretty weird to let the programmer choose his keyboard and mouse, but I think there are some very good arguments about it. First, a keyboard and a mouse are a host for many bacteria, I’d rather dwell in my own bacteria than anyone else’s. Second we get used to our equipment, I’d like to have the same equipment at work as I have at home.
Of course not everything is a variable¹, and we’ll have to have standard equipment. But the tools we use to get our things done should be chosen by us!
Every team or programmer shall have access to the information it, he or she needs to get the work done.
Most of the time we’ll hear something like “you’ve got Internet access, use it”, but that’s a poor argument. The quality of the information on the Internet is very variable, and finding something very specific is hard. We need to get access to online books (ACM offers access to several hundred books, O’Reilly’s Safari Bookshelf is also searchable).
Make customer conversations available to the programmer. Let your people get the raw material! Every post-processing loses some information.
Every team or programmer shall have an environment best suited for it, him or her.
The environment we’re working is is crucial to our work. It’s important to have a quiet place to work². Every team shall have a room for it’s own. Every team shall be allowed to change the room for their own needs. Every programmer shall have space for his stuff (paperwork, pens, utilities etc.), or the room will quickly look like a chimp-cage.
Every team or programmer shall have the freedom to choose the best process and methodology for it, him or her.
There are many, many processes and methodologies out there promising “to be the best”. Not every process suits every project or team, we need to choose which one is best for us. It’s important to get that straight: We’re building software for a customer, either internal or external, how this is done should be our responsibility because we’re responsible for the product!
Every team or programmer shall have the right to be involved in all stages of the software development.
To get a good understanding of the project, the requirements, and the constraints we need information. So we should have the right to be at meetings with customers, managers, testers, designers etc. It’s not necessary to dive in, talk with them or something, just listen, get the information needed and ask questions if something is unclear³!
Every team or programmer shall have time to reflect.
History favors those who take the time to learn from it. If a project went well, sit down and discuss why and what went well. If a project went down the line, sit down and discuss what could have caused it. The important thing is to reflect what went well and maybe more important what didn’t. Try to extract the essence of the success or failure. Learn from the past!
Every team or programmer shall have time to learn something new.
Our business is a very fast one. Whenever you stop to keep up to date we’ll quickly lose tracks. We have to constantly learn new things, stay on top and try to educate ourselves as well as our peers. It’s like the Race with the Red Queen, we have to run as fast as we can to stay where we are⁴.
Every team of programmer shall have the freedom to criticize.
We should take criticism seriously, we have to listen and react. It’s of no use to try to change the people, usually people don’t complain out of nothing. We have to take it as an opportunity. We should educate them if they got something wrong, or change our point of view and think about the criticism, maybe there is something about it⁵.

Jeff Atwood isn’t the only one to write a Bill of Rights, the C2 Wiki has one too as well as a Bill of Responsibilites.

¹ One man’s constant is another man’s variable. — Alan Perlis

² I once worked in a room with seven people and at least five different projects, imagine the noise level in there.

³ We often discussed issues which were dictated from above, but we didn’t know it, so we lost hours or days discussing alternatives which simply were out of question.

⁴ I know of a company which is very into communicating new ideas. They preserve time to educate their people, but they forgot to consider the learning time in the first place.

⁵ I once was asked to stop criticizing some product we were building. I was so perplex I couldn’t say anything about it. Isn’t criticism a way to improve things, or at least a pointer what is wrong with a particular product?

⁶ I used to like the Microsoft Natural Keyboard Pro while using a PC.

⁷ I once was asked to share a mouse with a colleague sitting in another room :-).

Darcs

2006-08-09T00:00:00+02:00

Darcs is a wonderful new revision control system. It’s quite different from CVS or Subversion. First it has no versions, second it’s distributed.

A version control system with no version? Silly me! Of course Darcs has versions, but you’re responsible to create them yourself.

Distributed means, that there is no dedicated central repository. Each repository exists on it’s own, if you’d like to have a central repository where all developers check in and out is up to you.

Darcs is being developed by David Roundy, a physics professor at Berkeley. It’s build upon a “theory of patches”. If this sounds very academic I should mention that it’s programmed in Haskell. But wait! It’s really good, and you don’t have to have several diplomas to understand it. Quite the opposite, it’s really very simple and elegant.

I’m writing this because I couldn’t find a good introduction into Darcs. Funny enough that was my fault, because even if Darcs is an advanced tool it’s surprisingly simple. Anyways I’ll present a quick introduction here, but for a complete reference see it’s manual.

At first I had a list of commands and what they do, but honestly it would not help anyone who knows to read a manual. So I’ll take another approach. I’ll try to cover the common uses of a versioning system and explain how to accomplish that task with Darcs.

What I like about it

Offline usage

You can work on your own repository on your own computer and as soon as you get connectivity, you upload the files to another repository.

Compare that to CVS: You make changes, and check them in with a single command – no distinction between different changes.

Cherry picking

Since Darcs operates on patches you always can choose which ones you check in, or check out. If you and your developers are disciplined enough, you can easily keep multiple branches in sync.

Documentation

Darcs is well documented, and it’s included help system is also very good.

Simple yet powerful

The patch-system of Darcs is very intuitive (really!), and still very powerful.

What I don’t like about it

The community behind Darcs is very able (Darcs is written in Haskell), but limited in size.

Darcs’ tool support (IDE plugins, GUIs, additional Software) is not very extensive. But there are some nice projects, like an Eclipse Plugin, a Web Frontend and even a Trac module.

Installing it on Mac OS X (using DarwinPorts) is easy, but takes forever since it depends on the Glasgow Haskell Compiler which takes a lot of time to compile itself. For Windows and Linux are binaries available. Gabriele Renzi pointed out that there are binaries for Mac OS X available, thanks!

You have to handle the user accounts on your own. Darcs doesn’t have to do anything with permissions. You either handle it via file-system permissions, SSH accounts or something like that.

Creating/Initializing a repository

If you’d like to create a new repository, for a new project, or for an existing — not version controlled project, darcs initialize is your friend.


mkdir ProjectDirectory
cd ProjectDirectory
darcs initialize

This command creates a directory _darcs which contains metadata and your patches, simply said it’s your repository. Please be sure to understand the difference between project files and repository. The repository lies under the _darcs directory and keeps track of the changes etc., the project files are the files you are working on.

Adding files to version control

If you’ve created some files, or would like to import your existing project files, you run darcs add.


darcs add * -r

This command adds all files in the project directory (-r means recursively include subtrees) to your repository. Please note, that your repository is not changed by this command. It only tells Darcs that these files are under version control.

Check what you’ve done since your last “commit”

Often you’d like to see what you’ve done since your last commit, and which files have changed, are new or have been removed.


darcs whatsnew -l

The -l parameter tells Darcs to show a summary, including new files. You’ll see a summary like


M ./modified.txt +1
R ./removed.txt
A ./underversioncontrol.txt
a ./newfile.txt

M…	Modified file (showing the changed lines).
R…	Removed, file is under version control, but has been removed.
A…	Added to the repository.
a…	New file, not under version control.

If you ommit the -l parameter, you’re shown the changes within the files in Darcs’ own patch format. You can always get an original diff file out of Darcs if you need one.

Commiting

So far we did not change our repository. To get your current changes into the repository (AKA commit), you simply “record” the changes with darcs record.

Now I have to explain patches. All Darcs operations work on patches. So if you commit something, you commit one or more patches, if you check something out, you check out patches etc.

So, if you call darcs record Darcs will ask you for each change if it should record it. One collection of changes together form a patch.

After you’re done with selecting your changes, you’re asked to give a short patch-name, and a longer patch-description.

If you’re sure that all changes need to be in the repository, you can use -a for all changes to skip the change selection. I usually use darcs record -am 'my meaningful patch name'. This will immediately record (commit) all my changes as a single patch.

Pushing changes to other repositories

As said above, Darcs has no central repository. You can push changes to any repository you’ve got write access to. It’s similar to a check-in in CVS.

So, to push one or more patches to another repository you use darcs push destination. The destination could be locally or via SSH.

Darcs asks for each patch which is not already in the destination repository, so you won’t transfer more data than necessary.


darcs push versioning.company.com:/path/to/your/project

The above command pushes the selected patches via SSH to a remote repository, and applies them immediately.

Now it’s easy to manage multiple branches the same time, just push a single patch to one or more repositories. You won’t have to merge anything, because you keep bugfixes on both, your HEAD repository, and your STABLE repository in sync. But because of Darcs’ innovative patch system even merging is simply, just pick the patches you need, that’s it. (Take that, CVS and SVN!).

Pulling changes from other repositories

Similar to pushing changes to a repository you can also pull them to your local repository. Just think of pulling (checking out) patches from your central repository, or from another branch of your project.


darcs pull versioning.company.com:/path/to/your/project

The above command pulls the selected patches via SSH from a remote repository, and applies them immediately.

With such features it’s very easy to manage multiple branches and versions of your software, or configuration files. Darcs’ manual calls this feature cherry picking.

Checking out a whole repository

To download the whole repository from you could create a new repository and download all patches from the source repository (darcs init followed by darcs pull ....). But there is a much more convenient and faster way:


darcs get versioning.company.com:/path/to/your/project

Notice, Darcs will create the project directory for you, if a directory with the same name already exists, the new one will get a suffix (_0, _1, _2…).

Creating a branch

So I’ve talked about branches earlier, how to we get one? This too is very easy, because you can think of a branch as just another repository of the same project.

So to create a brach you either could get your current repository from your local machine, or put your local repository to your remote machine. Often some computers aren’t accessible from other networks (NAT), so it’s necessary to be able to put them.


darcs put versioning.company.com:/path/to/your/project_STABLE

Again, it’s absolutely not necessary to have the branch on another computer, it’s perfectly good to put them on the same machine.

Creating/Tagging a version

Darcs does not create versions for each patch you record. You have to mark versions yourself. So, to tag the current state of your repository as a version you use darcs tag.


$ darcs tag
What is the version name? Version1
Finished tagging patch 'TAG Version1'

This versions are really what you would expect from versions if you don’t know CVS or SVN.

Sending patches via mail

Especially Open Source projects rely on patches from various people, but are not able to give anyone write access to the source code repository. Darcs enables you to send patches directly via mail instead of writing directly to the repository.

This gives the author of the project the possibility to check your changes and accept them or not.

Darcs needs to compare your local repository with another one (preferably the one you’d like to get your patch in).


darcs send versioning.opensource.org:/path/to/their/project

Again, Darcs will ask you for the changes, and the email address you’d like to send them. The receiver will use darcs apply to apply the patch if it’s good.

Looking at changes

Why use a version control system if you don’t watch your changes? To get a summary of all changes made to the repository you use darcs changes. It gives you a standard changelog you can include in your release notes, or changes.txt.

Assuring quality

Darcs has a lot of options, one of those is the ability to define a test program. This test procedure is being executed every time you record a patch. The patch is only recorded, if the test succeeds (determined via the shell-error codes).

To define a test procedure or test program, you simply type:


darcs setpref test "echo 'you really, really should test your code'"

This is written into your project/_darcs/prefs/prefs file, you can also define a system wide test procedure in ~/.darcs/prefs.

Tests are executed on a special working copy (speak: clean checkout from the repository).

Some advanced operations you might need

If you recorded a bad (buggy) patch, and you’d like to remove it before anything bad happens, you can quickly use darcs rollback. Rollback records an inverted patch, it does not affect your working directory.

If you’d like to physically remove patches from your repository (not working directory), darcs unrecord is just for you. If you’d also like to remove the changes from your working directory you should use darcs unpull, please not that you’ll loose the content of these patches!

If you forgot some change, or found a bug in a previous patch, you can use darcs ammend-record to add changes to the previous patch.

A feature you’re very unlikely to need is darcs trackdown, but it gives a peek view of what Darcs’ simple but powerful patch-system is capable of.

For example search for the patch that changed the password in some file:


darcs trackdown "grep 'adminPassword: admin' configuration.cfg"

Removes a patch after another until the command (grep in this case) returns true. Patches are removed on a special working copy, just like the tests.

Short Reference

This is a very short reference I wrote for myself, the manpage or the manuals does a better job. Maybe you’ll find it useful anyways.

`initialize`	sets the repository up. Can be executed in an empty directory or within your existing project directory.
`add`	adds one or more files to the repository. You can use `darcs add * -r` to add all files within this and subsequent directories to the repository.
`whatsnew`	shows all changes. You can use the command line switch `-l` to see a short summary and files which are not in the repository.
`record`	persits the changes to the repository. Each change is a patch of it’s own, so usually you’ll use `record` very frequently. `darcs record -am "comment"` applies the same description to all open changes.
`changes`	lists all changes (patches) recorded in this repository.
`push`	sends patches to another repository (local or remote), usually via SSH.
`pull`	gets patches from other repositories (local or remote).
`remove`	removes the file from the repository. Be aware that this will delete the file from remote repositories, but not your working repo.
`mv`	renames or moves a file. Using this command Darcs is able to handle it without losing consistency in other repos.
`replace`	changes tokens within your files. It’s like search and replace, but has some backup built-in.
`send`	packs a set of patches in a file and sends it via mail.
`apply`	takes the set of patches and applies it to the current repository.
`get`	fetches a complete repository (creates a new directory for the repository)
`put`	copies the current repository to another location. Creates a new directory if not already existent, and checks whether it’s “free” (not already a Darcs repository).
`tag`	is used to name the current state of the repository. It’s used to create versions. You can give `--tag` parameters to `get` for example to fetch a specific version.
`optimize`	speeds the operations on the repository up. Most useful for remote repositories.
`amend-record`	replaces a previous record with a new one. Has consistency problems and shouldn’t be used all too often.
`rollback`	logically removes the previous patch. Logically: the original patch remains, only a inverse patch is added. Doesn’t affect the working directory.
`unrecord`	physically removes a patch.
`revert`	removes changes made to the working directory. It’ll reflect the state of the repository.
`unrevert`	can undo the last revert.
`unpull`	removes a patch from the repository and from the working directory. Nothing left if you’re the only repository with that patch! (Is like unrecord and revert, just without the unrevert possibility)
`annotate`	gives the complete history of a file or directory.
`trackdown`	iteratively removes the last patch and runs the given command until it returns no more errors. For example `darcs trackdown 'grep TODO file.test'` returns the last patch that had “TODO” in “file.test”.

Ommited commands: repair, dist, resolve, diff, obliterate, check, setpref. I don’t think they’re important, but as you can see, Darcs isn’t very bloated.

Update: Link to binaries for Mac OS X added.

Ten questions

2006-08-01T00:00:00+02:00

Jaroslaw Rzeszótko sent a list of ten questions to some of the best known programmers in/on the net¹. It was a great idea, I love it. The questions are very basic, but what else would you ask? Anyway I started to answer them myself, so here they are²:

How did you learn programming? Were any schools of any use? Or maybe you didn’t even bother with ending any schools :)?

I taught myself GWA Basic on an Atari ST 2600. After I got a PC I started working with Turbo Pascal, Borland C++ and even Assembler.

Then I chose a technical school (HTBLuVA Villach) where I’ve been re-introduced to programming, this time in a more systematic way. We were taught C, C++, Cobol (CICS), Assembler and Java which was very, very good.

Now I’m happy that I was forced to learn Cobol and programming for the Windows using MFC since I would have never looked at it (and hopefully never have to do so again).

Later I went to the Technical University of Vienna. I’m not very satisfied with it’s introduction courses in programming (if I had to do them, I wouldn’t know much about programming), but the advanced courses are pretty interesting and challenging.

What do you think is the most important skill every programmer should possess?

Personal view:	Communication abilities
Esoteric view:	Focus and taste

Do you think mathematics and/or physics are an important skill for a programmer? Why?

I don’t think that physics is an important skill for programmers. Of course, everything you know helps you in some way, but we (as programmers) live in a idealized world where the laws of physics are abstracted. I think biology would be much more important than physics (think of genetic algorithms, cellular automata etc), but that’s only a guess.

Mathematics is very important, it’s hard to explain, but it gives you the ability to analyze your programs in a easily reproducible manner. If you know math, you’ll stand out of the programmer ~~mess~~ mass.

If you had three months to learn one relatively new technology, which one would you choose?

Currently I’m on the Web-side of life, so there is a lot I’d like to know and learn:

Compilers are always a hot topic for me.
Mobile computing will be big in the future.
Game-development for consoles, handhelds or even PCs would be cool too.
Artificial Intelligence and Desktop programming (on the Mac) are also interesting candidates.

What do you think makes some programmers 10 or 100 times more productive than others?

Honestly I think the most important habit is to keep learning and adapt to changing environments (call it requirements). You can be pretty fast by practicing bad habits, but learning new techniques will boost you in the mid/long term.

What are your favorite tools and why do you like them more than others?

Operating System
- I’m currently a big OS X fan, it gives me a Unix OS with commercial support and applications.
- Linux, because it had the biggest impact on me in my “early days”.
- Certainly not MS Win. I don’t know why, I just don’t like it. Maybe because “everyone” has it.
Programming Languages
- For Web applications and scripts
  - Ruby it is a lot of fun and gives you immediate satisfaction. Python and Perl didn’t win my heart.
- For larger programs (like my IMAP4 and POP3 Server)
  - Java because I’m very used to it and too lazy to learn equivalent programming languages like C#. C++ is the only language I ~~know~~ knew well enough to compare it with Java (I know these two languages far longer than all other languages combined).
- For Desktop applications
  - Sadly I don’t really know much about it. Java is no option (IMO), and my last native desktop applications (written in C++) is about seven years old. I guess I would choose Cocoa, C or C# depending on the operating system (Mac OS X, Linux, or Windows respectively).
- Functional Programming (I really like this style)
  - Haskell has many interesting ideas and is very powerful. Lisp/Scheme are nice too, but I’m more into Haskell.
Text Editor
- TextMate it made me dream – you can still make money with a text editor.
- VIm (although I could use some practice with it) because it is omnipresent.
Version Control System
- Darcs innovative and intuitive.
Shell
- Bash + Bash Completion (nevery really looked at something else).
Database Engine
- SQLite simple and fast (with duck-typing).
Other tools
- Firefox it’s extensible and does it’s job.

What is your favorite book related to computer programming?

I really like Donald Knuth’s The Art of Computer Programming 1-3. I never finished a single book of this series, but I know that Mr. Knuth put a tremendous amount of work into them³. They simply have the touch of “old school” and superiority.

I also love Test Driven Development by Kent Beck. It’s only a small book but had a huge impact on my programming style.

Then I’d like to include Agile Software Development. Principles, Patterns, and Practices from Robert C. Martin which gave me an idea what “professional software development” could be.

More on the “soft” side is The Pragmatic Programmer from Andrew Hunt and Dave Thomas.

What is your favorite book NOT related to computer programming?

Fiction	Neil Stephenson’s “Cryptonomicon”:
Non-Fiction	Edwart Tufte’s The Visual Display of Quantitative Information
Soft Skills	Tom DeMarco, Timothy Lister Peopleware

What are your favorite music bands?

System of a Down, Pearl Jam, Alanis Morissette…

¹ I think only because they are very present in/on the net, they are not necessarily the best programmers in the world.

² Please not that I don’t consider myself anywhere near the people Jaroslaw asked.

³ Almost any hard problem is scared only by opening one of them.

Don't over-reuse

2006-07-20T00:00:00+02:00

I’m no big fan of frameworks. This comes from bad experience with a framework drawn out of a specialized project. It had to be used in every project we had to implement whether it was appropriate or not. The argument was something like “it provides features you’d have to implement yourself, so it saves you time”. Usually these features would take about one hour to half a day to implement new — exactly fitting the problem. But we had to take all the clutter and deal with it. This was about six years ago, and as far as I know they still use this exact framework… .

Reuse is a good thing, we all know that, and we all draw our benefits from it. But you can certainly overdo it, not every project fits a specific framework.

Ben had a link to recordings of the last RailsConf (and insisted that I watch them) where Martin Fowler and David Heinemeier Hansson talked about what the essence of Rails really is. It does fit a few particular cases very well, but it doesn’t solve all problems for Web developers. Fowler’s talk is a bit more general on Software development, and DHH’s talk presents new ideas for the upcoming version of Rails, but starts with the philosophy and the goal as well as the non-goals of Rails. Both talks are highly recommended.

The reason I write about it is Bruce Eckel’s articel ‘When Reuse Goes Bad’.

This resulted in classic framework-itis. The contractor was no longer just trying to solve the customer’s problem, they were trying to solve all problems that looked like it. So for that reason alone the project would have required 10x the original time and money.

Read it, it’s a short one. I really felt affected by it.

Mental workout

2006-06-25T00:00:00+02:00

According to this article from Popular Science thinking really hard causes your brain to consume 15 times more calories than in “stand-by mode”. The calorie consumption goes up from 0.1 calorie per minute to 1.5 calories per minute.

So let’s start our mental workout with some easy warmup exercises

3D Logic

In 3D Logic you’ll have to connect two colored fields with a line in the same color on a 3D cube.

On this cube (1st level)

you’ll have to connect the two red squares with a red line, the two violet ones with a violet line and the green one… you got it. This game takes some time, but don’t be fooled it gets pretty hard in later levels!

Don’t be tired! Jump in to the next exercise

Planetary

Planetary requires you to make a non-planar graph planar (so that no edges intersect).

Transform this graph (3rd level):

to this one

At the end some stretching exercises

Sudoku

Of course Sudoku is the classic, it’s everywhere and of course online too — at WebSudoku for example. Currently there are also some multiplayer variants coming up, Sudoku Combat is a popular one.

Screenshot of SudokuCombat:

And? Lost some weight?

The year 2038 problem

2006-06-21T00:00:00+02:00

I just found some information on an upcoming problem, the year 2038 problem. On that page is also an program to test whether your operating system has a problem or not.

In summary: the system time is usually stored as seconds since 1970 in a 32-bit register (or variable). On 2038-01-19 at 03:14:07 the register will overflow… . The solution is to switch to an 64-bit register (sufficient more than 4,000,000,000,000 years).

Usually not only the operating system has to be tuned, all time variables have to be large enough. 32 years aren’t that much for some programs (I’ve heard :-)). BIOS variants may also have problems on that day.

Some operations do have that problem now! Just think of an average of two dates after Jan 2004 (a+b)/2! a+b is already too large for 32 bits.

I couldn’t resist to check it on my systems

Linux currently has a problem, Mac OS X doesn’t.

Mac OS X Tiger on a G5

Tue Jan 19 03:14:01 2038
Tue Jan 19 03:14:02 2038
Tue Jan 19 03:14:03 2038
Tue Jan 19 03:14:04 2038
Tue Jan 19 03:14:05 2038
Tue Jan 19 03:14:06 2038
Tue Jan 19 03:14:07 2038
Tue Jan 19 03:14:07 2038
Tue Jan 19 03:14:07 2038
Tue Jan 19 03:14:07 2038

Mac OS X Tiger on a G4

Tue Jan 19 03:14:01 2038
Tue Jan 19 03:14:02 2038
Tue Jan 19 03:14:03 2038
Tue Jan 19 03:14:04 2038
Tue Jan 19 03:14:05 2038
Tue Jan 19 03:14:06 2038
Tue Jan 19 03:14:07 2038
Tue Jan 19 03:14:07 2038
Tue Jan 19 03:14:07 2038
Tue Jan 19 03:14:07 2038

Linux (Kernel 2.6.16)

Tue Jan 19 03:14:01 2038
Tue Jan 19 03:14:02 2038
Tue Jan 19 03:14:03 2038
Tue Jan 19 03:14:04 2038
Tue Jan 19 03:14:05 2038
Tue Jan 19 03:14:06 2038
Tue Jan 19 03:14:07 2038
Fri Dec 13 20:45:52 1901
Fri Dec 13 20:45:52 1901
Fri Dec 13 20:45:52 1901

Manage your people

2006-04-22T00:00:00+02:00

Whereas I don’t have the experience to write about it, I’d like to show some lecture notes and summaries about leading in high-tech.

Science, technology and tools often hide the most important fact: your people are your capital! The better your people the better your products. Of course you can hire highly trained people, but it’s your obligation to give them space and opportunity to evolve.

Your employees are highly motivated and highly trained after their schooling, but all to often the get demotivated far too fast. Why? One thing I’ve blogged about earlier is to tell them how to do their task instead the goal. This leads to an interest conflict, the requirements may conflict with the employees liability. Simply put: I can’t be responsible for things I was forced to do.

Software Engineering is somehow special in some cases:

Ideas are the driving force
The outcome is often unsure
Research leads to knowledge gain; Development to products; these two often get mixed up and lead to nowhere
Progress is hard to measure (90% syndrome)
Unsafe starting-point (changing requirements; incomplete foundation)
High complexity of modern software products

To support the worker, you must make sure

to give them enough freedom to think
to enable quiet and uninterrupted working
to enable communication between the people
to supply the needed tools
not to condemn errors – these are necessary within a creative environment

Important points to keep:

bad ideas lead to bad products (often the first idea won’t be the best)
the best idea won’t lead to success if the routine work fails
methods enable creative thinking (they free you from thinking about routine tasks)
processes make projects faster! If a process slows you down (long-term), it’s the wrong one.
your people want to be proud of their work, make sure they can.

Growth

2006-04-22T00:00:00+02:00

I’ve seen companies which try hard to grow¹. I often can’t see the point in getting more people to work for them. Investing in the current people pays much more, I think. If you take the span of bad people to very good people which can be ten times more productive², wouldn’t it be worth to invest five times more money and time into these people than hiring more and more? Every single person increases the communication effort and slows the whole organization down. Whereas a slim, highly productive team increases the quality and lowers communication effort.

More productive?

There are many factors which influence the productivity of people in software development.

Speed (Function Points per Week)
Defect rate (bugs per Function Point)
Communication effort (knows how to use Google and Books)
Cost (Cost per Function Point)
many many more…

Lets compare two groups of developers, one slightly better than the other⁵.

Developer	FPs/Week	Bugs/Fp	Euro/Week	Euro/FP	Additional communication-effort
A	100	0.01	1200	12	0.05
B	66	0.02	800	12	0.10

So, developer A is clearly more productive than developer B, but costs more.

Imagine a project with 1,000 Function Points. It has to be code complete in five weeks.

Group of A-Developers

1,000 Function Points in five weeks are 200 Function Points per week. Our team has to consist of three A-developers. Ideally, three developers produce 300 Function Points, but we have to consider the communication effort: (3 × 100) x (0.95³) = max. 257 Function Points. Cost: 18,000 Euro with a combined defect rate of 0.0297³.

Group of B-Developers

This group has to consist of six developers. (6 × 66) x (0.9⁶) = max 210 Function Points. Cost: 24,000 Euro with a combined defect rate of 0.114.

Summary

As you can see, a few developers which are slightly better than another group can easily outperform a double sized team:

Team	Number of developers	max. FPs	Cost	Defect rate	Communication effort
A	3	257	18,000	0.0297	0.1426
B	6	210	24,000	0.114	0.4686

You could easily pay the A-Developers the double amount of the B-Developers and you would still produce a better⁴ product in the same time with the same expenses! Additionally the A-team still has more capacity left for changing requirements or unforeseen problems.

Again, wouldn’t it be far better to support and develop your current staff than simply adding more people?

¹ In terms of “hiring people”.

² The numbers came from two different sources. The first one is senior Siemens manager, and the second one is Paul Graham.

³ One minus the probability of no error: 1 – 0.99³

⁴ More features or superior usability and less defects.

⁵ Never trust a statistic you didn’t fake yourself.

The First Idea

2006-04-21T00:00:00+02:00

I don’t like first ideas. Most people working with me may have painfully experienced that¹.

I really think that idea generation is vital in our business. We can’t afford (anymore) to take the first idea and start working. Knowing the alternatives is much too important. All too often I see people thinking of a problem and the first idea that seems to make sense is it. These (non)efforts result in changing directions in the middle of the development, or in starting over. Both very expensive experiences.

Sometimes the fist glance (see Blink²) is very important and a very powerful tool. But in this case, with systems more complex than most other products in the world, that won’t work.

I really enjoyed Scott Berkuns “The Art of Project Management³”. Especially because he dedicated some pages on that problem. When we start to think about a problem we should explore the problem space as far as possible⁴. Know what’s allowed to do and think of as much possible solution within these boundaries as possible. At some point the solution space stops expanding and collapses. This is partly because some solutions are not feasible, and partly because the solutions overlap. At the end you should have a few very distinct solution-proposals. Now it’s time to analyze those. Do they really solve your problem? Is this what the users want? Is it feasible? How much does it cost?

At the end of this process you should have one solution which is very likely significantly different from the first idea. And you have a high chance that you’ve chosen one of the best solutions possible.

Bad ideas simply lead to bad products, but also good ideas lead to bad products if the rest fails. Nevertheless, many of the failing projects⁴ have problems with their foundation (requirements and ideas) than with the implementation (design and code). Unfortunately changing the foundation gets very expensive the further the project develops.

¹ On the other hand, my blog-articles are rarely more than a blink of an early idea.

² Blink by Malcolm Gladwell

³ The Art of Project Management by Scott Berkun

⁴ You have to know when to stop.

⁵ Number vary, but standish offered these: 31% are cancelled before completion, and 22% slip their schedules. Most of them had problems with their foundations.

The Pragmatic Programmer

2006-04-05T00:00:00+02:00

Wow, that’s a great book! I own over 80 non-fiction books but only few of them excite me that much. The Pragmatic Programmer from journeyman to master from Andrew Hunt and David Thomas is one of those. It’s stuffed full with practical wisdom and tips you can start using from the moment you read it.

They discuss all relevant topics of programming and software development, starting from text editors, IDEs, code generation, command line versus GUI etc… to estimation, handling requirements and documentation. It’s fantastic and really fun to read, too sad it has only 300 pages.

Usually the quotes on the back-flap of any book should be handled with care, but Ward Cunninghams quote If I’m putting together a project, it’s the authors of this book that I want. … And failing that I’d settle for people who’ve read their book. doesn’t really surprise me after reading just a few chapters.

(The second one equally full of wisdom, but with another focus, is Robert C. Martins Agile Software Development. Principles, Patterns, and Practices. Before buying several books on Design Patterns, Agile Methodologies and Software Development guidelines buy this one.)

Even if these books are from programmers for programmers, they’re extremely well written (meaning they don’t use nested parentheses (like me))!

Why you should use a plain text editor for programming

2006-04-03T00:00:00+02:00

I’m always impressed if someone codes like it’s all he or she has ever done, without IDE and still n-times faster than anyone with an IDE.

During the past few years I’ve done a lot of Java coding. I’ve used Eclipse most of the time, but until now I didn’t realize how little I really know about Java¹ — my skills largely depended on my IDE of choice.

Most of the time I don’t have a clue where (in which package) a specific class lies, as well as I don’t know (even roughly) what exceptions are thrown. For these tasks I asked my IDE to generate the code I needed (import, try/catch or throws statements).

The last few days I used ANTLR to generate a parser. I didn’t use an IDE (there is none for ANTLR 2 available for free), which meant I had to refer to the Java API to check whether my imports are right, how this or that method is named exactly and which arguments it’ll take (what Eclipse has done so far. It is to be blamed that I only know approximately the first three characters of a method name²).

If we are coding for productivity, in our day job or where time matters more than sharpening our skills, it’s fine (and sometimes crucial) to use an IDE. But if we want to learn something and get better (smarter) at our tasks it actually does harm: we’ll end up thinking man that’s easy I must know pretty much everything about this or that⁵… where in fact we only know how to use our IDE and don’t know s##t about the language itself³.

Another dangerous thing with code generators is, that it seems pretty easy to get up and running with a mid-sized application. We write a bit, generate a lot and we’re ready to go, isn’t that so? No, only at the beginning! If we (or someone else) need to change something in that code, maybe weeks or months later we have to understand every bit of the generated code (Ben has written about that too), now we’re screwed. If the code looks messy why should anyone bother to do his/her best to make clean changes?

If it’s easy to copy and generate lots of code why should one bother and think about abstraction, generalization and stuff? If I had to write getters and setters by hand, I would do anything to avoid it. Maybe (just maybe…) I’d just start designing my programs, making a real O-O design (or write a generator to write them for me, but that’s another story).

The previous problem can easily be avoided by hiding the generating the code. In Lisp one writes macros to generate code at compile time, xDoclet generates code as it’s own step, EJB 3 generates code at compile time too.

Now, what’s the point? Use a plain text-editor⁴ for educational purposes and avoid generating too much code with your IDE on your day job.

¹ Java in terms of packages structure, method names etc. Meta-concepts like algorithms, techniques, patterns etc. are out of question.

² That’s a great example for a quote out of Donald Normans book The Design of Everyday Things Knowledge in the head, and in the world: Precise behavior can emerge from imprecise knowledge, and, of course his Things that make us Smart book. None of these things make us an expert… merely an user.

³ You can be a pretty smart programmer, but don’t know a lot about the used programming language. To be a true expert you have to dive into the language itself and don’t rely on an IDE.

⁴ There are many high quality editors out there. I like TextMate most, but Emacs, Vim and JEdit are quite good too (and free, but keep the learning curve in mind).

⁵ See Unskilled and Unaware of it (PDF, 500kb). Somewhat related: Being a specialist (PDF, 164kb).

Ruby: future and delayed (lazy) variables

2006-03-12T00:00:00+01:00

Future variables

Future variables are a very popular technique to implement dataflow execution. A future variable gets assigned with a (potentially lengthy) operation which will be executed in a separated thread. The assignment is therefore non-blocking.

As soon as the variable gets read, it blocks if the calculation isn’t ready, or it immediately returns the result.

I’ve implemented that feature using some Ruby tricks, but the finest and most elegant solution would be in Lisp using it’s unique macros.

Anyways, here a quick example:


x = RFuture.future do
   # some lengthy operation with an interesting result
end

Now, that assignment doesn’t block, and we could do some other things, later on we may want to know what the result of our operation is:


puts x

This code could block if the result isn’t ready.

As you can see, there is no special handling for reading the variable (meaning using adopting future variables later on is quite easy).

In the best case scenario the saved execution time is pretty high (compared to sequential processing):

Delayed (lazy) variables

Another concept introduced in this library are delayed or lazy variables. These are variables which get assigned a operation which isn’t executed until we really need the result. Seems pretty pointless, but it’s used in some functional programming languages (like Hasekll).


x = RFuture.delay do
   # some operation with an interesting result we may need later on
end

Again, reading the variable doesn’t differ from usual variables:


puts x

Download the file here.

Installing SQLite, Lighttpd, FastCGI and Ruby bindings on Mac OS X

2006-02-28T00:00:00+01:00

I’ve had a hard time installing SQLite, Lighttpd, FastCGI together with Ruby bindings on OS X, so I thought I could share my experiences. Maybe someone has similar problems and finds some hints for solving them:

DarwinPorts

I used DarwinPorts to install the software, don’t forget to include the target directories to your path.

For example I install the packages to /opt/local, so I’ve included /opt/local/bin and /opt/local/sbin to may path.

Example .profile:


export PATH=/opt/local/bin:/opt/local/sbin:$PATH

SQLite and Ruby

That was the hard part since my Ruby installation was not properly installed. The best way to install it is to remove ruby and it’s dependencies first:


port -uf uninstall ruby

After that install SWIG, Ruby and Ruby Gems in that order:


port install swig
port install ruby
port install rb-rubygems

Mac OS X Tiger has SQLite3 pre-installed, if you choose to use SQLite 2 you have to install it from DarwinPorts:


port install sqlite

Now we’re ready to install the Ruby bindings for SQLite:


gem install sqlite3-ruby

or for SQLite 2:


gem install sqlite-ruby

My problem was that I had to remove Ruby, install swig and reinstall Ruby. If swig isn’t installed gem can’t compile the native bindings for Ruby and you’ll get something like:

sudo gem install sqlite
Attempting local installation of 'sqlite'
Local gem file not found: sqlite*.gem
Attempting remote installation of 'sqlite'
Building native extensions.  This could take a while...
ERROR:  While executing gem ... (RuntimeError)
    ERROR: Failed to build gem native extension.
Gem files will remain installed in /usr/lib/ruby/gems/1.8/gems/sqlite-2.0.1 for inspection.
  ruby extconf.rb install sqlite
checking for main() in -lsqlite... no
checking for sqlite.h... no

Lighttpd and FastCGI

Now we’re ready to install Lighttpd and FastCGI:


port install fcgi
port install pcre
port install lighttpd
gem install fcgi

Now what?

Now you’ve got a good starting point for other tutorials like:

How to use ActiveRecord outside Rails
A Quick Guide to SQLite and Ruby
or simply start working with Web.py, Ruby on Rails, Turbo Gears or Django

Using your home Internet connection with your mobile phone and a Mac

2005-11-16T00:00:00+01:00

After several hours of fooling around with my Nokia 6680 and some tools I’ve finally succeeded! I’m able to use my home Internet connection on my phone via Bluetooth! Now I’m able to load my phone with tons of software, data and experiment with other nice applications like Google Maps without paying 1.5 Euro per Megabyte!

Here a short step-by-step guide:

Setup a Serial Port on your Mac:
1. Open Bluetooth in your System Preferences
2. Make sure your mobile is paired with your Mac (it should appear in the Devices pane)
3. Go to the Sharing pane. Here you should have a Bluetooth-PDA-Sync click on it and enable Require pairing for security and make shure that your Port Type is set to RS-232.
  If you don’t have this port simply click on Add Serial Port Service and configure it as described above.
4. It should look like this:
5. Now run the share script (at the end of this article) to associate a pppd and share your connection
Download xans Version of GnuBox
Install it on your mobile.
1. Create a new Access Point named Bt on your mobile.
2. Set it up according to xans howto
3. Set your mobiles IP to 192.168.1.131 (or the IP you chose for it in the above script).
4. Setup the DNS IPs of your provider.
Start the gnubox application
1. Ignore any error messages at the start
2. Choose Options→Install→create records
3. Choose Options→Install→copy settings from GPRS
4. Choose Options→2box Bluetooth→Serial port
5. Choose Options→Debug→Bring up IF
6. That’s it! If you have any troubles, please refer to xans troubleshooting guide
7. I’ve used Opera to test the installation. You have to configure it exactly like here .

It’s likely that some of this steps aren’t necessary, but it’s working for me.

Here the script to share your serial port with your AirPort. You may want to adapt the script according to your setup

IP Address (be sure to change it on both ends, mobile and Mac),
Serial Port name,
interface to share (for example the en1 interface is your AirPort and the en0 your Ethernet)

   
 sudo /usr/sbin/pppd /dev/tty.Bluetooth-PDA-Sync 115200 \
   noauth local passive proxyarp \ 
   asyncmap 0 silent persist :192.168.1.131
 sudo /usr/sbin/sysctl -w net.inet.ip.forwarding=1
 sudo /usr/sbin/natd -same_ports -use_sockets -log -deny_incoming -interface en1
 sudo /sbin/ipfw add divert natd ip from any to any via en1

Unshare

To unshare your port use this script:

   
 sudo /sbin/ipfw -f flush
 sudo killall natd
 sudo /usr/sbin/sysctl -w net.inet.ip.forwarding=0
 sudo killall pppd

or simply restart your computer.

Further info

If you still have troubles try these ressources in order: xans howto xans troubleshooting guide support forum

There’s also guide for Macs and Nokia 6600 mobiles available.

A very handy tool for debugging is the Bluetooth PacketLogger, available if you’ve XCode installed (take a look at /Developer/Applications/Utilities/Bluetooth).

Bluetooth Hack Series60 Nokia

Productivity and programming languages

2005-06-10T00:00:00+02:00

We all know that programming languages affect productivity. For example writing a word processor in Assembler is much harder than writing it in C++. If seems natural to have a table with all available programming languages with their corresponding productivity value.

The main problem is to compare these languages. A common method is counting the “lines of code” a programmer needs for a given program. But who is going to implement the same specification in hundreds of languages? So to make the test applicable we need a common concept for the comparison.

This concept is called “Function Point Analysis”. This approach was introduced by IBM for exactly this task. Here you define function points (there are a lot of rules for doing that) and measure how many function points a programmer implements per month. More information about function points: Function Point FAQ and Fundamentals of FPA.

For the productivity measurement of programming languages we count the lines of code per function point.

The languages are categorized into levels – Assembler for example has level 1 with 320 lines of code per function point. A Macro Assembler has level 1.5 with 213 lines of code per function point etc…

Here an excerpt of this list:

Language	Level	LoC
Assembly (Basic)	1.00	320
Assembly (Macro)	1.50	213
C	2.50	128
C++	6.00	53
Haskell	8.50	38
JAVA	6.00	53
LISP	5.00	64
Objective-C	12.00	27
SMALLTALK 80	15.00	21

A lot of languages are special languages – like Excel with level 57. My guess is that a general purpose language is not able to go far beyond level 20.

Another table from here shows this:

Language	LoC
Assembler	320
Macro Assembler	213
C	150
Lisp	64
Basic	64
Objective – C	26
Smalltalk	21
Query Languages	16
Spreadsheet Languages	6

The values are the same or at least nearly the same for the languages (maybe they came from the same source?)

I would like to see a JavaScript, Ruby and Python included in these tables, but unfortunately I couldn’t find anything for these languages. I think JavaScript is no better than 10, Python and Ruby are together with Perl around 15, maybe Ruby has the highest level of these languages (Comments?).

The higher the language level the easier it is to read and change such a program. There is – of course – a trade off: the more code you write (for example in assembler) the more optimizations are possible. With increasing processor speed, and advanced algorithms this argument is getting more and more obsolete.

REST Web-services

2005-04-10T00:00:00+02:00

If people talk of “Web-Services”, they often mean SOAP as maintained by the W3C. But SOAP isn’t everything. In fact SOAP was a basically a RPC (Remote Procedure Call) protocoll over HTTP (therefore the name Simple Object Access Protocoll). Since W3C took over SOAP (from Microsoft and later IBM) it has been extended into more general directions.

With REST (Representational State Transfer) similar applications can be written. REST itself is only an architectural style (defining how you should design your application). REST was initiated by the IETF as the basic model for the modern Web. HTTP 1.1 was designed using REST as tool for discovering unproper features etc… .

What is the essence of REST?

REST sees the Web as a collection of resources connected to a common network. Each resource has a set of associated methods (PUT, GET, POST, DELETE) to operate on them. SOAP on the other hand has services with definable methods (like getCustomerRecord, setCPUHeat etc…).

Since REST heavily depends (or defines) on existing standards existing software can easily be used with it. For example firewall administrators can block writing methods (DELETE, PUT, POST) on some resources, or internet caches are able to store resources requested by GET. SOAP uses POST (non-cacheable) to transfer the data between the client and the server. Using SOAP a firewall administrator has no chance to control access to his resources – he can allow or disallow SOAP (meaning HTTP). SOAP applications have to manager their own security.

The REST/SOAP debate is often compared to pre-SQL eras. Think of a database providing “getCustomerRecord” and “setCPUHeat” commands instead of general SELECT and UPDATE statements on resources like “Customer” and “CPU”. The same applies to REST – here we have general methods, with SOAP we have specialized interfaces.

Why SOAP if REST is so good?

SOAP is a product (at least you can make products from the SOAP specification) – REST is “only” an architectural style. Guess with which one companies can make money? REST of course has it’s drawbacks too, for example you cannot buy libraries to ease some task… . REST isn’t widely known/accepted and few companies provide explicit support. Amazon calls it’s XML over HTTP a REST interface, del.icio.us and flickr have REST interfaces too (usually XML over HTTP). The biggest drawback of REST is the lack of a description language – SOAP has a very powerful one with WSDL.

Want a more in-depth treatment? REST article in PDF

Lessons of game design

2005-03-10T00:00:00+01:00

If you’re searching UML, Design Patterns or suchlike, you’re wrong here – this time I’ll try to cover some principles of puzzle design to general software from the point of the user:

Scott Kim, a well known Puzzle creator has written a great paper on Puzzle design called “The Puzzlemaker’s Survival Kit” (sorry, no link available).

He goes through principles, case studies and the design process of puzzle and game design. Some of these guides can (IMHO) applied to software design in general.

How? I think, puzzles in games and features in general software aren’t that different. Here the first principle of puzzle design written by Scott Kim:

He also writes:

bq.A puzzle is a problem that is fun to solve!
What makes puzzles different from problems is that they are fun to solve. A good puzzle is guaranteed to have a solution that is neither too easy nor too hard to find.

If we map puzzle to real world problem I want/have to solve we should design our software to allow this as easy as possible and powerful as needed. So, our software should enable the user to solve a problem with joy!

I wouldn’t limit “fun” to games – I think we could build software that every problem is fun to solve because the software helps us and doesn’t step in the way. It depends only on our imagination and our willing to invest a lot of time to build software that is fun to use. (Although game puzzles may be more fun to solve, but not as rewarding).

…neither too easy nor too hard… is another interesting phrase. Doesn’t that map directly to the power of any feature included into a piece of software? Conclusion: don’t treat your users as complete dumbs, design your features right – not too complex, but with enough power to perform a specific kind of task.

Scott Kim explains how to construct puzzles in games, let’s see if we can map these to software in general.

Think small.

Interesting, huh? Think small as advice! Isn’t that the same as Apples Think different – just take the iPod Shuffle or the Mac Mini? A lot of people try to get more and more features into a piece of software (Think big), but these systems tend to be overly complex and unusable (at least for me).

See how far you can get with the smallest number of features.

This is one of the most thrilling and adorable advices I’ve ever read! Don’t build battalions of special cases. Extract, refine and rewrite until you’ve got a truly elegant solution for the problems you want to solve. The web, meaning HTTP, follows exactly this advice. They striped out every special case and build a solid foundation for thousands of web applications by not disturbing or limiting the possibilities. See Axioms of Web architecture for more.

Many, if not all, of the brightest minds of history gave this advice – starting from Da Vinci Simplicity is the ultimate sophistication., Hoare The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay., Dijkstra Simplicity and elegance are unpopular because thy require hard work and discipline to achieve and education to be appreciated. or Simplicity is prerequisite for reliability.… .

Start with a tutorial.

Definitively a game design advice, or not? Eclipse, GMail and many more start with a tutorial – maybe we should think of including a quick and simple introduction into any given piece of software?

Make puzzles easy to author.
…
Construction puzzles need to be as easy and fun to author as they are
to solve.
…

This is a very underestimated advice in general software. Why do we think that we are the great masterminds and know every single case our users want to do with the software? There is no possible way for doing that with any given real world problem!

Open up your software and let users create features the need through an API, scripting capabilities or whatever.
Design these interfaces so, that no existing code has to be read to use the API, Interface or Scripting capabilities. Make it easy to build new functionality and don’t even try to lock in the user!

Even Microsoft allows users to build their own little programs within their office suite.

Another big advantage: You might use that feature yourself to build new features and customizations… .

Simplification!

Do it! Any feature you find “hard to use” is for the user simply unusable – your user doesn’t invest month or years learning (like you building) the software!

User interfaces which you – as the creator – find “super easy” are usually “just right” for your users.

[I’ve written this in a rush, so please excuse some weird phrases and formulations.]

Battery lifetime

2005-02-10T00:00:00+01:00

Rechargeable lithium-ion batteries are very common in gadgets like laptops, PDAs and cell phones, unfortunately these are very expensive. I had to replace my Sony battery for 250 Euro(!) – well I bought a cheaper clone for 100 Euro, but it has its drawbacks (at the beginning it worked for 3 hours, now for about one…).

So it pays if you keep your current battery up and running as long as possible. Here are a few facts I’ve collected over the last few months:

Lithium-ion batteries can be recharged whenever convenient, there is no memory effect
A new battery should be charged and discharged fully for the first three cycles (initial conditioning)
Keep the electrons moving (discharge/recharge the battery at least once every month)
Don’t keep your device plugged in all the time (this is very bad for the battery)
Keep the working temperature in mind (22°C is best for most devices)

I put my notebook battery out if I’m working plugged in for a longer time.

Notebook Li-Ion batteries have a typical lifetime of about three years, but this depends on how you use it. A friend of mine uses his notebook on a daily basis (on battery) and his battery works for about four years now – I’ve used my notebook occasionally (mostly plugged in) and the battery had to be replaced after 2.5 years… .

I’ve read some posts on bulletin boards telling me to put my battery into the refrigerator – these procedure should repair some structure and the battery should work like a new one. Well I’ve tried it with my old battery, but it didn’t work :-).

Broken batteries usually provide their full power but only on very low power consumption, for example my broken laptop battery could power my cell phone for days (weeks?) because it consumes much less power than the laptop.

More resources:
Apple
Cellpower

Well – I’m after all a software guy, so the electronic stuff written here should be used with caution :-).

Repetition is the root of all evil

2005-01-10T00:00:00+01:00

Yes, repetition is the root of all evil. Knuth (or was it Hoare?) said “Premature optimization is the root of all evil”, but I think it’s repetition. In fact repetition is often used to provide better performance (eg loop unrolling to suit your CPUs pipeline or the good old Duff’s device).

I’m talking about repetition in software – the typical copy and paste fault. Every programming book I’ve read so far deals in one or the other way with reducing repetition.

Think of patterns – each pattern tries to reduce the amount of code written by providing professional reuse mechanisms.
- Factory tries to concentrate the code for creating objects on one place,
- Strategy tries to supply a mechanism to traverse through interconnected objects,
- Facade lets an existing piece of code hide behind another interface.
- Singleton seems to be an exception because it reduces not the code replication rather the data replication.

In Test Driven Development the rules are simple:
- write a test (red bar)
- fake to get the bar green (unit tests are ok, when the “bar” is green)
- clean up and remove replications

Every programming language which doesn’t provide good facilities to reduce replication should be considered harmful! All functional languages have facilities to pass around functions, Python and Ruby introduced similar techniques, Java… well no, but you can simulate such a behavior with enormous amount of code (I don’t know C#, but I guess Anders Hejlsberg introduced something similar).

Whenever you look at a program or you’re writing one and you discover some kind of repetition: Remove it! Kill it! Destroy it… whatever but get rid of it!

Each time I introduced some sort of repetition and had to fix a bug in there, I forgot to fix the replica…

Now, what should we do to reduce replication? Sometimes it’s a bit hard to remove the replication. In the eighties the keyword was subclassing. Today subclassing is considered a performance killer and is sometimes referred to as hardwired code.
Now we have patterns – those magic constructs made by object gurus. I doubt that too many people do some kind of “pattern driven” design – those systems would get utterly complex and even harder to maintain than systems with a few repetitions.
So, whenever you encounter a repetition just remove it as best as you can, don’t hack! Think of a clean lean design and write some tests before changing something (just to be sure you didn’t break something).

That’s one of Extreme Programming’s weaknesses: there is no master plan the whole team adheres. Each pair of programmers implements their user stories and doesn’t know what the others do (unless the pairs are changed very (very) frequently). [Extreme Programming isn’t written in stone, every team can adopt it to it’s own needs – so maybe there are some who indeed have such a master-plan]

On the other hand repetition helps me to remember things, therefore:
Remove repetition in your software (and your friends software) as soon as you discover it!

IMHO

The Most Amazing PostgreSQL Database

hstore

JSON

PLv8

k-Nearest Neighbors

A REST API in Clojure

Creating the project scaffolding

Project configuration (project.clj)

The request handler (handler.clj)

The complete handler.clj

REST Framework Survey (Java, Haskell, Go, Node.js)

Step 1

Code

Step 2

Conclusion

Writing v1

Update to v2

Deployment

Summary

Programming Languages in Joy and Sorrow

Simplicity

Expressiveness

Performance

Productivity

Language assessment

Go Philosophical

Look for a language with good interop functionality.

Read some code before diving in

Enough with the subjective assessment, show me something objective!

In the beginning was the Test...

JUnit 4.9 Feature Roundup

Matchers

Assumptions

Categories

Theories

Rules

Misc additions

Infinitest

ClasspathSuite

Notebooks (dead wood)

How to Hide Google's "Google Plus Notification Count"

JavaScript

How to dive into Legacy Code

My steps

Haskell Books and Tutorials

Learn you a Haskell

Real World Haskell

Wikibook

Others

Git Overview

Basic Terms

Quick Intro

Commands

Adding/Removing changes to/from the Index

Checking differences between commits/index/working directory

Commiting changes

Sharing or “backing up” your work

How I use Git

Converting from Mercurial to Git

Missing some things from Mercurial

Which one is the right one?

Mercurial Version Control Status in the ZSH Command Line Prompt

Free and Open Source Math Programs

Hybrid

Computer Algebra System

Numerical Analysis/Computation

Statistics

Command-Line/Desktop Calculator

“Other” command-line/desktop calculators

GUI Calculators

Emacs Basics

Basic Concepts

Two things to remember

Major/Minor Modes

Emacs Lisp functions and key bindings

Emacs’ Architecture

Undo/Redo

Basic Movement

Search and Search & Replace

Advanced `rsync` parameters