IMHO

Haskell Books and Tutorials

Michael — 2008-10-23T22:25:31+00:00

Haskell is really a language very worth knowing. It does many things so different than most other languages which I really enjoy.

I’d like to use this post to mention a few really good resources for learning Haskell:

Learn you a Haskell

Learn you a Haskell is a very fun and entertaining tutorial for Haskell very much in the spirit of Why’s poignant guide to Ruby.

It isn’t finished yet, but it’s really good to start with. (Via Wadler’s Blog)

Real World Haskell

Real World Haskell is a upcoming book I really look forward to. It is freely available on its website, so check it out.

Wikibook

There is also a good collection of Haskell topics on Wikibooks – Programming/Haskell

Wikibook is more interesting if you already know a bit of Haskell and would like to understand it more in depth.

Others

I liked Yet another Haskell Tutorial very much, this aims at people with a bit of background, though.

Another more recent book on Haskell is Programming in Haskell which is nice, but also quite basic IMO.

Of course there is always the Gentle introduction to Haskell

Enjoy.

—
Haskell is doomed to succeed. —Sir Tony Hoare

Browser-JavaScript Performance on Mac OS X

Michael — 2008-09-29T20:53:07+00:00

JavaScript performance is one of the most important things in recent browser development.

Google did an awesome job with Chrome’s V8, and Mozilla’s Tracemonkey also raises the bar.

I usually compare the performance on this site

The tests were done on a MacBook Pro (2007), here the results:

Firefox 3.0.1 on Mac OS X

Firefox 3.1 Alpha (2008-09-29 nightly) on Mac OS X

Webkit (2008-09-29 nightly) on Mac OS X

Google Chrome running on top of Mac OS X in Parallels with Windows XP SP3

Note how fast Chrome is – even when running on top of two operating systems!

Opera 9.5 running on Mac OS X

—
The key to performance is elegance, not battalions of special cases. —Jon Bently and Doug McIlroy

Git Overview

Michael — 2008-08-24T15:42:12+00:00

My version control journey started with CVS, after that I looked at SVN, but never really used it. The shortcomings of centralized repositories were too obvious and with my increasing interest in Haskell I jumped on the distributed version control train with Darcs. I really, really liked it, but it had some nasty things too. After a while I was looking for something different and stumbled over Mercurial, again I was really happy with it but somehow my journey wasn’t over yet.

Recently I was watching two Git techtalks on YouTube:

The two talks are very different. The first one is more about why there’s place for yet another source code management with the rare opportunity to see Linus Torvalds giving a talk. The second one is about Git’s technicalities (commands etc.).

Basic Terms

Git distinguishes between the workspace, the index/staging area, the object store and remote repositories.

The workspace is where you do your actual work. It contains tracked files, untracked files and a special directory ”.git”.

The index (or staging area) is used for preparing commits. You can add files to the next commit or even only parts of files for the next commit. All the changes you’d like to commit get into the index first. A commit takes everything in the index and persists it in the object store.

The object store isn’t really an issue here, and remote repositories are more or less clear what they are.

Quick Intro

For a more complete introduction see the Git tutorial (the cool stuff is in the next section).

Somewhere on the KDE Project I found a really nice Cheat-Sheet for Git.

To create a new repository just issue

git init

To start tracking files (or content) issue

git add <files>

To commit the added changes (note, only the state of the files when you issued add will be recorded) issue

git commit

To push changes to a remote repository (ssh, filesystem etc.) issue

git push <remote url>

To clone a complete repository onto your local disk issue

git clone <remote url>

I like to do some general configuration for all my projects, something like this:


git config --global user.name "Your Name" 
git config --global user.email your.mail@host.com
git config --global color.ui auto
git config --global color.interactive auto

This tells Git who you are and enables some nice color-features of Git.

To get the most out of Git learn one of these commands:

git help <command>
man git-<command>

So, git help config is equivalent to man git-config, similarily, git config... is equivalent to git-config.... The latter seems to be discouraged (since Git switches to a library approach or something), but is often used in tutorials or manuals.

That may be the most used commands with Git, the more interesting ones – and the ones for which you may want to switch to Git follow:

Commands

Adding/Removing changes to/from the Index

To add all changes in a file, simply use add, some examples:

add . adds all files in the current directory and its sub-directories to the index
add <file> adds a specific file to the index
add -u adds all changed files to the index (that is, it adds alls files Git knows about and that have changed)
add --patch lets you decide whether you want to include smaller changes within a file instead of all changes in the file(s) see The Thing About Git for a larger example. This is extremely useful if you accidentally made two unrelated changes to a file and you want to keep your commits clean (you do want to keep your commits clean, don’t you?!)
add -i is an interactive version of add with a text-based GUI combining all the above capabilities

If you accidentally added some changes to the index, simply use reset to clean it, or specify the file to remove from the index, like reset <file>. reset can do more than simply cleaning the index, you can use it to restore the state of the latest commit by using reset --hard. reset also takes a parameter telling it which commit to consider – instead of using the latest HEAD, use HEAD^ for the parent of the actual HEAD, or give it a SHA1 hash… . Take a look at this post for a better treatment of git reset and its possibilities.

Checking differences between commits/index/working directory

Sometimes you need to know what exactly the differences between two commits or the working directory and the index (or some commit) is.

This is actually fairly trivial with any revision control system, and of course also with git.

diff compares your working directory with the index
diff --cached compares your index with the latest commit (or any named commit you supply)
diff <commit> compares your working directory with the named commit (use HEAD for the latest commit)
diff <commit> <commit> compares (surprise) two named commits (for example, diff HEAD HEAD^ shows the differences from the last two commits)

To get a high-level view of changed files, Git offers the status command.

status shows a summary of changed files, files added to the index and files not tracked.
status -a shows what the index would look like if you issued add -u or a commit -a (that is, all changed files get commited or added to the index).
status -u {no, normal, all} using the -u parameter, Git’s behavior concerning untracked files can be changed (don’t show untracked files, show only files and directories, or show all untracked files recursively).

Another way to get an overview of your controlled content is by using ls-files:

ls-files shows all tracked files
ls-files -t shows files with some additional information (like cached, modified etc.)
ls-files --others shows untracked files

Somehow I find this more interesting than the status output, but this may come from my Mercurial experiences where hg status would output a very similar list as ls-files -t.

Most projects have some files which should not be tracked, for example binaries built from source, or backup-files from your editor… . Using .gitignore you can tell Git to always ignore these files (unless you order it to show you also ignored files, though).

There are several ways to tell Git what to ignore. The most common one is by creating a .gitignore file in the root of your project. This file could be added to the index and would be shared. The nice thing with this file is, that you can keep one in each sub-directory overriding the configuration of the parent directory. See gitignore for more.

The second way, local to your repository, is to edit the file .git/info/exclude to your needs. On Mac OS X, Git automatically adds .DS_Store files to that exclude file.

The third way is to specify a core.excludesfile configuration (for example git config --global core.excludesfile "/absolute/path/.gitglobalexclude") and create a global ignore file (I think an absolute path is needed, but a patch enabling relative paths was submitted).

The format of the files is simple:

!file never ignore this file (use !asdf.o if you excluded *.o, but you want to track asdf.o)
/file ignore the file in the root of the project
file ignore the file anywhere in the project
*.o ignore all files with an ”.o” ending
dir/ ignore the directory “dir” but not the file “dir”

Commiting changes

So, let’s say we have a clean collection of changes in the index. The next step is to commit it.

The basic command is simple, just commit, but you can do more.

commit -a commit all changed files (adding every changed file to the index)
commit -m "commit message" supply the commit message at the command line
commit --amend edit the previous commit (that is, the new commit merges with the old one)

With Git it’s common to make branches for bigger experiments. Use a branch for example if you want to implement a new feature which requires many changes in your code-base, but it takes too long for a single commit.

So, to make a branch from the current HEAD, just issue checkout -b name. This creates a branch named “name” and switches to it. This is equivalent to branch name checkout name.

In this branch you can do whatever is needed to implement the new feature, but if necessary, you can switch back to the old branch (for maintenance or something). After you’ve completed your changes, just merge the branch with the new feature back to the original branch merge name. After that you can safely delete the “name” branch branch -d name.

The following picture shows two branches, Branch B is created during the working directory (red line) points to Branch A, we can switch between the branches with checkout and finally delete Branch B.

Of course there can be conflicts while merging the changes from the other branch into a branch which has evolved by itself. Either you resolve the conflicts all at once, or you keep your branch with the new feature more or less up to date with rebase. After each change to the master branch, you could issue a rebase master in the new branch to get the new changes of the master branch incorporated into your “name” branch.

Another very nice feature of rebase is to merge many commits into a single commit (or edit some of them). Suppose you made your changes within a new branch, now you’re ready but you’d like to have a single commit containing all changes of the branch. Just issue rebase -i master, and Git with alls commits leading from master to the current HEAD. Replace each “pick” with “squash” and Git merges all the commits into a single new one (you get to supply a new commit message). Look at git awsome-ness for a complete example.

The following diagram shows how commits are connected when a branch is made.

From the branch we can do a merge (first picture) or a rebase (second picture). Note that the merge automatically issues a commit if no conflicts arise!

Here a small illustration what you could do with a rebase command:

For smaller changes there’s also a special holding area for changes. Suppose you’re in the middle of a bigger change and you have to quickly fix something completely unrelated. Just issue stash and Git will put away all your current uncommited changes and present you with a clean working directory. Now make the quick-fix, commit and issue stash pop (equivalent to stash apply and stash drop) to continue working on your fancy new feature.

It’s also possible to stack stashes up, use stash list to see a list of stashes. If you come to the conclusion that your stashed changes deserve their own branch, issue stash branch foo to create a new branch “foo” and pop the latest stashed changes.

You never should temper with commits you already published to a remote repository (or other people)! If you find a mistake in a published commit use revert <commit> which records a special commit telling other people that something isn’t right with the specified commit.

Sharing or “backing up” your work

Sometimes you’re not the one who started the project, or you simply reinstalled or changed the computer. How do you get your repository back?

Nothing simpler than that: clone <url>. Unfortunately, clone cannot clone to a remote repository. I can’t clone my local project to my backup server – I have to either scp it to it or create a project there and pull the changes. After that I can use the standard command to publish or receive changes.

clone sets up all necessary configurations and you can simply issue pull to update your repository with remote changes or push to send your changes to the remote repository.

In a distributed environment it’s not uncommon that you might want to push or pull from different remote repositories. For that reason, Git allows you to configure shortcuts to other repositories. The main remote repository is called “origin”, and you usually push or pull the master branch. pull is a twofold operation, it first fetches the contents (using fetch) and then merges the new contents with your changes. Of course you could also use fetch, inspect the changes and merge afterwards.

So, how to you want clone the Git repository:

git clone git://git.kernel.org/pub/scm/git/git.git

After the clone, we can look at the remote sites and their respective URLs with remote -v. To look which branches are available at the remote site, issue branch -r.


me@host:git % git remote -v 
origin    git://git.kernel.org/pub/scm/git/git.git
me@host:git % git branch -r
  origin/HEAD
  origin/html
  origin/maint
  origin/man
  origin/master
  origin/next
  origin/pu
  origin/todo

Now let’s add another repository containing the Git project (note the git2 subdomain):

git remote add mirror git://git2.kernel.org/pub/scm/git/git.git

This tells Git to add the given URL with the shorthand “mirror” and to track all branches. I you’d like to track only some branches, for example only the master branch add -t master to the command.


me@host:git % git remote -v 
mirror    git://git2.kernel.org/pub/scm/git/git.git
origin    git://git.kernel.org/pub/scm/git/git.git
me@host:git % git branch -r 
  mirror/html
  mirror/maint
  mirror/man
  mirror/master
  mirror/next
  mirror/pu
  mirror/todo
  origin/HEAD
  origin/html
  origin/maint
  origin/man
  origin/master
  origin/next
  origin/pu
  origin/todo

You can either switch to a remote branch with checkout origin/todo or create a new branch based on some remote branch checkout -b myownbrach origin/todo, the latter also sets up “tracking” which means that push and pull automatically use the branch “todo” at the remote repository “origin”.

Another very cool feature of Git (and of course others) is the ability to “cherry-pick”. That is, we could take the patches (commits) interesting to us and incorporate them into our own branch. To accomplish that task, there’s cherry-pick at our disposal. Simply choose the commit you’d like to import from another branch and call cherry-pick <commit> at the branch you’d like to incorporate the patch (commit). Works like a charm.

How I use Git

Note that I just switched to Git, but since Mercurial was very similar I know most of the concepts to some degree.

I use version control for most of my files, be it configuration files in my home directory which get shared across my laptop, desktop and work machine or letters I write. Of course Git isn’t the best way to do everything, but it’s a better way than not syncing my files at all.

I have three “work-machines” which synchronise to a server accessible by all of my machines. For machine-specific files (in my home-dir for example) I made a special directory ”.machinedependent” and created sub-folders for each machine. In this directories I put the configuration files I’d like to be version controlled even if the are only interesting for one machine. Then I hard-link files and soft-link directories and I’m done. Git handles this quite nicely.

All of my coding projects are handled by Git and stored (for backup purposes and accessibility) on the mentioned server. I like the fact that my code is distributed across several machine nowhere near each other. Git-home-history has some capabilities to produce encrypted archives of your work too, but I didn’t dig into that by now.

This, of course, didn’t touch more than the bare surface of Git. Look at

for more information.

Converting from Mercurial to Git

I downloaded hg2git and converted all my repositories to Git (you should be in an empty directory and it may be an advantage to use zsh):


for i in path/to/repos/*
do
  proj=$i:t
  mkdir $proj
  cd $proj
  git init
  hg2git.sh -r path/to/repos/$proj
  git checkout default
  git checkout -b master
  git branch -d default
  cd ..
done

Missing some things from Mercurial

There are actually two features I somewhat miss in Git. First I cannot clone my local repository to a remote server without hassles (login to the remote server and create an empty project for example, or just scp the whole project over… something like that).

Second, Mercurial had very nice short-cuts. For example I could issue hg st instead of hg status, Git also has aliases, but you have to configure them on your own.

git config --global alias.ci "commit -a" adds an alias “ci” to commit all updated files (instead of adding each changed file individually to the index).

Which one is the right one?

Well the question of “which version control software should I use” comes up sooner or later (for some people so late, they won’t even notice it). I cannot tell you which one is right, the choice between centralized source code management and a distributed one is the first step and gives different possibilities (distributed is the way to go, IMHO).

Anyway, you’ve got plenty to choose from.

For me only Darcs, Mercurial and (only recently) Git were interesting enough to dig deeper. I dumped Darcs because installing it was not so easy, I’m not root on all the machines I have access to and to compile a Haskell program into my home-dir was too much work for me. And of course I heard about some nasty bugs in it… .

Mercurial was awesome (take a look at the Mercurial TechTalk), but Git won me over for absolutely no reason (mabye because it was envisioned by Linus Torvalds, maybe because I like C way more than Python).

I think most of the time we are not the ones to choose the version control software, anyway.

—
Sometimes “pi = 3.14” is (a) infinitely faster than the “correct” answer and (b) the difference between the “correct” and the “wrong” answer is meaningless. And this is why I get upset when somebody dismisses performance issues based on “correctness”. The thing is, some specious value of “correctness” is often irrelevant because it doesn’t matter. While performance almost always matters. And I absolutely detest the fact that people so often dismiss performance concerns so readily. —Linus Torvalds

Version Control Status in the Command Line Prompt

Michael — 2008-08-07T19:42:12+00:00

I sometimes forget to push or pull changes to or from a remote repository. To remedy the problem I wrote myself a little script to show me the status on the prompt.

The script is written for zsh and Mercurial, but can be easily adapted to other version control systems (I guess).

Everytime you change a directory the scripts looks up if there are uncommited changes, remote changes to be pulled or local patchsets to be pushed.

The prompt of a directory within a project with uncommited changes would look something like this:

  user@C2D:~/root-dir/subdir %            root-dir: U

Possible states:

U: Uncommited changes (use hg commit)
I: Incoming patchsets (use hg pull)
O: Outgoing patchsets (use hg push)

I have configured Mercurial to automatically update my repository after a pull, but some people might wish to add another status for “pending updates” or something like that.

The script uses “hg incoming -q” or “hg outgoing -q” to check for sets to be pulled or pushed, so you should configure the default paths in your .hg/hgrc file for pulling or pushing like this:

  [paths]
  default-push = ssh://yoursever.tld/Projects/PlanToTakeOverTheWorld/
  default = ssh://yourserver.tld/Projects/PlanToTakeOverTheWorld/

Unfortunately there are a few problems:

Accessing remote repositories over some network is costly. If you have a slow network connection you might want to temporarily switch off the remote status commands by using “export VCSNOREMOTE=true” please note that “export VCSNOREMOTE=false” has the same effect, in fact any string would switch off the remote commands. To enable them again use “export VCSNOREMOTE=”.

The second problem might be that you have to enter your credentials to access remote servers quite frequently. Use some sort of ssh-agent or something similar to avoid typing in your password all the time.

The third thing to remember is that updating the status with every prompt is useless, so I opted for updating after changing a directory (also if you type cd .) but at least after 60 seconds.

Installation: save anywhere on your disk and source it in your .zshrc file.

Download: vcs_status_prompt

— Life is painting a picture, not doing a sum. —Oliver Wendell Holmes

Emacs Basics

Michael — 2008-02-24T21:41:50+00:00

It’s been a while since I wrote my Vim Introduction and Tutorial (exactly one year). A lot happened between now and then, I chose to get a better feeling about Emacs for example.

The reasons aren’t easily explained; The most prominent reason is the awesome AucTex-mode since I’m working heavily with LaTeX lately.

Anyways, learning Vim and Emacs is better than learning only one of them :-).

Vim started in 1991 and is based on Bill Joy’s Vi written in 1976/77. Emacs started as set of macros for the TECO editor 1976 and was rewritten in 1984 (GNU Emacs). So both editors are based on concepts invented more than 30 years ago and both editors are very unlikely to change any time soon.

I still like Vim’s way of doing things, for example the basic commands are very easy and very fast to reach.

At first I wanted to compile something similar to my Vim Tutorial, but I ended up publishing just a few notes and more or less a cheat-sheet for Emacs. If you like it more official look at the Emacs Guided Tour and its very fine documentation.

The best way to get up and running with Emacs is the included tutorial (start by typing C-h t).

I’m using Aquamacs on my Mac and GNU Emacs on Linux. Both use GNU Emacs 22 at its core, but especially Aquamacs provides some extensions to make it (almost) seamless on OS X.

Basic Concepts

You can’t learn Emacs and lean back, every mode Emacs provides offers its own key bindings and has its own specialities you have to know.

So, it doesn’t matter if you use Vim for day-to-day tasks and Emacs for special purpose text editing (like typing LaTeX, or interacting with some math software). Only the most basic commands are the same for all modes.

Two things to remember

You can cancel any operation either by pressing C-g or three times <Esc>.

To exit Emacs, type C-x C-c.

Major/Minor Modes

A Major Mode provides Key-Bindings, Indentation, Display, Syntax etc. There can be only one Major Mode active at a given time. How would you mix two different syntax-highlighting schemes?

A Minor Mode provides additional bindings and some other behavior. For example spell-correction, automatic line-breaks etc.

Emacs Lisp functions and key bindings

You can access all functions of Emacs with: M-x (that’s <Alt>-x or <Meta>-x or <Esc> x or <Option>-x depending on your setup, preferences and terminology).

C-x means pressing Control and x at the same time, <Esc> x means pressing Escape and x after another.

There is no way to bind all functions to some key-combo and it might be impossible to remember the bindings. So, if you want to replace some string with some other string type “M-x repl<Tab><Tab>” and Emacs gives you possible completions, “replace-string” sounds good.

Emacs’ Architecture

Emacs has a small Lisp core written in C. But the “meat” of Emacs is the expansion-library written in Emacs Lisp. Essentially everything you do and write to extend and configure Emacs is written in Emacs Lisp.

The Bookworm has an introduction to the Emacs introductions.

Undo/Redo

To undo almost every action you did, you can use either undo as command or one of the following key-bindings: C-/, C-_ or C-x u

Undo → C-/

To redo, you have to switch the “undo-direction” with C-g, after that you can use the usual undo-key of your choice.

Switch undo-direction → C-g

Basic Movement

Character movement
- Up → C-p
- Down → C-n
- Left → C-f
- Right → C-b

Page movement
- Page Down → C-v
- Page Up → M-v

Line movement
- Start of Line → C-a
- End of Line → C-e

Sentence movement
- Start of Sentence → M-a
- End of Sentence → M-e

Paragraph movement
- Start of Paragraph → M-{
- End of Paragraph → M-}

Other
- Start of File/Buffer → M-<
- End of File/Buffer → M->

Search and Search & Replace

Searching/Replacing

Incremental Search forward → C-s
Incremental Search backward → C-r
Incremental Search forward with regular expressions → search-forward-regexp
Incremental Search backward with regular expressions → search-backward-regexp
Replace some string → replace-string
Replace using regular expressions → replace-regexp

Text Surgery

Deleting/Killing
- Delete char forward → C-d
- Delete char backward → <Backspace>
- Kill word forward → M-d
- Kill word backward → M-<Backspace
- Kill to end of line → C-k
- Kill to end of sentence → M-k

Transpose (switch elements)
- Transpose chars → C-t
- Transpose words → M-t
- Transpose lines → C-x C-t

Case-operations
- Capitalize word → M-c
- Uppercase word → M-u
- Lowercase word → M-l
Text-completion → M-/ (repeat to cycle through possible completions)

New-Lines
- New line → <Return> or C-m (insert a CR before the cursor)
- Open new Line → C-o (insert a CR after the cursor)
- New line and indent → C-j

I like to format my text so that no lines are longer than 78 characters. Either I use the minor mode auto-fill-mode or I use M-q to re-justify (fill) the current paragraph (necessary if you edit a paragraph after it was auto-filled).

Cut/Copy/Paste

Basic commands

Maybe you noticed that I changed the terminology of deleting a character to killing a word. The difference is, that “killed” text is put into the Yank-Ring. From this ring, you can get your text back (pasting, yank). The size of your yank-ring depends on your setup.

Yank → C-y, use M-y to cycle through the various text-parts.

If you’ve marked some text, use C-w to cut the text, or M-w to copy the text into the yank-ring:

Cut → C-w
Copy → M-w

Regions

To mark some text, use the mouse, or set a mark with C-<Space> and move the cursor to the end point of the region. The text between the mark and the current cursor-position (also called point) is the region.

Set mark → C-<Space>

You can jump between point and mark using C-x C-x, this switches the mark and the point, so that your region remains intact.

Switch point and mark → C-x C-x

If you’ve enabled the transient-mark-mode Emacs highlights the region when you create it. As soon as you move the cursor, the region is still there, but the highlight is gone (verify with C-x C-x for example).

Rectangles

The mark and point also specify a rectangle over your text (upper right and lower left edge).

Kill the rectangle → C-x r k
Yank the rectangle → C-x r y
Insert space into the rectangle → C-x r o
Replace text of the rectangle → C-x r t
Clear rectangle (replace with space) → C-x r c

There is an interesting minor mode for additional rectangle-functionality called CUA.

Frames, Windows and Buffers

Emacs has the concept of Frames, Windows and Buffers. A Frame is a Window in the common sense, a Window is a part of the frame. For example if you split the screen of your Emacs frame, you got two windows within one frame.

The last concept is the Buffer, it’s simply a real or virtual file (virtual == not saved to disk).

Buffer commands:

Load a file (buffer) → C-x C-f
Show loaded buffers → C-x C-b
Switch to another buffer → C-x b
Close a buffer → C-x k
Save a buffer → C-x C-s

Frame commands:

Open new frame with file → C-x 5 f
Close this frame → C-x 5 0
Switch to other frame → C-x 5 o

Window commands:

Split window vertically → C-x 2
Split window horizontally → C-x 3
Close all other windows → C-x 1
Close this window → C-x 0

The Minibuffer (the last line of your Emacs screen) is a buffer itself, but has some specialities I wont go into now.

Getting Help from Emacs

You could fill some very big books with Emacs-specific information, but Emacs calls itself a “self-documenting” editor. This means that the documentation reflects exactly how your Emacs is configured. If you change some key-bindings, the documentation reflects this changes and displays the right key-combo in the help files.

Some hints on how to get going using the help-system:

Show information on the current mode(s) → C-h m
Describe the command bound to the given key → C-h k
Search for a function → C-h a
Describe all key-bindings of the current mode(s) → C-h b

Further Information

Steve Yegge has some intersting articles on Emacs:

IBM has some nice Emacs tutorials too.

The EmacsWiki is another excellent source for Emacs related howtos, downloads and – to some extend – discussions.

Wish

I’d wish that Vim and Emacs would merge. That I could take advantage of Emacs’ awesome tool-support and Vim’s awesome way of editing small junks of text. I know there is Viper-mode and Vimpulse, but both are not exactly what I want. Viper-mode is plain Vi, and Vimpulse is early in development.

Is there a way to integrate a Vim window into Emacs without emulating and reimplementing the features?

For the time being, I’ll have to switch between the environments. Vim as my default-editor and Emacs for special purpose editing.

—
Emacs outshines all other editing software in approximately the same way that the noonday sun does the stars. It is not just bigger and brighter; it simply makes everything else vanish. —Neal Stephenson

Free and Open Source Math Programs

Michael — 2008-02-24T14:36:21+00:00

In this post I’ll go through some of the most prominent math programs available with source code.

This is by no means “original” work, I just collected the headlines and links to various mathematical software projects out there. This started with an offer of my University (Mathematica for 13 Euro), but I don’t want to invest time in a tool I won’t have (free, or almost free) access to for the rest of my life.

The programs differ in their respective foci, the main areas are Computer Algebra System, Numerical Analysis, Statistics and general purpose desktop calculators.

My current choice is SAGE and R for my math and statistics needs. For simple calculations I usually use the calculator included with the OS or some interactive programming-shell (ghci or irb).

Hybrid

SAGE

SAGE is the new star on the open-source math sky. Essentially it is a CAS written in Python, but it provides interfaces to many math programs available. This includes commercial programs like Mathematica and Maple as well as free programs like Maxima, Octave or Singular.

From the homepage:

Each interface requires that the corresponding software is installed on your computer. SAGE includes GAP, PARI, Singular, and Maxima, but does not include Octave (very easy to install), MAGMA (non-free), Maple (non-free), or Mathematica (non-free).

SAGE runs as service on your server or desktop and you use your browser to interact with it (you can also use the command-line interface, though). It is very well documented and gives you access to the source code of the algorithms right within your browser.

Use SAGE for studying a huge range of mathematics, including algebra, calculus, elementary to very advanced number theory, cryptography, numerical computation, commutative algebra, group theory, combinatorics, graph theory, and exact linear algebra.

The goal of the SAGE project, which started 2005, is to provide the best mathematical software in the world.

SAGE is pretty large (on my Mac it uses 722MB right after the installation), but you can find even more (optional packages etc) at their homepage.

It’s also possible to interact with it via Texmacs or Emacs (again, available from their download page).

SAGE is available for Windows, Linux and Mac OS X.

Computer Algebra System

Maxima

Maxima is a system for the manipulation of symbolic and numerical expressions, including differentiation, integration, Taylor series, Laplace transforms, ordinary differential equations, systems of linear equations, polynomials, and sets, lists, vectors, matrices, and tensors. Maxima yields high precision numeric results by using exact fractions, arbitrary precision integers, and arbitrarily precision floating point numbers. Maxima can plot functions and data in two and three dimensions.

Maxima is very mature and stood the test of time. It started in 1982 and is “free” since 1998.

Maxima is included in SAGE.

Axiom

Axiom is a general purpose Computer Algebra system. It is useful for doing mathematics by computer and for research and development of mathematical algorithms. It defines a strongly typed, mathematically correct type hierarchy. It has a programming language and a built-in compiler.

Yacas

YACAS is an easy to use, general purpose Computer Algebra System, a program for symbolic manipulation of mathematical expressions. It uses its own programming language designed for symbolic as well as arbitrary-precision numerical computations. The system has a library of scripts that implement many of the symbolic algebra operations; new algorithms can be easily added to the library. YACAS comes with extensive documentation (hundreds of pages) covering the scripting language, the functionality that is already implemented in the system, and the algorithms we used.

Homepage Downlaods, Screenshots, Tutorial, Manual and Demo are available there.

PARI/GP

PARI/GP is a widely used computer algebra system designed for fast computations in number theory (factorizations, algebraic number theory, elliptic curves…), but also contains a large number of other useful functions to compute with mathematical entities such as matrices, polynomials, power series, algebraic numbers etc., and a lot of transcendental functions. PARI is also available as a C library to allow for faster computations.

PARI/GP is included in SAGE.

Singular

SINGULAR is a Computer Algebra System for polynomial computations with special emphasis on the needs of commutative algebra, algebraic geometry, and singularity theory.

Singular is included in SAGE.

GAP

GAP is a system for computational discrete algebra, with particular emphasis on Computational Group Theory. GAP provides a programming language, a library of thousands of functions implementing algebraic algorithms written in the GAP language as well as large data libraries of algebraic objects.

GAP is included in SAGE.

Macaulay2

Macaulay 2 is a software system devoted to supporting research in algebraic geometry and commutative algebra.

Numerical Analysis/Computation

Octave

GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Euler

Euler is a powerful interactive numerical laboratory. It quickly evaluates numerical functions, visualizes results, and allows to test and program numerical algorithms. The system can handle real, complex and interval numbers, vectors and matrices, and additionally a long data type for exact computation. Most routines are written in the high level Euler language and can be modified or extended by the user.

Scilab

Scilab is a scientific software package for numerical computations providing a powerful open computing environment for engineering and scientific applications.

Statistics

R

If you want to do some statistics there is no way around “R”. At least not in the free and open-source software world.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible.

I think R is a tool everyone should be familiar with, at least with the most basic operations. Statistics is such an important every-day tool you can immediately put to work as soon as you know something, anything, about it.

Homepage
Documentation
Tutorial
Screenshots
Books (currently 53 books!)

Command-Line/Desktop Calculator

Sometimes you just want to calculate very simple things, for this task a full fledged system isn’t really appropriate.

bc

bc is an arbitrary precision numeric processing language. Syntax is similar to C, but differs in many substantial areas. It supports interactive execution of statements. bc is a utility included in the POSIX P1003.2/D11 draft standard.

bc is usually installed on any Linux/Unix system or available via its package management.

“Other” command-line/desktop calculators

Essentially you can use any interactive shell, like Python’s python or Ruby’s irb as well as Haskell’s ghci or hugs.

GUI Calculators

Despite the included Desktop Calculators there are some very fine products out there, here some pearls I could find:

Mac OS X
- Magic Number Machine
Linux
- Qualculate!
- Calcoo
Windows
- Calcoo for Windows
- Any suggestions?

If nothing else, there is always Google Calc.

—
Not being able to check the code of a computer-based calculation is like not publishing proofs for a mathematical theorem, it’s ludicrous. —William Stein, author of SAGE

Addendum to "Time Machine for every Unix out there"

Michael — 2007-12-06T10:55:36+00:00

My article about using rsync to mimic the behavior of Apple’s Time Machine generated a lot of traffic, and more important, a lot of feedback.

In this article I’ll summarize and try to clarify a few things.

First of all: I didn’t invent any of the things I’ve written in the article! All I did was to write it up in a manner I could understand it. The rsync developer(s) included the feature(s) for a specific purpose which might be exactly what we do now. But this was obvious, I hope.

Time Machine and Performance

Ars Technica has a very deep review of the new features of Leopard, including Time Machine.

Interestingly they do very much the same as I did in my script – each Backup gets its own folder and there is a link to the latest.

Some readers complained that Time Machine looks only at the changed files, where my script has to look at each file on the disk. For me, the difference is negligible. Apple introduced hard-links to directories, which saves a lot of time and a bit of space. Unfortunately this feature isn’t availabel on plain Unix, but it’s a neat trick!

Restoring the Files

Since we make full backups each time, you have your whole and complete directory structure in that Backup-directories.

Restoring a file is merely a simple copy command.

Or, if you want to restore the whole filesystem, use:

rsync -avz --delete host:/Backup/scurrent/ /target/

But keep in mind that —delete removes any file which is not in the backup!

Removing old Backups

Another huge advantage of full backups is: you can delete any backup you don’t need anymore.

Of course each backup-directory takes space, if only for the hard-links and directory structure. Cleaning up the backups is a breeze: just delete the directories you don’t need anymore. For example you could delete all directories older than 2 months or keep one week full, then only the monday backup. There are plenty of possibilities.

Other Tools

Many people suggested many interesting tools for doing backups. My argument against using such tools is that rsync is everywhere!

You find rsync in the smallest Linux distributions (like DSL), on PDAs like the Zaurus, on Windows using Cygwin and also on Nokia’s Internet Tablets.

One backup strategy for all your devices!

Space Requirements for the Links

Nothing in life is free, but the space used by the hardlinks for the backup is marginal. It is also dependent on the filesystem you use, ReiserFS seems to do a good job in packing more links in one cluster. But hey, these things are Backups you throw away space anyways!

Colon

There is one particular problem using a colon in the script: some operating systems don’t like them.

Here an updated version of the script without colons:


 #!/bin/sh

 date=`date "+%Y-%m-%dT%H_%M_%S"`
 rsync -azPE --link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date \
   && ssh $HOST "rm PATHTOBACKUP/current \
   && ln -s back-$date PATHTOBACKUP/current"

Another slight change in the script is, that all commands are connected with &&. This means the next command is only done when the previous command returned no error.

Of course, if anything messes up the “current” link, the script just makes another full backup without the links to the previous version.

Extended Attributes on OS X

One user suggested the addition of the -E parameter on Mac OS X. If you use the parameter on non-OS X machines it’s more or less redundant (-a contains executability), but on OS X some additional information is backed up.

Just use it.

Windows

Some users reported success with this strategy on Windows. I’m happy that we now have a unified backup strategy for all major operating systems without paying a single cent.

You have to use Cygwin to install rsync (there are binaries without the whole Cygwin on the net, here for example).

Measuring the Space Requirements

To lookup how much space your particular backups need, simply issue du -shc back-*. This prints each backup directory with its real memory usage:

  me@myserver:Backups % du -shc back-*                                                                                                               
  406M  back-2007-11-12T12:00:00
  6.8M  back-2007-11-13T20:28:56
  189M  back-2007-11-13T23:42:54
  2.6G  back-2007-11-14T08:15:43
  1.8M  back-2007-11-15T20:07:07
  7.9M  back-2007-11-18T23:57:09
  11M back-2007-11-19T21:53:34
  2.6M  back-2007-11-21T18:57:40
  2.1M  back-2007-11-21T19:11:02
  11M back-2007-11-22T01:19:38
  1.9M  back-2007-11-22T11:05:17
  19M back-2007-11-22T18:37:56
  7.4M  back-2007-11-22T22:45:04
  8.5M  back-2007-11-22T22:51:29
  9.2M  back-2007-11-24T12:45:12
  4.2M  back-2007-11-24T17:30:38
  11M back-2007-11-25T00:41:19
  1.9M  back-2007-11-27T20:41:40
  317M  back-2007-12-03T10:19:07
  12M back-2007-12-05T11:51:19
  3.6G  total
  me@myserver:Backups %

As you can see, the smallest backup (obviously without any changed file) requires 1.8MB.

—
Backups are typically that last line of defense against data loss, and consequently the least granular and the least convenient to use. —Yosemite Technologies

Time Machine for every Unix out there

Michael — 2007-11-13T20:55:57+00:00

rsync is one of the tools that have gradually infiltrated my day to day tool-box (aside Vim and Zsh).

Using rsync it’s very easy to mimic Mac OS X new feature called Time Machine. In this article I’ll show how to do it, but there is still a nice GUI missing – for those who like it shiny.

What Time Machine does

Time Machine makes a snapshot of your files every hour. The files are usually stored on a external hard drive connected to your Mac via USB or Firewire. Earlier Leopard versions (ADC preview versions) had the ability to make the backups to a remote drive (I’ve heard).

So if you lose a file, or did a devastating change to one of your files, simply go back in time until you find your file or a version that’s not corrupted.

Incrementally backing up all files every hour so that you can access them in reversed chronological order isn’t that hard with standard Unix utilities like rsync. The only missing thing is a nice GUI for which Apple is known to be quite good at.

Making full backups in no time every hour

You can use this method to make a backup every hour or every ten minutes if you like. There are many many features you can tune or configure to your own taste – excluding files that are larger than 1GB for example.

So, here the command to make the backup:


 rsync -aP --link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP

Lets go through the parameters step by step.

-a means Archive and includes a bunch of parameters to recurse directories, copy symlinks as symlinks, preserve permissions, preserve modification times, preserve group, preserve owner, and preserve device files. You usually want that option for all your backups.
-P allows rsync to continue interrupted transfers and show a progress status for each file. This isn’t really necessary but I like it.
--link-dest this is a neat way to make full backups of your computers without losing much space. rsync links unchanged files to the previous backup (using hard-links, see below if you don’t know hard-links) and only claims space for changed files. This only works if you have a backup at hand, otherwise you have to make at least one backup beforehand.
PATHTO/$PREVIOUSBACKUP is the path to the previous backup for linking. Note: if you delete this directory, no other backup is harmed because rsync uses hard-links and the operating system (or filesystem) takes care of releasing space if no link points to that region anymore.
$SOURCE is the directory you’d like to backup.
$CURRENTBACKUP is the directory to which you’d like to make the backup. This should be a non-existing directory.

As said earlier, rsync has many many features. To exclude files over a certain size for example, use the option --max-size (unfortunately this is not available on the rsync version shipped with Mac OS X Leopard). The man page or the documentation can give you plenty of ideas in this direction.

So much for the theory of the most important command for our purpose. Here a simple script that makes an incremental backup every time you call it:


 #!/bin/sh

 date=`date "+%Y-%m-%dT%H:%M:%S"`
 rsync -aP --link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$date
 rm $HOME/Backups/current
 ln -s back-$date $HOME/Backups/current

The script creates a file called “back” appended by the current date and time, for example back-2007-11-13T22:03:32 which contains the full backup. Then there is a symbolic link called “current” which points to the most recent directory. This directory-link is used for the --link-dest parameter.

You should look at the --exclude parameter (or better, --exclude-from= parameter) and learn how to exclude certain files or directories from the backup (you shouldn’t backup your backup for example).

The script above only works on the local machine because making links on a remote machine needs some extra work. But not much:


 #!/bin/sh

 date=`date "+%Y-%m-%dT%H:%M:%S"`
 rsync -azP --link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date
 ssh $HOST "rm PATHTOBACKUP/current && ln -s back-$date PATHTOBACKUP/current"

To get that working you either use a public/private key authentication scheme or something else to avoid typing in your password. Another possibility is, of course, to mount the remote file-system on the local computer using the above script.

On my setup the script takes about 6 seconds to synchronize 46968 files and 29GB – this takes 20MB for the file structure (with no actual files to transfer of course). But afterwards, I have a complete backup of my system in a new directory.

On a much bigger setup (1.2 million files and 50GB of data) the backup takes about 30 minutes and takes about 3GB of space (just for links!), so it isn’t exactly free, but very convenient.

The space needed for the backup is determined by the shape of your directory structure. On the larger setup I have lots of Maildirs and a very deep directory structure so it takes much more space than my home-directory backup above. 3GB is quite a lot, but 20MB doesn’t hurt.

Advanced `rsync` parameters

Additional to the parameters described above, I usually employ a combination of these parameters in my backup:

--delete and --delete-excluded this tells rsync to remove files from my backups either if they are gone on my local machine, or if I decided to exclude them from my backup.
--exclude-from=FILE the file specified here is a simple list of directories of files (one per line) which should not be backed up. My Trash folder oder some .cache folders are candidates for this file.
-P is used to give more information on how far the backup is, and how many files are to be backed up. Additional it could resume an interrupted transfer (which doesn’t apply here because we create a blank backup each time we call the script).
-x this one is important because it prohibits rsync to go beyond the local filesystem. For example if you backup you Linux-root partition, you should not include the /proc directory because rsync will get stuck in it. -x excludes all mounted filesystems from the backup which is probably what you want in most cases.

Hard-Links

Each file in a directory is a link to the actual data on your hard-disk. The file-system keeps track of how many links to a area point, and only if the last link is deleted, the whole area gets deleted (in contrast to soft-links, these are pointers to the file-name, not the contents).

Here an illustration of two backups with three files each. File1 and File2 are the same in both backups, only File3 changed between Backup1 and Backup2. So in Backup2, File3 (changed) has to point to a different area than File3 in Backup1.

BTW, there is a nice project for Linux out there which provides the same functionality as Time Machine including a nice GUI which is also based on rsync and the procedure presented here.

The End

Credit: The initial idea for this approach came from Mike Rubel – rsync snapshots.

Also interesting if you have to cope with Windows: Optimal remote backups with rsync over Samba.

There are quite a few approaches out there which more or less do the same, but rsync is available on virtually every Unix out there (even the DSL with its 50MB footprint includes rsync). So using other tools might be more convenient, but I’ll stick with the omnipresent rsync.

rsync offers the possibility to store only the differences to the previous backup (using --compare-dest which should point to a full-backup instead of --link-dest). It then doesn’t make links to the unchanged files, it just leaves them out. This way you get an incremental backup without the “directory-overhead” of the --link-dest approach. But you have to be extremely cautious which one of older backups you delete because the newer backups just don’t contain some of these files (think of full-backups as checkpoints)! Using the --link-dest you can delete all backups but the last and you still got all the files, so I’m happy to pay 20MB per backup for this safety.

—
The universe tends toward maximum irony. Don’t push it. —JWZ

My Mac OS X Setup

Michael — 2007-11-04T18:04:03+00:00

Installing Mac OS X on a Mac is a breeze, yet I usually do two things after the installation to improve stability.

Hibernation/Safe Sleep

The newer Macs come with a feature called “Hibernation” or Safe Sleep. Every Mac owner knows that you can put your Mac to sleep and wake it any time you want. In contrast to some (almost?) all other major operating systems this just works. The sleep mode still powers the RAM, so you still need some power. Now imagine you put your MacBook (Pro) to sleep and you run out of battery. This is where Hibernation comes to play.

Hibernation writes the contents of your RAM to a file on disk, an in case of total power loss, it restores the contents of your memory from that file → nothing is lost.

Unfortunately writing several gigabytes to disk takes some time, and I experienced hard crashes when I interrupt this process, for example by opening the Laptop shortly after I’ve closed it.

My solution – giving up the fallback option – is to disable Hibernation at all:

sudo pmset -a hibernatemode 0

The pmset utility offers some more options (Hibernation only on battery etc…).

Wake on Bluetooth

The default setting for Bluetooth and Power Management is to allow Bluetooth devices to wake the computer. For example you can press a button of your Bluetooth Keyboard or Mouse to activate your computer.

In my opinion this feature isn’t that hot on a Laptop. I once put my Mac into my bag and accidently pushed a button of my Keyboard. The computer came to live, but had no chance at all to cool itself → it ran pretty hot when I took it out of my bag… .

Since this accident I always disable “Wake on Bluetooth” because I wake my Mac by opening the lid, not by pressing some button.

These two options increased the stability of my machine quite significantly, and I don’t miss them in any way, although “Hibernation” would be nice – just in case… .

—
Boom. —Steve Jobs

Apple, Apple Care, MacBook Pro, and Mac OS X Leopard

Michael — 2007-11-01T15:15:11+00:00

About one year ago I bought a second generation MacBook Pro. At first everything was fine and I didn’t encounter any weird sounds or screen flickers known from previous models. Unfortunately after a few weeks, it started to make some weird sound sometimes when I open it (or switch it on).

The noise vanished a few seconds after starting the computer but it still bothered me, so I brought it to an Apple Service center. Three repairs later, the noise was still there (the technician also heard it – not only me), so I called Apple Care.

At first they said “it’s the usual operating sound of these machines”. I was tempted to trust them, but I started a small research on the Internet. It turned out, that there is indeed a sound (issued by the CPU) which is considered normal, but my particular one wasn’t included here.

So I called Apple Care again presenting my new findings and giving a very detailed description of my problem. Due to the unsuccessful repairs they offered me to replace the whole thing.

After a week or so I received a brand new MacBook Pro, not the second generation model, but a third generation MacBook Pro including Leopard.

Leopard

The new Mac OS X is a fine step forward. At first it isn’t very different from Tiger, only some minor user interface tweaks here and there.

At a closer look they have improved many details and introduced some very new features (Time Machine and Quick Look for example). I won’t go into the details here, other sites are much better suited for this. But Time Machine is very nice for backup purposes and offers some kind of versioning file system, Quick Look was one of these features I thought “nice, but not for me” until I got a PowerPoint presentation. I don’t have PowerPoint installed on my system, but Quick Look could open it without a hassle – nice!

Oh, yes – Leopard seems to be more responsive to me (faster if you like), especially Spotlight got a real performance boost.

Third Generation MacBook Pro

The new MBP has a LED backlit display which helps to increase the battery life time of the notebook (and lower the environmental hazard) together with Intel’s new Santa Rosa chipset which can independently regulate the CPU cores. Apple claims that these models can run up to 6 hours on battery. Well usually these numbers aren’t that realistic (you have to shut off Bluetooth, WLAN and set the brightness to the lowest level to get near them), but two movies with medium brightness and sound via the internal speakers are absolutely no problem (about 4-4.5 hours). That’s pretty good for a 15.4 inch widescreen laptop!

At first I was annoyed by the brightness adjustment feature, it turns the brightness down if the surrounding light is dimmed, and up as soon as the environment gets brighter. It’s only annoying when you see the change, but the eye is more relaxed and working on the laptop is more comfortable (turning the feature off is no problem either).

Boot Camp

Since I like to play a game every now and then, I thought I could try installing Windows using Boot Camp.

The installation was quite easy. I installed Windows Vista Business because my student edition of XP didn’t work (they are altered editions of the standard XP editions). The Boot Camp drivers allow nearly the same features as Mac OS X has (brightness adjustment, volume, sleep when lid is closed etc.). Vista also runs surprisingly fast (I disabled certain services like SuperFetch and the Indexer), but it uses very much space, more than I am willing to spend on a 20 GB partition… maybe I’ll give XP another try sometime.

Conclusion

Altogether I am very pleased with the outcome of my noise problems on my notebook. Apple was kind enough to replace the whole machine eleven months after my purchase and the new machine simply rocks.

Almost all hardware manufacturers have problems – whether it’s IBM/Lenovo, Dell or Apple. The difference in – my opinion – lies in the service. I’m not saying that other companies wouldn’t replace a faulty product, but replacing it after more than 90% of the warranty have passed is exceptional, isn’t it?

Well, not being able to repair an issue on three repairs is also exceptional (but can I blame Apple for independent service centers?).

The only other issue with expensive hardware was with my IBM T86 TFT monitor. I bought it on an internet auction (private seller, no bill or guarantee card etc). I had some problems with it, so I called IBM if there is a possibility to repair it. It turned out that the monitor had four years of warranty left and I was eligible for their rapid-replacement service (a carrier brings me a new one, and I send back the old one). Again, service matters, but you pay for it – the MBP costs about 2000 Euro, you can get 15,4 inch laptops as low as 500 Euro too, do you think these companies would ever replace the whole thing after 11 months because of some sound which doesn’t affect the usability of the device at all?

—
Have you tried turning it off and on again?—IT Crowd answering machine