IMHO

Emacs Basics

Michael — 2008-02-24T21:41:50+00:00

It’s been a while since I wrote my Vim Introduction and Tutorial (exactly one year). A lot happened between now and then, I chose to get a better feeling about Emacs for example.

The reasons aren’t easily explained; The most prominent reason is the awesome AucTex-mode since I’m working heavily with LaTeX lately.

Anyways, learning Vim and Emacs is better than learning only one of them :-).

Vim started in 1991 and is based on Bill Joy’s Vi written in 1976/77. Emacs started as set of macros for the TECO editor 1976 and was rewritten in 1984 (GNU Emacs). So both editors are based on concepts invented more than 30 years ago and both editors are very unlikely to change any time soon.

I still like Vim’s way of doing things, for example the basic commands are very easy and very fast to reach.

At first I wanted to compile something similar to my Vim Tutorial, but I ended up publishing just a few notes and more or less a cheat-sheet for Emacs. If you like it more official look at the Emacs Guided Tour and its very fine documentation.

The best way to get up and running with Emacs is the included tutorial (start by typing C-h t).

I’m using Aquamacs on my Mac and GNU Emacs on Linux. Both use GNU Emacs 22 at its core, but especially Aquamacs provides some extensions to make it (almost) seamless on OS X.

Basic Concepts

You can’t learn Emacs and lean back, every mode Emacs provides offers its own key bindings and has its own specialities you have to know.

So, it doesn’t matter if you use Vim for day-to-day tasks and Emacs for special purpose text editing (like typing LaTeX, or interacting with some math software). Only the most basic commands are the same for all modes.

Two things to remember

You can cancel any operation either by pressing C-g or three times <Esc>.

To exit Emacs, type C-x C-c.

Major/Minor Modes

A Major Mode provides Key-Bindings, Indentation, Display, Syntax etc. There can be only one Major Mode active at a given time. How would you mix two different syntax-highlighting schemes?

A Minor Mode provides additional bindings and some other behavior. For example spell-correction, automatic line-breaks etc.

Emacs Lisp functions and key bindings

You can access all functions of Emacs with: M-x (that’s <Alt>-x or <Meta>-x or <Esc> x or <Option>-x depending on your setup, preferences and terminology).

C-x means pressing Control and x at the same time, <Esc> x means pressing Escape and x after another.

There is no way to bind all functions to some key-combo and it might be impossible to remember the bindings. So, if you want to replace some string with some other string type “M-x repl<Tab><Tab>” and Emacs gives you possible completions, “replace-string” sounds good.

Emacs’ Architecture

Emacs has a small Lisp core written in C. But the “meat” of Emacs is the expansion-library written in Emacs Lisp. Essentially everything you do and write to extend and configure Emacs is written in Emacs Lisp.

The Bookworm has an introduction to the Emacs introductions.

Undo/Redo

To undo almost every action you did, you can use either undo as command or one of the following key-bindings: C-/, C-_ or C-x u

Undo -> C-/

To redo, you have to switch the “undo-direction” with C-g, after that you can use the usual undo-key of your choice.

Switch undo-direction -> C-g

Basic Movement

Character movement
- Up -> C-p
- Down -> C-n
- Left -> C-f
- Right -> C-b

Page movement
- Page Down -> C-v
- Page Up -> M-v

Line movement
- Start of Line -> C-a
- End of Line -> C-e

Sentence movement
- Start of Sentence -> M-a
- End of Sentence -> M-e

Paragraph movement
- Start of Paragraph -> M-{
- End of Paragraph -> M-}

Other
- Start of File/Buffer -> M-<
- End of File/Buffer -> M->

Search and Search & Replace

Searching/Replacing

Incremental Search forward -> C-s
Incremental Search backward -> C-r
Incremental Search forward with regular expressions -> search-forward-regexp
Incremental Search backward with regular expressions -> search-backward-regexp
Replace some string -> replace-string
Replace using regular expressions -> replace-regexp

Text Surgery

Deleting/Killing
- Delete char forward -> C-d
- Delete char backward -> <Backspace>
- Kill word forward -> M-d
- Kill word backward -> M-<Backspace
- Kill to end of line -> C-k
- Kill to end of sentence -> M-k

Transpose (switch elements)
- Transpose chars -> C-t
- Transpose words -> M-t
- Transpose lines -> C-x C-t

Case-operations
- Capitalize word -> M-c
- Uppercase word -> M-u
- Lowercase word -> M-l
Text-completion -> M-/ (repeat to cycle through possible completions)

New-Lines
- New line -> <Return> or C-m (insert a CR before the cursor)
- Open new Line -> C-o (insert a CR after the cursor)
- New line and indent -> C-j

I like to format my text so that no lines are longer than 78 characters. Either I use the minor mode auto-fill-mode or I use M-q to re-justify (fill) the current paragraph (necessary if you edit a paragraph after it was auto-filled).

Cut/Copy/Paste

Basic commands

Maybe you noticed that I changed the terminology of deleting a character to killing a word. The difference is, that “killed” text is put into the Yank-Ring. From this ring, you can get your text back (pasting, yank). The size of your yank-ring depends on your setup.

Yank -> C-y, use M-y to cycle through the various text-parts.

If you’ve marked some text, use C-w to cut the text, or M-w to copy the text into the yank-ring:

Cut -> C-w
Copy -> M-w

Regions

To mark some text, use the mouse, or set a mark with C-<Space> and move the cursor to the end point of the region. The text between the mark and the current cursor-position (also called point) is the region.

Set mark -> C-<Space>

You can jump between point and mark using C-x C-x, this switches the mark and the point, so that your region remains intact.

Switch point and mark -> C-x C-x

If you’ve enabled the transient-mark-mode Emacs highlights the region when you create it. As soon as you move the cursor, the region is still there, but the highlight is gone (verify with C-x C-x for example).

Rectangles

The mark and point also specify a rectangle over your text (upper right and lower left edge).

Kill the rectangle -> C-x r k
Yank the rectangle -> C-x r y
Insert space into the rectangle -> C-x r o
Replace text of the rectangle -> C-x r t
Clear rectangle (replace with space) -> C-x r c

There is an interesting minor mode for additional rectangle-functionality called CUA.

Frames, Windows and Buffers

Emacs has the concept of Frames, Windows and Buffers. A Frame is a Window in the common sense, a Window is a part of the frame. For example if you split the screen of your Emacs frame, you got two windows within one frame.

The last concept is the Buffer, it’s simply a real or virtual file (virtual == not saved to disk).

Buffer commands:

Load a file (buffer) -> C-x C-f
Show loaded buffers -> C-x C-b
Switch to another buffer -> C-x b
Close a buffer -> C-x k
Save a buffer -> C-x C-s

Frame commands:

Open new frame with file -> C-x 5 f
Close this frame -> C-x 5 0
Switch to other frame -> C-x 5 o

Window commands:

Split window vertically -> C-x 2
Split window horizontally -> C-x 3
Close all other windows -> C-x 1
Close this window -> C-x 0

The Minibuffer (the last line of your Emacs screen) is a buffer itself, but has some specialities I wont go into now.

Getting Help from Emacs

You could fill some very big books with Emacs-specific information, but Emacs calls itself a “self-documenting” editor. This means that the documentation reflects exactly how your Emacs is configured. If you change some key-bindings, the documentation reflects this changes and displays the right key-combo in the help files.

Some hints on how to get going using the help-system:

Show information on the current mode(s) -> C-h m
Describe the command bound to the given key -> C-h k
Search for a function -> C-h a
Describe all key-bindings of the current mode(s) -> C-h b

Further Information

Steve Yegge has some intersting articles on Emacs:

IBM has some nice Emacs tutorials too.

The EmacsWiki is another excellent source for Emacs related howtos, downloads and – to some extend – discussions.

Wish

I’d wish that Vim and Emacs would merge. That I could take advantage of Emacs’ awesome tool-support and Vim’s awesome way of editing small junks of text. I know there is Viper-mode and Vimpulse, but both are not exactly what I want. Viper-mode is plain Vi, and Vimpulse is early in development.

Is there a way to integrate a Vim window into Emacs without emulating and reimplementing the features?

For the time being, I’ll have to switch between the environments. Vim as my default-editor and Emacs for special purpose editing.

—
Emacs outshines all other editing software in approximately the same way that the noonday sun does the stars. It is not just bigger and brighter; it simply makes everything else vanish. —Neal Stephenson

Free and Open Source Math Programs

Michael — 2008-02-24T14:36:21+00:00

In this post I’ll go through some of the most prominent math programs available with source code.

This is by no means “original” work, I just collected the headlines and links to various mathematical software projects out there. This started with an offer of my University (Mathematica for 13 Euro), but I don’t want to invest time in a tool I won’t have (free, or almost free) access to for the rest of my life.

The programs differ in their respective foci, the main areas are Computer Algebra System, Numerical Analysis, Statistics and general purpose desktop calculators.

My current choice is SAGE and R for my math and statistics needs. For simple calculations I usually use the calculator included with the OS or some interactive programming-shell (ghci or irb).

Hybrid

SAGE

SAGE is the new star on the open-source math sky. Essentially it is a CAS written in Python, but it provides interfaces to many math programs available. This includes commercial programs like Mathematica and Maple as well as free programs like Maxima, Octave or Singular.

From the homepage:

Each interface requires that the corresponding software is installed on your computer. SAGE includes GAP, PARI, Singular, and Maxima, but does not include Octave (very easy to install), MAGMA (non-free), Maple (non-free), or Mathematica (non-free).

SAGE runs as service on your server or desktop and you use your browser to interact with it (you can also use the command-line interface, though). It is very well documented and gives you access to the source code of the algorithms right within your browser.

Use SAGE for studying a huge range of mathematics, including algebra, calculus, elementary to very advanced number theory, cryptography, numerical computation, commutative algebra, group theory, combinatorics, graph theory, and exact linear algebra.

The goal of the SAGE project, which started 2005, is to provide the best mathematical software in the world.

SAGE is pretty large (on my Mac it uses 722MB right after the installation), but you can find even more (optional packages etc) at their homepage.

It’s also possible to interact with it via Texmacs or Emacs (again, available from their download page).

SAGE is available for Windows, Linux and Mac OS X.

Computer Algebra System

Maxima

Maxima is a system for the manipulation of symbolic and numerical expressions, including differentiation, integration, Taylor series, Laplace transforms, ordinary differential equations, systems of linear equations, polynomials, and sets, lists, vectors, matrices, and tensors. Maxima yields high precision numeric results by using exact fractions, arbitrary precision integers, and arbitrarily precision floating point numbers. Maxima can plot functions and data in two and three dimensions.

Maxima is very mature and stood the test of time. It started in 1982 and is “free” since 1998.

Maxima is included in SAGE.

Axiom

Axiom is a general purpose Computer Algebra system. It is useful for doing mathematics by computer and for research and development of mathematical algorithms. It defines a strongly typed, mathematically correct type hierarchy. It has a programming language and a built-in compiler.

Yacas

YACAS is an easy to use, general purpose Computer Algebra System, a program for symbolic manipulation of mathematical expressions. It uses its own programming language designed for symbolic as well as arbitrary-precision numerical computations. The system has a library of scripts that implement many of the symbolic algebra operations; new algorithms can be easily added to the library. YACAS comes with extensive documentation (hundreds of pages) covering the scripting language, the functionality that is already implemented in the system, and the algorithms we used.

Homepage Downlaods, Screenshots, Tutorial, Manual and Demo are available there.

PARI/GP

PARI/GP is a widely used computer algebra system designed for fast computations in number theory (factorizations, algebraic number theory, elliptic curves…), but also contains a large number of other useful functions to compute with mathematical entities such as matrices, polynomials, power series, algebraic numbers etc., and a lot of transcendental functions. PARI is also available as a C library to allow for faster computations.

PARI/GP is included in SAGE.

Singular

SINGULAR is a Computer Algebra System for polynomial computations with special emphasis on the needs of commutative algebra, algebraic geometry, and singularity theory.

Singular is included in SAGE.

GAP

GAP is a system for computational discrete algebra, with particular emphasis on Computational Group Theory. GAP provides a programming language, a library of thousands of functions implementing algebraic algorithms written in the GAP language as well as large data libraries of algebraic objects.

GAP is included in SAGE.

Macaulay2

Macaulay 2 is a software system devoted to supporting research in algebraic geometry and commutative algebra.

Numerical Analysis/Computation

Octave

GNU Octave is a high-level language, primarily intended for numerical computations. It provides a convenient command line interface for solving linear and nonlinear problems numerically, and for performing other numerical experiments using a language that is mostly compatible with Matlab. It may also be used as a batch-oriented language.

Euler

Euler is a powerful interactive numerical laboratory. It quickly evaluates numerical functions, visualizes results, and allows to test and program numerical algorithms. The system can handle real, complex and interval numbers, vectors and matrices, and additionally a long data type for exact computation. Most routines are written in the high level Euler language and can be modified or extended by the user.

Scilab

Scilab is a scientific software package for numerical computations providing a powerful open computing environment for engineering and scientific applications.

Statistics

R

If you want to do some statistics there is no way around “R”. At least not in the free and open-source software world.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible.

I think R is a tool everyone should be familiar with, at least with the most basic operations. Statistics is such an important every-day tool you can immediately put to work as soon as you know something, anything, about it.

Homepage
Documentation
Tutorial
Screenshots
Books (currently 53 books!)

Command-Line/Desktop Calculator

Sometimes you just want to calculate very simple things, for this task a full fledged system isn’t really appropriate.

bc

bc is an arbitrary precision numeric processing language. Syntax is similar to C, but differs in many substantial areas. It supports interactive execution of statements. bc is a utility included in the POSIX P1003.2/D11 draft standard.

bc is usually installed on any Linux/Unix system or available via its package management.

“Other” command-line/desktop calculators

Essentially you can use any interactive shell, like Python’s python or Ruby’s irb as well as Haskell’s ghci or hugs.

GUI Calculators

Despite the included Desktop Calculators there are some very fine products out there, here some pearls I could find:

Mac OS X
- Magic Number Machine
Linux
- Qualculate!
- Calcoo
Windows
- Calcoo for Windows
- Any suggestions?

If nothing else, there is always Google Calc.

—
Not being able to check the code of a computer-based calculation is like not publishing proofs for a mathematical theorem, it’s ludicrous. —William Stein, author of SAGE

Addendum to "Time Machine for every Unix out there"

Michael — 2007-12-06T10:55:36+00:00

My article about using rsync to mimic the behavior of Apple’s Time Machine generated a lot of traffic, and more important, a lot of feedback.

In this article I’ll summarize and try to clarify a few things.

First of all: I didn’t invent any of the things I’ve written in the article! All I did was to write it up in a manner I could understand it. The rsync developer(s) included the feature(s) for a specific purpose which might be exactly what we do now. But this was obvious, I hope.

Time Machine and Performance

Ars Technica has a very deep review of the new features of Leopard, including Time Machine.

Interestingly they do very much the same as I did in my script – each Backup gets its own folder and there is a link to the latest.

Some readers complained that Time Machine looks only at the changed files, where my script has to look at each file on the disk. For me, the difference is negligible. Apple introduced hard-links to directories, which saves a lot of time and a bit of space. Unfortunately this feature isn’t availabel on plain Unix, but it’s a neat trick!

Restoring the Files

Since we make full backups each time, you have your whole and complete directory structure in that Backup-directories.

Restoring a file is merely a simple copy command.

Or, if you want to restore the whole filesystem, use:

rsync -avz --delete host:/Backup/scurrent/ /target/

But keep in mind that —delete removes any file which is not in the backup!

Removing old Backups

Another huge advantage of full backups is: you can delete any backup you don’t need anymore.

Of course each backup-directory takes space, if only for the hard-links and directory structure. Cleaning up the backups is a breeze: just delete the directories you don’t need anymore. For example you could delete all directories older than 2 months or keep one week full, then only the monday backup. There are plenty of possibilities.

Other Tools

Many people suggested many interesting tools for doing backups. My argument against using such tools is that rsync is everywhere!

You find rsync in the smallest Linux distributions (like DSL), on PDAs like the Zaurus, on Windows using Cygwin and also on Nokia’s Internet Tablets.

One backup strategy for all your devices!

Space Requirements for the Links

Nothing in life is free, but the space used by the hardlinks for the backup is marginal. It is also dependent on the filesystem you use, ReiserFS seems to do a good job in packing more links in one cluster. But hey, these things are Backups you throw away space anyways!

Colon

There is one particular problem using a colon in the script: some operating systems don’t like them.

Here an updated version of the script without colons:


 #!/bin/sh
 date=`date "+%Y-%m-%dT%H_%M_%S"`
 rsync -azPE --link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date \
 && ssh $HOST "rm PATHTOBACKUP/current \
 && ln -s back-$date PATHTOBACKUP/current"

Another slight change in the script is, that all commands are connected with &&. This means the next command is only done when the previous command returned no error.

Of course, if anything messes up the “current” link, the script just makes another full backup without the links to the previous version.

Extended Attributes on OS X

One user suggested the addition of the -E parameter on Mac OS X. If you use the parameter on non-OS X machines it’s more or less redundant (-a contains executability), but on OS X some additional information is backed up.

Just use it.

Windows

Some users reported success with this strategy on Windows. I’m happy that we now have a unified backup strategy for all major operating systems without paying a single cent.

You have to use Cygwin to install rsync (there are binaries without the whole Cygwin on the net, here for example).

Measuring the Space Requirements

To lookup how much space your particular backups need, simply issue du -shc back-*. This prints each backup directory with its real memory usage:

 me@myserver:Backups % du -shc back-* 
 406M back-2007-11-12T12:00:00
 6.8M back-2007-11-13T20:28:56
 189M back-2007-11-13T23:42:54
 2.6G back-2007-11-14T08:15:43
 1.8M back-2007-11-15T20:07:07
 7.9M back-2007-11-18T23:57:09
 11M back-2007-11-19T21:53:34
 2.6M back-2007-11-21T18:57:40
 2.1M back-2007-11-21T19:11:02
 11M back-2007-11-22T01:19:38
 1.9M back-2007-11-22T11:05:17
 19M back-2007-11-22T18:37:56
 7.4M back-2007-11-22T22:45:04
 8.5M back-2007-11-22T22:51:29
 9.2M back-2007-11-24T12:45:12
 4.2M back-2007-11-24T17:30:38
 11M back-2007-11-25T00:41:19
 1.9M back-2007-11-27T20:41:40
 317M back-2007-12-03T10:19:07
 12M back-2007-12-05T11:51:19
 3.6G total
 me@myserver:Backups %

As you can see, the smallest backup (obviously without any changed file) requires 1.8MB.

—
Backups are typically that last line of defense against data loss, and consequently the least granular and the least convenient to use. —Yosemite Technologies

Time Machine for every Unix out there

Michael — 2007-11-13T20:55:57+00:00

rsync is one of the tools that have gradually infiltrated my day to day tool-box (aside Vim and Zsh).

Using rsync it’s very easy to mimic Mac OS X new feature called Time Machine. In this article I’ll show how to do it, but there is still a nice GUI missing – for those who like it shiny.

What Time Machine does

Time Machine makes a snapshot of your files every hour. The files are usually stored on a external hard drive connected to your Mac via USB or Firewire. Earlier Leopard versions (ADC preview versions) had the ability to make the backups to a remote drive (I’ve heard).

So if you lose a file, or did a devastating change to one of your files, simply go back in time until you find your file or a version that’s not corrupted.

Incrementally backing up all files every hour so that you can access them in reversed chronological order isn’t that hard with standard Unix utilities like rsync. The only missing thing is a nice GUI for which Apple is known to be quite good at.

Making full backups in no time every hour

You can use this method to make a backup every hour or every ten minutes if you like. There are many many features you can tune or configure to your own taste – excluding files that are larger than 1GB for example.

So, here the command to make the backup:

rsync -aP --link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP

Lets go through the parameters step by step.

-a means Archive and includes a bunch of parameters to recurse directories, copy symlinks as symlinks, preserve permissions, preserve modification times, preserve group, preserve owner, and preserve device files. You usually want that option for all your backups.
-P allows rsync to continue interrupted transfers and show a progress status for each file. This isn’t really necessary but I like it.
--link-dest this is a neat way to make full backups of your computers without losing much space. rsync links unchanged files to the previous backup (using hard-links, see below if you don’t know hard-links) and only claims space for changed files. This only works if you have a backup at hand, otherwise you have to make at least one backup beforehand.
PATHTO/$PREVIOUSBACKUP is the path to the previous backup for linking. Note: if you delete this directory, no other backup is harmed because rsync uses hard-links and the operating system (or filesystem) takes care of releasing space if no link points to that region anymore.
$SOURCE is the directory you’d like to backup.
$CURRENTBACKUP is the directory to which you’d like to make the backup. This should be a non-existing directory.

As said earlier, rsync has many many features. To exclude files over a certain size for example, use the option --max-size (unfortunately this is not available on the rsync version shipped with Mac OS X Leopard). The man page or the documentation can give you plenty of ideas in this direction.

So much for the theory of the most important command for our purpose. Here a simple script that makes an incremental backup every time you call it:


 #!/bin/sh
 date=`date "+%Y-%m-%dT%H:%M:%S"`
 rsync -aP --link-dest=$HOME/Backups/current /path/to/important_files $HOME/Backups/back-$date
 rm $HOME/Backups/current
 ln -s back-$date $HOME/Backups/current

The script creates a file called “back” appended by the current date and time, for example back-2007-11-13T22:03:32 which contains the full backup. Then there is a symbolic link called “current” which points to the most recent directory. This directory-link is used for the --link-dest parameter.

You should look at the --exclude parameter (or better, --exclude-from= parameter) and learn how to exclude certain files or directories from the backup (you shouldn’t backup your backup for example).

The script above only works on the local machine because making links on a remote machine needs some extra work. But not much:


 #!/bin/sh
 date=`date "+%Y-%m-%dT%H:%M:%S"`
 rsync -azP --link-dest=PATHTOBACKUP/current $SOURCE $HOST:PATHTOBACKUP/back-$date
 ssh $HOST "rm PATHTOBACKUP/current && ln -s back-$date PATHTOBACKUP/current"

To get that working you either use a public/private key authentication scheme or something else to avoid typing in your password. Another possibility is, of course, to mount the remote file-system on the local computer using the above script.

On my setup the script takes about 6 seconds to synchronize 46968 files and 29GB – this takes 20MB for the file structure (with no actual files to transfer of course). But afterwards, I have a complete backup of my system in a new directory.

On a much bigger setup (1.2 million files and 50GB of data) the backup takes about 30 minutes and takes about 3GB of space (just for links!), so it isn’t exactly free, but very convenient.

The space needed for the backup is determined by the shape of your directory structure. On the larger setup I have lots of Maildirs and a very deep directory structure so it takes much more space than my home-directory backup above. 3GB is quite a lot, but 20MB doesn’t hurt.

Advanced `rsync` parameters

Additional to the parameters described above, I usually employ a combination of these parameters in my backup:

--delete and --delete-excluded this tells rsync to remove files from my backups either if they are gone on my local machine, or if I decided to exclude them from my backup.
--exclude-from=FILE the file specified here is a simple list of directories of files (one per line) which should not be backed up. My Trash folder oder some .cache folders are candidates for this file.
-P is used to give more information on how far the backup is, and how many files are to be backed up. Additional it could resume an interrupted transfer (which doesn’t apply here because we create a blank backup each time we call the script).
-x this one is important because it prohibits rsync to go beyond the local filesystem. For example if you backup you Linux-root partition, you should not include the /proc directory because rsync will get stuck in it. -x excludes all mounted filesystems from the backup which is probably what you want in most cases.

Hard-Links

Each file in a directory is a link to the actual data on your hard-disk. The file-system keeps track of how many links to a area point, and only if the last link is deleted, the whole area gets deleted (in contrast to soft-links, these are pointers to the file-name, not the contents).

Here an illustration of two backups with three files each. File1 and File2 are the same in both backups, only File3 changed between Backup1 and Backup2. So in Backup2, File3 (changed) has to point to a different area than File3 in Backup1.

BTW, there is a nice project for Linux out there which provides the same functionality as Time Machine including a nice GUI which is also based on rsync and the procedure presented here.

The End

Credit: The initial idea for this approach came from Mike Rubel – rsync snapshots.

Also interesting if you have to cope with Windows: Optimal remote backups with rsync over Samba.

There are quite a few approaches out there which more or less do the same, but rsync is available on virtually every Unix out there (even the DSL with its 50MB footprint includes rsync). So using other tools might be more convenient, but I’ll stick with the omnipresent rsync.

rsync offers the possibility to store only the differences to the previous backup (using --compare-dest which should point to a full-backup instead of --link-dest). It then doesn’t make links to the unchanged files, it just leaves them out. This way you get an incremental backup without the “directory-overhead” of the --link-dest approach. But you have to be extremely cautious which one of older backups you delete because the newer backups just don’t contain some of these files (think of full-backups as checkpoints)! Using the --link-dest you can delete all backups but the last and you still got all the files, so I’m happy to pay 20MB per backup for this safety.

—
The universe tends toward maximum irony. Don’t push it. —JWZ

My Mac OS X Setup

Michael — 2007-11-04T18:04:03+00:00

Installing Mac OS X on a Mac is a breeze, yet I usually do two things after the installation to improve stability.

Hibernation/Safe Sleep

The newer Macs come with a feature called “Hibernation” or Safe Sleep. Every Mac owner knows that you can put your Mac to sleep and wake it any time you want. In contrast to some (almost?) all other major operating systems this just works. The sleep mode still powers the RAM, so you still need some power. Now imagine you put your MacBook (Pro) to sleep and you run out of battery. This is where Hibernation comes to play.

Hibernation writes the contents of your RAM to a file on disk, an in case of total power loss, it restores the contents of your memory from that file -> nothing is lost.

Unfortunately writing several gigabytes to disk takes some time, and I experienced hard crashes when I interrupt this process, for example by opening the Laptop shortly after I’ve closed it.

My solution – giving up the fallback option – is to disable Hibernation at all:

sudo pmset -a hibernatemode 0

The pmset utility offers some more options (Hibernation only on battery etc…).

Wake on Bluetooth

The default setting for Bluetooth and Power Management is to allow Bluetooth devices to wake the computer. For example you can press a button of your Bluetooth Keyboard or Mouse to activate your computer.

In my opinion this feature isn’t that hot on a Laptop. I once put my Mac into my bag and accidently pushed a button of my Keyboard. The computer came to live, but had no chance at all to cool itself -> it ran pretty hot when I took it out of my bag… .

Since this accident I always disable “Wake on Bluetooth” because I wake my Mac by opening the lid, not by pressing some button.

These two options increased the stability of my machine quite significantly, and I don’t miss them in any way, although “Hibernation” would be nice – just in case… .

—
Boom. —Steve Jobs

Apple, Apple Care, MacBook Pro, and Mac OS X Leopard

Michael — 2007-11-01T15:15:11+00:00

About one year ago I bought a second generation MacBook Pro. At first everything was fine and I didn’t encounter any weird sounds or screen flickers known from previous models. Unfortunately after a few weeks, it started to make some weird sound sometimes when I open it (or switch it on).

The noise vanished a few seconds after starting the computer but it still bothered me, so I brought it to an Apple Service center. Three repairs later, the noise was still there (the technician also heard it – not only me), so I called Apple Care.

At first they said “it’s the usual operating sound of these machines”. I was tempted to trust them, but I started a small research on the Internet. It turned out, that there is indeed a sound (issued by the CPU) which is considered normal, but my particular one wasn’t included here.

So I called Apple Care again presenting my new findings and giving a very detailed description of my problem. Due to the unsuccessful repairs they offered me to replace the whole thing.

After a week or so I received a brand new MacBook Pro, not the second generation model, but a third generation MacBook Pro including Leopard.

Leopard

The new Mac OS X is a fine step forward. At first it isn’t very different from Tiger, only some minor user interface tweaks here and there.

At a closer look they have improved many details and introduced some very new features (Time Machine and Quick Look for example). I won’t go into the details here, other sites are much better suited for this. But Time Machine is very nice for backup purposes and offers some kind of versioning file system, Quick Look was one of these features I thought “nice, but not for me” until I got a PowerPoint presentation. I don’t have PowerPoint installed on my system, but Quick Look could open it without a hassle – nice!

Oh, yes – Leopard seems to be more responsive to me (faster if you like), especially Spotlight got a real performance boost.

Third Generation MacBook Pro

The new MBP has a LED backlit display which helps to increase the battery life time of the notebook (and lower the environmental hazard) together with Intel’s new Santa Rosa chipset which can independently regulate the CPU cores. Apple claims that these models can run up to 6 hours on battery. Well usually these numbers aren’t that realistic (you have to shut off Bluetooth, WLAN and set the brightness to the lowest level to get near them), but two movies with medium brightness and sound via the internal speakers are absolutely no problem (about 4-4.5 hours). That’s pretty good for a 15.4 inch widescreen laptop!

At first I was annoyed by the brightness adjustment feature, it turns the brightness down if the surrounding light is dimmed, and up as soon as the environment gets brighter. It’s only annoying when you see the change, but the eye is more relaxed and working on the laptop is more comfortable (turning the feature off is no problem either).

Boot Camp

Since I like to play a game every now and then, I thought I could try installing Windows using Boot Camp.

The installation was quite easy. I installed Windows Vista Business because my student edition of XP didn’t work (they are altered editions of the standard XP editions). The Boot Camp drivers allow nearly the same features as Mac OS X has (brightness adjustment, volume, sleep when lid is closed etc.). Vista also runs surprisingly fast (I disabled certain services like SuperFetch and the Indexer), but it uses very much space, more than I am willing to spend on a 20 GB partition… maybe I’ll give XP another try sometime.

Conclusion

Altogether I am very pleased with the outcome of my noise problems on my notebook. Apple was kind enough to replace the whole machine eleven months after my purchase and the new machine simply rocks.

Almost all hardware manufacturers have problems – whether it’s IBM/Lenovo, Dell or Apple. The difference in – my opinion – lies in the service. I’m not saying that other companies wouldn’t replace a faulty product, but replacing it after more than 90% of the warranty have passed is exceptional, isn’t it?

Well, not being able to repair an issue on three repairs is also exceptional (but can I blame Apple for independent service centers?).

The only other issue with expensive hardware was with my IBM T86 TFT monitor. I bought it on an internet auction (private seller, no bill or guarantee card etc). I had some problems with it, so I called IBM if there is a possibility to repair it. It turned out that the monitor had four years of warranty left and I was eligible for their rapid-replacement service (a carrier brings me a new one, and I send back the old one). Again, service matters, but you pay for it – the MBP costs about 2000 Euro, you can get 15,4 inch laptops as low as 500 Euro too, do you think these companies would ever replace the whole thing after 11 months because of some sound which doesn’t affect the usability of the device at all?

—
Have you tried turning it off and on again?—IT Crowd answering machine

XML as human interface and DSLs

Michael — 2007-10-30T22:58:41+00:00

While reading “The Definitive ANTLR Reference” by Terence Parr I encountered this quote. I’m a bit sceptic about XML beyond its use of data exchange, so the quote has completely won me over for the book.

Don’t be afraid to build a human-readable configuration file (I implore everyone to please stop using XML as a human interface!) or to build domain-specific languages to make yourself more efficient. Designing new languages and building translators for existing languages, when appropriate, is the hallmark of a sophisticated developer. —Terence Parr, The Definitive ANTLR Reference, Page 23.

—
Being an expert in XML is like being an expert in comma-separated values. —Terence Parr

Please Leopard, don't Suck

Michael — 2007-10-20T19:58:41+00:00

Apple’s next generation operating system is nearing its release date. Time to face the truth – is it as awesome as Steve told us?

Since I’m a Mac user (despite my Vista adventures), I can only repeat what the brave people of The Joy of Tech published in September: Please Leopard, don’t suck!

Tiger is already far better than Vista – it does the small things so much better than all Windows versions together. Of course on a competitive market these “features” don’t count. But for me as user these are the features I really really love on Mac OS X.

Here my list of these things make OS X the better operating system:

OS X always prints in the right order – on Windows I often had to reverse the pages.
I can print to PDF in almost all situations.
Pressing Shift+Cmd+4 I can make screenshots of a part of my screen (press the spacebar to make pictures of whole windows).
For certain tasks I’m simply so much faster on the (sane) command-line than by clicking through myriads of dialogs (but both ways are possible).
Unix! ‘nuff said.
Taste – yes, I know, this word was stressed a lot in combination with “Apple”, but it’s simply true – compare Windows to OS X. OS X isn’t just a conglomerate of desktop effects, but a fine-tuned composition of feedback-mechanisms. It’s like candy, but you can’t get sick by eating to much of it.
Innovation – being so much smaller than MS, Apple has to put in considerable effort into their products, but – and this is what I like about them – they don’t suffer featuritis (indeed they had a big disaster because of featuritis in the nineties, but they learned their lessons well).

Well, the list got a bit esoteric at some point, but many many things “just work”. Of course Apple is all but perfect, their “closed-system” policy is dangerous and very annoying. Doing things they don’t want you to do (for good or bad reason) is very hard (try to keep a Macbook running while the lid is closed, for example).

Anyway, closed systems also have their advantages, and if you don’t like it, you can always switch to some other operating system on your Intel Mac. Windows is easily installed on a Mac, and for Linux there are some guides found on the net (depending on your actual hardware).

At some point I didn’t want to care about operating systems anymore (Linux was great, but very time consuming), I wanted something that “just worked” (which ruled out Windows). OS X came to the rescue – and here I am several years later, more happy with my OS than ever.

—
Please Leopard, don’t suck!

Windows Vista has bad Behaviour

Michael — 2007-10-05T18:15:11+00:00

Vista constantly accesses my hard drive, here is how to do the exorcism (no, not the format-the-drive exorcism).

One thing I notice whenever I start Windows Vista (I usually rely on Linux) is that it always does something on my harddrive. I didn’t know what it was doing, so I started to investigate.

To keep it short, there are three things to disable to keep your harddisk from working permanently.

First, I disabled the “Indexing Service” which was an obvious candidate for accessing my drive(s). Unfortunately it wasn’t the only process digging on my harddrive.

The second service I disabled was the “System Restore” which was suggested here.

This still didn’t work out, so my quest wasn’t finished yet. The last process I disabled was “Superfetch”. This actually did the trick. I don’t know what it is supposed to do, but the OS didn’t suffer as far as I can tell.

I’ll leave the three processes disabled because I didn’t recognize a negative impact (well, the Index-Service might be important to some people).

—
If something’s expensive to develop, and somebody’s not going to get paid, it won’t get developed. So you decide: Do you want software to be written, or not?—Bill Gates

Why, oh why didn’t they choose otherwise?

Linux configuration

Michael — 2007-09-01T13:42:12+00:00

This weekend I tried to get my new keyboard (Microsoft Ergonomic Keyboard 4000) and my Logitech MX1000 Laser mouse to work properly. The Keyboard has many extra-keys I didn’t bother to count, and the mouse has 12 buttons which can be very useful at times.

Almost accidently I solved a bugging performance problem with the Firefox browser. It was incredibly slow when opening Google Spreadsheets, well the whole system was incredibly slow while loading the spreadsheet… .

Configuring the Microsoft Ergonomic Keyboard 4000 and the MX1000 Laser mouse

Most of the information I used to get my new keyboard working (except of 6 keys) came from the Gentoo-Wiki and the Ubuntu-Forum.

What you need is the

Make sure that you get the right version of the patch for your Kernel (in case that 2.6.22 is outdated).

Extract the Kernel, patch it (patch -p1 < hid.patch) and compile it. I assume you’ve got some experience compiling a Kernel (just follow the instructions at the Ubuntu-Forum).

Now for the configuration, I didn’t follow the Gentoo-Wiki guide and didn’t hack my xkb configuration because I couldn’t see any advantages… it seems that the “evdev” XkbModel does the right thing (if you don’t set the Model to evdev, the keys are really messed up).

I used the Gnome Keyboard-Shortcuts program to assign functionality to the keys. The keys I couldn’t get to work are the special keys 1, 2, 3, 4, 5, Fav, and Zoom-In (Zoom-In is recognized by X, but is already assigned). Since I use GMail for mail I assigned the Mail-button to firefox "http://www.gmail.com".

The mouse is equally simple, follow the instructions here.

Here the /etc/X11/xorg.conf config (relevant parts):

Section "InputDevice" 
 Identifier "Keyboard Ergo. 4000 1" 
 Driver "evdev" 
 Option "Protocol" "evdev" 
 Option "Device" "/dev/input/event1" 
 Option "CoreKeyboard" 
 Option "XkbRules" "xorg" 
 Option "XkbModel" "evdev" 
 Option "XkbLayout" "us" 
EndSection
Section "InputDevice" 
 Identifier "Keyboard Ergo. 4000 2" 
 Driver "evdev" 
 Option "Protocol" "evdev" 
 Option "Device" "/dev/input/event2" 
 Option "XkbRules" "xorg" 
 Option "XkbModel" "evdev" 
 Option "XkbLayout" "us" 
EndSection
Section "InputDevice" 
 Identifier "MX1000" 
 Driver "evdev" 
 Option "CorePointer" 
 Option "Device" "/dev/input/event0" 
 Option "Buttons" "12" 
 Option "Resolution" "800" 
EndSection
[...]
Section "ServerLayout" 
 Identifier "Default Layout" 
 Screen "Default Screen" 0 0
 InputDevice "Keyboard Ergo. 4000 1" "CoreKeyboard" 
 InputDevice "Keyboard Ergo. 4000 2" "SendCoreEvents" 
 InputDevice "MX1000" 
EndSection

The udev rules to fix the keyboard to event1 and event2 (/etc/udev/rules.d/10-nek4k.rules, the UID should not differ from yours):

KERNEL=="event*",SYSFS{modalias}=="usb:v045Ep00DBd0173dc00dsc00dp00ic03isc01ip01", MODE="0644", NAME="input/event1" 
KERNEL=="event*",SYSFS{modalias}=="usb:v045Ep00DBd0173dc00dsc00dp00ic03isc00ip00", MODE="0644", NAME="input/event2"

The udev rules for the mouse (/etc/udev/rules.d/11-mx1000.rules):

KERNEL=="event*",SYSFS{modalias}=="usb:v046DpC50Ed2500dc00dsc00dp00ic03isc01ip02", MODE="0644", NAME="input/event0"

The ~/.xbindkeysrc configuration I use simply activates the “Forward” “Backward” functionality (thumb buttons):

# Backward and Forward buttons
"xvkbd -text "\[Alt_L]\[Left]"" 
 m:0x10 + b:8
"xvkbd -text "\[Alt_L]\[Right]"" 
 m:0x10 + b:9

This maps the mouse button 8 to Alt-Left, and button 9 to Alt-Right. Remember to start xbindkeys at the start of you session (for example add it to .gnomerc or even better to .xprofile). To rearrange the mouse buttions how I like them, I use a Xmodmap file (/etc/X11/Xmodmap):

!MX1000
pointer = 1 10 3 4 5 7 6 8 9 2 11 12 14 13

This rearranges the second button (middle button on the scroll-wheel) to the thumb button, and reverses the tilt-buttons of the scroll-wheel. This could also be in .xmodmaprc.

Use xev to find out which number any button has (works for the keyboard too).

Performance issues with Firefox

I noticed severe performance issues with Firefox each time I opened a Google Spreadsheet. The interesting thing was that Xorg took 100% of the CPU, not Firefox itself. I resolved the issue more or less accidently: I installed the binary NVidia driver which gave the whole system a real performance boost.

Bootup Performance

The next thing I attacked was the boot-time. First I disabled many services I didn’t use. Usually I create a folder called disabled within the runlevel directories (/etc/rcS.d and /etc/rc2.d) and move the services I don’t use (or don’t know I use).

Using a coll utility called BootChart (stores it’s charts in /var/log/bootchart/) I found out that a big chunk of my startup time is taken by getting my IP by DHCP. I changed my configuration to a static IP, so I could reduce the startup time by 10 seconds.

My chart currently shows a lot of time (and I/O access) is coming from modprobe, I wonder if compiling the Kernel with all necessary drivers included into the Kernel would give me even less startup time… Maybe I’ll try that sometime too :-).

So, now my Ubuntu setup seems quite nice, unfortunately I edited a lot of files in the process, which are easily lost so this is more or less a tutorial for my future-me to get the stuff working again.

—
It’s also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: ‘For every file that is read from the disk, lets do a … write to the disk! And, for every file that is already cached and which we read from the cache ... do a write to the disk!—Linus Torvalds about atime