This is Graham Poulter

Cutting down on clutter with the Outbox Method

2013-03-31T22:58:00.002+02:00

We are only human, descendants of Homo Habilis the tool user, and we get emotionally attached to our
tools and memorabilia, loath to discard anything that has use or value or recalls memories. In time this causes clutter as we acquire more durable goods than we donate, recycle or discard. When I chose to move from Cape Town to Ireland, I also desired to strip away clutter and start afresh with only things I would actually use, plus a few sentimental or beautiful items. In the process, I've refined a trivially simple but effective method to cut down clutter and prevent accumulation sometimes called the Outbox Method.

Letting go
The insight I (and many others) had is that it is stressful to let go of a many of your belongings at at once, or to let go of a belonging quickly. So, the trick is to let go gradually, both in number of items and taking time over it. To this end I placed a box by the front door, and every now and then I would look through my clothes and belongings and put the least-wanted items in the box, no matter if they were perfectly functional or could fetch money on Gumtree. It didn't feel like I was throwing them away, more like I was segregating the things I wanted the least, and I could always pick it out of the box if I needed it or changed my mind - and sometimes I did. After some time, the box gets full, and this is when I would make a trip to the local charity shop and donate the whole lot.

Over the 4 months preceding the move, I made a trip to the charity shop roughly every two or three weeks, both hands holding bags of clothes and goods. In the end, I brought 29kg on the plane, 106kg in airfreight (of which 22kg were books), and left 65kg of filing in a cupboard at my parents' home. Since arriving in Ireland, I have donated yet another bag of clothes and three bags of goods - the latter bulked up with a lot of hangers I have no closet space for.

One in, one out
The outbox has worked great for a mission to cut down clutter, but how does one prevent it accumulating with every trip to the mall? For the last year or so I've been applying a "one-in, one-out" principle, that if I buy something new, I should put some equivalent amount of items in the outbox, preferably a similar item. Unless I am desperately short of t-shirts, for every new t-shirt I buy an old shirt goes in the outbox.

Dress better without buying anything
An aside on wardrobes: they tend to be tiered, with house clothes, casual clothes, work clothes, clothes for going out, and formal clothes. I've found that if you donate your house clothes, you don't suddenly find yourself with nothing to wear - rather your oldest casual clothes become your house clothes, and some evening-out clothes become casual clothes. In other words, you can improve your wardrobe not just by buying new items, but by donating old ones.

Right now I have very little clutter, everything fits in the cupboards, and I intend to keep it that way. Yes, that is my living room.

Taking notes in a notebook made from trees

2013-03-14T23:15:00.001+02:00

A few weeks ago I started a new job that calls for a lot of training, and I chose to take notes using a dead-tree notebook instead of a notebook computer (gasp!). The notebook itself was conference swag dating from 2011, and happened to be a a Moleskine Ruled Soft Notebook Large (13x21cm, 192 pages). I have since grown fond of it and finished the last page today. Yesterday I bought another despite the disturbingly high price per page (on Amazon it's about 30% less and still pricey).

The last time I did weeks worth of note-taking was 2005, so when I started again a few weeks ago my notes were messy with little structure, lacking ways to find notes or identify the subject of a page without reading it. But I've re-learned and refined some techniques over the many training sessions. Here are some methods I favor at the moment:

Write the title of the session in centered block capitals, keeping some empty space around it.
Write the date in the page corner (I write them like 2013.03.14) - these do double duty as a form of page numbering besides answering "When the hell did I write this?"
Adjacent the date include a compressed form of the title of the lecture or meeting, this labels the rough subject of the text and makes it easy to riffle to an entry.
Use colored pens to make important text stand out. The scheme I use is roughly:

Black: most text and section headings.
Red: newly-defined term or paragraph subject, and also key terms and points.
Blue: external references like URLs, and later annotations.

Use first 6 pages for a table of contents. Why 6? This leaves 192-6=186 pages, and I get about 23-25 titles per 30-line page, so 6 pages gives 140-150 titles.

For each entry, write the date in red followed by the title in black.
For conferences, put the conference title on it's own line, with dated talks indented on the following lines.

For later notes at least, I have a nicely indexed, easy to find-and-identify sequence of notes in which the subject keys and external references stand out clearly.

But why did I opt for the notebook technology state of the art in 1800 C.E., when I was also issued with a lightweight, latest-model 13" MacBook Air on my first day? Well, I did use the MacBook at first. But the first time I did so, another attendee tapped me on the shoulder to tell me the sound of typing was breaking her concentration. I suppose I was typing too fast and too much: the temptation for a fast typist is to take down nearly-verbatim what the speaker is saying as they say it. But I want key concepts, not a complete dictation! Quality over quantity. With digital notes I also feel like editing them during the session, or I'm tempted to quickly search on something the speaker mentions - when I should be paying attention to what the speaker is saying. Afterwards. even though I rarely revisit the notes, they take up visual space in my notes folder, vying for attention and turning up unwanted in search results, becoming clutter. And they're still editable, with the temptation to tend them later.

With a paper notebook, the slowness of handwriting forces me to stick to key concepts, the permanence of pen and paper prevents me from turning my attention to editing, and the linear succession of notes is it's own form of organisation. And using an attractively-bound notebook makes me feel like keeping the notes around instead of tossing them in the trash.

Review and summary of "The Blue Zones"

2012-10-21T16:58:00.000+02:00

The book is titled "The Blue Zones: Lessons for living longer from the people who've lived the longest" by Dan Buettner. The big idea is that some parts of the world have significantly more people living to 100 years old than the global average, so let's see what's different or notable about how them and their ways of living, because maybe there's something to learn from them. There's a smattering of input from scientists at the beginning and end of the book, but the bulk of the content is personal observations, anecdotes from centenarians, and travelogue of Buettner's "Blue Zone Quest".

Warning: "look at the successful people" is a common approach popularised by the "get rich" sub-genre of self-help books. But what the approach finds is possibly-spurious correlations. For example, Dan observed that several centenarians are fond of an evening glass of red wine , or port, or saki - but it is a massive jump from observing moderate red wine consumption in some old people to concluding that a nightcap will add years to your life. Yes, look at the centenarians to generate hypotheses, but life is much too short to take up everything that seems notable about their behaviour without solid evidence for cause-and-effect.

Why did I blog about this? The book has been on my shelf since 2009, mostly a waste of space and money. Blogging is the motivation to read it before giving it to charity. And, I've decided that if I read something (non-fiction), I may as well take some time to summarise it, and if I've summarised it I may as well blog it.

Buettner picks out four "Blue Zones" (Wikipedia article, worth reading): "Okinawa, Japan; Sardinia, Italy; Nicoya, Costa Rica; Ikaria, Greece; and among the Seventh Day Adventists in Loma Linda, California" - communities which have well-above-average number of centenarians per capita.

The next three paragraphs are actually all points from the first chapter:

About aging:

Twin studies find genetic factors explain about 25% of variation in lifespan.
Gerontologists define aging by the risk of dying. Increasing age is the overarching factor in the continuous risk of dying, but it's not the sole determiner.
Many of the changes described as associated with ageing - becoming farsighted, graying and loss of hair, loss of collagen in skin - are not universal.
On average, a 30 year old alive today has a reasonable chance of living to late 70's or early 80's. It would be a decade more, but for a few major risk factors (heart disease, cancer, stroke).
Bodies are like cars built for 200,000km: a few such cars will go to 300,000km or more, but all deteriorate over time even with the best upkeep. With deterioration comes frailty: when you hit a bump, you are less capable of bouncing back. At some point there's no bounce-back, and then you die (paraphrased from Tom Pearls)
Estimating chances depends on current age. Chance of a 80-year-old living to 100 is much better than a newborn. Looking backwards, most centenarians were quite healthy at 80.
Living to 100 is like picking 5 lottery balls to win. Removing major risk factors can reduce the required balls from 5 to 4.

Improving chances:

There is no pill that universally "extends life".

Anti-oxidants? Bah humbug: junk foods like twinkies are full of them to ensure long shelf-life.
Vitamins? Get your basic requirements, which is easily attainable by eating fruit and veg. Any more does not help, and large quantities may cause problems.
Hormones? Simply dangerous. Forget about it.

Diet: reasonable diet, looking for moderation in calories, and balancing the calories across carbohydrates, fats, proteins. Taking in what you really need.
Exercise: prefer changes to your lifestyle, over exercise for the sake of exercising. Bike or walk instead of driving. Exercise built into your lifestyle has better chances of being sustained.

Swimming is a great cardiovascular exercise. But for bones, exercises that use gravity (like walking) are better.
Marathon runners have great cardiovascular systems, but their joints give out.
Walking helps muscle and bone, without the joint pounding of running.

Living more "good years". Who wants to live an extra 2 years on life support? The real question is: how can you delay the onset of disability? Aim for "successful aging" rather than mere lifespan.

Things that may help with living longer:

Social connectedness. If nothing else, it makes life more worthwhile.

But it's very individual. For example, you can't say that "family support" is universally good: some people are very anxious and upset about their families.

Doing something you find interesting and worthwhile. If your work is driven by internal passion, rather than externalities like money, then it's less stressful.
Get rid of anti-aging quackery. It costs you money, and often harms you.

And this is all from the last chapter, which picks out the common observations from the previous five chapers:

Buettner's "9 lessons" for living longer:

Move Naturally: incorporate exercise into your lifestyle.
Hara Haichu Bi: Confucian reminder common in Okinawa, to stop eating when you are 80% full. Combined with eating the right foods, it keeps obesity away. It's the difference between stopping when you are full, and stopping when you are no longer hungry. Weight gain is not from stuffing yourself, rather it's from eating a bit more than you need every day. Prefer foods with lower caloric density.
Prefer plant foods. A mostly plant-based diet accented with meat. We do need protein at each meal, but not much. Also, eat legumes.
Wine at 5. Note: the "glass of red in the evening" advice is not on a solid evidential footing. When it comes to health benefits of alcoholic beverages, it's all swings and roundabouts.
Sense of purpose: live longer by having something worth getting up for in the morning. Take up something new, so you don't stagnate.
Have down-time: regular times to slow down, unwind, de-stress. "Life is short, don't run so fast you miss it"
Belong (to a community): Buettner lists religious communities. I would add that there are many of communities built around common interests, that don't require their members to believe in fairy tales.
Loved ones first: the centenarians live in multi-generational homes where younger generations care for the older ones, with strict "honor the elders" cultural norms. That's a rarity now in Western societies, and fewer healthy years for the elders is the price.
"Right Tribe" (being around the right people): the Blue Zones have tight-knit communities for social connectedness, and social circles that promote healthy lifestyles. The centenarians also tend to be likeable: "there was not one grump amongst them".

Summary of Time Management for System Administrators

2012-09-01T13:55:00.000+02:00

This post summarises and reviews the book "Time Management for System Administrators" (companion site: Everything Sysadmin) by Thomas A. Limoncelli. My role increasingly includes systems to administer, which means more interruptions taking away from development projects, so hopefully this will be useful.

Core Principles
I enjoyed the principles he presented with computer analogies, which I've paraphrased:

Conserve RAM: Working memory is like CPU cache, short-term memory is like RAM, long-term memory is like a hard disk - and a digital task-management system is like an SSD. Page out all future tasks and appointments to your task manager, freeing your short-term memory for the project at hand.

Conserve CPU: Cultivate routines, habits and precomputed decisions - "Think once, reuse the result many times" - so you can allocate more of your thought-cycles to projects.

Lock critical sections: Shield yourself from distractions and interruptions during Project Time.

Minimise context switches: Deal efficiently with interruptions to reduce context switching costs.

Prioritise like QoS: Too many packets arriving? Drop low-priority packets from the middle of the queue.

Then there were some major themes: deflecting interruptions, developing routines, and managing workflow. Below I've paraphrased summaries of key ideas under those headings.

Deflecting Interruptions
Interrupts costs a context switch, time to deal with it, another context switch back, and any time to fix mistakes you made due to broken concentration.

If someone has to be on-call for interruptions, take it in shifts with a buddy to be the "Interruption Shield" (or "Interrupt Handler" :)
Interruptors only need to feel acknowledged, most of the time the task can be done later.
Perform one of 3 actions while the interruptor is present, then say "thank you" and the interaction is over:

Delegate the task to a specific person.
Record the task in your ticket system for later.
Do the task there and then, while they watch you.

Worst outcome to avoid: for the interruptor to leave you there busy with their task.

Setting aside Project Time
Advice on working on large projects, as opposed to independent tasks.

Use your hour(s) of "Peak Brain Power" for Tough Project time. Often the first hours, before the office is full and people are interrupting (or late at night, if you lack a social life).
First hour is totally wasted if you use it to catch up on mail or news.
Remove distractions by turning off IM, email, feeds, notifications.
Deflect interruptions using the "Delegate, Record or Do" rule.
DO NOT check mail first thing. (1) no-one's there to read your replies anyway, (2) anything really important should be SMSd to you, or at least blink on your monitoring dashboard, (3) you know you would be compelled to read ALL the mail.

Developing Routines
Routines save both "RAM" because they are easier to remember, and "CPU" because you can compute the routine once and then use it over and over.

Some good routines and pre-computed choices:

First thing in the morning, make your daily list based on calendar, task list, request tracker.
One hour per day assigned to flush queued-up "interrupt" tasks.
Weekly meeting with your line manager.
Weekly customer walk-around to check for unreported issues.
Weekly hour devoted to catching up on news.
Weekly hour for clearing accumulated emails (delete, or to the wiki or ticket system)
Whenever using a door, check for your keys or access card.
Whenever editing an unversioned file, make a dated backup, whether it's important or not.
Call vendors early in the morning, gives them the whole day to work the bureaucracy.

Managing Workflow (the Cycle System)

Limoncelli first identifies some examples of bad systems: Pile of Post-it Notes, and the Giant Ever-Growing Todo List of Doom. Much of the book is about his agile-like "Cycle" system, where you create a separate list for each day of the year and can feel good about completing the just the day's tasks. It combines task list and calendar. His system also combines office life and the rest of his life, because sysadmins blur the two anyway. His one daily list contains both employer entries at the top, and personal entries torwards the bottom.

He stresses the importance of using just one system to handle everything. Unfortunately I have Calendar(s), JIRA GreenHopper board at work, Remember The Milk for personal tasks, and Trello to track personal projects. That's 5 systems, each with distinct advantages for different tasks. The problem is there's no one place where I can go to see what I'm doing today. To combine them I'm going to do the Daily List in the first 10 minutes of the morning, just a Dropbox text file, with short-hand titles of each task and appointment for the day, A/B/C priority and time limit, and maybe a link back to its source.

Some options for handling overflow, where there's too much for one day:

Punt low priorities to the next day.
Split large tasks
Cut the scope of large tasks
Ask boss for help prioritising
Delegate tasks

Prioritisation
Advice on prioritising, since there's no end to the tasks.

Drop low-priority tasks like QoS if you're flooded with new tasks.
Give higher priority to projects based on impact: Think of Return on Investment (ROI).
Reorder tasks by customer perception of how long it takes: what the customer thinks of as a "quick task" should go first.
"Hurry up and wait" tasks (like ordering new hardware) should also go first.
Special priority for requests from your boss.
Manage your boss: (1) share your career goals, (2) only delegate upwards if it requires boss's authority, (3) understand boss's goals and help accomplish them.
Fix conflicting priorities by getting the sources to communicate with each other.

General Advice
Collected a few unrelated bits and pieces:

Procrastination can arise from fear (of change) and doubt (of succeeding).
Self-esteem is like poker chips you need to make "bets" on life, chips that can be increased by kind acts - and you can be kind to yourself.
Fall asleep like a PC: dump contents of RAM to disk, then hibernate. RAM needs to be powered on and continually refreshed. Meaning: empty all the issues taking up your attention (RAM) onto paper (disk) so that your brain can hibernate without fear of forgetting.
Don't trust your brain. If someone asks you for something verbally and you cannot record it immediately, ask them to send you mail or open a ticket - as opposed to just saying "yes, I'll do it".
Don't work late. Most people have 4 or 5 productive hours in them each day, and the rest is spinning your wheels.
Vacations: single days and long weekends are not enough! Takes a few days to really unwind, and you don't want a backlog of chores. Take long vacations, with no computer.
Pick from a limited set of email actions: Filter, Delete Unread, Read and Delete, Read and File, Read and Reply Then Delete, Read And Delegate Then Delete, Do Now And Delete. Record any important information elsewhere (e.g. wiki), because email sucks at everything.
Make a shell alias or function any time it takes you a few minutes to construct a command line.
Time waster is anything with low ratio of reward to time spent. Tempting time wasters are irresistible and pull you in: you need a rule to set a timer to avoid wasting hours.

Mantras
Scattered about the book were a few sayings that he to hand:

Sooner is better than later.
Yes I can.
Don't trust your brain.
Think once, do many.

My evaluation: "Time Management for System Administrator" is a book worth reading at least once (the first half more so than the second half) if you experience a lot of interruptions because sysadmin makes up at least part of your job.

MemSQL's recipe for a SQL database that's as fast as memcached

2012-08-22T22:41:00.001+02:00

This post summarises the points I found interesting in the longer High Scalability post entitled "MemSQL Architecture - The Fast (MVCC, InMem, LockFree, CodeGen) And Familiar (SQL)" that gives an overview of the architecture of MemSQL, which is a SQL database reinvented for 21st-century hardware. MemSQL's developers claim it to be the fastest database in the world, or at least so fast that you won't benefit from memcached, thus simplifying your app architecture.

Below are what I find to be the most interesting points about its architecture:

All queries are served from memory: db size limited by RAM. The maximum RAM is malloc'd on startup and never paged out.
Uses MySQL protocol: works with existing MySQL client libraries. That's a stroke of genius right there.
Generates C++ from SQL templates and retains the generated code between restarts. MemSQL employs an ex-Facebook dev who knows Facebook's HipHop PHP-to-C++ compiler.
Tables and indexes built on lock-free lists, skip-lists and hash tables, permitting full concurrency on 64-core systems (but does take a row lock to handle write-write conflicts)
By default transactions queue the writes and return immediately, but you can force full durability by setting transaction buffer size to 0.
Background thread groups the writes and appends to logfiles as fast as the disk will allow. Logfiles on reaching 2GB are consolidated into structured snapshots for faster startup. Sequential writes only.
Shards are shared-nothing, so each write transaction can only write to one shard. Aggregation queries across shards work much like MapReduce.
Uses multi-version concurrency (MVCC): write transactions add new versions of the row to a linked list, so that concurrent transactions still see the old values.

There are some limitations though:

Restricted SQL support for now: no support for outer joins, views, stored procs, user functions, triggers, foreign keys or non-utf8 charsets.
READ COMMITTED isolation only.
Not free, currently licensed by cpu-hours

And commenters point out some more:

Skip-lists are not friendly to CPU caches - memory access jumps around a lot during traversal. Counter-comment: benefit of caching is in the hit ratio. Randomly querying a 64GB in-memory database when the CPU has a few MB cache, the hit ratio will be low no matter what data structure you use.
C++ compilation will make the first query for each pattern really slow
Will take many years to catch up to the features of existing SQL databases, if they can even be supported efficiently.

A comparison of file synchronisation software

2012-08-12T13:52:00.000+02:00

This is my biased comparison of the following free software for local file synchronisation: FreeFileSync, SyncToy, Create Synchronicity, cwRsync, WinRoboCopy, Unison, DirSyncPro and Synkron.

TL;DR: FreeFileSync is the best of the bunch, especially for two-way sync.

Given uncapped internet and money to spare, Dropbox would be an excellent all-round solution that puts your files on every computer you use and keeps them all in sync, and lets you recover deleted files. I also use Google Drive for for document storage: word processing, spreadsheets and PDFs. However, for those of us with multiple computers, need for backups, and caps on our home internet connection, there remains a requirement for offline software to synchronise camera photos, videos and other large files.

FreeFileSync

FreeFileSync

FreeFileSync does 1-way mirror and 2-way sync. For 2-way sync it builds a hidden ".sync.ffs_db" database in each root that enables it to speed up sync when files have been renamed, and determine in which direction to sync new or deleted files. The UI makes it easy to specify entire subfolders to sync to the right or left or exclude temporarily. FreeFileSync is a C++ app cross-platform for Win, Mac and Linux. It loads huge directories (100,000+ files) easily.

Tip: to sync a tree of symlinks, set "Symbolic Link Handling" to "Follow" under the "Compare" gear-icon. If part of the tree has complex changes coming from both sides, it's worth using the "temp exclude" feature to focus down to the tricky part and tweak the direction choices.

Cons: No biggies, but there's a learning curve for discovering the features. FFS is not fully automatic - it's worth checking the sync preview in case of conflicting changes on both sides.

SyncToy

SyncToy

SyncToy is a popular older offering, fast and simple to use. It creates a SyncToy_{GUID}.dat database file in the root of each folder.

Cons: SyncToy throws errors on junctions and directory links. It's Windows only and was last updated in 2009. The SyncToy_*.dat files in the sync roots are annoying to look at. 2-way sync works but the 1-way sync "Echo mode" does not restore files that were deleted on the destination so I can't recommend it for more than simple 2-way sync situations.

Create Synchronicity

Create Synchronicity is a tiny VB.Net app (230kb) with a simple user interface. It handles symlinks and junctions and compares up to 4000 files per second.

Cons: excluding files and folders gets finicky because excludes are defined in a single text box, and regex exclusions didn't work for me. Although it "should" run on Mono, it crashes with System.TypeLoadException on Linux.

cwRsync and Grsync

Grsync

Having used rsync on Linux for many years, and sometimes the GRsync, I have naturally used cwRsync on Windows to do my one-way backups for some time.

Cons: cwRsync costs money past version 4.0.3, which has no GUI. GRsync has a GUI. They both use Cygwin posix layer, and I find them very slow on Windows compared to native Windows apps. It gives a "cannot traverse non-regular file" error for Windows junctions and directory symlinks. You can sort-of do 2-way sync with rsync by running it twice but there's no easy way to pick and choose which side to favour in complex cases, so don't do it. The ".rsync-filter" files get unwieldly if you syn different subsets to different destinations. For NTFS destination they write strange security settings on the files that prevent other Windows installations from easily accessing the files.

WinRoboCopy

WinRoboCopy

WinRoboCopy is a GUI frontend to robocopy, which is the closest Windows has to rsync as a built-in command. Unlike cwRsync, it is robust and optimised for Windows directory links, junctions, encrypted data, remote shares and scheduled tasks. It had the fastest transfer rates: 57MB/s over USB3.

Cons: Like rsync it's designed for 1-way sync only.

Unison

Unison

Unison provides reliable 2-way sync, maintaining databases for each root with file hashes and metadata needed to detect renames accurately. It's an OCaml app with GTK UI.

Cons: due to hashing it's slow like molasses, only compares a few files per second, and it's several times slower running on Windows than on Linux. It's ok up to a few hundred megs of data. Tricky to run on Windows, as you have to install GTK+ separately.

DirSyncPro

DirSyncPro is a cross-platform Java app with the icon set of a QT app. It traverses symlinks and junctions and it's "Job Sets" can sync multiple pairs of folders in one click.

Cons: The UI is cluttered and complex, with way too many steps to configure things. Analysis takes much longer than FreeFileSync. It's annoying to have to hover over the base filename to see the relative path. Filters are disabled except in 2-way "Synchronization" mode.

Synkron

Synkron

Synckron is a cross-platform QT App. It appears to support N-way sync, not just 2-way.

Cons: Although it compares the files found under junctions and symlinks, it displays an error that it "cannot synchronise" them.

Coda: A few programs I've skipped over include JFileSync (last: July 2007), OneSync (windows-only, for sync via intermediate location), and Syncless (last: May 2010, focuses on real-time sync).

My Provisional Solution

My approach is to use mklink to create a tree of symlinks to the various folders that I want to sync, which means I can use one big sync job for everything and don't need lots of subfolder exclusions. I've ruled out cwRsync, SyncToy and Synkron on Windows as they don't deal with symlinks.

For one-way synchronisation (mirroring) I recommend rsync on Linux and WinRoboCopy on Windows.

For two-way synchronisation, I recommend FreeFileSync because it's fast, usable, flexible and cross-platform.

Powering the future

2012-07-11T16:30:00.000+02:00

I'm concerned about the potential for anthropogenic climate change to irrevocably damage ecosystems and biodiversity, and cause a lot of human suffering. I'm also excited about the potential of new technology to mitigate the damage by reducing carbon emissions.

I found myself thinking about the technologies I've read about that could help, and below I've collected some of my favourites that, in my amateur opinion, have the potential to greatly reduce carbon emission in residences, commerce and wheeled-transport.

Energy Efficiency
Greater efficiency can reduce the building of new coal power, as well as make it more feasible to supply residences and commerce from wind and solar resources.

Replacing residential and commercial incandescent and halogen lamps with LED lamps & computer-controlled timing. LEDs go one better than fluorescent.
Replacing street lamps with dimmable LED street lights. Traffic lights already under way.
Replacing residential and commercial electric water heaters with solar water heaters.
Replacing conventional electric and gas cookers with induction heating.
Use passive house technologies to greatly reduce energy needed for heating and cooling.

Need technology to get more affordable. Also the problem of old houses, less effective to retrofit.

Transportation without fossil fuels

Electric transport pushes up grid power requirements, but even electricity from coal is more efficient than burning fuels directly.

Replacing buses, taxis and commercial delivery vehicles with battery-electric vehicles.

Requires standardised batteries/charging stations, and automated battery-changing systems so that vehicles can get a fresh battery every time they return to base.

Self-driving electric taxis could serve most of the needs of most urban commuters - home, office, shopping, nights out. Use smartphone to schedule pickup times and dropoff locations.

Requires further development of self-driving cars

Energy Production and Storage
Moving to cost-effective renewable power generation is going to take more technical advances than efficiency or transport.

Exploiting onshore and offshore wind energy resources.

Requires generators not using expensive rare-earth permanent magnets. Promising: electromagnetic excitation, superconducting magnets. Also requires advances in offshore generation.

Exploiting solar energy resources

Requires cheap high-efficiency photovoltaic cells. Promising: vapor-deposited material on sheets of flexible substrate. Imagine: 100 megawatts from carpeting 1square km of desert with photovoltaic material.

Exploiting geothermal and tidal power resources. To a lesser extent, biomass.

Greater technical hurdles here: geothermal power is extremely deep down in most places. Tidal and wave power is also in very early stages, difficult to make machines reliable at sea. Re biomass, we'll never grow enough of it.

Pumped storage hydroelectric can smooth supply fluctuations. Also, gas turbines provide instant smoothing over fluctuatinos.
Batteries the size of shipping containers placed at substations to enable distributed energy storage and load balancing in a smart grid, also helps with smoothing supply fluctuations.

Requires cheap way to make giant batteries. Promising: molten-salt and liquid metal batteries.

Discussion

The demands of heavy industry will probably require supplementary fossil or nuclear power, and of course air and sea transport will still need liquid fuels. I don't think biofuels are a good solution, because of the dubious carbon and energy balance (oil-intensive fertilisers and harvesters and energy used in processing), and that we'll never grow enough with shrinking availability of water and arable land due to climate change.

Even so, I reckon the combined effect deploying these technologies worldwide would go a long way in reducing the burning of fossil fuels. Plus, most of them are really cool.

Maximum Wordpress Performance on an EC2 Micro Instance

2012-02-11T16:21:00.001+02:00

This is how to get Wordpress handling a reasonable continuous load on an Amazon EC2 t1.micro instance using only standard Ubuntu packages. I'm writing down the recipe before I forget how I managed it, and because I could not find a post that specifically details how to get all of Wordpress, Apache, mod_rewrite, mod_fastcgi, php-fpm and php-apc to play nicely together. Subsets, but not all of them. There's plenty out there for nginx though. Eventually I pieced an Apache solution together from many sources, and the synthesis follows. One post in particular from Brandon's Blog on FastCGI with PHP Opcode Cache does capture the basic principle and explains the benefits. Note: most of the benefit is from opcode caching, mod_php will be fine for most sites.

Why an EC2 micro instance? Micro instances are only $0.025 per hour. However, a stock Wordpress/mod_php install quickly runs out of memory and CPU resources at just one request per second. One user ctrl-clicking on posts in the archive for a minute, or a vulnerability scanner, or rude crawler, can DOS your site for everyone else. Without these optimisations, your remaining option is to rent a small instance at 3.8x the price of a micro. I realise the irony of going to all this effort myself and yet my own blog is hosted on Blogger.

Aside: If you are using mod_php on a micro, set MaxClients from 150 down to 50 to respect the micro's limited 593MB of system memory, because each Apache+mod_php process chews about 10MB. Even with FastCGI bringing Apache processes down to 2.5MB, keep it low because there's only one CPU core for all 50 to share. On a micro I also recommend setting MaxRequestsPerChild to 1000, KeepAliveTimeout to 5, and MaxKeepAliveRequests to 30 (about two-thirds of MaxClients).

List of ingredients:

apache2 - standard Apache 2.2 webserver on Ubuntu 11.10
php-apc - Alternative PHP Cache, providing shared opcode cache
libapache2-mod-fastcgi - FastCGI module capable of talking to php-fpm.
php5-fpm - PHP FastCGI Process Manager. Does what it says on the box.
APC Object Cache Backend - WP can use APC to retain objects between requests.
WP Super Cache - cache pages to static HTML

PHP FPM works with mod_fastcgi OR mod_fcgi_proxy but NOT mod_fcgid, because the latter does its own process management exclusively, so does not support proxying to an external process manager like PHP-FPM, and FPM is necessary to get the benefit of opcode cache sharing.

Basic Setup

First install Wordpress from scratch under Apache + mod_php to a DocumentRoot of /srv/wordpress. On Ubuntu, run sudo apt-get install libapache2-mod-php5 php5-mysql php5-gd to get dependencies.

Configure Permalinks using mod_rewrite, since they're pretty and getting mod_rewrite to play nice with FastCGI is part of the trick.

WP Super Cache

Install WP Super Cache from the plugin manager, using the default "PHP" caching. WP Super Cache is a huge win for minimal effort, serving cached HTML to anonymous users without regenerating the page.

Opcode and Object Caching

Install the APC opcode cache with sudo apt-get install php-apc. Run php -r 'phpinfo();'|grep apc to check that it's enabled. APC causes PHP sub-processes to inherit a warm opcode cache, greatly conserving CPU resources.

Extract /usr/share/doc/php-apc/apc.php.gz to /srv/wordpress/apc.php and visit /apc.php to see how the cache is doing. If fragmentation is substantial then shm_size is too low. Restrict access to apc.php.

apc.shm_size = 64M in /etc/php5/conf.d/apc.ini, because the default 32MB cache is too small for Wordpress (once you open Admin). Also set PHP memory_limit to 64M. Also set apc.slam_defense = 0 to prevent piles of "Potential cache slam averted for key" in Apache's error.log. The the slam_defense setting is deprecated by the default apc.write_lock anyway.

Install the trunk version (supports WP 3.1+) of APC Object Cache Backend simply by placing object-cache.php in the wp-content directory. Simply dropping the file in enables Wordpress to cache complex PHP objects between requests and is also a huge win for little effort.

Set up mod_fastcgi with FPM

FPM takes PHP out of Apache, cutting startup memory for new Apache workers from 10MB to 2.5MB per process.

WARNING: Brandon's benchmarks found reqs/second performance is not improved by switching from mod_php to FastCGI, most of the performance gain came from opcode caching. I now only recommend using FastCGI+FPM in case of (a) many concurrent requests and (b) serving a lot of static files (not just PHP content). In those cases the memory saving should be substantial.

Uncomment the "multiverse" lines in /etc/apt/sources.list and run apt-get update to make libapache2-mod-fastcgi available. Run sudo apt-get install libapache2-mod-fastcgi php5-cgi php5-fpm, which also enables mod_fastcgi and starts the "php5-fpm" service. For a micro instance, set FPM max_children to 6, max_requests to 500. Immortal PHP processes been known to go crazy after a while - after a couple of days I found one hogging 100% CPU - so don't let them live forever.

Add the following lines inside your WordPress VirtualHost, adapting "/srv/wordpress" to your DocumentRoot. It pretends that there exists a PHP5 executable called "/php5.fcgi". mod_fastcgi intercepts calls to php5.fcgi and passes them to FPM instead. Also enable the "actions" module to support the "Action" line.

<IfModule !mod_php5.c>
<IfModule mod_fastcgi.c>
        Alias /php5.fcgi /srv/wordpress/php5.fcgi
        FastCgiExternalServer /srv/wordpress/php5.fcgi -host 127.0.0.1:9000
        AddHandler php-fpm .php
        Action php-fpm /php5.fcgi
</IfModule>
</IfModule>

Making mod_fastcgi work with mod_rewrite

The final trick is avoiding infinite recursion between mod_rewrite and mod_fastcgi, which shows up as lots of these in error.log: the Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary.

The solution is to add a RewriteCond %{REQUEST_URI} !^/php5.fcgi just before the final RewriteRule . index.php [L] to prevent /php5.fcgi (the handler) from being re-written to /index.php, which then needs a handler (/php5.fcgi), which is then rewritten to /index.php, ad infinitum. The complete rule block looks as follows:

RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteRule ^files/(.+) wp-includes/ms-files.php?file=$1 [L]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
RewriteCond %{REQUEST_URI} !^/php5.fcgi
RewriteRule . index.php [L]

Now you are ready to a2enmod fastcgi && a2dismod php5 && service apache2 restart.

Try a test query. If anything goes wrong, you can immediately revert to mod_php5 by running a2enmod php5 && a2dismod fastcgi && service apache2 restart. Post a comment and I'll see if I can help.

Faster Windows 7 under Ubuntu by using raw SSD access

2011-11-15T09:00:00.000+02:00

This post is about how I made Windows 7 run fast as a guest under Ubuntu by running it from a raw partition of a Solid State Drive under VirtualBox 4.1.6.

This week I was given an ADATA S599 2.5" SATA II Solid State Drive in 115GB capacity from Mantech for my office workstation -- a 2008 Dell Precision T5400 specced as quad-Xeon with 4GB RAM, and installed Ubuntu 11.10 amd64 desktop on it. It now boots in 17 seconds, and the previously 30-second-long first-time login now takes less than 5 seconds. I use Python/Linux but most of the other devs use .NET/Windows so I require a Windows 7 virtual machine. In fact, the slowness of my Windows 7 VM on a rotating drive was the main motivator for buying the SSD.

I wanted Windows to take advantage of the TRIM command, to avoid the SSD slowing down once all its blocks have been written to. I created an extra partition for windows during Ubuntu installation rather than have the VM run from a file, because I don't think TRIM issued by the guest OS would be passed down to the SSD if the virtual disk is a file on the host filesystem.

To get ownership of the partition for VirtualBox, I created the following udev rule in /etc/udev/rules.d/customdisk.rules which permanently gives graham write permission to the Windows partition which is /dev/sdb1 on my machine. I ran udevadm info -a -n sdb1 to get the start and size attributes to prevent the rule matching any other sdb1. The rule sets UDISKS_PRESENTATION_HIDE to prevent Nautilus from displaying the partition, so I can't corrupt it by accidentally mounting and writing to it while the VM is running.

KERNEL=="sdb1", SUBSYSTEM=="block", ATTR{start}=="2048", ATTR{size}=="136716288", SYMLINK+="win7", OWNER="graham", GROUP="disk", ENV{UDISKS_PRESENTATION_HIDE}="1"

I ran the udevadm test to make the rule to take effect:

sudo udevadm test "$(udevadm info --query=path --name=sdb1)"

Following Virtual Box Manual Chapter 9 on Using a raw host hard disk from a guest I first listed partitions to get the correct partition number:

sudo VBoxManage internalcommands listpartitions -rawdisk /dev/sdb

And created a raw VMDK for partition 1 of sdb:

sudo VBoxManage internalcommands createrawvmdk -filename /home/graham/.Virtualbox/RawDisks/sdb1.vmdk -rawdisk /dev/sdb -partitions 1 -relative

From virtualbox I created a Windows 7 machine, selecting sdb1.vmdk as the disk. On first run of the VM I added the Official Windows 7 SP1 ISO from Digital River to the virtual CD/DVD drive, installed Windows, and installed VirtualBox guest editions with 3D enabled. VirtualBox setting changes included:

System: Base Memory=1024MB, Enable IO APIC=Ticked, Processors=2
Storage: SATA Controller: sdb1.vmdk: Solid-state drive=Ticked
Display: Enable 3D and 2D Acceleration. Video Memory=256MB

The 3D isn't perfect: I disabled animations in guest's system settings, the Windows Experience benchmark crashes, and IE9 scrolling is jumpy (turned out to be an IE9 bug).

But in general the VM is snappy and no longer slows down the Ubuntu host much.

Getting smoke smell out of a waterproof jacket

2011-11-08T20:46:00.000+02:00

Update: editing to sound less like a commercial.

In case anyone is interested, my waterproof jacket was infused with campfire smoke from the Cape Point overnight trail last weekend, and this is how I got the smell out.

The jacket uses "Vapour-Tex" breathable waterproofing, similar to Gore-Tex, so I couldn't clean it with normal detergents. I washed it with a non-detergent liquid cleaner (Nikwax Tech Wash), since normal detergents contain particles that can between the fibers of the jacket's durable water repellant (DWR) coating and impede its performance.

I hand-washed wash the garment in a basin filled with "hand-hot" water using a capful of the wash liquid. By "Hand-hot", I mean I could use my bare hands without getting burnt. It soaked for about an hour. After that even the water smells of smoke.

The bottle instructed rinsing the garment three times. I used hand-hot water for all three rinses. I did not soak on the first rinse, soaked it for 15 minutes on the second rinse, and 8+ hours on the third rinse.

After the final rinse the jacket smelled faintly of smoke, and I considered hanging it out. However, smoke particles get between the fluorocarbon fibres of the DWR layer, so I wanted it out entirely.

So, I washed it again in hand-hot water, soaking it for 8+ hours. Then I did the triple rinse again. After all of that, there was no perceptible smell of smoke on the jacket. It had been soaking wet for about 36 hours.

I hung the jacket out for a few hours to drip-dry and sprayed it with Nikwax TX Direct to maintain the durable water repellant layer. I wiped off excess after a few minutes and left the garment to dry properly.

Finally, I used heat to restore the DWR layer. The usual recommendation is tumble-drying on low or medium, or ironing with a towel in-between (risky). I used a hair-dryer in lieu of a tumble dryer, seems to work.

Logging the client IP behind Amazon ELB with Apache

2011-10-27T11:51:00.001+02:00

How can Apache HTTP Server log the actual remote client IP address when an Amazon Elastic Load Balancer (ELB) is proxying the client HTTP requests? The solution below involves SetEnvIf.

An ELB sets REMOTE_ADDR to the load balancer IP and sets the X-Forwarded-For HTTP header to a comma-delimited string of ip-addresses like client, proxy1, proxy2.

The various solutions I've seen for logging client IP suggest replacing %h (for REMOTE_ADDR) in the NCSA common log format (%h %l %u %t \"%r\" %>s %O) with the X-Forwarded-For header:

LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %O xfwd_common

This approach has two problems:

Broken log formatting
Comma-separated IP addresses violate the NCSA common and combined log formats and generally breaks applications that attempt to extract the log fields.

Above I added quotes around X-Forwarded-For to make it easier to extract by regex. Supporting this modified format in Splunk involves adapting the access-extractions transform to use [[qstring:clientip]] (quoted string) instead of [[nspaces:clientip]] (no-spaces string).

Missing IP for unproxied requests
Direct or unproxied HTTP requests lack the X-Forwarded-For header, so the clientip is logged as "". If all clients connect via the load balancer this won't happen, but in practice developers and monitoring agents may want to skip the load balancer.

Solution for logging the true client IP

I've worked out how to fix the log formatting and log unproxied IPs by using SetEnvIf to log the remote client IP whether the request is direct or proxied:

SetEnvIf REMOTE_ADDR "(.+)" CLIENTIP=$1
SetEnvIf X-Forwarded-For "^([0-9.]+)" CLIENTIP=$1
LogFormat "%{CLIENTIP}e %D %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" trueip_combined

The first line sets the environment variable CLIENTIP to the value of REMOTE_ADDR.
The second line then overwrites CLIENTIP with the first component of X-Forwarded-For if available.
The third line defines the custom trueip_combined log format that uses CLIENTIP in place of %h.
It also uses %D in the place of the never-used ident field (%l) to log request latency in microseconds.

The one downside is that depending on how ELB treats X-Forwarded-For, it may allow clients to spoof their source IP.

Hope people find this useful.

Show the current Git, Mercurial, Subversion or Bazaar branch in your prompt

2011-09-13T15:41:00.001+02:00

This bash script prefixes the prompt with the branch name whenever the working directory is in a Git, Mercurial, Subversion or Bazaar branch. The code is part of my bash-environ package. Here it is on Ubuntu Pastebin, and a plaint-text download.

The script modifies PS1 to reference the branch function which detects .git/.hg/.svn/.bzr version control systems and formats the branch name for the prompt. branch minimises calls to external commands and avoids "bzr nick" entirely, so there should be no noticeable delay in displaying the prompt.

Using dirname-ng (cl_dirname) will provide dirname as a function which is 10x faster than /usr/bin/dirname, for slightly faster check for .bzr, .git, .hg, .svn directories.

Update: Added Subversion checkout detection
Update: Removed use of "bzr nick", examine branch.conf & location files directly
Update: Used __git_ps1 pointed out by Anon. Removed env vars.
Update: Quoted all values to prevent word splitting when path contains spaces.

## Print nickname for git/hg/bzr/svn version control in CWD
## Optional $1 of format string for printf, default "(%s) "
function be_get_branch {
  local dir="$PWD"
  local vcs
  local nick
  while [[ "$dir" != "/" ]]; do
    for vcs in git hg svn bzr; do
      if [[ -d "$dir/.$vcs" ]] && hash "$vcs" &>/dev/null; then
        case "$vcs" in
          git) __git_ps1 "${1:-(%s) }"; return;;
          hg) nick=$(hg branch 2>/dev/null);;
          svn) nick=$(svn info 2>/dev/null\
                | grep -e '^Repository Root:'\
                | sed -e 's#.*/##');;
          bzr)
            local conf="${dir}/.bzr/branch/branch.conf" # normal branch
            [[ -f "$conf" ]] && nick=$(grep -E '^nickname =' "$conf" | cut -d' ' -f 3)
            conf="${dir}/.bzr/branch/location" # colo/lightweight branch
            [[ -z "$nick" ]] && [[ -f "$conf" ]] && nick="$(basename "$(< $conf)")"
            [[ -z "$nick" ]] && nick="$(basename "$(readlink -f "$dir")")";;
        esac
        [[ -n "$nick" ]] && printf "${1:-(%s) }" "$nick"
        return 0
      fi
    done
    dir="$(dirname "$dir")"
  done
}

## Add branch to PS1 (based on $PS1 or $1), formatted as $2
export GIT_PS1_SHOWDIRTYSTATE=yes
export PS1="\$(be_get_branch "$2")${PS1}";

DNS and GeoIP from the browser address bar

2010-12-14T08:43:00.006+02:00

When analysing logs in Splunk for attacks and spammers I need to do reverse DNS lookups, geolocation of IP addresses, and whois lookups to identify the source. Here is a shortcut that saves opening a terminal, by using a custom search engine to do the lookup straight from your web browser address bar.

Visiting DNSStuff.com adds a "Search Engine", at least in Chrome, which does reverse dns + geoip location from the address bar. Set the "keyword" on the search engine to "ip" so by typing "ip 173.194.33.104" in the address bar redirects to this page. The search engine can also be added manually. The %s is replaced by address bar contents on pressing enter:

http://www.dnsstuff.com/tools/ipall/?tool_id=67&token=&toolhandler_redirect=0&ip=%s

The DNSStuff IP lookup is free but their other tools require subscription. DomainToIP.com has a free bookmarklet (bottom left of the page) that does forward lookup and whois on the site currently in your address bar. Add a custom search engine with keyword "dns" so that typing "dns www.google.com" in the address bar directs to this page.

http://domaintoip.com/ip.php?domain=%s

There are many other uses for custom search engines: I have custom search keywords to search the Python documentation, for Google's "I'm Feeling Lucky" button, for Splunk documentation, Google site-search for Wikipedia, and so forth.

Note: In Splunk I also use the Google Maps App and the Reverse DNS App, but to do a single lookup its quicker just to use the address bar.

Growing large crystals of Copper (II) Sulphate

2010-09-26T15:22:00.005+02:00

My shiny ~~Khaydarin~~ CuSO4 crystal!

As a kid I grew crystals as a hobby, mostly from copper (II) sulphate and alum (potassium aluminium sulphate). I tried it again a couple of weeks ago, did it properly this time, and grew the one pictured to the right over about 4-5 days. Wayne's This and That has some good first-hand info on hobbyist crystal growing, and I have come up with a variation on "slow cooling" which I shall call "fast cooling", to drastically speed up crystal growth at the expense of clarity. The post describes my method, which only required about a R50 bill of materials.

Materials

2 small glass beakers (empty spice jars)
Distilled water (R7 from Dischem)
50g copper sulphate (R7 from Dischem)
0.2mm nylon fish line (R6 from AllSports)
Small funnel (R9 from Crazy Store)
Glass ash tray (R10 from Crazy Store)
Plastic spoon, toothpick, blob of prestik, cloth to cover jars.

Warning

Copper sulphate is poisonous, tastes horrible, and the solution stains everything porous. It even precipitates a copper coating onto steel spoons. Copper sulphate stains can be removed from cotton with "peroxy"-type stain removers, and take weeks to come out from under your fingernails. On the plus side, copper sulphate is anti-fungal which is why pharmacies stock it.

Instructions

Grow a seed crystal

Make a small amount of super-saturated solution (steps 2.1-2.5 with smaller quantities)
Fill the glass ashtray about 5mm deep with solution
Use the fridge to cool it down
After 3-12 hours pick out a seed crystal with tweezers: either a single 2-4mm crystal, or a clump grown overnight.
Tie the crystal to a length of nylon fishing line using a slip knot, or a reef knot and trimming excess length.

Make a supersaturated solution

Heat a mostly-full spice jar of distilled water in the microwave at medium power, watching like a hawk to avoid boiling it.
Add about 25g of copper sulphate (add more if it all dissolved)
Stir with plastic spoon till no more dissolves (takes about 2 minutes)
Pour solution into the second jar, leaving the dregs behind.
Place solution in the freezer for 15-20 minutes (or fridge for longer) to cool it down.
Suspend the seed in the jar: hang it over the toothpick, secure with prestik, and adjust so that the crystal doesn't touch the jar, then cover with a cloth.

Grow the crystal: takes 10 minutes every 12 hours

The seed should have grown and there should also be a mat of crystals at the bottom of the jar. The solution is now merely saturated.
Remove the seed and pour about 75% of the saturated solution into a clean jar.
Supersaturate the remaining 25% by microwaving on low power, stirring to dissolve residue, and maybe cool it in the freezer for a bit. If there was no mat of crystals on the bottom to dissolve, add a teaspoon of copper sulphate.
Add the supersaturated solution to the saturated solution, leaving the dregs behind.
Suspend the seed crystal again and wait 8-12 hours.

Tips

Nylon line prevents new crystals forming, and needs to be thin (0.2mm) to tie up a seed crystal. According to Wayne one can form a seed crystal directly on a thicker nylon line to avoid having a fuzzy core in an otherwise clear, slow-grown crystal.
Copper sulphate solubility varies greatly with temperature, and takes ages to evaporate unless you have warm, filtered air blowing over a large-diameter beaker, preferably sitting in a warm thermostat-controlled bath. Too much effort to set up, so I supersaturated the solution every 12 hours instead.
The surface of a finished crystal dehydrates after a while, making it lose its lustre. Complete dehydration would turn the CuS04.5H2O crystal to white powder. May as well store the crystal suspended in solution, or in an airtight jar with copper sulphate powder on the bottom to stabilise the humidity.

Calculating the true cost of car ownership

2010-07-12T16:45:00.010+02:00

In December 2008 I bought my 2004 model-year, 80000km Ford Fiesta 1.6i Ambiente for R72,800, freshly serviced. It now has 99,100km on the clock and went in early for the 100,000km service due to a fault that will yet cost me a couple grand more to get fixed properly.

Here are the costs so far. Maintenance includes the service, two front tyres and wheel alignment, a battery, car washing, and a couple of random repairs.

Parts & Maintenance: R4,539
Licensing: R872
Petrol: R11,854
Parking: R410
Fines: R615
Santam 3/F/T insurance: R2,989

Total cost: R21,279

But there's more: the car's current book value is R59,200 according to Santam, a depreciation of R13,600 since purchasing, bringing my total costs incurred during ownership to R34,879. Divide by 19,100km travel gives a complete running cost of R1.826 per kilometer. Yet of this cost, petrol only makes up 62c.

So, at least for me, petrol is only one-third of the all the costs I've incurred owning a car. Note: I've include costs not part of proper Total Cost of Ownership: namely avoidable costs (fines/parking) and mileage-dependent costs (petrol,repairs).

Edit: Added insurance costs split out licensing as Alapan suggested, qualified total as "costs incurred".

Reducing 3G data consumption when tethered

2010-07-09T17:22:00.010+02:00

My new place doesn't have ADSL yet, so I tethered my 3G iPhone to the computer. As a result, this post is about ways to make the computer sip bandwidth instead of guzzle it, and the changing break-even price between 3G and ADSL in South Africa.
Since I'm on MTN I needed to download a special config to enable tethering for the iPhone 3G, which is available from http://wan.to/iphone/. Here's a how-to for iPhone tethering in Ubuntu.

But 3G bandwidth in South Africa is expensive: R80 for a 100MB MTN bundle (although its cheaper in bulk: R389 for a 2GB MTN bundle). The first time I plugged the phone into Windows the OS grabbed 40MB for updates. After that I did all I could to minimise bandwidth consumption:

Operating System Tweaks

Disable automatic download of operating system updates.
If using Windows, disable 3rd-parter auto-updaters using CCleaner (disable Flash, Java, Google updaters and any others)
Monitor TCP connections to see if any unexpected services are using the internet and turn them off. Use lsof -i on linux, TcpView or Netlimiter Free (thanks Sam) on Windows.
In iTunes, turn off update checking and change the default Podcast settings to "check manually" and never automatically download episodes! That was another 20MB downloaded behind my back...

Firefox Preferences

Disable Firefox update and add-on updates.
Uncheck the Load images automatically checkbox. Text-only default saves a lot of bandwidth: I'm not going to pay to see sidebar ads.
Install FlashBlock and AdBlock Plus add-ons, to prevent downloading random content.
Install Firefox User Agent Switcher to pretend to be a mobile browser and get smaller mobile versions of websites. If pretending to be an iPhone, be aware that Firefox is not compatible with some iPhone-only webapps.

Google Chrome Options

In Options, "Under the bonnet", disable checking for updates, and various things like search-completion and DNS prefetching.
In "Content settings" select "Do not show any images" and "Do not allow any sites to use plug-ins"
Start Google Chrome with --user-agent="Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543 Safari/419.3" to pretend its an iPhone. Chrome uses WebKit and will work with many iPhone webapps.

With this setup I can do searching, mail and news-checking on PC tethered to the phone, without spending hundreds of rands for 3G bandwidth.

The iPhone is faster tethered??

The strangest thing is that iPhone 3G internet suddenly gets fast when I tether it. Safari on the phone at the same desk takes a while to load a heavy page, but plug the phone into the PC and use the PC browser and everything comes down at 500kbps. I should double-check. Does MTN have some way of knowing whether the phone is tethered and making 3G slow for mobile phones but fast when used like a 3G modem? If so, is there a way for the iPhone to pretend its tethered when its not?

Comparing the cost of ADSL and 3G

A Telkom Closer 2 calling plan (free installation and free off-peak calls) is R170 per month, plus R200 pm for an MWEB All-in-One 2GB ADSL package - total of R370 per month. That is almost as expensive as adding MTN's 2GB data bundle for R389 assuming an existing contract. For more than 2GB per month, ADSL will be far cheaper, since R519 will buy the R170 Closer 2 line and R349 MWEB all-in-one uncapped ADSL at 384kbps.

Edit: Added iTunes podcast-disabling, and section on Google Chrome.

The Panasonic Lumix DMC-TZ8/ZS5 Super-Zoom Camera

2010-04-18T19:55:00.010+02:00

For my trip to Zurich I bought a Panasonic Lumix DMC-ZS5, which was released in March 2010. Here are my Picasa albums for Zurich and my later trip to Tygerberg Zoo in Cape Town taken with the DMC-ZS5 (DMC-TZ8 in Europe).

Here are some of the reviews by photographyblog, dpreview and Panasonic's product page. The product page gives a summary of the camera's vital stats and features. Below are my findings from taking the camera on real-life overseas travel for a week.

On the confusing naming, I think ZS implies "Super Zoom" and TZ means "Travel Zoom". The ZS5 is the US camera specially imported by SA Camera, because the European version, the TZ8, is not yet available in South Africa through the Panasonic distribution channels. The difference is the charger (US vs EU plug) and NTSC vs PAL TV-out. The charger is a pain in Europe where travel adaptors feature a rim that prevents the US pins on the charger from actually reaching the socket. Fortunately the lithium battery lasted over 330 photos and a few minutes of movie (Friday to Monday), and only needed 10 minutes of charging at a Zurich camera shop to last the rest of Monday.

The ZS5/TZ8 is also the baby brother of the ZS7/TZ10, the difference being a smaller LCD, lack of RAW recording, lack of AVCHD encoding for hi-def video, and lack of a (power-sucking) GPS to geocode all your photos for you. Nonetheless the ZS5/TZ8 is a great travel companion: small and lightweight, it's happy to do all the hard thinking for you, and has great zoom for zoo visits, and you can even tell it what time zone and location tag to use for the dates of your trip. Lastly, the ZS5/TZ8 and ZS7/TZ10 are successor to the ZS3/TZ7 that was released in early 2009.

Most of the time I stick with the iA or "Intelligent Auto" mode, which on half-press will select a scene type (portrait, scenery, action, night portrait etc), detect normal vs macro mode, select ISO rating, shutter speed and aperture, flash mode, picks out the subjects to focus on. iA mode does have a few parameters including maximum ISO rating, forcing flash-off the resolution / aspect ratio (4:3, 16:9 or 3:2). If I don't like the settings, I release the button, half-press again and iA will often pick different settings, for example focusing on the zoo animal instead of the cage. Specifying "ISO-max 400" is handy for auto-mode because the higher ISO levels that it might pick - up to ISO 3200 for a motion-shot in the dark - are too grainy to be of any use.

The zoom is also pretty smart: the zoom toggle maxes out at 12x zoom, but hitting the extended zoom button will let it go up to 16x zoom at a lower resolution (center of CCD), and hitting the button again will digitally zoom to 32x. Once more goes all the way back to 1x zoom (wide-angle). The image stabiliser ensures photos are not blurry except if the subject was stationary at half-press (slow shutter selected), but then starts moving after the half-press. If it is very dark and your hands aren't steady the camera tries to compensate by using the highest ISO and fastest shutter feasible, for a dark and grainy but only slightly blurry photo. If you use a tripod for the night scenery it will select a longer exposure and take a good photo.

I also use the P or "Programmable" often in situations where iA gets it wrong, typically with funny lighting or macro scenes. I prefer programmable mode to selecting one of the three-dozen specific scene modes. P-mode will still select the shutter, aperture, ISO and auto-focus, but it has a quick-menu to configure all the scene related settings like exposure, white balance, macro mode, ISO mode (specific ISO or auto+ISO-max) flash mode, auto-focus mode, stabiliser mode and dozens of other settings. The P mode has an option to record 5 seconds of sound alongside the photo, which is handy for recording captions you would otherwise forget, like plant names.

For special shots I use manual (M), shutter-priority (S) or aperture-priority (A) modes. S-mode has manual shutter speed and auto-aperture, useful for long nigh-time exposures taken with a tripod. Aperture-priority mode is the opposite and is useful for shallow-field or deep-field shots. Manual mode takes manual shutter, aperture and exposure, but will still do automatic ISO, white balance and such if you let it.

The image quality is fine if you don't let the ISO go above 400. The small high-density CCD has a lot more per-pixel noise than the small low-megapixel CCD's of older cameras or the large CCDs of the four-thirds DSLRs. But the noise is mainly noticeable at 1:1 zoom - you don't see it at screen size, scaled down for web albums, or in print. The movie quality is meh, it's ok provided you pan very slowly, but so far I haven't taken anything worth uploading.

I do have a reliability complaint, in that the LCD is easily scratched by just one grain of sand. So do not let the camera anywhere near sand or dirt. Also, I expect that to sit on the camera or drop it once will be the end of it. It's also susceptible to humidity, temperature and pressure changes (well, to condensation), so I'm concerned that the dirt and dew found on overnight hikes will bring the camera to an early death if I regularly take hiking. So far the camera is two weeks in and going strong.

Back up Windows server with cwRsync and rsnapshot

2010-03-31T16:27:00.008+02:00

Here is how to backup data from Windows servers to a Linux backup server using rsync. Specifically using cwRsync server (a Cygwin + rsync package) running on the Windows server, and rsnapshot running on the Linux backup server. This was tested with rsync 3.0.7 and rsnapshot 1.3.1 on Linux (CentOS 5.4), and cwRsync 4.0.4 (includes rsync 3.0.7) on Windows (Server 2008).

First download the cwRsync server package and install it on the Windows server. It will attempt to create a service user for cwRsync, but the user creation did not work for me. Rather specify the name and password of an existing service account (specially-created if necessary) for RsyncServer service to run as. Use the services panel to configure RsyncServer to run automatically on boot. Before starting the service, configure the rsyncd.conf along these lines:

uid = 0
gid = 0
use chroot = false
strict modes = false
hosts allow = backups.example.com
log file = rsyncd.log
[examplemodule]
path = /cygdrive/c/example/path/to/back/up
read only = true
transfer logging = yes

The uid=0 and gid=0 must be added to avoid the @ERROR Invalid UID nobody. The hosts allow line is to only allow the backup server to access the data. Configure rsync modules for each local resource to back up, using Cygwin paths (/cygdrive/c). You can test the rsyncd.conf by running rsync from the cwRsync "ICW" directory in Program Files and attempting to rsync from it on the Linux server:

bin\rsync.exe --config=rsyncd.conf --daemon --no-detach

On the Linux backup server, install rsnapshot, and see the Debian rsnapshot article for an example of /etc/cron.d/rsnapshot to perform the hourly, daily, weekly backups. By default there are 6 hourly snapshots (one every 4 hours), 7 daily snapshots, 4 weekly snapshots and 3 monthly snapshots. For example, /etc/rsnapshot.conf would need a line like this to get data from the windows server. Type tabs between the items, not spaces:

backup rsync://windowsbox.example.com/examplemodule windowsbox

Also config the snapshot_root to where you want the files to go. The awesomeness of rsnapshot is that it uses the rsync --link-dest option: as it creates the new snapshot it detects unchanged files from the previous snapshot and hardlinks back to them. Thus, you get complete snapshots using only an incremental amount of space. To view the files as they were 3 days ago, visit the daily.3 directory under "windowsbox" in snapshot_root.

Cheap Phone: The Vodafone 135

2010-03-28T18:59:00.005+02:00

This weekend I bought a Vodafone 135 for ZAR 179 (24 USD). The Feb 2009 press release for the 135 had this to say:

The Vodafone 135 is a candy bar style mobile designed to make mobile communications affordable in developing markets thanks to a short two line black and white display suitable for calls and texts. Its design makes it the most affordable phone of its kind on the market. It will be available in the summer in black in developing markets on pre-pay.

I bought it after hearing about the announce of the Vodafone 150 at the Mobile World Conference. The 150 is currently the "cheapest mobile phone in the world", with an unsubsidised price of under 15 USD (111 ZAR), soon to be available in India, Turkey and parts of Africa, including South Africa. Nokia, in one-upmanship will shortly be announcing a 10 USD phone to launch in rural India. I think the 135 is a better-looking phone than the 150 - I like the flat buttons. It does everything you expect from a 1999 Nokia "blockia", but refined by 10 years of hindsight in designing phone interfaces.
The commercial mission of the cheapest phone in the world is to bring in the next 500 million subscribers, win-win for the people who will at last have a means of communication... and banking - since the phones support mobile payments. Watch for the M-Pesa mobile payment system to become the bank with the most customers in the world.

The phones are going to be great for lot of people who never could afford them before. But there's also tiny market for them as a backup phone for the few who live in one of the developing nations but own a smartphone. Smartphones tend to have a short lifespan on the beach, in the mountains, at backpacker hostels, in the rear pocket of jeans, and at large events frequented by pickpockets. Hence, I have bought myself a Vodafone 135 in lieu of paying much the same every month to insure my iPhone.

Sleep better by staying offline in the evening

2010-03-02T20:14:00.003+02:00

My sleep cycle continually drifts later: a spiral of going to sleep later, waking up later, feeling worse, starting to crave coffee, staying at work later, coming home later, working later on the computer... going to sleep later.

After I've been away hiking for a week, I come back and sleep well - because for a week I've been going to bed at sunset, waking up at sunrise and exercising the whole day. But, once back at work, the cycle begins again.

I think a major cause of the late bed-times and insomnia is using the computer in the evening: email, IM, browsing - damn you digg and reddit - and my pet projects, tasks and goals that I work on after-hours, often till after 11pm. And even if I'm off by 10pm, I simply cannot fall asleep diving from the computer onto the bed - even reading a few chapters of a book first. Sometimes falling asleep takes hours. I sometimes go out with friends during the week, but then I usually fall asleep quickly.

So, my new idea is to do all my home computer tasks early in the evening as the first thing when I get home from work. Once I'm tired, around 8pm, I'll shut it down, shower, have dinner, and spend the rest of the evening relaxing - TV, dead-tree books, and scribbling notes on paper, and go to bed as soon as I'm relaxed and bored.

Google Maps South Africa launches new features

2009-10-30T22:42:00.011+02:00

Today Jaroslav Bengl, the Business Product Manager for Google Switzerland GmbH, presented a developer overview of Google Maps features released in South Africa just yesterday.

As of yesterday, the South African Google Maps has driving directions, reverse geocoding ("what is near this coordinate"), and overlays for terrain elevation, photos, Wikipedia, Panoramio, and public transit routes. Google Street View cars are surveying Johannesburg, Cape Town, Durban, Pretoria and East London - and the street view should be ready before the World Cup.

Here is a view of the Koeberg Interchange which now has directional arrows used by the route finders:

All the effort is indeed in preparation for the World Cup, for the sake of all the tourists who have become accustomed to driving directions and complete road networks in the Google Maps of their own countries. Jaroslav says South Africa is getting driving directions and street view ahead of even some European countries and cities.

Google no doubt is paying a lot of license money for access to complete and up-to-date road network data, and we have the Fifa 2010 World Cup to thank for pushing us up the list of priorities. However, for the last few years as a Google Maps backwater, there were some people working to fill in the SA road networks and directions, namely the GPS-equipped volunteers at OpenStreetMap, who even took the trouble to mark on OpenStreetMap the changes to the Koeberg Interchange that are under construction:

On the Google Maps API side of things, the geocoding API (address->latitude/longitude) is now also available in South Africa, and the search box supports mixed languages: searching for "cape town ショッピング" where the second word is Japanese for "shopping" works as expected. However, "레스토랑 cape town" where the first word is Korean for "restaurant" brings up hotels instead. There are many APIs for overlaying your own data on Google Maps, for example using a public KML feed. The terms of service require in any case that apps using the free Google Maps API themselves be free and open to the public.

The South African User Experience Forum

2009-10-29T10:45:00.002+02:00

The South African User Experience Forum ("SA UX Forum") is a community of practice in the field of user experience design. I went to a SA UX Forum event last night at 24.com, organised by Phil Barrett of Flow Interactive and hosted by 20Four Labs, with wine from Stormhoek. About 50 people attended, mostly designers and architects, and a handful of developers like myself.

Phil Barrett spoke about innovation and sketching (especially how sketched interfaces can be iterated and refactored with the users' participation, infinitely faster than software iterations), Kath Roderick from Microsoft gave a demo of Sketchflow (part of Expression Blend), and Dennis Williams from DNA|Creative spoke about the basics of how to sketch - including what pens to use (about 5 different pens) and "how to draw stuff" for people who can't draw. They also demo'd Balsamiq Mockups a lo-fi sketching software for Adobe Air.

I thoroughly enjoyed it, even though by "interface" I usually think of an Application Progamming Interface. If you're interested, they announce events via a Google Group and Facebook. Here's a list of their previous meet-ups and other user experience resources

The 2004 Fiesta and the loose plastic door bits

2009-10-18T20:01:00.003+02:00

The 2004 Ford Fiesta has black plastic strips glued on between the front and back door. Twice now, a plastic strip has come loose and begun to vibrate when driving on the highway, sounding like its about to fall off. I have personally glued the strips back down with contact adhesive and test them from time to time to check if they are coming loose again.

After googling it, I believe this is a consistent manufacturing defect: the wrong choice of glue, and a design defect by not securing the strips properly. Also, the plastic grills on the left and right of the Fiesta bumper come out easily. To make it worse, each of these bits of plastic trimming cost a lot: R400 for plastic grill-bit for a bumper, probably R300-400 for the plastic bit for the door.

IOL article about the plastic trimming: http://www.iol.co.za/index.

QWERTY keyboard dumped for low-pain Colemak

2009-03-07T12:59:00.004+02:00

I can type simple lower-case words at over 100 words per minute on a QWERTY keyboard, slowing down for more complex texts with lots of numbers and punctuations or if the writing takes a lot of thought. However typing all day is causing finger strain, so I sought a keyboard layout to minimises finger motion. Colemak is that layout.

QWERTY is a pain

The QWERTY layout has two design goals (1) to be able to type "typewriter" on the top row for demonstrations, and (2) prevent typewriter keys from jamming. Goal (1) puts frequent keys like t,e,r and i on the top row, and goal (2) results in a lot of "same-finger jumping" where the same finger has to jump rows to type common pairs of letters - for example ed, ce, ju, im, mu, nu, mi, um, ol, lo, ki.

Colemak to the rescue

For a while I've looked at buying some sort of fancy ergonomic keyboard from a manufacturer that doesn't realise how broken QWERTY is. Then via some iPhone app news mentioning it I came across the Colemak website. Colemak is a keyboard layout designed in 2006, partially computer-optimised. According to its model, more than halves typing effort versus QWERTY, about the same as Dvorak but with features that make it easier to learn:

Home row keys (arstdhneio) put the 10 most frequent letters in English under your fingertips
Pinky finger is used only rarely
Loads of "hand-roll combos" where you type 2, 3 or even 4 keys in one smooth motion.
10 keys stay where they are in QWERTY (namely Q,A,Z,X,C,V,B,H,M)
Most windows keyboard shortcuts stay the same (see above)
All keys except E and P are typed with the same finger or same hand as on QWERTY.

Learning Colemak

Colemak can be learnt using TypeFaster on Windows or KTouch on Linux or the Keybr Flash applet, and download lessons from the www.colemak.com. In Windows you install the custom Colemak layout and switch between QWERTY and Colemak with shortcut keys. Colemak is for touch-typing, so you are not supposed to go to the effort of physically re-labelling your keyboard and may not even need to print a cheat sheet if you do a few hours of lessons first.

I started on Monday with GTypist lessons, and also used KTouch. On Windows, TypeFaster has an awesome feature where it generates personalised lessons that drill you on your slowest or least accurate keys. On Thursday, I switched my keyboards to Colemak, which was annoying for a while as my speed was under 20 words per minute and accuracy was low and I had to relearn some shortcuts.
It's now Saturday and I've hit 30 words per minute, though more like 20 right now but with better accuracy. I don't seem to have forgotten QWERTY. Given a minute to adjust, I can switch between QWERTY and Colemak as I please. I just need to remember to keep practising a few minutes a day.

Masochists may instead be interested in the TNWCLR, which increases typing effort 112% over QWERTY.

Bad Python

2009-01-29T13:34:00.289+02:00

Update 2012-07-16: Since this post is still popular 3.5 years later, let me mention that the examples fall into three general categories of bad practices that really are not specific to Python:

"Bad Python" is often "Old Python": using only the conveniences available in Python before 2.4/2.5. In a fast-moving language, the old ways are often going to look bad.
"Bad Python" is often "Java Python": Python written in Java idioms. It's poor form to write in one language using the idioms of another.
"Bad Python" is often "Bad Programming": many of the practices would be poor form in any programming language.

Original Post
I've seen quite a lot of bad Python, even though Python makes the Path of Good Code relatively easier to find than other languages where spaghetti is the result without extra discipline and years of dedicated study of the language on the part of the programmer. Such is the Tao of Perl.

Much bad Python however is from programmers who only knew statically typed OO languages (Java/C++/C#) and have not yet grokked dynamic typing, first-class functions, pervasive use of iterators, properties, etc, leading to eyesores such as:

Accessors such as getDistance() and setDistance(), instead of using an attribute. In Python, attributes can be turned into properties later, preserving the class interface.
Asserting the type of every argument and returned value, taking up maybe 30% of the code itself and 80% of the unit test code. Checking is usually pointless because the interpreter itself will let you know if someduck didn't .quack() like a duck, and makes the code less flexible.
Uber-private attributes (for no good reason), going so far as to use double-underscores on each side, which are supposed to be reserved for language features.
Dozens of customising parameters in constructors, such as reversed and strip and maxlen - when passing in a general transform function would be so much more elegant and could do so much more than just reverse the strings that the class works with.
Delegates where first-class function will do.
Wrapping things classes when dicts and tuples would be cleaner.

Partly, the developers don't recognise the ways in which many Design Patterns become trivial in Python, to the extent that they are more like one-line idioms than chapter-worthy Patterns with capital "P". Singleton Pattern? Write a module. Iterator? It's fundamental to the language. Need a Factory Pattern? Write a function and thanks to dynamic typing you can substitute makedummywidget for the makewidget during testing.Flyweight objects or Command Dispatch? Just use a dictionary. See also Python Patterns. (correction: previously referred to Abstract Factory, which is is not a factory but a group of related factories, e.g. for widgets from a given UI toolkit). Besides bad habits acquired from static OO, one can write bad code, in any language with

Vague and misleading identifiers (topic of future post)
Massive 'god' classes
Source files having no discernable structure.
Awkward decompositions of function

Forces similar logic to be repeated in dozens of places
Prevents parts from being reused e.g. in unit tests
Prevents dependencies from being stubbed out e.g. in unit tests
Mixing logic with orthogonal aspects like error-handling and logging for a harder-to-maintain mess.

Oh well. I recently spent a day re-doing about 30% of the functionallity of 8,600 lines of someone elses Java-style Python into 200 lines of real Python to support the greater flexibility I needed. Code can be that bad. (later expanded to just under 300 lines thanks to feature creep) From the comments:

Python is not Java (a related rant about using Java idioms in Python)
Idiomatic Python (if you want examples of idioms in Python)
Python Cookbook (preview online at Safari Books)
How not to write FORTRAN in any language especially on readability and how a language can help and hinder good design.