Vincent Liu - Coding Is An Art

Why you should Blame your Tools, sometimes.

2009-10-01T15:57:00.000-07:00

As software developers, we would all wish to have 3rd party libraries that works magically immediately after installation. While this is to be expected and holds true for popular and widely used libraries, people do tend to forget that it is still more of a rule of thumb than an iron-cast fact.

Like the urban myth goes, "a good workman never blames his tools", this mindset is so ingrained in the software development culture, that it is always bad form to assume that bug may reside in a library rather than in your code. After all, it is more likely that you are the offending culprit, given it is less likely a widely used code base will make a mistake when compared to code just seen by you alone. But with everything statistical, there is always a chance that an outlier event can happen.

And it did for me.

Internally, we rely on log4cxx as our logging library, which a peculiar bug seems to only be triggered upon the termination of our application. After spending more than 2 days of debugging, the error seemed to have consistently been at the line where the logger object is being accessed. And like all true developers, I bit my lip and kept soldiering on, reading through my own code, checking the same lines over and over again, trying to see if I had missed defining a variable, or if I had forgotten to allocate memory for it. Naturally, after while of doing so, I began to start questioning my personal sanity rather than the lines of code I've written, amid this 'impossibility' that the error could actually be from the library.

And then I popped.

I turned over to my colleague and mentioned to him that the code kept failing on the logging line. The team had known that I've been stuck on this problem for the last day. Even as I started, I could see the mild derision on his face, "it can't be the library. Maybe gdb is not showing what actually is going on."

So on and on the circle I went. Then someone else suggested using valgrind.

Since I was running out of options, I didn't see any harm trying. At most I'll waste an hour more to the day I've already wasted, anyway.

Valgrind didn't really show anything more different to what gdb was showing. Basically saying the same thing - that log4cxx is accessing an invalid memory location. Given I wasn't an expert on valgrind, I got someone else to come over and discussed what we were seeing on the stack trace.

No luck still.

It's gone past the point of frustration, into the point of desperation that I decided to just fucking comment out the offending statement and see if the problem goes away.

After recompling and re-running the application 30+ times without the bug re-surfacing, I was very certain to point at the logger library being the culprit. But of course if I did, I couldn't definitively prove it, and worse, I had no workaround; the entire code base is reliant on it for logging, and to take it away would mean a major re-engineering exercise.

Sometimes talking to others help. Sometimes they offer a difference in perspective, and other times, they may have caught something you hadn't.

Given that we had multiple threads that were still operating when a shutdown signal is received, I thought maybe it'll be worth trying to stop those threads safely before the termination signal is processed - since it's likely that some portions or memory may be freed while the threads are still utilising them. But I was just shooting wildly - it was a very long shot, given how the threading interactions were designed, it was difficult to believe that it could have been the problem. But trying something is better than doing nothing.

So I sat down with another colleague of mine, and talked him through what I wanted to do, and invariably the discussion lead to another where we talked about the valgrind backtrace that I had generated, which showed memory destruction caused by exit(). After a bit of theorising back and forth about the behaviour, and on how I'd would write a quick hack to test my hypothesis, my colleague decided to do a ubiquity search on 'log4cxx exit crash^H^H^H^H', which then our eyes popped when we saw the preview result before it changed into something else that he was meaning to type.

"Go back again", I said excitedly. He was already as excited as I was, so there wasn't really the need for any prompting.

It turns out that there is a known problem in log4cxx found months ago, and someone had even posted a patch on it already. But here's the thing about software: unless you are willing to roll your own fixes, be prepared to wait for a long time before someone eventually fixes it. And even if someone did fix it, the upstream maintainer will still take a while before committing it into the main trunk. Or even worse, he may just choose to ignore it. And yes it does happen - ask Ulrich Drepper. :P

On how to fix it, and package it up nicely for use, I'll write about that in a later post, for now I want to share an invaluable lesson when it comes to software development, which also equally applies in life: while established conventions are more often right than wrong, it never hurts to question them just in case they aren't. It may not be okay to blame your tools all the time, but it sometimes worth to know when to call a shoddy spade when you see one.

How to build a Debian Package for GDB

2009-09-22T15:14:00.000-07:00

I've resisted titling this post as 'building an Ubuntu package' even though I'm building it for Ubuntu - technically it's more proper to call it a Debian package given its lineage. Nevertheless the mechanism behind building your own packages is pretty much the same for the two.

I'll use GDB as an example of how to build your own package - for a good reason, firstly because the stock version of GDB that is shipped with Ubuntu is terribly broken. Here's what I mean:


% gdb --args java
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(no debugging symbols found)
(gdb) r
Starting program: /usr/bin/java Test
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New process 16487]
Executing new program: /usr/lib/jvm/java-6-openjdk/jre/bin/java
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
warning: Cannot initialize thread debugging library: generic error
[New process 16487]
[Thread debugging using libthread_db enabled]
Segmentation fault

The stock build of GDB doesn't handle multi-threaded applications properly, among other minor issue like not setting the path to point to the correct debug library paths, which makes it unusable for serious debugging tasks.

Secondly, GDB 7.0 has reversible debugging, which makes it doubly tempting to roll my own. Finally, GDB has minimal external and library dependencies, which is an easy example to build a package without going into the complexity of having to generate a chrooted environment.

But why not just do the typical 'configure && make install' combination?

The drawback in doing so, is that the process is one-way: once you've installed it like that, there is no easy way of uninstalling it, short of remembering the list of what's being installed and removing them manually. Do-able? Sure, but certainly cumbersome. The neater way to do this is to create a package and have the package manager deal with installation/uninstallation for you.

Building GDB

We need to perform the usual compiling and installing steps like we normally do; the only difference is that we want the installer to place all the resultant files into a separate directory for generating a package. Doing this is straightforward using the prefix flag provided by configure. The steps are commented and reproduced below:


# Assuming that you're in the current source directory /home/user/gdb-sources
% mkdir -p custom-gdb-7.0-amd64/usr/
% configure --prefix=/home/user/gdb-sources/custom-gdb-7.0-amd64/
% make && make install

There are dependencies that GDB will need in order to compile properly (things like bison, lex as far as I remember), but I'll assume that you know how to resolve these dependencies yourself. Otherwise, the source should finish compiling and installing to the destination /home/user/gdb-sources/custom-gdb-7.0-amd64/.

Generating the Control file

In order to generate a package, a Debian control file is required, which contains the information that the 'dpkg-deb' package generator will need. Here's how we write one:


% mkdir custom-gdb-7.0-amd64/DEBIAN
% cat > custom-gdb-7.0-amd64/DEBIAN/control
Package: customgdb
Version: 7.0
Section: base
Priority: optional
Architecture: amd64
Depends: lex, bison
Maintainer: Vincent Liu <blog@vinceliu.com>
Description: Custom build of GDB
This version of GDB provides cutting edge
capabilities that the stock package does not provide.
^D
%
# The control-D symbol above is to indicate the
# file termination character

There are plenty of details I've omitted here, and you will have to read more to understand and tune your own control file configuration. Here's the tutorial I referenced, and the Debian manual to help you figure out the details of each control field.

Generating and Installing the Package

Once you've got the control file generated, building the package is just a single dpkg-deb away:


% fakeroot dpkg-deb --build custom-gdb-7.0-amd64

You will get a resulting custom-gdb-7.0-amd64.deb package generated for installation. To install it, you'll have to remove the existing GDB package, as it conflicts with your new installation. Do the following:


# remove the original gdb
% dpkg -r gdb

# install the new gdb
% dpkg -i custom-gdb-7.0-amd64.deb

If you ever needed to revert back to the stock versions of GDB, you can now easily remove your custom version by dpkg -r customgdb, and reinstalling it using apt-get or your own favourite package manager.

GDB 7.0 Is Out!

2009-09-17T15:13:00.000-07:00

Read the announcement here. It's actually the pre-release version, nevertheless it's quite stable.

What makes this version interesting? It's got reversible debugging, which means that it's the first time you can actually make your code go back in time to find out what it did before it crashed.

I've tried the new commands, but I'm haven't been successful in stepping backwards still - well more experimenting ahead!

Update: The tutorial is available at
http://www.sourceware.org/gdb/wiki/ProcessRecord/Tutorial. (Thanks to Michael!)

Links for 2009-09-17 [del.icio.us]

2009-09-18T00:00:00-07:00

Bing Snags 10-Percent of U.S. Search - AppScout
People may hate Microsoft, but I actually admire the company for its tenacious competitive spirit. Don't bash me, my Microsoft hating lemmings, erm I meant friends - learn to recognise a good thing when you see one ;)

Links for 2009-09-12 [del.icio.us]

2009-09-13T00:00:00-07:00

Obama will not rest…
… until Americans are paying more for a set of tires than Indians pay for an entire car. Is America foolhardy enough to engage in a trade war with China?

Links for 2009-08-31 [del.icio.us]

2009-09-01T00:00:00-07:00

The Programmer's Wife
Be weary the next time some woman comes up to you and starts sprouting about Linux, Python, C++ and The Penguin. ;)

Links for 2009-08-23 [del.icio.us]

2009-08-24T00:00:00-07:00

The Boy Who Heard Too Much
He was a 14-year-old blind kid, angry and alone. Then he discovered that he possessed a strange and fearsome superpower — one that put him in the cross hairs of the FBI.

Links for 2009-08-20 [del.icio.us]

2009-08-21T00:00:00-07:00

Infiltrating a Botnet - Cisco Systems
Fascinating read into the mind of a botmaster.

Links for 2009-08-19 [del.icio.us]

2009-08-20T00:00:00-07:00

A Singapore Taxi Driver's Diary
Probably the only taxi driver in this world with a PhD from Stanford and a proven track record of scientific accomplishments, I have been forced out of my research job at the height of my scientific career, and unable to find another one, for reasons I can only describe as something "uniquely Singapore". As a result, I am driving taxi to make a living and writing these real life stories just to make the dull job a little more interesting. I hope that these stories are interesting to you too.

Links for 2009-08-18 [del.icio.us]

2009-08-19T00:00:00-07:00

Is there any way I can set the compiler in strict C++ mode, I don't want to learn C/C++ mix mode
How to become a Code-Nazi ;)

Bug Hunting with Revision Control using Mercurial

2009-08-13T13:52:00.000-07:00

My impression through reading other software developers' blogs is that, when presented a choice of distributed revision control, there seem to be a majority preference for Git over Mercurial. This popularity does not come as surprising, since it is the brainchild of the most prominent icon of open source software today, the famed Linux kernel hacker Linus Torvalds.

It feels like the Mercurial camp is a little bit unrepresented, but today's post isn't about a blow-by-blow account between the two software revision control systems; you will probably be more informed by experts out there who have better qualifications in critiquing the pros and cons among the two. More importantly, I feel these differences are mostly nitpicking to general laypeople - when the demands of our projects do not scale to a level where the performance differences are visible, it really doesn't matter.

People will have their biases, and the git-vs-mercurial battle will probably be not too different from the perennial ideological battles between the emacs-vs-vim crowds. And obviously, you should have detected my bias in this case already :)

Anyway, back to our original discussion.

I have only used git briefly, and I'll not comment much on it given my limited impression. I'd agree with most people that it is a remarkable software revision control tool and leave it as that. It must strike you as curious to why I am using Mercurial then - it's largely of based on pragmatism, given I've acquired much familiarity through constant use at work. Besides, if I haven't been complaining enough to be tempted to try out git again, it must have not been that bad, right?

I'll share with you the best thing I love about using these revision control tools, namely, for their amazing ability to find out where the sources of your bugs are. It's not uncommon for software to break between the revisions you've check in; and sometimes these breakages fail to be noticed until you've checked it in after a bunch of some other code, making your original error even harder to find.

Well, not anymore.

Mercurial has this ability called 'bisect' which allows you to toggle between revisions until you're able to zone down to the exact patch where your code first breaks. Given that most code check-ins tend to be of small increments, knowing which patch that causes the break helps you narrow down the error to a very small subset where you can focus your efforts on finding the bug.

How does this work? Let me show you a real life session of how bisect works in finding an erroneous commit:


vincentliu@vm1:~/replicode$ hg bisect -g 1614
vincentliu@vm1:~/replicode$ hg bisect -b 1671
Testing changeset 1666:ee86f6717c42 (57 changesets remaining, ~5 tests)
36 files updated, 0 files merged, 8 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -b 1666
Testing changeset 1630:6e221cda7176 (28 changesets remaining, ~4 tests)
28 files updated, 0 files merged, 2 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -b 1630
Testing changeset 1623:f7b12d17a79b (14 changesets remaining, ~3 tests)
10 files updated, 0 files merged, 0 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -b 1623
Testing changeset 1618:ac9135ec8e99 (9 changesets remaining, ~3 tests)
17 files updated, 0 files merged, 0 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -g 1618
Testing changeset 1620:032f83fb6b8c (5 changesets remaining, ~2 tests)
26 files updated, 0 files merged, 1 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -g 1620
Testing changeset 1621:1d8191199d0d (3 changesets remaining, ~1 tests)
5 files updated, 0 files merged, 0 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -g 1621
Testing changeset 1622:65b4f19e8941 (2 changesets remaining, ~1 tests)
9 files updated, 0 files merged, 0 files removed, 0 files unresolved
vincentliu@vm1:~/replicode$ hg bisect -g 1622
The first bad revision is:
changeset:   1623:f7b12d17a79b
parent:      1618:ac9135ec8e99
parent:      1622:65b4f19e8941
user:        Anonymous Person <xxx@xxx.xxx>
date:        Fri Aug 07 14:13:56 2009 +0100
summary:     Automated merge with http://xxx

Some of the output has been slightly modified to protect the innocent[1].

The first two bisect commands, with the -g and -b arguments indicate the two changesets that you know for certain which are good and bad respectively. With the boundaries set, Mercurial goes on its way and starts checking out changesets that are committed in between.

For each checked out changeset, you'll have the opportunity of testing it out; once you are done, you can indicate whether if the changeset is good or bad by using the same commands with the checked out changeset number. Mercurial then checks out the next changeset for you to test.

Here's where there is some serious voodoo at work; notice that I had 57 changesets in between. It's going to be a nightmare if I had to test out every single one of them - instead, Mercurial tries to subdivide through the changeset smartly to isolate the problem (hence the term 'bisect') and gives you a general estimate of how many more tests you have to do before Mercurial can definitively point out the problematic changeset. In my case, it took me 8 tries. Pretty impressive eh?

I certainly have my few criticisms about Mercurial, like my initial incredulity of how the developers came up with the command of hg when their software system is called 'Mercurial'; but past those kinks and initial gag-reflexes, I have to admit that my experiences using has largely remained satisfactory. And if you're one of those who are still at two minds about using a distributed revision control system, I urge you to give it a try. You just might like it!

[1] It's not nice to put the names of the people I work with on my blog without their consent.

The Recession is Over

2009-08-03T04:11:00.000-07:00

I'm no economist and I certainly did no research. I have neither volumes of literature, nor data or statistics to back me up. But rather than listening to the talking heads on CNBC and Bloomsberg, why not try one of my observations for a change?

I got reminded that we are in a recession by a French flutist at a gig I attended last night. If news has reached as far as to a band of travelling hippie-musicians, surely we must have lived through the worst of it all already?

You know the adage that it is time to get out of the markets when cab drivers are tipping you on stocks? Well, maybe this is just the same, only in reverse.

Eat your heart out Roubini, it's time for a celebration, Champagne anyone?

Java is not the JVM

2009-07-22T15:14:00.000-07:00

For many IT people, it sounds funny to assert that the Java language has nothing to with the JVM itself. But as incredulous as it sounds, this is actually true. Let me explain, using some code as a shallow illustration how this is the case.

When I was hacking at the Java bytecode level, one of the things that I do is to optimise for memory efficiency. There is a need for storing an array of booleans, and the most obvious way of saving memory is to store it at a bitwise level, by stashing 8 boolean values within a byte.

Within the JVM, booleans are stored as bytes (executionally, they are worse: the VM treats booleans as ints!). Furthermore, in Java, there isn't a low-level means of utilising booleans as integral types like C can. If you had to write code in pure Java, at best you'll end up writing code like this:

// assume z == boolean[8]
byte b = 0;
for ( int i=0; i < 8; i++ ) {
  if  ( z[i] == true ) {
  b |= ( 1 << i );
}
}

Unlike C, the code is clunky, as you are having to perform a conditional check on a boolean, before you can perform bitwise operations on the values, because Java considers booleans as a non-integral type. How annoying!

But this constrain only affects the Java language - the same rules do not apply when it comes to the JVM. On the VM, it is perfectly legit for you to express code like this:

// assume z == boolean[8]
byte b = 0;
for ( int i=0; i < 8; i++ ) {
  b |= z[i] << i;
}

However, just about any Java compiler disallows this code to compile - the operations on the boolean violates type-safety. But don't blame the compilers, they are just conforming to the language specifications. But since the JVM has nothing to do with the Java language, there is nothing illegal in doing so outside the Java language, let say by using bytecode assembly. Here's an equivalent, using jasmin assembly code:

.source BooleanToByte.j
.class BooleanToByte
.super java/lang/Object

.method public static main([Ljava/lang/String;)V
.limit stack 4
.limit locals 3

iconst_0
istore_1      ; byte b = 0;

iconst_0
istore_2      ; int i = 0;

LOOP:

iload_2
bipush 8
if_icmpge EXIT_LOOP:       ; if i>=z.length exit loop

; here's the magic code that allows you to do direct
; bitwise b |= z[i] << b="">
iload_1
aload_0
iload_2
baload
iload_2
ishl
ior
istore_1

iinc 2 1
goto LOOP:

LOOP_EXIT:

return

.end method

The jasmin code will probably assemble, but don't expect the JVM to execute it; it serves only as an example, and lacks a few things (I'm missing the constructor block and other nitty gritty little things that's needed to satisfy the bytecode verifier). It is but a case study to separate the JVM from the Java language as people typically assume.

There has been a number of other languages that has since mushroomed which relies on the JVM as its core; these languages include Groovy, Scala, Jython and JRuby, many of which are rather interesting, although they are more of a curiosity at this stage - I've yet to see any of these implementations deployed in a production environment, although I don't say that as a criticism of any of these languages. In fact, I am actually quite impressed with the JRuby, and I recommend you give it a try. It's very faithful to the actual Ruby implementation and allows you to use Java directly. Good fun, I'd say, especially when it combines the expressive of the former with the features of the latter. It's quite impressive that the JVM has been able to be so versatile in allowing other languages to plug into it directly.

GDB: Relaying Trapped OS Signals

2009-07-22T12:47:00.000-07:00

By chance, I have managed to land myself in a situation where the bug occurs only at a signal handler. In my situation, this means that it happens only when I try to kill the program using a SIGINT, or more commonly known as the 'ctrl-c' keystroke.

GDB usually traps this signal, and other signals such as SIGSEGV (Segmentation Fault), so that you can trace buggy behaviour that is causing your application to fail. But once in a while, the error may occur after the signal is sent, when the code failure resides within the signal handler.

However the default behaviour of GDB is to trap these signals and subsequently consume them, in effect preventing the bug from occurring. To prevent unwanted behaviour in rare cases such as mine, you need to issue the following command:


(gdb) handle SIGINT pass

Given that GDB utilises this signal internally, it will ask you for a confirmation to change it. Say 'y' to it, and GDB will correspondingly pass the signal to the application after trapping, which will give you a chance to debug the handler code that is causing the bug.

GDB's Conditional Breakpoints

2009-07-19T03:14:00.000-07:00

Conditional Breakpoints for Scalar Types

Let's assume that you, the brilliant hacker, has coded up some really uber-cool stuffs, like this piece of code below:


1: for ( int i = 0; i < gazillion; i++ ) {
2:   doSlightlyBuggyButUberCoolStuffs(i)
3: }
4:
5: void doSlightlyBuggyButUberCoolStuffs(int i) {
6:   // your code here that needs some
7:   // fixing before it becomes uber-cool
8: }

It is doing all the cool stuffs as intended, but somehow something always goes wrong when the code executes up to 2147483648, which is kind of puzzling.

So what to do?

You may be tempted to breakpoint at line 5, at the start of the doSlightlyBuggyButUberCoolStuffs():


(gdb) br doSlightlyBuggyButUberCoolStuffs

And gdb dutifully does what it's told; every single time doSlightlyBuggyButUberCoolStuffs() gets executed, it stops and waits for you to act on it:


Breakpoint 1, doBuggyButUberCoolStuffs (i=1) at test.cpp:6
6:        // start of your uber-cool code
(gdb) c

Breakpoint 1, doBuggyButUberCoolStuffs (i=2) at test.cpp:6
6:        // start of your uber-cool code
(gdb) c

.....

Breakpoint 1, doBuggyButUberCoolStuffs (i=100) at test.cpp:6
6:        // start of your uber-cool code
(gdb) c

After 100 iterations, you think you've had enough! So it's time to do it the smart way, by setting a conditional:


(gdb) br test.cpp:2
Breakpoint 1 at 0x1234: file test.cpp, line 2.
(gdb) cond 1 i==2147483648
(gdb) run

After the breakpoint is set, gdb only notifies you when the loop is at its 2147483648th iteration:


Breakpoint 1 at 0x5678: file test.cpp:2
2:   doBuggyButUberCoolStuffs(i)
(gdb) s
6:         // start of your uber-cool code
(gdb) p i
$1 = 2147483648

Jackpot! You're now at the 2147483648th iteration! And very soon after, you found the offending piece of code, caused by a numerical overflow of a signed integer. Another bug trampled, and peace returns to your realm once more.

Conditional Breakpoints for char* Strings
But very soon after, you run into another irritating problem which is happening within another section of your uber-cool code. This time, the conditional depends on parsing a huge portion of text that comes from, um..., /dev/random :P


1: while ( true ) {
2:   char* c = getStringFromDevRandom();
3:   launchNuclearMissileIfCodeMatch(c);
4:}

Somehow, you are absolutely convinced that /dev/random will eventually provide correct codes to launch the nuclear missile, but given that launchNuclearMissileIfCodeMatch() is a really top-secret and highly obfuscated code residing in an external library called libtopsecret.so, it isn't such a good idea to debug into the call unless you want the NSA bursting through your front doors...

But since you do know the launch code (it's one of those things that you'll have to kill your friends if you ever told them), you can perform a conditional check on the string, and breakpoint at it to tell you if the secret code is ever generated by /dev/random to find out if launchNuclearMissileIfCodeMatch() is really a hoax:


(gdb) br test.cpp:3
Breakpoint 1 at 0xdeadbabe: file test.cpp, line 3.
(gdb) set $secret_code = "MyUberSecretivePassword"
(gdb) cond 1 strcmp ( $secret_code, c ) == 0
(gdb) run

And then, you let your code run... (!)

Well, unfortunately, you get sick of sitting around and waiting for it to happen after a whole day. It seems like /dev/random doesn't really generate your uber-secret nuclear launch codes as frequently as you would like to think. In the meantime, the world thanks their lucky stars that you haven't caused a nuclear winter to materialise just yet... :)

How to lose weight and be healthy

2009-07-11T09:50:00.000-07:00

I have little patience with people who complain about their weight problems, and for a good reason - I have a really simple way for losing weight which does not involve going on a crazy exercise binge or starving yourself silly. Sounds miraculous? Well, read on.

The easy part that you already know, is to make some wise choices about your diet, exercise moderately, and be consistent at both. Yes, it is both sensible and achievable.

The second part is usually what squeezes the life out of one, when I say:

"Stop eating chocolates, cake and ice-cream!"

My advice makes me sound like I'm a Nazi Dictator violating their 8th Amendment rights. But no, and let me reassert this again, it is that simple.

I'm deadly serious.

Chocolates, cake and Ice-cream belong to a broader category I classify as 'leisure food'. It's leisure because people eat these food for their own enjoyment, not for sustenance. This applies to all other snacks (cookies, biscuits, sweets, chips and flavoured drinks) - if that is what you are usually consuming when you're not hungry, then you shouldn't be having it.

"But, but... that's impossible!"

I'll tell you a story about a friend of mine. Let's call him 'Bryan'. (Oh, that is your real name! Let's see how long before someone points you to read this entry :P) Bryan has a minor weight issue, and he once told me about this amazing story that his household fridge was like The Bag of Holding; each time he opens it, there will be a slice of chocolate cake or other sweet delights waiting for him. No surprises to why he isn't able to keep his hands off the sugary stuffs.

Now let me tell you another story of another friend of mine. Let's call him 'Vincent'. (Oh hey, that's me!) Vincent grew up in a poor household where there's no cake awaiting for him in the fridge any time he opens it. In fact, there's nothing there for him whenever he opens it. Poor Vincent! You might even begin think that his childhood is real deprived!

On the contrary, Vincent had never been hungry in his life; there is always food on the table, and the meals involve a simple serving of rice, vegetables, and either eggs, chicken, fish or other protein sources. Given the tight budget Vincent's mum had, there wasn't enough money to buy chocolates, snacks and fizzy drinks anyway. 20 years later, Vincent do not have much cravings for these 'leisure foods' . To others, he just seems to have an incredible self-control.

No, this isn't a tit-for-tat competition here, all I'm saying is that good moderate choices makes harder things easier in the long run.

So what do you do if you grew up in the life of Bryan?

Start small, and be consistent. Consistency is the key, my friend. In the long run, it will get easier for you. Just don't get into the delusion of wanting instant results.

There is this one other issue, like what if:

"I get hungry really easily!"

That is why you should stop eating chocolates, cake and ice-cream! And eat more meat!

Some of you might be puzzled. More meat? Let me explain why in my own personal, anecdotal example. If you ever read food labels, you'll find that a 40g bar of Cadbury's chocolate contains 205 Calories of energy. In comparison, 125g of chicken breast contains only 133 Calories of energy!

I'll answer how these number translate to in human terms. If I have more than a good portion of chicken for lunch, say 250g, I'll have consumed 266 Calories and it will keep me from being hungry till dinner. In comparison, if I had two chocolate bars for lunch, I would have consumed 410 Calories of energy AND I'll be hungry again 2 hours later!

I'm aware that there are some scientific studies out there that supports my claim, but what's more important, is that I know it works for me. This I want you to keep in mind. I certainly can tell that something within meat (or protein, I suspect) keeps one from feeling hungry sooner. Carbohydrates just don't do the same thing. It's no wonder some people swear by faddish regimes like the Atkin's Diet.

But I do not recommend anybody from trying funny diet plans, or simply going on diets at all. Firstly, if you are starving yourself, you are doing it wrong. Because it'll never work - primal instinct will always override any artificial discipline that you try inflicting on yourself. Secondly, diets promote deprivation, and the deprivation of essential nutrients to your body is certainly always harmful. All I'm saying, is to cut down on unnecessary leisure food carbohydrates, and have more meat if your body reacts to hunger the same way as I do. In addition, do stick to what the doctors recommend with regards to healthy servings of fruits and vegetables.

Eating less is good for you

Recently, it's been proven that caloric restriction will improve health and extend longevity. And this includes us primates, and not just in lab mice. If you're too lazy to read the links, here's a visual difference between two monkeys where one is on caloric restriction and the other isn't. Guess which one is which?

If you're cringing in horror about how miserable the rest of your life will be starving yourself, think again. You'll be surprised how it does not involve starving. There's no contradiction, and I'll use myself as an example. For the numbers to make sense, we need to know how much Calories I need - and there are plenty of calculators out there that will help to figure this out, Google it.

Now, for a 30 year-old, 1.75m, 65kg guy, my Caloric requirement is about +/- 2500 Calories, depending on which calculators used. A 25% Caloric reduction is what they've used in the scientific studies, so that brings my figure to 1875 Calories. Now, lets calculate what I eat on a given day, which includes breakfast, lunch and dinner with no snacks.

Breakfast
4 slices of bread, 68g - 200 Calories
Nutella, 18g spread! - 98 Calories
Tea - 2 Calories

Lunch
200g Chicken - 212 Calories
250g Rice - 867 Calories

Dinner
300g Salmon - 447 Calories
100g Broccoli - 35 Calories
133g Apple - 65 Calories

Total Calorie Count : 1926 Calories

It's slightly more than 1875, but it's not too far off range. And I'm not rationing myself either - I've given generous estimates for the spreads (18g spreads to 67g of bread!), and dinner is 2 portions of Salmon (supermarkets estimate of portion sizes are ridiculous!). Give and take, that is what I normally have without having to feel hungry at all, it's quite amazing to know that I'm still within their bounds of 'Caloric Reduction'!

If you've noticed, it is the carbohydrate foods that are giving an amazingly high Caloric count - look at rice at 867 Calories at 250g! And I never knew this until I looked this up myself!

More chicken from now on, please!

Supersized Nations

Being fat is a peculiar problem that exists only in industralised nations. No surprises, since that is where all the money is. Indulgence is an economic problem if you actually think about it - it only afflicts people who can afford it. I'm just 'lucky' I grew up in a circumstance where I do not have a luxury to indulge in leisure foods, although this doesn't make the make me the model case study for solving the fat nations problem.

It is really not a problem if you understand this is how capitalism works. Businesses are incentivised to create things you want, and for food companies, to create foods that are tasty, so that you'll buy them. Tasty foods means that you'll want to have more. People often cite the reasons of growing fat on the an abundance of food and the general lack of exercise in the modern world - it is true, but it's no more correct to blame it on the companies who make tasty foods, than to attribute blame on themselves. If food companies are predatory to the addiction of your taste buds, should we go after them like we do on hard drugs?

I like capitalism - it is your prerogative to see how best to spend your money, so it is within your rights to eat yourself silly, and then spend a fortune on treatment - that is a personal choice.

But it irks me to see how governments are socialising these costs at the expense of sensible people - it's just one of the things in our society I like to describe as 'mad'. If it's 'too big to fail', nobody should have allowed it to grow that big in the first place. So far, governments are just plain inept at dealing with both supersized companies and supersized people.

So, do not bring weight loss problems as a conversation to my dinner table. And if anybody does have the temerity to raise it, be prepared for a session of ridicule - and if he or she takes it in good humour, maybe I'll refer them to what I have to say on this subject matter.

Scottish Whisky

2009-07-03T15:25:00.001-07:00

Feels like you've drunk a cigar.

I ♥ Amsterdam!

2009-06-24T03:14:00.000-07:00

First Stop, Brussels. Ok, you must be scratching your head now, and asking "Brussels? Like did you read your maps upside-down and got lost?" Well, the truth can't be any simpler: it is cheaper if I fly into Charleroi and drove to Amsterdam! Seeing two cities for the price of one, nice!

That's the town centre of Brussels, gorgeous!

In one part of the city, there's a wall with the mural of Tin Tin. There's quite a bit of Tin Tin stuffs here, no surprises, given this the birthplace of its author, Herge.

Belgium waffles! Quite delectable, I had the one covered with strawberries and laced with chocolate, mmmmh! And it was free! Not from the shop, but from a new friend of mine that I've met while travelling out from Dublin airport, a fellow Singaporean traveller, which is as rare as hen's teeth! Surprisingly, he was from Singapore's Ministry of Foreign Affairs, and remarked that I'm quite possibly the only Singaporean living here! I feel so special already! :)

The architecture in the town centre is just incredible. Look at all the life-like statutes that are part of the walls of the buildings. For the lack of time, since I had a long drive ahead, that's all from Brussels. After which I had to take off and drive down to my newest favourite city, Amsterdam!

My new friend had actually spent a few days in Amsterdam already, so essentially I've gotten a free guide to show me around, which was really cool. It's really a quirk of fate, given that most Singaporeans may gladly give Ireland a pass, but not him - being an avid reader, and a fan of James Joyce, which was the main reason why we got to cross paths.

Weed! You can now understand why I find this place so charming - not the soft drugs if that's what you're thinking about! It's the liberalism, dammit! People are free to do whatever they want here, but take the responsibility for their actions, of course. Still can't believe that I'm travelling with a ranked civil servant here - hopefully he picks up a few lessons and report to his political masters that it is fine to have more liberalism! There aren't any major law and order problems here just because soft drugs are available - proves that we certainly don't need to hang people just because they have a bit of grass in their pockets. Well, even Barack Obama has inhaled it ... had he been living in Singapore rather than America, they would have snuffed the life out of him before he can ever live up his potential to become the 44th President of the United States.

Sorry guys, if you are looking for NSFW pictures - for consolation, you can see that I'm in the 'red light district' of Amsterdam. There's an Erotic Museum where I was told you'll be able to learn all you need to know about the history of sex. Fortunately, or unfortunately, Amsterdam has cleaned up quite a fair bit the last couple of years - I was told that half the city centre used to be areas where you can see skimpily dressed women behind glass boxes. Today, it's reduced to a only small section within one alley - and there isn't much to see as well. Hell, there's probably more naked bodies lying on Bondi Beach (NSFW!) any given summer day than in Amsterdam!

The only nakedness I've witnessed was from a drunken Englishman who had decided that his penis was probably a better show than all the ladies there. The crowd were wild, clapping, cheering and cat-whistling while he's twirling his thing out of his pants and flashing it to the crowd. On a related note, the booth you see up there is a peeing booth - yes, you go in and pee on the side of the streets, pretty visible to everybody else. No prizes for guessing what that dark patch on the floor is. ;)

Amsterdam is full of canals and quaint little houses, some tilted with age, as you can see up there. Quite charming, although I wouldn't want to be living in there if there's a serious risk of it toppling over.

That's how beautiful Amsterdam is. Oh, yeah you can see that it's devoid of people in the picture, which is kind of cool - that's because I woke up at 5am on a Sunday morning to take a wander about. The only people left on the streets were the few revellers who were probably too drunk to know how to go home.

I'll leave you with a final picture of one of the canals of Amsterdam. It's absolutely gorgeous for a place, and certainly one of the best cities I've been to in my life, and if you ever have the opportunity to visit it, go - I assure you won't regret it!

Configuring your Linux Firewall using iptables

2009-06-21T03:14:00.000-07:00

When I first started out using Linux, I was quite daunted by 'iptables', the built-in firewall that is bundled with the Linux kernel. Given there is a general misconception from a lot of people's that configuring it is anything but easy has also compounded towards my reluctance to try to learn it in detail initially - but no surprises here, as the good tutorial I've referenced has 16 chapters and 10 appendices! It's little wonder why some people might be scared away by that.

But there is a good reason why a tutorial about iptables is that big - computer security is all about the details, most of the time you know all the details on the different aspects of network security to understand the whole picture before you can design a comprehensive firewall that provides all features you want without letting malicious traffic through.

Still, if you're just setting up a simple home network + firewall, it shouldn't be that difficult. And it isn't really. I'll show you a few recipes you can use to set things up properly without having too much RTFM.

For illustration, I'll use the following setup that I'm running at home as an example:

My server is an old Celeron PC which acts as the firewall. It has an ethernet card which connects to a wireless switch where the Internet connection gets shared by all my laptops connected to My LAN. How the server connects to the Internet is via my Huawei E220 broadband modem. It just convenient to have my configuration this way as well, since my old iBookG4 has no suitable drivers. The broadband device is recognized as ppp0 as shown in the diagram above. Let me now show you a few interesting things you can do with your 'iptables' firewall.

Recipe #1 Forward Internet Connections using IP Masquerading
You want to let your LAN make connections from the Internet. This is one of the cool features that iptables provide that makes it more than just a firewall. Before you make changes to your firewall entries, you'll need to make some changes to your kernel's configuration to enable it to forward IP traffic. To do this dynamically, run the following command:


echo 1 > /proc/sys/net/ipv4/ip_forward

The changes you've made above will be lost the next time you reset your computer. To make this change permanent, you have to make changes to /etc/sysctl.conf to include the following line:


net.ipv4.ip_forward=1

Once that's set up, we can issue the commands to iptables to start forwarding traffic from the LAN to the Internet:


iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o ppp0 -j ACCEPT
iptables -A FORWARD -i ppp0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

The formal name for forwarding network traffic is called 'Network Address Translation', or NAT for short. This explains why the first iptables command has nat in it. It basically instructs the firewall to remember the connection that gets forwarded out the Internet. It needs to do this do multiplex different connections from the LAN into a single connection out to the Internet, and then smartly demultiplexes the received data back to the requesters. The next two iptables directives tell the firewall to allow forwarding of packets to the Internet from the LAN, and only allow data packets from the Internet to be sent back to the LAN only if there previously are connections requesting for it. This effectively denies any illegal traffic from coming into the LAN unless computers within it explicitly allows it to.

Recipe #2 Differentiating Traffic between LAN and the Internet
Often, you'll want to assign different rights to traffic from your LAN vs. the Internet. Traffic from your LAN is usually trusted, and hence is within the safe boundary, while Internet traffic is regarded as hostile, hence classified as unsafe. Like the diagram shown in my example above, data from the Internet via device ppp0 is the unsafe network which I'll want to have differentiating rules from my safe LAN network originating from eth0.

Firstly we want to create the two different chains to represent traffic from eth0 and ppp0:


iptables -N ETH0_FILTER
iptables -N PPP0_FILTER

Once the chains are created, we have to tell the main INPUT traffic chain to segregate the traffic between the two networks:


iptables -A INPUT -i eth0 -j ETH0_FILTER
iptables -A INPUT -i ppp0 -j PPP0_FILTER

Once the different changes are linked to the main input, we can now provide rules to treat the different networks separately. For example, if we want to let our LAN network access everything, and only allow SSH traffic from the Internet, we can put rules like these:


iptables -A ETH0_FILTER -j ACCEPT
iptables -A PPP0_FILTER -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A PPPo_FILTER -j DROP

This will drop all other traffic except SSH on ppp0. For other interesting ways of configuring how you want to filter your traffic between the different chains keep reading the remaining tips.

Recipe #3 Logging Suspicious Traffic
How would you know if you are under attack by malicious Internet traffic? Simple, by logging these intrusions. Here's one way of logging these intrusions:


iptables -A PPP0_FILTER -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 --name DEFAULT --rsource -j LOG --log-prefix "DROPPED:"

The example above says that if there are more than 2 consecutive connections from the Internet to my SSH port (22) within the last 60 seconds, then LOG the message with the prefix "DROPPED:". Obviously, this line only logs the connection, what I've omitted is to drop the connection (see Recipe #4 below).

Recipe #4 Rate Limit Spam Traffic
Bots and spammers usually rely on software that repeatedly scan and access your server to try to bruteforce their way in. On machines with a noisy harddisk like mine, the repeated clicking sound is a dead-giveaway (not to mention the annoyance!) So to stop from them from repeatedly doing so, we enact a rule that drops packets if too many incoming new connections are attempted within a short period of time:


iptables -A PPP0_FILTER -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 --name DEFAULT --rsource -j DROP
iptables -A PPP0_FILTER -p tcp -m tcp --dport 22 -j ACCEPT

The first line tells the firewall to log all new incoming connections - if more than 2 new connections are attempted within 60 seconds, then all the remaining connections will be dropped until the 60 second period times out. Given my default policy of my firewall is to drop connections, the second line is included to explicitly ACCEPT the connection, if the first rule does not match (ie, no more than 2 connections are seen within the last 60 seconds).

Recipe #5 Fight Back Spammers By Tarpitting
A tarpit connection is one that delays incoming network connections as long as possible. This technique causes spam connections to slow down, limiting the amount of computers that it can spam. However the iptables version of tarpit is a slightly more advanced variant: it sets the TCP acknowledgement window of the network packet to 1, forcing the sender to generate a full TCP packet per 1 byte of data it tries to send, making it computationally costly for spammers as it saps the computer's CPU resources. If you like to fight back against spammers, then this tip is for you.

To enable tarpitting, this requires you to patch and recompile your kernel, which is an entire post itself, so read my more detailed post on how to enable tarpitting.

Recipe #6 Making Your Firewall Changes Permanent
After making all those nifty changes, it would be a shame if they got lost the next time your computer rebooted. So here's how you can make these firewall settings permanent. Once you are satisfied with all the changes you are making to your firewall, save it by invoking iptables-save:


iptables-save > /etc/iptables.rules

The above command pipes all the configuration into /etc/iptables.rules file. Once you have that, you'll want to restore the configuration every single time your computer starts up. There are quite a few places where you can start restoring the firewall, I do it in my /etc/rc.local file, after my ppp connection is started, where I insert the following line:


iptables-restore < /etc/iptables.rules

And you're all done. Now you can sit back, relax and enjoy the security features of your firewall!

Getting System Information from Linux

2009-06-17T15:14:00.000-07:00

Here are some commands that I commonly use to find information about my system. The amount of information you can get on your computer can be vast and varied - it depends on how detailed you want to go into each of the subsystem on your computer. I'll try to group them in order that is most sensible, and also, note that these commands may be Ubuntu/Debian specific.

Listing devices on your mainboard:

 
biosdecode                 # information about your BIOS
lshw                       # gets quite a bit of information on everything about your CPU
lspci                      # get devices on your PCI bus
lsusb                      # list devices on your USB
dmidecode                  # get device information via BIOS
fdisk -l                   # get partition info on your harddisk

Getting information on your OS:


cat /proc/cpuinfo          # get information about your processor 
cat /proc/meminfo          # shows memory usage
free                       # show available free memory
top                        # detailed memory usage by process
htop                       # a better version of top
lsof                       # shows which file handle is opened by processes
lsmod                      # shows loaded kernel modules
dmesg                      # output bootup information
lsb_release -a             # see which distro of OS you're using
ps -e aux                  # list all running processes
df --si                    # show amount of free disk space
hdparm -t harddisk_device  # show performance of harddisk
ifconfig                   # show network configuration
route                      # show network routing configuration
iwconfig                   # show wireless network information

Ubuntu on iBook G4

2009-06-14T01:23:00.000-07:00

People must think I am going gaga; I have installed Ubuntu on every different CPU architecture I have laid my hands on, and now on my Macbook G4!

Mac Zealots won't be pleased. But, don't you worry - the Mac OSX image is still living somewhere in the system. Unfortunately Ubuntu isn't as efficient in power utilisation as the Mac OSX is on the iBook G4: the machine gets hotter much quicker and you can hear the fan whirring at a much more regular interval.

So I've got Ubuntu/Xubuntu living in various incarnations now; on an UltraSparc, PowerPC, x86 and AMD64 (ok, I've double counted if you consider 64-bit as a variant of the x86 architecture ;)

Before I get labelled an Ubuntu zealot, I need to clear the air a little. I've installed Linux because it has plenty of development tools that a software developer needs; and Ubuntu because it's an easy distro for installation. Still I'm no less impressed by the vast amount of hardware Linux supports.

I certainly think Linux takes the crown for being an ubiquitous OS, in spite of being driven by a purely free software development movement - remember that nobody gets paid to do this, and yet people are generous enough to donate code and effort to make this all happen. The irony in this, is that it is exactly of Linux's free nature that makes supporting so many different hardware possible in the first place.

Related Posts: It's Alive! (Linux on UltraSparc)

Setting up a tarpit on Ubuntu Linux

2009-06-11T07:57:00.000-07:00

It's amazing to see how big botnets can grow up till these days, and they really have plenty of computing power to spare. So what do botnet owners do with these unused free computing power after looting all valuable information from the poor victim? They waste it on scanning on any potential possibilities no matter how minute a chance of finding an opening is.

In the days when computer resources are scarce, computer bots don't bother port scanning addresses when ping requests doesn't provide a response. But not anymore. They know that there are people out there who are slightly more tech-savvy and do not want to be annoyed - so today's bots have no qualms in trying to scan every single port on a network address, even if ping does not respond.

Well, my computer security philosophy is simple: scanning the ports on my computer constitutes as aggression - if you engage in such activity, then it means I am free to retaliate in response to it.

Even so, I do not mean launching an attack on the infected computer; but I'll make your bots waste it's resources by making connections that leads to a dead end. On the flip side, in the process of doing that, this scheme will not waste my own resources by doing it. Typically, an activity like this is termed as 'tarpitting'. So let's see how we can set up a tarpit to fight these bots.

Patching the Kernel

In order to perform tarpitting, we need to rely on Linux's firewall, iptables and the 'tarpit' module. But since the 'tarpit' module on iptables isn't supported on default on Debian/Ubuntu anymore, the only way to enable it is to patch the kernel and recompile it. This may sound daunting to a novice user, but there really isn't a need to; all you need is just some basic knowledge and patience to set things up.

Firstly, a patch to the kernel becomes necessary. It's currently unofficially maintained at http://enterprise.bih.harvard.edu/pub/tarpit-updates/, and marked as being 'unsupported' or 'obsolete' by netfilter team themselves, which essentially means use at your own risk! I'm usually a risk-taker (only when it comes to computer software ;) so it's not a big issue. You should work out if this is right for you.

You'll first need to download the kernel sources, and set up the corresponding environment for recompiling your kernel. A detailed step-by-step procedure is provided in the Ubuntu Wiki. I'm just going to skim through the details from the wiki, and show you the commands that is relevant for version Ubuntu Intrepid:

% apt-get install linux-kernel-devel fakeroot build-essential makedumpfile
% apt-get install build-dep linux
% apt-get source linux-source

Now you need to find out what version of the kernel you're running before you can download and apply the corresponding patch. The version is shown as the directory name of the source you've downloaded, eg:

% ls -l /usr/src/
linux-source-2.6.27

What we are interested is the number indicated in bold. In my case, it's 2.6.27. We need to do a few things here: firstly we want to inherit all the old configuration that came with your currently working kernel, so that the newly compiled kernel will be the same as the original. Then we can download the patch and apply it to the linux source, so that only change is the addition of the tarpit feature:

% cd /usr/src/linux-source-2.6.27
% make oldconfig
% wget http://enterprise.bih.harvard.edu/pub/tarpit-updates/tarpit-2.6.27.patch
% patch -p1 < tarpit-2.6.27.patch

The patch should apply cleanly, which means now you have the tarpit feature in the kernel. But that's not enough, you need to make sure tarpit is compiled, as a module generally. To do this run:

% make menuconfig

And select 'M' from the menu options

Networking Support -> Network packet filtering framework (Netfilter) ->Core Netfilter Configuration -> "TARPIT" target support

.

Compile Time!

This is when you need to sit back, go make yourself a cup of coffee, and be patient. On my 500Mhz Celeron box, it took about 6-8 hours of compilation time on a Saturday morning. Essentially, I just left it compiling while I went out to enjoy a bit of the sunshine - you should too, especially if you are compiling on a slow computer like me.

There really isn't anything too exciting watching a computer churn out code, kind of like watching grass grow. :)

Issue the following commands to start the compiling process, and then wait:

make-kpkg clean # only needed if you want to do a "clean" build
fakeroot make-kpkg --initrd --append-to-version=-tarpit kernel-image kernel-headers

If Ubuntu complains about not finding make-kpkg, then you may have to install 'kernel-package' (apt-get install kernel-package). This will start off the compilation. Once you've completed there should be 2 Debian packages resulting from the compilation. All that's left to do is to install them:

% ls *.deb
linux-headers-2.6.27.18-tarpit_2.6.27.18-tarpit-10.00.Custom_i386.deb
linux-image-2.6.27.18-tarpit_2.6.27.18-tarpit-10.00.Custom_i386.deb

% dpkg -i linux-image-2.6.27.18-tarpit_2.6.27.18-tarpit-10.00.Custom_i386.deb
% dpkg -i linux-headers-2.6.27.18-tarpit_2.6.27.18-tarpit-10.00.Custom_i386.deb

The installer will make modifications to the boot loader (usually GRUB these days), and adds two new entries into your boot menu. If you haven't made any customised changes to it, usually the installation process will not require any intervention and should complete automatically.

Reboot your computer and you're set for setting a tarpit up!

Configuring 'iptables' for Tarpitting

To utilise tarpit, you need to configure the rules on your firewall (iptables) to tarpit on incoming connections. There are plenty of excellent tutorials out there explaining how to use iptables to achieve what you want to do with your firewall, and it's beyond the scope of my entry to cover it all here. I'll just give a few simple examples on how you can use it to waste the resources of bots and spammers.

To tarpit SMTP connections (assuming that you are not running an SMTP server):

iptables -A INPUT -p tcp -m tcp --dport 25 -j TARPIT

To tarpit incoming botnet bruteforce attacks on SSH:

iptables -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --rsource
iptables -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 --rsource -j TARPIT

The example limits SSH attempts to 2 connections in 60 seconds. And if any connection tries to connect at a rate higher than that, then the connection is sent to the tarpit immediately. My actual configuration is even more stringent than that; given that my SSH connections are verified by keys and not by password, there is never a chance that I could have sent a wrong password and hence tarpitting myself. For an average user who accidentally connects to my server, it isn't really too much of a problem - the connection will eventually time out.

But lets see what happens when a spambot tries to connect repeatedly. I'll simulate this by using nc to act as a spammer. Let see what happens when I set the rule to just DROP:

# iptables -I INPUT 1 -p tcp -m tcp --dport 25 -j DROP
# nc localhost 25
^C
# nc localhost 25
^C
# nc localhost 25
^C
# nc localhost 25
^C
# nc localhost 25
^C
# netstat -apn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4227/sshd

DROP just does what it's told; that is to drop the packet, and that's the end of the story. The spambot will just shrug its shoulders and move on to find another spamming target. But see what happens when when we turn tarpitting on:

# iptables -D INPUT 1
# iptables -I INPUT 1 -p tcp -m tcp --dport 25 -j TARPIT
# nc localhost 25
^C
# nc localhost 25
^C
# nc localhost 25
^C
# nc localhost 25
^C
# nc localhost 25
^C
# netstat -apn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4227/sshd   
tcp        0      1 127.0.0.1:36183         127.0.0.1:25            FIN_WAIT1   -           
tcp        0      1 127.0.0.1:36185         127.0.0.1:25            FIN_WAIT1   -           
tcp        0      1 127.0.0.1:36184         127.0.0.1:25            FIN_WAIT1   -           
tcp        0      1 127.0.0.1:36181         127.0.0.1:25            FIN_WAIT1   -           
tcp        0      1 127.0.0.1:36182         127.0.0.1:25            FIN_WAIT1   -

As you can see, the connections are stuck in the FIN_WAIT1 state, waiting for socket time outs to occur. So tarpitting works like a reverse syn-flood attack, but in this case the 'damage' is self-inflicted - the more aggressive a spambot is in trying to make a connection to us, the more it gets its resources exhausted. This helps to use up the computing resources of the spam computer, and engaging it in unproductive activities, thus preventing it from spamming more targets.

What if the 90% of the world Tarpits?

Unfortunately, most spambot code writers have wisen up to these techniques, and correspondingly have adapted their system to make their socket timeouts relatively short, thereby minimising the impact of such a defensive system. However, if most of the computer systems in the world utilises such a system, it will make it prohibitively expensive for spammers to engage in such activities.

But the reality is, the majority of computer users do not understand the implications of this philosophy for it to work out. In fact, tarpitting will have been a good way of deterring most spam without adding more costs to paying customers like us. Imagine if 90% of all the computers are adversarial like this; then spam bots will have been wasting their resources 90% of the time. That should make the economics of spam a bad proposition to spammers, rather than the reverse situation we are having today - the majority of spam is handled by ISP's filtering, wasting 90% of the Internet's email traffic on spam, annoying email users, and charging consumers money to take away the problem.

If you haven't noticed it yet, in essence, we are indirectly paying for the costs these spammers incur. And that pisses me off.

As a parting note to my post, I hate all spammers with a passion, so let this be a warning to all link-spammers on my blog - as much as I dislike spammers that I'll tarpit their connections, I do not take kindly to your link spam on my blog. Don't even bother do try, they are screened, and if your comments are just superficial irrelevant stuffs, you can bet your ass that it's never going to see the light of the day! And don't ever let me get my hands on your IP address ... :P

Examining binary files in Linux

2009-06-03T18:42:00.000-07:00

A few different tips assembled together for one to find out information about an executable binary in Linux.

To assert that the file is a binary executable (or some other file types):

file file.bin

To see what the legible strings within the binary file is:

strings file.bin

To do a hexdump of the file:

od -tx1 file.bin

To disassemble a compiled binary:

readelf -b file.bin -m i8086

To disassemble an binary object file:

objdump -DaflSx -b file.bin -m i8086

To list the symbols in an object file:

nm file.bin

To see what shared library it's being linked with:

ldd file.bin

To see a trace of what libraries it calls / files open dynamically:

dtrace file.bin

To debug through it's execution:

gdb file.bin

To unmangle function names if code is compiled with C++:

echo "<mangled_symbol_name>" | c++filt

How to 'make' a Euro / Sterling Key In Linux

2009-06-01T13:21:00.000-07:00

I never had to deal with the problem of handling foreign currency symbols, given that the countries I've lived in the past use the same terminology, where the only difference is the prefixing of their respective country name to the word 'dollar'.

But living in the Eurozone and for being so near to the UK, the idea of expressing money in dollars is relatively quaint experience to them as much 'a quid' is to me. This difference is visibly noticed when it comes to computer keyboards.

Keyboards for Europe with the exception of the UK have their default currency symbols mapped to the '€/£' symbol by default - there are key other layout quirks which make these keyboards infuriating to use but I'll leave them for another day.

Even though I still reflexively swap Euros for Dollars in my daily conversations, at least my 'foreign' accent helps people to contextually frame what I meant, but typing '$' signs when you mean '€' certainly confuses people. My workaround in the past was to type 'Euros' at every instance when I mean currency, which is really becoming tiresome.

So, the impetus aside, here's a quick tutorial to show you how to generate a Euro sign.

First, we need to find out the keycodes of the keys that we want to remap. We do this by invoking 'xev', which traps all keystrokes and mouse movements. The keys we want to trap are the currency symbol, which is usually the same key as the numerical key '4' on the alphabetical side of the keyboard, and the right 'alt' key, which I will use as the special shift key to get € and £ without losing the $ symbol. A capture of xev looks like this:

% xev
KeyPress event, serial 31, synthetic NO, window 0x2800001,
  root 0x6b, subw 0x0, time 2804155, (256,85), root:(807,409),
  state 0x0, keycode 13 (keysym 0x34, 4), same_screen YES,
  XLookupString gives 1 bytes: (34) "4"
  XmbLookupString gives 1 bytes: (34) "4"
  XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x2800001,
  root 0x6b, subw 0x0, time 2804251, (256,85), root:(807,409),
  state 0x0, keycode 13 (keysym 0x34, 4), same_screen YES,
  XLookupString gives 1 bytes: (34) "4"
  XFilterEvent returns: False

KeyPress event, serial 34, synthetic NO, window 0x2800001,
  root 0x6b, subw 0x0, time 2807796, (256,85), root:(807,409),
  state 0x0, keycode 108 (keysym 0xff7e, Alt_R), same_screen YES,
  XLookupString gives 0 bytes:
  XmbLookupString gives 0 bytes:
  XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x2800001,
  root 0x6b, subw 0x0, time 2807933, (256,85), root:(807,409),
  state 0x2000, keycode 108 (keysym 0xff7e, Alt_R), same_screen YES,
  XLookupString gives 0 bytes:
  XFilterEvent returns: False

ClientMessage event, serial 34, synthetic YES, window 0x2800001,
  message_type 0x11a (WM_PROTOCOLS), format 32, message 0x118 (WM_DELETE_WINDOW)

There's a number of other events being truncated so that I'm only showing the relevant portions. The first two keypress/keyrelease set shows the keycode for '4' as 13 and the second set shows that my right 'alt' key has the keycode of 108.

Armed with these numbers, let's create a .xmodmaprc file in your home directory:

keycode 108 = Mode_switch
keycode 13 = 4 dollar EuroSign sterling

Once the file is created, to activate the change immediate, simply issue xmodmap:

% xmodmap ~/.xmodmaprc

And viola*, by pressing 'right alt' + '4' gives me '€' and 'shift' + 'right alt' + '4' gives me '£'!

* Don't even get me started on umlauts and accents ;P

Huawei E220 Modem on Linux

2009-05-29T11:00:00.000-07:00

Unfortunately for mobile broadband Internet connections, network setups are usually difficult and inconsistent experiences between the different network providers. Getting the modem to work can be a rather frustrating experience if things don't work straight out of the box.

Before I start, I'll let you know that some of the settings here may be specific only to my provider, O2 Ireland, which you may have to do your own specific tweaks, and as the saying goes, "your mileage may vary".

On the good side, the Huawei E220 modem seems to be a rather popular and well supported device, and it did on one instance worked straight out of the box on one of my friend's computer running Ubuntu 9.04 with Network Manager. It doesn't seem to work on 8.10, not on my machine (Ubuntu) or my laptop (Xubuntu), which may just boil down to configuration issues, or not. On the funny side, when I tried to get the settings off my friend's computer by right-clicking on it, it simply froze the machine entirely. (Windows users, insert your jibes here ;p)

This problem is probably specific to O2, since I had a Vodafone modem dongle that I had borrowed before, which worked flawlessly on when plugged into my laptop, which I had assumed will be the case when I got O2, but it turned out not to be so.

Update: Found the reason why Network Manager will work in 9.04 but not 8.10, the newer release had included Modem Manager which had specific setup that will request for a PIN which the O2 card was set up in default. By comparison, the Vodafone dongle did not require a PIN, hence network manager worked without a hitch.

Anyhow, maybe these information I've gleaned from will help you find out what you need to get things to work.

What the lights on the modem really means.

Ignore what the documentation says (partially) that came with the subscription. From my personal observation, if the light is green and flashing, it means that the modem is active, just that it's not authenticated to the provider. That should be a sign that your modem is working.

The modem is also capable of flashing in blue colour (which is nowhere explained in the booklet). This means that your connection is authenticated with the provider, but currently is not having ppp connection established.

If the lights are in solid green, blue, or light blue, it means that your connection is active, and in various different operating speeds (GPRS, 3G, HSDPA respectively) as explained in the booklet. From my observations, it seems like the connection typically reside in the 3G mode (dark blue colour) when it's passive, and only switches to HSDPA mode whenever you start sending or receiving data from the network.

Modem doesn't work with Network Manager

A number of sources suggests that the E220 modem works straight with Linux via Network Manager, but it certainly didn't work for me straight off. So I had do some reading on the wireless broadband forums to try to find answers. Most of them are geared towards solving problems for Vodafone, and the information is really spotty when it comes to O2. Given that I had no idea where to look to find out what's actually happening inside Network Manager, I had to try some other alternatives.

Update: I didn't know where to look before, but I've since found that Network Manager logs to /var/log/daemon.log - still the messages are not too helpful to actually tell you what exactly is the problem.

The saviour - wvdial

'wvdial' is the alternative application that got the connection to work after a bit of reading. The documentation on wvdial can be confusing, and even as I've gotten it to work, I still don't fully understand the relationship between wvdial and pppd. Here's the excerpted config I had in '/etc/wvdial.conf', it's a little half-baked, and sometimes still fails:

[Dialer O2]
ISDN = 0
Baud = 460800
Modem = /dev/ttyUSB0
Phone = *99#
Modem Type = Analog Modem
Stupid Mode = 1
Username = gprs
Password = gprs
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2
Init3 = AT+CPIN="1234"
Init5 = AT+CGDCONT=1,"IP","open.internet"

[Dialer O22]
ISDN = 0
Baud = 460800
Modem = /dev/ttyUSB0
Phone = *99#
Modem Type = Analog Modem
Stupid Mode = 1
Username = gprs
Password = gprs
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2
#Init3 = AT+CPIN="1234"
Init5 = AT+CGDCONT=1,"IP","open.internet"

Do replace '1234' with the actual PIN number that you have.

You can see that I have 2 entries for the connection, which the only difference between the two is that the second entry doesn't have a PIN authentication command. This is the intentional, because wvdial does not know whether the modem is in authenticated mode or not (ie, whether if it's flashing blue or flashing green. The ATZ command does not seem to reset this authenticated state.)

If you started your computer fresh and your modem is showing a flashing green light, invoke wvdial this way:

% wvdial O2

That should work if you started your modem cold. Sometimes the first connection gets dropped for no reason, and to try to reestablish the connection, run wvdial with the O22 connection instead. If you succeed, you should see an output like this:

--> WvDial: Internet dialer version 1.60
--> Cannot get information for serial port.
--> Initializing modem.
--> Sending: ATZ
ATZ
OK
--> Sending: ATQ0 V1 E1 S0=0 &C1 &D2
ATQ0 V1 E1 S0=0 &C1 &D2
OK
--> Sending: AT+CGDCONT=1,"IP","open.internet"
AT+CGDCONT=1,"IP","open.internet"
OK
--> Modem initialized.
--> Sending: ATDT*99#
--> Waiting for carrier.
ATDT*99#
CONNECT
--> Carrier detected.  Starting PPP immediately.
--> Starting pppd at Sat May 30 08:20:44 2009
--> Pid of pppd: 7312
--> Using interface ppp0
--> pppd: H�c X�c ��c
--> pppd: H�c X�c ��c
--> pppd: H�c X�c ��c
--> pppd: H�c X�c ��c
--> pppd: H�c X�c ��c
--> pppd: H�c X�c ��c
--> local  IP address 89.204.199.133
--> pppd: H�c X�c ��c
--> remote IP address 10.64.64.64
--> pppd: H�c X�c ��c
--> primary   DNS address 62.40.32.33
--> pppd: H�c X�c ��c
--> secondary DNS address 62.40.32.34
--> pppd: H�c X�c ��c

Trying Network Manager Again

If you have established a connection successfully before, but got dropped for some reason, the light on your modem should be flashing blue. In this case your modem is in the authenticated state, and Network Manager will start working happily if you wanted to use it now.

As I said, Network Manager did work with one of my friend's computer - the difference being that when I tried connecting, his version of Network Manager prompted me to key in my PIN, while my didn't. Even manually setting the PIN in the configuration won't make it work.

But at least it'll work indirectly, and tells us that the problem lies within authentication.

Fun things to do with the modem - AT Commands

Through using wvdial, I realised that the usb modem actually uses a variant of the AT commands of the phone modems I used to have for dialups and BBSes. It kind of piqued my interest a bit, and good for reliving the good old days of fiddling around with AT commands on my modem.

To do so, we'll need to find the interface in which you can send and receive commands to - 'dmesg' will be helpful for these occasions:

[   21.839161] usb-storage: device found at 2
[   21.839163] usb-storage: waiting for device to settle before scanning
[   21.849151] usbserial: USB Serial support registered for GSM modem (1-port)
[   21.849175] option 4-1:1.0: GSM modem (1-port) converter detected
[   21.849429] usb 4-1: GSM modem (1-port) converter now attached to ttyUSB0
[   21.849441] option 4-1:1.1: GSM modem (1-port) converter detected
[   21.849519] usb 4-1: GSM modem (1-port) converter now attached to ttyUSB1
[   21.849535] usbcore: registered new interface driver option
[   21.849538] option: USB Driver for GSM modems: v0.7.2

So, /dev/ttyUSB0 is the interface in which Network Manager/wvdial uses to connect to the mobile phone provider, which kind of perplexed me why there is an additional /dev/ttyUSB1 interface. One of the things that came up from googling was an out-of-date kernel support page for the modem.

It provided a tool to read the signal strength of the modem, which out of curiosity, I downloaded the source code and waded through it. That's when I realised that /dev/ttyUSB1 is the interface in which you can issue AT commands to.

Armed with that knowledge, we can now start issuing commands straight into the device! Relying on a primitive method, do this by starting two terminal windows side by side. On one window, do:

$ cat /dev/ttyUSB1

This shows you what's the output coming out from the commands issued. On the other window it is where you issue your commands. For example:

$ echo "AT" > /dev/ttyUSB1

You should see "OK" coming out from the other window, showing that the modem has acknowledge your 'attention' command. Pretty cool eh?

Fun Things To Do #1: Disabling PIN Authentication

Remember the problem with PIN authentication that prevented Network Manager from working properly? Well you can side-step the problem by disabling the PIN authentication feature on the SIM card:

echo 'AT+CLCK="SC",0,"1234"' > /dev/ttyUSB1

Replace '1234' with the actual PIN number that you have. This should disable the need for authentication. A word of caution: do this only if you're not too concerned about the physical security of your modem, otherwise if it gets lost or stolen, others can start using your Internet connection for free!

Fun Things To Do #2: Get SMS Messages

For linux users, we aren't provided with any GUI for us to access and send SMS messages from the SIM card. Unfortunately the O2 site registration assumes that we are all Windoze users, which is the only way in which we can pull out the authentication SMS message that it sends to the mobile phone.

Well fret no more, here's how we can gain access to SMS messages simply by using AT commands:

$ echo 'AT+CMGF=1' > /dev/ttyUSB1
$ echo 'AT+CMGL="ALL"' > /dev/ttyUSB1

This should turn on SMS mode on the modem and dump out all the received SMSes. And from the output messages, you can pick out the authentication message that looks like this:

+CMGL: 0,"REC READ","02",,"25/05/27,20:33:07+04"

Welcome to O2 Broadband! Should you have any queries, visit www.o2.ie/broadbandfaq or our interactive forum on http://forums.o2online.ie. Best wishes, O2.

+CMGL: 1,"REC READ","0000000000",,"26/05/28,21:04:54+04"

Your verification code is XXXXXXXX. Please go to o2.ie and continue your registration.

This is at least useful if you don't want the hassle of manually pulling out the SIM card to put in into your phone to get the SMS message for authentication.

More References

The Wikipedia page on the Hayes modem command set has a set of good starter commands on AT commands. Or to read up more on SMS AT write commands or read commands can be found in these links.

Internet, Finally!

2009-05-28T14:30:00.000-07:00

I finally have Internet at home!

After 3 long months of going cold turkey involuntarily. I have finally gotten an Internet connection to my home, via mobile broadband. It's clocking about 300kbps download at best, which isn't great, but at least I can use it to type whimsical blog posts like these in the leisure of my home, rather than having to deal with late nights in the office. With commuting time after hours being unreliable and subjected to long half hour waits, it seriously isn't worth wasting the time just to be able to use the Internet, really.

Surprisingly, the new place that I've moved in does not have a fibre-to-the-curb connection to my apartment - and it's only built 2 years ago! Not only that, the apartment does not even come with a phone line, which means I can't even get ASDL. I can get a technician to get it especially wired to my place, but that means installation charges + line rental for a phone that I do not want in order to get Internet access, making it a rather expensive proposition.

So the only alternative left was to rely on cheap wireless broadband, which at 20 Euros is the cheapest by Irish standards (That's excluding VAT, which is another major bloodsucker in this country :P). That's roughly about $40 dollars AUD/SGD, which I suppose would have gotten me a decent network connection, and without any download restrictions (on a land line)?

Oh well, at least for an el-cheapo bastard like me, it should suffice, since I don't do much that consumes a large amount of bandwidth anyway. 10Gb is quite hard to use up no matter how much blog posts, email or news I do in a day, but obviously it's never quite enough for the YouTube generation.

Which is fine for an 'old fogey' like me - I belong to the '640k ought to be enough for anybody' generation anyway, ;)

A Good Friday It Is!

2009-04-18T11:58:00.000-07:00

Decided that I wanted a reckless, unplanned holiday over the Easter weekend, so I packed my bag with only a toothbrush, a set of clothes, and took off on my bicycle.

Given that I had a 4-day weekend, I had this 'delusion of grandeur' that I'll try to cycle to Waterford and back, which when I recounted my little adventure to my friends later, they all had thought that I was a little 'ambitious'. I'm sure they were just be polite with me not to point out that I was deranged to try to cycle 320km to and back. :)

Without even a map or any plans on how I'm supposed to get there, I cycled aimlessly south, passing through Shankill, Bray then Greystones. I had never been in those places before, and each of them were a lovely little towns of their own, with Bray the biggest, Shankill the smallest, and Greystones just nice. In a way this Goldilocks (yours truly) is certainly tempting fate without even planning for where he's going - it is really not funny trying to tough it out on the streets; there's still plenty of cold in mid-April to try to sleep on the streets when you're in Ireland!

Stopped in each town for a while, and had plenty of soft-serve vanilla ice-cream cones along the way, which was when I noticed this Irish anomaly on their menus called the 'Special 99'. Even after later when I try to figure out they story behind the name by asking my Irish colleagues, nobody seems to know why having a stick of chocolate in the ice-cream makes it a 'Special 99'. Very well then, I'm just going to have to make up my own story, and bluff anybody who asks that an Irishman invented this in '99, which took the whole country by storm which is how the name came about. So take note that you've heard it here first!

Along the way, I met a couple of French backpackers whom I started chatting up with. They were asking for directions and I found out that they are travelling to Glendalough (pronounced 'Glen-da-lock'), where the main attraction there is some old ruins. They were lost, but they had a plan where I had none, so I thought I could give them a hand by trying to look up the place they are going on my Blackberry-cum-GPS, which only gave me a rough idea of where we were, and had no relevance to any of the landmarks that they have shown me on their map. ( A mental note to self: don't ever rely on the Blackberry's GPS to find your way out of the woods if your life depended on it! )

Anyhow, given that I didn't have a plan, and they were quite adamant to go there, I thought they must be on to something, so I decided to abandon my original plan to Waterford via zipping along the coastal fringe, and follow them inland instead.

With hindsight, this had probably turned out to be a pretty good choice.

Given that they were travelling by foot, and didn't have too much luck with hitch-hiking, we decided to camp out in someone's paddock:

Obviously it was a little illegal, given that we're trespassing, but certainly the tripedation made it a little bit more exciting. Claire set up the tent while Alexandre and I started to look around for some wood to start a little campfire of our own. (When I recounted the story to an older guy later, he was curious where I got my firewood from - unbeknown my insanely dumb luck in retrospect, he was surprised that we managed to get some wood for burning; someone had left some strewn around when cutting down some overgrowth.)

Spent the evening camping out with them, to which later they mentioned the fact that I had no place to stay. They had only a two-men tent, and didn't have any room for me, so I had go try my luck when they pointed out that there is a "Bed and Breakfast" somewhere adjacent to the street across the paddock. It just might be as well, to leave them some privacy to have some romantic personal time under the nice Irish moonlight. I think they did enjoy their camp there, but they did try their darn best not to make me feel left out by admitting to me about how cold the night was before, and that I must have had a better time living under a properly heated shelter.

To me, living in a 'Bed & Breakfast' was a rather interesting experience . I had never lived in one before this, and certainly had no knowledge of the existence of such a concept. I had this mistaken idea that it was just another style of commercial hotelier gimmicks where they charge you extra by tagging breakfast to your bill compulsorily. Armed with this half-baked idea in my head, I was kind of worried when I made my approach to the residential property the BnB was signposted, and getting rather concerned that I won't be having a place to stay for the night.

While the place did turn out to be a BnB, my lack of knowledge became even more of a comedy later when I was seated with the hostess in the living room for a rest when I remarked about how much the place resembled a family home, to which she responded in confusion, "but, aren't you looking for a Bed and Breakfast?" She would have gotten into bed that night confused, not only until the next morning would she have found out that I had absolutely no idea what I was in for when confessed that I thought I was going to be living in some funny-styled, breakfast-included hotel.

The next Saturday morning past Good Friday turned out to be another glorious day, with the sun streaking above the country fields. I joined the couple once they got packed up and started on our way again:

Even when one is outside a city, nobody can really escape into what is truly countryside anymore, as a helicopter hovered nearby in mid-air:

Glendalough was still a while away, and the French couple hitched their way there, while I was left to cycle in the vast country road in the early morning. It was an absolute bliss, all nice and serene, which is only irregularly punctured by a car whizzing through once in a while. And then I got to my intended destination:

Glendalough is like a nice national park, certainly reminded me of Yosemite, just a lot less crowded and commercial. If I had my way, I can certainly spend the remainder of my life here - if not only for this damn thing in life called 'work'!

Along the way, I got to know a few more friends - this is Fabio from Italy. I bumped into him along the way cycling. He and his mate happens to be heading towards the same direction as me, so I became the new party in their convoy. It's always fun to meet people from different cultures, you get to learn how very much people are different from you first hand. One of the early things I've come to learn ab0ut the Italians are they're rather fond of appending 'no' after their phrases, when they are actually wanting to mean 'yes' - talk about confusion!

It was my good fortune to bump into them, as they had been instrumental in my return journey. I had no map, and had a lame-GPS that was running out of battery. Since they knew their way, and happen to be cycling their way back to Dublin, I decided to not push my luck any further, and tag along their return journey home.

Along the way back, there's more of the lovely Irish countryside:

Sheep! The damn cutest thing in the whole wide world! Baaaaa!

In the Chinese Almanac, I'm born in the year of the Sheep/Goat/Lamb (what's the difference?), maybe that explains why I have this weird sense of affection towards these cuddly little creatures. :)

For an absolutely unplanned weekend, I must say the outcome has certainly been wildly positive. I can't believe my incredible luck I had for the couple of days - had a few firsts, made a few friends, and enjoyed an incredibly lovely weather for a country more renowned for cloudy, rainy weather - I must have had a 4-leafed clover stuck in the underside of my pants somewhere! So then, all that's left for me to do in this country is to catch some Leprechauns and find the pot of gold at the end of the rainbow! (Apologies to my Irish friends for such corny jokes ;p)

Happiness is ...

2009-03-26T02:10:00.000-07:00

Happiness is a warm steaming bowl of instant noodles
when you are feeling cold and hungry.

Happiness is a warm fuzzy feeling of a full tummy
after a piping hot bowl of instant noodles.

Sometimes happiness isn't too hard to attain after all :)

Reanabling FCKEditor for MoinMoin Wiki

2009-02-13T04:41:00.000-08:00

The Costs of Free Software

FCKEditor, the rich text editor feature in MoinMoin wiki has been removed from the Debian/Ubuntu distributions from a decision the maintainer(s) made after notification of a security issue of enabling it. Obviously I wasn't the only person who got annoyed by its removal, given that the rich text editing feature has been the first place why many, like I, have used the wiki altogether.

So even the 'free' as in 'freedom' in software turns out to be not so free anyway - given that I had no freedom in weighing my own pros and the cons of dealing with the potential security problem. For me, the answer is clear as day - the wiki is only used in a secured local area network and has no editing rights, not even reading rights to anybody outside the intranet - so why should anybody dictate that I can't use FCKEditor with MoinMoin?

(No) thanks to them, this post wouldn't have existed if not for the shortsightedness in destroying the appeal of what otherwise would be an attractive software package. For all that FOSS is worth, it's main Achilles heel is certainly the lack of attention to the needs of end customers. That's why FOSS companies have to make their money out of service - because there is none!

How to Reanable FCKEditor for Debian/Ubuntu's MoinMoin

The workaround to using Debian/Ubuntu's version of MoinMoin is downloading the current version from the website and install it. Converting it to use your existing settings should be a relatively painless task.

Download the package and install a local version of MoinMoin. Follow the same instructions as given on how to install as given in BasicInstallation section. I'll assume that you've installed it in /usr/local/ as was given in the example:

% python setup.py install --prefix='/usr/local' --record=install.log

Using /usr/local will separate the Debian/Ubuntu copy from your self installed copy.

You can then proceed on to installing MoinMoin using the same instructions from Ubuntu 7.04 - the instructions are still valid, but the thing that you have to watch out is to remember to replace all references of /usr/share/ with /usr/local/share.

In addition, you will have to modify the moin.cgi file. In downloaded version, it does not point to the /etc/moin configuration path. In order to reuse your old configuration, add the following line in red into your moin.cgi file:

#sys.path.insert(0, '/path/to/wikiconfigdir')
#sys.path.insert(0, '/path/to/farmconfigdir')
sys.path.insert(0, '/etc/moin')

This will make the wiki start reading from the /etc/moin directory like your Debian/Ubuntu package does.

Debian/Ubuntu has made further changes in the configuration to ensure you don't use the GUI editor, so you have to revert them back by adding or modifying the two lines your /etc/moin/mywiki.py file:

# The GUI WYSISYG editor is not installed with Debian.
# See /usr/share/doc/$(cdbs_curpkg)/README.Debian for more info
editor_force = True
editor_default = 'text'  # internal default, just for completeness

editor_force=False
editor_default='gui'

Once the changes are made, restart apache, and FCKEditor should be reenabled again.

Setting Dual-Head Displays with Radeon HD 3650 in Ubuntu Linux

2009-02-11T06:55:00.000-08:00

I'm using a Sapphire's ATI Radeon HD 3650 card, which may help you if you are using the same thing and want to set up a dual-head display properly.

From my experiences, I found that Ubuntu versions 7.10, 8.04 and 8.10 will get the card to work straight out of the box, but I had no luck to get it to work for dual displays perfectly except for version 8.10 (You'll still need some modifications to xorg though, see below).

For the two older version 7.10 and 8.04, the closest thing that I have come to making it work on the older versions is by downloading and installing ATI's driver straight from their support site.

With the older versions, both the xserver-xorg-fglrx (did not work - black screen) and the xserver-xorg-fglrx-envy (defers to the Mesa driver, making it horridly slow) packages weren't the most fruitful experiences when I tried installing them, so avoid unless you're keen on experimenting.

Installing via the ATI Installer

I'll recommend not to use this option if you have planning to upgrade to Ubuntu 8.10. There are a few quirks with the installer version that will not work well with your window manager if you wanted a rotated screen (See later section).

The steps are straightforward in this case: just execute the downloaded installer and follow the instructions. It's important to note the that the uninstaller is located at /usr/share/ati/fglrx-uninstall.sh, which may become necessary to do to prevent conflicting installations if you decided to apt-get from Ubuntu's repository later.

Upgrading and Installing from Ubuntu 8.10's Repository

These steps are probably not necessary if you've installed Ubuntu 8.10 fresh, but may be essential if you got here through an upgrade path like I did.

The main things you want to apt-get are: xorg-driver-fglrx, jockey-common, jockey-gtk. As a non-dependent package, jockey-common is surprisingly crucial to getting your setup to work, as it contains working versions of amdpcsdb and various important files in /etc/ati.

Once you the packages installed, if you are running gnome, go to 'System -> Administration -> Hardware Drivers' and you should see the ATI Driver available for installation. Click on 'Activate' to download the driver and install. It may take a while before the installation completes.

After the installation, you may be prompted to restart your machine. Before you do, just check that the 'fglrx' driver is mentioned in /etc/X11/xorg.conf. Check that the following line is present in your "Device" Section, eg:

Section "Device"
Identifier    "Configured Video Device"
Driver            "fglrx"
EndSection

Add the line in red into your xorg.conf if it isn't present. Here's an example xorg.conf that may help. Once you are done with the changes, restart your machine.

Checking that fglrx Driver is Running

With luck, your X should be running on your system after the installation. The first thing to do is you check that you have the 'fglrx' driver running. You should get something like this from the output of 'fglrxinfo':

% fglrxinfo
display: :0
OpenGL vendor string: ATI Technologies Inc.
OpenGL renderer string: ATI Radeon HD 3600 Series
OpenGL version string: 2.1.8087 Release

You can also run fgl_glxgears to verify that the card is running correctly visually.

Setting up the Dual Display Configuration

( Update: Compiz does work OK with dual-display, and out of the box as well, just not with rotated-screens. If you don't use rotated screens, this should be fine, so skip the following section. )

The dual-head display set up does not work well with the compositing manager - this is troublesome for Gnome because the default manager is Compiz. You'll have to pass the eye-candy and rely Metacity instead. If you don't, your reconfigured X server either conks out with a backtrace failing on some deprecated calls, or you're greeted with just a wallpaper background or a black screen.

To get access to the settings, you'll need gconf-editor which is an X application, so you should change this setting before you try restarting X in dual-head mode.

Run gconf-editor and change the key value in /desktop/gnome/applications/window_manager/default and /desktop/gnome/applications/window_manager/current from '/usr/bin/compiz' to '/usr/bin/metacity'

Next is to make changes in your xorg.conf file. While ATI provided the default tools like 'aticonfig' and ATI's Catalyst Control Centre ('amdcccle'), I'll recommend you avoid them for now (see Troubleshooting for why).

Here's an excerpt from my xorg.conf as an example:

Section "Device"
Identifier    "ATI Radeon HD 3650 [0]"
Driver         "fglrx"
BusID         "PCI:1:0:0"
Screen       0

# only need to set it one time for a dual-head card
#  Option       "UseFastTLS" "1"
#  Option       "VideoOverlay"    "on"
#  Option       "OpenGLOverlay"    "off"
EndSection

Section "Device"
Identifier    "ATI Radeon HD 3650 [1]"
Driver      "fglrx"
BusID       "PCI:1:0:0"
Screen            1

# Rotation: Not supported as an option directly at the moment
#    Option          "RandRRotation"
#    Option            "Rotate" "CCW"
EndSection

Section "Monitor"
Identifier    "Monitor 0"
EndSection

Section "Screen"
Identifier            "Screen 0"
Monitor              "Monitor 0"
Device                "ATI Radeon HD 3650 [0]"
DefaultDepth     24
SubSection "Display"
   Viewport   0 0
   Depth        24
EndSubSection
Option               "DPMS"
EndSection

Section "Screen"
Identifier           "Screen 1"
Monitor             "Monitor 1"
Device               "ATI Radeon HD 3650 [1]"
DefaultDepth    24
SubSection "Display"
   Viewport   0 0
   Depth        24
EndSubSection
Option               "DPMS"
EndSection

Section "ServerLayout"
Identifier    "Default Layout"
Screen          "Screen 0"
Screen          "Screen 1" RightOf "Screen 0"
EndSection

Section "ServerFlags"
# have to disable xinerama if I want screen rotation
Option        "Xinerama" "off"
EndSection

The full xorg.conf file can be found here. Make the relevant changes, keep your fingers crossed, and restart your computer.

Screen Rotation

If you've managed to get this far, I'd assume that you have your dual head display working by now. If you wanted more fun out of your display, you can try the rotation capabilities provided by randr. The randr extensions are disabled by default; if you tried xrandr without enabling the extensions you're going to get this message:

% xrandr --output default --rotate left
xrandr: output default cannot use rotation "left" reflection "none"

To enable the randr extenstions, use the 'aticonfig' tool:

% aticonfig --set-pcs-str='DDX,EnableRandr12,TRUE'

Restart X or your computer after you made the changes.

While the ATI installer versions works without a hitch on older versions of Ubuntu, rotation is the only case it doesn't work right. For your window manager to detect it's dimensions properly, you'll have to install Ubuntu's version of fglrx, which somehow takes care of the rotation properly compared to the ATI installer version. Rotation also will not work if you are running in Xinerama configuration - you'll end up with areas on your workspace not being accessible by your mouse.

While Gnome allows the option to change its screen rotation from its menu, it doesn't seem to be able to rotate the screen even when the option is set. This means the screen you want rotated will always start landscape every single time it starts up. At least xrandr now can detect all the screens that you have available on your card and perform rotation on it:

% xrandr
Screen 1: minimum 320 x 200, current 1024 x 1280, maximum 1280 x 1280
DFP1 disconnected (normal left inverted right x axis y axis)
DFP2 connected 1024x1280+0+0 left (normal left inverted right x axis y axis) 338mm x 270mm
1280x1024      60.0*+   75.0     75.0     70.0     60.0*
1280x960       60.0     60.0
1152x864       75.0     70.0     60.0
1280x768       59.9
1280x720       60.0
1024x768       75.0     75.0     72.0     70.1     60.0
800x600        72.2     75.0     70.0     60.3     56.2
720x480        60.0
640x480        75.0     72.8     75.0     60.0
640x432        60.0
640x400        75.1     59.9
512x384        60.0     74.9
400x300        75.0     60.7
320x240        75.6     60.0
320x200        75.5     60.1
CRT1 disconnected (normal left inverted right x axis y axis)
CRT2 disconnected (normal left inverted right x axis y axis)
TV disconnected (normal left inverted right x axis y axis)

To do rotation, you'll have to invoke this command every single time you log in:

% xrandr --output DFP2 --rotate left

Replace DFP2 with whatever screen xrandr reports you have.

Troubleshooting

1) There are a few times I had my run-ins with the utility tools ATI provides. Firstly with 'aticonfig'. While it proves useful sometimes, at others, it screws up your xorg.conf configurations. What I'll recommend is that you play around with aticonfig, and copy the relevant parts it generated into your own xorg.conf file, instead of relying on it blindly.

2) The other tool, ATI's Catalyst Control Centre ' - amdcccle', is just pure evil IMHO. The ATI drivers install an important file called '/etc/ati/amdpcsdb', which it loads independently from xorg.conf. If you're finding that X isn't working anymore even with an xorg.conf file you know to have worked previously, it's most likely that the amdpcsdb has been corrupted, which I find that happening from tinkering around with amdcccle, which is probably the biggest time wasted in trying to hunt down spurious problems.

If your card hangs even if Sysreq keys doesn't work no matter what xorg.conf parameters you've changed, the first thing is probably to try to restore the database. The way I got it to work again was by coping over the amdpcsdb.default file to amdpcsdb, eg:

% cp /etc/ati/amdpcsdb.default /etc/ati/amdpcsdb

Do make backups of the original file before you do that, I'm not responsible for any loss or damage it would cause!

3) If X turns into a black screen after you've logged out, try adding or modifying the following line inside /etc/gdm/gdm.conf to true:

# If you are having trouble with using a single server for a long time and want
# GDM to kill/restart the server, turn this on.  On Solaris, this value is
# always true and this configuration setting is ignored.
AlwaysRestartServer=true

Confessions of a 'Ctrl-Alt-Del' Addict

2009-02-09T10:52:00.000-08:00

Growing up in a world where MS-DOS was the first operating system I've ever used certainly leaves me rather brain-damaged when it comes to the 3-finger salute of 'Ctrl-Alt-Del'. That key combination literally is the 'One Keystroke That Rules Them All' - more or less, it's the last thing that will probably still work in the face of a catastrophic computer failure.

Obviously, I never did wean off from my un-elite MS heritage, so it should come as no surprise that I sometimes still use 'Ctrl-Alt-Del' to solve my problems in Linux. Still, it's nice to know that Linux on the whole has always been accommodating enough to include that magic keystroke to satisfy the likes of people like me - most distributions secretly sneak in a line in /etc/inittab (or /etc/event.d for Ubuntu's Upstart) to make it reboot should I fancy myself into fondling those keys when my urge arises, although I must say I have less urges to do so these days ever since I've migrated to Linux, so thanks for asking!

Even when in graphical mode (it's called 'X-Windows' by the way, but you can simply call it 'X' if you want to be a smarty pants, but calling it 'Windows' is plainly an insult!), most window-managers still associate it with some behaviour, like invoking 'xkill' or popup for a shutdown menu. I think some window-managers even invoke the screensaver for that! (I don't even want to go there - that's just plain wierd!)

Shame to say, I've only recently discovered the equivalent of 'Ctrl-Alt-Del' in the Linux world after using it for 8 years!

The magic keys are known as 'System Request' (Sysreq) keys, the all powerful keystrokes a man can invoke (then pray) if your system ever gets f**ked or becomes totally unresponsive. Before you even hit the power switch, try holding down the 'Alt-PrintScreen' keys, and type in the following letters: r-s-e-i-u-b

A good way to remember it is by the phrase:

"Reboot System Even If Utterly Borked"

With luck, that will try to kill all processes gracefully, sync all data to your drives, unmount them, and reboot your system. There's more to the Magic Sysreq keys than I have let on, and more details can be found here.

There are a few other neat tricks with the Sysreq keys, like killing off errant applications that's chewing off too much of your system memory with 'Alt-PrintScreen' + r-f.

A pity that I've only learnt it after such a long while, for it would have been really helpful when I was tearing my hair off a while ago when Rubygems was chewing up all my free memory and thrashing my harddisk from all that silly swapping.

Ubuntu 8.04's Generic Kernel Hangs Machine

2009-02-07T10:31:00.000-08:00

(Updated: To solve your problem, upgrade to Ubuntu 8.10. The newer kernel does not suffer from the problem mentioned below)

If you are using a Vostro 200, this post may apply to you, but bear with my few paragraphs of rant before I give you a fix to the problem. :)

Bloody linux has done it again, borking up my machine during an upgrade cycle. That's always why I'm hesitant on upgrading unless there is no choice. I'll leave the FOSS purists to argue about the concept of 'free' in whatever way they like, but I've always been mindful that there is a hidden cost involved in using Open Source software - you're paying it with your time as a guinea pig for the developers to solve the problems you've encountered.

One way to mitigate that cost is by not using the most cutting edge versions of software - in that way, you're relatively safe from any initial teething issues that still needs to be fixed. But the bad news is, stable/working probably equals to 5 years or more in the FOSS world, and computer hardware moves faster than that.

Given the fast turnover of computer peripherals, no sensible shop will stock 5 year old hardwares to sell it to you, and hence my problems started when I added a new ATI Radeon card to the system. The last known working kernel for the Vostro had been 2.6.22-14-generic, which I had found out a long time ago when I had 'apt-getted' a 2.6.22-16-generic kernel. The obvious quick-fix then was to be make GRUB boot my older kernel, to which I left it at that.

But given that I had installed a new card and needed to install ATI's newest driver, the driver installer, in cahoots with dkms, decided that I wanted a 2.6.22-16-generic kernel module, which obviously screws up X, and hangs my machine whenever the xserver tries to start.

So I'm left with the situation that either the kernel will hang, or X will. Greeeeat.

Since I had been holding off upgrading from Gutsy to Hardy for a long while, I decided to take a gamble to see if the upgrade to a newer kernel would have solved the problem, since it's probably 1 year past the last release cycle. But it turns out to be a real awful mistake - upgrading to 2.6.24-23-generic still screws up the SATA controller, ejecting me to busybox's prompt after giving the following messages:

ata1.00: qc timeout (cmd 0x27)
ata1.00: failed to read native max address (err_mask=0x4)
ata1: failed to recover some devices, retrying in 5 secs

...

ata2.00: qc timeout (cmd 0x27)
ata2.00: failed to read native max address (err_mask=0x4)
ata2: failed to recover some devices, retrying in 5 secs

...

I can imagine Mac users laughing with derision at me right now - 'Get a Mac, it just works!'

After wasting two nights of precious time booting into my old kernel and trying to coax xserver and the fglrx driver to play nice, I decided to revisit the issue of fixing the kernel problem. After a bit of digging, I found the solution in launchpad's bug report: 'Ata Revalidation Failed'. To solve the problem, add the argument 'irqpoll all_generic_ide' into the kernel line in GRUB:

title        Ubuntu 8.04.2, kernel 2.6.24-23-generic
root        (hd0,0)
kernel     /boot/vmlinuz-2.6.24-23-generic root=UUID= ro irqpoll all_generic_ide
initrd        /boot/initrd.img-2.6.24-23-generic
quiet

I don't know what exactly the kernel flag does, but I suppose the SATA controller wouldn't jive with the kernel and needed a jolt of forcible of polling from the kernel for it to work.