Matteo Caprari

The Slow Motion Syndrome

2011-08-05T00:00:01+00:00

Some days it's hard to focus, to-do lists grow long, emails go unread and the ours slowly go by. I call it the Slow Motion Syndrome.

Maybe you have had it too sometimes: you slow down for some reason and slip to a low point from where you can't climb out. Stuff accumulates, catching up becomes difficult and you end up short of breath and nothing much done.

While it's ok to have few slow days, if it goes on for some time, it gets really difficult to shake it off.

I have not found a way to prevent it, but I have developed a strategy to get back on track.

1: Today I'm useless and I have to live with that

Do not start dunking coffee to counter it; no coffee or tea at all is best. Do not set any goals for the day, small or big. Do not even pretend you are working. Stay at home if you can. Walk slowly.

Just accept that today is slow.

2: Get bored

Do not try to kill time. Do not do anything that you don't feel strongly about: no going on random wikipedia pages, no cleaning your inbox, no reading old xkcd comics. No fake work, especially.

Stare at the screen if you have to. I mean it.

3: Do one thing

After some time of total boredom, your brain will be screaming for action and will be piping out with ideas. With some luck you'll remember something that you always wanted to do if you only had time: learn a new language, learn a new programming language, resolve a Rubik cube, start a blog, whatever.

Well, now you have the time, so get to it. With some luck you'll get into it and cycle up to higher activity level. Once you get there, you'll naturally pick up your other stuff.

Do you suffer from SMS? Do you have an exit strategy?

-teo

Getting to know the Java JIT compiler

2011-06-30T19:00:39+00:00

What is the Java JIT compiler? How does it affect my programs? Should I care? I've looked for answers to this questions and this is what I found.

The JIT is in

Most java VMs have a built in Just In Time compiler ("the JIT").

The JIT analyzes the behaviour of a program while it runs and looks for opportunities to optimize the bytecode and re-compile it to machine code using the most appropriate native instruction. All mainstream java VMs have a JIT compiler and you can bet that all your java programs are using it right now.

The JIT is awesome and magical and very opaque and the gains will be different on different machines, VMs and configurations.

The JIT is on by default, but it can be disabled using -Djava.compiler=none

Because it is opaque and unpredictable, it is generally a bad idea to rely on it for the performance of a program. Still, it is interesting to see it at work.

But is it faster?

Let's first work out if this JIT optimization works at all.

I just added -Djava.compiler=none to my eclipse.ini and restarted it. Itfeels slower on "big operations" like clean & rebuild all or searching for the string "this" in all java files in the workspace.

That's good but very difficult to measure.

This is simple class that does some float multiplcations

public static void main(String[] args) {       
    int  count = Integer.parseInt(args[0]);     
    System.out.print("Run " + count + " times with JIT " + args[1] + "...\t");        
    long start = System.nanoTime();
    
    float f = 1f;
    for (int i=1; i<count; i++)
        f *= i;
    
    long end = System.nanoTime();                                        
    System.out.println("completed in " + (end-start) + " nanoseconds ");      
}

And this is a simple script that executes the code alternatively with the JIT ON and OFF:

COUNT=50000000
for i in $(seq 5); do
    echo $i"-----"
    java -cp bin net.caprazzi.WWJD $COUNT ON
    java -Djava.compiler=none -cp bin net.caprazzi.WWJD $COUNT OFF
done

It's immediately clear that with the JIT on, the execution is reliably 50 times faster:

$ sh test.sh
1-----
Run 50000000 times with JIT ON...       completed in 166548543 nanoseconds
Run 50000000 times with JIT OFF...      completed in 8765290860 nanoseconds
2-----
Run 50000000 times with JIT ON...       completed in 161432520 nanoseconds
Run 50000000 times with JIT OFF...      completed in 8775846265 nanoseconds
3-----
Run 50000000 times with JIT ON...       completed in 180772121 nanoseconds
Run 50000000 times with JIT OFF...      completed in 9166550254 nanoseconds
4-----
Run 50000000 times with JIT ON...       completed in 152595716 nanoseconds
Run 50000000 times with JIT OFF...      completed in 8824074862 nanoseconds
5-----
Run 50000000 times with JIT ON...       completed in 157337479 nanoseconds
Run 50000000 times with JIT OFF...      completed in 8738365122 nanoseconds

Cool. Let's change the code a bit and try again:

float[] floats = new float[count];
float f = 1f;
for (int i=1; i < count; i++) {
    f *= i;
    floats[i] = f;
}

Run 50000000 times with JIT ON...       completed in 102871363 nanoseconds
Run 50000000 times with JIT OFF...      completed in 9154895167 nanoseconds

Sweet, here the optimized version is 80 times faster.

Always always faster faster?

This code was tested on an laptop with an intel multicore processor and windows 7 (java version: ~~sun~~ oracle 6.0.26, client VM, 32 bit)

Let's see what happens when we run it on

A scruffy linux virtual box, java OpenJDK 6 client VM 32 bit 6.0.20

Run 50000000 times with JIT ON...       completed in 17990143548 nanoseconds
Run 50000000 times with JIT OFF...      completed in 27172812381 nanoseconds

Boooo - that's just 1.5 times faster!

A different linux virtual box, java oracle 6.0.20, server VM, 64 bit

Run 50000000 times with JIT ON...       completed in 305513000 nanoseconds
Run 50000000 times with JIT OFF...      completed in 2135685000 nanoseconds

Mhhhh - 7 times faster.

A 1997 Mac Pro, oracle 6.0.24, server VM, 64 bit

Run 50000000 times with JIT ON...       completed in 435386000 nanoseconds
Run 50000000 times with JIT OFF...      completed in 1844200000 nanoseconds

That's 4 times faster.

Ok, so there is a definite but very variable performance loss in handling floats and array when turning off the JIT.

Let's try with strings

String s = "0";
for (int i=0; i < count; i++) {
    s += "0";
}
s.charAt(0);

Windows 7, intel cpu, java client VM 6.0.26: no difference

Run 50000 times with JIT ON...  completed in 1859603439 nanoseconds
Run 50000 times with JIT OFF... completed in 1832164432 nanoseconds

linux virtual box, java 6.0.20, server vm 64 bit, 1.4x

Run 50000 times with JIT ON...  completed in 1362850000 nanoseconds
Run 50000 times with JIT OFF... completed in 2003949000 nanoseconds

Magic magic magic

The JIT can speak

-XX:-PrintCompilation is an option to the Oracle's VM supports that makes the jit output some information on what it compiles a method. This allows to see how the JIT intervenes differently on each run of the same program.

If we execute our test 1 time, we see no output:

$ java -XX:+PrintCompilation -cp bin net.caprazzi.WWJD 1 ON
Run 1 times with JIT ON...      completed in 23000 nanoseconds

I've incremented the size of the test loop until at 3250 iterations we start seeing some output:

$ java -XX:+PrintCompilation -cp bin net.caprazzi.WWJD 3250 ON
Run 4000 times with JIT ON...   ---   n   java.lang.System::arraycopy (static)
completed in 40601000 nanoseconds

At 4500 iteration, more output:

    n   java.lang.System::arraycopy (static)
    1   java.lang.Object:: (1 bytes)

At 5000:

    n   java.lang.System::arraycopy (static)
    1       java.lang.Object:: (1 bytes)
    2       java.lang.String::getChars (66 bytes)
    3       java.lang.AbstractStringBuilder::append (60 bytes)
    4       java.lang.StringBuilder::append (8 bytes)

10000:

    n   java.lang.System::arraycopy (static)
    1       java.lang.Object:: (1 bytes)
    2       java.lang.String::getChars (66 bytes)
    3       java.lang.AbstractStringBuilder::append (60 bytes)
    4       java.lang.StringBuilder::append (8 bytes)
    5       java.lang.Math::min (11 bytes)
    6       java.lang.String:: (72 bytes)
    7       java.util.Arrays::copyOfRange (63 bytes)
    8       java.lang.AbstractStringBuilder:: (12 bytes)
    9       java.lang.StringBuilder::toString (17 bytes)
    10       java.lang.String::valueOf (14 bytes)
    11       java.lang.StringBuilder:: (18 bytes)

And finally, at 15000 the most verbose yet:

    n   java.lang.System::arraycopy (static)
    1       java.lang.Object:: (1 bytes)
    2       java.lang.String::getChars (66 bytes)
    3       java.lang.AbstractStringBuilder::append (60 bytes)
    4       java.lang.StringBuilder::append (8 bytes)
    5       java.lang.Math::min (11 bytes)
    6       java.lang.String:: (72 bytes)
    7       java.util.Arrays::copyOfRange (63 bytes)
    8       java.lang.AbstractStringBuilder:: (12 bytes)
    9       java.lang.StringBuilder::toString (17 bytes)
    10       java.lang.String::valueOf (14 bytes)
    11       java.lang.StringBuilder:: (18 bytes)
    12       java.lang.String::toString (2 bytes)
    1%      net.caprazzi.WWJD::main @ 59 (133 bytes)

But what is it trying to say?

I found little info about decoding this output, but from what I can understand, each line shows a method that has been compiled to native code and a progressive id. 1% at the end means that WWJD::main has used "On stack replacement". I have ino idea of what 'n' means.

The little I know about -XX:+PrintCompilation, I got it from Do Java 6 threading optimizations actually work? - Part II. The article is excellent and a hard read. It links two detailed explaination of PrintCompilation output, but they are both dead.

So what?

Ok, so we have seen that the JIT is definitely there, that it can make a program faster, but that it's difficult to gauge how much so, and with a so wide range of possible outcomes the safest strategy is to assume that the JIT won't gain you anything.

This post is far from a complete exploration of the JIT, and it raises more questions than it answers. While writing I found a lot of interesting cues and I'll be back with more stuff once I have followed them.

Enforcing design principles in software

2011-06-27T17:33:31+00:00

Before starting to write a new piece of software, I come up with some simple design principles that I think will do some good. For smallish projects I just keep the principles in mind and behave so, but for bigger stuff I try to actively enforce the policy.

"In theory, theory and practice are the same. In practice, they are not."
-- Lawrence Peter Berra

Actively "defending" your code from your own mistakes is controversial; in theory if you stick to a principle, there is no need for enforcement. A friend of mine has never written a single bug in his life. (It's true, I swear). He clearly can go without a safety net.

Another friend added that proper code testing makes enforcement useless.

I hate to admit it, but I do do bugs sometimes, and some tests I wrote turned out to be not so great. I've learnt that I'm a better coder when there are some safety checks. One immediate benefit is that checking rules makes me less likely to cut corners "just this once".

More deeply, I find that established check policies, much like widely used vaccines, have the potential to eradicate some categories of bugs.

No country for nulls

The principle I use most frequently is nulls are no good. Nulls are the source of Null Pointer Exceptions (NPE) and null checks. NPEs are a pain because they tend to happen far from where the assignment has happened, while null checks are just plain ugly. If I could, I would change the language and remove the concept of nullability

Failing that, I have to turn the principle in a rule and implement it in the code. At first the best rule may seem "always check for nullity", but I'll show that in this case it is enough to implement half of the rule to cover your ass: "never assign a null"

Never assign a null

Let’s begin with an example - everybody likes the bang of a popping balloon, so I have implemented a balloons collection that allows to pop many balloons at once for a bigger, better bang.

class Balloons {
    List<Balloon> balloons = new ArrayList<Balloon>();
    void addBalloon(Balloon balloon) {
        this.balloons.add(balloon);
    }
    void pinchAll() {
        for (Balloon b : balloons)
                   b.pinch();
    }
    public static void main(String[] args) {
        Balloons balloons = new Balloons();
        balloons.addBalloon(new Balloon());
        balloons.addBalloon(null); // one of your little helpers has run out of gas
        balloons.pinchAll();
    }
}

Not only this program will end in NPE, but also the stack trace will not clarify were the null comes from:

Exception in thread "main" java.lang.NullPointerException
at Balloons.pinchAll(Balloons.java:16)
at Balloons.main(Balloons.java:23)

To get rid of the NPE, you can just add a nullcheck inside pinchAll.

void pinchAll() {
    for (Balloon b : balloons)
        if (b != null)
            b.pinch(); 
}

True, this version avoids the exception, but if you ignore null values, you’re better off not putting them in your collection in the first place, yes?

Let’s try again:

void addBalloon(Balloon balloon) {
    if (balloon == null)
        trow new IllegalArgumentException(“balloon must not be null”);
    this.balloons.add(balloon);
}

Much better now: the output will actually help to pin down the cause of the problem.

Exception in thread "main" java.lang.IllegalArgumentException: 
    balloon must not be null
at Balloons.addBalloon(Balloons.java:10)
at Balloons.main(Balloons.java:22)

Additionally, since you have protected your code so that balloons will never contain a null, we can remove the check in pinchAll() making the code a little neater.

I know, adding != null all over your code is hardly neat, so I use some helper methods to make it less ugly. Here some example of how I use the "Protect" class" (full code at the bottom of the post):

void addBalloon(Balloon balloon) {
    Protect.valid(balloon);
    this.balloons.add(balloon);
}
void addMany(Balloon one, Balloon two) {
    Protect.valid(one, two);
        //....
}
void addWithLabel(String label, Balloon balloon) {
    // this will also check for empty strings
    Protect.valid(label, balloon);
    //....
}

I have picked up the habit of testing all arguments with Protect.notNull, and then think if I really need it or if this code is implicitly protected by it call hierarchy.

Once the check is in place in most of your methods, the way all the tests interlock makes for a warm feeling.

Yes, it’s bulky and your lean and clean code will look bloated, but mind that you'll save a lot of !=null checks after invoking methods. Once you get used to it, it also doubles as a form of documentation as any reader will quickly get that nulls are not accepted.

Null protection is probably at its best when used at the constructor of an immutable class: you know that all instances of that class do not contain nulls

// a bunga bunga you can trust
public class BungaBunga {
    private final String secretLocation;
    public BungaBunga(String secretLocation) {
        Protect.valid(secretLocation);
        this.secretLocation= secretLocation;
    }
    // ....
}

Beware, testing for nullity will waste some cpu - on my laptop about a ns for the simple check (object == null) and maybe 10 for the version with varargs. It’s not much but they may add up, so in you may want to skip the checks in some regions of your codes, especially method invocations in loops. As most things in life, don't overdo it.

It's possible to add more complex checks or other kinds. In a recent piece of code I was enforcing all the calls to a class to happen from one single thread. In another app that was using some simple spatial geometry I enforced all the ints in some classes to be positive.

My Protect class is a very crude piece of code, but it does the job and it's easy to understand. Note how empty strings are treated as nulls.

public class Protect {
    static void valid(Object object) {
        if (object == null)
            throw new IllegalArgumentException("Object in position 0 is null");
    }        
    static void valid(String string) {
        if (string == null || string.trim().length() == 0)
            throw new IllegalArgumentException("Str in pos 0 is null or empty");
    }
    static void valid(String a, String b) {
        if (a == null || a.trim().length() == 0)
            throw new IllegalArgumentException("Str in pos 0 is null or empty");
        if (b == null || b.trim().length() == 0)
            throw new IllegalArgumentException("Str in pos 1 is null or empty");
    }
    static void valid(String... strings) {
        for (int i=0; i<strings.length; i++) {
            if (strings[i] == null || strings[i].trim().length() == 0)
                throw new IllegalArgumentException("Str in pos " + i + " is null or empty");
        }
    }
    static void valid(Object a, Object b) {
        if (a == null)
            throw new IllegalArgumentException("Obj in pos 0 is null or empty");
        if (b == null)
            throw new IllegalArgumentException("Obj in pos 1 is null or empty");
        if (a instanceof String && ((String) a).trim().length() == 0)
            throw new IllegalArgumentException("Str in pos 0 is null or empty");
        if (b instanceof String && ((String) b).trim().length() == 0)
            throw new IllegalArgumentException("Str in pos 1 is null or empty");
    }
    
    // beware, this is maybe 10 times slower than the non-varargs version
    static void valid(Object... objects) {
        for (int i=0; i<objects.length; i++) {
            if (objects[i] == null)
                throw new IllegalArgumentException("Obj in pos " + i + " is null");
            if (objects[i] instanceof String && ((String) objects[i]).trim().length() == 0)
                throw new IllegalArgumentException("Str in pos " + i + " is null or empty");
        }
    }                
}

Thanks for reading this far, and always test your code, better if before writing it.

Java bytecode, string concatenation and StringBuilder

2011-06-23T10:08:43+00:00

In my earlier post I was making a fuss over picking the faster hash algorithm, when I realised I was using + to concatenate strings. Should I always use a StringBuilder? Should I care even for small strings? Heck, if I use theStringBuilder I'll be creating one extra object anyway?

I tried some variations of the test and I did not find any performance difference when comparing simple concatenation to using the string builder, even when trying with bigger strings and other silly combinations. I got curious and wrote a very simple class and looked at the resulting bytecode.

This java code:

public static void main(String[] args) {
    String cip = "cip";
    String ciop = "ciop";
    String plus = cip + ciop;
    String build = new StringBuilder(cip).append(ciop).toString();
}

Generates this - see how the two concatenation styles lead to the very same bytecode:

 L0
    LINENUMBER 23 L0
    LDC "cip"
    ASTORE 1
   L1
    LINENUMBER 24 L1
    LDC "ciop"
    ASTORE 2
// cip + ciop
   L2
    LINENUMBER 25 L2

    NEW java/lang/StringBuilder
    DUP
    ALOAD 1
    INVOKESTATIC java/lang/String.valueOf(Ljava/lang/Object;)Ljava/lang/String;
    INVOKESPECIAL java/lang/StringBuilder.<init>(Ljava/lang/String;)V
    ALOAD 2
    INVOKEVIRTUAL java/lang/StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder;
    INVOKEVIRTUAL java/lang/StringBuilder.toString()Ljava/lang/String;

    ASTORE 3
// new StringBuilder(cip).append(ciop).toString()
   L3
    LINENUMBER 26 L3

    NEW java/lang/StringBuilder
    DUP
    ALOAD 1
    INVOKESPECIAL java/lang/StringBuilder.<init>(Ljava/lang/String;)V
    ALOAD 2
    INVOKEVIRTUAL java/lang/StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder;
    INVOKEVIRTUAL java/lang/StringBuilder.toString()Ljava/lang/String;

    ASTORE 4
   L4
    LINENUMBER 27 L4
    RETURN

The compiler has transformed "cip+ciop" into "new StringBuilder(cip).append(ciop).toString()". In other words, "+" is effectively a shorthand for the more verbose StringBuilder idiom.

Update: Javadoc for java 1.4 says "String concatenation is implemented through the StringBuffer class", while in java 1.5 it's "String concatenation is implemented through the StringBuilder(or StringBuffer) class". It makes sense, since StringBuilder was introduced in 1.5 as a faster (non thread-safe) version of StringBuffer.

The compiler will do same trick for cip + "ciop" and "cip" + ciop. (In case you wonder, "cip" + "ciop" will just be compiled as "cipciop").

This is great, but beware, the compiler is not a worthy substitute for you thinking at what you do.

This code

String big = "both";
big += cip;
big += ciop;

Will be compiled into

String big = "both";
big = new StringBuilder(bag).append(cip).toString();
big = new StringBuilder(bag).append(ciop).toString();

While of course the most efficient way is

String big = new StringBuilder("both").append(cip).append(ciop).toString();

One more example:

String boo = "both";
for (int i=1; i<100; i++)
     boo += cip + ciop;

Now the compiler will do the obvious thing and instantiate one new StringBuilder at each iteration, as if I had written

String boo = "both";
for (int i=1; i<100; i++)
     boo += new StringBuilder(boo).append(cip).append(ciop).toString();

For better efficiency, I should have done this:

StringBuilder foo = new StringBuilder("both");
for (int i=1; i<2; i++)
    foo.append(cip).append(ciop);
String boo = foo.toString();

UPDATE:

Following Christian comment below, I had a look at how string.concat is handled:

String foo = "foo";
String bar = "bar";
String baz = "baz";

String foobarbaz = foo.concat(bar).concat(baz);        
String foobarbazX = foo + bar + baz;

Gets compiled in:

// .. snip
   LINENUMBER 20 L3
    ALOAD 1
    ALOAD 2
    INVOKEVIRTUAL java/lang/String.concat(Ljava/lang/String;)Ljava/lang/String;
    ALOAD 3
    INVOKEVIRTUAL java/lang/String.concat(Ljava/lang/String;)Ljava/lang/String;
    ASTORE 4
   L4
    LINENUMBER 21 L4
    NEW java/lang/StringBuilder
    DUP
    ALOAD 1
    INVOKESTATIC java/lang/String.valueOf(Ljava/lang/Object;)Ljava/lang/String;
    INVOKESPECIAL java/lang/StringBuilder.<init>(Ljava/lang/String;)V
    ALOAD 2
    INVOKEVIRTUAL java/lang/StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder;
    ALOAD 3
    INVOKEVIRTUAL java/lang/StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder;
    INVOKEVIRTUAL java/lang/StringBuilder.toString()Ljava/lang/String;
    ASTORE 5
// .. snip

So, is String.concat the best choice in terms of bytecode size?

-teo

Evaluating relative speed of java digest (hashing) algorithms

2011-06-22T14:27:20+00:00

It's best practice to encrypt security tokens such as passwords and sessions ids in your database. I was just doing that to the session tokens for a project at work, and I wondered which algorithm to pick if speed was the only consideration.

I cranked up some code that measures the wall-clock that it takes for each algorithm to hash a bunch of strings (10 millions). I found that MD5 is the fastest, and MD2 is the slowest, taking roughly twice the time. I would have gone for MD5 anyway as it is the standard choice, but it's good to see that it's quick too.

The usual disclaimers apply: this test is very un-scientific and is only relevant for my specific use case (the tokens I handle are the same length of the random strings in the test) so don't plan your business on it. That's pretty quick stuff anyway (15 to 30 microsecond each encryption), so you may want to pick an algorithms for its security features rather then for its execution speed. See the docs for more info.

Output:

Creating 10000000 random strings... Created.
Testing algo MD2... Completed in 339126 milliseconds
Testing algo MD5... Completed in 169690 milliseconds
Testing algo SHA-1...   Completed in 200398 milliseconds
Testing algo SHA-256... Completed in 211560 milliseconds
Testing algo SHA-384... Completed in 303999 milliseconds
Testing algo SHA-512... Completed in 316265 milliseconds
Test Complete.

And code:

import java.math.BigInteger;
import java.security.MessageDigest;
import java.security.SecureRandom;

public class CompareHashFunctions {

    private static SecureRandom random = new SecureRandom();

    public static void main(String[] args) throws Exception {

        int runs = 10000000;               

        System.out.print("Creating " + runs + " random strings... ");
        String salt = randomString();
        String[] strings = new String[runs];
        for (int i=0; i<strings.length; i++) {
            strings[i] = randomString();
        }

        System.out.println("Created. ");    

        runTest(salt, strings, "MD2");
        runTest(salt, strings, "MD5");
        runTest(salt, strings, "SHA-1");
        runTest(salt, strings, "SHA-256");
        runTest(salt, strings, "SHA-384");
        runTest(salt, strings, "SHA-512");        

        System.out.println("Test Complete.");
    }

    static void runTest(String salt, String[] strings, String algo) throws Exception {
        System.out.print("Testing algo " + algo + "...\t");
        MessageDigest instance = MessageDigest.getInstance(algo);
        long start = System.nanoTime();
        for (int i=0; i<strings.length; i++) {
            byte[] bytes = (salt + strings[i]).getBytes("UTF-8");
            MessageDigest clone = (MessageDigest)instance.clone();
            new String(clone.digest(bytes));
        }
        long elapsed = (System.nanoTime() - start) / 1000000;
        System.out.println("Completed in " + elapsed +  " milliseconds ");
    }        

    /**
    * Random string
    * @return a random string of 25 or 26 chars
    */
    static String randomString() {
        // 130 bit random integer converted to string in base 32
        return new BigInteger(130, random).toString(32);
    }
}

-teo

Where are generics stored in java compiled classes?

2011-02-24T18:00:39+00:00

I was having a go at learning some Java bytecode and started looking at how generics were handled. As expected, the compiler was emitting cast instructions when generic types where used, but nothing where the types where declared: generics in Java are implemented using a technique called "erasure".

Straight from the docs:

When a generic type is instantiated, the compiler translates those types by a technique called type erasure — a process where the compiler removes all information related to type parameters and type arguments within a class or method. Type erasure enables Java applications that use generics to maintain binary compatibility with Java libraries and applications that were created before generics. from java docs

But sure not ALL information about the type parameters is lost. That would mean that once I compile my code, all other developer would use it "the old way", with casts and all, but clearly this is not the case. Let's write two simple classes:

public class GenericClass<T> {}

public class StandardClass {}

And decompile them:

$ javap -c GenericClass
public class learn.GenericClass extends java.lang.Object{
public learn.GenericClass();
  Code:
   0:   aload_0
   1:   invokespecial   #8; //Method java/lang/Object."<init>":()V
   4:   return
}

$ javap  -c StandardClass
Compiled from "StandardClass.java"
public class learn.StandardClass extends java.lang.Object{
public learn.StandardClass();
  Code:
   0:   aload_0
   1:   invokespecial   #8; //Method java/lang/Object."<init>":()V
   4:   return
}

As it should, the bytecode looks exactly the same. No type parameters.

Let's invoke the same command again with -verbose to see some more detail:

$ javap -verbose StandardClass
Compiled from "StandardClass.java"
public class learn.StandardClass extends java.lang.Object
  SourceFile: "StandardClass.java"
  minor version: 0
  major version: 50
  Constant pool:
const #1 = class        #2;     //  learn/StandardClass
// ... snip
const #15 = Asciz       StandardClass.java;
// ... snip

$ javap -verbose GenericClass
Compiled from "GenericClass.java"
public class learn.GenericClass extends java.lang.Object
  SourceFile: "GenericClass.java"
  Signature: length = 0x2
   00 13
  minor version: 0
  major version: 50
  Constant pool:
const #1 = class        #2;     //  learn/GenericClass
// ... snip
const #19 = Asciz       Ljava/lang/Object;;

The two outputs are different at last: the constant_pool section of a class file has an optional "signature" field. This optional can specify the full signature of the class, including type parameters. The compiler can then use this information to do the right thing when turns sources into binaries.This stuff is specified in the section "4.8.8 The Signature Attribute" of the updated JVM Specification.

simple webserver and rest with jetty and NO XML

2011-02-16T13:57:40+00:00

Starting a new web project (maybe with rest?) in Java requires mastery of many complex abstract components. You need a webapp for tomcat (fiddle with some xml files), spring (import plenty of jars and add some more xml files), pick a rest framework like spring, resteasy, restlset (fiddle with annotations), deploy to tomcat and go.

I won't deny all this components add structure, reusability and a lot of good stuff, but sometimes I have simple needs that scream for simple solutions.

Few days ago I needed just that: a simple database with a REST api, and the ability to serve a few static files.

I used jetty embedded and it took was one maven dependency, 3 classes, 150 lines of java, 15 minutes and NO XML.

I was so satisfied with the result work that I served myself a beer and posted the code on github. The actual project implemented a datastore on neo4j, but for sake of simplicity I released it on github as a dumb in-memory key-value store.

According to apache bench, with as little as 8m memory (-Xmx8m) the server was able to handle 2k concurrent users at a rate of 1500 requests per second.

The main is as simple as this:

public static void main(String[] args) throws Exception {
    Server server = new Server(8080);
    Context root = new Context(server, "/");

    // configure the default servlet to serve static files from "htdocs"
    root.getInitParams().put("org.mortbay.jetty.servlet.Default.resourceBase", "htdocs");
    root.addServlet(new ServletHolder(new DefaultServlet()), "/");

    // use /uuid to get a fresh id
    root.addServlet(new ServletHolder(new UUIDServlet()), "/uuid");

    // the actual key/value store
    root.addServlet(new ServletHolder(new KeyValueServlet()), "/store/*");
    server.start();
}

Get the code: Simple webserver and REST with jetty and NO XML on GitHub

Most Advanced Yet Acceptable

2010-05-18T09:06:08+00:00

Raymond Loewy believed that

The adult public's taste is not necessarily ready to accept the logical solutions to their requirements if the solution implies too vast a departure from what they have been conditioned into accepting as the norm.

Quite a statement. He synthesised his observation in a principle and further compressed it in acronym: Most Advanced Yet Acceptable - MAYA He was a radical, forward thinker so I guess he was constantly refitting his ideas into the Acceptable yet Advanced. As a industrial designer, he was trying to have his designs mass produced and sold to actual people. He succeeded and his designs reveal the tension between the future rolling in and the people holding back. Of course, it's common sense. Had he designed something Unacceptable, his biography would have been different. But MAYA is also a very real constraint, the sort that makes or kills your proposition. No matter what you are selling to whom: a new song, a new recipe, a new software; to your fans, to your girlfriend, to your boss. Being advanced is not enough, it must be acceptable. Not enough advanced, it is not interesting. You can ignore the principle and be a scientist. Or never challenge it and be an engineer or just be boring. Some would say that a sciengineer will balance his act and live up to the MAYA principle. As a software developer and technology entusiast I find it hard to hit the sweet spot.

ten principles of good design and two books

2010-05-08T23:02:04+00:00

I'm watching the first episode of The Genius of Design, a BBC documentary series exploring the history of design. It's not about software design, but one could argue that the process and the craft of design are interesting not matter what is being designed. Anyway. They briefly interview Dieter Rams, a very prolific designer who made the history of industrial design and came up with ten commandments. Read them slowly. The 10th is pure genius.

Good design is innovative
Good design makes a product useful
Good design is aesthetic
Good design makes a product understandable
Good design is unobtrusive
Good design is honest
Good design is long-lasting
Good design is thorough down to the last detail
Good design is environmentally friendly
Good design is as little design as possible

(We are grateful to Mr Rams for such pearls of wisdom, and so is apple...)

All this talk about design reminds me of the design of everyday things. It's a brilliant book, inspiring and foundational. I keep suggesting it, especially to programmers and makers alike. Another title worth mentioning is the design of design, from the same folk who wrote "the mythical man-month". The new book is not as ground-breaking as the old one, but it's a good read of its own right.

On the subject of exploring and finding a design, watch this video on ted: build a tower, build a team (6 minutes).
Is there something like imdb or wikipedia for books? I keep linking amazon but I'd prefer to link to a website with a more informational angle.

openid authentication handler for couchdb

2010-01-13T17:09:09+00:00

Over at couchdb-openid, there is my implementation of OpenID version 1.1 for couchdb, based on http://github.com/etnt/eopenid

It seems fairly stable but has only been tested against myopenid.com, so it is definitely not production ready.

I plan to add support for openid 2.0 and to make couchdb work as openid endpoint.

The handler code would love to be reviewed by someone with some erlang and couchdb experience.

Demo

Try the login page at fortytwo.

Quick install:

cd couchdb_install_path/lib/couchdb/erlang/lib/
git clone git://github.com/mcaprari/couchdb-openid.git
cd couchdb-openid
make
edit local.ini [httpd]/authentication_handlers (or do it form futon) and add {couch_httpd_openid_auth, openid_authentication_handler} BEFORE the default handlers
restart couchdb

Quick test:

http://caprazzi.net:5984/_session?openid=auth-request&openid-identifier=<your_openid>

What to expect:

Only openid 1.1 is supported and it has only been tested with myopenid.com as openid provider.

When a client hits the initiation url (above), it is redirected to the openid provider and prompted to authorise the association.

Then it's redirected back to the couch and

if the client is not logged in in and supplies a new openid, a new user is created with username=openid and the client is logged in
if the client is not logged in in and supplies a mapped openid, the client is logged in as the mapped user
if the client is logged in and supplies a new openid, the supplied openid is added to current user, and the client keeps the current login
if the client is logged in and supplies a mapped openid
- if openid is mapped to the same user, the client keeps the current login
- if openid is mapped to a different user, the operation fails 400
if user is logged in AS ADMIN and supplies a new openid the operation fails 500

TODO:

try erl_openid for openid 2.0 support
decide if it is wise to map openids to admins (if at all possible)
cleanup ets table after auth confirm (or maybe find an alternative to ets tables)
reduce dependence from eopenid (dict access routines at least)

Couchdb runtime statistics viewer

2010-01-05T14:16:49+00:00

CouchDB comes with a runtime statistics module that lets you inspect how CouchDB performs. The statistics module collects metrics like requests per second, request sizes and a multitude of other useful stuff. From Couchdb wiki. I created a simple app that uses chronoscope to display how some metrics change over time. Jump to the demo, go to the source or use the docs.

Install

couchdb-stats is a simple couchapp plus a python script that hits /_stats and stores the results in couchdb server Attention: this app is tested with Couchdb 0.11.x, not yet released at the time of this post.

$ git clone git://github.com/mcaprari/couchdb-stats.git
$ cd couchdb-stats
$ couchapp push app http://localhost:5984/stats
$ python stats_copy.py localhost localhost stats 60

Couchdb reports metrics since server start or of the last 1, 5 or 15 minutes. There is no way to see yesterday's stats. To do that, we need to keep hitting /_stats and store results in a couchdb database. stats_copy.py does just that.

At this point it's possible to write several views, each focused on a particular metric. See the documentation for more details.

questions, comments and suggestions welcome

generating SVG charts with couchdb

2009-12-08T14:28:43+00:00

In this article I describe how I got couchdb to produce SVG charts using list functions

This post is long, so I'll report the results first:

group_level=1 yearly averages svg - png

group_level=2 monthly averages svg - png

group_level=3 daily values svg - png

Now go and read how I did it:

generate some test data
upload test data to couchdb
create and manage a design document with couchapp
write a simple view with map/reduce
write a _list function and render the charts!
Conclusions

Apache CouchDB is a document-oriented database server, accessible via a RESTful JSON API. It has some advanced features, such as the ability to write 'views' in a map/reduce fashion and to further transform the results using javascript. It's a young but very promising project.

Try this at home

You can browse browse or download all code discussed here. All comments and corrections are welcome.

Generate some test data

To get started with this exploration we need some data to render, and a quick way to visualize it before our application is ready. This Python script generates a series of data points that simulate the goings of someone's bank account.

# test_data.py. Usage: python test_data.py <simulation_length>
import sys
import random

days = int(sys.argv[1])
savings = 10000
pay = 2000
for i in range(0, days):
	if ( i%30 == 0):
		savings = savings + pay
	savings = savings - random.randint(0, pay/16) - 2
	print i, (int(savings))

Use the script to generate a sample set with 3000 points:

$ python test_data.py 3000 > test_data.txt
$ cat test_data.txt
0 11947
1 11882
2 11813
...

Our final output will be similar to a line chart made with some bash and gnuplot:

#!/bin/sh
# gnuplot.sh generates a plot of a series piped in stdin
(echo -e "set terminal png size 750, 500\nplot \"-\" using 1:2 with lines notitle"
cat -
echo -e "end") | gnuplot

$ cat test_data.txt | sh gnuplot.sh > test_data.png

Upload test data data to couchdb

We need our data in json format so that it can be uploaded to couchdb. This python scripts converts each input line to a json object. Each object will become a document in couchdb. All lines are collected in the 'docs' array, to make the output compatible with couchdb bulk document api. It also adds a tag to each document, so it's easier to upload and manage multiple datasets.

# data_to_json.py. builds json output suitable for couchdb bulk operations
import sys
import datetime
date = datetime.datetime(2000, 01, 01)
tag = sys.argv[1]
print '{"docs":['
for line in sys.stdin:
	day, value = line.strip().split(' ')
	datestr = (date + datetime.timedelta(int(day))).strftime("%Y-%m-%d")
	if (day <> "0"): print ","
	sys.stdout.write('{"tag":"%s", "date":"%s", "amount":%s}'%(tag, datestr, value)),
print '\n]}',

$ cat test_data.txt | python data_to_json.py test-data > test_data.json
$ cat test_data.json
{"docs":[
{"tag":"test-data", "date":"2000-01-01", "amount":11896},
{"tag":"test-data", "date":"2000-01-02", "amount":11876},
....
{"tag":"test-data", "date":"2008-03-17", "amount":18703},
{"tag":"test-data", "date":"2008-03-18", "amount":18643}
]}

Create a new database with name svg-charts-demo

$ curl -i -X PUT http://localhost:5984/svg-charts-demo/
HTTP/1.1 201 Created
...
{"ok":true}

Upload the test data

$ curl -i -d @test_data.json -X POST http://localhost:5984/svg-charts-demo/_bulk_docs
HTTP/1.1 100 Continue

HTTP/1.1 201 Created
....

Verify that 3000 documents are in the database.

$ curl http://localhost:5984/svg-charts-demo/_all_docs?limit=0
{"total_rows":3000,"offset":3000,"rows":[]}

Create and manage a design document with couchapp

Design documents are special couchdb documents that contain application code such as views and lists. CouchApp is a set of scripts that makes it easy to create and manage design documents. In most cases installing couchapp is matter of one command. If you have any problems or want to know more, visit Managing Design Documents on the Definitive Guide.

$ easy_install -U couchapp

This command creates a new couchapp called svg-charts and installs it in couchdb

$ couchapp generate svg-charts

$ ls svg-charts/
_attachments  _id  couchapp.json  lists  shows  updates  vendor  views

$ couchapp push svg-charts http://localhost:5984/svg-charts-demo/
[INFO] Visit your CouchApp here:
http://localhost:5984/svg-charts-demo/_design/svg-charts/index.html

Write a simple view with map/reduce

This view will enable us to group the test data year, month or day and see the average for each group.

// map.js
// key is array representing a date [year][month][day]
// value is each doc amount field (a number)
function(doc) {
	// dates are stored in the doc as 'yyyy-mm-dd'
	emit(doc.date.split('-'), doc.amount);
}

// reduce.js
// this reduce function returns an array of objects
// {tot:total_value_for_group, count:elements_in_the_group}
// clients can than do tot/count to get the average for the group
// Keys are arrays [year][month][day], so count will always be 1 when group_level=3
function(keys, values, rereduce) {
	if (rereduce) {
		var result = {tot:0, count:0};
		for (var idx in values) {
			result.tot += values[idx].tot;
			result.count += values[idx].count;
		}
		return result;
	}
	else {
		var result = {tot:sum(values), count:values.length};
		return result;
	}
}

Update the design document and test the different groupings

$ couchapp push svg-charts http://localhost:5984/svg-charts-demo/

Call the view with group_level=1 to get the data grouped by year

$ curl http://localhost:5984/svg-charts-demo/_design/svg-charts/_view/by_date?group_level=1
{"rows":[
{"key":["2000"],"value":{"tot":4247068,"count":366}},
...
{"key":["2008"],"value":{"tot":1529286,"count":78}}
]}

Call the view with roup_level=2 to get the data grouped by month

$ curl http://localhost:5984/svg-charts-demo/_design/svg-charts/_view/by_date?group_level=2
{"rows":[
{"key":["2000","01"],"value":{"tot":343578,"count":31}},
{"key":["2000","06"],"value":{"tot":345282,"count":30}},
...

Call the view with roup_level=3 to get the data grouped by day. As all the keys are different at the third level, this returns a single row for each document.

$ curl -s http://localhost:5984/svg-charts-demo/_design/svg-charts/_view/by_date?group_level=3
{"rows":[
{"key":["2000","01","01"],"value":{"tot":11896,"count":1}},
{"key":["2000","01","04"],"value":{"tot":11747,"count":1}},
...

Same as above but limiting the response to a range of days

$ curl -s 'http://localhost:5984/svg-charts-demo/_design/svg-charts/_view/by_date?group_level=3
&startkey=\["2008","01","01"\]&endkey=\["2008","01","04"\]'
{"rows":[
{"key":["2008","01","01"],"value":{"tot":20050,"count":1}},
{"key":["2008","01","02"],"value":{"tot":20019,"count":1}},
{"key":["2008","01","03"],"value":{"tot":19974,"count":1}},
{"key":["2008","01","04"],"value":{"tot":19878,"count":1}}
]}

Write a _list function and render the charts!

function(head, req) {
	start({"headers":{"Content-Type" : "image/svg+xml"}});
	
	// some utility functions that print svg elements
	function svg(width, height) {
		return '<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"'+
		' style="fill:black"'+
		' width="'+width+'" height="'+height+'">\n';
	}
	function line(x1, y1, x2, y2, color) {
		return '<line x1="'+x1+'" y1="'+y1+'" x2="'+x2+'" y2="'+y2+'"
			style="stroke-width: 0.2; stroke:'+color+'"/>\n';
	}
	function rect(x, y, width, height, color, fill) {
		return '<rect x="'+x+'" y="'+y+'" width="'+width+'" height="'+height+'"
			style="fill:'+fill+'; stroke:'+color+'"/>\n';
	}
	function text(x,y, text) {
		return '<text x="'+x+'" y="'+y+'" font-size="11"
			font-family="sans-serif">'+text+'</text>\n';
	}
	
	// import query parameters
	var x_size = req.query.width || 750;
	var y_size = req.query.height || 500;
	var level = parseInt(req.query.group_level);
	
	// find max and min values
	// collect values and labels
	var y_max = null;
	var y_min = null;
	var values = [];
	var labels = [];
	var count = 0;
	while(row = getRow()) {		
		var value = Math.ceil(row.value.tot/row.value.count);
		if (y_max==null || value>y_max) { y_max=value; }
		if (y_min==null || value<y_min) { y_min=value; }
		values[count] = value;
		labels[count] = row.key.join('-');
		count++;
	}
	// calculate scalig factors
	var in_width = x_size-(2*pad);
	var in_height = y_size-(2*pad);
	var in_x_scale = in_width/count;
	var in_y_scale = in_height/(y_max-y_min);
	
	// free space surrounding the actual chart
	var pad = Math.round(y_size/12);
	
	send('<?xml version="1.0"?>');
	send(svg(x_size, y_size));
	
	// background box	
	send(rect(1,1, x_size, y_size, '#C6F1C7', '#C6F1C7'));
	
	// chart container box
	send(rect(pad,pad, x_size-(2*pad), y_size-(2*pad), 'black','white'));

	// draw labels and grid
	var y_base = y_size - pad;
	var lastx = 0;
	var lasty = 0;
	for(var i=0; i<count; i++) {
		var x = pad+Math.round(i*in_x_scale);
		if (i==0 || x-lastx > (30+12*level)) {
			send(line(x, y_base+(pad/2), x, pad,'gray'));
			send(text(x+3, y_base + (pad/2), labels[i]));
			lastx = x;
		}	
		var y = Math.round(y_base - ( (values[i]-y_min) * in_y_scale));
		if (i==0 || lasty-y > 15) {
			send(line(5, y, pad+in_width, y,'gray'));
			send(text(5, y-2, values[i]));
			lasty = y;
		}
	}
	// draw the actual chart
	send('<polyline style="stroke:black; stroke-width: '+ (4-level) +'; fill: none;" points="');
	for(var i=0; i<count; i++) {
		if (i>0) send(',\n');
		var x = pad+Math.round(i*in_x_scale);
		var y = Math.round(y_base - ( (values[i]-y_min) * in_y_scale));
		send( x + ' ' + y);
	}
	send('"/>');
	
	send('</svg>');
}

Update couchapp, and execute the list function 'chart-line' against the view 'by_date'. Use different group_level settings, to obtain different charts:

curl http://localhost:5984/svg-charts-demo/_design/svg-charts/\
_list/chart-line/by_date?group_level=3 > chart-line_level-3.svg

curl http://localhost:5984/svg-charts-demo/_design/\
_list/chart-line/by_date?group_level=2 > chart-line_level-2.svg

curl http://localhost:5984/svg-charts-demo/_design/\
_list/chart-line/by_date?group_level=1 > chart-line_level-1.svg

group_level=1 yearly averages svg - png

group_level=2 monthly averages svg - png

group_level=3 daily values svg - png

Concusions

It worked.

I didn't expect to use a single list function for all grouping levels. I'm particularly happy of how it worked out, and even more considering that the whole thing is about 100 lines of code.

The output isn't too nice, but I think I can be made presentable with under 500 lines of code and some effort.

Couchdb is always a pleasure to work with and it goas a long way in minimizing "Time To something Done".

Learning Closure: managing dependencies and compiling

2009-12-04T10:42:36+00:00

Google's closure library offers semantics and tools to manage dependencies between modules. This is specially useful when building single-page javascript applications.

To find out how smoothly this works, I'll build a simple javascript application that given a string, displays its MD5 hash. We'll

create a simple component
crate a template with closure templates
import a third party library (jshash library) to do the md5 hashing
build a dependency tree that includes all of the aobove using the Dependency Calculation Script
concatenate all dependencies in one script

If you have used jQuery you'll find that this code is verbose and painfully similar to java.

To ease your mind, think that structured libraries like Closure and YUI tend to promote more readable and maintainable code (at the expense of coolness).

Closure and YUI are a better fit when the goal is to build a complex application.

Try this at home

See the demo of the sample application discussed below. You can browse browse or download all source code discussed here, including all 3rd party libraries. I welcome all comments and corrections.

Project setup

Create an empty directory (i called mine goog-dependencies-example), enter it and download all the stuff

Download the closure library

svn export http://closure-library.googlecode.com/svn/trunk/ closure-library

Download closure template tools and expand it in closure-templates/
Download the jshash library and unzip it

You should now have a structure like this:

goog-dependencies-example\
  |-- closure-templates\
  |-- closure-library\
  |-- jshash-2.2\

Publish this directory on a web server.

During development, I use lighttpd with this minimal configuration file to quickly publish just a directory. Save the file in the directory you want to publish, run lighttpd -D -f lhttpd-minimal.txt and browse to http://localhost:3030/. Lighttpd is available on ubuntu, macos (via macports) and windows (via cygwin).

UI: template

We need a ui component with a textarea to input some text, a button to execute and and a textarea to see the results. Using closure template is definitely overkill here, but it's useful to demonstrate how to load templates a dependencies. Create the file md5.ui.template.soy

{namespace net.caprazzi.md5.ui.template}

/**
 * md5 ui template
 * To retrieve its contents, invoke the function
 * net.caprazzi.md5.template.ui.main()
 */
{template .main}
<div class="md5-ui">
	<textarea class="md5-input"></textarea><br/>
	<button class="md5-action">Hash It!</button><br/>
	<textarea class="md5-output"></textarea><br/>
</div>
{/template}

And execute this command to compile it to a javascript file:

$ java -jar closure-templates/SoyToJsSrcCompiler.jar \
	--shouldProvideRequireSoyNamespaces \
	--outputPathFormat js/md5.ui.template.js md5.ui.template.soy

// This file was automatically generated from md5.ui.template.soy.
// Please don't edit this file by hand.

goog.provide('net.caprazzi.md5.ui.template');

goog.require('soy');
goog.require('soy.StringBuilder');

net.caprazzi.md5.ui.template.main = function(opt_data, opt_sb) {
  var output = opt_sb || new soy.StringBuilder();
  output.append('<div class="md5-ui">
	<textarea class="md5-input"></textarea><br/>
	<button class="md5-action">Hash It!</button><br/>
	<textarea class="md5-output"></textarea><br/></div>');
  if (!opt_sb) return output.toString();
};

UI: javascript

Now let's write some code that hashes the contents of the first textarea when the button is clicked. Name the file js/md5.ui.js.

goog.provide('net.caprazzi.md5.ui');

// NOTE the inclusion of our template
goog.require('net.caprazzi.md5.ui.template');

// NOTE the inclusion of jshash
// as defined in third_party_deps.js
goog.require('jshash.md5');

goog.require('goog.events');
goog.require('goog.dom');

net.caprazzi.md5.Ui = function(parent) {
	this.parent = parent;
}

net.caprazzi.md5.Ui.prototype.render = function() {
	var html = net.caprazzi.md5.ui.template.main();
	this.parent.innerHTML = html;
	this.input = goog.dom.$$('textarea', 'md5-input', this.parent)[0];
	this.button = goog.dom.$$('button', 'md5-action', this.parent)[0];
	this.output = goog.dom.$$('textarea', 'md5-output', this.parent)[0];

	var self = this;
	goog.events.listen(this.button,
			goog.events.EventType.CLICK,
			function() { self.onButton_()});
}

net.caprazzi.md5.Ui.prototype.onButton_ = function() {
	this.output.value = hex_md5(this.input.value);
}

Application entry point

This file exposes the function main(), that will be executed at page load. It only requires the two modules that uses directly

goog.provide('net.caprazzi.md5.application');

goog.require('net.caprazzi.md5.ui');
goog.require('goog.dom');

net.caprazzi.md5.application.main = function() {
	var container = document.getElementById('md5-ui-container');
	var ui = new net.caprazzi.md5.Ui(container);
	ui.render();
}

deps.js: the dependencies file

Calcdeps.py is a script that comes with closure library. It parses the source files in search of goog.require() and goog.provide() declarations. It then uses those declarations to build a dependency tree.

The dependency tree is stored in a file as a list of goog.addDependency calls. Each calls describes a file and the modules it provieds and requires. This line declares that md5.component.js will provide 'net.caprazzi.md5' and require 'net.caprazzi.md5.template' and others:


goog.addDependency('js/md5.component.js',
    ['net.caprazzi.md5'],
    ['net.caprazzi.md5.template', 'goog.events', 'goog.dom', 'jshash']);

Run this command in the root of the project to generate the deps.js all the dependecies (including google's). This may take a while.

$ python closure-library/closure/bin/calcdeps.py \
    -p closure-library/closure \
    -p closure-library/third_party \
    -p closure-templates \
    -p js \
    -i js /md5.* \
    -o deps \
> deps.js

The option '-o deps' tells calcdeps.py to output a dependency tree. The other options specify the source directories and files. Spend a moment to review the contents of the resulting file.

Managing the third party library 'md5.js'

As I explained above, calcdeps uses special declarations in the javascript files to make sense of the dependencies. We know that by 'jshash.md5' i mean 'jshash-2.2/md5.js' but calcdeps could only figure this out if the file included a correct goog.require statement.

At this point we may just add the line to the file and go on with our lives. But we all agree that editing 3rd party libraries is not a good practice.

I maintain a text file 'extradeps.txt' where each line associates a module to a file, in this case the contents are

jshash.md5 jshash-2.2/md5-min.js

Then a simple python script parses that file and generates the missing addDependency() statements:

import sys
for mod, path in [  line.strip().split(' ') for line in file(sys.argv[1]) ]:
	print "goog.addDependency('%s',['%s'],[]);" % (path, mod)

Execute the script to see what the output looks like. Then remember to concatenate its output to deps.js each time your rebuild:

$ python fix_deps_file.py extradeps.txt >> deps.js

in dev: index_dev.html

The last piece of the puzzle is the one that holds everything together: the html

<html>
	<head>
		<title>Hash me, hash me</title>
		<script>
			CLOSURE_NO_DEPS=true;
			CLOSURE_BASE_PATH="./";
		</script>
		<script src="closure-library/closure/goog/base.js"></script>
		<script src="deps.js"></script>
		<script>
			goog.require('net.caprazzi.md5.application');
		</script>
	</head>
	<body onload="net.caprazzi.md5.application.main();">
		<h3>Hash me, hash me</h3>
		<div id="md5-ui-container"></div>
	</body>
</html>

Note the two global directives before the inclusion of base.js:

CLOSURE_NO_DEPS stops closure from loading its own deps.js
CLOSURE_BASE_PATH tells closure file loader how to build the script urls.

Note that the html file only imports closure's base.js and deps.js then invokes directly code that comes from md5.ui.js

Navigate to index.html, the application should be working.

Use firebug or another network monitor, to see how the browser is downloading many different files to satisfy all the dependencies. This is the perfect behaviour for development, because separate files are easier to debug.

going live: the single js file and index_live.html

Loading many small javascript files is an evil practice in production. We definitely want to have all our javascript files concatenated in one big blob.

I previously used calcdeps.py with the option '-o deps' to build a dependency tree. Calcdeps can also concatenate all your dependancies right away, using the option '-o script'. Unfortunately if you try to run it against our code, it will terminate with an exception (

Exception: Missing provider for (jshash.md5)

. Clearly, again, the dependency script doesn't know which files provides that module

Python to the rescue again: we'll reuse the extradeps.txt file with a new python script that takes all the external deps and puts them in a file along with the correct goog.provide() statements.

import sys
mods = [  line.strip().split(' ') for line in file(sys.argv[1]) ];
for mod, path in mods:
	print "goog.provide('%s');" % (mod)
for mod, path in mods:
	for line in file(path):
		print line

Running this script produces a calcdeps-friendly javascript file:

$ mkdir tmp/
$ python concat_extra_deps.py extradeps.txt > tmp/extra.js

We can now tell calcdeps to look into tmp/ to find what's needed to build our big target file:

$ python closure-library/closure/bin/calcdeps.py \
	-p closure-library \
	-p tmp \
	-p js \
	-p closure-templates \
 	-i js/md5.application.js \
    -o script \
> md5_application.js

Now we can produce the final artifact and go live with index_live.html:

<html>
	<head>
		<title>Hash me, hash me</title>
		<script src="md5_application.js"></script>
	</head>
	<body onload="net.caprazzi.md5.application.main();">
		<h3>Hash me, hash me</h3>
		<div id="md5-ui-container"></div>
	</body>
</html>

Conclusions

I've done this exercise to understand if the closure's dependency management would be usable in a production environment.

There was some initial misunderstanding between calcdeps and me and I had to look at base.js to understand what was going on and to find out about the global directives that influence the loader.

The syntax is neat and serves the double purpose of managing namespaces and dependencies.

My impression is that at least the goog.require/provide syntax and the calcdeps.py script can be used in production with confidence.

Closure dosen't allow to specify css files as dependencies, while it's possible with YUI3 (that sports a cool dependency management mechanism of his own).

As an exercise you can think about adding a compilation step using google closure compiler.

-teo

Book: Programming Collective Intelligence

2009-12-02T18:47:06+00:00

Programming books are seldom page turners, even for geeks like us. And programming books often fail to go beyond the technology. Programming books are rarely as good as this one. Allow me to digress for a moment. Online there is a lot of data about you. And me. Our reactions have been logged, our desires noted. You know where I am now. I know your status. Ok, I'm being dramatic, but bear with me. I have a point. All this data tells one story and million stories in their gritty details. And since past behaviour is a good predictor of future behaviour, I can read your very future in those million stories. Not so easy, of course. All those records are generated by humans and computers. It's cybernetic data, its scale and complexity are beyond belief. We can only make sense of that data using computers. Back to the topic. This very thick yet enjoyable book is a programmer's journey in the world of machine learning and statistics. The author, Toby Segaran, is brilliant. He will hold your hand while describing many algorithms an techniques. He'll tell you just enough theory to make sense of what's going on. And then, plain simple python, down with the implementation. Interested in building and thinking about a recommendation engine? What about picking a good solution to a proble, amongst millions? Curious about genetic programming? Wanna have your own spam filter? The last great thing about this book are the example. Real world, not your usual fictional pet shop. You'll fetch data from your delicious account, correlate live stock market data from yahoo, analyze your favorite blogs and more stuff. Tony writing is unpretentious and the book is easy to read, but beware there is a lot of meat to digest. Don't rush.

Learning Closure. Ajax with goog.net.*

2009-11-12T00:20:52+00:00

Google released closure library, a javascript library good to make rich internet applications. As this is going to be an important library in software developement, I decided to try it. 'Ajax' networking is a good starting point, so I looked into the package goog.net and found two promising classes, HxrIo and XhrManager, which I used for my tests. The documentation is clear but sometimes shy about who does what. Fortunately there are plenty of links to the source code, which is well written and documented. The interesting code is below, but there's a more extend version of my closure-library tests. Just hit a url without waiting for a response


// load the modules
goog.require('goog.net.XhrIo');
goog.require('goog.Uri');

// the easy way: static function call with no callback
goog.net.XhrIo.send('url/');

// using the send() method of an instance.
new goog.net.XhrIo().send('urlA'); // GET, default
new goog.net.XhrIo().send('urlB', 'POST');

// also works with an Uri object
var uri = new goog.Uri('urlC');
new goog.net.XhrIo().send(uri, 'PUT');

// no http method validation, this is a valid call
new goog.net.XhrIo().send('urlF', 'XXXX');

// send some data with the POST, works as expected
new goog.net.XhrIo().send('urlG', 'POST', 'p1=v1&p2=v2);

// ATTENTION data will not be attached to a GET
// so this call won't do what you may expect.
new goog.net.XhrIo().send('urlH', 'GET', 'p1=v1&p2=v2');

Request data from the server


// an html file as text
goog.net.XhrIo.send('sample.html', function(event) {
	var text = event.target.getResponseText());
});

// an xml file as document object
goog.net.XhrIo.send('sample.xml', function(event) {
	var document = event.target.getResponseXml());
});

// a json file as javascript object
goog.net.XhrIo.send('sample.json.js', function(event) {
	var data = event.target.getResponseJson());
});

// spot the 404
goog.net.XhrIo.send('gone_fishin', function(event) {
	var success = event.target.isSuccess()); // false
	var status = event.target.getStatus()); // 404
	var statusText = event.target.getStatusText()); // not found
});

// !!! exceptions in the callback are swallowed, so check your conditions
// (there is interesting error handling stuff in goog.debug.*)
goog.net.XhrIo.send('gone_fishin.js', function(event) {
	// I expected some data, but the file is not there
	// and this will raise an exception...
	var data = event.target.getResponseJson();
	var field_data = data.field;
	// ...so this statement will not execute...
	alert('bonk');
	// ...BUT you won't see anything in your console.
});

// a 404 using an XhrIO instance and closure event management;
var io = new goog.net.XhrIo();
goog.events.listen(io, goog.net.EventType.COMPLETE, function(event) {
	var status = event.target.getStatus()); // 404
});
io.send('gone_fishin');

Using the connection manager


goog.require('goog.net.XhrManager');

// goog.net.XhrManager(opt_maxRetries, opt_headers, opt_minCount, opt_maxCount, opt_timeoutInterval)
var mgr = new goog.net.XhrManager(2, null, 0, 2);
goog.events.listen(mgr, goog.net.EventType.COMPLETE, function(event) {
	// this is fired once for each send(), even if they are retried
	var xhr = event.xhrIo;
});
mgr.send('id_one','closure.js');
mgr.send('id_two','/urlTwo');
mgr.send('id_three','/urlThree');
mgr.send('id_four','/urlFour');

// reusing an id before the request is complete, causes an exception...
try { mgr.send('id_one','/x'); }
catch (e) { /*[goog.net.XhrManager] ID in use*/ }

// ...unless the connection is aborted
mgr.abort('id_one');
mgr.send('id_one','/x');

mgr.send('other_id','/someurl', null, null, null, function(event) {
	// it's ok to reuse an id an after the request completed
	mgr.send('other_id', '/fool');
});

book: forms that work

2009-10-23T15:13:00+00:00

Forms that Work: Designing Web Forms for Usability There is no html in this book, it's not about programming forms. And that's why I liked it. This short book clarifies what good form design is about, in plain english. Plenty of images help, too. I'm still not too good at designing forms but at least now I know the name of the game. -teo

book: don't make me think

2009-10-15T14:14:00+00:00

Don't Make Me Think!: A Common Sense Approach to Web Usability

Delightful one-night read. Some good common sense advice on web design and two brilliant chapters on usability testing on a budget.

common sense is underrated

netflix prize contest, a new one

2009-10-08T12:57:00+00:00

With netflix announcing a new contest, it's time to look back at how the Netflix Prize ended: BellKor's Pragmatic Chaos team (AT&T Research engineers) won over the ensemble (a creek of lower-ranked contestants), just minutes before the deadline.

Both teams blended together different algorithms to improve by 10% the hit rate of existing netflix suggestion engine.

This is a story of monkeys and challenges, interesting beyond bytes and keyboards. Wired has good coverage, but you can google for more.

I'm reading Programming Collective Intelligence, which so far is ranking high: it uses easy-to-read-and-write python to build suggestion engines using many different algorithms. For programmers.

story coverage:

Above the Clouds: The View from Berkeley

2009-08-03T11:34:00+00:00

RAD Lab released Above the Clouds: The View from Berkeley (pdf) an high-level overview of the cloud. It's great read for anyone interested in the subject, no need to be involved in software development.

It's the best text on the subject that I've read so far: the authors start and stay above the cloud, avoid stepping on technical traps, yet they never become too theoretical.

The paper will tell you what the cloud is and what is not, how it fits in today's IT ecology. It also describes the "economics of the cloud" with no-nonsense formulas and pragmatic statements, making sensible assumptions and predictions.

If anything, I think it underestimates the impact of private clouds for small and medium organizations in the future. Designing your data center as cloud buys you a greater potential for scalability and flexibility.

Add that even a tiny company benefits from exposing its data layer as a service with domain specific logic (that is, a SaaS with an API), and that couples perfectly with a cloud architecture.

This is straightforward only for new organizations; a massive paradgim switch in an existing company could well kill it...

Falickaway: a flickr viewer

2008-10-03T21:57:00+00:00

Available now: flickaway (demo)

I love flickr.

I also love my pictures, and sometimes flickr feels too... generic? impersonal?
Anyway. I decided to spend some time coding this flickr viewer so to have a website with my style and the same pictures I have on flickr.

I thought someone else might find it useful. If you like it, just go there, run the wizard, copy the generated html in an html file. Deploy to a web server. done.
Yes, the wizard is javascript-only as well.

I almost forgot: that html page is less than 2k.
All the javascript and the styles will be fetched from my website (or google, or yahoo), so you'll always get the newest and best version.

A brief on technology:
I'm using jQuery (from googleapis.com), google feed apis and flickr rest api (on the wizard, to discover your feeds). All files are gzipped then deployed on Amazon S3.

Matteo

[C Testing] MinUnit - less is more

2008-09-08T11:48:00+00:00

- Can a whole framework be 3 lines long? -

The C project I'm working on is quite small in LOC and files. It won't grow much as it's deployed on a chip with 4k program space and 256 bytes of ram.

No matter how small, test all your code. Always. Period.

Problem is, where to start?

Approaching unit testing may go down two different paths.

Write few functions with ifs and printfs, call them from a main(), execute and see. It's prone to code duplication and you'll soon start reinventing the wheel. You are paving your way to a maintainability hell. Anyway, It might work for small projects and is very easy to get started with.

Otherwise you can embrace a framework such as CppUnit o Check, learn it and start writing tests. You get more power and less duplication. You'll benefit from standardization, automation and much more. It's the best approach, except for the steep learning curve. Add that writing tests is not exciting nor easy, and you might end up being so frustrated that you'll never get your test suite done.

Introducing the solution: MinUnit, a unit testing framework in 3 lines of code (!!!!):

 /* file: minunit.h */
 #define mu_assert(message, test) do { if (!(test)) return message; } while (0)
 #define mu_run_test(test) do { char *message = test(); tests_run++; \ )  extern int tests_run;
                                if (message) return message; } while (
0

(This kind of magic is why I really do miss macros in Java)

At least it is the solution for my small-scale project.

Just copy those few lines in a header file, include it and start writing tests. It is really that easy. And gone is your excuse for not testing.

Really, this might be all you need even for a mid-sized project. Especially if you are introducing unit testing in an existing project, the jump start can really pay off.

One important feature of (some) bigger frameworks is that the test runner executes your test code in a separate address space. This way if your buggy code picks the wrong pointer, it won't bring down the test runner, but only that unit. The runner will report it as a failure and move on. Do that with MinUnit and probably you'll have no clue of what hit you.

Make no excuses, test-your-code.

For the records, I modified it a little bit, adding a specialized assert:

#define mu_assert_eq(expected, actual) {\
int a = (expected); \
int b = (actual); \
do { if ((a) != (b)) { \
sprintf(minunit_msg, "FAILURE %s:%d expected %d, got %u", __FILE__, __LINE__, a, b);\
return minunit_msg; }; \
} while(0); }

There is a complete example on the project page or have a look at a real world example: test_ring_buffer.h and test.c

Matteo

CYZ_RGB alpha 2 - feature complete

2008-09-07T19:03:00+00:00

The second release of CYZ_RGB, an alternative firmware for BlinkM, is now ready for download.

The latest release implements all the features of the original firmware.

The program has been slightly tested at a functional level (sitting there and watching the lights go on end off), but has no unit tests.

I usually write loads of tests, possibly before the actual code; this time I was wandering in a completely new area and I knew most of the code was going to be highly disposable. Now that all hurdles have been surpassed, the refactoring stage will go hand-to-hand with writing a thorough test suite. Wish me good luck.

Go read the details.

Goals for next release:

Write a comprehensive test suite
Decouple cyz_cmd from cyz_rgb
Decouple usiTwiSlave from cyz_cmd
Reduce binary footprint (now 4072b of 4096 available)
Add ability to write 2 scripts

-Matteo

How (not) to code and new functionalities

2008-08-15T11:43:00+00:00

How (not) to write code for a small chip

The project has two main set of procedures for color control and commands execution. Loving encapsulation and separation of concerns, I built two structs with function pointers, to mimic objects:

typedef struct CYZ_CMD {

 unsigned char rcv_cmd_buf[8];
 unsigned char rcv_cmd_buf_cnt;
 unsigned char rcv_cmd_len;
 Cyz_rgb* cyz_rgb;
 void (*execute)(void* this, unsigned char* buf);
 void (*receive_one_byte)(void* this, unsigned char rcv);
} Cyz_cmd;

then allocating an "instance" with malloc, doing some initialization work and finally executing "methods" with

cyz_cmd->execute(cyz_cmd, play);

Smart you say? Not at all. I only ended up eating all available memory (a flattering 256 Bytes). I reverted to global variables and functions, dropped all the dynamic memory thing and learnt how not write code for an embedded system. True, neat and clean code is more maintainable, but I was spending precious memory to buy only an apparent advantage.

Adding new commands and functionalities will further reduce available memory: I'll probably learn some neat tricks. Can't wait...

Added some commands (only in svn head)

'W' Write script line

Script 0 can be modified and up to 10 lines added and executed. With some limitations:

line duration is ignored
script lines only support fade to rgb and go to rgb
script is not persisted and will disappear at next power off
script number is ignored, can only write script 0 (as in original blinkm)

'p' Play script

After writing a script with 'W', it can be xecuted with 'p'. Only applies to script 0

'o' Stop script

No limitations here :)

See the project page for details.

Matteo

CYZ_RGB alpha - blinkm alternative firmware

2008-08-11T20:33:00+00:00

The first release of CYZ_RGB, an alternative firmware for BlinkM, is now ready for download.

Go read the details.

BlinkM is a Smart LED produced by ThingM:

BlinkM is a “Smart LED”, a networkable and
programmable full-color RGB LED for hobbyists,
industrial designers, prototypers, and experimenters.

Just connect it to an I2C bus and start sending commands like "show rgb color" or "fade to hsv value" and so on and so forth (it does much more; see the human-readable data sheet).

A bit of explanation: I liked the idea to do a few more things with the leds and decided to modify the firmware. The original firmware is actually closed source (due to legal issues) so I had to start  writing a new one from scratch. This first release is very basic but hopefully adding new features won't be that hard.

The new firmware has two interesting additional features in the works: the ability to act as a master on the I2C bus and support for dallas 1-wire protocol, to leave one pin free for your creativity.

Download

If you decide to download and give it a try, you'll find slave.hex and master.hex: the first is the actual blinkm firmware replacement, the other is the experimental master. You can flash the files on two different blinkms, connect them the to the same i2c bus and watch they change color together.

Browse the source code.

Download the binary release.

Get the sources with svn: svn checkout http://codalyze.googlecode.com/svn/cyz_rgb/trunk cyz_rgb

Interested? I need help.

I'd like to hear from you. If you have any patch to submit it will be welcome.

Please not that my experience in C is very limited. And it approaches zero when it comes to microcontroller programming. So, if you are any good at it, please be patient and expect the worse... i'm a java programmer ;). When you stop feeling sick after reading the code, well, drop me a line

Matteo