I would like to announce ogaraK, a simulator of malaria population genetics.

ogaraK is a Java Web Start application developed in Groovy and Java which allows to simulate parasite population genetics. It can be used, for instance, to compare the effects of different drug deployment policies.

A set of Python scripts are made available to analyze the results (mainly frequencies of resistance loci over time).

ogaraK is free software (GPL v3).

ogaraK can also be used to simulate some of common theories about sex: Epistasis, Red-Queen and spatial heterogeneity (but not based on size, as the underlying model has no concept of population numbers).

I would like to ask the reader for some basic help: If you can, could you please test the application (A single click to run, if you have Java 1.5+)? I know most people will not understand the results or the parameters, but just a simple run would help (and reporting back if something goes wrong!). The objective is to make the application available to epidemiologists, and due to their lack of IT knowledge, a robust application is needed. Any comments would really be appreciated (I make no financial profit from this free application). If you know Java Web Start, then if you could activate the console and return any error detected (even from a blind execution), that would be most appreciated.

My main assumption in defense of Groovy is that you are, a Java person. Java is your day to day programming language and you are comfortable with that. Though you are comfortable with that you want to try something different: maybe you want to try a scripting language, maybe you do not like too much boilerplate code.

Why Groovy? Because it gives you a lot of goodies with essentially no learning curve. An illustrative example: I’ve spent the whole morning thinking that I was editing a Java source file, but I was indeed working on a Groovy file. You see, Groovy not being a superset of Java ends up being almost that: Code in Java is, in most cases already code in Groovy. So, if you know Java, you can write Groovy. You will not write the best idiomatic Groovy, you will not gain any of Groovy’s goodies (and you will pay a performance penalty, BTW). But this is an amazing head-start if you want to go in the direction of higher-level languages (I would argue that it is even smoother than C from C++ as the paradigm does not change from Java to Groovy, it is OO to OO). If you go the Jython way then you have to learn a new language. If you go Scala or Clojure then you have to learn a whole new paradigm (and deal with the impedance of imperative semantics in typical Java libs against standard functional semantics).

For little to no cost you have now many goodies associated with scripting languages (dynamic, low boiler-plate coding, DSLs, better meta-programming, …). In some cases Groovy even out-competes supposedly more elegant languages (are Scala meta-programming facilities still as bad as in the past?).

Another interesting advantage of Groovy is that, if you want to revert back to Java then it is much easier. Why would you want to do that? Well, for performance reasons. In a Groovy application that I have, a small part of the code is extremely intensive, so I had to rewrite it in Java. This revealed to be a trivial exercise (similar syntax, similar semantics).

Again, let me stress out this: your requirements and your personal path are fundamental in any decision you take. There is no true language (OK, Prolog…. ). Different people, different approaches. All I am saying is: if your background is strongly grounded on Java and you feel comfortable with that, then Groovy is probably the way to go.

Disclaimer: While I have a couple of applications made in Groovy, most of my scripting efforts in the JVM world involve Jython (another fine language implementation, appropriate in a different set of circumstances) and I also believe, that from a declarative and highly expressive language point of view, what is being done with Clojure certainly deserves mention.

Oracle.

Let me continue my little story: Fast-forward a few years and I accepted a job at the same university as data centre manager. The 3 biggest universities in the city had bought the same accounting/payroll system. This system had a Oracle backend: database, application server and financial extensions. It so happened that each component had completely different licensing terms: one was on concurrent connections another on concurrent users, yet another on the total number of users. The notion of user also varied: The financial application had thousands of users, but it connected with a single user to the application server. The usage of multiple CPUs was also subjected to licensing. I hate to think on the countless hours spent by the data centre managers just parsing all this. It still gives me a headache.

But it does not stop here: even as a manager I was still very technically inclined (and had previous experience as a database manager): I would subscribe to the view that Oracle databases are a too-much complex database system that seem to be highly profitable to Oracle and the contractors that are payed to maintain such a complex system. Between Oracle DB and IBM DB/2 I would take DB/2 any time (at least 5 years ago where my professional path changed). Of course, for most applications PostgreSQL is more than enough…

The point is, as you probably have noticed by now: I am not comfortable with Oracle’s corporate culture. I am not comfortable with Oracle taking the lead on Java. In fact I can think of no worse corporation to lead Java.

I would also like to draw your attention to size and power bias. My point is this: people tend to dislike Microsoft, but part of that dislike does not come from corporate culture but from the influence and power that Microsoft has (still) in computing. I dread to think of the consequences of people like Larry Ellison or Steve Jobs having the same influence as Bill Gates. I am not defending Bill, just suggesting that it could be much worse. We do not feel the pain of Oracle (or Apple for that matter) because they do not have the massive OS/Office market share that MS has.

I always had a love relationship with Java (though I have a background in declarative languages). It was amazing to see the appearance of new languages on top of the JVM (I use Groovy, Clojure and Jython). The idea of a shared VM (and shared libraries) which a set of cooperating and competing languages on top is wonderful…

But I am starting to doubt that the JVM world will stay an open community where the best ideas can be incepted and flourish.

If IBM had bought Sun…

I do not trust you, Larry!

Recommending readings (from James Gosling’s blog):
Quite the firestorm
The shit finally its the fan
Cynical chuckes

• [jvm] YouDebug – A debug with a scriptable UI (not a GUI). Allowing for automated debugging. The main justification for this is e.g. on costumer sites where you cannot fire a graphical debugger. But an automated debugging framework can be the ground for some cool debugging tools. Furthermore it is scriptable in Groovy!
• [general]Are we there yet? – “Rich Hickey advocated for the reexamination of basic principles like state, identity, value, time, types, genericity, complexity, as they are used by OOP today, to be able to create the new constructs and languages to deal with the massive parallelism and concurrency of the future”. This is a 1 hour video (with synchronized slides). While I disagree with the basic premise that concurrency is fundamental in the future, this presentation is one of the most invaluable videos about the design of programming languages.
• [lisp] A Lisp User-group meetings calendar (Google based)
• [clojure] Be mindful of Clojure’s binding – Pitfalls of lazy sequences and thread bindings. The lazy sequence stuff seems more of an example of global variables are bad thing, as the example is dependent on an external symbol evaluation, but clearly one has to be mindful of lazy versus non-lazy evaluation.

There is a certain austerity tradition coming from the CS community (at least from the BEST part of it). Most people see syntactic sugar as a waste of time and energy. Furthermore everything is just trees and a linear representation with the root at the head is just fine to represent trees. Lisp embodies this tradition to the full. And for the most part is a great tradition.

This is all fine, and I would understand the defensiveness considered that we live in a world polluted with overly complex languages which just bring problems and no advantages to compensate.

But some level of syntactic sugar is important for humans. And while meta-programming is nice, I suppose most programming languages are still used by human programmers .

It would suggest to have a look at the literature on human cognition, namely the strategies used to read text: it is messy. But I prefer to go in a different direction. I just want to expose how some subtle representation issues might have a massive impact: Have you ever tried to do basic arithmetic with roman numerals? Have a look here, namely “How can I use Roman numerals to do arithmetic problems?”. While this is a different problem, it serves to emphasize that representation matters in the most strangest ways.

Even if it doesn’t, humans are a conservative species, they are not ready to change everything at the same time. Having a to jump to imperative+heteroiconic+infix to functional+homoiconic+prefix is too much. Lisp derived languages loose potential to attract more people. And in this case is is really worth it? Is it really THAT IMPORTANT to be prefixed? Is that the main conceptual selling point of Lisp?

But most importantly there are solutions that are not dirty:.

First nobody is forcing infix notation, one can have both at the same time (check Prolog :- op – you can do BOTH +(1 2) and 1+2).

Second hard-coding operators is not necessary at all: again check the Prolog suggestion. You can have all infix operators that you want (including none)

It is easy to conceptualize a superset of Lisp with infix notation and ZERO INITIAL infix operators.

If I can shout something is this: Check Prolog’s :- op ! Simple, elegant. And fully coherent with the Lisp philosophy at its core.

I would like to discuss a bit the first (syntax) and the last (macros). The macro part discussed somewhere in the future. For now I would like to concentrate on syntax.

Lisp syntax is strange for most humans, we simply do not write (+ (* 2 3) (/ 4 2)). Lisp proposers are probably aware that this will not be very popular, as people are used to something completely different. It is not only different but is also very uneconomical to both read and write: 2*3 + 4/2 is arguably easier to read. One can even make some intuitive declarative things. Notice that I’ve written 2*3 (no space around *) but 2*3 + 4*2 (spaces around +). This allows to give queues to readers about the precedence of operators. You can say that these issues are just simple and ridiculous syntactic sugar. Well, it seems that Homo sapiens has evolved to appreciate it. And I don’t really see a reason not to accommodate this requirement.

Yes, I read you already: infix syntax forces hard-coding of operators and precedences (e.g. + precedes *) . Makes parsing much more complex (there goes the beautifully simple Lisp parser out of the window) and in fact forcing a certain structure with hard-coded operators.

Well… no.

Take a look at Prolog op built-in. :- op allows you to define operator precedence and association rules. So the parser is really not much complex: it starts with a simple, uber-flexible, blank slate which you can fill in.

So, for (- 4 3 2) you can say – is an infix operator with first argument with precedence over the second (i.e. (4 – 3) – 2) and, voila, you can write 4 – 3 – 2. No need for hard-coded operators, all dynamically defined. You could, maybe for sadistic reasons, change the argument precedence so 4 – 3 – 2 would become 4 – (3 – 2). The flexibility is there.

You can also state that + and – have higher precedence than * and /. So that 2*3 + 2 is interpreted as (+ (* 2 3) 2). Again, you can change the precedence rules.

Ah… and you can have operators that are atoms (alphanumeric symbols). In Prolog you can say “X is 3 +2″. “is” is an operator. You can redefine operator rules, define new operators.

Operators are nothing more than a set of syntactic sugar rules to convert a more intuitive representation to another. But they really make things much more pleasant and humane.

I would imagine that adding this to Clojure would be DEAD EASY, by the way.

Regarding the issues about macros, I will leave it for another post.

So far, nothing new. I for one, am in the group of the explicit naming of everything.

Sometimes the name has some impedance between the average expectation and the functionality exposed. That can be a source of lots of grief. Lets have a look at a concrete example to make things clear.

Clojure has a function named contains? . How do you expect such a function to operate? Let me give a few examples (and if you know how it really operates in reality, try to forget for now what you know).

What you think would be the result of:

1
2
3
4
5

(contains? [3 5] 3)
(contains? [3 5] 1)
(contains? "bla" 1)
(contains? "bla" "a")
(contains? '(1 2) 1)

Does [3 5] contains 3?
Does [3 5] contains 1?
Does “bla” contains 1?
Does “bla” contains “a”?
Does (1 2) contains 1?

Actually (contains? [3 5] 3) is false. Why? Lets have a look at the documentation:

clojure.core/contains?
([coll key])
Returns true if key is present in the given collection, otherwise
returns false.  Note that for numerically indexed collections like
vectors and Java arrays, this tests if the numeric key is within the
range of indexes. [...]

So contains? is looking at the KEYS of a collection. The vector [3 5] has two keys: 0 and 1 (the indexes with occupation of the said vector), that is why (contains? “bla” 1) is true (the string “bla” has 3 keys 0, 1 and 2.

By the way, the list (1 2) is not a collection, so that false is caused by a type error.

Note that contains? actually has some utility : it is used to check if a map has a certain key.

There is an slight impedance between what most people would expect from “containing something” and contains? It so happens that slight impedances are much worse than big ones, because big ones are so damn obvious that they are easy to spot and normally end up being corrected (imagine a function drawLine, that draws a circle: it is a dead obvious problem that one will easily notice and the developer most probably correct). Slight impedance problems are more obnoxious:

• They will confuse newbies (like myself), whose expectations will be aligned with the general knowledge of a certain name or action.
• They will increase the cognitive load of experts which will need to be aware of the dissonance between the meaning of a certain word in the programming language and the meaning in the natural language.
• It will make reading and understanding a program more difficult.
• They will be a source of bugs, as developers, even experienced one, will sometimes forget the local meaning of a word and unconsciously apply the general meaning.

By the way, my personal solution for contains? Well, quite simply rename it hasKey? and only apply it to maps. Note that this post is not about contains? in particular, but about this “impedance” problem in general.

I would imagine that some readers might think this as irrelevant and nitpicking. In my subjective and personal point of view, this issue is a potential source of hard to find bugs and a waste of mental energy. In fact this blog is called Cognitive _Consonance_ for some reason…

BioJava’s parser relies on JGraphT to create a representation of the “tree”. The tree is actually really more an acyclic graph than a tree, though some trees are rooted (and therefore trees in the proper sense). Manipulating the JGraphT weighted graph is the complicated part, not really the BioJava interface. Note that JGraphT objects can be easily rendered using the JGraph library (yeah, it is confusing: there is one lib with graph algorithms called JGraphT and another, for vizualisation, called JGraph).

In this small tutorial, we will only try to write a textual representation of a tree.

Imagine this simple nexus file:

#NEXUS

Begin TREES;
	tree test1 = (1,2);
	tree test4 = (1:0.1,(2:0.2,3:0.3):0.4);
End;

We just want to draw this:

coder@move-on:~/development/biobug/test$ java Test test1.nex test1
Will process file test1.nex tree test1
p0
  1: 1.0
  2: 1.0
coder@move-on:~/development/biobug/test$ java Test test1.nex test4
Will process file test1.nex tree test4
p0
  1: 0.1
  p1: 0.4
    2: 0.2
    3: 0.3

So, tree 1 is composed of nodes (leaves) 1 and 2 and the inner node which was named p1. Tree 2 has distances.

By the way, we will also want to know which trees are in the file.

Lets start!.

So, we start by loading and parsing the file:

import org.biojavax.bio.phylo.io.nexus.*;
 
[...]
        //file is a String with the name of the file to be processed
        NexusFileBuilder builder = new NexusFileBuilder();
        NexusFileFormat.parseFile(builder, new File(file));
        NexusFile nexus = builder.getNexusFile();

Nexus files have several blocks (Taxa, Data, Tree, Set). We are interested in getting the Tree block, lets do a function for that:

    TreesBlock getTreeNode(NexusFile nexus) {
        Iterator it = nexus.blockIterator();
        NexusBlock block;
        while(it.hasNext()) {
                block = (NexusBlock)it.next();
                if (block.getBlockName().equals("TREES")) {
                        return (TreesBlock)block;
                }
            }
            return null;
    }

We get the nexus block iterator and go through it until we find a block whose name is TREES, and return that block.

No that we have the TREES block, lets print the names of all trees:

    void printTrees(NexusFile nexus) {
            TreesBlock node = getTreeNode(nexus);
            Map trees = node.getTrees();
            Set keys = trees.keySet();
            System.out.println("Trees:");
            for (Object obj : keys) {
                System.out.println(obj);
            }
    }

We get a map, where the key is the name of the tree and the value would be the tree as, essentially, a String based (but with some annotated info) representation (not a graph). Now, given a certain name, lets get the graph:

import org.biojava.bio.seq.io.ParseException;
import org.jgrapht.*;
import org.jgrapht.graph.*;
 
[...]
 
    WeightedGraph<string , DefaultWeightedEdge> getTree(NexusFile nexus, String name)
    throws ParseException {
        String topNode;
        TreesBlock node = getTreeNode(nexus);
        WeightedGraph</string><string , DefaultWeightedEdge> graph = node.getTreeAsWeightedJGraphT(name);
        topNode = node.getTopNode();
        System.out.println("The top node is: " + topNode);
        return graph;
    }
</string>

Note that getTreeAsWeightedJGraphT will do some parsing, so ParsingException can be raised. Note also that the top node name can be retrieved (in the case of tree test1, that will be named p1). Some considerations: You can change the rules to create internal nodes; if there are clashes of names inner nodes will be renamed (not leaves!).

Regarding the top node, we call it top node and not root node. While from a data structure perspective the tree has a root, from a phylogenetic perspective the tree might be rooted or not (in which case being root has no meaning, and it is really just a simple weighted acyclic graph). How to know if the tree is rooted? Remember the function to get all trees (getTrees)? The value of the map has a method called getRootType. So, to know if is rooted, you need to use that function. Not the best design… but at least works.

Ok, now we just need to print a tree…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

    static void dump(WeightedGraph<string , DefaultWeightedEdge> graph,
            String parent, String node, String depth) {
        Set</string><string> verts = graph.vertexSet();
        String vertex = "";
        for (String candidate : verts) {
            if (candidate.equals(node)) {
                vertex = candidate;
                break;
            }
        }
        System.out.print (depth + vertex);
        if (parent != null) {
            System.out.print (": " + graph.getEdgeWeight(graph.getEdge(parent, vertex)));
        }
        System.out.println();
        for(DefaultWeightedEdge e: graph.edgesOf(vertex)){
            if (graph.getEdgeSource(e).equals(node)) {
                dump(graph, vertex, graph.getEdgeTarget(e), "  "+ depth);
            }
        }
    } 
</string>

Ok, this is the complicated part. Note the following:

• The complexity has to do with processing graphs
• dump is a recursive function
• Node is synonym with Vertex
• Notice the important bit, if you know that there is a node called “bla”, it is not enough to do graph.containsVertex(“bla”). The answer will probably be false. Remember that one thing is reference (which we have here, i.e. ==) and not content equality (.equals). See below, a remainder
• Finally we go through all edges referencing the current vertex and choose the ones that start on the current one. Again, if the tree is unrooted, the notion of direction does not apply, but it is still good to do a “tree” traversal

We end here.

Regarding the “equal” issue remember that:

        System.out.println("a" == new String("a"));
        System.out.println("a".equals(new String("a")));

Returns false, true. By this order. This is important when traversing the graph. If you know that the reference is equal (and it is when we getEdgeTarget) than one could use it. If you don’t know (like you pass a String that you have constructed yourself or got from some other place), then one needs to go through the vertex/node list and do a .equals to get the correct vertex.

A small example with all the above, is here, ready to use.

Simple Phylogenetic viewer

Preamble:

1. This is newbie code: Handle with care! My main objective is not to make a tree viewer but a tree comparer. So this is no more than a learning step.
2. You can test it yourself as it is a Java WebStart application, just click here. You don’t need to have a phylogenetic tree file yourself. I supply an example inside.
3. This makes use of JGraph and Archaeopteryx (the PhyloXML parser)

I do maintain this code on github. I have one project for the viewer and another for general utilities. All the code is still very crude, but you might be interesting in stealing some of the swing code, either as a crude example of how to interact with swing or taking my micro-DSL for menus. If you want to interact with JGraph, this might be a starting point. I don’t want, in any way, suggest that this code is any good.

Some lessons that I’ve learned and that I would like to share:

1. Some of the clojure.contrib code is a bit green. I tried to use the graph library, but it is very small and specific. I ended up starting doing my own. Mine is even smaller and specific, no claims of generality.
2. I don’t appreciate some of the core functions of Clojure (I’ve written on this before and will write more in the near future). The great thing about Clojure is that you can import only what you want from the core and extend it yourself. I intend to do just that for my personal use. This is a PLUS point for Clojure: the flexibility that is made available to change many of the decisions of the language implementor (in the great tradition of declarative and homoiconic languages)
3. While I can change the core for my uses, I think defnk should really be core for everybody! I fact I wander if defn should not become defnk…
4. I am pretty sure that when *warn-on-reflection* is activated and action taken to correct the warnings, lots of code will increase in performance. With the more important side effect of annotating the code with type info.
5. I have quite a lot of recursive code that doesn’t use recur. Something to learn and master…
6. JGraph layout algorithms are not fantastic. I’ve tried with much bigger trees and the result was far from perfect (I also noticed performance problems in my own code).

The biggest hurdle that I’ve found was the construction of user interfaces and how verbose Clojure Java interop can become. Of course one can create functions (and that was done) to create buttons, frames, menus, etc. But the creation of Java container structures (think frame contains menubar which contains menu with menus inside and so on) would benefit from a dialect where, when a certain (container) object was created it’s (Java) namespace would become easily available.

Imagine constructing a Structure like this:

MenuBar[
    Menu(File) [
        Menu(New)
        Menu(Close)
        Separator
        Menu
    ]
    Menu(Edit) [
        Menu(Cut)
        Menu(Paste)
    ]
]

it would be nice to be able to write something like:

((new JMenuBar)
   (add ((new JMenu "File")
       (add (new JMenu "New"))
       (add (new JMenu "Close"))
       (addSeparator)
    )
   (add ((new JMenu "Edit")
       (add (new JMenu "Cut"))
       (add (new JMenu "Paste"))
    )
)

“add” and “addSeparator” are Java methods. All this would be dynamic against the Java object hierarchy (not a hand-written library!). Note that there is no doto special form (or variants) and, most importantly, note that, given a list (a b c d), if a is a Java object b c d are evaluated as methods of a. If b is (i (y x s)), x and s would be evaluated as methods of y, if they failed then as methods of a, if this fails interpreted as normal Clojure. Something like this (rough sketch).
This would be useful, e.g., to construct Swing hierarchies by hand in a expedient way (not suggesting anything more, especially not to do big programs with outside scope).
I am going to try to write some code that does this in the next few days.

It is not clear to me that the exercise is conducted in the same way as the original one [Update: the author actually says this in the preamble]. Let me explain:

The original Prolog problems are solved without the help of the (existing) Prolog libraries, just using the basic language mechanisms. They are a good at illustrating the underlying declarative power of Prolog.

For instance, problem 1, finding the last element of a list is solved with this in Prolog:

% P01 (*): Find the last element of a list
 
% my_last(X,L) :- X is the last element of the list L
%    (element,list) (?,?)
 
% Note: last(?Elem, ?List) is predefined
 
my_last(X,[X]).
my_last(X,[_|L]) :- my_last(X,L).

Notice the in-code comment that “last is predefined”. In fact, using the Prolog library this could be done with a one liner:

?- last([1,2,3],E).
E = 3.

The offered solution in Clojure is also a one liner:

user=> (last [1 1 2 3 5 8])
8

Given a sufficiently large and clever library (and Clojure has a very nice library) all problems on the list could be solved with a one-liner.

In my opinion, an apples-to-apples comparison with the original solutions would not use the core library.
It would probably be like this for the same problem:

(defn mylast [l]
  (let [mynext (next l)]
    (if (nil? mynext)
      (first l)
      (mylast mynext)
    )
  )
)

Yep, next is in the core library also, but being a call to clojure.lang.RT, I think it is fair game to use it.

Ok, better yet, with recur, as it is on core (this is essentially a copy of the core version):

(defn mylast [s]
  (if (next s)
    (recur (next s))
    (first s)
  )
)

The Prolog exercise exposes the declarativeness and expressive power of Prolog. The Clojure example exposes mostly the cleverness of the core library.

Both are interesting points of view (I am not criticizing the Clojure solution), but they cannot be used for comparison purposes.