cleancode.se

Refreshing caches in ehcache

2012-03-31T00:00:00Z

TL;DR

To (almost) never have a request hit a cold cache:
annotate your cached methods with:

@Cacheable(cacheName="propertyCache"
	,decoratedCacheType= DecoratedCacheType.REFRESHING_SELF_POPULATING_CACHE
	,refreshInterval=[oldTimeToLiveInterval])

and in ehcache.xml change timeToLive to timeToIdle.

The full story

Ehcache has several ways to populate its cache. The sandardway is When a request is made and the value isnt in the cache. The cache forwards the request to the method beeing cahed. If you for example have a database call cached the call to the database will be made. This is whats commonly known as a read through cache. There is one thing to note: The method call is not blocking. If a second call to the same method is done before the first call populates the cache. The second call will also forward to the method.

Ehcache has a way to change this behaviour called decorators. And ehcache spring annotations provide some with annotations.

The first one solves the problem mentioned above. The only difference to the standard cache is that it is blocking. So if 10 calls comes at the same time, it will still just render one call to the database. To use the blocking cache annotate the method as follows:

@Cacheable(cacheName="propertyCache"
	,decoratedCacheType= DecoratedCacheType.SELF_POPULATING_CACHE)
    public List<String> getProperties(int id) {
	//call database
	}

You can also add selfPopulatingTimeout=1234 to set the time to wait before giving up and forwarding to the method.

I would say this solution is good enough. But, the second annotation provided is a self refreshing cache. all you need to do is to change the above to:

    @Cacheable(cacheName="propertyCache"
    ,decoratedCacheType= DecoratedCacheType.REFRESHING_SELF_POPULATING_CACHE)
    public List<String> getProperties(int id) {
	//call database
	}

you can also specify the interval for the refresh by adding “refreshInterval=1000” to the annotation. I added “refreshInterval=[value i had for timeToLive in ehcache.xml]”, to keep the caches to be updated in the same interval as before the change.
This will cause ehcache to save the metod calls used when getting a value, and in the background run it again every [refreshInterval] ms. This means your cache will be updated without any http request needing to wait for the slow database call.
You probably also want to open up your ehcache.xml and change timeToLiveSeconds to timeToIdleSeconds. Since the cache will be updated anyway, It is probably not a good idea to expire the cache based on when it was first used (the refresh thread wont reset the timeToLive timer). Since if you do,tThe items will still expire and cause them to be removed and that will cause an empty cache hit. Have in mind, that timeToIdle is probably going to be reset alot more then timeToLive. So if you keep it at the same value as you had timeToLive, you will probably end up with more items in the cache, taking up more memory. But that is dependent on how your application is used.

It is worth noting. That if possible, it is even more effective to be sure to always update the caches when changing a value. Then you can set your caches ttl to extremly long times, and never have to call the database at all. But unfortunatly, that is often very hard to accomplish. For one, you most likely need to use something like terracotta to cluster your jvm’s. And to keep track of all chages is most likely a very hard task. It can be made easier if you plan for it from the begining. For me and the application i am working on. We dont realy have a database load problem and can easily let the caches be updated pretty often and just refresh some keys or entire caches when something changes.

Productivity tools for java developers

2012-03-03T00:00:00Z

As a developer, i use alot of tools, some help more then other. For a while now, i have been forcing myself to learn new tools to see what could increase my productivity. I thought maybe my learnings could be of some help.

jrebel

I remember the time before i found jrebel. I will never go back. Not having to redeploy is incredibly important. The time from implementation to feedback needs to be kept short, and with jrebel it is close to 0. It isnt perfect, there are some things that cant be reloaded, it uses some extra memory but is worth it.

tmux

I have used screen for a long time togheter with irssi. But not much else. I also loved the split screens in emacs for a long time. With tmux you can have split screen terminals that live between sessions. Using split screens and multiple terminals in one window makes it easier for me to find the terminal i want. I have a terminal full screen on one monitor, and switch window for when im coding, deploying, or any other type of task.

zsh

Although bash is great, zsh is the next step. smarter auto-completion, a history that works, and so much more. you can probably get your bash to do most of the great stuff from zsh, but unless you are a hardcore bash lover. I dont see any point in that. zsh just works. Also take a look at oh-my-zsh (https://github.com/robbyrussell/oh-my-zsh). For plug in and theme handling.

sublime

I must admit I’m very new to sublime, just about a week, but the love came fast. It does most of what my beloved netbeans does, but faster, and while using way less memory. Dont get me wrong, it is not an ide, it is an editor. But i still use it on my laptoptop for some java. It is just easier to get up and running. And with jrebel and maven cli, it just works great. But mostly i use it as a text editor, and to edit non java code.

vi(m)

first of all, im not saying vim is better then emacs (or the other way around). You should learn vim anyway. Not to replace your ide or editor of choice. But instead of using nano to edit config files and similar while on ssh. I dont think i have ever used a server without vi installed, but they hardly ever have for example emacs. And if im going to use vim, i should be good at it.

bash/awk/sed/cut/a good scripting language

Even if you are working with java al day long. Some tasks are more suitable for a small bash script or a oneliner with awk/sed. Writing a small shellscript is often incredibly fast. And when you know what you can do, you will find more simple scripts to make your everyday easier.
I also recomend learning a more powerfull scripting language, i prefer python but ruby or any other language that you most likely can run on any server is just as good. Some tasks are just to big for bash, but you still want it as a script.

Simple configuration handling in tomcat

2012-02-13T00:00:00Z

Keeping configuration in sync and making it easy to change is extremly important. When doing changes in production you want to know that thera are no human errors in editing. And you want to be sure that the exact same configuration was tested in development before putting it live. There are several solutions that solve this problem. My favorite and probably the most known is puppet. But although puppet seems awesome, for many reasons we are not ready to implement puppet at work now. But i still wanted to fix my problem with configuration handling, atleast for some things. My first target beeing server.xml of tomcat. Mostly since we have been tuning our connectionpools alot lately, and it is tedious to change a number in 15+ places everytime. I looked around and didnt find anything simple enough. So i went the classic way and did it myself. I decided to use python (despite i don’t really know it), mostly because i know al the
servers can run python and its a simple language.

My script ended up as follows:

import os, sys,host	Info
f=open("server.xml", "r")
text = f.read();
text = text.format(**hostInfo.info);
print text

where hostinfo is a python file with a map

info={
'mysqlHost': '192.168.65.1,
'hostname': 'myawesomeserver',
'logdir' : '/mount/logs/myawesomeserver',
'workerName' : 'tomcat1'
}

And thats basicly it. i can now in my server.xml for example write:

   <Resource name="jdbc/MySQL/myawesomedb" auth="Container"
                   type="javax.sql.DataSource"
                   initialSize="1" maxActive="10" maxIdle="4" maxWait="10000"
                   username="user" password="topsecret" driverClassName="com.mysql.jdbc.Driver"
                   url="jdbc:mysql://{mysqlHost}:3306/myawesomedb"
                   />

Now if i change the configuration, i can simply push it out to our stage enviroment first. Where i have one mysqlHost. And when i know the change works and its time for production. I can be sure to push out the exact same configuration to al my production servers without any risk of human errors. And al it took was 5 lines of python.

This can (and should) be more generalised to take other files then server.xml. To actually push the configuration out to servers i can think of several ways. Either run it local for each server and pipe the output into files and push thoose into git/svn. Or you can keep the template in svn, and just keep the hostinfo.py on each server and run the script there.

For me, this was a solution so obvious and simple that i at first didnt think it was worth a blog post. But then on the other hand. It took me several years to think of this obvious simple idea. So maybe, just maybe. Im not alone.

Lessons learned from ehcache.

2012-02-07T00:00:00Z

I recently took over a webpage that is using ehcache for caching. I had never used ehcache before and from start i was far from impressed. It was alot of xml, i found the get from caches was much slower then what i am used to see in for example memcached. After some more looking around in the code i found a CacheManager implementation and a cacheKey generator. The later was my biggest surprise. To generate a key for a cache (get or put) it used a deep json representation of al the arguments of the method called. This included some methods that used Lists and other complex datastructures as arguments. This ofcource had to be changed. So i upgraded ehcache and added ehcache-spring-annotations to get rid of the xml. And i used the default deep hashcode key implementation provided by the annotations library. The upgrade and conversion went fairly smooth. But i learnt some things that i thought i should share:

Ehcache doesnt have a default eviction policy, or rather. the default eviction policy is to not remove dead elements until the cache is full. An element is only ever removed if the cache is full. Or when a get of the element realizes that the element is to old, then it is purged and a new item is generated. This behaviour can be changed in your applicationcontext by adding an eviction policy and setting a time in minutes for how often the cache should be cleaned.

    <ehcache:config cache-manager="cacheManager">
        <ehcache:evict-expired-elements interval="60"/>
    </ehcache:config>

One hard to find bug was introduced in the migration. Some values seemed to randomly suddenly get the default value, it came and went by flushing the cache. After alot of debbuging, we found that it was since one method that took an Integer as argument, made a difference between 0 and null in the inside the method. And it seems that in the hashcode based key generator, null and 0 get the same hashcode. Not a big supprice but still. And the real problem was that the code used null as a value, but it is something to watch out for.

Since the annotations create a proxy of the class, they require an interface, and that the call to the method is from otuside the class. otherwise it wont pass through the proxy created by the annotation. You wont get notified by this in any other way then that the cache will stay empty.

I had pretty much only used memcached before, so this came as a big surprise to me. By default, when you get from the cache, you get the actual instance of the object you put there. This of cource makes sense when you think about it, but i wasn’t ready for it. The effect is that if you change a mutable item after you get it from the cache, you also change it in the cache. You cant treat the instance you get as a unique instance. There is a setting to enable copy on read and copy on write. I decided to take care of that myself, since most items are immutable anyway and those that aren’t i know how to copy faster then just serializing them back and forth. The best solution is of cource to only have immutable objects.

Most (if not all) of this is things you can find in the documentation of ehcache and ehcache-spring-annotations. But i guess im not the only one who has a hard time to find it al. Especially if you arent looking.

Overall i must say, that now when i am getting used to ehcache. It feels pretty nice. I love the fact that you dont have to do the normal get from cache, check if null dance that im used to from just using memcached.

Getting started with Moustache and Enlive

2011-01-04T00:00:00Z

A while ago i wrote a tutorial for compojure. Since i have switched to using enlive and moustache almost al the time. I think it is time for them to get a tutorial aswell.

The goal for this tutorial will be to write a simple page for bookmarking.

Getting started

I will be using Leinigen, you can ofcource use Cake without changing much (if anything at al).
Start with

lein new FeedTutorial

I have added all the dependencies that will be used, so we dont have to go back later:
My project.clj

(defproject FeedTutorial "1.0.0-SNAPSHOT"
  :description "FIXME: write"
  :dependencies [[org.clojure/clojure "1.2.0"]
                 [org.clojure/clojure-contrib "1.2.0"]
                 [enlive  "1.0.0-SNAPSHOT"]
                 [net.cgrand/moustache "1.0.0-SNAPSHOT"]
                 [ring/ring-core "0.2.5"]
                 [ring/ring-devel "0.2.5"]
                 [ring/ring-jetty-adapter "0.2.5"]])

Now, open up your core.clj. The first thing we need is the basic code for moustache. We will need jetty to run it, and we will need a simple route to display something:

(ns FeedTutorial.core
  ( :use net.cgrand.moustache
         net.cgrand.enlive-html
         ring.util.response
         [ring.adapter.jetty :only [run-jetty]]))

(def my-app-handler
     (app     
      [""] (-> "My own page!" response constantly)))

(run-jetty #'my-app-handler {:port 8080
                                  :join? false})

app is macro, it helps us to create routes in an easy syntax.
the [""] mapps to the root, ie when jetty is running you can go to http://localhost:8080/ and se “My own page!”
if you change [""] to for example [“wow”] it will instead map to http://localhost:8080/wow

The “response” function takes a string and creates a response map. We can run it in the repl:
FeedTutorial.core> (→ “Sometext” response)
{:status 200, :headers {}, :body "Sometext"}

In case you havent seen → before. it is a macro that just makes the code easier to read.
(→ a b c d) is the same as (d (c (b (a))))

So we could write:

FeedTutorial.core> (response "Sometext")
{:status 200, :headers {}, :body "Sometext"}

The constantly function just takes anything and creates a function that takes any number of arguments and returns what you gave it.
For example:

FeedTutorial.core> ((constantly 1) 2)
1
FeedTutorial.core> ((constantly 1) 2 3 4 5 6 7)
1

In other words:
(→ “My own page!” response constantly))
creates a function that always returns a response with the body “My own page!”

Adding a template

As i said earlier, i will use enlive, wich is a templating engine. Lets see some code, we create our first template:

(deftemplate index "FeedTutorial/index.html"
  [])

Now all this template does, is that it loads the FeedTutorial/index.html file. This ofcource means we have to create it. It should be located in:
FeedTutorial/src/FeedTutorial/index.html that is the same folder as your core.clj.
At this point it doesnt realy mather how it looks, i created the following:

<html>
  <head>
    <title>My page!</title>
  </head>
<body>
   I am using a template!
</body>
</html>

Calling the function at the repl yields:
FeedTutorial.core> (index)
(“\n \n My page!\n \n\n I am using a template!\n\n\n”)

Note: if you get a nullpointerexception, you probably put the index.html in the wrong folder.

As you can see it is simply the html code in a list as a string.

We also need to add a route for it. I moved our old route to “hello”, you can remove it if you like. My routes look like the following:

(def my-app-handler
     (app
      [""] (-> (index) response constantly)
      ["hello"] (-> "My own page!" response constantly)))

As you can see it is the same as before, except that i wrap “index” in paranthesis to call it. rendering the string.
Now if you browse http://localhost:8080/ you will see the html you wrote in index.html

Making changes to the template

Displaying a static .html file is not realy magic. To make changes, we first add something in the .html file that we want to change.
I added:

   <div id="bookmarks">
     
   </div>

To add some text in that div, using enlive. We expand our template:

(deftemplate index "FeedTutorial/index.html"
  []
  [:div#bookmarks] (content "I changed the html"))

As you might have guessed. that row says:
“Find the div with id ‘bookmarks’ and set the content to ‘I changed the html’”
If you happen to know jquery, it is equal to:

$("div#bookmarks").html("jquery changed me");

If you now browse to http://localhost:8080 you should see the text “I changed the html” (in addition to any other text you had on the page before)

Now, lets say we have a vector of strings:

(def bookmarks ["Something happend!" "More news." "Very cool stuff"])

And we want that to be a list on our page?
First: we add a list tag to our html.

   <ul id="items">
	<li>This will be replaced</li>
   </ul>

Then we update our template:

(deftemplate index "FeedTutorial/index.html"
  [items]
  [:div#bookmarks] (content "I changed the html")
  [:ul#items :li] (clone-for [item items]
                          (content item)))

And since we added an argument (items) to the template, we need to update our route:

      [""] (-> (index bookmarks) response constantly)

As you can see, it now takes one argument “items” that is the sequence to be displayed as a list.
[:ul#items :li] selects the list tag in the unorderd list items. Then the clone-for function makes a copy of the list tag for each item in items. and sets its content to the content of item.

in other words you should now see:

Something happend!
More news.
Very cool stuff

Very pretty!
Now, maybe we want to display the link to the source of the bookmark aswell.

We can update our bookmarks vector to contain the urls aswell:

(def bookmarks [ {:title "Something happend!" :url "www.someurl.com"}
                  {:title "More news." :url "www.newssite.com"}
                  {:title "Very cool stuff" :url "www.verycoolsite.com"}])

A list is not very suitable to display this. So lets change the html file to contain a table instead.

   <table id="items">
    <tr class="bookmark">
     <td class="title"></td>
     <td class="url"></td>
    </tr>
    </table>

We ofcource also need to update the template:

(deftemplate index "FeedTutorial/index.html"
  [items]
  [:div#bookmarks] (content "I changed the html")
  [:table#items :tr.bookmark] (clone-for [item items]
                                [:td.title] (content (:title item))
                                [:td.url] (content (:url item))))

I changed the ul to be the table. and point the first selector to the :tr. Since that has children, ie two :td elements. We can clone them and change their content. I find this an extremly elegant way to do this. We can move the table anywhere we like in the html, we can style it with css. We can do anything view related without havint to see any clojure code.
If you visit localhost:8080 and view the source. it should look something like:

   <table id="items"> 
    <tr class="bookmark"> 
     <td class="title">Something happend!</td> 
     <td class="url">www.someurl.com</td> 
    </tr><tr class="bookmark"> 
     <td class="title">More news.</td> 
     <td class="url">www.newssite.com</td> 
    </tr><tr class="bookmark"> 
     <td class="title">Very cool stuff</td> 
     <td class="url">www.verycoolsite.com</td> 
    </tr> 
 
    </table>

We can now for example add a table header in our html file:

   <table id="items">
    <tr class="header">
      <th>Title</th>
      <th>URL</th>
    </tr>
    <tr class="bookmark">
     <td class="title"></td>
     <td class="url"></td>
    </tr>
   </table>

To get the page to update you need to evaluate the enlive template again.

My full project now can be found at:

Here at my GitHub account

That is my files as they looked at this point.

I removed the initial “i am using a template” text since it isnt used.

We should use the bookmarks div for something more usefull. We can put a header there for example!

Change the Template to:

  [:div#bookmarks :*] (content "My own bookmarkingpage!")

The :* is not a smiley, it selects anything. I choose to use it so we can change from h1 to h2 or something else. Without having to go into the clojure code.

In the html file i put:

  <div id="bookmarks">
    <h1></h1>
  </div>

Adding new content

We have a very static page, since the only way to change the content now is to change the clojure code itself. We can change this by adding a /addBookmark route

First we need a function to handle the request:

(defn add-bookmark [request]
  (response (str request)))

For now, it just returns a response displaying the http request as a string
And add a route for it and tell moustache that it should call add-bookmark2 .

      ["addBookmark"] add-bookmark

As you can see, we dont say anything about the request, or dont pass any parameter into add-bookmark, even though it requires one. Moustache takes care of this for us.

If you browse to your localhost:8080/addBookmark you will now see your request map. mine looks like this:

{:remote-addr "0:0:0:0:0:0:0:1", :scheme :http, :request-method 
:get, :query-string nil, :content-type nil, :uri "/addBookmark", 
:server-name "localhost", 
:headers {"accept-charset" "ISO-8859-1,utf-8;q=0.7,*;q=0.3", 
"accept-language" "en-US,en;q=0.8", "accept-encoding" "gzip,deflate,sdch", 
"user-agent" "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.15 (KHTML, like Gecko) Chrome/10.0.612.1 Safari/534.15",
 "accept" "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
 "connection" "keep-alive", "host" "localhost:8080"}, 
:content-length nil, :server-port 8080, :character-encoding nil, 
:body #<Input org.mortbay.jetty.HttpParser$Input@1faf67f0>}

Since we are adding a new bookmark, we want to use request-method post. To be able to post you can use curl, find a plugin for your browser, or write a simple html form. I will use curl.
I want the parameters to be called “title” and “url” so my curl command would be:



curl -d “title=a title&url=www.myurl.com” localhost:8080/addBookmark

The response i get now is:

{:remote-addr "0:0:0:0:0:0:0:1", :scheme :http, 
:request-method :post, :query-string nil, 
:content-type "application/x-www-form-urlencoded", 
:uri "/addBookmark", 
:server-name "localhost", 
:headers {"content-type" "application/x-www-form-urlencoded", 
"content-length" "27", "accept" "*/*", "host" "localhost:8080", 
"user-agent" "curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18"}, 
:content-length 27, :server-port 8080, :character-encoding nil, 
:body #<Input org.mortbay.jetty.HttpParser$Input@769652dd>}

It says nothing about my title or url! We need to tell moustache to wrap the parameters. we use the ring middleware “wrap-params” for this. My routes now look like:
(after removing the now unused “hello” route)

(def my-app-handler
     (app
      wrap-params
      ["addBookmark"]  add-bookmark
      [""] (-> (index bookmarks) response constantly)
      ["raw"] (-> (raw) response constantly)))

You also need to use wrap params in the ns declaration:

(ns FeedTutorial.core
  ( :use net.cgrand.moustache
         net.cgrand.enlive-html
         ring.util.response
         ring.middleware.params
         [ring.adapter.jetty :only [run-jetty]]))

Now if we try our curl again:

curl -d "title=a title&url=myurl.com" localhost:8080/addBookmarks
{:remote-addr "0:0:0:0:0:0:0:1", 
:scheme :http, :query-params {}, 
:form-params {"url" "myurl.com", "title" "a title"}, 
:request-method :post, :query-string nil, 
:content-type "application/x-www-form-urlencoded", 
:uri "/addBookmarks", 
:server-name "localhost", 
:params {"url" "myurl.com", "title" "a title"}, 
:headers {"content-type" "application/x-www-form-urlencoded", 
"content-length" "27", "accept" "*/*", "host" "localhost:8080", 
"user-agent" "curl/7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18"}, 
:content-length 27, :server-port 8080, :character-encoding nil, 
:body #<Input org.mortbay.jetty.HttpParser$Input@44fc63be>}

Now it is there! nice and tidy in a map we can reach using the :params key. To clean up a little, we can use destructuring in the add-bookmark function and only fetch the params.h

(defn add-bookmark [{params :params}]
  (response (str params)))

We now get a much nicer response:


 curl -d "title=a title&url=myurl.com" localhost:8080/addNews
{"url" "myurl.com", "title" "a title"}

To be able to add bookmarks in a nice way. We can wrap it as an agent:

(def bookmarks (agent [{:title "Something happend!" :url "www.someurl.com"}
                       {:title "More news." :url "www.newssite.com"}
                       {:title "Very cool stuff" :url "www.verycoolsite.com"}]))

We then have to replace references to bookmarks with @bookmarks. The route for example will now be:

      [""] (-> (index @bookmarks) response constantly)

To add to a vector, we can use conj. And since our bookmarks are in an agent we have to use send to send a function and its arguments to the agent.

An operation with conj without the agent could look like:

(conj @bookmarks {:title "test" :url "www.test.com"})
[{:title "Something happend!", :url "www.someurl.com"} 
{:title "More news.", :url "www.newssite.com"} 
{:title "Very cool stuff", :url "www.verycoolsite.com"} 
{:title "test", :url "www.test.com"}]

Here i deref bookmarks to get the vector.

if we deref bookmarks again, we will see it hasnt changed.

@bookmarks
[{:title "Something happend!", :url "www.someurl.com"} 
{:title "More news.", :url "www.newssite.com"} 
{:title "Very cool stuff", :url "www.verycoolsite.com"}]

Since we want the change to be persistent. We have to send conj yo our agent:

(defn add-bookmark [{params :params}]
  (send bookmarks conj {:title (params "title") :url (params "url")})
  (response (str params)))

As you can see, send is used by sending the function and its arguments to an agent. that function is then applied to the content of the agent.

Using curl to add new bookmarks is not a good way in the long run. I took a look at instapaper, read it later and similar. And changed one of their bookmarklets to work with our own bookmarking tools!

javascript:function%20iprl5(){
var d=document,z=d.createElement('scr'+'ipt'),
b=d.body,l=d.location;try{if(!b)throw(0);
z.setAttribute('src',l.protocol+'//localhost:8080/addBookmark?title='+encodeURIComponent(d.title)+'&url='+encodeURIComponent(l.href));b.appendChild(z);}
catch(e){alert('Please wait until the page has loaded.');}}iprl5();void(0)

You can create a bookmark in your browser with this as url. Now you can click on it while on any page, and it will add it to your bookmarks page.
I am not a javascript genie, But it works. It sends a call to localhost:8080/addBookmark with title set to the title of the current page, and url to the url of the current page. Very nice!

I think this will be enough for this time. There is still alot you can do, and maybe il write a part two.
Dont be afraid to leave feedback, good or bad. I cant improve unless i know what is wrong.

The “done” version can be found here:
Here at my GitHub account

What is Clojure good at?

2010-10-17T00:00:00Z

I was recently asked the question “What is Clojure’s sweetspot?”. I ofcources started rambling about how nice it is for concurent applications, that you can make a map statement run in multiple threads by simply changing “map” to “pmap”. That you dont have to care about locks and so on.
But then i was asked about a real world problem or something that you would encounter in your real day job. I found this alot harder to answer, the only thing i could come up with on the spot was data intense applications like flight caster. But i couldnt realy let the question go. Lets start with what clojure is not.

Clojure is not a better java

I often hear “scala is a better java” and i agree with it. With scala, you can continue to write code like you do with java. It will just have less and more effective syntax. This is even the prefered aproach, and makes it easy to introduce scala to new developers. With clojure, this is not the case. You have to think in totaly new ways and it can be realy hard in the begining. But in the end, it is very rewarding. Clojure (and lisp in general) is one of thoose languages that if you learn it, and never get to use it in production. You will still have had great benefit from knowing clojure and the functional ways of thinking. It will affect your coding in java.

Clojure is great at java

Clojure realy is great at java, maybe even better than java itself. It might sound weird at first, but let me explain.
First of all, the repl is a killer. This is not something that is unique to clojure, but that doesnt make it less good. Beeing able to try out new api’s and fumble forward is great.
Second, there is no compile time. It is not that the compile time is short, it doesnt exist. Making a change to a webpage will be visible on next reload. All you do is re-evaluate the function you changed.
Third, functions without state and side effects are alot easier to test then java code. Yes you can still create hard to test code, but it is harder.
Fourth, you can abstract away more code. Take memcached for example. When using memcached, you have to

 collect the value from the cache,

check if it is null, 

if so calculate it. 

store it in memcached 

return it.

You cant get rid of thoose 4 lines of extra code in every method (you probably can with annotations but i havent seen it?). In clojure you write a function call it for example

 with-memcached

You can then simply

 (with-memcached 

	(some-heavy-code) “memcachedkey”)

and it will check the cache and so on for you. Same goes for performance timing. how many times have you written

long start = System.currentTimemillis(); // (or nanos..) 

{some code } 

System.out.println("time: "+ (System.currentTimeillis()-start);

in clojure you can just do

(time (my-function))

I could go on, but im not going to, my point is not to say that Clojure is the best language ever made, or that everybody should learn Clojure. My point is, that if you want to learn and use clojure, it is not only suited for concurent code with big amounts of data. It is suited for most things. It might not be the best language for everything, but then again, no language is.

A small disclaimer, note that i am not saying that Clojure is better then Scala or Groovy or anything else. I am just saying that Clojure is great. You can do alot/everything i mention in for ex Scala, that is besides the point.
I would love too see someone else write something similar about scala/jruby/groovy/whatnot, It might even already be some posts about it? I would love if someone could help me with more arguments for clojure in everyday life, there is probably a blogpost somewhere out there about that aswell?

Writing pretty Database code in java.

2010-10-14T00:00:00Z

I write alot of database code in java. As an effect, i have a code macro in netbeans. So when i write:

sql[tab]
it expands to:

Yes that is alot of code. And it doesnt even collect the result, or contain the sql needed.
I got very tired at looking at al this code. I thought alot, tried to extract some usefull method or anything to show the intent of the code more.

Fortunatly, in java 7 this will be improved. Thanks to the new resource management.
It will then look something like:

This is a great improvement, but i think we can do better. Also, we wont see java 7 anytime soon. And even when we do, its not sure our legacy projects can use it.

Enter project lombok: http://projectlombok.org/
Lombok does some nice magic with annotations. We can now get the following:

No try catch at all, we pretty much dont have to add any code that we dont want. Creating a connection, preparing a statement, executing an sql, collecting each row into something. Return the Collection of objects. Simple!

But i have been working in PHP alot lately, and this hurts to say. In php, you can abstract away even more.
You dont even have to create one method for each sql query. you can create a general query. So you can do something like this:

The only way to come close to this in java. Is to just use Object everywhere and cast everything as you need. But then you arent coding in java anymore. Instead, alot of java projects now use something like Hibernate. I am not the biggest fan of it. But it seems to do the trick most of the time.

But, lets say we broaden our minds. Java is so much more then just the language, it is a platform. We can change java the language to something that suits our needs more. Enter groovy!
I havent used groovy alot, but i should have. This is why:

Example Taken from this Tutorial

sql is just an object created with the connection details. This can be reused for all our queries.
Id and firstname are ofcource columns in the table you select from. And you can instead of printing add objects to a collection or do something else that is usefull. And all in one (1!) row. Groovy also has some other nice features for jdbc. If you do alot of jdbc, you should really take a look!

If you want to go abit further from java. But still stay on the jvm, try clojure! Then your code can look like:

Example taken from this page

I love clojure, but between the groovy and the clojure examples of jdbc, i prefer groovy.

I bet you can create Java POJO from sql with jdbc using groovy or clojure, in less code then with Hibernate. And you dont have to clutter your model objects with annotations, you dont have to let Hibernate do bad sql queries, and you get to learn a new language. Go try it!

Getting started with compojure, part 2

2010-09-03T00:00:00Z

This is part two of my getting started with clojure web dev tutorial. In the first part we built a small tasklist, it is not very functional yet. But we made it ourselves!

In this part, we will add javascript and ajax!
To do this, we need to be able to serve static files. This is pretty simple.
Start by creating a folder “files” in your project folder (ie the same folder as your project.clj)
In the files folder, create a javascript.js and enter som text in it.

Now, back to core.clj.
We have to tell compojure/ring that it should serve files. we do this, wrapping our call to myroutes in the jetty function, to make it easier to read, we also move it to a new function:

we will also have to add: ring.middleware.file-info and ring.middleware.file to the (:use ) part of the ns declaration.

Now if you go to http://localhost:8080/javascript.js you should see the contents of your file.

Time for some ajax magic, to make our life simple, we will use jquery, and a jquery plugin called jquery.form.
you can find jquery.form here: http://github.com/malsup/form/raw/master/jquery.form.js?v2.43
and jquery from jquery.org or from google: http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js
download them and put them in your files folder.

Try loading them in your browser: http://localhost:8080/jquery.min.js it should just works.

To include them in our clojure code, we can use some helper functions in hiccup.

Load the form in your webpage again and you should see: <script src=“/jquery.js” type=“text/javascript”></script> and so on in your source.

Now we can do some javascript coding!
Open your javascript.js and enter the following:

#formresult, is the css id of the tag where the result of the formpost should be placed. we want that in a div just under the form. A div is created like this:
[:div {:id "formresult"} “form result will be here”]

#myform, is the css id of the form we want to turn into an ajaxform. We add this to our form:
(form-to {:id "myform"} [:post “/form” ]

so the entire function should look like this:

Now, evaluate the function and try your form. When you hit enter, it will not reload the page. “form result here” will just be replaced with task3 [“task1” “task2”], if you have entered task1,task2 and task3 in your form. Although that is pretty cool. We want it to look as pretty as our /view does. Simple!

What we do is simply to move the part that creates a vector of all our tasks to the let assignment. Then we have it handy and can once again use the (unordered-list) function to make our tasklist pretty. Note the (html that is also added.

Incase you got something wrong, here is my full core.clj

core.clj

Now, this is not realy a tasklist, we cant remove tasks we are done with. This will be the next step:

We start by creating a new route:

(GET “/delete/:task” {session :session, params :params} (delete-task (params “task”) session))

Note, if we didnt need the session, we could have used the short version:
(GET “/delete/:task” [task] (delete-task task session)

The :task turns whatever you write there, into a parameter. So if you go to http://localhost:8080/delete/mytask it will call delete-task with the value mytask.

We define our delete-task
(defn delete-task [task] )
Now for the question, how do ve remove from the vector? Vectors only support removal by index. The easiest way to solve this is to change our vector to a set. Simply change vector? to set? and (vector to (hash-set. Everything still works!

Now we can write our function:

now, create some tasks. Then enter http://localhost:8080/delete/sometask and sometask is deleted! you can verify by using the /view we did in part one.

But lets be honest, this is not a usefull way to handle this, We want to be able to click on a item to remove it.

We solve this with some more jquery:

What we do here, is we create a function called done, this adds a clickListener to all list items. The clickListener calls /delete/list-item-text and on success, hides the item.

This will be enough for the second part.
My full core.clj

update: a running version can be viewed here: http://cljtasks.appspot.com/
Thanks to the excelent guides at: http://compojureongae.posterous.com/ for helping me get it up on app engine!

In the next part, we will try to store our tasklist in a NoSQL database.

Getting started with compojure

2010-08-30T00:00:00Z

The goal of this tutorial is a simple tasklist written with clojure and compojure
I will be using Lein, so if you dont have that. Head over to: http://github.com/technomancy/leiningen and follow the instructions.
With that installed, lets start with:

lein new SimpleTasks

This will create a new project for us.
Lets begin by adding some dependencies that we will need. Open SimpleTasks/project.clj and add the following dependencies:

Save the file and invoke “lein deps”. This will download all the dependencies we need.
If you dont want to use lein, enclojure in netbeans for example works great. All you need to do then is to go to clojars.org and find the dependencies listed above and add them to your project.
The
:dev-dependencies [[swank-clojure “1.2.1”]]
is to be able to start a swank server and connect emacs to it. If you wont be using that, you can leave it out.

Now we can start coding!

Open up src/SimpleTasks/core.clj in your favorite editor. I use emacs with clojure mode and slime. But anything works.

you will see:

(ns SimpleTasks.core)

Now, lets start by getting a webpage running!

This is the basic setup of a simple compojure webbapp.
If you run this and point your browser to localhost:8080 you will see “Hello!”
If you dont know how to run it. the simplest way is to use

“lein repl” wait for the repl to load, and then:
clojure.core=> (require ’SimpleTasks.core)

Now what does al this code do?

(defonce server (run-jetty #’myroutes
{:join? false
:port 8080}))

Starts an instance of jetty at port 8080
join? false, makes the repl return instantly instead of blocking aslong as jetty is running.
#’myroutes tells compojure what function it should called when the server recieves a request.

(defroutes myroutes
(GET “/” [] (display)))

Here we define our routes, as you can see this is the method called by jetty.
It simply forwards any call to “/” ie “localhost:8080/” to the function display

(defn display []
(html [:h1 “hello!”]))

Finally we end up in display. Al it does is render html page with “hello!” in a <h1> tag

Now for a great thing about clojure and compojure.
edit the display funktion so it renders something else. ex:
(defn display []
(html [:h1 “hello world!”]))

To make compojure display this instead, al you need to do is to evalute that function, and reload your browser.

Ok, lets start with the tasklist. The first thing we want is a form to add tasks:

We start by adding another function to the routes:

And this is the render-form function:

Note: you also have to add: hiccup.form-helpers to the :use list in the ns declaration

Now, evalute the defroutes and the render-form functions and point your browser to http://localhost:8080/form
Hint: if you are using the repl you can use: (require ’SimpleTasks.core :reload) to evaluate it again.

your core.clj should now look like: http://gist.github.com/555496

Try to type something in the box and submit the form. You will get a “Not found” error in your browser.
This is because al our routes ar “GET” and the form use “POST”

So we need to add another route:

(POST “/form” {params :params} (handle-form (params “task”)))

Note: if you want a route to take all forms of requests, you can use (ANY “/” (myfunction))
The {params :params} gives us a map of the parameters from the form. Since we only care about the “task” parameter. we pick that and send it to handle-form.

Lets check so the form works and just print the task:

(defn handle-form [task]
(html [:h2 task]))

Load http://localhost:8080/form and enter something and submit. you should see the task you just entered.

In a normal tasklist, you would probably store the tasks in a database or in textfiles. To make it simple, and to learn more of compojure/ring/hiccup. we will use the session.

We begin by taking care of the result from the post:

(defn handle-form [task]
{:body (str “stored” task)
:session {:tasks task}})

Instead of using the (html) function from earlier, we specify the body and session. After this, the task will be stored under the key :tasks in the users session.

To make ring/compojure understand and store data in the session, we also need:

(wrap! myroutes :session)

We also want a way to see what is in the session, this is pretty easy. we start by adding a new route:

(GET “/view” {session :session} (view session))

A simple GET route that in the same way we saw with the parameters from the form, take the session and forward it to a function.

The function that is called:

(defn view [session]
(html [:h1 (str "tasks: " (:tasks session))]))

My full core.clj at this point:
http://gist.github.com/557879

Just beeing able to store one task is not going to work very long. Time to expand our tasks and store it in a vector.
This means that we want to append more tasks to the session, not only replace the information for every task we store.
We will use a simple vector as our storage format. To be able to do this, we need not only the parameters from the form, but also the current session. We change our POST route:

(POST “/form” {session :session, params :params} (handle-form (params “task”) session)))

we now have the session and the task parameter from the form.
we change the form handler:

What this does, except for print the task from the form, and the tasks from the session. Is that it checks if the value of the session is a vector.
If it is. the task is added to the current vector of tasks. If it is not a vector, it is most likely empty, and we create a new vector with our task.

We dont need to change anything to our view route.
Try it out! We can now add multiple tasks to our simple tasklist!

Time to make that view look like a real tasklist.
We will do this the easy way, add hiccup.page-helpers to the ever groving list of namespaces in :use.
then simply wrap our task vector in (unordered-list)

We now have a list of tasks. This is how mine looks after adding three tasks:

Tasks

task1
task2
task3

This is enough for the first part.
We still have some work todo. We cant set tasks as completed or delete tasks. Later, we will also use jquery to make it smother too use.

update: A question came up on the clojure mailing list HERE about how to reload a namespace if you are not using emacs. Thanks to mark Rathwell who posted the answer:
(require ’[foo.something :as something] :reload)
(require ’[foo.something :as something] :reload-all)

Code reviews

2010-06-15T00:00:00Z

Code reviews are great. They are also often misunderstood.
If someone asks you to check his code, this does not mean that he wants you to make it your mission to find any mistake he made, so you can point and laugh at him. He simply wants you to help him improve the code.
But why should you help him with HIS CODE! it’s his work, cant he do his work on his own? If you have to check al the code he writes, you can just as well write it yourself and take his paycheck, right? No, ofcource not.
First, its not his code, its your code, its your teammates code, and all your future teammates code. YOU are responsible for every single line in that code base, yes everything. I do ofcource not mean you as a person, I mean you as a team. And by that i do not mean that the sum of the code responsibilities of all team-members is the entire codebase, i mean that the team as an entity is responsible for everything.
Second, the point of a code review is to improve, not only to improve your teammate, but also to improve you, and the teamwork of you and the one who wrote the code. You are not better than your teammate because he asks you to review his code, honestly, if he asks for reviews repeatedly and you dont. He is the better developer. Good developers know that they can improve, and that they can learn from everyone around them.

What should you look for in a review? You should make sure you understand what everything does without having to think too much, that the flow is simple to follow and that you dont spot anything obviously wrong. And finally ofcource check so its properly tested and that the tests are clean.
When you find something you dont like or dont understand, discuss it, ask how the writer thought when he wrote it, and why he wrote it that way. It does not have to be something big; it can be a method name that can be misunderstood or a variable that could have a shorter scope.
Also, when you find something that you really like, tell him. Its always good with positive feedback.

When should you do a code review? Well, when you feel you need it, if you are done with something. Ask someone or just everyone in general if anyone has time to check your code. Then take a conference room or just roll over with your chair and sit next to each other. I would even say its preferable to just sit at your workspace, that way, the entire team can hear you in the background and if they hear something that interests them, they can join the discussion.

The best code review, is ofcource the continuous review that goes on al the time, also known as Pair programming. Try it!

Your tools should help you!

2010-06-14T00:00:00Z

The tools and libraries you use should help you, they can help your hardware and so on. but in the end, they should make your life easier.

Take memcached for example, who does memcached help? Does it help your database server? yes, and thats a good thing! Does it help you? no, not realy, it forces you to make sure you invalidate keys, collect values from multiple sources and clutters your code and so on. Im not saying you shouldnt use memcached, im saying maybe you should use something in between, maybe you should find another library that helps you with using memcached. If you cant find such a library, maybe there is something else that can solve your problem, instead of solving the symptom of the problem.

Let me explain what i mean, il continue to use memcached as the example. Why is it you use memcached? because your database cant keep up. The symptom of this is that your webpage gets slow. You “solve” this by giving the database less to do. But what your realy doing is solving the symptom, ie adding memcache and asking your database less, speeds upp the webpage again. But your problem is realy still there: you cant get the data up from the database fast enough. You are just giving your users old data. yes, this data can be unchanged. but its still old.
A solution to the problem, would be to store the data somewhere else, perhaps some nosql storage if that works. Or perhaps you dont even need to store it in a database, if your in java maybe something like terracotta can solve it. I dont realy care, but solve the problem, not the symptom. And more important, the solution should help you, not make you work harder.

This is not meant as a bash against sql databases or memcached, they can be good tools. But they are often missused. My point is, your tools should not force you to work harder, or make your code more error prone. They should help you, and make your life easier, not only your hardwares life. Its not about what tools you use, its how you use them.

Writing code is hard work!

2010-06-11T00:00:00Z

This post started out as a comment to: http://wolfie.github.com/2010/06/09/method-hierarchies.html
But when i had posted the comment i felt that i had more to say, so here we go:

You do not get clean, readable, maintainable code by just writing short methods, or by just writing some junit tests, or by just refactoring some code now and then. You get clean code by hard work, practise, and discussions.

If you encounter a bug in legacy code, and realise it is in a 1500 line class, you dont just find the problem and fix it. You treat the problem.
You do this by writing unit tests until you have isolated the problem (yes it is possible to write tests for the code, if you dont think so. go buy michael feathers book: http://amzn.to/9q7cNn).
When you have isolated the problem with a failing test, you make the test pass. If you have to change code that is not under test, you write tests for it first. Then you fix the problem, in as small increments as possible, with a run of tests for each increment.
When you have fixed the problem. You have most likely improved the design and readability of the class, if you havent. Do one small change to improve it.

Why should you write small methods? why not just keep theese 4 lines in the other method? its the only place it is used anyway!
For several reasons:
It raises the abstractionlevel to extract methods.
It is easier to click on a methodcall to find what it does, then to ignore 4 lines in a method.
It is easier to verify that 4 lines does what they are expected to do, if they are in a method with a good name. Then just 4 lines among 100 in a big method.
It is easier to write a junit test for a 4 line method, then for 4 lines in a 100 line method.

The problem with your codebase is not that someone wrote crap 4 years ago. It is that you didnt improve it everytime you touched it.
It is not someone elses code that is hard to read, everyone should own the code, it is your code that is hard to read!

I love code reviews, not the code reviews that involve one “chief architect” saying “thats not god enough, fix it”. But the code reviews that involve 2-3 people disucssing someones code, why does that method have that name? could we improve a junit test here? could we extract another method? does this class do to many things? and so on. Dont review every single row of code, but thoose lines you review, do it well, discuss it, learn from it, improve from it. The goal of a code review is not perfect code. It is that one year from now, you have improved from it.

I realise now that i havent mentioned TDD or pair-programing. But you already do that anyway, right?

dont forget, software craftmanship is not about speed, it is about quality.