Dev Time

Softwerkskammer Rhein-Main Open Space

2013-02-19T00:39:00.001-08:00

On Saturday I attended an Open Space in Wiesbaden, organized by members of Softwerkskammer Rhein-Main, a very active chapter of the German software craftmanship community. The event took place in the offices of Seibert Media above a shopping mall including a nice view of the city.

The Format

Open Space conferences are special as there is no predefined agenda. All the attendees can bring ideas and propose those in the opening session and choose a time slot and room. Sessions are not necessarily normal presentations but rather discussions so it's even OK to just propose a question that you have or a topic you'd like to learn more about from the attendees. Also, there are some guidelines and rules: sessions don't need to start and end in time, you can always leave a session in case you feel you can't contribute and you shouldn't be disappointed if nobody shows up for your proposed session.

Personal Kanban

Dennis Traub presented a session on Personal Kanban. As I did Kanban style development in one project already I was eager to learn how to apply the principles to personal organization. Basically it all works the same as normal Kanban. Tasks are visualized on a board where a swimlane defines the state of a task with work items flowing from left (todo) to right (done). You can define swimlanes as it fits your habits, e.g. one for todos, one for in progress and one for blocked. The in progress lane needs to have a Work in Progress limit which is the amount of tasks you start and process in parallel. An important aspect is that you don't have to put all your backlog items to the todo lane but you can also keep them in a seperate place. This keeps you from getting overwhelmed when looking at the board.

It sounds like Kanban is a good way for organizing your daily life. For me personally the biggest hindrance is that I am working from my living room and I'd rather not put a Kanban board in my living room. If I'd use a separate office I guess I'd try it immediately.

Open Source

An attendee wanted to know some experiences with Open Source communities. Two full time committers, Ollie for Spring and Marcel for Eclipse, shared some of their experiences. I am still surprised that a lot of Open Source projects have quite some bugs in their trackers that could easily be fixed by newcomers. A lot of people like Open Source software but not that many seem to be interested in contributing to a project continuously. Most of the interaction with users in the issue trackers are one time reports, so the people report one bug and move on. Even for big projects like Spring and Eclipse it's hard to find committers. One way to motivate people is to organize hack days where users learn to work with the sources of the projects but this also needs quite some preparation.

Freelancing

The topic of freelancing was discussed all over the day. Markus Tacker presented his idea of the kybernetic agency, a plan to form a freelance network with people who can work on projects together. We discussed benefits and possible problems, mainly of legal type. A quite inspiring session that also made me think about the difference of freelancing in the Java enterprise world compared to PHP development. Most of the freelancers I know would prefer not to work 5 days a week for one client exclusively but that is often a prerequisite for projects in the enterprise world.

Learning

Learning is a topic that is very important to me so I proposed a session on it. I already switched from 5 to 4 days the last months of my employment at synyx because I felt the need to invest more time in learning which is often not possible when working on client projects. Even now as a freelancer I keep one day for learning only. What works best for me is writing blogposts that contain some sample code. I can build something and when writing the post I make sure that I have a deep understanding of the topic I am writing about. Other people also said that the most important aspect is to have something to work on, reading or watching screencasts alone is no sustainable activity. I also liked the technique of another freelancer: whenever he notices that he could do something different on the current project he stops to track the time for the customer and tries to find ways to improve the project, probably learning a new approach. This is something you are doing implicitly as a freelancer, you often spend some of your spare time thinking about client work but I like this explicit approach.

Summary

All in all this was a really fruitful, but also exhausting, day. Though I chose meta topics exclusively I gained a lot from visiting. Thanks a lot to the organizers (mainly Benjamin), moderators, sponsors and all the attendees that made this event possible. I am looking forward to meeting a lot of the people again at Socrates this year.

Book Review: Gradle Effective Implementation Guide

2013-02-01T03:10:00.000-08:00

PacktPub kindly offered me a free review edition of Gradle Effective Implementation Guide written by mrhaki Hubert Klein Ikkink. As I planned to read it anyway I agreed to write a review of it.

Maven was huge for Java Development. It brought dependency management, sane conventions and platform independent builds to the mainstream. If there is a Maven pom file available for an open source project you can be quite sure to manage to build it on your local machine in no time.

But there are cases when it doesn't work that well. Its phase model is rather strict and the one-artifact-per-build restriction can get in your way for more unusual build setups. You can workaround some of these problems using profiles and assemblies but it feels that it is primarily useful for a certain set of projects.

Gradle is different. It's more flexible but there's also a learning curve involved. Groovy as its build DSL is easy to read but probably not that easy to write at first because there are often multiple ways to do something. As a standard Java developer like me you might be unsure about the proper way of doing something.

There are a lot of helpful resources online, namely the forum and the excellent user guide but as I prefer to read longer sections offline I am really glad that there now is a book available that contains extensive information and can get you started with Gradle.

Content

The book starts with a general introduction into Gradle. You'll get a high level overview of its features, learn how to install it and write your first build file. You'll also learn some important options of the gradle executable that I haven't been aware of.

Chapter 2 explains tasks and how to write build files. This is a very important chapter if you are not that deep into the Groovy language. You'll learn about the implicitly available Task and Project instances and the different ways of accessing methods and properties and of defining tasks and dependencies between them.

Working with files is an important part of any build system. Chapter 3 contains detailed information on accessing and modifying files, file collections and file trees. This is also where the benefit of using Groovy becomes really obvious. The ease of working with collections can lead to very concise build definitions though you have all the power of Groovy and the JVM at your hands. The different log levels are useful to know and can come in handy when you'd like to diagnose a build.

While understanding tasks is an important foundation for working with Gradle it's likely that you are after using it with programming languages. Nearly all of the remaining chapters cover working with different aspects on builds for JVM languages. Chapter 4 starts with a look at the Java plugin and its additional concepts. You'll see how you can compile and package Java applications and how to work with sourceSets.

Nearly no application is an island. The Java world provides masses of useful libraries that can help you build your application. Proper dependency management, as introduced in Chapter 5, is important for easy build setups and for making sure that you do not introduce incompatible combinations of libraries. Gradle supports Maven, Ivy and local file based repositories. Configurations are used to group dependencies, e.g. to define dependencies that are only necessary for tests. If you need to influence the version you are retrieving for a certain dependency you can configure resolution strategies, version ranges and exclusions for transitive dependencies.

Automated testing is a crucial part of any modern software development process. Gradle can work with JUnit and TestNG out of the box. Test execution times can be improved a lot by the incremental build support and the parallelization of tests. I guess this can lead to dramatically shorter build times, something I plan to try on an example project with a lot of tests in the near future. This chapter also introduces the different ways to run an application, create distributions and how to publish artifacts.

The next chapter will show you how you can structure your application in separate projects. Gradle has clever ways to find out which projects need to be rebuild before and after building a certain project.

Chapter 8 contains information on how to work with Scala and Groovy code. The necessary compiler versions can be defined in the build so there is no need to have additional installations. I've heard good things about the Scala integration so Gradle seems to be a viable alternative to sbt.

The check task can be used to gather metrics on your project using many of the available open source projects for code quality measurement. Chapter 9 shows you how to include tools like Checkstyle, PMD and FindBugs to analyze your project sources, either standalone or by sending data to Sonar.

If you need additional functionality that is not available you can start implementing your own tasks and plugins. Chapter 10 introduces the important classes for writing custom plugins and how to use them from Groovy and Java.

Gradle can be used on several Continuous Integration systems. As I've been working with Hudson/Jenkins exclusively during the last years it was interesting to also read about the commercial alternatives Team City and Bamboo in Chapter 11.

The final chapter contains a lot of in depth information on the Eclipse and IDEA plugins. Honestly, this contains more information on the Eclipse file format than I wanted to know but I guess that can be really useful for users. Unfortunately the excellent Netbeans plugin is not described in the book.

Summary

The book is an excellent introduction into working effectively with Gradle. It has helped me to get a far better understanding of the concepts. If you are thinking about or already started working with Gradle I highly recommend to get a copy. There are a lot of detailed example files that you can use immediately. Many of those are very close to real world use cases and can help you thinking about additional ways Gradle can be useful for organizing your builds.

Make your Filters Match: Faceting in Solr

2013-01-24T00:49:00.000-08:00

Facets are a great search feature that let users easily navigate to the documents they are looking for. Solr makes it really easy to use them though when naively querying for facet values you might see some unexpected behaviour. Read on to learn the basics of what is happening when you are passing in filter queries for faceting. Also, I'll show how you can leverage local params to choose a different query parser when selecting facet values.

Introduction

Facets are a way to display categories next to a users search results, often with a count of how many results are in this category. The user can then select one of those facet values to retrieve only those results that are assigned to this category. This way he doesn't have to know what category he is looking for when entering the search term as all the available categories are delivered with the search results. This approach is really popular on sites like Amazon and eBay and is a great way to guide the user.

Solr brought faceting to the Lucene world and arguably the feature was an important driving factor for its success (Lucene 3.4 introduced faceting as well). Facets can be build from terms in the index, custom queries and ranges though in this post we will only look at field facets.

As a very simple example consider this schema definition:

<fields>
    <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" /> 
    <field name="text" type="text_general" indexed="true" stored="true"/>
    <field name="author" type="string" indexed="true" stored="false"/>
</fields>

There are three fields, the id, a title that we'd probably like to search on and an author. The author is defined as a string field which means no analyzing at all. The faceting mechanism uses the term value and not a stored value so we want to make sure that the original value is preserved. I explicitly don't store the author information to make it clear that we are working with the indexed value.

Let's index some book data with curl (see this GitHub repo for the complete example including some unit tests that execute the same functionality using Java).

curl http://localhost:8082/solr/update -H "Content-Type: text/xml" --data-binary \
    '<add><doc>
            <field name="id">1</field>
            <field name="text">On the Shortness of Life</field>
            <field name="author">Seneca</field>
    </doc> 
    <doc>
            <field name="id">2</field>
            <field name="text">What I Talk About When I Talk About Running</field>
            <field name="author">Haruki Murakami</field>
    </doc> 
    <doc>
            <field name="id">3</field>
            <field name="text">The Dude and the Zen Master</field>
            <field name="author">Jeff "The Dude" Bridges</field>
    </doc>
    </add>'
curl http://localhost:8082/solr/update -H "Content-Type: text/xml" --data-binary '<commit />'

And verify that the documents are available:

curl http://localhost:8082/solr/query?q=*:*
{
  "responseHeader":{
    "status":0,
    "QTime":3,
    "params":{
      "q":"*:*"}},
  "response":{"numFound":3,"start":0,"docs":[
      {
        "id":"1",
        "text":"On the Shortness of Life"},
      {
        "id":"2",
        "text":"What I Talk About When I Talk About Running"},
      {
        "id":"3",
        "text":"The Dude and the Zen Master"}]
  }}

I'll omit parts of the response in the following examples. We can also have a look at the shiny new administration view of Solr 4 to see all terms that are indexed for the field author.

Each of the author names is indexed as one term.

Faceting

Let's move on to the faceting part. To let the user drill down on search results there are two steps involved. First you tell Solr that you would like to retrieve facets with the results. Facets are contained in an extra section of the response and consist of the indexed term as well as a count. As with most Solr parameters you can either send the necessary options with the query or preconfigure them in solrconfig.xml. This query has faceting on the author field enabled:

curl "http://localhost:8082/solr/query?q=*:*&facet=on&facet.field=author"
{
  "responseHeader":{...},
  "response":{"numFound":3,"start":0,"docs":[
      {
        "id":"1",
        "text":"On the Shortness of Life"},
      {
        "id":"2",
        "text":"What I Talk About When I Talk About Running"},
      {
        "id":"3",
        "text":"The Dude and the Zen Master"}]
  },
  "facet_counts":{
    "facet_queries":{},
    "facet_fields":{
      "author":[
        "Haruki Murakami",1,
        "Jeff \"The Dude\" Bridges",1,
        "Seneca",1]},
    "facet_dates":{},
    "facet_ranges":{}}}

And this is what a configuration in solrconfig looks like:

<requestHandler name="/select" class="solr.SearchHandler">
  <lst name="defaults">
    <str name="q">*:*</str>  
    <str name="echoParams">none</str>
    <int name="rows">10</int>
    <str name="df">text</str>
    <str name="facet">on</str>
    <str name="facet.field">author</str>
    <str name="facet.mincount">1</str>
  </lst>
</requestHandler>

This way we don't have to pass the parameters with the query anymore and can see which parts of the query change.

Common Filtering

When a user chooses a facet you issue the same query again, this time adding a filter query that restricts the search results to any that have the value for this certain fields set. In our case the user would only see books of one certain author. Let's start simple and pretend that a user can't handle the massive amount of 3 search results and is only interested in books on Seneca:

curl 'http://localhost:8082/solr/select?fq=author:Seneca'
{
  "responseHeader":{...},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "id":"1",
        "text":"On the Shortness of Life"}]
  },
  "facet_counts":{
    "facet_queries":{},
    "facet_fields":{
      "author":[
        "Seneca",1]},
    "facet_dates":{},
    "facet_ranges":{}}}

Works fine. We added a filter query that restricts the results to only those that are written by Seneca. Note that there is only one facet left because the search results don't contain any books by other authors. Let's see what happens when we try to filter the results to see only books by Haruki Murakami. We need to URL encode the blank, the rest of the query stays the same:

curl 'http://localhost:8082/solr/select?fq=author:Haruki%20Murakami'
{
  "responseHeader":{...},
  "response":{"numFound":0,"start":0,"docs":[]
  },
  "facet_counts":{
    "facet_queries":{},
    "facet_fields":{
      "author":[]},
    "facet_dates":{},
    "facet_ranges":{}}}

No results. Why is that? The default query parser for filter queries is the Lucene query parser. It tokenizes the query on whitespace, so even if we store the field unanalyzed it's not the query we are probably expecting to use. The query that is the result of the parsing process is not a term query as in our first example. It's a boolean query that consists of two term queries author:Haruki text:murakami. If you are familiar with the Lucene query syntax this won't be a surprise to you. If you prefix a term with a field name and a colon it will search on this field, otherwise it will search on the default field we declared in solrconfig.xml.

How can we fix it? Simple, just turn it into a phrase by surrounding the words with double quotes:

curl 'http://localhost:8082/solr/select?fq=author:"Haruki%20Murakami"'
{
  "responseHeader":{...},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "id":"2",
        "text":"What I Talk About When I Talk About Running"}]
  },
  "facet_counts":{
    "facet_queries":{},
    "facet_fields":{
      "author":[
        "Haruki Murakami",1]},
    "facet_dates":{},
    "facet_ranges":{}}}

Or, if you prefer, you can also escape the blank using the backslash, which yields the same result:

curl 'http://localhost:8082/solr/select?fq=author:Haruki\%20Murakami'

Fun fact: I am not that good at picking examples. If we are filtering on our last author we will be surprised (at least I scratched my head for a while):

curl 'http://localhost:8082/solr/select?fq=author:Jeff%20"The%20Dude"%20Bridges'
{
  "responseHeader":{...},
  "response":{"numFound":1,"start":0,"docs":[
      {
        "id":"3",
        "text":"The Dude and the Zen Master"}]
  },
  "facet_counts":{
    "facet_queries":{},
    "facet_fields":{
      "author":[
        "Jeff \"The Dude\" Bridges",1]},
    "facet_dates":{},
    "facet_ranges":{}}}

This actually seemed to work though we neither turned it into a phrase nor did we escape the blanks. If we look at how the Lucene query parser handles this query we see immediately why this returns a result. As with the last example this is turned into a boolean query, only the first query is executed against the author field. The other two tokens are searching on the default field and in this case "The Dude" matches the text field: author:Jeff text:"the dude" text:bridges. If you just want to match on the author field you can escape the blanks as we did in the example before:

curl 'http://localhost:8082/solr/select?fq=author:Jeff\%20\"The\%20Dude\"\%20Bridges'

I'll spare you with the response.

Using Local Params to set the Query Parser

At ApacheCon Europe in November Eric Hatcher did a really interesting presentation on query parsers in Solr where he introduced another, probably cleaner way to do this: You can use the local param syntax for choosing a different query parser. As we have learnt, the query parser defaults to the Lucene query parser. You can change the query parser for the query by setting the defType parameter, either via request parameters or in the solrconfig.xml but I am not aware of any way to set it for the filter queries. As we have unanalyzed terms the correct thing to do would be to use a TermQuery, which can be built using the TermQParserPlugin. To use this parser we can explicitly set it in the filter query:

curl 'http://localhost:8082/solr/select?fq={!term%20f=author%20v='Jeff%20"The%20Dude"%20Bridges'}'

Or, for better readability, without the URL encoding:

curl 'http://localhost:8082/solr/select?fq={!term f=author v='Jeff "The Dude" Bridges'}'

The local params are enclosed by curly braces. The value term is a shorthand for type='term', f is the fiels the TermQuery should be built for and v the value. Though this might look quirky at first this is a really powerful feature, especially since you can reference other request parameters from the local params. Consider this configuration of a request handler:

<requestHandler name="/selectfiltered" class="solr.SearchHandler">
  <lst name="defaults">
    <str name="q">*:*</str>  
    <str name="echoParams">explicit</str>
    <int name="rows">10</int>
    <str name="wt">json</str>
    <str name="indent">true</str>
    <str name="df">text</str>
    <str name="facet">on</str>
    <str name="facet.field">author</str>
    <str name="facet.mincount">1</str>
  </lst>
  <lst name="appends">
    <str name="fq">{!term f=author v=$author}</str>
  </lst>
</requestHandler>

The default configuration is the same as we were using above. Only the appends section is new, which adds additional parameters to the request. There are similar local params as we were using via curl, but the real filter query is replaced by the variable $author. This can now be passed in cleanly via an aptly named parameter:

curl 'http://localhost:8082/solr/selectfiltered?author=Jeff%20"The%20Dude"%20Bridges'

There are a lot of powerful features in Solr that are not that commonly used. To see this example in Java have a look at the Github repository of this blogpost.

JUnit Rule for ElasticSearch

2013-01-09T23:03:00.000-08:00

While I am using Solr a lot in my current engagement I recently started a pet project with ElasticSearch to learn more about it. Some of its functionality is rather different from Solr so there is quite some experimentation involved. I like to start small and implement tests if I like to find out how things work (see this post on how to write tests for Solr).

ElasticSearch internally uses TestNG and the test classes are not available in the distributed jar files. Fortunately it is really easy to start an ElasticSearch instance from within a test so it's no problem to do something similar in JUnit. Felix Müller posted some useful code snippets on how to do this, obviously targeted at a Maven build. The ElasticSearch instance is started in a setUp method and stopped in a tearDown method:

private EmbeddedElasticsearchServer embeddedElasticsearchServer;

@Before
public void startEmbeddedElasticsearchServer() {
    embeddedElasticsearchServer = new EmbeddedElasticsearchServer();
}

@After
public void shutdownEmbeddedElasticsearchServer() {
    embeddedElasticsearchServer.shutdown();
}

As it is rather cumbersome to add these methods to all tests I transformed the code to a JUnit rule. Rules can execute code before and after a test is run and influence its execution. There are some base classes available that make it really easy to get started with custom rules.

Our ElasticSearch example can be easily modeled using the base class ExternalResource (see the full example code on GitHub):

public class ElasticsearchTestNode extends ExternalResource {

    private Node node;
    private Path dataDirectory;
    
    @Override
    protected void before() throws Throwable {
        try {
            dataDirectory = Files.createTempDirectory("es-test", new FileAttribute []{});
        } catch (IOException ex) {
            throw new IllegalStateException(ex);
        }

        ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.settingsBuilder()
                .put("http.enabled", "false")
                .put("path.data", dataDirectory.toString());

        node = NodeBuilder.nodeBuilder()
                .local(true)
                .settings(elasticsearchSettings.build())
                .node();
    }

    @Override
    protected void after() {
        node.close();
        try {
            FileUtils.deleteDirectory(dataDirectory.toFile());
        } catch (IOException ex) {
            throw new IllegalStateException(ex);
        }
    }
    
    public Client getClient() {
        return node.client();
    }
}

The before method is executed before the test is run so we can use it to start ElasticSearch. All data is written to a temporary folder. The after method is used to stop ElasticSearch and delete the folder.

In your test you can now just use the rule, either with the @Rule annotation to have it triggered on each test method, or using @ClassRule to execute it only once per class:

public class CoreTest {

    @Rule
    public ElasticsearchTestNode testNode = new ElasticsearchTestNode();
    
    @Test
    public void indexAndGet() throws IOException {
        testNode.getClient().prepareIndex("myindex", "document", "1")
                .setSource(jsonBuilder().startObject().field("test", "123").endObject())
                .execute()
                .actionGet();
        
        GetResponse response = testNode.getClient().prepareGet("myindex", "document", "1").execute().actionGet();
        assertThat(response.getSource().get("test")).isEqualTo("123");
    }
}

As it is really easy to implement custom rules I think this is a feature I will be using more often in the future.

12 Conferences of 2012

2013-01-03T01:52:00.000-08:00

I went to a lot, probably too many conferences in 2012. As the year is over now I'd like to summarize some of the impressions, maybe there's a conference you didn't know about and you'd like to attend this year.

FOSDEM

FOSDEM is the Free and Open Source Software Developer European Meetup, a yearly event that takes place in Brussels, Belgium. There are multiple tracks and developer rooms on a multitude of topics ranging from databases to programming languages and open source tools. The rooms are spread across some buildings at the University so there might be some walking involved when switching tracks. What is rather special is that there's no registration involved, you just go there and that's it. The amount of people can be overwhelming, especially in the main entrance area. Unfortunately I was rather disappointed with the talks I chose. The event is a very good fit if you are working on an Open Source project and, as the name of the conference suggests, want to meet other developers of the project.

Berlin Expert Days

BED-Con is a rather young conference organized by the Java User Group Berlin-Brandenburg. I haven't been there in the first year but in 2012 it still had a small and informal feeling. The conference takes place in three rooms of the Freie Universität Berlin, the content selection was an excellent mixture of technical and process/meta talks, most of them in German. If you can afford the trip to Berlin I'd definitively recommend going there.

JAX

The largest and best known German Java conference. There are two editions, JAX in April in Mainz (the one I attended) and W-JAX in November in Munich. There's one huge hall and several smaller rooms and a wide variety of topics you can choose from. I never planned to go there as the admission fee is rather high (thanks to Software & Support for sponsoring my ticket) but I have to admit that it really can be worth the money. There were excellent talks by Charles Nutter, Tim Berglund and many more. The infrastructure (food, coffee, schedules) is very good, if you are on a business budget you can gain a lot by visiting.

Berlin Buzzwords

A niche conference on Search, Scale and Big Data. A lot of people are coming from overseas just to visit. If you are interested in these technologies definitively go there. For more information see this post.

Barcamp Karlsruhe

My first real Barcamp. Really fun event but of course there are always some sessions that are not as interesting as anticipated. Topics ranged from Computer and work stuff to more soft content. I always thought this is a nerd only event but as there is so much to choose from Barcamps might even be interesting for people who are not that much into computers. Very well organized, no admission fee, interesting sessions.

Socrates

The International Software Craftmanship and Testing Conference. Awesome setting and the first open space conference I attended. It takes place in a seminar center in the middle of nowhere which makes it a very intense experience. Besides the sessions there are a lot of informal discussions going on around the day with the very enthusiastic attendants. The 2012 event started Thursday evening with a world cafe, Friday and Saturday open space and an optional code retreat on Sunday. I'd say there were three kinds of sessions: informal discussion rounds, practical hands on sessions and talks. It seems that most people liked the practical sessions best, so if I could choose again I'd go to more of those. Thanks to the sponsors all we had to pay was the accommodation and one meal which additionally makes it an incredible cheap event. Be quick with registration as space is limited.

FrOSCon

The Free and Open Source Software Conference is a great community weekend event with different tracks on admin and development topics. I like it a lot because of the variation of talks and the very informal setting. It's a mixture of holiday and learning and for me a chance to get information on topics that are not presented at the other developer conferences I attend. Talks are partly english, partly german. You can either stay in St. Augustin or in Bonn, it's only a short tram ride.

JAX On Tour

JAX On Tour is another event I attended because of the generous sponsorship of Software & Support. It's not a conference but a training event with longer talks that are grouped together. It's a small event and there's always time for questions. I learned a lot, mainly about documenting architectures. This is a really good alternative to a normal conference to grasp a topic in depth.

OpenCms-Days

If you are into OpenCms you probably already heard about OpenCms-Days, and if not this is probably not for you. Two days of OpenCms only, used by Alkacon to present new features and by the community to present extensions and projects. I am always impressed that there are people who fly around the world just to attend, but of course this is the only conference of its kind worldwide. There is always something new to learn and it's fun to meet the community.

DevFest Karlsruhe

A one day event on Google technologies. There have been several events in different cities worldwide, this one organized by the Google Developer Group Karlsruhe. The organizers were really unlucky as multiple speakers canceled on short notice, nevertheless there were some really good talks. Kudos to the organizers who managed to get this event started in a really short time frame.

ApacheCon Europe

I originally went to ApacheCon because it took place in Sinsheim, which is close to Karlsruhe. Fortunately in this year the conference also hosted the LuceneCon Europe, so there were lots of interesting talks for me. Additionally I had been voted as a committer to the ODFToolkit just before it and I was able to meet some other people that are involved in the project. The location was really special (soccer stadium) but I think a lot of non-locals had to suffer a bit because of the lack of hotels and taxis. This community event can be really interesting even if you are only a user of a project.

Devoxx

Devoxx is the largest Java conference in Europe, organized by the Java User Group Belgium. They attract a lot of high class speakers so this is the place to keep you informed on the Java universe. Located in a large multiplex cinema in a suburb of Antwerp, very comfy chairs and huge screens. The week starts with two university days that contain longer talks and are usually less crowded. This year I went there on Wednesday, due to a train strike in Belgium I arrived quite late. I have to admit that it probably wasn't worth the hassle for only 1.5 days. If you are going there I recommend to stay the whole week.

2013

So this was a lot last year. I don't plan to visit that many conferences again. I am sure that I will be going to Berlin Buzzwords, Socrates, FrOSCon. There probably will be more, but it won't get 13 this year :).

Finally, I couldn't have afforded to pay for all those conferences myself, so thanks to synyx, who paid for FOSDEM and BEDCon when I was still employed there and provided me with a free ticket to OpenCms-Days. Thanks to Software & Support for letting me attend JAX and JAX On Tour for free, those guys are fantastic supporters of the Java User Group Karlsruhe. Also, thanks to the Devoxx team for letting me attend as an ambassador of our JUG.

Gradle is too Clever for my Plans

2012-12-19T22:16:00.000-08:00

While writing this post about the Lucene Codec API I noticed something strange when running the tests with Gradle. When experimenting with a library feature most of the times I write unit tests that validate my expectations. This is a habit I learned from Lucene in Action and can also be useful in real world scenarios, e.g. to make sure that nothing breaks when you update a library.

OK, what happened? This time I did not only want to have the test result but also ran the test for a side effect, I wanted a Lucene index to be written to the /tmp directory to manually have a look at it. This worked fine for the first time, but not afterwards, e.g. after my machine was rebooted and the directory cleared.

It turns out that the Gradle developers know that a test shouldn't be used to execute stuff. So once the test is run successfully it is just not run again until its input changes! Though this bit me this time this is a really nice feature to speed up your builds. And if you really need to execute the tests, you can always run gradle cleanTest test.

Looking at a Plaintext Lucene Index

2012-12-07T03:45:00.001-08:00

The Lucene file format is one of the reasons why Lucene is as fast as it is. An index consist of several binary files that you can't really inspect if you don't use tools like the fantastic Luke.

Starting with Lucene 4 the format for these files can be configured using the Codec API. Several implementations are provided with the release, among those the SimpleTextCodec that can be used to write the files in plaintext for learning and debugging purposes.

To configure the Codec you just set it on the IndexWriterConfig:

StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_40);
IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_40, analyzer);
// recreate the index on each execution
config.setOpenMode(IndexWriterConfig.OpenMode.CREATE);
config.setCodec(new SimpleTextCodec());

The rest of the indexing process stays exactly the same as it used to be:

Directory luceneDir = FSDirectory.open(plaintextDir);
try (IndexWriter writer = new IndexWriter(luceneDir, config)) {
    writer.addDocument(Arrays.asList(
            new TextField("title", "The title of my first document", Store.YES),
            new TextField("content", "The content of the first document", Store.NO)));

    writer.addDocument(Arrays.asList(
            new TextField("title", "The title of the second document", Store.YES),
            new TextField("content", "And this is the content", Store.NO)));
}

After running this code the index directory contains several files. Those are not the same type of files that are created using the default codec.

ls /tmp/lucene-plaintext/
_1_0.len  _1_1.len  _1.fld  _1.inf  _1.pst  _1.si  segments_2  segments.gen

The segments_x file is the starting point (x depends on the amount of times you have written to the index before and starts with 1). This still is a binary file but contains the information which codec is used to write to the index. It contains the name of each Codec that is used for writing a certain segment.

The rest of the index files are all plaintext. They do not contain the same information as their binary cousins. For example the .pst file represents the complete posting list, the structure you normally mean when talking about an inverted index:

field content
  term content
    doc 0
      freq 1
      pos 1
    doc 1
      freq 1
      pos 4
  term document
    doc 0
      freq 1
      pos 5
  term first
    doc 0
      freq 1
      pos 4
field title
  term document
    doc 0
      freq 1
      pos 5
    doc 1
      freq 1
      pos 5
  term first
    doc 0
      freq 1
      pos 4
  term my
    doc 0
      freq 1
      pos 3
  term second
    doc 1
      freq 1
      pos 4
  term title
    doc 0
      freq 1
      pos 1
    doc 1
      freq 1
      pos 1
END

The content that is marked as stored resides in the .fld file:

doc 0
  numfields 1
  field 0
    name title
    type string
    value The title of my first document
doc 1
  numfields 1
  field 0
    name title
    type string
    value The title of the second document
END

If you'd like to have a look at the rest of the files checkout the code at Github.

The SimpleTextCodec only is an interesting byproduct. The Codec API can be used for a lot useful things. For example the feature to read indices of older Lucene versions is implemented using seperate codecs. Also, you can mix several Codecs in an index so reindexing on version updates should not be necessary immediately. I am sure more useful codecs will pop up in the future.

Getting rid of synchronized: Using Akka from Java

2012-08-22T23:02:00.000-07:00

I've been giving an internal talk on Akka, the Actor framework for the JVM, at my former company synyx. For the talk I implemented a small example application, kind of a web crawler, using Akka. I published the source code on Github and will explain some of the concepts in this post.

Motivation

To see why you might need something like Akka, think you want to implement a simple web crawler for offline search. You are downloading pages from a certain location, parse and index the content and follow any links that you haven't parsed and indexed yet. I am using HtmlParser for downloading and parsing pages and Lucene for indexing them. The logic is contained in two service objects, PageRetriever and Indexer, that can be used from our main application.

A simple sequential execution might then look something like this:

public void downloadAndIndex(String path, IndexWriter writer) {
    VisitedPageStore pageStore = new VisitedPageStore();
    pageStore.add(path);
        
    Indexer indexer = new IndexerImpl(writer);
    PageRetriever retriever = new HtmlParserPageRetriever(path);
        
    String page;
    while ((page = pageStore.getNext()) != null) {
        PageContent pageContent = retriever.fetchPageContent(page);
        pageStore.addAll(pageContent.getLinksToFollow());
        indexer.index(pageContent);
        pageStore.finished(page);
    }
        
    indexer.commit();
}

We are starting with one page, extract the content and the links, index the content and store all links that are to be visited in the VisitedPageStore. This class contains the logic to determine which links are visited already. We are looping as long as there are more links to follow, once we are done we commit the Lucene IndexWriter.

This implementation works fine, when running on my outdated laptop it will finish in around 3 seconds for an example page. (Note that the times I am giving are by no means meant as a benchmark but are just there to give you some idea on the numbers).

So we are done? No, of course we can do better by optimizing the resources we have available. Let's try to improve this solution by splitting it into several tasks that can be executed in parallel.

Shared State Concurrency

The normal way in Java would be to implement several Threads that do parts of the work and access the state via guarded blocks, e.g. by synchronizing methods. So in our case there might be several Threads that access our global state that is stored in the VisitedPageStore.

This model is what Venkat Subramaniam calls Synchronize and Suffer in his great book Programming Concurrency on the JVM. Working with Threads and building correct solutions might not seem that hard at first but is inherintly difficult. I like those two tweets that illustrate the problem:

Adding the "synchronized" keyword to Java was a mistake. Makes people believe they can write multi-threaded code.
— Erik Dörnenburg (@erikdoe) Oktober 25, 2011

95% of syncronized code is broken. The other 5% is written by Brian Goetz. - Venkat Subramaniam at #s2gx
— Ronny Løvtangen (@rlovtangen) Oktober 26, 2011

Brian Goetz of course being the author of the de-facto standard book on the new Java concurrency features, Java Concurrency in Practice.

Akka

So what is Akka? It's an Actor framework for the JVM that is implemented in Scala but that is something that you rarely notice when working from Java. It offers a nice Java API that provides most of the functionality in a convenient way.

Actors are a concept that was introduced in the seventies but became widely known as one of the core features of Erlang, a language to build fault tolerant, self healing systems. Actors employ the concept of Message Passing Concurrency. That means that Actors only communicate by means of messages that are passed into an Actors mailbox. Actors can contain state that they shield from the rest of the system. The only way to change the state is by passing in messages. Each Actor is executed in a different Thread but they provide a higher level of abstraction than working with Threads directly.

When implementing Actors you put the behaviour in a method receive() that can act on incoming messages. You can then reply asynchronously to the sender or send messages to any other Actor.

For our problem at hand an Actor setup might look something like this:

There is one Master Actor that also contains the global state. It sends a message to fetch a certain page to a PageParsingActor that asynchonously responds to the Master with the PageContent. The Master can then send the PageContent to an IndexingActor which responds with another message. With this setup we have done a first step to scale our solution. There are now three Actors that can be run on different cores of your machine.

Actors are instantiated from other Actors. On the top there's the ActorSystem that is provided by the framework. The MasterActor is instaciated from the ActorSystem:

ActorSystem actorSystem = ActorSystem.create();
final CountDownLatch countDownLatch = new CountDownLatch(1);
ActorRef master = actorSystem.actorOf(new Props(new UntypedActorFactory() {

    @Override
    public Actor create() {
        return new SimpleActorMaster(new HtmlParserPageRetriever(path), writer, countDownLatch);
    }
}));

master.tell(path);
try {
    countDownLatch.await();
    actorSystem.shutdown();
} catch (InterruptedException ex) {
    throw new IllegalStateException(ex);
}

Ignore the CountdownLatch as it is only included to make it possible to terminate the application. Note that we are not referencing an instance of our class but an ActorRef, a reference to an actor. You will see later why this is important.

The MasterActor contains references to the other Actors and creates them from its context. This makes the two Actors children of the Master:

public SimpleActorMaster(final PageRetriever pageRetriever, final IndexWriter indexWriter,
    final CountDownLatch latch) {

    super(latch);
    this.indexer = getContext().actorOf(new Props(new UntypedActorFactory() {

        @Override
        public Actor create() {

            return new IndexingActor(new IndexerImpl(indexWriter));
        }
    }));

    this.parser = getContext().actorOf(new Props(new UntypedActorFactory() {

        @Override
        public Actor create() {

           return new PageParsingActor(pageRetriever);
        }
    }));
}

The PageParsingActor acts on messages to fetch pages and sends a message with the result to the sender:

public void onReceive(Object o) throws Exception {
    if (o instanceof String) {
        PageContent content = pageRetriever.fetchPageContent((String) o);
        getSender().tell(content, getSelf());
    } else {
        // fail on any message we don't expect
        unhandled(o);
    }
}

The IndexingActor contains some state with the Indexer. It acts on messages to index pages and to commit the indexing process.

public void onReceive(Object o) throws Exception {
    if (o instanceof PageContent) {
        PageContent content = (PageContent) o;
        indexer.index(content);
        getSender().tell(new IndexedMessage(content.getPath()), getSelf());
    } else if (COMMIT_MESSAGE == o) {
        indexer.commit();
        getSender().tell(COMMITTED_MESSAGE, getSelf());
    } else {
        unhandled(o);
    }
}

The MasterActor finally orchestrates the other Actors in its receive() method. It starts with one page and sends it to the PageParsingActor. It keeps the valuable state of the application in the VisitedPageStore. When no more pages are to be fetched and indexed it sends a commit message and terminates the application.

public void onReceive(Object message) throws Exception {

    if (message instanceof String) {
        // start
        String start = (String) message;
        visitedPageStore.add(start);
        getParser().tell(visitedPageStore.getNext(), getSelf());
    } else if (message instanceof PageContent) {
        PageContent content = (PageContent) message;
        getIndexer().tell(content, getSelf());
        visitedPageStore.addAll(content.getLinksToFollow());

        if (visitedPageStore.isFinished()) {
            getIndexer().tell(IndexingActor.COMMIT_MESSAGE, getSelf());
        } else {
            for (String page : visitedPageStore.getNextBatch()) {
                getParser().tell(page, getSelf());
            }
        }
    } else if (message instanceof IndexedMessage) {
        IndexedMessage indexedMessage = (IndexedMessage) message;
        visitedPageStore.finished(indexedMessage.path);

        if (visitedPageStore.isFinished()) {
            getIndexer().tell(IndexingActor.COMMIT_MESSAGE, getSelf());
        }
    } else if (message == IndexingActor.COMMITTED_MESSAGE) {
        logger.info("Shutting down, finished");
        getContext().system().shutdown();
        countDownLatch.countDown();
    }
}

What happens if we run this example? Unfortunately it now takes around 3.5 seconds on my dual core machine. Though we are now able to run on both cores we have actually decreased the speed of the application. This is probably an important lesson. When building scalable applications it might happen that you are introducing some overhead that decreases the performance when running in the small. Scalability is not about increasing performance but about the ability to distribute the load.

So it was an failure to switch to Akka? Not at all. It turns out that most of the time the application is fetching and parsing pages. This includes waiting for the network. Indexing in Lucene is blazing fast and the Master mostly only dispatches messages. So what can we do about it? We already have split our application into smaller chunks. Fortunately the PageParsingActor doesn't contain any state at all. That means we can easily parallelize its tasks.

This is where the talking to references is important. For an Actor it's transparent if there is one or a million Actors behind a reference. There is one mailbox for an Actor reference that can dispatch the messages to any amount of Actors.

We only need to change the instanciation of the Actor, the rest of the application remains the same:

parser = getContext().actorOf(new Props(new UntypedActorFactory() {

        @Override
        public Actor create() {

            return new PageParsingActor(pageRetriever);
        }
}).withRouter(new RoundRobinRouter(10)));

By using a router the Akka framework automatically takes care that there are 10 Actors available. The messages are distributed to any available Actor. This takes the runtime down to 2 seconds.

A word on Blocking

Note that the way I am doing network requests here is not recommended in Akka. HTMLParser is doing blocking networking which should be carefully reconsidered when designing a reactive system. In fact, as this application is highly network bound, we might even gain more benefit by just using an asynchronous networking library. But hey, then I wouldn't be able to tell you how nice it is to use Akka. In a future post I will highlight some more Akka features that can help to make our application more robust and fault tolerant.

Slides and demo code for my talk at JUG KA available

2012-07-05T09:50:00.001-07:00

I just uploaded the (german) slides as well as the example code for yesterdays talk on Lucene and Solr at our local Java User Group.

The demo application contains several subprojects for indexing and searching with Lucene and Solr as well as a simple Dropwizard application that demonstrates some search features. See the README files in the source tree to find out how to run the application.

Dropwizard Encoding Woes

2012-06-29T00:57:00.000-07:00

I have been working on an example application for Lucene and Solr for my upcoming talk at the Java User Group Karlsruhe. As a web framework I wanted to try Dropwizard, a lightweight application framework that can expose resources via JAX-RS, provides out of the box monitoring support and can render resource representations using Freemarker. It's really easy to get started, there's a good tutorial and the manual.

An example resource might look like this:

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Path("/example")
@Produces(MediaType.TEXT_HTML)
public class ExampleResource {

    @GET
    public ExampleView illustrate() {
        return new ExampleView("Mot\u00f6rhead");
    }

}

The Resource produces HTML using Freemarker, which is possible if you add the view bundle in the service. There is one method that is called when the resource is addressed using GET. Inside the method we create a view object accepting a message that in this case contains the umlaut 'ö'. The view class that is returned by the method looks like this:

import com.yammer.dropwizard.views.View;

public class ExampleView extends View {

    private final String message;

    public ExampleView(String message) {
        super("example.fmt");
        this.message = message;
    }

    public String getMessage() {
        return message;
    }
}

It accepts a message as constructor parameter. The template name is passed to the parent class. This view class is now available in a freemarker template, an easy variant looks like this:

<html>
    <body>
        <h1>${message} rocks!</h1>
    </body>
</html>

If I run this on my machine and access it with Firefox it doesn't work as expected. The umlaut character is broken, something Lemmy surely doesn't approve:

Accessing the resource using curl works flawlessly:

curl http://localhost:8080/example
<html>
    <body>
        <h1>Motörhead rocks!</h1>
    </body>
</html>

Why is that? It's Servlet Programming 101: You need to set the character encoding of the response. My Firefox defaults to ISO-8859-1, curl seems to use UTF-8 by default. How can we fix it? Tell the client which encoding we are using, which can be done using the Produces annotation:

@Produces("text/html; charset=utf-8")

So what does it have to do with Dropwizard? Nothing really, it's a JAX-RS thing. All components in Dropwizard (Jetty and Freemarker notably) are using UTF-8 by default.

Running and Testing Solr with Gradle

2012-06-20T05:28:00.000-07:00

A while ago I blogged on testing Solr with Maven on the synyx blog. In this post I will show you how to setup a similar project with Gradle that can start the Solr webapp and execute tests against your configuration.

Running Solr

Solr is running as a webapp in any JEE servlet container like Tomcat or Jetty. The index and search configuration resides in a directory commonly referred to as Solr home that can be outside of the webapp directory. This is also the place where the Lucene index files are created. The location for Solr home can be set using an environment variable.

The Solr war file is available in Maven Central. This post describes how to run a war file that is deployed in a Maven repository using Gradle. Let's see how the Gradle build file looks like for running Solr:

import org.gradle.api.plugins.jetty.JettyRunWar

apply plugin: 'java'
apply plugin: 'jetty'

repositories {
    mavenCentral()
}

// custom configuration for running the webapp
configurations {
    solrWebApp
}

dependencies {
    solrWebApp "org.apache.solr:solr:3.6.0@war"
}

// custom task that configures the jetty plugin
task runSolr(type: JettyRunWar) {
    webApp = configurations.solrWebApp.singleFile

    // jetty configuration
    httpPort = 8082
    contextPath = 'solr'
}

// executed before jetty starts
runSolr.doFirst {
    System.setProperty("solr.solr.home", "./solrhome")
}

We are creating a custom configuration that contains the Solr war file. In the task runSolr we configure the Jetty plugin. To add the Solr home environment variable we can use the way described by Sebastian Himberger. We add a code block that is executed before Jetty starts and sets the environment variable using standard Java mechanisms. You can now start Solr using gradle runSolr. You will see some errors regarding multiple versions of slf4j that are very like caused by this bug.

Testing the Solr configuration

Solr provides some classes that start an embedded instance using your configuration. You can use these classes in any setup as they do not depend on the gradle jetty plugin. Starting with Solr 3.2 the test framework is not included in solr-core anymore. This is what the relevant part of the dependency section looks like now:

testCompile "junit:junit:4.10"
testCompile "org.apache.solr:solr-test-framework:3.6.0"

Now you can place a test in src/test/java that either uses the convenience methods provided by SolrTestCaseJ4 or you can instantiate an EmbeddedSolrServer and execute any SolrJ actions. Both of these ways will use your custom config. This way you can easily validate that configuration changes don't break existing functionality. An example of using the convenience methods:

import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.SolrServerException;
import org.junit.BeforeClass;
import org.junit.Test;
import java.io.IOException;

public class BasicConfigurationTest extends SolrTestCaseJ4 {

    @BeforeClass
    public static void initCore() throws Exception {
        SolrTestCaseJ4.initCore("solrhome/conf/solrconfig.xml", "solrhome/conf/schema.xml", "solrhome/");
    }

    @Test
    public void noResultInEmptyIndex() throws SolrServerException {
        assertQ("test query on empty index",
                req("text that is not found")
                , "//result[@numFound='0']"
        );
    }

    @Test
    public void pathIsMandatory() throws SolrServerException, IOException {
        assertFailedU(adoc("title", "the title"));
    }

    @Test
    public void simpleDocumentIsIndexedAndFound() throws SolrServerException, IOException {
        assertU(adoc("path", "/tmp/foo", "content", "Some important content."));
        assertU(commit());

        assertQ("added document found",
                req("important")
                , "//result[@numFound='1']"
        );
    }

}

We extend the class SolrTestCaseJ4 that is responsible for creating the core and instanciating the runtime using the paths we provide with the method initCore(). Using the available assert methods you can execute queries and validate the result using XPath expressions.

An example that instanciates a SolrServer might look like this:

import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer;
import org.apache.solr.client.solrj.response.FacetField;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.SolrParams;
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

import java.io.IOException;

public class ServerBasedTalkTest extends SolrTestCaseJ4 {

    private EmbeddedSolrServer server;

    @BeforeClass
    public static void initCore() throws Exception {
        SolrTestCaseJ4.initCore("solr/conf/solrconfig.xml", "solr/conf/schema.xml");
    }

    @Before
    public void initServer() {
        server = new EmbeddedSolrServer(h.getCoreContainer(), h.getCore().getName());
    }

    @Test
    public void queryOnEmptyIndexNoResults() throws SolrServerException {
        QueryResponse response = server.query(new SolrQuery("text that is not found"));
        assertTrue(response.getResults().isEmpty());
    }

    @Test
    public void singleDocumentIsFound() throws IOException, SolrServerException {
        SolrInputDocument document = new SolrInputDocument();
        document.addField("path", "/tmp/foo");
        document.addField("content", "Mein Hut der hat 4 Ecken");

        server.add(document);
        server.commit();

        SolrParams params = new SolrQuery("ecke");
        QueryResponse response = server.query(params);
        assertEquals(1L, response.getResults().getNumFound());
        assertEquals("/tmp/foo", response.getResults().get(0).get("path"));
    }

    @After
    public void clearIndex() {
        super.clearIndex();
    }
}

The tests can now be executed using gradle test.

Testing your Solr configuration is important as changes in one place might easily lead to side effects with another search functionality. I recommend to add tests even for basic functionality and evolve the tests with your project.

Reading term values for fields from a Lucene Index

2012-06-16T00:13:00.000-07:00

Sometimes when using Lucene you might want to retrieve all term values for a given field. Think of categories that you want to display as search links or in a filtering dropdown box. Indexing might look something like this:

IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_36, new StandardAnalyzer(Version.LUCENE_36));
IndexWriter writer = new IndexWriter(directory, config);

Document doc = new Document();

doc.add(new Field("Category", "Category1", Field.Store.NO, Field.Index.NOT_ANALYZED));
doc.add(new Field("Category", "Category2", Field.Store.NO, Field.Index.NOT_ANALYZED));
doc.add(new Field("Author", "Florian Hopf", Field.Store.NO, Field.Index.NOT_ANALYZED));
writer.addDocument(doc);

doc.add(new Field("Category", "Category3", Field.Store.NO, Field.Index.NOT_ANALYZED));
doc.add(new Field("Category", "Category2", Field.Store.NO, Field.Index.NOT_ANALYZED));
doc.add(new Field("Author", "Theo Tester", Field.Store.NO, Field.Index.NOT_ANALYZED));
writer.addDocument(doc);

writer.close();

We are adding two documents, one that is assigned Category1 and Category2 and one that is assigned Category2 and Category3. Note that we are adding both fields unanalyzed so the Strings are added to the index as they are. Lucenes index looks something like this afterwards:

Field	Term	Documents
Author	Florian Hopf	1
	Theo Tester	2
Category	Category1	1
	Category2	1, 2
	Category3	2

The fields are sorted alphabetically by fieldname first and then by term value. You can access the values using the IndexReaders terms() method that returns a TermEnum. You can instruct the IndexReader to start with a certain term so you can directly jump to the category without having to iterate all values. But before we do this let's look at how we are used to access Enumeration values in Java:

Enumeration en = ...;
while(en.hasMoreElements()) {
    Object obj = en.nextElement();
    ...
}

In a while-loop we are checking if there is another element and retrieve it inside the loop. As this pattern is very common when iterating the terms with Lucene you might end with something like this (Note that all the examples here are missing the stop condition. If there are more fields the terms of those fields will also be iterated):

TermEnum terms = reader.terms(new Term("Category"));
// this code is broken, don't use
while(terms.next()) {
    Term term = terms.term();
    System.out.println(term.text());
}

The next() method returns a boolean if there are more elements and points to the next element. The term() method then can be used to retrieve the Term. But this doesn't work as expected. The code only finds Category2 and Category3 but skips Category1. Why is that? The Lucene TermEnum works differently than we are used from Java Enumerations. When the TermEnum is returned it already points to the first element so with next() we skip this first element.

This snippet instead works correctly using a for loop:

TermEnum terms = reader.terms(new Term("Category"));
for(Term term = terms.term(); term != null; terms.next(), term = terms.term()) {
    System.out.println(term.text());
}

Or you can use a do while loop with a check for the first element:

TermEnum terms = reader.terms(new Term("Category"));
if (terms.term() != null) {
    do {
        Term term = terms.term();
        System.out.println(term.text());
    } while(terms.next());
}

You can't really blame Lucene for this as the methods are aptly named. It's our habits that lead to minor errors like this.

Berlin Buzzwords 2012

2012-06-06T09:20:00.000-07:00

Berlin Buzzwords is an annual conference on search, store and scale technology. I've heard good things about it before and finally got convinced to go there this year. The conference itself lasts for two days but there are additional events before and afterwards so if you like you can spend a whole week.

The Barcamp

As I had to travel on sunday anyway I took an earlier train to attend the barcamp in the early evening. It started with a short introduction of the concepts and the scheduling. Participants could suggest topics that they either would be willing to introduce by themselfes or just anything they are interested in. There were three roomes prepared, a larger and two smaller ones.

Among others I attended sessions on HBase, designing hybrid applications, Apache Tika and Apache Jackrabbit Oak.

HBase is a distributed database build on top of the Hadoop filesystem. It seems to be used more often than I would have expected. Interesting to hear about the problems and solutions of other people.

The next session on hybrid relational and NoSQL applications stayed rather high level. I liked the remark by one guy that Solr, the underdog of NoSQL, often is the first application where people are ok with dismissing some guarantees regarding their data. Adding NoSQL should be exactly like this.

I only started just recently to use Tika directly so it was really interesting to see where the project is heading in the future. I was surprised to hear that there now also is a TikaServer that can do similar things like those I described for Solr. That's something I want to try in action.

Jackrabbit Oak is a next generation content repository that is mostly driven by the Day team of Adobe. Some of the ideas sound really interesting but I got the feeling that it still can take some time until this really can be used. Jukka Zitting also gave a lightning talk on this topic at the conference, the slides are available here.

The atmosphere in the sessions was really relaxed so even though I expected to only listen I took the chance to participate and ask some questions. This probably is the part that makes a barcamp as effective as it is. As you are constantly participating you keep really contentrated on the topic.

Day 1

The first day started with a great keynote by Leslie Hawthorn on building and maintaining communities. She compared a lot of the aspects of community work with gardening and introduced OpenMRS, a successful project building a medical record platform. Though I currently am not actively involved in an open source project I could relate to a lot of the situations she described. All in all an inspiring start of the main conference.

Next I attended a talk on building hybrid applications with MongoDb. Nothing new for me but I am glad that a lot of people now recommend to split monolithic applications into smaller services. This also is a way to experiment with different languages and techniques without having to migrate large parts of an application.

A JCR view of the world provided some examples on how to model different structures using a content tree. Though really introductionary it was interesting to see what kind of applications can be build using a content repository. I also liked the attitude of the speaker: The presentation was delivered using Apache Sling which uses JCR under the hood.

Probably the highlight of the first day was the talk by Grant Ingersoll on Large Scale Search, Discovery and Analytics. He introduced all the parts that make up larger search systems and showed the open source tools he uses. To increase the relevance of the search results you have to integrate solutions to adapt to the behaviour of the users. That's probably one of the big takeaways for me of the whole conference: Always collect data on your users searches to have it available when you want to tune the relevance, either manually or through some learning techniques. The slides of the talk are worth looking at.

The rest of the day I attended several talks on the internals of Lucene. Hardcore stuff, I would be lying if I said I would have understood everything but it was interesting nevertheless. I am glad that some really smart people are taking care that Lucene stays as fast and feature rich as it is.

The day ended with interesting discussions and some beer at the Buzz Party and BBQ.

Day 2

The first talk of the second day on Smart Autocompl... by Anne Veling was fantastic. Anne demonstrated a rather simple technique for doing semantic analysis of search queries for specialized autocompletion for the largest travel information system in the Netherlands. The query gets tokenized and then each field of the index (e.g. street or city) is queried for each of the tokens. This way you can already guess which might be good field matches.

Another talk introduced a scalable tool for preprocessing of documents, Hydra. It stores the documents as well as mapping data in a MongoDb instance and you can parallelize the processing steps. The concept sounds really interesting, I hope I can find time to have a closer look.

In the afternoon I attended several talks on Elasticsearch, the scalable search server. Interestingly a lot of people seem to use it more as a storage engine than for searching.

One of the tracks was cancelled, Ted Dunning introduced new stuff in Mahout instead. He's a really funny speaker and though I am not deep into machine learning I was glad to hear that you are allowed to use and even contribute to Mahout even if you don't have a PhD.

In the last track of the day Alex Pinkin showed 10 problems and solutions that you might encounter when building a large app using Solr. Quite some useful advice.

The location

The event took place at Urania, a smaller conference center and theatre. Mostly it was suited well but some of the talks were so full that you either had to sit on the floor or weren't even able to enter the room. I understand that it is difficult to predict how many people attend a certain event but some talks probably should have been scheduled in different rooms.

The food was really good and though it first looked like the distribution was a bottleneck this worked pretty well.

The format

This year Berlin Buzzwords had a rather unusual format. Most of the talks were only 20 minutes long with some exceptions that were 40 minutes long. I have mixed feelings about this: On the one hand it was great to have a lot of different topics. On the other hand some of the concepts definitively would have needed more time to fully explain and grasp. Respect to all the speakers who had to think about what they would talk about in such a short timeframe.

Berlin Buzzwords is a fantastic conference and I will definitively go there again.

Content Extraction with Apache Tika

2012-05-11T10:58:00.003-07:00

Sometimes you need access to the content of documents, be it that you want to analyze it, store the content in a database or index it for searching. Different formats like word documents, pdfs and html documents need different treatment. Apache Tika is a project that combines several open source projects for reading content from a multitude of file formats and makes the textual content as well as some metadata available using a uniform API. I will show two ways how to leverage the power of Tika for your projects.

Accessing Tika programmatically

First, Tika can of course be used as a library. Surprisingly the user docs on the website explain a lot of the functionality that you might be interested in when writing custom parsers for Tika but don't show directly how to use it.

I am using Maven again, so I add a dependency for the most recent version:

<dependency>
    <groupId>org.apache.tika</groupId>
    <artifactId>tika-parsers</artifactId>
    <version>1.1</version>
    <type>jar</type>
</dependency>

tika-parsers also includes all the other projects that are used so be patient when Maven fetches all the transitive dependencies.

Let's see what some test code for extracting data from a pdf document called slides.pdf, that is available in the classpath, looks like.

Parser parser = new PdfParser();
BodyContentHandler handler = new BodyContentHandler();
Metadata metadata = new Metadata();
InputStream content = getClass().getResourceAsStream("/slides.pdf");
parser.parse(content, handler, metadata, new ParseContext());
assertEquals("Solr Vortrag", metadata.get(Metadata.TITLE));
assertTrue(handler.toString().contains("Lucene"));

First, we need to instanciate a Parser that is capable of reading the format, in this case PdfParser that uses PDFBox for extracting the content. The parse method expects some parameters to configure the parsing process as well as an InputStream that contains the data of the document. Metadata will contain all the metadata for the document, e.g. the title or the author after the parsing is finished.

Tika uses XHTML as the internal representation for all parsed content. This XHTML document can be processed by a SAX ContentHandler. A custom implementation BodyContentHandler returns all the text in the body area, which is the main content. The last parameter ParseContext can be used to configure the underlying parser instance.

The Metadata class consists of a Map-like structure with some common keys like the title as well as optional format specific information. You can look at the contents with a simple loop:

for (String name: metadata.names()) { 
    System.out.println(name + ": " + metadata.get(name));
}

This will produce an output similar to this:

xmpTPg:NPages: 17
Creation-Date: 2010-11-20T09:47:28Z
title: Solr Vortrag
created: Sat Nov 20 10:47:28 CET 2010
producer: OpenOffice.org 2.4
Content-Type: application/pdf
creator: Impress

The textual content of the document can be retrieved by calling the toString() method on the BodyContentHandler.

This is all fine if you exactly know that you only want to retrieve data from pdf documents. But you probably don't want to introduce a huge switch-block for determining the parser to use depending on the file name or some other information. Fortunately Tika also provides an AutodetectParser that employs different strategies for determining the content type of the document. The code above all stays the same, you just use a different parser:

Parser parser = new AutodetectParser();

This way you don't have to know what kind of document you are currently processing, Tika will provide you with metadata as well as the content. You can pass in additional hints for the parser e.g. the filename or the content type by setting it in the Metadata object.

Extracting content using Solr

If you are using the search server Solr you can also leverage its REST API for extracting the content. The default configuration has a request handler configured for /update/extract that you can send a document to and it will return the content it extracted using Tika. You just need to add the necessary libraries for the extraction. I am still using Maven so I have to add an additional dependency:

<dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr</artifactId>
    <version>3.6.0</version>
    <type>war</type>
</dependency>
<dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr-cell</artifactId>
    <version>3.6.0</version>
    <type>jar</type>
</dependency>

This will include all of the Tika dependencies as well as all necessary third party libraries.

Solr Cell, the request handler, normally is used to index binary files directly but you can also just use it for extraction. To transfer the content you can use any tool that can speak http, e.g. for curl this might look like this:

curl -F "file=@slides.pdf" "localhost:8983/solr/update/extract?extractOnly=true&extractFormat=text"

By setting the parameter extractOnly to true we advice Solr that we don't want to index the content but want to have it extracted to the response. The result will be the standard Solr XML format that contains the body content as well as the metadata.

You can also use the Java client library SolrJ for doing the same:

ContentStreamUpdateRequest request = new ContentStreamUpdateRequest("/update/extract");
request.addFile(new File("slides.pdf"));
request.setParam("extractOnly", "true");
request.setParam("extractFormat", "text");
NamedList<Object> result = server.request(request);

The NamedList will contain entries for the body content as well as another NamedList with the metadata.

Update

Robert has asked in the comments what the response looks like.
Solr uses configurable response writers for marshalling the message. The default format is xml but can be influenced by passing the wt attribute to the request. A simplified standard response looks like this:

curl -F "file=@slides.pdf" "localhost:8983/solr/update/extract?extractOnly=true&extractFormat=text"
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1952</int></lst><str name="slides.pdf">

Features                                                                                                                                                                            
                                                                                                                                                                                    
HTTPSchnittstelle                                                                                                                                                                  
XMLbasierte Konfiguration                                                                                                                                                          
Facettierung                                                                                                                                                                        
Sammlung nützlicher LuceneModule/Dismax                                                                                                                                            
                                                                                                                                                                                    
Features                                                                                                                                                                            
                                                                                                                                                                                    
HTTPSchnittstelle                                                                                                                                                                  
XMLbasierte Konfiguration                                                                                                                                                          
Facettierung                                                                                                                                                                        
Sammlung nützlicher LuceneModule/Dismax                                                                                                                                            
JavaClient SolrJ                                                                                                                                                                   
                                                                                                                                                                                    
[... more content ...] 
                                                                                                                                                                                   
</str><lst name="slides.pdf_metadata"><arr name="xmpTPg:NPages"><str>17</str></arr><arr name="Creation-Date"><str>2010-11-20T09:47:28Z</str></arr><arr name="title"><str>Solr Vortrag</str></arr><arr name="stream_source_info"><str>file</str></arr><arr name="created"><str>Sat Nov 20 10:47:28 CET 2010</str></arr><arr name="stream_content_type"><str>application/octet-stream</str></arr><arr name="stream_size"><str>425327</str></arr><arr name="producer"><str>OpenOffice.org 2.4</str></arr><arr name="stream_name"><str>slides.pdf</str></arr><arr name="Content-Type"><str>application/pdf</str></arr><arr name="creator"><str>Impress</str></arr></lst>                                                                            
</response>

The response contains some metadata (how long the processing took), the content of the file as well as the metadata that is extracted from the document.

If you pass the atrribute wt and set it to json, the response is contained in a json structure:

curl -F "file=@slides.pdf" "localhost:8983/solr/update/extract?extractOnly=true&extractFormat=text&wt=json"             
{"responseHeader":{"status":0,"QTime":217},"slides.pdf":"\n\n\n\n\n\n\n\n\n\n\n\nSolr Vortrag\n\n   \n\nEinfach mehr finden mit\n\nFlorian Hopf\n29.09.2010\n\n\n   \n\nSolr?\n\n\n   \n\nSolr?\n\nServerization of Lucene\n\n\n   \n\nApache Lucene?\n\nSearch engine library\n\n\n   \n\nApache Lucene?\n\nSearch engine library\nTextbasierter Index\n\n\n   \n\nApache Lucene?\n\nSearch engine library\nTextbasierter Index\nText Analyzer\n\n\n   \n\nApache Lucene?\n\nSearch engine library\nTextbasierter Index\nText Analyzer\nQuery Syntax \n\n\n   \n\nApache Lucene?\n\nSearch engine library\nTextbasierter Index\nText Analyzer\nQuery Syntax \nScoring\n\n\n   \n\nFeatures\n\nHTTPSchnittstelle\n\n\n   \n\nArchitektur\n\nClient SolrWebapp Lucene\nhttp\n\nKommunikation über XML, JSON, JavaBin, Ruby, ...\n\n\n   \n\nFeatures\n\nHTTPSchnittstelle\nXMLbasierte Konfiguration\n\n\n   \n\nFeatures\n\nHTTPSchnittstelle\nXMLbasierte Konfiguration\nFacettierung\n\n\n   \n\nFeatures\n\nHTTPSchnittstelle\nXMLbasierte Konfiguration\nFacettierung\nSammlung nützlicher LuceneModule/Dismax\n\n\n   \n\nFeatures\n\nHTTPSchnittstelle\nXMLbasierte Konfiguration\nFacettierung\nSammlung nützlicher LuceneModule/Dismax\nJavaClient SolrJ\n\n\n   \n\nDemo\n\n\n   \n\nWas noch?\nAdminInterface\nCaching\nSkalierung\nSpellchecker\nMoreLikeThis\nData Import Handler\nSolrCell\n\n\n   \n\nRessourcen\nhttp://lucene.apache.org/solr/\n\n\n\n","slides.pdf_metadata":["xmpTPg:NPages",["17"],"Creation-Date",["2010-11-20T09:47:28Z"],"title",["Solr Vortrag"],"stream_source_info",["file"],"created",["Sat Nov 20 10:47:28 CET 2010"],"stream_content_type",["application/octet-stream"],"stream_size",["425327"],"producer",["OpenOffice.org 2.4"],"stream_name",["slides.pdf"],"Content-Type",["application/pdf"],"creator",["Impress"]]}

There are quite some ResponseWriters available for different languages, e.g. for Ruby. You can have a look at them at the bottom of this page: http://wiki.apache.org/solr/QueryResponseWriter

Importing Atom feeds in Solr using the Data Import Handler

2012-05-07T09:58:00.001-07:00

I am working on a search solution that makes some of the content I am producing available through one search interface. One of the content stores is the blog you are reading right now, which among other options makes the content available here using Atom.

Solr, my search server of choice, provides the Data Import Handler that can be used to import data on a regular basis from sources like databases via JDBC or remote XML sources, like Atom.

Data Import Handler used to be a core part of Solr but starting from 3.1 it is shipped as a separate jar and not included in the standard war anymore. I am using Maven with overlays for development so I have to add a dependency for it:

<dependencies>
  <dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr</artifactId>
    <version>3.6.0</version>
    <type>war</type>
  </dependency>
  <dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr-dataimporthandler</artifactId>
    <version>3.6.0</version>
    <type>jar</type>
  </dependency>
</dependencies>

To enable the data import handler you have to add a request handler to your solrconfig.xml. Request handlers are registered for a certain url and, as the name suggests, are responsible for handling incoming requests:

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">data-config.xml</str>
  </lst>
</requestHandler>

The file data-config.xml that is referenced here contains the mapping logic as well as the endpoint to access:

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>
    <dataSource type="URLDataSource" encoding="UTF-8" connectionTimeout="5000" readTimeout="10000"/>
    <document>
        <entity name="blog"
                pk="url"
                url="http://fhopf.blogspot.com/feeds/posts/default?max-results=100"
                processor="XPathEntityProcessor"
                forEach="/feed/entry" transformer="DateFormatTransformer,HTMLStripTransformer,TemplateTransformer">
            <field column="title" xpath="/feed/entry/title"/>
            <field column="url" xpath="/feed/entry/link[@rel='alternate']/@href"/>
            <!-- 2012-03-07T21:35:51.229-08:00 -->
            <field column="last_modified" xpath="/feed/entry/updated" 
                dateTimeFormat="yyyy-MM-dd'T'hh:mm:ss.SSS" locale="en"/>
            <field column="text" xpath="/feed/entry/content" stripHTML="true"/>
            <field column="category" xpath="/feed/entry/category/@term"/>
            <field column="type" template="blog"/> 
        </entity>
    </document>
</dataConfig>

First we configure which datasource to use. This is where you alternatively would use another implementation when fetching documents from a database.

Documents describe the fields that will be stored in the index. The attributes for the entity element determine where and how to fetch the data, most importantly the url and the processor. forEach contains an XPath to identify the elements we'd like to loop over. The transformer attribute is used to specify some classes that are the available when mapping the remote XML to the Solr fields.

The field elements contain the mapping between the Atom document and the Solr index fields. The column attribute determines the name of the index field, xpath determines the node to use in the remote XML document. You can use advanced XPath options like mapping to attributes of elements where only another attribute is set. E.g. /feed/entry/link[@rel='alternate']/@href points to an element that determines an alternative representation of a blog post entry:

<feed ...> 
  ...
  <entry> 
    ...
    <link rel='alternate' type='text/html' href='http://fhopf.blogspot.com/2012/03/testing-akka-actors-from-java.html' title='Testing Akka actors from Java'/>
    ...
  </entry>
...
</feed>

For the column last_modified we are transforming the remote date format to the internal Solr representation using the DateProcessor. I am not sure yet if this is the correct solution as it seems to me I'm losing the timezone information. For the text field we are first removing all html elements that are contained in the blog post using the HTMLStripTransformer. Finally, the type contains a hardcoded value that is set using the TemplateTransformer.

To have everything in one place let's see how the schema for our index looks like:

<field name="url" type="string" indexed="true" stored="true" required="true"/>
<field name="title" type="text_general" indexed="true" stored="true"/>
<field name="category" type="text_general" indexed="true" stored="true" multiValued="true"/>
<field name="last_modified" type="date" indexed="true" stored="true"/>
<field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>
<field name="type" type="string" indexed="true" stored="false"/>

Finally, how can you trigger the dataimport? There is an option described in the Solr wiki, but probably a simple solution might be enough for you. I am using a shell script that is triggered by a cron job. These are the contents:

#!/bin/bash
curl localhost:8983/solr/dataimport?command=full-import

The data import handler is really easy to setup and you can use it to import quite a lot of data sources into your index. If you need more advanced crawling features you might want to have a look at Apache ManifoldCF, a connector framework for plugging content repositories into search engines like Apache Solr.

Testing Akka actors from Java

2012-03-07T01:02:00.008-08:00

If you're looking for a general introduction into using Akka from Java have a look at this post

In a recent project I've been using Akka for a concurrent producer-consumer setup. It is an actor framework for the JVM that is implemented in Scala but provides a Java API so normally you don't notice that your dealing with a Scala library.

Most of my business code is encapsulated in services that don't depend on Akka and can therefore be tested in isolation. But for some cases I've been looking for a way to test the behaviour of the actors. As I struggled with this for a while and didn't find a real howto on testing Akka actors from Java I hope my notes might be useful for other people as well.

The main problem when testing actors is that they are managed objects and you can't just instanciate them. Akka comes with a module for tests that is documented well for using it from Scala. But besides the note that it's possible you don't find a lot of information on using it from Java.

When using Maven you need to make sure that you have the akka-testkit dependency in place:

<dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-testkit</artifactId>
    <version>2.1-SNAPSHOT</version>
    <scope>test</scope>
</dependency>

I will show you how to implement a test for the actors that are introduced in the Akka java tutorial. It involves one actor that does a substep of calculating Pi for a certain start number and a given number of elements.

To test this actor we need a way to set it up. Akka-testkit provides a helper TestActorRef that can be used to set it up. Using scala this seems to be rather simple:

val testActor = TestActorRef[Worker]

If you try to do this from Java you will notice that you can't use a similar call. I have to admit that I am not quite sure yet what is going on. I would have expected that there is an apply() method on the TestActorRef companion object that uses some kind of implicits to instanciate the Worker object. But when inspecting the sources the thing that comes closest to it is this definition:

def apply[T <: Actor](factory: ⇒ T)(implicit system: ActorSystem)

No sign of implicit for the factory. Something I still have to investigate further.

To use it from Java you can use the method apply that takes a reference to a Function0 and an actor system. The actor system can be setup easily using

actorSystem = ActorSystem.apply();

The apply() method is very important in scala as it's kind of the default method for objects. For example myList(1) is internally using myList.apply(1).

If you're like me and expect that Function0 is a single method interface you will be surprised. It contains a lot of strange looking methods that you really don't want to have cluttering your test code:

TestActorRef workerRef = TestActorRef.apply(new Function0() {

    @Override
    public Worker apply() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public void apply$mcV$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public boolean apply$mcZ$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public byte apply$mcB$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public short apply$mcS$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public char apply$mcC$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public int apply$mcI$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public long apply$mcJ$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public float apply$mcF$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public double apply$mcD$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }
}, actorSystem);

The only method we really are interested in is the normal apply method. Where do those other methods come from? There is no obvious hint in the scaladocs.

During searching for the solution I found a mailing list thread that explains some of the magic. The methods are performance optimizations for boxing and unboxing that are automatically generated by the scala compiler for the @specialized annotation. Still, I am unsure about why this is happening exactly. According to this presentation I would have expected that I am using the specialized instance for Object, maybe that is something special regarding traits?

Fortunately we don't really need to implement the interface ourself: There's an adapter class, AbstractFunction0, that makes your code look much nicer:

@Before
public void initActor() {
    actorSystem = ActorSystem.apply();
    actorRef = TestActorRef.apply(new AbstractFunction0() {

        @Override
        public Pi.Worker apply() {
            return new Pi.Worker();
        }
           
    }, actorSystem);
}

This is like I would have expected it to behave in the first place.

Now, as we have setup our test we can use the TestActorRef to really test the actor. For example we can test that the actor doesn't do anything for a String message:

@Test
public void doNothingForString() {
    TestProbe testProbe = TestProbe.apply(actorSystem);
    actorRef.tell("Hello", testProbe.ref());

    testProbe.expectNoMsg(Duration.apply(100, TimeUnit.MILLISECONDS));
}

TestProbe is another helper that can be used to check the messages that are sent between cooperating actors. In this example we are checking that no message is passed to the sender for 100 miliseconds, which should be enough for execution.

Let's test some real functionality. Send a message to the actor and check that the result message is send:

@Test
public void calculatePiFor0() {
    TestProbe testProbe = TestProbe.apply(actorSystem);
    Pi.Work work = new Pi.Work(0, 0);        
    actorRef.tell(work, testProbe.ref());

    testProbe.expectMsgClass(Pi.Result.class);     
    TestActor.Message message = testProbe.lastMessage();
    Pi.Result resultMsg = (Pi.Result) message.msg();
    assertEquals(0.0, resultMsg.getValue(), 0.0000000001);
}

Now we use the TestProbe to block until a message arrives. When it's there we can have a look at using the lastMessage().

You can look at the rest of the test on Github. Comments are more than welcome as I am pretty new to Scala as well as Akka.

Update

As Jonas Bonér points out I've been using the Scala API. Using the Props class the setup is easier:

    @Before
    public void initActor() {
        actorSystem = ActorSystem.apply();
        actorRef = TestActorRef.apply(new Props(Pi.Worker.class), actorSystem);
    }

Legacy Code Retreat

2012-02-19T08:22:00.007-08:00

Yesterday I attended the first german Legacy Code Retreat in Bretten. The event was organized by Softwerkskammer, the german software craftsmanship community.

A legacy code retreat doesn't work like a common code retreat where you implement a certain functionality again and again. It instead starts with some really flawed code and the participants apply different refactoring steps to make it more testable and maintainable. There are six iterations of 45 minutes with different tasks or aims. For each iteration you work with a different partner and after a short retrospective with all participants you mostly start again from the original code.

The github repository for the legacy code contains the code in several languages among which are Java, C++, C# and Ruby.

Iteration 1

The first iteration was used to get to know the functionality of the code. There were no real rules so the participants were free to explore the code in any way they liked.

I paired with Heiko Seebach who I already knew to be a Ruby guy. We were looking at the code with a standard text editor, already quite unfamiliar to standard Java IDE work. I already got enough Ruby knowledge to understand code when I see it so this was no problem. For quite some time we tried to understand a certain aspect that was happening when running the code. It turned out that this was a bug in the Ruby-version of the code. Next we tried to setup RSpec and get starting with some tests.

During this iteration I didn't learn that much about the legacy code but more about some Ruby stuff.

Iteration 2

The target of the second iteration was to prepare a golden master test that could be used during all of the following iterations. The original legacy code is triggered by random input (in the Java version using java.util.Random) and writes all its state to System.out. We should capture the output for a certain input sequence and write it to a file. This can then automatically be compared to the output of a modified version. If both files are the same there are likely no regresions in the code.

I paired with another Java guy and we were working on my machine in Netbeans. I noticed how unfamiliar I am with standard Netbeans project setup as I am using Maven most of the time. We were doing the test and started some refactorings, all in all a quite productive iteration. Things I learned: java.util.Random really only uses the seed for its number generation so if you are using the same seed again and again you always get the same result. Also, when doing file stuff in plain Java I really miss commons-io.

Iteration 3

In Iteration 3 we were supposed to use an antipattern for testing: Subclass to Test. You take the original class and overwrite some methods in it that are called from the method to test.

It turned out that the original code is not suited well for this approach. There are only few methods that really rely on other methods. Most of the methods are accessing the state via the fields directly. Me and my partner therefore didn't really overwrite the methods but instead use an initializer block for prepareing the state of method calls. This is similar to an approach for Map-initialization that I started to apply only just recently:

Map data = new HashMap() {
    {
        put("key", "value");
    }
};

The approach worked quite fine for the given code but it's probably true that the tests won't stay maintainable.

Iteration 4

Iteration 4 was based on the previous iteration. All the methods that have been subclassed for testing should be moved to delegates and passed into the original class using a dependency injection approach.

I paired with a C++ guy who is doing embedded stuff during his day job on his C++ code. It turned out that we had quite different opinions and experiences. He was really focused on performance and couldn't understand why you would want to move methods to another class just to delegate to them as you are introducing overhead with the method call.

I haven't done any C++ programming since University. Eclipse seems to be suited well for development but compared to Java it still seems to lack a lot of convenience functionality.

Iteration 5

On Iteration 5 I paired with Tilman, a Clean Code Aficionado who I already knew from our local Java User Group. We were supposed to change as many methods as possible to real functions that don't work on fields but on parameter values only.

A lot of people were struggling with this approach at first. But it turns out if you are doing this you have a really good starting position for doing further refactorings more easily.

My partner was doing most of the coding with some input from me. We were taking some directions I wouldn't have taken by myself but the resulting code was really well structured and could be reduced in size. Also we worked with an interesting Eclipse plugin I had seen before already: Infinitest always runs the tests in the background, no need to run the tests manually. Have to check if there's something like this available for Netbeans as well.

Iteration 6

To be honest, I don't know what the goal of the sixth iteration really was. I was pairing with a developer that was still fighting with the failing tests from the previous iteration. Most of the iteration we tried to get these running again. In the last few minutes we managed to extract some clases and clean up some code.

Conclusion

The first german legacy code retreat really was a great experience. I learned a lot and, probably even more important, had a lot of fun.

The food and the location both were excellent. Thanks to the organizers Nicole and Andreas as well as the sponsors for making it possible. It's great to be able to attend a high quality event totally for free.

Running my Tests again

2012-01-10T13:29:00.000-08:00

For some time I've been bugged by a Netbeans problem that I couldn't find any solution to. When running a unit test from within Netbeans from time to time it happended that the tests just failed. They seemed to be executed in an old state. Running them again didn't help either, it seemed that some parts of the project didn't get recompiled. When executing the tests from a command line Maven build there were never any problems and afterwards the tests could be run again from Netbeans. The problem only occured very infrequently but nevertheless it was really annoying. I started not running the tests from Netbeans at all but only using Maven. That is also not a good solution as you either run all tests or have to edit the command line all the time for running only a single test.

Recently I noticed what caused the problem: Netbeans has its compile on save feature on for tests. This means it is using its internal incremental compile feature which doesn't seem to work fine at least for some project setups.

You can disable it in the project properties on the Build/Compile node. I haven't seen any problems since disabling it. Saves me a lot of time to run the tests from the IDE again.

Talking about Code

2011-12-29T01:22:00.000-08:00

Yesterday I attended the Softwerkskammer Karlsruhe meetup for the first time. Softwerkskammer tries to connect the Software craftmanship community in Germany.

The topic for the evening was simple: More Code. We looked at a lot of samples from a real project and discussed what's wrong with them and what could be done better. There were a lot of different opinions but that's a good thing as I got to question some habits I have when programming.

This has been the first time I've been to a meeting where there is a lively discussion like this. The conferences and user groups I attend mostly have classic talks with one speaker and far less audience participation. Talking about code is a really good way to learn and this won't be the last time I attended a meetup. Thanks to the organizers.

Spring in Action

2011-12-26T07:46:00.000-08:00

Sometimes it's comfortable to not be an absolute expert in a certain technology. This makes it really easy to learn new stuff, e.g. by profane methods like reading a book. Even if you are a Spring expert it is still likely that you will take something from the latest edition of Spring in Action by Craig Walls as it covers a wide range of topics. I haven't read one of the predecessors but people told me that those are even better.

Having finished the book recently I just wanted to take the time to write down two interesting small configuration features that I learned from it.

p-Namespace

A feature that I just didn't know before but seems to be quite useful is the p-Namespace. It's a namespace that is not backed by a schema and allows to configure beans in a really concise way. For example look at how a bean might be configured normally:

    <bean id="foo" class="foo.bar.Baz">
        <property name="myLongProperty" value="2"/>
        <property name="myStringProperty" value="Hallo"/>
    </bean>

The properties we'd like to set are children of the bean node. Netbeans comes with nice autocompletion support for the property names as you can see from the screenshot.

The p-Namespace is a more concise version where the property names itself become attributes of the bean node:

    <bean id="foo" class="foo.bar.Baz"
        p:myLongProperty="2" p:myStringProperty="Hallo"/>

See that Netbeans is also clever enough to offer code completion here as well.

I am not sure if I will use the short form of the p-Namespace a lot. A consistent use of the features in a project is quite important so I think if the short form is used it should be used everywhere in the project.

Accessing Constants

Sometimes you need to access some constants in your Spring configuration files. There are several ways to handle this, one of it using the util-Namespace:

<property name="day">
    <util:constant static-field="java.util.Calendar.WEDNESDAY"/>
</property>

Another way can be to use the Spring Expression Language to access it:

<property name="day" value="#{T(java.util.Calendar).WEDNESDAY}"/>

I think this can be used more commonly as the value doesn't need to be registered as a subnode. For example I had some problems using util:constant as key or value in a util:map. That would have been easy just using the EL version.

Not another Diamond Operator Introduction

2011-12-07T13:46:00.000-08:00

I just returned from the talk "Lucky Seven" of our local Java User Group. It was far better than I expected. Not that I expected Wolfgang Weigend to be a bad speaker but though I organized the event I got the feeling that I had seen one too many Java 7 introductions already. But there was more ...

One of the interesting aspects that I haven't been paying that much attention to is the merge of the JRockit and Hotspot VM. Hotspot will be the base of the new development and JRockit features will be merged in. Some of these features will already be available in OpenJDK during the JDK 7 timespan.

One of the changes got some amount of interest lately: The PermGen space will be removed. Sounds like a major change but, once it works, it will definitively be a huge benefit.

JRockit has been highly respected for its monitoring features. Among those is the interesting Java Flight Recorder that reminds me of the commercial project Chronon. It will be an always on recording of data in the JVM that can be used for diagnostic purposes. Sounds really interesting!

The overall goal of the convergence is to have a VM that can tune itself. Looking forward to it!

The (mixed german and english) slides of the talk are available for download.

Getting started with Gradle

2011-10-26T01:58:00.000-07:00

Maven has been my build tool of choice for some years now. Coming from Ant the declarative approach, useful conventions as well as the dependency management offered a huge benefit. But as with most technologies the more you are using it the more minor and major flaws appear. A big problem is that with Maven builds are sometimes not reproducable. The outcome of the build is influenced by the state of your local repository.

Gradle is a Groovy based build system that is often recommended as a more advanced system. The features that make it appealing to me are probably the easier syntax and the advanced dependency cache.

For a recent project that I just uploaded for someone else I needed to add a simple way to build the jar. Time to do it with Gradle and see what it feels like.

The build script

The purpose of the build is simple: compile some classes with some dependencies and package those to a jar file. Same as Maven and Ant, Gradle also needs at least one file that describes the build. This is what build.gradle looks like:

apply plugin: 'java'

repositories {
    mavenCentral()
    mavenRepo url: "http://bp-cms-commons.sourceforge.net/m2repo"
}

dependencies {
    compile group: 'org.opencms', name: 'opencms-core', version: '7.5.4'
    compile group: 'javax.servlet', name: 'servlet-api', version: '2.5'
}

sourceSets {
    main {
        java {
            srcDir 'src'
        }
    }
}

Let's step through the file line by line. The first line tells gradle to use the java plugin. This plugin ships with tasks for compiling and packaging java classes.

In the next block we are declaring some dependency repositories. Luckily Gradle supports Maven repositories so existing repositories like Maven central can be used. I guess without this feature Gradle would not gain a lot of adoption at all. There are two repos declared: Maven central where most of the common dependencies are stored and a custom repo that provides the OpenCms dependencies.

The next block is used to declare which dependencies are necessary for the build. Gradle also supports scopes (in Gradle: configurations) so for example you can declare that some jars are only needed during test run. The dependency declaration is in this case similar to the Maven coordinates but Gradle also supports more advanced features like version ranges.

The last block isn't really necessary. It's only there because my Java sources are located in src instead of the default src/main/java. Gradle uses a lot of the Maven conventions so it's really easy to migrate builds.

Building

To build the project you need Gradle installed. You can download a single distribution that already packages Grooovy and all the needed files. You only need to add the bin folder to your path.

Packaging the jar is easy: You just run the jar task in the java plugin: gradle :jar. Gradle will start to download all direct and transitive dependencies. The fun part: It uses a nice command line library that can display text in bold, rewrite lines and the like. Fun to watch it.

I like the simplicity and readability of the build script. You don't need to declare anything if you don't really need it. No coordinates, no schema declaration, nothing. I hope I will find time to use it in a larger project so I can see what it really feels like in the daily project work.

Book Review: Solr 1.4 Enterprise Search Server

2011-02-28T11:08:00.000-08:00

I've been interested in Solr since I read about it the first time, must have been some time in 2008, doing some research for a search centric web page that was supposed to be run on OpenCms but unfortunately was never developed. At that time I wouldn't have used it as I hadn't heard about it before but I liked the idea a lot. After having attended the Devoxx university session by Eric Hatcher on Solr in 2009 I was completely sure that the next search system I would implement would be based on Solr. The project's nearly finished now, time to recap what I took out of the book I got for learning Solr.

First of all, when learning a new technology I prefer paper books over internet research. Though there are other books available, Solr 1.4 Enterprise Search Server by David Smiley and Eric Pugh seems to be the one that is most often recommended.

The book starts off with a high level introduction into what Solr and Lucene are, some first examples and interestingly, how to build Solr from source. Though the book was released before Solr 1.4 the authors seemed to have the foresight that some features might still be lacking and had to be included manually. In fact, I've never seen an open source project where applying patches is such a common thing as it seems to be the case for Solr.

Schema configuration and text analysis are the topics for the second chapter. It begins with an introduction into MusicBrainz, a freely available data set of music data is used as an example throughout the book. This chapter is crucial to the understanding of Solr as it introduces a lot of Lucene concepts that probably not every reader is familiar with.

After quite some theory chapter 3 starts with the practical parts, covering the indexing process. Curl, the command line http client, is used to send data to solr and retrieve it. Another option, the data import handler, that directly imports data from a database, is also introduced.

Chapter 4 to 6 walk the reader through the search process and several useful components to enhance the users search experience like faceting and the dismax request handler. This is the part where Solr really shines as you can see how easy it is to integrate new features in your application that probably would have taken a long time to develop using plain Lucene.

Deploying Solr is covered in Chapter 7 with quite some useful information on configuring and monitoring a Solr instance. Chapter 8 looks at some client APIs from different programmin languages, SolrJ being the most important to me. The book ends with an in-depth look at how Solr can be tunded and scaled.

I can say that this is a really excellent book, as an introduction to Solr as well as a reference while developing your application. The most common use cases are covered, the examples make it really easy to adopt the concepts in your application. There are lots of hands on information that prove useful during development and deployment of your application.

Some slight drawbacks I don't want to keep to myself: As the common message format for Solr is a custom XML dialect, there is a lot of XML in the book to digest. As it's so common to use it that's not necessarily a bad thing but you might get quite dizzy looking at a lot of angle brackets. From a readers perspective some variety would have been nice e.g. by mixing XML with the Ruby format or JSON or introducing client APIs earlier. Also, while it's a good idea to use a data set that is freely available, MusicBrainz probably isn't the best format for demoing some features. There are no large text sections or documents, which are often what a search application will be build on. And finally, not really an issue of the authors but rather of the publisher, PacktPub: When skimming through the book it's quite hard to see when a new section begins. The headlines do not contain a numbering scheme and are of a very similar size.

Nevertheless, if you have to develop an application using Solr, you should by all means buy this book, you won't regret it.

Running Ruby on Rails Tests in Netbeans

2011-01-22T17:55:00.000-08:00

I don't get it. Netbeans is often recommended as an excellent IDE for Ruby on Rails development, not only when targeting the JVM. Nevertheless, even some basic features don't seem to be working with the default setup. You can't even run the tests, which is fundamental to developing using a dynamic language.

What's happening? Suppose you have a simple app and you want to run some tests using the test database. Not sure if this is mandatory when using the built in JRuby but it seems to be normal to use the jdbcmysql adapter. When you try to run the tests you will see something like this:


1) Error:
test_index_is_ok(ContactsControllerTest):
ActiveRecord::StatementInvalid: ActiveRecord::JDBCError: Table 'kontakt_test.contacts' doesn't exist: DELETE FROM `contacts`

followed by the stack trace that isn't really helpful as it's not the root cause. Rails somehow doesn't create the tables in the test database. You'll see a more helpful output when starting the rake task "db:test:prepare" directly in debug mode:


** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
rake aborted!
Task not supported by 'jdbcmysql'
/path/to/netbeans-6.9.1/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rails-2.3.8/lib/tasks/databases.rake:380
/path/to/netbeans/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
/path/to/netbeans/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
/path/to/netbeans/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
[...]
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment 
** Execute db:test:purge

The task fails in the database task in the rails lib. You can open up the source code by opening the node Libraries/Built-in-JRuby/rails-2.3.8/lib/tasks/databases.rake in Netbeans.

At line 357 you can see the problem: Rails only expexts some hardcoded adapters, jdbcmysql not being one of them. It skips the task for unknown adapters. Two options to fix it: Insert a regular expression that matches both:


when /mysql/ # instead of when "mysql"

or add the jdbcmysql adapter as a second option:


when "mysql","jdbcmysql"

Now the tests are running and hopefully passing. The same kind of error might occur for other tasks as well as there are some more checks for the mysql adapter in this file. You should be able to fix them the same way.

I wouldn't have expected to have to patch the rails code for using it in Netbeans but this doesn't seem to be uncommon. Using a recent active record version is supposed to fix the problem as you can use mysql as an adapter name then but I didn't find a way to run the jdbc generator from Netbeans. It isn't available in the list of generators and I didn't find a generator gem to download.

What's to be learned for me from this? I got a better understanding of how the build process works using rake. But more importantly: even technologies that have been hyped for a long time might not be that flawless as you would expect.

Refactoring in Git

2011-01-09T04:49:00.000-08:00

To me, when using SVN, the most important reason for using an IDE plugin was the refactoring support: SVN doesn't notice when you rename a file, you have to explicitly call svn mv.

I thought this would be a major problem with Git, as a Java refactoring changes the content and the filename in one go. As the content changes the SHA1-checksum also changes and you'd run into problems. Fortunately, that's not the case.

With Git, you don't need a special operation: It detects renames with minor changes automatically.

Time for a test. Suppose you have a simple Java class like this:

public class TestClass {

    public static void main(String [] args) {
        System.out.println("Hello Git");
    }

}

Commit it to the Git repository:

flo@hank:~/git-netbeans$ git add src/TestClass.java
flo@hank:~/git-netbeans$ git commit -m "added test class"
[master 9269c2f] added test class
 1 files changed, 7 insertions(+), 0 deletions(-)
 create mode 100644 src/TestClass.java

Rename the class (either by using an IDE or by executing a manual refactoring by changing the file name and the class name):

public class TestClassWithNewName {

    public static void main(String [] args) {
        System.out.println("Hello Git");
    }

}

git status will tell you something like this:

flo@hank:~/git-netbeans$ git status
# On branch master
# Changed but not updated:
#   (use "git add/rm ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#       deleted:    src/TestClass.java
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       src/TestClassWithNewName.java
no changes added to commit (use "git add" and/or "git commit -a")

Doesn't look that good yet. It detects an added and a removed file. Next, stage the changes and have another look at the status:

flo@hank:~/git-netbeans$ git rm src/TestClass.java
rm 'src/TestClass.java'
flo@hank:~/git-netbeans$ git add src/TestClassWithNewName.java
flo@hank:~/git-netbeans$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       renamed:    src/TestClass.java -> src/TestClassWithNewName.java
#

Neat, Git detected a rename. Let's commit and see the log:

flo@hank:~/git-netbeans$ git commit -m "refactored class"
[master 4acd7f1] refactored class
 1 files changed, 1 insertions(+), 1 deletions(-)
 rename src/{TestClass.java => TestClassWithNewName.java} (72%)
flo@hank:~/git-netbeans$ git log src/TestClassWithNewName.java
commit 4acd7f19ccd6cc02816ee7f1293ea5a69d7a4ca7
Author: Florian Hopf 
Date:   Sun Jan 9 14:27:59 2011 +0100

    refactored class

Hmmm, only the last commit? Looks like we have to tell that we want to follow renames:

flo@hank:~/git-netbeans$ git log --follow src/TestClassWithNewName.java
commit 4acd7f19ccd6cc02816ee7f1293ea5a69d7a4ca7
Author: Florian Hopf 
Date:   Sun Jan 9 14:27:59 2011 +0100

    refactored class

commit 9269c2fd194b2bd2b93a18ab88f21fb2180c5870
Author: Florian Hopf 
Date:   Sun Jan 9 13:48:35 2011 +0100

    added test class

What do I take from this experiment? I guess I won't use the Netbeans Git plugin for now. I still have to get acquainted to the command line and its better to learn the basics first.