OUTERTHOUGHT.BLOG

Lily 1.2 is out!

2012-04-19T12:06:10.000Z

We’re very happy to announce the release of Lily 1.2. It contains a number of improvements and exciting new features, both on the open source and the enterprise side of things.

Lily Core 1.2

First of all, Lily has been upgraded and verified to be CDH3u3 compatible. We’re also closely tracking CDH4 evolutions and are confident to be among the first to ship a CDH4-compliant version later this year.

The most important two new features of Lily Core 1.2 are along improved support for HBase-style operations, more importantly Scanners and Map/Reduce support.

Lily Scanners

Similar to HBase Scanners, Lily Scanners allow you to scan over records, making use of the natural ordering of records using their row key or record ID. As you might know, Lily supports both system-generated as well as user-specified IDs, and experience taught us the latter are commonly used. Using user-specified IDs, Lily Scanners allow you to influence the storage order of records in Lily, and iterate over them based on ID, range of IDs (start/stop), in an indexed fashion, i.e. fast.

Even better, Scanners support the use of filters to select records based on filter expressions, which means if you’re smart about key design, you might no longer need other search indexes to Lily content (such as Solr) for simple search and retrieval operations.

You can selectively return only a subset of records fields while scanning, and filter based on RecordType and record ID prefix. Also, using filters, this allows for (Solr) index rebuilds of partial data sets.

Lily 1.2 ships with a CLI tool allowing to do simple command line Scans.

Lily Map/Reduce

Even though Map/Reduce was already being used for scalable batch rebuilding of search indexes, it wasn’t generally available for people that wanted to process Lily data in map-reduce style for other purposes.

With the 1.2 release, Lily supports a LilyScanInputFormat which allows you to use Scanners (and filters) to specify the input to a Lily-based map/reduce job. Lily ships with Maven artefacts for generating a skeleton M/R job, too.

Lily testing framework

For those of you using the Lily test framework to develop Lily client applications, you will find startup and tear-down times of the ‘Lily Embed’ tool to be much shorter. Being software engineers ourselves for a long time, we care a lot about the engineer’s experience when developing against Lily. With this improvement, running a Lily development workbench on a laptop or workstation, building and testing has become much more efficient.

Lily Enterprise 1.2

The 1.2 release of Lily Enterprise contains two major improvements: a better, more stable cluster installer, and support for Pentaho Kettle for ETL operations.

Pentaho Data Integration support

Lily is often used as the central data aggregation tier in enterprise deployments, and we want to make the task of bringing data into Lily as easy as possible.

Lily Enterprise 1.2 contains a Kettle LilyOutput plugin. Kettle, or Pentaho Data Integration, is a powerful ETL tool that connects to many different data sources. Using a simple, UI-based configuration, the Lily Kettle integration allows you to effortlessly, and most importantly without any coding, set up a transformation and loading pipeline to inject data into the Lily Data Repository from a variety of sources, such as relational databases, static files, and even enterprise back-end systems.

Roadmap

Starting with this release, we are now committed to regular releases of Lily Core and Enterprise. We will have three major releases per year – with Lily Enterprise customers having access to schedule and planned features.

Thanks for reading this far, and we hope you will enjoy using Lily 1.2 as much as we enjoyed building it.

Outerthought joins NGDATA

2012-02-17T08:32:38.000Z

I have some really exciting news to share with you today: Outerthought is now part of a larger company that is committed to bringing Lily to the next level.

As interest in Lily was growing at an increasingly fast pace, it became clear that it deserved a strong, growth-targeted and globally-oriented company with a world-class management team.

I'm therefore extremely excited to announce that Outerthought is now part of NGDATA, a Big Data software company that makes sense of your data. NGDATA is planning to secure additional capital to accelerate the development and roll-out of Lily’s roadmap and to support business development activities in the United States. We will make Lily into the best Big Data management platform out there, with a special focus on large-scale machine learning and recommendations in real-time.

My role will be VP Product of NGDATA, listening to and working with customers and partners, market research and analysts, and generally making sure Lily has an awesome roadmap. I welcome Luc Burgelman (CEO), Frank Hamerlinck (COO) and Jürgen Ingels as co-founders and private investors of NGDATA, and Anne-Mie Poot as our VP HR.

I'm really proud of what Outerthought has achieved in the Big Data space so far, and the future is looking really promising. I'd also like to thank the Outerthought team for their effort and contribution in getting us where we are today! Thanks to anyone who helped us along the way - you won't be disappointed with what comes next.

The official announcement can be read on the NGDATA website - http://ngdata.com/site/news/96-ng.html.

Steven Noels
CEO Outerthought,
VP Product NGDATA.

A First Exploration Of SolrCloud

2012-02-08T15:54:05.000Z

SolrCloud has recently been in the news and was merged into Solr trunk, so it was high time to have a fresh look at it.

The SolrCloud wiki page gives various examples but left a few things unclear for me. The examples only show Solr instances which host one core/shard, and it doesn’t go deep on the relation between cores, collections and shards, or how to manage configurations.

In this blog, we will have a look at an example where we host multiple shards per instance, and explain some details along the way.

The setup we are going to create is shown in this diagram.

SolrCloud terminology

In SolrCloud you can have multiple collections. Collections can be divided into partitions, these partitions are called slices. Each slice can exist in multiple copies, these copies of the same slice are called shards. So the word shard has a bit of a confusing meaning in SolrCloud, it is rather a replica than a partition. One of the shards within a slice is the leader, though this is not fixed: any shard can become the leader through a leader-election process.

Each shard is a physical index, so one shard corresponds to one Solr core.

If you look at the SolrCloud wiki page, you won’t find the word slice [anymore]. It seems like the idea is to hide the use of this word, though once you start looking a bit deeper you will encounter it anyway so it’s good to know about it. It’s also good to know that the words shard and slice are often used in ambiguous ways, switching one for the other (even in the sources). Once you know this, things become more comprehensible. An interesting quote in this regard: “removing that ambiguity by introducing another term seemed to add more perceived complexity”. In this article I'll use the words slice and shard as defined above, so that we can distinguish the two concepts.

In SolrCloud, the Solr configuration files like schema.xml and solrconfig.xml are stored in ZooKeeper. You can upload multiple configurations to ZooKeeper, each collection can be associated with one configuration. The Solr instances hence don’t need the configuration files to be on the file system, they will read them from ZooKeeper.

Running ZooKeeper

Let’s start by launching a ZooKeeper instance. While Solr allows to run an embedded ZooKeeper instance, I find that this rather complicates things. ZooKeeper is responsible for storing coordination and configuration information for the cluster, and should be highly available. By running it separately, we can start and stop Solr instances without having to think about which one(s) embed ZooKeeper.

For the purpose of this article, you can get it running like this:

download ZooKeeper 3.3. Don’t take version 3.4, as it’s not recommended for production yet.
extract the download
copy conf/zoo_sample.cfg to conf/zoo.cfg
mkdir /tmp/zookeeper (path can be changed in zoo.cfg)
./bin/zkServer.sh start

And that’s it, you have ZooKeeper running.

Setting up Solr instance directories

We are going to run two Solr instances, thus we’ll need two Solr instance directories.

Let’s create two directories like this:

mkdir -p ~/solrcloudtest/sc_server_1
mkdir -p ~/solrcloudtest/sc_server_2

Create the file ~/solrcloudtest/sc_server_1/solr.xml containing:

<solr persistent="true">
  <cores adminPath="/admin/cores" hostPort=”8501”>
  </cores>
</solr>

Create the file ~/solrcloudtest/sc_server_2/solr.xml containing (note the different hostPort value):

<solr persistent="true">
  <cores adminPath="/admin/cores" hostPort=”8502”>
  </cores>
</solr>

We need to specify the hostPort attribute since Solr can’t detect the port, it falls back to the default 8983 when not specified.

This is all we need: the actual core configuration will be uploaded to ZooKeeper in the next section.

Creating a Solr configuration in ZooKeeper

As explained before, the Solr configuration needs to be available in ZooKeeper rather than on the file system.

Currently, you can upload a configuration directory from the file system to ZooKeeper as part of the Solr startup. It is also possible to run ZkController’s main method for this purpose (SOLR-2805), but as there’s no script to launch it, the easiest way right now to upload a configuration is by starting Solr:

export SOLR_HOME=/path/to/solr-trunk/solr

# important: move into one the instance directory!
# (otherwise Solr will start up with defaults and create a core etc.)
cd sc_server_1

java \
 -Djetty.port=8501 \
 -Djetty.home=$SOLR_HOME/example/ \
 -Dsolr.solr.home=. \
 -Dbootstrap_confdir=$SOLR_HOME/example/solr/conf \
 -Dcollection.configName=config1 \
 -DzkHost=localhost:2181 \
 -jar $SOLR_HOME/example/start.jar

Now that the configuration is uploaded, you can stop this Solr instance again (ctrl+c).

We can now check in ZooKeeper that the configuration has been uploaded, and that no collections have been created yet.

For this, go to the ZooKeeper directory, and run ./bin/zkCli.sh, and do the following commands:

[zk: localhost:2181(CONNECTED) 1] ls /configs
[config1]

[zk: localhost:2181(CONNECTED) 2] ls /collections
[]

You could repeat this process to upload more configurations.

If you would like to change a configuration later on, you essentially have to upload it again in the same way. The various Solr cores that make use of that configuration won’t be reloaded automatically however (SOLR-3071).

Starting the Solr servers

All SolrCloud magic is activated by specifying the zkHost parameter. Without this parameter, you run Solr ‘classic’, with the parameter, you run SolrCloud. If you look into the source code, you will see that this parameter causes the creation of a ZkController, and at various places checks of the kind ‘zkController != null’ are done to change behavior when in cloud mode.

Open two shells, and start the two Solr instances:

export SOLR_HOME=/path/to/solr-trunk/solr
cd sc_server_1
java \
 -Djetty.port=8501 \
 -Djetty.home=$SOLR_HOME/example/ \
 -Dsolr.solr.home=. \
 -DzkHost=localhost:2181 \
 -jar $SOLR_HOME/example/start.jar

and (note: different instance dir & jetty port)

export SOLR_HOME=/path/to/solr-trunk/solr
cd sc_server_2
java \
 -Djetty.port=8502 \
 -Djetty.home=$SOLR_HOME/example/ \
 -Dsolr.solr.home=. \
 -DzkHost=localhost:2181 \
 -jar $SOLR_HOME/example/start.jar

Note that now, we don’t have to specify the boostrap_confdir and collection.configName properties anymore (though that last one can still be useful as default sometimes, but not with the way we will create collections & shards below).

We have neither added the -Dnumshards parameter, which you might have encountered elsewhere. When you manually assign cores to shards as we will do below, I don’t think it serves any purpose.

So the situation now is that we have two Solr instances running, both with 0 cores.

Define the cores, collections, slices, shards

We are now going to create cores, and assign each core to a specific collection and slice. It is not necessary to define collections & shards anywhere, they are implicit by the fact that there are cores that refer them.

In our example, the collection is called 'collectionOne' and the slices are called 'slice1' and 'slice2'.

Let’s start with creating a core on the first server:

curl 'http://localhost:8501/solr/admin/cores?action=CREATE&name=core_collectionOne_slice1_shard1&collection=collectionOne&shard=slice1&collection.configName=config1'

(in the URL above, and the solr.xml snippet below, the word 'shard' is used for 'slice')

If you have a look now at sc_server_1/solr.xml, you will see the core was added:

<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="true">
  <cores adminPath="/admin/cores" zkClientTimeout="10000" hostPort="8501"
hostContext="solr">
    <core
       name="core_collectionOne_slice1_shard1"
       collection="collectionOne"
       shard="slice1"
       instanceDir="core_collectionOne_slice1_shard1/"/>
  </cores>
</solr>

AFAIU the information in ZooKeeper takes precedence, so the attributes collection and shard on the core above serve more as documentation, or they are of course also relevant if you would create cores by listing them in solr.xml rather than using the cores-admin API. Actually listing them in solr.xml might be simpler than doing a bunch of API calls, but there is currently one limitation: you can’t specify the configName this way.

In ZooKeeper, you can verify this collection is associated with the config1 configuration:

[zk: localhost:2181(CONNECTED) 5] get /collections/collectionOne
{"configName":"config1"}

And you can also get an overview of all collections, slices and shards like this:

[zk: localhost:2181(CONNECTED) 0] get /clusterstate.json
{
  "collectionOne": {
    "slice1": {
      "fietsbel:8501_solr_core_collectionOne_slice1_shard1": {
        "shard_id":"slice1",
        "leader":"true",
        "state":"active",
        "core":"core_collectionOne_slice1_shard1",
        "collection":"collectionOne",
        "node_name":"fietsbel:8501_solr",
        "base_url":"http://fietsbel:8501/solr"
      }
    }
  }
}

(the somewhat strange "shard_id":"slice1" is just a back-pointer from the shard to the slice to which it belongs)

Now let’s create the remaining cores: one more on server 1, and two on server 2 (notice the different port numbers to which we send these requests).

curl 'http://localhost:8502/solr/admin/cores?action=CREATE&name=core_collectionOne_slice2_shard1&collection=collectionOne&shard=slice2&collection.configName=config1'

curl 'http://localhost:8501/solr/admin/cores?action=CREATE&name=core_collectionOne_slice2_shard2&collection=collectionOne&shard=slice2&collection.configName=config1'

curl 'http://localhost:8502/solr/admin/cores?action=CREATE&name=core_collectionOne_slice1_shard2&collection=collectionOne&shard=slice1&collection.configName=config1'

Let’s have a look in ZooKeeper at the current state of the clusterstate.json:

[zk: localhost:2181(CONNECTED) 0] get /clusterstate.json
{
  "collectionOne": {
    "slice1": {
      "fietsbel:8501_solr_core_collectionOne_slice1_shard1": {
        "shard_id":"slice1",
        "leader":"true",
        "state":"active",
        "core":"core_collectionOne_slice1_shard1",
        "collection":"collectionOne",
        "node_name":"fietsbel:8501_solr",
        "base_url":"http://fietsbel:8501/solr"
      },
      "fietsbel:8502_solr_core_collectionOne_slice1_shard2": {
        "shard_id":"slice1",
        "state":"active",
        "core":"core_collectionOne_slice1_shard2",
        "collection":"collectionOne",
        "node_name":"fietsbel:8502_solr",
        "base_url":"http://fietsbel:8502/solr"
      }
    },
    "slice2": {
      "fietsbel:8502_solr_core_collectionOne_slice2_shard1": {
        "shard_id":"slice2",
        "leader":"true",
        "state":"active",
        "core":"core_collectionOne_slice2_shard1",
        "collection":"collectionOne",
        "node_name":"fietsbel:8502_solr",
        "base_url":"http://fietsbel:8502/solr"
      },
      "fietsbel:8501_solr_core_collectionOne_slice2_shard2": {
        "shard_id":"slice2",
        "state":"active",
        "core":"core_collectionOne_slice2_shard2",
        "collection":"collectionOne",
        "node_name":"fietsbel:8501_solr",
        "base_url":"http://fietsbel:8501/solr"
      }
    }
  }
}

We see we have:

one collection named collectionOne
two slices named slice1 and slice2
each slice has two shards. Within each slice, one shard is the leader (see "leader":"true"), the other(s) are replicas. Of each slice, one shard is hosted in each Solr instance.

Adding some documents

Now let’s try our setup works by adding some documents:

cd $SOLR_HOME/example/exampledocs
java -Durl=http://localhost:8501/solr/core_collectionOne_slice1_shard1/update -jar post.jar *.xml

We sent the request to one specific core, but you could have picked any other core and the end result would be the same. The request will be forwarded automatically to the leader shard of the appropriate slice. The slice is selected based on the hash of the id of the document.

Use the admin stats page to see documents got added to all cores:

http://localhost:8501/solr/core_collectionOne_slice1_shard1/admin/stats.jsp
http://localhost:8502/solr/core_collectionOne_slice1_shard2/admin/stats.jsp

http://localhost:8502/solr/core_collectionOne_slice2_shard1/admin/stats.jsp
http://localhost:8501/solr/core_collectionOne_slice2_shard2/admin/stats.jsp

In my case, the cores for slice1 got 16 documents, those for slice2, 12. Unlike the traditional Solr replication, with SolrCloud updates are sent directly to the replica's.

Querying

Let’s just query all documents. Again, we send the request to one particular shard:

http://localhost:8501/solr/core_collectionOne_slice1_shard1/select?q=*:*

You will see the numFound=”28”, the sum of 16 and 12.

What happens internally is that when you sent a request to a core, when in SolrCloud mode, Solr will look up what collection the core is associated with, and do a distributed query across all slices (it will pick one shard for each slice).

The SolrCloud wiki page gives the suggestion that you can use the collection name in the URL (like /solr/collection1/select). In our example, this would then be /solr/collectionOne/select. This is however not the case, but rather a particularity of that example. As long as you don’t host more than one slice and shard of the same collection in one Solr server, it can make sense to use such a core naming strategy.

Starting from scratch

When playing around with this stuff, you might want to start from scratch sometimes. In such case, don’t forget you have to remove data in three places: (1) the state stored in ZooKeeper (2) the cores defined in solr.xml and (3) the instance directories of these cores.

When writing the first draft of this article, I was using just one Solr instance and tried to have all the 4 cores (including replica's) in one Solr instance. Turns out there was a bug that prevents this from working correctly (SOLR-3108).

Managing slices & shards

Once you have defined a collection, you can not (or rather should not) add new slices to it, since documents won’t be automatically moved to the new slice to fit with the hash-based partitioning (SOLR-2595).

Adding more replica shards should be no problem though. While above we have used a very explicit way of assigning each core to a particular slice, you can actually leave that parameter off and Solr will automatically assign it to some slice within the collection. (I guess here the -Dnumshards parameter kicks in to decide whether the new core should be a slice or a shard)

How about removing replicas? It can be done, but manually. You have to unload the core and remove the related state in ZooKeeper. This is an area that will be improved upon later. (SOLR-3080)

Another interesting thing to note is that when your run in SolrCloud mode, all cores will automatically take part in the cloud thing. If you add a core without specifying the collection, a collection named after that core will be created. You can’t mix ‘classic’ cores and ‘cloud’ cores in one Solr instance.

Conclusion

In this article we have barely touched the surface of everything SolrCloud is: there’s the update log for durability and recovery, the sync’ing between replica’s, the details of distributed querying and updating, the special _version_ field to help with some of the these points, the coordination (election of overseer & shard leaders), … Much interesting stuff to explore!

As becomes clear from this article, SolrCloud isn't as easy to use yet as ElasticSearch. It still needs polishing and there's more manual work involved in setting it up. To some extent this has its advantages, as long as it's clear what you can expect from the system, and what you have to take care of yourself. Anyway, it's great to see that the Solr developers were able to catch up with the cloud world.

Lily 1.1 is out: what's new?

2011-12-14T14:18:53.000Z

We have a nice Christmas present for you: Lily 1.1 is out, and there's improvements for everyone: developers, administrators and Lily hackers. Read more about the exciting new stuff in Lily 1.1 below!

Complex Field Types

Lily adds a high-level data model on top of HBase. Originally, the model was a simple list of fields stored within records, but we added some field types making that model a whole lot more interesting. A first addition is the RECORD value type. You can now store records inside records, which is useful to store structured data in fields. For indexing purposes, you can address sub-record data as if it are linked records, using dereferencing.

Two other cool new value types are LIST and PATH, which allow for far more flexible modeling than the previous multi-value and hierarchy field properties. At the schema level, we adopted a generics style of defining value types, for instance LIST<LIST<STRING>> defines a field that will contain a list of lists of strings. Finally, we also added a BYTEARRAY value type for raw data storage.

Conditional updates

If you're familiar with multi-user environments you sure know about the problem of concurrent updates. For these situations, Lily now provides a lock-free, optimistic concurrency control feature we call conditional updates. The update and delete methods allow one to add a list of mutation conditions that need to be satisfied before the the update or delete will be applied.

For concurrency control, you can require that the value of a field needs to be the same as when the record was read before the update.

Test framework

Lily 1.1 ships with a toolchest for Java developers that want to run unit tests against an HBase/Lily application stack. The stack can be launched embedded or externally, with simple scripts straight out of the Lily distribution. You can also request a 'state reset', clearing a single node instance of Lily for subsequent test runs. Yes, you can now run Lily, HBase, Zookeeper, HDFS, Map/Reduce and Solr in a single VM, with a single command.

Server-side plugins

For the fearless Lily repository hacker, we offer two hooks to expand functionality of the Lily server process. There's decorators which can intercept any CRUD operation for pre- or post-execution of side-effect operations (like modifying a field value before actually committing it).

Rowlog sharding

The global rowlog queue is now distributed across a pre-split table, with inserts and deletes going to several region servers. This will lead to superior performance on write-or update-heavy multi-node cluster setups.

API improvements

Our first customers (*waves to our French friends*) found our API to be a tad too verbose and suggested a Builder pattern approach. We listened and unveil a totally new (but optional) method-chaining Builder API for the Java API users.

Whirr-based cluster installer

For Lily Enterprise customers, we rewrote our cluster installer using Apache Whirr, being one of the first serious adopters of this exciting Cloud- and cluster management tool. Using this, installing Lily on many nodes becomes a breeze. Here's a short movie showing off the new installer.

Performance

Thanks to better parallelization, Lily has become considerably faster. You can now comfortably throw more clients at one Lily cluster and see combined throughput scale fast.

All in all, Lily 1.1 was a great release to prepare. We hope you have as much fun using Lily 1.1 as we had building it. Check it out here: www.lilyproject.org.

Innovation and collaboration on Big Data recommendations

2011-10-31T15:08:11.000Z

(press release)

Outerthought and Oxynade, two software companies from the Belgian Ghent area, are collaborating in the context of the TWIRL project, a European research project on open platforms able to process, mine, interlink and fuse data originating from real world applications and on-line data sources. The research groups IBBT/Ghent University (IBCN) and Sirris join their effort.

The research project sits at the root of new developments in Lily, the Big Data content repository from Outerthought, that will combine storage, indexing search with profile management, analytics and content recommendations in its next versions. The Lily platform is based on Apache Hadoop and HBase and is the world's first NOSQL/Big Data content repository.

Hadoop and HBase are being used in large organizations such as TomTom, Netlog, Twitter, Facebook and Yahoo!, but the technology is still regarded as complex and finicky to use. Lily makes this technology easy to adopt and use for every organization with large-scale data management needs.

"Research without field validation has no purpose", says Steven Noels, Outerthought CEO, "so we are very happy to be able to collaborate with Oxynade on the TWIRL project. They provide us with a lot of practical experience and knowledge about event recommendations, and they possess an immense set of practical trial data. Also, due to their international growth ambitions, they are in need of a data platform that can scale widely, which means Lily is a great fit for them."

The TWIRL project also got the EU ITEA label, which guarantees high-quality industry research.

Outerthought and Oxynade receive a Flemish research grant for their collaboration to the amount of 550.000 EUR for a 100 manmonth project. The tangible collaboration plans between the industry group and Sirris and IBBT as research partners had a positive impact on the grant approval.

Contact:

Steven Noels - Outerthought - +32 9 338 82 20
Niko Nelissen - Oxynade - +32 9 233 40 09

Cloudera partnership and Accenture Innovation Awards

2011-08-16T09:39:46.000Z

As mentioned in our previous newsletter, this Summer is Hot in Lily-land. We have some more exciting announcements to make:

Cloudera Connect^TM Partnership

We're very happy and proud to be included in the initial list of Cloudera Connect^TM Partners, announced just a couple of days ago. Lily is also being prominently featured in a Solution Spotlight, explaining the value-add of Lily on top of the Cloudera Distribution of Hadoop. We're excited to work together with Cloudera on making Big Data and HBase easier to use for enterprise developers in retail, media and news.

If you want to learn more about how Lily fits well with Cloudera, a solution brief is available as well (application/pdf, 160.9 kB, info).

Accenture Innovation Awards

Also, we're participating with the Accenture Innovation Awards, a yearly contest for novel business and technology ideas.

Lily Summer season

2011-07-22T06:48:18.000Z

This Summer, we’re hard at work on some exciting new features of Lily, while helping some customers to kick off their new Lily-based projects. It’s great to see our product being used in practice, a truly educational experience for ourselves as well. Let’s have a look at the new stuff.

1. Lily Test Framework

We’ve always said Lily is about developer conveniencing: making the hard bits easy. At the core of Lily, we’re solving the really hard problem of consistent index maintenance in between HBase and Solr. On the outside however, we also wanted to make it easy for enterprise devs to Get Things Done with Lily - as if it is just another database component they already know.

That means it should be easy to call Lily from inside unit tests, and that you don’t need half-a-cluster per developer to just use Lily and its API to program against. We wanted something that allows you to launch Lily and all of its constituents with a single call, embedable, “laptop-class”-compatible (i.e. not relying on virtualization tools to launch pre-made Linux images with Lily, just to run Lily on a developer workstation).

So we went off and created a Lily test framework, that launches Lily, HBase, Hadoop, Solr and friends quickly, easily and efficiently, with sane single-node preconfigurations and some hooks to cycle data initialization as well. It’s available from trunk since a couple of days. Some people will be really happy to learn that this Lily test framework supports Windows (except currently for map/reduce operations) as well. The Lily Test Framework allows you to call Lily from inside unit tests, either standalone or embedded. The standalone mode can also be used to quickly set up a single-node, single-process instance of Lily to play around with.

Adding onto that, there’s a Maven goal available now to quickly set up a fresh, empty Lily project.

All this might sound pretty mundane, and to some extend it is, but it's one of the babysteps we see necessary to emphasize that we're really serious about bringing Lily to the enterprise, not requiring PhD-level hackers to get started with Big Data.

2. Lily cluster installs

For Lily Enterprise customers, we’re currently preparing a Whirr-based cluster installation that supports both cloud-based installs (e.g. Amazon EC2) and "bring your own nodes" installations - on your own cluster. Whirr is an Apache project under incubation for installing, setting up and running cloud services in a platform-neutral way. We’ve been happily contributing a slew of patches and issue reports (221, 338, 339, 240 and 342) while working on Lily support for Whirr.

Using Whirr and our Lily Enterprise installation packages, setting up complex multi-node installs of Lily will be a breeze.

3. The Lily Adoption Program

As mentioned during our well-attended first Lily webinar, we've set up a Lily adoption roadmap helping enterprises to easily discover, explore, adopt, deploy and support Lily inside their organization. It's a multi-step, facilitated process with workshops, proof-of-concept support, training and implementation assistance based on our many years of project experience. The 2-day workshop is still available at a discounted rate (-20%) until end of September, so don't hesitate to get in touch with us.

It’s a long (temperature-wise not so hot unfortunately) Summer over here in Lily-land, and we’re making great progress whipping up some exciting new features. Stay tuned for more!

AppsForGhent in review.

2011-05-16T07:19:09.000Z

The AppsForGhent hacking event and contest last Saturday was a success on the many scales you can measure. For me the most important one in any case was the participation both in quality and quantity of the various teams, kuddos to all participating and giving it their best shot.

Also nice to see is that more and more stuff (code and ideas) is being shared out there. Here is our presentation by the way:

20110514 appsforghent

View more presentations from Outerthought.

I'm looking forward to seeing the teams (and the city) showing a prolonged commitment in the 2nd challenge of the event: creating a working application during the next two weeks and will be keeping an eye on twitter @AppsForGhent to track progress.

Lily Hacking @AppsForGhent

2011-05-14T08:19:29.000Z

Today a genuine 'hack the goverment (of Ghent)' event is taking place in the offices of ibbt. We got invited to sponsor the event and decided to throw in some more then just the typical goodies too share, and what better to way too show off some relevance by just delivering some hacker infrastructure?

So we set-up a full enterprise seven node cluster for the event on virtual servers in the ibbt demo network (not public though) and preloaded it with the KML files we received from the city.
Being about geospatial data, the nice side-effect is some first experience with the spatial queries in SOLR

The setup is rather straightforward, and involved:

Installing the cluster through the lily-provisioning (see enterpise documentation)
Parsing the placemarks from the KML files and producing lily-import-json through a naive convertor

Creating a simple lily schema describing those records

{
  namespaces: {    "be.appsforghent.kml": "k"  },
  fieldTypes: [
    {      name: "k$name",             valueType: { primitive: "STRING" },      scope: "versioned"    },
    {      name: "k$kind",             valueType: { primitive: "STRING" },      scope: "versioned"    },
    {      name: "k$description",      valueType: { primitive: "STRING" },      scope: "versioned"    },
    {      name: "k$coordinate",       valueType: { primitive: "STRING" },      scope: "versioned"    },
    {      name: "k$address",          valueType: { primitive: "STRING" },      scope: "versioned"    }
  ],
  recordTypes: [
    {
      name: "k$Placemark",
      fields: [
        {name: "k$name"       , mandatory: true  },
        {name: "k$kind"       , mandatory: true  },
        {name: "k$description", mandatory: false },
        {name: "k$coordinate" , mandatory: false },
        {name: "k$address"    , mandatory: false }
      ]
    }
  ]
}

Create the needed solr schema. (listing only relevant parts)

<schema name="example" version="1.3">
 <types>
...
    <!-- A specialized field for geospatial search. If indexed, this fieldType must not be multivalued. -->
    <fieldType name="location" class="solr.LatLonType" subFieldType="double"/>
...
 </types>
 <fields>
   <!-- Fields which are required by Lily -->
   <field name="lily.key" type="string" indexed="true" stored="true" required="true"/>
   <field name="lily.id" type="string" indexed="true" stored="true" required="true"/>
   <field name="lily.vtagId" type="string" indexed="true" stored="true"/>
   <field name="lily.vtag" type="string" indexed="true" stored="true"/>
   <field name="lily.version" type="long" indexed="true" stored="true"/>
 <!-- Your own fields -->
   <field name="name" type="text" indexed="true" stored="true" required="false"/>
   <field name="kind" type="text" indexed="true" stored="true" required="false"/>
   <field name="coord" type="location" indexed="true" stored="true"/>
   <field name="addr" type="text" indexed="true" stored="true" required="false"/>
 </fields>
</schema>

Creating an index-configuration and add that to lily

<?xml version="1.0"?>
<indexer xmlns:k="be.appsforghent.kml">
  <records>
    <record matchNamespace="k" matchName="Placemark" matchVariant="*" vtags="last"/>
  </records>
  <fields>
    <field name="name"  value="k:name"/>
    <field name="kind"  value="k:kind"/>
    <field name="coord" value="k:coordinate"/>
    <field name="addr"  value="k:address"/>
  </fields>
</indexer>

Upload the records
Enjoy the SOLR geo filtering results

Klik om te vergroten

Lily 1.0: Smart Data, at Scale, made Easy

2011-04-23T21:38:23.000Z

We’re really proud to release the first official major release of Lily - our flagship repository for scalable data and content management, after 18 months of intense engineering work. Along this event, we are also launching our commercial Lily services, and announcing some early-stage customers and partners. We’re thrilled being first to launch the first open source, general-purpose, highly-scalable yet flexible data repository based on NOSQL/BigData technology: read all about it below.

What

Lily is Smart Data, at Scale, made Easy. Lily is a data and content repository made for the Age of Data: it allows you to store and manage vast amounts of data, and in the future will allow you to monetize user interactions by tracking and analyzing audience data.

Lily makes Big Data easy with a high-level, developer-friendly data model with rich types, versioning and schema management. Lily offers simple Java and REST APIs for creating, reading and managing data. Its flexible indexing mechanism supports interactive and batch-oriented index maintenance.

Lily is the foundation for any large-scale data-centric application: social media, e-commerce, large content management applications, product catalogs, archiving, media asset management: any data-centric application with an ambition to scale beyond a single-server setup. Don't focus on scale and infrastructure: we'll do that for you while you can focus on real differentiators.

Lily is dead serious about Scale. The Lily repository has been tested to scale beyond any common content repository technology out there, due to its inherently distributed architecture, providing economically affordable, robust, and high-performing data management services for any kind of enterprise application.

For whom

Lily puts BigData technology within reach of enterprise and corporate developers, wrapping high-care leading-edge technology in a developer-and administrator-friendly package. Lily offers the flexibility and scalability of Apache HBase, the de-facto leading Google BigTable implementation, and the sophistication and robustness of Apache SOLR, the market leader of open source enterprise and internet search. Lily sits on the shoulders of these Big Data revolution leaders, but provides additional ease of use strongly needed for corporate adoption.

Lily Enterprise

Lily Enterprise makes easy easier. Start using Lily with a turn-key technology adoption process featuring training, proof-of-concept consulting, mentoring, enterprise support and a host of enterprise add-on tools, such as cluster deployment tools, administrative user interface, and easy installation packages. Lily Enterprise is a subscription-based software and services offer available worldwide.

What’s next

Beyond scalable data management, Lily will evolve into an all-in-one database/data warehouse/business intelligence platform, providing its users with insights on all aspects of data usage and audience data. Lily 1.5 will have built-in attention data collection and analytics. Lily 2.0 will do real-time content recommendations based on attention data and user-provided domain knowledge. A Lily SaaS offer is also planned.

Early adopters and partners

Join the ranks of ADEO, the world's 4th DIY retail group that adopted Lily as the foundation of its next-generation e-commerce platform. Explore Lily together with Bonnier, a leading multinational publishing group. Lily runs great on the Cloudera Distribution for Hadoop (CDH) and even better on Cloudera Enterprise. Lily is certified CDH-compatible. Early implementation partners are Cronos Group (BE), SFEIR (F) and Infosys.

Thanks

Lily builds further upon the best data and search technology out there: Apache HBase and SOLR. HBase is in use at some of the largest data properties out there: Facebook, StumbleUpon and Yahoo! SOLR is rapidly replacing all proprietary enterprise search solutions and is one of the most popular open source projects at the Apache Software Foundation. We're thankful for the developer communities working hard on these projects, and strive hard to contribute back where possible. We're also appreciative of the commercial service suppliers backing these projects: Lucid Imagination and Cloudera.

Where

Everything Lily can be found at www.lilyproject.org. Enjoy!

Visualizing HBase Flushes And Compactions

2011-02-21T10:01:10.000Z

I was looking into more detail at how HBase compactions work, and given my experience collecting metrics for Lily, and also inspired by this blog post on Lucene, I thought it would be nice to do some tests and make some graphs of the HBase flush and compact process.

Background

When something is written to HBase, it is first written to an in-memory store (memstore), once this memstore reaches a certain size, it is flushed to disk into a store file (everything is also written immediately to a log file for durability). The store files created on disk are immutable. Sometimes the store files are merged together, this is done by a process called compaction.

This buffer-flush-merge strategy is a common pattern, for example also used by Lucene, and is described in a paper called the Log-Structured Merge-Tree (never read it though). In contrast to Lucene though, were one usually has only one IndexWriter, HBase typically has hundreds of regions open.

There are two kinds of compactions:

the minor compactions: these are triggered each time a memstore is flushed, and will merge some of the store files, determined by an algorithm described below.
the major compactions: these run about every 24 hours (after the currently oldest store file was written), and merge together all store files into one. The 24 hours is adjusted with a random margin of up to 20% to avoid many major compactions happening at the same time. Major compactions can also be triggered manually, via the API or the shell.

There is another difference between minor and major compactions: major compactions process delete markers, max versions, etc, while minor compactions don't. This is because delete markers might also affect data in the non-merged files, so it is only possible to do this when merging all files.

If, after a compaction, a newly written store file is greater than a certain size (default 256 MB, see property hbase.hregion.max.filesize), the region will be split into two new regions.

The basic compaction process

The illustration below shows a simple scenario: in a loop we perform a put of a row with a single 1k value. The table has one column family and only one region. The flush size size of the memstore has been set to 100MB (hbase.hregion.memstore.flush.size), the maximum file size (hbase.hregion.max.filesize) has been set high enough to avoid splits from happening. All tests in this blog have been done on a single node (my laptop).

The main goal here is to illustrate how the minor compaction process decides when and what to merge.

We see that the memstore grows gradually until it reaches its flush size of 100MB, when this happens it is flushed to disk in a new store file. This process repeats itself until we have 3 store files. At this point, a compaction is performed which merges the files together into one file.

The second compaction is only performed after 4 store files are written. How does HBase decide this?

The algorithm is basically as follows:

Run over the set of all store files, from oldest to youngest
If there are more than 3 (hbase.hstore.compactionThreshold) store files left and the current store file is 20% larger then the sum of all younger store files, and it is larger than the memstore flush size, then we go on to the next, younger, store file and repeat step 2.
Once one of the conditions in step two is not valid anymore, the store files from the current one to the youngest one are the ones that will be merged together. If there are less than the compactionThreshold, no merge will be performed. There is also a limit which prevents more than 10 (hbase.hstore.compaction.max) store files to be merged in one compaction.

If you need more detail, have a look at the source of the Store.compact() method.

Now, with this knowledge, we can see why the second compaction does not run after flush 5: the oldest file was about ~240MB, the two younger ones together ~160MB, 240 > 160 * 1.2

When the second compaction runs after flush 6, it does merge all the files together, instead of just the 3 youngest, since ~240 > 3*80*1.2 is not true. After flush 9, we have ~480 > 3*80*1.2, so the oldest store file is not included in the merge.

With the default configuration of HBase, if a store file would be 480MB, the region will be split. I have on purpose augmented the limit to illustrate what happens if there are more store files, though I could also have lowered the memstore flush size.

If you wonder how the store file merging behaves over longer time, below is the result of a longer run with 157 flushes of 10 MB. Click on the image below to see it bigger (10000px by 800px).

The largest number of store files at any time is 7 (see number 143 in the chart), though the oldest (biggest) one is only merged occasionally. It is a good thing that the number of store files stays small, because when you read a row from HBase it needs to check all the store files and the memstore (this can be optimized through bloomfilters or by specifying timestamp ranges).

The number 7 might make you think of the property hbase.hstore.blockingStoreFiles, which has a default value of 7, but I augmented it for this test to be sure it did not have any influence.

Compactions and deletes

When you perform a delete in HBase, nothing gets deleted immediately, rather a delete marker (a.k.a. thombstone) is written. This is because HBase does not modify files once they are written.

The deletes are processed during the major compaction process, at which point the data they hide and the delete marker itself will not be present in the merged file.

Lets illustrate this with an extreme scenario. The following graphs are from a scenario where, in a loop, we put a row and then immediately delete the row. So conceptually, if we look at the table during the test, it will contain at most one row. Nevertheless, the store files will only grow.

What we see is:

the memstore grows and flushes three times, three store files are created
after the third store file is written, a compaction is performed
the result of the compaction is a new store file of size 0.

This is a surprising result: only major compactions process deletes, and major compactions supposedly only run every 24 hours.

What happens is that if the set of store files selected for compaction is the set of all store files, then HBase decides to do a major compaction. In fact, it should, otherwise the 24H check, which looks at the time of the oldest store file, would mistakenly assume that this one file was the result of a major compaction.

Besides explicit deletes, there are other cases where store files will shrink after a major compaction: cells that are removed because there are more than max-versions versions of it, or when using the TTL feature.

Multiple column families

Each column family has its own memstore and its own set of store files. But all the column families within a table stick together. They will all flush their memstore at the same time (this might change). They are also splitted in the same regions, so when one of the column families grows to large, the other ones, how small they might be, will also split.

The illustration below shows a scenario where a table with two column families is used. In family 2, we only do a put for every 10 puts we do in family 1. So family 1 will grow much faster than family 2 (family 2 is sparse family).

The memstore and storefile count graphs show the data for both families counted together. Note how at each memstore flush, the number of store files increases with two.

You can also see how a second compaction is performed for family 2 but not for family 1. This is because all store files in family 2 are smaller than the memstore flush size of 100 MB.

Multiple regions

Now lets do a scenario with a table that has multiple regions.

For this test, a table was created with 10 initials regions, divided so that random UUIDs would be evenly spread over these regions. This time the memstore flush size was set to 30 MB.

As the puts are performed, the memstores of each region should fill up evenly, and so we expect them to all reach the flush size at about the same time, and thus to flush at about the same time. However, in the HBase region server there is only one thread which performs flushes, so they happen in sequence. This means that the memstores of some regions will keep growing above their limit. The property hbase.hregion.memstore.block.multiplier controls how many times a memstore can grow above its limit before HBase blocks updates. The default is 2, in our example here this means the memstore would be allowed to grow to 2*30=60MB maximum.

Similarly, there is one thread for doing the compactions and splits, so these are performed in sequence too. Still, it is a period where resources are used which would otherwise have been usable to serve client requests. If you would run a similar scenario a cluster, the different nodes could still run there compactions in parallel. In such a scenario, were all regions are filled very evenly and decide to do compactions at about the same time, you can see some negative spikes when observing the hbaseRequestCount metric. In the chart here you can notice that memstore grows a bit less steep during the compactions.

Memstore upperLimit

All the above illustrations were quite synthetic. On a real system, there will often be much more regions, and more than one table, and there will often not be enough memory to let memory stores grow to their configured flush size.

HBase uses by default 40% of the heap (see property hbase.regionserver.global.memstore.upperLimit) for all memstores of all regions of all column families of all tables. If this limit is reached, it starts flushing some memstores until the memory used by memstores is below at least 35% of heap (lowerLimit property).

Let's simulate this scenario by again using 10 regions, but now with an upper limit of 100 MB. The regionserver in this test has only 1GB of memory, so 400MB available for the memstores. Given that we have 10 memstores (1 table, 1 column family, 10 regions), and that we will fill them evenly, we expect the first flushes to happen when the memstores reach 400/10=40MB (instead of going on to their 100MB flush size).

And, hurary huray, the theory is confirmed by the plot below.

Some final notes:

during flushes & compactions, HBase keeps processing put and get requests, always giving a consistent view of the data
certain properties, like the flush size, can be configured on the level of the table

The charts were made with gnuplot, afterwards annotated with Inkscape. The metrics were collected through HBaseAdmin's ClusterStatus and JMX. The store file sizes were read directly from HDFS. I set hbase.regionserver.msginterval to 1000 to get fine-grained data from ClusterStatus. Lily's clientmetrics package was used to collect & output the metrics, and to generate the necessary input files for gnuplot.

Lily 0.3 - the Valentine release!

2011-02-14T15:43:17.000Z

What better fitting way to celebrate Valentine than with a nice flower? Here's a new and beautiful Lily for you!

This release is the result of 3 months of hard work since Lily 0.2 last October. Our focus was stabilization, performance and robustness, providing a platform we can continue building upon. More than 50 tickets were resolved during this development sprint, and we're slowly readying ourselves for the 1.0 release. Lily 0.3 brings many gradual improvements over Lily 0.2. It has various important performance improvements, a more solid implementation of the blob fields, automatic retry of operations that fail due to IO exceptions (between Lily client and Lily server), and other miscellaneous improvements.

Lily 0.3 also is the result of a lengthy proof-of-concept project for our first paying Lily customer, who stressed the importance of performance, robustness and flexibility. The resulting, large improvements on the tester tool are all part of the Lily source tree now, for everybody's benefit.

In the mean time, we're plugging Lily everywhere, and with great success. We're also slowly disclosing the Lily future roadmap - Smart Data at Scale - where Lily will be the first (and only) data repository tightly integrating scalable storage, robust indexing, flexible searching, real-time usage analytics and built-in Data Insights, all from a single open source product to be installed on your infrastructure or hosted in the cloud.

Everything Lily can be found at www.lilyproject.org. We're also sharing the details of our commercial software subscription service with select prospects, let us know if you're interested!

Here's a concise list of improvements since Lily 0.2:

Repository

Performance / space improvements

Shorter column key encoding (field id's)
Reduction of number of column families used
Avoid duplicate values in the table: make use of sparseness of the table
Drop the use of HBase rowlocks, which do not survive region splits/moves.
Use byte[] as keys in RecordType FieldType cache

Added a new method createOrUpdate which creates or updates a record depending on whether it already exists. This new method has the advantage over the create method that it can be retried in case of IO exceptions, i.e. it is idempotent, similar to PUT in HTTP/REST.
Allow updating versioned-mutable fields without specifying the record type.
Throw a RecordLockedException instead of generic exception when a record is locked, this allows Lily clients to retry the operation in that case.

Clear historical data when deleting a record and remove any referenced blobs.
The link index stores record IDs and field IDs as bytes instead of strings.
The record ID string representation was changed to use comma instead of semicolon to separate variant properties, since the use of semicolons was problematic in the JAX-RS based REST interface implementation.

Upgrade to Apache HBase 0.90

Blobs

Rework blobstore functionality

Blobs can only be accessed through the record they are used in, not directly by using their blob key. This is to allow for future record-level access control.
Introduce a Repository.getBlob() method, which returns a BlobAccess object, which provides access to the blob meta data (Blob object) and the blob input stream. This avoids the need to read the record in case you need the blob metadata.
Uploaded blobs which are never used in a record are cleaned up.

The HDFS-stored blobs are stored in a hierarchical structure.

RowLog improvements

Performance improvements

the RowLog processor uses a Zookeeper based notification system instead of Netty based.
Optimize queue scanning: avoid scanning over deleted rows in the table, fix too-frequent scanning, fix endless scanning loop on startup in case of no repository activity.
The RowLog processor only processes messages of a minimal age (avoid conflicts with direct processing of wal messages).

Extended RowLogConfigurationManager to add/update rowlog configuration information.
Avoid and remove stale messages in the queue.
Allow the rowlog to either use row-level locks (wal use case) or executionstate-level locks per subscription (mq use case) when processing messages.
Added a WAL processor which handles open WAL messages.

REST interface

Adapted blob-support to new blobstore functionality. Content-Length header is now set when downloading blobs. Multi-value or hierarchical blobs are now accessible.
Support updating versioned-mutable fields.
Fixed various smaller bugs reported by users.

HBase index library

Allow to add/remove multiple entries in one call.
Performance

Fixed important performance issue whereby row scanning always ran to the end of the index table.
Enable scan caching.
Added a performance testing tool.

Indexer

Upgrade to Tika 0.8
Performance

Avoid FieldNotFoundException when evaluating field values

the SOLR request-writer and response-parser implementation configurable. This allows to use the XML format instead of the javabin format.

LilyClient

Automatically retry operations on IOExceptions, this allows operations to survive node failures.
Automatic balancing over all Lily nodes. Each method called on the Repository object will automatically be performed on an arbitrarily selected Lily node.
Avro: switch from HTTP to Netty transport. For this, upgraded to an Avro 1.5 snapshot with patch AVRO-747.

Tester tool

Allows to configure test scenarios and indexer and solr configuration.
Has extended logging, metrics and metrics plotting (gnuplot integration) capabilities allowing for performance evaluations.
Introduces general performance testing library.

Lily server process

Abilitly to create tables with multiple initial regions at first cluster startup (record table, linkindex, blobincubator, ...). Also allows to set the max file size and the memstore flush size.
The initial Lily startup can now be performed on multiple nodes concurrently, previously this failed because the table creation code did not handle failures in case of concurrent table creation.
Configuration files changed so that they allow for inheritance (= fallback from one conf dir to another, to the built-in conf). Include default configuration in Kauri-module jars. All this will help in maintaining Lily configuration across Lily versions.

We hope you'll enjoy this new Lily as much as we did making it. Let us know how we're doing!

What's in store for 2011?

2010-12-22T17:20:27.000Z

As we're gently spinning down for Christmas and Year's End parties, it's time to take a quick peek at what we have in store for next year.

But first some fun off-topic news: we've been hosting a benefit for the Red Cross / Studio Brussel action to support orphans of AIDS victims this week, basically doing some live DJing and music streaming during lunch. It was called "lunchmusic4life" and the result can be seen at our UStream show page. We aren't done yet, if you have some time tomorrow tune in between 12AM and 1PM Belgian Time.

We're planning to release a new beta of Lily somewhere end of January introducing a number of performance enhancements we're currently researching with the nice (hardware) assistance of the Sizing Servers lab of HOWEST. I'll be giving a NOSQL intro presentation during YaJUG's inaugural NOSQL User Group meeting that month as well.

We're still on target to have our 'final' 1.0 release end of March, which means we're planning for 2.0 already. The idea for 2.0 is simple: at first, you'll use Lily to 'just' store and index your data, but next, with 2.0, we'll give you some tools to play with your data, basically the plumbing to provide Data Insights around your information collection.

We'll do this together with partners, but on the low-level technical side of things, we'll provide a processing framework that will allow you to easily have recommendations, patterns, semantic augmentation and other advanced (meta)data processing exposed as part of the Lily data model. During the process, we'll also take 'audience data', basically usage data into account, as these will provide data insights as well.

In the mean time, we're working on some proof-of-concept Lily projects for customers, putting Lily through its paces and see how well it performs. Needless to say, we're thankful and thrilled to have these early adopters on board.

On the Daisy side, projects are doing well, we just signed up a new customer who's going to manage medico-operational care procedures for a Dutch oncology center.

We'll be merrily continuing our open source product activities in 2011, with some other exciting announcements ahead of us. The entire Outerthought team wishes you a great Christmas season, and see you on the other side!

@ Devoxx 2010

2010-11-25T15:17:44.000Z

Last week Antwerp (devoxx) was the place to be for sensing the pulse of the Java community. The annual great European Java-developers-feast was again a joyful get-together of Java-enthusiasts, great speakers, and other smart people. The formula and the venue (great speakers, even greater screens) has proven itself over time, but the organizers keep having an eye for trying out new and innovative ways (sure enough tweets, but loads of whiteboards as well!) to foster the thought exchange and hackerish tinkering.

Sure enough there is side-tracks and comments on how Oracle is translating its "ownership" into "stewardship", but mostly the conference is just about the techy stuff, some showing off, and quite some well-deserved awe.

We got the floor for two sessions during the conference focussing more on the REST (kauri) side of our activities. For those who missed the slides or want to rehash them, see below.
The first slides on the "wonderful web machine" are there to invite the audience into thinking about "the web" as a single-person-invention. An enormous design and clever architecture of a vast successfully interoperating massively distributed computing system. No less! :) In my mind mankind really has surpassed himself on this one. Getting carried away in my enthusiasm I even encouraged people to contemplate about this wonder at least once a week. Joking aside, "the wonder view" serves as my motivation for advocating REST as in "going with the web grain": follow the techniques for making your web application an integral part of this link-in wonder, rather then a stand-alone island in this digital universe.

Overall it was great to see how much air-time (too much according to some) this conference has been giving to REST. Mostly thanks to the great presentations of Paul Sandoz on Jax-RS and the plans for 2.0. Other important 'trend' to watch was the obvious attention to the BigData/Cloud/NoSQL track. Credits to the fine line-up of speakers there of course.

My two biggest takeaways of the week finally: "Java Modules" and "CDI" (and how they relate to the future of kauri) deserve their own blogposts. (promise!)

Here are the slides from the talks we gave:

"Lab" Session (3h) on restfull programming in Java

Devoxx 2010 | LAB : ReST in Java

View more presentations from Outerthought.

"Tools in Action" Session (30') on Kauri and Lily as the groundwork for web-scale applications

Devoxx 2010 | Tools In Action : Kauri and Lily

View more presentations from Outerthought.

Rowlog : A message queue system on top of HBase

2010-11-24T08:14:45.000Z

Rowlog : A message queue system on top of HBase

The need for a MQ

In our Lily content repository project (cfr www.lilyproject.org), each time we update (create, update, delete) a record we want to update a SOLR-based index. To accomplish this we need a message queue system (MQ) that delivers messages about updates on records to an indexer component which updates the SOLR index.
The MQ needs to comply to a few requirements:

The context of the messages are events that happen on Lily records
Messages can be delivered in an asynchronous way to the indexing component
Messages about a specific record should be delivered in-order
No two messages of a same record can be consumed by the indexer component at the same time
A message consumer should be able to retrieve all oustanding messages related to a record
There can be no message loss

Why build our own?

MQ systems already exist in many flavors and implementations like for instance RabbitMQ, ActiveMQ, etc. Still we decided to build our own system on top of HBase.
First of all we want to keep the amount of different components and storage technologies (e.g. MySQL) co-existing in the Lily space as low as possible since each new technology adds an extra level of complexity with respect to configuration, administration, deployment etc.
Secondly we want all components of Lily to behave the same with respect to scalability (partitioning), failover, reliablity (replication). Since we were already using HBase that has our desired scalabiliy properties we decided to use the same technology to build a MQ system.

A MQ on HBase named 'Rowlog'

Until now this blog has been talking about updates and events that apply to Lily records. But in fact we have built an MQ system that is usable outside the context of Lily. The context of the messages are events that happen on a HBase row. That is also where the name "Rowlog" comes from.

Subscriptions

Each client interested in processing or consuming messages should register a subscription on the rowlog. In case of Lily this is the indexer updating the SOLR index, but this could just as well be an e-mail notification client or anything else.
A message will then be offered to each registered subscription for consumption by one of its listeners.

Message Queue Table

Each message is stored in a message table (which is stored in a HBase table). The key of the message contains a subscription id, a timestamp of when the event happened and the rowkey of the row the message is about.
A Rowlog processor will regularly scan the message table and process the messages. The messages are sorted based on their timestamp. They will therefore be processed in the order the events happened, even across different rows.

Row-local Queue

Next to the message queue table, there is also the concept of a row-local queue. In a row-local queue we keep information about events on a row in messages that are stored on the same row where the message is about, but in a column family separate from the actual data.
Doing this gives us a few advantages:

Adding a message can happen in the same atomic put operation as updating the data itself (thereby avoiding the need for transactions).
All outstanding messages of a row can easily be retrieved. In the message queue table the messages of one row are interleaved with the messages of other rows.
All message information is available next to the data. If the message queue table would be lost it can be rebuild based on the row-local queue. (This is more a hypothetical advantage since we don't expect to be loosing the message queue table.)

The information stored in the row-local queue is an execution state for each subscription (executed or not) and a payload (extra information about the event that happened). By storing this information here, the messages stored in the message queue table can be kept very light-weight.

Write-Ahead Log (WAL)

The message queue is not the only reason why we came up with the rowlog solution.
At the same time we also needed a solution to be able to execute actions as a consequence of an update on a record, i.e. 'secondary actions'. The requirements for these actions were :

Guaranteed, eventual execution (cfr no message loss)
Preferably synchronous execution (can't be fulfilled in case of failures)ability to resume in case of failures (e.g. a node crash after an update but before a secondary action got executed)
Secondary actions should be able to see the state of the record corresponding to the update. Therefore, secondary actions should be executed before the next update on the record happens.
Ordered execution of multiple secondary actions following the same update

The above requirements don't completely apply to what is usually understood under a write-ahead log. For instance, we don't need are transactions across multiple updates on multiple rows. But we stick with the name.

Rowlog as a solution for the WAL

While designing a solution for the WAL we noticed that we ended up with something that was very similar to our MQ solution and that in fact we could distill the same solution for both use cases : the "Rowlog".

For a more technical explanation of the Rowlog, it's architecture and how to use it, take a look here.

Something for the weekend: Lily 0.2 is OUT !

2010-10-29T14:26:21.000Z

Three months after the highly anticipated proof of architecture release, we're living up to our own promises, and are releasing Lily 'CR' 0.2 today - a fully-distributed, highly scalable and highly available content repository, marrying best-of-breed database and search technology into a powerful, productive and easy-to-use solution for contemporary internet-scale content applications.

For whom

You're building content applications (content management, archiving, asset management, DMS, WCMS, portals, ...) that scale well, either as a product, a project or in the cloud. You need a trustworthy underlying content repository that provides a flexible and easy-to-use content model you can adapt to your own requirements. You have a keen interest in non-relational/HBase technology but need a higher-level API, and scalable indexing and search as well.

Foundations

Lily builds further upon Apache HBase and Apache SOLR. HBase is a faithful implementation of the Google BigTable database, and provides infinite elastic scaling and high-performance access to huge amounts of data. SOLR is the server version of Lucene, the industry-standard search library. Lily joins HBase and SOLR in a single, solidly packaged content repository product, with automated sharding (making use of multiple hardware nodes to provide scaling of volume and performance), high availability and automated index maintenance. Lily adds a sophisticated, yet flexible and surprisingly practical content schema on top of this, providing the structuredness of more classic databases, versioning, secondary indexing, queuing: all the stuff developers care for when fixing real-world problems. All this is made accessible through a powerful Java/AVRO API and a platform-neutral REST interface.

Key features of this release

Fully distributed: Lily has a fully-distributed architecture making maximum use of all available hardware for scalability and availability. ZooKeeper is used for distributed process coordination, configuration and locking. Index maintenance is based on an HBase-backed RowLog mechanism allowing fast but reliable updating of SOLR indexes.
Index maintenance: Lily offers all the features and functionality of SOLR, but makes index maintenance a breeze, both for interactive as-you-go updating and MapReduce-based full index rebuilds
Multi-indexers: for high-load situations, multiple indexers can work in parallel and talk to a sharded SOLR setup
REST interface: a flexible and platform-neutral access method for all Lily operations using HTTP and JSON
Improved content model: we added URI as a base Lily type as a (small) indication of our interest in semantic technology

More importantly, we commit ourselves to API and data format layout compatibility from this release onwards. Lily offers the final API we want to support in the final release. Lily 0.2 is our contract for content application developers, upgrading to Lily final should require them to do as little code changes as possible.

From where

Download Lily from www.lilyproject.org. Open source, Apache license, no strings attached.

Enterprise support

Together with this release, we're rolling out our commercial support services (and signed up a first customer, yay!) that allows you to use Lily with peace of mind. Also, this release has been fully tested and depends on the latest Cloudera Distribution for Hadoop (CDH3 beta3) - Cloudera being the leading Hadoop enterprise services provider. We're proud to collaborate with a true market leader here.

Next up

Lily 1.0 is planned for March 2011, with an interim release candidate in January. We'll be working on performance enhancements, feature additions, and are happily - eagerly - awaiting your feedback and comments. We'll post a roadmap for Lily 0.3 and onwards by mid November.

If you want to keep track of Lily's on-going development, join the Lily discussion list or follow our company Twitter @outerthought.

Thank you

I'd like to thank Bruno and Evert for their hard work so far, the Flemish IWT government fund for their partial financial support, and all of our early Lily adopters and enthusiasts for their much valued feedback. You guys rock!

Devoxx is coming! Come and see us!

2010-10-26T14:58:48.000Z

In just under three weeks from now, Europe's largest Java-fest is happening again, with loads of cool things happening in or around Metropolis in Antwerp. We've been helping with organizing this year's edition, we'll be there, in full force, and we'd love to meet you in Belgium's second nicest city (Ghent coming first of course ;-) November 15-19. Here's what we contributed to Devoxx' ninth edition!

The NoSQL/Cloud track

It's no secret that NoSQL and Cloud technology, with a good dash of Devops tactics, is this year's Devoxx key theme. As a matter of fact, I've been asked to join the Devoxx steering group specifically for this purpose, reaching out to some great BigData speakers all around the globe, and have been plugging this 'conference-inside-the-conference' at various other events, like in the picture above from ParisJUG in September.

As you can see from my own personal agenda, Devoxx will be very busy - attending all sessions related to NoSQL, Cloud and Devops will already be a full-time task on its own. We have 3h university sessions on Cassandra, HBase, MongoDB and Hadoop, and 45' conference sessions on both these products plus Voldemort (from LinkedIn) and Mahout, and great case-studies from Twitter, Facebook and Adobe. That's BigData Adobe, not Flex/AIR. ;)

To top this all off, I'll be co-hosting a NoSQL BOF meeting Thursday night between 9 and 10PM (yes, sessions run that late during Devoxx).

RESTy stuff

On the Kauri/REST/Java side of things, we haven't been sitting around idle as well. Marc will be presenting a 'Tools In Action' talk on Kauri and Lily Tuesday afternoon, before co-hosting the REST BOF that evening. But the most important news is that Marc will be presenting a Hands-on Lab on RESTful, Kauri-based development Tuesday morning. People who've been subject to Marc's tutoring in the past know that this kind of hands-on sessions are his favorite pet peeve, so I can only strongly advise you to join that Lab: it will be great!

Be there!

Devoxx will be selling out, this week if not sooner. So if you haven't booked your passes yet, better now than sorry. See you in Antwerp!

Final countdown for Lily 0.2

2010-10-23T12:18:03.000Z

Next week, it's final countdown for Lily 0.2, which for all imaginable reasons is a pre-qualifier for 1.0 soon after that. Our primary focus for this important 0.2 release is two-fold: full distributability and developer contracts. That means we're trying to stabilize APIs and data/disk formats so that devs can safely start playing with Lily with the assurance that upgrading to 1.0-final will be as easy as possible. In the mean time, it's crunch time around Lily, tying up various loose bits, testing the robustness in case of node failures, some performance and load testing (tweaking of which will be the primary focus for 1.0), and making sure everything is well documented.

There's a small sample application showing loading and indexing of mail archives (mbox) into Lily which you'll find in the Lily source tree, and I made a little screencast of that.

Overall, Lily is in good shape for this next release, and we are really excited of the reactions we got so far. So keep your browsers ready, we're shooting for 0.2 next week on Friday, or the Monday after that (because that's how it goes, no?)

Daisy 2.4 is out, and it's worth the wait!

2010-10-19T16:02:51.000Z

This is bound to end up being the largest product release announcement we've ever written, as it's the result of both fortunate and unfortunate circumstances. First, the good news: we've been very busy in the last 15 months, working both on our new product Lily, but also adding new features and improving the already rich feature set of Daisy.

People who have been in touch with us during trade shows or events over the past year heard the story already: we're focusing Daisy onto the wonderful world of techdoc, internal collaborative document/book authoring and legal/QDoc publishing, with sophisticated document models and a feature set to match that.

The bad news is that it took us 15 months to release this new version, but I can honestly testify that it's entirely worth the wait. Daisy 2.4 is better than ever, with a new build system, new features and new developer conveniencing, and we're very proud to release it today. So without further ado, I'll pass the mic to Karel, who has been shepherding this new Daisy release for a long time now, and who will introduce you to all things new in Daisy 2.4.

Thanks for all the hard work that went into this release!

Steven.

Main features

Point in time (docs)

Go back in time and pick an arbitrary date: we'll show you the matching versions of your content and even better: We allow you to search and navigate facets in the complete corpus back into time. This is a feature you can't find elsewhere in the open source CMS market.

Replication service (docs)

Handle higher loads and fail-safe setups through replication between multiple repository instances in master-slave mode.

Simpler repository installation

Keep enjoying that next installation of Daisy.

Up to Daisy 2.3, the installation script would ask you a lot of questions without giving the possibility to change your answers without starting over. Now the installation process is driven by a configuration file: you start with a template installation file. Easy!

Inline editing (docs)

Customize the editing environment to match the presentation layout.

Your users can now edit the content in the familiar layout of your site. The tabbed paradigm is still the default, but can easily be modified into a true inline editing environment.

Single sign-on (docs) and account manager support (link)

Skip the login page! Yay! Finally!

Use Kerberos authentication (Windows or other ), or
Plug in your own authentication scheme, or
Use "Account Manager", an Online Identity project by the Mozilla Labs people - for now, only an Firefox add-on with a prototype implementation exists, but as the spec gains attention, more implementations should arise.

Updating authentication schemes (docs)

Keep your Daisy user data in sync with your own backend (e.g. LDAP or your own)

Document task improvements

Better leverage your document handling batch processes

We've added a 'retry count' to tasks. This overcomes failed tasks due to document locks or other temporary situations, and gives them possibly another run later on.
Document tasks now survive server restarts
Restarting document tasks is made much easier
Adding custom document tasks (Java) as part of your customization-project is made a lot easier.

Developer features

Because happy developers are effective developers.

Maven 2 build system

We switched from maven to Maven 2 for building Daisy.

Maven 2 plugin & sample-project archetype (docs)

We don't just use Maven 2 for building Daisy, but for projects using Daisy. The daisy-maven-plugin allows this to be done in a very efficient and professional manner.

Using the sample-project archetype, you can give your Daisy project the best start it can get, for the complete dev team, in no time!

During your project customization project we offer the flexibility to switch and test various Daisy versions (which is important for facing the cutting edge challenges of working on top of Daisy trunk, but also could be useful in longer-term projects)

Spring context component

A wrapper which lets you use the popular Spring framework for developing your own wiki extensions, rather than the less-well known/used Avalon framework that Cocoon provides.

Context doc in search page

The query search page lets you set a context document. This makes it easier to test queries using the ContextDoc() function.

Custom Lucene analyzers

Optimize the full-text indexing process for your content using a custom index analyzer and get better full-text search results by using custom query analyzers. (e.g. handling custom synonym lists)

Convenience upgrades and fixes

Important libary upgrades

FOP 1.0 (from 0.95)
JQuery 1.4.2 (from 1.2.6)
Lucene 3.0.1 (from 2.2.0)

Conditional namespaces (docs)

A Daisy repository can now manage multiple namespaces and decide in which namespace each document should go. Playing with multiple content repositories and reshuffling their content over time was never easier.

Configuration

Various changes making configuration easier and more flexible.

Version mode cookie

By storing the version mode in a cookie, guests can also benefit from the version mode and point in time features across your customizations

Document browser

This was introduced in 2.3, and got some usability fixes and general attention.

Publisher: custom link annotation

In the advanced section: publisher requests now support adding custom link-annotation metadata (available to the layout styling) based on value-expressions.

Summary

We realize there has been an exceptional long time lapse between Daisy 2.4 and the previous 2.3 release, but we are confident this exhaustive feature-list makes up for your wait.

Obviously the Point-in-Time support makes us specially proud. Again Daisy sets itself apart from the CMS-pack with a new compelling feature.

Adding this to a growing differentiator list from previous releases (remember Books, document tasks, workflow,...) makes for more persuasive argumentation to envy both end-users and project-customers. And delivering on the promises should be a breeze with the added developer support.

As always, there's our Daisy mailing list which gets your questions answered, and our professional services you can tap into if you want your project to be done with Daisy by its makers.

May all your Daisy projects be merry, and enjoy this new release!

Go and get the good stuff at www.daisycms.org.

Conference season: NoSQL and REST ahoy!

2010-10-08T21:09:45.000Z

Conference season is heating up rapidly in the next few weeks. Or rather, it seems like we're onto something with this NoSQL/HBase/SOLR/BigData thingy, to the extent that all the pieces of the puzzle are falling together quite nicely. Hurray!

Next Monday, I'll be (very briefly) presenting Lily to the Bay Area HBase MeetUp group, who has temporarily moved places to New York for the occasion of Hadoop World 2010, the (now yearly) Cloudera/Hadoop lovefest. Not only that, I'll finally get to meet all those great folks that make the HBase-clock tick, something which puts a really big smile on my face. I have a couple of more business-like meetings scheduled as well, so things will be pretty busy.

Tuesday, I have the cumbersome challenge of choosing between all those Hadoop World conference sessions and absorb as much info as possible to relay to my colleagues at Outerthought HQ. Even though it's my first stay in NYC, I doubt there will be much time for touristy things. Oh well, such is life.

The weeks thereafter, leading up to the end of October, it's Lily release crunch time again, preparing a decent set of release info materials and making sure all the right channels are properly informed. That, and continuing work on the Lily partnership model, convincing fellow techy folks about the value-add Lily can provide to their business.

Mid November, during Devoxx, it's the NoSQL bandwagon ride again, but we're topping this off with something hot off the presses: a Lab session on RESTful development with Java, combining the power of REST design patterns, Java and Kauri in a 3 hour hands-on mega-exercise for Devoxx University attendees. So if you happen to attend Devoxx during University days and want to get a proper introduction into REST and Kauri, join Marc for RESTful development with Java. People who have heard Marc presenting before, know that such training sessions are his favorite pastime, so I'm sure you'll be in for a great ride.

OUTERTHOUGHT.BLOG

Lily 1.2 is out!

Outerthought joins NGDATA

A First Exploration Of SolrCloud

SolrCloud terminology

Running ZooKeeper

Setting up Solr instance directories

Creating a Solr configuration in ZooKeeper

Starting the Solr servers

Define the cores, collections, slices, shards

Adding some documents

Querying

Starting from scratch

Managing slices & shards

Conclusion

Lily 1.1 is out: what's new?

Complex Field Types

Conditional updates

Test framework

Server-side plugins

Rowlog sharding

API improvements

Whirr-based cluster installer

Performance

Innovation and collaboration on Big Data recommendations

Cloudera partnership and Accenture Innovation Awards

Cloudera ConnectTM Partnership

Accenture Innovation Awards

Lily Summer season

AppsForGhent in review.

Lily Hacking @AppsForGhent

Lily 1.0: Smart Data, at Scale, made Easy

What

For whom

Lily Enterprise

What’s next

Early adopters and partners

Thanks

Where

Visualizing HBase Flushes And Compactions

Background

The basic compaction process

Compactions and deletes

Multiple column families

Multiple regions

Memstore upperLimit

Lily 0.3 - the Valentine release!

Repository

Upgrade to Apache HBase 0.90

Blobs

RowLog improvements

REST interface

HBase index library

Indexer

LilyClient

Tester tool

Lily server process

What's in store for 2011?

@ Devoxx 2010

Rowlog : A message queue system on top of HBase

Rowlog : A message queue system on top of HBase

The need for a MQ

Why build our own?

A MQ on HBase named 'Rowlog'

Subscriptions

Message Queue Table

Row-local Queue

Write-Ahead Log (WAL)

Rowlog as a solution for the WAL

Something for the weekend: Lily 0.2 is OUT !

For whom

Foundations

Key features of this release

From where

Enterprise support

Next up

Follow us

Thank you

Devoxx is coming! Come and see us!

The NoSQL/Cloud track

Cloudera Connect^TM Partnership