Claus Ibsen (@davsclaus) riding the Apache Camel

CamelFly - A mashup of real time flight positions

2012-02-24T10:30:00.000+01:00

Last year the university of Applied Sciences at Brandenburg Germany posted a solution to gather earthquake data.

This year they build a mashup example, CamelFly, for tracking plane positions in German speaking countries, and enriching the data with weather details, and to send mails for critical alerts etc.

A snippet of the diagram is shown below.

CamelFly - Snippet of EIP architecture diagram

They wrote a nice wrapup of the example, with architecture diagrams using EIP patterns, and provide the full source code of the example.

You can read about the example here. And they provide source code in their git repo.

Apache Apollo 1.0 Released

2012-02-03T16:34:00.000+01:00

Yeah this is a great day. Hiram announced the next generation message broker named Apache Apollo (eg ActiveMQ 6) today. Rob Davies blog about this as well and he explains why FuseSource started the work on Apollo a couple of years ago. And why we think the new Scala based, purely asynchronous threading model of Apollo will be able to handle the next decade of messaging load, with peta and zeta bytes of data.

If you have been following Hirams blog, then you may have noticed he has posted some incredible performance metris of Apollo against ActiveMQ 5.x, RabbitMQ, HornetMQ.

Correlating logs from redelivered messages made easier in Camel

2012-02-03T09:57:00.000+01:00

Apache Camel handles redelivery of messages based on two principles
- internally
- externally

The internal redelivery is essentially the Camel error handler, that allows you to retry processing a failed message. This is all handled by Camel itself.

However the situation changes when the redelivery is commenced externally, such as from a message broker like Apache ActiveMQ. Previously from a Camel point of view, when the message broker was redelivering a message, it was regarded as a new message.

This made it a bit harder for end users of Camel to figure out this was a redelivered message, and also how to correlate logs.

This has been improved in the upcoming releases of Apache Camel and Fuse Mediation Router.

Example
Given you have configured a Camel route to consume messages from ActiveMQ and use transactions as shown in the following snippet.

<route id="myRoute">
  <from uri="activemq:queue:foo"/>
  <transacted/>
  <process ref="myProcessor"/>
  <process ref="myOtherProcessor"/>
</route>

The route is kept simple for demonstrations purpose. However it indicate the route is transacted, which allows Camel to support externally redeliveries.

We have configured the Camel activemq component to use JMS transacted acknowledgment (transacted=true)

</bean>

Now suppose Camel consumes a message from the foo queue. And processing the message fails upon the first two attempts, and on the 3rd attempt the message is processed successfully.

Camel will by default log at WARN level when a transaction is rolled back. What has been improved is the details in this log, which now includes information whether the message was externally redelivered, as well both the message id and exchange id. This allows you to correlate the logs. More about this a bit later. Notice in the 1st attempt, we have redelivered(false), and in the 2nd attempt its redelivered(true). Also notice that the MessageID is the same, where as ExchangeID is unique.

1st attempt

Transaction rollback (0x3e4ac866) redelivered(false) for (MessageId: ID:davsclaus.lan-55814-1328257420734-2:2:1:1:1 on ExchangeId: ID-davsclaus-lan-55809-1328257419533-0-3) caught: java.lang.IllegalArgumentException: Forced exception

2nd attempt

Transaction rollback (0x3e4ac866) redelivered(true) for (MessageId: ID:davsclaus.lan-55814-1328257420734-2:2:1:1:1 on ExchangeId: ID-davsclaus-lan-55809-1328257419533-0-4) caught: java.lang.IllegalArgumentException: Forced exception

3rd attempt

Success, no WARN logging.

Configuring logging details

When a transaction is rolled back, Camel will not log the stracktrace, so its less noisy. But you can configure the logging details from the Camel transaction error handler as shown below. If you want to see the failed exception and stack trace, then you can set the logStackTrace and logExhausted options to true.

</errorHandler>

That is not all, the activemq component can also configure logging details. The activemq component will be default log any uncaught exceptions at WARN level including stack traces. So out of the box you may see the exception being logged twice. One time by the Camel transaction error handler, and a 2nd time by the activemq component. You can customize the logging level on the activemq, so for example to suppress logging, you can disable logging using the two errorHandler options as shown below:

</bean>

Okay so lets take a look at how to correlate the logs.

Correlating logs

When ActiveMQ redeliveries a message, it will use the same MessageID, which allows you to correlate the logs. So in the example above. You can correlate the message by MessageID, which ends with 2:2:1:1. It can get even easier if you enable MDC logging, which means you can have a column in the log file, that contains the MessageID.

Here is a example where we use MDC logging, and have exchangeId and messageId in two columns. We have shortened the ids, as the are often unique in the last part. So we have kept the last 10 chars, configured by %-10.10X{exchangeId} - %-10.10X{messageId}. I also added a route step to log the message using <to uri="log:before"/>.

1st message

INFO before - 471691-0-3 - -2:2:1:1:1 - Exchange[ExchangePattern:InOnly, BodyType:String, Body:Hello World]

WARN TransactionErrorHandler - 471691-0-3 - -2:2:1:1:1 - Transaction rollback (0x4ab34646) redelivered(false) for (MessageId: ID:davsclaus.lan-56013-1328258472923-2:2:1:1:1 on ExchangeId: ID-davsclaus-lan-56008-1328258471691-0-3) caught: java.lang.IllegalArgumentException: Forced exception

2nd messge

INFO before - 471691-0-4 - -2:2:1:1:1 - Exchange[ExchangePattern:InOnly, BodyType:String, Body:Hello World]

WARN TransactionErrorHandler - 471691-0-4 - -2:2:1:1:1 - Transaction rollback (0x4ab34646) redelivered(true) for (MessageId: ID:davsclaus.lan-56013-1328258472923-2:2:1:1:1 on ExchangeId: ID-davsclaus-lan-56008-1328258471691-0-4) caught: java.lang.IllegalArgumentException: Forced exception

3rd message

INFO before - 471691-0-5 - -2:2:1:1:1 - Exchange[ExchangePattern:InOnly, BodyType:String, Body:Hello World]

So what you can see is that the exchangeID is unique, as it goes 0-3, 0-4, 0-5. Where as MessageID is the same as its 2:2:1:1:1.

That is not all, I have also added a new JMX attribute, so you from JMX can see how many external redelivered messages a Camel route has processed.

JMX Statistics

So the JMX statistics for the route would be as follows for this example.

1st message

ExchangesTotal: 1

ExchangesCompleted: 0

ExchangesFailed: 1

ExternalRedeliveries: 0

Redeliveries: 0 (internal)

2nd message

ExchangesTotal: 2

ExchangesCompleted: 0

ExchangesFailed: 2

ExternalRedeliveries: 1

Redeliveries: 0 (internal)

3rd message

ExchangesTotal: 3

ExchangesCompleted: 1

ExchangesFailed: 2

ExternalRedeliveries: 2

Redeliveries: 0 (internal)

The new attribute is ExternalRedeliveries. Notice in the 3rd message, we can see that we have processed 3 messages, however 2 of them failed, and 1 was successfully. Beforehand you could not see from this information that it was in face the same message. It could just as well be 3 different messages, where it just happens only 1 could be processed successfully. Now you can know how many unique messages a route has processed by calculating: ExchangesTotal - ExternalRedeliveries. So in our example that would mean: 3 - 2 = 1.

I am considering improving this a bit more. I fell that that Camel should log at INFO level if a external redelivered message was committed. This would allow you to see this from the logs. This allows you to know that the message was eventually processed successfully. As the alternative would be that ActiveMQ would move the message to its Dead Letter Queue. And when this happens, Camel would not know, and you cannot see this from the Camel logs. Likewise I consider making adjusting the logging levels for transaction begin and commit configurable. They are currently fixed at DEBUG level.

If you have feedback upon this then feel free to comment on the blog, or reach out to the Camel community.

Unit testing OSGi Blueprint with Apache Camel made easier

2012-01-27T10:33:00.000+01:00

Apache Camel 2.10 is currently in development, and recently Guillaume Nodet added a new camel-test-blueprint component, to make unit testing Camel with OSGi blueprint much easier.

I have then polished the camel-test-blueprint to make it easier to use, and also fix a few glitches etc. So what's the story then? Well camel-test-blueprint, works exactly as the regular camel-test you may be familiar with already.

So all you have to do is to extend the CamelTestSupport class, and override the getBlueprintDescriptor method, and return the path for your OSGi blueprint XML file. Then the rest is the usual story, to add test methods.

The Camel documentation has been updated, and you can find details here.

This is a great step forward for Camel end users who are on the OSGi blueprint wagon, as its becoming easier to develop and unit test along the way. All you have to do is to right click from your editor and run the unit tests as shown in the screenshot below.

JUnit testing Camel with OSGi Blueprint

The Camel Maven archetype camel-archetype-blueprint has been updated and include the test kit out of the box with a sample code. I used this archetype to generate a new project, imported the project into Fuse IDE / Eclipse, and then I opened the unit test source code file, and right clicked -> run as -> JUnit test. This is all standard functionality of Eclipse.

The camel-test-blueprint component will be part of Apache Camel 2.10 release, and as well in the upcoming Fuse Mediation Router 2.9 release (our distribution of Apache Camel).

A rare book

2012-01-24T10:11:00.000+01:00

Jonathan and I hear often from our readers of Camel in Action, that the book is great, that it helped them get up and riding the Camel, and so on.

Camel in Action - a rare book, and a rare book cover

The Camel in Action book has now been out there over a year. And over the time readers have been kind to post reviews of the book.

I just spotted the last review on Amazon, which I though was much inline what we have heard people say about the book. Let me just quote that review.

It isn't often that I give a technical book a 5 star rating. This is the exception because it is an exceptional book.

This book is fascinating reading and how often can you say that about a technical reference? As one goes through the material the author always seems to be at the place you need him to be at that moment. You don't get that "now waitaminit" feeling that other books can leave you with when they switch topics.

There are smooth context switches from how one uses the system to how one implements the details and then back up again. This makes it easy to follow and is one of the reasons it is actually fun to read.

Boy, fun to read, is that ever an exceptional technical book.

If you have spotted it, the above image of the book cover is also rare. In fact it was the first draft of the cover, where the Camel is standing up. However Manning later decide to have the Camel sitting instead, which is the actual used cover. You can see the actual cover on the sidebar of my blog.

EIP Flashcards

2012-01-20T15:36:00.000+01:00

FuseSource makes available the EIP patterns as a PDF download.

EIP patterns as PDF download

The design patterns defined in this PDF deck are based on the book Enterprise Integration Patterns (Addison-Wesley). Gregor Hohpe and Bobby Woolf documented their combined experience in the integration space and created this notation, which has since been adopted as the standard for describing integration patterns. Our FuseSource community has found, as an example, a key benefit to this format of standard notations is the efficiency of using the cards to design complex integration routes in a conference room.

The EIP patterns flashcard is available at the Enterprise Integration Patterns page.

FuseSource January Webinar - ActiveMQ with High Availabilty

2012-01-18T16:20:00.000+01:00

I am possible a bit late to the game to post this blog, but I wanted to spread the word. FuseSource is having a webinars this month, actually tomorrow on the 19th.

First Scott Cranton is giving a webinar tomorrow about deploying ActiveMQ with High Availability.

Scott Cranton is presenting ActiveMQ with High Availability, and sharing his experience from the trenches, in the upcoming webinar.

This is a topic beyond the basic introduction levels. So great stuff for people there. And believe me Scott have seen this in real life with very large customer installations, such as world leading retailers.

Webinar Title
Deploying Apache ActiveMQ with High Availability

Date: January 19th, 2012
Speaker: Scott Cranton, Principal Solutions Engineer
Time: 11:00AM EST

Webinar Description
Learn how to use open source Apache ActiveMQ for reliable messaging in mission-critical applications where uptime is paramount. This webinar will focus on enterprise deployments where applications need to be available 24x7 without suffering an outage to the messaging system.

We will cover techniques for deploying ActiveMQ with high availability (HA), so your application will survive network outages, hard disk crashes or machine losses. Focus will be on configuring Master/Slave a Network of Brokers, and explaining what's happening within ActiveMQ. We will also look at using ActiveMQ in global organizations that span many different geographical locations. Finally, we will talk about how to configure ActiveMQ so that it can scale to support thousands of concurrent connections, queues and topics.

This webinar will include a live demo and Q & A session.

Registration
You can register for the webinar from this page.

FuseSource has previous held webinars in our archives. So feel free to take a look and watch any webinar at your leisure.

FuseSource has professional Camel documentation

2012-01-13T12:25:00.000+01:00

Eric Johnson from the FuseSource documentation team, recently blogged about the fact, we from FuseSource, have plenty of professional documentation.

Quoting Eric

I find it frustrating that so few people, people who use our products, realize the amount of documentation FuseSource offers on their products. I was talking to some users the other day who were shocked to discover that FuseSource offerers Javadoc for Fuse ESB. It rapidly becomes clear that people who complain about our documentation are usually complaining about the documentation they find at the Apache projects that make up the core of our offerings. I understand their frustration with the information on the Apache sites. It is often outdated, confusing, and hard to search. It is written by ninjas for other ninjas. The FuseSource Documentation team writes professional, versioned content targeted at more typical corporate IT professionals. We strive to make the content task oriented and relevant. We use better sample code, vet the content for accuracy, and organize it so that it can be searched effectively. Our documentation can be found at:

Fuse ESB (Apache ServiceMix)

Fuse Message Broker (Apache ActiveMQ)

Fuse Mediation Router (Apache Camel)

Fuse Services Framework (Apache CXF)

It is available for free and, hopefully, is useful. We love to get feedback, so if you find issues or have ideas about how we can improve the content, leave feedback in the comment forms on the bottom of the pages. One of the writers will get back to you pretty quick.

I can second Eric's observations. The level and quality of the documentation of the four Apache projects, can at best be used as a reference material.

As Eric says, FuseSource, have a professional team, who help writes documentation you cannot find at Apache, such as migration guides, performance tuning guides, deployment guides, transaction guides, management guides, and an operators handbook, etc. You may find pieces here and there scattered in the Apache documentation.

For example, when people ask me, what is Apache Camel, and how do they get started? Then I usually point them to an article - Open Source Integration with Apache Camel and How Fuse IDE Can Help written by Jonathan Anstey. This article introduces you to Camel, tells how its related to Enterprise Integration Patterns, shows a high level architecture overview, and talks about some of its concepts. And tie it all together with a sample use-case, with source code, that are explained.

Another good source is the chapter 1 (free) of the Camel in Action book. Jonathan is a co-author of this book, together with me.

There is of course also the FuseSource Mediation Router (Camel) introduction documentation as well.

Apache Camel 2.9 Released - Top 10 Changes

2012-01-03T20:57:00.000+01:00

On the last day of 2011 the Apache Camel artifacts just managed to be pushed to the central maven repo, just shy 1.5 hours before champagne bottles was cracked and we entered 2012.

The 2.9 release is a record breaking release with about 500 JIRA tickets resolved since the 2.8 released 5 months ago.

Here is a break down of 10 of the most noticeable improvements and new features:

1.
JAR dependencies reduced. The camel-core JAR now only depend on the API from slf4j. On top of that about 15 components, no longer depends on Spring JARs.

I have previously blogged about this.

2.
The Simple language has been overhauled and has a much improved syntax parser, which gives precise error details, what is wrong. You can now also have embedded functions inside functions as well. And we have unary operators, such as ++ to easily increment counters. I also started experimenting with ternary operators, so expect Conditional and the Elvis operator to be introduced in the future :)

I have previously blogged about this.

3.
The Bean Component has been much improved as well. Now you can define bindings explicit in the method name option, to fully 100% decouple your bean code from Camel, when using more complicated bindings. Likewise you can pass in values such as literals, numbers, booleans etc as well. The bean component can now also invoke static methods directly, as well invoking private class beans if an interface exists.

I have previously blogged about this.

4.
Splitting big XML files in a streaming mode with low memory footprint is now possible. There is a tokenizer solution, that is pure String based by scanning tokens. And another solution to use the StAX and JAXB APIs. The former requires no JAXB bindings, as required by the latter solution.

I have previously blogged about these two solutions [1] and [2].

5.
More cloud components. We now have 2 new AWS components for Simple Email Service, and Simple DB. There is also a new JClouds component.

6.
Using request-reply over JMS with fixed reply queues now supports a new exclusive option which performs faster, than the default assumed shared queue. Likewise the JMS consumer supports a new asyncConsumer option, to allow the JMS consumer to leverage the asynchronous non-blocking routing engine. All good stuff that if enabled can make JMS goes faster under certain use-cases.

7.
Added a new number of JMX annotations to allow custom components to easily expose custom JMX attributes and operations. We also have JMX load statistics on the ManagedCamelContext MBean which is similar to the unix top command, which has average load stats for the last 1-minute, 5-minutes, and 15-minutes.

8.
The camel-cxf component now supports OSGi blueprint configuration for the CXF-RS as well.

9.
There is a number of new Apache Karaf Camel commands for further managing your Camel applications from the command shell.

10.
And as usual there is a lot of minor improvements and bug fixes as well. For example the file/ftp components now support the sendEmptyMessageWhenIdle to .. yeah send an empty message when there was no files to poll. Likewise the script and language components now more easily allow to load scripts from file/classpath. And the Camel Test Kit, now have more juice for swapping endpoints before unit testing, which makes it easier to swap real endpoints with mocks and whatnot without touching your route code in the tests.

And we have as usual upgraded to the latest and greatest of 3rd party libraries, such as Apache CXF 2.5.1, Groovy 1.8.5, Jackson 1.9.2, AWS 1.2.12, Spring 3.0.6, and JPA2 etc.

You can see more details at the 2.9 release notes, such as details about other improvements and bug fixes etc.

Apache Camel - 2011 in Numbers

2012-01-02T13:31:00.000+01:00

The last couple of years I did this blog post about numbers for 2010, and 2009, so I thought I should do it again this year.

Just to do a quick post on some of the numbers for the Apache Camel project in year 2011.

Number of releases in 2011: Camel 2.x = 12 releases (incl. 1 RC release).
Number of posts on Camel user forum in 2011: 8315 (+1393 over last year)
Number of commits in 2011: 4926 (+990 over last year)
Total number of tickets created at end of 2011: 4844
Number of tickets created in 2011: 1367 (where as 189 are still unresolved).
Number of tickets resolved in 2011: 1286
Number of committers doing commits in 2011: 26 (+8 over last year) (source: git shortlog --after 2010-12-31 --before 2012-01-01 -ns).

I am sure you can dig more numbers if you got the time to hunt them down.

An introduction to Apache Camel - Blog Series

2011-12-30T14:22:00.000+01:00

Recently on twitter I noticed some tweets about Apache Camel, which covers getting started with Apache Camel.

So far there are three blogs, and there are more coming. If you want to get more familiar with what Camel is, as well how to get started with a new project, and kick the tires yourself. Then the blog series is well worth a read.

Part 1 - Introduction to Apache Camel
Part 2 - Maven project setup for a sample camel spring dsl project
Part 3 - Writing Camel Routes Using Spring DSL based XML

Happy reading. And happy new year.

Apache Camel - A little Scala DSL example

2011-12-16T09:14:00.002+01:00

So we have a Scala DSL in Apache Camel for many years now, and I guess its about time I wrote a little blog entry about this (has been on my todo list for a while).

So the Scala DSL is of course using the Scala programming language which has many bells and whistles over plain Java. However the uptake of the Scala DSL is not very high, as the Java and XML DSL's is good enough for most people.

Anyway I guess one of the nice thing about the Scala DSL would be using closures as expressions and predicates etc. So let's do a little example using the Filter EIP pattern and use a closure as the filter predicate.

When using the Scala DSL you should use the org.apache.camel.scala.dsl.builder.RouteBuilder, which is the Scala empowered DSL.

So in the following we have a FilterRoute class where we use the Scala DSL, when we define the createMyFilterRoute function.

class FilterRoute {

def createMyFilterRoute = new RouteBuilder {

from("direct:start")

.filter(_.in("gold") == "true")

.to("mock:gold")

}

As you can see from the code above, inside the scope of RouteBuilder we have the Scala DSL at our disposal. Then we use the Filter EIP which accepts a function with the Exchange as parameter, which gets defaulted into the _ symbol. The result of that function is the evaluated as a predicate, using the Scala powerful (but a bit scary) type system with implicit type converter and case matching etc.

The in("gold") is a function on a Exchange wrapped we have in the Scala DSL which adds additional methods on the Camel Exchange (RichExchange), the in is a function that look up a header.

To unit test this route I wanted to use the existing and powerful camel-test module. This module offers the CamelTestSupport class you can extend for your unit tests.

So the unit tests can be almost like in Java, but you would need to add the trait org.apache.camel.scala.dsl.RouteBuilderSupport which helps bridge the Scala RouteBuilder with the Java RouteBuilder, that the CamelTestSupport expects and uses. The code below shows an unit test example.

class FilterRouteTest extends CamelTestSupport with RouteBuilderSupport {

override def createRouteBuilder() = new FilterRoute().createMyFilterRoute

@Test

def testFilterRouteGold() {

getMockEndpoint("mock:gold").expectedMessageCount(1)

template.sendBodyAndHeader("direct:start", "Hello World", "gold", "true")

assertMockEndpointsSatisfied()

}

@Test

def testFilterRouteNotGold() {

getMockEndpoint("mock:gold").expectedMessageCount(0)

template.sendBodyAndHeader("direct:start", "Hello World", "gold", "false")

assertMockEndpointsSatisfied()

}

As you can see we use the RouteBuilderSupport trait, and then override the createRouteBuilder function to return the Scala DSL empowered RouteBuilder we created previously. The rest of the code is standard and plain Java code with JUnit @Test annotations.

This example is provided in the source code of the camel-scala module, as part of an unit test.

If you are a Scala fan and interested in Camel as well, then the Camel community could use people who are dedicated to Scala and help with the Scala DSL. The Camel team is often busy with other issues in our lives, so we only have a bit time to have fun and play with Scala.

Upcoming webinar - Getting started developing with Apache Camel and Fuse IDE

2011-12-09T13:00:00.000+01:00

I am giving a webinar next week on getting started developing with Apache Camel with Fuse IDE.

Date: December 15th
Speaker: Claus Ibsen
Time: 11:00AM EST

This webinar is a hands-on practical tutorial on how to get started integrating applications with Apache Camel using Fuse IDE.

The webinar begins with a brief overview of Apache Camel, and why it makes integration tasks much easier. We then quickly get practical and develop our first integration application using Camel and Fuse IDE. Slowly we progress and show how to implement integration solutions using, but not limited to, files, ftp, jms, web servies, csv, xml, objects.

We also work with a number of Enterprise Integration Patterns such as Content Based Router, Recipient List, Splitter, Aggregator, Message Translator, Service Gateway and Idempotent Consumer. This webinar also touches on how you can gain insight into running Camel applications, to better understand what's happening, using message tracing. All together this is a very practical webinar with fewer slides and more live code and development.

Accompanying this webinar we have an eBook with sample code to ensure that after the webinar, you have material to successfully get started integrating applications with your Apache Camel journey.

You can register here.

Java Day Riga

2011-12-01T15:37:00.001+01:00

I was invited to speak at the Java Day Riga Conference.

The conference took place at the Tallin Riga Hotel, which is a very nice hotel, just opened a couple of years ago. The hotel is also placed in the city centre, with walking distances to the nice areas.

I would like to thank Dmitry, for inviting me, and Oracle for sponsoring this event. According to Dmitry, the event would not be possible without the good faith from the Oracle. The event was free, so we had about 200+ attendees.

I like the badges, having a big and easy to read name.

Conference badget with easy to read names

The evening before the conference, we went out on a guide tour to see Riga. We hired a history professor, whom was very enthusiastic and knew a lot about what has happened to Riga over the many years. Riga is a very old city, named after the river, and it has been taken over many times between Germans, Polishes, Swedes, Russian etc.

Monument of freedom in Riga centre

The big monument in the city centre is the monument of independence, and on top it holds three stars, one for each of the three regions of Latvia.

Unfortunately after one hour it become too cold to pay attention, and we have already seen 4 churches, and many historical buildings. We found a place selling mulled wine, so we could get a bit of warm.

History Tour in Riga. Drinking mulled wine to get a bit warmth

After the tour we went for a later dinner, and just a few beers. The conference starts the next morning.

The keynote talks was good talks about the upcoming lambda expressions in Java 8. As well a talk about JEE 7, which has focus on the cloud.

Martijn Verburg preparing for his talk

Martijn Verburg, did a great talk titled "Diabolical Developer". It's a sort of pep-talk with common sense, where Martin says what some people may think is controversial. But what he really want to say, is that "think for yourself".

Martijn thinks developers are awesome

I gave a talk about Enterprise Integration Patterns and Apache Camel. In the talk I also demonstrated live coding, how to setup a new Camel project, and get that up and running in Eclipse etc. As well some of the powers of the Fuse IDE, which has graphical notation of your Camel routes, using the industry standard EIP icons. Likewise Fuse IDE is capable of doing runtime insights into running applications, which helps you pin point what is going on, as well being able to see performance statistics, with graphical representation etc.

In the end of the talk, the audience was giving a choice of either talking about Apache ServiceMix or the Camel roadmap. The audience chose the latter, so I gave a round about what to expect in the next Camel 2.9 release. Then we ran out of time. A talk for 50 minutes, is frankly a bit on the short side. 1h is usually the minimum.

In the airport on my way home. I spotted the Camel.

Camel spotting at the airport

Splitting big XML files with Apache Camel - Part 2

2011-11-24T09:45:00.001+01:00

In my previous blog about splitting big files with Apache Camel, I said we were working on another solution, which is a new camel-stax component. The work is now complete, and the component will be part of the next Apache Camel 2.9.0 release. Thanks to Romain for his contribution.

The stax component
The stax component allows you to split big XML files as well, but it requires using JAXB and StAX. This means you need to define a POJO class(es) with JAXB annotations, to bind to the XML schema.
However the benefit is that you then work with the POJO classes in Camel.

For example the records example from the previous blog could be written with camel-stax as follows in the Java DSL

from("file:inbox")
.split(stax(Record.class)).streaming()
.to("activemq:record");

Where Record is the POJO class which has the JAXB annotations. And stax is a static import from the class org.apache.camel.component.stax.StAXBuilder.

If you are using XML DSL, then consult the camel-stax documentation which has such an example.

A little test
So I run the equivalent test from the previous blog as well with 40.000 elements, and the memory usage delta was about 8mb.
The test logs the time: Processed file with 40000 elements in: 55.962 seconds.

Running the test with 200.000 elements results in 9mb memory usage delta, and test time of: 250 seconds.

The unit test is in camel-stax component as the org.apache.camel.component.stax.StAXXPathSplitChoicePerformanceTest class.

Apache Camel 2.8.3 Released

2011-11-24T09:41:00.001+01:00

This is just a quick blog entry to say that Apache Camel 2.8.3 has been released. The JIRA tracker has about 60 tickets resolved for this release.

Apache Camel 2.9.0-RC1 Released

2011-11-12T10:05:00.001+01:00

The Camel team is working hard on the last pieces for the upcoming Apache Camel 2.9.0 release. In the mean time we decided to cut a release candidate; due to some larger changes like core API refactorings, Spring dependency changes, rewritten simple expression language, etc.

We would highly appreciate any feedback from the community in terms of any upgrade glitches, or other issues discovered in the release candidate.

The release is available to download from Apache, and as well from Central Maven repo.

For the release notes we suggest to take a look at the current in-progress release notes for the 2.9.0 release.

Coffe Machine and Camel in Action

2011-11-07T12:05:00.002+01:00

Jonathan and I got published at the very end of last year when Manning announced that the Camel in Action book was available in print. As authors we are entitled to royalties of the sales of the book. Before you ask, we only get pocket changes compared to the amount of work we put into the book.

So what's the story with coffee then? Well as I have a home office I have to make my own coffee, as there is no fancy coffee machines around I can use. So for years I have been living off regular filtered, instant or stempel -coffee.

I made a promise to myself that I would buy a coffee machine when I get my royalty cheque. Well the cheque has arrived a month ago.

There is a lot of different types and brands of coffee machines. So I spend a while reading the web and watching you tube reviews of the machines. I also got advice from Johan Edstrom, who has a machine.

As I do not want to be my own barrista I was looking for a full automatic machine. At first I got my eyes on a machine from Gaggia, but recently spotted a new machine from Jura. I wanted a small machine as the daily use would be at my home office for a single person; our dog don't dring coffee, and my wife only drinks coffee in the weekends. The latest Jura ENA 9 Micro seemed like a great machine. It's small and compact, full automatic, and has a nosel to take in milk directly from the carton (or from a thermal bottle). So basically it's a one push button machine making coffee, expresso, cappuccino or latte.

So last wednesday I put in the order, and this morning the machine arrived at my doorstep.

My new coffee machine

The machine costs about 1300$, which would have been out of my normal price range. Well I could of course take the cash out of my regular pay cheque, but I wanted to stick to what was affordable from the royalties. That would also justify going for a higher priced model.

If I have to guess how many hours I have spend working on the book, then 1300 hours would be a good guess. So that's 1$ per hour. And that's before tax.

Just wanted to share this with the readers, so you know how I spend my royalties.

Thank you, the readers, for making my coffee a pleasure to drink from this day forward.

Splitting big XML files with Apache Camel

2011-11-04T12:32:00.000+01:00

In the upcoming Apache Camel 2.9 we have improved the support for splitting big XML files using streaming and very low memory footprint.

In previous versions and examples provided on the Camel website, often showed examples of using XPath to split XML files using the Splitter EIP pattern.

Unfortunately the underlying XPath framework do not support an iterator based result, as its limited to the types defined by the JDK in the XPathConstants. That means a NODESET would be used as result type, which causes the XPath framework to return a NodeList instance which contains the entire XML payload in memory. There is nothing you can do about this despite using StAXSource, SAXSource or other stream types as input to the XPathExpression. Regardless what it would return a NodeList as result.

Tokenizer solution
So the Camel team have two solutions in the works. The first is already implemented in the upcoming 2.9 release. Its based on the tokenizer language which supports an iterator stream based. This means we can split any big file one a piece by piece without causing the entire content loaded into memory. So I enhanced the tokenizer to support two additional modes:

pair
xml

pair mode
The pair mode is to be used when you need to grab piece by piece and you have a known start and end tokens to denote a record. For example if you have a [START] and [END] markers in the content.

from("file:inbox")
.split(body().tokenizePair("[START]", "[END]")).streaming()
.to("activemq:record");

xml mode
This was used as the foundation for the xml mode as well, as the idea is similar. So you define a child tag name as the record to grab. For example to split by then you do as follows:
from("file:inbox")

.split(body().tokenizeXML("record")).streaming()
.to("activemq:record");

The XML content may look like
<records>
<record id="1">

</record>

  <record id="2">
  
  </record>
...
  <record id="99999">
  
  </record>
</records>

Now what about namespaces? Suppose you have a common namespace in the parent/root tag as follows:
<records xmlns="http://acme.com/records">

...
</records>

Then you can instruct the tokenizeXML to inherit namespaces from a parent/root tag by providing the name of the tag as the 2nd parameter as shown:
from("file:inbox")

.split(body().tokenizeXML("record", "records")).streaming()
.to("activemq:record");

Which means each splitted message will contain the namespace included:
<record id="1" xmlns="http://acme.com/records">


</record>

What I like about the tokenizer is fully stream based and returns data as String content, which means there is no intermediate DOM or POJO objects or anything like that. Which mean it can split any kind of XML payload without having any model of it in the java code. So if you just need as in these examples to split the big XML file and send each splitted message to a JMS queue, then that is fast as there is no unnecessary to/from object marshaling.

A little test
I ran a little test on my laptop to process 40.000 records and the memory usage delta was about 4mb.
The test logs the time: Processed file with 40000 elements in: 53.676 seconds

Running the same test with XPath reneders a memory usage delta about 100mb.
The test is in fact a little faster: Processed file with 40000 elements in: 49.941 seconds

The reason is that after all content is loaded into memory, then its a pure CPU processing, where as the tokenizer does load the content from disk piece by piece.

That was just a small XML file with 40.000 records with a file size of about 7mb.
Now image if the XML file was a 500mb size with a million records. The XPath will be very slow and most likely cause a OOME exception on your server.

The unit tests is in camel-core which you can play with in the src/test/org/apache/camel/language directory.

What about the other solution
Its a community effort together with Romain who have created a Camel StAX component with a stream based iterator as well. However it requires a POJO model that has been JAXB annotation to be used. We will continue working on this and have his work contributed into the Apache Camel distribution.

Todays Webinars about Database Integration with Camel

2011-10-27T07:38:00.000+02:00

Today my colleague Charles Moulliard is giving a webinar about database integration with Apache Camel. Charles will give the presentation twice today, one time slot suited for EMEA and another for US.

Enterprise integration projects often include databases, and creating Apache Camel message routes in the context of database connection and transaction management are nontrivial. The first part of this 2-part webinar series will introduce you to several integration components and demonstrate how easy it is to configure them to give access to the information persisted in a database. Particular attention will be given to the pros and cons of each component, and when to use each. We will also cover how to avoid losing messages during database connection calls using transactions.

Links for more details

If you miss the webinar, then its recorded and will be available later on demand.

About Camel 2.7.4, 2.8.2 and 2.9 Releases

2011-10-25T13:02:00.000+02:00

Today the Camel team announced the availability of the Apache Camel 2.7.4 and 2.8.2 releases. These release are mostly bug fix releases to ensure stability on those branches. Albeit 2.8.2 has more than 100 tickets resolved. So I suggest to take a bit extra care if upgrading to 2.8.2 due the many changes.

In terms of the upcoming Apache Camel 2.9.0 release, then we have started discussing this at Apache. The plan is to cut a RC1 release first, to give the community a chance to give it a test spin and provide feedback. The reason is because Camel 2.9.0 have a number of API changes and cleanup in the camel-core. Therefore we would love for people in the community to download and try out RC1, especially if you have plans to upgrade to Camel 2.9.0 or later. We would like feedback on the upgrade issues you may encounter, and we will try to correct them before the actual 2.9.0 release.

If everything goes smoothly then hopefully Camel 2.9.0 is GA in end of november. This release has some great wins for the community (such as the improved simple language), which I will blog about later.

Fish and Chips in London, and Oysters in Paris

2011-10-20T10:20:00.000+02:00

Last week I was speaking at the Fuse Community Days in London and Paris. This is a blog entry of my impressions from that trip.

So this was my 2nd time I attend and speak at the London Fuse Community Days, which is located in the Skills Matters facility. The facility is a bit rustic and gives an impression of more informal and hands-down atmosphere. This is what I really like about these community days. It gives every body attending a great opportunity to meet and great and share ideas and solutions.

Fuse Community Day in London, at Skills Matters

It was a great day, and I spoke to many Camel end users, who are not necessary FuseSource customers, but they all spoke well of Camel - there is a lot of love for Camel. For example Julien from BBC who I helped by implementing the done file function for the file and ftp components.

However this blog title is about Fish and Chips, so the day before I went on a stroll in London, and find myself a nice little restaurant where I could enjoy the trademark dish of England - Fish and Chips.

Fish and chips with peas in Lodon

After the Lodon event, we traveled to Paris for the upcoming community day. The day before the event we went out with fellow speakers to a fancy restaurant in Paris. Because we was a large party we were only given two choices for starters, main and desert. As I really dislike four grass I had to go for the oysters. I prefer my oysters spiced or with soya etc, not raw, the seafood taste is not a favorite. As I did not finish up all my oysters the waitress though I disliked them, and asked if I wanted a salad or something. Well the oysters was fine, but I cannot eat 10 for starters. The main dish was a big steak, with french friese. Not as fancy as I would have thought. The desert was some italian dish, which is bonds with ice cream and hot chocolate on top.

Debbie pouring hot chocolate on my desert. Its Larry Alston or CEO next to me.

It was time for a photo of the desert, so Debbie setup and poured more chocolate on my desert. After the restaurant we had a late night beer at the hotel, before turning in.

The next day we attended the Fuse Community Day in Paris. It was another successful day. This time we have a mix of talks in English and French. Even though it was my 2nd time in Paris, my French haven't improved beyond "bonjour". I am ending this blog entry with "au revoir".

CamelOne 2011 Videos Available

2011-10-19T08:05:00.002+02:00

I have previously blogged about my thoughts about the CamelOne 2011 conference, where I said all the videos will become available.

Word of mouth is that more videos is now online, which you can find here.
I had a great time at CamelOne and looking forward to next year.

Upcoming Fuse Community Days in London and Paris

2011-09-29T13:22:00.001+02:00

I am going to be speaking in London on October 11th and Paris on October 13th speaking at free Fuse events - if anyone would like to meet-up for a beer and chat then come meet us.

I have the pleasure of doing a co-shared talk with James Strachan about Camel, Enterprise Integration Patterns and Fuse IDE. Knowing James I am sure you will get a glimpse of the latest installment of the Fuse IDE which has some remarkable new features and ... yes looks much nicer.

Skills Matter has provided a site with an agenda and more details you can check out here.

FuseSource UK Community Day
Tuesday 11th October
Skills Matter, London, EC1V 7DP

FuseSource FR Community Day
Thursday 13th October
La Défense, Paris

So if you are nearby those locations, then I suggest to stop by. Its free and ad-free.

Todays webinar - Getting Started with Apache Camel Fuse IDE

2011-09-27T08:18:00.001+02:00

Today FuseSource is hosting a 1 hour webinar on getting started with Apache Camel with Fuse IDE.

FuseIDE is a graphical, Eclipse-based tool for integrating software components that work with Apache ServiceMix, ActiveMQ and Camel and can help you with your messaging and integration projects.

Developing Camel applications with Fuse IDE in Eclipse

If you are not currently using Apache Camel you might not know that it is a powerful and flexible integration toolset that allows users to create integration routes between software components using the Camel domain-specific language (DSL). Fuse IDE goes one step further and allows users to connect components visually. With a simple drag and a drop developers and architects can create, debug, and deploy simple or complex routes and integrate systems quickly and easily.

The webinar starts at 5pm central european time.

You can find more details here, and how to register for the webinar.