I’ve been profiling recently a legacy J2EE system in an attempt to improve its performance as the system is expected to get much bigger hit in near future. Turned out that only severe architectural changes and essentially rewriting core components would dramatically improve throughput of the system. The underlying issue was the long standing enemy: distributed transactions. This experience led me to thinking more about high performance systems and principles to follow.

Executing a distributed, long transaction and blocking until everything is committed will always be a bottleneck, so we need to make sure to avoid mistakes from the past as we can see everywhere increased business demand for greater scalability. Therefore we can forget about traditional model of the two phase commit. However, in most cases, we can’t trade the reliability and consistency of the system for its performance. So, what are our options? I think the keywords would be:

• loosely coupled
• locally transactional
• operating on local data
• taking advantage of multi-core/machine power

Which leads to:

• data grids
• concurrent programming languages (Erlang, Scala, Clojure, …)
• actor model

To achieve high throughput system that scales but remains consistent and reliable we have to change couple of things. Instead of distributed transactions consider using smaller, independent stages (workflow) that can fail or succeed independently. No need for transaction coordination. Operations by mean of partitioning would be executed where the data regarding the operation exists. The partitioning which is given in data grid solutions like Coherence can be a bit tricky with solutions which don’t provide this feature (Terracotta, at least the version 2.7.3).

Personally for me database is dead.  Personally I would go for Scala.

Share This

Interesting task I have to say. I think I will approach this by addressing both project management and engineering requirements by setting up an iterative process based on bits and pieces from Scrum and XP. It shouldn’t be a surprise to anyone involved in the past in Web21C Experiment. The Scrum process will kick off with pretty standard set of features hence I will not bother you with details.

What I want to achieve in field of team organisation is to:

• grow a successful team that delivers business value (doh!) as soon as possible (doh!)
• improve internal and external communication and information flow
• share common understanding of the domain and the system
• learn how to cope with inevitable change
• activate developers to be more encouraged to introduce new ideas and solutions
• put the team in position to be more in the partner relationship with business
• improve motivation and satisfaction from work
• encourage being passionate about technology and sharing that passion

In the same time in field of engineering practices I’ll stand my ground on:

• automation, automation, automation
• continuous integration
• code coverage
• automated integration tests
• patterns
• non mandated on-request pairing (you wonder what it is?)
• lightweight solutions
• shared code-ownership

There is an interesting lean Kanban process on horizon which looks as a good match for the environment however I will not be pushing for this initially. As the process should evolve over time there is always time to navigate original agile scrum process towards something which can be maybe a better choice. Only time will prove.

Comments, advice?

Share This

I participated yesterday in the Bupa London 10k race, second edition of this, as organizers were trying to sell, premiere London 10k run. They are not far from the truth to be honest. Click on the image of the route. With starting line on next to Buckingham Palace and St. James’s Park and with the finish on The Mall, taking runners almost for a sightseeing including Westminster Abbey, Houses of Parliament, Big Ben, The London Eye, Cleopatra’s Needle, Tate Gallery, Millennium Bridge, The Monument, Lloyds of London, Leadenhall Market, Bank of England, Mansion House, St. Paul’s Cathedral, Horseguards, Nelson’s Column and Admiralty Arch is a running event in the capital not to miss for medium to long distance runners. It is a race organized by the same people working hard each year to make the London Marathon what it is, so everything what was needed was there. Even weather, comparing with previous year’s torrential rain, was just perfect.

I haven’t been preparing for this race in particular, mainly because after finishing Paris Marathon I decided to have a month off running. Only a week ago I started some training, first week of the 18 weeks programme which I am going to follow this time as preparation for the Berlin Marathon, Royal Parks Half Marathon and the Frankfurt Marathon. Yes, 2 marathons and a half in 5 weeks. I know, I know. I will write a bit more about this another time.

Coming back to the Bupa 10k. My time was 48 min 26 sec (5k split 24m56s), the official new personal best for this distance. Taking into account irresponsible but much appreciated BBQ the night before I am really happy with the time. It is slightly worse than my unofficial 46m52sec for first 10k from the Lisbon Half Marathon. I am using Garmin 405 for GPS tracking now so the details of the run are available on the Garmin Connect site.

The public registration for the London Bupa 10k 2010 opens next week on the 10th of June. It seems pretty early for a run happening in 12 months, but if someone wants to take part in the next year’s event I recommend signing up asap as I am sure the places will be sold out quickly.

Congratulations to Dave for achieving his dream target of sub-1h, Milan for not showing up, Andrew and Rachel for finishing their first 10k in a very good style and Maciej for being just scary fast for a first timer.

Share This

It is an awesome tour of Paris. The route leads from the Champs-Elysées to the  Place de la Concorde, Rue de Rivoli along the Louvre, then Place de la Bastille and roundabout of the Place de la Nation. Not far next is the second biggest park in the city: Bois des  Vincennes where runners make a 7km loop. Half marathon mark is on the way back from the park to Bastille where slight downhill helps save some energy for the second and final part. From Bastille the route goes towards the river Saine and the next 10km give everyone chance to enjoy river side views of Paris. The famous-infamous 31km mark is next to the Eiffel Tower where couple of hundred meter up hill stints remind about the distance behind and the already drained to minimum resources in the body. The biggest park in Paris, Parc de Boulogne, is a place for final effort where each passed kilometer brings us closer to the finish line on the Avenue Foch overlooking Arc de Triomphe.

It was the fastest race in Paris of all time, this year. Winner of the men’s event Vincent Kipruto of Kenya set a course record of 2:05:47.

On personal note I am happy with my own performance, far from above mentioned time, but within my sub-4h target. The official split times from the chip were:

5km      00:28:29
10km     00:56:34
15km     01:22:50
21.1km 01:55:44
25km     02:18:02
30km     02:47:28
35km     03:15:43
42,2km  03:57:21

Second part, let say from 32km hasn’t been easy but I managed to avoid hitting the wall and kept my pace at reasonable level. GPS track from the race is here and here (my phone crashed!). The race course and conditions were very good, maybe slightly too hot but is there anyone who knows how to control it? Great experience in a beautiful city though! I paid my personal price for this by missing the best snowboarding trip of the year to Laax but I think at the end it was worth it. That’s what she said.

So, the first marathon done! I already have got plans for the next one. Berlin in September, here we go.

Share This

For me it was a tune-up race before the Paris Marathon. I tried to take this as easy as possible which in race conditions is not a trivial task to do. As a result of this I have beaten my personal best by 11min :) Official time from the chip was 1h 47min 14sec. As always more details on the run together with GPS data is on the Nokia Sports tracker site.

Interesting city, ok organisation of the race, nice route and superb weather. Clearly worth doing.

Countdown to Paris continues. 5 days left, excitement is getting more and more obvious every day. I have to say I am really looking forward for that one. I am really lucky with my health, no major injuries during the prep. As it turns out at the Runner’s World forum there is a quite big group of runners from UK so I believe I will meet some of them in Eurostar on the way to Paris this Friday. If not then certainly in the Corcoran’s after the run!

Race starts on Sunday at 8.45am local time. Target time 3h 55min. Wish me good luck.

Share This

My plan

It is a 12 week program which intends to accommodate my existing sport activities to replace some of the shorter runs with just around 30km of actual running a week. It is intended for people who are already in overall good shape as the plan is rather short. The week structure is:

Monday - rest

Tuesday - 5-a-side foolball - 60 min

Wednesday - squash 45 min

Thursday - fast run <= 10km

Friday - squash 45min

Sathurday - gym - cross training

Sunday - long run >= 20 km

The intention is that the squash and football replace hills repetitions and intervals often used in marathon preparation plans. Important to mention is that exercises at gym are not focused on legs. Rather everything else but legs. The distance of the long run should start with around 15km, then gradually get to 25km with maybe one 30km run during the training cycle. Last two weeks is a tampering time when  training impact should be decreased. Allocation of activities to the days of the week is flexible with only the long run to be done preferably on weekend.

My plan doesn’t really follow the rule mentioned above about making as much distance a week as possible but instead gives me a range of different activities. I believe it prevents fatigue often caused by excessive running which then might lead to injuries and overall lower performance.

Certainly it prevents boredom and allows you to have a social life for those 12 weeks! Also the length of the training is suitable for preparation for European marathons taking place at the beginning of April allowing someone to have normal Christmas and New Year’s break unlike the longer schedules. You know what I am talking about, don’t you?

I should know how well the training works in just 5 weeks as I will stand on the start line of the Paris Marathon on April the 5th. I replaced a usual long run 2 weeks before the final race day with a start in the Lisbon half marathon. That will be the gear, hydration and feeding strategy test day. More to come.

Share This

First make sure you have installed latest Apple’s Xcode Developer Tools which will provide gcc compiler among the other toys.

The next step is to download Erlang sources and man pages. At the time of writing this post the latest version of Erlang was R12B-5. Consult the Erlang download page for details of current release. Let’s get the above mentioned files :

%  wget http://erlang.org/download/otp_src_R12B-5.tar.gz
%  wget http://erlang.org/download/otp_doc_man_R12B-5.tar.gz

Untar the sources and change current directory:

%  tar xzvf otp_src_R12B-5.tar.gz
%  cd otp_src_R12B-5

What we have to do now is to do usual configure/make/make install steps of compilation. To check available configuration options run:

%  ./configure --help

I will configure Erlang with:

%  ./configure --prefix=/sw/erlang --enable-threads
--enable-smp-support --enable-kernel-poll

Next steps are creating the target directory, compilation and installation, uncompressing the man pages:

%  sudo mkdir /sw/erlang
%  make
%  sudo make install
%  sudo cp ../otp_doc_man_R12B-5.tar.gz /sw/erlang/
%  cd /sw/erlang
%  sudo tar xzvf otp_doc_man_R12B-5.tar.gz
%  sudo rm otp_doc_man_R12B-5.tar.gz

The last step will create man folder in /sw/erlang. Everything almost ready. We should add Erlang to the PATH and man pages to the MANPATH i.e. in ~/.bashrc.

export PATH=$PATH:/sw/erlang/bin
export MANPATH=$MANPATH:/sw/erlang/man

The last step is to load the updated .bashrc file:

%  . ~/.bashrc

Everything should be up and running now. Try:

%  man erlang
%  erl

Share This

• com.tc.cluster.event.thisNodeConnected — The local client connecting to a Terracotta server
• com.tc.cluster.event.thisNodeDisconnected — The local client disconnecting from a Terracotta server
• com.tc.cluster.event.nodeConnected — Another client connecting to a Terracotta server
• com.tc.cluster.event.nodeDisconnected — Another client disconnecting from a Terracotta server

To consume the events we will use javax.management classes and first we will get reference to a MBeanServer:

MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer();

Then we should get the cluster bean. I created small helper class ClusterUtils as described in the Terracotta JMX Guide:

ObjectName clusterBean = ClusterUtils.getClusterBean(mbeanServer);

We can alrady check what is the identificator of the current node:

String nodeId = mbeanServer.getAttribute(clusterBean, "NodeId").toString();

The only remaining step is registration to the JMX notifications:

mbeanServer.addNotificationListener(clusterBean, new SimpleJmxNotificationListener(), null, null);

Where the SimpleJmxNotificationListener implements javax.management.NotificationListener. We are ready now to consume the events and react accordingly to the changes in the size of the cluster. There is also a JMX Util library that makes using Terracotta Cluster Events even more easy.

By default the cluster nodes will be indentified by strings in format ClientID[SEQ_NUMBER] like ClientID[85]. If we wanted to set different identification in theory we could use -Dtc.node-name=NAME property when starting a given JVM. However there is currently (Terracotta 2.7.1) an open problem with handling of that property which hopefully will be resolved soon.

Everything what is needed to create a dynamic cluster manager is in place now. Cluster manager would be deployed on each node and share own state in Terracotta cluster. It is not difficult to imagine creating a bit of logic to handle connections/disconnections of new nodes in a cluster and dynamically reconfigure the existing nodes to accommodate to the change. There is always a recurring topic of partitioning when using Terracotta, how to follow paradigm of locality of reference when building a low latency, scalable systems. Combination of the cluster management described above with dynamic subscription to messaging topics with sticky routing could be a nice solution to the problem. Just musing…

Share This

“The last value queue type causes logically updated versions of previous messages to appear to overwrite the older messages. To use this feature, add a qpid.last_value_queue key to the arguments supplied. The value of the key is irrelevant. Messages published to the queue then need to specify a value for the qpid.LVQ_key in the headers of messages they publish. The last value queue uses the value of the qpid.LVQ_key to determine whether a newly published message is an update to an existing message on the queue. If this is the case, the new message will appear to overwrite the older message. A subscriber that requests messages after this has occurred will see only the newer message.”

Common example would be a queue with latest price updates per symbol when we are interested only in the last value per instrument. The documentation of MRG Messaging mentions only last value queues defined by providing qpid.last_value_queue parameter when creating a queue. There is also slightly different implementation of this feature defined by parameter qpid.last_value_queue_no_browse. The difference is described on qpid website.

Let start and create a last value queue using qpid-config tool:

qpid-config add queue lvq --last-value-queue

Check if the queue has been created:

qpid-config queues

Queue Name                                        Attributes
======================================================================
lvq                                               --last-value-queue

The messages with the same value of the qpid.LVQ_key property will be replaced, so if we wanted to write simple price publisher that publishes 12 messages of four symbols SYMBOLS = new String [] {”Crude”, “Gold”, “Silver”, “Copper”} which uses the last value queue feature the code would look like:

Destination destinationQueue = new AMQQueue("amq.direct", "lvq");
MessageProducer producerQueue = session.createProducer(destinationQueue);
TextMessage txtQueueMsg = session.createTextMessage();
for (int i=1; i < = 12; i++) {
    String symbol = SYMBOLS[(i-1)%4];
    txtQueueMsg.setStringProperty("qpid.LVQ_key", symbol);
    txtQueueMsg.setText(String.format("Tick %d for symbol %s", i, symbol));
    producerQueue.send(txtQueueMsg);
    log.info(String.format("Producer: sent message => %s",  txtQueueMsg.getText()));
}

Let’s create a synchronous client to consume the messages:

Destination destinationQueue = new AMQQueue("amq.direct", "lvq");
MessageConsumer consumerQueue = session.createConsumer(destinationQueue);

boolean finished = false;
while(!finished) {
    TextMessage message = (TextMessage) consumerQueue.receive(5000);
    if (message != null) {
        log.info(String.format("Consumer: received message %s key=%s", message.getText(), message.getStringProperty("qpid.LVQ_key")));
        message.acknowledge();
    } else {
        log.info("Timeout");
        finished = true;
    }
}

Assume that consumer subscribed to the queue after publisher finished publishing the 12 messages. In this case the consumer will receive only 4 messages, one per symbol:

Consumer: received message Tick 9 for symbol Crude key=Crude
Consumer: received message Tick 10 for symbol Gold key=Gold
Consumer: received message Tick 11 for symbol Silver key=Silver
Consumer: received message Tick 12 for symbol Copper key=Copper

Share This

I will explain quickly how to install mc on mac (Mac OSX Leopard 10.5.6) using Fink. Installation with Fink should be painless and this blog post should not exist but I experienced an issue with the process I wanted to share quick solution:

1. If you don’t have Fink yet install it
2. Run:

sudo apt-get install mc

3. Try to start Midnight Commander:

mc

4. Fink should install all dependencies but if you encounter following error go to step 5, if not you are done:

dyld: Library not loaded: /sw/lib/libintl.1.dylib  Referenced from: /sw/bin/mc

Reason: image not found

5. Small dirty hack to fix the previous problem:

sudo ln -s /sw/lib/libintl.3.4.3.dylib /sw/lib/libintl.1.dylib

I know that there are more clean solutions than this one. Share this with me! What’s your favourite OFM?

Share This