Our Doctor, always busy.

Christmas 2011: The Doctor, the Widow and the Wardrobe

Ohm, I saw this episode a long time ago, so my memory of it is not that great. I liked the teleporting box and the whole mystery. I usually don’t like the Christmas episodes, but this one was all right.

Our Doctor just loves bow ties

Asylum of the Daleks (S07E01)

The Daleks kidnap the Doctor, Amy and Rory and throw them to a prison plant. On that planet there are Daleks which went crazy, and a souffle backing woman. And somehow the souffles never turn out well. Why does she keep backing souffles? Why did the Daleks send the Doctor to the prison?

I love the craze Dalkes and woman in this episode. Is fun!

The Daleks are also busy.

Dinosaurs on a Spaceship (S07E02)

The Doctor goes to a Spaceship, which has lost it’s pilot. To our surprise there are Dinosaurs on the spaceship. And a bad guy.

I didn’t like the story, but Dinosaurs are always awesome!

Dinosaurs, uuiiihh!

A Town Called Mercy (S07E03)

UUUAAAH, a western episode, with a evil robot, a alien and gun wielding men. Why does the evil robot hates this small city? Why is there a alien in the city?

I love the western with sci-fi genre. The story itself is OK.

The good, the bad and the ugly!

The Power of Three(S07E04)

Boring episode ㅠㅠ. The episode wants to become mysterious. Strange black cubes suddenly appear. The cubes don’t do anything. Therefore our team has to wait, and wait, and wait.

I already said that i found this episode to be boring.

Many boxes!

The Angels Take Manhattan(S07E05)

In New York a private investigator receives a strange case. Angel statues which seem to move. However, no one sees them moving, yet they move. So, the detective goes into a hotel, where he meets a old himself. What is going on?

Ah, I like the Weeping Angels. And this episode. It has a lot of time traveling and Weeping Angels. It’s the best episode of series 7 so far.

Weep, Weep, Weep

Christmas 2012: The Snowmen(S07E06)

Suddenly there are many snowmen and they are evil. Also the souffle making woman is here. Strange!

Hmm, I think the episode is OK.

Love Snowmen

Conclusion

So far I enjoyed series 7. I’m waiting for the new episodes. On March 30th the BBC will start airing the new episodes. Here are some trailers:

Series 7, First Half

Series 7, Second Half

The setup looked like this:

- Amazon EC2: Mysql Extra Large instance
- Two Amazon EC2: M1 Large instance as web tiers. Running the Play Application.
- Two Amazon EC2:M1 Medium instance as load tiers. Using Grinder to run scripts which use the web app.

We take a look at latency and throughput.

Test Setup

Blog-App

First let’s look a small blog app. The app serves posts, which have tags and authors. So the application gets the latest posts and fetches the tags for each post. Also it shows the most popular tags.

A blog

Latency in Blog Application

We can see that with ADBCJ requests are completed faster. For example with JDBC, it takes around 50ms on average to return the result; the ADBCJ version can do it in about 38ms.

Let’s also look at the throughput. In a small, focused test, where we access the index page in JSON format, ADBCJ performs way better. However, in a more realistic scenario, where we access different pages, the throughput of ADBCJ was about the same.

Blog: Latency vs Throughput

So you are now might tempted to use ADBCJ for a blog like application. Think again, a blog is fairly static application. So we can easily use caching to store the rendered pages. A good cache beats JDBCs and ADBCJs asses! Adding a cache boosts throughput and latency.

Blog: Latency vs Throughput

Blog: Latency With Caching

User Content App

Let’s look at a more dynamic application, a “Facebook” clone. The main page lists posts from the people a person follows. Each post has ‘likes’ and comments. Also the site suggests users to follow. Furthermore it also contains messages, which user can send to each other.
About 50% of the requests require complex read operations, the other 50% are simple writes, like inserting a new comment.

First, let’s look again at the latency. We can see that the complex read operation suddenly is three times faster. However, all writes requests are slowed down. This is maybe a tradeoff which you can make. Reading requests are the ones where users have to wait, while write requests can be done in the background via AJAX.

Latency User Content App

So, I wondered if there a fundamental issue with ADBCJ for writes, or if it’s an interaction effect. I ran test with separate read and write requests only. Then ADBCJ has always better latency. So, it seems to be an unfortunate interaction.

Reads / Writes Only

Finally, let’s take a throughput. ADBCJ does way better. Even more, when the system is under pressure, also ADBCJ’s latency for writes start to get better compared to JDBC.

Throughput User Content

Latency under large Load

Stats Collection App

The last application is a ‘stats collection app’, which collections information about the user’s actions. It is very write heavy, with only some rare, long running reads. Here JDBC and ADBCJ latency was about the same. However, the throughput of ADBCJ was much better.

Stats Collecting

Stats App Throughput

Conclusion

So, ADBCJ can speed up your web applications. Of course, only the data serving parts. Keep in mind caching, avoiding requests etc). Of course you need to do your own experiment for your use case. Next time I’m going to explain why ADBCJ can support better latency and throughput, with code examples. Stay tuned!

ADBCJ and Scala, new buddies

Grabbing the Bits

Of course, first we need to grab the bits. We can get it via SBT. Currently there are only snapshots available on my snapshot repository. We pull the Scala ADBCJ API, the ADBCJ connection pool and the MySQL driver.

Getting Access to the Database

The next step is to get access the the database. We do this with the Database object, by passing in the connection URL, username and password. This create a instance which is used to access this database. Usually you should have one instance only per database in your application.

Then we can create new connections. The most basic way is to create a connection directly. However, you are then responsible for closing the connection.

Closing Connections Automatically

Now closing the connection manually can be error prone and tedious. Therefore helper methods exists. One for using a connection and one for using a connection with a new transaction.
The closure has to return a future. Since everything is asynchronously, we cannot close the connection when we leave the future. Instead we need to close it when everything is done. So, the connection will be close, when the future returned by the closure is completed.

Some Real Work

Let’s do some real work. We create the schema if it doesn’t exist and then insert a few items. In general, I highly recommend to read the Akka documentation about futures, which gives an idea how to deal with tem.

Query

Let’s also query for some data. When running queries, your future will contain a immutable result set, which supports the standard collection operations.

Prepared Statements

Of course, there prepared statement support:

One Last Hint, Debugging

So, this was a small tour through the API. However I want to share one last important thing, which helps you dealing with failures.

Let’s assume you ran a back query:

org.adbcj.mysql.codec.MysqlException: 42S02Table 'adbcj-demo.thistabledoesnotexist' doesn't exis 
  at org.adbcj.mysql.codec.packets.ErrorResponse.toException(ErrorResponse.java:49)
	at org.adbcj.mysql.codec.decoding.ExpectQueryResult.handleError(ExpectQueryResult.java:31)
	at org.adbcj.mysql.codec.decoding.ResponseStart.parse(ResponseStart.java:33)
	at org.adbcj.mysql.codec.decoding.AcceptNextResponse.parse(AcceptNextResponse.java:26)
	at org.adbcj.mysql.codec.MySqlClientDecoder.doDecode(MySqlClientDecoder.java:96)
	at org.adbcj.mysql.codec.MySqlClientDecoder.decode(MySqlClientDecoder.java:67)
	at org.adbcj.mysql.netty.Decoder.decode(MysqlConnectionManager.java:177)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:113)
	...

Then you get a future which contain the error. However, the stack trace of the error does not contain any hint where in the code the query was issues. It just contains a stack trace with the internal event loop:

You can use the debug flat to get a stack trace which points at the location which sent the request. Either add -Dorg.adbcj.debug=true to the JVM flags. Or set the appropriate property when crating the database access. Note that this feature is quite expensive, so I recommend to use it while developing or during an investigation.

So, that’s it for now. Future posts will go into more aspects of ADBCJ the interaction with Scala.

I’ve been continuing work on this, mainly on the basics, the ADBCJ driver implementation . Here’s a overview of the updates:

• Simple connection pool: This helps to avoid the costs of creating connections. It also gives a nice control how many connections are used.
• H2-Support: Implemented a driver for the H2 database
• MySQL Driver Restructuring: The driver has been massively simplified, mainly in term of synchronization. This makes it easier to debug and reason about it.
• DBFuture implementation moved to CAS operations instead of locks: Less overhead, simplifies implementation and avoids potential dead locks
• Dropped PostgreSQL driver for now: I never maintained that driver, so I dropped it for now. If someone has interrest in a PostgreSQL driver, talk to me.
• Countless bugfixes

H2? What? Ah, the database.

Connection Pool

In the JDBC world connection pools are the standard. And this has it’s reasons. Connection to a database needs some initial work. With a connection pool this overhead is avoided. Plus it gives you control how many connections are used. ADBCJ has now a connection-pool too.

First you need to add the pool implementation to your project:

libraryDependencies += "org.adbcj" % "adbcj-connection-pool" % "0.5-SNAPSHOT"

After that you add the prefix pooled to the connection string. This will load the ADBCJ connection pool. For examples:

ConnectionManager plainMysql 
      = ConnectionManagerProvider.createConnectionManager(
            "adbcj:mysql://localhost/adbcjtck", "root", "");

ConnectionManager pooledMysql 
      = ConnectionManagerProvider.createConnectionManager(
            "adbcj:pooled:mysql://localhost/adbcjtck", "root", "");

The connection pool can be configured by providing properties for it. Here are the supported properties with the defaults:

Map<String,String> poolConfig = new HashMap<String, String>();
poolConfig.put("pool.maxConnections","50");
poolConfig.put("pool.maxWaitForConnection","500");

ConnectionManager pooledMysql
  	= ConnectionManagerProvider.createConnectionManager(
  		"adbcj:pooled:mysql://localhost/adbcjtck",
		"root",
		"",
		poolConfig);

H2-Support

I really like the H2 database. It’s simply to handle, embed, run and operate. I often use it for smaller projects, testing, default embedded database and so on. So no wonder, I’ve added support for it. You can add it via Maven/sbt.

First you need to add the driver to your project:

libraryDependencies += "org.adbcj" % "h2-async-driver" % "0.5-SNAPSHOT"

And then you can use the driver:

// Plain connection
ConnectionManager h2 
       = ConnectionManagerProvider.createConnectionManager(
       	  "adbcj:h2://localhost:14242/adbcjtck;MVCC=TRUE",
  	  "sa",
	  "sa");

// With connection pool, recommended
ConnectionManager h2Pooled 
       = ConnectionManagerProvider.createConnectionManager(
       	   "adbcj:pooled:h2://localhost:14242/adbcjtck;MVCC=TRUE",
	   "sa",
	   "sa");

Restructuring Synchronization of MySQL driver

The synchronization of the requests and running operations in the MySQL driver was quite complex. This caused a lot of head aches fixing race conditions and finding errors. I’ve rewritten the internal synchronization of the MySQL driver to be simpler and more understandable.
Another side effect is, that the driver now pipelines request much more aggressively. This should improve performance in some scenario drastically. The cost of this is that most requests cannot be cancelled anymore for now.

Source, etc.

The source is still on GitHub (https://github.com/gamlerhart/adbcj). Also remember that I’ve a improved Scala-API here.

The stuff is on my Maven-Repo:
https://github.com/gamlerhart/gamlor-mvn/raw/master/snapshots

Group-ID and Version for all ADBCJ projects:
GroupID: org.adbcj
Version: 0.5.0-SNAPSHOT

API:
ArtifactID: adbcj-api

Mysql:
ArtifactID: mysql-async-driver

H2:
ArtifactID: h2-async-driver

Connection-Pool:
ArtifactID: adbcj-connection-pool

That’s it for now =)

Testing Collections, Ahoi

The Google Guava Testlib brings a whole test suite which tests the behavior of Set, List and Map implementations. You can specify which features your collections support and then profit from the hundreds of tests Google has written for you.

Getting Started

First we need to get the library. The easiest way to get it is with Maven:

<dependency>
	<groupId>com.google.guava</groupId>
	<artifactId>guava-testlib</artifactId>
	<version>13.0</version>
	<scope>test</scope>
</dependency>

After that we can start to write a test suite. There are test-suite builders for this purpose, the most important ones are ListTestSuiteBuilder, SetTestSuiteBuilder and MapTestSuiteBuilder. These builder allow you to create a test-suite according to your needs.
First you specify the collection to be used and the test data with the using-method. Afterwards you can specify what features are implemented by your collection. Here’s an example:

Take a good look into the available constants in CollectionFeature, ListFeature and MapFeature for the features sets to test.
Anyway, BAAAMM! Now I’ve already a huge suite of test running, which cover a lot of dirty details. For example my implementation seems to have issues with the equals method of the returned sub-lists:

Oh Oh

Setup/Tear-Down, Suppressing Tests

Now the actual tests do run in already written test classes. So what if your collection needs some setup- / tear down code? For example I use the test-suite to test collections which are backed up by a database. That means I need a setup- / tear down phase.
This can be done by providing a runnable for the setup and tear down:

You also can suppress very specific methods of the test suite. This is done by giving the methods which should be suppressed:

Your Own TestData

Perhaps your collection implementation can only hold very specialized elements. In that case you might need to generate the test-items yourself, which are added to the list. Instead of using the Google test data provider, you can use your own. Here’s an example, which tests the collection with a custom type:

Other Types

Above I’ve shown examples for a List implementation. For Sets and Maps it works more or less exactly the same way. Instead of ListSomeThingSomething the classes are called SetSomethingSomething or MapSomethingSomething. Otherwise the same principals apply.

Conclusion

If you ever implement your own Java collections, take a look at the Guava Test lib. It saves you a ton of work!

I just recently started to look into some J2EE stuff. Still feels super scary. Don’t want to say much more about it. Yes,  J2EE got a lot better, but is still a giant monster. And don’t worry, there are tons of other monsters out there which are as scary as J2EE.

Misfits 3 (Wikipedia, Trailer)

I’ve already covered Misfits here. And Misfits it still damn good, I really can recommend it. The third season is maybe even the best one. I especially love the Zombie episode. If you like fantasy series, and haven’t tried Misfits yet, give it a watch.

Star-O-Meter: (4.5/5) (Yes, I even slightly increased it).

Zombies! Run!

It’s Always Sunny in Philadelphia (Wikipedia, Trailer)

May people really like this sitcom. Therefore I’ve watched one season. However I didn’t really like it. Usually I enjoy black comedy. This series is for me just average, but not great. So I stopped watching after the first season.

Star-O-Meter: (2.5/5)

You don’t wanna go to this bar.

Psych (Wikipedia, Trailer)

In this series a guy who can remember anything pretends to be a psychic. With his abilities he help the police solving difficult cases. Psych is a humor-full detective series, which I quite like. When it’s boring this series is nice to kill a hour or two.

Star-O-Meter: (3/5)

There are no ghosts, but big lies

Pretty Little Liars (Wikipedia, Trailer)

After a young woman dies, her 4 friends receive strange texts on their phones from a unknown person. These message contain secrets and lies of them. Who sends those messages? Who does he or she know all secrets? The plot sounds interesting, but the series isn’t. I found it rather boring.

Star-O-Meter: (2/5)

Hello. Hello? Hello??!!??

Mad Man (Wikipedia, Trailer)

Mad Man is boring. Don’t know why people like it.

Star-O-Meter: (1.5/5)

In Mad Man people really don’t work

Life Special Investigation Team (DramaWiki)

Life Special Investigation Team is another of these countless crime stories series. This time about insurance fraud. It’s alright. However it has really strange poop/fart ‘jokes’ in there. The aren’t even jokes, just strange scenes.
Star-O-Meter: (2.5/5)

People go crazy because of insurance money

Dream High 1 and 2 (Wikipedia, Trailer)

I really like Glee . Dream High is a series about music, so I expected something like Glee. However Glee is way better than Dream High. The first season is alright, but not great. The second season is the worse than the first one.

Star-O-Meter: (2.0/5)

Dreaming

When It’s At Night (DramaWiki)

In this series Heo Cho Hwi searches for old national treasures to then put them in museum. I really like the first episode, which is quite awesome. However after the first episode it gets more and more boring.

Star-O-Meter: (1.5/5) (However, for the first episode 3.5/5)

Catch the thief!

Mary stayed out all Night (Wikipedia)

I first though this drama revolves around some kind of marriage frauds. However it’s about an arranged marriage and love and so I lost interest. And the lead actress looks like a 14 old girl and not like the 20-25ish she’s supposed to be. Totally miss casted for this role.

Star-O-Meter: (0.5/5)

A fraud marriage is fun

Secret Garden (Wikipedia)

o.O, WOW, this thing is horrible, horrible, horrible. I cannot believe how bad this thing is (In fact, I watched the whole thing, just to see if it stays that bad). The story is a mess, I hate every character in it, the story makes no sense. Only Ha Ji Won’s character is alright. I probably could rage an hour about how bad it is.

Star-O-Meter: (0.0/5)

The secret garden is in the secret forest.

Office Girls (Wikipedia)

Office Girls is also bad. Especially the music is obnoxious, it always the some crappy song. And also the sound mix is horrible. The volume between voices, music and background is inconsistent and often ill balanced. Only the comic relief characters Liu Yu Le(劉禹樂) and Shi Ta Long(史特龍) are a little bit funny.

Star-O-Meter: (0.0/5)

Yep, who want’s to work?

Korean Peninsula (Wikipedia, Trailer)

In this series North- and South Korea together try to research on a new fuel source. However this dangerous project suffers an terror attack and all people are having a hard time.
I found this thing boring and stopped watching it quickly. (Maybe it get more interesting later on?)

Star-O-Meter: (1.5/5)

Three, Two, One….BAAAAAAAAMMMMM!

Conclusion

I watch to many TV series.

Racing Stuff

Good Benchmarking, Science for itself

Good benchmarking is a small science for itself. Creating a balanced benchmark, ensure equal conditions for multiple runs, cranking out usable numbers, do proper analysis etc. I didn’t do that. I just wrote some code, ran that stuff and compared the numbers.
In the end it heavily depends on your scenario anyway. For example when you run trivial queries against a local database, then JDBC will probably win, since it only blocks a thread very briefly. If you add web-like latencies to the database communication, then my ADBCJ solution has an huge advantage.

The Benchmark Scenario

This is basically the benchmark scenario which I measured. It simulates a web like application.

• A request comes in
• We query for the most important data. Also we query for some less important data. The more important data is returned asynchronously to the client first, while the other queries are still running. This should simulate a page, which first shows the important stuff and loads the rest in the back ground, like many websites do.
• When the first part of the result arrives at the client, another request is sent, which updates some data
• When all queries, updates and operations complete, we’re done. We measure the time between issuing the request and completing everything

For the plain JDBC implementation I used the Akka future API, to run that work in the background and continue with the other operations. The ADBCJ implementation relies on its asynchronous API.
For these number I ran the database locally, so there is no huge round trip overhead. Only the queries need a bit of time to complete, from sub milliseconds up the around 6 milliseconds.

Result, JDBC vs ADBCJ

Well ADBCJ is as expected faster for this task. Mainly because it can send new queries, while processing the in streaming result of previous queries. Maybe the JDBC implementation also ends up exhausting the thread pool with blocked threads, but I didn’t check it if that’s the case:

Here’s also a box plot of the data. There is quite a high variance in the data, and a few latency spikes:

JDBC vs ADBCJ

JDBC with Connection Pool, ADBCJ via JDBC bridge

Now this is a little unfair, since you usually use a connection pool for JDBC. And we didn’t check out the ADBCJ via JDBC fallback driver. So I ran the benchmark as well with the BoneCP connection pool and ADBCJ via JDBC.
Guess what, JDBC with a connection pool reacts the fastest. This means that a connection pool has a huge effect in this benchmark, since it opens and closes database connection often. The conclusion is that ADBCJ desperately needs its own connect pool implementation, to handle similar scenarios.
Another interesting fact is that ADBCJ via JDBC performs the worst here. It’s clear that it is probably a bit slower the JDBC version, because I manually split the JDBC calls reasonably across threads. The ADBCJ to JDBC bridge has no insight how to smartly dispatch the blocking calls. However the huge difference between JDBC and ADBCJ via JDBC is surprising. This could indicate that there are some implementation issues present in the ADBCJ to JDBC bridge.

Again a box plot of this data. Again high variance and quite a few spikes:

JDBC Connection Pool, ADBCJ via JDBC

High Load Performance

Well, I wanted to see if the picture doesn’t change if the load is much higher. So I cranked the executed request up, so that the machine is fairly busy. The database is still running locally. The picture doesn’t change much, maybe there is a tendency that the performance advantage of ADBCJ gets bigger. But I didn’t to more scaling testing to verify that.
Also the ADBCJ to JDBC bridge crashed, because it just issued way to many connections. Well, as mentioned, the ADBCJ to JDBC bridge is more testing facility anyway.

And again in a box plot:

Higher Load: JDBC vs ADBCJ

Conclusion

Well, don’t draw to many conclusion from these numbers. We certainly can see that ADBCJ can be faster for some scenarios. We also can clearly see, that the missing connection pool for ADBCJ is a huge performance penalty for many applications.

슈퍼 퍼워로 뭘 하기 몰라요

I really like The Fades and think that it is fun. The story is good, the actors and special effects decent. I really like the atmosphere and tone, especially the horror movie like parts. There one thing I didn’t really like, and that’s Paul, the main character. His so insecure and he’s a whiny person. He never got my sympathy. Also I never could believe that Paul and Mac are geeks. Mac’s movie quotes are too old and out of place, to believe that his character is a teenager.

Anyway, I enjoyed the Fades, so I can recommend it if you like this kind of show.

Star-O-Meter: (4/5)

And here’s a little trailer: