ORMBattle.NET Blog: ORM benchmarks & news

Link: Devart dotConnect Team writes about ORMBattle.NET and its test suite

Administrator — Fri, 13 Aug 2010 19:37:00 +0000

"ORMBattle.NET have recently updated their results of testing various .NET Framework ORMs and added LinqConnect to their lists. LinqConnect demonstrates one of the best results it their tests.

ORMBattle.NET is an open-source project for ORM comparison, supported by a development team from X-tensive.com. This project contains a set of tests for feature-oriented comparison, but it is more focused on the performance tests (CRUD operation performance, object materialization, etc.).

LinqConnect development team agrees with most of the tests. The results on the whole are reliable enough to consider them when choosing ORM framework.

We already are studying the situations where LinqConnect showed bad results and working on performance improvement."

Original post: http://www.devart.com/blogs/dotconnect/?p=3553.

Scorecard is updated

Alex Yakunin — Fri, 30 Jul 2010 12:57:00 +0000

It finally happened! Almost 9 months has passed since last scorecard update, and now ~~the baby is born :)~~ you can see the newest results.

Most important changes:

Test suite was migrated to .NET 4.0.
All tested ORM tools are updated to their most recent versions. New versions are mentioned on their description pages.
There is a new player, Devart LinqConnect ORM, showing perfect results right on the first round.
LightSpeed ORM is removed from this test suite by decision of Mindscape.
Finally, all the tests were performed on 64-bit Windows 7.

Brief overview of new results:

Most of players show nearly perfect LINQ test results, so I suspect they're quite comparable from this point now. All the typical cases are fully covered. I hope ORMBattle LINQ test played a noticeable role in this process.
There are no changes in LINQ test pass results for Subsonic (that's predictable - the project frozen for at least one year) and NHibernate. The case with NHibernate is a bit more complex: currently we test its latest stable version (v2.1.2 GA), although NHibernate 3.0.0 alpha1 is already available. But we were unable to update our tests for it (see the commit comment here). Thus any help in resolving this issue is appreciated.
BLToolkit is still definitive leader in performance tests; all the other commercial players are quite comparable there now. As I hope, that's partly because of this test. There are no changes in NHibernate and Subsonic results here.
Lots of people were interested in EF 4.0 results: EF team has improved LINQ support (they got additional 5% here), but its performance results are noticeable worse - from 10 to 20% in most of the cases, and nearly twice - on materialization test. So if earlier it was one of leaders on this test, now it's closer to losers. If you know any possible reasons of this, please describe it in comments to this post.
New LINQ to SQL shows much worse performance on CRUD operations. Please let know us, if you know the reason \ possible fix - the difference looks quite suspicious; probably, there is some mistake in test (this could appear e.g. because of change in default behavior).
Our own ORM, DataObjects.Net, also shows some decrease in materialization performance. The only reason I can imagine here is lack of almost any profiling of this part during last year. So hopefully, we'll improve our own results soon.

That's all for now. Since there are few suspicious results and upcoming upgrade to NHibernate 3.0.0 alpha1, the next update is near.

Enjoy the new scorecard.

Kind regards,
Alex Yakunin

"Can ORM make your application faster?" series

Administrator — Mon, 29 Mar 2010 10:04:00 +0000

In March I started to write a series of posts called "Can ORM make your application faster?". 6 posts are already there, although the whole sequence isn't finished yet (planning to do this in two weeks). The topic is perfectly fitting to this website, so I sharethe link here.

I'd appreciate if you could give me more links to similar articles (~ optimization techniques used by ORM tools). Please leave them in comments - I'll share the most interesting ones (or all) in this blog.

Repository at Google Code is migrated to Mercurial

Alex Yakunin — Wed, 17 Mar 2010 03:09:00 +0000

We decided to do this before making further changes. So to checkout the code now, you should run the following command:

hg clone https://ormbattle.googlecode.com/hg/ OrmBattle

Alternatively, you can use TortoiseHg GUI.

I'm back ;)

Administrator — Sat, 06 Mar 2010 06:10:00 +0000

Hi everyone!I

I didn't post here for pretty long time - we worked really hard on update of our own product. Now this part is finished, and we're going to spend some time on improving ORMeter (yes, we renamed it - you might notice it even by new header).

What's planned:

Add collection performance tests (CRUD & simple bulk load tests)
Add few more LINQ tests - see http://groups.google.ru/group/ormbattle-development/t/6dbeb2497f3544a9
Implement HTML output mode for TestRunner. Currently it prints out text tables, and actually I hate converting them to Excel, and then - to nicely looking HTML with pictures. But if it would export HTML, relatively simple jQuery script + Google Chart API could make this manual conversion process unnecessary.
Update the scorecard there using the most recent versions of all products.
Likely, we'll do some minor changes related to design and scorecard layout.

I hope we'll be able to finish at least testing-related part of this plan in March 2010.

As usual, any opinions and ideas are welcome.

ORMBattle.NET Development Google Group

Administrator — Thu, 12 Nov 2009 17:55:00 +0000

We opened ORMBattle.NET Development Google Group for public access. So if you'd like to share an idea on how to improve the test suite, it is the best place for doing this.

Scorecard is updated

Administrator — Thu, 05 Nov 2009 15:22:00 +0000

Most important changes:

There are LINQ to SQL tests now - thanks to Igor Tkachev, the author of BLToolkit.
DataObjects.Net 4 is returned back to the scorecard. I think now this is fully honest.
BLToolkit got its LINQ tests - again, thanks to Igor Tkachev.
There are significant changes in the scorecard, but I'm going to comment this in separate post a bit later.
Our test suite now compiles and runs on any PC without any third-party tools: mainly, all you need is to download it, or checkout and compile it. More complete instruction is here. So if you'd like to add another ORM tool or develop your own test - you're welcome!

Follow to the scorecard to discover new results.

Get ready for update!

Administrator — Thu, 29 Oct 2009 10:36:00 +0000

He everyone!

We've been keeping silence for pretty long period, so time for the next update is come. I announce:

The scorecard will be updated on this week.
You'll see LINQ to SQL performance tests there - thanks to Igor Tkachev, the author of BLToolkit.
DataObjects.Net 4 will be returned back to the scorecard. I think now this is fully honest - it is not definite leader there anymore.
Likely, you will see one more new player here.
There are some significant changes in the scorecard - we know this, because preliminary tests are already performed. Btw, they prove the usefulness of ORMBattle.NET test suite in action: it really pushes ORM vendors to pay attention on LINQ implementation quality and performance.
Our test suite became quite easy to run on any PC: all you need is to XCopy (or checkout) and compile it.

I'm looking forward to see you back soon.

New ORMBattle.NET face

Administrator — Fri, 11 Sep 2009 19:08:00 +0000

The time has come - we're planning to change ORMBattle.NET image. Take a look at the pictures and leave us a comment :)

Btw, it was a good battle ;)

A honest comparison of Entity Framework and NHibernate?

Administrator — Tue, 08 Sep 2009 14:24:00 +0000

Here is the link. Check out my comments - there are plenty of them below the post. Currently the main question for me is: "Is it really true that EF is JUST 2 times slower than NH on this test, although it sends ~ 2500 queries vs ~ a single one in NH?"

Of course I admit there is either my own (it isn't fully clear what is ctx.EFCompanySet there) or Fabio's mistake. But at least for the first look it looks quite suspicious.

Anyway, EF test code is here:

using (var ctx = new EFContainer()) {
  var companies = from c in ctx.EFCompanySet select c;
  foreach (var company in companies) {
    company.Employees.Load();
    foreach (var employee in company.Employees)
      employee.Reports.Load();
  }
}

And NH test code:

IList<NHCompany> result;
using (var session = Sf.OpenSession()) {
  // My comment: why I don't see similar lines in EF test?
  // See e.g. this post.
  result = session.GetNamedQuery("NHCompany.All").List<NHCompany>(); 
}
foreach (NHCompany company in result) {
  writer.WriteLine(company);
  foreach (NHEmployee employee in company.Employees) {
    writer.WriteLine(employee);
    foreach (NHReport report in employee.Reports) {
      writer.WriteLine(report);
    }
  }
}

Since test code isn't available, I just quoted the code I found in Fabio's blog (in the first article from this cycle Fabio wrote to some other guy asking for the source code: "I’m working as consulter, if you want see the code I will happy to send you the price.", so I hardly have a chance to take a look at this beauty).

And... I feel I must quote few Fabio's sentences in the end:

"In this last weeks I saw the birth of a new Sport: Kick NHibernate." - but Fabio, in reality you devoted 3 posts to actually kick ORMBattle.NET ;)
"The next time you will see a competitions, before look at results, have a look to the driver." - no comments.
"Anyway my opinion about MDTD continuing being the same (MDTD = Monkey Driven Test Development)." - no comments as well.

P.S. If you don't know the story, Oren Eini (Ayende @ Rahien) and Fabio Maula, likely, are the most well-known criticizers of ORMBattle.NET. As well as the most known NHibernate developers.