So, today Erjang made it's fist public appearance [read: live demo] at one of Trifork's open geek nights, and I talked a little about Erjang performance towards the end. So here are some initial numbers.
I have been reluctant to publish numbers because it is really difficult to say anything when the implementation is incomplete. But I think that for these tests I have the machinery and error checking in place; so as far as I can see there should be no extra error checking or the like that will be needed going forward. Luckily, I have lots of ideas for how to improve performance. Erjang is still very young.
Computing Fibonacci Numbers
The first test is a tail recursive fibonacci. The code is Fast tail recursive fibonacci, and also in the github repo here. The version I'm using runs a series of fibonacci computations three times, and I take the result of the third run. On a JVM, there is a significant gain in letting the virtual machine "warm up". On the BEAM VM, the numbers are very consistent.
The x-axis is simply the input to the fibonacci function; y-axis is the log of the running time. You can see that Erjang catches up with BEAM on fib(N), when N gets large. As numbers get higher, time is dominated by big number multiplication, and less so by erjang/erlang proper.
For applications that have some non-trivial Java code, this is a good outlook. The more "plain java" code in the application, the more the VM can optimize. Such as the implementation of bignums, which in Java's case is written in pure Java.
This JVM profile shows how over all computation time is dominated by bignums.
Flat profile of 12.31 secs (157 total ticks): Thread-0
Interpreted + native Method
94.8% 73 + 0 java.math.BigInteger.multiply
2.6% 2 + 0 java.math.BigInteger.subtract
1.3% 1 + 0 erjang.m.erlang.ErlBif.eqxp
98.7% 76 + 0 Total interpreted
Thread-local ticks:
51.0% 80 Blocked (of total)
1.3% 1 Class loader
So it's not really a test of Erjang. The only visible erlang piece in there is ErlBif.eqxp which is short for =:= used as a guard. Hmm... Looking forward to have enough there to see some real tests.
Next Up: "The Ring"
The other interesting example is how Well Erjang does on context switch. Sriram Srinivasan and Alan Mycroft published these numbers at ECOOP'2008, so I had my hopes high.
With the various overheads imposed by implementing Erlang semantics, numbers ended up being a little disappointing. We're only at-par. The numbers below represent this ring example from Alex Miller's blog, actual used code is here at github
Running 10,000 cycles in a 10,000-process ring, total of 100,000,000 messages (each with a context switch) takes ~120 seconds, for both BEAM and Erjang on my MacBook Pro. Over time, Erjang seems a little faster as the JVM warms up.
Actual run times here [BEAM]
2> ring:main().
{1263,330690,192865} Starting message, cycles=10000
{1263,330812,106399} Around ring 10000 times
10000 == 10000 ... stop
Start={1263,330690,192952} Stop={1263,330812,106697} Elapsed=121913745us
{1263,330812,193086} Starting message, cycles=10000
{1263,330955,949775} Around ring 10000 times
10000 == 10000 ... stop
Start={1263,330812,193298} Stop={1263,330955,950028} Elapsed=143756730us
{1263,330956,39000} Starting message, cycles=10000
{1263,331078,695922} Around ring 10000 times
10000 == 10000 ... stop
Start={1263,330956,45864} Stop={1263,331078,696187} Elapsed=122650323us
{1263,331078,770597} Starting message, cycles=10000
{1263,331225,267266} Around ring 10000 times
10000 == 10000 ... stop
Start={1263,331078,770701} Stop={1263,331225,267941} Elapsed=146497240us
{1263,331225,341222} Starting message, cycles=10000
{1263,331370,629837} Around ring 10000 times
10000 == 10000 ... stop
Start={1263,331225,341327} Stop={1263,331370,629947} Elapsed=145288620us
ok
and Erjang [io:format is not really implemented, in this output I just print out the arguments]:
io:format "~p Starting message, cycles=~p~n" [{1263,329908,204000},10000]
io:format "~p Around ring ~p times ~n" [{1263,330069,516000},10000]
io:format "~p == ~p ... stop ~n" [10000,10000]
io:format "Start=~p Stop=~p Elapsed=~pus~n" [{1263,329908,497000}, {1263,330069,516000},161019000]
io:format "~p Starting message, cycles=~p~n" [{1263,330069,673000},10000]
io:format "~p Around ring ~p times ~n" [{1263,330194,857000},10000]
io:format "~p == ~p ... stop ~n" [10000,10000]
io:format "Start=~p Stop=~p Elapsed=~pus~n" [{1263,330069,693000},{1263,330194,858000},125165000]
io:format "~p Starting message, cycles=~p~n" [{1263,330194,915000},10000]
io:format "~p Around ring ~p times ~n" [{1263,330309,547000},10000]
io:format "~p == ~p ... stop ~n" [10000,10000]
io:format "Start=~p Stop=~p Elapsed=~pus~n" [{1263,330194,927000},{1263,330309,548000},114621000]
io:format "~p Starting message, cycles=~p~n" [{1263,330309,910000},10000]
io:format "~p Around ring ~p times ~n" [{1263,330450,834000},10000]
io:format "~p == ~p ... stop ~n" [10000,10000]
io:format "Start=~p Stop=~p Elapsed=~pus~n" [{1263,330309,922000},{1263,330450,834000},140912000]
io:format "~p Starting message, cycles=~p~n" [{1263,330450,867000},10000]
io:format "~p Around ring ~p times ~n" [{1263,330567,340000},10000]
io:format "~p == ~p ... stop ~n" [10000,10000]
io:format "Start=~p Stop=~p Elapsed=~pus~n" [{1263,330450,886000},{1263,330567,340000},116454000]
The Erjang numbers are for a configuration of a size-1 thread pool. If I increase the pool size, then a congestion problem shows up in Kilim's scheduler; here is a snapshot from jprofiler. The more threads I add, the worse the problem.
So this seems like something that can be improved. It looks like there is allocation going on inside a synchronized block, and that is never good. But at least somewhere to look to improve it; an ideal project for a new contributor? Let me know.
Conclusions?
So ... it looks like we are on-track to match BEAM on performance. We will need much more info than this to really assert overall performance. Micro benchmarks are not really a good indication of application performance, but it looks like we're in the right ballpark.
One thing that needs to be looked into is also memory usage. It looks like I use significantly more memory than BEAM, which does not surprise me, but is an issue of cause. I think that a good chunk of the memory is used by .class data, and so it might make very good sense to write a byte code interpreter; and then only generate java code for "HotSpots". If so, then we could do even more aggressive compilation and inlining.
Cheers!
Update. @hilmer wrote up this nice post