Comments for ostinelli|net

Comment on Boost message passing between Erlang nodes by Roberto Ostinelli

Roberto Ostinelli — Fri, 12 Jun 2009 08:14:37 +0000

i completely agree. this benchmark test is very specific.

i will need to reperform the local benchmark, as i prefer not bugging the otp team without being 100% sure of the results i’m experimenting. asa i get to have some time, i will. i’ll get back to you then to know who to contact :)

Comment on Boost message passing between Erlang nodes by Ulf Wiger

Ulf Wiger — Thu, 11 Jun 2009 11:45:51 +0000

@Roberto: Those were interesting numbers. One perpetual problem with benchmarking is that you have to gauge how close to reality your test comes. In your test, you begin by spawning 200,000 processes that all get into line to send a distributed message. This will probably result in an unusually great propability for lock contention, at least compared to most ‘normal’ erlang applications.

The local case is interesting. You should probably let the OTP team know. My guess is that they have verified with their own benchmarks that local message passing does *not* deteriorate with each new release, but you may have hit some corner case that their benchmarks don’t cover.

Comment on Boost message passing between Erlang nodes by Roberto Ostinelli

Roberto Ostinelli — Wed, 10 Jun 2009 18:16:28 +0000

ok. still it beats me why qr is actually performing so well.

i would just like to understand where i should/should not use the qr mechanism, since as i already said - it is, after all, quite weird to implement a queue mechanism at a higher level than tcp.

i am far from having the insights and experience you have in erlang; thus again, thank you for your feedback, it is warmly welcomed.

Comment on Boost message passing between Erlang nodes by Roberto Ostinelli

Roberto Ostinelli — Wed, 10 Jun 2009 18:10:08 +0000

hi ulf, i had seen the readme of the release, and i said to myself this was perfect timing. when you wrote i just had erlang R13B01 installed and was starting to perform the test :)

here are the results. again, on my macbook with leopard.

1. to my surprise, sending speed of messages within the same node has consistently decreased. it is now 3.6 million/min instead of 5.3.

2. speed between two erlang nodes are similar, however setting the dist_nodelay option to false now produces far better results, bringing message passing between erlang nodes up to 1 million [instead of ~810K].

3. qr still beats everything: 2.3 million/min between different erlang nodes.

Comment on Boost message passing between Erlang nodes by Ulf Wiger

Ulf Wiger — Wed, 10 Jun 2009 18:02:51 +0000

@Roberto: I’m not at all sure that qr is worse on multicore. All messages are going out on the same port eventually anyway, and sending to the same process gives little lock contention as appending to the message queue is one of the operations most optimized in the Erlang VM - and a very small critical section. Sending to the port (until R13B01), the critical section actually includes the term_to_binary and updating of the atom cache, as well as the actual send to the port. This means that the risk of lock contention is much higher. It’s the contention that’s costly, not the actual locking.

Comment on Boost message passing between Erlang nodes by Ulf Wiger

Ulf Wiger — Wed, 10 Jun 2009 17:57:15 +0000

In an unusually timely fashion, OTP has just released R13B01, which, among other things, delivers significant speedups on port communication, e.g. Distributed Erlang. Specifically, it dramatically reduces the lock contention on the port.

It would be interesting to see how R13B01 affects this benchmark.

Comment on Boost message passing between Erlang nodes by Roberto Ostinelli

Roberto Ostinelli — Wed, 10 Jun 2009 13:39:30 +0000

thank you ulf and richard for these precious insights. ulf, unfortunately i believe that it’s already too late for me, as my brains seems to be thinking in slices too :)

still, i still haven’t had a grab on:

1. why qr is actually working [or if it's working only with such a benchmark];
2. what should i beware of when using qr.

to my understanding up until now, qr could be particularly costly on multi-core machines. is this correct? this alone could give me enough reasons to think again before using qr.

fact is, i’ve designed a multi-node architecture from the start [i.e. not based on clusters] and i would really like to find a “best practice” way to get the speed between nodes as fast as it can be.

i’m even considering developing an alternative carrier, but this seems to be a really complex and overwhelming task.

again, thank you for any insights you feel like sharing.

cheers,

Comment on Boost message passing between Erlang nodes by Ulf Wiger

Ulf Wiger — Wed, 10 Jun 2009 13:23:58 +0000

…what’s more (my brain seems to schedule in slices, just like Erlang does - beware: this may happen to you too, in time!)

There are good reasons to make ! fully asynchronous in SMP, while it isn’t in non-SMP. There’s a tiny backoff control that checks the length of the receiver’s inbox, and punishes the sender if it’s too long. This is *very* costly on many-core. Similarly, sending to a port is a blocking operation, which favors solutions where only one process sends to the port. Perhaps this too should be made fully asynchronous…? This is easier to accomplish for Distributed Erlang than e.g. for gen_tcp, since it would break the gen_tcp API unless it is extended.

Comment on Boost message passing between Erlang nodes by Ulf Wiger

Ulf Wiger — Wed, 10 Jun 2009 13:15:57 +0000

…commenting on my own comment above, of course in a multicore world, it is by no means certain that this invalidates the benchmark, or the utility of qr, much like ets has been a popular optimization on single-core, but is much less attractive on SMP due to the low-level locking.

Comment on Boost message passing between Erlang nodes by Ulf Wiger

Ulf Wiger — Wed, 10 Jun 2009 13:12:12 +0000

@Richard: Actually, since the send() function spawns processes in a tight loop, one could imagine that there will indeed be lock contention on the distributed send. Spawn is non-blocking, and returns as soon as it has inserted the other process in the ready queue. It’s therefore likely to spawn a great number of processes before it yields, e.g. due to context reductions.