StatsD is a NodeJS daemon, listening for udp message. It parses the message received, get the metrics and then flush it at periodic times. Its the perfect tool for monitoring anything like your own application metrics, without any overheads on your servers or your coders for starting using it.

StatsD benefits on other systems ?

• Super  easy on client side: the client just send udp packet with a simple text protocol. No configuration needed, you just have to know the destination ip and port. You have client implementation on ruby, js, python, c etc…
• Super light on client side: udp is fire and forget. On heavy loaded machines, you have a very very low overhead.
• Manage data on server side: keep track of counters for you.
• Trivial api: you deal with counters, timers and gauges. thats it !
• Plug to other system like graphite: statesd don’t graph anything, but it natively supports graphite backend to flush data for graphing. I’ve also implemented a udp proxy backend in node, sending every datas to a eventmachine websocket broker: easy !
• Business / Application ready api: just perfect for tracking db response times, user interactions on frontend, temperature, api response times, login errors / success etc…

Why i use StatsD and Graphite

• i use StatsD because its the lightest solution in term of installation and code integration.
• I use graphite because with graphs like munin we are very far from the real time (1 or 5 minutes) and with graphite we can really customize dashboards: its perfect !

Code examples (client side)

I  use statsd-ruby for my application. With this lib, code looks like this.

Graphite

Graphite performs two tasks:

1. Storing numbers changing over time
2. Graphing them

More infos on StatsD

The StatsD code and doc at github.

StatsD is develop by the start-up Etsy: here is their blog post about this tool.

Streams and topology

• stream a continuous sequence of tuples
• spouts a source of stream
• bolts stream transformation

Storm guarantees that every message flowing through a topology will be processed, even if a machine goes down and the messages it was processing get dropped. How Storm accomplishes this without any intermediate queuing is the key to how it works and what makes it so fast.

Benefits of storm

After all, what benefits toward zeromq and some hand made code ? This tricky contraints are managed:

• Guaranteed message processing
• Robust process management workers are monitored
  
• Fault detection and automatic reassignment
  
• Efficient message passing zeromq rocks !
  

More infos:

http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html

My pres at Barcamp Bordeaux 2012. HTML5 is not only markup. We have a lot of improvements  in the js API with a deep impact on architecture of tommorow web applications.

Do you know graylog2 ? Its big data for your logs (mongodb + elasticsearch + ruby on rails).

I’ve setup a syslog-ng server centralizing  about 20 servers logs (from syslog). Then pipe it to a ruby eventmachine broker to distribute a websocket stream and graylog2 server at same time. I will make a post about this later: this post is a teasing.

You can see in this video a realtime graph of log count of about 20 servers every 1/10 seconds. Stay tuned !

I was yesterday at the bordeaux JUG (Java user group) for the conference “Objectif Cloud” by Nicolas de Loof.

• A very goog conference about Neo4j, the graph database is incoming at the jug
• Nicolas was a contributor to maven and is a co-author of the book of “Apache Maven”. Engineer at cloudbees.
• The question: where to deploy ?
• Jenkins could be used as a “reomte control via ssh system”
• I should take a look at builder and chef….
• Cloud ?
• Elastic and fast (get server instance in minutes)
• Self service
• Shared ressources
• Standard network access (ip / http …)
• Measure service for billing (io / network io / ram ….)
• IaaS -> Paas -> Saas -> there is a big gap !
• Cloudbees product wrap a lot of stuff: do one thing one time the best way and share it
• mongodb
• couchdb
• mysql
• loadbalancer
• monitoring

i will be at the event “Objectif Cloud” by the bordeaux #java user group 10:31am via @mathieuel

Local event in bordeaux by the java user group. Each month we have a presa from a strong skills tech.

github rubybdx hello world d3.js + sinatra + elasticsearch cc @rubybdx http://t.co/UIb7h1qU 11:36am via@mathieuel

This is the source code for my demo / presa at ruby bdx, the ruby user group in bordeaux.

#fr my pres at rubybdx screencast / vidéo #d3.js + #sinatra + #elasticsearch + #capucine http://t.co/83qbsggB10:40am via @mathieuel

My fisrt presa et rubybdx ruby bordeaux user group. I really enjoy it and i will make another one next time.

slideshare of my pres at #ruby usergroup in #bordeaux d3.js + ruby sinatra + #elasticsearch http://t.co/K5hLIRhCscreencast incoming ! 8:04pm via @mathieuel

Slideshare of my presa.

Realtime Search: Solr vs Elasticsearch – ES rocks ! http://t.co/koQy1jio via @socialcast 12:44pm via @mathieuel

More and more migrating form Solr to Elasticsearch ?

#fr Screencast / vidéo of my pres at ruby bordeaux group with a live demo at the end. Talking about elasticsearch, d3.js, sinatra and capucine.

Get the source code on github: http://github.com/rubybdx/hello-world-d3js-elasticsearch

Presentation i’ve made at the ruby user group in bordeaux / france

http://blog.evernote.com/tech/2012/02/23/whysql/

Evernote think ACID properties of SQL database is more suitable than double checks on “code side”.

Data is cleanly sharded by users: all of data related to a user is kept inside a node. Disk are added when user data grows in a single node, and all the new users account creation are routed to a new pool of servers.

But Evernote could go to NoSQL on another features, offcourse !

http://www.woothemes.com/support/wootumblog/

Your wordpress like a tumblr !