Last night I launched a new player for the ChillOut Everyday page. It is very simple and stylish and you can find there dozens of good music non-stop. More players with other genres will be launched soon.

Last weekend I went to OPorto, Portugal, to participate in the Hackafone: a mobile application contest where participants have to brainstorm, create an idea and produce it in 18 hours. The two winners would win a Samsung Galaxy Tab 10.1 and, above all, the app will be marketed by Vodafone in their app store and communication channels.

The event was hosted in the Geeks Lab of muchBeta, a web creative saas power house, located in Afurada, near the Douro River; p.s. great view!
The app was created with @ricardjorg and @luismvaz. We are going to finish it until the end of January 2012 and you can expect it in an Android, iPhone and iPad near you. Following is a sneak preview of the splash page;

Thank you muchBeta team, Vodafone for believing in new talents, the other hackthletes and my team.

Pictures from the event

Live Stream Videos: http://www.ustream.tv/channel/hackafone

Media News:

http://abertoatedemadrugada.com/2011/11/hackafone-vodafone-muchbeta.html

http://economico.sapo.pt/noticias/descubra-os-projectos-portugueses-que-vao-para-silicon-valley_131855.html

http://www.pordefeito.com/2011/10/25/hackafone-na-afurada-valley/

Read the full Report

The objective for this system is to give hourly social metrics like page fans, page views, posts shares, from Facebook, but also how much likes, shares and tweets the posts from several websites received. This would enable, later, the visualization how page posts, that link to the website content, influence the ecosystem.

MySQL Approach

I first modeled a simple schema: a table for pages, links and snapshots. A snapshot is an entry with the pageId, metric type, datetime and the value. Every 15 min a cron job would run and go through a list of pages, take their fan pages and other metrics, run trough a list of links, feed from several rss, and collect their shares and tweets. This is the social timeline of a link:

Even tough this analytics system was design to hold only a few thousand metrics a day, which MySQL, can handle pretty easily I would like to perform some data analysis from it. One idea was to calculate which was trending in the last X hours, X days. To do this I would simply compare the current value of shares/tweets with the value from X hours ago. Calculate the differences, keep it in an array, order it and return the top 10 links. The problem was to return the correct snapshots. I would have to do two queries: one for the current values (latest snapshot of each link) and other to get the values of the past X hours. This could get tricky because I was not sure when the snapshot would have been taken – the cron job can take less or more time to run and span through different minutes – and need to search for large time spaces.

Other problems involved summing up all tweets in a specific hour or day, retrieving top posts of the day, etc.

Using Redis

Redis is, simply explained, a key/value datastore. You tell him to save the value “bar” in the key “foo”.

redis> SET foo bar
OK
redis> GET foo
"bar"
redis> GET dne
(nil)

Imagine PHP arrays, Python dictionaries, etc. Or if you are familiar with memcached its the same principle. For a good introduction to Redis check this post from Michael J. Russo. Redis offer advantages over other key/value datastores: lists, sets, sorted sets and hashes. This enable us to keep data organized in a very simple and fast way!

Modeling for Redis

When using Redis we need to focus on modeling it for the specific purpose it will be used. Its important to retain that we can only get data through the keys and never search through all the values. Its mandatory that we create several indexes. Let’s go.

To keep data about analytics and having only key/value available how do we do it? The key must contain the information we are going to look for in the future: ids, keys, etc. Here’s an example of a key: metric:page_fans:12345678:2011-05-01. In the future we will want to look for the daily page fans for the page with id 12345678  in a specific date. Easy!

redis> GET metric:page_fans:12345678:2011-05-01
41012

No complicated SQL queries or calculations. What if we want for a full month? We use the mget command, which enables us to demand several keys in one request:

redis> MGET metric:page_fans:12345678:2011-05-01 metric:page_fans:12345678:2011-05-02
41012
41210

For simplicity I just wrote two days but for a full month we would request all the days in one swoop. Simple, fast and accurate.

Managing the data

This looked pretty simple but what if we want to look for all available statistics? That’s where sets, or lists, come handy. The purpose of sets is to save several values. In the old memcached days we would have a list on our program, serialize it and then save the output in a key. If we wanted to change it? We would need to retrieve it, parse it locally, remove the last entry, insert the new one and finally save it again. Boring, easy errors can happen here, no atomic operations, etc. How to do it in Redis?

Before explaining how to do it just a quick note: in this sets we are not gonna keep the values, because then we would not know where they belong, but instead we will save the keys. This will act as “table”, a directory. Lets see the example:

redis> SET  metric:page_fans:12345678:2011-05-01    41012
OK
redis> SADD  page:12345678:page_fans   metric:page_fans:12345678:2011-05-01
1    //this is the response of Redis for true, the operation succeed


If we do this for every metric we get then to see all the metrics for a specific page we just need to get the set entries:

redis> SMEMBERS   page:12345678:page_fans
metric:page_fans:12345678:2011-04-28
metric:page_fans:12345678:2011-04-29
metric:page_fans:12345678:2011-04-30
metric:page_fans:12345678:2011-05-01
metric:page_fans:12345678:2011-05-02
metric:page_fans:12345678:2011-05-03
..


Then we just give all of these values through an mget command and we know all the values related to a post. There’s a lot to explore in this: in my application, for example, I keep metrics for days and also for hourly stats. If I want to know a whole month I request the daily values. If I want to be more precise I just request all the hours of specific day(s). The use cases of this can give another blog post.

Extra useful use case: top posts daily, weekly, ..

Now you want to know the top posts by day. You can maintain a table in the database and every time a post is updated, or receives a page view, a like or a tweet you re-calculate everything and update this table. Boring. What about using Redis again? With Redis sorted sets you can maintain a ranked list easily. Pick a key, like “top-posts-twitter”, insert the value and that’s it. Then you just need to request the top 10 entries and you get yourself a super fast ranked list. If you want to have daily lists just add the current date to the key when you perform the update and you will have different sets. The syntax is “ZADD key score member”

redis> ZADD   top-posts-twitter:2011-05-20   230   post:11111111112
redis> ZADD   top-posts-twitter:2011-05-20   450   post:11111111113
redis> ZRANGE   top-posts-twitter:2011-05-20  0 10  WITHSCORES
post:11111111113
450
post:11111111112
230
...


When inserting just give the score (page views, likes or tweets), the key (postId) and you’re done. Then just request the top 10 members with ZRANGE and you’re done.

Conclusion

I want to state clearly that all of this could be done in MySQL but some operations would be more complex, slow or simply less direct. The fact that you can manipulate the indexes, work with simple structures and perform more direct queries simplifies the whole layout of the application. This is also a work in progress to test how we can use this simple-modeled-technologies to overcome the complexity of a full modeled MySQL database. In the future I will try to expand this topic.

Further Reading

• Offical Redis.io Website
• This post was very influenced by this blog post from a Disqus Engineer.
• You can learn Redis basics in Redis, from the Ground Up.

Comments and Reactions

Also Doubts? Leave all of them in the comment box. We can chit-chat.

Since May 2011 I am working on Sanoma Media, a subsidiary of Sanoma which operates in 20 european countries and employed over 15k last year.

I will be producing web applications mainly in Facebook and other social networks. More updates in the future.

Pesquisa as tuas bandas favoritas faz “Seguir” e recebe notificações sempre que sejam agendados novos concertos. Faz do Myout.net a tua agenda.

• Utilizando a pesquisa geral poderás encontrar facilmente eventos, espaços ou artistas
• Filtra os eventos por cidade, preço ou data
• Também podes ver os próximos eventos na vista semanal
• Segue os teus artistas favoritos, espaços favoritos e utilizadores. Recebe notificações quando forem criados novos eventos
• Recebe recomendações de eventos baseados nos teus gostos
• Deixa o teu comentário em eventos e espaços
• Escreve a tua opinião e classifica espaços e eventos interagindo com a comunidade MyOut
• Faz login com o Facebook e terás automaticamente os teus artistas e espaços favoritos importados
• Divulga os teus eventos e partilha (a criação é rápida e gratuita)
• Promove o teu espaço: podes adicionar eventos, personalizar a página e atrair novos seguidores

http://www.myout.net

Read full article

Read the full article

Both tools on this post are free, will sit on your tray and won’t bother you, while maintaining your contacts and calendar synced. I’m gonna show you the easiest way to do it.

Google Calendar Sync

Download page: http://www.google.com/support/cal..wer=89955
(Direct link to download)

Installation

1. Read through the Google Calendar Sync Terms of Service, and click I Agree.
2. Follow through the Installation Options and click Install to finish the set-up process.
3. Once Google Calendar Sync is installed on your computer, the Google Calendar Sync Settings window will appear:

Just enter your email/password and click Save. If you don’t want to sync back to Google Calendar choose “1 way: Google Calendar to Microsoft Outlook”

You are now syncing Google Calendar with Outlook!

Go Contact Sync

As the name says this is the plugin that’s gonna sync your Gmail Contacts. It supports contact categories and contact photos so your contacts will look exactly the same.

Download page: http://www.webgear.co.nz/Pr../GOContactSync.aspx
(Direct link to download)

Installation

1. Follow the setup instructions..
2. After installed, choose Start > All Programs > GO Contact Sync

Just enter your email/password, make sure Sync Profile is not empty (enter any text, the program doesn’t allow it to be empty) and click Sync. Also check “Run program at startup“ .

You may be prompted by Outlook that a program is trying to access it. Grant permission to it for 5 minutes.

You are now syncing Gmail Contacts with Outlook!

Conclusion

Because Go Contact is still in beta you may some notices from the program that an error has ocurred. I found that a restart to the program solves it. I don’t know what the error means but a future release should correct the bug.

Also you are not limited to Outlook. With your Outlook synced you can keep your mobile also synced without spending money; I keep my Nokia N70 synced by bluetooth (maybe a future blog post). I just tell Nokia Suite to sync Outlook with my Nokia everytime its in range and now I have Google Calendar and Gmail Contacts on my pocket 24/7.

Questions/Problems

If you have any questions or problems just reply on the comments.