Jean-Lou Dupont's WEBlog

Notes on Hadoop on Amazon Elastic MapReduce

2012-09-08T14:32:00.001-04:00

I am on a project which requires Hadoop for crunching millions of documents. Being new to Hadoop, I was faced with a learning curve. Here are a couple of notes on the experience.

Hadoop Streaming

Working with streaming jobs on Hadoop is straightforward enough, except when it comes to controlling how Hadoop treats the input files.

The input files I needed to process were large (100's MB). Hadoop insisted on chunking those to smaller size but unfortunately the default splitting wasn't compatible with my file type. I ended up fooling Hadoop in accepting bigger minimum split size:

-D mapred.min.split.size=10737418240

Hadoop with Custom JAR

Although one has full control of Hadoop when using custom JAR jobs, I must say I had quite a rocky ride getting my stuff to work.

First off, I needed a a way to package the dependent JAR librairies inside the main JAR. I was forced to write a custom ANT build file to do this.

The trick is to get the dependent jars in the directory ./lib for Hadoop EMR (I don't know about the standard Hadoop, sorry) to be able to include those in the Java classpath.

Second, I wanted to use Amazon S3 as both input and output for files. The trick there is to grab FileSystem objects based on the URI scheme of the files.

Path opath=new Path(outputPath);

FileSystem ofs=opath.getFileSystem(conf);

Forget a single of those and the whole job crashes and you end up paying Amazon for nothing... $100's of dollars down the drain ironing up this one.

Lego Style Software Design

2012-02-22T09:08:00.000-05:00

My company recently got the chance to work on a very interesting project (which unfortunately I can't divulge at the moment). What made the project fun to work on, aside from the product level challenges, was the fact that we were given much more architectural flexibility than usual.

Amongst the work emerged the following:

Python Functional Tools

2012-02-05T19:44:00.001-05:00

I got the chance to come by a pretty cool project recently: Python Moka. It consists of functional programming friendly implementations of standard Python dictionary and list classes. Then it struck me: wouldn't be nice to have Erlang-ish pattern matching functionality to Python?

So I crafted a small Python package scratching an itch I have had for way too long: function dispatching based on pattern matching. For those interested, here are the relevant links:

Project on Github
Main page on Systemical

Feedback welcome :)

EC2 architecture notes

2012-02-02T22:11:00.001-05:00

I've updated the home page of Systemical: you'll find a bunch of useful links to documents:

Enjoy :)

Amazon AWS tools - jldaws

2012-01-20T16:01:00.000-05:00

Today I am open-sourcing yet another project. It consists in a collection of Linux scripts related to Amazon Web Services (AWS).

The project's home page can be found here whilst the code repository is there.

My life in the Cloud

2011-12-08T14:27:00.001-05:00

I am pretty much 100% in the Cloud now. Here's how I do it.

Services

Email: Google GMAIL (with "Tasks")
Calendar: Google Calendar
Documents: Google Docs (with a sprinkle of DropBox, going away soon enough)
Music: combination of Rdio and Grooveshark
Movies: Netflix (on my Android devices, Apple TV and MacBook Pro)
Code: Github
Project Issue Tracker: Lighthouse

What I gain:

Mobility
Availability
Persistence
Version Control

I don't need to worry about loosing / forgetting my laptop for work: I just grab a spare one at work. Hence, business continuity is greatly improved over the traditional Microsoft "stack".

So, on my Linux/OSX PCs, I only need the following installed software:

Google Chrome (the auto-sync feature for bookmarks, extensions etc. is way cool :)
Eclipse (with PyDev, Google Plugin, Egit)
git

When it comes time to setting-up a complete working environment, I can turnaround one in under 1/2 hour!

Security

The first reaction I get when I expose my workflow: is this Cloud stuff secure? To which I respond with a combination of the following:

I would rather trust Google and their army of sysadmins than me when it comes to external security threats
I never write down passwords nor credit card numbers in documents. Period.
Doing business comes down to trust - being part of society requires trust
My data is more secure located in the Cloud than on my laptop / desktop : those are too easy to lift

Availability

Another common reaction is about availability. I don't know about you guys, but Internet Access is pretty available nowadays and certainly Google is. For those of you worried about this dimension, consider:

I have access through my Android mobile devices (carrier #1)
I have access through my 3G dongle (carrier #2)
I have access through my Cable Modem (carrier #3)

Yes I subscribe to 3 different carriers - I don't remember the last time I lost connectivity and was out of commission, business wise.

Software based Message Switch

2011-10-29T21:59:00.001-04:00

Media-Keys over Web-Socket

2011-08-14T09:03:00.002-04:00

My newest open-source project is a Media-Keys over Web-Socket application. The project's details can be found here.

Application
The main use-case I wanted to address with this component is the ability to control music playback of web-sites such as GrooveShark and Rdio through the keyboard media-keys.

Usage
A Google Chrome extension will follow shortly.

Using Google Sites as Content Management System

2011-05-18T13:17:00.000-04:00

I recently had to build a new web site from ground-up. One thing I didn't want to do is go off installing Wordpress / Drupal etc. on a virtual machine. As much as I love technology, I can't be bothered anymore with administrative tasks such as backups.
I decided to give Google Sites another go but I quickly was confronted with the limitations of this platform. That's when I decided to couple to great Google products: Sites and AppEngine.

View the presentation

A Service Registration Web-Service API

2011-04-15T19:33:00.000-04:00

It's been a while since I have last blogged... been very busy with my company Data-Tester Inc.

I have the pleasure to announce the immediate availability of another Web-Service hosted on Systemical : ipreg. This service allows, for example, mobile devices (such as Android) to "discover" available LAN services).
Why is this useful at all you might ask? For the curious: it boils down to limitations related to multicast support in mobile devices. For more details, read the details here.

iTunes playcount updater script

2010-10-11T20:52:00.000-04:00

I am pretty busy these days with my new job ( I am heading a start-up in the telecommunications market ) but I took a couple of hours to craft a couple of scripts.

The first script (lastfm_gettracks) downloads all of a user's song track information from Last.fm. It can be found on Pypi here. More information can be perused here.

The second script takes as input the file generated by the former script and updates the iTunes database on an OSX machine (sorry, I don't do windozes). Documentation for this script can be found here whilst package details can be found on Pypi here.

Logo contest

2010-09-09T15:34:00.001-04:00

Help me choose a logo for my new company. The logo contest on 99designs has ended and it is now time to vote!

VOTE HERE

Thanks!

Chrome Framed extension

2010-09-05T09:08:00.000-04:00

I have just published my latest Google Chrome extension: Chrome-Framed. The extension allows opening a tab with 2 columns 'frame'. This functionality enables a user 2 pages side-by-side in the same chrome window.

Enjoy!

Musicbrainz and Last.fm proxies updated

2010-08-17T20:21:00.001-04:00

Just a quick note to let users of musicbrainz-proxy-dbus & lastfm-proxy-dbus applications that new versions are available on my PPA. Notable changes include:

- Tray icon support: the main application window can now be closed and a tray icon can control them
- Help button linking to a web accessible page

Web Services available on Systemical.com

2010-06-20T21:33:00.001-04:00

I've decided to open access to some web services I built. These services are hosted on Google AppEngine and the documentation can be found here.

Google Moderator: an RSS feed for a "series"
Stackoverflow: an RSS feed for a user's statistics on http://stackoverflow.com/
Github: an RSS feed for a user's statistics on http://github.com/

I will be opening access to more services so stay tuned!

Last.fm Proxy with DBus interface

2010-06-17T09:00:00.000-04:00

I just realized I haven't made a public announcement regarding an application that constitutes a critical piece of my Rhythmbox arsenal: " lastfm-proxy-dbus ".

This Linux Gnome application serves as "proxy" to a Last.fm user's scrobbling history. The data collected through this proxy serves to populate the "playcount" field of Rhythmbox's database.

More information can be found here.

Rhythmbox Synclastfm plugin

2010-06-15T15:13:00.003-04:00

Today I announce the availability of a brand new version (2.02) of my Rhythmbox plugin "Synclastfm" (see previous post).

This version packs the much anticipated "batch update" feature i.e. the plugin will walk the user's history on Last.fm in order to update Rhythmbox tracks 'playcount' field.

More information can be found here. Have fun and don't forget to provide feedback!

Musicbrainz Proxy service over DBus

2010-05-30T23:28:00.002-04:00

I have just finished another building block on my quest to taming my music library: a Linux based application serving as 'proxy' (with caching of course) to Musicbrainz webservice. The service is exposed through DBus.

Why?
Correlating music tracks across disparate applications / services can be daunting. For example, suppose one scrobbles his/her tracks to Last.fm. Furthermore, let's say one wishes to keep his/her favorite Music Player in sync with his/her Last.fm profile. This goal appears, on the surface, to be an easy one but as it turns out, support for finding & cataloging music tracks coherently across various applications / services remains elusive at the moment.

Also worth noting: access to Musicbrainz's webservice is monitored. Webservice API access is limited to 1call/sec: if many applications make uncoordinated access to the service, the user's machine might get blacklisted. As I intend to make the track mbid a central piece of information for correlating my music tracks, I needed to address this issue.

Solution
With this application, it will be easier to integrate the various other building blocks of my music management arsenal. See the other project SyncLastfm that will benefit from this addition.

Project
The project's home page can be found here. The associated Ohloh page can be found there.

Update: the application will be installed under Applications / Other.
Update2: more information is accessible here.

Some Google Gadgets

2010-05-10T13:26:00.001-04:00

I am in the process of building up my company website based on Google Apps, I need some Google Gadgets to embed but as I am unsure about the majority of the gadgets listed in the directory, I have created some of my own.

Skype Account Information
Ohloh Profile Flair
Stackoverflow Profile Flair
Twitter Profile Flair with latest tweets

Links to the relevant gadgets are provided on my projects page. I would rather not duplicate them here as I am most likely to forget to update them whereas on the project page they'll get my attention.

Last.fm Web Service desktop interface through DBus

2010-02-08T09:20:00.005-05:00

Not too long ago, I started my quest to Music Library Faiyland (my daughter would love this quote). On this journey, I have completed a couple of intermediate steps ( Rhythmbox plugin - Syncing with Last.fm and Rhythmbox plugin for Last.fm desktop client ) and now I am happy to announce another milestone: a DBus accessible interface to Last.fm's Web Service API.

API
The current DBus interface is rudimentary but can be easily extended ( I have abstracted Last.fm's API to a manageable degree ). I will certainly be expanding the methods in the near future based on my needs and "customer feedback" ;-) Below is a non-exhaustive list of the API:

Account related:

get/set Username
getAuthUrl : retrieve a URL for authenticating the service against a user
setApiKey , setSecretKey

Track related

track.addTags
track.getTags
track.removeTag

I have only implemented these methods because they get me what I need at the moment: the possibility to "rate" my music tracks through a hack using the tags to hold the rating for a track. E.g. a 5 stars track would get "R5" as tag.

Message Based Design Technique
In this project, I have further experimented with a design technique based on "Message Passing". I have made some notes on the subject here.

Project
The home page of this project can be found on my Github repository. A Debian package can be found on my Launchpad PPA.

Notes on a Message Passing design

2010-01-29T17:36:00.001-05:00

I am an avid supporter of "message based" systems. I took the time to perform a quick "brain dump" of some of my thoughts on the subject.

Follows a simple Python based implementation.

I will very likely make other contributions to my blog along the same lines.

Message Bus for Gnome Object based Python projects

2010-01-21T20:04:00.017-05:00

Whilst developing my latest software project (Rhythmbox plugin Last.fm sync), I have crafted a "Message Bus" based on Gnome GObject. Since I didn't have any experience at all with Gnome GObject previously, it took me sometimes to work out the details of the architecture. For an example usage, please refer to my Git Repository.

""" 
    GObject based "Message Bus"
    
    @author: Jean-Lou Dupont
"""

import gobject #@UnresolvedImport

class Signals(gobject.GObject):
    """
    List of the application level signals
    """
    __gsignals__ = {

         ## NOTE: customization required here
         ## =================================
         ## Announces changes in the user's Last.fm properties
        "lastfm_username_changed":  (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, (gobject.TYPE_STRING,)) 
        ,'lastfm_password_changed': (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, (gobject.TYPE_STRING,))

        ## Used to signal a change in the currently playing track
        ,"playing_song_changed":    (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, (gobject.TYPE_OBJECT,))
        
        ## Used to report a failure when accessing Last.fm web service
        ,'lastfm_request_failed':   (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, ())
        
        ## Used for distributing the results of the query against the Last.fm web service
        ,"user_track_info":         (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, (gobject.TYPE_OBJECT,))
        
        ## Used to pass around the "shell" global object
        ,"rb_shell":                (gobject.SIGNAL_RUN_FIRST, gobject.TYPE_NONE, (gobject.TYPE_OBJECT,))              
    }

    def __init__(self):
        gobject.GObject.__init__(self)
        
        
class Bus(object):
    """
    Message Bus
    
    Borg Pattern
    """
    _signals=Signals()

    @classmethod
    def emit(cls, name, *pa, **kwa):
        cls._signals.emit(name, *pa, **kwa)

    @classmethod
    def add_emission_hook(cls, name, callback):
        gobject.add_emission_hook(cls._signals, name, callback)
    
mbus=Bus()

## =============================================================== Tests

if __name__=="__main__":

    def callback(signal, data):
        print "callback: signal=%s, data=%s" % (signal, data)
    
    Bus.add_emission_hook("lastfm_username_changed", callback)
    
    Bus.emit("lastfm_username_changed", "jldupont")

Relevant Stackoverflow posts
Here are two Stackoverflow posts in relation with my exercise: question 1, question 2.

Rhythmbox plugin - Syncing with Last.fm

2010-01-21T19:13:00.001-05:00

In my quest to manage my music collection seamlessly between media players, I've decided to use Last.Fm as my main database. Last.Fm provides ways to store:

History of played tracks
Playcount associated with played tracks
Tags associated with tracks
Playlists
etc.

Through their API (see mindmap), it is possible to have access to all this information (and much more).

Rhythmbox

My music collection is currently organized through Rhythmbox - it wasn't my first choice at the time I've moved from my Windows desktop to my Linux Ubuntu one but it has proven very useful. RB can be extended with Python scripts quite painlessly.

Last.Fm Syncing

The plugin I have developed this time supports the following features:

When a track is played, it downloads the corresponding information from the user's Last.Fm account and updates the following fields of the Rhythmbox database:

Playcount: if the "last.fm playcount" associated with the track is greater than 0, the local playcount of RB is updated with it
Rating: if the "last.fm love" field is set for the track, the local "rating" field of the track is set to 5.0 (5 stars)

With this functionality, it is easier to update a local Rhythmbox media player database with information stored on a user's account on Last.fm. The standard Last.Fm plugin of Rhythmbox must be enabled and configured correctly.

Installation

The plugin can be installed by following the procedure found at the project's home page. A direct link to the plugin Debian/Ubuntu repository is provided here.

Future

The project is far from being completed - I have many other features in the pipeline. Stay tuned!

Related project

There is a related project to this one: Rhythmbox - Last.Fm Desktop Client plugin.

Feedback is very welcome!

UPDATE: see here for details on the new release.

Rhythmbox plugin for Last.fm desktop client

2010-01-11T16:50:00.018-05:00

I've recently lost my music database to the hands of Banshee (actually, the Sqlite3 database file is still somewhat salvageable). I have longed for an application that would make my life easy in managing my music collection (which is pretty extensive) and do this day haven't found one. The recent event pushed me to the edge: I am embarking on a quest to scratch my itch on this matter.

First step

The first step consisted in getting acquainted with Rhythmbox, my new music media player. Why Rhythmbox? because it supports a Python based extension system (compared to Banshee's C# based on) and I happen to really enjoy writing software using Python.

To this effect, I have crafted a plugin that sends the current playing state (START/PAUSED/RESUMED) of Rhythmbox to Last.fm's Desktop Client.

Plugin

The project is located on Github.

Project

I intend to make several other contributions focused on media players. My goal is to be able to synchronize my music collection seamlessly between Rhythmbox, Last.fm and iTunes (with possibly also Musicbrainz). Stay tuned!

Google Wave Invites

2010-01-06T20:35:00.002-05:00

I've got some Google Wave invites left... first come, first served!