Sharing Technologies

SoapUI / REST / JSON / variable namespaces

2012-02-13T21:29:00.001-05:00

As a developer of large Web applications, I'm used to relying on a few proven test strategies:

Unit tests with JUnit / Cobertura and DOH / JSCoverage;
Functional and Integration tests with Selenium 2.

When I joined my current development team to work on the project TradeInsight, I was introduced to SoapUI for the functional tests. I like it's ability to convert JSON to XML, enabling then the writing of XPath match assertions.

There's one little caveat related to the conversion and the XPath assertions:

When the server response contains a single JSON object, the conversion introduce a namespace into the generated XML.
And this namespace depends on the server address.

The following figures show the original JSON response and its conversion to XML with the inferred namespace.

JSON response produced by a REST service.

Transformation into a XML payload with an inferred namespace.

This automatic mapping to namespace server dependent does not allow to write server-agnostic code if you follow the suggested solution!

The following figures show a XPath match expression as documented in the SoapUI training materials. Sadly, running it against an XML with a namespace, this error is reported:

XPathContains assertion failed for path [//startDate/text()] : Exception missing content for xpath [//startDate/text()] in Response.

Simple XPath expression with the corresponding error message.

Corrected XPath expression as suggested, now server dependent :(

A simple solution consists in replacing the specified namespace with the meta-character '*', which match any namespace. As all elements of the XML document are under the scope of the inferred namespace, it's important to prefix all nodes of the XPath expression with '*:', as illustrated by the following figure.

Use of the '*' prefix to produce assertion server independent.

I hope this helps.

A+, Dom

Un nouveau chapître

2011-06-27T21:47:00.000-04:00

Il y a quelques 18 moins, j'ai choisi de devenir un entrepreneur à temps plein, consacrant mon temps et mes ressources financières au développement du service AnotherSocialEconomy. Cela a été une expérience très riche en enseignements.

Ceux qui me connaissent savent à quel point il était important que je sois vraiment maître de mon destin professionnel :

Développer un outil qui offre une vraie valeur à son groupe d'utilisateurs.
Baser les décisions sur les faits et leurs conséquences sans laisser places aux interférences politiques (tellement immobilisantes dans les grandes entreprises).
Utiliser la technologie pour servir d'abord les usagers, puis pour construire les services et faciliter leur développement. En ce sens, la méthodologie Lean Startup et son principe Build-Measure-Learn offre un excellent cadre de travail.
Utiliser la méthodologie Agile (sans oublier les outils de XP) pour développer une application de grande envergure.

Avec mon associé Steven, cela a été très intéressant de définir mes propres objectifs, mon organisation de travail, et diriger le développement de l'entreprise.

Mais il y a un mois, principalement à cause de l'absence d'une perspective de revenus stables dans un futur proche, je me suis mis à la recherche de contrats et de postes permanents. C'est sûr que c'est déchirant, mais l'outil est en production et je peux le faire évoluer dans mon temps libre. En appliquant donc un principe de Lean une fois de plus ("Failure to change is a vice!", Hiroshi Okuda, Président de Toyota Motor Corp.), je replonge du côté salarié.

Finalement, après plusieurs entretiens, mon choix s'est porté sur la compagnie MEI qui développe une plate-forme Web pour aider ses clients (des producteurs de biens de consommation emballés--consumer packaged goods) à suivre leurs campagnes de promotion. Jusqu'il y a deux ans, MEI offrait principalement son service aux grands manufacturiers. Maintenant, dans une offre SAAS, MEI développe une nouvelle version pour les producteurs petits et intermédiaires, sous la marque TradeInsight. La synergie entre mon expérience (Java, JavaScript, cloud, mobile, Agile) et l'équipe est indéniable.

Au fait : MEI embauche encore !

L'aventure d'AnotherSocialEconomy continue, à temps perdu de mon point de vue et dans mes interactions avec les groupes de la communauté technique de Montréal (NewTech, Android, NodeJS, etc.). Pour plus d'informations quant aux partenariats possibles, veuillez prendre contact avec Steven.

A+, Dom

OAuth authorization handling in a Android application

2011-06-07T15:17:00.001-04:00

Note : This post is part of the series "Lessons learned as an independent developer". Please refer to the introduction (in French) for more information. Cet article fait partie de la série intitulée « Leçons d'un développeur indépendant ». Au besoin, lisez mon introduction pour plus d'informations.

Context

AnotherSocialEconomy APIs are based on standards as much as possible: OpenID and OAuth for the authentication and authorization, HTTP-based REST interface (1, 2) for the communication protocol, JSON for the data payload format, etc.

This post is about setting up a Android application which get authorization tokens from a OAuth provider.

OAuth provider

There are many known OAuth providers like Netflix, Twitter, Facebook (coming to OAuth 2.0 soon), Yahoo!, Google, etc. If these providers are convenient, they don't offer much flexibility if some debugging is required.

For this experiment, I'm going to use Google App Engine Java and their OAuth support. For a complete walk-through, refer to Ikai Lan's post: Setting up an OAuth provider on Google App Engine, especially for the part which describes how to get the public and secret keys for your client application to sign communications with the provider.

OAuth client - Work flow

Strategy: Upon creation, the process checks if the authorization tokens have been saved as user preferences.

If they are present, they are loaded to be used to sign each future communication with the application on the server.
If they are missing, the OAuth work flow is triggered:

With the server application keys, a signed request is sent to get a temporary request token.
With this request token, a URL to the authentication page is required and an Intent is created to load the corresponding page in a browser. At this step, the application is stopped.
The user enters his credentials in the browser and grants access rights to the mobile application. The return URL has a custom format: ase://oauthresponse.
The mobile application, which has an Intent registered for that custom URL, is restarted and is given the return URL. A verification code is extracted from this URL.
The verification code is used to issue a signed request asking for the access tokens.

The access tokens are saved as part of the user preferences only if she selected a 'Remember me' option.

Figure 1: Authorization work flow

Alternative: If the mobile application offers anonymous services, like browsing the list of registered stores in the case of AnotherSocialEconomy.com, it can be friendlier to delay the authorization verification.

OAuth client - Initiating the authorization process (1, 2, 3)

To simplify the application development, I have decided to use oauth-signpost, a library provided by Matthias Käppler who wanted a slick and simple way to access Netflix services.

Signpost is the easy and intuitive solution for signing HTTP messages on the Java platform in conformance with the OAuth Core 1.0a standard. Signpost follows a modular and flexible design, allowing you to combine it with different HTTP messaging layers.

Note that this library is also good to manage remotely Twitter accounts.

This section is about initiating the authorization process, which occurs if the application is not called by the application on the server (with the verification code, see next section) and if the OAuth token could not be found in the user preferences. This is the path with the steps {1, 2, 3} in Figure 1.

if (!justAuthenticated && Preferences.get(Preferences.OAUTH_KEY, "").length() == 0) {
    // Display the pane with the warning message and the sign in button
    setContentView(R.layout.main_noauth);

    // Update the 'Remember me' checkbox with its last saved state, or the default one
    final String saveOAuthKeysPrefs = Preferences.get(Preferences.SAVE_OAUTH_KEYS, Preferences.SAVE_OAUTH_KEYS_DEFAULT);
    ((CheckBox) findViewById(R.id.app_noauth_keepmeconnected)).setChecked(Preferences.SAVE_OAUTH_KEYS_YES.equals(saveOAuthKeysPrefs));

    // Attach the event handler that will initiate the authorization process up to opening the browser with the authorization page
    findViewById(R.id.app_noauth_continue).setOnClickListener(new OnClickListener() {
        @Override
        public void onClick(View v) {
            // Check if the 'Keep me connected' check box state changed and save its new state
            boolean keepMeConnected = ((CheckBox) findViewById(R.id.app_noauth_keepmeconnected)).isChecked();
            if (Preferences.SAVE_OAUTH_KEYS_YES.equals(saveOAuthKeysPrefs) != keepMeConnected) {
                Preferences.set(Preferences.SAVE_OAUTH_KEYS, keepMeConnected ? Preferences.SAVE_OAUTH_KEYS_YES : Preferences.SAVE_OAUTH_KEYS_NO);
            }
                    
            // Set up the OAuth library
            consumer = new CommonsHttpOAuthConsumer("<your_app_public_key>", "<your_app_secret_key>");

            provider = new CommonsHttpOAuthProvider(
                    "https://<your_app_id>.appspot.com/_ah/OAuthGetRequestToken",
                    "https://<your_app_id>.appspot.com/_ah/OAuthAuthorizeToken",
                    "https://<your_app_id>.appspot.com/_ah/OAuthGetAccessToken");
                    
            try {
                // Steps 1 & 2:
                // Get a request token from the application and prepare the URL for the authorization service
                // Note: the response is going to be handled by the application <intent/> registered for that custom return URL
                String requestTokenUrl = provider.retrieveRequestToken(consumer, "ase://oauthresponse");

                // Step 3:
                // Invoke a browser intent where the user will be able to log in
                startActivity(new Intent(Intent.ACTION_VIEW, Uri.parse(requestTokenUrl)));
            }
            catch(Exception ex) {
                Toast.makeText(Dashboard.this, R.string.app_noauth_requesttoken_ex, Toast.LENGTH_LONG).show();
                Log.e("Dashboard no auth", "Cannot initiate communication to get the request token\nException: " + ex.getClass().getName() + "\nMessage: " + ex.getMessage());
            }
        }
    });
}

Figure 2 below illustrates the pane main_noauth displaying the warning message and the action button, and figure 3 shows the authorization page as provided by Google for the hosted applications on App Engine.

Figure 2: Pane displayed if application not yet authorized

Figure 3: Google authorization page

Whatever action the user takes, the application is going to be called with the URL ase://oauthresponse. The next section covers this work flow path.

OAuth client - Processing the authorization (4, 5)

The application is registered with an Intent associated to the scheme ase and the host oauthresponse. The labels themselves are not important, only their uniqueness and the correspondence with the return URL specified at Step 2.

<intent-filter>
    <action android:name="android.intent.action.VIEW"/>
    <category android:name="android.intent.category.DEFAULT" />
    <category android:name="android.intent.category.BROWSABLE"/>
    <data android:scheme="ase" android:host="oauthresponse"/>
</intent-filter>

The following code snippet implements the steps 4 and 5 as described in Figure 1.

private boolean checkOAuthReturn(Intent intent) {
    boolean returnFromAuth = false;
    Uri uri = intent.getData();

    if (uri != null && uri.toString().startsWith("ase://oauthresponse")) {
        // Step 4:
        // Get the request token from the Authentication log in page
        String code = uri.getQueryParameter("oauth_verifier");
            
        try {
            // Step 5:
            // Get directly the access tokens
            provider.retrieveAccessToken(consumer, code);
            returnFromAuth = true;
                
            // Persist the tokens
            if (Preferences.SAVE_OAUTH_KEYS_YES.equals(Preferences.get(Preferences.SAVE_OAUTH_KEYS, Preferences.SAVE_OAUTH_KEYS_DEFAULT))) {
                Preferences.set(Preferences.OAUTH_KEY, consumer.getToken());
                Preferences.set(Preferences.OAUTH_SECRET, consumer.getTokenSecret());
            }
        }
        catch(Exception ex) {
            Toast.makeText(Dashboard.this, R.string.app_noauth_accesstoken_ex, Toast.LENGTH_LONG).show();
            Log.e("Dashboard no auth", "Cannot complete communication to get the request token\nException: " + ex.getClass().getName() + "\nMessage: " + ex.getMessage());
        }
    }
       
    return returnFromAuth;
}

The Dashboard class definitions are available in a gist on GitHub. This gist contains also a wrapper of the SharedPreferences class, the application manifest with the declaration of the Intent for the custom return URL, and the layout definition of the pane with the warning and the sign in button.

OAuth Client - The quirks

My Android application is very simple and is configured with the launch mode singleTop. As such, if the system does not destroy the application when the code starts an activity to browse the Authentication service URL, the invocation of the ase://oauthresponse URL by the browser should trigger a call to the onNewIntent() method. It never happened during my tests and on my phone... Every time, the application is recreated and a call to onCreate() is issued. So both functions delegate to the helper checkOAuthReturn().

@Override
protected void onNewIntent(Intent intent) {
    checkOAuthReturn(intent);
}

In this example, I've decided to select the view to associate to the first screen of the application according to the knowledge of the OAuth access token (read from the user preferences or retrieved dynamically thanks to the verification code coming with the ase://oauthresponse URL). The following snippet illustrates this flow. In some occasions, it can be better to start a separate activity if the main pane is instrumented to disable the triggers to protected actions. This approach with a separate activity is also better for the portability.

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    Preferences.setPreferenceContext(PreferenceManager.getDefaultSharedPreferences(getBaseContext()));

    boolean justAuthenticated = checkOAuthReturn(getIntent());
        
    if (!justAuthenticated && Preferences.get(Preferences.OAUTH_KEY, "").length() == 0) {
        setContentView(R.layout.main_noauth);

        // Instrumentation of the pane to initiate the authorization process on demand
        // ...
    }
    else {
        setContentView(R.layout.main);
    }
}

I hope this helps.
A+, Dom

Partage d'expérience dans le développement d'applications Android

2011-06-02T21:57:00.001-04:00

Contexte

Note : Cet article est le premier d'une série intitulée « Leçons d'un développeur indépendant ». Le niveau de la discussion dans cet article est général. Les articles suivant iront plus en profondeur et seront illustrés de bouts de code.

Dans le cadre de mon entreprise AnotherSocialEconomy.com, j'ai eu l'occasion de mettre en œuvre plusieurs bonnes pratiques et je vais en partager quelques unes ici. Je vais notamment me concentrer sur les développements autour de l'application cliente Android, depuis l'identification des usagers jusque l'émission de notifications asynchrones.

AnotherSocialEconomy.com, ou ASE, offre un service connectant consommateurs et détaillants :

Les consommateurs à la recherche d'un produit ou service n'ont qu'à décrire leur demande depuis l'un des multiples points d'entrée de l'application : une page Web faite pour AdWords, le site de ASE ou affilié, la page de l'application Facebook, un message direct depuis Twitter, etc.
Les détaillants participants sont notifiés des demandes en fonction de leurs préférence. Les détaillants sont libres de faire une ou plusieurs propositions en fonction de leur disponibilité.
Au fur et à mesure que les propositions sont composées, les consommateurs en sont notifiés et peuvent à tout moment les décliner ou les confirmer.
Les confirmations sont notifiés aux détaillants qui réservent alors le produit ou le service pour le consommateur.
Ce dernier n'a plus qu'à payer et à en prendre possession.

Pour faire simple : ASE connecte les consommateurs avec les détaillants qui ont les produits ou services qu'ils recherchent en inversant le processus de recherche.

Le moteur de ASE est présentement codé en Java et est hébergé sur l'infrastructure Google App Engine. Dans la suite de cet article, pour généraliser le propos, le moteur de ASE est référencé en tant qu'application serveur.

La gestion des utilisateurs sur le serveur

Depuis le départ, le service des usagers de l'application serveur repose sur OpenID. Avec OpenID, l'identification des utilisateurs est confiée à des services tiers de confiance (Yahoo!, Google, AOL, etc.) sans que l'application serveur ne voit le mot de passe des utilisateurs, seulement leur identifiant OpenID. Ce mode de gestion règle aussi plusieurs problèmes :

Les utilisateurs n'ont pas à créer un énième compte pour le service ASE.
Le serveur est moins à risque car il n'y a pas de mots de passe enregistré là.
La gestion des sauvegardes est plus simple (toujours parce qu'il n'y a pas de mots de passe).
En cas de bris de leur mot de passe, les utilisateurs peuvent assurément mieux s'appuyer sur les services de leur fournisseur OpenID que sur les miens :)

Plus tard dans le cycle de développement, notamment parce qu'il a s'agit de développer une application Facebook, les mécanismes d'identification de Facebook, ceux de Twitter et de ceux de Microsoft Live ont été intégrés à l'application serveur. Des trois, le mécanisme de Twitter est le plus standardisé (OAuth), donnant aussi accès aux données de l'utilisateur du service. Mais tous ont été intégrés de manière à agir comme des services OpenID.

OpenID est un bon système d'identification pour une application cliente Web. Avec les restrictions de sécurité des navigateurs (SSL et sandbox), une fois que l'identité de l'utilisateur est confirmée par un fournisseur OpenID, tant que cette identité reste associée à la session Web, l'envoi de données vers les navigateurs reste protégé.

Par contre, quand l'application cliente est native (sur un ordinateur ou sur un téléphone mobile), il n'est pas possible de s'appuyer sur un mode de session Web robuste comme celui des navigateurs. Aussi une application malicieuse pourrait intercepter l'identifiant de session et s'en servir à l'insu de l'usager. Pour se prémunir contre cette attaque, il est souhaitable d'utiliser OAuth qui signe chaque échange entre l'application cliente et le serveur, rendant caduc l'utilisation de l'identifiant de session Web.

L'authentification des usagers sur le client

Chaque téléphone Android est associé à un utilisateur. Si la carte SIM de l'opérateur téléphonique est changée, les données de l'utilisateur précédent ne sont plus accessibles. Chaque application à accès à son propre espace de stockage protégé, mais l'utilisateur peut réclamer cet espace à tout instant. Ce n'est donc pas une solution de stockage à long terme.

Dans le modèle d'authentification OAuth, les échanges de données sont signés par l'application cliente grâce à un jeton émis par l'application serveur. Grâce à la signature, l'application serveur est assurée de l'identité de l'utilisateur à chaque échange de données.

Pour avoir un jeton, le protocole à observer par l'application cliente est relativement simple :

Émettre une requête pour recevoir un premier jeton dit d'accès.
Ce jeton est utilisé pour initier un appel vers une page d'autorisation.
L'application serveur présente alors une page d'identification où l'utilisateur doit, s'il n'est pas déjà authentifié, entrer son identifiant et son mot de passe, puis accepter que l'application cliente accède aux données qui sont gérées par l'application serveur.
L'application serveur retourne un second jeton attestant de l'acceptation par l'utilisateur de l'accès aux données. Ce jeton a une durée de vie limitée.
Ce second jeton peut être utilisé pour obtenir deux jetons (clé publique et clé secrète) qui permettront à l'application cliente de signer les échanges de données de telle sorte que l'application serveur les associera à l'utilisateur concerné.
Souvent ces deux jetons ont une grande durée de vie (pas d'expiration dans le cas de Twitter), et peuvent donc être sauvegardés par l'application cliente pour signer de manière transparente tous les futurs échanges.
Il faut cependant tenir compte que l'utilisateur peut révoquer ces deux jetons n'importe quand, ou qu'ils peuvent expirer n'importe quand (à cause d'un changement de stratégie du côté de l'application serveur, par exemple) aussi il faut être prêt à exécuter le processus pour obtenir deux nouveaux jetons à n'importe quel moment.

Il est important de noter que la sauvegarde des jetons d'authentification doit être très sécuritaire. Il n'est pas acceptable de les sauvegarder dans un simple fichier texte situé sur une carte d'extension mémoire par exemple. Si le risque d'accès à ces jetons est trop grand, il faut mieux rejouer le scénario ci-dessus pour obtenir un nouveau jeu de jetons.

Au moment où j'écris cet article, le matériel de la série Samsumg S et la tablette Motorola Xoom ont des systèmes de fichiers encryptés. À ma connaissance, même Android 3.1 n'offre toujours pas de solution bas niveau de sécurité maximale...

La réception des notifications asynchrones

Si de plus en plus de fondeurs de silicium mettent l'accent sur la puissance du processeur central (Qualcomm) et leur nombre (NVidia vient d'annoncer un Tegra avec 4 cœurs), si l'augmentation de la bande passante (de HPSA+ à LTE par exemple) permet des échanges de données de plus en plus rapide même loin de tout réseau informatique, la capacité énergétique des téléphones portables modernes reste leur point faible. Par le passé, j'ai eu des téléphones Nokia et Sony Ericsson capables de rester en veille plus d'une semaine. Maintenant, je dois brancher mon téléphone HTC Desire chaque soir, et cela même avec une navigation somme toute restreinte !

Dans ces conditions, maintenir une application éveillée pour pouvoir interroger l'application serveur à intervalles réguliers (technique dite de polling) est à proscrire.

Il y a deux ans, en développant une application pour la plate-forme BlackBerry 5, j'ai utilisé la technique suivante :

L'application client sur le téléphone écoutait un certain nombre de messages du système (changement de type de réseau, perte du réseau, etc.) et les colligeait dans une base de données interne.
C'était l'application serveur qui décidait du moment de transmission de ces données statistiques en envoyant un SMS à chaque téléphone.
À la réception de ce SMS, l'application client ouvrait une connexion HTTP pour transmettre en rafale ses données colligés.
Une fois l'ensemble de données rapatriés de chaque téléphone, l'application serveur établissaient des rapports de couverture pour l'opérateur.

Depuis la version 2.2, il existe le protocole AC2DM: Android Cloud to Device Messaging. Quand une application cliente configurée pour AC2DM s'initialise, elle doit s'enregistrer auprès du serveur local AC2DM et reçoit en retour un identifiant d'enregistrement. C'est la responsabilité de l'application cliente d'envoyer cet identifiant à l'application serveur pour que celle-ci ait la clé pour envoyer les notifications asynchrones à cette application cliente, et à elle seule.

Quelque part, l'approche du AC2DM est semblable à ma méthode d'activation par SMS. Il se peut même qu'elle utilise en sous main cette technique ;) La principale différence réside dans l'aspect service : avec AC2DM, l'application cliente n'a pas à rester active pour recevoir les notifications, c'est le serveur local de notifications qu'il l'activera au besoin.

L'application pour les consommateurs

L'application cliente pour les consommateurs doit offrir plusieurs fonctionnalités :

recevoir les notifications concernant les demandes et les propositions en attente
gérer la liste des demandes et propositions en attente
créer de nouvelles demandes
avec un accès au carnet d'adresses du téléphone pour pouvoir inclure ses « amis » en copie des demandes
avec un accès au système de localisation géographique du téléphone pour faciliter la création des demandes
modifier ou annuler des demandes en attente
confirmer ou annuler des propositions en attente

Le principal objectif de l'application cliente sur les téléphones mobiles est le relais des notifications de mise-à-jour de demande, en réaction à la réception de nouvelles propositions ou de modifications de proposition de la part de détaillants. En quelques « clics », l'utilisateur doit pouvoir accéder rapidement au détail de la demande concernée, au détail de la proposition et à des informations sur le magasin ou bureau du détaillant. Pour faciliter cet accès, la plupart des informations sont sauvegardées sur le téléphone au fur et à mesure qu'elles sont requises. Pour garder une structure proche du modèle de données produit par l'application serveur, le stockage utilisé est le service de base de données interne du mobile (SQLite sur Android, par exemple).

L'application pour les détaillants

L'application cliente pour les consommateurs doit offrir plusieurs fonctionnalités :

recevoir les notifications de nouvelles demandes
créer et gérer des propositions (possiblement avec un accès à la caméra pour scanner les codes barre)
confirmer les livraisons
gérer la liste des demandes et propositions en attente

Parce que les services offerts aux consommateurs sont très différents de ceux offerts aux détaillants, ils sont proposés dans deux applications différentes. Cela réduit les risques de confusion de contexte pour les utilisateurs agissant autant en tant que consommateur que détaillant.

À suivre...

Dans les prochains articles, je décrirai en détail les différentes implantations que j'ai réalisées. Il y a plusieurs techniques qui ne sont pas évidentes, comme celle gérant l'authentification avec OAuth, et j'imagine que cela sera utile à bien des développeurs ;)

A+, Dom

Google App Engine, scheduled tasks, and persisting changes into the datastore: the risk of a race condition

2011-04-16T10:56:00.000-04:00

This post is about a race condition I've accidentally discovered and hopefully fixed. It occurred in App Engine and was generated by tasks I created for immediate execution...

Context

When I started developing in Java for Google App Engine, I decided to give a try with JDO, mainly because it is datastore agnostic (*). Operations managing my entities are organized in DAOs with set of methods like the followings.

public Demand update(Demand demand) {
    PersistenceManager pm = getPersistenceManager();
    try {
      return update(pm, demand);
    }
    finally {
        pm.close();
    }
}

public Demand update(PersistenceManager pm, Demand demand) {
    // Check if this instance comes from memcache
    ObjectState state = JDOHelper.getObjectState(consumer);
    if (ObjectState.TRANSIENT.equals(state)) {
        // Get a fresh copy from the data store
        ...
        // Merge old copy attributes into the fresh one
        ...
    }
    // Persists the changes
    return pm.makePersistent(demand);
}

I knew that changes are persisted only when the PersistenceManager is closed; closing it after an update is safe attitude. I decided anyway to separate the PersistenceManager instance management from the business logic updating the entity for clarity.

This decision offers the additional benefit of being able to share PersistenceManager instance with many operations. The following code snippet illustrates my point: a unique PersistenceManager instance is used for two entity loads and one save.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        // Get the identified demand (can come from the memcache)
        Demand demand = getDemandOperations().getDemand(pm, demandKey, ownerKey);

        // Check if the demand's location is changed
        if (command.contains(Location.POSTAL_CODE) || command.contains(Location.COUNTRY_CODE) {
            Location location = getLocationOperations().getLocation(pm, command);
            if (!location.getKey().equals(demand.getLocationKey())) {
                command.put(Demand.LOCATION_KEY, location.getKey());
            }
        }

        // Merge the changes
        demand.fromJson(command);

        // Validate the demand attributes
        ...

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);

        // Report the demand state to the owner
        ...
    }
    finally {
        pm.close();
    }
}

For my service AnotherSocialEconomy which connects Consumers to Retailers, the life cycle for a Demand is made of many steps:

State open: raw data just submitted by a Consumer;
State invalid: one verification step failed, requires an update from the Consumer;
State published: verification is OK, and Demand broadcasted to Retailers;
State confirmed: Consumer confirmed one Proposal; Retailer reserves the product for pick-up, or delivers it;
State closed: Consumer notified the system that the transaction is closed successfully;
State cancelled: ...
State expired: ...

In my system, some operations take time:

Because of some congestion in the environment, which occurs sometimes when sending e-mails.
Because some operations require a large data set to be processed–like when a Demand has to be broadcasted to selected Retailers.

Because this time constraint and the 30 second limit, I decided to use tasks extensively (tasks can run for 10 minutes). In some ways, my code is very modular now, easier to maintain and test.

So I updated my code to trigger a validation task once the Demand has been updated with the raw data submitted by the Consumer. The code snippet shows the task scheduling in the context of the processDemandUpdate() method illustrated above.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        ...

        // Update the state so the entity is ready for the validation process
        demand.setState(State.OPEN);

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);

        // Create a task for that demand validation
        getQueue().add(
            withUrl("/_tasks/validateOpenDemand").
                param(Demand.KEY, demandKey.toString()).
                method(Method.GET)
        );
    }
    finally {
        pm.close();
    }
}

Issue

Until I activated the Always On feature, no issue has been reported for that piece of code: my unit tests worked as expected, my smoke tests were fine, the live site behaved correctly, etc.

Then the issue started to appear randomly: sometimes, updated Demand instances were not processed by the validation task anymore! A manual trigger of this task from a browser or curl had however the expected result...

For the task to be idempotent, the state of the Demand instance to be validated is checked: if set with open, the Demand attributes are checked with the result of the state being set with invalid or published. Otherwise nothing happens. With that approach, Demands already validated are not processed a second time...

What occurred?

Without the Always On feature activated, because of the low traffic in my application, the infrastructure was delaying the process of the validation task a bit and it was executed once the request process finished.
Thanks to that soft process serialization, the datastore update commanded by the instruction pm.close() had all chances to be completed before the start of the validation task!
With the Always On feature activated, the infrastructure had much more chance to get one of the two other application instances to process the validation task... which could happen before the datastore update...
As it started before the datastore update, the validation task found a task in the state set by the previous run of the task for this Demand instance: invalid or published. Then it exited without reporting any error.

Solutions

The ugly one:
Add a delay before executing the task with the countdownMillis() method.

// Create a task for that demand validation
        getQueue().add(
            withUrl("/_tasks/validateOpenDemand").
                param(Demand.KEY, demandKey.toString()).
                method(Method.GET).
                countdownMillis(2000)
        );
    }
    finally {
        pm.close();
    }
}

A tricky one:
Use memcache to store a copy of the Demand, which the validation will use instead of the reading it from the datastore. Because there's no warranty that your entity won't be evicted before the the run of the validation task, this is not a solution I can recommend.

The simplest one:
Move the code scheduling the code outside the try...finally... block. The task will be scheduled only if the updates of the Demand instance have been persisted.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        ...

        // Update the state so the entity is ready for the validation process
        demand.setState(State.OPEN);

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);
    }
    finally {
        pm.close();
    }

    // Create a task for that demand validation
    getQueue().add(
        withUrl("/_tasks/validateOpenDemand").
            param(Demand.KEY, demandKey.toString()).
            method(Method.GET)
    );
}

The most robust one:
Wrap everything withing a transaction. When a task is scheduled within a transaction, it's really enqueued when the transaction is committed.

Be aware that adopting this solution may require a major refactoring.

Conclusion

Now I understand the issue, I'm a bit ashamed of it. For my defense, I should say the defect has been introduced as part of an iteration which came with a series of unit tests. Before the activation of the Always On feature, it stayed undetected, and later it occurred only rarely.

Anyway, verifying the impact of all calls to external tasks before persisting any changes is one point in my review check list.

I hope this helps,
A+, Dom

--
Notes:
* These days, I would start my application with Objectify. This blog post summarizes many arguments I agree on too in favor to Objectify.

State of the AnotherSocialEconomy Initiative

2011-04-14T11:26:00.002-04:00

When my partner Steven and I started our startup adventure a few years ago, our main goal was to demonstrate our ability to convert an idea into a live project. As we used our experience to build a viable product, we knew it would add value to our resume.

Over the months, the project evolved slowly:

The core idea is: help the consumers who look for a specific product to find the retailer who has it in stock, and help retailers to connect with consumers online and drive them in-store. Our moto: the missing link between shopping online and buying offline.
The proof-of-concept was made of screenshots, live Twitter accounts, and a piece of Python code connecting those accounts together. This material allowed us to be among the semi-finalist companies of TechCrunch50 in 2009!
The first implementation of the engine connected consumers and retailers, each of them interacting with the system with direct messages (DMs), sent from their own Twitter account. At that time, the tool was named Twetailer.
Later, we figured out Twitter was too geeky and we added a connector to accept and generate e-mails. Since then, the engine has a XMPP (instant messaging) connector, another one for Facebook, and a plan for VOIP (with Twilio).
At one point, we were approached to start an experiment for golfers: usually avid golfers have to spend a lot of time on the phone to get three buddies to play with and to book a tee-time. In two months, we created ezToff.com, developed an embeddable widget to ease the creation of a tee-off request, and developed a Web console for the golf course staff. The experiment was shut down because of a lack of traction...
Recently, we started another experiment in the used car market, under the name AnotherSocialEconomy. Our market researches found that the average time to buy a used car is six weeks in Quebec. Typically, consumers start on the Web, grab listings, and call dealerships one after the other. In the Montreal area, there are 300,000 pre-owned cars bought per month: ⅓ from dealerships, ⅓ from wholesalers, and ⅓ from individuals. Dealerships control 45% of the market value.
So far, this experiment has been a partial success: we get demands from consumers and forward proposals from used car sales people. We helped our first customer finding a car in only two weeks! But, as the dealership staff is not used to new technologies (sic), we manage the service for them and it's very time consuming...

What's next?

We are very happy with consumers trusting us. We work hard to continue to improve their experience, on our landing pages and in our communication by e-mail. The priority is to have them qualifying more their demands upfront.

Our focus right now is more on the retailer-side, in order to have sales people in the dealerships interacting with the system by e-mail too. If around 80% of them agree to work with us to serve our users, we have to prepare the proposal details and to reach them out for approval. For the business to scale, they should prepare and post the proposals themselves.

For now, we need more data to determine trends. This is a prerequisite for used car dealers to adopt our methodology. It is also possible that will lead to another pivot.

Lessons learned?

The first one is an obvious one now: nobody can be as committed as the founders! Since I left the company Compuware to become a full time entrepreneur, Steven and I have met many people we expected to work/partner with: a technology company CEO, a former manufacture owner and now real estate agent, a UX designer, a few VCs, a marketer, two successful startup founders, etc. If we sometimes got excellent feedback, none joined us.

The second one is related to the product development: two techies are not enough to make a great product! They can talk about their product at length, but they don't know how to convince decision makers. They need the help of a marketing genius!

Another one is related to the importance of the contact network. If you don't know the right people, very few will listen to you. Having a large address book or friends with deep pockets definitively helps a lot.

And the last one: developing a tool for the general public is difficult! Following the Lean Startup process can really help. Check Ash Maurya's blog, for example.

Technologies learned?

I continue to find Google App Engine an awesome environment. The recent addition of the Channel API which allows the back-end logic to push asynchronous notifications into Web consoles really improves the user experience. On the maintenance side, I appreciate the Java Remote API which simplifies the development of maintenance and data extraction tasks.

Web console side, I've started upgrading the code to Dojo 1.6 and its new HTML5 compliant syntax. I don't use the AMD loader yet, but I'm waiting for the one coming with 1.7. I have recently started to use Selenium 2 for my smoke tests and I really like it!

Mobile side, I wish to have more spare time to update my Android application for ezToff and to benefit from the Android Cloud To Device Messaging (C2DM) API. But I'm also thinking of building application with the awesome dojox.mobile.

To develop our customer base in the used car experiment, we have created two AdWords campaigns: one for each language, both in the Montreal area. Using AdWords and optimizing the campaigns was very instructive. There are many concepts to master: long tail, auto bid, average CPC, conversion rate, landing page quality score, etc. I know understand why so many people choose to become an AdWords certified partner ;)

Wrapping up 2010, preparing 2011

2011-01-06T23:39:00.000-05:00

2010 Summary

2010 was an interesting year for me professionally. Inspired by similar lists online, I present what I did (or can remember at least):

Left Compuware and joined my partner Steven at Milstein and Associates inc. end-of-January, to focus 150% of my time on AnotherSocialEconomy (formely known as Twetailer).
Adapted the Amazon Flexible Payment Service (FPS) library to the App Engine environment—freely available on github.
Refactored the communication layer to be able to send details e-mails to customers, in addition to short ones sent over Twitter and Instant Messaging services.
Built the first Web consoles for Golf players and Golf courses staff, based on Dojo and using the freshly delivered REST API—check ezToff.com.
Built the first Android application for Golf players using their GPS & Address book to ease the tee-off booking process with AnotherSocialEconomy—freely available on github.
Helped preparing pitches to Golf Canada representatives and to Golf staff members and owners.
Developed the AnotherSocialEconomy widget, ready to be embedded in participant websites and loading the AnotherSocialEconomy wizard on demand
Reviewed the book Google App Engine Java and GWT Application Development.
Continued to develop my open-sourced library offering tools for globalizable generic resource bundles (TMX)—on github too.
Developed a prototype of a Facebook application.
Augmented the AnotherSocialEconomy engine to support the used car dealers: buyers don't buy immediately, but collect car information and offers for a while before committing with one dealer => the engine work flow has been adapted to support this slower path of interaction.
Attended presentations to few car dealerlship owners.
Attended meetings with various mentors and potential investors.
Attended meetings of Montreal NewTech, Android Montreal, Augmented Reality Montreal communities

I’m pretty happy with what I have done so far and am looking forward to doing even more.

New technologies

It was also fun to play around some hot new technologies:

Ubuntu 10.04 and 10.10
Android 2.2 and push mechanism on my HTC Desire
App Engine 1.4.0 and channel api
Node.js and WebSockets

2011 Goals and Plans

2011 is going to be critical for AnotherSocialEconomy. The application runs and passed usability tests. The focus point is now on the business development!

Concentrate on one domain (used car market) and get a significant traffic in the Montreal area.
Gather customer feedback (consumer looking for second hand cars and used car dealers), tune the system, and increase traffic. Repeat until 100% satisfaction ;)
Once the system is proven by the traffic and testimonies, involve investors and/or partners to 1) expand the business to other areas or 2) to target another domain or 3) both expand geographically and vertically.
Develop data mining tools for retailers.
Develop domain oriented interfaces for consumers (Web/HTML5 for tablets and PC, native apps for iPhone, Android, BlackBerry).
Add more communication channels (like voice messages with Twilio, for example).
Offer my services as Software Architect & Developer consultant on designing & developing highly scalable and highly available applications on Google App Engine and mobile applications on Android.

A+, Dom

Reviewed book 'Google App Engine Java and GWT Application Development' is out!

2010-12-03T11:57:00.000-05:00

I know that's a pity not to post more regularly! It's just I'm too busy with the developments for AnotherSocialEconomy.com ;)

Here is a little news for Google App Engine developers:

Over the summer, I've been asked to review the draft of the book Google App Engine Java and GWT Application Development. Even with my experience, I learn few techniques, like with the object relationships (chapter 5). A very good book for beginners/intermediates, and still an interesting book for experts.

Enjoy!
A+, Dom

Note: I've no incentive to sell the book, just the pleasure to share a good reference ;)

Securing accounts on the Web

2010-09-20T22:22:00.001-04:00

Situation

Few days ago, my partner Steven got his Google account compromised for a short period of time:

Tweet #1 at 8:52 PM on Sept. 9: Just received 2 calls from friends wondering if I'm being held at a London hotel. FYI, I'm not.
Tweet #2 at 12:23 AM on Sept. 10: Re: Being held in London. My Google account password was changed by an IP address in Nigeria. I've got it back now but with no Contacts.
Tweet #3: at 3:04 PM on Sept. 10: Re Stuck in London: I thought I had everything under control last night but needed http://bit.ly/czgYdg Google Security Breach help to fix.

The thieves used his account to send a scam to few of his friends asking for money because he was supposedly blocked in London without resources.

If Steven's password was not very strong, there's no chance it has been discovered after only few attempts. At no time, Google reported that attempts to log into his account were conducted from computers with IP addresses in Liberia! Steven saw the first warning only when he recovered the access!

Encountered risk

The goal of these thieves was limited to getting money as soon as possible. So they reached out few of Steven's contacts, ones he contacts only occasionally, and they asked for a money to be transferred by Western Union. As they kept the control of his account, they would have been able to get the transaction MTCN (money transfer control number) via his inbox. Western Union maintains a page listing the Common Scams.

Others could have decided to change his password, to just spy his incoming message stream (these ones enabled the POP3 and IMAP accesses), to ask for password reset when Steven is not online, and then to steal his identity in many online services.

Because Steven reacted promptly and because his contacts detected the scam, the thieves did not get any benefit from this operation. They are probably trying to get someone else now, maybe someone from his contact list.

How to reduce the exposure

The first protection consists in defining strong passwords. A lot of services offer information about how to produce strong passwords. I would recommend this Microsoft site Strong Passwords | Microsoft Security—I'm confident that they don't provide the online password checker to enhance a grey dictionary ;)

The second protection would be to use a unique and strong password per account. This is probably the most difficult part! I may use probably 20 to 30 online services, some I use regularly, others I use very rarely. There's no way I can remember so many strong passwords...

My solution: Keepass + DropBox

Keepass is an open source password manager. The tool has been ported on many platforms: Windows, Mac, Linux, iPhone, Android, etc.—Full list on the download page.
DropBox (link with my referral id ;) is an online file sharing system that, thanks to a program installed on each computer/mobile in your network, maintains in sync the corresponding set of files. DropBox is a nice companion to Keepass as it duplicates your password database transparently, reducing the risk to loose the passwords if the original computer is lost.

The combination of the password generator and Keepass secure edit controls makes the tool especially useful:

It's easy to generate a strong passwords (remember: 16 characters or plus ;)
You don't have to remember them as a simple Ctrl+C / Ctrl+V allows to copy securely them in your browser! (the computer clipboard is automatically flushed after few seconds.)

In final, I just have one very strong password (30+ characters) to remember and to change periodically.

Known limitations

Some sites ask users to give a secret answer for a series of predefined questions. If you look at the Apple page below, you'll see that some questions might weaken users more than offering a protection... These days, it's pretty simple to find the responses online!

List of predefined security questions on Apple.com website

Many sites only accepts alphanumerical characters only or don't accept passwords over 20 characters. Oddly enough, most of the bank websites I use prevent too long and too complex passwords! I guess they have other tools to detect intrusions...

List of predefined security questions on Apple.com website

Last minute update

Today, Google announced on its Online Security blog that they will offer a Two-Step Authentication mechanism to log into Google services. This One-Time Password authentication is simpler than distributing a one-time password generator, as Amazon does for example, while providing a still strong security enhancement.

I hope it helps.
A+, Dom

Dilbert: Social Media age

2010-09-14T09:53:00.000-04:00

I cannot resist to re-post this strip!

Are such attitudes only due to managers that need to keep a straight control over their reports in order to seamlessly justify their position? I already covered that aspect in my blog post Manager Attitude back in Dec. 2008. I then stated that the situation would be smoother is such managers could fully assume their facilitator role...

These days, I think it's more related to the difference in each person's cognitive age!

In the traditional chronological order, you could become a manager because you knew the company internal mechanics, the management dynamic, and its long term vision. People finger-pointing the Generation Y (see my post Work around the general laziness, for example) have good reasons to ask new employees to first acquire good experiences before giving them rewards and responsibilities.

However, with the emergence of multi-stream platforms, young people have a tremendous possibilities to learn so much in short periods of time and to overpass older ones. My involvement in edu.cyn.in proved this statement many times: kids are much more prolific at producing online content and sharing with their peers than their own teachers!

On the other end of the spectrum, a lot of retired people who don't have to compete again to maintain their social rank, have gain the possibility to become relevant again by being more connected with younger crowds, more socially involved.

This Dilbert strip illustrates a direct confrontation of the socially aged person and the just chronologically aged one!

I hope you like it too.
A+, Dom

Update business-side

2010-09-06T23:05:00.000-04:00

It's been a long time since my last post. The summer went by so quickly. With my partner Steven Milstein, we have focused on building a generic tool for golfers and golf courses. This post summarizes the latest developments.

The product name

The product name Twetailer is now AnotherSocialEconomy.com. Twetailer was chosen when Twitter started to become mainstream and was made of a combination of Twitter and Retailer. The project Dretailer (more in a future post) was name from Android and Retailer, and we thought about having a variation per connector type.

AnotherSocialEconomy.com is better because it really illustrates that our offer is about a business platform in a global economy, with social networks involved, in a new way of connecting people—and it is not tinted by any service du jour. More information on Steven's blog post The Twouble with Twetailer.

ezToff.com: a dedicated implementation

AnotherSocialEconomy.com provides a generic multi-channel communication engine, conveying messages among consumers and retailers according to the given location and some search criteria. Whenever we explained what's the product is about, potential consumers understood but were blocked when it was time to formulate one of their needs... We knew then we needed to offer a very simple interface where users can post demands without thinking twice about it!

At one point, we met Marc Bienstock who offered to help us building such an interface for golfers! According to Marc:

When a golfer has a chance to play (because his wife gave him the permission ;), he usually calls his buddies to find three of them who are free too.
Then he has to call a golf courses to book a round at the agreed on time, maybe with one or many pull carts, one or two golf carts, etc.
If the booking goes well, he calls his buddies to give them the golf course coordinates.
If it does not work, he calls another club or calls his buddies with the time of the available rounds.
Way too many phone calls to be able to spend a minimum of $200 for 4 players!
On the golf course-side, they have to deal with so many phone calls; some of them have a system queuing calls up to 50!
Average time spent on the phone per caller is too long, and many are useless because they cannot screen them.
When a round stays free, it's an average of $200 lost.

So end-of-July, we launched ezToff.com which allows:

One golfer can submit a request for a tee-off from one central place and it will be broadcasted to all participating golf courses around the given location. The golfer can specify the e-mail addresses of his buddies so they'll be cc'ed for all exchanged messages.
Golf courses are provided a Web console which displays all requests posted in their area. According to their schedule, they can propose rounds—the price per round and the total cost should be documented. Golf courses don't have to watch the console indefinitely as ezToff.com can notify them by e-mail, SMS or tweet.
Proposals are sent back to request initiator by e-mail (his buddies receive a copy if he gave their e-mail addresses). At this step, the golfer can wait for more proposals to come. If he wants to book it, a simple link in the e-mail will generate a response to be sent by e-mail to ezToff.com.

With ezToff.com, golfers can get tee-off proposals with just few clicks. Golf courses get all demands electronically and can focus on the ones they can propose a round to (others are simply declined). Buddies of the golfers are notified at each step of the process automatically. No more infinite phone calls ;)

We interviewed many golfers and they confirmed the golfer's pain. With some visits to golf courses, we've confirmed the courses' pain too. The difficulty we have is that we're not sales people and it's hard to close a deal with golf courses... Knowing that the season is almost finished here, that our business model is probably too expensive for the golf courses, that we need good marketing materials, we're going to tune the offer and be ready for the 2011 season.

The AnotherSocialEconomy.com reseller and influencer programs

The ezToff.com project has been beneficial for us on many points:

With the widget, we have another way to reach consumers: influencers (bloggers, Facebook groups, associations' site, for example) can embed it into their webpages.
If dedicated Web consoles are available to golfers and golf courses, all operations can be done with e-mails, which is the most pervasive communication tool.
The original messages produced by the engine were short to accommodate the Twitter limitation of 140 characters per message. These ones were cryptic to too many people. The variation of these messages for e-mail are now much friendlier and contains links ready to forward the readers' response.
We'll simplify the pricing model thanks to the received feedback.

But the most significant development is the offer of the influencer program:

Each influencer will receive 25% of the revenues generated by consumers confirming demands posted with their copies of the widget.
An influencer can propose one or many widgets anywhere on the Web he reaches his/her community.
We'll work with influencers to customize the widgets for his/her community.

In parallel, we developed the reseller program to share 25% of the revenues generated by retailers proposing goods or services with AnotherSocialEconomy.com.

It's possible that some well organized enterprises will be their own reseller (cut of 25%) and will drive requests with their copies of the widget (another cut of 25%) but that's fine. We'll make some money if they make some, and they'll have a good discount if they help driving more traffic (which means more business to them).

Next steps

On business side, Steven and I are looking for resellers that will help to bring ezToff.com to the next level. We are also trying to develop another domain specific implementation, another skin on the top of the AnotherSocialEconomy.com engine.

On the technical side, if everything works perfectly for the ezToff.com users, I need to document the widget usage and the REST API used by the Web consoles. I need also to enable the Facebook connector for influencers to communicate on this platform as they can do on Twitter. There's also the Android application (project dretailer mentioned above) to update to benefit from the Android Cloud to Device Messaging (c2dm) mechanism and then to push notifications asynchronously. Still a lot to do but no road block ;)

A+, Dom

How to resize VirtualBox disk images?

2010-07-19T23:56:00.001-04:00

-- Update on May 8, 2012 --

Having to resize an image again, I looked at the VirtualBox documentation before following the CloneVDI route a second time. And I was pleasantly surprised that the version 4.1 of VBoxManage accepts a parameter --resize to the command modifyhd!

The process can be done in a matter of minutes:

Stop the hosted system (Win7 in my case).
Run the following command in the folder of your VDI file
VBoxManage modifyhd <name>.vdi --resize <size-in-mb>
Start the virtual machine.
Open the disk manager tool (Use the Menu Windows, type disk man in the search box, and select 'Create and format hard disk partitions').
You should see your drive with the initial partition(s) and new free space.
Click on the partition to extend and choose the command 'Extend Volume' in the contextual menu.
And voilà.

No need to copy the VDI on the host machine. Very fast and robust process.

'Extend Volume' option in the contextual menu.

-- Original post on July 19, 2010 --

Quick post to share a wonderful VirtualBox companion: CloneVDI!

Almost two months ago, I switched from Windows XP to GNU/Linux with Ubuntu 10.04 distribution. Everything goes very well and I do not regret the move.

In my day-to-day job, as the architect/developer/tester of the Twetailer project, engine and of many of its clients, I still need to run programs on the Windows OS, especially the series of Internet Explorer 7 & 8 (IE 6 is killed, isn't it?).

To verify my test suites for Internet Explorer, I rely on VirtualBox running the initial Windows XP release. Because the disk size requirement for the initial version is low, I stupidly created a small 4 GB disk image! Then it required hours to load and install the service packs 2 & 3, plus IE 8, plus the latest .Net framework, plus the security updates. Be careful: 4 GB is way too small to store the system, the virtual memory page file, and the additional stock coming with the service packs and others!

Few days after the initial setup, I was facing the "Not enough space available" warning :-( Instead of wasting another set of hours in a re-installation, I googled virtualbox increase vdi size, and the first article I found was from the VirtualBox forums. I thought it was a good sign and I started reading... but I became quickly disappointed because the given explanations required many tricks and time to setup too. Then I jumped to the last pages (page 6 to be precise) to read:

...

The new way:
Run the CloneVDI tool. Enter name of source VDI and desired disk size into dialog box. Click "Proceed" button. Expect to spend maybe 5 minutes for a typical VDI, longer of course with big drives.

The CloneVDI tool has existed since mid September 2009. It was created specifically so as to remove the need to recommend an embarassingly complicated rigmarole for performing what should have been a simple task. So, in late May 2010 it is quite disheartening to see people still joining the site in order to provide uninformed endorsement of the obsolete procedure.

The post was from Don Milne, alias mpack, the creator of the CloneVDI tool.

Then I loaded CloneVDI from the referenced post, installed Wine with the Ubuntu Software Center, ran CloneVDI, selected my initial image, specified a new name and a new size (now 10 GB), and all the magic transformation occurred in less than 1 minute!

CloneVDI pane: clone an virtual image with an increased size in just a few clicks!

Warning: It seems the cloning only works up to the first snapshot, as my clone did not get the snapshots. This was not a issue for me but be careful on your side because it might be necessary to merge the snapshots. As the cloning is very fast, producing a new image per snapshot should work-around the issue.

Anyway, I wholeheartedly recommend CloneVDI when it's time to allocate more disk space to a VirtualBox machine!

A+, Dom

The joy of Ubuntu

2010-06-07T12:21:00.001-04:00

Few decades ago, most of my work was done on a SUN hardware running Solaris. The Université de Rennes I, France, was providing the Internet access and Mosaic, followed few times later by Netscape, was my favorite browser. At that time, I was preparing a PhD and most of the online discussions were conducted in newsgroups and emails.

One week ago, I definitively switched the OS of my Thinkpad T61 from Windows XP to Ubuntu Lucid Lynx (10.04). In some ways, the Ubuntu environment is not that different from the Solaris I used to work on. For example, I was very pleased to use again an efficient Virtual Desktop system and to benefit from the native Compose key mechanism that allows me to type any crazy sequences producing foreign characters.

Having been pushed on the Microsoft side by my various employers, it seems I forgot all the good sides of the Unix environments. Microsoft marketing has been very brilliant: as many others, I accepted the PC environment limitations! Do you remember the "cooperative multitasking" of Windows 3.1? Do you remember that changing a registry key transformed your Windows NT Workstation in a much capable Windows NT Server? Do you remember that Windows 7 is really the first shiny interface with transparency and gadgets? (Sorry: Vis-what? I don't know ;)

To be honest, let's recognize that the Unix environments did not progress much. On the open source side, the Linux distributions provide a very fragmented offer. The success of some shiny distributions like Knoppix and Ubuntu is fairly recent.

As a geek, I would say that the most impressive feature in these distributions is the 3D rendering engine Compiz and its Cube plugin! It's just amazing.

When I decided to switch to Ubuntu, I wondered if I would lose some features. So far, all have their counterpart for the Linux environment, sometimes with many additional benefits. I did not have to even install Wine, the Windows emulator! Here are the tools that were important to me and I carried over:

Dropbox: has Linux/MacOS/Windows distributions.
Keepass: had to convert the database to 1.x format and now use KeepassX which has Linux/MacOS/Windows distributions.
Aptana/Android SDK/App Engine SDK: equivalent Java packages.
Firefox/Chrome/Add-ons: have Linux/MacOS/Windows distributions.
Skype (for VOIP calls and screen shares)...
OpenOffice.
VirtualBox.
git.

I am so impressed with the system that I've also switched my old Toshiba M40 (one P4 core and a NVidia card) and now my kids have an easy access to many free games and educational tools!

A+, Dom

What motivates us?

2010-05-28T10:16:00.001-04:00

One year and half ago, I wrote the blog post Career Advice '08. The key idea was about encouraging people to develop their own expertise to be successful in their career. Don't wait for your management to give you something exciting, do it yourself!

Two weeks ago, I wrote another blog post Work around the general laziness. In some ways, I reported on my disappointment of not seeing enough autonomous experts, and that entrepreneurs have to compose with variously skilled teams.

I finished my second expose by mentioning the Twetailer's work-for-attribution offer. I explained that many people liked the concept and that few of them have committed to deliver something with their own schedule. Each one at its own pace builds an extended skill set.

What are the commonalities between the contributors?

They have a regular job so the money is not an issue.
They are performers and have a high level of satisfaction.

I was such a contributor when I started working with Steven, before jumping on the entrepreneur-side to help developing the business from the inside.

I had not really identified the sources of my motivation until I saw Dan Pink's illustration of "What motivates us", for a talk given by the Royal Society for the encouragement of Arts, Manufactures and Commerce (theRSA.org).

The three factors that lead to performance and personal satisfaction:

Autonomy

Mastery

Purpose.

I strongly encourage you to look at that video, part for the fun of viewing Dan Pink in action, part for the delivered message. I encourage you also to share your experience as a comment below ;)

A+, Dom

Work around the general laziness ;)

2010-05-18T15:48:00.007-04:00

This blog entry is primarily inspired by a rant from Jason Calacanis, during the episode #47 of This Week in Startups (full show, or directly starting near 5:20). During that show, Jason goes against the Generation Y people who are usually lazy and want everything before actually delivering...

Another source of inspiration is Mark Suster, the co-host of This Week in Venture Capital, with a series of posts culminating with The Long-Term Value of Loyalty.

What's their point-of-view?

Jason is mostly focusing on a group of people, a group that has been educated with a different mindset from his own one. Gen-Y people have more concerns about ecology in general, for example, they are used to getting free or cheap digital goods, and many of them live with their parents longer than before...

If I can agree on the fact that many Gen-Y people seem lazier than the the Gen-X or the baby boomers at the same age, I think the key differentiators is their relationship with money: the Gen-Y people think they need less money, so they don't work that hard to get always more!

In his first post of the series, Mark pinpoints job hoppers, employees who can resign anytime, without much respect for their commitment or for the situation of the company. If Mark readjusts the context in the second post by writing that “quitting a job because it's a mess is OK”, Mark still thinks that being “loyal” is important and everyone should stay loyal for long term benefits...

I can agree with Mark that working with such individuals is risky, that developing a business with not-so-loyal people is tough. However, I think that building a business with people from different mindsets and backgrounds is more valuable. To me, diversity, if you can manage it, is more important than loyalty!

Is laziness a syndrome limited to Generation Y?

T-shirt Laziness Pays Off Now from SNORGTEES

Before becoming an entrepreneur, I was an employee like many others. In France, I worked for a very small company and then for a multinational corporation before immigrating to Canada. My first job here was for a medium Montreal company which was later bought by Oracle. I quitted Oracle to go with IBM Rational, which I quitted to go with Compuware. In addition to my professional life, I've been involved in many non governmental organizations.

Along my life, I've always been curious, interested in learning new subjects, debating around them, and trying to implement the best ideas, especially when I'm passionate. I've been lucky to team up with great people (note that I learned a lot too from the failed partnerships) and I've gotten excellent mentors. When I write “lucky”, I mean “I worked hard to consider myself as lucky.”

All in all, I can honestly say I haven't seen a lot of people working very hard at work, not that they are lazy, just that they have different priorities, different motivations. Although I've already proposed many times to be a mentor, to give lunch-n-learn talks, to organize specific trainings, etc., -on my own time, for the only benefit of the recipient- only few people have followed.

I've observed the “good enough” attitude with any types of people: old or young, men or women, immigrants or native country, Europeans or Americans, etc. In Western countries, we don't have to fight for a shelter or to find food, so the sense of urgency is blunted. Why would people with already enough (enough money, enough responsibilities, enough social involvement, etc.) go for more?

Maybe the Gen-Y people are worse than the others,because their “good enough” level is lower than before. IMHO, they are just like the common crowd, just normal people in our modern world.

How to deal with the general laziness?

As mentioned before, I think the diversity is important. Trying to stay in closed vacuum with the elite can help a bit, but not for a long time. The key point is to compose teams with top elements and less skilled ones. The newbies can learn from the experienced people and more help around them will reduce the “single point of failure” risk.

For sure, you cannot reward the in-learning people as you do reward the top team members. As I explain to my kids, there's a consequence to everything! If you work hard, if your help is valuable, if you go beyond your tasks, the system should reward you accordingly. If it does not come immediately, it should be clear that it will come eventually if everything goes well. If the hard workers don't get a tangible ROI, I think it's normal to expect them to slow down or to quit... (to expect them slowing down ?)

To me, the key factor in a successful project is the commitment of the participants. It's not that important that this intern has a lower velocity than an experienced engineer because we can plan accordingly. What's more important is that you trust that he's going to deliver as expected, or if he has some troubles that he's going to report them as soon as possible.

In the case of software developers evolving in an Agile environment, measuring their work progress is not really an issue. If the team is correctly equipped, everyone is accountable. In the peripheral teams (product managers, marketing, sales representatives, etc.), commitments are more difficult to get, and the more sources of non-productivity there are, the fewer chances of success you get.

A real case, please!

My partner Steven and I have developed the concept of work for attribution. Because we're a startup, we cannot offer salaries in exchange to work. However, we offer to attribute back the work to the ones who have delivered it. Immediately, contributors can use our environment as a lab to test and develop new ideas. We have a working product and it's up to them to adapt it, to make it better.

If our projects are successful, if we can cash them at one point, the contributors will be part of the success and then we'll try to reward them. If a big player wants to acquire us, top contributors will be probably part of the deal.

So far, many people have liked the concept and a few of them have committed to deliver something specific. Some of them are Gen-Y people, some are immigrants, and everyone work at different rates. If I can rely on people I trust, on people who are going to deliver what they have committed to, whoever they are, it's very cool!

A+, Dom

Social Software, Cynapse, and Open Platforms

2010-04-21T12:22:00.003-04:00

What's a Social Software?

These days, most of white collar workers have to deal with computers on a daily basis. Computers are everywhere: from the front desk to allow receptionists to get to the company directory up to garage doors checking who's in, who's out. There are so different types of computers that some of them go unnoticed!

The scope of this post is limited computers used to collaborate:

Make appointments
Produce documents
Review & comment documents
Forward documents
Poll users
Manage tasks

When it's time to collaborate, the vast majority of computer users rely on e-mails. E-mail is probably the most essential tool now, and without e-mails, many are lost. Do you remember the BlackBerry syndrome of the RIM product early adopters? Yeah, the one that wakes up people middle of the night so they can get new e-mails ;)

With tools like Microsoft Outlook, Apple iCal, and Google Calendar, more users relying on calendar tools to organize their time. This is neat because with such tools you can sometimes see up front if the targeted time slot is good for other attendees. And when attendance confirmation comes, the meeting information are updated automatically without requiring a specific triage in the mailbox.

When people have to share pictures, because the ones modern digital camera can generate are so big and very few people have the knowledge to reduce their size nicely, more and more people resort to online hosting services like flickr or Picasa. In addition to process images to transit on the Web (while still available in high resolution for printing purposes), sharing pictures via links is way lighter for the mail system. It has also the additional benefit that pictures can be removed safely without one someone else to purge his/her mailbox.

With review sites like cnet or Epinions.com, more and more users have started to give their opinions online to share the joy of being a owner of a wonderful gadget, or to inform others that such a gadget is crap! Involving readers and customers to share their experiences is the key element of the social software movement.

So what's the relation between Cynapse and social software:

Cynapse is a central place where people can collaborate online with the most practical tools without relying on e-mails, a place where collaboration and communication do not loose their context.

Why Cynapse?

When my partner Steven and I started to work on the !twetailer, we knew that we wanted to collaborate online, within a reliable and protected environment that could handle many document types: user stories, specifications, diagrams, mock ups, pictures, bookmarks, discussions, etc.

As IBM-ers, we looked among the tremendous IBM products and we selected Lotus GreenHouse which was still in beta. The main flaw we experienced was its disruptive slowness. Another issue to us was our quasi-impossibility to influence the development path to fix the painful issues. Just 2 guys in a big user community have a very little impact...

We then decided to switch to Elgg, a free and open source software, that our hosting service offered. Elgg is probably a good tool but it did not match our needs. The variety of entities was limited as our ability to fix the issues. Probably, it would have been better to host the service at home and to extend its model. Because it would have distracted us from our project, we looked for a better solution.

At this time, Steven was studying the social software offering and Cynapse came out as a good fit to us:

The service out-of-the-box was promising and was still under active development.
They had an affordable offer: free self hosted service, managed service for a small subscription, managed hardware also for a fee.
They had an active user community which was hosted on the tool itself (the “eat your own dog food” principle).
Whatever solution we choose, we are free to upgrade/downgrade/exit.

Cynapse is an Indian company and its managers are really great. For practical reasons, we decided to go with the managed service online. When Steven contacted them, asking for a rebate in exchange to us blogging about our experience with the tool and us feeding them with enhancement requests from the development point of view, they accepted and sustained our involvement.

Disclosure: Because the tool and the team is great, Milstein & associates is now a business partner and can resell and offer services on the top of the tool.

After one full year and two upgrades, we are very happy with the tool and its support. At one point, we have developed an offer for schools available at edu.cyn.in which has been extremely well adapted by the kids!

In addition to the collaboration aspect, Cynapse with its offering of various tools in one platform allows users to develop their online reputation in a controlled environment. As Craig Newmark, the founder of craigslist, mentions in a blog post:

People use social networking tools to figure out who they can trust and rely on for decision making. By the end of this decade, power and influence will shift largely to those people with the best reputations and trust networks, from people with money and nominal power. That is, peer networks will confer legitimacy on people emerging from the grassroots.

The Cynapse environment allows users to highlight their work: statistics about contributions and comments are shared on the main page, readers can note the published materials, etc. When people are new to social software, Cynapse offers a simple way to identify the people others “trust” and allows good contributors to build their reputation.

Aside our work on the !twetailer project, because the tool fits our needs, because kids adopted very well, we have proposed it to traditional companies (the ones that relies on Microsoft Outlook and shared folders for their collaboration). For now, the feedback is positive but it's too early to advance any adoption rate ;)

Why Open Platforms?

As a developer, I'm an heavy user of open source software (development environment, source control, build system, etc.). I am also a contributor myself with two open libraries hosted on github:

A set of utilities for Web application developers (Java, Python, JavaScript) which offers:

Globalization features: from one set of central repositories (TMX format) to programming language dependent resource bundles;
JSON related objects: to ease the parsing/generation of JSON structure

An adaptation of the Amazon Flexible Payment Service (FPS) library for the Java environment of Google App Engine.

And the content of this blog is offered under the Common Creative Licence By-NC-SA, which allows using beyond the traditional “fair-use” as long as you cite the source ;) I very like open platform for the reasons Larry Lessig gave in speech on . Open source is good for innovation for geeks like myself. For enterprises, I would argue that the key point is the data access: at anytime, someone can get the data out of an open source project without risking any patent infringement, without risky reverse-engineering. For sure, it won't be free as in “free beer” but they'll be free to get their data back. Some people will argue that close software often offer a way to export data in a standard format (like a SQL dump for a database) and allow then to import them somewhere else, but that's only true if no feature are dropped during the export and if all features from one can be activated in two during the import.

To summarize my point, I would say that open source software allow anyone to exit at anytime while continuing to control the data.

As a collaboration tool, Cynapse is maybe not the best one but it offers competitive advantages for the right price (for free if you have the team and expertise to manage it, for fee if you choose the hassle-free solution of the hosted service on Amazon AWS) while letting users migrating to another platform anytime.

Important point to consider during the selection of an open source solution: the quality of its community. More active is the community, better are your chances that issues you are facing have been documented, better your chances to see your (excellent) enhancement requests supported by others. Being able to contribute to an active community (by submitting bug reports, by answering others' questions, by submitting patches) have also the side benefit of improving your reputation.

A+, Dom

Amazon FPS library for the Google App Engine environment

2010-03-21T23:14:00.002-04:00

Here is a post on Amazon Flexible Payments System (FPS) and the beauty of the open source model!

Amazon FPS as the payment platform

!twetailer is an application connecting people to people, consumers to retailers and vice-versa. The business model is freemiun-based:

Posting demands, receiving proposals, and accepting proposals is available for free to any customers.
Listening to demands, proposing products or services, and closing deals is available for free to any registered retailers.
At one point, !twetailer is going to offer retailers to ask consumers to pay for the accepted products or services. The payment platform is going to be Amazon FPS¹.

Amazon FPS is a really neat platform, the only one to my knowledge allowing to organize money transfers between third-parties. With Amazon FPS, !twetailer will be able to convey money from the consumers directly to the retailers, without the money transiting on !twetailer own account! This is a really safe mechanism.

As a quick introduction to Amazon FPS, I would strongly suggest you listen to that one hour webcast introduced on Amazon Payments blog on April 7, 2009: Monetize your innovation with Amazon FPS. If you use the open-source tool VideoLAN VLC, you can load the ASX file directly from Akamai from here.

Amazon and the open-source model

Amazon FPS, as many others Amazon Web Services (AWS), allows third-party applications to use its services through very simple APIs which are HTTP based! The libraries that developers need to use are mostly wrappers over HTTP connections with some specific controllers formatting the requests and signing them (to avoid a man-in-the-middle process tampering them).

Because HTTP is an open protocol and because Amazon could not probably develop its libraries for all possible Web servers, Amazon opened the libraries' source and attached to them a very liberal license².

This is a very respectable attitude regarding their customers and also very well thought on the business-side: if developers can adopt their libraries for their own needs, Amazon won't have to pay for the corresponding development and it will enlarge the set of applications their platform can serve!

Amazon FPS on Google App Engine platform

The !twetailer server-side logic is Java based and dropping the Amazon FPS library freshly compiled in war/WEB-INF/lib is simple. However, the Amazon FPS code cannot run as-is because of few App Engine limitations...

The first one is encountered when the application needs to build the URL that will launch the co-branded service, a page that will allow consumers to pay for the service or product previously proposed by a retailer.

The static method HttpURLConnection.setFollowRedirects(boolean) controls the VM behavior and is then guarded by a JVM permission.

Read the incident report in the Google App Engine discussion group.

Fixing this issue is simple: tune the ability to follow redirection on the connection itself instead of applying the settings globally.

The second issue is really major:

The library uses the Jakarta Commons HttpClient component to convey payment requests from the application to the Amazon infrastructure. And many of its underlying calls are blocked in Google App Engine Java environment.

I asked for advices on AWS FPS forums. But without response, I have decided to go with my own wrapper of the Google URL Fetch mimicking the HttpClient HttpConnectionManager and HttpConnection classes.

Wrappers of Google URL Fetch for Amazon FPS

Following Amazon's leadership, I offer the URL Fetch wrappers that allows Amazon FPS to work on Google App Engine platform:

Same Apache license, version 2.0.
Code hosted on GitHub, in the repository DomDerrien/amazon-fps-gaej.
For now, only two Amazon classes have been modified: com.amazonaws.ipnreturnurlvalidation.SignatureUtilsForOutbound and com.amazonaws.fps.AmazonFPSClient.
My two additions are the classes domderrien.wrapper.UrlFetch.UrlFetchConnectionManager and domderrien.wrapper.UrlFetch.UrlFetchHttpConnection.

The code currently available works in the simple scenario !twetailer needs. But it is still under development. And the test suite covering it is not yet completed.

UrlFetchConnectionManager class definition

/******************************************************************************* 
 *  Adaptation for the Amazon FPS library to work on the Java platform of
 *  Google App Engine.
 *  
 *  Copyright 2010 Dom Derrien
 *  Licensed under the Apache License, Version 2.0
 */

package domderrien.wrapper.UrlFetch;

import org.apache.commons.httpclient.ConnectionPoolTimeoutException;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpConnection;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.params.HttpConnectionManagerParams;

public class UrlFetchConnectionManager implements HttpConnectionManager {

 private HttpConnectionManagerParams params;
 private HttpConnection connection;
 
 public void closeIdleConnections(long timeout) {
  throw new RuntimeException("closeIdleConnections(long)");
 }

 public HttpConnection getConnection(HostConfiguration hostConfiguration) {
  throw new RuntimeException("getConnection(HostConfiguration)");
  // return null;
 }

 public HttpConnection getConnection(HostConfiguration hostConfiguration, long timeout) throws HttpException {
  throw new RuntimeException("getConnection(HostConfiguration, long)");
  // return null;
 }

 public HttpConnection getConnectionWithTimeout(HostConfiguration hostConfiguration, long timeout) throws ConnectionPoolTimeoutException {
  // As reported in http://code.google.com/appengine/docs/java/urlfetch/usingjavanet.html#Java_Net_Features_Not_Supported
  // > The app cannot set explicit connection timeouts for the request.
  if (connection != null) {
   releaseConnection(connection);
  }
  connection = new UrlFetchHttpConnection(hostConfiguration);
  return connection;
 }

 public HttpConnectionManagerParams getParams() {
  return params;
 }

 public void releaseConnection(HttpConnection connection) {
  connection.releaseConnection();
 }

 public void setParams(HttpConnectionManagerParams params) {
  // Parameters set in AmazonFPSClient#configureHttpClient:
        // - ConnectionTimeout: 50000 ms
  // - SoTimeout: 50000 ms
  // - StaleCheckingEnabled: true
  // - TcpNoDelay: true
  // - MaxTotalConnections: 100 (as proposed in the default config.properties file)
  // - MaxConnectionsPerHost: 100 (as proposed in the default config.properties file)

  this.params = params;
 }
 
}

UrlFetchConnection class definition

/******************************************************************************* 
 *  Adaptation for the Amazon FPS library to work on the Java platform of
 *  Google App Engine.
 *  
 *  Copyright 2010 Dom Derrien
 *  Licensed under the Apache License, Version 2.0
 */

package domderrien.wrapper.UrlFetch;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetAddress;
import java.net.MalformedURLException;
import java.net.Socket;
import java.net.SocketException;
import java.net.URL;

import javamocks.io.MockInputStream;
import javamocks.io.MockOutputStream;

import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpConnection;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.params.HttpConnectionParams;
import org.apache.commons.httpclient.protocol.Protocol;

import com.google.appengine.api.urlfetch.FetchOptions;
import com.google.appengine.api.urlfetch.HTTPHeader;
import com.google.appengine.api.urlfetch.HTTPMethod;
import com.google.appengine.api.urlfetch.HTTPRequest;
import com.google.appengine.api.urlfetch.HTTPResponse;
import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.appengine.api.urlfetch.URLFetchServiceFactory;

public class UrlFetchHttpConnection extends HttpConnection {

 private static URLFetchService urlFS = URLFetchServiceFactory.getURLFetchService();

 private HostConfiguration hostConfiguration;
 private HTTPRequest _request;
 private HTTPResponse _response;
 private MockOutputStream _requestBody = new MockOutputStream();
 private MockInputStream _responseBody = new MockInputStream();

 
 private HTTPRequest getRequest() throws MalformedURLException {
  if (_request == null) {
   _request = new HTTPRequest(
     new URL(hostConfiguration.getHostURL()), 
     HTTPMethod.POST, // AmazonFPSClient#invoke(Class, Map) uses only POST method
     FetchOptions.Builder.disallowTruncate().followRedirects()
   );
  }
  return _request;
 }
 
 private static final String SEPARATOR = ": ";
 private static final int SEPARATOR_LENGTH = SEPARATOR.length();
 private static final String NEW_LINE = "\r\n";

 private HTTPResponse getResponse() throws MalformedURLException, IOException {
  if (_response == null) {
   // Get the response from the remote service
   _response = urlFS.fetch(getRequest());
   // Rebuild stream of HTTP headers (except the HTTP status retrieved from readLine(String) method)
   StringBuilder buffer = new StringBuilder();
   for (HTTPHeader header: _response.getHeaders()) {
    buffer.append(header.getName()).append(SEPARATOR).append(header.getValue()).append(NEW_LINE);
   }
   buffer.append("Content-Length: ").append(_response.getContent().length).append(NEW_LINE);
   buffer.append(NEW_LINE);
   // Rebuild stream of HTTP content (chunked-encoded)
   buffer.append(Integer.toString(_response.getContent().length, 16)).append(";chunk size").append(NEW_LINE);
   buffer.append(new String(_response.getContent())).append(NEW_LINE);
   buffer.append("0;").append(NEW_LINE);
   _responseBody.resetActualContent(buffer.toString());
  }
  return _response;
 }

 /**
  * Default constructor
  * @param hostConfiguration
  */
 public UrlFetchHttpConnection(HostConfiguration hostConfiguration) {
  super(hostConfiguration);
  this.hostConfiguration = hostConfiguration;
 }

 @Override
 protected void assertNotOpen() throws IllegalStateException {
  throw new RuntimeException("assertNotOpen()");
 }

 @Override
 protected void assertOpen() throws IllegalStateException {
  assert(_response != null);
 }

 @Override
 public void close() {
  // Nothing to do!
 }

 @Override
 public boolean closeIfStale() throws IOException {
  // Safe call, passed to the inherited method
  return super.closeIfStale();
 }

 @Override
 protected void closeSocketAndStreams() {
  throw new RuntimeException("closeSocketAndStreams()");
 }

 @Override
 public void flushRequestOutputStream() throws IOException {
  getRequest().setPayload(_requestBody.getStream().toString().getBytes());
 }

 @Override
 public String getHost() {
  return hostConfiguration.getHost();
 }

 @Override
 public HttpConnectionManager getHttpConnectionManager() {
  throw new RuntimeException("getHttpConnectionManager()");
 }

 @Override
 public InputStream getLastResponseInputStream() {
  throw new RuntimeException("getLastResponseInputStream()");
 }

 @Override
 public InetAddress getLocalAddress() {
  throw new RuntimeException("getLocalAddress()");
 }

 @Override
 public HttpConnectionParams getParams() {
  return new HttpConnectionParams();
 }

 @Override
 public int getPort() {
  return hostConfiguration.getPort();
 }

 @Override
 public Protocol getProtocol() {
  return hostConfiguration.getProtocol();
 }

 @Override
 public String getProxyHost() {
  throw new RuntimeException("getProxyHost()");
 }

 @Override
 public int getProxyPort() {
  throw new RuntimeException("getProxyPort()");
 }

 @Override
 public OutputStream getRequestOutputStream() throws IOException, IllegalStateException {
  return _requestBody;
 }

 @Override
 public InputStream getResponseInputStream() throws IOException {
  return _responseBody;
 }

 @Override
 public int getSendBufferSize() throws SocketException {
  throw new RuntimeException("getSendBufferSize()");
 }

 @Override
 protected Socket getSocket() {
  throw new RuntimeException("getSocket()");
 }

 @Override
 public int getSoTimeout() throws SocketException {
  throw new RuntimeException("getSoTimeout()");
 }

 @Override
 public String getVirtualHost() {
  throw new RuntimeException("getVirtualHost()");
 }

 @Override
 protected boolean isLocked() {
  throw new RuntimeException("isLocked()");
 }

 @Override
 public boolean isOpen() {
  // Safe call, passed to inherited method
  return super.isOpen();
 }

 @Override
 public boolean isProxied() {
  // Safe call, passed to inherited method
  return super.isProxied();
 }

 @Override
 public boolean isResponseAvailable() throws IOException {
  return _response != null;
 }

 @Override
 public boolean isResponseAvailable(int timeout) throws IOException {
  return _response != null;
 }

 @Override
 public boolean isSecure() {
  return hostConfiguration.getPort() == 443;
 }

 @Override
 protected boolean isStale() throws IOException {
  throw new RuntimeException("isStale()");
 }

 @Override
 public boolean isStaleCheckingEnabled() {
  throw new RuntimeException("isStaleCheckingEnabled()");
 }

 @Override
 public boolean isTransparent() {
  // Safe call, passed to the inherited method
  return super.isTransparent();
 }
 
 @Override
 public void open() throws IOException {
  // Nothing to do
 }

 @Override
 public void print(String data, String charset) throws IOException, IllegalStateException {
  // Save the passed HTTP headers for the request
  int idx = data.indexOf(SEPARATOR);
  if (idx != -1) {
   String name = data.substring(0, idx);
   String value = data.substring(idx + SEPARATOR_LENGTH).trim();
   getRequest().addHeader(new HTTPHeader(name, value));
  }
  // Other information are just ignored safely 
 }

 @Override
 public void print(String data) throws IOException, IllegalStateException {
  throw new RuntimeException("print(string): " + data);
 }

 @Override
 public void printLine() throws IOException, IllegalStateException {
  throw new RuntimeException("printLine()");
 }

 @Override
 public void printLine(String data, String charset) throws IOException, IllegalStateException {
  throw new RuntimeException("printLine(string, String): " + data + " -- " + charset);
 }

 @Override
 public void printLine(String data) throws IOException, IllegalStateException {
  throw new RuntimeException("printLine(string): " + data);
 }

 @Override
 public String readLine() throws IOException, IllegalStateException {
  throw new RuntimeException("readLine()");
 }

 private boolean waitForHttpStatus = true;
 
 @Override
 public String readLine(String charset) throws IOException, IllegalStateException {
  if (waitForHttpStatus) {
   // Dom Derrien: called only once to get the HTTP status, other information being read from the response output stream
   int responseCode = getResponse().getResponseCode();
   String line = "HTTP/1.1 " + responseCode;
   switch(responseCode) {
    case HttpStatus.SC_OK: line += " OK"; break;
    case HttpStatus.SC_BAD_REQUEST: line += " BAD REQUEST"; break;
    case HttpStatus.SC_UNAUTHORIZED: line += " UNAUTHORIZED"; break;
    case HttpStatus.SC_FORBIDDEN: line += " FORBIDDEN"; break;
    case HttpStatus.SC_NOT_FOUND: line += " NOT FOUND"; break;
    case HttpStatus.SC_INTERNAL_SERVER_ERROR: line += " INTERNAL SERVER ERROR"; break;
    case HttpStatus.SC_SERVICE_UNAVAILABLE: line += " SERVICE UNAVAILABLE"; break;
    default: line = "HTTP/1.1 " + HttpStatus.SC_BAD_REQUEST + " BAD REQUEST";
   }
   waitForHttpStatus = false;
   return line;
  }
  throw new RuntimeException("readLine(String)");
 }

 @Override
 public void releaseConnection() {
  // Do nothing, connection closed automatically...
 }

 @Override
 public void setConnectionTimeout(int timeout) {
  throw new RuntimeException("setConnectionTimeout(int)");
 }

 @Override
 public void setHost(String host) throws IllegalStateException {
  throw new RuntimeException("setHost(String");
 }

 @Override
 public void setHttpConnectionManager(HttpConnectionManager httpConnectionManager) {
  throw new RuntimeException("setHttpConnectionManager(HttpConnectionManager");
 }

 @Override
 public void setLastResponseInputStream(InputStream inStream) {
  // Safe call, passed to inherited method
  super.setLastResponseInputStream(inStream);
 }

 @Override
 public void setLocalAddress(InetAddress localAddress) {
  throw new RuntimeException("setLocalAddress(InetAddress)");
 }

 @Override
 protected void setLocked(boolean locked) {
  // Safe call, passed to inherited method
  super.setLocked(locked);
 }

 @Override
 public void setParams(HttpConnectionParams params) {
  throw new RuntimeException("setParams(HttpConnectionParams)");
 }

 @Override
 public void setPort(int port) throws IllegalStateException {
  throw new RuntimeException("setPort(int)");
 }

 @Override
 public void setProtocol(Protocol protocol) {
  throw new RuntimeException("setProtocol(Protocol)");
 }

 @Override
 public void setProxyHost(String host) throws IllegalStateException {
  throw new RuntimeException("setProxyHost(String)");
 }

 @Override
 public void setProxyPort(int port) throws IllegalStateException {
  throw new RuntimeException("setProxyPort(int)");
 }

 @Override
 public void setSendBufferSize(int sendBufferSize) throws SocketException {
  throw new RuntimeException("setSendBufferSize(int)");
 }

 @Override
 public void setSocketTimeout(int timeout) throws SocketException, IllegalStateException {
  // Safe call, passed to inherited method
  super.setSocketTimeout(timeout);
 }

 @Override
 public void setSoTimeout(int timeout) throws SocketException, IllegalStateException {
  throw new RuntimeException("setSoTimeout(int)");
 }

 @Override
 public void setStaleCheckingEnabled(boolean staleCheckEnabled) {
  throw new RuntimeException("setStaleCheckingEnabled(boolean)");
 }

 @Override
 public void setVirtualHost(String host) throws IllegalStateException {
  throw new RuntimeException("setVirtualHost(String)");
 }

 @Override
 public void shutdownOutput() {
  throw new RuntimeException("shutdownOutput()");
 }

 @Override
 public void tunnelCreated() throws IllegalStateException, IOException {
  throw new RuntimeException("tunnelCreated()");
 }

 @Override
 public void write(byte[] data, int offset, int length) throws IOException, IllegalStateException {
  throw new RuntimeException("write(byte[], int, int): " + new String(data) + ", " + offset + ", " + length);
 }

 @Override
 public void write(byte[] data) throws IOException, IllegalStateException {
  throw new RuntimeException("write(byte[]): " + new String(data));
 }

 @Override
 public void writeLine() throws IOException, IllegalStateException {
  // Safe call, new line being inserted automatically by the HTTPRequest renderer
 }

 @Override
 public void writeLine(byte[] data) throws IOException, IllegalStateException {
  throw new RuntimeException("writeLine(byte[]): " + new String(data));
 }
}

Anyone is free to fork it for his own needs. Be careful with the code because I deliver it without warranties! If you have issues to report, if you can document how to reproduce them, depending on my workload, I will help you. If you fix the issue on your side, I will be happy to merge the corresponding patches into my main branch.

I hope this helps,
A+, Dom
--
Notes:

At least in United States of America until Amazon extends its coverage to company without a US bank account.
Apache License, Version 2.0, January 2004, which allows users to make modifications while keeping them private.

Ladies and gentlemen, here is !twetailer

2010-03-11T10:37:00.000-05:00

On Tuesday March 9 evening, my partner Steven Milstein and I attended a Montreal NewTech event. As three other companies, Steven pitched our project !twetailer to a crowd of 30-40 people. After few closed presentations, it was our first public pitch and it went very well! Read Steven's blog post for the details.

Background

I met Steven while working for IBM Rational. He was the Business Analyst while I was the Software Architect for the Web client of the Rational Portfolio Manager product. When Steven came to me with the Reverse Retailing idea, we agreed to develop it as a proof-of-concept for our own expertise.

We worked so well that we submitted our project to TechCrunch50 (TC50). If we were not among the 50 finalists, we passed the first selection round on 1000+ applicants and we had the chance to present it to Jason Calacanis during 15 minutes last August.

Boosted by the appreciation we got from Jason, we continued to focus on developing user stories and the corresponding code. We were ready to demo the full cycle early this January. Two months later, after many tests and fine tuning, we are opening !twetailer to broader audience!

Presentation

!twetailer first objective is to “connect supply and demand”, specifically “connect consumers to retailers” in its first version.

Look at this original presentation to get the sense of !twetailer ;) We made it months ago but it's still very accurate. Don't miss the part describing the hub, starting at 3:00.

After the presentation, we got many feedback from the audience and most of our interlocutors got it right. Here is the interpretation of Max Maheu, the presenter of the SolidWild company:

Max: If I register my company and listen for the tags “3d printing prototype logo”, all people using !twetailer will get to me?
Me: Exactly. All demands posted with one or many corresponding tags in the Montreal area will be forwarded to you, will be routed to you for free ;)
...
Me: At this stage, it's possible you'll get unrealistic demands, many of them with the #demo tag. But if you respond to them, that means if you propose something, your message will be routed back to the consumers and your business information will be displayed to them!
Max: It's like advertising my business then! Cool.

What's next?

As mentioned by Guy Kawasaki in Montreal in March 2009, we have chosen to "launch early and to correct progressively". If all the delivered features are fully functional, there is still a long road to go in delivering the full feature set.

At this step, we need to get users, that means consumers and retailers, playing with the system and giving us their feedback.

To collect the information, we have organize the community site twetailer.cyn.in, implemented with the excellent social software from Cynapse. To value the contributions, we have setup a Work-for-attribution protocol: any community member that makes a significant contribution will have his/her work publicly recognized. Any volunteer?

We need also to work on the marketing side:

The brand !twetailer seems too tightly related to Twitter, while using Twitter is just one among the various set of connectors taking to our engine.
If !twetailer initial targets are consumers and retailers in a public market, it can work with closed markets, where wholesalers communicate with manufacturers, for example. This aspect needs to be documented and illustrated.
To ensure a vibrant life to !twetailer, we have plan to open its API to third-party developers, a bit a-la Twitter. Closed friends have already accepted to develop clients (under the Work-for-Attribution CLA) that will exercise it, but we need a stronger communication there too.

Thanks a lot to our families and closed friends for encouraging us on the entrepreneurship path. Thanks to anyone for the feedback because they help us improving !twetailer.

Call to contributors

!twetailer is a big project with a development still growing, so there's a lot of room for anyone to showcase their knowledge!

If you're a developer with Java/JavaScript skills,
If you're a tester with automation experience,
If you're a UI designer & Interactivity specialist,
If you're marketer with a Web 2.0 & Social software experience,
If you're a simple consumer in touch with a vibrant community,
If you're a business owner looking for new ways to reach your customers,
Etc.

Don't hesitate to contact Steven (Steven@Twitter or Steven@Twetailer.com) or myself (Dom@Twitter or Dom@Twetailer.com) and we'll exchange on our community site at twetailer.cyn.in.

A+, Dom

Ma nouvelle vie en tant qu'entrepreneur !

2010-02-08T23:43:00.003-05:00

La démission

Il y a trois semaines, j'ai démissionné de mon poste de Consultant technique pour la compagnie Compuware, à Montréal. Plusieurs facteurs m'ont poussé à prendre cette décision :

Mon projet [toujours secret] personnel arrive à maturité et les conditions de sa mise en marché sont à envisager sérieusement;
Mon boss Paul Czarnik m'assigne sur un projet en relation avec l'intégration des outils de Gomez (récemment achetée par Compuware) dans le produit phare Vantage.
Le récent chaos provoqué par le tremblement de terre en Haïti offre une opportunité de proposer mon projet personnel aux organismes d'aide, dans la même veine de ce qui sera proposé aux organismes de Microfinance, comme Diku Dilenga et Jamii Bora.

D'un côté, il y avait un assignement plus prenant avec mon employeur, et de l'autre, il y avait une demande d'attention plus importante pour aborder une phase critique. J'ai finalement opté pour la démission. Youpi, je suis maintenant indépendant !

Certains amis m'ont demandé si cela avait un rapport avec mon âge (ils se souviennent de mon articule Le cap des 40 ans). Réponse simple: pas du tout, pas de middle-age crisis en vue ;)

C'est plus le cotoiement, même par Internet ou livre interposé, avec des gens inspirant comme Guy Kawasaki (GK's keynote in Montreal) qui me poussent à aller de l'avant, pour « faire quelque chose qui a du sens. »

Et maintenant ?

Du temps où je travaillais en tant qu'Architecte logiciel pour IBM Rational, à Montréal, j'avais eu le plaisir de travailler avec Steven Milstein qui avait le rôle d'Analyste d'affaires.

L'idée originale du projet qui m'occupe dorénavant est de Steven. Avec mon expérience, j'ai pris en charge le développement en nous appuyant sur des outils très accessibles comme :

git et GitHub pour la gestion des sources;
Pivotal Tracker pour le suivi des tâches de développement;
Google App Engine pour l'infrastructure en production, initiallement avec la version Python et puis en version Java (voir Google App Engine Meets Java);
Eclipse et ant comme principaux outils de développement (il y a aussi JUnit, Corbertura, Dojo Toolkit, JSUnit, etc.);

Le travail de collaboration (user stories, blogues techniques, support à la communauté des participants, etc.) est colligé sur la plateforme cyn.in, sélectionnée par Steven pour sa grande qualité et l'évolution rapide de son développement.

Dans l'immédiat, je me concentre sur la mise au point de l'outil, passant d'un rythme de 15 à 20 heures par semaine à quelques 60 heures et plus. En ce moment, le code représente environ 13.000 lignes de code pour 33.000 lignes de test, rien que pour la partie serveur !

Si nous avons fait plusieurs présentations réelles du produit, nous sommes fin prêts pour faire des présentations plus larges, toujours sous le coup d'accord de non divulgation cependant. Idéalement, nous souhaitons avoir des partenariats pour avoir une masse critique et aller de l'avant en ouvrant les portes à tout à chacun ! Excitant, non ?

Parce que nous pensons que notre produit peut aider à la coordination des équipes d'aide aux sinistrés en Haïti, nous avons pris des contacts pour le leur proposer à titre gratuit, mais pour lequel il faut assumer les frais d'exploitation. Comme les contacts se font au niveau gouvernemental, du Canada et du Québec, ou au niveau de grandes ONGs comme la Croix Rouge ou MSF, il risque de se passer un certain temps avant que notre proposition trouve un écho favorable. Le suivi des contacts se poursuit.

Le futur

Pour le moment, je suis sans revenu. N'ayant pas de dettes colossales, ayant un train de vie modeste, je pense pouvoir être en roue libre pendant quelques mois sans que ma famille en patisse.

Avec Steven, en attendant que le grand projet avance, et en exploitant notre expérience des outils de socialisation, nous avons développé une offre reposant sur cyn.in et dirigée vers les écoles et l'initiation des élèves. Le projet s'appelle edu.cyn.in—voir le blog de Steven pour plus de détails.

Pendant la période de lancement, étant toujours « programmeur » sur des technologies récentes, j'ai le sentiment que mon capital de « connaissance technologique » restera solide. Ma communication sur ce blogue en témoignera ;) Le cas échéant, je pourrais toujours aller à la recherche de contrat de consultant pour partager mes connaissances.

Mon activité pour l'organisme de microfinance Diku Dilenga continue, toujours pour le soutien technique, moins en tant qu'investisseur privé. Si le grand projet décolle par contre, je consacrerai du temps pour son adaptation aux organismes de microfinance avec pour objectif d'aider les microentrepreneurs et de d'offrir une source de revenus aux dits organismes.

À suivre donc,
A+, Dom

Unit tests, Mock objects, and App Engine

2009-11-20T18:49:00.010-05:00

For my [still a secret] project which is running on Google App Engine infrastructure [1], I want to make it as solid as possible from the beginning by applying most of the best practices of the Agile methodology [2].

Update 2009/12/05:
With the release of the App Engine Java SDK 1.2.8 (read release notes, I had to update my code and this post on two points:

Without the specification of the JDO inheritance type, the environment assumes it's superclass-table. This type is not supported by App Engine. Only subclass-table and complete-table are supported. In the Entity class described below, I had to add @Inheritance(strategy = InheritanceStrategy.SUBCLASS_TABLE). Read the documentation about Defining data classes for more information.
With the automation of the task execution, the MockAppEngineEnvironment class listed below had to be updated to serve an expected value when the Queue runs in the live environment. Read the details on the thread announcing the 1.2.8. SDK prerelease on Google Groups.

Now, all tests pass again ;)

As written on my post from September 18, I had to develop many mock classes to keep reaching the mystical 100% of code coverage (by unit tests) [3]. A good introduction of mock objects is given by Vincent Massol in his book “JUnit in Action” [4]. To summarize, mock objects are especially useful to inject behavior and force the code using them to exercise complex control flows.

Developing applications for Google App Engine is not that complex because the system has a good documentation and an Eclipse plug-in ease the first steps.

Use case description

Let's consider a simple class organization implementing a common J2EE pattern:

A DTO class for a Consumer;
The DAO class getting the Consumer from the persistence layer, and sending it back with updates; and
A Controller class routing REST requests. The Controller is an element of the implemented MVC pattern

Use case illustration

The code for the DTO class is instrumented with JDO annotations [5]:

Consumer DTO class definition

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
@Inheritance(strategy = InheritanceStrategy.SUBCLASS_TABLE)
public class Consumer extends Entity {
    @Persistent
    private String address;
 
    @Persistent
    private String displayName;
 
    @Persistent
    private String email;
 
    @Persistent
    private String facebookId;
 
    @Persistent
    private String jabberId;
 
    @Persistent
    private Long locationKey;
 
    @Persistent
    private String twitterId;
 
    /** Default constructor */
    public Consumer() {
        super();
    }
 
    /**
     * Creates a consumer
     * @param in HTTP request parameters
     */
    public Consumer(JsonObject parameters) {
        this();
        fromJson(parameters);
    }
 
    public String getAddress() {
        return address;
    }
    
    public void setAddress(String address) {
        this.address = address;
    }
    
    //...
}

My approach for the DAO class is modular:

When the calling code is doing just one call, like the ConsumerOperations.delete(String) method deleting the identified Consumer instance, the call can be done without the persistence layer knowledge.
When many calls to the persistence layer are required, the DAO API offers the caller to pass a PersistenceManager instance that can be re-used from call to call. With the combination of the detachable="true" parameter specified in the JDO annotation for the Consumer class, it saves many cycles.

Excerpt from the ConsumerOperations DAO class definition

/**
 * Persist the given (probably updated) resource
 * @param consumer Resource to update
 * @return Updated resource
 * @see ConsumerOperations#updateConsumer(PersistenceManager, Consumer)
 */
public Consumer updateConsumer(Consumer consumer) {
    PersistenceManager pm = getPersistenceManager();
    try {
        // Persist updated consumer
        return updateConsumer(pm, consumer);
    }
    finally {
        pm.close();
    }
}
 
/**
 * Persist the given (probably updated) resource while leaving the given persistence manager open for future updates
 * @param pm Persistence manager instance to use - let opened at the end to allow possible object updates later
 * @param consumer Resource to update
 * @return Updated resource
 */
public Consumer updateConsumer(PersistenceManager pm, Consumer consumer) {
    return pm.makePersistent(consumer);
}

The following piece of the abstract class BaseOperations shows the accessor made availabe to any controller code to get one handle of a valid PersistenceManager instance.

Excerpt from the abstract BaseOperations DAO class definition

/**
 * Accessor isolated to facilitate tests by IOP
 * @return Persistence manager instance
 */
public PersistenceManager getPersistenceManager() {
    PersistenceManager pm = getPersistenceManagerFactory().getPersistenceManager();
    pm.setDetachAllOnCommit(true);
    pm.setCopyOnAttach(false);
    return pm;
}

To finish the use case setup, here is a part of the controller code which deals with incoming HTTP requests and serves or operates accordingly. This specific piece of code replies to a GET request like:

Invocation: http://<host:port>/API/Consumer/43544"

Response:

{key:43544, displayName:"John", address:"75, Queen, Montréal, Qc, Canada", 
locationKey:3245, location: {id:3245, postalCode:"H3C2N6", countryCode:"CA",
latitude:43.3, longitude:-73.4}, ...}

Excerpt from the ConsumerRestlet Controller class definition

@Override
protected JsonObject getResource(JsonObject parameters, String resourceId, User loggedUser) throws DataSourceException {
    PersistenceManager pm = getBaseOperations().getPersistenceManager();
    try {
        // Get the consumer instance
        Consumer consumer = getConsumerOperations().getConsumer(pm, Long.valueOf(resourceId));
        JsonObject output = consumer.toJson();
        // Get the related information
        Long locationKey = consumer.getLocationKey();
        if (locationKey != null) {
            Location location = getLocationOperations().getLocation(pm, locationKey);
            output.put(Consumer.LOCATION, location.toJson());
        }
        // Return the complete set of information
        return output;
    }
    finally {
        pm.close();
    }
}

Simple mock

Now, it's time to test! To start slowly, let's deal with the Restlet getResource() method to verify:

Just one and only one instance of PersistenceManager is loaded by the function;
The PersistenceManager instance is cleanly closed at the end of the process;
There's a call issued to get the identified Consumer instance;
There's possibly a call issued to get the identified Location instance;
The output value has the expected information.

In the corresponding unit test series, we don't want to interfere with the App Engine infrastructure (the following chapter will address that aspect). So we'll rely on a mock for the PersistenceManager class that will be injected into the ConsumerRestlet code. The full source of this class is available on my open source project two-tiers-utils: javax.jdo.MockPersistenceManager.

Custom part of the mock for the PersistenceManager class

public class MockPersistenceManager implements PersistenceManager {
    private boolean closed = false; // To keep track of the "closed" state
    public void close() {
        closed = true;
    }
    public boolean isClosed() {
        return closed;
    }

    // ...
}

Here are the unit tests verifying the different flow paths:

When an exception is thrown, because the back-end does not serve the data for example;
When the Consumer instance returns without location coordinates;
When the Consumer instance is fully documented.

Three tests validating the behavior of the ConsumerRestlet.getResource() method

@Test(expected=IllegalArgumentException.class)
public void testUnexpectedError() {
    // Test prepration
    final PersistenceManager pm = new MockPersistenceManager();
    final BaseOperations baseOps = new BaseOperations() {
        boolean askedOnce = false;
        @Override
        PersistenceManager getPersistenceManager() {
            if (askedOnce) {
                fail("Expects only one call");
            }
            askedOnce = true;
            return pm;
        }
    };
    final Long consumerId = 12345L;
    final ConsumerOperations consumerOps = new ConsumerOperations() {
        @Override
        Consumer getConsumer(PersistenceManager pm, Long id) {
            assertEquals(consumerId, id);
            throw new IllegalArgumentException("Done in purpose!");
        }
    };
    ConsumerRestlet restlet = new ConsumerRestlet() {
        @Override BaseOperation getBaseOperations() { return baseOps; }
        @Override ConsumerOperation getConsumerOperations() { return consumerOps; }
    }
    
    // Test itself
    JsonObject response = restlet.getResource(null, consumerId.toString, null);
}

@Test
public void testGettingOneConsumer() {
    // Test prepration
    final PersistenceManager pm = new MockPersistenceManager();
    final BaseOperations baseOps = new BaseOperations() {
        boolean askedOnce = false;
        @Override
        PersistenceManager getPersistenceManager() {
            if (askedOnce) {
                fail("Expects only one call");
            }
            askedOnce = true;
            return pm;
        }
    };
    final Long consumerId = 12345L;
    final ConsumerOperations consumerOps = new ConsumerOperations() {
        @Override
        Consumer getConsumer(PersistenceManager pm, Long id) {
            assertEquals(consumerId, id);
            Consumer consumer = new Consumer();
            consumer.setId(consumerId);
            return consumer;
        }
    };
    final Long locationId = 67890L;
    final LocationOperations locationOps = new LocationOperations() {
        @Override
        Location getLocation(PersistenceManager pm, Long id) {
            fail("Call not expected here!");
            return null;
        }
    };
    ConsumerRestlet restlet = new ConsumerRestlet() {
        @Override BaseOperation getBaseOperations() { return baseOps; }
        @Override ConsumerOperation getConsumerOperations() { return consumerOps; }
        @Override LocationOperation getLocationOperations() { return locationOps; }
    }
    
    // Test itself
    JsonObject response = restlet.getResource(null, consumerId.toString, null);
    
    // Post-test verifications
    assertTrue(pm.isClosed());
    assertNotSame(0, response.size());
    assertTrue(response.containsKey(Consumer.ID);
    assertEquals(consumerId, response.getLong(Consumer.ID));
}

@Test
public void testGettingConsumerWithLocation() {
    // Test prepration
    final PersistenceManager pm = new MockPersistenceManager();
    final BaseOperations baseOps = new BaseOperations() {
        boolean askedOnce = false;
        @Override
        PersistenceManager getPersistenceManager() {
            if (askedOnce) {
                fail("Expects only one call");
            }
            askedOnce = true;
            return pm;
        }
    };
    final Long consumerId = 12345L;
    final Long locationId = 67890L;
    final ConsumerOperations consumerOps = new ConsumerOperations() {
        @Override
        Consumer getConsumer(PersistenceManager pm, Long id) {
            assertEquals(consumerId, id);
            Consumer consumer = new Consumer();
            consumer.setId(consumerId);
            consumer.setLocationId(locationId);
            return consumer;
        }
    };
    final LocationOperations locationOps = new LocationOperations() {
        @Override
        Location getLocation(PersistenceManager pm, Long id) {
            assertEquals(locationId, id);
            Location location = new Location();
            location.setId(locationId);
            return location;
        }
    };
    ConsumerRestlet restlet = new ConsumerRestlet() {
        @Override BaseOperation getBaseOperations() { return baseOps; }
        @Override ConsumerOperation getConsumerOperations() { return consumerOps; }
        @Override LocationOperation getLocationOperations() { return locationOps; }
    }
    
    // Test itself
    JsonObject response = restlet.getResource(null, consumerId.toString, null);
    
    // Post-test verifications
    assertTrue(pm.isClosed());
    assertNotSame(0, response.size());
    assertTrue(response.containsKey(Consumer.ID);
    assertEquals(consumerId, response.getLong(Consumer.ID));
    assertTrue(response.containsKey(Consumer.LOCATION_ID);
    assertEquals(locationId, response.getLong(Consumer.LOCATION_ID));
    assertTrue(response.containsKey(Consumer.LOCATION);
    assertEquals(consumerId, response.getJsonObject(Consumer.LOCATION).getLong(Location.ID));
}

Note that I would have been able to override just the PersistenceManager class to have the Object getObjectById(Object arg0) method returning the expected exception, Consumer, and Location instances. But I would have pass over the strict limit of a unit test by then testing also the behavior of the ConsumerOperations.getConsumer() and LocationOperations.getLocation() methods.

App Engine environment mock

Now, testing the ConsumerOperations class offers a better challenge.

As suggested above, I could override many pieces of the PersistenceManager class to be sure to control the flow. But to do a nice simulation, I almost need to have the complete specification of the Google App Engine infrastructure to be sure I mock it correctly. This is especially crucial when processing Query because Google data store has many limitations [6] that others traditional database, like MySQL, don't have...

Because this documentation is partially available and because Google continues to update its infrastructure, I looked for a way to use the standalone environment made available with the App Engine SDK [1]. This has not been easy because I wanted to have the test running independently from the development server itself. I found first some documentation on Google Code website: Unit Testing With Local Service Implementations, but it was very low level and did not fit with my JDO instrumentation of the DTO classes. Hopefully, I found this article JDO and unit tests from App Engine Fan, a great community contributor I mentioned many times in previous posts!

By cooking information gathered on Google Code website and on App Engine Post, I've produced a com.google.apphosting.api.MockAppEngineEnvironment I can use for my JUnit4 tests.

Three tests validating the behavior of the ConsumerRestlet.getResource() method

package com.google.apphosting.api;
 
// import ...
 
/**
 * Mock for the App Engine Java environment used by the JDO wrapper.
 *
 * These class has been build with information gathered on:
 * - App Engine documentation: http://code.google.com/appengine/docs/java/howto/unittesting.html
 * - App Engine Fan blog: http://blog.appenginefan.com/2009/05/jdo-and-unit-tests.html
 *
 * @author Dom Derrien
 */
public class MockAppEngineEnvironment {
 
    private class ApiProxyEnvironment implements ApiProxy.Environment {
        public String getAppId() {
          return "test";
        }
 
        public String getVersionId() {
          return "1.0";
        }
 
        public String getEmail() {
          throw new UnsupportedOperationException();
        }
 
        public boolean isLoggedIn() {
          throw new UnsupportedOperationException();
        }
 
        public boolean isAdmin() {
          throw new UnsupportedOperationException();
        }
 
        public String getAuthDomain() {
          throw new UnsupportedOperationException();
        }
 
        public String getRequestNamespace() {
          return "";
        }
 
        public Map getAttributes() {
            Map out = new HashMap();

            // Only necessary for tasks that are added when there is no "live" request
            // See: http://groups.google.com/group/google-appengine-java/msg/8f5872b05214...
            out.put("com.google.appengine.server_url_key", "http://localhost:8080");

            return out;
        }
    };
 
    private final ApiProxy.Environment env;
    private PersistenceManagerFactory pmf;
 
    public MockAppEngineEnvironment() {
        env = new ApiProxyEnvironment();
    }
 
    /**
     * Setup the mock environment
     */
    public void setUp() throws Exception {
        // Setup the App Engine services
        ApiProxy.setEnvironmentForCurrentThread(env);
        ApiProxyLocalImpl proxy = new ApiProxyLocalImpl(new File(".")) {};
 
        // Setup the App Engine data store
        proxy.setProperty(LocalDatastoreService.NO_STORAGE_PROPERTY, Boolean.TRUE.toString());
        ApiProxy.setDelegate(proxy);
    }
 
    /**
     * Clean up the mock environment
     */
    public void tearDown() throws Exception {
        // Verify that there's no pending transaction (ie pm.close() has been called)
        Transaction transaction = DatastoreServiceFactory.getDatastoreService().getCurrentTransaction(null);
        boolean transactionPending = transaction != null;
        if (transactionPending) {
            transaction.rollback();
        }
 
        // Clean up the App Engine data store
        ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
        if (proxy != null) {
            LocalDatastoreService datastoreService = (LocalDatastoreService) proxy.getService("datastore_v3");
            datastoreService.clearProfiles();
        }
 
        // Clean up the App Engine services
        ApiProxy.setDelegate(null);
        ApiProxy.clearEnvironmentForCurrentThread();
 
        // Report the issue with the transaction still open
        if (transactionPending) {
            throw new IllegalStateException("Found a transaction nor commited neither rolled-back." +
                    "Probably related to a missing PersistenceManager.close() call.");
        }
    }
 
    /**
     * Creates a PersistenceManagerFactory on the fly, with the exact same information
     * stored in the /WEB-INF/META-INF/jdoconfig.xml file.
     */
    public PersistenceManagerFactory getPersistenceManagerFactory() {
        if (pmf == null) {
            Properties newProperties = new Properties();
            newProperties.put("javax.jdo.PersistenceManagerFactoryClass",
                    "org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory");
            newProperties.put("javax.jdo.option.ConnectionURL", "appengine");
            newProperties.put("javax.jdo.option.NontransactionalRead", "true");
            newProperties.put("javax.jdo.option.NontransactionalWrite", "true");
            newProperties.put("javax.jdo.option.RetainValues", "true");
            newProperties.put("datanucleus.appengine.autoCreateDatastoreTxns", "true");
            newProperties.put("datanucleus.appengine.autoCreateDatastoreTxns", "true");
            pmf = JDOHelper.getPersistenceManagerFactory(newProperties);
        }
        return pmf;
    }
 
    /**
     * Gets an instance of the PersistenceManager class
     */
    public PersistenceManager getPersistenceManager() {
        return getPersistenceManagerFactory().getPersistenceManager();
    }
}

With such a class, the unit test part is easy and I can build complex test cases without worrying about the pertinence of my mock classes! That's really great.

Excerpt of the TestConsumerOperations class

public class TestConsumerOperations {
 
    private MockAppEngineEnvironment mockAppEngineEnvironment;
 
    @Before
    public void setUp() throws Exception {
        mockAppEngineEnvironment = new MockAppEngineEnvironment();
        mockAppEngineEnvironment.setUp();
    }
 
    @After
    public void tearDown() throws Exception {
        mockAppEngineEnvironment.tearDown();
    }
 
    @Test
    public void testCreateVI() throws DataSourceException, UnsupportedEncodingException {
        final String email = "unit@test.net";
        final String name = "Mr Unit Test";
        Consumer newConsumer = new Consumer();
        newConsumer.setDisplayName(name);
        newConsumer.setEmail(email);
        assertNull(newConsumer.getId());
 
        // Verify there's no instance
        Query query = new Query(Consumer.class.getSimpleName());
        assertEquals(0, DatastoreServiceFactory.getDatastoreService().prepare(query).countEntities());
 
        // Create the user once
        ConsumerOperations ops = new ConsumerOperations();
        Consumer createdConsumer = ops.createConsumer(newConsumer);
 
        // Verify there's one instance
        query = new Query(Consumer.class.getSimpleName());
        assertEquals(1, DatastoreServiceFactory.getDatastoreService().prepare(query).countEntities());
 
        assertNotNull(createdConsumer.getId());
        assertEquals(email, createdConsumer.getEmail());
        assertEquals(name, createdConsumer.getName());
    }
    
    // ...
}

Conclusion

As a big fan of TDD, I'm now all set to cover the code of my [still a secret] project efficiently. It does not mean everything is correct, more that everything I thought about is correctly covered. At the time of this writing, just for the server-side logic, the code I produced covers more than 10,000 lines and the unit tests bring an additional set of 23,400 lines.

When it's time to refactor a bit or to add new features (plenty of them are aligned in my task list ;), I feel comfortable because I know I can detect most of regressions (if not all) after 3 minutes of running the test suite.

If you want to follow this example, feel free to get the various mock classes I have added to my two-tiers-utils open-source project. In addition to mock classes for the App Engine environment, you'll find:

Basic mock classes for the servlet (see javax.servlet) and javamocks.io packages -- I had to adopt the root javamocks because the JVM class loader does not accept the creation on the fly of classes in the java root).
A mock class for twitter4j.TwitterUser -- I needed a class with public constructor and a easy way to create a default account.
A series of mock class for David Yu's Project which I use to allow users with OpenID credentials to log in. Read the discussion I had with David on ways to test his code, in fact the code he produced and I customized for my own needs and for other security reasons.

For other details on my library, read my post Internationalization and my two-tiers-utils library.

I hope this helps.

A+, Dom
--
References:

Google App Engine: the homepage and the SDK page.
See my post on Agile: SCRUM is Hype, but XP is More Important... where I mentionned the following techniques: Continuous Integration (CI), Unit testing and code coverage (CQC), and Continuous refactoring.
I know that keeping 100% as the target for code coverage numbers is a bit extreme. I read this article Don't be fooled by the coverage report soon after I started using Cobertura. In addition to reducing the exposition to bugs, the 100% coverage gives a very high chance to detect regressions before pushing the updates to the source control system!
Vincent Massol; JUnit in Action; Editions Manning; www.manning.com/massol and Petar Tahchiev, Felipe Leme, Vincent Massol, and Gary Gregory; JUnit in Action, Second Edition; Editions Massol; www.manning.com/tahchiev. I was used to asking any new developer joigning my team to read at least this chapter 7: Testing in isolation with mock objects.
JDO stands for Java Data Objects and is an attempt abstract the data storage manipulation. The code is instrumented with Java annotations like @Persistent, it is instrumented at compile time, and dynamically connect to the data source thanks few properties files. Look at the App Engine - Using the Datastore with JDO documentation for more information.
For general limitations, check this page Will it play in App Engine. For JDO related limitations, check the bottom of the page Usin JDO with App Engine.

Canadian Wireless Management Forum - my review

2009-10-16T19:58:00.000-04:00

Last week, I attended the Canadian Wireless Management Forum in Montreal. Honestly, I've been a bit disappointed because none of the presenters were as great as the ones I met last year. Over the day, I've still been able to gather bits of information I want to share here ;) Thanks to my company Compuware to have allowed sparing one day there.

iPhone and other smart phones deployed in enterprises

The main focus of the conference is around managing wireless (read mobile phone) communications in enterprises. Some presenters talked about how to control expenses, from sharing guidelines up to using monitoring tools suggesting policies from the statistical analysis of the monthly bills. Testimonies showed that applying a strict control on the mobile phone usage and on the contracts allowed cutting costs from 10 to 30%!

At one point, Nicolas Arsenault made a strange point. Here is what I remind from his talk:

Credits: smoothouse

Credits:Josh Bancroft

Deploying an iPhone application for your employees, when compared to a BlackBerry application, has a much lower cost because the employee already owns the device or he's more likely to buy it. With the employee paying for the phone, probably paying to get new phones regularly (like to move from a iPhone 3G to the latest 3GS), employers can just assume the data plans and can spend more resources on the application development, that at one point can be offered to the company customers. Another benefit is related to the technical support: phone owners stop annoying enterprises' help desk for their phone, they contact the manufacturers or the software vendors directly (they are on their own)...

If this approach have evident economical benefits, I disagree with it because:

In general, letting employees owning their work mobile phones cause problems when they leave the company. Most of employees have a non concurrence clause in their contract, usually valid 6 months after the employment contract termination. During that period, they cannot compete with their previous employer. Imagine a salesperson, an account manager, or a consultant who owns his phone number, who is referenced in the phone directories all his previous contacts. When these contacts want to deal with the company, who do you think they'll call: the mobile phone or the company front line? Now that these employees are out, they don't have to submit their phone bills anymore (as they did with their expenses reports). No one can detect the issues anymore… And this is without mentioning that the employee replacement or his colleagues have no way to get the mobile phone contact list to continue the business as usual!

In the company I worked, none let me use my own computer on their network! For the IT departments, this practice would raise too much security concerns. I even know cases that just installing a VPN software on your own machine install silently a bunch of monitoring tools that can mess up your systems (thanks to VirtualBox, it easy to limit these nasty side effects ;). In the old days, when phones were just stupid, just able to handle voice call and exchange text messages (SMS), the risk of phone infection by viruses were pretty negligible. The widely used phone operating system is Symbian, used on Nokia and Sony Ericsson phones, and J2ME is a common application framework, even on phones running Windows Mobile. If the identified viruses are not a lot, and if they are not some damageable (send on your behalf SMS to expensive services, for example), the newest smart platforms provide much more threats because the corresponding phones can host tons of applications. In such an environment, how can a company force software upgrades on systems it does not own? How can it force an employee to upgrade to a new hardware because the current one is compromised?

My last point is more ethical: how far companies should go with putting the burden on their employees? An iPhone costs around 800$. It's often offered around 200$ with a 3-year contract, which costs basically around 60$/month with a simple data plan. If the telecom operators (telco) subsidize so aggressively such a phone, they surely expect higher average revenue per user (ARPU). In addition to be linked to the telco for a long period of time (compare 3 years with the 3 to 6 months between technological evolution: 6 to 12 times longer!), employees have to support costs the company should assume. Usually, companies try to provide a comfortable work environment to get most of their employees, and that's fair to me: if the company gives more than the salary, employees are more likely to deliver better work. With the incitation of employees assuming the cost of the new mobile phones, I see a regression: companies give less while expecting more (reachable—possibly traceable—outside the office hours, for example). IMO, it's yet another example of a technological progress that might worsen fragile people condition...

Mobile payment and Near-field-communication (NFC)

Shortly on this topic, I want to mention presentation made by Daniel Martin for Atlas Telecom Mobile, David Robinson for Rogers Wireless, and Prakash Hariramani for Visa. They talked about an experiment conducted downtown Toronto where they were able, thanks to Motorola phones equipped with a NFC emitter and a good number of retailers there, to allow mobile payment over the mobile phone network. Visa's solution, called PayWave™ (MasterCard's one is called PayPass™), was inserted into the Motorola phone extension and allow consumers to pay for their purchase quite easily.

During the discussion, Mr. Robinson talked about Rogers Wireless approach being strictly based on standards. He mentioned the recent u-turn of telco who continued to invest in closed and proprietary solutions (like CDMA) and now move to standardized ones (like HSPA which is an upgrade of GSM, on the road to LTE—see example of Bell and Telus in Canada). Rogers Wireless' approach is then to work with the rest of the major industry players (Orange, Vodafone, etc.) to define a solution for anyone anywhere. Mr. Robinson talked about the possibility to define an extended SIM card (for Subscriber Identity Module; the card contains a micro-SIMCARD processor which manage very securely information). This new SIM card will have NFC capabilities and will be able to interact with contactless payment terminals. It's possible that these SIM cards will contain additional information like driver license identifier that police officers will be able to read, insurance number for the government agencies, etc. The phone will provide the interface to enable the data access, and smart phones with touch screens open the door to various and robust verification techniques.

Because more and more people are more attached to their phone than to their wallet, this approach will possibly be more convenient, more secure, and smaller (no more cash, business cards, credits cards, etc. ;) Who said that implanting the SIM under anyone's skin, a SIM that can unlock your phones, cars, houses, computers, etc, is just science fiction?

Telco business model in danger!

iBwave offers solutions improving in-building wireless coverage. Knowing the fact that 60%-80% of mobile usages are indoor, that telco have difficulties to boost the power or to multiply outdoor antennas near high density areas, these places stay mostly uncovered. Mr. Bouchard illustrated his point with the simulation of the poor performance of the traditional networks on the McGill campus—really amazing! In conclusion, iBwave sits in a very nice and promising niche ;)

The last point of interest to me came from Mario Bouchard, from iBwave.

To introduce his company activities, Mr. Bouchard shows two diagrams which made me think for a while. Let me try to reproduce them.

Mr. Bouchard states that 15 years ago, the innovation came from the network manufacturers: they invented the technology, others created cell phones to connect to the new network, some operators offered (very expensive) plans, and consumers (locked with long term contracts) tried to communicate.

Credits: DigitalAlan

Today, the order has been scrambled:

Thanks to the rapid technology evolution, designers of mobile devices can embed many types of sensors into communicant machines, and with the increasing miniaturization, such machines are pervasiveness!
Because of the reduced delay between big technology evolutions (think about the iPhone which is just 2 years old), consumers choose their devices carefully, and bargain more to get the best quality-price ratio.
Network manufacturers now to their best to provide networks than can deliver at the rhythm the devices can consume. When the European community created the Groupe Spécial Mobile (initial meaning of the GSM acronym, known now for Global System for Mobile communications) in 1982, we had to wait up to 1991 to experiment the first GSM network. GSM is also known the 2G technology. The EDGE (Enhanced Data rates for GSM Evolution, or 2.5G) has been introduced in 1999, the first 3G technology (HSPA/UMTS, and EV-DO/CDMA) has been delivered late 2001, and the coming 4G ones (LTE for Long Term Evolution or WiMax) are on the bench.
At the end of the line, now there are the telcos:

Investments are phenomenal
The competition is rude (not rude enough in Canada, IMO)
Customers are volatile and they always want the latest phone
They have to subsidize the phones to lower the barriers to entry
They have to provide the best coverage everywhere
Customers are quick to leverage social tools to complain about them
Customers don't respect the old rules (read: jailbreak their phone)
Customers are not the cash cows they used to be...

I don't think the telco future looks very nice. As traditional telecommunication service providers, they are more and more just Internet providers. Personally, I'm fine with communication on Internet (VOIP/SIP), with the possibility to stream on Internet (Qik.com, Layar.com), to receive instant messages (IM) instead of text messages (SMS). And look, when I've a chance to connect my phone to a wifi network, I'm happy to get a better connectivity while saving few bucks.

Interesting developments to follow, aren't they?

A+, Dom

Internationalization and my two-tiers-utils library

2009-10-05T21:31:00.006-04:00

This is a follow-up article of Internationalization of GAE applications, which itself is part of the series Web Application on Resources in the Cloud.

In my initial article, I explained some hurdles of globalizing applications, especially the ones being implemented with many programming languages. In this article, I'm going to describe few use-cases and how my open-source library two-tiers-utils can ease the implementation. Here are the covered topics:

Get the user's preferred locale
Display messages in different locales
Handle localized messages with different programming languages
Generate the localized bundles per programming language
Bonus

Get the user's preferred locale

For this use-case, let's only consider the Java programming language. Another assumption is the availability of the localized resources in the corresponding Java format (i.e. accessible via a PropertyResourceBundle instance).

In a Web application, the user's preferred locale can be retrieved from:

The HTTP headers:

locale = ((HttpServletRequest) request).getLocale();

The HTTP session (if saved there previously):

HttpSession session = ((HttpServletRequest) request).getSession(false);
if (session != null) {
  locale = new Locale((String) session.getAttribute(SESSION_USER_LOCALE_ID));
}

The record containing the user's information:

locale = ((UserDTO) user).getPreferredLocale();

To ease this information retrieval, the two-tiers-utils library provides the domderrien.i18n.LocaleController class

Excerpt of public methods offered within domderrien.i18n.LocaleController

This class can be used in two situations:

In a login form, for example, when we can just guess the desired locale from the browser preferred language list or from an argument in the URL.
In pages accessible to identified users thanks to the HTTP session.

Usage example of the domderrien.i18n.LocaleController.detectLocale()

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<%@page
    language="java"
    contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"
    import="org.domderrien.i18n.LabelExtractor"
    import="org.domderrien.i18n.LocaleController"
%><%
    // Locale detection
    Locale locale = LocaleController.detectLocale(request);
%><html>
<head>
    <title><%= LabelExtractor.get("dd2tu_applicationName", locale) %></title>
    ...
</head>
<body>
    ...
    <img
        class="anchorLogo"
        src="images/iconHelp.png"
        width="16"
        height="16"
        title="<%= LabelExtractor.get("dd2tu_topCommandBox_helpIconLabel", locale) %>"
    />
    ...
</body>
</html>

The same message in different locales

The previous example introduces also a second class: domderrien.i18n.LabelExtractor. Being given an identifier, an optional array of Object references, and a locale, the get static method loads the corresponding string from the localized resource bundle.

Excerpt of public methods offered within domderrien.i18n.LabelExtractor

A series of localized entries like en:“Welcome {0} {1}”, fr:“Bonjour {2} {1}”, and ja:“お早う {0}{1}” can be easily invoked with a simple command like: LabelExtractor.get("welcome_message", new Object[] { user.getFirstName(), user.getLastName() }, user.getLocale());.

The same message used from different programming languages

Java is a pretty neat language with a large set of editors and code inspectors. But Java is not the only languages used for Web applications. If the two-tiers-utils library provides nice Java features, the delivery of the same library interfaces for the programming languages JavaScript and Python libraries makes it way more valuable!

Code of the domderrien.i18n.LabelExtractor.get() method for the JavaScript language.

(function() { // To limit the scope of the private variables

    /**
     * @author dom.derrien
     * @maintainer dom.derrien
     */
    var module = dojo.provide("domderrien.i18n.LabelExtractor");

    var _dictionnary = null;

    module.init = function(/*String*/ namespace, /*String*/ filename, /*String*/ locale) {
        // Dojo uses dash-separated (e.g en-US not en_US) and uses lower case names (e.g en-us not en_US)
        locale = (locale || dojo.locale).replace('_','-').toLowerCase();

        // Load the bundle
        try {
            // Notes:
            // - Cannot use the notation "dojo.requirelocalization" because dojo parser
            //   will try to load the bundle when this file is interpreted, instead of
            //   waiting for a call with meaningful "namespace" and "filename" values
            dojo["requireLocalization"](namespace, filename, locale); // Blocking call getting the file per XHR or <iframe/>

            _dictionary = dojo.i18n.getLocalization(namespace, filename, locale);
        }
        catch(ex) {
            alert("Deployment issue:" +
                    "\nCannot get localized bundle " + namespace + "." + filename + " for the locale " + locale +
                    "\nMessage: " + ex
                );
        }

        return module;
    };

    module.get = function(/*String*/key, /*Array*/args) {
        if (_dictionary == null) {
            return key;
        }
        var message = _dictionary[key] || key;
        if (args != null) {
            dojo.string.substituteParams(message, args);
        }
        return message;
    };

})(); // End of the function limiting the scope of the private variables

The following piece of code illustrates how the JavaScript domderrien.i18n.LabelExtractor class instance should be initialized (the value of the locale variable can come from dojo.locale or a value injected server-side into a JSP page) and how it can be invoked to get a localized label.

Usage example of the domderrien.i18n.LocaleController.get()

(function() { // To limit the scope of the private variables

    var module = dojo.provide("domderrien.blog.Test");

    dojo.require("domderrien.i18n.LabelExtractor");

    var _labelExtractor;

    module.init = function(/*String*/ locale) {
        // Get the localized resource bundle
        _labelExtractor = domderrien.i18n.LabelExtractor.init(
                "domderrien.blog",
                "TestBundle",
                locale // The library is going to fallback on dojo.locale if this parameter is null
            );

        ...
    };

    module._postData = function(/*String*/ url, /*Object*/ jsonParams) {
        var transaction = dojo.xhrPost({
            content : jsonParams,
            handleAs : "json",
            load : function(/*object*/ response, /*Object*/ioargs) {
                if (response == null) {
                    // Message prepared client-side
                    _reportError(_labelExtractor.get("dd2tu_xhr_unexpectedError"), [ioargs.xhr.status]);
                }
                if (!response.success) {
                    // Message prepared server-side
                    _reportError(_labelExtractor.get(response.messageKey), response.msgParams);
                }
                ...
            },
            error : function(/*Error*/ error, /*Object*/ ioargs) {
                    // Message prepared client-side
                _reportError(error.message, [ioargs.xhr.status]);
            },
            url : url
        });
    };

    var _reportError = function(/*String*/ message, /*Number ?*/xhrStatus) {
        var console = dijit.byId("errorConsole");
        ...
    };

    ...

})(); // End of the function limiting the scope of the private variables

The following series of code excerpts show the pieces involved in getting the localized resources with the Python programming language.

LabelExtractor methods definitions from domderrien/i18n/LabelExtractor.py

# -*- coding: utf-8 -*-

import en
import fr
 
def init(locale):
    """Initialize the global dictionary for the specified locale"""
    global dict
    if locale == "fr":
        dict = fr._getDictionary()
    else: # "en" is the default language
        dict = en._getDictionary()
    return dict

Sample of a localized dictionary from domderrien/i18n/en.py

# -*- coding: utf-8 -*-
 
dict_en = {}
 
def _getDictionary():
    global dict_en
    if (len(dict_en) == 0):
        _fetchDictionary(dict_en)
    return dict_en
 
def _fetchDictionary(dict):
    dict["_language"] = "English"
    dict["dd2tu_applicationName"] = "Test Application"
    dict["dd2tu_welcomeMsg"] = "Welcome {0}."
    ...

Definitions of filters used by the Django templates, from domderrien/i18n/filters.py

from google.appengine.ext import webapp
 
def get(dict, key):
    return dict[key]
 
def replace0(pattern, value0):
    return pattern.replace("{0}", str(value0))
 
def replace1(pattern, value1):
    return pattern.replace("{1}", str(value1))
 
...
 
# http://javawonders.blogspot.com/2009/01/google-app-engine-templates-and-custom.html
# http://daily.profeth.de/2008/04/using-custom-django-template-helpers.html
 
register = webapp.template.create_template_register()
register.filter(get)
register.filter(replace0)
register.filter(replace1)
...

Django template from domderrien/blog/Test.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>{{ dictionary|get:dd2tu_applicationName }}</title>
    ....
</head>
<body>
    ...
    <div class="...">{{ dictionary|get:dd2tu_welcomeMsg|replace0:parameters.loggedUser }}</div>
    ...
</body>
</html>

Test handler from domderrien/blog/Test.py

from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template

def prepareDictionary(Request):
    locale = request.get('lang', 'en')
    return LabelExtractor.init(locale)

class MainPage(webapp.RequestHandler):
    def get(self):
        parameters = {}
        parameters ['dictionary'] = domderrien.i18n.LabelExtractor.init(self.request)
        parameters ['loggedUser'] = users.get_current_user()
        path = os.path.join(os.path.dirname(__file__), 'domderrien/blog/Test.html')
        self.response.out.write(template.render(path, parameters))

application = webapp.WSGIApplication(
    [('/', MainPage)],
    debug=True
)
 
def main():
    webapp.template.register_template_library('domderrien.i18n.filters')
    run_wsgi_app(application)
 
if __name__ == "__main__":
    main()

Generate the localized bundles per programming language

In my previous post Internationalization of GAE applications, I suggest to use a dictionary format that would be programming lnaguage agnostic while being known by translator: TMX, for Tanslation Memory eXchange.

Snippet of a translation unit definition for a TMX formatted file

<tu tuid="dd2tu_welcomeMessage" datatype="Text">
 <tuv xml:lang="en">
  <seg>Welcome {0}</seg>
 </tuv>
 <note>{0} is going to be replaced by the logged user's display name</note>
 <prop type="x-tier">dojotk</prop>
 <prop type="x-tier">javarb</prop>
 <prop type="x-tier">python</prop>
</tu>

The two-tiers-utils library provides a Java runtime domderrien.build.TMXConverter that generates the resource bundles for Java/JavaScript/Python. If a simple series of XSL-Transform runs can do the job, the TMXConverter does a bit more by:

Comparing the modification dates of the generated files with the TMX one to generate them only if needed
Check the uniqueness of the label keys
Generate the list of supported languages

Invoking the TMXConverter runtime from an ant build file is very simple, while a bit verbose:

Ant target definition invoking the TMXConverter

<target name="step-tmx-convert">
    <mkdir dir="${temp.dir}/resources" />
    <mkdir dir="src/WebContent/js/domderrien/i18n/nls" />
    <java classname="domderrien.build.TMXConverter" fork="true" failonerror="true">
        <classpath refid="tmxconverter.classpath" />
        <classpath location="${temp.dir}/resources" />
        <jvmarg value="-Dfile.encoding=UTF-8" />
        <arg value="-tmxFilenameBase" />
        <arg value="${dd2tu.localizedLabelBaseFilename}" />
        <arg value="-sourcePath" />
        <arg value="${basedir}\src\resources" />
        <arg value="-jsDestPath" />
        <arg value="${basedir}\src\WebContent\js\domderrien\i18n\nls" />
        <arg value="-javaDestPath" />
        <arg value="${temp.dir}/resources" />
        <arg value="-languageFilenameBase" />
        <arg value="${dd2tu.languageListFilename}" />
        <arg value="-buildStamp" />
        <arg value="${dd2tu.stageId}" />
    </java>
    <native2ascii
        src="${temp.dir}/resources"
        dest="${temp.dir}/resources"
        encoding="UTF8"
        includes="*.properties-utf8"
        ext=".properties"
    />
    <copy
        file="${temp.dir}/resources/${dd2tu.localizedLabelBaseFilename}.properties"
        tofile="${temp.dir}/resources/${dd2tu.localizedLabelBaseFilename}_en.properties"
    />
    <mkdir dir="src/WebContent/js/domderrien/i18n/nls/en" />
    <copy
        file="src/WebContent/js/domderrien/i18n/nls/{dd2tu.localizedLabelBaseFilename}.js"
        todir="src/WebContent/js/domderrien/i18n/nls/en"
    />
</target>

With the TMX file as the source of thruth for the label definitions, it is just a matter of altering the value a <prop/> tag and running the build once again to move one label definition from one programming language to another. No more error prone copy-and-paste of text between different file formats!

Excerpt of the generated Java resource bundle

bundle_language=English
unit_test_sample=N/A
dd2tu_applicationName="Test Application"
dd2tu_welcomeMessage=Welcome {0}
...
x_timeStamp=20091001.1001

Excerpt of the generated JavaScript resource bundle

({bundle_language:"English",
unit_test_sample:"N/A",
dd2tu_applicationName:"Test Application",
dd2tu_welcomeMessage:"Welcome ${0}",
...
x_timeStamp:"20091001.1001"})

Excerpt of the generated Python class definition

# -*- coding: utf-8 -*-
 
dict_en = {}
 
def _getDictionary():
    global dict_en
    if (len(dict_en) == 0):
        _fetchDictionary(dict_en)
    return dict_en
 
def _fetchDictionary(dict):
    dict["_language"] = "English"
    dict["dd2tu_applicationName"] = "Test Application"
    dict["dd2tu_welcomeMsg"] = "Welcome {0}."
    ...
    dict["x_timestamp"] = "20091001.1001"

Bonus

The TMXConverter being part of the build process and going over all localized TMX files, it generates the list of supported languages.

JSP code fetching a HTML &ltselect/> box with the list of supported languages

<span class="topCommand topCommandLabel"><%= LabelExtractor.get("rwa_loginLanguageSelectBoxLabel", locale) %></span>
<select
    class="topCommand"
    dojotType="dijit.form.FilteringSelect"
    id="languageSelector"
    onchange="switchLanguage();"
    title="<%= LabelExtractor.get("rwa_loginLanguageSelectBoxLabel", locale) %>"
><%
    ResourceBundle languageList = LocaleController.getLanguageListRB();
    Enumeration<String> keys = languageList.getKeys();
    while(keys.hasMoreElements()) {
        String key = keys.nextElement();%>
        <option<% if (key.equals(localeId)) { %> selected<% } %> value="<%= key %>"><%= languageList.getString(key) %></option><%
    } %>
</select>

The following figures illustrates the corresponding code in action.

Part of a login screen as defined with the default (English) TMX file.

Part of a login screen as defined with the French TMX file.

Conclusion

The two-tiers-utils library is offered with the BSD-like license. Anyone is free to use it for his own purposes. but I'll appreciate any feedback, contribution, and feature requirement.

See you on github.com ;)
“May the fork be with you.”

A+, Dom

Progress update

2009-09-18T23:39:00.001-04:00

Almost three months without publishing anything! I am definitively not proud of this score...

I have been busy on three fronts:

At work, I continue working on mobile related development. The easiest platform to play with is the Android one (see my post on Android Dev Phone 1) and I thrilled to see new handsets being made available, like the HTC Dream and Motorola Cliq, with their respective HTC Sense and Moto BLUR custom UIs (see videos below). Even if its SDK is not as rich as Android's one, even if the gadgets are not as polished as Android's ones, I also like developing for BlackBerry phones, like the BlackBerry Storm.

For my side project, running on Google App Engine infrastructure, it is progressing very well. Today, I reached a milestone: the first part of [still a secret] runs live. In terms of coding, it represents ~6,000 lines for the source files and ~12,000 lines for the test. I have always considered source vs test as being 50-50; it seems I should re-evaluate the balance to 1/3-2/3 ;)

Thanks to different contributors, I have developed a series of mock classes allowing to test transactions with BigTable, the database used by App Engine. A really neat piece of code I am going to describe here later.

On the social side, I am working with the board of Diku Dilenga Canada (board I am member of) to move as its Executive Director (still as a volunteer). The move has been inspired by Jean-Pierre Tchang, founder of IRIS Mundial. I hope this update will make Diku Dilenga (Canada) as successful as IRIS Mundial.

I had a lot of activities on this front recently: a fundraiser thanks to Louis Lamontagne walking between Saint Jean Pied de Port (France) and Santiago de Compostela (Spain), a trip of 780km, a first series of computers to be shipped to Kananga, Democratic Republic of the Congo, etc.

The plan to link the [still a secret] project with Diku Dilenga activities have been formally approved by the board during the summer. This side, there is a possibility I will do a presentation to the 2010 Africa/Middle East Regional Microcredit Summit (AMERMS) to be held in Nairobi, Kenya April 7-10, 2010. It would be nice to participate, isn't it?

Stay tuned, I should be back in few days with technical information about unit testing transactional code on Google App Engine ;)

A+, Dom
--
Videos:

HTC Hero, the first phone with the HTC Sense, a customized UI scheme on the top of Android.

Motorola Cliq, the first phone with Moto BLUR, a customized UI scheme on the top of Android.

BlackBerry Storm, first BlackBerry phone with a touch screen

JavaOne Conference

2009-06-12T14:55:00.003-04:00

Duke and myself ;)

It has been a long week in San Francisco while I was attending the JavaOne conference. Sessions started at 8:30 AM and finished after 9:30 PM!

Among all reviews that have been published, read this ones: Community day on Monday, Conference day 1 on Tuesday, Day 2 on Wednesday, Day 3 on Thursday, and Day 4 on Friday. Sun's website contains also a complete list of conference articles. Pictures of the event are available on Sun's Photo Center website.

I went there with two objectives:

See JavaFX in action;
Look at all efforts around JavaME and the MSA initiative.

JavaFX technology stack

Eric Klein, Vice President, Java Marketing, Sun Microsystems

I came without preconceived idea around JavaFX, just with my background as a JavaScript/Java developer and my knowledge of Flex and AIR.

The first deception came from looking at the scripting language: Man! they invented yet another language :( For sure, the JavaFX scripting language is nicely handled by NetBeans 6.5+ (except the code formatting) but new paradigms and new conventions are a big barrier to adoption to me. Can you figure out what the following code is doing?

public class Button extends CustomNode {
  public var up: Node;
  content: Node = up;
  public override function create(): Node {
    return Group {
      content: bind content
    }
}

If the visual effects to animate texts, pictures, and video are really great, the native support of a sound library is really missing! For example, I would expect to be able to synchronize KeyFrame objects with sound tracks but the KeyFrame.time attribute has to be set manually—not very flexible when it's time to change the sound track...

The pros are:

A coming visual editor to prepare clips;
A large library of image and video effects;
The multi-platform support, especially for mobile devices.

Platform dependent packaging is nicely handled: NetBeans project properties pane provides choices among {Standard Execution, Web Start Execution, Run in Browser, Run in Mobile Emulator}. As a Web developer, I am just sorry to see that the application window size cannot be expressed in percentage, that the auto-resizing is handled transparently, which is not better than usual Flex applications.

Last point: I have not seen how to invoke JavaFX handlers from JavaScript ones, and vice-versa, when the application is deployed to run in browsers. If you have a source for these information, please, drop the link in a comment.

JavaME technology stack

Christopher David, Rikko Sakaguchi and Patrik Olsson
during Sony Ericsson General Session

This was definitively the most interesting domain to me. During one session, it has been announced that 60% of shipped mobile devices in 2008 are Java-enabled. In 2009, the market share should grow up to 70%. In relation with the iPhone, Scott McNeally did this joke: the possible future Sun owner Larry Ellison might succeed to open the iPhone platform to Java because he is a well known friend of Steve Jobs.

Compared to the Java Standard Edition (J2SE) and the Java Enterprise Edition (J2EE), the Java Mobile Edition (J2ME) has more external contributors. Under the Mobile Service Architecture (MSA) initiative, a lot of mobile device manufacturers and telecommunication operators participate to the Java Community Process (JCP) to deliver Java Specification Requests (JSRs). Note that MSA itself is defined as a JSR: JSR 248. As of today, most of the recent phones are MSA 1.1 compliant (this is a mandatory requirement for the telco Orange, for example). Nokia and Sony Ericsson have shipped a lot of MSA-compliant handsets, LG, Samsung, and Motorola shipped very few ones. The standard MSA 2 (JSR 249) is being finalized and it contains the promising JSRs:

Additional APIs I imagine many developers are looking forward: JSR 257 Contactless communication API and JSR 229 Payment API.

MSA evolution and its JSR set
(from the MSA specification documentation)
(click to enlarge)

All major manufacturers have opened or are opening "App Stores" a-la Apple. They open also their development platforms. More companies will be able to adapt their software offering to mobile devices. Even Sony allows anyone to write application to run in Blu-ray Disc players. The main difficulty on developer-side is the fragmentation: there is no standard API allowing to discover the features supported by a device! Developers have to rely on each manufacgturer's feature list and on exception handling :(

The Blackberry platform is pretty well controlled and should be easy to develop on. Then follows Sony Ericsson which provides consistent phone classes (i.e. what works for one phone in the JP 8.4 class work for all phones in that class). The delivery of the Sun JavaME SDK 3.0 containing many third-party emulators (even one for Windows Mobile devices) added to better on-device deployment and on-device debugging capabilities, should motivate more and more developers.

I have not enough experience with Android (just got one Android dev phone two weeks ago) to compare it to the JavaME technology stack. I don't know neither about Symbian (Nokia devices) or LiMo (Motorola devices) platforms.

Exhibition hall

Besides the visit of mobile device manufacturers (RIM and Ericsson) booths, I visited:

Sun's project Fuji (open ESB) with a Web console using <canvas/> from HTML5, like Yahoo! Pipes.
Convergence, the Web client for Sun's communication suite, built on the top of Dojo toolkit ;)
INRIA (French national R&D lab) for its static code analysis Nit.
Isomorphic Software for its SmartClient Ajax RIA System.
eXo Platform (on OW2 booth) for its eXo Portal offering.
Liferay, Inc. for its eponymous portal.

Other discoveries

James Gosling and myself ;)

I attended very good presentations, like the opening keynote which was fun. Among the good presenters, I can mention (ordered alphabetically):

Ben Galbraith and Dion Almaer (see also Ajaxian.com), Mozilla.
BJ Hargrave, IBM, and Peter Kriens, aQute, about OSGi.
Doug Tidwell, IBM, who gave a funny speech about REST-WS* in the cloud.
Erik Hellman, Sony Ericsson.
James Gosling, Sun Microsystems.
Patrick Chanezon, Google, about Open Web.

If you have a Sun Developer Network (SDN) account (by the way, it's free), you can view the slides of the technical sessions at: http://developers.sun.com/learning/javaoneonline/.

Special mention

I want also to mention the call to developers by MifOS people who have been awarded by James Gosling during the Friday morning general session. This organization develops open source software for microfinance institutions (MFIs) to help them managing loans and borrowers (see demo). Really nice initiative started by the Grameem Bank!

Excerpt from James Gosling Toy Show report:

Microfinancing Through Java EE Technology

Gosling next introduced a group whose great innovation with Java technology was social and not technical. Sam Birney, engineering manager and Mifos alumnus, and Van Mittal-Hankle, senior software engineer at the Grameen Foundation, took the stage to receive Duke's Choice awards for their work using Java EE to serve 60,000 clients worldwide in microfinancing, a highly successful means of helping poor people get small loans and start businesses.

Mifos is open-source technology for microfinance that is spearheaded by the Grameen Foundation.

"Sometimes excellence comes not from technical innovation but in how technology is used," explained Gosling. "This is an example of using Java technology to really improve people's lives."

With an estimated 1.6 billion people left in the world who could benefit from microfinance, the men put out a call for volunteers to contribute to the Mifos project..

What's next?

What are my resolutions? Get a Mac as the development platform (Eclipse works on MacOS and I can use a Win7 image within VirtualBox), and start development on Java enabled phone (at least MSA 1.1 compliant).

A+, Dom

Internationalization of GAE applications

2009-06-10T15:08:00.000-04:00

Costs of badly planned internationalization

In my experience, internationalizing an application is very expensive if it is not planned upfront.

The first source of costs is due to developers who are used to lazily hard-coding labels. Extracting them at the end of the development process is always error prone because developers may have done assumptions on exact labels. Good regression test suites can help detecting such situations but they cannot avoid the cost of the required fix and its corresponding test runs. With late label extraction, developers without enough context tend to add extra dictionary entries. With the label extraction near the release milestone, developers in a rush and without the initial context tend also to produce non documented dictionaries—that is limit their re-usability and increase the difficulty of possible defect fixings.

Non localized Java code

protected String formulatePageIndex(int index, int total) {
    String out = "Page " + index;
    if(0 < total) {
        out += " of " + total;
    }
    return out;
}

In the example above, I have seen the extraction of the labels Page and of instead of Page %0 and Page %0 of %1. The quick extraction leads to the impossibility to invert the two arguments! Think about the people naming conventions: in many countries, the last name is displayed before the first name, in others the first name precedes the last name (%last %first compared to %first %last), and in Japan both names are printed without separator in between (%last%first).

Localized Java code

protected String formulatePageIndex(int index, int total) {
    // Get the resource bundle already initialized for the correct locale
    ResourceBundle rb = getCurrentResourceBundle();
    // Prepare values
    Object[] values = Object[] {Integer.valueOf(index), Integer.valueOf(total)};
    // Get the right localized label
    String label = rb.getString("PageIndexLabel_Page");
    if(0 < total) {
        label = rb.getString("PageIndexLabel_PageOf");
    }
    // Return the localized label with injected values
    return Message.format(label, values);
}

The second source of costs is due to the missed opportunities of deploying localized builds early in the development process. In Agile environments, we expect to get runnable builds on regular basis (at the end of each sprints, each 4 to 6 weeks, for example). And these fool-proof builds can be demoed to customers to get early feedback. If the development organization can work with translators iteratively, there are a lot of chance to detect localization defects while their fixing cost is not too high. In the past, I have seen product developments hugely hit when bi-directional languages (like Arabic and Hebrew) had been introduced...

Different aspects of the internationalization

Internationalization (i18n) [1] has two aspects:

The translation of the labels;
The localization (l10n) of these labels.

The localization takes into account the language and the country, sometimes with variants in a country. For example, the Spanish language spoken 19 identified countries. In Mexico (ES_MX), the language is slightly different from the one generally spoken in Spain (ES or ES_ES). In Spain, there are many regional languages like the Catalan (CA_ES). The different locales are normalized by the Unicode consortium (ISO-639 and ISO-3166). Codes are composed of a sequence of two letters for the language plus two letters for the country plus two letters for the region. If letters are missing after the language, most of programming languages fallback on common defaults.

In order to ease application localizations, Unicode references a Common Locale Data Repository (CLDR) [2]. This repository is used and updated by many companies like IBM, Sun, Microsoft, Oracle, etc. The repository describes rules on how to:

Localize currencies;
Localize metrics (distance, speed, temperature, etc.);
Localize dates and calendars.

As of today, I think only timezone definitions are still not centrally managed... This is especially bad because conversions between Universal Time (UTC) dates and local dates are operating system dependent (Sun Solaris have small differences with Microsoft Windows, for example). Many tools use the Unicode CDLR information. For example, each release of the Dojo toolkit use its information to provide the Calendar widget for 27 locales [3].

Internationalization with different programming languages

Almost all programming languages have ways to facilitate application globalization. Java provide resource bundles (*.properties file), Microsoft .Net has resource files (.rc files), Python has dictionaries, etc. If JavaScript lacks of native support for globalization, some libraries offer various support. To my knowledge, Dojo toolkit is the first providing a full support.
If developing an application on Google App Engine infrastructure can be done with only one programming language, Python or Java as this time of writing, it is highly possible that developers will use some JavaScript libraries to speed up their development. This is without counting the delivery of a similar program front-end as a native application (made with Adobe AIR, Microsoft .Net, C/C++, Groovy, etc.).

In different situations, I have seen developers moving manually label definitions from one environment to another one. Sometimes, definitions were left over, cluttering the system. In Agile environments, developers should focus on the requirements for the current sprint, leaving some tuning for later sprints. For example, at one point during the development, some labels defined in a JavaScript bundle might be moved to a Java bundle because the localization will be done server-side into a JSP file.

My solution is to put all labels in one localized central repository. The dispatch among the different programming languages is done at build time. When I looked for this repository format, my solution was selected against the following criteria:

Easily editable;
Has a standard format;
Usable by static validation processes;
Has excellent re-usability factors;
Easily extensible to new programming languages.

I chose the TMX format (TMX for Translation Memory eXchange [4]). This is an XML based format (good for edition, extensibility, and use by static validation tools) which has been defined to allow translation memory export/import between different translation tools like DejaVu. The XlDiff format would have been another good candidate.

The following table illustrates the flow of interactions between the different actors in a development team. This sequence diagram shows that, once the developers have delivered a first TMX file, testers and translators can work independently to push tested and localized builds to the customer. As explained later, if developers tune the TMX entries without updating the labels themselves, translators and testers (at least from the l10n point-of-view) can stay out of the loop—only steps [1, 7, 8] are replayed.

Simplified view of the overall interaction flow
Developers	Testers	Translators	Build process	End-users
1. Write labels in one language into the TMX file. These labels are extracted from design documents.
			2. Generate the application with for one locale.
	3. Produce a generic bundle to identify non extracted labels.
			4. Generate the application with for two locales.
				5. Use the application in one locale (switching to the `test` language is hidden).
		6. Use the initial TMX to produce `n` localized TMXs.
			7. Generate the application with for `2 + n` locales.
				8. Can use the application in `1 + n` locales.

The following code snippet shows how an entry into the base TMX file is defined.

Snippet of a translation unit definition for a TMX formatted file

<tu tuid="entry identifier" datatype="Text">
 <tuv xml:lang="locale identifier">
  <seg>localized content</seg>
 </tuv>
 <note>contextual information on the entry and relations with other entries</note>
 <prop type="x-tier">dojotk</prop>
 <prop type="x-tier">javarb</prop>
</tu>

The key features of the TMX format are:

The format can be validated with an external XSD (XML Schema Description);
One entry (tu: translation unit) can contain many localized contents (tuv: translation unit value);
Developers have a normalized placeholder (<note/>) to register contextual information;
Extensions are used by the build process to target the type of resource bundle to receive the localized label.

With such an approach, I have seen a drastic reduction of translation mistakes, especially thanks to the <note.> tag. Sometimes, graphical elements contain inter-related labels that cannot be grouped under a generic entity. The following set of elements illustrates the situation. The TMX approach saves translators headaches because they are simply informed about the relation between four entities.

Search:
The conversion from the TMX to the various resource bundles is done by an XSL-Transform. With the continuous integration handled by ant, the corresponding task generates the output after having appended the XSLT file coordinates to a copy of the TMX file and after asked for the transformation with the corresponding <xsl/> ant task. Depending on the machine performance, depending on the TMX file size, I found that the process can be time consuming. If this is your case too, I suggest you write your own little Java program to handle it. You can also use mine ;)

Stylesheet transforming label definitions for the Dojo toolkit

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
 <xsl:output method="text" />
 <xsl:template match="/tmx/body">
  {
  <xsl:for-each select="tu">
    <xsl:for-each select="prop">
      <xsl:if test="@type='x-tier' and .='dojotk'">
        "<xsl:value-of select="../@tuid" />":"<xsl:value-of select="../tuv/seg" />",
      </xsl:if>
    </xsl:for-each>
  </xsl:for-each>
  "build", "@rwa.stageId@"}
 </xsl:template>
</xsl:stylesheet>

Use of the stylesheet above to convert Dojo toolkit related definitions from the TMX files by an Ant task [5]

<target name="convert-tmx">
  <style
    basedir="src/resources"
    destdir="src/resources"
    extension=".js"
    includes="*.tmx"
    style="src/resources/tmx2dojotkxsl"
  />
</target>

A+, Dom
--
Sources:

Introduction to internationalization and localization on Wikipedia.
Unicode Common Locale Data Repository (CLDR).
Dojo toolkit API: dojo.cldr, dojo.i18n, private dijit._Calendar, dojox.widget.Calendar, and dojox.widget.DailyCalendar.
Definition of the Translation Memory eXchange (TMX) format.
Reference of the XSLT/tyle task for Ant scripts.