Sharing Technologies

The JavaScript technology stack

2014-05-29T21:03:00.000-04:00

Context

I've been developing with the JavaScript language since 1999. I was then involved in the development of an administrative console for clusters of calendar servers. Coming from an experience in Windows development with the MFC environment, I quickly implemented the MVC pattern in the browser with a set of frames: one frame taking all the visible space to render the interface, another frame kept hidden (with height="0") for exchanging data with my service (implemented in C++ as a FastCGI module), and the frameset to save the state.
Later, while working for IBM Rational, I had the chance to discover the Dojo Toolkit in its early days (v0.4). After that experience, even if I extended my expertise on mobile (mainly native Android and cross-platform with Unity), I continued to contribute to Web projects. However, I did not have a chance to start a project from scratch, always having to deal with legacy code and constrained schedules... until I joined Ubisoft last February!

The presentation layer and user's input handling

Most of "Web developers" are more hackers than developers: they use PHP on the server to assemble HTML components, they use JavaScript to control the application behaviour in the browser, they control the interface from any place, and they deploy often without the safety net of automated tests... Their flexibility is a real asset for editorial teams but a nightmare for the quality assurance teams!
Here are my recommendations for the development of Web applications:

Use the server as little as possible when preparing the views, just to detect the browser user agent and the user's preferred language in order to deliver the adequate HTML template;
Define all the presentation elements with HTML, CSS, and images. Don't use the JavaScript to create HTML fragments on the fly, only if it clones parts of the HTML template;
Use the JavaScript to inject data and behaviour: switch from one section to another, submit a request to the server, display data and notifications, etc.

The main advantage of letting the presentation coded with HTML, CSS, and images is that it can be delegated to designers. Developers can take the materials designers have produced with Dreamweaver, for example, and insert the identifiers required to connect the JavaScript handlers. Or they provided the initial skeleton instrumented with these identifiers, and designers iterate over them freely. There are so many tools to optimize the presentation layer:

HTML minifier, going up to drop optional tags;
CSS minifier and tools to detect unused rules, like uncss;
Image optimizer and sprite generators.

Designers can then focus on defining on the best interface regardless optimization. The only limitations: don't let them introduce or depend on any JavaScript library!

As for choosing a JavaScript library, I recommend to consider the following points:

A mechanism to define modules and dependencies. The best specification is AMD (for Asynchronous Module Definition), implemented by RequireJS, Dojo, and jQuery, among others. Note that AngularJS has its own dependency injection mechanism.
Until ES6 (or ECMAScript 6) makes it standard, use a library that provide an implementation of Promise, especially useful for its all() method;
A test framework that reports about the code coverage. IMHO, without the code coverage statistics, it's difficult to judge the quality of the tests, then determine our ability to detect regressions early...
Even if you cannot rely on a dispatcher that detects user-agents and preferred languages, use the functionality of the has.js library or equivalent to expose methods for test purposes. Coupled with a smart build system, the exposed methods will be hidden in production.
Just minifying the JavaScript code is not sufficient. It's important to have the dead code removed (especially the one exposed just for test purposes). The Google Closure compiler should be part of your tool set.

The data access layer

In the last 10 years, my back-end services were always implemented in Java, sometimes with my own REST compliant library, sometimes with other libraries like Spring, RestEasy, Guice, etc. Java is an easy language to develop with and, with all the available tooling, ramping up new developers is not difficult. Without counting services like Google App Engine which hosts low profile applications for free.
On the other end, Java is really verbose and not done for asynchronicity. It lacks also the support of closures. And, as with many programming languages, nothing prevents you to use patterns to clarify the code behaviour.

At Ubisoft, I've been given the opportunity to host the back-end services on Node.js. The major factor in favor to Node.js is its WebSocket support (not yet decided between ws and engine.io). The second factor is related to the nature of the application: 99% of the transactions between the clients and the server are short lived. Operations that require long computations are handled by services based on Redis and Hadoop. And finally, Node.js scales well.
When I joined the project, the team had a small working environment made a-la Node.js style: no clear definition of dependencies, a lot of nested callbacks to handle asynchronous behaviours, all presentation built with Jade, no commonly adopted patterns to organize the code logic, no unit tests (obvious as nested callbacks are nearly impossible to test!). I then rebooted the project with a better approach for the separation of concerns:

AMD as the format to define modules, with the Dojo loader to bootstrap the application;
Express to handle the RESTful entry points
A layered structure to process the requests:

Resource: one class per entity, to gather data from the input streams, and forwarding them to the service layer;
Service: one class per entity, possibly communicating with other services when data should be aggregated for many entities;
DAO: one class per entity, controlling data from the file system, from MongoDB, from MySQL or from another service over HTTP.

A set of classes modelling each entities; I imposed this layer to serve two needs: 1) allow some restrictions of entities (attributes can be declared mandatory or read-only, or need to match a regular expression) and 2) support non destructive partial updates.
Unit tests to cover 100% of the implemented logic.

At this stage, the most complex classes are the base classes for the MongoDB and the MySQL DAOs (I judge their complexity by the tests that requires 3 times more code). But with the help of Promises, the code is elegant and compact ;)

/**
 * Select the identified resources, or all resources if not filter nor range are specified
 *
 * @Param {Object} filters bag of key/value pairs to be used as a filter the resources to be returned
 * @Param {Range} range limit the number of results returned
 * @Param {Object} order bag of key/value pairs to be used to order the results
 * @Return a Promise with a list of resources as the parameter of the onSuccess method
 *
 * @Throw error with code 204-NO CONTENT if the selection is empty, as a parameter of the onFailure method of the promise
 */
select: function (filters, range, order) {
    return all([this._select(filters, range, order), this._count(filters)]).then(function (responses) {
        range.total = responses[1];
        responses[0].range = range;
        return responses[0];
    });
},

// Helper forwarding the SELECT request to the MySql connection
_select: function (filters, range, order) {
    var query = this.getSelectQuery(filters, range, order),
        ModelClass = this.ModelClass;

    return this._getConnection().then(function (connection) {
        var dfd = new Deferred();

        connection.query(query, function (err, rows) {
            connection.release();
            if (err) {
                _forwardError(dfd, 500, 'Query to DB failed:' + query, err);
                return;
            }

            var idx, limit = rows.length,
                entities = [];
            if (limit === 0) {
                _forwardError(dfd, 204, 'No entity match the given criteria', 'Query with no result: ' + query);
                return;
            }
            for (idx = 0; idx < limit; idx += 1) {
                entities.push(new ModelClass(rows[idx]));
            }
            dfd.resolve(entities);
        });

        return dfd.promise;
    });
},

// Helper forwarding the COUNT request to the MySql connection
_count: function (filters) {
    var query = this.getCountQuery(filters);

    return this._getConnection().then(function (connection) {
        var dfd = new Deferred();

        connection.query(query, function (err, rows) {
            connection.release();
            if (err) {
                _forwardError(dfd, 500, 'Query to DB failed:' + query, err);
                return;
            }

            dfd.resolve(rows[0].total);
        });

        return dfd.promise;
    });
},

Code sample: the select() method of the MySqlDao, and its two direct helpers.

Few comments on the code illustrated above:

Each helper wraps the asynchronicity with callbacks of the MySQL plugin into Promise (via the Deferred class);
The main entry point relies on the Promise all() method to convey the request result only when the responses from the two helpers are ready;
The method _getConnection() returns a Promise which is resolved with a connection from the MySQL pool (i.e. from mysql.createPool().getConnection());
The method _forwardError() is a simple helper logging the error and rejecting the Promise; at the highest level, express use the error code as the status for the HTTP response;
The method _select() converts each results into an instance of the specified model, providing transparently a support for field validation and partial updates;
With the use of given ModelClass, this MySqlDao class acts like a Java and C# Generics.

The persistence layer

I'm not a DBA and will never be one. Regarding the piece of code above, one of my colleague proposed to update the query for the SELECT in order to get the record count at the same time. It would be repeated with each rows but it would save a round trip. At the end, I decided to keep the code as-is because it's consistent with the code of the MongoDB DAO. We'll measure the impact of the improvement later when the entire application is ready.
If I'm not an expert, I always had to deal with databases: Oracle 10g, IBM DB2, MySQL, PostgreSQL for the relational databases and Google datastore and MongoDB for the Non-SQL ones. My current project relies on MongoDB where player's session information are stored and on MySQL which stores static information. I like working with MongoDB because of its ability to work with documents instead of rows of normalized values. It is very flexible, well aligned with entities used client-side. And MongoDB is highly scalable.
Once the DAOs have been correctly defined, implemented, and tested, dealing with any database at the service level is transparent. Developers can focus on the business logic while DBAs optimize the database settings and deployments.

The continuous integration

Java is an easy language to deal with. First, they are a lot of very good IDE, like Eclipse and IntelliJ. Then, they are plenty of test tools to help verifying the code behaves as expected&em;my favorites ones are JUnit, Mockito, and Cobertura. And finally Java applications can be remotely debugged, profiled, and even obfuscated.

In the past, I controlled the quality of my JavaScript code with JSUnit and JSCoverage. Now I recommend Intern to run unit tests efficiently with Node.js and functional tests with Selenium. I really like Intern because it's AMD compliant, it produces coverage reports, and it mainly do organize my tests as I want! A small run of around 1,000 unit tests by Node.js takes around 5 seconds. The functional test suite with 20 green-path scenarios takes 20 seconds to run in Firefox and Chrome in parallel.
Here is a small nonetheless important point about Intern flexibility:

I want my tests to work on modules totally isolated one from another. To have them isolated, I inject mocks to replace the injected dependencies in the module to be tested.
Intern suggested way requires:

Removing the module to be tested from the AMD cache, with require.undef([mid]);;
Replacing references of dependent classes by mock ones, with require({map: { '*': { [normal-mid]: [mock-mid] } });;
Reloading the module to be tested that will now use the mock classes instead of the original ones;
Calling and verifying the behaviour of the module.

Currently, I prefer instrumenting the modules with the help of dojo/has to be able to access private methods and replace on-the-fly dependent classes with mock ones. Each test injects the required mocks, and the afterEach test method restore all original dependent classes.
My Intern configuration file contains the definition used by dojo/has to expose test-friendly methods, while my index.html and my app.profile.js (used by the Dojo build system) leave it undefined. So these methods are not accessible from the browser, and not even defined in the built code.
With the help of Mockery, I can test everything, up to the classes controlling the access to MySQL, as illustrated above.

In the Java world, maven has replaced ant as the configuration tool of choice. In the JavaScript world, developers have to rely on many tools:

Node.js, npm, and bower to manage the libraries required server-side (npm) and client-side (bower);
Grunt to run administrative tasks like: building CSS file from Stylus ones, running the tests, compiling the code, deploying the built code, etc.
Intern produces test and coverage reports for Jenkins and TeamCity CI tools.

The development environment

My editor of choice is Brackets, by Adobe. It's a very powerful tool, still being actively developed, so continuously better than before. It has a lot of extensions, like an interactive linter and the Theseus debugger. And debugging and fixing extensions to fit your needs is very easy.

MongoDB consumes as much as memory as possible. To avoid cluttering your development environment while keeping your database on hand, I suggest you use vagrant to configure a virtual machine where MongoDB and your Node.js server run in isolation. Coupling a Vagrantfile with a provisioning script allows all your collaborators to benefit from the same configuration.
When it's time to push on production environments, check if there isn't a Grunt extension that can help you sending the compiled code via FTP or SSH on a remote machine or on Amazon Web Service or on Google Cloud Engine.
I hope this helps, Dom

Review of the book: Getting Started with Grunt

2014-03-23T17:33:00.001-04:00

Disclaimer: I've been offered a electronic copy of the book in exchange to this review. I have no contractual relationship with Packt Publishing nor the book author Jamie Pillora.

Context

Since early 2000, I've been on the Java technology stack for implementing three-tier applications (Web or mobile clients, J2EE Web server, Relational databases). I considered the move from make to ant as a real improvement. My first public project on GitHub are still relying on ant.

I was so happy with ant that I switched to maven only around 2010. To me, the main benefit of maven is its management of the dependencies: from the maven repository, I can depend simply on many Java libraries like Google App Engine, RestEasy and Mockito. I can also get resources for the Web client, like the Dojo Toolkit. Over the years, I wrote few plugins to cover the same features I used to get from ant.

Almost one, I heard about Grunt, the JavaScript build tool. I liked that many plugins were also provided (the officially supported plugins with a name prefixed by grunt-contrib-). Because I was happy with my maven environment, I then started to use Grunt only for Stylus and the grunt-contrib-watch.

Recently, I joined the game company Ubisoft to work on a project involving Node.js server-side. My switch to Grunt was immediate ;) I love the extensive set of plugins and the easiness of writing my own ones if needed. The pair npm / Grunt is rock solid: npm which manages the dependencies and Grunt runs the tasks.

When Packt Publishing contacted me for the review of the book Getting Started with Grunt, I saw an occasion to consolidate my knowledge on Grunt, to compare what I know with someone else experience. And reviewing a published book is way more difficult and lengthy than working on drafts, a work I did one for the book Google App Engine and GWT ;)

Book content

As revealed by the title, the book targets new users of Grunt, or people evaluating the technology. If I compare to my own experience, Grunt power users won't learn much by reading the book.

If the book does not explain how Grunt works, it however describes a lot of topics extensively:

The transpiling aspect: from CoffeeScript, from Stylus/Sass/Less;
The code processing: verification with JSLint and JSHint, minification with Uglify;
The code testing: with Mocha and PhantomJS;
The deployment: assembling many files in one, sending it over FTP, publishing on Amazon S3;
The customization: writing and publishing plugins.

Opinion

This is book really targets new Grunt adopter. It helps understanding the basic tasks. It also describes how to setup a build environment for Web clients. I think exposing more grunt plugins, like grunt-exec to run non JavaScript tool, could have set the book as a reference book...

The author decided to focus on one type of application: a Web client based on Jade, the HTML template engine. I think describing tasks for the application logic on a Node.js server, like grunt-express or grunt-nodemon, would have interest a wider audience. Grunt is a really versatile tool.

I hope it helps,
A+, Dom

2013 Products I Can't Leave Without

2014-01-31T22:16:00.000-05:00

Having some spare time before starting a new job at Ubisoft in Montreal, I want to give a try of my old 2009 Products I Can't Leave Without post.

Update April 27, 2014: as many others, I have decided to stop using the Dropbox service. For now, I use Google Drive and QNAP qsynch.

2013

Blogger
Chrome Canary
Dropbox
Eclipse
Feedly
Firefox
Gimp
Git
GMail
Google App Engine
Google Search
KeePassX
SourceTree
Unity3D
VirtualBox
YouTube

Blogger
This is the tool used to publish this blog. I use WordPress in other projects. WP has definitely a larger feature set, without counting its amazing plugin list (commercial and free). However, Blogger is just fine for the type of edition I conduct here.

Company:	Google
Website:	blogger.com
Launch Date:	August 18, 2007

Blogger is a blog publishing platform formerly known as Pyra Labs before Google acquired it in February 2003. Blogger blogs are mostly hosted internally with the “dot Blogspot” domain but they can also be hosted externally on a user’s own server.

Blogger provides bloggers with a WYSIWYG editor that allows them to drag-and-drop widgets, change fonts and edit page elements. Also, Feedburner’s feed management tools are tightly integrated with Blogger blogs due to Google’s recent acquisition.

Credits: CrunchBase

Chrome Canary
Google declines the Chrome browser in 4 versions, and Chrome Canary is its most bleeding-edge version. I still use Firefox for most of the Internet browsing, but I now use Chrome for my development related tasks: Webapp debugging on the desktop, remote debugging on a tablet, device emulation (iPad, Nexus, etc.). As the vast majority of mobile browsers are WebKit based as Chrome, it's definitely a must-have development tool.

Company:	Google
Website:	google.ca/intl/en/chrome/browser/canary.html

Google Chrome is an based on the open source web browser Chromium which is based on Webkit. It was accidentally announced prematurely on September 1, 2008 and slated for release the following day. It premiered originally on Windows only, with Mac OS and Linux versions released in early 2010.

Credits: CrunchBase

Eclipse
For an early adopter of IntelliJ IDEA by JetBrains, I had to use Eclipse (company's tool when working with IBM Rational, cheap when working with Compuware). I should recognize that it is going better and better (especially with the refactoring features and the JavaScript support) and it has more plugins than IntelliJ. It is also a platform for OSGi and for Rich Applications.

Eclipse Website
IntelliJ IDEA Website
OSGi Alliance Website and OSGi introduction on Wikipedia.

Company:	FLOSS
Website:	eclipse.org

Eclipse is a Java-based Integrated Development Environment (IDE). In addition to supporting Java (standard and J2EE), Eclipse supports other programming languages like C++ and Python thanks to plugins. It also offers extensions for developing on Android, BIRT, databases, etc.

Feedly
When Google Reader disappeared, I tried Feedly. It's not a great tool, but it does the job: I can continue to easily read the continuous new stream from the Internet ;)

Company:	Feedly
Website:	feedly.com
Launch Date:	June 2008

Credits: CrunchBase

Firefox
Having been a Web application developer for a long time, I adopted Firefox (then know as Firebird) in 2003. With the introduction of the Firebug extension (in 2005), it became with primary browser and it had never lost this status. Its early integration of Google search was also a serious advantage. These days, with the faviconize extension and Firefox ability to start with the previous configuration, my browser always starts with: iGoogle, GMail, Google Calendar.

Company:	Mozilla
Website:	getfirefox.com
Launch Date:	November 9, 2004

Firefox 4 Hits 100 Million Downloads After A Month (4/22/11).

Credits: CrunchBase

Gimp
I never the budget and training for Adobe Photoshop. So I started using Gimp. If you can pass over its weird interface (too many windows, IMO), Gimp offer tons of features for Web application developers: to adjust pictures, to generate textures, to resize images, etc. And there are plenty of free tutorials on the Web.

Company:	FLOSS
Website:	gimp.org

My favorite video series on Photoshop, starting with the first episode: You Suck at Photoshop #1: Distort, Warp, & Layer Effects.

Git
As a developer, I always want to put my code into a source control system. It is not just because I am afraid that my laptop crashes, then wasting hours of work. It is mainly because I want to keep track of the update history. At work, over the years, I used ClearCase, CVS, and Subversion. For my personal development, I used Subversion a lot and now I use Git.

Company:	FLOSS
Website:	git-scm.com

Free hosting service of open-sources Github - Charges applied for private hostings.

GMail
When I started working, I dealt with many machines and I hated having to start one just to look at a specific inbox. With GMail, my account is available anywhere. When I read Turn Gmail Into Your Personal Nerve Center, I started to use GMail as my knowledge database.

Company:	Google
Website:	gmail.com
Launch Date:	April 1, 2004

Gmail, also known as Google Mail, is a free email service with innovative features like “conversation view” email threads, search-oriented interface and plenty of free storage (almost 7GB). Gmail opened in private beta mode in April 2004 by invite only. At first, invites were hard to come by and were spotted up for sell on auction sites like eBay. The email service is now open to everyone and is part of Google Apps. Paul Buchheit, an early employee at Google, is given credit for building the product.

Another Gmail feature is the organization, tracking and recording users’ contact lists. For instance, if you start typing the letter C into the “To” field Gmail will bring up a list of every email address and contact name starting with the letter. This feature helps when you can’t quite remember a name. Plus, Gmail automatically adds and updates email addresses and names to your contact list when you write emails.

Credits: CrunchBase

Google App Engine
My first job in Montréal, Canada, was with a small company named Steltor (bought few years later by Oracle). The core business was the development of a distributed calendar system (servers in cluster, native clients, web client, mobile client, etc.). Since then, I am used to tracking my work with an electronic calendar. Google Calendar and its ability to mix many agendas is excellent.

Company:	Google
Website:	developers.google.com/appengine
Launch Date:	April 2008

Google App Engine offers a full-stack, hosted, automatically scalable web application platform. The service allows developers to build applications in Python, Java (including other JVM based languages such as JRuby) which can then use other Google services such as the Datastore (built on BigTable) and XMPP. The service allows developers to create complete web application that run entirely on Google’s computing infrastructure and scale automatically as the application’s load changes over time. Google also provides an SDK for local development and site monitoring tools for measuring traffic and machine usage.

Google’s offering competes with Amazon’s Web Services suite, including EC2, S3, SQS, and SimpleDB.

Credits: CrunchBase

Google Search
Google Search is an amazing tool: recently, I was trying to find a solution to a tough technical problem and I found it thanks to Google Search which pointed toward a blog post written the same day, just few hours before, in Europe! Incredible... When I give a conference into universities, I often say: “If I asked a question today and you have no clue about the response, that's fine. If you still have no clue tomorrow, you're in trouble...”

Company:	Google
Website:	google.com
Launch Date:	September 4, 1998

Search is Google’s core product and is what got them an official transitive verb addition to the Merriam Webster for “google”. The product is known for its Internet-crawling Googlebots and its PageRank algorithm influenced heavily by linking.

When users type keywords into the home page search box they are returned with relevant results that they can further refine. Google also has more specific search for blogs, images, news and video. Google will also return search results from your own computer files and emails via Google Desktop.

Credits: CrunchBase

KeePassX
KeePassX is an open source Password Safe, an multi-platform extension of KeePass. I use it in conjunction with Dropbox so my precious list of account coordinates are available on all my devices (desktops, tablets, and phones).

Company:	FLOSS
Website:	keepassx.org

SourceTree
As a developer, I much prefer using command line tools like maven, git, and other ant and Python scripts. Git is a really powerful tool but I worked with team members having some difficulties to deal with the branches, cherry-picking, and stashing for example. The freeware SourceTree offers a neat interface on the top of git. I'm more a command line user than a GUI one. Hoever

Company:	Atlassian
Website:	atlassian.com/software/sourcetree/

Unity3D
For my own company AnotherSocialEconomy.com, I only developed native applications for Android. As a software architect at Electronic Arts, in the (now defunct) Play Mantis franchise lab in Montréal, I worked with Unity developers. I then learned its cross-platform capabilities, it's physics engine, and it's powerful C# library. Since then, I've created few project with Unity and I think it's a powerful ecosystem.

Company:	Unity Technologies
Website:	unity3d.com/

VirtualBox
Developing software requires sometimes specific configurations. Testing them requires always specific configurations (at least to replay always the same test cases every time the source control system, like Git, is updated). There are the famous VMWare products (Workstation, Player, ESX) and Microsoft VirtualPC. VirtualBox is an open source product provided by SUN Microsystems, and it has nice features while being powerful.

Company:	Oracle
Website:	virtualbox.org

VirtualBox is a family of powerful x86 virtualization products for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers, it is also the only professional solution that is freely available as Open Source Software under the terms of the GNU General Public License (GPL).

Presently, VirtualBox runs on Windows, Linux, Macintosh and OpenSolaris hosts and supports a large number of guest operating systems including but not limited to Windows (NT 4.0, 2000, XP, Server 2003, Vista), DOS/Windows 3.x, Linux (2.4 and 2.6), Solaris and OpenSolaris, and OpenBSD.

YouTube
YouTube is famous because of fun videos. But it also hosts technical videos.

Company:	Youtube
Website:	youtube.com
Launch Date:	December 11, 2005

YouTube was founded in 2005 by Chad Hurley, Steve Chen and Jawed Karim, who were all early employees of PayPal. YouTube is the leader in online video, sharing original videos worldwide through a Web experience. YouTube allows people to easily upload and share video clips across the Internet through websites, mobile devices, blogs, and email.

Everyone can watch videos on YouTube. People can see first-hand accounts of current events, find videos about their hobbies and interests, and discover the quirky and unusual. As more people capture special moments on video, YouTube is empowering them to become the broadcasters of tomorrow.

In November 2006, within a year of its launch, YouTube was purchased by Google Inc. in one of the most talked-about acquisitions to date.

Credits: CrunchBase

I hope it helps,
A+, Dom

Singleton uniqueness issue in a C# generic class

2014-01-30T13:01:00.002-05:00

I code in C# within the Unity environment (I really like the co-routine mechanism and the yield return instruction).

At one point, I developed a set of base classes to provide a default behavior to entities that must be exchanged on the wire and cached on the device. And I used generic/template classes to be able to generate instances of the final classes from within the methods of the super class. Information saved on the device are signed with a piece of information obtained from the server.

public class LocalCache<T> where T : BaseModel {

    public static string signatureKey { get; set; }

    public T Get(long id) {
        // Get the JSON from PlayerPrefs
        // Verify its signature
        return (T) Activator.CreateInstance(typeof(T), new object[] { JSON });
    }

    public void Save(T entity) {
        // Get the JSON representing the data
        // Sign the JSON
        // Save the signed JSON into PlayerPrefs
    }

    public void Reset(long id) { ... }

    private string SignData(string data) { ... }
}

In Java, there would be only one instance of the signatureKey variable per JVM, whatever the number of class instances. In C#, I've been surprised to find that this attribute is not unique!

public class LocalCache<T> where T : BaseModel {

    // Other code as shown above...

    public static void TestDifferentInstanciations() {
        LocalCache<System.Int64>.signatureKey = "Set for Int64";

        UnityEngine.Debug.Log("Int16: " + LocalCache<System.Int16>.signatureKey);
        // => produces: Int16:

        UnityEngine.Debug.Log("Int16: " + LocalCache<System.Int64>.signatureKey);
        // => produces: Int64: Set for Int64

        UnityEngine.Debug.Log("string: " + LocalCache<System.string>.signatureKey);
        // => produces: string:
    }
}

My surprise comes from the fact that the concept of generic class only exist for the definitions. At runtime in C# only exist types and the type of LocalCache<Int16> is different from the type of LocalCache<Int64>. Then there are multiple copies of the signatureKey attribute, exactly one per constructed types...

The solution is to set the singleton into a separated non generic class!

internal static class StaticLocalCacheInfo {
    public static string signatureKey { get; set; }
}

public class LocalCache<T> where T : BaseModel {

    public static string signaturePartFromServer {
        get { return StaticLocalCacheInfo.signatureKey; }
        set { StaticLocalCacheInfo.signatureKey = value; }
    }

    // Rest of the business logic...

    public static void TestDifferentInstanciations() {
        LocalCache<System.Int64>.signatureKey = "Set for Int64";

        UnityEngine.Debug.Log("Int16: " + LocalCache<System.Int16>.signatureKey);
        // => produces: Int16: Set for Int64

        UnityEngine.Debug.Log("Int16: " + LocalCache<System.Int64>.signatureKey);
        // => produces: Int64: Set for Int64

        UnityEngine.Debug.Log("string: " + LocalCache<System.string>.signatureKey);
        // => produces: string: Set for Int64
    }
}

Note that I got the confirmation of the behavior on Unity forum.

I hope it helps!
A+, Dom

Objectify and conflicting OnSave / OnLoad callbacks

2014-01-28T11:04:00.000-05:00

Since the beginning, I've been using Google App Engine. I tried first its Python implementation, then I moved to its Java implementation. The move to Java was dictated by the simplicity of the language and the largest set of tooling to code, test, and debug the applications.

My first implementation was using JDO as the ORM layer with my own servlet infrastructure implementing a REST interface. Nowadays, I use Objectify in place of JDO and RestEasy (with Guice and JAX-RS).

In my latest project, I've implemented two sets of entities:

One set which tracks with a creation date and a version number;
One set which extends the first one and tracks the identifier of their owners.

Both base classes implement an @OnSave callback to control their core information. Here are a short version of the these classes.

@Index public abstract class AbstractBase<T> {
    @Id private Long id;
    @Unindex protected Date creation;
    @Unindex protected Long version;

    // Accessors
    // ...

    @OnSave protected void prePersist() {
        if (creation == null) {
            creation = new DateTime().toDate();
            version = Long.valueOf(0L);
        }
        version ++;
    }
}

@Index public abstract class AbstractAuthBase<T> extends AbstractBase<T> {
    @Index private Long ownerId;

    // Accessors
    // ...

    @OnSave protected void prePersist() {
        if (ownerId == null || Long.valueOf(0L).equals(ownerId)) {
            throw new ClientErrorException("Field ownerId is missing");
        }
    }
}

When I ran the implementation of the code above, I faced a weird behavior:

The entities of the User class extending only AbstractBase had their creation and version fields correctly set when persisted.
The entities of the Category class extending AbstractAuthBase had none of these two fields set!

It appears the issue comes from Objectify which was only invoking the first method! And twice BTW...

I looked then at the Objectify implementation, precisely at the methods ConcreteEntityMetadata<T>.processLifeCycleCallbacks() and ConcreteEntityMetadata<T>.invokeLifeCycleCallbacks(). In the first method, you can see that the reference of the methods annotated @OnSave and @OnLoad are accumulated in two lists. In the second method, the given list is parsed and the method is applied to the current object.

My issue arose because applying twice the method prePersist() on a instance of a Category class was always calling the method of the base class AbstractAuthBase<T>! I fixed the issue by renaming the callbacks (one in checkOwnerId() and the other in setCreationAndVersion()).

A+, Dom

SoapUI / REST / JSON / variable namespaces

2012-02-13T21:29:00.001-05:00

As a developer of large Web applications, I'm used to relying on a few proven test strategies:

Unit tests with JUnit / Cobertura and DOH / JSCoverage;
Functional and Integration tests with Selenium 2.

When I joined my current development team to work on the project TradeInsight, I was introduced to SoapUI for the functional tests. I like it's ability to convert JSON to XML, enabling then the writing of XPath match assertions.

There's one little caveat related to the conversion and the XPath assertions:

When the server response contains a single JSON object, the conversion introduce a namespace into the generated XML.
And this namespace depends on the server address.

The following figures show the original JSON response and its conversion to XML with the inferred namespace.

JSON response produced by a REST service.

Transformation into a XML payload with an inferred namespace.

This automatic mapping to namespace server dependent does not allow to write server-agnostic code if you follow the suggested solution!

The following figures show a XPath match expression as documented in the SoapUI training materials. Sadly, running it against an XML with a namespace, this error is reported:

XPathContains assertion failed for path [//startDate/text()] : Exception missing content for xpath [//startDate/text()] in Response.

Simple XPath expression with the corresponding error message.

Corrected XPath expression as suggested, now server dependent :(

A simple solution consists in replacing the specified namespace with the meta-character '*', which match any namespace. As all elements of the XML document are under the scope of the inferred namespace, it's important to prefix all nodes of the XPath expression with '*:', as illustrated by the following figure.

Use of the '*' prefix to produce assertion server independent.

I hope this helps.

A+, Dom

Un nouveau chapître

2011-06-27T21:47:00.000-04:00

Il y a quelques 18 moins, j'ai choisi de devenir un entrepreneur à temps plein, consacrant mon temps et mes ressources financières au développement du service AnotherSocialEconomy. Cela a été une expérience très riche en enseignements.

Ceux qui me connaissent savent à quel point il était important que je sois vraiment maître de mon destin professionnel :

Développer un outil qui offre une vraie valeur à son groupe d'utilisateurs.
Baser les décisions sur les faits et leurs conséquences sans laisser places aux interférences politiques (tellement immobilisantes dans les grandes entreprises).
Utiliser la technologie pour servir d'abord les usagers, puis pour construire les services et faciliter leur développement. En ce sens, la méthodologie Lean Startup et son principe Build-Measure-Learn offre un excellent cadre de travail.
Utiliser la méthodologie Agile (sans oublier les outils de XP) pour développer une application de grande envergure.

Avec mon associé Steven, cela a été très intéressant de définir mes propres objectifs, mon organisation de travail, et diriger le développement de l'entreprise.

Mais il y a un mois, principalement à cause de l'absence d'une perspective de revenus stables dans un futur proche, je me suis mis à la recherche de contrats et de postes permanents. C'est sûr que c'est déchirant, mais l'outil est en production et je peux le faire évoluer dans mon temps libre. En appliquant donc un principe de Lean une fois de plus ("Failure to change is a vice!", Hiroshi Okuda, Président de Toyota Motor Corp.), je replonge du côté salarié.

Finalement, après plusieurs entretiens, mon choix s'est porté sur la compagnie MEI qui développe une plate-forme Web pour aider ses clients (des producteurs de biens de consommation emballés--consumer packaged goods) à suivre leurs campagnes de promotion. Jusqu'il y a deux ans, MEI offrait principalement son service aux grands manufacturiers. Maintenant, dans une offre SAAS, MEI développe une nouvelle version pour les producteurs petits et intermédiaires, sous la marque TradeInsight. La synergie entre mon expérience (Java, JavaScript, cloud, mobile, Agile) et l'équipe est indéniable.

Au fait : MEI embauche encore !

L'aventure d'AnotherSocialEconomy continue, à temps perdu de mon point de vue et dans mes interactions avec les groupes de la communauté technique de Montréal (NewTech, Android, NodeJS, etc.). Pour plus d'informations quant aux partenariats possibles, veuillez prendre contact avec Steven.

A+, Dom

OAuth authorization handling in a Android application

2011-06-07T15:17:00.001-04:00

Note : This post is part of the series "Lessons learned as an independent developer". Please refer to the introduction (in French) for more information. Cet article fait partie de la série intitulée « Leçons d'un développeur indépendant ». Au besoin, lisez mon introduction pour plus d'informations.

Context

AnotherSocialEconomy APIs are based on standards as much as possible: OpenID and OAuth for the authentication and authorization, HTTP-based REST interface (1, 2) for the communication protocol, JSON for the data payload format, etc.

This post is about setting up a Android application which get authorization tokens from a OAuth provider.

OAuth provider

There are many known OAuth providers like Netflix, Twitter, Facebook (coming to OAuth 2.0 soon), Yahoo!, Google, etc. If these providers are convenient, they don't offer much flexibility if some debugging is required.

For this experiment, I'm going to use Google App Engine Java and their OAuth support. For a complete walk-through, refer to Ikai Lan's post: Setting up an OAuth provider on Google App Engine, especially for the part which describes how to get the public and secret keys for your client application to sign communications with the provider.

OAuth client - Work flow

Strategy: Upon creation, the process checks if the authorization tokens have been saved as user preferences.

If they are present, they are loaded to be used to sign each future communication with the application on the server.
If they are missing, the OAuth work flow is triggered:

With the server application keys, a signed request is sent to get a temporary request token.
With this request token, a URL to the authentication page is required and an Intent is created to load the corresponding page in a browser. At this step, the application is stopped.
The user enters his credentials in the browser and grants access rights to the mobile application. The return URL has a custom format: ase://oauthresponse.
The mobile application, which has an Intent registered for that custom URL, is restarted and is given the return URL. A verification code is extracted from this URL.
The verification code is used to issue a signed request asking for the access tokens.

The access tokens are saved as part of the user preferences only if she selected a 'Remember me' option.

Figure 1: Authorization work flow

Alternative: If the mobile application offers anonymous services, like browsing the list of registered stores in the case of AnotherSocialEconomy.com, it can be friendlier to delay the authorization verification.

OAuth client - Initiating the authorization process (1, 2, 3)

To simplify the application development, I have decided to use oauth-signpost, a library provided by Matthias Käppler who wanted a slick and simple way to access Netflix services.

Signpost is the easy and intuitive solution for signing HTTP messages on the Java platform in conformance with the OAuth Core 1.0a standard. Signpost follows a modular and flexible design, allowing you to combine it with different HTTP messaging layers.

Note that this library is also good to manage remotely Twitter accounts.

This section is about initiating the authorization process, which occurs if the application is not called by the application on the server (with the verification code, see next section) and if the OAuth token could not be found in the user preferences. This is the path with the steps {1, 2, 3} in Figure 1.

if (!justAuthenticated && Preferences.get(Preferences.OAUTH_KEY, "").length() == 0) {
    // Display the pane with the warning message and the sign in button
    setContentView(R.layout.main_noauth);

    // Update the 'Remember me' checkbox with its last saved state, or the default one
    final String saveOAuthKeysPrefs = Preferences.get(Preferences.SAVE_OAUTH_KEYS, Preferences.SAVE_OAUTH_KEYS_DEFAULT);
    ((CheckBox) findViewById(R.id.app_noauth_keepmeconnected)).setChecked(Preferences.SAVE_OAUTH_KEYS_YES.equals(saveOAuthKeysPrefs));

    // Attach the event handler that will initiate the authorization process up to opening the browser with the authorization page
    findViewById(R.id.app_noauth_continue).setOnClickListener(new OnClickListener() {
        @Override
        public void onClick(View v) {
            // Check if the 'Keep me connected' check box state changed and save its new state
            boolean keepMeConnected = ((CheckBox) findViewById(R.id.app_noauth_keepmeconnected)).isChecked();
            if (Preferences.SAVE_OAUTH_KEYS_YES.equals(saveOAuthKeysPrefs) != keepMeConnected) {
                Preferences.set(Preferences.SAVE_OAUTH_KEYS, keepMeConnected ? Preferences.SAVE_OAUTH_KEYS_YES : Preferences.SAVE_OAUTH_KEYS_NO);
            }
                    
            // Set up the OAuth library
            consumer = new CommonsHttpOAuthConsumer("<your_app_public_key>", "<your_app_secret_key>");

            provider = new CommonsHttpOAuthProvider(
                    "https://<your_app_id>.appspot.com/_ah/OAuthGetRequestToken",
                    "https://<your_app_id>.appspot.com/_ah/OAuthAuthorizeToken",
                    "https://<your_app_id>.appspot.com/_ah/OAuthGetAccessToken");
                    
            try {
                // Steps 1 & 2:
                // Get a request token from the application and prepare the URL for the authorization service
                // Note: the response is going to be handled by the application <intent/> registered for that custom return URL
                String requestTokenUrl = provider.retrieveRequestToken(consumer, "ase://oauthresponse");

                // Step 3:
                // Invoke a browser intent where the user will be able to log in
                startActivity(new Intent(Intent.ACTION_VIEW, Uri.parse(requestTokenUrl)));
            }
            catch(Exception ex) {
                Toast.makeText(Dashboard.this, R.string.app_noauth_requesttoken_ex, Toast.LENGTH_LONG).show();
                Log.e("Dashboard no auth", "Cannot initiate communication to get the request token\nException: " + ex.getClass().getName() + "\nMessage: " + ex.getMessage());
            }
        }
    });
}

Figure 2 below illustrates the pane main_noauth displaying the warning message and the action button, and figure 3 shows the authorization page as provided by Google for the hosted applications on App Engine.

Figure 2: Pane displayed if application not yet authorized

Figure 3: Google authorization page

Whatever action the user takes, the application is going to be called with the URL ase://oauthresponse. The next section covers this work flow path.

OAuth client - Processing the authorization (4, 5)

The application is registered with an Intent associated to the scheme ase and the host oauthresponse. The labels themselves are not important, only their uniqueness and the correspondence with the return URL specified at Step 2.

<intent-filter>
    <action android:name="android.intent.action.VIEW"/>
    <category android:name="android.intent.category.DEFAULT" />
    <category android:name="android.intent.category.BROWSABLE"/>
    <data android:scheme="ase" android:host="oauthresponse"/>
</intent-filter>

The following code snippet implements the steps 4 and 5 as described in Figure 1.

private boolean checkOAuthReturn(Intent intent) {
    boolean returnFromAuth = false;
    Uri uri = intent.getData();

    if (uri != null && uri.toString().startsWith("ase://oauthresponse")) {
        // Step 4:
        // Get the request token from the Authentication log in page
        String code = uri.getQueryParameter("oauth_verifier");
            
        try {
            // Step 5:
            // Get directly the access tokens
            provider.retrieveAccessToken(consumer, code);
            returnFromAuth = true;
                
            // Persist the tokens
            if (Preferences.SAVE_OAUTH_KEYS_YES.equals(Preferences.get(Preferences.SAVE_OAUTH_KEYS, Preferences.SAVE_OAUTH_KEYS_DEFAULT))) {
                Preferences.set(Preferences.OAUTH_KEY, consumer.getToken());
                Preferences.set(Preferences.OAUTH_SECRET, consumer.getTokenSecret());
            }
        }
        catch(Exception ex) {
            Toast.makeText(Dashboard.this, R.string.app_noauth_accesstoken_ex, Toast.LENGTH_LONG).show();
            Log.e("Dashboard no auth", "Cannot complete communication to get the request token\nException: " + ex.getClass().getName() + "\nMessage: " + ex.getMessage());
        }
    }
       
    return returnFromAuth;
}

The Dashboard class definitions are available in a gist on GitHub. This gist contains also a wrapper of the SharedPreferences class, the application manifest with the declaration of the Intent for the custom return URL, and the layout definition of the pane with the warning and the sign in button.

OAuth Client - The quirks

My Android application is very simple and is configured with the launch mode singleTop. As such, if the system does not destroy the application when the code starts an activity to browse the Authentication service URL, the invocation of the ase://oauthresponse URL by the browser should trigger a call to the onNewIntent() method. It never happened during my tests and on my phone... Every time, the application is recreated and a call to onCreate() is issued. So both functions delegate to the helper checkOAuthReturn().

@Override
protected void onNewIntent(Intent intent) {
    checkOAuthReturn(intent);
}

In this example, I've decided to select the view to associate to the first screen of the application according to the knowledge of the OAuth access token (read from the user preferences or retrieved dynamically thanks to the verification code coming with the ase://oauthresponse URL). The following snippet illustrates this flow. In some occasions, it can be better to start a separate activity if the main pane is instrumented to disable the triggers to protected actions. This approach with a separate activity is also better for the portability.

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    Preferences.setPreferenceContext(PreferenceManager.getDefaultSharedPreferences(getBaseContext()));

    boolean justAuthenticated = checkOAuthReturn(getIntent());
        
    if (!justAuthenticated && Preferences.get(Preferences.OAUTH_KEY, "").length() == 0) {
        setContentView(R.layout.main_noauth);

        // Instrumentation of the pane to initiate the authorization process on demand
        // ...
    }
    else {
        setContentView(R.layout.main);
    }
}

I hope this helps.
A+, Dom

Partage d'expérience dans le développement d'applications Android

2011-06-02T21:57:00.001-04:00

Contexte

Note : Cet article est le premier d'une série intitulée « Leçons d'un développeur indépendant ». Le niveau de la discussion dans cet article est général. Les articles suivant iront plus en profondeur et seront illustrés de bouts de code.

Dans le cadre de mon entreprise AnotherSocialEconomy.com, j'ai eu l'occasion de mettre en œuvre plusieurs bonnes pratiques et je vais en partager quelques unes ici. Je vais notamment me concentrer sur les développements autour de l'application cliente Android, depuis l'identification des usagers jusque l'émission de notifications asynchrones.

AnotherSocialEconomy.com, ou ASE, offre un service connectant consommateurs et détaillants :

Les consommateurs à la recherche d'un produit ou service n'ont qu'à décrire leur demande depuis l'un des multiples points d'entrée de l'application : une page Web faite pour AdWords, le site de ASE ou affilié, la page de l'application Facebook, un message direct depuis Twitter, etc.
Les détaillants participants sont notifiés des demandes en fonction de leurs préférence. Les détaillants sont libres de faire une ou plusieurs propositions en fonction de leur disponibilité.
Au fur et à mesure que les propositions sont composées, les consommateurs en sont notifiés et peuvent à tout moment les décliner ou les confirmer.
Les confirmations sont notifiés aux détaillants qui réservent alors le produit ou le service pour le consommateur.
Ce dernier n'a plus qu'à payer et à en prendre possession.

Pour faire simple : ASE connecte les consommateurs avec les détaillants qui ont les produits ou services qu'ils recherchent en inversant le processus de recherche.

Le moteur de ASE est présentement codé en Java et est hébergé sur l'infrastructure Google App Engine. Dans la suite de cet article, pour généraliser le propos, le moteur de ASE est référencé en tant qu'application serveur.

La gestion des utilisateurs sur le serveur

Depuis le départ, le service des usagers de l'application serveur repose sur OpenID. Avec OpenID, l'identification des utilisateurs est confiée à des services tiers de confiance (Yahoo!, Google, AOL, etc.) sans que l'application serveur ne voit le mot de passe des utilisateurs, seulement leur identifiant OpenID. Ce mode de gestion règle aussi plusieurs problèmes :

Les utilisateurs n'ont pas à créer un énième compte pour le service ASE.
Le serveur est moins à risque car il n'y a pas de mots de passe enregistré là.
La gestion des sauvegardes est plus simple (toujours parce qu'il n'y a pas de mots de passe).
En cas de bris de leur mot de passe, les utilisateurs peuvent assurément mieux s'appuyer sur les services de leur fournisseur OpenID que sur les miens :)

Plus tard dans le cycle de développement, notamment parce qu'il a s'agit de développer une application Facebook, les mécanismes d'identification de Facebook, ceux de Twitter et de ceux de Microsoft Live ont été intégrés à l'application serveur. Des trois, le mécanisme de Twitter est le plus standardisé (OAuth), donnant aussi accès aux données de l'utilisateur du service. Mais tous ont été intégrés de manière à agir comme des services OpenID.

OpenID est un bon système d'identification pour une application cliente Web. Avec les restrictions de sécurité des navigateurs (SSL et sandbox), une fois que l'identité de l'utilisateur est confirmée par un fournisseur OpenID, tant que cette identité reste associée à la session Web, l'envoi de données vers les navigateurs reste protégé.

Par contre, quand l'application cliente est native (sur un ordinateur ou sur un téléphone mobile), il n'est pas possible de s'appuyer sur un mode de session Web robuste comme celui des navigateurs. Aussi une application malicieuse pourrait intercepter l'identifiant de session et s'en servir à l'insu de l'usager. Pour se prémunir contre cette attaque, il est souhaitable d'utiliser OAuth qui signe chaque échange entre l'application cliente et le serveur, rendant caduc l'utilisation de l'identifiant de session Web.

L'authentification des usagers sur le client

Chaque téléphone Android est associé à un utilisateur. Si la carte SIM de l'opérateur téléphonique est changée, les données de l'utilisateur précédent ne sont plus accessibles. Chaque application à accès à son propre espace de stockage protégé, mais l'utilisateur peut réclamer cet espace à tout instant. Ce n'est donc pas une solution de stockage à long terme.

Dans le modèle d'authentification OAuth, les échanges de données sont signés par l'application cliente grâce à un jeton émis par l'application serveur. Grâce à la signature, l'application serveur est assurée de l'identité de l'utilisateur à chaque échange de données.

Pour avoir un jeton, le protocole à observer par l'application cliente est relativement simple :

Émettre une requête pour recevoir un premier jeton dit d'accès.
Ce jeton est utilisé pour initier un appel vers une page d'autorisation.
L'application serveur présente alors une page d'identification où l'utilisateur doit, s'il n'est pas déjà authentifié, entrer son identifiant et son mot de passe, puis accepter que l'application cliente accède aux données qui sont gérées par l'application serveur.
L'application serveur retourne un second jeton attestant de l'acceptation par l'utilisateur de l'accès aux données. Ce jeton a une durée de vie limitée.
Ce second jeton peut être utilisé pour obtenir deux jetons (clé publique et clé secrète) qui permettront à l'application cliente de signer les échanges de données de telle sorte que l'application serveur les associera à l'utilisateur concerné.
Souvent ces deux jetons ont une grande durée de vie (pas d'expiration dans le cas de Twitter), et peuvent donc être sauvegardés par l'application cliente pour signer de manière transparente tous les futurs échanges.
Il faut cependant tenir compte que l'utilisateur peut révoquer ces deux jetons n'importe quand, ou qu'ils peuvent expirer n'importe quand (à cause d'un changement de stratégie du côté de l'application serveur, par exemple) aussi il faut être prêt à exécuter le processus pour obtenir deux nouveaux jetons à n'importe quel moment.

Il est important de noter que la sauvegarde des jetons d'authentification doit être très sécuritaire. Il n'est pas acceptable de les sauvegarder dans un simple fichier texte situé sur une carte d'extension mémoire par exemple. Si le risque d'accès à ces jetons est trop grand, il faut mieux rejouer le scénario ci-dessus pour obtenir un nouveau jeu de jetons.

Au moment où j'écris cet article, le matériel de la série Samsumg S et la tablette Motorola Xoom ont des systèmes de fichiers encryptés. À ma connaissance, même Android 3.1 n'offre toujours pas de solution bas niveau de sécurité maximale...

La réception des notifications asynchrones

Si de plus en plus de fondeurs de silicium mettent l'accent sur la puissance du processeur central (Qualcomm) et leur nombre (NVidia vient d'annoncer un Tegra avec 4 cœurs), si l'augmentation de la bande passante (de HPSA+ à LTE par exemple) permet des échanges de données de plus en plus rapide même loin de tout réseau informatique, la capacité énergétique des téléphones portables modernes reste leur point faible. Par le passé, j'ai eu des téléphones Nokia et Sony Ericsson capables de rester en veille plus d'une semaine. Maintenant, je dois brancher mon téléphone HTC Desire chaque soir, et cela même avec une navigation somme toute restreinte !

Dans ces conditions, maintenir une application éveillée pour pouvoir interroger l'application serveur à intervalles réguliers (technique dite de polling) est à proscrire.

Il y a deux ans, en développant une application pour la plate-forme BlackBerry 5, j'ai utilisé la technique suivante :

L'application client sur le téléphone écoutait un certain nombre de messages du système (changement de type de réseau, perte du réseau, etc.) et les colligeait dans une base de données interne.
C'était l'application serveur qui décidait du moment de transmission de ces données statistiques en envoyant un SMS à chaque téléphone.
À la réception de ce SMS, l'application client ouvrait une connexion HTTP pour transmettre en rafale ses données colligés.
Une fois l'ensemble de données rapatriés de chaque téléphone, l'application serveur établissaient des rapports de couverture pour l'opérateur.

Depuis la version 2.2, il existe le protocole AC2DM: Android Cloud to Device Messaging. Quand une application cliente configurée pour AC2DM s'initialise, elle doit s'enregistrer auprès du serveur local AC2DM et reçoit en retour un identifiant d'enregistrement. C'est la responsabilité de l'application cliente d'envoyer cet identifiant à l'application serveur pour que celle-ci ait la clé pour envoyer les notifications asynchrones à cette application cliente, et à elle seule.

Quelque part, l'approche du AC2DM est semblable à ma méthode d'activation par SMS. Il se peut même qu'elle utilise en sous main cette technique ;) La principale différence réside dans l'aspect service : avec AC2DM, l'application cliente n'a pas à rester active pour recevoir les notifications, c'est le serveur local de notifications qu'il l'activera au besoin.

L'application pour les consommateurs

L'application cliente pour les consommateurs doit offrir plusieurs fonctionnalités :

recevoir les notifications concernant les demandes et les propositions en attente
gérer la liste des demandes et propositions en attente
créer de nouvelles demandes
avec un accès au carnet d'adresses du téléphone pour pouvoir inclure ses « amis » en copie des demandes
avec un accès au système de localisation géographique du téléphone pour faciliter la création des demandes
modifier ou annuler des demandes en attente
confirmer ou annuler des propositions en attente

Le principal objectif de l'application cliente sur les téléphones mobiles est le relais des notifications de mise-à-jour de demande, en réaction à la réception de nouvelles propositions ou de modifications de proposition de la part de détaillants. En quelques « clics », l'utilisateur doit pouvoir accéder rapidement au détail de la demande concernée, au détail de la proposition et à des informations sur le magasin ou bureau du détaillant. Pour faciliter cet accès, la plupart des informations sont sauvegardées sur le téléphone au fur et à mesure qu'elles sont requises. Pour garder une structure proche du modèle de données produit par l'application serveur, le stockage utilisé est le service de base de données interne du mobile (SQLite sur Android, par exemple).

L'application pour les détaillants

L'application cliente pour les consommateurs doit offrir plusieurs fonctionnalités :

recevoir les notifications de nouvelles demandes
créer et gérer des propositions (possiblement avec un accès à la caméra pour scanner les codes barre)
confirmer les livraisons
gérer la liste des demandes et propositions en attente

Parce que les services offerts aux consommateurs sont très différents de ceux offerts aux détaillants, ils sont proposés dans deux applications différentes. Cela réduit les risques de confusion de contexte pour les utilisateurs agissant autant en tant que consommateur que détaillant.

À suivre...

Dans les prochains articles, je décrirai en détail les différentes implantations que j'ai réalisées. Il y a plusieurs techniques qui ne sont pas évidentes, comme celle gérant l'authentification avec OAuth, et j'imagine que cela sera utile à bien des développeurs ;)

A+, Dom

Google App Engine, scheduled tasks, and persisting changes into the datastore: the risk of a race condition

2011-04-16T10:56:00.000-04:00

This post is about a race condition I've accidentally discovered and hopefully fixed. It occurred in App Engine and was generated by tasks I created for immediate execution...

Context

When I started developing in Java for Google App Engine, I decided to give a try with JDO, mainly because it is datastore agnostic (*). Operations managing my entities are organized in DAOs with set of methods like the followings.

public Demand update(Demand demand) {
    PersistenceManager pm = getPersistenceManager();
    try {
      return update(pm, demand);
    }
    finally {
        pm.close();
    }
}

public Demand update(PersistenceManager pm, Demand demand) {
    // Check if this instance comes from memcache
    ObjectState state = JDOHelper.getObjectState(consumer);
    if (ObjectState.TRANSIENT.equals(state)) {
        // Get a fresh copy from the data store
        ...
        // Merge old copy attributes into the fresh one
        ...
    }
    // Persists the changes
    return pm.makePersistent(demand);
}

I knew that changes are persisted only when the PersistenceManager is closed; closing it after an update is safe attitude. I decided anyway to separate the PersistenceManager instance management from the business logic updating the entity for clarity.

This decision offers the additional benefit of being able to share PersistenceManager instance with many operations. The following code snippet illustrates my point: a unique PersistenceManager instance is used for two entity loads and one save.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        // Get the identified demand (can come from the memcache)
        Demand demand = getDemandOperations().getDemand(pm, demandKey, ownerKey);

        // Check if the demand's location is changed
        if (command.contains(Location.POSTAL_CODE) || command.contains(Location.COUNTRY_CODE) {
            Location location = getLocationOperations().getLocation(pm, command);
            if (!location.getKey().equals(demand.getLocationKey())) {
                command.put(Demand.LOCATION_KEY, location.getKey());
            }
        }

        // Merge the changes
        demand.fromJson(command);

        // Validate the demand attributes
        ...

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);

        // Report the demand state to the owner
        ...
    }
    finally {
        pm.close();
    }
}

For my service AnotherSocialEconomy which connects Consumers to Retailers, the life cycle for a Demand is made of many steps:

State open: raw data just submitted by a Consumer;
State invalid: one verification step failed, requires an update from the Consumer;
State published: verification is OK, and Demand broadcasted to Retailers;
State confirmed: Consumer confirmed one Proposal; Retailer reserves the product for pick-up, or delivers it;
State closed: Consumer notified the system that the transaction is closed successfully;
State cancelled: ...
State expired: ...

In my system, some operations take time:

Because of some congestion in the environment, which occurs sometimes when sending e-mails.
Because some operations require a large data set to be processed–like when a Demand has to be broadcasted to selected Retailers.

Because this time constraint and the 30 second limit, I decided to use tasks extensively (tasks can run for 10 minutes). In some ways, my code is very modular now, easier to maintain and test.

So I updated my code to trigger a validation task once the Demand has been updated with the raw data submitted by the Consumer. The code snippet shows the task scheduling in the context of the processDemandUpdate() method illustrated above.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        ...

        // Update the state so the entity is ready for the validation process
        demand.setState(State.OPEN);

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);

        // Create a task for that demand validation
        getQueue().add(
            withUrl("/_tasks/validateOpenDemand").
                param(Demand.KEY, demandKey.toString()).
                method(Method.GET)
        );
    }
    finally {
        pm.close();
    }
}

Issue

Until I activated the Always On feature, no issue has been reported for that piece of code: my unit tests worked as expected, my smoke tests were fine, the live site behaved correctly, etc.

Then the issue started to appear randomly: sometimes, updated Demand instances were not processed by the validation task anymore! A manual trigger of this task from a browser or curl had however the expected result...

For the task to be idempotent, the state of the Demand instance to be validated is checked: if set with open, the Demand attributes are checked with the result of the state being set with invalid or published. Otherwise nothing happens. With that approach, Demands already validated are not processed a second time...

What occurred?

Without the Always On feature activated, because of the low traffic in my application, the infrastructure was delaying the process of the validation task a bit and it was executed once the request process finished.
Thanks to that soft process serialization, the datastore update commanded by the instruction pm.close() had all chances to be completed before the start of the validation task!
With the Always On feature activated, the infrastructure had much more chance to get one of the two other application instances to process the validation task... which could happen before the datastore update...
As it started before the datastore update, the validation task found a task in the state set by the previous run of the task for this Demand instance: invalid or published. Then it exited without reporting any error.

Solutions

The ugly one:
Add a delay before executing the task with the countdownMillis() method.

// Create a task for that demand validation
        getQueue().add(
            withUrl("/_tasks/validateOpenDemand").
                param(Demand.KEY, demandKey.toString()).
                method(Method.GET).
                countdownMillis(2000)
        );
    }
    finally {
        pm.close();
    }
}

A tricky one:
Use memcache to store a copy of the Demand, which the validation will use instead of the reading it from the datastore. Because there's no warranty that your entity won't be evicted before the the run of the validation task, this is not a solution I can recommend.

The simplest one:
Move the code scheduling the code outside the try...finally... block. The task will be scheduled only if the updates of the Demand instance have been persisted.

public void processDemandUpdateCommand(Long demandKey, JsonObject command, Long ownerKey) throws ... {
    PersistenceManager pm = getPersistenceManager();
    try {
        ...

        // Update the state so the entity is ready for the validation process
        demand.setState(State.OPEN);

        // Persist them
        demand = getDemandOperations().updateDemand(pm, demand);
    }
    finally {
        pm.close();
    }

    // Create a task for that demand validation
    getQueue().add(
        withUrl("/_tasks/validateOpenDemand").
            param(Demand.KEY, demandKey.toString()).
            method(Method.GET)
    );
}

The most robust one:
Wrap everything withing a transaction. When a task is scheduled within a transaction, it's really enqueued when the transaction is committed.

Be aware that adopting this solution may require a major refactoring.

Conclusion

Now I understand the issue, I'm a bit ashamed of it. For my defense, I should say the defect has been introduced as part of an iteration which came with a series of unit tests. Before the activation of the Always On feature, it stayed undetected, and later it occurred only rarely.

Anyway, verifying the impact of all calls to external tasks before persisting any changes is one point in my review check list.

I hope this helps,
A+, Dom

--
Notes:
* These days, I would start my application with Objectify. This blog post summarizes many arguments I agree on too in favor to Objectify.

State of the AnotherSocialEconomy Initiative

2011-04-14T11:26:00.002-04:00

When my partner Steven and I started our startup adventure a few years ago, our main goal was to demonstrate our ability to convert an idea into a live project. As we used our experience to build a viable product, we knew it would add value to our resume.

Over the months, the project evolved slowly:

The core idea is: help the consumers who look for a specific product to find the retailer who has it in stock, and help retailers to connect with consumers online and drive them in-store. Our moto: the missing link between shopping online and buying offline.
The proof-of-concept was made of screenshots, live Twitter accounts, and a piece of Python code connecting those accounts together. This material allowed us to be among the semi-finalist companies of TechCrunch50 in 2009!
The first implementation of the engine connected consumers and retailers, each of them interacting with the system with direct messages (DMs), sent from their own Twitter account. At that time, the tool was named Twetailer.
Later, we figured out Twitter was too geeky and we added a connector to accept and generate e-mails. Since then, the engine has a XMPP (instant messaging) connector, another one for Facebook, and a plan for VOIP (with Twilio).
At one point, we were approached to start an experiment for golfers: usually avid golfers have to spend a lot of time on the phone to get three buddies to play with and to book a tee-time. In two months, we created ezToff.com, developed an embeddable widget to ease the creation of a tee-off request, and developed a Web console for the golf course staff. The experiment was shut down because of a lack of traction...
Recently, we started another experiment in the used car market, under the name AnotherSocialEconomy. Our market researches found that the average time to buy a used car is six weeks in Quebec. Typically, consumers start on the Web, grab listings, and call dealerships one after the other. In the Montreal area, there are 300,000 pre-owned cars bought per month: ⅓ from dealerships, ⅓ from wholesalers, and ⅓ from individuals. Dealerships control 45% of the market value.
So far, this experiment has been a partial success: we get demands from consumers and forward proposals from used car sales people. We helped our first customer finding a car in only two weeks! But, as the dealership staff is not used to new technologies (sic), we manage the service for them and it's very time consuming...

What's next?

We are very happy with consumers trusting us. We work hard to continue to improve their experience, on our landing pages and in our communication by e-mail. The priority is to have them qualifying more their demands upfront.

Our focus right now is more on the retailer-side, in order to have sales people in the dealerships interacting with the system by e-mail too. If around 80% of them agree to work with us to serve our users, we have to prepare the proposal details and to reach them out for approval. For the business to scale, they should prepare and post the proposals themselves.

For now, we need more data to determine trends. This is a prerequisite for used car dealers to adopt our methodology. It is also possible that will lead to another pivot.

Lessons learned?

The first one is an obvious one now: nobody can be as committed as the founders! Since I left the company Compuware to become a full time entrepreneur, Steven and I have met many people we expected to work/partner with: a technology company CEO, a former manufacture owner and now real estate agent, a UX designer, a few VCs, a marketer, two successful startup founders, etc. If we sometimes got excellent feedback, none joined us.

The second one is related to the product development: two techies are not enough to make a great product! They can talk about their product at length, but they don't know how to convince decision makers. They need the help of a marketing genius!

Another one is related to the importance of the contact network. If you don't know the right people, very few will listen to you. Having a large address book or friends with deep pockets definitively helps a lot.

And the last one: developing a tool for the general public is difficult! Following the Lean Startup process can really help. Check Ash Maurya's blog, for example.

Technologies learned?

I continue to find Google App Engine an awesome environment. The recent addition of the Channel API which allows the back-end logic to push asynchronous notifications into Web consoles really improves the user experience. On the maintenance side, I appreciate the Java Remote API which simplifies the development of maintenance and data extraction tasks.

Web console side, I've started upgrading the code to Dojo 1.6 and its new HTML5 compliant syntax. I don't use the AMD loader yet, but I'm waiting for the one coming with 1.7. I have recently started to use Selenium 2 for my smoke tests and I really like it!

Mobile side, I wish to have more spare time to update my Android application for ezToff and to benefit from the Android Cloud To Device Messaging (C2DM) API. But I'm also thinking of building application with the awesome dojox.mobile.

To develop our customer base in the used car experiment, we have created two AdWords campaigns: one for each language, both in the Montreal area. Using AdWords and optimizing the campaigns was very instructive. There are many concepts to master: long tail, auto bid, average CPC, conversion rate, landing page quality score, etc. I know understand why so many people choose to become an AdWords certified partner ;)

Wrapping up 2010, preparing 2011

2011-01-06T23:39:00.000-05:00

2010 Summary

2010 was an interesting year for me professionally. Inspired by similar lists online, I present what I did (or can remember at least):

Left Compuware and joined my partner Steven at Milstein and Associates inc. end-of-January, to focus 150% of my time on AnotherSocialEconomy (formely known as Twetailer).
Adapted the Amazon Flexible Payment Service (FPS) library to the App Engine environment—freely available on github.
Refactored the communication layer to be able to send details e-mails to customers, in addition to short ones sent over Twitter and Instant Messaging services.
Built the first Web consoles for Golf players and Golf courses staff, based on Dojo and using the freshly delivered REST API—check ezToff.com.
Built the first Android application for Golf players using their GPS & Address book to ease the tee-off booking process with AnotherSocialEconomy—freely available on github.
Helped preparing pitches to Golf Canada representatives and to Golf staff members and owners.
Developed the AnotherSocialEconomy widget, ready to be embedded in participant websites and loading the AnotherSocialEconomy wizard on demand
Reviewed the book Google App Engine Java and GWT Application Development.
Continued to develop my open-sourced library offering tools for globalizable generic resource bundles (TMX)—on github too.
Developed a prototype of a Facebook application.
Augmented the AnotherSocialEconomy engine to support the used car dealers: buyers don't buy immediately, but collect car information and offers for a while before committing with one dealer => the engine work flow has been adapted to support this slower path of interaction.
Attended presentations to few car dealerlship owners.
Attended meetings with various mentors and potential investors.
Attended meetings of Montreal NewTech, Android Montreal, Augmented Reality Montreal communities

I’m pretty happy with what I have done so far and am looking forward to doing even more.

New technologies

It was also fun to play around some hot new technologies:

Ubuntu 10.04 and 10.10
Android 2.2 and push mechanism on my HTC Desire
App Engine 1.4.0 and channel api
Node.js and WebSockets

2011 Goals and Plans

2011 is going to be critical for AnotherSocialEconomy. The application runs and passed usability tests. The focus point is now on the business development!

Concentrate on one domain (used car market) and get a significant traffic in the Montreal area.
Gather customer feedback (consumer looking for second hand cars and used car dealers), tune the system, and increase traffic. Repeat until 100% satisfaction ;)
Once the system is proven by the traffic and testimonies, involve investors and/or partners to 1) expand the business to other areas or 2) to target another domain or 3) both expand geographically and vertically.
Develop data mining tools for retailers.
Develop domain oriented interfaces for consumers (Web/HTML5 for tablets and PC, native apps for iPhone, Android, BlackBerry).
Add more communication channels (like voice messages with Twilio, for example).
Offer my services as Software Architect & Developer consultant on designing & developing highly scalable and highly available applications on Google App Engine and mobile applications on Android.

A+, Dom

Reviewed book 'Google App Engine Java and GWT Application Development' is out!

2010-12-03T11:57:00.000-05:00

I know that's a pity not to post more regularly! It's just I'm too busy with the developments for AnotherSocialEconomy.com ;)

Here is a little news for Google App Engine developers:

Over the summer, I've been asked to review the draft of the book Google App Engine Java and GWT Application Development. Even with my experience, I learn few techniques, like with the object relationships (chapter 5). A very good book for beginners/intermediates, and still an interesting book for experts.

Enjoy!
A+, Dom

Note: I've no incentive to sell the book, just the pleasure to share a good reference ;)

Securing accounts on the Web

2010-09-20T22:22:00.001-04:00

Situation

Few days ago, my partner Steven got his Google account compromised for a short period of time:

Tweet #1 at 8:52 PM on Sept. 9: Just received 2 calls from friends wondering if I'm being held at a London hotel. FYI, I'm not.
Tweet #2 at 12:23 AM on Sept. 10: Re: Being held in London. My Google account password was changed by an IP address in Nigeria. I've got it back now but with no Contacts.
Tweet #3: at 3:04 PM on Sept. 10: Re Stuck in London: I thought I had everything under control last night but needed http://bit.ly/czgYdg Google Security Breach help to fix.

The thieves used his account to send a scam to few of his friends asking for money because he was supposedly blocked in London without resources.

If Steven's password was not very strong, there's no chance it has been discovered after only few attempts. At no time, Google reported that attempts to log into his account were conducted from computers with IP addresses in Liberia! Steven saw the first warning only when he recovered the access!

Encountered risk

The goal of these thieves was limited to getting money as soon as possible. So they reached out few of Steven's contacts, ones he contacts only occasionally, and they asked for a money to be transferred by Western Union. As they kept the control of his account, they would have been able to get the transaction MTCN (money transfer control number) via his inbox. Western Union maintains a page listing the Common Scams.

Others could have decided to change his password, to just spy his incoming message stream (these ones enabled the POP3 and IMAP accesses), to ask for password reset when Steven is not online, and then to steal his identity in many online services.

Because Steven reacted promptly and because his contacts detected the scam, the thieves did not get any benefit from this operation. They are probably trying to get someone else now, maybe someone from his contact list.

How to reduce the exposure

The first protection consists in defining strong passwords. A lot of services offer information about how to produce strong passwords. I would recommend this Microsoft site Strong Passwords | Microsoft Security—I'm confident that they don't provide the online password checker to enhance a grey dictionary ;)

The second protection would be to use a unique and strong password per account. This is probably the most difficult part! I may use probably 20 to 30 online services, some I use regularly, others I use very rarely. There's no way I can remember so many strong passwords...

My solution: Keepass + DropBox

Keepass is an open source password manager. The tool has been ported on many platforms: Windows, Mac, Linux, iPhone, Android, etc.—Full list on the download page.
DropBox (link with my referral id ;) is an online file sharing system that, thanks to a program installed on each computer/mobile in your network, maintains in sync the corresponding set of files. DropBox is a nice companion to Keepass as it duplicates your password database transparently, reducing the risk to loose the passwords if the original computer is lost.

The combination of the password generator and Keepass secure edit controls makes the tool especially useful:

It's easy to generate a strong passwords (remember: 16 characters or plus ;)
You don't have to remember them as a simple Ctrl+C / Ctrl+V allows to copy securely them in your browser! (the computer clipboard is automatically flushed after few seconds.)

In final, I just have one very strong password (30+ characters) to remember and to change periodically.

Known limitations

Some sites ask users to give a secret answer for a series of predefined questions. If you look at the Apple page below, you'll see that some questions might weaken users more than offering a protection... These days, it's pretty simple to find the responses online!

List of predefined security questions on Apple.com website

Many sites only accepts alphanumerical characters only or don't accept passwords over 20 characters. Oddly enough, most of the bank websites I use prevent too long and too complex passwords! I guess they have other tools to detect intrusions...

List of predefined security questions on Apple.com website

Last minute update

Today, Google announced on its Online Security blog that they will offer a Two-Step Authentication mechanism to log into Google services. This One-Time Password authentication is simpler than distributing a one-time password generator, as Amazon does for example, while providing a still strong security enhancement.

I hope it helps.
A+, Dom

Dilbert: Social Media age

2010-09-14T09:53:00.000-04:00

I cannot resist to re-post this strip!

Are such attitudes only due to managers that need to keep a straight control over their reports in order to seamlessly justify their position? I already covered that aspect in my blog post Manager Attitude back in Dec. 2008. I then stated that the situation would be smoother is such managers could fully assume their facilitator role...

These days, I think it's more related to the difference in each person's cognitive age!

In the traditional chronological order, you could become a manager because you knew the company internal mechanics, the management dynamic, and its long term vision. People finger-pointing the Generation Y (see my post Work around the general laziness, for example) have good reasons to ask new employees to first acquire good experiences before giving them rewards and responsibilities.

However, with the emergence of multi-stream platforms, young people have a tremendous possibilities to learn so much in short periods of time and to overpass older ones. My involvement in edu.cyn.in proved this statement many times: kids are much more prolific at producing online content and sharing with their peers than their own teachers!

On the other end of the spectrum, a lot of retired people who don't have to compete again to maintain their social rank, have gain the possibility to become relevant again by being more connected with younger crowds, more socially involved.

This Dilbert strip illustrates a direct confrontation of the socially aged person and the just chronologically aged one!

I hope you like it too.
A+, Dom

Update business-side

2010-09-06T23:05:00.000-04:00

It's been a long time since my last post. The summer went by so quickly. With my partner Steven Milstein, we have focused on building a generic tool for golfers and golf courses. This post summarizes the latest developments.

The product name

The product name Twetailer is now AnotherSocialEconomy.com. Twetailer was chosen when Twitter started to become mainstream and was made of a combination of Twitter and Retailer. The project Dretailer (more in a future post) was name from Android and Retailer, and we thought about having a variation per connector type.

AnotherSocialEconomy.com is better because it really illustrates that our offer is about a business platform in a global economy, with social networks involved, in a new way of connecting people—and it is not tinted by any service du jour. More information on Steven's blog post The Twouble with Twetailer.

ezToff.com: a dedicated implementation

AnotherSocialEconomy.com provides a generic multi-channel communication engine, conveying messages among consumers and retailers according to the given location and some search criteria. Whenever we explained what's the product is about, potential consumers understood but were blocked when it was time to formulate one of their needs... We knew then we needed to offer a very simple interface where users can post demands without thinking twice about it!

At one point, we met Marc Bienstock who offered to help us building such an interface for golfers! According to Marc:

When a golfer has a chance to play (because his wife gave him the permission ;), he usually calls his buddies to find three of them who are free too.
Then he has to call a golf courses to book a round at the agreed on time, maybe with one or many pull carts, one or two golf carts, etc.
If the booking goes well, he calls his buddies to give them the golf course coordinates.
If it does not work, he calls another club or calls his buddies with the time of the available rounds.
Way too many phone calls to be able to spend a minimum of $200 for 4 players!
On the golf course-side, they have to deal with so many phone calls; some of them have a system queuing calls up to 50!
Average time spent on the phone per caller is too long, and many are useless because they cannot screen them.
When a round stays free, it's an average of $200 lost.

So end-of-July, we launched ezToff.com which allows:

One golfer can submit a request for a tee-off from one central place and it will be broadcasted to all participating golf courses around the given location. The golfer can specify the e-mail addresses of his buddies so they'll be cc'ed for all exchanged messages.
Golf courses are provided a Web console which displays all requests posted in their area. According to their schedule, they can propose rounds—the price per round and the total cost should be documented. Golf courses don't have to watch the console indefinitely as ezToff.com can notify them by e-mail, SMS or tweet.
Proposals are sent back to request initiator by e-mail (his buddies receive a copy if he gave their e-mail addresses). At this step, the golfer can wait for more proposals to come. If he wants to book it, a simple link in the e-mail will generate a response to be sent by e-mail to ezToff.com.

With ezToff.com, golfers can get tee-off proposals with just few clicks. Golf courses get all demands electronically and can focus on the ones they can propose a round to (others are simply declined). Buddies of the golfers are notified at each step of the process automatically. No more infinite phone calls ;)

We interviewed many golfers and they confirmed the golfer's pain. With some visits to golf courses, we've confirmed the courses' pain too. The difficulty we have is that we're not sales people and it's hard to close a deal with golf courses... Knowing that the season is almost finished here, that our business model is probably too expensive for the golf courses, that we need good marketing materials, we're going to tune the offer and be ready for the 2011 season.

The AnotherSocialEconomy.com reseller and influencer programs

The ezToff.com project has been beneficial for us on many points:

With the widget, we have another way to reach consumers: influencers (bloggers, Facebook groups, associations' site, for example) can embed it into their webpages.
If dedicated Web consoles are available to golfers and golf courses, all operations can be done with e-mails, which is the most pervasive communication tool.
The original messages produced by the engine were short to accommodate the Twitter limitation of 140 characters per message. These ones were cryptic to too many people. The variation of these messages for e-mail are now much friendlier and contains links ready to forward the readers' response.
We'll simplify the pricing model thanks to the received feedback.

But the most significant development is the offer of the influencer program:

Each influencer will receive 25% of the revenues generated by consumers confirming demands posted with their copies of the widget.
An influencer can propose one or many widgets anywhere on the Web he reaches his/her community.
We'll work with influencers to customize the widgets for his/her community.

In parallel, we developed the reseller program to share 25% of the revenues generated by retailers proposing goods or services with AnotherSocialEconomy.com.

It's possible that some well organized enterprises will be their own reseller (cut of 25%) and will drive requests with their copies of the widget (another cut of 25%) but that's fine. We'll make some money if they make some, and they'll have a good discount if they help driving more traffic (which means more business to them).

Next steps

On business side, Steven and I are looking for resellers that will help to bring ezToff.com to the next level. We are also trying to develop another domain specific implementation, another skin on the top of the AnotherSocialEconomy.com engine.

On the technical side, if everything works perfectly for the ezToff.com users, I need to document the widget usage and the REST API used by the Web consoles. I need also to enable the Facebook connector for influencers to communicate on this platform as they can do on Twitter. There's also the Android application (project dretailer mentioned above) to update to benefit from the Android Cloud to Device Messaging (c2dm) mechanism and then to push notifications asynchronously. Still a lot to do but no road block ;)

A+, Dom

How to resize VirtualBox disk images?

2010-07-19T23:56:00.001-04:00

-- Update on May 8, 2012 --

Having to resize an image again, I looked at the VirtualBox documentation before following the CloneVDI route a second time. And I was pleasantly surprised that the version 4.1 of VBoxManage accepts a parameter --resize to the command modifyhd!

The process can be done in a matter of minutes:

Stop the hosted system (Win7 in my case).
Run the following command in the folder of your VDI file
VBoxManage modifyhd <name>.vdi --resize <size-in-mb>
Start the virtual machine.
Open the disk manager tool (Use the Menu Windows, type disk man in the search box, and select 'Create and format hard disk partitions').
You should see your drive with the initial partition(s) and new free space.
Click on the partition to extend and choose the command 'Extend Volume' in the contextual menu.
And voilà.

No need to copy the VDI on the host machine. Very fast and robust process.

'Extend Volume' option in the contextual menu.

-- Original post on July 19, 2010 --

Quick post to share a wonderful VirtualBox companion: CloneVDI!

Almost two months ago, I switched from Windows XP to GNU/Linux with Ubuntu 10.04 distribution. Everything goes very well and I do not regret the move.

In my day-to-day job, as the architect/developer/tester of the Twetailer project, engine and of many of its clients, I still need to run programs on the Windows OS, especially the series of Internet Explorer 7 & 8 (IE 6 is killed, isn't it?).

To verify my test suites for Internet Explorer, I rely on VirtualBox running the initial Windows XP release. Because the disk size requirement for the initial version is low, I stupidly created a small 4 GB disk image! Then it required hours to load and install the service packs 2 & 3, plus IE 8, plus the latest .Net framework, plus the security updates. Be careful: 4 GB is way too small to store the system, the virtual memory page file, and the additional stock coming with the service packs and others!

Few days after the initial setup, I was facing the "Not enough space available" warning :-( Instead of wasting another set of hours in a re-installation, I googled virtualbox increase vdi size, and the first article I found was from the VirtualBox forums. I thought it was a good sign and I started reading... but I became quickly disappointed because the given explanations required many tricks and time to setup too. Then I jumped to the last pages (page 6 to be precise) to read:

...

The new way:
Run the CloneVDI tool. Enter name of source VDI and desired disk size into dialog box. Click "Proceed" button. Expect to spend maybe 5 minutes for a typical VDI, longer of course with big drives.

The CloneVDI tool has existed since mid September 2009. It was created specifically so as to remove the need to recommend an embarassingly complicated rigmarole for performing what should have been a simple task. So, in late May 2010 it is quite disheartening to see people still joining the site in order to provide uninformed endorsement of the obsolete procedure.

The post was from Don Milne, alias mpack, the creator of the CloneVDI tool.

Then I loaded CloneVDI from the referenced post, installed Wine with the Ubuntu Software Center, ran CloneVDI, selected my initial image, specified a new name and a new size (now 10 GB), and all the magic transformation occurred in less than 1 minute!

CloneVDI pane: clone an virtual image with an increased size in just a few clicks!

Warning: It seems the cloning only works up to the first snapshot, as my clone did not get the snapshots. This was not a issue for me but be careful on your side because it might be necessary to merge the snapshots. As the cloning is very fast, producing a new image per snapshot should work-around the issue.

Anyway, I wholeheartedly recommend CloneVDI when it's time to allocate more disk space to a VirtualBox machine!

A+, Dom

The joy of Ubuntu

2010-06-07T12:21:00.001-04:00

Few decades ago, most of my work was done on a SUN hardware running Solaris. The Université de Rennes I, France, was providing the Internet access and Mosaic, followed few times later by Netscape, was my favorite browser. At that time, I was preparing a PhD and most of the online discussions were conducted in newsgroups and emails.

One week ago, I definitively switched the OS of my Thinkpad T61 from Windows XP to Ubuntu Lucid Lynx (10.04). In some ways, the Ubuntu environment is not that different from the Solaris I used to work on. For example, I was very pleased to use again an efficient Virtual Desktop system and to benefit from the native Compose key mechanism that allows me to type any crazy sequences producing foreign characters.

Having been pushed on the Microsoft side by my various employers, it seems I forgot all the good sides of the Unix environments. Microsoft marketing has been very brilliant: as many others, I accepted the PC environment limitations! Do you remember the "cooperative multitasking" of Windows 3.1? Do you remember that changing a registry key transformed your Windows NT Workstation in a much capable Windows NT Server? Do you remember that Windows 7 is really the first shiny interface with transparency and gadgets? (Sorry: Vis-what? I don't know ;)

To be honest, let's recognize that the Unix environments did not progress much. On the open source side, the Linux distributions provide a very fragmented offer. The success of some shiny distributions like Knoppix and Ubuntu is fairly recent.

As a geek, I would say that the most impressive feature in these distributions is the 3D rendering engine Compiz and its Cube plugin! It's just amazing.

When I decided to switch to Ubuntu, I wondered if I would lose some features. So far, all have their counterpart for the Linux environment, sometimes with many additional benefits. I did not have to even install Wine, the Windows emulator! Here are the tools that were important to me and I carried over:

Dropbox: has Linux/MacOS/Windows distributions.
Keepass: had to convert the database to 1.x format and now use KeepassX which has Linux/MacOS/Windows distributions.
Aptana/Android SDK/App Engine SDK: equivalent Java packages.
Firefox/Chrome/Add-ons: have Linux/MacOS/Windows distributions.
Skype (for VOIP calls and screen shares)...
OpenOffice.
VirtualBox.
git.

I am so impressed with the system that I've also switched my old Toshiba M40 (one P4 core and a NVidia card) and now my kids have an easy access to many free games and educational tools!

A+, Dom

What motivates us?

2010-05-28T10:16:00.001-04:00

One year and half ago, I wrote the blog post Career Advice '08. The key idea was about encouraging people to develop their own expertise to be successful in their career. Don't wait for your management to give you something exciting, do it yourself!

Two weeks ago, I wrote another blog post Work around the general laziness. In some ways, I reported on my disappointment of not seeing enough autonomous experts, and that entrepreneurs have to compose with variously skilled teams.

I finished my second expose by mentioning the Twetailer's work-for-attribution offer. I explained that many people liked the concept and that few of them have committed to deliver something with their own schedule. Each one at its own pace builds an extended skill set.

What are the commonalities between the contributors?

They have a regular job so the money is not an issue.
They are performers and have a high level of satisfaction.

I was such a contributor when I started working with Steven, before jumping on the entrepreneur-side to help developing the business from the inside.

I had not really identified the sources of my motivation until I saw Dan Pink's illustration of "What motivates us", for a talk given by the Royal Society for the encouragement of Arts, Manufactures and Commerce (theRSA.org).

The three factors that lead to performance and personal satisfaction:

Autonomy

Mastery

Purpose.

I strongly encourage you to look at that video, part for the fun of viewing Dan Pink in action, part for the delivered message. I encourage you also to share your experience as a comment below ;)

A+, Dom

Work around the general laziness ;)

2010-05-18T15:48:00.007-04:00

This blog entry is primarily inspired by a rant from Jason Calacanis, during the episode #47 of This Week in Startups (full show, or directly starting near 5:20). During that show, Jason goes against the Generation Y people who are usually lazy and want everything before actually delivering...

Another source of inspiration is Mark Suster, the co-host of This Week in Venture Capital, with a series of posts culminating with The Long-Term Value of Loyalty.

What's their point-of-view?

Jason is mostly focusing on a group of people, a group that has been educated with a different mindset from his own one. Gen-Y people have more concerns about ecology in general, for example, they are used to getting free or cheap digital goods, and many of them live with their parents longer than before...

If I can agree on the fact that many Gen-Y people seem lazier than the the Gen-X or the baby boomers at the same age, I think the key differentiators is their relationship with money: the Gen-Y people think they need less money, so they don't work that hard to get always more!

In his first post of the series, Mark pinpoints job hoppers, employees who can resign anytime, without much respect for their commitment or for the situation of the company. If Mark readjusts the context in the second post by writing that “quitting a job because it's a mess is OK”, Mark still thinks that being “loyal” is important and everyone should stay loyal for long term benefits...

I can agree with Mark that working with such individuals is risky, that developing a business with not-so-loyal people is tough. However, I think that building a business with people from different mindsets and backgrounds is more valuable. To me, diversity, if you can manage it, is more important than loyalty!

Is laziness a syndrome limited to Generation Y?

T-shirt Laziness Pays Off Now from SNORGTEES

Before becoming an entrepreneur, I was an employee like many others. In France, I worked for a very small company and then for a multinational corporation before immigrating to Canada. My first job here was for a medium Montreal company which was later bought by Oracle. I quitted Oracle to go with IBM Rational, which I quitted to go with Compuware. In addition to my professional life, I've been involved in many non governmental organizations.

Along my life, I've always been curious, interested in learning new subjects, debating around them, and trying to implement the best ideas, especially when I'm passionate. I've been lucky to team up with great people (note that I learned a lot too from the failed partnerships) and I've gotten excellent mentors. When I write “lucky”, I mean “I worked hard to consider myself as lucky.”

All in all, I can honestly say I haven't seen a lot of people working very hard at work, not that they are lazy, just that they have different priorities, different motivations. Although I've already proposed many times to be a mentor, to give lunch-n-learn talks, to organize specific trainings, etc., -on my own time, for the only benefit of the recipient- only few people have followed.

I've observed the “good enough” attitude with any types of people: old or young, men or women, immigrants or native country, Europeans or Americans, etc. In Western countries, we don't have to fight for a shelter or to find food, so the sense of urgency is blunted. Why would people with already enough (enough money, enough responsibilities, enough social involvement, etc.) go for more?

Maybe the Gen-Y people are worse than the others,because their “good enough” level is lower than before. IMHO, they are just like the common crowd, just normal people in our modern world.

How to deal with the general laziness?

As mentioned before, I think the diversity is important. Trying to stay in closed vacuum with the elite can help a bit, but not for a long time. The key point is to compose teams with top elements and less skilled ones. The newbies can learn from the experienced people and more help around them will reduce the “single point of failure” risk.

For sure, you cannot reward the in-learning people as you do reward the top team members. As I explain to my kids, there's a consequence to everything! If you work hard, if your help is valuable, if you go beyond your tasks, the system should reward you accordingly. If it does not come immediately, it should be clear that it will come eventually if everything goes well. If the hard workers don't get a tangible ROI, I think it's normal to expect them to slow down or to quit... (to expect them slowing down ?)

To me, the key factor in a successful project is the commitment of the participants. It's not that important that this intern has a lower velocity than an experienced engineer because we can plan accordingly. What's more important is that you trust that he's going to deliver as expected, or if he has some troubles that he's going to report them as soon as possible.

In the case of software developers evolving in an Agile environment, measuring their work progress is not really an issue. If the team is correctly equipped, everyone is accountable. In the peripheral teams (product managers, marketing, sales representatives, etc.), commitments are more difficult to get, and the more sources of non-productivity there are, the fewer chances of success you get.

A real case, please!

My partner Steven and I have developed the concept of work for attribution. Because we're a startup, we cannot offer salaries in exchange to work. However, we offer to attribute back the work to the ones who have delivered it. Immediately, contributors can use our environment as a lab to test and develop new ideas. We have a working product and it's up to them to adapt it, to make it better.

If our projects are successful, if we can cash them at one point, the contributors will be part of the success and then we'll try to reward them. If a big player wants to acquire us, top contributors will be probably part of the deal.

So far, many people have liked the concept and a few of them have committed to deliver something specific. Some of them are Gen-Y people, some are immigrants, and everyone work at different rates. If I can rely on people I trust, on people who are going to deliver what they have committed to, whoever they are, it's very cool!

A+, Dom

Social Software, Cynapse, and Open Platforms

2010-04-21T12:22:00.003-04:00

What's a Social Software?

These days, most of white collar workers have to deal with computers on a daily basis. Computers are everywhere: from the front desk to allow receptionists to get to the company directory up to garage doors checking who's in, who's out. There are so different types of computers that some of them go unnoticed!

The scope of this post is limited computers used to collaborate:

Make appointments
Produce documents
Review & comment documents
Forward documents
Poll users
Manage tasks

When it's time to collaborate, the vast majority of computer users rely on e-mails. E-mail is probably the most essential tool now, and without e-mails, many are lost. Do you remember the BlackBerry syndrome of the RIM product early adopters? Yeah, the one that wakes up people middle of the night so they can get new e-mails ;)

With tools like Microsoft Outlook, Apple iCal, and Google Calendar, more users relying on calendar tools to organize their time. This is neat because with such tools you can sometimes see up front if the targeted time slot is good for other attendees. And when attendance confirmation comes, the meeting information are updated automatically without requiring a specific triage in the mailbox.

When people have to share pictures, because the ones modern digital camera can generate are so big and very few people have the knowledge to reduce their size nicely, more and more people resort to online hosting services like flickr or Picasa. In addition to process images to transit on the Web (while still available in high resolution for printing purposes), sharing pictures via links is way lighter for the mail system. It has also the additional benefit that pictures can be removed safely without one someone else to purge his/her mailbox.

With review sites like cnet or Epinions.com, more and more users have started to give their opinions online to share the joy of being a owner of a wonderful gadget, or to inform others that such a gadget is crap! Involving readers and customers to share their experiences is the key element of the social software movement.

So what's the relation between Cynapse and social software:

Cynapse is a central place where people can collaborate online with the most practical tools without relying on e-mails, a place where collaboration and communication do not loose their context.

Why Cynapse?

When my partner Steven and I started to work on the !twetailer, we knew that we wanted to collaborate online, within a reliable and protected environment that could handle many document types: user stories, specifications, diagrams, mock ups, pictures, bookmarks, discussions, etc.

As IBM-ers, we looked among the tremendous IBM products and we selected Lotus GreenHouse which was still in beta. The main flaw we experienced was its disruptive slowness. Another issue to us was our quasi-impossibility to influence the development path to fix the painful issues. Just 2 guys in a big user community have a very little impact...

We then decided to switch to Elgg, a free and open source software, that our hosting service offered. Elgg is probably a good tool but it did not match our needs. The variety of entities was limited as our ability to fix the issues. Probably, it would have been better to host the service at home and to extend its model. Because it would have distracted us from our project, we looked for a better solution.

At this time, Steven was studying the social software offering and Cynapse came out as a good fit to us:

The service out-of-the-box was promising and was still under active development.
They had an affordable offer: free self hosted service, managed service for a small subscription, managed hardware also for a fee.
They had an active user community which was hosted on the tool itself (the “eat your own dog food” principle).
Whatever solution we choose, we are free to upgrade/downgrade/exit.

Cynapse is an Indian company and its managers are really great. For practical reasons, we decided to go with the managed service online. When Steven contacted them, asking for a rebate in exchange to us blogging about our experience with the tool and us feeding them with enhancement requests from the development point of view, they accepted and sustained our involvement.

Disclosure: Because the tool and the team is great, Milstein & associates is now a business partner and can resell and offer services on the top of the tool.

After one full year and two upgrades, we are very happy with the tool and its support. At one point, we have developed an offer for schools available at edu.cyn.in which has been extremely well adapted by the kids!

In addition to the collaboration aspect, Cynapse with its offering of various tools in one platform allows users to develop their online reputation in a controlled environment. As Craig Newmark, the founder of craigslist, mentions in a blog post:

People use social networking tools to figure out who they can trust and rely on for decision making. By the end of this decade, power and influence will shift largely to those people with the best reputations and trust networks, from people with money and nominal power. That is, peer networks will confer legitimacy on people emerging from the grassroots.

The Cynapse environment allows users to highlight their work: statistics about contributions and comments are shared on the main page, readers can note the published materials, etc. When people are new to social software, Cynapse offers a simple way to identify the people others “trust” and allows good contributors to build their reputation.

Aside our work on the !twetailer project, because the tool fits our needs, because kids adopted very well, we have proposed it to traditional companies (the ones that relies on Microsoft Outlook and shared folders for their collaboration). For now, the feedback is positive but it's too early to advance any adoption rate ;)

Why Open Platforms?

As a developer, I'm an heavy user of open source software (development environment, source control, build system, etc.). I am also a contributor myself with two open libraries hosted on github:

A set of utilities for Web application developers (Java, Python, JavaScript) which offers:

Globalization features: from one set of central repositories (TMX format) to programming language dependent resource bundles;
JSON related objects: to ease the parsing/generation of JSON structure

An adaptation of the Amazon Flexible Payment Service (FPS) library for the Java environment of Google App Engine.

And the content of this blog is offered under the Common Creative Licence By-NC-SA, which allows using beyond the traditional “fair-use” as long as you cite the source ;) I very like open platform for the reasons Larry Lessig gave in speech on . Open source is good for innovation for geeks like myself. For enterprises, I would argue that the key point is the data access: at anytime, someone can get the data out of an open source project without risking any patent infringement, without risky reverse-engineering. For sure, it won't be free as in “free beer” but they'll be free to get their data back. Some people will argue that close software often offer a way to export data in a standard format (like a SQL dump for a database) and allow then to import them somewhere else, but that's only true if no feature are dropped during the export and if all features from one can be activated in two during the import.

To summarize my point, I would say that open source software allow anyone to exit at anytime while continuing to control the data.

As a collaboration tool, Cynapse is maybe not the best one but it offers competitive advantages for the right price (for free if you have the team and expertise to manage it, for fee if you choose the hassle-free solution of the hosted service on Amazon AWS) while letting users migrating to another platform anytime.

Important point to consider during the selection of an open source solution: the quality of its community. More active is the community, better are your chances that issues you are facing have been documented, better your chances to see your (excellent) enhancement requests supported by others. Being able to contribute to an active community (by submitting bug reports, by answering others' questions, by submitting patches) have also the side benefit of improving your reputation.

A+, Dom

Amazon FPS library for the Google App Engine environment

2010-03-21T23:14:00.002-04:00

Here is a post on Amazon Flexible Payments System (FPS) and the beauty of the open source model!

Amazon FPS as the payment platform

!twetailer is an application connecting people to people, consumers to retailers and vice-versa. The business model is freemiun-based:

Posting demands, receiving proposals, and accepting proposals is available for free to any customers.
Listening to demands, proposing products or services, and closing deals is available for free to any registered retailers.
At one point, !twetailer is going to offer retailers to ask consumers to pay for the accepted products or services. The payment platform is going to be Amazon FPS¹.

Amazon FPS is a really neat platform, the only one to my knowledge allowing to organize money transfers between third-parties. With Amazon FPS, !twetailer will be able to convey money from the consumers directly to the retailers, without the money transiting on !twetailer own account! This is a really safe mechanism.

As a quick introduction to Amazon FPS, I would strongly suggest you listen to that one hour webcast introduced on Amazon Payments blog on April 7, 2009: Monetize your innovation with Amazon FPS. If you use the open-source tool VideoLAN VLC, you can load the ASX file directly from Akamai from here.

Amazon and the open-source model

Amazon FPS, as many others Amazon Web Services (AWS), allows third-party applications to use its services through very simple APIs which are HTTP based! The libraries that developers need to use are mostly wrappers over HTTP connections with some specific controllers formatting the requests and signing them (to avoid a man-in-the-middle process tampering them).

Because HTTP is an open protocol and because Amazon could not probably develop its libraries for all possible Web servers, Amazon opened the libraries' source and attached to them a very liberal license².

This is a very respectable attitude regarding their customers and also very well thought on the business-side: if developers can adopt their libraries for their own needs, Amazon won't have to pay for the corresponding development and it will enlarge the set of applications their platform can serve!

Amazon FPS on Google App Engine platform

The !twetailer server-side logic is Java based and dropping the Amazon FPS library freshly compiled in war/WEB-INF/lib is simple. However, the Amazon FPS code cannot run as-is because of few App Engine limitations...

The first one is encountered when the application needs to build the URL that will launch the co-branded service, a page that will allow consumers to pay for the service or product previously proposed by a retailer.

The static method HttpURLConnection.setFollowRedirects(boolean) controls the VM behavior and is then guarded by a JVM permission.

Read the incident report in the Google App Engine discussion group.

Fixing this issue is simple: tune the ability to follow redirection on the connection itself instead of applying the settings globally.

The second issue is really major:

The library uses the Jakarta Commons HttpClient component to convey payment requests from the application to the Amazon infrastructure. And many of its underlying calls are blocked in Google App Engine Java environment.

I asked for advices on AWS FPS forums. But without response, I have decided to go with my own wrapper of the Google URL Fetch mimicking the HttpClient HttpConnectionManager and HttpConnection classes.

Wrappers of Google URL Fetch for Amazon FPS

Following Amazon's leadership, I offer the URL Fetch wrappers that allows Amazon FPS to work on Google App Engine platform:

Same Apache license, version 2.0.
Code hosted on GitHub, in the repository DomDerrien/amazon-fps-gaej.
For now, only two Amazon classes have been modified: com.amazonaws.ipnreturnurlvalidation.SignatureUtilsForOutbound and com.amazonaws.fps.AmazonFPSClient.
My two additions are the classes domderrien.wrapper.UrlFetch.UrlFetchConnectionManager and domderrien.wrapper.UrlFetch.UrlFetchHttpConnection.

The code currently available works in the simple scenario !twetailer needs. But it is still under development. And the test suite covering it is not yet completed.

UrlFetchConnectionManager class definition

/******************************************************************************* 
 *  Adaptation for the Amazon FPS library to work on the Java platform of
 *  Google App Engine.
 *  
 *  Copyright 2010 Dom Derrien
 *  Licensed under the Apache License, Version 2.0
 */

package domderrien.wrapper.UrlFetch;

import org.apache.commons.httpclient.ConnectionPoolTimeoutException;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpConnection;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.params.HttpConnectionManagerParams;

public class UrlFetchConnectionManager implements HttpConnectionManager {

 private HttpConnectionManagerParams params;
 private HttpConnection connection;
 
 public void closeIdleConnections(long timeout) {
  throw new RuntimeException("closeIdleConnections(long)");
 }

 public HttpConnection getConnection(HostConfiguration hostConfiguration) {
  throw new RuntimeException("getConnection(HostConfiguration)");
  // return null;
 }

 public HttpConnection getConnection(HostConfiguration hostConfiguration, long timeout) throws HttpException {
  throw new RuntimeException("getConnection(HostConfiguration, long)");
  // return null;
 }

 public HttpConnection getConnectionWithTimeout(HostConfiguration hostConfiguration, long timeout) throws ConnectionPoolTimeoutException {
  // As reported in http://code.google.com/appengine/docs/java/urlfetch/usingjavanet.html#Java_Net_Features_Not_Supported
  // > The app cannot set explicit connection timeouts for the request.
  if (connection != null) {
   releaseConnection(connection);
  }
  connection = new UrlFetchHttpConnection(hostConfiguration);
  return connection;
 }

 public HttpConnectionManagerParams getParams() {
  return params;
 }

 public void releaseConnection(HttpConnection connection) {
  connection.releaseConnection();
 }

 public void setParams(HttpConnectionManagerParams params) {
  // Parameters set in AmazonFPSClient#configureHttpClient:
        // - ConnectionTimeout: 50000 ms
  // - SoTimeout: 50000 ms
  // - StaleCheckingEnabled: true
  // - TcpNoDelay: true
  // - MaxTotalConnections: 100 (as proposed in the default config.properties file)
  // - MaxConnectionsPerHost: 100 (as proposed in the default config.properties file)

  this.params = params;
 }
 
}

UrlFetchConnection class definition

/******************************************************************************* 
 *  Adaptation for the Amazon FPS library to work on the Java platform of
 *  Google App Engine.
 *  
 *  Copyright 2010 Dom Derrien
 *  Licensed under the Apache License, Version 2.0
 */

package domderrien.wrapper.UrlFetch;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetAddress;
import java.net.MalformedURLException;
import java.net.Socket;
import java.net.SocketException;
import java.net.URL;

import javamocks.io.MockInputStream;
import javamocks.io.MockOutputStream;

import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpConnection;
import org.apache.commons.httpclient.HttpConnectionManager;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.params.HttpConnectionParams;
import org.apache.commons.httpclient.protocol.Protocol;

import com.google.appengine.api.urlfetch.FetchOptions;
import com.google.appengine.api.urlfetch.HTTPHeader;
import com.google.appengine.api.urlfetch.HTTPMethod;
import com.google.appengine.api.urlfetch.HTTPRequest;
import com.google.appengine.api.urlfetch.HTTPResponse;
import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.appengine.api.urlfetch.URLFetchServiceFactory;

public class UrlFetchHttpConnection extends HttpConnection {

 private static URLFetchService urlFS = URLFetchServiceFactory.getURLFetchService();

 private HostConfiguration hostConfiguration;
 private HTTPRequest _request;
 private HTTPResponse _response;
 private MockOutputStream _requestBody = new MockOutputStream();
 private MockInputStream _responseBody = new MockInputStream();

 
 private HTTPRequest getRequest() throws MalformedURLException {
  if (_request == null) {
   _request = new HTTPRequest(
     new URL(hostConfiguration.getHostURL()), 
     HTTPMethod.POST, // AmazonFPSClient#invoke(Class, Map) uses only POST method
     FetchOptions.Builder.disallowTruncate().followRedirects()
   );
  }
  return _request;
 }
 
 private static final String SEPARATOR = ": ";
 private static final int SEPARATOR_LENGTH = SEPARATOR.length();
 private static final String NEW_LINE = "\r\n";

 private HTTPResponse getResponse() throws MalformedURLException, IOException {
  if (_response == null) {
   // Get the response from the remote service
   _response = urlFS.fetch(getRequest());
   // Rebuild stream of HTTP headers (except the HTTP status retrieved from readLine(String) method)
   StringBuilder buffer = new StringBuilder();
   for (HTTPHeader header: _response.getHeaders()) {
    buffer.append(header.getName()).append(SEPARATOR).append(header.getValue()).append(NEW_LINE);
   }
   buffer.append("Content-Length: ").append(_response.getContent().length).append(NEW_LINE);
   buffer.append(NEW_LINE);
   // Rebuild stream of HTTP content (chunked-encoded)
   buffer.append(Integer.toString(_response.getContent().length, 16)).append(";chunk size").append(NEW_LINE);
   buffer.append(new String(_response.getContent())).append(NEW_LINE);
   buffer.append("0;").append(NEW_LINE);
   _responseBody.resetActualContent(buffer.toString());
  }
  return _response;
 }

 /**
  * Default constructor
  * @param hostConfiguration
  */
 public UrlFetchHttpConnection(HostConfiguration hostConfiguration) {
  super(hostConfiguration);
  this.hostConfiguration = hostConfiguration;
 }

 @Override
 protected void assertNotOpen() throws IllegalStateException {
  throw new RuntimeException("assertNotOpen()");
 }

 @Override
 protected void assertOpen() throws IllegalStateException {
  assert(_response != null);
 }

 @Override
 public void close() {
  // Nothing to do!
 }

 @Override
 public boolean closeIfStale() throws IOException {
  // Safe call, passed to the inherited method
  return super.closeIfStale();
 }

 @Override
 protected void closeSocketAndStreams() {
  throw new RuntimeException("closeSocketAndStreams()");
 }

 @Override
 public void flushRequestOutputStream() throws IOException {
  getRequest().setPayload(_requestBody.getStream().toString().getBytes());
 }

 @Override
 public String getHost() {
  return hostConfiguration.getHost();
 }

 @Override
 public HttpConnectionManager getHttpConnectionManager() {
  throw new RuntimeException("getHttpConnectionManager()");
 }

 @Override
 public InputStream getLastResponseInputStream() {
  throw new RuntimeException("getLastResponseInputStream()");
 }

 @Override
 public InetAddress getLocalAddress() {
  throw new RuntimeException("getLocalAddress()");
 }

 @Override
 public HttpConnectionParams getParams() {
  return new HttpConnectionParams();
 }

 @Override
 public int getPort() {
  return hostConfiguration.getPort();
 }

 @Override
 public Protocol getProtocol() {
  return hostConfiguration.getProtocol();
 }

 @Override
 public String getProxyHost() {
  throw new RuntimeException("getProxyHost()");
 }

 @Override
 public int getProxyPort() {
  throw new RuntimeException("getProxyPort()");
 }

 @Override
 public OutputStream getRequestOutputStream() throws IOException, IllegalStateException {
  return _requestBody;
 }

 @Override
 public InputStream getResponseInputStream() throws IOException {
  return _responseBody;
 }

 @Override
 public int getSendBufferSize() throws SocketException {
  throw new RuntimeException("getSendBufferSize()");
 }

 @Override
 protected Socket getSocket() {
  throw new RuntimeException("getSocket()");
 }

 @Override
 public int getSoTimeout() throws SocketException {
  throw new RuntimeException("getSoTimeout()");
 }

 @Override
 public String getVirtualHost() {
  throw new RuntimeException("getVirtualHost()");
 }

 @Override
 protected boolean isLocked() {
  throw new RuntimeException("isLocked()");
 }

 @Override
 public boolean isOpen() {
  // Safe call, passed to inherited method
  return super.isOpen();
 }

 @Override
 public boolean isProxied() {
  // Safe call, passed to inherited method
  return super.isProxied();
 }

 @Override
 public boolean isResponseAvailable() throws IOException {
  return _response != null;
 }

 @Override
 public boolean isResponseAvailable(int timeout) throws IOException {
  return _response != null;
 }

 @Override
 public boolean isSecure() {
  return hostConfiguration.getPort() == 443;
 }

 @Override
 protected boolean isStale() throws IOException {
  throw new RuntimeException("isStale()");
 }

 @Override
 public boolean isStaleCheckingEnabled() {
  throw new RuntimeException("isStaleCheckingEnabled()");
 }

 @Override
 public boolean isTransparent() {
  // Safe call, passed to the inherited method
  return super.isTransparent();
 }
 
 @Override
 public void open() throws IOException {
  // Nothing to do
 }

 @Override
 public void print(String data, String charset) throws IOException, IllegalStateException {
  // Save the passed HTTP headers for the request
  int idx = data.indexOf(SEPARATOR);
  if (idx != -1) {
   String name = data.substring(0, idx);
   String value = data.substring(idx + SEPARATOR_LENGTH).trim();
   getRequest().addHeader(new HTTPHeader(name, value));
  }
  // Other information are just ignored safely 
 }

 @Override
 public void print(String data) throws IOException, IllegalStateException {
  throw new RuntimeException("print(string): " + data);
 }

 @Override
 public void printLine() throws IOException, IllegalStateException {
  throw new RuntimeException("printLine()");
 }

 @Override
 public void printLine(String data, String charset) throws IOException, IllegalStateException {
  throw new RuntimeException("printLine(string, String): " + data + " -- " + charset);
 }

 @Override
 public void printLine(String data) throws IOException, IllegalStateException {
  throw new RuntimeException("printLine(string): " + data);
 }

 @Override
 public String readLine() throws IOException, IllegalStateException {
  throw new RuntimeException("readLine()");
 }

 private boolean waitForHttpStatus = true;
 
 @Override
 public String readLine(String charset) throws IOException, IllegalStateException {
  if (waitForHttpStatus) {
   // Dom Derrien: called only once to get the HTTP status, other information being read from the response output stream
   int responseCode = getResponse().getResponseCode();
   String line = "HTTP/1.1 " + responseCode;
   switch(responseCode) {
    case HttpStatus.SC_OK: line += " OK"; break;
    case HttpStatus.SC_BAD_REQUEST: line += " BAD REQUEST"; break;
    case HttpStatus.SC_UNAUTHORIZED: line += " UNAUTHORIZED"; break;
    case HttpStatus.SC_FORBIDDEN: line += " FORBIDDEN"; break;
    case HttpStatus.SC_NOT_FOUND: line += " NOT FOUND"; break;
    case HttpStatus.SC_INTERNAL_SERVER_ERROR: line += " INTERNAL SERVER ERROR"; break;
    case HttpStatus.SC_SERVICE_UNAVAILABLE: line += " SERVICE UNAVAILABLE"; break;
    default: line = "HTTP/1.1 " + HttpStatus.SC_BAD_REQUEST + " BAD REQUEST";
   }
   waitForHttpStatus = false;
   return line;
  }
  throw new RuntimeException("readLine(String)");
 }

 @Override
 public void releaseConnection() {
  // Do nothing, connection closed automatically...
 }

 @Override
 public void setConnectionTimeout(int timeout) {
  throw new RuntimeException("setConnectionTimeout(int)");
 }

 @Override
 public void setHost(String host) throws IllegalStateException {
  throw new RuntimeException("setHost(String");
 }

 @Override
 public void setHttpConnectionManager(HttpConnectionManager httpConnectionManager) {
  throw new RuntimeException("setHttpConnectionManager(HttpConnectionManager");
 }

 @Override
 public void setLastResponseInputStream(InputStream inStream) {
  // Safe call, passed to inherited method
  super.setLastResponseInputStream(inStream);
 }

 @Override
 public void setLocalAddress(InetAddress localAddress) {
  throw new RuntimeException("setLocalAddress(InetAddress)");
 }

 @Override
 protected void setLocked(boolean locked) {
  // Safe call, passed to inherited method
  super.setLocked(locked);
 }

 @Override
 public void setParams(HttpConnectionParams params) {
  throw new RuntimeException("setParams(HttpConnectionParams)");
 }

 @Override
 public void setPort(int port) throws IllegalStateException {
  throw new RuntimeException("setPort(int)");
 }

 @Override
 public void setProtocol(Protocol protocol) {
  throw new RuntimeException("setProtocol(Protocol)");
 }

 @Override
 public void setProxyHost(String host) throws IllegalStateException {
  throw new RuntimeException("setProxyHost(String)");
 }

 @Override
 public void setProxyPort(int port) throws IllegalStateException {
  throw new RuntimeException("setProxyPort(int)");
 }

 @Override
 public void setSendBufferSize(int sendBufferSize) throws SocketException {
  throw new RuntimeException("setSendBufferSize(int)");
 }

 @Override
 public void setSocketTimeout(int timeout) throws SocketException, IllegalStateException {
  // Safe call, passed to inherited method
  super.setSocketTimeout(timeout);
 }

 @Override
 public void setSoTimeout(int timeout) throws SocketException, IllegalStateException {
  throw new RuntimeException("setSoTimeout(int)");
 }

 @Override
 public void setStaleCheckingEnabled(boolean staleCheckEnabled) {
  throw new RuntimeException("setStaleCheckingEnabled(boolean)");
 }

 @Override
 public void setVirtualHost(String host) throws IllegalStateException {
  throw new RuntimeException("setVirtualHost(String)");
 }

 @Override
 public void shutdownOutput() {
  throw new RuntimeException("shutdownOutput()");
 }

 @Override
 public void tunnelCreated() throws IllegalStateException, IOException {
  throw new RuntimeException("tunnelCreated()");
 }

 @Override
 public void write(byte[] data, int offset, int length) throws IOException, IllegalStateException {
  throw new RuntimeException("write(byte[], int, int): " + new String(data) + ", " + offset + ", " + length);
 }

 @Override
 public void write(byte[] data) throws IOException, IllegalStateException {
  throw new RuntimeException("write(byte[]): " + new String(data));
 }

 @Override
 public void writeLine() throws IOException, IllegalStateException {
  // Safe call, new line being inserted automatically by the HTTPRequest renderer
 }

 @Override
 public void writeLine(byte[] data) throws IOException, IllegalStateException {
  throw new RuntimeException("writeLine(byte[]): " + new String(data));
 }
}

Anyone is free to fork it for his own needs. Be careful with the code because I deliver it without warranties! If you have issues to report, if you can document how to reproduce them, depending on my workload, I will help you. If you fix the issue on your side, I will be happy to merge the corresponding patches into my main branch.

I hope this helps,
A+, Dom
--
Notes:

At least in United States of America until Amazon extends its coverage to company without a US bank account.
Apache License, Version 2.0, January 2004, which allows users to make modifications while keeping them private.

Ladies and gentlemen, here is !twetailer

2010-03-11T10:37:00.000-05:00

On Tuesday March 9 evening, my partner Steven Milstein and I attended a Montreal NewTech event. As three other companies, Steven pitched our project !twetailer to a crowd of 30-40 people. After few closed presentations, it was our first public pitch and it went very well! Read Steven's blog post for the details.

Background

I met Steven while working for IBM Rational. He was the Business Analyst while I was the Software Architect for the Web client of the Rational Portfolio Manager product. When Steven came to me with the Reverse Retailing idea, we agreed to develop it as a proof-of-concept for our own expertise.

We worked so well that we submitted our project to TechCrunch50 (TC50). If we were not among the 50 finalists, we passed the first selection round on 1000+ applicants and we had the chance to present it to Jason Calacanis during 15 minutes last August.

Boosted by the appreciation we got from Jason, we continued to focus on developing user stories and the corresponding code. We were ready to demo the full cycle early this January. Two months later, after many tests and fine tuning, we are opening !twetailer to broader audience!

Presentation

!twetailer first objective is to “connect supply and demand”, specifically “connect consumers to retailers” in its first version.

Look at this original presentation to get the sense of !twetailer ;) We made it months ago but it's still very accurate. Don't miss the part describing the hub, starting at 3:00.

After the presentation, we got many feedback from the audience and most of our interlocutors got it right. Here is the interpretation of Max Maheu, the presenter of the SolidWild company:

Max: If I register my company and listen for the tags “3d printing prototype logo”, all people using !twetailer will get to me?
Me: Exactly. All demands posted with one or many corresponding tags in the Montreal area will be forwarded to you, will be routed to you for free ;)
...
Me: At this stage, it's possible you'll get unrealistic demands, many of them with the #demo tag. But if you respond to them, that means if you propose something, your message will be routed back to the consumers and your business information will be displayed to them!
Max: It's like advertising my business then! Cool.

What's next?

As mentioned by Guy Kawasaki in Montreal in March 2009, we have chosen to "launch early and to correct progressively". If all the delivered features are fully functional, there is still a long road to go in delivering the full feature set.

At this step, we need to get users, that means consumers and retailers, playing with the system and giving us their feedback.

To collect the information, we have organize the community site twetailer.cyn.in, implemented with the excellent social software from Cynapse. To value the contributions, we have setup a Work-for-attribution protocol: any community member that makes a significant contribution will have his/her work publicly recognized. Any volunteer?

We need also to work on the marketing side:

The brand !twetailer seems too tightly related to Twitter, while using Twitter is just one among the various set of connectors taking to our engine.
If !twetailer initial targets are consumers and retailers in a public market, it can work with closed markets, where wholesalers communicate with manufacturers, for example. This aspect needs to be documented and illustrated.
To ensure a vibrant life to !twetailer, we have plan to open its API to third-party developers, a bit a-la Twitter. Closed friends have already accepted to develop clients (under the Work-for-Attribution CLA) that will exercise it, but we need a stronger communication there too.

Thanks a lot to our families and closed friends for encouraging us on the entrepreneurship path. Thanks to anyone for the feedback because they help us improving !twetailer.

Call to contributors

!twetailer is a big project with a development still growing, so there's a lot of room for anyone to showcase their knowledge!

If you're a developer with Java/JavaScript skills,
If you're a tester with automation experience,
If you're a UI designer & Interactivity specialist,
If you're marketer with a Web 2.0 & Social software experience,
If you're a simple consumer in touch with a vibrant community,
If you're a business owner looking for new ways to reach your customers,
Etc.

Don't hesitate to contact Steven (Steven@Twitter or Steven@Twetailer.com) or myself (Dom@Twitter or Dom@Twetailer.com) and we'll exchange on our community site at twetailer.cyn.in.

A+, Dom

Ma nouvelle vie en tant qu'entrepreneur !

2010-02-08T23:43:00.003-05:00

La démission

Il y a trois semaines, j'ai démissionné de mon poste de Consultant technique pour la compagnie Compuware, à Montréal. Plusieurs facteurs m'ont poussé à prendre cette décision :

Mon projet [toujours secret] personnel arrive à maturité et les conditions de sa mise en marché sont à envisager sérieusement;
Mon boss Paul Czarnik m'assigne sur un projet en relation avec l'intégration des outils de Gomez (récemment achetée par Compuware) dans le produit phare Vantage.
Le récent chaos provoqué par le tremblement de terre en Haïti offre une opportunité de proposer mon projet personnel aux organismes d'aide, dans la même veine de ce qui sera proposé aux organismes de Microfinance, comme Diku Dilenga et Jamii Bora.

D'un côté, il y avait un assignement plus prenant avec mon employeur, et de l'autre, il y avait une demande d'attention plus importante pour aborder une phase critique. J'ai finalement opté pour la démission. Youpi, je suis maintenant indépendant !

Certains amis m'ont demandé si cela avait un rapport avec mon âge (ils se souviennent de mon articule Le cap des 40 ans). Réponse simple: pas du tout, pas de middle-age crisis en vue ;)

C'est plus le cotoiement, même par Internet ou livre interposé, avec des gens inspirant comme Guy Kawasaki (GK's keynote in Montreal) qui me poussent à aller de l'avant, pour « faire quelque chose qui a du sens. »

Et maintenant ?

Du temps où je travaillais en tant qu'Architecte logiciel pour IBM Rational, à Montréal, j'avais eu le plaisir de travailler avec Steven Milstein qui avait le rôle d'Analyste d'affaires.

L'idée originale du projet qui m'occupe dorénavant est de Steven. Avec mon expérience, j'ai pris en charge le développement en nous appuyant sur des outils très accessibles comme :

git et GitHub pour la gestion des sources;
Pivotal Tracker pour le suivi des tâches de développement;
Google App Engine pour l'infrastructure en production, initiallement avec la version Python et puis en version Java (voir Google App Engine Meets Java);
Eclipse et ant comme principaux outils de développement (il y a aussi JUnit, Corbertura, Dojo Toolkit, JSUnit, etc.);

Le travail de collaboration (user stories, blogues techniques, support à la communauté des participants, etc.) est colligé sur la plateforme cyn.in, sélectionnée par Steven pour sa grande qualité et l'évolution rapide de son développement.

Dans l'immédiat, je me concentre sur la mise au point de l'outil, passant d'un rythme de 15 à 20 heures par semaine à quelques 60 heures et plus. En ce moment, le code représente environ 13.000 lignes de code pour 33.000 lignes de test, rien que pour la partie serveur !

Si nous avons fait plusieurs présentations réelles du produit, nous sommes fin prêts pour faire des présentations plus larges, toujours sous le coup d'accord de non divulgation cependant. Idéalement, nous souhaitons avoir des partenariats pour avoir une masse critique et aller de l'avant en ouvrant les portes à tout à chacun ! Excitant, non ?

Parce que nous pensons que notre produit peut aider à la coordination des équipes d'aide aux sinistrés en Haïti, nous avons pris des contacts pour le leur proposer à titre gratuit, mais pour lequel il faut assumer les frais d'exploitation. Comme les contacts se font au niveau gouvernemental, du Canada et du Québec, ou au niveau de grandes ONGs comme la Croix Rouge ou MSF, il risque de se passer un certain temps avant que notre proposition trouve un écho favorable. Le suivi des contacts se poursuit.

Le futur

Pour le moment, je suis sans revenu. N'ayant pas de dettes colossales, ayant un train de vie modeste, je pense pouvoir être en roue libre pendant quelques mois sans que ma famille en patisse.

Avec Steven, en attendant que le grand projet avance, et en exploitant notre expérience des outils de socialisation, nous avons développé une offre reposant sur cyn.in et dirigée vers les écoles et l'initiation des élèves. Le projet s'appelle edu.cyn.in—voir le blog de Steven pour plus de détails.

Pendant la période de lancement, étant toujours « programmeur » sur des technologies récentes, j'ai le sentiment que mon capital de « connaissance technologique » restera solide. Ma communication sur ce blogue en témoignera ;) Le cas échéant, je pourrais toujours aller à la recherche de contrat de consultant pour partager mes connaissances.

Mon activité pour l'organisme de microfinance Diku Dilenga continue, toujours pour le soutien technique, moins en tant qu'investisseur privé. Si le grand projet décolle par contre, je consacrerai du temps pour son adaptation aux organismes de microfinance avec pour objectif d'aider les microentrepreneurs et de d'offrir une source de revenus aux dits organismes.

À suivre donc,
A+, Dom

Unit tests, Mock objects, and App Engine

2009-11-20T18:49:00.010-05:00

For my [still a secret] project which is running on Google App Engine infrastructure [1], I want to make it as solid as possible from the beginning by applying most of the best practices of the Agile methodology [2].

Update 2009/12/05:
With the release of the App Engine Java SDK 1.2.8 (read release notes, I had to update my code and this post on two points:

Without the specification of the JDO inheritance type, the environment assumes it's superclass-table. This type is not supported by App Engine. Only subclass-table and complete-table are supported. In the Entity class described below, I had to add @Inheritance(strategy = InheritanceStrategy.SUBCLASS_TABLE). Read the documentation about Defining data classes for more information.
With the automation of the task execution, the MockAppEngineEnvironment class listed below had to be updated to serve an expected value when the Queue runs in the live environment. Read the details on the thread announcing the 1.2.8. SDK prerelease on Google Groups.

Now, all tests pass again ;)

As written on my post from September 18, I had to develop many mock classes to keep reaching the mystical 100% of code coverage (by unit tests) [3]. A good introduction of mock objects is given by Vincent Massol in his book “JUnit in Action” [4]. To summarize, mock objects are especially useful to inject behavior and force the code using them to exercise complex control flows.

Developing applications for Google App Engine is not that complex because the system has a good documentation and an Eclipse plug-in ease the first steps.

Use case description

Let's consider a simple class organization implementing a common J2EE pattern:

A DTO class for a Consumer;
The DAO class getting the Consumer from the persistence layer, and sending it back with updates; and
A Controller class routing REST requests. The Controller is an element of the implemented MVC pattern

Use case illustration

The code for the DTO class is instrumented with JDO annotations [5]:

Consumer DTO class definition

@PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
@Inheritance(strategy = InheritanceStrategy.SUBCLASS_TABLE)
public class Consumer extends Entity {
    @Persistent
    private String address;
 
    @Persistent
    private String displayName;
 
    @Persistent
    private String email;
 
    @Persistent
    private String facebookId;
 
    @Persistent
    private String jabberId;
 
    @Persistent
    private Long locationKey;
 
    @Persistent
    private String twitterId;
 
    /** Default constructor */
    public Consumer() {
        super();
    }
 
    /**
     * Creates a consumer
     * @param in HTTP request parameters
     */
    public Consumer(JsonObject parameters) {
        this();
        fromJson(parameters);
    }
 
    public String getAddress() {
        return address;
    }
    
    public void setAddress(String address) {
        this.address = address;
    }
    
    //...
}

My approach for the DAO class is modular:

When the calling code is doing just one call, like the ConsumerOperations.delete(String) method deleting the identified Consumer instance, the call can be done without the persistence layer knowledge.
When many calls to the persistence layer are required, the DAO API offers the caller to pass a PersistenceManager instance that can be re-used from call to call. With the combination of the detachable="true" parameter specified in the JDO annotation for the Consumer class, it saves many cycles.

Excerpt from the ConsumerOperations DAO class definition

/**
 * Persist the given (probably updated) resource
 * @param consumer Resource to update
 * @return Updated resource
 * @see ConsumerOperations#updateConsumer(PersistenceManager, Consumer)
 */
public Consumer updateConsumer(Consumer consumer) {
    PersistenceManager pm = getPersistenceManager();
    try {
        // Persist updated consumer
        return updateConsumer(pm, consumer);
    }
    finally {
        pm.close();
    }
}
 
/**
 * Persist the given (probably updated) resource while leaving the given persistence manager open for future updates
 * @param pm Persistence manager instance to use - let opened at the end to allow possible object updates later
 * @param consumer Resource to update
 * @return Updated resource
 */
public Consumer updateConsumer(PersistenceManager pm, Consumer consumer) {
    return pm.makePersistent(consumer);
}

The following piece of the abstract class BaseOperations shows the accessor made availabe to any controller code to get one handle of a valid PersistenceManager instance.

Excerpt from the abstract BaseOperations DAO class definition

/**
 * Accessor isolated to facilitate tests by IOP
 * @return Persistence manager instance
 */
public PersistenceManager getPersistenceManager() {
    PersistenceManager pm = getPersistenceManagerFactory().getPersistenceManager();
    pm.setDetachAllOnCommit(true);
    pm.setCopyOnAttach(false);
    return pm;
}

To finish the use case setup, here is a part of the controller code which deals with incoming HTTP requests and serves or operates accordingly. This specific piece of code replies to a GET request like:

Invocation: http://<host:port>/API/Consumer/43544"

Response:

{key:43544, displayName:"John", address:"75, Queen, Montréal, Qc, Canada", 
locationKey:3245, location: {id:3245, postalCode:"H3C2N6", countryCode:"CA",
latitude:43.3, longitude:-73.4}, ...}

Excerpt from the ConsumerRestlet Controller class definition

@Override
protected JsonObject getResource(JsonObject parameters, String resourceId, User loggedUser) throws DataSourceException {
    PersistenceManager pm = getBaseOperations().getPersistenceManager();
    try {
        // Get the consumer instance
        Consumer consumer = getConsumerOperations().getConsumer(pm, Long.valueOf(resourceId));
        JsonObject output = consumer.toJson();
        // Get the related information
        Long locationKey = consumer.getLocationKey();
        if (locationKey != null) {
            Location location = getLocationOperations().getLocation(pm, locationKey);
            output.put(Consumer.LOCATION, location.toJson());
        }
        // Return the complete set of information
        return output;
    }
    finally {
        pm.close();
    }
}

Simple mock

Now, it's time to test! To start slowly, let's deal with the Restlet getResource() method to verify:

Just one and only one instance of PersistenceManager is loaded by the function;
The PersistenceManager instance is cleanly closed at the end of the process;
There's a call issued to get the identified Consumer instance;
There's possibly a call issued to get the identified Location instance;
The output value has the expected information.

In the corresponding unit test series, we don't want to interfere with the App Engine infrastructure (the following chapter will address that aspect). So we'll rely on a mock for the PersistenceManager class that will be injected into the ConsumerRestlet code. The full source of this class is available on my open source project two-tiers-utils: javax.jdo.MockPersistenceManager.

Custom part of the mock for the PersistenceManager class

public class MockPersistenceManager implements PersistenceManager {
    private boolean closed = false; // To keep track of the "closed" state
    public void close() {
        closed = true;
    }
    public boolean isClosed() {
        return closed;
    }

    // ...
}

Here are the unit tests verifying the different flow paths:

When an exception is thrown, because the back-end does not serve the data for example;
When the Consumer instance returns without location coordinates;
When the Consumer instance is fully documented.

Three tests validating the behavior of the ConsumerRestlet.getResource() method

@Test(expected=IllegalArgumentException.class)
public void testUnexpectedError() {
    // Test prepration
    final PersistenceManager pm = new MockPersistenceManager();
    final BaseOperations baseOps = new BaseOperations() {
        boolean askedOnce = false;
        @Override
        PersistenceManager getPersistenceManager() {
            if (askedOnce) {
                fail("Expects only one call");
            }
            askedOnce = true;
            return pm;
        }
    };
    final Long consumerId = 12345L;
    final ConsumerOperations consumerOps = new ConsumerOperations() {
        @Override
        Consumer getConsumer(PersistenceManager pm, Long id) {
            assertEquals(consumerId, id);
            throw new IllegalArgumentException("Done in purpose!");
        }
    };
    ConsumerRestlet restlet = new ConsumerRestlet() {
        @Override BaseOperation getBaseOperations() { return baseOps; }
        @Override ConsumerOperation getConsumerOperations() { return consumerOps; }
    }
    
    // Test itself
    JsonObject response = restlet.getResource(null, consumerId.toString, null);
}

@Test
public void testGettingOneConsumer() {
    // Test prepration
    final PersistenceManager pm = new MockPersistenceManager();
    final BaseOperations baseOps = new BaseOperations() {
        boolean askedOnce = false;
        @Override
        PersistenceManager getPersistenceManager() {
            if (askedOnce) {
                fail("Expects only one call");
            }
            askedOnce = true;
            return pm;
        }
    };
    final Long consumerId = 12345L;
    final ConsumerOperations consumerOps = new ConsumerOperations() {
        @Override
        Consumer getConsumer(PersistenceManager pm, Long id) {
            assertEquals(consumerId, id);
            Consumer consumer = new Consumer();
            consumer.setId(consumerId);
            return consumer;
        }
    };
    final Long locationId = 67890L;
    final LocationOperations locationOps = new LocationOperations() {
        @Override
        Location getLocation(PersistenceManager pm, Long id) {
            fail("Call not expected here!");
            return null;
        }
    };
    ConsumerRestlet restlet = new ConsumerRestlet() {
        @Override BaseOperation getBaseOperations() { return baseOps; }
        @Override ConsumerOperation getConsumerOperations() { return consumerOps; }
        @Override LocationOperation getLocationOperations() { return locationOps; }
    }
    
    // Test itself
    JsonObject response = restlet.getResource(null, consumerId.toString, null);
    
    // Post-test verifications
    assertTrue(pm.isClosed());
    assertNotSame(0, response.size());
    assertTrue(response.containsKey(Consumer.ID);
    assertEquals(consumerId, response.getLong(Consumer.ID));
}

@Test
public void testGettingConsumerWithLocation() {
    // Test prepration
    final PersistenceManager pm = new MockPersistenceManager();
    final BaseOperations baseOps = new BaseOperations() {
        boolean askedOnce = false;
        @Override
        PersistenceManager getPersistenceManager() {
            if (askedOnce) {
                fail("Expects only one call");
            }
            askedOnce = true;
            return pm;
        }
    };
    final Long consumerId = 12345L;
    final Long locationId = 67890L;
    final ConsumerOperations consumerOps = new ConsumerOperations() {
        @Override
        Consumer getConsumer(PersistenceManager pm, Long id) {
            assertEquals(consumerId, id);
            Consumer consumer = new Consumer();
            consumer.setId(consumerId);
            consumer.setLocationId(locationId);
            return consumer;
        }
    };
    final LocationOperations locationOps = new LocationOperations() {
        @Override
        Location getLocation(PersistenceManager pm, Long id) {
            assertEquals(locationId, id);
            Location location = new Location();
            location.setId(locationId);
            return location;
        }
    };
    ConsumerRestlet restlet = new ConsumerRestlet() {
        @Override BaseOperation getBaseOperations() { return baseOps; }
        @Override ConsumerOperation getConsumerOperations() { return consumerOps; }
        @Override LocationOperation getLocationOperations() { return locationOps; }
    }
    
    // Test itself
    JsonObject response = restlet.getResource(null, consumerId.toString, null);
    
    // Post-test verifications
    assertTrue(pm.isClosed());
    assertNotSame(0, response.size());
    assertTrue(response.containsKey(Consumer.ID);
    assertEquals(consumerId, response.getLong(Consumer.ID));
    assertTrue(response.containsKey(Consumer.LOCATION_ID);
    assertEquals(locationId, response.getLong(Consumer.LOCATION_ID));
    assertTrue(response.containsKey(Consumer.LOCATION);
    assertEquals(consumerId, response.getJsonObject(Consumer.LOCATION).getLong(Location.ID));
}

Note that I would have been able to override just the PersistenceManager class to have the Object getObjectById(Object arg0) method returning the expected exception, Consumer, and Location instances. But I would have pass over the strict limit of a unit test by then testing also the behavior of the ConsumerOperations.getConsumer() and LocationOperations.getLocation() methods.

App Engine environment mock

Now, testing the ConsumerOperations class offers a better challenge.

As suggested above, I could override many pieces of the PersistenceManager class to be sure to control the flow. But to do a nice simulation, I almost need to have the complete specification of the Google App Engine infrastructure to be sure I mock it correctly. This is especially crucial when processing Query because Google data store has many limitations [6] that others traditional database, like MySQL, don't have...

Because this documentation is partially available and because Google continues to update its infrastructure, I looked for a way to use the standalone environment made available with the App Engine SDK [1]. This has not been easy because I wanted to have the test running independently from the development server itself. I found first some documentation on Google Code website: Unit Testing With Local Service Implementations, but it was very low level and did not fit with my JDO instrumentation of the DTO classes. Hopefully, I found this article JDO and unit tests from App Engine Fan, a great community contributor I mentioned many times in previous posts!

By cooking information gathered on Google Code website and on App Engine Post, I've produced a com.google.apphosting.api.MockAppEngineEnvironment I can use for my JUnit4 tests.

Three tests validating the behavior of the ConsumerRestlet.getResource() method

package com.google.apphosting.api;
 
// import ...
 
/**
 * Mock for the App Engine Java environment used by the JDO wrapper.
 *
 * These class has been build with information gathered on:
 * - App Engine documentation: http://code.google.com/appengine/docs/java/howto/unittesting.html
 * - App Engine Fan blog: http://blog.appenginefan.com/2009/05/jdo-and-unit-tests.html
 *
 * @author Dom Derrien
 */
public class MockAppEngineEnvironment {
 
    private class ApiProxyEnvironment implements ApiProxy.Environment {
        public String getAppId() {
          return "test";
        }
 
        public String getVersionId() {
          return "1.0";
        }
 
        public String getEmail() {
          throw new UnsupportedOperationException();
        }
 
        public boolean isLoggedIn() {
          throw new UnsupportedOperationException();
        }
 
        public boolean isAdmin() {
          throw new UnsupportedOperationException();
        }
 
        public String getAuthDomain() {
          throw new UnsupportedOperationException();
        }
 
        public String getRequestNamespace() {
          return "";
        }
 
        public Map getAttributes() {
            Map out = new HashMap();

            // Only necessary for tasks that are added when there is no "live" request
            // See: http://groups.google.com/group/google-appengine-java/msg/8f5872b05214...
            out.put("com.google.appengine.server_url_key", "http://localhost:8080");

            return out;
        }
    };
 
    private final ApiProxy.Environment env;
    private PersistenceManagerFactory pmf;
 
    public MockAppEngineEnvironment() {
        env = new ApiProxyEnvironment();
    }
 
    /**
     * Setup the mock environment
     */
    public void setUp() throws Exception {
        // Setup the App Engine services
        ApiProxy.setEnvironmentForCurrentThread(env);
        ApiProxyLocalImpl proxy = new ApiProxyLocalImpl(new File(".")) {};
 
        // Setup the App Engine data store
        proxy.setProperty(LocalDatastoreService.NO_STORAGE_PROPERTY, Boolean.TRUE.toString());
        ApiProxy.setDelegate(proxy);
    }
 
    /**
     * Clean up the mock environment
     */
    public void tearDown() throws Exception {
        // Verify that there's no pending transaction (ie pm.close() has been called)
        Transaction transaction = DatastoreServiceFactory.getDatastoreService().getCurrentTransaction(null);
        boolean transactionPending = transaction != null;
        if (transactionPending) {
            transaction.rollback();
        }
 
        // Clean up the App Engine data store
        ApiProxyLocalImpl proxy = (ApiProxyLocalImpl) ApiProxy.getDelegate();
        if (proxy != null) {
            LocalDatastoreService datastoreService = (LocalDatastoreService) proxy.getService("datastore_v3");
            datastoreService.clearProfiles();
        }
 
        // Clean up the App Engine services
        ApiProxy.setDelegate(null);
        ApiProxy.clearEnvironmentForCurrentThread();
 
        // Report the issue with the transaction still open
        if (transactionPending) {
            throw new IllegalStateException("Found a transaction nor commited neither rolled-back." +
                    "Probably related to a missing PersistenceManager.close() call.");
        }
    }
 
    /**
     * Creates a PersistenceManagerFactory on the fly, with the exact same information
     * stored in the /WEB-INF/META-INF/jdoconfig.xml file.
     */
    public PersistenceManagerFactory getPersistenceManagerFactory() {
        if (pmf == null) {
            Properties newProperties = new Properties();
            newProperties.put("javax.jdo.PersistenceManagerFactoryClass",
                    "org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory");
            newProperties.put("javax.jdo.option.ConnectionURL", "appengine");
            newProperties.put("javax.jdo.option.NontransactionalRead", "true");
            newProperties.put("javax.jdo.option.NontransactionalWrite", "true");
            newProperties.put("javax.jdo.option.RetainValues", "true");
            newProperties.put("datanucleus.appengine.autoCreateDatastoreTxns", "true");
            newProperties.put("datanucleus.appengine.autoCreateDatastoreTxns", "true");
            pmf = JDOHelper.getPersistenceManagerFactory(newProperties);
        }
        return pmf;
    }
 
    /**
     * Gets an instance of the PersistenceManager class
     */
    public PersistenceManager getPersistenceManager() {
        return getPersistenceManagerFactory().getPersistenceManager();
    }
}

With such a class, the unit test part is easy and I can build complex test cases without worrying about the pertinence of my mock classes! That's really great.

Excerpt of the TestConsumerOperations class

public class TestConsumerOperations {
 
    private MockAppEngineEnvironment mockAppEngineEnvironment;
 
    @Before
    public void setUp() throws Exception {
        mockAppEngineEnvironment = new MockAppEngineEnvironment();
        mockAppEngineEnvironment.setUp();
    }
 
    @After
    public void tearDown() throws Exception {
        mockAppEngineEnvironment.tearDown();
    }
 
    @Test
    public void testCreateVI() throws DataSourceException, UnsupportedEncodingException {
        final String email = "unit@test.net";
        final String name = "Mr Unit Test";
        Consumer newConsumer = new Consumer();
        newConsumer.setDisplayName(name);
        newConsumer.setEmail(email);
        assertNull(newConsumer.getId());
 
        // Verify there's no instance
        Query query = new Query(Consumer.class.getSimpleName());
        assertEquals(0, DatastoreServiceFactory.getDatastoreService().prepare(query).countEntities());
 
        // Create the user once
        ConsumerOperations ops = new ConsumerOperations();
        Consumer createdConsumer = ops.createConsumer(newConsumer);
 
        // Verify there's one instance
        query = new Query(Consumer.class.getSimpleName());
        assertEquals(1, DatastoreServiceFactory.getDatastoreService().prepare(query).countEntities());
 
        assertNotNull(createdConsumer.getId());
        assertEquals(email, createdConsumer.getEmail());
        assertEquals(name, createdConsumer.getName());
    }
    
    // ...
}

Conclusion

As a big fan of TDD, I'm now all set to cover the code of my [still a secret] project efficiently. It does not mean everything is correct, more that everything I thought about is correctly covered. At the time of this writing, just for the server-side logic, the code I produced covers more than 10,000 lines and the unit tests bring an additional set of 23,400 lines.

When it's time to refactor a bit or to add new features (plenty of them are aligned in my task list ;), I feel comfortable because I know I can detect most of regressions (if not all) after 3 minutes of running the test suite.

If you want to follow this example, feel free to get the various mock classes I have added to my two-tiers-utils open-source project. In addition to mock classes for the App Engine environment, you'll find:

Basic mock classes for the servlet (see javax.servlet) and javamocks.io packages -- I had to adopt the root javamocks because the JVM class loader does not accept the creation on the fly of classes in the java root).
A mock class for twitter4j.TwitterUser -- I needed a class with public constructor and a easy way to create a default account.
A series of mock class for David Yu's Project which I use to allow users with OpenID credentials to log in. Read the discussion I had with David on ways to test his code, in fact the code he produced and I customized for my own needs and for other security reasons.

For other details on my library, read my post Internationalization and my two-tiers-utils library.

I hope this helps.

A+, Dom
--
References:

Google App Engine: the homepage and the SDK page.
See my post on Agile: SCRUM is Hype, but XP is More Important... where I mentionned the following techniques: Continuous Integration (CI), Unit testing and code coverage (CQC), and Continuous refactoring.
I know that keeping 100% as the target for code coverage numbers is a bit extreme. I read this article Don't be fooled by the coverage report soon after I started using Cobertura. In addition to reducing the exposition to bugs, the 100% coverage gives a very high chance to detect regressions before pushing the updates to the source control system!
Vincent Massol; JUnit in Action; Editions Manning; www.manning.com/massol and Petar Tahchiev, Felipe Leme, Vincent Massol, and Gary Gregory; JUnit in Action, Second Edition; Editions Massol; www.manning.com/tahchiev. I was used to asking any new developer joigning my team to read at least this chapter 7: Testing in isolation with mock objects.
JDO stands for Java Data Objects and is an attempt abstract the data storage manipulation. The code is instrumented with Java annotations like @Persistent, it is instrumented at compile time, and dynamically connect to the data source thanks few properties files. Look at the App Engine - Using the Datastore with JDO documentation for more information.
For general limitations, check this page Will it play in App Engine. For JDO related limitations, check the bottom of the page Usin JDO with App Engine.