Weird Al's not the only one who knows how to go large with Jackson

A couple of months ago I was asked to build a processor that would take a JSON file; perform a few elementary checks and transformations and upload the resulting records into a Couch DB. I hadn’t done any JSON, or Couch and, naturally, the whole thing had to be done yesterday – but at first glance it didn’t look like much of a challenge.

However, looking a little deeper, my processor was going to be part of a replication process across two databases housed in separate enterprises. The replication was going to be based on a daily snapshot – each JSON file would be a copy of the entire database – and a few back-of-the-envelope calculations suggested that the files may become rather large (40+GB) over time.

There was also a fairly tight processing window within which the upload had to be complete. Repressed memories of chasing down and remediating an old DOM based XML project plagued with OutOfMemoryExceptions and very slow processing times warned that some thought was required.

Obviously I wasn’t going to be able to be able to read the whole file into RAM to transform it and fortunately, I didn’t need to – the file was a collection of a couple of million simple objects with no cross references. So to minimize the amount of memory required by the processor, I simply needed to parse the large file into a stream of objects and transform them either individually or in small collections.

Jackson

Key “JSON Parsing” into your favourite search engine and you find about thirty different technologies competing for your attention. Also about a hundred different opinions as to which was “the best”. After a brief review, Jackson recommended itself though the tutorial which indicated that it included a Streaming API, as well as a DOM like Tree Model and JAXB style object bindings.

Given the memory and time concerns above, my first concept was to use the Streaming API to identify object boundaries within the input file to break it down into small chunks and to then use a JAXB style object binding to do the simple transformations required (mostly field name changes). The object binding piece was beautifully trivial:

public class Business {
    @JsonView({ BriefView.class, FullView.class })
    private String businessId;

    @JsonView({ BriefView.class, FullView.class })
    private String name;

    @JsonView({ FullView.class })
    private String address;

    ...
}

Using the JsonView concept, I was able to use the same value class to render different JSON requests for various external services supporting the checking and transformations required.

The Streaming API piece wasn’t going so smoothly. Although it was straight-forward to identify the object boundaries in the JSON token stream, there didn’t seem to be a natural (i.e. easy) way of accessing the underlying character stream in order to create the smaller work unit for the object bindings to use.  A re-think was necessary.

The object bindings were giving me an easy way of creating a collection of value objects in memory from a JSON stream – and conversely of creating a JSON stream from a collection of value objects. If I could get the Jackson object binding to create a stream of value objects, then I could dispense with the streaming API altogether. Some trawling through the Jackson API soon showed this wasn’t only possible, but easy:

JsonFactory f = new MappingJsonFactory();
JsonParser jp = f.createJsonParser(reader);

do {
    Business business = jp.readValueAs(Business.class);

    if (business == null) {
        handler.complete(context);
        break;
    }

    numberOfBusinessesReceived++;
    handler.handleBusiness(business, context);

} while (true);
jp.close();

The object handler here applies some cheap validation rules and aggregates a bunch of objects into a file for later (in memory) processing and upload to Couch. The context object carries some extra information – like the name of the file being processsed.

The magic provided by Jackson is the readValueAs method, which reads enough of the stream to create an object. Pushing a 10GB of two million records file through the processor running on an ordinary desktop with 256M of heap space yielded a processing time of 6 minutes – well within the required processing time window.\

Conclusion

The creators of Jackson have learnt the lessons from XML parsing well. Jackson offers a range of parsing approaches that allows developers to choose the right approach to the task at hand. In short, Jackson eats massive JSON files – without raising a sweat.

Yow! Brisbane 2011

I recently had the pleasure of attending the Brisbane Yow! Conference.  This was a great conference with talks to interest developers across many languages platforms and experience.

This blog is a summary of the more interesting talks that I attended and my thoughts on them.

Keynote – Top 10 JVM Erroneous Zones – Cameron Purdy

The opening talk of day one was a very technical discussion from an Oracle VP, taking us through what he thought were areas that can/could/should be improved in the Java Virtual Machine.  It was a reasonably low level talk for a Keynote but this was a developer’s conference so not entirely unexpected.

Purdy spoke about low-level extensibility in terms of abstracting on primitive types and the limitations of building word length into compilers.  He also touched on precision, lack of continuations support and made comparisons with other languages such as C++ and Scala to support his points.

These points were then used to relate back to development work being done at Oracle on the JVM.  The case was made that since Oracle took over JVM development and maintenance, many improvements and efficiencies in the language have been made. As Java remains a flagship product for Oracle they will continue to do so going forward.

What I took from the talk was that someone driving development is being constructively critical of the language, which is undergoing a lot of criticism in  light of the popularity of dynamic languages such as Groovy.

Lean and Agile  – Continuous Design – Mary Poppendieck

This talk from the Lean Luminary was not quite what I expected. It focused more on a wider view of product design from conception to market place whereas I thought it would be covering incremental design, how best to manage that within an agile environment and how best to integrate that with a continuous delivery approach.

Although the talk was a little broader than I expected, Mary continually reaffirmed the 3M philosophy of make a little, sell a little and learn a little.  This was then discussed in terms of product cycles and how important design conception is and how it can then be refined through iterations of testing and using the design to the point that it can be taken to market.

A lot of comparisons were made between Google, Apple and Dropbox in terms of defining and refining products before they go to market and what comes after go live in terms of design.

An interesting point was her discussion on enabling toggling of features in production so that unfinished features do not disrupt the continuous delivery cycle. She discussed doing small releases via load balancing to test out new features on a smaller audience in case they fail and drive people away.

Overall this was a great talk to be listening to if you were taking a new product to market or worked within a start up type environment.   Of course it was quite a privilege to hear Mary Poppendieck present and she is an excellent presenter.

 DevOps – We’re all Developers Now – Julian Simpson

This talk eventually got underway after a few technical hitches with the speaker’s laptop not co-operating with the projector.  You definitely don’t want technical difficulties in front of a room full of developers.

Once the talk began, it was basically a quick overview of DevOps and then mainly a comparison of Puppet and Chef.  Both Puppet and Chef are automation tools for Sys Admins.   The talk was very interesting showing how a sys admin would manage the installation of an application using the Nginx web server and Jetty.  Examples of how this is configured in Puppet and Chef where then shown illustrating how the configuration is then version controlled to provide change control management and one version of truth.

The power of this was demonstrated very well as Julian discussed how the configuration defined where specific components were to be installed across many virtual servers, allowing the administrator to roll out changes very quickly to a scalable number of machines.  Very powerful and quite dangerous – all the more reason for source control and proper change control!

DevOps is a fast growing and very interesting development in the industry and is leading to more responsibility for the developer as the gap between development and production is closing….

I thought this was a very good talk and I am looking closely at the various DevOps movements within our clients.

Transitioning To The Cloud – Challenges Transitioning To The Cloud – Rod Johnson

A very interesting talk from Rod Johnson, the founder of Spring.  This talk was Rod discussing what he sees as the issues and challenges facing the drive of software into the cloud.

He defined the differences between Infrastructure as a service (IaaS), Platforms as a service (PaaS) and Software as a service (SaaS) and then spoke of the trends he thinks will happen with these.  These were namely that there will be more growth in PaaS and likened it to the second coming of the main frame.  He talked about a standardized stack such as a Ruby or a Java stack, which might see developer’s choice narrow and will also see the tools as part of the platform.

It was very interesting hearing someone of Rod’s experience and foresight discussing the cloud and what he saw it meaning for developers in the future.

The Interactive Web – The Live Web – Dan Ingalls.

Dan Ingalls is one of the founders of Smalltalk so it was quite surreal to have him present a demonstration on a JavaScript web IDE for building HTML 5 applications.

The Living Kernel is a component architecture for HTML and runs in the browser.  It lets developers build and compose with existing components in a canvas like fashion and then tie them together with events, animations and scripting to quickly pull together visual applications.

While the examples shown in the talk were all relatively lightweight there was some scope of the power of this show when he quickly generated a graph based on Unix output to show CPU utilization of a server.  This would then be saved as a HTML report.  His entire presentation was also built using the tool, which showed its versatility.

What was missing in this talk was the business benefits of these tools.  He mentioned towards the end that they generate some reporting for come clients using this but it was only mentioned in passing.

Better Testing for Less Work – QuickCheck – John Hughes

This very entertaining talk was based around the speakers experience using QuickCheck.

QuickCheck is a testing tool, which based on a set of rules will then generate a specified number of tests with random data.  If an error is found, then the tool will focus on reproducing that error as many times as possible to determine the simplest cause of the error from the random data it generated.

John Hughes explained the tool at a high level and briefly showed how to specify a property rule but the basis of his talk was his experiences using QuickCheck in industry.  He discussed the use of the tool in car industry software and how he identified an issue in a production database.

John Hughes was such an entertaining presenter the entire audience was hanging on every word and we all erupted in applause at the end of the talk.  His passion for his work was evident and this translated into a surprisingly interesting talk on a low level-testing tool.

Keynote – Escape from the Ivory Tower – Simon Peyton-Jones

You could be forgiven for calling Simon Peyton-Jones  “Monty” and John Hughes’ “Python”.

Both went to university together at Cambridge and have the same passion, humour and comic delivery which makes for an excellent presentation. This was a wonderful follow on from the John Hughes talk.

This talk covered the history of Haskell, a functional programming language which has largely been academic until recently.

With the rise of parallel programming Haskell has attracted far more interest and suddenly Simon as a co-creator of Haskell is very much in demand as a speaker.

The speech centered around the history of Haskell and the differences between Haskell and other functional programming languages.  For non-functional programmers Simon recommended some exposure to Haskell to help them think about programming in a different way.

Again the speaker was very passionate about his topic and his presentation was so well put together that the audience was very much engaged.

JavaScript Everywhere

The next few talks that I attended painted a better picture for me on the rise and rise of JavaScript.

Web As A Platform – Dart, a new programming language for structured web programming  – Kasper Lund

This talk introduced Dart, a new programming language (almost) released by Google, still very much in its formative years.

Through coding up examples, the presenter highlighted a number of the language features mainly comparing them to JavaScript.   I took it that Dart was positioning itself to rival JavaScript. Dart can either run on it’s own VM or can be compiled down to JavaScript… which leads me into the next talk.

Web As A Platform – The Web Browser is a Transitional Technology – Allen Wirfs-Brock

This talk was based on the premise that we are moving into a new era of computing. We went from corporate computing to personal computing and now into what Mozilla’s Allen Wirfs-Brock labeled the Ambient Computing era.

Ambient is defined as “completely enveloping” so I take it he was trying to capture the fact the now people have multiple devices which they want and use all the time, therefore we are enveloped in the technology.  Bit of a stretch for me I’m afraid… I don’t think that phrase is going to stick.

In any case, he made good points that the new gadgets entering the market do not typically use a browser any more to interact with the web.  However a lot of them still use the same technology stack just without the browser chrome.  This web platform can sit directly on new hardware losing the PC and the browser.  To relate back to the title of the speech, the browser was a PC based, transitional technology to get us to this point.

The biggest point I believe Allen made was that JavaScript was directly built into this technology stack and was the only language present.  By this fact alone it foretells a bright future for JavaScript as it would be very difficult for an emerging technology (such as Dart) to proliferate to the point that all vendors across the Internet and gadget market will replace JavaScript in this platform.

I guess that’s why the Dart guys have ensured Dart can compile down to JavaScript….

Web As A Platform – NodeJS and the JavaScript-Everywhere Strategy – Matthew Eernisse

Yammer’s Matthew Eernisse talked about JavaScript on the server side and the rise of NodeJS.  In the talk Eernisse wrote a web server from scratch to show how quickly you can get off and running with NodeJS.  He discussed some of the tools available to help with NodeJS development and highlighted some of the trouble shooting lessons he had learned.  With so many callbacks and asynchronous code it can be difficult finding root causes of issues.  Eernisse showed how he changed his debug to help address those issues.

The majority of the Yammer upload apps are NodeJS applications and Shine Technologies has also done some major project work with a Melbourne client on various NodeJS apps.

This talk complemented the previous ones on highlighting the rise and rise of JavaScript to me.

The only other talk I should mention was the Keynote by Damien Conway on extending Perl to implement quantum physics. This was genius in PowerPoint but while I enjoyed it and followed it to a degree, I could not hope to summarise it due to its complexity.  Presentations like Conway’s go a long way to inspiring people in the field to take their profession far beyond what anyone thought was possible.  Sensational stuff!

Summary

There were some other interesting talks, which rounded out the conference plus a good chance to meet other developers and people in the industry as well as the speakers themselves who were very approachable.

This was a very well run conference and I will be lining up next year again for it.

Alistair Sutton is a Senior Consultant for Shine Technologies and has recently moved to Brisbane to open a Brisbane branch for Shine.

Synchronising advanced data structures with a backend sucks. Especially when the sync is both ways; user updates to server, server updates to user. Couchbase for Mobile attempts to solve this issue by bringing CouchDb and it’s spiffy replication to Android.

What? Why?

Why would you ever want CouchDb on your phone, you ask. Well, along with giving you some fancy pants replication, which I’ll rant about soon, some of the benefits over Android’s built in SQLite support are:

• Unstructured data storage in JSON documents
• Map/reduce for querying
• Versioning of documents
• And more..

In a bit more detail ..

JSON documents with whatever you want in them

The data in CouchDb is document oriented, in that it stores and returns your content in JSON documents. These documents can have whatever fields and whatever JSON acceptable values you wish without any table changes. This gives you the power to support dynamically added fields. If you wanted to create a form that accepts additional user defined fields, CouchDb or something like it is what you want.

A CouchDb document, requested with a GET to /database/boo, is read and written something like

{
  "_id": "booboo",
  "_rev": "gibberish",
  "favouriteFoods": [ "Bangers", "Mash" ],
  "bodyWeight": 300
}

While a horribly non standard representation of a SQL table structure, that allows dynamically added fields, would be something like

Form#1()
FormValue#1(
  formId: 1,
  fieldName: "favouriteFoods",
  fieldValue: "Bangers",
  fieldType: "String"
)
FormValue#2(
  formId: 1,
  fieldName: "favouriteFoods",
  fieldValue: "Mash",
  fieldType: "String"
)
FormValue#3(
  formId: 1,
  fieldName: "bodyWeight",
  fieldValue: "300",
  fieldType: "Number"
)

Retrieved with queries like

SELECT * FROM Form
SELECT * FROM FormValue WHERE formId = 1

As you can see the CouchDb approach is much nicer, and in SQLite’s case the structure of Form and FormValue need to be defined when the table is created. Meaning updates to the structure of these tables involve fancy scripts or complicated SQLiteOpenHelper.onUpgrade functions.

Map/reduce functions

CouchDb has a different method of querying the data compared to normal SQL databases, depending on your usage this could be a deal maker or a deal breaker.

In CouchDB you create views of your data, written in Javascript, emitting a key and a value. You then look up the data you want using the key you specified. You can also reduce the values for summing or other fun things. For example:

"byName": {
  "map": function (doc) {
    if (doc.name) {
      emit(doc.name, 1);
    }
  },
  "reduce": function (key, values, rereduce) {
    return sum(values);
  }
}

When this has been indexed you will have an index like:

"Jim", 1
"Jim", 1
"Jack", 1

Calling it with reduce will give you:

"Jim", 2
"Jack", 1

Versioning of documents

CouchDb versions documents. Every time you update a document the previous revision will still exist and be accessible (until compaction which I’ll ignore), which may be useful for basic undo or rollback functionality.

Everything CouchDb gives you

And all of the other things CouchDb provides, like hugs, kittens and obese indexes (don’t quote me).

Replication

The main benefit you’ll see with using CouchDb for your application (presumably) is data synchronisation via replication.

CouchDb supports really easy start and stop replication, with it’s own conflict resolution. You could have multiple databases that haven’t seen each other in months, tell them about each other and each of them will eventually catch up with the other as far as documents go. This is awesome for “when I want” syncing, and offline editing of data that syncs when a connection is established again. Some use cases:

Taking notes

This would be great for a note taking application that supports mobile as well as web edits. The user could add notes on both sides and sync to make both sides consistent when they want / when a wireless connection is available.

Real time when available

You could have replication going any time the application is in the foreground. This would allow for applications that work in real time but don’t explode when the network is lost.

And I’m sure other more exciting things my boring brain chose to shield me from.

I’m sold. How do I make the magic happen?

First up, make an Android 2.x project in your favourite Java IDE (Eclipse), which you have obviously already set up for Android development.

Next you need to get the latest version of CouchDb for Android (still in Developer preview), here. This website also covers the instructions for getting your project CouchDb friendly, basically it’s “run an Ant script and hope it’s not doing anything too horrible”. Without going too into it, it’s configuring your project for CouchDb, Erlang (which CouchDb uses behind the scenes) and some funky native libraries.

You can also add Ektorp, a Java API for CouchDb with Android support. This wraps the CouchDb calls so you don’t have to generate your own HTTP requests. That’s really up to you, but the code examples below use this, and Couchbase recommends it as far as Android CouchDb libraries go.

I’m listening..

If you’re anything like me, you’ve probably skipped to this section or closed the page when you didn’t instantly see code, but here goes.

Starting CouchDb

First we’ll set up a CouchbaseDelegate in our main Activity. You use this to capture startup errors and kick off initialisation steps / replication when the database has started.

ICouchbaseDelegate couchDelegate = new ICouchbaseDelegate() {
  @Override
  public void exit(String error) {
    // Show the error to the user
  }

  @Override
  public void couchbaseStarted(String host, int port) {
    // CouchDb has started, set up your connector
    // in our case initialise Ektorp
    startEktorp(host, port);
  }
};

Now we’ll actually request for CouchDb to be started, referencing the delegate we created above. In my case I’m calling this when the activity starts.

private void startCouch() {
  CouchbaseMobile couch = new CouchbaseMobile(
    getBaseContext(), couchDelegate
  );
  couchConnection = couch.startCouchbase();
}

CouchbaseMobile will do its magic and eventually call your delegate.

Connecting to the database

Now that CouchDb has started, we’ll connect using the host and port the delegate gave us.

clientBuilder = new AndroidHttpClient.Builder();
httpClient = (AndroidHttpClient)clientBuilder
  .host(host)
  .port(port)
  .build();
dbInstance = new StdCouchDbInstance(httpClient);
// The last parameter true will create the database
// if it doesn't exist.
dbConnector = dbInstance.createConnector(DATABASE_NAME, true);

The dbConnector is now ready for normal database use. Let the CRUD begin.

Creating the initial design documents

CouchDb is pretty useless without a view over your data (unless you know ids or want to just work with all documents). So after CouchDb has started you should create (or update) the design documents specifying your views.

designDocId = "_design/slyncy";
byNameView = "byName";
byNameMap = "function(doc) {if (doc.name) emit(doc.name, doc);}";
DesignDocument dDoc = new DesignDocument(designDocId);
dDoc.addView(byNameView, new DesignDocument.View(
  byNameMap
));
dbConnector.create(dDoc);

Putting a list adapter on top of the view

Assuming you’re displaying the data in a list, you’d want to wrap a view in a custom CouchbaseViewListAdapter. There’s a horrible implementation in the example code showing basic view mapping.

ViewQuery query = new ViewQuery()
  .designDocId(designDocId)
  .viewName(byNameViewName)
  .descending(true);

// The third parameter specifies that you should listen to
// the _changes feed from CouchDb enabling this means the
// list will automatically update for you when data changes
adapter = new CouchbaseViewListAdapterImpl(
  dbConnector, query, true
);

The final parameter (_changes: true) is pretty powerful. It works like Android content provider data change notifications, allowing the list to update when the data changes. Delete a document and it will be removed from the list.

Something Ektorp is missing with their base CouchbaseViewListAdapter class is support for include_docs when querying the view. This means your document will have to be emitted as an indexed value, which could lead to obese indexes if your data set is big enough.

Starting replication

Starting replication is pretty simple. You build a ReplicationCommand, specifying the source, target and continuous (once until complete (false), or continually run until shut down (true)).

pushReplicationCommand = new ReplicationCommand.Builder()
  .source(DATABASE_NAME)
  .target(syncUrl)
  .continuous(true)
  .build();

EktorpAsyncTask pushReplication = new EktorpAsyncTask() {
  @Override
  protected void doInBackground() {
    dbInstance.replicate(pushReplicationCommand);
  }
};
pushReplication.execute();

Switch the source and target around for pull replication. Depending on how your system is you would probably want both (user updates -> server, server updates -> user).

CRUD

After the connection is set up, CRUD is simple. You just call create/update on a document (where document is a Map in my case), or delete with a given id and revision.

dbConnector.create(document);
dbConnector.get(Map.class, id);
dbConnector.update(document);
dbConnector.delete(id, revision);

Summing up

So, as you can see, connecting to CouchDb on Android is fairly simple, if a little verbose, but that’s the Java way.

You can find a working (you know, sometimes) example application here, based on AndroidGrocerySync.

Concerns

Data Security

Sure, replication sounds dandy, but how do I stop the user replicating changes of documents they don’t own?

CouchDb has a method for this called Validation Functions. Basically you can define a Javascript function that will look at an update and allow you to reject the change. The last parameter of these functions is a user context. So using this you can work out who’s writing and if their write is valid. For example:

function (newDoc, savedDoc, userCtx) {
  if (savedDoc) {
    // This is an update, make sure the user
    // isn't watching the world burn.
    if (newDoc.userName !== userCtx.name) {
      throw({ unauthorised: 'You shall not pass ' + userCtx.name });
    }
  }
}

Internal phone security

A completely uninformed concern (I’m sure Couchbase has thought of it) is security from other applications running on the phone. What if you have private data you don’t want Mr Naughty’s application to push to his cloud of deceit?

Completely based on a Stack Overflow thread here, it seems that connecting to Couch and access the database itself is secure enough, but the data is written to external storage, which isn’t secure at all.

Conclusion

I’ll sum it up in pros and cons form.

Pros:

• CouchDb on Android makes replication easy. It’s basically allout of the box, just point and say go
• The JSON strucure of CouchDb documents gives your application a lot of flexibility and a nice way to store data.
• The actual code behind it isn’t too painful or that far away from ‘typical Android’ code.

Cons:

• Starting up CouchDb is slow, from what I’ve heard mostly due to loading an Erlang interpeter. You’ll see the lag between starting the application and initialising CouchDb if you run the example application
• Map/Reduce while providing some friendliness also removes the possibility of on the fly SQL style queries
• Packaging CouchDb with your application adds about 10mb of bloat to your installation.

All up I’ll give it 3.33 out of 5 overly intelligent mice, ready to take over the world.

Links / More interesting reads:

• Example Github project: https://github.com/dbrain/slyncy
• A nice setup guide here

Note: Images randomly pilfered from Google Image Search.

Getting started with the development of location-based services on Android is relatively easy thanks to the well documented location API. However, getting more serious shows there is still much uncharted territory. One such area concerns the accuracy of real-life location data, which I have recently taken a close look at. In short, location data can be far more accurate than Google’s conservative estimates – at least with the phone that I used.

The Android developer documentation gives estimates on expected accuracy of the location data. This estimate varies according to the location provider used. Android distinguishes two location provider types with the following accuracy:

• Fine-grained location provider like the GPS provider with an accuracy of less than 100 m.
• Coarse-grained location provider like the network location provider with an accuracy of 100 – 500 m

One look at the Google Maps application makes it clear that the estimates for the accuracy are very conservative: in many cases the user location is shown on Google Maps with an accuracy of a few meters, much better than the estimates stated above.

My interest is in the individual accuracy of real-world data rather than an average accuracy estimate. Each time a location provider sends a location update, the accuracy of this individual location is given as well. In this article I look at this accuracy as reported by the location provider, and  compare the location accuracy of the two main location providers to actual locations.

Location Providers

Location providers are the sources of location data in Android. Even though the Android API specifies location providers in a very generic way, in practice there are two main location providers: The GPS location provider and the network location provider. Because they use different ways to obtain location data they differ in accuracy, minimum notification interval, time to first fix (TTFF), and battery consumption.

The way you receive location information in Android is to register a Location Listener with the Location Manager. At registration the location provider, notification interval and notification distance have to be specified:

locationManager.requestLocationUpdates(provider, notificationInterval, notificationDistance, locationListener);

The Location Manager  will call the onLocationChanged() method of the listener if the distance since the last location update for the registered location provider is greater then the notificationDistance, or the time since last location update is greater than the notificationInterval.

GPS Location Provider

The most obvious location provider is the GPS. GPS provides data with high accuracy: Less than 100 m is the official number, but real accuracy is often around 2 m – 20 m. Once it is connected to satellites it can provide location updates in intervals as short as a few seconds.

Sounds like an ideal location provider, but the GPS also has its draw-backs. For one, the initial connection to the satellites – Time to First Fix (TTFF) – can be quite slow. GPS traditionally needs to connect to at least three satellites for a 2D location, and initial transmission of so-called Ephemeris data takes a minimum of 30 seconds for a satellite. Once the Ephemeris data has been transmitted and connection to the satellite is made, location updates are very fast.

To speed up the TTFF, manufacturers have come up with a new way to deliver the initial Ephemeris data to the GPS: They use an Internet connection to deliver the data much faster than the transmission from the satellite. Android uses this so-called Assisted GPS (A-GPS) to speed up TTFF to 5-15 seconds, but only if you have an Internet connection.

Another draw-back of the GPS provider is that the battery consumption is quite high, and continued use of the the GPS will drain the battery. Even though GPS location updates can be received every few seconds, the Android developers guide recommends update intervals for the GPS of greater than 10 minutes. This limits the type of applications that can benefit from using the GPS.

Last but not least: Since GPS uses a radio signal, solid objects – especially those made from metal – will obstruct the signal. In practice, this means the GPS will not work in most buildings (GPS is not affected by clouds, rain, or snow, though).

Accuracy

Each location update from a location provider comes with the accuracy of the data in meters. This accuracy is given for each location update. To visualise the location data and the accuracy for each location I wrote an Android application that records the location data in a database. Drawing the recorded data on a Google Map helps visualise the data. The following map shows the data recorded for the GPS provider during a walk through Melbourne.

GPS provider data compared to actual location

The black line is the actual path I took during the walk. The white dots are the locations recorded by the GPS provider, while the red circles around the dots indicate the accuracy per location as reported by the GPS provider.

As expected the accuracy of the GPS provider is quite good: Most of the time the accuracy is within 30 m, some times as good as 2 m. That is much better than the conservative estimate of less than 100 m stated in the documentation.

An interesting observation is that sometimes the actual location is not within the circle of accuracy given by the GPS provider. The location marked with 1 on the map is about 25 m off the actual location, with an accuracy radius of about 5 m. Even with the radius of accuracy the result is still 20 m off the actual location.

Another observation that shows the draw-back of the GPS: if we are indoors or have otherwise bad reception there may not be a GPS signal, which means the location is completely lost. Marker 2 shows a section of the walk without any GPS signal. First I walked through a building from the Flight Centre to the cafe “The Journal”. Even after I passed though this building and walked down down the narrow Flinders Lane the GPS reception didn’t improve, so there are no readings at all during this time.

Network Location Provider

The network location provider uses both WiFi hotspots and cell towers known to the Android device to approximate a location of a user. When the location provider is polled, the IDs of the WiFi hotspots and cell towers in the area are sent via Internet to the Google Location Server, a database with location information on WiFi hotspots and cell towers. Google Location Server returns an approximate location of the user.

The data comes from a combination of cell and WiFi hotspot information, and can only be obtained when the device can access the Internet to query the Google Location Server. Accuracy of the data is specified somewhere between 100 m – 1000 m depending on the information available. WiFi hotspots allow an accuracy of 100 m – 500 m, while cell towers only allow for an accuracy greater 500 m. This means the network provider will be very inaccurate in areas without WiFi hotspots. Conversely, the more hotspots and cell towers are in the area the greater the accuracy.

The network provider can give first location results within seconds. This is much faster than the GPS location provider, which even using A-GPS may take up to a minute to give back first results.

Accuracy

The following map visualises data recorded for the network provider when walking through Melbourne.

Network provider data compared to actual location

The black line again is the actual path taken, while the white dots are the locations recorded by the network provider. The turquoise circles indicate the accuracy reported by the network provider. The walk took about 30 minutes, during that time 16 points where within a reasonable accuracy (I discarded those points with an accuracy of greater than 100 m as too inaccurate, just to make the map clearer).

The map shows that at least in an urban area the network provider gives enough data with an accuracy of less than 100 m. Most of them were actually in the area of 50 m. This is not as good as the GPS, but still quite impressive. The network provider also has number of advantages over the GPS: It works both indoors and outdoors, starts up quickly, and doesn’t drain the battery.

WiFi Hotspot Location Data

During recording of the data I stumbled across an interesting anomaly: The network location provider put me in a totally different location than where I actually was. Curiously this happened every time I connected to my WiFi hotspot at home.

Wrong WiFi Hotspot Location

The location marked with 1 is near my home, while the location marked with 2 is the place where I used to live. Once I arrived home the network provider placed me in location 2 instead of near location 1. Apparently Google thought I was in my old home once I logged into my home network, because it had previously registered the location of my WiFi hotspot in location 2.

This made me think about how Google actually obtains the location of a WiFi hotspot. It used to be possible to register the location of your WiFi hotspot with Google by hand. This option apparently does not exist anymore. As this discussion shows, the data for the network provider actually comes from data Android GPS providers sent to the Google Location Server at some other time:

“[...] before the GPS acquires a “fix”, Android will send the Wifi’s MAC/SSID and Cell Tower base stations CID/LAC around you to Google’s server and Google’s server will send back your location information; after the GPS acquires a “fix”, your device will send the Wifi MAC/SSID, Cell Tower CID/LAC, and your GPS location to Google’s server and Google’s server will collect this information to refine their own Wifi database.”

The discussion also shows an Email reply from a Google employee stating that updating location information on WiFi hotspots is done automatically when a GPS provider sends new location data, but “may require a significant amount of data from you and other users before changes are made to Google’s location database”.  That explains why my WiFi hotspot is still registered at my old home in the Google Location Server.

Conclusion

Comparing the location data received from the location providers with the actual location data shows that both location providers are much better in accuracy than the conservative estimates of the documentation.

For the hightest accuracy the GPS provider needs to be used, but is not suitable for all situations. The limitation of GPS to outdoor use is quite constraining, as is the battery consumption and TTFF, especially when used without A-GPS.

The results of network provider for me came as a surprise: It is quite accurate while doing without the draw-backs of GPS like power-consumption,  TTFF, and restriction to outdoor use. It does lead to large inaccuracies in areas without WiFi hotspots.

Clearly a combination of both provides will yield the best results. How they are mixed depends on the application. A foreground application with emphasis on accuracy would use the GPS provider as the primary provider, with the network provider as a backup, while a background application with emphasis on low power consumption would use the network provider as its primary location provider with the GPS as a backup. Varying the update intervals allows fine-tuning the providers to a specific application.

Code

If you want to have a look at the location data on your Android device I have uploaded the code to GitHub. It consists of an Android application that records the data as well as an HTML page that displays the data on Google Maps.

Of the many excellent sessions at this week’s Swipe Conference, the one titled “Build Better Cocoa Apps Using Game Mechanics” by Paris Buttfield-Addison was unique in its topic. In it, Paris outlined how gamification is currently viewed as something that can be quickly slapped onto an existing product in order to increase its appeal, whereas in reality, successful gamification requires carefully thought-out integration, for when done correctly it should form a core part of your app’s user experience.

In case you’re new to the concept, gamification is the application of game-like mechanics to non-game products in an effort to increase engagement with the user, motivating them to spend more time or money on your product.

The current trend in gamification is rather worrying, with seemingly everyone adding it to the list of things their product needs – as though it were a requirement for being successful. The reality of the situation is that these ‘gamification grafts’ do nothing to enhance the user experience at all, providing mainly virtual badges for completing rather menial tasks – for example, pushing a button. This lack of challenge contributes to a failure to keep a user engaged, for they quickly become bored and move on.

A better approach to gamification would be to find a way to enhance the core functionality through the use of game-like mechanics. An example of successful gamification is Stack Overflow. The core functionality is to provide people with a place to both ask and answer questions. However, game play elements come into it by rewarding users with a reputation system; the more you participate on the site, the more your reputation increases. Your reputation is then an indication of how helpful you are in the community. With higher reputation comes privileges that extend to helping with moderation of the site. This reputation system sets itself up in a unique way so that it encourages people to actively take part in the community and rewards them in the process.

Perhaps the most important point that was made during Paris’ session was that not all products are suited to gamification. An app that displays the weather has no use for leader boards or an elaborate point system – they bring nothing meaningful to the user experience. However something like Foursquare, whose core functionality is to provide you with a means of sharing your location with others, benefits greatly from the addition of game play elements, effectively turning an otherwise dull concept into an engaging experience that rewards and encourages exploration of your surroundings.

In conclusion, the concept of gamification is definitely here to stay, but as developers and designers, we need to be vigilant and not give in to the urge to simply slap on a half-baked implementation. Instead we need to take the extra time to evaluate if it will actually enhance the existing user experience.

We’ve all heard developers say it: “I’m a terrible drawer” or “I’ve got no design skills”. Perhaps we’re even guilty of saying it ourselves – I know I am. But after attending this year’s Swipe Conference I now subscribe to the opinion that this is no longer acceptable. We are all responsible for the design of the app we are building; developer, designer, tester, or producer: every member of the team is accountable for helping shape the app’s design and interactions.

For far too long this has represented how teams have collaborated.

Why must we all be accountable?

We – as developers – are iPhone experts. We know what can and can’t be done technically. We have toyed with and tried building (with both successes and failures) many ideas before. But before that, we are users; in fact, not just users but experienced super users. Sure we know how to code, but this is really just a small part of creating an app with an amazing user experience.

But I can’t draw

During the Swipe Conference we had a 30-minute session that addressed this exact problem. But the fact is, that it’s not a problem at all. When talking about working together to help build an intuitive interactive app, I’m not talking about creating the entire visual representation of your app. I’m talking about presenting ideas, and thinking through them as a team.

This 30-minute session taught us such a simple lesson, how to use a pen, how to draw a basic shape, a square, a circle, and how to colour them in. This is all you need to do, in order to present your ideas to the team. Sketch some simple shapes that represent the objects in your app, shade the background to bring the elements to the front, and colour them in. It’s simpler than a picture you would have drawn in kindergarten. “I can’t draw” is not an excuse!

So I’ve drawn my picture, now what?

At the end of this session, we were all encouraged to hand in the sketches of the iPhone interfaces that we had created. The organisers then took these sketches and stuck them up on the wall. And guess what, with just a black pen, a grey texta, and an orange texta (the swipe conference colour), we had hundreds of ideas that both conveyed an idea and were immediately understandable.

Just as we were able to do at Swipe Conference, everyone in an iPhone team can and must spend time doing the same thing. Pin your ideas on a wall, tear them down, change them, evolve them, give feedback, and work together to evolve from an infinite number of ideas from each team member into one highly refined and fantastic experience.

Brilliant apps with intuitive interactions

This is how we need to think about working

By having a team where everyone can provide input, we help to create an experience that can only be beneficial. I know from some of the projects I’ve worked on, that it can be increasingly frustrating when you’re involved building the inner workings of the app, but cannot influence its overall direction. With such a great device to work on, and the potential it provides to user and developers alike, it would be a gross injustice to continue working in a counter productive way.

Let the designers turn your ideas into beautiful pixel-perfect designs, and let the developers transform these designs into interactive working interfaces – but also work together with everyone to allow your app to become a masterful interactive experience.

Yesterday was the last day of Swipe Conference so I thought I would take this time to reiterate one of the points I took from the first presentation by Josh Clark.  Josh covered quite a few topics and if you haven’t already you should check out his book Tapworthy: Designing Great iPhone Apps.

In short:

• Gestures can be brilliant … if the context they are used in feels natural
• If you’re using gestures make sure your users will find them
• If you don’t think your users will figure out your gestures easily, don’t overload your users with lots of help hints all at once; instead, let them “unlock” them over time, like a reward for using your app
• Downside though is that there is no consensus on what a 3 finger swipe gesture might do. Every iOS app that uses this gestures decides it for themselves.

It was obvious from the way Josh presents that he has so much passion for touch / gesture based devices. What I personally took away from his presentation was that gestures can be awesome shortcuts within your app. In a lot of cases gestures are a natural progression with how we interact with real world items.

So let’s drill in a little more to some of these points.

Gestures are brilliant… if used in the correct context

How cool are gestures?! As a developer I have the freedom to implement any gesture I want, to perform any actions I desire. But it’s not really that simple – you need to consider if the gesture you’re using makes sense for the functionally you’re tying it to. For example, do you believe your users will associate a cool 4 finger pinch gesture in your app with bringing up a feedback email form?

Bad example, I know – because of course you would say: thats just stupid; why would you do that?! But these are exactly the kind of questions we need to ask ourselves – not only as developers, but as UX designers.

How will users learn your gestures?

If you want to add a gesture to your app because it’s a cool shortcut, how are you going to let your users know those gestures exist within your app?

Take a look at Apple for example. Their iBook app allows a user to turn a page by swiping from right to left; and yet in Contacts app, performing this action will most likely prompt you to delete one of your contacts. This is just confusing to a user.

I highly recommend that you watch Cathy Shive‘s presentation (when the Swipe Conference presentations are released – I will update this post when it happens). Cathy highlighted a lot of great reasons why it’s so important to get your metaphors right. If it looks like a book and doesn’t function at all like a book, then why are you making your app look like a book?

If the gesture you’re adding into your app is not something a user will instinctively know to try out, then you need to give them some hints to be able to find this shortcut.

There are a number ways you can do this:

• Possibly draw attention to the part of the app that has this gesture – maybe it could pulse for the first time to signify it is interactive
• Perform an animation that hints to the type of gesture that may be used to replay something
• If your app has quite a few shortcuts maybe only provide one hint each time the user starts up your app

Don’t forget to give your users a chance to explore your app. Users love to explore. They may stumble upon your gestures themselves.

Don’t overload your new users with “helpful” hint messages

I really liked the example Josh highlighted, which was if you were to be given an in-depth explanation before you were to read a book, you would instantly feel like using a book was quite complicated. For example: you turn the page from left to right, you read from left to right (depending on culture) and when you reach the end of a line you jump down to the next line and continue; to bookmark you can fold over a page and return to it in the future; when done, simply close the book (there’s a funny explanatory diagram along similar lines that you can find here).

The same goes for an iOS app. Feel free to explore gestures; make use of them as shortcuts within your app, but if you need to explain to a user how to use your app, then don’t just throw up a big old welcome message that tells a user every gesture they can perform – I guarantee you they will glaze over it. If it looks and feels complicated, then you’re just putting off your users.

In short, ease your users into finding these hidden gems within your app.

I hope swipeconf runs again next year!

Swipe Conference was great! I had a great time and all the presenters were excellent. It was cool to see so many iOS developers / UX designers out there who are really trying to build the best iOS apps they can.

Shine was recently involved with helping a client bring an outsourced mobile web site in-house. The site was essentially a guide for browsing business and event information for cinemas, restaurants and bars in your area. Bringing this web site in-house was mainly an exercise in re-writing the outsourced instance as a component to the client’s own internally developed NodeJS/Express app server (currently used to serve their main – non-mobile – web site).

The NetBiscuits platform was used to handle all the device specific porting which allowed the development effort to focus almost entirely on business logic and backend data processing. The combined learning curve of NetBiscuit’s own markup language with Shine’s Jazz template system was simple enough to make the presentation layer development a breeze. This blog will talk about the process involved in developing for NetBiscuits and some of the problems and solutions encountered on the way.

So what is NetBiscuits?
NetBiscuits is a hosted platform technology that enables web apps to reach mobile devices. It’s essentially a knowledge base that maps mobile devices and their browsers to a presentation profile, translating your mobile site into the format expected by whatever mobile client that is making the request.

It’s worth noting that the concept of porting web content to mobile devices and their browsers is big. Seriously. The impact of standardized mobile web development is perhaps lost on those of us who’ve never worked in it before. Where common web development concerns itself with compliance across 4 or 5 main browsers, mobile web development must cater to hundreds. Even in the days when Nokia ruled the world the support and maintenance of mobile web apps for an ever-growing device list was not just for the final phase of a project but an ongoing full time job.

Maximizing your audience still means supporting the lowest denominator, but these days you can expect your low-end browsers to run javascript. This old school WAP and J2ME developer acknowledges that times have moved on but pays due credit to the people behind NetBiscuits and their mobile platform.

Working with NetBiscuits
You develop your mobile web app in mostly the same way as any other web app with the difference that instead of dynamically generating HTML you’re generating BiscuitML. Your production URL will point to your NetBiscuits account (hosted on their cloud) which in turn points to your app server.

For as long as your app server talks to the NetBiscuits platform in BML, NetBiscuits will talk fluently to all devices that hit your mobile site. This means your development efforts can focus purely on the business logic without the concern of tailoring content to multiple screen sizes and browser types.

So what can you do with BML? Pretty much everything you can do with HTML. For most of the devices hitting your mobile site NetBiscuits will convert your BML to HTML. Writing markup in BML is done with Biscuit tags that comprise of familiar page elements such as; list, table, form, image etc. More specialised tags are available to handle ad-banners, analytics, maps and media. A script tag exists to provide various levels of javascript support ranging from handling browser events completely within the markup to simply relying on an external js file.

Challenges with Session Handling
Environmental constraints meant that client side session management was the only option available. This in turn meant that support for cookies and javascript was a requirement for any browser hitting the site. Cookies would store the user’s state code (see below) and the preferred list of cinema venues for returning movie times.

<TEXT>
    <richtext>
        Select your state:[br]
    </richtext>
    ...
    <richtext>
        <event type="onclick">
            <!-- Custom set cookie function -->
            <action eval="setStateCodeCookie('VIC');" />
            <action eval="return true;" />
        </event>
        [url="path/to/main/page/?{URLParams}"]VIC[/url]
    </richtext>
    ...
</TEXT>

Maintaining sessions within cookies is certainly made easier in the Tomcat/JSP model where the HTTPSession interface provides easy access to user data. In such a scenario, session data are largely managed by the server and a user is identified by a cookie-stored session id. In our instance we ended up pulling session data directly from cookies after discovering some limitations with BML.

Amidst the wealth of documentation for BML and its rich internet features there were examples of how to get/set/delete cookies from script tags. This led us to the false belief that a cookie could be interrogated and manipulated fully from within the markup. Wrong. BML’s getCookie function exists only to test the presence of a cookie, not to read/extract its data. Cookie handling was therefore relegated to an external js file which also meant taking control away from the markup for certain browser events. Specifically, the onclick event had to be hijacked from certain anchor tags to implement redirects for re-written URLs.

<TEXT>
    <richtext>
        Find Movies:[br]
    </richtext>
    ...
    <richtext>
        <event type="onclick">
            <!-- Override default behaviour and use
                 an externally defined custom handler
                 to perform redirect -->
            <action eval="getPreferredCinemasForMovieSearch('{URLParams}');" />
            <action eval="return false;" />
        </event>
        [url="#"]Browse by Cinemas[/url]
    </richtext>
    ...
</TEXT>

Hijacking the Onclick Event
Retrieving a user’s cinema preferences (above) is one example of URL re-writing trickery. Retrieving their state code is another. A state code would populate all URLs presented to the user. New users would have their state code persisted in URL parameters for the duration of their session. Returning users would need their session initialised by extracting their state code from the cookie.

Since cookie data wasn’t available at BML generation time, extraction would have to happen at URL click time. So this meant triggering a javascript function on the browser using the anchor tag’s onclick event.

Performing cookie data extraction at ‘click time’ effectively turns the handler function into an event-swallowing black hole, since it must also perform the page redirect leaving no control to pass back to the browser and it’s anchor tags. A consequence is that any URL parameters normally processed within the markup must now be passed to the javascript function for re-writing to the URL.

Ideals, Alternatives and Trade Offs
Relying on the BML to handle all things cookie/session related would have been the most ideal solution. URLs could have been re-written at BML generation time using cookie data without any javascript intervention. With the benefit of hindsight, the next best alternative would have been to utilize the onload event instead of hijacking the onclick event. This is because the onload event allows for the markup to retain control of its anchors – since re-writing happens well before URL click time.

This works fine for initialising a returning user’s state code since ALL URLs are modified with this data in the same way. However in situations where we needed to be more selective (extracting a user’s cinema preferences for only certain URLs), additional effort is required to identify/isolate the subset of URLs to operate on. Not all URLs need specific preference data appended to them. So the trade off is down to the assembly and redirect of URLs for a pre-determined anchor using the onclick method versus isolating a target set of anchors and appending data to an existing URL using the onload method.

<page>
    <script>
        <event type=”onload”>
            <!-- Append cookie data to a
                 target set of anchor tags -->
            <action eval="appendCookieDataToTargetURLs('{cookieName}',
                                                       '{targetId}',
                                                       '{URLParams}');" />
        </event>
    </script>
...
</page>

Conclusion
On the one hand it makes sense that BML can only provide superficial support for cookies given that its markup is never processed directly by the browser. On the other hand, if the NetBiscuits platform is communicating with client browsers and processing the BML on our behalf, could it not set its own session cookie and expose it through a set of functions within the markup? Where is NetBiscuit’s HTTPSession?! Perhaps the logistics of supporting potentially thousands of such sessions makes this concept infeasible.

To summarise, there may be further options available for enabling a mobile website that warrant a comparison but our experience with NetBiscuits was a happy and successful one. Cookie handling aside, the presentation layer development for our mobile app was super smooth. We also made use of a couple of the specialised tags for analytics and maps with relative ease.

The absence of full cookie support just means a little extra work. Perhaps we’ll see this built out in a future implementation of BML? In the meantime, its just old-school javascript know-how to enable cookie based sessions for your web apps. Of course, all this needs to be put in perspective with the main advantage provided by NetBiscuits: standardised mobile web development! Easy to overlook this significance when granted in such an accessible and ready-to-use format.

Cliffano Subagio from Shine Technologies presented together with Carl Husselbee from Sensis “JavaScript from Nose to Tail” at the August MelbJS Meetup. The slides and the video of the presentation are now online.

There have been many good examples of using Hudson for cross-platform builds and automatic execution of tests, but Hudson also provides a great environment for empowering non-developers to execute particular tests whenever they want. We have found this to be particularly the case when automating data migration tests.

This article will discuss the what automatic data migration testing is, and how Hudson can make it easier. Whilst it refers to Hudson, the same techniques could also be used with Jenkins.

About automated data migration testing

Data migration testing is usually quite complicated and time-consuming. This is because you need to make sure all the migrated records are checked for accuracy from the source database to the destination database. It not only require testers to identify how the migrated records are mapped from the source to the destination, but also to perform checking on the values of the records.

The complexity of data migration testing increases when the testing needs to be done on huge amount of data and the mapping from the source to the the destination is quite complicated.

In order to reduce the complexity of data migration testing and achieve better testing performance, it becomes necessary to apply automatic verification. In some cases manual testing can be completely superseded by automatic testing.

A simple example

The following diagram illustrates a simple method for automating data migration testing that we have taken in the past:

We see that testers only need to provide test inputs, each of which specifies:

• the table(s) that the data is being sourced from
• the table(s) that the data is being sent to
• how the migrated data is mapped from the source to the destination

Inputs can be things like custom JMeter Test Scripts or Excel Spreadsheets.

Once the test inputs have been prepared, an automatic data migration testing program can load them and perform checking on the data from the source database to the target database.

After the testing is completed, the program will generate the test reports which can be either XML or HTML format.

Limitations of a stand-alone approach

Whilst it provides a good baseline to work from, the aforementioned approach is not particular user-friendly:

1. Test programs are usually executed from the command-line, meaning each test execution needs to be triggered on a separate terminal. Therefore, multiple terminals will be required to execute a number of different tests if test output for each test execution needs to be monitored and kept.
2. Testers need to tell the program which test is to be executed, and the environment that the test is to be executed against. The execution can become quite complex when there are a lot potential tests to choose from, or a number of different environments.
3. During the process of data migration, there can be frequent changes to data mapping rules, which requires testers to frequently edit their existing test inputs accordingly, or to add new test inputs

Fortunately, there is a tool that helps with some of these problems: the Hudson continuous integration server.

How Hudson helps

By using Hudson to run tests, users get an out-of-the-box interface to trigger tests, monitor test progress, and track test results. Specifically:

1. Testers do not have to execute tests from the command-line
2. Testers can specify parameters and easily pass them to test programs; for example: the test input to be executed and the environment that the test is to be executed against
3. Multiple test executions can be triggered simultaneously
4. Test reports and console output are well maintained
5. Test inputs can be managed and fetched from a central repository
6. Specific descriptions can be generated for each test execution, which provides better maintenance and tracking
7. Each tester may log onto Hudson with a specific user name and password, which helps identify who ran each test

Setting it up

So how does one go about setting up such a project? Firstly, a Hudson project needs to be created. The project can be created by simply clicking on “New Job” on the Hudson homepage, and provide the project name and project type:

Click “OK” button to save the project, and the project will be shown on the Hudson homepage:

We can then click through to the project, and click the ‘Configure’ link. In order to allow testers to specify the test to be run and the database environment the test is to be run against, the project needs to have two input parameters defined for it:

Now let’s specify what the build actually executes. Normally, this would be a complex Ant script. For the purposes of simplicity, we’ll just specify a dummy shell script:

In this case, the script will:

1. Check if the specified input test file exists. If the input test file does not exist, then it will exit the current build job gracefully; otherwise it will proceed the build job.
2. Check if the test reports have been generated successfully for the current build job.
3. At the end of each build job, a custom build description will be set which will clearly tell which test was executed on which test environment – for example “Execution of test_input1.xlsx on Source Database 2″.

Finally, as test reports need to be kept for for tracking purposes, we define them as build artifacts:

Taking it for a spin

To run this build:

1. Click ‘Build Now’ for the ‘Automatic Data Migration Testing’ project
2. Select a specific test input to run:
3. Select the test environment to execute the test input against:
4. Click ‘Build’ to trigger the build to start executing the test input.
5. Track test execution progress by accessing the build console output:

Once the build job is completed, testers can:

• Check the test reports by accessing the build artifacts:
• Use the custom build description to keep track of what this build did:
  In this case, the test input file is “test_input1.xlsx” and the test environment is “Source Database 3″; therefore the build description will be set as “Execution of test_input1.xlsx on Source Database 3″.
• Review the build console output

Conclusion

We’ve seen how Hudson is a great environment for empowering testers to run their own data migration tests. It does this by:

• Managing test inputs and environments
• Executing single or multiple tests
• Tracking test executions
• Storing test reports

It’s a tribute to the original Hudson developers that it can be used in new and interesting ways.