Java Code Trips & Tips

Adding a new sudoer in Ubuntu

2012-01-18T09:35:00.000-08:00

This is something I have to do from time to time, but not frequently enough that I manage to memorize it - but trivial enough that it annoys me to have to look it up every time.

Here's to future memory:

create a new user `bob`
sudo adduser bob
check that the `admin` group is the one for the 'sudoers' on the machine
sudo cat /etc/sudoers
add Bob to the admin group
sudo addgroup bob admin

done!

Disabling SSH root access

Incidentally, always worth disabling SSH root access for the machine: it's the one account hackers know for a fact it must be there, and the one they'll try to crack - the additional positive side-effect is that, at the first attempt to ssh as root, they'll be left hanging waiting for a response from sshd, which is nice.

Unless you have already done so, you must first create your own personal account and add yourself to the sudoers, as shown above - then try that this all Works As Intended, by logging out from ssh, logging back in as `bob` and executing a harmless, but root-only-allowed command:

ssh -l bob server

bob@server:~$ whoami

bob

bob@server:~$ sudo touch /etc/blah

Password: <------ NOTE: this is Bob's password, NOT root's

bob@server:~$ ls -l /etc/blah

-rw-r--r-- 1 root root 0 2011-11-12 13:28 /etc/blah <-- NOTE `root` owner

bob@server:~$ sudo rm /etc/blah

bob@server:~$ ls -l /etc/blah

ls: cannot access /etc/blah: No such file or directory

bob@server:~$ groups

bob admin

Now that we know we can manage our instance from the safety of a non-root account, we can disable SSH access to root: head straight for the sshd (SSH daemon) configuration file and find the line below, change it to "no":

bob@server:~$ sudo vim /etc/ssh/sshd_config

PermitRootLogin no

save the file and you're done.

Try it out: logout, try ssh root@server and you will see it just hangs there - enjoy that warm fuzzy feeling induced by the knowledge that the hacker's machine will be just equally sitting there, like a clueless idiot, just waiting for a response that'll never come.

Sugarcoating: passwordless access & short-name resolving

Your server may have a name such as bob.remote.server.blah.foobar.com - typing this every time you want to ssh into it may grow pretty thin after a few times: just add a line to your /etc/hosts file on your box (remember, you have to do this as root):

$ sudo vim /etc/hosts

##

# Host Database

#

# localhost is used to configure the loopback interface

# when the system is booting. Do not change this entry.

##

127.0.0.1 localhost

255.255.255.255 broadcasthost

::1 localhost

fe80::1%lo0 localhost

# This is the line to add, if the IP is assigned statically:
110.22.8.111 bob.remote.server.blah.foobar.com bob.remote

This way, you can just log into your remote host using:

ssh -l bob bob.remote

However, you still will be asked for your password every time you want to SSH into that instance: to avoid this, you will need to create a private/public key pair, and upload the public key to your RS instance's authorized_keys file (more details here).

On your box:

$ ssh-keygen -t rsa

DO NOT ENTER a passphrase, but protect the private key:

$ chmod 600 .ssh/id_rsa

$ ls ~/.ssh

total 32

drwx------ 5 marco staff 170B Oct 26 13:26 .

drwxr-xr-x+ 39 marco staff 1.3K Nov 11 16:30 ..

-rw------- 1 marco staff 1.6K Oct 26 13:24 id_rsa

-rw-------@ 1 marco staff 401B Oct 26 13:24 id_rsa.pub

-rw-r--r-- 1 marco staff 4.4K Nov 10 15:55 known_hosts

$ cat ~/.ssh/id_rsa.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVJ1+T1erKLiZKxsc5g9qEoeSS5T+3JSYS8NgCEACXxDWOUSw92SkFI32YsKcL7f8MFUHQhgyT8eMY8v2Eaz91/3l3IeGPOHmUgTLvoNaWNOmEC95Jj7pmx+qOHrK/4DbS2WLiJ4PmQ5kPsV/lzlfzcvY2FfOa71/1bch5izXzDRTDMi0spe7S1wUMO+rsLPf59jKiQ66l8Jof+b4F5A1eMFBjENFOdZBD5jUnu9dcGNn00ykDLmGUq3y3L9w8AOwOe9Ig0dZlLw5URU7jW8yut7sJUHlUJEoZt0q0xpMfwDeUt63Blrf2sJgte4Tw2HSofAs1W1xpE+oyed6lXQHH marco@snaplogic.com

NOTE -- this is a PUBLIC key, I don't care if you post it on lulz.com :)

Copy this (Command-C)

$ ssh -l bob bob.remote

Password: <-- You WILL be asked for the password

[user@bob.eng.snaplogic.com:~]$ vim .ssh/authorized_keys # this may be empty, that's ok

Paste the key (Command-V) and save the file; if this creates the file, make sure it's only writeable by you

[user@bob.eng.snaplogic.com:~]$ chmod 600 .ssh/authorized_keys

[user@bob.eng.snaplogic.com:~]$ exit

$ ssh -l bob bob.remote

Welcome to Ubuntu 11.04 (GNU/Linux 2.6.35.4-rscloud x86_64)

* Documentation: https://help.ubuntu.com/

Last login: Sat Nov 12 16:21:51 2011 from 12.90.36.218

Look, ma, no password!

CDT Indexer & Google gtest framework

2011-09-13T00:03:00.000-07:00

As I mentioned in a previous post of mine, Eclipse's CDT (C++ Development Tools) has a few issues when used with Google C++ Unit Testing framework, but it generally works.

Where CDT really gets confused is around the 'exclusion' of the unit tests for the non-test build configurations: as mentioned in my other post, one needs to add the G-Test's include/ directory to include search path (-I).

Default Preferences

However, I recently found out that the indexer gets confused too, if one uses the default preferences in Window > Preferences > C/C++ > Indexer (see screenshot on the left): the critical item is the "Index unused headers" which leads to a bunch of syntax errors on the unit tests, due to the fact that the entire testing:: namespace is not indexed, when the Active Build Configuration is not the Test one.

Whilst this may (possibly) be a "feature," it annoys the hell out of me, nor did I find it particularly useful (as it's very difficult to tell the genuine errors from the artifacts).

I have found that by enabling the indexing of unused headers, causes the syntax errors to disappear from the *_test.cc files and makes for an altogheter more pleasant working environment (see below for my settings).

One other thing worth noting is that the "Build configuration for the indexer" preference is best left alone: choosing the one that would intuitively make the most sense (Use active build configuration) causes again a whole bunch of (spurious) syntax errors to appear in your test sources.[1]

I've personally not experimented enough with the "Index source and header opened in the editor," but if anyone can figure out what the best choice would be, I'd love to hear your comments.

Recommended Preferences

Finally, it's worth noting that, if you have recently updated your install (or even, moved from Helios to Indigo) it is possibly that the configuration, indexer, etc. of your projects got seriously hosed: what got me into the quest for the "perfect indexer settings" was a bunch of mysterious syntax errors in my unit tests, which I was totally unable to resolve (although the code itself build just fine with g++).

If you do come across a similar situation (a project that used to build, and where you get unreasonable and unexplained syntax errors in Eclipse's editor) try and change the Preferences for the C++ Indexer (even just opening the page, changing a few random settings, hitting 'Apply' and then 'Restore Defaults' and 'Ok' should work).

[1] Worth noting that the (default) setting is a bit 'circular' as it delegates to the "build configuration specified in the project's indexer settings;" however, the latter (unless otherwise specified) seems to defer to the Workspace settings: which must be different from just using the 'active build configuration' as choosing the latter causes a whole bunch of errors to appear in the test sources: hence, there must be a default, non-obvious setting which makes it all work, but no indication as to what that is.

Android ADK and mbed.org

2011-09-06T23:42:00.000-07:00

The 'mbed kit' just landed on my doorstep, sent courtesy of the good guys at ARM (Cambridge, UK).
This gadget supports the Android ADK (http://mbed.org/cookbook/mbed-with-Android-ADK)

which is fully documented here and, to have an idea of the geeky-fun that can be had with these gizmos, I suggest you check out this video.
I fear that my weekends are likely gone for quite a while now....

Just as an example of what is possible, the following snippet will make the LEDs flash, as well as connect to a terminal on the host PC to use as stdout; this is a very basic adaption of the HelloWorld.cpp app, and some tutorials from mbed.org: the point being that it took me less than 30min to get going!

// AlertAvert.com (c) 2011. All rights reserved
//
// The code below has been adapted from tutorials on
// mbed.org - credits should go to ARM Ltd.
//
// Author: M. Massenzio (m.massenzio@gmail.com)

#include "mbed.h"

// USB out to the host PC - from a terminal window access
// using: screen /dev/ttyACM0
Serial pc(USBTX, USBRX); // tx, rx

// Board LEDs
DigitalOut myled(LED1);
DigitalOut yourled(LED2);

bool on = true;
void quit() {
  on = false;
}

void toggle(DigitalOut d1, DigitalOut d2) {
  d1 = !d1;
  d2 = !d2;
}

int main() {
  pc.printf("Starting LED flashing...\r\n");
  Timeout to;
  myled = !(yourled = 1);
  to.attach(&quit, 5);
  while(on) {
    toggle(myled, yourled);
    wait(0.2);
  }
  myled = yourled = 0;
  pc.printf("Exiting now\r\n");
}

And this is what came out of it:

Use a common Tag for all your activities' logging

2011-08-22T00:12:00.000-07:00

It is pretty common for any non-trivial Android app to be composed of several Activities which together interact with the user for the furtherance of whatever is the objective of the app itself.

It is also pretty common, especially during the development phase, to have a need to emit logs so that one can check certain events happen when they are supposed to, and invariants, for example, are satisfied.

The Android ADT (Eclipse plugin) has a very nice feature in the form of the LogCat View that shows all the system logs, including those coming from your applications: as there may well be tens of apps running (and logging) at the same time (especially on a live device -- as to why you should use a device for your testing and debugging, you may want to read this) the LogCat enables one to 'filter in' a subset, based on a 'tag'.

The tag used by the LogCat is the value of the string used for the tag parameter in the Log.x() calls:

Log.d("myApp.tag", "All is going well");

It is pretty obvious that, in any given Activity, we would want a constant to use as the tag:

private static final String TAG = "myApp.tag";

but this has the drawback that we would have to either duplicate that same statment across all Activities (and, even worse, keep them all in sync if we decide, for whatever reason to change them) or resign ourselves to use different tags for different activities.

Alternatively, we could use a common 'Constants' class to keep, amongst others, a constant Tag:

public interface Constants {
  public static final String TAG = "myApp.tag";
  // other constants...
}

This is not very friendly, however, as every log statement (and there may well be hundreds in your code) would look something like:

Log(Constants.TAG, "All is going well");

it's only 10 extra characters, but multiply that for several hundrends, and you've a veritable case of carpal tunnel syndrome on your hands (well, ok...).

A much better option is to use Android's mechanism to store common application strings, which also gives the additional benefit that it's dead easy to change the tag in one full swoop, should that ever be necessary: in one of the XML files in the res/values/ directory, have a line of the form

myApp.tag

and in your code, you can do something like:

public class MyActivity extends Activity {
  public static String TAG; // note no 'final' here

  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    TAG = getResources().getString(R.string.TAG);
    Log.d(TAG, "Starting User Preferences activity...");
    // other initialization follows...
  }
}

This can be done in each and every of your App's activities, and then you can easily create a new filtered view in LogCat so that you can see only your App's logs, from all the activities, also with the reassurance that, should ever want to change that, it's just a one minute edit of the XML file

HTTP POST for Android (with a twist: Protocol Buffers)

2011-08-13T18:15:00.000-07:00

One of this blog's most visited entries (ranking always around the Top 5 Google results for "HTTP POST" too) elaborated on how to send data across to a server from a J2ME Midlet.

I thought it was high time for bringing this into the new Century, and provide an update as to how to do this from an Android smartphone.
It turns out that (a) the code is relatively straightforward and (b) that there is not so much difference after all from writing the same code for a desktop client: credit for this ought to go to the Android Google team who have done such a superb job of adapting the J2SE API to the Android platform.

As usual, I will not elaborate on the basics of how to set-up your Android development environment (I strongly recommend the excellent tutorials on Android.com) or about the basics of an HTTP POST (please refer to the original article).
Also, whilst not explicitly noted in the code below, it should be obvious that this may be a long-running request and/or anything could go wrong and block, and you should never run a network action from within the UI thread, as it would otherwise make your application unresponsive (this is even more critical in Android, where the system wisely detects such cases of kiddie-programming and pops a request to the user to the effect that they may want to terminate the offending slouching app... yet another reason to love the Android platform!): more details can be found in this post, which is still very much up-to-date.

Given that the code to send an HTTP POST request requires some data to be sent in the first place, and so as to make it all a bit more interesting, I decided to add a twist to the tale, by providing an example of how you could use Protocol Buffers in you communication between your mobile app and the server back-end.

Protocol Buffers are a massively useful technology developed (and open-sourced) by Google and we use them all over the place both as data payloads for our RPC infrastructure, as well as ways to encapsulate our domain model objects in our Bigtable persistence store (very often, PB's are the model objects and are used as first class objects).
Elaborating on their usefulness is outside the scope of this post, and I would strongly encourage you to experiment with them, especially if your application needs advanced serialization capabilities and/or is based on (or would benefit from) mixed-language (Java / C++) code.

Please download the source code from here, and then use the source code to build a protobuf-2.4.x.jar library that you will need to add to both the Android and Server projects (note this warning when adding the JAR to the Android'd build path: do not add it as a User Library in Eclipse, but rather as a single "external" JAR, or it won't be added to the APK and the application will fail at runtime).

If you are using Eclipse for your development (and I can't honestly see why not) it's worth noting that my team just released a new protobuf editor that is freely available and it will eventually released as open-source; it is based on the awesome XText 2.0 framework for DSL development, go check it out (the latter only works for Eclipse 3.7).

The following code is based on my Android Receipts applications (you can install it from Android Market - the current version does not implement the server communication yet, but sends the receipts' images and data to your email account).

Alright, having done intros and niceties, here we go!

As the starting point, here is the code for the protobuf messages:

// receipts.proto   created 2011-08-10, M. Massenzio
// Copyright AlertAvert.com (c) 2011.  All rights reserved.

package receipts;
option java_package = "com.alertavert.receiptscan.model.proto";
option java_outer_classname = "ReceiptsProtos";

message ReceiptProto {
  // The unique ID associated with the receipt, 
  // may be left unassigned until persisted
  optional string id = 99;
  
  // A name associated with this receipt, it should be unique for the user
  required string name = 1;
  
  required string merchant = 2;
  required float amount = 3;
  
  // The currency for the amount, if not present assumed to be
  // the default for the locale
  optional string currency = 4;
  optional string notes = 5;
  
  // The scanned image of the receipt, may not be there 
  // (as its URI may be retrieved from the persistence layer).
  optional bytes image = 6;
}

/*
 * A server request payload, contains enough information to 
 * uniquely associate the request with
 * an ongoing user session and one or more receipts.
 */
message ReceiptsPayload {
  required string token = 1;
  repeated ReceiptProto receipts = 2;
  optional string username = 3;
}

Fairly straightforward as you can see, this will compile to both C++ (.h / .cc) and Java source code, which one can then use in one's own code - the serialization is implemented by the protoc libraries (you'll see an example below) and the framework takes care of all the rest.

The command line to compile this is:

protoc --cpp_out=../src-gen/cpp --java_out=../src-gen/java receipts.proto

assuming that your code is in the receipts.proto file.

I use normally a Makefile to automate all this, which looks something like this:

# Makefile for protobufs
#
# Created by M. Massenzio, 2011-04-16
# Updated 2011-08-10
#
# Assumes the following structure (works best with Eclipse):
#
# Project
#   src           --> contains all java packages
#   src-gen/java  --> contains generated Java files in their packages
#   src-gen/cpp   --> contains generated C++ files in their namespaces
#   proto         --> all source .proto files
#
# In general, Protocol Buffer libraries are expected to be found
# in /usr/local/lib (see also below).

all: build

clean:
	@echo "Removing all generated code and recreating directories"
	@rm -rf ../src-gen/*
	@mkdir -p ../src-gen/cpp
	@mkdir -p ../src-gen/java

# WARNING: ../src-gen/cpp and ../src-gen/java-gen MUST be already present,

#          use make clean
build: receipts.proto
    # fails if protobuf is not installed
	@pkg-config --cflags protobuf >/dev/null
	@protoc --cpp_out=../src-gen/cpp --java_out=../src-gen/java receipts.proto
	@echo "Code successfully generated.\nRemember to include these flags in the compilation:\n" `pkg-config --cflags --libs protobuf`

#
# Flags for the compilation:
# compiler: -pthread -I/usr/local/include
# linker:   -pthread -L/usr/local/lib -lprotobuf -lz
#
# In Eclipse, use Project > Properties > C/C++ Build > Settings
#
# C++ Compiler: Miscellaneous --> add -pthread in "Other flags"
#               Directories   --> add a 'workspace linked' /gen directory
# Linker: Libraries  (-l) protobuf (-L) /usr/local/lib
#         Miscellaneous --> Linker flags: -pthread
#
# For Java projects, remember to add protobuf-java-2.4.0a.jar
# (on this system, under /opt/google/protobuf-2.4.0a/java/target)
#

If all went according to plan, you should have, in src-gen/java/com/alertavert/receiptscan/model/proto a ReceiptsProtos.java file, which you can open to have a look at (I strongly discourage editing it - nothing good can come of it, and it will be overwritten anyway once one regenerates the protobufs).

Time to go back to the Android project, add the protobuf-2.4.x.jar to the build path, add src-gen/java to the build path as a source folder (if src-gen is not already part of the project's folder, add it as a "linked folder," then right-click on the 'java' folder and select Build path > Add as source folder).

Verify that Eclipse can build the project without syntax errors on the ReceiptsProtos.java class, and then create (or update) an existing class that will send the receipts objects to the server.

Mine looks something like this:

// Copyright AlertAvert.com (c) 2011.  All rights reserved.
// Created by M. Massenzio
package com.alertavert.android.applications.receipts.connectivity;

// imports omitted
// ...

public class HttpSender implements Sender {
  /** Use a custom user-agent to distinguish the sender and for logging purposes */
  private static final String USERAGENT = 
      "AndroidReceipts-server-alertavert_1.0.0;android/receipts";
  URL destination;
  private String reason;
  
  @Override
  public boolean send(Collection receipts) {
    if (receipts.isEmpty()) {
      return false;
    }
    BufferedOutputStream os = null;
    HttpURLConnection connection = null;
    try {
      connection = (HttpURLConnection) destination.openConnection();
      if (connection == null) {
        reason = "Could connect to server " + destination;
        return false;
      }
      // Uses POST
      connection.setDoOutput(true);
      // get the default chunk length
      connection.setChunkedStreamingMode(0);
      // set a custom user agent, so the server can choose the best way 
      // to handle the request
      connection.setRequestProperty("User-Agent", USERAGENT);
      os = new BufferedOutputStream(connection.getOutputStream());
      byte[] data = wrapIntoProtoPayload(receipts).toByteArray();
      Log.d(ControllerActivity.TAG, "Proto serialized into "
          + data.length + " bytes");
      os.write(data);
      if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
        // TODO notify the listener of a successful send
        Log.d(ControllerActivity.TAG, "Receipts successfully uploaded");
        return true;
      } else {
        // TODO call the listener and return the error code and message
        Log.e(ControllerActivity.TAG, "Error uploading receipts: " +
            connection.getResponseMessage());
        reason = connection.getResponseMessage();
        return false;
      }
      return true;
    } catch (IOException ex) {
      ex.printStackTrace();
      return false;
    } finally {
      try {
        if (os != null) os.close();
        if (connection != null) connection.disconnect();
      } catch (IOException stupidestExceptionEver) {
        // just ignore this idiocy and praise JDK7's Try-With-Resources
        // ... not one day too soon!
      }
    }
  }
  
  private ReceiptsProtos.ReceiptsPayload wrapIntoProtoPayload(
      Collection receipts)
      throws IOException {
    ReceiptsProtos.ReceiptsPayload.Builder builder =
       ReceiptsProtos.ReceiptsPayload.newBuilder();
    // TODO obtain these values from the SessionManager or a SecAuth layer
    builder.setToken("123456abcdef");
    builder.setUsername("anuser");
    for (Receipt r : receipts) {
      builder.addReceipts(wrapIntoReceiptProto(r));
      Log.d(ControllerActivity.TAG, "Adding receipt " + r.getName());
    }
    return builder.build();
  }
  
  private ReceiptProto wrapIntoReceiptProto(Receipt r) throws IOException {
    ReceiptProto.Builder builder = ReceiptProto.newBuilder();
    if (r.getId() != 0) {
      builder.setId("" + r.getId());
    }
    builder.setName(r.getName());
    builder.setAmount(r.getAmount().getFloatValue());
    if (r.getAmount().getCurrency().length() == 0) {
      // TODO(marco) extract the default currency for the locale 
    } else {
      builder.setCurrency(r.getAmount().getCurrency());
    }
    builder.setMerchant(r.getMerchant());
    if (r.getNotes().length() > 0 ) {
      builder.setNotes(r.getNotes());
    }
    // Extract the image data and wrap into the protocol buffer
    byte[] buf = new byte[1024];
    InputStream is = new BufferedInputStream(new FileInputStream(
        r.getImageUri().getPath()));
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    int numBytesRead;
    while ((numBytesRead = is.read(buf)) != -1) {
      bos.write(buf, 0, numBytesRead);
    }
    is.close();
    ByteString image = ByteString.copyFrom(bos.toByteArray());
    builder.setImage(image);
    return builder.build();
  }
  ////////////////////////
  // Lots of other stuff here omitted, in particular Listeners that will be
  // notified when this method completes, etc.
  // ...
}

As you can see, the core of sending an HTTP POST request is pretty straightforward: 5-6 lines of code at most, with all the rest necessary to wrap the data around (note that in the sample code above I do virtually no error checking on the returned response code, nor use any data that the server may be sending back - maybe in a later post...).

The all-important line is

connection = (HttpURLConnection) destination.openConnection();

with destination the URL of the server (something like: https://receipts.alertavert.com/Receipts/upload); when testing it from your Android device, you have essentially two options:

emulator - slow and painful, I discourage it;
a real Android device connected via USB to your development box (highly recommended).

Depending on which option you use, and if #2 is being use with WiFi active (and possibly connected to your internal A/P) you will have to use different URLs. Assuming that your server (more about that in a second) is running on the same box as your development environment, and that you use either the emulator or the phone with WiFi OFF, use 10.0.2.2 as the address to get to your machine.

If you are using a device, with WiFi on, you can just use the PC's IP internal address (say, 192.168.0.3, or whatever it is you have set it up at on your LAN - or even an externally facing IP if your router/LAN allows you to do so).

Remember that if you are running a plain vanilla Tomcat instance, that one will be listening on port 8080, so the full URL will be something like:

http://10.0.2.2:8080/ReceiptsServer/upload

Which brings us finally to the server-side code, a pretty run-of-the-mill Servlet class:

public class UploadServlet extends HttpServlet {
  private static final long serialVersionUID = 1L;
  private static final Logger log = Logger.getLogger(UploadServlet.class.getCanonicalName());
  private static final String JPEG = ".jpg";
  private static final String FILE_SCHEME = "file";

  /**
   * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse
   *      response)
   */
  protected void doPost(HttpServletRequest request, HttpServletResponse response)
      throws ServletException, IOException {
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    BufferedInputStream bis = new BufferedInputStream(request.getInputStream());
    try {
      int numBytesRead;
      byte[] buf = new byte[256];
      while ((numBytesRead = bis.read(buf)) != -1) {
        bos.write(buf, 0, numBytesRead);
      }
      log.info("Received " + bos.size() + " bytes");
      ReceiptsProtos.ReceiptsPayload payload = ReceiptsProtos.ReceiptsPayload
          .parseFrom(bos.toByteArray());
      if (payload != null) {
        // TODO(marco): validate token with a SessionManager
        log.info("Got a valid payload: " + payload.getToken() + ", with "
            + payload.getReceiptsCount() + " receipts enclosed, from user "
            + payload.getUsername());
        if (payload.getReceiptsCount() > 0) {
          List receipts = new ArrayList();
          for (ReceiptsProtos.ReceiptProto proto : payload.getReceiptsList()) {
            // simple proto -> object model adapter method
            Receipt receipt = protoReceiptAdapter(proto);
            // extract the image data, save to a file and set receipt.setImageUri() 
            // to point to it
            extractImage(proto.getImage(), receipt);
            log.info("Received a receipt: " + receipt.toString());
            receipts.add(receipt);
          }
          // TODO pass receipts to a DAO for persistence
        } else {
          log.severe("No receipts received");
        }
      } else {
        log.severe("Could not parse payload into a valid ReceiptsProto object");
      }
    } finally {
      bos.close();
      bis.close();
    }
  }

  /**
   * A simple implementation of an Adapter, this would most likely be 
   * delegated off-servlet too.
   * 
   * @param proto
   * @return
   */
  private Receipt protoReceiptAdapter(ReceiptsProtos.ReceiptProto proto) {
    Receipt receipt = new Receipt();
    receipt.setName(proto.getName());
    receipt.setMerchant(proto.getMerchant());
    int intVal = (int) Math.floor(proto.getAmount());
    int decVal = (int) (proto.getAmount() - intVal);
    receipt.setAmount(new Money(intVal, decVal, proto.getCurrency()));
    receipt.setNotes(proto.getNotes());
    return receipt;
  }

  /**
   * Saves the image data to a temporary file - obviously this is just for demonstration
   * purposes, the actual persistence would be delegated elsewhere,
   * away from servlet processing.
   * 
   * @param image
   * @param receipt
   * @throws IOException
   */
  private void extractImage(ByteString image, Receipt receipt) throws IOException {
    File imageFile = File.createTempFile(receipt.getName(), JPEG);
    BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(imageFile));
    bos.write(image.toByteArray());
    bos.close();
    try {
      receipt.setImageUri(new URI(FILE_SCHEME, imageFile.getAbsolutePath(), ""));
      log.info("Image file saved in " + imageFile.getAbsolutePath());
    } catch (URISyntaxException e) {
      log.severe("Could not create URI for " + imageFile.getAbsolutePath()
          + ", original error was "
          + e.getLocalizedMessage());
    }
  }
}

Again, there is virtually no error processing here, nor any error codes are returned to the client - definitely not production-class, but at least gives you an idea of how to get data from a mobile Android client and save it someplace server-side.

Remember to add to your web.xml the following:

<servlet>
    <description></description>
    <display-name>UploadServlet</display-name>
    <servlet-name>UploadServlet</servlet-name>
    <servlet-class>
        com.alertavert.receipts.server.servlets.UploadServlet
    </servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>UploadServlet</servlet-name>
    <url-pattern>/upload</url-pattern>
  </servlet-mapping>

so that it knows how to direct the request.

Hope this becomes as popular as the other, please feel free to link to this page from your blog and / or forum posts (I'd love to hear from you if you do so, but you don't need any special permission) and let me know what you think of it.

(Images courtesy of E. Massenzio (c) 2011 - see her other work here)

Automating Eclipse launch

2011-08-09T23:26:00.000-07:00

I've recently taken up the Eclipse team at Google (we integrate Google's internal build tools and distributed development environment with the Eclipse Platform) and we are constantly faced with the challenge of maintaining several distributions in a multi-user environment.

I thought I'd use some of the goodness we've come up with in my home environment too, so have put together a much-scaled-down version of our launch script, for my personal use - and thought I'd share this more broadly, as I'm sure other folks will find this useful too.

I generally have multiple installation and user configuration profiles, and even from the desktop launcher, I much prefer to use launch scripts than just pointing the shortcut to the binary executable.

Here is my generic configuration script:

#!/bin/bash
#
# Runs the stable version of Eclipse.
# By default, it will launch Indigo, installed in /opt/eclipse; different
# versions can be launched by invoking this script and setting the
# ECLIPSE_INSTALL and CONFIG variables to point to the correct places.
#
# The default configuration directory is set to '/home/$USER/.eclipse/eclipse37'
# but this can be changed exporting a different value for CONFIG.
# See /home/marco/bin/eclipse-helios for an example of how to do this.

##############################
# Globals (default value):
#
#   LD_LIBRARY_PATH           (/usr/local/lib:/usr/lib)
#     library search path for C++ dynamic libraries
#
#   ECLIPSE_INSTALL           ()
#     installation directory
#
#   CONFIG                    ()
#     user's configuration directory
#
#   MEM_ALLOC_POOL, PERM_SIZE (2,048MB, 256MB)
#     Memory allocation and PermGen size
#
#   JVM                       (/usr/local/java/bin/java)
#     Java VM executable (java)
#
# By setting any of the variables above after source`ing this script
# and before invoking launch_eclipse() the default value(s) can be changed
##############################
declare ECLIPSE_INSTALL
declare CONFIG
declare MEM_ALLOC_POOL
declare PERM_SIZE
declare JVM="/usr/local/java/bin/java"

##############################
# Concatenates input paramters to LD_LIBRARY_PATH, if not already present
##############################
munge_library_path() {
  for dir in $@; do
    if [ -n "${dir}" ]; then
      local is_present=`echo $LD_LIBRARY_PATH | grep ${dir} | wc -l`
      if [ ${is_present} -eq 0 ]; then
        export LD_LIBRARY_PATH=${dir}:$LD_LIBRARY_PATH
      fi
    fi
  done
}

##############################
# Sets up global variables with default values if not already set
##############################
check_dirs() {
  # Installation directory
  if [ -z "${ECLIPSE_INSTALL}" ]; then
    echo "You must set ECLIPSE_INSTALL to the Eclipse installation directory"
    exit -1
  fi
  if [ ! -d ${ECLIPSE_INSTALL} ]; then
    echo "Cannot find Eclipse installation directory (${ECLIPSE_INSTALL})"
    exit -1
  fi

  # Configuration directory, user-specific
  if [ ! -d "${CONFIG}" ]; then
    mkdir -p ${CONFIG}
  fi
}


##############################
# The location of the JVM, currently the default (OpenJDK 6)
##############################
locate_jvm() {
  if [ ! -e "${JVM}" ]; then
    echo "Could not locate a valid JVM at ${JVM}"
    echo "Trying to locate a valid java installation"
    JVM=`which java`
    if [ -z "${JVM}" ]; then
      echo "Could not find a valid JRE, giving up"
      return -1
    fi
    echo "Found a JRE at ${JVM}"
  fi
}

##############################
# Runs a few checks, sets up the LD library path and then launches Eclipse
#
# Param:
#   clean   will set the '-clean' Eclipse option that will clear the plugins cache
##############################
launch_eclipse() {
  check_dirs
  locate_jvm
  munge_library_path /usr/local/lib /usr/lib

  if [ "$1" == "clean" ]; then
    CLEAN="-clean"
  fi

  $ECLIPSE_INSTALL/eclipse $CLEAN -vm $JVM -configuration $CONFIG \
      -bundlepool $CONFIG/plugins \
      -vmargs -Xmx${MEM_ALLOC_POOL:-"2048M"} -XX:MaxPermSize=${PERM_SIZE:-"256M"}
}

This is used in the actual shell script (called, unimaginatively, eclipse):

#!/bin/bash
#
# Runs the stable version of Eclipse.
# By default, it will launch Indigo, installed in /opt/eclipse; different
# versions can be launched by invoking this script and setting the
# ECLIPSE_INSTALL and CONFIG variables to point to the correct places.
#
# See eclipse_config.sh for more details

source /home/marco/bin/eclipse_config.sh

ECLIPSE_INSTALL="/opt/eclipse"
CONFIG="/home/${USER}/.eclipse/eclipse37"

launch_eclipse

What is left is to just point a desktop (or menu) launcher to this script and replace the generic icon with the one in the eclipse/ folder (icon.xpm)

CREDITS -- I would very much like to gratefully acknowledge Machtelt Garrels and his much-consulted "Bash Guide for Beginners;" I never seem able to remember the bash conditional operators!

Eclipse (Indigo) Update site

2011-07-06T22:50:00.000-07:00

When executing a "clean run" of Eclipse (typically, after p2 has completely messed up your installation and you've lost critical functionality / plugins -- the only reliable remedy I've found thus far, is to wipe ~/.eclipse, and restart from scratch) the 'Available Update Sites' is an empty list.

Googling the information proves every time a complete random shot in the dark, so I thought I'd collect the info once and for all in a place I'm likely to find again in the future.

The wiki page (impossibly difficult to find in the maze that Eclipse documentation is):
http://wiki.eclipse.org/Eclipse_Project_Update_Sites

Gives this link for...
The Eclipse Project Updates - http://download.eclipse.org/eclipse/updates/3.7

Executing an update of the Eclipse Platform, adds another 'update site' that provides a much
richer selection of available plugins:

Indigo - http://download.eclipse.org/releases/indigo

Hope this helps someone else too!

Ubuntu: Network Manager is evil

2011-05-24T23:27:00.000-07:00

Warning - this post if filled with hatred, if you believe in forgiveness, you may want to look away now.

In my book, the only sin worse than uselessness, is actively causing damage and waste of time.
What punishment ought to be meted then, to something that, having proven to be useless, causes untold waste of time.

Yes, NetworkManager, and nm-applet: I'm looking at you two.

Quite apart from never having quite understood what use one could have for either, I have found a number of situations in which NM has caused waste of time, only the last of which has been to prevent a KVM virtual machine I just created to connect to the network, despite all my efforts, searching on the Web, trying out countless variations on configurations and, yes, several reboots.

I have been following along the instructions given here, only to be unable to connect in UserMode, barely managing (via the Bridged interface) to connect to my host machine, and totally failing to obtain any connection to the outside world.

Enter

sudo apt-get purge network-manager network-manager-gnome

and, hey, presto, all is good and well, and I can happily connect from my guest OS to the Internet.
(see also here)

Simply changed my /etc/network/interfaces file to reflect my settings (YMMV, adjust to your LAN settings) and all works just fine:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto br0
iface br0 inet static
        address 192.168.1.10
        network 192.168.1.0
        netmask 255.255.255.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

Just remember to restart the network after you have made the changes:

sudo /etc/init.d/networking restart

Die in pain, Network Manager, and may the world be a better place without you.

Creating a C++ callback function from a class instance method

2011-05-22T00:26:00.000-07:00

I have been playing around with the Boost library and, in particular, the boost::thread multithreading library.

There a thread is simply created by constructing an object of type boost::thread, and passing in to the constructor a pointer to the function to be executed in the newly spawned thread.

There is much more to it, and I encourage you to read the documentation, but that gave me an interesting opportunity to ponder the issue of how to create a thread by instead passing in a pointer to a class's method: the need typically arises when one does not want to use global variables (which need to be guarded against race conditions) or where parameters are necessary for the execution of the method and/or return values are expected.

Granted, the boost::thread constructor allows you to pass in up to nine parameters, so that's rarely an issue, but I just wanted to find out a reasonably general way to achieve this; it turns out that this is far from trivial, and I thought I'd share my findings.

For the impatient, this is the solution:

template<typename T, typename V, typename R>
class MakeCallback {

  // Member method in T(ype), takes parameter V(alue),
  // and returns an object of type R(esult)
  // Note the parentheses around (T::*): without them, 
  // the compiler gets confused.
  typedef R(T::*func)(const V&);
  func f_;
  T& t_;
  V value_;
  R* res_;
public:
  MakeCallback(T& type, func f, const V& value, R* res = NULL) :
      f_(f), t_(type), value_(value), res_(res) { }
  virtual ~MakeCallback() { }
  void operator()() {
    if (res_)
      // Note here the parenthesis around t_.*f_ 
      // They are necessary, or a compiler error will be generated
      *res_ = (t_.*f_)(value_);
  }

  // This allows the pointer to the result value to
  // be set after the object has been created
  void set_res(R* res) { res_ = res; }
};

and this is how one uses it:

int main(int argc, char[}* argv) {
  A a(33);
  int res;
  // Generally, you don't need the & operator to take a function's pointer.
  // But in this case it's mandatory, or the compiler will complain
  // about doSomething() not being static.
  // In any event, the use of & makes your intent clearer (you are taking
  // the method's address,
  // not invoking it) and I encourage you to use it consistently, even where
  // this is not stricly necessary.
  MakeCallback<>A, int, int> mc(a, &A::doSomething, 22, &res);
  boost::thread do_it(mc);
  do_it.join();
  std::cout << "And the result is : " << res << std::endl;

  B b;
  boost::thread another(MakeCallback<A, std::string, B>(a, &A::doSomethingElse, "22.13", &b));
  another.join();
  std::cout << "..and B is " << b.get() << std::endl;
}

given the following declarations for A and B:

class B {
  float x;
public:
  B(float s = 0.0) : x(s) {}
  float get() { return x; }
};

class A {
  int num_;
public:
  A(int num) : num_(num) { }

  int doSomething(const int& k) {
    return num_ + k;
  }

  B doSomethingElse(const std::string& s) {
    double b = ::atof(s.c_str()) + num_;
    return B(b);
  }
};

A couple of points that have caused me much head-scratching and I thought it worth passing on:

you must use the '&' operator in front of A::doSomething, or the compiler will complain about it not being a static function;
note we are, in fact, calling an instance method: in other words, we expect to have instance-specific values stored in the class (A in this case) that our method (doSomething()) will use
note also the use of a pointer (R* res_) to store the return value: this is important; without it, the return value of the method call ((t_.*f_)(value_)) will be lost!

This is because, in the call to boost::thread(mc) the compiler automatically created a copy of our object (using the compiler-generated copy constructor; see Scott Meyer's "Effective C++", Item 5)
Note also the calls to thread::join() - without them, you'd have a race on the returned values: possibly reading them, before the actual method (doSomething()) had any chance of updating it;

Just to be clear, this rather trivial implementation is not meant to be used in real code (if you need something along these lines, boost::bind is what you want) but it was an interesting exercise in how to deal with generics, functors and callbacks.

Skype on Ubuntu 10.04 - mic doesn't work

2011-05-08T11:36:00.000-07:00

I got the following fix from here, but it does not tell the entire story:
(my setup: Acer Aspire One AO751h, Ubuntu Lucid 10.04.2 LTS)

I had the same problem on an Acer Timeline for a long time, but the balance trick works.

Solution:
A better way than changing the output balance is to use alsamixer or pavucontrol and mute one of the input channels. In alsamixer: F4 to change to recording, left/right to move to microphone, and z or c to mute one channel. In pavucontrol go to the "input devices" tab, unlock the channels, and mute one of the channels.

It seems the normal volume control uses one balance slider for all inputs and outputs together.

The reality is that two poorly designed (and worse implemented) pieces of software interact in a way to make it virtually impossible for the average user to figure out what's going on:

Pulse Audio -- it still amazes me how a simple problem (enable audio on a PC) could be made so complicated and unusable;
Skype for Linux -- a real disgrace that a company of the might and power of Skype could let us down so badly: I just wish they had the decency of release the software open source, so we could fix it ourselves!

Be that as it may, in addition to the 'fix' above, you need to also disable the "Allow Skype to adjust mixer levels" in the Sound Options tab of Skype - otherwise you can see skype continually adjusting your microphone's gain down until it gets to the point of being unaudible.

Leaving it here for future reference: I don't expect Skype to fix this anytime soon!

Eclipse CDT and Google GTest

2011-04-10T00:25:00.000-07:00

If you have read my previous post about adding the gTest framework to your C++ project, you are very likely to have spent quite some time scratching your head trying to figure out how to make it play nicely with Eclipse's CDT, and its (let's face it) not terribly well-developed concept of 'Build Configurations'.

Especially coming from Java and the extremely intuitive integration in Eclipse with JUnit, CDT's approach to unit testing leaves a lot to desire.

I will not go through the basic setup necessary to add gTest to your project: please read the Get Started intro on Google Code, or my previous post; it's pretty trivial anyway.

However, there are a couple of corrections (or perhaps, CDT 7.0 has fixed some of the limitations that required the workarounds I mentioned there).

Add gtest.a to the Test Configuration that you define for your project
The best way to add the gTest libraries would be to create a symbolic link from /usr/local/lib to the

/make/gtest.a

library, created by the default install process)

$ sudo ln -s make/gtest.a /usr/local/lib/libgtest.a

and then add /usr/local/lib to the Linker's library search path (-L/usr/local/lib) and libgtest.a to the list of included libraries (-lgtest) using Project > Properties:

Add the includes from gTest to ALL build configurations

Do this by adding the

/include/ folder to your project's include search path (Project > Properties);

Now, to make the unit tests run, we need to add a fairly trivial main() method (taken from gTest's Primer doc):

#include "gtest/gtest.h"

int main(int argc, char **argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

along with the normal test fixture(s) in either one or multiple .cc files, whichever way works best for you.
(personally, I prefer the latter: one C++ class per .cc file, and the associated *_test.cc file alongside - keeps me sane! This, obviously, requires that one just has one main() method in a separate source file, which will drive the entire test suite).

However, this will cause trouble when linking with a Debug/Release configuration, and obviously you don't want anyway to ship your code with the tests binaries to bloat it.

This is (in theory) where the 'Build configurations' come into play: just exclude the *_test.cc files from Debug/Release, and equally the file containing the main() method of your program (I like to call those Main.cc and have the bare minimum to get it all started, but nothing more) be excluded from your Test configuration.

However, if you do that, and hence would not include the

/include

folder to all the Build Configurations (for which this would be rather pointless, as it would only be needed by the Test config) you lose the syntax highlight in all the files that make up your test fixture(s): for some reason, excluding a file from even ONE Build Configuration, confuses the hell out of CDT and the editor seems unable to cope even when the Active Configuration is the one (eg, Test) that you actually defined the file to be part of -- this is bizarre, because the various configurations still build correctly and run as expected.

So here is the workaround (which is not pretty, but looks like it's the only way to keep one's sanity, and adds minimal hassle): just remember to add the gtest/include folder to all the build configurations settings.

A screenshot of what happens when you go for the "wholesome" approach and exclude from build all the *_test files as well as the gTest includes from the settings for the Debug/Release configurations [note however how the compilation/build completes without errors for the Test config]:

Let's hope CDT 8.0 comes out soon, alongside XText 2.0 -- all slated to be released along with Indigo (Eclipse 3.7) sometime in June 2011.

Mobile templates

2011-03-26T00:59:00.000-07:00

Just enabled "Mobile Templates" for this blog: check it out on your iPad, Android, iPhone, or mobile device: it looks really cool on my HTC Evo!

Run Android tests in a connected device

2011-01-17T23:58:00.000-08:00

Not quite sure as to why it never occurred to me, but it is totally possible and legal to run unit tests inside a real device, connected to your dev box via a USB cable.

An obvious advantage is that you are now running your tests in a more "realistic" environment; there are no drawbacks that I can think of (unless your device has some weird 'lockdown' imposed by the manufacturer / cell provider); and... that's when you realize just how much faster than an emulator that is.

So here's the day's top-tip: if you are in a hurry, and have a lot of unit test you want to run, get that USB cable out!

Using the same model classes in Android, GWT and JPA (Part II)

2011-01-15T00:20:00.000-08:00

In a recent post, I've demonstrated how one can use the same domain-specific (Model) classes end-to-end, from the Mobile layer, to the GWT Front-End, up to server-side and Persistence layer (JPA).

Recently, I've done a serious refactoring of my Android Receipts mobile app, using java.net.URIs to keep track of the receipts' images location in the (local) filesystem on the mobile: whilst these are not used at all (and, in fact, have no meaning) on the GWT front-end (although, may "regain" semantics on the server side, where may be needed to track the location of the persisted images) they cause a bit of confusion, as the java.net.URI class is not supported by GWT.

What I then decided to do, was to find out how one could overcome such a problem, and here is what I found.

The key part in replacing a JRE class that GWT does not support is in using <super-source> that does a virtual `chroot` on the directory structure:

# GwtPlayground/lib/src/com/alertavert/receiptscan/URI.gwt.xml
<module rename-to='ReceiptscanURI'>
  <inherits name='com.google.gwt.user.User'/>
  <!-- Specify the paths for translatable code, by redefining the super-source
       so that the class under jre/java/net/URI will be correctly identified by the
       compiler as java.net.URI
    -->
  <super-source path='jre'/>
</module>

The URI.java file goes under jre/java/net/ - the GWT compiler will take care of the rest.
Remember that in URI.java the package is java.net:

# com/alertavert/receiptscan/jre/java/net/URI.java

// Copyright AlertAvert.com (c) 2011. All rights reserved.
// Created Jan 14, 2011, by M. Massenzio (marco@alertavert.com)

package java.net;

/**
 * Horrible hack to support Receipt's use of java.net.URI location for the
 * actual image file, which is however not supported in GWT code
 */
public interface URI {
  public abstract String getPath();
}

This must be "kept hidden" from Eclipse, or it will flag it as a compilation error (the folder structure and the package declartion won't match).

The GWT compiler only needs access to the source files (and, as far as URI.java is concerned, that's all that's needed too).
However, both Eclipse and the actual server-side code will need access to the model classes (eg, Receipt.java) and also you want them someplace where it's easy to edit them, and the updates be reflected reasonably quickly in the compiled code (as well as in the running web app).

Keep them in a `linked folder` in your GWT project (to the Android project, where they belong) and add as well a symlink to the source folder:

$ ls -lad src/com/alertavert/receiptscan/model
  lrwxrwxrwx src/com/alertavert/receiptscan/model -> 
      /workspaces/main/polaris/common/com/alertavert/receiptscan/model

Here, you can place the ReceiptscanModel module:

# GwtPlayground/lib/src/com/alertavert/receiptscan/ReceiptscanModel.gwt.xml
<module rename-to='ReceiptscanModel'>
  <inherits name='com.google.gwt.user.User'/>
  
  <!-- Mock implementation of java.net.URI which GWT does not support, and
       in fact, is not necessary server-side, but Receipts declares and would break
       GWT compilation.
       This enables the GWT module to compile, and will never be used at runtime.
    -->
  <inherits name='com.alertavert.receiptscan.URI'/>
  
  <!-- No need to specify the entry point class or anything else   -->
  <!-- for that matter: this is only needed so that classes in     -->
  <!-- the .model folder will be incluse in the transitive closure -->
  <!-- of GWT compiler.                                            -->
  <!-- Specify the paths for translatable code                     -->
  <source path='model'/>
</module>

Last step, is to create a JAR that the GWT compiler will use (needs to be in the classpath of the GWT Eclipse project):

#!/bin/bash
#
# GwtPlayground/lib/create-jar.sh
#
# Simple script to create JAR file containing the java.net.URI replacement for GWT
# Uses the -C option to descend into the src/ dir before packaging the files.

USAGE="Please enter a version number for the JAR (eg, 1.0)"

if [ -z $1 ]; then
  echo $USAGE
  exit -1;
fi

echo "Creating jar file receiptscan_gwt_model_$1.jar"
jar -cvf receiptscan_gwt_model_$1.jar -C src com >/dev/null 2>&1

Note that every time you edit any of the Java files in the `model` package, the server will need to be restarted (there is a button in Eclipse's GWT Development console that allows you to quickly do so) and the create-jar.sh needs to be re-run, so that the new code is made accessible to the GWT compiler.

Android Draw 9-patch seems broken too

2010-12-23T17:03:00.000-08:00

As far as I can tell, the current release of the Android SDK (R08) ships without the Swing Desktop jar, and this causes draw9patch to fail.

~$ draw9patch &
[2] 52452
~$ Exception in thread "AWT-EventQueue-0" java.lang.NoClassDefFoundError:
       org/jdesktop/swingworker/SwingWorker
 at com.android.draw9patch.Application$1.run(Application.java:48)
 at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
 at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
 at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:269)
 at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:190)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:184)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:176)
 at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

[2]+  Done                    draw9patch

The 'fix' is rather simple: just download the JAR for Swing Desktop from here, and drop the swinglabs-0.8.0.jar file into the [sdk-install-dir]/tools/lib folder.

The Swing Desktop project can be found here.

Update -- Richard just left a comment here and pointed me to his site (Android 9 Patch) who's got plenty of 9-patch icons that really look awesome, and I thought I'd share the goodness (especially, given he's so kind as to share them at no charge).

Go check out Patch 15, it's pretty impressive!

Carrier BIlling comes to Android!

2010-12-23T13:35:00.000-08:00

Some of you folks may have recently noticed a little remark at the bottom of an email from the Android Market folks:

Finally, we wanted to bring to your attention that Android Market now offers a new form of payment for users on the AT&T network -- Direct Carrier Billing. This payment option lets Android users on the AT&T network purchase applications more easily.

This is an awesome achievement on the part of the Google folks, which integrates payments made by users into their phone bills, if they are AT&T customers: in turn, this enables us (the developers) to receive our payments via fewer user clicks and in a more streamlined way, so as to smooth the path for the user to purchase our Apps.

And all that, at virtually no additional complexity for us, which is, I'd say, pretty awesome!

I would expect more integrations and more awesomeness in the next few months: stay tuned!

Latest update for Android SDK breaks for Ubuntu Karmic

2010-12-08T23:32:00.001-08:00

If you have recently updated your Android SDK to R08 (Gingerbread, 2.3) on Linux and are using Ubuntu Karmic, chances are that you will not be able to run the emulator: It will simply die with the following error:

$ ./emulator
./emulator: /lib32/libc.so.6: version `GLIBC_2.11' not found (required by ./emulator)

This is caused by an incompatibility with Karmic's installed GLIBC - the new SDK/Emulator works just fine on Lucid.

The Android team is working on a fix - in the meantime, a workaround is to download the _r07 of the tools from here, and replace emulator in the tools/ folder under the android_sdk installation folder.

A bit hacky, but works.

Using Boost in Ubuntu with Eclipse

2010-11-30T22:38:00.001-08:00

Boost is an open source library of extremely useful and carefully designed C++ classes and methods ranging from graph algorithms, to regular expressions matching, to multi-threading.
Part of the Boost library was also integrated into the C++ standard as the TR1 set of libraries.
You can learn more about Boost here.
Using it with Eclipse in Ubuntu is definitely possible, but not straightforward; so I decided to post this simple tip here, to help folk spare some of the grief.
The first step in using Boost in your code is to download the header (.hpp) files and (optionally) source code - the latest release is 1.45 and is available from Boost's website.
In theory, you could compile it, and build the libraries' binaries yourself: by all means, go ahead and do it, but there is an easier way, if you can live with a slightly 'older' version.

Ubuntu 9.10 (Karmic Koala) comes with Boost 1.34 pre-configured (I think, this is, at any rate, the version that I had on my system):

$ ls /usr/lib/libboost*
/usr/lib/libboost_date_time-gcc41-1_34_1.so.1.34.1
/usr/lib/libboost_date_time-gcc41-mt-1_34_1.so.1.34.1
/usr/lib/libboost_date_time-gcc42-1_34_1.so.1.34.1
/usr/lib/libboost_date_time-gcc42-mt-1_34_1.so.1.34.1
/usr/lib/libboost_filesystem-gcc41-1_34_1.so.1.34.1
/usr/lib/libboost_filesystem-gcc41-mt-1_34_1.so.1.34.1
/usr/lib/libboost_filesystem-gcc42-1_34_1.so.1.34.1
/usr/lib/libboost_filesystem-gcc42-mt-1_34_1.so.1.34.1
/usr/lib/libboost_iostreams-gcc41-1_34_1.so.1.34.1
/usr/lib/libboost_iostreams-gcc41-mt-1_34_1.so.1.34.1
/usr/lib/libboost_iostreams-gcc42-1_34_1.so.1.34.1
/usr/lib/libboost_iostreams-gcc42-mt-1_34_1.so.1.34.1
/usr/lib/libboost_regex-gcc41-1_34_1.so.1.34.1
/usr/lib/libboost_regex-gcc41-mt-1_34_1.so.1.34.1
/usr/lib/libboost_regex-gcc42-1_34_1.so.1.34.1
/usr/lib/libboost_regex-gcc42-mt-1_34_1.so.1.34.1
/usr/lib/libboost_signals-gcc41-1_34_1.so.1.34.1
/usr/lib/libboost_signals-gcc41-mt-1_34_1.so.1.34.1
/usr/lib/libboost_signals-gcc42-1_34_1.so.1.34.1
/usr/lib/libboost_signals-gcc42-mt-1_34_1.so.1.34.1
/usr/lib/libboost_thread-gcc41-mt-1_34_1.so.1.34.1
/usr/lib/libboost_thread-gcc42-mt-1_34_1.so.1.34.1

However, by using Synaptic, you can actually install the more recent 1.40.0 version:

This will install a bunch of libboost_*.so.1.40.0 files into /usr/lib: in order to match the libraries with the header files (and avoid introducing incomprehensible compilation errors at best, and subtle bugs at worst) you should download the matching boost_1_40_0.tar.bz2 from Boost download archives.

We would be almost there, were it not for the fact that the soname does not really comply with the -l gcc linker option.

In gcc the -L option specifies a 'search directory' (this would not be needed, as /usr/lib is searched for libraries by default) and the -l specifies additional dynamic (.so) or static (.a) libraries: you specify the file to include during the link step by omitting the `lib` prefix and the .so or .a extension.
In other words, if your HelloBoost source uses code from Boost Regex (libboost_regex.so) you would specify it like thus:

$ gcc -Wall hello_boost.cc -o HelloBoost -lboost_regex

You can see where I'm heading with all this: the last step will be to add symbolic links that do away with the soname suffixes of Boost libraries, and all will be well:

$ ln -s /usr/lib/libboost_regex.so.1.40.0 /usr/lib/libboost_regex.so

Given that there are in total 24 files for the whole of Boost 1.40, I've done a bit of find & replace magic, and have come up with the following shell file (the other advantage being that, when a later version becomes available for Ubuntu, I will not have to change any of my Eclipse projects setting, but just change the VERSION value in the script):

#!/bin/bash
#
# Simple script to enable Boost with simple -l gcc option
# Use: gcc -Wall hello_world.cc -o HelloWorld -lboost_regex
#
# Created by M. Massenzio, 2010-11-30

VERSION=1.40.0

for libname in libboost_date_time \
            libboost_filesystem \
            libboost_graph_parallel  \
            libboost_graph      \
            libboost_iostreams  \
            libboost_math_c99f  \
            libboost_math_c99l  \
            libboost_math_c99   \
            libboost_math_tr1f  \
            libboost_math_tr1l  \
            libboost_math_tr1   \
            libboost_mpi        \
            libboost_prg_exec_monitor \
            libboost_program_options  \
            libboost_python-py25      \
            libboost_python-py26      \
            libboost_regex            \
            libboost_serialization    \
            libboost_signals          \
            libboost_system           \
            libboost_thread           \
            libboost_unit_test_framework \
            libboost_wave \
            libboost_wserialization 
do
  ln -s /usr/lib/$libname.so.$VERSION /usr/local/lib/$libname.so
done

The last and final step is to tell Eclipse what to look for when building your binary: right-click on your project's folder (in the C++ Perspective, Project Explorer) then Properties > C/C++ Build > Settings, select GCC C++ Linker > Libraries and add the 'stripped' library names in the option list:

Note for the curious: the additional libraries listed there are for Google C++ Unit testing framework, GUnit, available here - you will have to build it, but that's pretty straightforward, resulting in the two libgtest.a and libgtest_main.a libraries - be careful to add those only to your 'Test' configuration and exclude your unit tests from build in your Release/Debug configurations.

Using the same model classes in Android, GWT and JPA

2010-11-23T00:20:00.000-08:00

These days, it is rather common to have a mobile-enabled web service or application, where you essentially enable your users to access the service both via a browser-enable desktop application, as well as from their mobiles whilst on the go.

It's usually the case that the main business logic, as well as the business Model, are shared between the browser components and your mobile app: however, it is not obvious how to re-use code (or even use the same source code) when you are essentially using two completely separate SDKs, and even different JREs!

In fact, if you are using Google Web Toolkit on the front-end, there's not even a JRE involved: as your classes will be compiled in JavaScript.

This short video shows how you can actually use the very same source files in all three layers, so that when you make changes (fix bugs, add functionality) the improvements are immediately reflected everywhere - and even better, you avoid subtle bugs that may be caused by inadvertently the code going "out-of-sync" in one of the layer.

The example is based on my Android Receipts application, which can also be freely downloaded in Android Market.

Please let me know what you think, and whether you are finding the videos useful - or you prefer the written word!

Update Someone commented on the low resolution of the video (480p) as making it difficult to follow: I understand, and, in fact, the original video was at a higher res (720x576, 25 fps): unfortunately, uploading to YouTube the resolution gets dropped during the transcoding.
Disappointing, I know, but hardly something I can do anything about.

I would suggest instead to see also Part II of this tutorial, which further elaborates on the topic, and provides some code samples.

Dump CPU temperature data to a file in Ubuntu

2010-11-20T23:59:00.016-08:00

Update: I've figured out that it makes a lot more sense to have a reading of the CPU load to correlate with the temperature reading, so I've added that too using /usr/bin/uptime.

I have written a very simple Java utility to take readings from the CPU temperature sensors utility (/usr/bin/sensor) and dump it to a CSV file.
Although of fairly limited utility, I just wanted to see how to run a binary from Java and capture its output - and, let's face it, it is a rainy Saturday in Silicon Valley this afternoon and I'd just installed this new CPU cooler and just felt like doing some fuzzying around.
All the code, javadocs and a full working jar are available from the usual place, just remember to create a sensor.properties file under a .sensors directory in your home dir.

Configuring and Running the program
The program itself can be run from the command line with:

$ java -jar sensors_0.1alpha.jar

and a simple properties file looks something like:

# Configuration file for the sensor data storage utility
#
# Created by M. Massenzio, 2010-11-20

sensors.data.file=/home/marco/tempdata/sensors.data.csv
sensors.data.timestamp.format=hh:mm:ss
sensors.cores=2
sensors.interval.msec=60000

# Default is /usr/bin/sensors, change it here if your installation is different
sensors.command=/usr/local/bin/sensors

The data is appended at the end of the CSV file, so it's ok to leave the property unchanged between runs, or even have this run regularly from a cron job (you'll have to do a bit of shell magic to find out it PID and have another job kill it after a given time).

Obviously, an improvement to be to have an additional property configuration to have it terminate after a given amount of time (or a set number of readings).

If you have ideas how to improve this (the obvious next step would be to add a simple JDBC adapter and dump the data on a MySQL database) and/or make changes to the code, please let me know!

The Code
The source code itself is packaged inside the JAR, and consists of a single class, where everything happens, to an extent, just inside the main() method (although, there's a Runnable thrown in to be run as a TimerTask, so it's not as banal as it sounds!).

Executing an external command

Running a Process from inside Java code, is a simple matter of executing:

String sensorProcessLocation = System.getProperty("sensors.command", 
    "/usr/bin/sensors");
Process p = Runtime.getRuntime().exec(sensorProcessLocation);
LineNumberReader reader = new LineNumberReader(
    new InputStreamReader(p.getInputStream()));

The last lines captures the output from the process (I know, blatantly confusing) and you can then use to 'pipe' its output into your code for processing.

The RegEx to parse the output
Although the output from sensors would be pretty easy to parse 'by hand' I thought it much more elegant to use a Regular Expression, and also throw in the concept 'capturing groups' to have an orderly output (although, it's a safe bet sensors will consistently output the temperature readings in 'sorted cores order'):

REGEX = "Core([01]).*\\+([0-9.]*).*"

will correctly parse output from the version of sensors installed on my distro (Ubuntu 9.10, Karmic Koala):

$ lsb_release -a
   No LSB modules are available.
   Distributor ID: Ubuntu
   Description: Ubuntu 9.10
   Release: 9.10
   Codename: karmic
 $ sensors
   k8temp-pci-00c3
   Adapter: PCI adapter
   Core0 Temp:  +42.0°C                                    
   Core1 Temp:  +39.0°C

This is the code that parses the output from sensors, and puts the correct reading in the respective core's array slot:

Pattern p = Pattern.compile(System.getProperty("sensors.data.regex", 
             REGEX));
  Matcher m = p.matcher(reading);
  if (m.matches()) {
    int core = Integer.parseInt(m.group(1));
    if (core < coreTemps.length) {
       coreTemps[core] = Float.parseFloat(m.group(2));
     }
  }

The rest of the code (available inside the Jar to download, alongside with the javadoc in the /apidocs folder) is pretty much simple housekeeping.

Changing the value returned by getModuleName

2010-11-11T23:17:00.000-08:00

When refactoring the name of the GWT module (which typically involves changing the name of the Module.gwt.xml file, see this post) you also have to change the returned value from getModuleName() in every GWTTestCase:

This is way I usually factor the name of the module out into a utility class, and reference a static constant from every test case: saves a lot of pain and effort following some heavy refactoring.

If you do want to watch the video, here's a few tips:

if your connection is fast enough, choose 720p as the resolution, or the various menu items and text in general, will be very blurry;
this is best viewed full-screen;
a description as to how to create a reusable GWT Module can be found here

Change a GWT Module name

2010-11-11T22:27:00.005-08:00

This is really a trivial how-to, but wanted to try out how easy it was to capture Eclipse's window using xvidcap, and then upload to Youtube and embed the video here.

This was all relatively simple (from having the idea, to blog posted and tested, took around 15 minutes) so I plan to do a few more.

If you do want to see the video, here's a few tips:

if your connection is fast enough, choose 720p as the resolution, or the various menu items and text in general, will be very blurry;
this is best viewed full-screen;
do remember if you follow along, and you have unit tests, to change the module name there too (or see this video);

Implementing a Remote Service in Android - Part I

2010-11-07T13:25:00.001-08:00

When I decided that a particular idea of mine could be best implemented as an Android Service, running in the background, I found that there is not much information available on the web, beyond some very basic examples, android.com's API Demo sample, and the AIDL tutorial page.

However, those are rather "sparse" guidelines, and several gaps are left for oneself to fill out by trial and error: so I decided to post this brief how-to, along with the code itself.

As noted in the Android Developer's guide:

A service doesn't have a visual user interface, but rather runs in the background for an indefinite period of time. For example, a service might play background music as the user attends to other matters, or it might fetch data over the network or calculate something and provide the result to activities that need it. Each service extends the Service base class.

There are actually two ways of implementing a service: one is to just extend the Service class, implement the onBind() method and then implement the Binder interface in your service implementation class (this is the approach described here); the other is to use AIDL (Android Interface Description Language), followed by the API Demo sample (and further described in Part II of this blog - coming soon).

So, here we go: let's assume that we want to implement a simple service, which enables developers to add a simple UI element to their app to allow delighted users to make donations, but freeing them from having to implement all the machinery: our service will take a Developer Key and an Amount value ($ cents), and will credit the amount to the developer's account (we assume that the user's credentials can be retrieved from the system - how to do that is outside the scope of this blog); the app developer herself will only have to implement a simple UI (or a fancy complicated one: that's up to her) and connect to our service.

The service class definition is pretty trivial:

public class DonateService extends Service {
  @Override
  public IBinder onBind(Intent intent) {
    return new DonateServiceImpl(getResources());
  }
}

The service implementation itself is rather simple too:

public class DonateServiceImpl extends Binder { ... }

with all the 'action' being in its onTransact() method:

@Override
  protected boolean onTransact(int code, Parcel data, Parcel reply, int flags) {
    if (code != res.getInteger(R.id.SERVICE_CODE)) {
      Log.e(getClass().getSimpleName(), "Transaction code should be " +
          res.getInteger(R.id.SERVICE_CODE) + ";" + " received instead " + code);
      return false;
    }
    Bundle values = data.readBundle();
    String devKey = values.getString(res.getString(R.string.DEV_KEY));
    int amountInCents = values.getInt(res.getString(R.string.AMT));
    Log.i(getClass().getSimpleName(), getUser() + " wants to donate " +
        amountInCents + " to " + devKey);
    if (amountInCents <= 0) {
      Log.e(getClass().getSimpleName(), "Amount should be a positive integer (" +
          amountInCents + " is not).");
      return false;
    }
    Log.d(getClass().getSimpleName(), "Sending request to server");
    // This is where we would implement our HTTPS connection service, most likely to
    // some RESTful service
    
    return true;
  }

And this is pretty much all there is to it, as far as it concerns to handling a service call (please do read the description on the Service class, as well as the notes about a service's lifecycle: similar to an Activity, there is an onCreate(), onDestroy() etc.) and retrieving the data marshalled by the system, from the calling Activity.

If you now try to 'test' it by using the Android Instrumentation framework from a 'sibling' test project (you create one typically at the same time as you create a new Android project in Eclipse), your code should look something like this:

public void testServiceRuns() {
    Resources myRes = getContext().getResources();
    assertNotNull("The test case resources are null", myRes);
    
    Intent i = new Intent(ACTION);
    IBinder binder = bindService(i);
    assertNotNull(binder);
    
    Bundle values = new Bundle();
    values.putString(myRes.getString(R.string.DEV_KEY), "12345ABCDE");
    values.putInt(myRes.getString(R.string.AMT), 99);
    Parcel data = Parcel.obtain();
    data.writeBundle(values);
    
    try {
      int serviceCode = myRes.getInteger(R.id.SERVICE_CODE);
      assertTrue(binder.transact(serviceCode, data, null, 0));
      Log.i("test", "Service executed successfully");
    } catch (Exception ex) {
      Log.e("test", "Could not transact service: " + ex.getLocalizedMessage(), ex);
      fail(ex.getLocalizedMessage());
    };
  }

Funnily enough, this will work even if you mistype the name of the service's implementation class in the Android Manifest (AndroidManifest.xml):

<application android:label="@string/service_label" 
                     android:icon="@drawable/app_icon">
   <service android:exported="true" 
            android:name=".DonateService"
            android:process=":remote">
      <intent-filter>
         <action android:name="@string/ACTION_DONATE" />
      </intent-filter>
   </service>
</application>

(try changing .DonateService to .blah and this will still work) -- all this to say: the mechanics of Android unit testing are different from a remote invocation, and may succeed even though your service may be unavailable to other applications.

Notice in particular android:process=":remote" - this is what tells Android application manager to lookup your service's intent-filter too, when looking for possible targets, when another process invokes startService(Intent).

Finally, your test case should extends ServiceTestCase - but make sure you change the constructor, from Eclipse's auto-generated one to something like this:

public DonateServiceBinderTest(String name) {
    super(DonateService.class);
    setName(name);
  }

as explained elsewhere.

The <intent-filter> is the trick that does all the magic here: when matched against the Intent's 'action' will cause your service to be invoked.
Which brings us to the next topic: how do we invoke the service from a separately developed, completely independent application?

I have created a very simple Activity (download service_example_usage-0.2_beta from our site) that does exactly that.

In the following, I will largely ignore the niceties of building an Android UI, as these are widely explained elsewhere and are largely outside the scope of this post, and will focus instead on the specifics of calling a Service.

As a general rule, when invoking a Service, you must do it outside of your primary UI thread: for starters, you have no idea how long will it take to complete and, secondly, you want to give the users the ability to terminate it, should it take too long.

Hence:

Rule #1 - execute a service invocation from within a Runnable that executes outside your main UI thread, and set up the service call, so that it has a callback handler;

Which neatly leads to

Rule #2 - to handle the service outcome where this needs to influence changes in the UI (and it most likely will, even for the trivial task of notifying the user that the call succeeded/failed, whatever...) use a Handler, or your app will crash unceremoniously.

Let us dissect the code in SimpleApplication.java (activateService(final int amt)) one step at a time:

1. Visually notify the user that we're engaging in something that will take time to complete, and we aren't quite sure how long:

LinearLayout panel = (LinearLayout) findViewById(R.id.ProgressPanel);
    panel.setVisibility(View.VISIBLE);

the progress bar has been defined in the layout/main.xml file as 'indeterminate' and as a spinning wheel:

<ProgressBar android:id="@+id/ProgressBar" android:layout_height="wrap_content" 
                android:layout_marginLeft="15px" android:layout_width="wrap_content"
                android:indeterminate="true" android:indeterminateBehavior="repeat" 
                android:visibility="visible"/>

2. Create an intent, whose action will match the service's action's intent-filter (see note below[*] regarding sharing common strings between the service and its intended users):

Intent i = new Intent(getResources().getString(R.string.ACTION_DONATE));

3. Using this Activity's Context, bind it to a service that can service this Intent, by providing an implementation of a ConnectionService interface:

boolean isConnected = bindService(i, new ServiceConnection() {
     @Override
     public void onServiceConnected(ComponentName name, IBinder service) {
       Bundle values = new Bundle();
       values.putString(getResources().getString(R.string.DEV_KEY),
           getResources().getString(R.string.DEVELOPER_KEY_VALUE));
       values.putInt(getResources().getString(R.string.AMT), amt);
       Parcel data = Parcel.obtain();
       data.writeBundle(values);
       boolean res = false;
       try {
         res = service.transact(serviceCode, data, null, 0);
       } catch (RemoteException ex) {
         Log.e("onServiceConnected", "Remote exception when calling service", ex);
         res = false;
       }
       Message msg = Message.obtain(h, serviceCode, amt, (res ? 1 : -1));
       msg.sendToTarget();
     }

     @Override
     public void onServiceDisconnected(ComponentName name) {
     }
   }, Context.BIND_AUTO_CREATE);

4. Wrap the above into a Runnable, and then kick the Thread alive:

Thread serviceThread = new Thread(new Runnable() {
     @Override
     public void run() { //... }
   });
   serviceThread.start();

5. To enable this newly created thread to 'callback' your activity and carry out tasks inside the UI thread, you need to implement a Handler class, create a Message to wrap your returned results and then configure your handler as the message's target:

// outside the Runnable:
  final Handler h = new ServiceCompleteHandler(result, panel, ctx);
// this will be invoked inside the UI thread when called
// at the end of onServiceConnected, in the service activation thread
  Message msg = Message.obtain(h, serviceCode, amt, (res ? 1 : -1));
  msg.sendToTarget();

Your Handler needs to override the default handleMessage() method (that does nothing otherwise) and will be run by the System inside your UI thread (and will thus have access to your Activity's widgets, without causing a RuntimeException):

public static class ServiceCompleteHandler extends Handler {
  @Override
  public void handleMessage(Message msg) {
    int amt = msg.arg1;
    boolean outcome = msg.arg2 > 0;
    serviceComplete(amt, outcome);
  }

  void serviceComplete(int amt, boolean outcome) {
    panel.setVisibility(View.INVISIBLE);
    Rect r = new Rect(0, 0, 48, 48);
    Drawable icon;
    if (outcome) {
      icon = getResources().getDrawable(R.drawable.accepted_48);
    } else {
      icon = getResources().getDrawable(R.drawable.cancel_48);
    }
    icon.setBounds(r);
    result.setCompoundDrawables(icon, null, null, null);
    int dollars = amt / 100;
    int cents = amt % 100;
    result.setText(getResources().getString(outcome ? 
        R.string.thanks : R.string.sorry) + dollars +
        "." + cents + "\n" + getResources().getString(R.string.promo));
    result.setVisibility(View.VISIBLE);
  }
}

And this is pretty much all there is to it: sure, it's a bit more convoluted than just invoking a method in class, but we're talking here inter-process communication (IPC) and remote invocation (RMI) and they ain't pretty in any of the other frameworks either (think J5EE or, God forbid, CORBA).

In a subsequent post, I will show how to use AIDL to make invoking the service a more painless experience for the clients, but at the cost of having a slightly more convoluted development process on the service implementation's side.
---
[*]A note about re-using common strings (see the file res/values/donate_service_strings.xml): it is good practice not to use hard-coded strings in your service calls and data maps, and even better to have them in a commonly shared file (instead of duplicated everywhere in each applications strings.xml file).

Ideally, if you publish a service, you should make such a file freely available to your service users (eg, downloadable from your API docs site's pages); equally, if you use a service, either use the available strings file (hopefully made available from your provider -- if they don't, question your desire to use their service...) or partition those service-specific strings in its ownd dedicated file.

Eclipse users: while developing your service, you will certainly want to test it out with a simple app (very much like I describe here). The obvious way to heed the above advice would then be to add the 'strings' file as a "link" in Eclipse Package Explorer, so that any changes made during the service development, would be automatically picked up the 'testing' app.

This won't work (Android's plug-in does not pick linked .xml files in the res/... tree, to generate the R.java file). A simple workaround would be to create a soft link to the one file in the res/values/ directory:

$ ln -s ../DonateService/res/values/donate_service_strings.xml res/values/service_defs.xml

(you can give whatever name you wish to the link).
In the downloadable code, I've added a physical copy to the file in each package, to avoid people having troubles when re-using it: if you do use both to follow along and/or make changes, I recommend removing one copy and making a link as suggested here.

Unit testing Android Activity

2010-06-06T14:22:00.001-07:00

As I mentioned earlier, testing on Android is not for the faint-hearted (or the man-in-a-hurry) - documentation is very thin on the ground (although I've recently seen appear a few testing-related articles in the Developer documentation for the latest SDK - haven't checked them out yet, though) and the API is cumbersome at the best of times (and outright misleading, at the worst).

The "master class" to test an Activity class MyActivity, is ActivityInstrumentationTestCase2<MyActivity> that instruments and initializes it.

Its documentation is marginally better than that for android.test package:

A framework for writing Android test cases and suites.

but, hey, we haven't set the bar too high here.

Here are a few 'top tips' as how to avoid some of the grief, that I've discovered whilst developing unit tests for my AndroidReceipts application:

Do not use the 'default' constructor, but the one that uses ActivityUnitTest(String)
In Eclipse, you can right-click on a class in the project explores and then select New/Other... to create a new JUnit TestCase: in the ensuing dialog box, then select ActivityInstrumentationTestCase2 as the 'super' class, the plugin will helpfully auto-generate the full class with the following constructor:

// DON'T DO THIS - it won't compile
public MyActivityTest(String name) {
  super(name);
}

Here, Eclipse will (correctly) complain that there is no such thing as a ActivityInstrumentationTestCase2(String name) and you will quickly figure out that a (possible) super call may be something like this:

// DON'T DO THIS - it won't work
public MyActivityTest(String name) {
  super(PKG_NAME, MyActivity.class);
}

This will, sadly, cause the test runner (android.test.InstrumentationTestRunner) to happily ignore your test class:

[2010-06-06 22:33:20 - PolarisTest] Test run failed: Test run incomplete. Expected 51 tests, received 4

the trick is to add a call to setName(name) so that the test runner will find your tests (name, incidentally, is the name of the method being run).

// Do this instead:
public ScanActivityTest(String name) {
  // NOTE -- API Level 8 have deprecated this constructor, and replaced with one that simply takes the Class<T> argument
  super(PKG_NAME, MyActivity.class);
  setName(name);
}

If you are using the SDK 2.2 (L8) version, then there appears to be a new constructor that only takes the name of the Activity's Class under test (while the constructor shown above is deprecated): I have not tried it out, and targetting L8 devices, at the moment, rather severely restricts your target market.

Beware of Eclipse (ADB) missing a change in project source
The typical cycle is to write some code, run the tests, make changes, run the tests again.

This generally works, but, from time to time (I've been unable to discern a pattern) ADB misses a change in your source code and just re-runs the same tests as before.
As this typically happens when you make changes to the 'main' project (as opposed to the 'test' one) I suspect this happens when you do not save the modified source file, this in turn does not trigger a re-build of the APK, which change would have been picked up by the deployer of the 'test' project.

Be that as it may, keep an eye on the Console view and check that the a new version of the APK for the 'main' or 'test' (or both) projects gets installed on the emulator.

If you 'remove' the app manually, you also must 'clean' the project
On the same token, at times it turns out that the only way to get ADT out of its own hole, is just to go into the Emulator's Settings/Applications/Manage Applications and just remove either or both of the installed projects.

This will make the test runner deeply unhappy, and it will manifest its unhappiness by refusing to deploy the APKs and giving out an

[2010-06-06 19:28:31 - PolarisTest] Application already deployed. No need to reinstall.

The one way to quickly 'fix' this is to go into Project/Clean... and clean one or both of the projects.

The default tearDown() calls on onPause() (but not onStop() and even less onDestroy())
Well, I was surprised to find that out - it would have been reasonable to expect the test fix to run through the whole Activity lifecycle, and shut it down "gracefully.

In case you have some 'session management' (and who doesn't these days, in a serious Android app? you want to preserve state so that the user can come back to your app and find it exactly the way she left it) this may cause some surprising results when testing.

Considering that, in the Android process management system, there is no guarantee of a 'graceful shutdown' (essentially the scheduler wants to feel free to kill your proces without having to wait for your app to get itself sorted out -- and quite rightly so: we don't want a "Windows Experience" where some poorly-paid and even less-trained programmer can bring the whole system to its knees by sheer incompetence) this is just as well: in fact, the more I look into it, the more I find myself doing state management in the onPause / onCreate / onRestart lifecycle methods, and essentially ignoring the onStop and onDestroy (in particular the latter, I wonder sometimes why it's there at all)

Whatever, words to the wise: your Activity's onStop/onDestroy won't be called, unless you do it yourself.

Unit test run concurrently, but, apparently, @UiThreadTest prevents this for the UI thread
I must confess I'm not entirely clear about the full implications of using the @UiThreadTest annotation (apart from ensuring that the tests will run sequentially in the UI thread, thus avoiding a predictable chaos if they were all allowed to try to access UI resources concurrently) but one fundamental implication of this is that sendKeys cannot be called from within a test that is annotated with the @UiThreadTest.

Despite the documentation being conspicuosly silent about this minor detail, it turns that it must not be run in the UI thread; all we know about this method is that "Sends a series of key events through instrumentation and waits for idle." - whatever that means, the bottom line is that it does not return and your test will eventually fail with a timeout exception.

From my limited experimentation, the only workaround is to either use it only in tests thare are annotated with something such as @SmallTest or similar (but not @UiThreadTest) or to make it run in a separate thread (just create a Runnable that will execute once you are sure the UI elements have been initialized - critically, the layout has been 'inflated').

And, on this topic...

Activity.findViewById(id) - only use after you have 'inflated' the View (typically, by calling the setContentView() on an R.layout.my_layout resource)

Nothing much to add here, really, just be careful about how you sequence your asserts, if you need to verify conditions on UI elements (typically, Widgets) as the findViewById will invariably return null until the view is 'inflated' from the XML (if this is, indeed, the way you build your View).

On a similar note, I've found the Window.getCurrentFocus() (and similarly, Activity.getCurrentFocus()) pretty much useless: naively, I thought that, once the Layout had been inflated, the 'focused window' would have been the main container (or some random widget therein: that would have worked for my tests): in fact, this call most invariably returns null (unless, I presume, you sendKeys to move the focus where you want it to be: this is rather cumbersome, in my opinion, and makes the tests rather brittle and too tightly coupled with the UI layout, which, in my book anyway, is A Bad Thing).

So, here is what I do instead:

	@UiThreadTest
	public void testOnDisplayReceipts() {
  	  Receipt r = new Receipt();
	  instance.accept(r);
	  instance.onDisplayReceipts();
	  assertNotNull(instance.mGallery);

	  // verifies that the Gallery view has been 'inflated'
	  View gallery = instance.findViewById(R.id.gallery_layout);
	  assertNotNull(gallery);
	}

Ok, it's not pretty, I'll give you that, but it works (and is rather independent of what I do in my Gallery view, what widgets are there and how they are arranged).

Beware of super.tearDown()
Worth to feature in one of Bloch's Puzzlers, what does this code do, when run with an InstrumentationTestRunner?
You may also want to know that the test passes, and, upon exiting from testOnCreateSQLiteDatabase the stub field contains a valid
reference to the SQLiteDatabase just created and opened in its db package-visible field.


public class ReceiptsDbOpenHelperTest extends ActivityInstrumentationTestCase2 {

  public ReceiptsDbOpenHelperTest(String name) {
    super("com.google.android.applications.receiptscan", ScanActivity.class);
    setName(name);
  }

  public static final String DB_NAME = "test_db";
  ReceiptsDbOpenHelperStub stub;

  protected void setUp() throws Exception {
    super.setUp();
  }

  protected void tearDown() throws Exception {
    super.tearDown();
    if (stub != null) {
      String path = stub.db.getPath();
      Log.d("test", "Cleaning up " + path);
      if (path != null) {
	File dbFile = new File(path);
	boolean wasDeleted = dbFile.delete();
	Log.d("test", "Database was " + (wasDeleted ? "" : "not ") + "deleted");
      }
    }
  }

  /**
   * Ignore the details, but this does "work as intended," opens the database and returns a reference
   * to it in the db variable.
   * The {@code stub} is a class derived from {@link ReceiptsDbOpenHelper} and simply gives us access to
   * some protected / private fields
   */
  public void testOnCreateSQLiteDatabase() {
    stub = new ReceiptsDbOpenHelperStub(getActivity(), DB_NAME, null, 1);
    SQLiteDatabase db = stub.getReadableDatabase();
    assertNotNull(db);
    assertTrue(stub.wasCreated);
    assertEquals(stub.db, db);
  }
}

Well, you'll be surprised to know that, our tearDown() does absolutely nothing: after the call to super.tearDown(), stub is null (one can check it out using a debugger session; at least, that's what I did: upon entering ReceiptsDbOpenHelperTest.tearDown() stub is a perfectly valid reference, just after the call to super.tearDown() it's a null: apparently a call to super.tearDown() on an ActivityInstrumentationTestCase2<MyActivity> wipes out all the 'context-related' instance variables.

Yes, I was too.

The fix is, obviously, trivial: move the call to super.tearDown() to the bottom of your tearDown() (luckily, this is not a constructor, so there's no reason why not to).

Summary
There seems to be light at the end of the tunnel: some "official" documentation is starting to appear on Android.com, the SDK is (slowly) moving to be (marginally) more user-friendly, and writing (and running) unit tests is no longer as painful as it used to be in version 1.0.

However, one would have wished that it wouldn't have taken until release 2.2 of the SDK (and version 8 of the API) to get where we are now: for one thing, it's rather likely that Android will be the first (or main) programming platform that many kids will take up when starting to explore computing and software development - and whilst programming in Android is fun, productive and gives an immediate sense of accomplishment, I am concerned that, having given testing (and unit testing, in particular) such a back seat, the wrong lesson may be learnt by young computer scientists: namely, that testing (and coding for testing) is something that can be done another day, when we'll get on the next release...

It isn't - writing unit tests is vital to write bug-free, solid and portable code; it also encourages the design of clean APIs: there's nothing like writing a few tests to figure out that one's just written a cumbersome API that needs fixing: and the sooner one finds out, the better!

Blogged with the Flock Browser

Setting up a shared repository for Mercurial

2010-05-02T05:14:00.001-07:00

Mercurial -- Source Versioning Control

(this is also available as a Google Doc)

Mercurial -- Source Versioning Control

Use of Mercurial over SSH

Creating a private/public key pair
Setting up your username
Pulling a changelist from the remote repository
Making changes to the files in the (local) repository
Pushing changes to the (remote) repository
Pulling changes from the (remote) repository
Pulling / Pushing changes in Eclipse

Use of Mercurial over SSH

Please review the section named: Using the Secure Shell (ssh) protocol in Mercurial's "Definitive Guide" -- you may also find useful to review this blog entry (but there are differences with my 'paranoid' settings....)

I have configured my SSH port away from the standard sshd (22) and I'm using Dynamic DNS to use with my broadband connection (see http://dyndns.org for more info); further, I recommend only allowing SSH connections from a (limited) number of trusted IP (ranges) by editing /etc/hosts.allow; if you can't get this one to work, please figure out which IP address are you connecting from (http://www.formyip.com) and tweak your hosts.allow accordingly:

ssh -p 666 -l [user] yourdomain.dyndns.org

at this stage, you should be prompted for your password, that is to be expected.
If you can't connect, make sure you can DNS-lookup the server (nslookup yourdomain.dyndns.org must return something meaningful -- this is achieved via inadyn, so if you get an IP address, but still can't connect, it is -marginally- possible that inadyn is down and the IP was changed by your ISP).

Creating a private/public key pair

On the local Linux^[1] box do the following:

ssh-keygen -t rsa

[1] if are using Windoze, you're too far down in the evolution chain for me to waste any time educating you, but you can do worse than Google  "putttygen key pairs" or some such thing.

this will generate a private/public key pair in your ~/.ssh directory, one of which should be named something like id_rsa.pub -- copy that one (not the id_rsa private key!) on the remote machine's ~/.ssh directory (or in /tmp, it doesn't really matter). 
You must then add this public key to your ~/.ssh/authorized_keys file (if it does not exist, you will need to touch it, remember to chmod it to 500 -- only YOU must be able to rw it);

touch ~/.ssh/authorized_keys
chmod 500 ~/.ssh/authorized_keys
cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys

logout from the remote machine, then re-issue the ssh command above, you should then be logged in, without the remote SSH server asking for the password.

Setting up your username

It's always good practice to ensure that your changes can easily be tracked back to you: as an essential part of this, setting up a meaningful username for HG is critical: you can use the HGUSER environment variable (export HGUSER="Marco Massenzio <m.massenzio@gmail.com>") or set it up in a more 'permanent' way by editing the Mercurial configuration file (~/.hgrc):

# This is a Mercurial configuration file.
[ui]
username = Marco Massenzio <m.massenzio@gmail.com>

Pulling a changelist from the remote repository

The directory where we will place HG repositories is in /usr/local/hgrepo, each directory will be a project's repository -- a simple test repo is 'hello' (/usr/local/hgrepo/hello) and you should be able to clone it onto your machine by issuing the following (from a local shell -- do not ssh into the remote server):

hg clone ssh://user@yourdomain.dyndns.org:666//usr/local/hgrepo/hello

A couple of things worthy of note:

note the // after 666 (//usr) this tells SSH to start at the root of the filesystem (/) instead of your homedir (~/)
replace the username in user@ with your actual username -- do not send the password in cleartext, you should in fact not be asked for a password (if you are, then something went wrong with the private/public key pair setup above)
you should now have a hello directory, containing a few files on your local machine; if something went wrong, you should see an error message.

Making changes to the files in the (local) repository

Follow the usual HG practices -- edit the file with your favourite editor, save it, and once you are happy with it, commit it to the (local) repository:

hg ci -m "Did something astounding"

Pushing changes to the (remote) repository

Once you are happy with the changes, and think it's time for the other developers to see them, push them back to the remote repository:

hg push -v ssh://user@yourdomain.dyndns.org:666//usr/local/hgrepo/hello

By design, doing so, will not make the changes the 'tip' (or head) of the remote repository -- see the book as for why -- you should run an hg update command on the remote directory: the simplest way I found is to SSH into your remote machine and then run an hg update on the repo:

ssh -p 666 -l user yourdomain.dyndns.org
cd /usr/local/hgrepo/hello
hg status
# this should be an empty line, no editing of files should happen on the remote machine
hg glog -l 6
# see the 'graphic log' extension section in the hg manual
# this will show a list of the last 6 CLs, the 'active' one marked with an @
hg update
# this will make 'head' the 'active' revision

if no conflicts arise then this should update head; if there are conflicts, you will have to run a merge command:
hg merge
# I discourage doing merges on the remote machine however: merges are best done using a visual
# editor, suck as tkdiff, or similar.

possibly resolving the merge conflicts as they arise.

Pulling changes from the (remote) repository

Before starting any work, you should pull the latest version of the repository from ibw:

hg pull -v ssh://user@yourdomain.dyndns.org:666//usr/local/hgrepo/hello

By design, doing so will not make the changes the 'tip' (or head) of the local repository -- see the book as for why -- you should run an hg update command on the local directory:

hg up

if no conflicts arise then this should update head; if there are conflicts, you will have to run a merge command:

hg merge

Pulling / Pushing changes in Eclipse

Use the Mercurial Eclipse plugin [follow instructions on the site to install -- update site: http://www.vectrace.com/eclipse-update/] then use the right-click context sensitive Team menu with the Project folder selected, and choose accordingly.

The Repository URL to use is:
ssh://user@yourdomain.dyndns.org:666//usr/local/hgrepo/hello

but do not enter username/password in the dialog box, SSH will pick up the private key and proceed to conduct the handshake (if the pub/priv keys setup does not work for any reason, Eclipse will still be able to connect to the repository, but you will be asked for the password a couple of times, or possibly more).

Personally, I'm not using the Eclipse plugin (the initial delay in starting up at Eclipse launch is most irritating) and only use hg's command-line interface.