ajack.org

Quickly making files publicly accessible with Dropbox and Automator

2012-05-16T00:00:00+01:00

Quite often I want to take a screenshot, put it in my Dropbox's public directory and share the link with somebody. Typically this entails taking the screenshot, opening Finder, moving the file to Dropbox/Public, right clicking the file, choosing "Copy Public URL" and shraing the link. I knocked together a little Automator service to take some of the monotony out of it.

Open Automator and choose to create a new Service.

First, change the "Service receives selected" drop down to "files or folders" in "Finder". This ensures the service is only available in the context menu when clicking on items in Finder.

Choose "Files and Folders" from the library and drag "Copy Finder Items" into the main window. In the "To:" dropdown, select "Other..." and browse to your Dropbox Public directory. Check the "replace existing files" if that's what you feel like doing.

We'll need to get your public URL first though, so browse to your Dropbox directory in Finder, right click a file and hit Dropbox > Copy Public Link. Paste this somewhere and remove the filename from the end, you'll end up with something like "http://dl.dropbox.com/u/123456/" where "123456" is unique to your account.

Now choose "Utilities" from the library, and drag "Run Shell Script" into the main window, underneath the existing action. We're going to write a little script that will get the filename, append it to your Dropbox public URL and copy it to the clipboard.

First, make sure you change the "Pass input:" dropdown to "as arguments".

In the input field, paste in the following code replacing the URL in the first line with your public URL we got earlier.

dropbox_url="http://dl.dropbox.com/u/XXXXXX/"
echo -ne ${dropbox_url}$(basename $1) | pbcopy

That's it. Hit File > Save and give the service a name (I called mine "Copy to Dropbox").

Now, when you right click a file in Finder there'll be an option to use the service. Select it and the file will be copied to the public directory and the Dropbox URL for it will be copied to your clipboard. I should note here that while the URL will be immediately copied to your clipboard, the file will take a certain time to upload to Dropbox, so it'll probably not be ready immediately. I just check the Dropbox icon in the menu bar - if it's still flashing then the file's not ready yet.

I chose to make two versions of this, one which copies the file to the public directory and one which moves it. To move instead of copy, simply drag the "Move Finder Items" action from the library in the first step, instead of "Copy Finder Items"

Using Growl for long-running scripts

2012-01-10T00:00:00+00:00

Often it's nice to know when a long-running process has finished without having to continually check the terminal it's running in. I use this small function to call Growl to notify me when a command has finished.

function gn {
    /usr/local/bin/growlnotify -m "Starting \"$*\""
    $*
    if [ $? == 0 ]; then
        /usr/local/bin/growlnotify -m "\"$*\" succeeded"
    else
        /usr/local/bin/growlnotify -m "\"$*\" failed"
    fi
}

This simply wraps the command in a couple of calls to growlnotify. Add this to your ~/.bash_profile and simply prepend gn to your long-running command

source ~/.bash_profile
gn nvm install v0.6.7

Grabbing SMSs from a USB connected Android phone

2011-12-02T00:00:00+00:00

Since moving abroad I've found myself in the position of being one of those guys that own more than one phone. It's a privileged and elite club for sure but I'll save the praise for another day. Now having two phones is useful as I can keep in contact with people in both countries for the smallest payout (although using my UK phone in the Netherlands is obviously expensive). I tend to carry my Dutch phone around with me during the day and leave my UK phone at home. That usually works fine but it's annoying if I get a text on the phone I don't have with me. Since the phone I leave at home is always connected via USB to a small server that sits on my desk, I figured it'd be easy to grab the new messages off it and shoot 'em over the internet to wherever I might be. Turns out it is.

A note here: this technique almost certainly only works on rooted phones - I don't imagine you'd have access to the SMS database with a non-rooted device. For reference, I'm using a rooted T-Mobile G1 running CM5 (because I'm too lazy to update it).

The method here is pretty straightforward: use adb to grab the SMS database off the phone, use python and sqlite3 to parse its contents, and send an email. In reality though there was no need to grab the database - everything could be done over adb. The first step was figuring out where the database was and what data it kept.

The database is located at

/data/data/com.android.providers.telephony/databases/mmssms.db

and can be browsed using sqlite3 directly on the phone. The schema for the sms table is:

_id INTEGER PRIMARY KEY,
thread_id INTEGER,
address TEXT,
person INTEGER,
date INTEGER,
protocol INTEGER,
read INTEGER DEFAULT 0,
status INTEGER DEFAULT -1,
type INTEGER,
reply_path_present INTEGER,
subject TEXT,
body TEXT,
service_center TEXT,
locked INTEGER DEFAULT 0

Some interesting data, but all I'm interested in is address, date and body. Now that we've got this far, we can get all messages with one call of adb from the server, like so:

$ adb shell 'sqlite3 /data/data/com.android.providers.telephony/databases/mmssms.db "SELECT address, date, body FROM sms"'

This gets us all of the messages on the phone. Since I wanted to just get new messages, we can limit the SELECT to just return anything newer than some date. I've already read all the messages on there so I only want to see new ones. So I created a text file, wrote the current date in milliseconds (since that's the format Android uses to timestamp its SMSs) to it and used that to limit the database query.

adb shell 'sqlite3 /data/data/com.android.providers.telephony/databases/mmssms.db "SELECT address, date, body FROM sms WHERE date > \'SOME_DATE\'"'

I wrapped this up in a Python script and scheduled it as a cron job, each time it runs it writes the current date to the file, which is read back in each execution. The server that the phone's connected to has a small landing page on it which I use to monitor backups and the like, so I figured that'd be a nice central point to display any new messages.

The SMS is also delivered to my Gmail inbox which I typically have open, so I get a notification usually within a couple of minutes of receiving an SMS.

Polling the phone every couple of minutes isn't the most optimal solution but I don't need to know about SMSs in realtime, so approximate is good enough.

As usual the code is on GitHub so check it out.

Further work

The nice thing about this setup is that it's not just useful for notifying about new messages but can be used to control the server by sending the phone a certain string. For example, I could send a message of "restart apache" to the phone, check for that string in the Python script then perform the necessary action. Options!

Quick Gmailer

2011-10-06T00:00:00+01:00

At work I very rarely use my company-supplied computer, opting to use my personal machine with all my stuff on it. The only drawback to this is that I can't access the internal wiki or print. Drawbacks indeed. Since I typically have a terminal open, I wrote this script to send myself internal links that I might get via e-mail or files I want to visualise on some dead trees. It's very simple but means I don't have to go to Gmail and do it all manually.

It accepts a range of parameters depending on whether you want a subject, body, attachment or mixture. The following examples assumed the script is aliased to 'g' (which the chicks dig)

g 'http://www.some-obscure-internal-link.com'
g ~/Desktop/secret_documents.txt
g 'Some subject' 'Some body text....' /path/to/attachment
g # Read from stdin

Not much to say about it! Check it out at Github

Extracting single databases from MySQL dump

2011-03-27T00:00:00+00:00

If you've ever used mysqldump (or PHPMyAdmin) to make a dump of every database on a host to a single file, you'll know how much of a pain it is to then go through that file to see which databases it contains. A while ago I dumped all the databases on my old desktop so I could import them on my MacBook. I did a straight dump of all the databases in one go, then when it came to importing them I decided I'd probably like to go through and sort them out, deleting any I didn't want. The file came in at around 300MB and so choked any GUI text editor I tried to open it in. Vim coped with it fine, but it was a chore to go through the thousands of lines just to see what was going on. So I wrote a small python script that parses the file, displays a list of the databases it contains and allows extraction of single databases into separate files.

Simply pass it the path of the SQL dump, choose a number from the list and it'll be written out to a new file in the current working directory under the name (database).sql.

Get the script here or at Github.

A motion interface with HTML5

2011-03-26T00:00:00+00:00

I was looking at the HTML5 Rocks slides yesterday and came across the Device Orientation slide which allows a compliant browser to access the orientation data from an accelerometer on a supported device. Then I noticed the Chrome logo moving and saw that Chrome is able to access the sensor data from my MacBook's Sudden Motion Sensor. Awesome! I had no idea this data was available to the browser.

I remembered seeing a few articles a while ago (when Apple started putting the accelerometers in MacBooks) where people had got access to the raw data and used it for such things as turning their machines into lightsabres and changing desktops by hitting the sides of the laptop. For some reason the latter always stuck in my head so when I saw the device orientation event in HTML5, I thought I could mimic this behaviour. Getting the sensor data is very simple, first you need to create the event handler.

var orientationListener = function(event) {
    var alpha = event.alpha;
    var beta = event.beta;
    var gamma = event.gamma;
    // Do stuff here
}

You can then bind it to the "deviceorientation" event handler like so

window.addEventListener('deviceorientation', this.orientationListener, false);

Easy! For some reason, the alpha value is always null. The beta value represents tilting the device left and right (roll) and the gamma value towards and away from you (pitch).

I wrote a simple Flickr browser which uses left and right tilting to browse a set of images. Here's a video of it in action.

Fixing a Sansa Clip+ stuck in a boot loop

2011-03-21T00:00:00+00:00

This is a quick post to outline the steps needed to sort out an 8GB Sansa Clip+ that's stuck on the firmware boot screen. I have had Rockbox installed on my Clip+ for some time and everything has worked flawlessly. A couple of weeks ago, however, I was having trouble booting into either Rockbox or the default Sansa firmware. Rockbox would get stuck on the loading screen while the default Sansa firmware would get stuck on the "Refreshing your media" screen.

I wasn't able to reinstall Rockbox (which I figured to be the root of the problem) but I finally managed to get into the default firmware (if you've got Rockbox installed, hold left while powering the Clip on). The disk did not mount though so I couldn't update the Rockbox install. A quick look at dmesg showed there was a problem.

There was no dice mounting /dev/disk1 manually either, so I popped open Disk Utility to see what it had to say. After hitting "verify disk," I was confronted with a bunch of filesystem errors. Hitting "repair disk" sorted this out no sweat.

If you're running Linux, just hit up some fdisk-fu, it'll achieve the same results. After this my Clip was working like a damn pro. I expect this will work if you're just running the standard firmware, but I haven't tried it.

Writing an autonomous Android app

2011-03-20T00:00:00+00:00

I have recently been working on a project for my PhD that involved installing devices in people's homes in an attempt to change behaviours around recycling and food waste. In a nutshell, I needed to unobtrusively and reliably capture the contents of a kitchen bin. It was decided that the best way to do this was to augment the bin with a 3g or wifi enabled mobile phone which would be able to to the capturing, processing and transmitting of the data in a single, small device. Given that I have experience writing Android apps, and the fact that there is a multitude of different Android devices on the market, the Android platform was clearly the best option available to me. The overall aim of the project is to run a study with multiple student households in geographically different environments over a period of six weeks. The phones are securely encased within the bins, which makes them difficult to physically access, and the problem is further complicated due to the fact I have recently moved abroad as part of my PhD and so I am not even in the same country as the devices. This post serves to document my experiences in creating an autonomous Android app that requires little to no user intervention.

Crash reporting

The first thing I considered when starting to write the app was "what's going to happen when the app crashes?" Note the "when", not "if" – if you think there is no chance that your app is going to crash at some point, you're almost entirely wrong. So before I think about how I'm going to recover from the crash, I'm going to want to see some details about it. This includes - at a minimum - a timestamp, a unique device identifier and the stack trace (so we know exactly where the error occurred and why). A friend of mine introduced me to ACRA, a fully comprehensive crash reporting tool. Of course it's possible to implement this yourself (see details in this thread) but ACRA generates crash data and uploads it directly to a Google Docs spreadsheet, giving a nice overview of all crash information. Additionally, it's possible to set up e-mail notification when the spreadsheet has been changed, so you get informed of a crash reasonably quickly. The documentation for ACRA is very helpful and allows you to add crash reporting to your application within minutes.

Since I have four Android devices in my project, I need a way of identifying which one has crashed. ACRA supports adding custom fields to the report, letting me include the IMEI number of the device. I use the IMEI number throughout my code as a way of uniquely identifying the devices, using the following code.

public static String getIMEI(Context ctx) {
    TelephonyManager telephonyManager = (TelephonyManager)ctx.getSystemService(Context.TELEPHONY_SERVICE);
    return telephonyManager.getDeviceId();
}

We'll need the READ_PHONE_STATE permission to get the IMEI number, so add the following to your AndroidManifest.xml file.

<uses-permission android:name="android.permission.READ_PHONE_STATE" />

Adding it to the ACRA report is as simple as the following:

String imei = Util.getIMEI(this);
ErrorReporter.getInstance().putCustomData("IMEI", imei);

ACRA reports back a lot of information about a crash. Sometimes this can be a little overwhelming when looking at the spreadsheet, so I suggest hiding some of the columns that are irrelevant for you. Since I know a lot about my target devices (it's not an app that's released in the wild), I can hide several fields that don't tell me anything I don't already know.

Crash recovery

Now we have a nice way of visualising the crash report data, we should think about how to recover from a crash. Now Android has different ways of dealing with unexpected events. The system might generate an ANR (Application not responding) if it receives no response to an input event within 5 seconds (reference) or it might just straight up crash if an exception is thrown that is unhandled. This manifests itself in the all-too-commonly-seen force close dialog.

Since my app doesn't deal with user input at all, this is largely irrelevant. But what happens when my software generates an exception? I'm dealing a lot with bitmaps and network transmissions so there's plenty of scope for things to go wrong. This is where Thread.setDefaultUncaughtExceptionHandler() comes into play. Calling this method, and passing it your own UncaughtExceptionHandler allows you to get control in the case any unhandled exception occurs. This is a nice thing to have in place because we don't want the Android system showing a dialog saying the app is not responding and prompting the user to close it because the environment in which we're running the app is not user-oriented. Intervening in this process allows us to control what happens when an unhandled exception is thrown. The following code snippet shows how we can do this. I only have one core activity in my app that does all of the capturing and transmitting of the images, so I put this line in there.

Thread.setDefaultUncaughtExceptionHandler(new MyDefaultExceptionHandler(pintent));

That's all we need to handle those unhandled exceptions. (Note: I'm passing in a variable to the constructor, this will be explained later). Now we define our actual UncaughtExceptionHandler, like so:

public class MyDefaultExceptionHandler implements UncaughtExceptionHandler {
    private PendingIntent restartIntent;
    public MyDefaultExceptionHandler(PendingIntent pi) {
        this.restartIntent = pi;
    }
        
    public void uncaughtException(Thread arg0, Throwable arg1) {
        // Send the ACRA report
        ErrorReporter.getInstance().handleSilentException(arg1);

        // Restart the app using the PendingIntent
        AlarmManager mgr = (AlarmManager) restartContext.getSystemService(Context.ALARM_SERVICE);
        mgr.set(AlarmManager.RTC, System.currentTimeMillis() + 2000, restartIntent);
        System.exit(2);
    }
}

I'll outline what the rest of the code does below - of course, it's entirely up to you what you do when you encounter an unhandled exception but my actions are as follows.

Send an ACRA report

I want to report this exception as I would any other exception the app generates, which means sending an ACRA report to my Google Docs spreadsheet. Assuming you've set up ACRA as defined in the documentation, all you need to do is send a silent report.

ErrorReporter.getInstance().handleSilentException(arg1);

Restart the app

I want to restart my application right away, so the phone can continue to do its job. It's no good handling unexpected crashes then doing nothing. To restart the app, we can do the following. In my main activity which is running constantly, I create a PendingIntent that will remain in existence even if the activity that created it no longer exists.

PendingIntent pintent = PendingIntent.getActivity(this.getBaseContext(), 0, new Intent(getIntent()), getIntent().getFlags());

This PendingIntent is passed to my UncaughtExceptionHandler above so it can be used with the AlarmManager to restart my app. In the overridden uncaughtException() method, we get an instance of the AlarmManager class, which allows us to schedule events for the future. As the above code shows, an event is scheduled 2000ms in the future (just to give a bit of a pause before we start the app again) and then the PendingIntent is passed to the alarm manager so it's given something to do (i.e. start the PendingIntent). Finally we exit so the process is killed.

It is probably a good idea to only restart the app a certain number of times within a given timeframe. If there is a fundamental - and therefore consistent - problem with your code, your app is going to crash, restart, crash, restart, etc. continually. A simple rule saying "if we've restarted n times in the past m minutes, don't restart the app, just close it and send a notification". Maybe the phone needs a reboot, which is covered below.

We also want to make sure the app starts up when the phone boots. This way, if any manual intervention has to occur, we can just turn the phone off, turn it back on again and put it back in its housing. Once the phone boots up, the app will start automatically. To do this, we need to create a BroadcastReceiver which will receive a notification when the phone boots up. When it receives this notification, we can tell it to start the app. We'll need to create a new class for the BroadcastReceiver.

public class BootReceiver extends BroadcastReceiver {
    public void onReceive(Context ctx, Intent arg1) {
        Intent i = new Intent(ctx, MyActivity.class);
        i.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK); 
        ctx.startActivity(i);
    }
}

Each time the phone successfully boots, it'll create an intent to start MyActivity (you should replace this with the activity you want to start).

We'll need to add the following to the AndroidManifest.xml file now to create the intent filter. Additionally, we'll require the RECEIVE_BOOT_COMPLETED permission.

<receiver android:name=".BootReceiver">
    <intent-filter>
        <action android:name="android.intent.action.BOOT_COMPLETED" />
    </intent-filter>
</receiver>

<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED"/>

Conserving battery

Depending on your situation, your phone might not be constantly connected to a mains power source, in which case you'll need to minimise your battery usage. I used a couple of techniques to keep the power consumption down.

Turning off the radio

The majority of the time my app was just sitting there waiting for an event to happen in which the accelerometer triggered an action. Only then would a picture be uploaded to the server. It makes sense, then, that the radio only needs to be enabled when the transmission is happening, and it should be turned off the rest of the time. The simplest way to do this is to enable airplane mode when we don't need data connectivity, and disable it when we do. Airplane mode will cut all radio activity and therefore decrease battery consumption. We can write a function that enables airplane mode when we pass true, and disable it when we pass false.

private void airplaneMode(boolean enable) {
    Settings.System.putInt(getContentResolver(), Settings.System.AIRPLANE_MODE_ON, enable ? 1 : 0);
    Intent intent = new Intent(Intent.ACTION_AIRPLANE_MODE_CHANGED);
    intent.putExtra("state", enable);
    sendBroadcast(intent);
}

Dimming the screen

While this probably won't make much difference as the screen will turn off after a few seconds anyway, I kept the screen dimmed as much as possible while the app was running - no user intervention means nobody is going to be looking at the screen anyway.

public static void screenBrightness(Window window, float val) {
    WindowManager.LayoutParams lp = window.getAttributes();
    lp.screenBrightness = val;
    window.setAttributes(lp);
}

To set the screen brightness to its lowest value:

screenBrightness(getWindow(), 0.05f);

Status information console

I wanted a way to remotely view the status of each of my devices and provide me with a way to manage them to some degree. This is of course invaluable if physical access to the devices is infeasible. A web interface is the most obvious solution here, especially as Android makes it easy to interface with HTTP services. I drafted a list of information I'd like a quick overview of for the devices.

Time at which the app last started up
Battery level
Last activity
Available memory

I wrote a PHP script that handles the transmitting of the information - which are supplied via GET parameters - and updates a database. Again, I used the phones' IMEI numbers to identify each one. Each time the app starts, it sends the current timestamp to the server and each time it captures a photo, it uploads the photo and current timestamp, battery level, and available memory. The following code creates an HttpClient and executes a call to the PHP script in question.

HttpClient httpClient = new DefaultHttpClient();
HttpGet request = new HttpGet("http://www.my-server.com/update.php?action=an_action&imei=xxx");
httpClient.execute(request);

Needless to say, we'll need the INTERNET permission in order to do this, so this needs adding to the AndroidManifest.xml file.

<uses-permission android:name="android.permission.INTERNET" />

Time at which the app last started up

This information lets me know when the app was last started. This is useful to know as it lets me see when the last time the app crashed. Of course this information is available in the ACRA report but it's nice to have it on the console - which I visit more regularly.

Battery level

This is not necessarily a requirement but it's a nice thing to know. The devices are to be deployed into student houses (with a wall adapter so the phone has a continuous power source) but there's always the possibility that they'll get unplugged for whatever reason. So each time the phone uploads a photo, it'll also send the current battery level. If the battery level on any of the phones falls below a certain threshold, I get an e-mail notification and I can let the people in the household know that the phone has been unplugged. This the code you need to get the battery level.

public static double getBatteryLevel(Context ctx) {
    Intent batteryIntent = ctx.getApplicationContext().registerReceiver(null, new IntentFilter(Intent.ACTION_BATTERY_CHANGED));
    int rawlevel = batteryIntent.getIntExtra("level", -1);
    double scale = batteryIntent.getIntExtra("scale", -1);
    double level = -1;
    if (rawlevel &gt;= 0 &amp;&amp; scale &gt; 0) {
        level = rawlevel / scale;
    }
    return level;
}

You could do a nice visualisation to show battery level on your console, I just show the percentage. I should mention that this method will return a number between 0 and 1, so you'll need to multiply by 100 to get the percentage.

Last Activity

This is quite project specific, as my phones are taking pictures and uploading them - which provides the whole data source for the project - so I'd like to know if there's a lull in activity from any of the phones which might indicate something is wrong. To this end, each time an upload happens, I send the current timestamp to the server so I know how long ago the last activity was.

Available memory

As I'm dealing with bitmaps (and doing some resizing on the device), it's nice to know if the device is low on physical memory. Sending the amount of available memory on each image upload lets me see this. ACRA reports a the same data but only on a crash, this lets me see the memory availability on a more frequent basis.

Remote management console

Being able to see various bits of information about the status of each device is all well and good but it would be really nice if we could actually control the device remotely. Really the only feature I wanted from this was the ability to reboot a device. It would be handy also to be able to do this from within the app, if, for example, the app is restarting itself over and over again the phone could be rebooted to see if this solves the problem. Helpfully, the PowerManager class provides us with the handy reboot() method. Easy, right? Well yes and no. When I was exploring how to remotely reboot the device, I found this method and went about writing the code to do it. My strategy was simple: use the web console to provide a link to reboot the device which, upon clicking, updates a flag in the database for the phone. When the app on the device starts, it executes an HTTP GET request to a script on the server which, if I've clicked the "reboot" link, returns a given string (e.g. "reboot"). The phone gets the response from the server and if it equals "reboot", we reboot the device. Very simple but effective. It was only when I got to calling the PowerManager.reboot() method that I realised it's only available for API level 8 and higher (i.e. Android 2.2). The phones I chose for this project are Sony Ericsson XPeria X10 minis - chosen for their small physical size and having an on-board camera flash - which run Android 1.6. People have managed to put 2.1 onto the phones, but seemingly not 2.2, so I was really stumped. I am yet to come up with a method of rebooting the devices (short of rooting them, which is difficult since they're in a different country). This method, however, would work fine on 2.2 and above.

Further work

One feature I'm missing which would be extremely useful is the ability to do a remote update of the app. I've got all this infrastructure to inform me of problems and let me view device stats but no way of being able to really fix the problems. There's only one way I can think of to achieve this which involves a couple of issues for me: Android 2.2 and upwards support automatic updating of installed apps. However, this requires both devices running Android 2.2 (which mine do not) and also that the app is installed via the Android Market (which I wanted to avoid as it's not a publicly released app). The latter issue could be resolved by restricting the app to run on devices with specific IMEI numbers, and showing a message on other devices telling the user to uninstall the app as it will serve no purpose. This is a bit of a hack but it would solve the problem.

I suspect with a rooted device it would be possible to put the APK on a server and use the methods outlined above to check if an update is needed, download the APK and install it locally. Since my devices aren't rooted, I haven't looked into this but I expect if you can get shell access, you could do this.

Do a pilot study

It is best when embarking on a project like this to schedule a pilot study first. This is especially invaluable if, like me, your testing strategies aren't all that extensive. Running the software for a short period of time before the real study will help you iron out any problems you might experience, or add any additional features you think are required.

Conclusions

Android is a great platform for using in projects like this – the flexibility, ease of creating apps and feature-packed SDK really allow for software to be written that requires minimal, if any, user intervention. However, Android was not designed for such an environment and that sometimes gets in the way. The platform is intended to provide end users with a rich interactive experience – quite the opposite of what I wanted here. It's also created in such a way that attempts to protect the user from any unscrupulous activity from less-than-honest apps (that's a good thing!) which makes it more difficult when you're creating something that you know will only be deployed on one device and you'd like to get real system level access and do things that maybe wouldn't be looked upon too kindly if distributed to the entire Android population. Still, as I've outlined here, it's certainly not an impossibility.

If you're going to use Android devices for a similar project, I'd suggest using ones that can run 2.2 and upwards so you can remotely reboot it.

The pilot study for this project will begin in the coming week so I'll be able to see how well these ideas worked.

Firefox Usage Visualisation

2010-12-15T00:00:00+00:00

Recently I've been finding myself reading a lot about data visualisation and I've become quite interested in the many different ways it's possible to display data. I've fiddled a little with Processing in the past – I like the way it's super easy to show something tangible with very little effort (especially as graphics isn't my strong point). In my recent readings about visualisation, I came across the Mozilla Open Data Visualization Competition, a competition held by Mozilla to visualise data gathered from users participating in Test Pilot program. Test Pilot anonymously gathers data relating to various aspects of typical Firefox usage from users who have chosen to participate.

I spent a while thinking what parts of this data to visualise, then realised the deadline was a couple of days away so chose something reasonably quick and easy. I wanted to see how the open source nature of Firefox affected other devices and software its users choose. People use Firefox for many different reasons but I wanted to find out what the proportion of people using it due to the fact that it's open source. The surveys and data gathered in Test Pilot cover a wide range of aspects of Firefox usage but there was some data that was useful for this. First things first, the visualisation.

There were two aspects I was interested in: what smartphones are Firefox users using and what other browsers they use. The circular graphs on the left indicate which specific phones/browsers are in use and the pie charts on the right show the total proportion of these in terms of open and closed source.

It's clear from the smartphone graph that the majority of users don't use a smartphone (which I have to say surprised me). I should point out here that this question in the survey was optional, although mostly it was answered by everyone. The next largest proportion was the iPhone, followed closely by Android. With regard to software licensing, the open/closed source split was reasonably close. I categorised the operating systems as follows:

Open source: Android, Maemo, Symbian
Closed source: iOS, Windows Mobile, Palm, RIM

As for alternative browsers, Internet Explorer still clearly has a slight lead (again I should point out that Test Pilot is a survey conducted on users who use Firefox most probably as their primary browser and are "savvy" enough to know about, opt into and install Test Pilot, so it really isn't indicative of internet users on a broad scale at all).

It was slightly more challenging to categorise in browsers in terms of open/closed source. A lot of Chrome is open source (although potions aren't), so I decided that because the majority is, I would include it in that category. Internet Explorer and Safari clearly aren't and Opera is mostly proprietary (although has components that are open source) so all three go into the "closed source" category. I'm bring rather generic here by only having two categories (and not, for example, "Mostly closed source but with some open compents", etc. because I wanted to get a broad idea of whether open source software licensing was important to Firefox users). The pie chart clearly shows most other browsers being used are closed source.

So to summarise, it doesn't look like there is a strong relationship between using an open source browser and using other open source software in other areas, which indicates people typically use Firefox for other reasons.

Saving Flash Videos on Mac OS

2010-12-04T00:00:00+00:00

I recently bought a Mac and realised that saving a Flash video that's buffered in the browser is a bit more difficult than it is on my Ubuntu machine. Sometimes I like to watch a Flash video when I'm somewhere without net access so I use this technique reasonably often. Under Ubuntu, the FLV is nicely kept in /tmp and copying it to a different location for watching later is as easy as firing up Nautilus, browsing to /tmp and copying the file. (The file is removed when you close the relevant browser window). In Mac OSX it's buried somewhere under /private and is a bit of a pain to find manually. So I wrote this script to look through /private for flash files (actually only files that start with "Flash"), show them in a list with their file sizes and provide an easy method of copying them so they don't get removed when the browser window is closed. Note, ALL flash files will be listed, including small flash adverts, etc., which you probably don't want. This is why the filesize is listed – if it's a video it'll probably be a few megabytes (or more), as opposed to an advert which will typically be a few Kb.

Get the script here.

You'll need to change line 7 to an existing directory you want the files to be copied to. Use the script how you want; a suggested method is below:

$ cd /dir/of/script
$ chmod +x flash_getter.sh
$ sudo mv flash_getter.sh /usr/bin/flash_getter
$ flash_getter

Hospital Records podcasting with Python

2010-10-23T00:00:00+01:00

I've recently started listening to the Hospital Records podcast while at work when I can't think of anything to listen to on Spotify. Since I have no music on my work computer, and therefore no proper media player, I lack the ability to use the podcasting service properly and so I wrote a small Python script to do it for me.

./hospitalrecords.py --list

Shows a list of the recent podcasts in reverse chronological order. The numbers printed on each line are identifiers for each podcast.

./hospitalrecords.py --info 2

Show the date and tracklisting for the second podcast in the list.

./hospitalrecords.py --get 2

Downloads the second podcast in the list. The mp3 will be saved in the current working directory.

Get the script here.