Java Thinking

My first linux contribution is a disaster

paul — Sun, 24 Jan 2010 02:56:35 +0000

So I recently submitted a patch to the NetworkManager component: https://bugzilla.gnome.org/show_bug.cgi?id=607731 - but it turns out to be a duplicate because this problem was addressed in https://bugzilla.gnome.org/show_bug.cgi?id=600779 - by default Bugzilla doesn’t seem to include ‘resolved’ defects in the search results. You have to go to advanced search and select all statuses.

So, if my patch wasn’t needed, and this issue has been resolved, then why doesn’t the source code in trunk have this change? Bug 600779 mentions that service providers is deprecated, but by what? Ubuntu 9.10 still uses it, because when I edited /usr/share/mobile-broadband-provider-info/serviceproviders.xml it worked fine for me.

If service providers is deprecated, then why isn’t there something on the homepage? Bugzilla doesn’t seem to report which source files were changed in a bug - something I quite like in JIRA. See comment #1:

VirginInternet would still be used for tethering a phone, but I've added
VirginBroadband.  Thanks!

But what got changed? That would give me a clue.

Checking the version I’m running:

paul@paul-laptop:~$ dpkg -s mobile-broadband-provider-info
Package: mobile-broadband-provider-info
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 180
Maintainer: Ubuntu MOTU Developers
Architecture: all
Version: 20091009-0ubuntu1
Breaks: libmbca0
Description: database of mobile broadband service providers
Contains service provider specific settings of mobile broadband providers in
different countries.
Homepage: http://live.gnome.org/NetworkManager/MobileBroadband/ServiceProviders
Original-Maintainer: Antti Kaijanmäki (Wellark)

So if Ubuntu are still using it then why wouldn’t there be patches and maintenance releases?

Now, searching for Ubuntu specific information turns up NetworkManager provider information which doesn’t have the Virgin APN I need. So how do I fix this? Well, it says:

If you discover you need provider information from here or elsewhere, please file a bug against the mobile-broadband-provider-info package so we can improve the experience for all users to a level where all providers Just Work out of the box. https://bugs.launchpad.net/ubuntu/+source/mobile-broadband-provider-info

As today (16/01/09) bug #317860 at Launchpad cover a lot of Service Providers missing. (https://bugs.edge.launchpad.net/ubuntu/+source/mobile-broadband-provider-info/+bug/317860)

I’m was stunned to find out that this type of generic information was window manager specific (GNOME) now it looks like it is distribution specific (Ubuntu). If anyone can tell me the reasoning behind this, I’d love to know. Because right now, it looks crazy to me. Especially from a point of view that I just want to contribute something. It’s kind of weird that it’s even Operating System specific really, it really just should be a web service available to anyone and everyone, and the Operating Systems just update from it. Anyway… continuing the saga…

This bug report makes interesting reading. Apparently everything was fixed last year: https://bugs.launchpad.net/ubuntu/+source/mobile-broadband-provider-info/+bug/280490

It also says that the Virgin website documents the APN being VirginBroadband. But that page doesn’t exist any more. I can only find one reference to APN on their site, and thats the wrong one.

So, I’ve updated the Ubuntu wiki to add the Virgin APN to the list of providers. I wasn’t sure whether to update the Mobile Broadband Cards section with the Huawei e160e, since they’ve already got the 160 and 160g. But I can’t find any source code - I got as far as finding this: http://packages.ubuntu.com/source/lucid/mobile-broadband-provider-info - but the trail ends there.

Anyway, I’ve spent way too long on this. What started out as a simple patch has only shown me how little I know about the Linux environment, packages, and how the source is maintained. Maybe over time I’ll get more familiar with it all.

Week in Review - 2010-03

paul — Fri, 22 Jan 2010 07:22:00 +0000

Its been a good week. I got my Virgin Mobile Broadband Prepaid 3G modem working, and learnt a lot in the process, and actually contributed to the GNOME project: https://bugzilla.gnome.org/show_bug.cgi?id=607731

As part of this, I installed Pidgin so I could join the IRC channel and talk to developers, but I haven’t yet figured out how to view the history for a channel - so I can see what happened while I was offline…

It does surprise me though, that the database for mobile broadband providers is specific to the window manager (GNOME) instead of being a system level thing. I’m sure there are good reasons, and it probably doesn’t matter, but what do the KDE (and others) users do? Define it all over again?

Worse than that though is that Virgin don’t seem to make the APN easily available. I found it by searching the web, and finding it in forums, but why isn’t it in the FAQ on a Virgin site?

I didn’t anticipate how much I’d enjoy having access to the internet while training it back and forth from work (1hr train ride) but I’m loving it. I’m pretty sure the documentation that came with the modem says to keep it at least 15cm away from the body, so its attached via the provided usb cable, and sits in my bag on the floor a good distance away.

I’ve taken to using VLC Media Player as my favourite media player - it lets me use a minimal interface (video only, no controls displayed) to minimise real estate and has an ‘always on top’ option. This means I can put the video in a corner of the screen and program away - it works well.

I’ve also been playing with Wordpress:

customising the Thesis theme,
using a static page for the home page,
learning that I can turn commenting off for individual pages/posts
specifying excerpts so the teasers are better

Also, I’ve read up a bit on Joomla. I haven’t installed it yet, but I’ve been meaning to look into CMS alternatives to Wordpress for a while. It was frustrating understanding exactly what it could do until a friend pointed me to this resource : http://www.cmsmatrix.org/matrix/cms-matrix/joomla

Ubuntu, Virgin Mobile Internet Pre-paid (in Sydney), and the Huawei e160e

paul — Wed, 20 Jan 2010 21:12:34 +0000

I recently bought a Virgin Mobile Broadband (Australia) PrePaid USB modem - a Huawei e160e. I activated it over the phone, and when I plugged it in to my Dell Inspiron 1525 running Ubuntu 9.10, I could see that it was connected to the network (this modem flashes a blue light every 3 seconds when connected to the mobile network). But, try as I might, I couldn’t get it connected. Every time I tried, it would just disconnect.

Looking in the system logs, I’d see:

Jan 6 12:42:57 paul-laptop pppd[2846]: Plugin /usr/lib/pppd/2.4.4/nm-pppd-plugin.so loaded.
Jan 6 12:42:57 paul-laptop pppd[2846]: pppd 2.4.5 started by root, uid 0
Jan 6 12:42:57 paul-laptop pppd[2846]: Using interface ppp0
Jan 6 12:42:57 paul-laptop pppd[2846]: Connect: ppp0 <–> /dev/ttyUSB0
Jan 6 12:42:57 paul-laptop pppd[2846]: CHAP authentication succeeded
Jan 6 12:42:57 paul-laptop pppd[2846]: CHAP authentication succeeded
Jan 6 12:43:06 paul-laptop pppd[2846]: Modem hangup
Jan 6 12:43:06 paul-laptop pppd[2846]: Connection terminated.
Jan 6 12:43:07 paul-laptop pppd[2846]: Exit.

I tried out using usb_modeswitch, but I never really figured out how that is supposed to be used - and it looked to me like the system knew it was a 3G modem anyway.

Using another machine connected to the internet, I googled around and found an answer that worked.

A couple of notes from my experience:

This modem was not automatically detected - I had to manually start the ‘Network Connections’ to add the connection.
When selecting Australia and Virgin in the Network Connections wizard, it incorrectly set the APN to ‘VirginInternet’ - maybe this is for post-paid accounts, but I have a pre-pay, and the APN I use is ‘VirginBroadband’ - This is probably because for Virgin, there is only one plan to select (default) - how do we get more plans defined (i.e. post-paid and pre-paid)?
In the PPP settings, I needed to disable CHAP - doing this in Network Connections was enough, I didn’t have to edit /etc/ppp/options
The username and password don’t seem to matter

lsusb shows the device - interesting that it doesn’t label it e160e, but E220 / E270 ???:

paul@paul-laptop:~$ lsusb
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 002: ID 12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem / E270 HSDPA/HSUPA Modem
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 05a9:2640 OmniVision Technologies, Inc. OV2640 Webcam
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Now that I have it working, this is the system log from insertion of the USB device to connecting to the network:

Jan 21 07:37:54 paul-laptop kernel: [ 1281.336164] usb 2-2: new high speed USB device using ehci_hcd and address 2
Jan 21 07:37:54 paul-laptop kernel: [ 1281.479581] usb 2-2: configuration #1 chosen from 1 choice
Jan 21 07:37:55 paul-laptop kernel: [ 1281.581686] Initializing USB Mass Storage driver…
Jan 21 07:37:55 paul-laptop kernel: [ 1281.583023] scsi5 : SCSI emulation for USB Mass Storage devices
Jan 21 07:37:55 paul-laptop kernel: [ 1281.583231] usb 2-2: USB disconnect, address 2
Jan 21 07:37:55 paul-laptop kernel: [ 1281.583324] usbcore: registered new interface driver usb-storage
Jan 21 07:37:55 paul-laptop kernel: [ 1281.583331] USB Mass Storage support registered.
Jan 21 07:38:01 paul-laptop kernel: [ 1288.144163] usb 2-2: new high speed USB device using ehci_hcd and address 3
Jan 21 07:38:01 paul-laptop kernel: [ 1288.287791] usb 2-2: configuration #1 chosen from 1 choice
Jan 21 07:38:01 paul-laptop kernel: [ 1288.305526] scsi8 : SCSI emulation for USB Mass Storage devices
Jan 21 07:38:01 paul-laptop kernel: [ 1288.320440] scsi9 : SCSI emulation for USB Mass Storage devices
Jan 21 07:38:01 paul-laptop kernel: [ 1288.392549] usbcore: registered new interface driver usbserial
Jan 21 07:38:01 paul-laptop kernel: [ 1288.392579] USB Serial support registered for generic
Jan 21 07:38:01 paul-laptop kernel: [ 1288.392646] usbcore: registered new interface driver usbserial_generic
Jan 21 07:38:01 paul-laptop kernel: [ 1288.392650] usbserial: USB Serial Driver core
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406201] USB Serial support registered for GSM modem (1-port)
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406282] option 2-2:1.0: GSM modem (1-port) converter detected
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406451] usb 2-2: GSM modem (1-port) converter now attached to ttyUSB0
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406471] option 2-2:1.1: GSM modem (1-port) converter detected
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406561] usb 2-2: GSM modem (1-port) converter now attached to ttyUSB1
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406600] usbcore: registered new interface driver option
Jan 21 07:38:01 paul-laptop kernel: [ 1288.406604] option: v0.7.2:USB Driver for GSM modems
Jan 21 07:38:06 paul-laptop kernel: [ 1293.315709] scsi 8:0:0:0: CD-ROM HUAWEI Mass Storage 2.31 PQ: 0 ANSI: 2
Jan 21 07:38:06 paul-laptop kernel: [ 1293.325975] scsi 9:0:0:0: Direct-Access HUAWEI MMC Storage 2.31 PQ: 0 ANSI: 2
Jan 21 07:38:06 paul-laptop kernel: [ 1293.338298] sr1: scsi-1 drive
Jan 21 07:38:06 paul-laptop kernel: [ 1293.338641] sr 8:0:0:0: Attached scsi generic sg2 type 5
Jan 21 07:38:06 paul-laptop kernel: [ 1293.338970] sd 9:0:0:0: Attached scsi generic sg3 type 0
Jan 21 07:38:06 paul-laptop kernel: [ 1293.353699] sd 9:0:0:0: [sdb] Attached SCSI removable disk
Jan 21 07:38:17 paul-laptop pppd[2933]: Plugin /usr/lib/pppd/2.4.4/nm-pppd-plugin.so loaded.
Jan 21 07:38:17 paul-laptop pppd[2933]: pppd 2.4.5 started by root, uid 0
Jan 21 07:38:17 paul-laptop pppd[2933]: Using interface ppp0
Jan 21 07:38:17 paul-laptop pppd[2933]: Connect: ppp0 <–> /dev/ttyUSB0
Jan 21 07:38:17 paul-laptop pppd[2933]: PAP authentication succeeded
Jan 21 07:38:17 paul-laptop kernel: [ 1304.425235] PPP BSD Compression module registered
Jan 21 07:38:17 paul-laptop kernel: [ 1304.431699] PPP Deflate Compression module registered
Jan 21 07:38:24 paul-laptop pppd[2933]: Could not determine remote IP address: defaulting to 10.64.64.64
…

When the system wasn’t working (using CHAP and the wrong APN), you can clearly see the failure:

Jan 6 12:42:27 paul-laptop kernel: [ 332.937183] usb 2-4: new high speed USB device using ehci_hcd and address 3
Jan 6 12:42:27 paul-laptop kernel: [ 333.080647] usb 2-4: configuration #1 chosen from 1 choice
Jan 6 12:42:27 paul-laptop kernel: [ 333.176235] Initializing USB Mass Storage driver…
Jan 6 12:42:27 paul-laptop kernel: [ 333.179119] scsi5 : SCSI emulation for USB Mass Storage devices
Jan 6 12:42:27 paul-laptop kernel: [ 333.179385] usbcore: registered new interface driver usb-storage
Jan 6 12:42:27 paul-laptop kernel: [ 333.179392] USB Mass Storage support registered.
Jan 6 12:42:27 paul-laptop kernel: [ 333.180499] usb 2-4: USB disconnect, address 3
Jan 6 12:42:34 paul-laptop kernel: [ 339.752070] usb 2-4: new high speed USB device using ehci_hcd and address 4
Jan 6 12:42:34 paul-laptop kernel: [ 339.905181] usb 2-4: configuration #1 chosen from 1 choice
Jan 6 12:42:34 paul-laptop kernel: [ 339.921561] scsi8 : SCSI emulation for USB Mass Storage devices
Jan 6 12:42:34 paul-laptop kernel: [ 339.922840] scsi9 : SCSI emulation for USB Mass Storage devices
Jan 6 12:42:34 paul-laptop kernel: [ 339.948581] usbcore: registered new interface driver usbserial
Jan 6 12:42:34 paul-laptop kernel: [ 339.948610] USB Serial support registered for generic
Jan 6 12:42:34 paul-laptop kernel: [ 339.948706] usbcore: registered new interface driver usbserial_generic
Jan 6 12:42:34 paul-laptop kernel: [ 339.948710] usbserial: USB Serial Driver core
Jan 6 12:42:34 paul-laptop kernel: [ 339.970568] USB Serial support registered for GSM modem (1-port)
Jan 6 12:42:34 paul-laptop kernel: [ 339.970692] option 2-4:1.0: GSM modem (1-port) converter detected
Jan 6 12:42:34 paul-laptop kernel: [ 339.970862] usb 2-4: GSM modem (1-port) converter now attached to ttyUSB0
Jan 6 12:42:34 paul-laptop kernel: [ 339.970881] option 2-4:1.1: GSM modem (1-port) converter detected
Jan 6 12:42:34 paul-laptop kernel: [ 339.970976] usb 2-4: GSM modem (1-port) converter now attached to ttyUSB1
Jan 6 12:42:34 paul-laptop kernel: [ 339.971016] usbcore: registered new interface driver option
Jan 6 12:42:34 paul-laptop kernel: [ 339.971020] option: v0.7.2:USB Driver for GSM modems
Jan 6 12:42:38 paul-laptop pppd[2789]: Plugin /usr/lib/pppd/2.4.4/nm-pppd-plugin.so loaded.
Jan 6 12:42:38 paul-laptop pppd[2789]: pppd 2.4.5 started by root, uid 0
Jan 6 12:42:38 paul-laptop pppd[2789]: Using interface ppp0
Jan 6 12:42:38 paul-laptop pppd[2789]: Connect: ppp0 <–> /dev/ttyUSB0
Jan 6 12:42:38 paul-laptop pppd[2789]: CHAP authentication succeeded
Jan 6 12:42:38 paul-laptop pppd[2789]: CHAP authentication succeeded
Jan 6 12:42:38 paul-laptop kernel: [ 343.964939] PPP BSD Compression module registered
Jan 6 12:42:38 paul-laptop kernel: [ 344.026662] PPP Deflate Compression module registered
Jan 6 12:42:39 paul-laptop kernel: [ 344.925236] scsi 9:0:0:0: Direct-Access HUAWEI MMC Storage 2.31 PQ: 0 ANSI: 2
Jan 6 12:42:39 paul-laptop kernel: [ 344.925960] sd 9:0:0:0: Attached scsi generic sg2 type 0
Jan 6 12:42:39 paul-laptop kernel: [ 344.930619] scsi 8:0:0:0: CD-ROM HUAWEI Mass Storage 2.31 PQ: 0 ANSI: 2
Jan 6 12:42:39 paul-laptop kernel: [ 344.942337] sd 9:0:0:0: [sdb] Attached SCSI removable disk
Jan 6 12:42:39 paul-laptop kernel: [ 344.957198] sr1: scsi-1 drive
Jan 6 12:42:39 paul-laptop kernel: [ 344.957525] sr 8:0:0:0: Attached scsi generic sg3 type 5
Jan 6 12:42:47 paul-laptop pppd[2789]: Modem hangup
Jan 6 12:42:47 paul-laptop pppd[2789]: Connection terminated.
Jan 6 12:42:48 paul-laptop pppd[2789]: Exit.
Jan 6 12:42:57 paul-laptop pppd[2846]: Plugin /usr/lib/pppd/2.4.4/nm-pppd-plugin.so loaded.
Jan 6 12:42:57 paul-laptop pppd[2846]: pppd 2.4.5 started by root, uid 0
Jan 6 12:42:57 paul-laptop pppd[2846]: Using interface ppp0
Jan 6 12:42:57 paul-laptop pppd[2846]: Connect: ppp0 <–> /dev/ttyUSB0
Jan 6 12:42:57 paul-laptop pppd[2846]: CHAP authentication succeeded
Jan 6 12:42:57 paul-laptop pppd[2846]: CHAP authentication succeeded
Jan 6 12:43:06 paul-laptop pppd[2846]: Modem hangup
Jan 6 12:43:06 paul-laptop pppd[2846]: Connection terminated.
Jan 6 12:43:07 paul-laptop pppd[2846]: Exit.

When I tried out usb_modeswitch, I didn’t really know what I was doing. I guessed that the parameters (-v and -p) would be those identified in lsusb. But as for how to know what mode it was in or how to switch it, I don’t know.

paul@paul-laptop:~$ usb_modeswitch -v 12d1 -p 1003 -H -i 1

* usb_modeswitch: tool for controlling “flip flop” mode USB devices
* Version 1.0.2 (C) Josua Dietze 2009
* Works with libusb 0.1.12 and probably other versions

Looking for default devices …
Found default devices (1)
Accessing device 000 on bus 002 …
Using endpoints 0×01 (out) and 0×82 (in)
Not a storage device, skipping SCSI inquiry

Device description data (identification)
————————-
Error: could not get description string “manufacturer”
Manufacturer:
Error: could not get description string “product”
Product:
Serial No.: not provided
————————-
Sending Huawei control message …
Error: sending Huawei control message failed (error -1). Aborting.

paul@paul-laptop:~$

UPDATE:

I’ve been wondering how I get another ‘plan’ into the Network Connection wizard. If my assumption about the APN is correct (that ‘VirginInternet’ applies to post paid, and ‘VirginBroadband’ applies to pre paid) then I’d want to add a plan called ‘Prepaid’ with the APN ‘VirginBroadband’ and if possible without CHAP.

Searching around, I found mobile-broadband-provider-info which, if you browse the source you’ll find there is a serviceproviders.xml file which contains all of the definitions.

Using

find / -name serviceproviders.xml

I see the file exists at

/usr/share/mobile-broadband-provider-info/serviceproviders.xml

Editing this file, I can add the following under the Virgin provider, in the Australia region (just search for Virgin):

guest
guest
Prepaid

Now, I can open Network Connections and add a connection, selecting my new pre paid plan, and all I have to do is remember to disable CHAP as an option.

I love how open this all is and how easily I could find all this out. It took only a couple of minutes, while on the train traveling home from work. Awesome.

Looking at the README file in the source lead me to the information about ServiceProviders which includes how to contribute.

Now I just need to email this info to the code maintainers so that others have an easier time connecting to Virgin. And, I still have to investigate if there is any way to disable CHAP via the config.

A bit of xmas admin support

paul — Mon, 28 Dec 2009 13:03:43 +0000

It started off simple enough: “Can you install Photoshop Elements please?” - A friend got Photoshop Elements for xmas, since it was plugged as an easy, effective way to manage and share photos. (Toshiba laptop, running Vista)

Installing it was easy enough, but when coming to share photos via email it got a bit scary because, he uses Yahoo mail, not a fat email client. When I started looking at it I though okay, I just need to set up Outlook with the mail server settings. Going into the help menu in Yahoo mail gave some pretty clear instructions on how to set Outlook up - but it didn’t work. It kept asking for the username and password, which never worked. Searching around the web showed people talking about POP access to Yahoo mail, some suggesting it was a paid feature. But the Yahoo documentation didn’t make any mention of needing a paid account.

After wasting too much time on this, I just set up gmail - I knew I could easily access POP and SMTP here. So after creating a gmail account I set the reply to email in Outlook to be the Yahoo account and told gmail to forward any mail to Yahoo. Thus the gmail involvement would be pretty much invisible. I’ve sent a test email to the gmail account and I’m still waiting for it to arrive at yahoo…

So now I can share photos via email okay. Great. Not particularly easy for the novice user, and how come I can’t just tell PSE to use a web mail provider? AND, it looks like PSE maintains its OWN list of contacts. Not related to the Yahoo or Outlook contact lists. Jeeezz.

Next, to upgrade AVG virus checker. I was stunned to find AVG Free wouldn’t let me switch the Firefox search box provider to Google. Apparently it switches your default to yahoo, and denies you when you try to change it. I think its all configurable, but thats a crazy thing to deny you by default. Crazy.

So, I download the 1MB installer. Which then proceeds to download who knows how many MBs - but it takes a LONG time over a mobile internet connection (Three). Then the install starts, and promptly stops telling me to uninstall Microsoft Live OneCare - which I do, which means a reboot, and then when I restart the installer, it downloads everything again. This time, I did not install the link checker hoping to avoid the “cripple my computer” feature.

The last thing to do - disable the touch pad when the usb mouse is plugged in - was easy (via the control panel mouse settings).

3 Hours later I’m done. But we’ve got a long way to go baby! This was way too hard. I hope I’m totally wrong - Perhaps there are easier ways, but they weren’t obvious and I don’t have another 8 hours to figure it out. If I was a billionaire, I know what I’d be doing - sorting this sh*t out!

A simple task queue

paul — Tue, 24 Nov 2009 11:15:13 +0000

I’m working on a little sample framework - really only to keep my sanity and practice my chosen craft - that allows you to string together tasks in a pipeline for processing. To exercise that framework - to flush out the pros and cons of the implementation - I’m writing a sample application.

The basic idea is that each task is interested in an event. That event could be the arrival of a file, or the completion of another task etc. To string tasks together, I’ve created a simple database queue system - as a task completes, it writes to the queue and then other tasks which are interested will see that event and then begin. Tasks complete either successfully (by returning no errors) or unsuccessfully (by returning more than one error).

An important concept is to be able to add (or remove) tasks from the pipeline programmatically and without changing the database schema. So, quite simply, when a task completes, that event is written to the queue - but how do we find which events another task is interested in? And how do we do this in a way that won’t take longer over time?

So, lets assume Job2 is interested in the successful completion of Job1. We need to find all successful Job1 events which haven’t already been processed by Job2 (successfully or otherwise):


SELECT * FROM QUEUE where job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where job='job2' and result is not null)

I’m no SQL ninja, so this is my first simple solution. It works, but it doesn’t scale. With a thousand rows, it completes pretty quickly, but with tens of thousands of rows it takes way too long (nearly 8 minutes with 40000 rows running on my Dell Inspiron laptop).

Pragmatically, we can restrict the query based on time - we don’t need to search the whole table, just the more recent events:


SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result='SUCCEEDED' and task not in (SELECT task FROM QUEUE where tstamp >= ? and  job='job2' and result is not null)

You could easily just look over the events in the last hour if your tasks are short running and you get less than a thousand events an hour - it’d probably perform okay. To give an indication of performance I’ve written a little groovy script to show some basic trends - included below.
paul@paul-laptop:~$ groovy sqltest 2005 rows took: 4 4005 rows took: 5 6005 rows took: 3 8005 rows took: 3 10005 rows took: 2 12005 rows took: 3 14005 rows took: 3 16005 rows took: 4 18005 rows took: 5 20005 rows took: 3

Notice though, that this is a great example of where indexes really help - without the index times get larger as the rowcount gets larger:
paul@paul-laptop:~$ groovy sqltest 2005 rows took: 13 4005 rows took: 34 6005 rows took: 123 8005 rows took: 134 10005 rows took: 229 12005 rows took: 237 14005 rows took: 330 16005 rows took: 342 18005 rows took: 428 20005 rows took: 442

Its a pity I have to worry about the time restriction, but it does make the solution workable and at least in my intended application, appropriate. I was hoping to use the SQL MINUS function, but it appears MYSQL doesn’t support it.

Here’s the groovy script I used to generate the results. The difference between using the time restriction and the index is so dramatic and obvious, it is a great example.


import groovy.sql.Sql
import groovy.grape.Grape

Grape.grab(group:'mysql', module:'mysql-connector-java', version:'5.1.10', classLoader: this.class.classLoader.rootLoader)

enum STATUS { SUCCEEDED, FAILED }

def go() {
	def sql = Sql.newInstance("jdbc:mysql://localhost:3306/spike", "spike","password", "com.mysql.jdbc.Driver")
	try {
	   sql.execute("drop table QUEUE")
	} catch(Exception e){}

	sql.execute("CREATE TABLE QUEUE (id INTEGER  NOT NULL,job varchar (20) NOT NULL,task varchar (20) NOT NULL,result VARCHAR (20) NOT NULL, tstamp timestamp, PRIMARY KEY (id)) ENGINE = MyISAM")
	sql.execute("create index idx1 on QUEUE(tstamp, job, result)")

	(1..10).each() {
		def d = createHistory(sql,1000*it)
		findJobsToProcess(sql, d)
	}
}

def findJobsToProcess(sql, d) {
	def rowcount = 0
	sql.eachRow("select count(*) from QUEUE") { row ->
		rowcount = row[0]
	}

	long start, end

	start = System.currentTimeMillis()
	// now find the 'job1' items that haven't been processed by 'job2'
	sql.eachRow("SELECT * FROM QUEUE where tstamp >= ? and job='job1' and result=? and task not in (SELECT task FROM QUEUE where tstamp >= ? and  job='job2' and result is not null) order by tstamp asc", [d,STATUS.SUCCEEDED.name(),d]) {
//	    println "${it.id} ${it.job} ${it.task} ${it.result} ${it.tstamp}"
	}
	end = System.currentTimeMillis()
	println "${rowcount} rows took: ${end-start}"

}

def createHistory(sql, count) {
	sql.execute("truncate table QUEUE")
	int id = 1
	int task = 1
	// create a "history" of previous jobs
	STATUS.each() { status ->
		(1..count/2).each() {
			sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),status.name(), new Date()])
			sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job2',task.toString(),status.name(), new Date()])
			task++
		}
	}
	sleep(1000)
	def d = new Date()
	// insert some successful 'job1' items which 'job2' hasn't processed
	(1..5).each() {
		sql.execute("insert into QUEUE (id,job,task,result,tstamp) values (?,?,?,?,?)", [id++,'job1',task.toString(),STATUS.SUCCEEDED.name(), new Date()])
		task++
	}
	return  d
}

go()

In running this test, I used:

Dell Inspiron 1525, Intel(R) Core(TM)2 Duo Processor T5550, 1.83 GHz, 2MB Cache, 667 MHz FSB, 2GB (2 X 1024MB) 667MHz Dual Channel DDR2 SDRAM, 160GB 7200RPM Performance Hard Drive
Ubuntu 9.10 32 bit desktop edition
Groovy Version: 1.6.5
JVM: 1.6.0_03
mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (i486)

Rapache on Ubuntu 9.10

paul — Sun, 15 Nov 2009 05:02:24 +0000

I’ve just now stumbled across Rapache, a useful GUI tool that makes configuring apache easy. I found it by accident in the Ubuntu Software Center, but unfortunately it would freeze while trying to add a new domain. I searched the web for answers, and found a bug report.

This didn’t specifically reference Ubuntu 9.10 (rather, 9.04) and the file that needed to be patched didn’t exist in the given location.

I found it easily enough:

paul@paul-laptop:~$ sudo find / -name RapacheGui.py[sudo] password for paul:
/usr/lib/pymodules/python2.5/RapacheGtk/RapacheGui.py
/usr/lib/pymodules/python2.6/RapacheGtk/RapacheGui.py
/usr/share/pyshared/RapacheGtk/RapacheGui.py
paul@paul-laptop:~$

I edited the last one (/usr/share/pyshared/RapacheGtk/RapacheGui.py) as documented in comment 23 to add the following at line 79:

        if not Shell.command.ask_password(): sys.exit(1)

Note, this line MUST be preceeded by 8 spaces - indentation is important in Python.

Now, rapache would prompt for the system password and then close!

I’ve got it working now, by starting it with sudo:

sudo rapache

Everything seems to work. I’ve added a domain, turned on the include module, and added “AddOutputFilter INCLUDES .html” to the virtual host definition - all very quickly using rapache.

So, server side includes now work on my new virtual host - which is what I started out wanting to do before going off on this tangent.

Tagging with Appengine DataStore

paul — Fri, 06 Nov 2009 11:54:21 +0000

I’ve been looking into how to implement a tagging mechanism with Appengine (Python). By using a StringListProperty, you can associate a list of tags with an entity. The model would look something like this:

from google.appengine.ext import db

class Sample(db.Model):
  name = db.StringProperty(required=True)
  tags = db.StringListProperty()

Now, assuming we want to find the entities tagged with a given word, we can use a query like this:

q = db.GqlQuery(”SELECT * FROM Sample where tags = :1″,’z')

If we want to find all entities that have ANY of these tags (OR) we can query with GqlQuery or filter:

q = db.GqlQuery(”SELECT * FROM Sample where tags in :1″,['a','z'])

q = domain.Sample.all()
q.filter(”tags in “, ['b','d'])

When we want to find entities with ALL of the given tags:

q = db.GqlQuery(”SELECT * FROM Sample where tags = :1 and tags = :2″,’b',’c')
q = domain.Sample.all()
q.filter(”tags = “, ‘b’)
q.filter(”tags = “, ‘c’)

Too easy!

If you have an appengine project, you can try this code out by adding the entity (Sample - listed above) to your domain model and run the following in your development console (http://localhost:8080/_ah/admin/interactive):

import domain

domain.Sample(name='t1',tags=['a','b','c']).put()
domain.Sample(name='t2',tags=['b','c','d']).put()
domain.Sample(name='t3',tags=['c','d','e']).put()
domain.Sample(name='t3',tags=['x','Y','z']).put()

def display(r):
 for r in results:
  print r.name +" tags = "+' '.join(r.tags)

print "FIND WITH THIS TAG"
q = db.GqlQuery("SELECT * FROM Sample where tags = :1",'z')
results = q.fetch(5)
display(results)

print "FIND WITH ANY OF THESE TAGS (OR)"
q = db.GqlQuery("SELECT * FROM Sample where tags in :1",['a','z'])
results = q.fetch(5)
display(results)

print "FIND WITH ANY OF THESE TAGS (OR)"
q = domain.Sample.all()
q.filter("tags in ", ['a','z'])
results = q.fetch(5)
display(results)

print "FIND WITH ANY OF THESE TAGS (AND)"
q = db.GqlQuery("SELECT * FROM Sample where tags = :1 and tags = :2",'b','c')
results = q.fetch(5)
display(results)

print "FIND WITH ALL OF THESE TAGS (AND)"
q = domain.Sample.all()
q.filter("tags = ", 'b')
q.filter("tags = ", 'c')

results = q.fetch(5)
display(results)

This should produce the following result:

FIND WITH THIS TAG
t3 tags = x Y z
FIND WITH ANY OF THESE TAGS (OR)
t1 tags = a b c
t3 tags = x Y z
FIND WITH ANY OF THESE TAGS (OR)
t1 tags = a b c
t3 tags = x Y z
FIND WITH ANY OF THESE TAGS (AND)
t1 tags = a b c
t2 tags = b c d
FIND WITH ALL OF THESE TAGS (AND)
t1 tags = a b c
t2 tags = b c d

So, it’s easier than I thought. But I’m not experienced with DataStore yet, and I don’t know if there are inherit limitations with this approach - remember, there are limits on the way you retrieve data.

No module named _multiprocessing

paul — Mon, 26 Oct 2009 10:13:03 +0000

I’m getting back into appengine at the moment, and I came across this error:

No module named _multiprocessing

A google search turned up this error report:

http://code.google.com/p/googleappengine/issues/detail?id=1504

which lead to this post:

http://code.google.com/p/soc/wiki/GettingStarted

The problem was of-course, that I was using Python 2.6 instead of 2.5.

I love Linux, the solution is trivial:

The workaround for this is to install Python 2.5 on your machine. It will be then be accessible by using python2.5 on the command line. Then open thirdparty/google_appengine/dev_appserver.py in your favorite text editor and replace the first line,

#!/usr/bin/env python

with

#!/usr/bin/env python2.5

Note, I already had Python 2.5 installed via:

sudo apt-get install python2.5

Now, I’ll promptly get back to figuring out how best to use DataStore to implement my solution!

Amazon web services via Grails

paul — Sat, 10 Oct 2009 04:15:21 +0000

I just recently noticed that Amazon have changed the way you invoke their Product Advertising API - you now need to sign requests as described in the Example REST Requests.

I obviously haven’t been reading my emails, since the first clue came when I saw:

MissingParameterThe request must contain the parameter Signature.

Luckily, they have a java example demonstrating how to make an API request.

If you extract the source from this sample into your /src/java directory, then you can invoke the amazon api with code like:


import org.codehaus.groovy.grails.commons.ConfigurationHolder
import com.amazon.advertising.api.sample.*

class AmazonService {
  private static final String ENDPOINT = "ecs.amazonaws.com";
  private static final String AWS_SECRET_KEY = "put your secret key here";
  private static final String AWS_ACCESS_KEY_ID = "put your access key here";

  boolean transactional = false

  /**
   * Call amazon lookup api.
   * Returns xml document.
   */
  def getData(String asin) {
    SignedRequestsHelper helper = SignedRequestsHelper.getInstance(ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_KEY);

    String requestUrl = null;
    String title = null;

    Map params = new HashMap();
    params.put("Service", "AWSECommerceService");
    params.put("Version", "2009-03-31");
    params.put("Operation", "ItemLookup");
    params.put("ItemId", asin);
    params.put("ResponseGroup", "Small,Medium");

    requestUrl = helper.sign(params);

    def xml = new URL(requestUrl).text
    return new XmlSlurper().parseText(xml)
  }
}

You could, of-course, define your access keys in the Config.groovy class if you prefer.

(To find your identifiers, see http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.html?ViewingCredentials.html)

Locked subversion working copy

paul — Fri, 09 Oct 2009 02:06:22 +0000

I don’t know how it happened, but my local working copy got locked. This meant that every time I tried to commit, I got a ‘working copy is locked’ error.

The solution was easy, after I found a reference in the FAQ:

svn cleanup