Unfortunately it appears to be yet another Australian site reinventing the open-data-catalogue-wheel. So far we have:

• data.gov.au: custom WordPress site
• data.brisbane.qld.gov.au: custom WordPress site
• data.nsw.gov.au: custom PHP (Drupal?) site
• data.mosman.nsw.gov.au: custom proprietary site
• data.vic.gov.au: custom proprietary site

Open Source Data Catalogues

How come none are using and contributing to an open source data catalogue like CKAN or Open Data Catalog? These projects are so much more advanced than the sites that have been deployed in Australia and it’s no wonder – CKAN’s maturity is because it has been developed for over four years.

Not only that but using an open source open data catalogue simply means it’s cheaper for Australian governments to provide these services. If data.gov.au had used and contributed to an open source data catalogue we’d probably have had the vast majority of Australian local customisations done (if any were needed) and subsequent catalogues by Victoria, NSW and now councils would border on trivial to set up and deploy.

Doing It Right

The open source catalogues I mention above also get some key things right that Australian sites deployed so far don’t – namely open and public data requests and an API (oh the irony of open data catalogues not having a good any API!).

Open and public requests are critical and would highlight the lacklustre data released to date in Australia. The Brisbane catalogue mentioned above is a prime example, there’s a clear and demonstrable need for development application data to be opened and freed but it hasn’t been chosen as one of the items to be included in the data that launched with the site. If there was a way for data developers to make public requests and not a glorified contact form we might see this data opened at some stage. As it stands, and while there’s a scraper gathering the data that simply ignores the onerous conditions, I doubt we’ll see this.

How do we fix it?

However there’s no point in moaning without suggesting a way to fix things. One way the open data community in Australia could promote the use more advanced open source data catalogues would be to simply set up our own one. A system like CKAN can slurp in data from all of the existing catalogues, thereby providing a comprehensive centralised source of all Australian open data (not just one level of government and not just community projects like OpenAustralia). This would also provide an API to that data and a central place where everyone can log requests for open data and track what data is really in demand by open data hackers in Australia.

Maybe someone should take this on as a project for #odhd? ;)

tl;dr

The problem

To get my regular news many years ago I used to visit SMH.com.au every few hours and see what was happening in the world. This worked well for a number of years until SMH started to dramatically decline in quality compared to the print version of the Sydney Morning Herald. This decline appears to have been driven by the analytics that Fairfax was getting on news stories based on more bored office workers reading salacious stories instead what I’d consider more useful journalism (some top stories on the SMH front page as I write this are “Suds to the blokes” and “France’s femme fatale“).

As my use of RSS grew, I sought out a different approach to getting my news and set up hand-picked RSS feeds from the ABC. Once again this worked well for a few years until a couple of months ago the ABC updated their RSS feeds and inexplicably changed all of the URLs to the RSS feeds from nice “/news/sydney.rss” to “/news/feed/10232/rss.xml” without providing redirects. This meant I would’ve had to go through and update each of my feeds manually which was not a huge task but gave me pause for thought.

It was around this time that my girlfriend decided to switch off from news completely. She was sick of the fact that in Australia our “national debate” has seemed to be fixed on the same issues rehashed over and over and over for the last few years (think refugees, climate change, etc.), which got me thinking how I was consuming news.

As my feeds contained every single story for a particular topic or geographic region it was quite a firehose of news items and a lot to get through. Being a feed, I’d be constantly be seeing new items flow in and the temptation was to sit there reading until I’d read each item I was interested in, by which time there’d often be even more stories that had flowed into the feed.

This didn’t feel right – it felt like the news was consuming me, rather than me consuming the news.

A solution?

A couple of weeks before all of this had happened I’d come across Phil Gyford’s Today’s Guardian, which he wrote to address some specific shortcomings he’d identified with online news – Friction, Readability and Finishability. Take some time to read his post and understand Phil’s motivations.

For me the idea of finishability was really appealing as I was sick of the firehose of information. I didn’t want every news story, I wanted the best news stories and when I’d read them I wanted to know I could switch off.

Rage breeds innovation

This all lead to me porting Phil’s code to work with the ABC here in Australia and you can check out the result that gets updated each day at 08:00:

http://daily-paper.henaredegan.com/

I’ve had it running for a couple of weeks now and whilst I tried to use it for the first week or so I’m not finding it fitting in with my routine so I never get around to checking it. The funny thing is I don’t feel any less informed and I seem to be just as aware of the day’s events as the next person. My currently theory behind this is that other channels must be doing a good enough job at getting news in front of me – things like Twitter or links that friends send me directly over IM or email.

I’m not sure what to make of this all but I’d be very interested to hear how you’re using online news and if something like Phil’s daily paper would help you.

I wrote a little post about creating a PlanningAlerts scraper for the Northern Territory for my mum.

We ran an OpenAustralia hackfest last weekend and I wrote a blog post that was cross posted on the Official Google Australia Blog.

I wrote a guest post for ScraperWiki about our hackfest.

And I gave a talk at the hackfest:

and wrapped up the hackfest:

ScraperWiki also featured my ACMA scraper a while back (I must get back to that project):

Why the wrong horse? Well seven years ago WordPress wasn’t in the game but I do remember evaluating Drupal and whilst it has a vibrant, active community the same cannot be said for phpWebSite.

I wanted to give our site a visual refresh, make it easier for people to contribute and to move to a more secure platform than the out of date version of phpWebSite we were running on. The obvious choice was WordPress.

I’ve looked before for tools to migrate from phpWebSite to WordPress but never found anything so I decided to write a tool myself. As I was getting started writing a tool, the friend I was working on the migration with discovered a CSV importer plugin already written for WordPress so we decided to see how hard it would be to export data from phpWebSite as CSV that this plugin could understand.

As we didn’t have huge amounts of content it turned out to be much easier to export the Announcement posts as CSV using phpMyAdmin and manually recreate everything else (just a handful of comments and some image galleries).

The trick with exporting the Announcement posts was to use the CSV for MS Excel option of phpMyAdmin and then manipulate the data using LibreOffice Calc into the format expected by the CSV Importer plugin.

Since we only had a handful of comments I simply recreated these using the standard WordPress UI and manually set the dates to match phpWebSite. Photos are stored under images/<module name>/ so I just copied the images/photoalbum directory and uploaded all the images in each gallery using the usual WordPress uploader.

In the author’s announcement of the bot he said,

The idea of overhearing machines talking about what they’re doing is, to my mind, quite delightful.

and I tend to agree.

So I thought I’d look at setting something up locally. I live close to the flight path so while I was pondering this a plane flew over and I thought I’d set up a bot for the Sydney Airport.

They have listings of arrivals and departures so I hacked together a bot and this was the result:

So why is this a failed experiment? Well after running it for a time I realised that the source data, scraped from the arrivals and departures page of the Sydney Airport site, didn’t really match what I was trying to do with it.

The data is focused on providing information to passengers, not an accurate picture of what really is coming and going from the airport.

The lesson? Data can’t always be hacked to suit other purposes and if it’s the foundation of a project, as it so often is, then spend the time ensuring it matches up with what you’re trying to do.

That said, I don’t consider this a waste of time. Failure’s part of the process.

We’ve written a bunch of scrapers that are being used by PlanningAlerts and I’ve also created a few for other bits of data I am interested in or that I’ve been commissioned by clients to write.

For the second time, my creations have been featured on the front page, isn’t that nice? :)

Here’s three simple steps that will massively increase your online security. It won’t make you bullet-proof and you should still do all those other things security people have been telling you for years but these are really easy to do, anyone can do them and they plug the biggest security hole that people have these days – insecure and reused passwords.

Step 1. Create a complex password

Wait! Don’t stop reading because you think it’s going to be hard to remember – complex passwords can be easy to remember.

A software tool called pwgen can generate complex passwords that are pronounceable and therefore easier for us humans to remember.

There’s an online version you can use to generate these. Go off and generate one now and use it for your most important online tool, such as your Gmail account.

Step 2. Start using a password manager

If you’re reading this, chances are you have heaps of username and password combinations for all of the online services you use. To manage these you need to be using a password manager.

Password managers generate complex passwords for services that you don’t need to remember. When you open your password file you just enter one master password and it unlocks all of your other varied, complex passwords.

I use KeePass because it is open source and works on the different platforms I use (Linux and Android). I also use Dropbox to ensure it’s synchronised between all my computers and is always available for download over the Internet.

Go and download it now and use the password you generated in the last step as your master password.

Step 3. Move all of your passwords into your password manager

This sounds hard but the trick is not to go around laboriously changing all of your passwords up front. Whenever you touch a service (e.g. login to Twitter, upload photos to Flickr, etc.), just go to your settings page and use your password manager to generate a new password. It’ll take you 30 seconds.

Off you go – go and change a few now. You’ll feel more secure right away.

Update:

Jonathan Holmes responds, “easy! You don’t understand how freaked out people like me get when confronted by something like KeePass“. That’s a fair point and I feel embarrassed I’ve done the stereotypical geek thing and assumed something easy for me wouldn’t be difficult for a non-geek.

I’m off to eat some humble pie :) In the mean time, hopefully these instructions at least inspire some geeks to fix up their own online security and, in finest nerdy tradition, show their friends and family.

Given that I’ve not delved too deep into Drupal and certainly never developed on it, I thought I had better find out a bit about it before the conference.

I decided to write a module similar to oa4wp that displays your MP’s most recent speeches in a block. After a few hours hacking on Sunday afternoon, here it is:

OpenAustralia Drupal in a content block

I was pleased to get something usable in such a short amount of time and found the development reasonably straight-forward, despite the documentation for developers not really being there yet for version 7.

The block also works quite well as a sidebar, depending on what you want to use it for.

Install it, just like any other module, by downloading it and putting it in its own directory in the sites/all/modules folder of your Drupal site.

I finally decided to look it up and it’s as simple as: Ctrl-Shift and the character code from Gnome’s Character Map application.

So to type ™, go to Applications > Accessories > Character Map. Then note the character code:

Hold down Ctrl + Shift and type u2122. Simple.

Installation

Install the required software and load the module into the running kernel without restarting:

Drive creation

Set up encryption on the disk and mount it:

Mounting

How to mount the drive:

Unmounting

How to unmount the drive: