Once in a while, Oxford Archaeology get called upon to do some really big archaeological projects, like road schemes and airport expansion, that cover huge areas, go on for years, and generate loads of data. We love these, because it’s not very often that you get to look at whole landscapes- how multiple prehistoric villages interact, for example, rather than tantalising snapshots where you have to play “join the dots”.

Not only do we generate loads of data (thousands of artefacts, records, environmental samples, photos etc), but we ask difficult questions of them, based on statistical analysis of finds distribution, travelling salesman algorithms, best-path analysis. I think you can call this “real GIS”, though often people are surprised to hear that archaeologists use GIS at all.

As part of our ongoing “open approach“, and to prove that we put our money where our mouth is, we are now trying to do all of this in open source software rather than using the “standard” proprietary packages.  I’ve blogged previously about how pleased I was with the integration between QGIS and PostgreSQL, and how easy it was to manage large amounts of data without regress to proprietary packages, well  our current large project needs 3D analysis and large amounts of imagery manipulation, and again we’re finding that the open source tools out there do the job splendidly. Furthermore, we have a choice of tools, so if one approach doesn’t quite work the way we expect or want, then we can choose another. Now that can’t be bad, can it?

So, in brief, we’re using Quantum GIS and GvSIG pretty much interchangeably for our desktop GIS. All the vector data is in PostgreSQL. We use the QGIS Grass plugin to get data into a sensible format for 3D display and analysis in Paraview and Visit. We’re mosaicing up aerial photos using GDAL tools, and using Geoserver to publish everything to people who just need read-only access, and a direct connection to PostgreSQL for those that need to edit. We’ve developed a workflow for creating high-quality cartographic output by exporting to Inkscape, and the next step is a project website with links to our database and a nice openlayers map. Simples!

A couple of weeks ago was the OSGIS 2010 conference at the Centre for Geospatial Sciences at the University of Nottingham. This was expanded on the previous year’s event- it was spread over 2 days with workshops on the first day and the conference on the second. We also held the 2nd AGM of the UK OSGeo local chapter after the conference (a shameless attempt to get as many people as possible to attend).

The workshops went well- I reprised my “Databases and Web Mapping the Open Source way” workshop using the OSGeo Live DVD in the process. If you last tried Live DVDs a few years ago and think they are going to be unworkably slow and clunky then think again. The Live DVD team have done a very good job with the OSGeo version too!

As is usually the case when you’re involved in running an event, my memory of the actual papers on day two is somewhat hazy. Stand-out themes were TinyOWS and GvSIG in my book. The GvSIG team in particular were there in force, and very impressive.

The local chapter AGM went well, although I’ve committed my first mistake in calling it the local chapter, as we decided that was a bit too hells-angels, so we’re now called OSGeo:UK (branding updates to follow). Perhaps due to my (even more) shameless emotional blackmail in my introductory talk, there was a lot of new interest. We now have a mailing list of over 100 people for a start. The mailing list is actually the best place to go for my quick round-up of the AGM- check out the archive for more details. In particular though, if you are involved in promoting any workshops using open source software and you want some publicity then get in touch as we want to get a good list together of all the events that are taking place in the UK.

I followed up events in Nottingham with assisting at another two-day workshop in Leicester, at which I repeated my “databases and web mapping…” talk yet again, alongside talks on spatial statistics with R, and QGIS/GRASS. My talks for all these events are up on slideshare if you’re interested, and will follow in the downloads section of this website when time permits.

I’m helping to teach at a workshop on open source GIS at the University of Leicester in a couple of week’s time. As usual, this means running around trying to get all the software that we need installed on the university computers. As usual, what the course organisers think will be OK, and what the IT department think will be OK, are two different things!

We’d rather not use a LiveDVD, as we want the students to work in the environment which they are used to- in this case windows. We don’t really want to saddle the IT department with lots of PostgreSQL and Apache configuration, so we’re going to try using Portable GIS (gulp).

The current iteration (as available here), has an out of date version of Quantum GIS, doesn’t contain the statistical package R, and also contains a bunch of other stuff that we won’t use on the course (MySQL, GvSIG etc). The beauty of open source cross platform software though, is that rolling a customised version, containing just what software we need, and the latest versions, was quite easy.  Adding in the data, and even the course notes, will be straightforward (once we’ve written them, of course!). Installation is simply a case of copying everything onto each pc, and the students can take it all home with them when they are done.

The open source license is not the big deal here, despite the title of the post, but the added benefits that it brings are pretty cool. Free software means we can run the course without having to worry about buying licenses, or sending students home with limited demo versions afterwards. The cross-platform nature of the software means most of the donkey work is done in human readable files that can easily be edited to work in a portable fashion, and finally, because it’s open source, we can do that legally.

I think that’s all pretty cool…

(Small Plug) If you like the idea of using Portable GIS for a course, but don’t fancy customising or preparing it yourself, then get in touch!

I went to the Open Knowledge Foundation conference, OKCON in London a few weeks ago, and have been meaning do a review of it ever since. Whilst little of what I saw had a direct relevance to what I do, it was invigorating to be in a room with a whole bunch of people with imagination, who believe knowledge should be free to anyone, and who basically like to disrupt the status quo.

A few quotes:

[2009 was] the year open data went mainstream (Rufus Pollock)

The threat [to the record industry] is not piracy, but obscurity (Glynn Moody)

What would happen if every school had a reprap? (Ben O’Steen)

One paper that was quite relevant to me was “Dig the new breed” by Anthony Beck, about opening up archaeological data. There are related articles and mailing list posts here and here. I feel uneasy about some of the details (it’s not lethargy or ethics that dictates the data we do or don’t release, it’s money) but in general it’s a no-brainer. We’re doing our bit with our Eprints library, but we’re at an early stage with getting reports on there.

The only problem I see is figuring out who to lobby- I would add the developers and the county-level curators to the list as well as the actual archaeological units. I know of specific cases where developers would not provide the money for making the results of an excavation public access, and also where a regional Historic Environment Record would not allow “their” data to be shown on a web map.

Otherwise, I particularly enjoyed Glynn Moody’s paper on The Post-Analogue World- focussing on the “plight” of the record industry, struggling to cope with the transition from analogue to digital, and Ben O’Steen’s paper on Making the Physical from the Digital. Bookbinding, repraps, MP’s expenses and Cory Doctorow all in one talk. Can’t be bad!

Many of the talks are available to download here, and OKFN have working groups for both archaeology and geospatial data if you’re interested. I hope to have more involvement with both, and to investigate links between OKFN and OSGeo, now I have some time and mental space. More to come…

… and… relax!

We’ve just completed a really intense GIS project covering the entirety of the North-West of England- which we’ve been working on since Christmas. This week is the first time I have come up for air since before Christmas, and it’s been really great! I’ve been working on integrating the new free Ordnance Survey data into our nascent SDI, and updating our sites map with new features, and new data from some of our other offices. It might not look like much, but there’s lots of open source goodness in there! My colleague Lucian, who knows much more about PostgreSQL than I do has put in a function that automatically updates this map when a new project is added to our archives database. Next steps- more data integration- with our grey literature server, and sites from our two French offices, and adding the raster mapping from the Ordnance Survey to our SDI.

Anyhow, it’s great to be back in the land of blogging- I hope to do a post soon on how to deal with this great new data now it’s available, and also to recap on the great OKCON 2010 conference that I went to at the weekend!

So, we Brits are getting all excited because the Prime Minister gave a big speech about how Britain is going to be all fab,  broadbandy, and “totally, like Web 2.0, man”  soon. And all this data is going to be given away- look here’s some of it now. We’re all going to have instant access to Government services, free Ordnance Survey data, and a shiny Institute of Web Science to stick all the bits together.

Now, of course it’s a total coincidence that this is a nice positive announcement, and there’s to be an election shortly, for which we don’t yet know the date. Let’s not go there, since this is not a political blog. Let’s also not contrast this with the Digital Economy Bill, fast-tracking it’s way through parliament on somewhat dodgy grounds as we speak, that threatens to take a lot of this nice shiny stuff away if you’re so much as caught within a mile of an mp3 that you didn’t buy from ITunes. Let’s not talk about carrots and sticks, or nice cops and nasty cops or other such comparisons.

Instead, let’s concentrate on the data.  They will, on April 1st, release “a substantial package of information held by ordnance survey freely available to the public, without restrictions on re-use”. The consultation only finished 5 days ago! How can they possibly have had time to analyse the consultation responses yet? Personally, while I look forward to the release of the data, and it is a massive win for the Free Our Data campaign, I would prefer that they gave the consultation responses some respect by analysing them in detail, or at least pretended to read them properly.  I told someone off for assuming that the decision had already been made, and that the consultation was a waste of time, but perhaps they were right.

The rest of it, I quite like, though I’m yet to see a Government IT project really work properly. We’ve got (amongst other things):

• “core reference datasets that contain the precise names and co-ordinates of all 350 thousand bus stops, railway stations and airports in Britain”
• “an inventory of all non-personal datasets held by departments and arms-length bodies – a “domesday book” for the 21st century”
• “information on each set of data including its size, source, format, content, timeliness, cost and quality”

All sounds good. I could get excited by this, but I just want to wait and see what happens, shall we, and not take our eye off the other balls?

On Friday and Saturday I attended the inaugural WhereCampEU “un-conference” in London.  The short review: if one comes anywhere near you- go for it! It’s well worth it and I enjoyed every minute.

The slightly longer review: since there were no themes or papers organised beforehand, I guess it was a good opportunity to take the temperature of a certain part of the UK/EU geospatial community. So we got lots and lots of OpenStreetMap papers and iPhone apps! Don’t get me wrong, that’s not a bad thing, but personally, the papers that stood out for me were Steven Feldman’s talk on Business Models and how we are all f*cked without one, and, as always, Bob Barr’s talk on the true cost of “Free” data.  That particular talk was timely, coming just before the deadline for the consultation on freeing Ordnance Survey data.

On business models- Steven gave the talk twice due to popular request (one advantage to the un-conference format), and got remarkably different responses. The gist was whether you could come up with a one-minute elevator pitch on your business, your customers, and your prices. If you can’t do that, then you should be able to. There was quite a split in the audience.  On one hand there were people who are genuinely trying to make a business out of what they do- who have mortgages to pay, employees to pay, and kids to feed, who can see the point in a business model, and on the other hand there were people with well-paid day-jobs and a lucrative sideline in making iPhone apps who couldn’t see the point. That’s an over-generalisation, but I have work to do, so forgive me. I did start musing about doing a 140 character “tweet-pitch” too but maybe that’s just jumping on the bandwagon!

Overall, at the end of the two days I didn’t feel like I’d been at a conference, even though it was just as packed and even more fast-paced than usual. It was just more relaxing and informal, and the lack of corporate salesmen helped too! The venues were fantastic, as was the food, and the evening geo-beer was much appreciated. The team were keen to point out that next year’s “un-conference” (if it happens) should be somewhere else in Europe. At the time there weren’t that many takers, perhaps because there was a UK bias to the attendees, but I’m sure the enthusiasm and positive feedback will percolate around and we’ll get some volunteers.

After that I spent a couple of days being a tourist in London and catching up with old friends. As a test, I did my London navigation with a zoomable paper map, which I can say works very well and is pleasingly analogue (no batteries or data costs). Crikey though- who needs that many Starbucks, Costa and Caffe Neros?

There’s a fantastic article in today’s Guardian (via Computer World Magazine) about the International Intellectual Property Alliance, who say that countries advocating the use of open source software should be put on a “Specialist 301 list” (ie a trading watch list) because open source “weakens the software industry” and “fails to build respect for intellectual property”.  The IIPA is an umbrella group for organisations like the RIAA and the MPAA, who are of course well-known for their open-mindedness and forward thinking.

Go read the articles, I’ll wait…

As well as the obvious lack of understanding of, well, anything, this is all quite bizarre. It implies disapproval of almost the entire internet infrastructure, as well as the big (US) players like Google. It shows a complete lack of understanding about companies (in the US) making a revenue from an open source business model, or even including open source software within their offerings. And how you can claim that the open source licenses somehow harm intellectual property is beyond me.

Oh well, I guess that puts the British Government on the watchlist, as well as the US government, and presumably the IIPA need to put themselves on if they use Apache at all…