By my estimate that means:

5 * 52 possible working days per year = 260

– 8 UK public holidays per year

– an average of 32 days off per year (to make a nice round number)

= 220 working days  per year

At an (underestimated) average of 8.5 hours per day, for 20 oh-so-long years, that is:

220 * 8.5  * 20 =

37,400 hours

Time for a rest.

TL;DR – OGC looks like it is being played

A couple of weeks ago I sent this ill-tempered, ill-mannered – but quite funny – email rant (to *@*.com, so you’ve probably seen it already):

http://lists.osgeo.org/pipermail/standards/2013-April/000576.html

“The submitting organizations of the GeoServices REST API are and have been supporters of OGC since its earliest years. They have actively contributed to many of today’s OGC standards and have implemented many of them in their software products. They continue to be committed to the goals and mission of the OGC.”

All very motherhood and apple pie. I’m sure that the submitters help old ladies across the road too. What does it have to do with the standard though?

“Most aspects raised in the justification comments are not new and have been discussed at length before.”

The fact that the comments have been discussed before does not mean that they have been satisfactorily dealt with.

“It is also worth to point out again that the backward compatibility requirement with the existing implementations applies to the version of the candidate standard that is under vote. Once the standard has been adopted, changes will follow the OGC procedures and, for example, a future major revision could also break such backwards compatibility. However, OGC policies emphasize the importance of backwards compatibility between revisions of a standard and the careful considerations that are necessary in this process.”

So backwards compatibility overrides every other consideration now, but not in the future, except that it will because of existing OGC policies? This makes no sense at all.

“As part of the SWG deliberations, the specification has been reviewed and changes have been made to remove any implementation specific artifacts.”

The specification precisely, and in great detail, describes the entire functionality of a single piece of existing software. Simply deleting the company name from well-known constants does not constitute “removing implementation specific artifacts”.

And so on, and so forth.

Some of my procedural, er, concerns, were misplaced, but a significant one remains. The current OGC TC Policies and Procedures for TC votes allow for an appeal to arbitration to the OGC Review Board (RB). Unfortunately, the RB no longer exists, having been subsumed – more or less – into the OGC Architecture Board (OAB). However, the OAB clarified the voting procedure (leading, in part, to my rant), and does not appear to have left open the prospect of an appeal. I can’t get a definitive answer right now, so that might change. But if it doesn’t then, frankly, it stinks.

I sent that email in response to the, I felt, weasel words of the OGC GeoServices REST API Standards Working Group which were in turn a response to several No votes in the OGC standardisation process. I’d complained about the proposed standard before, and did not get a reply:

http://lists.osgeo.org/pipermail/standards/2012-October/000527.html

Setting aside whether or not the intentions of the submitting organisations are entirely altruistic; *and* setting aside whether or not the proposed standard is in-and-of-itself well enough written to be implementable; *and* setting aside whether or not there is too much overlap with existing OGC standards; *and* setting aside whether or not the proposed standard deserves to bear the moniker “REST”; the fact remains that the proposed standard simply presents the public face of the private data structures and functionality of a single piece of software (and why shouldn’t it – that is precisely what it was designed to do).

As such *all* other implementations must either map their capabilities to the proposed standard, or add capabilities to match the proposed standard. Whichever is necessary, the original implementation will short-to-medium term at the *very* least remain the gold standard.

This is not even about the submitting organisations, nor even the precise content of the proposed standard. Exactly the same arguments would hold true were we looking at UMN MapServer “map” files as a proposed OGC standard, for example. (Now I’ve gone and planted a crazy idea in a few heads…) The same arguments also applied over KML.

As long as the SWG can – legitimately, by the rules – cite “backwards compatibility” as a reason to reject comments made in the RFC period, as is rumoured, then everyone else just has to take the proposed standard more or less as-is.

Things have snowballed somewhat since my rant:

http://lists.osgeo.org/pipermail/discuss/2013-May/011599.html

http://lists.osgeo.org/pipermail/discuss/2013-May/011632.html

http://lists.osgeo.org/pipermail/standards/2013-May/000581.html

And:

http://wiki.osgeo.org/wiki/Geoservices_REST_API

(As I write this, more than 60 people have signed up to a draft letter to OGC from OSGeo on that wiki page.)

Even OGC-couldn’t-care-less-er James has weighed in:

http://www.spatiallyadjusted.com/2013/05/09/esri-and-an-ogc-standard/

(I particularly like the first comment on that post, which makes the serious, valid point that the OGC list of standards is often used as a pick-and-mix way to generate a requirements specification. It won’t be a surprise to see W*S and GeoServices for REST (assuming that the worst happens, and it is adopted) both appearing as requirements for the same project soon.)

Where we are now is that both those in favour and those against are busy politicking, trying to get their own way. Esri even had a conference call with several OGC staffers, which I think is beyond the pale: the SWG should be standing behind this standard, not a single member, however large and influential. Of course the call might have been entirely innocent, but I can’t know for sure because no-one has told me (thus far).

I believe that this standard is bad for OGC, bad for the vast majority of  OGC members, and bad for the wider OGC community because:

• It has too much overlap with existing OGC standards
• It has had next to no changes despite multiple comments and requests, because backwards compatibility (with a single source, remember) trumped everything
• It sets an extremely unhealthy precedent for the next single-source API that comes in to OGC (maybe I’ll sponsor UMN MapServer MAP files after all…)

Howard Butler put the over-arching software engineering problem succinctly:

@jeffharrison @bflood @bartvdeijnden All of it. “Yours” or “mine” is proprietary by definition. “Ours” is an agreement.

— Howard Butler (@howardbutler) May 6, 2013

If you are reading this and are an as-yet undecided voter, then I urge you to vote No when the vote re-opens. If you are reading this and you are Ed Parsons, then shame on you for voting Yes.

My opposition to this standard does not mean that I think OGC or the W*S standards are somehow perfect; far from it.

The W*S standards suffer from the basic framework being designed back in 1998 (two years before Roy Fielding’s dissertation, RESTafarians). They have too naive a view of the web (eg abusing HTTP POST to stuff large requests into the server). In fact, I’ve argued that they are for the internet, not the web. On the other hand they have  too sophisticated a view of GIS/databases (eg feature editing, which is a good thing, but which is wrapped in transactions and feature-level locks, which are really an edge case). They suffer, like many standards – OGC and otherwise, from clever people thinking that they know best: GIS developers pontificating about the web; geodesists designing data structures; etc.

But adding more of the different-but-the-same, particularly in this manner, simply won’t help.

I’ll give myself the final word:

Painfully aware that the #OGC argle-bargle over #GeoServices is a tedious sideshow. The world is moving on from W*S, but not to more OGC

— Martin Daly (@mpdaly) April 8, 2013

• the “EPSG” code 900913, or
• the “EPSG” codes in the 42000+ range – you Canuck swine, or
• the so-called “MIME type” “image/png8”, or
• an OGC W*S implementation that, after installation references “http://localhost” in the schemaLocation in the GetCapabilities XML response when accessed from a forward-facing URL,

… then your name will also go on the list.

Doubtless my list will be just as effective as the one in this classic Dad’s Army sketch:

A couple of quotes re. closed source:

“an untenable dependence on closed source software”
“hampered with closed source software”

Has someone been on an NLP course recently, by any chance? Of course NLP is bullshit, so I’m afraid it didn’t work on me.

The so-called “paper” referenced in the post then has less than subtle digs at not only nasty, evil, blah, blah, blah, closed source software (yawn), but also “unsupported open source”. More NLP smell (at the expense of open source developers who work on contract, but are not enterprisey enough for OpenGeo).

Clowns to the left of me, jokers to the right, here I am stuck in the middle with OpenGeo?

My absolute favourite thing in the paper is the pseudo scientific equation to calculate “the value of software systems to enterprises in a market”:

We propose that the value … can be expressed with a simple equation, where V = Value, F = Functionality and Power, O = Operational Costs, C = Control. That relationship can be expressed as:

         V = (F/O) * C

It is beyond the scope of this paper to definitively break down these value elements into all of their subcomponents, or to measure precise values for them. However, it is quite possible to see how relative changes in each drives value and therefore to compare alternative software systems on value or V. In this way enterprises can make judgements about which are better for their operations.

So the “paper” proposes the equation, bases the conclusions on the equation, but can’t even explain or defend it? Colour me unimpressed. Personally, I’m working on reducing the “Operational Costs” of all the software we use to zero. That way the value will be infinity and I’ll be rich beyond my wildest dreams! Kerching!

If it looks like FUD, swims like FUD, and quacks like FUD, then it probably is FUD.

Prospective parent (PP, not me), to current pupil (CP):

“Do you do any actual programming?”

CP:

“We do some Delphi.”

PP:

“What about web design or development?”

CP:

“We do some Dreamweaver.”

On the bright side, Francesca absolutely trashed the touch-typing test, scoring 50% more than the nearest high score. So there is some hope amid the despair.

There are way too many things in this release to even begin to list here, but here are a couple of before (SIS 7.0, on the left) and after (SIS 7.1, on the right) Map Modeller screenshots, as a small clue about one major change.

Henceforth Cadcorp will be pronounced See-Eh-Dee-See-Oh-Arr-Pee. Click here to hear our marketing department explaining the change.

Back of the net! Yet another key differentiator. +1, as the Google would have us say.

Update: What I should have said was that the 30% improvement is over the previous GeognoSIS 7.1 results, post-benchmark from November 2010, which were, in turn better than the GeognoSIS 7.0 results from the original benchmark.

The summary chart of all of the Cadcorp vector results – for Shapefiles, PostGIS and Cadcorp-proprietary FDB data sources – now looks like this:I would say “bring on the 2011 Benchmark“, but that would be asking for trouble…

As I explained here, we, Cadcorp, took part in the FOSS4G 2010 WMS Benchmarking exercise. I predicted that we could get our ars*s kicked, and we did. Allow me to explain (at length) why that was.

First, an entry from the release notes of SIS 7.0 Service Release 2, from May 28^th 2010:

• The Microsoft SQL Server 2008 cursor-based Dataset now pads very small query extents to avoid invalid geometry errors.

Hang on (you say), SQL Server 2008 was not part of the benchmark: I cry foul! Well, that “fix” was deemed sensible enough to propagate into all of our cursor-based Datasets, including those for ESRI Shapefiles and PostGIS, which were used in the benchmark.

Rewinding to that change, it was necessary because SQL Server 2008 (as I have told Spatial Ed many times) is very prissy about what geometry it accepts. The problem arose when a spatial query was used that, for example, asked for all rows in a table where the geometry in the row contained a Point.

Being good spatial database clients (or possibly being ones bearing the scars of deathly slow query times back in the day), we provide a primary filter as an envelope intersection, where possible, and the spatial query itself. If the test geometry is a Point then the envelope is degenerate, and SQL Server 2008 choked on it. While padding sounds simple enough, it is necessary to take the units of the coordinate reference system into account, so that, for example you don’t pad by one in a CRS using degrees, even though this might seem reasonable for a CRS using metres.

Luckily we have a nice simple COM object that will tell us sensible CRS-dependent tolerances that we could use, and so we did. What we did not realise was that a side-effect of that simple call was that another COM object was created in order to try and match the units of the CRS against the well-known ones supplied in the EPSG Geodetic Parameter Dataset. What was even worse was that involved initialising the database that we now use to store the EPSG data, and that the database initialisation was protected by a multi-threaded lock and, even worse than that, the lock was static, thus applying to all instances of that object, even though the database itself was a member of the object class.

(Please note that throughout that previous passage we were using The Royal We.)

Here is an example of the result of that problem – from the first test, for ESRI Shapefiles in their native CRS – taken directly out of the spreadsheet of results used in the final presentation (or here in Google Docs):

That’s us down there pootling along at the bottom, feeling sorry for ourselves.

We might not have been down there on the day had I not been on holiday in the week before the benchmark and had the Windows machine not been down all day on the Monday and the Tuesday morning before the presentation – with no-one around to reboot it because of the Labor Day holiday – and only Tuesday afternoon in which to run tests while also on a conference call. But we were.

We had the chance, as taken by a couple of the other participants – after what was by all accounts an extraordinarily long meeting in Barcelona – to pull our results, but they definitely happened that way, as we always knew they could, and so it seemed churlish not to make them public.

We’ve carried on testing intermittently since then, as time allows, have found and fixed the CRS database bottleneck described above – which gave us an immediate boost up the chart – and made a few other less dramatic improvements (in code; in the server configurations; and the way we used the data files – GeoTIFF especially), and now that chart looks like this (with our old results still there, mostly down at the bottom, for reference):

We have also re-run all of the other tests, put all of the results in the online spreadsheet shared by the participants, and updated the associated charts (I also made all of the charts use consistent colours throughout, so you can quickly find your favourite WMS):

That chart is now misleading, because we were not the only ones to suffer from multi-threading lock problems, as you can see from this post (including an updated chart) about changes made to Mapnik after the benchmark was over.

We have also been able to run some tests on the same vector data stored in the Cadcorp-proprietary Feature Database (FDB) format, with the – best so far – results shown here, alongside Shapefiles and PostGIS:

Finally (honest), we also ran the raster tests using a single ECW file (in the native CRS only) that I “borrowed” from the ERDAS benchmark data folder:

Shhhh. Don’t tell anyone about that chart – the new ERDAS ECW JPEG2000 SDK doesn’t allow us to do that without an additional licence, which we don’t yet have.

After all that, what did we learn?

• Don’t wait until the last minute before running the full suite of tests
• Don’t go on holiday just before the tests
• Don’t spend time worrying about data quality and coverage: it is what it is
• Don’t spend ages futzing around with data mangling for “best effort” tests that you then have no time to run
• Do try more ways of referencing multiple data files, e.g. the 56 GeoTIFFs used here
• Do try more server configurations
• Do spend time making a video of all 2200 images that can be used as a backdrop to Ivan Sanchez’s WMS song
• Finally, do publish, regardless

Lovely Prezi skills, no?