swirrl

Launch of the Shakespeare Review

Thu, 16 May 2013 07:56:05 +0000

I had an interesting morning yesterday at the launch of the Shakespeare Review, Stephan Shakespeare’s independent review of public sector information.

There was an interesting panel session and discussion at the event and the twittersphere seems to have given it a mainly positive reception.

I had the opportunity, via the Data Strategy Board, to discuss the review with Stephan on several occasions and it’s great to see it finalised and opened up for a broader conversation.

It covers a wide range of important topics, but I’d like to highlight two of the recommendations here, that are particularly relevant to the work of Swirrl and have the potential to make big impacts.

One is the recommendation that all government departments should make an immediate commitment “to publish their Core Reference Data to an agreed timetable, to a high standard to maximise linkability,…ease of use and free access. They should commit to maintaining this data and keeping it regularly updated”.

We’ve been arguing for some time that making this type of data available is essential infrastructure for the broader exploitation of open data: allowing other datasets to be built on top of the core, enabling diverse data sources to be connected together and providing a solid foundation that private and public sector can build useful services on top of. (And the point about maintaining it is essential).

Paul Maltby, head of Transparency at the Cabinet Office, blogged yesterday on how we might decide what should go into a set of national reference data.

There are clearly some details to be worked out in how this is implemented, but linked data certainly has an important role to play. There is an opportunity here for the linked data community to provide some guidance on how this data can be made as useful and re-usable as possible.

The other point I’d like to highlight is the recommendation that:

“We should expect systematic and transparent use of administrative data and other types of PSI in the formulation, implementation, monitoring and adaptation of government policy and service delivery, and formally embed this in the democratic process”.

There’s a huge potential for the public sector to exploit it’s own data to make development and evaluation of policy more effective – a huge amount of money is spent by government. Few would dispute that it makes sense to try to measure what works, and increasing data availability and the capacity to analyse it effectively can make this possible and practical in a way that it hasn’t been hitherto.

There’s a lot of other important points raised in the review. Not all with simple obvious solutions, but raising areas that should be investigated and debated: for example reform of the trading funds and finding a balance between privacy protection and making effective use of anonymised personal data.

I’m looking forward to seeing the government’s response to the review – and hope the bulk of the recommendations are taken forward for implementation.

Hampshire Land Supply Linked Data

Mon, 11 Mar 2013 11:53:34 +0000

A short time ago we produced a Linked Data site for Hampshire County Council. We modelled their planned home building plans over the next six years as Linked Data and published it through PublishMyData.

As an example of what could be done with this data via the APIs, and to help people explore it, we built an embeddable map based visualisation (see below).

Working with Hampshire to create this data and visualisation further developed their understanding of Open and Linked Data, which will help inform their future open data initiatives.

Tripod: A new SPARQL ORM for Ruby. Read and Write RDF in your Rails apps

Tue, 05 Mar 2013 09:43:56 +0000

As we’ve long been promising, we’ve started to release some open source code to help people publish and consume Linked Open Data. The first component that we’d like to talk about is Tripod. It’s an RDF and SPARQL ORM, designed for use in Ruby-on-Rails apps.

What’s the point of Tripod?

If you’re writing a Ruby application that reads or writes a lot of RDF data from SPARQL endpoints, then Tripod should make life easier for you. Tripod lets you define ActiveModel-compliant Rails models for use in your applications, providing a familiar interface for use in your Rails controllers and views.

To use all features of Tripod, you need to have access to a SPARQL 1.1 database endpoint (over HTTP). We use Fuseki, part of the Apache Jena project. If you want to be able to persist any changes to data, the endpoint needs to support SPARQL UPDATE (and you’ll need the appropriate permissions), but note that Tripod is also useful for read-only use cases too.

How to use Tripod

Just include Tripod::Resource into your model class, and specify as much or as little information as required. Here’s quite a specific Person model, with a default type and graph:

class Person
  include Tripod::Resource

  # these are the default rdf-type and graph for resources of this class
  rdf_type 'http://person'
  graph_uri 'http://people'

  field :name, 'http://name'
  field :knows, 'http://knows', :multivalued => true
  field :aliases, 'http://alias', :multivalued => true
  field :age, 'http://age', :datatype => RDF::XSD.integer
  field :important_dates, 'http://importantdates', :datatype => RDF::XSD.date, :multivalued => true
end

… and here’s a fairly free-form Resource model, with only the label predicate specified.

class Resource
  include Tripod::Resource
  field :label, RDF::RDFS.label
end

Defining a field just creates the relevant getters and setters on the instances of the model.

Note: If you don’t specify a type or graph at the class-level, you can still set them on specific instances. For now, we assume that everything in the database is stored in named graphs, so if you don’t set a graph at all then you won’t be able to persist a resource to the db.

p = Person.new('http://ric')
p.name = 'Ric'
p.age = 31
p.save

r = Resource.new('http://myresource', 'http://mygraph')
r.label = 'my resource'
r.save

person = Person.find('http://ric')
person.name #=> Ric

There’s also chainable query DSL which helps you DRY-up your code by preventing you from having to write lots similar sparql queries throughout your application.

Person.where("?uri <http://name> ?name").limit(1).offset(0).order("DESC(?name)")

This is just a taster of what Tripod can do. For full installation and usage details see the Github repo.

Uses of Tripod in the wild

We first used Tripod in the SmartJourney application we worked on for Aberdeen City and County councils. And we’re currently working on an open source version of our PublishMyData engine, which uses Tripod for database access.

Licence

We’ve released Tripod under the MIT Licence, which means that there are almost no restrictions on what you can do with it. We’d love to hear about what you build with it.

Swirrl awarded new DCLG Linked Open Data contract

Mon, 04 Mar 2013 09:30:52 +0000

We are pleased to announce that we were recently awarded a contract to work on a production version of DCLG’s open data initiative, OpenDataCommunities. This exciting project was awarded through G-Cloud, and we are delighted that the department chose us to help them execute their vision for open data.

The current version of OpenDataCommunities is a Linked Open Data pilot study, which we began work on over a year ago. It provides a selection of Local Government statistics, as well as supporting geographical data, released under the Open Government Licence.

The new project will establish a sustainable and reliable way that DCLG themselves can publish their data. We’ll be working with the department to create internal APIs for streamlining and automating the data publishing process.

We are also learning about how people want to use the open data and encouraging more people to use it. So, in addition to the technical tasks, we will be involved in fostering partnerships and building relationships amongst data owners and users. Importantly, DCLG is now committed to maintaining this service, which means that data users can rely on it being available into the future.

The public-facing service will be an evolution of the existing OpenDataCommunities, but we will be basing it on a brand new, open source version of our Linked Data publishing platform, PublishMyData (more details coming soon!). As such, the initial release of the new service, in about a month, will see the removal of a few existing features, providing us with a solid base on which to build new data-browsing and discovery tools, for a broad range of users.

We will also be publishing a large amount of new data over the coming months.

Creating Asynchronous, Embeddable JavaScript Widgets

Mon, 11 Feb 2013 15:28:19 +0000

For a recent project, we needed to the ability to allow users to embed portions of the site in their own websites and blogs by copy-pasting a little bit of code. This blog post explains how we arrived at a solution for this.

Warning! This blog post gets pretty techy and is intended for other nerds who may find this technique useful in their own work.

More detailed requirements

As we don’t know what browsers our users’ sites are targetting, we want embedding to work on a range of browsers, including older versions of Internet Explorer (we aimed to support IE6+).
The code snippet should have no prerequisites (other than JavaScript being enabled) and be relatively short.
In case our service is being temporarily slow to respond for any reason, we don’t want to block the rest of the target page from loading.
We shouldn’t interfere with any other events on the target page.
We can’t rely on any external JavaScript libraries, and we shouldn’t pollute the global namespace of the target page.

Our embed script

Here’s an example of the code snippet for embedding a widget from the DCLG’s Impact Indicator Dashboard.

<script type="text/javascript" id="dclg-impact-indicators-embedder-5b8cf3a4-b0f0-13e9-3737-d5a159268ae6" class="dclg-impact-indicators-async-script-loader">
  (function() {
    function async_load(){
      var s = document.createElement('script');
      s.type = 'text/javascript';
      s.async = true;
      var theUrl = 'http://opendatacommunities.org/impact_indicators/indicators/housing-starts/dashboard_widget';
      s.src = theUrl + ( theUrl.indexOf("?") >= 0 ? "&" : "?") + 'ref=' + encodeURIComponent(window.location.href);
      var embedder = document.getElementById('dclg-impact-indicators-embedder-5b8cf3a4-b0f0-13e9-3737-d5a159268ae6');
      embedder.parentNode.insertBefore(s, embedder);
    }
    if (window.attachEvent)
      window.attachEvent('onload', async_load);
    else
      window.addEventListener('load', async_load, false);
  })();
</script>

Let’s break down what’s going on here…

A simple solution for modern browsers: HTML5 async attribute.

Much of the complexity above is there to support older browsers. If we only wanted to support modern browsers, then we could have achieved most of the desired functionality in one line (I’ll get onto what the remote script actually does later on).

<script async src="http://opendatacommunities.org/impact_indicators/indicators/housing-starts/dashboard_widget"></script>

The key thing to note here is the use of the async attribute. This tells the browser to execute the script asynchronously. The problem is, though, the async attribute is not supported on old browsers.

Without support for the async attribute, any content after the script tag is blocked from loading until its src has loaded and executed. This isn’t very polite at the best of times, and if our service was being slow or temporarily down, then it would be very annoying for users of the target page.

Creating a script tag with JavaScript.

To get around lack of support for async, our snippet creates a script tag with JavaScript and appends it to the document immediately before our snippet’s script tag. Note: We’ve used a guid in the id of the script tag to make sure we find the right one, if multiple widgets have been embedded.

<script type="text/javascript" id="dclg-impact-indicators-embedder-5b8cf3a4-b0f0-13e9-3737-d5a159268ae6" class="dclg-impact-indicators-async-script-loader">
  var s = document.createElement('script');
  s.type = 'text/javascript';
  s.async = true;
  s.src = 'http://opendatacommunities.org/impact_indicators/indicators/housing-starts/dashboard_widget';
  var embedder = document.getElementById('dclg-impact-indicators-embedder-5b8cf3a4-b0f0-13e9-3737-d5a159268ae6');
  embedder.parentNode.insertBefore(s, embedder);
</script>

We’re getting there, but the code above would block onload from triggering on some browsers until the script has been loaded and run, which isn’t really acceptable if we’re aiming to be a polite little widget.

Waiting until after onload

To prevent us from blocking onload, we wrap everything in a function, and call that once the page has finished loading (in a cross-browser compatible way). Note that we don’t just use window.onload to avoid stomping on any existing onload events on the target page.

Wrap the code in a closure, and add a referer parameter onto the url (so we can track who’s embedding our widgets), and we have our final solution. Scroll back up for a reminder of what it looks like.

The remote script

So far we’ve talked a lot about how to call a remote script without blocking the target page from loading, but what does the remote script actually do?…

In our case, the src url points to a Rails application route which responds with some JavaScript. The response creates a div in the target page, and injects our content into it. We also add a style tag into the head of the target page so that we can style our widgets.

Here’s a (slightly simplified) example of the response for a widget:

(function (global) {
  // add array index of for old browsers (IE<9)
  if (!Array.prototype.indexOf) {
    Array.prototype.indexOf = function(obj, start) {
      var i, j;
      i = start || 0;
      j = this.length;
      while (i < j) {
        if (this[i] === obj) {
          return i;
        }
        i++;
      }
      return -1;
    };
  }

  // make a global object to store stuff in
  if(!global.OpenDataCommunities) { global.OpenDataCommunities = {}; };
  var OpenDataCommunities = global.OpenDataCommunities;

  // To keep track of which embeds we have already processed
  if(!OpenDataCommunities.processedScripts) { OpenDataCommunities.processedScripts = []; };
  var processedScripts = OpenDataCommunities.processedScripts;

  if(!OpenDataCommunities.styleTags) { OpenDataCommunities.styleTags = []; };
  var styleTags = OpenDataCommunities.styleTags;

  var scriptTags = document.getElementsByTagName('script');
  var thisRequestUrl = '<%= raw(request.url) %>';

  for(var i = 0; i < scriptTags.length; i++) {
    var scriptTag = scriptTags[i];

    // src matches the url of this request, and not processed it yet.
    if (scriptTag.src == thisRequestUrl && processedScripts.indexOf(scriptTag) < 0) {

      processedScripts.push(scriptTag);

      // add the style tag into the head (once only)
      if(styleTags.length == 0) {
        // add a style tag to the head
        var styleTag = document.createElement("link");
        styleTag.rel = "stylesheet";
        styleTag.type = "text/css";
        styleTag.href =  "http://opendatacommunities.org/assets/impact_indicators_embed.css";
        styleTag.media = "all";
        document.getElementsByTagName('head')[0].appendChild(styleTag);
        styleTags.push(styleTag);
      }

      // Create a div
      var div = document.createElement('div');
      div.id = '<%= div_id %>';

      // add the cleanslate classs for extreme-CSS reset.
      div.className = 'dclg-impact-indicators-embeddable .dashboard_widget cleanslate';

      scriptTag.parentNode.insertBefore(div, scriptTag);

      div.innerHTML = '<%= j(render(partial_name)) %>';

    }
  }
})(this);

Again, let’s break this down a bit…

Support for Array.indexOf

First, to make our lives easier later on, we add an indexOf function to Array for browsers that don’t already have it:

// add array index of for old browsers (IE<9)
if (!Array.prototype.indexOf) {
  Array.prototype.indexOf = function(obj, start) {
    var i, j;
    i = start || 0;
    j = this.length;
    while (i < j) {
      if (this[i] === obj) {
        return i;
      }
      i++;
    }
    return -1;
  };
}

Namespaced objects for tracking progress

Next, we make some objects (under an OpenDataCommunities namespace in our case) to track:

Which embed scripts have already been processed (in case there are multiple embeds on the target page)
How many style tags have been injected into the head.

Find all our embed scripts on the target page, and loop over them.

For the scripts we haven’t processed yet, and where the request url matches the src of the script tag, we’ll do our magic. (Again, this caters for when people have embedded multiple widgets).

var scriptTags = document.getElementsByTagName('script');
var thisRequestUrl = '<%= raw(request.url) %>';

for(var i = 0; i < scriptTags.length; i++) {
  var scriptTag = scriptTags[i];

  // src matches the url of this request, and not processed it yet.
  if (scriptTag.src == thisRequestUrl && processedScripts.indexOf(scriptTag) < 0) {

    processedScripts.push(scriptTag);
    // do our magic here!
  }
}

Note: for those not familiar with Rails, the content inside <% %> tags are little snippets of embedded ruby (ERb) that are run inline before the response is returned from the server.

Add a style tag into the head of the target page.

Nothing much to say here, except that in our case, we included the cleanslate extreme CSS reset in our CSS, so that we can be sure that no existing styles in the target page bleed into our widget.

// add the style tag into the head (once only)
if(styleTags.length == 0) {
  // add a style tag to the head
  var styleTag = document.createElement("link");
  styleTag.rel = "stylesheet";
  styleTag.type = "text/css";
  styleTag.href =  "http://opendatacommunities.org/assets/impact_indicators_embed.css";
  styleTag.media = "all";
  document.getElementsByTagName('head')[0].appendChild(styleTag);
  styleTags.push(styleTag);
}

Create the div and inject its content

// Create a div
var div = document.createElement('div');
div.id = '<%= div_id %>';

// Add the cleanslate class for extreme-CSS reset.
div.className = 'dclg-impact-indicators-embeddable .dashboard_widget cleanslate';

scriptTag.parentNode.insertBefore(div, scriptTag);

div.innerHTML = '<%= j(render(partial_name)) %>';

We use Erb again here to allow us to provide a unique div id, and render the contents of the widget into the innerHTML of that div.

Resources

I found thse pages useful while figuring this stuff out:

DCLG Impact and Input Indicators Dashboard

Wed, 06 Feb 2013 21:51:09 +0000

We recently worked with DCLG (Department for Communities and Local Govermnent) to produce an online dashboard that enables users to see how the department is performing against key indicators in priority areas:
Impact Indicators: the performance or output of public services.
Input Indicators: how the efficiency and productivity of public services are improving.

The data on the dashboard widgets and detail pages is pulled in ‘live’ from APIs on OpenDataCommunities.org (DCLG’s linked open data store, which we helped create). This data-driven approach means that as new data arrives for the indicators in the data store, the dashboard is automatically kept up to date.

Additionally, the dashboard widgets can be easily embedded into any website or blog (like we have done below) by copying and pasting a little bit of JavaScript.

As the widgets pull in data directly from the data store, figures will always be kept current. We feel this is a valuable technique for sharing interesting or useful data and we’ll blog again soon with details of how the embedding works.

(Here’s the actual embedded widget):

If you’re interested in the raw data, the relevant datasets can be found in the Business Plan section of OpenDataCommunities.

Steve Peters from DCLG has also written about the dashboard (with more background) on his own blog.

SmartJourney: Realtime travel and weather for Aberdeen and Aberdeenshire

Fri, 21 Dec 2012 16:32:02 +0000

A couple of weeks ago, we launched SmartJourney. Built in collaboration with Aberdeen City and Aberdeenshire Councils, and part-funded by Nesta, SmartJourney combines crowd-sourced and official information and weather and travel conditions in the North-East of Scotland.

Anyone can view existing issues, and report new ones through the mobile-focused web interface. By signing up for a free account, users can additionally subscribe to email alerts for updates to issues they’ve reported, or for issues reported in their chosen geographic zones of interest.

For us, one of the most interesting aspects of this project is that all data generated by users of the site (except for the personal details of users, of course) is published in realtime as fully queryable and browsable Linked Open Data at data.smartjourney.co.uk. For convenience, we’ve also provided a RESTful JSON API for reading and writing issue-reports.

Another outcome of this project was that we built a Ruby ORM for working with RDF data in a Ruby on Rails app, that we call Tripod. Tripod is by no means finished, and we’ll be adding more features soon (more details in another blog post). In fact, this is the first of many open source PublishMyData components that we’ll be releasing over the next few months.

PublishMyData now available through G-Cloud

Thu, 15 Nov 2012 09:51:17 +0000

We are pleased to announce that UK public sector organisations can now buy our hosted PublishMyData service through Cloudstore. This is a new initiative, set up by the UK government G-Cloud Programme, to simplify the process of buying cloud based ICT services for public sector organisations.

As regular readers of this blog will know, PublishMyData is Swirrl’s Linked Data publishing platform, which we offer as a hosted supported service, together with associated implementation and consultancy. As more government organisations are now starting to publish linked data as part of their open data strategy, we aim to make that as simple for them as possible by looking after the technology and maintenance, letting the data owners concentrate on the data itself.

The Cloudstore site has a detailed explanation of how the buying process works and a list of which organisations are eligible to use Cloudstore – it’s a very comprehensive list of UK public sector organisations. G-Cloud is very much for the whole public sector, not just central government.

You can find all the details of our Cloudstore offering here. Just search for Swirrl or PublishMyData.

Swirrl is Hiring! Help shape the Future of Linked Open Data

Tue, 18 Sep 2012 21:45:51 +0000

We’ve seen strong demand for our services since launching our Linked Data Publishing platform, PublishMyData, and now we need some extra help. So we’re looking for someone to join our small team here in Manchester (UK), either on a permanent or long-term contract basis

What we’re looking for

We’re looking for a motivated and sharp web developer to work predominantly in our central Manchester office on our Linked Data related products and projects.

Ideally you’d be experienced in Ruby on Rails, JavaScript, TDD and NoSQL databases, although we will consider proven polyglot programmers with other core skill-sets.

Some knowledge of RDF, Linked Data and triple stores would be a bonus but we appreciate it’s a fairly niche skill at the moment.

One of our core aims is to enable all kinds of users (from laypeople interacting with visualisations through to software developers using our APIs) to understand, use, and get value from open data. So although we don’t expect you to be a fully fledged designer, a strong aesthetic sense and a good feel for user experience would come in very handy.

Finally, but importantly, we’d like to find someone who shares our passion for open data and open source, and wants to make a difference in that area.

We think Swirrl would be a great place to work

You’d be making a big impact on our small company, helping to shape our products and future.

This will be a very challenging and rewarding role, working on interesting projects for high-profile clients, as well as developing our own suite of Linked Data products. We’re looking for the best, and will offer a competitive salary/day-rate, a healthy holiday entitlement, and profit share for the right candidate. You would also get to spec your own development machine.

Although we’re looking for someone to work mostly with us in our informal but dynamic central-Manchester office (which we share with a few other startups), we’d be happy to accommodate some remote working.

Get in touch

If you’re interested, send us an email or call us on 0845 0035 389, and we can arrange to have an in-person chat over lunch or a drink. All enquiries/negotiations will be treated in the strictest confidence.

Strictly no recruiters or agencies, please.

Linked Open Data Business Models

Wed, 11 Jul 2012 09:15:27 +0000

In the wake of our respected competitor, Talis, retiring from the general Linked Data software and consulting market, I thought it would be a good time to explain how we perceive the market, and position ourselves at Swirrl.

Over the last couple of years, we’ve found a slow but steady increase in demand for Linked Data hosting, consulting, and bespoke software development, which we’ve served by building our PublishMyData platform and selling related services. By bootstrapping our business and calling on our network of trusted freelancers to help meet bumps in workload, we’ve been able to ride the wave of demand at what we feel is the right pace. With each additional project, we’re able to improve our product, adding value for our customers and data users, and at the same time, making the process more efficient for us.

Despite being proud Linked Open Data advocates at Swirrl, we firmly believe in giving people what they actually need and want. This means not forcing any particular standard or technology on users or customers, or assuming every problem is a nail for our RDF hammers. For us, this means judicious use of standards where appropriate, while recognising that not everyone are SPARQL and RDF nerds like us – these skills are definitely not mainstream yet – although we have experienced an encouraging willingness to learn at nearly every turn.

But it’s not all about the technology. I personally think that one of the reasons for Linked Data not permeating through to popular Web-consciousness is that mature user interface patterns don’t yet exist for describing and navigating the kinds of multi-dimensional data that RDF modelling produces. The tech is relatively simple; the hard parts are understanding your client’s problem-domain, and creating an effective and pleasant user experience for the various audiences for the data. This is by no means a solved problem, but we think we’re already going a pretty good job of making the data accessible, and we’re constantly trying to improve in this area.

I’m happy to say that we’ve been turning a modest but healthy profit from our Linked Data activities and that gives us confidence to keep investing. We’re currently busy working on several Linked Data projects, and have a few more prospects in the pipeline. For now we have the capacity to match our workload, but we expect to need more freelance or permanent help in the future. If you’re a passionate LoD geek who’d like to work with us, we’d love to hear from you.

Organisations interested in our Linked Data consulting, hosting or development services, should check out our PublishMyData site for more details. You can get in touch by emailing hello@swirrl.com (other ways of contacting us are listed on our company website).

Be among the first to hear our PublishMyData news by signing up to our newsletter. You’ll get no more than 1 email per month, and we’ll never sell your address or spam you.