Today we are very happy and excited to announce the final release of CKAN 2.0. This is the most significant piece of CKAN news since the project began, and represents months of hectic work by the team and other contributors since before the release of version 1.8 last October, and of the 2.0 beta in February. Thank you to the many CKAN users for your patience – we think you’ll agree it’s been worth the wait.

CKAN 2.0 is a significant improvement on 1.x versions for data users, programmers, and publishers. Enormous thanks are due to the many users, data publishers, and others in the data community, who have submitted comments, code contributions and bug reports, and helped to get CKAN to where it is. Thanks also to OKF clients who have supported bespoke work in various areas that has become part of the core code. These include data.gov, the US government open data portal, which will be re-launched using CKAN 2.0 in a few weeks. Let’s look at the main changes in version 2.0. If you are in a hurry to see it in action, head on over to demo.ckan.org, where you can try it out.

Summary

CKAN 2.0 introduces a new sleek default design, and easier theming to build custom sites. It has a completely redesigned authorisation system enabling different departments or bodies to control their own workflow. It has more built-in previews, and publishers can add custom previews for their favourite file types. News feeds and activity streams enable users to keep up with changes or new datasets in areas of interest. A new version of the API enables other applications to have full access to all the capabilities of CKAN. And there are many other smaller changes and bug fixes.

Design and theming

The first thing that previous CKAN users notice will be the greatly improved page design. For the first time, CKAN’s look and feel has been carefully designed from the ground up by experienced professionals in web and information design. This has affected not only the visual appearance but many aspects of the information architecture, from the ‘breadcrumb trail’ navigation on each page, to the appearance and position of buttons and links to make their function as transparent as possible.

Under the surface, an even more radical change has affected how pages are themed in CKAN. Themes are implemented using templates, and the old templating system has been replaced with the newer and more flexible Jinja2. This makes it much easier for developers to theme their CKAN instance to fit in with the overall theme or branding of their web presence.

Authorisation and workflow: introducing CKAN ‘Organizations’

Another major change affects how users are authorised to create, publish and update datasets. In CKAN 1.x, authorisation was granted to individual users for each dataset. This could be augmented with a ‘publisher mode’ to provide group-level access to datasets. A greatly expanded version of this mode, called ‘Organizations’, is now the default system of authorisation in CKAN. This is much more in line with how most CKAN sites are actually used.

Organizations make it possible for individual departments, bodies, groups, etc, to publish their own data in CKAN, and to have control over their own publishing workflow. Different users can have different roles within an Organization, with different authorisations. Linked to this is the possibility for each dataset to have different statuses, reflecting their progress through the workflow, and to be public or private. In the default set-up, Organization user roles include Members (who can read the Organization’s private datsets), Editors (who can add, edit and publish datasets) and Admins (who can add and change roles for users).

More previews

In addition to the existing image previews and table, graph and map previews for spreadsheet data, CKAN 2.0 includes previews for PDF files (shown below), HTML (in an iframe), and JSON. Additionally there is a new plugin extension point that makes it possible to add custom previews for different data types, as described in this recent blog post.

News feeds and activity streams

CKAN 2.0 provides users with ways to see when new data or changes are made in areas that they are interested in. Users can ‘follow’ datasets, Organizations, or groups (curated collections of datasets). A user’s personalised dashboard includes a news feed showing activity from the followed items – new datasets, revised metadata and changes or additions to dataset resources. If there are entries in your news feed since you last read it, a small flag shows the number of new items, and you can opt to receive notifications of them via e-mail.

Each dataset, Organization etc also has an ‘activity stream’, enabling users to see a summary of its recent history.

Programming with CKAN: meet version 3 of the API

CKAN’s powerful application programming interface (API) makes it possible for other machines and programs to automatically read, search and update datasets. CKAN’s API was previously designed according to REST principles. RESTful APIs are deservedly popular as a way to expose a clean interface to certain views on a collection of data. However, for CKAN we felt it would be better to give applications full access to CKAN’s own internal machinery.

A new version of the API – version 3 – trialled in beta in CKAN 1.8, replaced the REST design with remote procedure calls, enabling applications or programmers to call the same procedures as CKAN’s own code uses to implement its user interface. Anything that is possible via the user interface, and a good deal more, is therefore possible through the API. This proved popular and stable, and so, with minor tweaks, it is now the recommended API. Old versions of the API will continue to be provided for backward compatibility.

Documentation, documentation, documentation

CKAN comes with installation and administration documentation which we try to keep complete and up-to-date. The major changes in the rest of CKAN have thus required a similarly concerted effort on the documentation. It’s great when we hear that others have implemented their own installation of CKAN, something that’s been increasing lately, and we hope to see even more of this. The docs have therefore been overhauled for 2.0. CKAN is a large and complex system to deploy and work on improving the docs continues: version 2.1 will be another step forward. Where people do run into problems, help remains available as usual on the community mailing lists.

… And more

There are many other minor changes and bug fixes in CKAN 2.0. For a full list, see the CKAN changelog.

Installing

To install your own CKAN, or to upgrade an existing installation, you can install it as a package on Ubuntu 12.04 or do a source installation. Full installation and configuration instructions are at docs.ckan.org.

Try it out

You can try out the main features at demo.ckan.org. Please let us know what you think!

Python Developers – CKAN/Services team

Role Description

The Open Knowledge Foundation is recruiting senior Python web developers, primarily to work on CKAN, our open source, open data catalogue. There will also be further responsibilities to develop data visualisation and other data-driven services.

CKAN is an open source web-based software project written in Python. It allows users to submit, search for and find open data. As well as powering datahub.io, CKAN is the catalogue behind the UK government’s data.gov.uk website. It also powers over 30 other data catalogues around the world. If you are an outstanding Python web developer, with a keen interest in open data, and enjoy working on open-source products, we’d love to hear from you.

You’d be involved in:

• customising CKAN for different governments and organisations;
• helping develop new features for CKAN;
• supporting existing software deployments;
• working on other Python programming projects in the OKFN.

Requirements

Essential:

• Web app development experience in Python;
• PostgreSQL;
• Linux (Ubuntu/Redhat);
• Git;
• Enthusiasm about open data and open knowledge.

Bonus points for any of these (not essential though):

• Solr;
• Knowledge of the CKAN codebase and extensions;
• Project management/consultancy experience;
• Semantic web/RDF;
• jQuery and CSS;
• S3 and EC2;
• Experience with Agile methods;
• Knowledge of the geo-spatial community.

About the Open Knowledge Foundation

The Open Knowledge Foundation (OKF) is a multi-award winning community-based, not-for-profit organisation. The Foundation now has projects and partnerships throughout the world and is especially active in Europe. We build tools and communities to create, use and share open knowledge – content and data that everyone can use, share and build on. We believe that by creating an open knowledge commons and developing tools and communities around this we can make a significant contribution to improving governance, research and the economy.

We’re changing the world by promoting a global shift towards more open ways of working in government, arts, sciences and much more. We don’t just talk about ideas, we deliver extraordinary software, events and publications.

How to apply

If you’d like to apply, please email jobs@okfn.org with the subject line ‘Python Developer – CKAN/Services’ and a copy of your CV by 30 April and we’ll take it from there. We are flexible on employee versus contractor but we normally contract. Being based in and around London would be a plus but we are happy to consider applications from elsewhere.

A new feature in CKAN 2.0 enables you to add custom previews for different file types.

CKAN’s ability to preview resources gives users a quick way to check if they have the dataset they need – as well as to begin to explore the data. CKAN has built-in previews for certain filetypes, such as images and CSV files. However, if you have resources in another format, users cannot preview them. Custom previews provide a simple way to add previews for more filetypes, or even modify existing preview methods.

Map preview of CSV file in CKAN

New previews can be build as a CKAN extension with the help of the IResourcePreview interface. This is the same interface as CKAN already uses for built-in previews. (Note that, while most of these are enabled by default, the built-in PDF preview is not. To enable PDF preview, you need to edit the CKAN .ini file and add pdf_preview to ckan.plugins.)

A preview extension must implement three methods:

• can_preview to indicate that it can preview the dataset,
• preview_template to return the template for the preview, and
• setup_template_variables to add the data that should be rendered to the template.

If you happen to write an extension that previews files that are fetched via an Ajax call, you should also have a look at the resource proxy extension. This offers a workaround for the same origin policy which normally prevents files being fetched from a domain that is different from the domain of the CKAN site.

To get started with your own preview extension, I recommend that you read the extension documentation linked above, and then have a look at the built-in preview extensions at ckanext. If you have any questions, let us know on the mailing list and we’ll try to help.

CKAN’s user interface is available in over 20 languages. The new version introduces over 450 new words and strings in the interface, and they all need translating before it’s released. If you speak one of the relevant languages, you can join that language team to help with the translation effort.

If you are a speaker of a language where CKAN isn’t available yet, this would also be a great time to start a translation project for it!

If you can help with translating the strings and keeping CKAN open to people across the world, please sign up here to contribute.

We are very proud that the Spanish region of Aragón finally has its own open data website: Aragón Open Data was launched in February, using CKAN. At the Government of Aragón ICT department, we are really concerned with the value of data and information. This CKAN-powered website is our first step to providing citizens with easier access to data. For the launch, we published more than 230 datasets rated with 2, 3 and 4 Tim Berners-Lee stars.

Why choose CKAN?

When we made the decision to publish the region’s open data in a single place, we wanted a ‘CMS for data’, and we took a look at the different options. CKAN was the best solution for our needs for several reasons, but a few stand out.

Firstly, the API. For the Aragón Open Data team, it is important to make it easy for developers to access data. This problem is solved with CKAN´s API. The API lets you get a reliable ID of a dataset, and use it to query for the current version of the dataset. This allows developers to have a unique object to download and also know when the dataset has changed. For developing apps based on the data, this is a perfect way to ensure you always have the latest data.

ARAGON OPEN DATA

Secondly, the CKAN back-end. It has been developed as a kind of CMS specialised for data, and the data management tools that CKAN provides are really useful to get your metadata ‘tidy’. It is easy to modify the different attributes of a dataset and to add new extra fields to improve the description of your datasets.

To give structured formats to advanced users that want to develop applications, we wanted a metadata file for each dataset: a file expressing all the metadata in RDF format using the DCAT vocabulary. CKAN helped as it has built-in RDF output, though there is no link to this by default. To make it more prominent, we were easily able to write a CKAN extension which put a link on each dataset page to the RDF metadata.

Finally, having made the decision to use CKAN, it was easy and intuitive to install using the published documentation and to develop the website using it. We are now working on extending the website using more of the posibilities that CKAN provides.

Open data, open spending

To reinforce our commitment to open data, we have also developed an open spending application: presupuesto.aragon.es. Through this app it is possible to ‘follow the money’ from citizens’ taxes through to public expenditure. You can also see in what policy area money has been spent during the last 8 years. You can calculate, depending on your earnings, how much of your money goes to each policy area. Finally there is a brief glossary of spending terms to help users better understand the budget’s vocabulary.

We hope you find useful data in the Government of Aragón’s data catalogue, and we will be very happy if you develop any data-based applications with it. Enjoy Aragón Open Data!

The ‘researcher’ user group

On 18 February, over 40 people participated in a workshop in London which focused on the use of CKAN for ‘research data management’. The event was led by the Orbital project at the University of Lincoln.

After some initial presentations, participants broke into groups to gather a list of requirements for CKAN – or any data management system – in a research setting. You can read all about the day, and what we produced, over on the Orbital project blog. There are also write-ups from the DCC, Datapool, data.bris and LSE.

We hope that the workshop marked the beginning of a CKAN user and developer community within the higher education and research sectors. If you’re interested in joining us, feel free to add yourself to the new CKAN4RDM mailing list. The results of the workshop will also form part of the research for a paper I am presenting at the IASSIST conference in May. The abstract is as follows:

This paper offers a full and critical evaluation of the open source CKAN software <http://ckan.org> for use as a Research Data Management (RDM) tool within a university environment. It presents a case study of CKAN’s implementation and use at the University of Lincoln, UK, and highlights its strengths and current weaknesses as an institutional Research Data Management tool. The author draws on his prior experience of implementing a mixed media Digital Asset Management system (DAM), Institutional Repository (IR) and institutional Web Content Management System (CMS), to offer an outline proposal for how CKAN can be used effectively for data analysis, storage and publishing in academia. This will be of interest to researchers, data librarians, and developers, who are responsible for the implementation of institutional RDM infrastructure. This paper is presented as part of the dissemination activities of the JISC-funded Orbital project <http://orbital.blogs.lincoln.ac.uk>.

I hope to see some of you there.

We went back to basics and redesigned the theme to focus on the core parts of CKAN: discoverability of data for end users and better data management tools for publishers.

Instructions to install or upgrade are below and you can go play around with it on beta.ckan.org – just create an account to try out all the funcitonality.

You’ll see a more engaging search page that highlights the use of facets and search result sorting as well as the keyword search; a redesigned dataset page showcasing resources, tags, activity streams and re-use of data (the “related” tab); a new multistep dataset registration process to guide publishers as well as a scalable authorization model for managing access rights to datasets (more info below under ‘Organisations’).

Highlights

Organisations

We’ve simplified the authorisation model for datasets. Previously a dataset was ‘owned’ by the user who added it to the portal – edit or admin access to the dataset had to be manually added one by one. This made it harder to distribute management of datasets for large portals with numerous publisher accounts for multiple publishing organizations (e.g. Departments or Agencies).

In 2.0, by default, datasets always belong to an organisation (e.g. Department of Health, Office of National Statistics etc) and access rights are automatically allocated depending on the organisation access level members have. So if your department has 1 admin, 5 editors (add/edit/delete datasets, but no member management access) and 10 members (can see org private datasets, member, editor, admin accounts, but not add or edit anything) then any dataset that one of the editors adds will be editable by any of the other 4 editors or the admin.

Activity Streams

Datasets, users and groups now generate activity streams that can be ‘followed’.

Templating

We have switched our templating engine to use Jinja2, making it easier to customize CKAN. We’ve found this to be much more designer friendly and intuitive than Genshi. It’s also generally cleaner and renders faster. We’re also able to extend templates (for example by extensions) in a less invasive way than previously using the {% ckan_extends %} template tag.

Note: Jinja2 templates do not support the IGenshiStreamFilter extension point, so extensions should either extend templates or provide template helper functions via the ITemplateHelpers extension point.

Installing or upgrading

This first beta release is aimed mainly at developers, and is only available via source install. You can refer to the “Install from Source” instructions or the “Upgrading a source install” sections on the documentation depending on your needs.

In either case, make sure to point the repository to the 2.0 release branch (release-v2.0):

• For first time source installs, in section 2b use:

pip install -e 'git+https://github.com/okfn/ckan.git@release-v2.0#egg=ckan'

• For source install upgrades, in section 2 use:

git checkout release-v2.0

Our supported extensions are currently being upgraded, and the degree of support for CKAN 2.0b varies depending on the extension. If you are using custom extensions on your instance, it is likely that you will need some work to upgrade them to 2.0. We are working on new documentation explaining how to upgrade from 1.X to 2.0, but in meantime to test this beta release we recommend disabling all extensions and enabling them one by one to see which ones will require updating. In case of doubt, feel free to send an email to the mailing list or ask on the IRC channel.

As with all beta releases, you are bound to find bugs or things that need improvement. We are still working on the proper stable release, and collecting all pending issues on this milestone on GitHub Issues:

https://github.com/okfn/ckan/issues?milestone=1&state=open

We would love to hear any feedback and bug reports, feel free to send them to the mailing list or add an issue to GitHub.

What’s coming next

We’re now working on a dataset approval workflow for moving datasets from private to public state as well as support for bulk actions and harvesting management interfaces. We’ll also be adding tweaks and improvements as people use the beta and provide feedback, as well as activity stream support for organisations.

CKAN is emerging as a de facto standard, according to a report into open data in Australia by the Office of the Australian Information Commissioner (OIAC).

The report, published last week, presents the result of a survey by the Commissioner into the state of open public sector information (PSI) across government agencies. It is structured around the Open PSI Principles published by OIAC in 2011.

191 agencies responded to the survey out of 245 contacted, and this high response rate illustrates the key finding of the report, that “Australian Government agencies are actively embracing an open access and proactive disclosure culture”. The report also reveals what agencies report as their greatest obstacles in the way of opening access to information more fully. These include technical barriers, such as whether existing information systems are well designed to support the move, and cultural ones, such as the need for more awareness and buy-in of the Principles inside agencies.

The report considers ‘next steps’ in Open Data and concludes, “The logical next step in facilitating reuse of government data is building data portals … In the arena of data portals, a small number of dominant software solutions are emerging as de facto standards. For example, CKAN has been widely adopted on national and regional levels.”

The report also mentions CKAN’s federation features, likely to be useful in Australia where a large amount of data is held by individual states and territories. The state of Queensland recently went live with the first CKAN-powered data portal in the country (above), built by the Open Knowledge Foundation and Australia-based partner XVT. The state plans to expand it soon to include a new site. The OKF has also built a demonstration platform for the state of South Australia.

The European Commission (EC) has unveiled a new data portal, which will be used to publish data from the EC and other bodies of the European Union.

This major project was announced last year, and it went live in December for testing before today’s announcement. The portal includes extensive CKAN customisation and development work by the Open Knowledge Foundation, including a multilingual extension enabling data descriptions (metadata) to be made available in different languages: at present the metadata is offered in English, French, German, Italian and Polish. The portal was originally planned for EC data, but it will now also hold data from the European Environment Agency, and hopefully in time a number of other EU bodies as well.

The EU has been a key mover in driving the Open Data agenda in member states, so it is fitting that it is now promoting transparency re-use of its own data holdings by making them available in one place. It has for some years been encouraging member states to publish data via dedicated portals, and it also supports the OKF’s work on publicdata.eu, a prototype of a pan-European data portal harvesting data from catalogues across the Union, via the LOD2 research project.

As the benefits of Open Data are recognised more and more widely, more data portals are appearing. Here are some recent new platforms with hot fresh data baked in a CKAN oven, as well as a couple that are coming soon. Only about half the portals mentioned here have had direct involvement by the Open Knowledge Foundation, a striking testament to CKAN’s open-source licence.

For a fuller list of CKAN data catalogues around the world, see this page.

Greek Open Data hub

The newest CKAN site is part of the Greek Open Data hub, which was announced today by the Open Knowledge Foundation’s Greek chapter. It includes a CKAN data repository as well as apps and info about the Greek Linked Open Data cloud.

Uruguay

Uruguay’s national open data portal catalogodatos.gub.uy was launched on 5 December. National and local government agencies can add datsets to the catalogue.

Aragon, Spain

Aragón Open Data provides data held by the Spanish region of Aragón. It is published by the Aragón government and went live this month.

Queensland, Australia

The state of Queensland launched the first governmental CKAN portal in Australia, Queensland Government data, in December 2012, with the help of the Open Knowledge Foundation. As the state premier said at the time, “The data held by the Queensland Government is the property of the people of Queensland. Therefore, where it is suitable for release, it will be released.”

publicdata.eu

The recent upgrade to publicdata.eu means that it is currently the only live site running the codebase for CKAN’s forthcoming version 2.0. Read more about the upgrade here.

Coming soon

Many other data publishers are looking at how to implement open data platforms. Here are a couple that have made announcements recently.

USA

As we noted in another post, the US government recently announced the decision to move its Open Data portal data.gov to run CKAN. This will for the first time combine general data in a single catalogue with the geodata previously catalogued at geo.data.gov. The Open Knowledge Foundation is working with the US government on the transition, including building new harvesting capabilities for geodata.

Germany

The German government has announced its intention to launch a national governmental data portal at govdata.de. The portal is being built by Fraunhofer Fokus, whose detailed description of the architecture of the site shows how CKAN will sit at the centre of the portal.