Introduction

If you’ve not heard of Headless DMS before, that’s because you’re reading it here for the first time. However, while the term may be new, the concept itself is not. CKAN, an open-source DMS, has been operating headlessly for years – the term headlessness just didn’t exist yet. It wasn’t until CMSs began calling themselves headless that the name gained traction. Here at Datopian, though, we were working with headless software long before it became cool.

In order to understand how Headless DMS – and specifically CKAN – is significantly improving the ways in which organisations manage their data, we first need to clarify a key term: headlessness.

What does it mean for a software to be ‘headless’?

To understand headless software, we first need to know some basic information about how software is structured. In software engineering, a distinction is made between the ‘front end’ part of a software, the part seen by the user, and the ‘back end’, the behind-the-scenes part. The back end is made up of a storage component (e.g. content repository or database), an editor (e.g. an admin user interface) and an API, which is a tool for delivering stored contents to the front end. The front end acts as a renderer, turning stored contents into a themed display (e.g. a webpage). Traditional DMSs, which contain both a front end and a back end that run in the same process, are known as monolithic DMSs.

Sometimes, software engineers choose to decouple the front end of the software (the ‘head’) from the back end (the ‘body’). There are many reasons for doing this, and these will be explored further on in this article.

You may remember the software stick-person from a previous article – here they are again, but this time they have been decoupled, leaving a headless part and a head:

Why go headless?

Datopian believes that shifting focus away from monolithic DMSs to a decoupled DMS approach could significantly benefit data-driven organisations. There are two main reasons for this.

Reason 1: greater specialisation

Using a monolithic DMS, in which the front end and back end are tied together, limits the extent to which each function of the DMS can be customised. The following table demonstrates the limitations of monolithic DMSs:

By decoupling the head and headless part of a DMS, we can build both parts using the latest technologies and practices specialised for each purpose. It also makes life easier for data portal developers, who now don’t have to worry about tackling the back end to make changes to the front end. This way, it’s easier to find developers for either end, and data portals are cheaper, faster and more flexible to build.

Reason 2: unlimited options

Monolithic DMSs can no longer keep up with the changing demands of users. Nowadays, users want to be able to integrate multiple sources, or push data from one database to multiple systems and devices. This is not possible with a monolithic DMS, which can only provide one back end and one front end.

For example, imagine a company using a DMS no longer just wants to display their data through the ‘attached’ data portal, but also wants to push this data to a smartphone/smartwatch app or a website. It can’t do this with a monolithic DMS. Neither could the company suddenly decide they wanted to have one front end (e.g. a data portal) that integrated information from multiple sources.

Let’s look at the different options presented by decoupling in turn.

Option 1: push data from one database to multiple devices (the ‘one body, multiple heads’ approach).

Option 2: integrate data and content from multiple sources (the ‘one head, multiple bodies’ approach).

Where does CKAN fit into all of this?

We mentioned earlier that CKAN was operating headlessly before headlessness became cool. Back in 2010, CKAN was used in headless mode to build data.gov.uk. Nowadays, most of our clients use CKAN as a monolithic DMS, i.e. with the back end and the front end unified as one system.

Recently, however, we at Datopian have been building a decoupled head for CKAN in javascript, called frontend v2. This is already in production with a number of clients, and allows us to deliver CKAN in two pieces: the headless component, and the head. We are currently working hard to make the head even better using the latest frontend technologies, React and Next.js.

From the perspective of our clients, not much has changed in terms of the way they use CKAN. However, deploying CKAN in decoupled mode improves their overall experience with the software. This is because frontend v2 makes it easier for clients to integrate content and data from both DMS and CMS for unified display via the same ‘head’. This is all made possible by CKAN’s rich API.

Is this the future

As mentioned earlier, through our work on CKAN, we were fine tuning the approach to headless software years before it even became cool. Thanks to recent developments in content management, we’ve now been able to give it a name: headless DMS. You heard it here first.

Technical Appendix:

There is currently no straightforward way to create unified front ends that integrate content and data. Here are some of the possible options:

• You start to develop a CMS (back end) in the DMS. This is far from ideal, as CMSs are good at what they do, given their rich admin UI and good structure.
• You put the DMS in front of the CMS (i.e. replicate content into the DMS). This would mean having to replicate content into the DMS and you have to develop theming in the DMS for content.
• You put a CMS in front of the DMS. This is even worse, as data portal functionality is data-focused, so you now have to replicate that functionality into the CMS.
• Or, you take the side-by-side option. This would mean having to maintain two themes and have a bifurcated user experience (and you may have to replicate things like user accounts).

Note: this is a post, written by Datopian. You can see the original post on the Datopian Blog.

You’ve heard of a content management system (CMS), but have you heard of a data management system (DMS)? In this article, we show that the two aren’t all that different. Just as many companies turn instinctively to CMSs to manage their content, we’ll explain why DMSs should be the natural go-to for any data-driven organisation.

Many companies are familiar with the term content management system (CMS). For those producing large amounts of content, investing in a CMS is the established practice. Type ‘content management’ into a web browser and well-worn software like WordPress and Contentful are top of the search.

Far fewer companies count the term ‘data management system’ (DMS) among their passive vocabulary. Enterprises producing large amounts of data, unlike their content-producing counterparts, have not traditionally enjoyed go-to solutions for managing their assets. They have only really had access to ad-hoc solutions for storing their data, like Dropbox or Sharepoint.

At least, until now. As a range of organisations begin collecting more and more data, the term DMS is starting escape expert circles. So, too, is knowledge about the open-source data management software CKAN.

What is a CMS?

A content management system is a software that can be used to manage the creation, modification and display of website content. In other words, it is a tool that allows non-expert users to create a website without having to code one from scratch. A well-known CMS is WordPress.

A traditional CMS allows users to:

• Store content, such as blog posts and images
• Edit, update and add content
• Display themed content on a website
• Share content internally

What is a DMS?

A data management system is a software that can be used to manage the storage, modification and display of data(sets) in a data portal. These could be internal data portals, used to manage data within an organisation, or open data portals, which are used to share data with the public.

DMSs are becoming increasingly popular among a wide range of data-driven organisations, from governments to private companies. This is because DMSs allow organisations to do much more with their data than simply store it across ad-hoc solutions like Dropbox or OneDrive. A traditional DMS allows you to:

• Store metadata about data stored elsewhere
• Discover data
• Edit, update and add data and/or metadata
• Display and visualise data
• Share data internally and externally

CKAN is an open-source DMS, which means it can be extended to provide new features based on different user needs.

Comparing CMS and DMS

The design and function of a DMS is very similar to a CMS. Both systems are made up of the same basic components:

• A system for storing information
• An interface for creating and editing information
• A component for rendering the stored information in a user interface (UI) and often in an API

You might think of a CMS and a DMS in terms of a human body, with the head as the front end and the body and legs as the back end (note: in software engineering, a distinction is made between the ‘front end’, the part of the software seen by the user, and the ‘back end’, the behind-the-scenes part of the software). Each part of the body serves a certain function.

TIP

It is important to clarify here that the type of CMS and DMS under discussion in this article are traditional, or ‘monolithic’ CMSs/DMSs. For these software to function properly, each part of the body has to work together as a unified system. More recently, new approaches to management system software (known as headlessness or decoupling), in which the different parts operate independently of one another, are gaining traction. Look out for an upcoming post on this from us soon!

Let’s compare the two management systems in more detail.

Whereas a CMS publishes web pages, a DMS publishes datasets. That being said, new approaches to data management make it possible to display both content (such as blog posts) and datasets via the same front end. This is thanks to the ‘headless’ movement within content and data management, which forms the subject of an upcoming Datopian post.

Alongside the ‘eGovernment Act (EGovG)’, the Federal Government’s ‘Digital Administration 2020’ programme made working towards digitisation in Germany a legal requirement for public administrations. For the first time in the federal republic’s history, the federal government outlined a plan for the provision of open data by means of the open data provision in Section 12a EGovG.

N.B. In this context, open data is defined as data that is made publicly accessible under a free licence and may be used and reused without restrictions.

But… what about data security?

Despite these legal initiatives, many administrations still hold back when it comes to publishing data openly. Data privacy is a deep-rooted concern among many Germans and open data is thus often greeted with suspicion. In order to make the most of the opportunity presented by the country’s new legislation, German public authorities must be granted a better understanding of the advantages created by open data.

After all, the advantages are considerable. Firstly, open data allows public administrations to supplement their own data with that of other organizations, thereby strengthening their basis for informed decision making. A great example is the Open.NRW, as well as the Vergabeportal.NRW, which facilitate the exchange of open data between authorities in the German state of Nordrhein-Westfalen. In this sense, publishing public data openly affords both administrations and the general public a level of autonomy that would not otherwise have been possible when it comes to making fact-based, accountable decisions. It also encourages public trust in government and enables new forms of cooperation and communication between civil society, business and the state.

Open data success stories

Before public data is published openly, the authorities publishing it cannot know exactly what will happen to their data, and this can feel daunting. However, public administrations should try to think of this uncertainty as a positive. It is perhaps obvious to say that the chances of one person in a single organization coming up with the optimal use for that organization’s data are slim. If you really want to unlock your data’s potential, then you need to let everyone look at it, because the likelihood is that the best idea for how to use your data will be discovered by someone else.

Moreover, there are numerous open data success stories in public administration, and many of these were aided greatly by CKAN. In Great Britain, the home of the now international NGO the Open Knowledge Foundation, many institutions are already reaping the benefits of open data. With the help of Open Knoweledge Foundation and Datopian, the government of Northern Ireland has built a CKAN-based data portal, for which new use cases are constantly emerging. Belfast City Council, for example, has identified commercial properties that were marked as available by estate agents, when in fact they were being used by businesses, which has enabled the Government of Northern Ireland to collect £350,000 in unpaid taxes.

To encourage more such use cases, the initiative ‘Open Data Northern Ireland’ has established a fund to support the design of open, data-driven apps. Anyone who has an idea for an app that uses open data to somehow benefit the public is eligible to receive a grant of between £300-£5000. As part of the initiative, the authorities have also created an online form where citizens can make suggestions for datasets that they would like to see collected.

Like many of public administrations in the EU, the cities of Scotland also wanted to use open-source software to empower their open data programme. Some data sets on their CKAN portal have received thousands of hits, with some unlikely data sets proving very popular, such as one that provides information on cemeteries in the city of Sterling. Following the publication of this data set, which provides records of gravestones, several citizens have contacted the authorities to inform them of inaccuracies. This is a great example of how public administrations can use CKAN open data portals to improve the quality of existing data.

Promoting open data in Europe and Germany

Upon Britain’s exit from the EU, a large part of Europe’s open data knowledge will most likely go with them. With the statutory Open Data regulation in § 12a EGovG and the government programme “Digital Administration 2020”, Germany is already on the right path to filling this knowledge gap.

Note: this is a post, written by Datopian. You can see the original post on the Datopian Blog.

This work is inspired by the excellent ckanext-composite extension and replaces ckanext-repeating repeating_text fields.

You must be using CKAN 2.8 or later and a custom IPackageController plugin to index datasets with repeating subfields.

Repeating Subfields

Repeating subfields let you define a group of fields to repeat within dataset and resource forms. All of the features available for normal ckanext-scheming fields can be used in subfields.

repeating_label may be used to provide a singular label for each group of subfields.

repeating_subfields contains a list of field definitions to repeat.

- field_name: submission
  label: Submissions
  repeating_label: Submission
  repeating_subfields:

  - field_name: date
    label: Date
    preset: date
    required: true

  - field_name: text
    label: Text
    preset: markdown
    required: true

  - field_name: flags
    label: Flags
    preset: multiple_checkbox
    choices:
    - label: Draft
      value: D
    - label: Approved
      value: A
    - label: Response Required
      value: R

Data stored in subfields is represented as lists of JSON objects in the API.

"submission": [
  {
    "date": "2021-02-01",
    "text": "an example submission",
    "flags": [
      "D"
    ]
  },
  {
    "date": "2021-02-05",
    "text": "another one",
    "flags": [
      "A",
      "R"
    ]
  }
],

Normal and custom validation rules apply and are displayed in the form by referencing the subfield group and the field with the error, e.g. “Submission 2: Text: Missing value”

Indexing Repeating Subfields

Repeating subfields can’t be indexed in Solr with CKAN’s default schema and indexing code since they expect extra fields to contain simple string values. Create a new IPackageController before_index plugin and Solr schema to handle indexing repeating subfields the best way for your own site.

class SubmissionsIndexPlugin(p.SingletonPlugin):
    """
    Map submission dataset fields to Solr fields
    """
    p.implements(p.IPackageController, inherit=True)

    def before_index(self, data_dict):
        flags = set()
        text = []
        for sub in data_dict.get('submission', []):
            text.append(sub['text'])
            flags |= set(sub['flags'])

        # replace list of dicts with plain text to prevent Solr errors
        data_dict['submission'] = '\n'.join(text)
        # index flags present in any submission
        data_dict['submission_flags'] = sorted(flags)

        return data_dict

For submission_flags to accept multiple values we must add a multivalued field to our Solr schema <fields> configuration:

<field name="submission_flags" type="string" indexed="true" stored="true" multiValued="true"/>

These new fields will now be available for use with CKAN advanced search e.g. submission:example or submission_flags:D.

If you don’t need advanced search or faceting based on repeating subfields you may use the included scheming_nerf_index plugin. This plugin passes repeating fields to Solr as JSON strings to prevent indexing errors instead and doesn’t require a customized Solr schema.

Future CKAN support for dynamic fields in Solr will simplify this required configuration.

Future Work

Some features not yet supported:

• validating the number of subfield groups (e.g. requiring at least one)
• nesting repeating subfields

If you need these features consider discussing your work on a new issue and working a pull request.

Multiple Text

Multiple text fields improve on the repeating_text fields from ckanext-repeating with a dynamic form with add and remove buttons.

- field_name: contributors
  label: Contributors
  preset: multiple_text

Data stored in repeating text fields is represented as lists of strings in the API.

"contributors": [
  "Person A",
  "Person B",
  "Person C"
],

required: true may be used to require at least one entry. Per-field and other types of validation are not yet implemented. Add a comment to the multiple text validation issue if you would like to work on this feature.

original post

These patch releases fix important bugs and security issues, so users should upgrade as soon as possible to the latest patch release for the CKAN version they
are using.

Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes.

• https://docs.ckan.org/en/latest/changelog.html#v-2-9-2-2021-02-10
• https://docs.ckan.org/en/latest/changelog.html#v-2-8-7-2021-02-10
• https://docs.ckan.org/en/latest/changelog.html#v-2-7-10-2021-02-10

In case of doubt don’t hesitate to ask for support in the mailing list or the Gitter channel.

As stated in the release policy, the latest patch release is the only one officially supported.

For details on how to upgrade, see the following links depending on your install method:

Package upgrade

Source upgrade

These patch releases fix important bugs and security issues, so users should upgrade as soon as possible to the latest patch release for the CKAN version they
are using.

Patch release upgrades are very straight-forward and do not contain any backwards incompatible changes.

• https://docs.ckan.org/en/latest/changelog.html#v-2-9-1-2020-10-21
• https://docs.ckan.org/en/latest/changelog.html#v-2-8-6-2020-10-21
• https://docs.ckan.org/en/latest/changelog.html#v-2-7-9-2020-10-21

In case of doubt don’t hesitate to ask for support in the mailing list or the Gitter channel.

As stated in the release policy, the latest patch release is the only one officially supported.

For details on how to upgrade, see the following links depending on your install method:

Package upgrade

Source upgrade

The CHANGELOG contains all details about this release. Check the installation documentation for instructions on how to get it up and running, if you are planning on upgrading an existing site there are also upgrade instructions. Make sure to check the migration notes  as there are important considerations and changes needed on existing sites.

What’s new

Here are some highlights of what’s included in this release:

• Dataset collaborators, for more granular permissions management at the dataset level

• API Tokens  allow more flexible and secure integrations with other tools (legacy API keys are being phased out)

• The new package_revise API action allows safe concurrent updates and simultaneous uploads

The CHANGELOG covers all changes in detail.

For developers

There are many more changes under the hood to make live easier for developers. CKAN 2.9 culminates the migration to Flask, and adds support for Python 3. CKAN 2.9 will be the only transition version where both Python 2 and 3 are supported. The next CKAN version will only support Python 3. There is a migration guide for extensions available on this wiki page.

Across the board there has been work to modernize different components and replace them with modern alternatives: new pytest based test suite, CLI and development server, database migrations, static assets, etc.

Patch releases for 2.7 and 2.8

Existing sites should upgrade to the latest patch releases (2.7.8 and 2.8.5) to include fixes and address security issues. The latest patch release is the only one supported by the CKAN team. For details on how to upgrade, see the following links depending on your install method: Upgrading CKAN.

If you find any issue, you can let the Tech team know in the mailing list or the Gitter channel.

Datopian are the co-creators, co-stewards and one of the main developers of CKAN. They work with government, NGO and enterprise clients worldwide to build CKAN-based, data-driven solutions, from open data portals to internal data management systems. Through monitoring client use cases for data across the private, public and third sectors, Datopian ensure that CKAN is responding to genuine challenges faced by real organizations. As with many of their feature releases, the suscribe to datasets option was built to address client needs.

#Why subscribe to datasets with CKAN?

In order to provide clients with a robust messaging system, Datopian needed to build a feature outside of the main application process.

Before Datopian developed a subscribe to datasets feature, data portal users had no good way of finding out about changes to datasets. Approaches to notifying users of changes include using RSS feeds or CKAN’s built-in email integration. However, these approaches were not applicable for their client’s context because:

• Some datasets and resources can change rapidly, and many different types of stakeholders can subscribe to change notifications. This means that anywhere from 50,000 to 200,000 notifications may be broadcast in a given month.
• Datopian’s client wants to extend the notification feature to support additional notification channels as well as email. A next iteration will add SMS notifications, giving users the choice to receive notifications by SMS, email, or both.

Another advantage of the feature is that the granularity is high. Users can currently receive the following information via email notifications:

• The name of the datasets in which a change has taken place.
• Whether the change was applied to a whole dataset, or a single resource.
• Whether there were changes to the metadata.

Here’s an example notification:

#Overview

#Current features

1. Configure notification frequency – system administrators can determine the frequency with which users receive email notifications. This is particularly helpful for users subscribed to very large datasets that are updated multiple times per minute/hour.
2. Disable notifications for certain datasets – system administrators may opt to disable notifications for certain datasets for a number of reasons. In particular, companies using CKAN data portals may choose to disable notifications for datasets that are updated frequently, should the cost of mass emailing become too high.
3. Subscribe to new datasets – CKAN users can receive emails notifying them when new datasets are added to the portal. This is particularly helpful for users monitoring all portal activity.

#How can I get the new feature?

The data subscriptions service is currently available for use. If you are interested in deploying it against your existing CKAN installation, please reach out to us by visiting the project on GitHub here and creating an issue. Additionally, contact Datopian to discuss how we can deploy a data subscription integration for your platform.

Want to work with Datopian? We are data management experts providing open-source tooling and related services to organisations worldwide. Check our website for more information or contact us.

© Datopian (CC Attribution-Sharealike (by-sa)).

We will cover the new features in much more detail in future posts, but expect Python 3 support, a completely revamped interface for tracking changes in datasets over time, more granular permissions at the dataset level, API tokens, concurrent safe API endpoints and much more.

CKAN is a community-led project and this has really shown in the build up towards the 2.9 version, which has received patches from over 100 contributors, more than any CKAN version before. Now that we are at the final stages, all CKAN friends can help make this the best version ever.

Here’s how you can help:

• Testing, testing, testing! Help us iron out issues by giving the new version a go. https://beta.ckan.org should contain the latest release branch.
• If you are comfortable installing CKAN, you can do a source install using the dev-v2.9 branch. It’s especially useful to test it with extensions and non-default configurations.
• We mentioned that CKAN 2.9 is the first version to support Python 3, but there are still several extensions that haven’t been ported. This page in our wiki lists the most popular and their status. If your organization uses any of them please consider contributing Python 3 support for it. The page contains details on the steps needed, and there are several extensions where work is already under way.
• Translations. If you are a non-English speaker and want to help make CKAN available in as many languages as possible you can register to Transifex and use the online editor to edit the 2.9 resource.

Please flag any issues you find during testing in the CKAN repository issue tracker, or drop by the Gitter channel if you have any question.

 
Featured image by skeeze from Pixabay

Since the portal’s launch, several design enhancements and data request workflows have been implemented as customizations on top of stock CKAN capabilities and plugins. The portal is available in both English and Turkish. 

The IMM Open Data Portal

At the time of writing this post, the portal holds:

• More than 120 datasets, amounting to 50 GB of raw data.
• More than 120 new dataset requests, over 60% of which have been resolved.
• Around 500 registered users.

The IMM open data portal is one of the first CKAN portals to offer search integration with Google Datasets. The CIO of IMM, Erol Ozguner, is committed to Open Data and believes that this initiative will provide a launch pad for open data in Istanbul. He also added that new capabilities and datasets will be deployed before the end of 2020, including better API-based data access, data format coherency and improved compatibility with mobile devices.