Joe Geldart

The Adoption of Renewable Energy in Europe

2010-12-17T00:00:00-08:00

Eurostat (tables ten00076 and ten00081), Freebase.

This visualisation shows the adoption of renewable energy sources within the EU over the period 1997 to 2008. The orange wedges show the percentage of primary (i.e. produced within the country) energy that comes from non-renewable sources. The blue wedges that which is renewable. The size of each pie chart is proportional to the amount of primary energy produced within tha country.

It was written using Protovis, with the code based on the Dorling cartogram example. The data was merged and exported to JSON using Google Refine.

Energy Use per Year By Sector

2010-12-16T00:00:00-08:00

Validating with Verity

2010-11-21T00:00:00-08:00

Validations are important in many systems for maintaining data integrity with respect to some domain model. ActiveModel includes a set of validations which can be used in Rails for describing conditions on model objects, but these are limited to persisted objects and lack expressiveness and reusability; slightly altering the nature of a validation often requires a new custom validation to be written. Verity provides a generic framework for validating any Ruby object, whether persisted or not, and composing validations with an expressive, readable DSL.

Why do we validate?

Validation is important for expressing domain constraints, ensuring that the data which is stored in an information system makes sense. In a stock control system (for example) it would be nonsense for the weight (or quantity) of a stock item to be negative. Validations allow us to prevent such data from being persisted. Validations also allow us to check the sanity of the results of operations as a form of in process error check. Any errors from these validations can be presented to the users, or put into a log, in a readable form that explains exactly what went wrong and how to fix it. These improves the usability of the system.

A good validation library makes it easy to express arbitrary aspects of the domain model. The complexity of the code should be proportional to the complexity of the condition. A vital part of this is allowing us to decompose complex conditions into small, reusable parts that may be glued back together in many different ways.

Verity is my attempt to provide an expressive validation system. Unlike ActiveModel, the validations are expressed as a readable DSL. Unlike Semantic Attributes, there is no tie to ActiveRecord.

A humane validation library

Verity’s aim is to make validations readable and composable and, therefore, more maintainable. After spending too long writing complex, interdependent validations for a project with an (admittedly) tricky data model, I decided that enough was enough. Validations are a vital part of the definition of a data model in any real-world application, where data noise can be introduced by its users and code and must be prevented. As such, they should be as nice to define as possible.

Verity accomplishes this in two main ways:

It provides a DSL for defining the validations in a readable syntax similar to RSpec; and
It treats validations lazily, so that they can be composed to construct more complex validations out of simpler ones.

The main inspirations for the project are RSpec and Semantic Attributes, a similar attempt to define a DSL for validations (although without the composability).

As additional goals, Verity aims to:

Be agnostic as to what persistence framework, if any, is being used; it is as useful on plain Ruby objects as it is on ActiveRecord records;
Be easy to extend, with new predicates that can be shared with other developers and projects; and
Integrate well with whatever internationalisation framework is being used to support multi-lingual and multi-region applications.

Possible future goals include integration with YARD, for easy documentation of model constraints, and RSpec and Cucumber, for testing these constraints.

Using Verity

Enabling Verity's DSL and core API on an object is simply a matter of using the class method verifiable in the class definition. This is an extension added to Object when the gem is loaded:

class User
  verifiable
end

Instances of this class now have a number of methods defined upon them for performing validations. #valid? returns false if and only if the instance fails to satisfy the defined conditions and true otherwise (note that it will return true if no validations are defined at all). #validation_errors returns a Hash, keyed by attribute name, of any errors found on the instance. This allows you to report failures back to your users in a natural fashion.

Validations are defined using the syntax attribute_name_must (for positive validations) or attribute_name_must_not (for negative ones).

These validations are built using predicates, objects that represent abstract tests against values. The simplest predicate is a nil check, which matches values that are nil:

class User
  verifiable

  attr_accessor :login
  login_must_not be_nil
end

Multiple validations may be defined against an attribute and all will be tested:

class User
  verifiable

  attr_accessor :login
  login_must_not be_nil
  login_must have_length(:at_least => 4)
end

Predicates can take late-bound parameters, either positional or named:

class User
  verifiable

  attr_accessor :login, :star_count
  login_must have_length, :at_least => 4
  star_count_must be_greater_than, 5
end

Defining new predicates is as simple as defining a new class that inherits from Verity::Predicates::Base and a class method (available to your class) that constructs an appropriate instance of your class. Methods defined within Verity::Predicates are automatically available to all classes which define verifiable. As with RSpec, it is through judicious use of these methods that we gain our readability, so think carefully about how your definitions 'read'. In particular, be sparing with positional parameters (and treat them like direct and indirect objects in English) and use named parameters with keys that read like prepositional phrases (such as with_domain or at_least).

Lazy parameters

Often we want to validate based on data only available at run-time, and often specific to the record being validated. Named predicate parameters in Verity support lazy evaluation using blocks. A block-valued parameter is evaluated at validation time to produce a value, which is then set on that predicate. The blocks may, optionally, take a parameter which is set to the record instance being validated. This may be used to retrieve attribute values at run-time.

class Message
  verifiable

  attr_accessor :length, :message
  message_must have_length, :exactly => lambda{|r| r.length}
end

In fact, this usage is so important that we have a sugar defined for it using a method the:

class Message
  verifiable

  attr_accessor :length, :message
  message_must have_length, :exactly => the(:length)
end

It is planned to support this for positional parameters as well very shortly but this hasn't been implemented yet.

Future work

This library is in a very early stage right now. There are lots of places it could go depending on how I, and others, find use for it and the API is not settled yet. I have considered a few of the possibilities, and have concrete plans for some.

ActiveRecord integration

Like it or not, ActiveRecord is a very important library that is used in many projects. Unfortunately, ActiveRecord’s validations (now provided through ActiveModel) are cumbersome and this was one of the main motivations for Verity. I plan to allow Verity to be used as a drop-in replacement for ActiveModel’s validations. This will probably require some changes to the API to align the two. Ideally, I’d avoid an adapter library (such as many systems use) and a core dependency is out of the question; as mentioned previously, an aim of Verity is to be usable with any Ruby object and definitely outside of Rails.

Internationalisation

The world is full of variety, and especially linguistic variety. Verity’s message system is very crude at the moment, and it is a priority that it supports multiple locales. This does not only include translation of messages, but also localised validations for things such as dates, currencies, addresses and telephone numbers. Localisation and internationalisation should be supported in some form no matter what system Verity is used in, although we could still leave the translations to the outer system.

Conditional validation

Composability brings some exciting possibilities. One of these is allowing a boolean logic of validations. A given validation can be said to apply (or not apply) only when another validation passes. This could be used to avoid inundating the user with irrelevant messages or to define complex domain constraints within a single model. As an example, imagine a validation on a message body that checks it doesn’t contain any links, but only if it is HTML:

class Message
  verifiable
  body_must_not contain_selector("a"), :if => is_valid_html
end

Implementing these conditionals may require some changes to the core DSL to make them expressive yet humane.

Soft validations

We often deal with hard constraints, requiring that the user enter only perfect data. Sometimes, however, it can be useful to be more forgiving. I want to support soft validations that can be used to present a warning to the user without blocking saving of the object. As an example, an email should normally have a subject, but this is not required. Many email clients present a warning if you try to send an email without a subject. It would be useful to be able to express this warning in the domain model:

class Email
  verifiable
  subject_should_not be_nil
end

The violation of a soft constraint could trigger a confirmation message, or simply display a note remarking that a non-critical validation failed.

Quantifications

Expressing constraints across collections can be difficult in non-composable systems. We often want to define the validity of some object in terms of the validity of some, or all, of the members of a collection of associated objects. With a composable system, this becomes much easier:

class Book
  verifiable
  chapters_must all, be_valid
  contact_emails_must some, be_verified
  reviewers_must none, be_banned
end

The use of quantificational modifiers like all, some or none allows us to apply validations to one-to-many associations without adding irrelevant validations to those model classes. In the above example, a user isn’t invalid if they are banned but they are an invalid member of the reviewers association for a book.

Conclusions

Verity provides an expressive, readable DSL for defining domain constraints on arbitrary Ruby objects. It does this by defining a composable framework of predicates, similar to RSpec, and combinators. A certain amount of syntactic sugar is given to make it easy to not only use Verity to define validations, but also to make writing new predicates as simple as possible.

There are many exciting directions that this library could go in, and I look forward to getting feedback from others. I have outlined some of my ideas above. Composability gives some great opportunities for defining new syntax.

This library is intended, above all else, to be useful. Currently its utility is limited by the paltry number of predicates available. I hope to define, over time, a large library of predicates that will come out of the box. I also hope that others will build predicate libraries, and provide them to the community.

If you have any questions, remarks or constructive criticism please leave a comment on this essay. Even if you don’t plan on using the library yourself, or contributing any code, any help with firming up the DSL and API would be appreciated.

Hypercode

2010-05-02T00:00:00-07:00

Introduction

Yesterday, I read an essay by Sean B Palmer on his thoughts (and criticisms) regarding literate programming. The main thrust of his criticism is that literate programming is, all too often, used in the style of a programming language tutorial. Rather than documenting the rationale behind the code, it instead documents the boilerplate. This is attributed to a fundamental mismatch between the syntax of natural language and the chaotic, woven nature of human thought.

The joy of programming for many, as this article rightly states, is solving problems. The joy is not in the syntax, but rather it is in the conception. As with mathematics, a good problem is one which makes conception clearer, allowing us to understand and solve other problems.

This article introduces an attempt to realise the vision of programming contained within that essay. By combining linked data, linked process and literate programming, I propose a more humane programming system comprising a development method and a new linked process language. This is only a first attempt at a solution to this problem, and no doubt my opinions will change as I explore this space further, but I believe this captures my essential understanding of this fascinating problem.

A Brief Introduction to Linked Process

Linked Process is the behavioural equivalent of the, currently, more common idea of Linked Data. Linked data has begun to achieve a critical density on the Web with sites such as Facebook, the BBC and Best Buy exposing machine readable information embedded within their existing HTML pages. Linked Process hopes to capitalise on this by exposing programs as machine readable data over the Web. A linked process language would use the Web as its module system, and allow for orchestration and adaptation of behaviour in a decentralised fashion.

One of the first linked process languages was Ripple, a concatenative language using RDF as its data model. This language adopted a stack-based operational semantics, based on a stream of stacks model that was both familiar and powerful. The stream model allowed the language to exploit the non-determinism inherent in an open system like the Web.

Ripple, however, was not the first language to attempt to provide an RDF expressed ‘bytecode’. The Haystack project developed an a language called Adenine, with a loosely Python based syntax, for expressing operations on their RDF-based internal data-model. The representation was, however, purely syntactic and did not leverage either the semantics of RDF or the conceptual semantics of the developers.

A couple of years ago, I wrote a paper exploring the possibility of expressing the semantics of RDF using category theory. This was motivated by Ripple, the Cat programming language and my research into higher-order uncertain logics. Since that paper, I have been fascinated by the possibilities of linking research into web science with general research into category theory and, in particular, institution theory. This is an area with a lot of potential, particularly if we can connect Joseph Goguen’s work on cognitive blends in terms of institutions to the idea of linked data spaces and named graphs.

The remainder of this article looks at the potential for combining these ideas with those of literate programming to produce documents which are both human-readable specifications and testable realisations of those specifications.

Executable Specifications

A specification is, essentially, a description of the problem to be solved. Specifications can include many sorts of constraint. Some are purely functional, containing elaborate detail on the executable behaviour of a solution. Others are purely non-functional, and simply describe the motivations behind the desire for a solution to the problem. Most specifications are somewhere in the middle, combining some executable behaviour with business cases.

Recently, Behaviour Driven Development (BDD) has sought to formalise this balancing act. A specification is seen as a set of features. Each feature consists of a business case, usually expressed as a user story, coupled with a set of scenarios which turn that story into a testable narrative. There are a number of popular BDD frameworks such as Cucumber, JBehave and jspec.

These tools have a weakness, however, in that the implementation of the specification is utterly distinct from its description. Acceptance test tools like Fitnesse attempt to solve this by providing a user interface that can link tests to source code files. This is crude; you cannot see how a particular part of the specification is realised by a particular part of the implementation. This reduces traceability of requirements, and requires that the specification, including tests, be manually kept up-to-date with the implementation. In my experience in development, writing code frequently alters the understanding of the problem itself; in fact, that is the most enjoyable part of the process. These changes of understanding need to be manually percolated through to the specification. This is error-prone, but necessary if the client is to be kept in the loop on the development process.

On the Web, we navigate through a mind-boggling complex sea of information by looking at small fragments at a time. These fragments, which we call pages, are connected together by hyperlinks which can be navigated by a user, in their own time, to create a custom narrative exploration of the available information. These links also express relational information, a fact exploited by search engines such as Google, which may be used to tie together the meaning of the documents. This is the key insight of linked data.

I propose viewing specifications as linked data and linked process combined. A specification is then not only a description of a system in a human readable form, but also its realisation. When necessary they can be viewed together, or separately, depending on what is needed. They exist on the Web and can, therefore, exploit all the features and benefits of the Web architecture such as decentralisation, openness and partiality of description. Since the specification and the implementation are fundamentally linked, changes to the specification may automatically change the implementation, and vice versa. The specification is automatically and fundamentally in sync with the implementation at all times, and thus serves as accurate documentation of a project.

Whilst many representations of a program are possible, probably the best (for this purpose) is RDFa embedded in HTML. The HTML provides the specification whilst the RDFa annotates this specification with an implementation. As an example, consider the following definition of a Fibonacci number.

Fibonacci Numbers

Given a natural number n, the n^th Fibonacci number is defined as follows:

The 0^th Fibonacci number is 0;

The 1^st Fibonacci number is 1; and

If n >= 2, the n^th Fibonacci number is the sum of the n-1^th Fibonacci number, and the n-2^th Fibonacci number.

A specification for Fibonacci numbers.

To begin turning this specification into a linked process, we must first give it a URI by adding an about attribute. We may then add some basic metadata, such as the sort of executable unit being described by the specification (in this case it is a pure predicate) and the parameters it takes:

  <figure class="specification" typeof="prog:PurePredicate" id="fib" about="#fib" prefix="prog: http://www.example.com/hypercode/core.html#">
<blockquote>
<h3 property="dc:title">Fibonacci Numbers</h3>
<p>
<span rel="prog:arg"><span about="#fib-n" id="fib-n" typeof="prog:Argument">Given a 
  <a rel="rdf:type" href="http://www.example.com/hypercode/core-types.html#Natural">natural number</a> n</span></span>, 
<span rel="prog:outputArg"><span about="#fib-out" id="fib-out" typeof="prog:OutputArgument">the n<sup>th</sup> Fibonacci number</span></span> 
is defined as follows:
</p>
...

Basic metadata for an executable unit.

Of note in the above is the use of existing vocabularies, such as Dublin Core. One can freely annotate the specification with any desired vocabularies. For example, each executable unit could be annotated with its own licence in a machine readable fashion, or with its creator or with topics drawn from a thesaurus. The only limit here is one’s imagination.

Also worth noting is the use of two different sorts of argument; plain arguments and output arguments. One of the ideas behind this programming language is that predicates may be used in both a logical and a functional fashion. If an argument is an output argument, and there may be at most one of these for an executable unit, then the result of this, when used functionally, is the value of the expression.

The body of the specification consists of a number of cases describing the behaviour of this executable unit depending on a set of conditionals. Let us annotate the specification to implement the first two cases:

  <ul>
<li rel="prog:case">
<div typeof="prog:Case">
The <span rel="prog:condition">
    <span typeof="prog:equals">
      <span rel="prog:arg" resource="#fib-n"></span>
      <span property="prog:arg">0</span><sup>th</sup> Fibonacci number
    </span> 
  </span>
<span rel="prog:body">
<span typeof="prog:equals">
  <span rel="prog:arg" resource="#fib-out"></span>
  <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#equals">is</a>
  <span property="prog:arg">0</span>;
</span>
</span>
</div>
</li>
<li rel="prog:case">
<div typeof="prog:Case">
The <span rel="prog:condition">
    <span typeof="prog:equals">
      <span rel="prog:arg" resource="#fib-n"></span>
      <span property="prog:arg">1</span><sup>st</sup> Fibonacci number
    </span>
  </span> 
  <span rel="prog:body">
    <span typeof="prog:equals">
      <span rel="prog:arg" resource="#fib-out"></span>
      <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#equals">is</a>
      <span property="prog:arg">1</span>; and
    </span>
  </span>
</div>
</li>
...

The first two cases in the Fibonacci specification.

Each case has a set of conditions; if any of these conditions are satisfied then the case may execute. Execution procedes by finding a coherent ordering of all of the bodies. In this case, with only a single body, this is trivial.

Calls of executable units are encoded using classes; each executable unit is effectively the class of its executions. This allows execution traces to be represented as linked data, and thus we can bring to bear all the same tools.

It should be noted that, where possible, normal HTML hyperlinks are used to connect (for example) arguments with their definitions, and executable units with their specifications. This allows for normal hypertext usage, as discussed earlier.

Turning to the third case, we shall see an example of functional calling easing the specification:

  <li rel="prog:case">
<div typeof="prog:Case"> 
<span rel="prog:condition">
If
<span typeof="prog:Predicate">
  <a rel="prog:arg" href="#fib-n">n</a> 
  <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#gte">is greater than or equal to</a> 
  <span property="prog:arg">2</span>
</span>
</span> 
<span rel="prog:body"> 
then
<span typeof="prog:Predicate">
  <a rel="prog:arg" href="#fib-out">the n<sup>th</sup> Fibonacci number</a> 
  <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#equals">is</a> 
  <span rel="prog:arg">the 
    <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#add">sum</a> of 
    <span rel="prog:arg">the 
      <span rel="#fib-n">
        <a rel="prog:subtractand" href="#fib-n">n</a> 
        <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#subtract">-</a> 
        <span property="prog:subtractor">1</span>th 
      </span>
      <a rel="rdf:type" href="#fib">Fibonacci number</a>
    </span> and 
    <span rel="prog:arg">the 
      <span rel="#fib-n">
        <a rel="prog:subtractand" href="#fib-n">n</a> 
        <a rel="rdf:type" href="http://www.example.com/hypercode/core.html#subtract">-</a> 
        <span property="prog:subtractor">2</span>th 
      </span>
      <a rel="rdf:type" href="#fib">Fibonacci number</a>
    </span>.
  </span>
</span>
</span>
</div>
</li>
</ul>

The third case in the Fibonacci specification.

Other than its greater complexity, there is little new in this case apart from the use of a functional calling style. This allows the implementation to adhere to the structure of the specification more closely. I anticipate that in real specifications, a mixture of a relational style and a functional style would be used to match the way the specification was written.

I will finish this section by providing the fully annotated specification. If you look at the source code for this page, or run it through an RDFa extractor, you will find the executable definition.

Fibonacci Numbers

Given a natural number n, the n^th Fibonacci number is defined as follows:

The 0^th Fibonacci number is 0;

The 1^st Fibonacci number is 1; and

If n is greater than or equal to 2 then the n^th Fibonacci number is the sum of the n - 1th Fibonacci number and the n - 2th Fibonacci number .

The completed specification for Fibonacci numbers.

Of course, we are not limited to just HTML. We could annotate equations written in MathML, for example, or a state-machine diagram in SVG. Rather than translating whatever is the best notation for our problem into some programming language in a separate file, we simply add enough information to our specification to make it run. I believe this will eliminate a lot of the disconnect between conception and realisation, and reduce the number of bugs resulting from the mental conversion of concepts.

Tooling

Whilst I feel this way of writing programs as linked data annotations on specifications has a certain elegance, I believe this idea will really shine with good tool support. IDEs have become a common way to navigate code, but with this your browser can be your IDE working on your specification.

Imagine taking the above specification, before the annotations were added, and selecting the whole block. With a single command, the block becomes an executable unit, perhaps automatically setting the dc:title property on the heading. Select the text ‘Fibonacci number’ in the third case and another command could search through all linked processes to find those which look similar, in this case the very predicate we're defining. We can use the browser to incrementally mark-up our specification with behaviour.

Should we want a different view, say as a schematic table, then since the behaviour is described in linked data, this is trivial. In fact, people could write web applications that simply provide a different visualisation on the executable units in different web pages. Programs become data for mash-ups. An entrepreneurial developer could produce a service that turns a program into an execution diagram, or a dependency diagram, or anything for which there is a need.

With the wealth of data that is exposed by providing executable specifications in the form of linked data, search engines can be used to find routines others have written and published. Using them in your code is as simple as using their URI. If the routines are annotated with licencing information, then an IDE would be able to automatically work out whether or not there is a conflict. Search engines could also filter their results by compatible licences.

I believe that there are many opportunities afforded by this method, and by having the structure of the code available explicitly rather than serialised into a linear form. Rich comments can be defined, shown, hidden, linked and thrown away without touching the code itself. A web application could host collaborative comments for a team, without needing to touch the code just by using linked data principles. I am sure there are many more ideas that I haven’t thought of just waiting for someone out there to develop them.

Living Code

By making specifications executable, we can allow for live specifications. By embedding a Javascript interpreter in the page, we could run our implementation, test it and also test the assumptions in the requirements. This allows for a much tighter feedback cycle than current tools. By providing a link to the specification to a client, we can push that through to the full development cycle, allowing our customers to see how the specification works just by playing with it.

Test suites can be added to the specification as annotations, and previous executions (remember, they're instances of the specification) can be trivially turned into tests. Through the same distributed extensibility mechanisms as I have already mentioned, others can easily extend the testing mechanisms. Want to add something like QuickCheck? No problem. Just define the propositions in RDF and write a web application, or a piece of Javascript, to test them. At any point, one may navigate through an execution trace as if it were a piece of the Web, because it is.

We could also use this method to allow for linked process web-services over simple HTTP. A suitable server extension could react to POST requests on the document by running the appropriate code. Clients could find new services simply by using a semantic web search engine. A GET request on the document would yield a precise description of its parameters and behaviour, which could be used in the matchmaking and orchestration process.

Further Work

At the time of writing, this idea is less than 24 hours old so there is plenty of work to do. I need to define a vocabulary for expressing the executable behaviours. I need to define a formal semantics for the execution model. I need to write a Javascript interpreter and maybe a command-line one too. There needs to be a good IDE.

Please bear with me as I explore these thoughts further. I was excited to feel so many pieces of what I have worked on over the past ten years come together so nicely so I would like to thank Sean for his provocative essay. If you are interested in the ideas here, please contact me.