But quietly and unannounced, Hedtek has been expanding over previous months.

First we employed Ashish Sehra and more often than not, contractor Danny Ghilea. They both contribute agile developer skills in the Rails and Java areas. For our future development we also look to Ash’s past mobile experience. More recently Apache Software Foundation Member and friend Robert Burrell Donkin joined us. Robert brings twelve years of heavy-duty concurrent Java skills and contributes a very strong agile focus to our already very agile team. At the same time we appointed Alex Tse as a salaried Intern.

So now together with Hedtek staffer David Workman, we have a superb team of developers. A dream team, in fact.

All to often sudden growth like this destroys development companies, or leads to gross inefficiencies in development practices as the new staff members dilute the good habits that the company previously possessed.

We have been very careful about sudden growth, to the extent of waiting more than two years for the people who would fit with Hedtek’s development culture. As such, we were looking to employ people (i) who we already knew and had worked with and (ii) who have the key skills and the right kind of enthusiastic team-oriented personality to fit with us.

With the exception of our Intern, Alex, who comes well recommended, we have all worked together previously. We already know what to expect of each other; we all subscribe to very similar and compatible ways of working; and we all strive to help each other and apply continuous improvement practices.

It is this mutuality of experience and practice that has enabled smooth, effective and rapid growth.

Of course, we would be nowhere without our administrator and office manager, Renata Vitovjakova. Renata gets the prize for being the world’s most efficient person. Particularly, we suspect that she concurrently runs several kanban boards dedicated to administration and finance in her head.

Finally, we have Anita Workman working part time as a user-level and acceptance tester, and use Anita’s carefully honed skills to good effect in providing an independent opinion on the usability of our software.

It feels like Hedtek has been blessed

Past contracts have been

1. A small contract to provide assistance migrating the Jorum from Edina to Mimas.
2. A larger contract to provide temporary technical direction, diverse technical support, and to identify and start work on a way to be able to add front-end services to Jorum and upgrade Jorum’s DSpace core.

The third contract is

3. To supply technical direction (now handed over to a new member of Jorum staff, Ben Ryan), to supply diverse DSpace technical support (e.g. we are deep in DSpace’s OAI-PMH import code right now), and most importantly, to build a new front end for Jorum, while also ensuring sustainability, thatJorum technical staff have worked on and can continue to work on the new front end.

We hope to demonstrate this front end at the next (early Feb) Jorum Steering Committee Meeting; demonstrating resource discovery by browse and search facilities, resource pages, resource content download, and deposit of new resources.

Having Jorum as a client is great: The work is interesting and enjoyable, and Jorum staff are great people to work with.

Also, for me (Mark) it feels a little like coming home; some of the work that we are doing feels like a continuation of things I identified in a previous small consultative report for Jorum.

Achieving some rapid gains in user experience

We’ve been doing some mods to the existing Jorum/DSpace user interface to tidy up some aspects of that interface. This, however, can only be only a partial fix: Beyond making a few simple but effective mods to the existing user interface, it rapidly becomes becomes too expensive to modify further.

Providing a new Jorum user experience

The most cost effective way of providing a better user experience is to start again.  The preferable approach is to use the DSpace REST API to enable the use of a new front-end service for Jorum. This architecture is shown in the first diagram.

So far, Jorum’s user-knowledgeable staff have been acting as user representatives in collaborative evaluation of incremental deliveries with co-design of improvements for the next incremental delivery. Our plans are then to provide a public beta, enlist test users, and provide a web-based user feedback mechanism to involve beta users in progressive enhancements.

As it stands, the new front end already offers more useful functionality than is available via the current version of Jorum.

Opening up the Jorum architecture

Proving an API also allows the use of specialist front ends with Jorum, illustrated in the second diagram.

This is great, because Jorum does get approached for and wants to support this kind of (specialist) functionality. In effect, the API opens up the use of Jorum at an architectural level, enabling others to create their own front ends to access and contribute to Jorum’s content.

Notes 

Under the hood

The work here has been to extend the Jorum/DSpace architecture.

In order for a front end to obtain data from Jorum, the DSpace REST API was retrofitted to to the version of DSpace used by Jorum. The read side of the API was unit tested to ensure correct operation of the API.

Deposit into Jorum  is via DSpace’s SWORD 1.3 API, authenticating using Shibboleth.

The front end service is being built using Ruby on Rails and client-side technologies that include HTML, CSS3 and JavaScript.

Jorum is built on the DSpace repository, and part of our work involves building a new front end to Jorum. For this we need an API to DSpace, and while, conveniently, there is the DSpace REST API [1] module available for DSpace, it has not been used with the version of DSpace that Jorum uses, and until recently, it had no automated tests available for it. Given the centrality of this API for Jorum’s future development, we have started developing a suite of automated tests for the API. This post discusses progress, and mentions where our tests can be found on github.

Because the first front-end work we intend to perform is for OER resource discovery and download, so far we have only focussed on the ‘read’ functionality of the API. For this we have tested endpoints (URLs) that focus on read capabilities for:

• communities
• collections
• items
• bitstreams
• searching
• metadata harvest

For all of these, we wrote integration tests that tested the API endpoints running on DSpace,and then re-ran the tests on a full Jorum build.

The test development process

To implement the tests, we had to create a framework that would run the DSpace REST API using Jetty and control this programmatically.

We also needed to load database fixtures in order to have a known DSpace state to test against. When we started writing tests, loading SQL fixture files was problematic. We went through several iterations where, using a given means of loading our fixtures,  all our existing tests would run, but frustratingly, for the very next test we tried, our fixture load mechanism would fail. Eventually we discovered that the appropriate way of loading a fixture is via the DatabaseManager class in DSpace.

We also needed to trace a bug in the DSpace REST API code where database connections weren’t being closed. This bug caused intermittent freezes during our test runs, which in turn made a stable test framework impossible while the bug existed.

Later, fixture creation became quite intricate when we needed to test file downloads were correct, as this involved finding and editing download directories in the data. This was solved neatly by the creation of a command line script that generates a database dump from a DSpace database, processes it, and replaces all the download paths with a Maven token that can be replaced with the actual file path location when the tests are run.

To test search, we also needed to create and load a Lucene index for DSpace. This was primarily done by using DSpace to construct a more intricate test fixture for the search actions and then taking a copy of the Lucene index from the DSpace installation. The index is then copied into place for use by the embedded Jetty server during the test run.

The tests themselves do a lot of validation of JSON structure. This is done with the aid of a simple JSON library which processes the JSON into JSONObject and JSONArray instances. This was a good starting point and was used to write the first few tests. As more tests were written, there was a lot of messy test fixture matching going on. To reduce this, we refactored our tests and started building up a set of matchers and matcher methods that allowed us to write tests much more quickly. The end result of this process was the beginning of a DSpace-specific test matcher library. This process could be taken further, but the stage we got to was enough to enable us to write tests quickly and with a minimum of effort.

Running the tests

The tests are built and run using Maven, so if you want to run the tests yourself (e.g. on a new DSpace release), you just need to get a copy of the REST API with our tests added, available at github.

Once you have the code, you will need to create a postgres database for the test run. This is aided with the script ‘create_integration_test_db’. Once the database is set up, you can run the tests with the command ‘mvn test’.

Here is the end of a successful test run (please click on this thumbnail image to see the output detail):

There is a lot of ‘noise’ on the output of a test run. This is unfortunately caused by various parts of the REST module that are logging directly to the console, rather than using a logging framework. Hopefully this will be fixed in future development on the module.

[1] We would prefer to call the API restful, since it does not follow the HATEOAS principle. For further details see any good reference on REST.

So as a record, simply a list of highlights of some of the things that came out of the sessions:

Learning

Group work as learning

Content creation as a means of learning

Active learning

Learning through teaching

Experiential learning through problem finding and problem solving

Processes

Process creation for learning processes

Self-organised structure

Peer support

Curriculum

Self-designed curriculum

Community-responsive curriculum

Accreditation

Self accreditation in a public manner

This is a post that may disappear from this blog, it’s for COMP61511, a postgraduate course unit that we teach annually to postgraduate computer scientists at the University of Manchester.

This is a course that will be unlike any others you have done in your education to date; co-operative peer-assisted looms large: you end up working and learning co-operatively in teams.

The agile component of the course will have lab work in Ruby and RSpec. You learn the Ruby with some help from us, we teach you the fundamentals of how to use RSpec and then, again, you help yourself and others to learn.

Ruby Resources

Library books

• There are a bunch of Ruby books on short loan in the library near the Student Support Office on the LF floor of Kilburn.

Online books

• The Little Book of Ruby
• The first version of Programming Ruby (now a little out of date, but good enough to still use)
• WikiBook’s Ruby Programming
  
• Something a bit wackier, Why the Lucky Stiff‘s book: why’s (poignant) Guide to Ruby, links to various formats here, soundtrack to the book here (watch out for your speakers if you are listening loud).
• Yawn, yet more books …

Other resources

• RubyMine, the IDE in use on the lab machines and any home machines you need it on. It needs a key to work.
• You don’t need this IDE to learn Ruby, you can use any editor and the ruby interpreter (command ruby on the lab’s Linux image) or, better still, to mess around with Ruby, use the Interactive Ruby Shell (command irb on the lab’s Linux image).
• Or start by just trying Ruby in your browser (hint may have to hit return twice to cause expression evaluation to happen, note the use of the word “start”, this week you should do more than start)
• ruby-doc.org contains tutorial material, documentation for the Ruby Core API and the standard libraries
  
• Ruby User’s Guide
• There is one book there on Metaprogramming Ruby, this is advanced mateial, but some people may find it easyist (and more interest) to learn Ruby from this book
• General interest only: See this Smashing Magazine tribute to _Why
• In learning a new programming language, remember, Google is your friend, just Google the error message, Google the Ruby keyword, Google everything….

A few course thoughts

We provide an experience of authentic education, in the week 3-5 labs you are expected to work as part of a self-motivated professional team that helps each other. Setting your expectations, in this course there is no escape from becoming part of a co-operative self-assisting team, it is your unavoidable destiny.

We start by being relatively authoritarian in approach, this is because you have been institutionalised by your previous experience in educational systems, and we need to remove all the bad habits that your past education has taught you. Then we gradually and gently encourage you to take control of your own education. By the end of the course you are working successfully in groups, helping each other and learning from each other.

Some notes:

• First two weeks: You are programming in pairs first learning Ruby, and then learning RSpec.
• Last three weeks: You are in groups, and doing a programming project. You get to choose the project, or you can choose one we supply. Ambitious projects which dont have a chance of being finished are OK, part of learning to be a computer scientist, of gaining the identity and habits of a computer scientists, are to be able to make decisions about difficult projects, about what to do next. Remember, the most interesting systems that you can build are never finished, there is always something else to do, another extenion to implement.
• Weekly commitment: 2.5 days, as for any other course unit. Thursday is the formally scheduled day but we will have another day in a lab, with agile coaches. If necessary, we hold labs on Saturdays; expect this in weeks 3-5.
• Arriving late is a bad habit that many students have. As part of your commitment to this course, please NEVER be late, it just wastes your time and other people’s time in waiting around and in catch-up activities. Once wasted, time is irreclaimable.
• If you are pair programming with someone, or in a group with someone, you will need that person’s mobile phone number. Make sure you get it. If you dont have a mobile, or a UK mobile number, please organise to get those in the next week.

Except for the paragraph under the last heading this post has not talked about agile methods. The latter is really closely connected with you taking control of your work while working in teams. We guide you through all of that stuff.

The course also has a self-contained part about the structure of systems and UML, taught by John, this is not discussed in this post.

• We are very close to releasing a strategy report we have been commissioned to write for Society of College, National and University Libraries (SCONUL) on library user experience for users of Higher Education Libraries. We are pleased that thanks to work by Dave Randall, we have incorporated some limited-scope ethnographic work to underpin the strategic material that is brought out in the report.  So far well received, this report is out for review at the moment. If you’d like to help in that process please email mark -a-t- hedtek.com.
• In February we finished our work on the  JISC-funded fishDelish Project. We partnered with the University of Manchester’s School of Computer and the FishBase Information and Research Group Inc (FIN) to convert FishBase’s relational data to linked data, and to build two innovative systems around the linked data. Hedtek was responsible for all technical work, and during the course we created about 1.4T (109) linked data triples from FIN’s relational data and built two the systems that exploited that data, one to create ‘live documents’ based on the data, and one to support field observations.
• We have become involved in planning aspects of a large JISC Collections ERM system funded by the University Modernisation Fund. This project is, at the most fundamental level, directed at increasing the quality of scholarly resources in Higher Education. The system itself is a shared service to support electronic resource licensing and management information for JISC Collections and participating Higher Education institutions. So far, Hedtek have been involved in scoping and early design activities for this service.
• We are participating in a project to synthesise the outputs of JISC Activity Data Programme, where we have particular responsibility for the technical synthesis. Exploitation of activity data is an area that we first brought to JISC’s attention in the DPIE 1 Project (way back when we were called Personal Learning Environments Ltd). Eventually activitydata.org will host this work, for not it is strictly a site in progress.In a Higher Education context, attention data enables the construction of a variety of different kinds of services, from search engines and recommenders based on cohort activity (eg a prototype social search engine we developed for the MOSAIC Project) to learning analytics, where the idea is to identify students at risk of failing or dropping out from their behaviours, and then to intervene to help and try to retain the students in education.
• Recently we have been appointed to contribute to the technical direction and assist in the development of Jorum, a national web service to share learning and teaching resources within the UK Further and Higher Education community.  So far our work has been divided between operational and technical topics, soon we will turn to questions of the future development and assist in producing the next Jorum roadmap.
• In part, we’ve also been heads down in semi-stealth mode developing a large WebApp application for our product portfolio. Less said at the moment the better.
• We are starting to gear up for our annual hands-on postgraduate course on Agile Development in the University of Manchester’s School of Computer Science. For technical readers, this year we are changing the technologies we use in this course from Java and JBehave, and instead using Ruby with RSpec.
• In fact, the last six months have pleasingly seen somewhat of a blossoming of our activities in promoting agile methods: We are about to run our second agile event for the JISC/UKOLN DevCSI, we gave two master classes at Harper Adams, and together with Robert Burrell Donkin we ran two community events, one for XP-Manchester, and one for Agile Yorkshire.

First what is the OpenURL Router, and what is its data? What we need to know here is that the Router effectively enables library and other services to find the URLs for online bibiographic resources (more detail). A simplification is that the Router supply a translation from bibliographic data to the URL in question. The OpenURL router is funded by JISC and administered by EDINA in association with UKOLN.

Suitably anonymised OpenURL Router Data has been published by a JISC-funded University of Edinburgh/EDINA Project, the Using OpenURL Activity Data Project. This project is participating in JISC’s Activity Data Programme where Hedtek is collaborating in the synthesis of the outputs of the projects participating in this proramme. Thus my interest in the data and what can be done with it.

My initial interest was in who has what proportion referrals. Tony computed this, and I wanted to replicate his results. In the end I had a slightly different set of results.

Downloading and decompressing this CSV data was pretty easy, as was honing in on one field of interest, the source of the data being referred to. Tony’s post and the OpenURL Router Data documentation made it pretty easy to hone in on the 40th field in each line of this CSV formatted data.

My first attempts were to use a ruby gem, CSV, from the ruby interpreter irb. This went well enough but I soon discovered that CSV wouldn’t handle fields with a double quote in them. Resorting to the my OS X command line

  tr \" \'   < L2_2011.csv   > nice.csv

soon sorted that out.

It soon emerged that I needed to write a method, so I flipped to the excellent RubyMine, and soon hacked up a little script. Interestingly, I found that the representation of the site with the requested resource often had a major component and a minor component, separated by a colon, thus

 EBSCO:edswah
 EBSCO:CINAHL with Full Text
 etc

Having been excited by previous mention of Mendeley by Tony and wanting to find out the percentage of references to Mendeley’s data for another piece of work I am doing, I stripped out the minor component, and came up with the following code

While its open to a good refactoring, it did the job well enough, producing an unsorted list of results. A quick refeactor resulted in the following, which also coalesced both mendeley.com and mendeley.com/mendeley into one result.

To sort the output I used a command line sort after the script invocation

 ruby totals.rb | sort -nr

and obtained the following, here only listing those sites with more than 1000 references

99453 	www.isinet.com
44870 	EBSCO
34186 	undefined
34004 	mendeley.com
27545 	OVID
9446 	Elsevier
6938 	CSA
6180 	EI
4353 	Ovid
3399 	wiley.com
2558 	jstor
2553 	mimas.ac.uk
2175 	summon.serialssolutions.com
2070 	Dialog
1034 	Refworks

The rest, working out percentages, is easy thanks to Excel, see the middle column

A total of 25 funded projects were examined. Project outputs included analysis, user research, prototyping, service development and examination of key technical and operational issues. The projects concerned were comprised fifteen major projects and ten rapid innovation projects.

The major projects are Automatic Metadata Generation, Consent Management, CRIS, DPIE 1, DPIE 2, EIE, EnTag, Gold Dust, Intute Repository Search, MOSAIC, MyGeo, PERTAINS, OU TELSTAR, TicTOCs and TILE.

The rapid innovation projects are Clipper, CloudBank, Journal TOCs API, Library Social Widget, List8D, Scrutiny, Shuffl, SPACER, Visual History and Walking Through Time.

While we took Hedtek’s broad definition of personalisation as our starting point, the major areas addressed in the projects boiled down to six areas as shown in the following chart (dark and light green indicate implementation, cream indicates only consideration). To some extent this reflects what was current at the time; note how ten of these projects addressed personalisation as a “my stuff” content collection activity expressed in the rightmost bar in the chart. However, this will change; our prediction is that future funded personalisation activities will be in areas represented by the leftmost four bars in the chart; cf. the geospatial and activity data areas in this recent JISC call.

(In all charts here the y-axis represents numbers of projects.)

The beneficiaries of the projects were:

Of the projects surveyed, a significantly large number are either in production use or exist in demonstrator form:

The synthesis activity produced some key thinking that appears in the landscape section of our micro-site, and two technical recommendations:

Recommendation 1
Adopt the broad definition of personalisation: Personalisation is the supply of services and/or data based on a model of a user.

Recommendation 2
Field trial and improve personalisation mechanisms and their user interfaces as part of the regular pattern of construction and rollout of personalisation mechanisms.

Recommendation 1 broadens out the working definition of personalisation used in the JISC world, and, importantly, allows for a definition of personalisation in education that allows for self-directed education. Recommendation 2 will help ensure the applicability of personalisation work, avoiding mere flights of fantasy as to what users want and need.

Notes

Hedtek has been very active in the JISC personalisation activities, having enacted the first JISC personalisation project, Development of Personalisation for the Information Environment 1. DPIE 1 proposed an architecture for personalisation extensions to the JISC Information Environment, which we explored further in the Extensions for the Information Environment project.  Hedtek then, with Sero and other partners, undertook the TILE and MOSAIC projects; concerned with the exploitation of activity data for personalisation purposes. Next Hedtek performed the MOSAIC Architectural Recommendations project. Finally, together with Sero, we performed the survey and synthesis described in this post, and also provided advice to JISC on further funding activities.

Acknowledgement

In the work described above Tom Franklin of Franklin Consulting provided assistance with some survey activities.

Commissioned by JISC, this document provides an architectural approach for MOSAIC search and recommendation services. The approach incorporates iterative cycles of development, where each cycle is preceded by an agile needs-driven approach to determine the next targets for implementation and roll out.

From the document:

The MOSAIC search and recommendation approach was a successful outcome of the MOSAIC project. [1] The MOSAIC search engine demonstrated personalised search based on user context and activity data. However, we are in an early stage of development of MOSAIC search and recommendation: Search has been prototyped, and a successful user evaluation has been performed. Recommendation has not been prototyped, though the University of Huddersfield has developed a local recommender service.

Through the MOSAIC project, we extended and refined our understanding of the approach’s functionality and now consider that MOSAIC provides the following approaches

• MOSAIC search implements social search
• Individual resources are prioritised in the search results according to activity data, specifically use data derived from library circulation records. At a later stage this approach can be generalised to include data derived from access to electronic resources, from such as e-journals, VLEs and repositories.
• A user can personalise the search according to their own identity and results are then prioritised according to the user’s social group, their class cohort.
• Users can refine search results using facets to discover who borrowed or accessed what in another year, on another course, at another institution, and/or at another progression level.

• MOSAIC social recommendation is of various forms that include
  “Borrowers like you borrowed …” and “Borrowers of this item next borrowed …”

[1] See, inter alia:

van Harmelen , M.  TILE project: architectural proposals for creating context and enabling contribution. 2008. http://ie-repository.jisc.ac.uk/295/

Kay, D, Chad, K, van Harmelen , M, Pattern, D, Miller, P and Harrop, H. Making Our Scholarly Activity Information Count: The JISC MOSAIC Project. 2010. http://ie-repository.jisc.ac.uk/446/