As one of the UK’s leading research universities, we are renowned for the strength, quality and breadth of our research and teaching capabilities.

To further bolster our commitment to research excellence, we are delighted to share that we have implemented a new, state-of-the-art High-Performance Computing (HPC) system.

The names of the new facility, “Ada”, is a nod to Ada Lovelace, the gifted mathematician recognised for creating the first computer programme. The name is a fitting tribute to this female pioneer, who also had ancestral ties to Nottinghamshire.

Ada replaces our former HPC facility, Augusta, which was in place for five years.

Professor Jonathan Hirst, Professor of Computational Chemistry comments: “technology continues to evolve at an extraordinary pace and it is crucial to ensure our HPC infrastructure can meet growing demand accordingly. Ada will be available for experimentation beyond larger, funded projects, leading to more ambitious research and enabling funding, more high-quality outputs and increased inter-disciplinary collaboration”.

Professor Phil Williams, Professor of Biophysics adds: “the new facility will enhance research capabilities across a wide range of disciplines, fuelling innovation and growth in strategic areas of the University, such as quantum technologies, nanoscience, artificial intelligence, imaging, and bioinformatics. It’s a really exciting step forward for the University of Nottingham”.

What is a HPC facility?

The on-site High-Performance Computer (HPC) is available to any UoN research student or member of academic staff.

The HPC allows users to process, analyse and store increasingly large amounts of data and perform complex calculations at high speed to enable high quality and valuable research outputs.

The new HPC service will also support computationally intensive research performed in areas of the university that are not traditional users of high-performance computing.

Strategic partner OCF was responsible for the design, development, and installation of the cutting-edge HPC facility.

Access, guides, and more information

For access, guides, and more information, please visit our dedicated SharePoint pages.

The post Introducing Ada, the University of Nottingham’s new, most powerful HPC service appeared first on Digital Research.

The University’s High Performance Computing (HPC) service is a research facility available to academic staff and research students, from any School or Faculty, who have the need for computing resource substantially greater than a standard PC.

Enabling research that changes the world

As one of the UK’s leading research universities, UoN is renowned for the strength of its research, and quality of data, through which ground-breaking scientific discoveries are made and innovations are fuelled.

But, as technology continues to evolve at an exponential pace, it is crucial to ensure our HPC infrastructure can meet growing demand accordingly.

Our current HPC service (Augusta) will reach end of life soon. We are therefore in the process of implementing a more powerful service, to ensure we have the facility to process and analyse increasingly large amounts of data, and perform complex calculations at high speed to enable high-quality and valuable research outputs in key research areas.

Implementing a new, more powerful HPC

The new HPC service is anticipated to be fully functional by January 2024.

The new hardware has been procured, and we are currently awaiting delivery, after which the hardware and software will be installed, tested, and all existing services migrated to the new service.

What are the benefits of the new HPC service?

• The new HPC service will be more widely accessible to support increasingly compute-intensive work performed in areas of the University that are not traditional users of HPC, enabling better accessibility to compute service and skills
• In addition, there will be support staff in place to make it even easier for new users to utilise the HPC service
• The service will also be available for experimentation beyond larger, funded projects, leading to more ambitious research and resulting funding, more high-quality outputs, increased inter-disciplinary collaboration, attracting researchers and PGRs
• A higher capacity and more versatile on-premise compute service will allow for more ambitious calculations
• Better data storage and connectivity
• Higher throughput of jobs and support for more efficient use
• Increased ambition for research on national-level HPC facilities

Want to know more or provide your feedback?

Should you have any questions regarding the HPC, or would like to know more, please visit these pages or contact Colin Bannister in the first instance.

The post Accelerating the University’s research ambitions with a more powerful HPC service appeared first on Digital Research.

In this blog, we describe how to translate text using some simple Python code and the Azure translator service.

Azure Text Translator

Azure Cognitive Services Translator is a cloud-based service that enables quick and accurate translation across many languages. The translator service can be used for:

• Language detection
• One-to-one or one-to-many translation
• Script transliteration (text conversion from its native script to an alternative script)

Supported Languages

The Azure Translator Service (v3.0) supports 129 languages. If the source language is known, you can specify this. If the source language is unknown, Azure can detect it. For a list of supported languages (along with their language codes), please refer to the language support page.

Pricing

Azure provides a free tier for the translator service. The free tier allows for 2 million characters per month (about 1,500 pages of single-spaced, 12-pt font text). If you intend to translate more than this, see the pricing page (UK South region).

How to use the Translator API

Create The Translator Resource

You will first need an Azure account. A free account can be created here. After creating your account, a translator resource must be provisioned:

• Login to the Azure Portal.
• Create your translator service resource by selecting it from the menu at this link: Create Translator.
• Select the region where you want the resource to be located. The region dictates where your data will be stored and processed. If you want your data to be stored and processed in the UK, then select either “UK South” or “UK West”.
• For pricing, select either the Free (F0) pricing tier or the pay-as-you-go (S0) tier.
• Click Review + Create

Get Your Authentication Keys and Endpoint

To use the API, you will need the authentication keys and endpoint of the resource you have just generated.

• After the resource is generated, click “Go to resource”.
• In the left column, select “Keys and Management”.
• This will open the following page:

Copy Key 1 or Key 2 as well as the endpoint URL. These will be used in the python code (see below) to make the translation request.

Python Setup and Code

Once you have your keys and endpoint, please visit this SharePoint page for instructions on setting up your python environment and for the copy-and-paste code. This is the final step to using the service and translating your texts.

If you are interested in talking about this process, please feel free to contact one of the team.

The post Text Translation using Azure Cognitive Services Translator appeared first on Digital Research.

In this blog, we explore the speed and efficiency of using the programming language Julia on the University’s high-performance computer.

This blog has been guest-authored by Jamie Mair, a PhD researcher in the School of Physics and Astronomy.

The repository for the code given below, which was presented at this year’s annual UoN HPC conference, can be found here.

For more information on the University’s high-performance computer, please see here.

What is Julia?

Julia is a modern, general purpose, programming language. It was designed for high-performance scientific programming. According to a blog post written by the creators, Julia was borne out of the desire for a programming language which is as fast as C, yet easy to use like Python, MATLAB and R. Personally, I use Julia because it is incredibly easy to prototype new code for my research, while being flexible enough to execute my code across my home machine, my GPU or an entire cluster. And all of this with minimum effort on my part!

In this blog, I discuss what’s possible in Julia for very little investment. You will see how to write reusable code and take advantage of every level of parallelism, from the hardware parallelism of SIMD, to executing native Julia code on the GPU and scaling it across a cluster. The ease of use and flexibility of this parallelism is unmatched and provides a very low bar to entry.

Julia is fast

Julia has many advantages when compared to other languages. However, the main claim that will interest our HPC community is speed. Julia claims to be as fast as C, so let’s put that to the test.

I will use the example of simulating a Monte-Carlo process, specifically a random walk. This process is simple, but the basics are used in a wide variety of fields and are common in HPC workloads.

The code for running a random walk is:

This function performs n random walks, all of length T. We can write some code in C++ that is roughly equivalent, to see empirical differences in performance.

This implementation in C++ is not optimal at all (I am no C++ programmer), but is indicative of what a typical PhD student might write for simulations. We find that the Julia version is between 4 and 7 times faster (depending on the size of n).

While Julia has the advantage here, I don’t think that Julia is inherently faster than C/C++. A better C/C++ developer could make the C/C++ code perform similar to the Julia code. The main point I want to make is that you can achieve C/C++ levels of performance for minimal effort, while keeping your code simple and trusting Julia’s “Just in Time” compiler (the standard LLVM compiler) to produce fast machine code. Not all code will run quickly, but there are performance tips that will let you achieve these C-like speeds.

Multi-threading in Julia

Almost every modern CPU has multiple cores, thus providing additional compute power. Julia provides lightweight multi-threading support natively. The language provides many ways of parallelising your code, but here we’ll just use a macro to make this process easy. A macro is code that writes other code. For executing a loop in parallel, we just add an @threads macro from the standard Threads library:

This is essentially the same source code as above, with just a simple change in front of the for-loop. We can add this to the performance graph:

We can see that this is pretty good performance out of the box. Unfortunately, we can see that threading doesn’t reach the theoretical maximum speedup, though there are techniques to remedy this.

Moving to the GPU

The modern GPU excels at array-based operations at scale, such as standard BLAS operations (matrix multiplications, etc.), which make up the backbone of a lot of numerical and scientific computing. Writing code for a GPU is notoriously difficult (custom kernels in C present a high barrier to entry for beginners).

The good news is that Julia has a successful package, called CUDA.jl, which allows you to write native Julia code that will compile directly to GPU code. Let’s take it for a spin!

Up to now, we have focused on using the standard Julia libraries, but there is a rich ecosystem of packages available. Adding a package is as simple as running the following code:

This uses Julia’s built-in package manager to download the CUDA.jl package. What makes this process painless is that when you use CUDA for the first time, it will download the CUDA libraries you need to get things working instead of having to install CUDA manually (as you need to do in Python libraries like Tensorflow and PyTorch).

GPUs are very different devices to CPUs and have a slightly different programming model. The main difference is that scalar indexing of an array has really poor performance on a GPU, and so the CUDA.jl package errors when you try to do this. Instead, we can use Julia’s excellent “broadcasting” notation, which allows us to write concise “vectorised” code. This notation also allows us to write non-allocating code, which means that we can reuse memory. This can make a huge performance difference. Here’s the same random walk code, rewritten to use array-based notation and avoiding allocating memory:

Please note that the use of the ! at the end of the function name is just notation that tells us that the function mutates the input arguments.

This code runs fine on the CPU with arrays, and is very similar to how someone would write this in MATLAB or numpy (in Python). We can benchmark this function and add it to the graph:

Notice that this has slightly better performance than the serial implementation, as it can make use of hardware SIMD instructions. What is really powerful about this approach is that we can directly re-use this code to run the code on the GPU:

There are no changes to our code!

The only thing we changed was the type of our array. Since Julia has a Just-in-Time compiler, which specialises on the type of the input arguments, it knows to perform all of the array-based operations on x on the GPU. We can see the performance below:

Here we can see that it is only really worth using the GPU for many elements in our array. If we require millions of calculations, the GPU can be around 100x faster than the CPU! We were able to get this huge increase in performance by only changing the input type to our function, and were able to reuse our original code.

Here, we have really only scratched the surface of GPU programming, as you can always move onto writing your own kernels from scratch. For a more detailed review, check out the CUDA.jl documentation.

Scaling up to cluster

This wouldn’t be an HPC blog without talking about using a cluster. While it is certainly possible to use a single node with multi-threading, Julia also has native support for multiprocessing to scale across multiple nodes. Julia has native support for multiprocessing (with the Distributed.jl standard library that comes with Julia), and support for schedulers like SLURM with ClusterManagers.jl.

In order to connect all of the nodes, all we need to do is run the following code:

We could use the parallel primitives as described in the Distributed.jl documentation. But let’s reuse some of our previous code and take the array based programming model. There is a package called DistributedArrays.jl that gives us the DArray type. As the name suggests, this distributes the memory of an array in blocks to each process:

We would hope that we could copy the method of implementing the GPU extension:

Unfortunately, the above code does not work on its own, but needs to implement the Random.randn! function for the DArray type. But since our function is easily mapped to each block in the DArray, we can just write a specific implementation of our random_walk! for this type, which we can do with a little help from the DistributedArrays.jl documentation:

We have to use the @everywhere macro to load the function on each of the workers. Usually, you can put all the code that each worker needs in a separate file and just call @everywhere once.

This code just tells Julia how to call our original random_walk! function on each block of the array, reusing our code from before. The performance (for 8 workers) is added to our graph:

Multiprocessing always has a much higher overhead than multi-threading, but we can scale to potentially millions of processes, and use memory and resources from hundreds of different computers.

What’s next?

Hopefully, in this blog, I have shown you the potential of using Julia to enable a range of parallelism, and convince you that there is a low barrier to entry to get started! All of the code in this blog post can be found in the blog folder in this GitHub repository.

If you want to learn more about Julia, you can visit the website and if you want to learn the language, start with the Julia manual.

In this blog, I haven’t gone through many of the other reasons that people use Julia, such as multiple dispatch or state-of-the-art support for differential equations and scientific machine learning. If there are packages that you need and are only available in Python, there is also a package called PyCall.jl which will let you let you run and write Python code directly from Julia, giving you access to the entire Python ecosystem.

Finally, if you want to ask questions, there is a very active community on the Julia discourse.

For information and getting connected to the University of Nottingham HPC, please see here.

Happy programming!

The post Using Julia on the HPC appeared first on Digital Research.

The UoN Trusted Research Environment (TRE) Service is now live.

The University of Nottingham’s pioneering Trusted Research Environment (TRE) platform is now live, enhancing the University’s research capabilities.

Hosted by Digital and Technology Services (DTS) in Microsoft Azure, the secure data platform is available to researchers, partnered institutions and trusted organisations working with highly sensitive data.

A secure digital infrastructure

Access to the University’s TRE platform extends across all Faculties. The platform provides researchers with the tools and capabilities needed to access sensitive data in a secure environment.

A TRE provides a computing environment (with controlled access to data in that environment) for projects that require a high level of data security and the ability to perform audits.

The centralised platform provides users with the tools needed to access, process, analyse and utilise sensitive data in a secure environment.

Pioneers in research

The platform’s extensive capabilities and digital infrastructure will enhance research opportunities, empowering the University’s academic community to continue to produce world-changing research through effective collaboration and invention.

For more information regarding the TRE platform, including the application process, its suitability for your work, platform security and/or associated costs, please visit the Digital Research SharePoint site.

The post Introducing Trusted Research Environments appeared first on Digital Research.

This week, we trialled HoloLens 2 technology with colleagues from the School of Life Sciences, the Biodiscovery Institute, the School of Education, the Nanoscale & Microscale Research Centre, and the Hounsfield Facility. 

The workshop was led by InterReality Labs, with support from Microsoft.

The HoloLens 2 is an untethered, wearable device that can project holographic models into the wearer’s real-world environment. This is known as augmented or mixed reality. It allows users to see digital 3D models or other artefacts before their eyes, embedded in a real-world physical environment. There are many applications. Our workshop focussed on research, collaboration, and facilities.

Research: you can use the HoloLens glasses to view 3D models in three dimensions. This is a fundamentally different experience from seeing such models on a two-dimensional screen. Users can walk around the models, view them from all angles, even walk inside them. Models can be rotated, scaled, situated in a real environment, all with a few simple hand gestures. We tried it out on a life-size model of a human skeleton and a train carriage, but any 3D model can be loaded into the device.

Collaboration: the HoloLens 2 shines when it comes to collaboration. The view through the glasses can be projected on a large screen and thus shared with other room occupants. Beyond that, it is possible to join a Teams meeting, whereby the call participants not only see the view through the HoloLens, but can also annotate this in real time. Sharing 3D models in a three-dimensional environment becomes possible with partners located anywhere in the world.

Facilities: another major application of the HoloLens is remote training. Users of complex research equipment can be guided from afar and in real time. Alternatively, expert technicians can demonstrate how to use equipment from the perspective of the user (and record this, if required). No more 100-page user documents! You can also use the HoloLens to create tours, and there is a package for making interactive mixed-reality guides.

Colleagues from across the University will now be testing the HoloLens in their respective research groups and facilities.

If you would like to try the technology or borrow a HoloLens 2, please get in touch with a member of the team!

The post HoloLens 2 and research appeared first on Digital Research.

In this blog, we consider approaches to tracking research impact, with a focus on digital tools.

We will outline reasons for gathering evidence of impact, what tools you might need to do this, and how digital approaches can help.

In short: how can you gather material to show the positive societal change that your work has enabled?

Why gather evidence for impact?

To demonstrate change: show to others that things are heading in the desired direction, satisfy stakeholders who are invested in your work, show that promises have been kept and increase your accountability, boost your own confidence.

To convince: build trust among potential partners, increase support for your work, make the case for scaling up, reduce scepticism.

To motivate: spark the curiosity of others, provoke questions and critical thought, inspire others to act, astound your audiences.

What do you need in your toolbox?

To demonstrate change: clear and credible baselines, testimonies from people affected by your work, user or access data, financial models and information, explorable data dashboards.

To convince: impressive comparisons (before and after your intervention), digestible data visualisations, a trusted messenger, a message tailored to your audience.

To motivate: inspiring stories, wide-reaching messages, equip your audience to take action.

How can digital approaches help (and the CART principles)?

Digital methods for gathering evidence of impact can help:

Speed: gather evidence quickly

Reach: gather evidence from a large group of stakeholders

Automation: gather evidence with less effort

Variety: gather evidence of different flavours

Cost: gather evidence cheaply

Digital tools can help you gather lots of evidence quickly, at low cost, and without much effort. We go into detail about some of these digital tools below, looking at their pros and cons.

However, just because we can do something, that does not mean we should. This is where the ‘CART principles‘ can help. CART stands for Credible, Actionable, Responsible, Transportable. In short:

Credible: privilege high-quality data over mass data collection; avoid bias in your data collection, for example by using randomised controlled trials; ensure that your counterfactuals (what would have happened if I had not intervened) are believable and supported with evidence.

Actionable: commit to collecting data you can actually use; collect data that will be useful also to others; make sure your data collection matches the needs of your stakeholders.

Responsible: ensure that the benefit of tracking impact outweighs the cost; ensure your data collection methods respect ethical guidelines.

Transportable: collect data that can generate knowledge for others; gather data that demonstrates why (or why not) your intervention could work in different contexts.

Following the CART principles when designing and carrying out your evidence collection will avoid waste and help maximise the value of your efforts.

Some specific approaches (and things to watch out for)

Build impact evidence-gathering into your project from the outset. In that way, you can plan the optimal time(s) for gathering evidence, establish ways to diversify the sources of your evidence, and even build the evidence-gathering process into your output (for example, ensuring from the very start of your project that you are able to see how many people download your toolkit, rather than waiting until after the toolkit is published to think about this).

The table below outlines some common tools for measuring impact, what they are good for, as well as some things to watch out for (you can open the image in a new tab for a larger view):

Take online surveys as an example. These are great for reaching a wide audience and collecting different types of information about the impact of your intervention. The cost is low (often free), surveys are flexible, and they help reduce biases that can occur in face-to-face settings. That said, surveys need careful designing to get the information you need, they often provide merely a snapshot of your impact at one moment of time (rather than regularly tracked impact), and they can be problematic in cases where your audience has low literacy levels.

Or how about tracking the number of people who have downloaded your resources? This is a great way of measuring the reach of your outputs over time. The process can be automated and it is free. However, this sort of tracking will not tell you how your resources are being used. So while it provides good quantitative data, it is less convincing when trying to construct a powerful narrative about the positive change you have enabled.

All the other tools in the table have their advantages, their challenges, and their optimal uses. The upshot is that one of the best approaches to impact evidence-gathering is to employ a combination of tools. This will allow you to gather a mix of quantitative and qualitative evidence and reduce the risk that key (i.e., persuasive) information is missed.

Last thoughts: planning you impact evidence-gathering

Planning your impact evidence-gathering will save you time in the long-run and set you up for demonstrable success.

Think early about what evidence you hope to gather (testimonies, cost savings, geographical reach, downloads of documents, etc.). Make sure you have good baselines, because you will want to be able to show that change has truly taken place and that this is linked to your intervention. Then think about the different audiences for your evidence, and what you might need to adapt to suit their needs – some audiences may prefer to see hard figures, whereas others will be persuaded by emotive stories. Finally, think about how you are going to collect the evidence (and how often), potentially using a mix of the tools outlined in the table above. If you can do all this at the start of your work, you will be in a fantastic place to demonstrate your impact to various audiences, when the time comes.

Further reading and inspiration

There is a wealth of resources on measuring and demonstrating impact effectively. Here is a selection:

A. Ebrahim (2019) Measuring Social Change. Performance and Accountability in a Complex World. Stanford – how to track performance toward worthy goals, with case studies from the non-profit sector

M.K. Gugerty & D. Karlan (2018) The Goldilocks Challenge. Right-Fit Evidence for the Social Sector. Oxford – using the CART principles to assess impact evidence-gathering and build better systems for tracking positive change

The Stanford Social Innovation Review (ssir.org) – for the latest thinking and innovations around impact

thinknpc.org (with various toolkits) – New Philanthropy Capital thinktank for the charity sector that aims at maximising positive social impact

Examples of socially conscious data visualisation and data dashboards (Periscopic)

If you have questions about any of the material discussed above, please don’t hesitate to contact a member of the team. We’d be happy to advise.

The post Approaches to tracking impact appeared first on Digital Research.

In this guide, we show you how to download and package a website for archiving.

If you have produced a website for your research project, you may wish to archive it (or be obliged by your funder to do so).

Here, we show you how to do two things:

(a) convert a website to a static version for continued hosting on e.g. github pages

(b) download and package a website in WARC format, which is an international standard for the archiving of digital assets.

We will use a a free tool called WGET. We will focus on Windows users, but if you have a Mac, an Internet search will reveal how to install WGET (for Linux users, WGET comes pre-installed). For Windows users, you can download the latest version of WGET here.

Download the EXE file (it’s only about 5MB), but don’t open it.

(To find out if your computer is 32-bit or 64-bit, type ‘System Information’ in the Windows search bar and look at ‘System Type’.)

Now, in the Windows search bar, type ‘cmd’ and open the Command Prompt.

When you have opened the Command Prompt, type ‘path’ and hit ENTER. You will see something like this:

The location highlighted above in red is where you need to move the WGET.exe file that you have just downloaded. Move the file there now, then restart the Command Prompt. (Please note that you will need admin rights on your computer to do this – if you are working on a University computer without admin rights, please request these temporarily via the UoN IT Self-Service Portal.)

To check that WGET is working, type:

wget -h

Hit ENTER and you should see something like this:

It is helpful to create a new directory (a new folder) for your downloaded website(s). For our purposes, we will call this new folder ‘archivedSite’. In the Command Prompt, type the following and hit ENTER to create the folder:

md archivedSite

Then type the following and hit ENTER:

cd archivedSite

Now we are ready to download your website(s) into the new folder. You should only do this for websites where you own the content.

We’ll use the following website as an example: https://www.resilient-decarbonised-energy-dtc.ac.uk/

Static version of the website

To get a static version of the website, type (or copy/paste) the following into the Command Prompt (all in one line) and hit ENTER:

wget --mirror --recursive --convert-links --adjust-extension --random-wait

--page-requisites --local-encoding=UTF-8 --no-parent -R "*.php, *.xml" https://www.resilient-decarbonised-energy-dtc.ac.uk

This will download a folder with HTML + CSS files for every page of the website, as well as accompanying media, such as images. All the hyperlinks in the website will work, including those that point externally to other websites. The downloaded files could be archived or displayed as a live static website on e.g. github pages, for free.

Nb. the download can take several minutes, depending on the size of the website.

WARNING:  static websites won’t handle interactive elements (such as fancy drop-down menus, submission forms, and other sophisticated plug-in functionality) nor can they deal with connections to underlying databases and search/filter functionality. If possible, you should remove these elements from your website before downloading it.

WARC version of the website

To get a WARC version of the website for archiving, type (or copy/paste) the following into the Command Prompt (all in one line) and hit ENTER:

wget --mirror --recursive --convert-links --adjust-extension

--random-wait --page-requisites --no-parent -R "*.php, *.xml" --warc-cdx

--warc-file=TYPE_YOUR_FILENAME_HERE https://www.resilient-decarbonised-energy-dtc.ac.uk

This will download a (zipped) WARC file and an accompanying CDX file (which is an index of the downloaded material). Both can be uploaded to a repository, such as the University’s Research Data Management Repository, for permanent preservation.

Nb. the download can take several minutes, depending on the size of the website.

And there you have it!

If you have any questions about the above, please do not hesitate to get in touch with a member of the team.

Further reading and resources

The full WGET user manual: https://www.gnu.org/software/wget/manual/wget.html

An overview of the most common WGET commands/options: https://www.computerhope.com/unix/wget.htm

Conifer: a US-based web archiving service, which includes a click-through version of the above process

The UK Web Archive, a partner of the UK Legal Deposit Libraries and responsible for preserving UK web content for future generations. You can ask the UK Web Archive to Save a UK Website for you.

For the University of Nottingham’s own digital preservation activity, notably around preserving elements of the main website, see this blog.

The post Archiving a website appeared first on Digital Research.

In this blog, we discuss an automated process for anonymising interview transcripts, patient notes, or other free-text data containing personal information. 

Colleagues wishing to share participant notes or interview transcripts, for example as publication appendices or in a research data repository, will likely need to anonymise the data. Anonymisation also comes with a number of benefits, such as these listed by the UK Information Commissioner’s Office:

• developing greater public trust and confidence that data is being used for the public good, while privacy is protected
• incentivising researchers and others to use anonymous information instead of personal data, where possible
• economic and societal benefits deriving from the availability of rich data sources

We have been exploring ways to automate the anonymisation process.

Below, we provide you with a link to some copy-and-paste code that will anonymise texts automatically. But first:

What is de-identification?

De-identification is the process of removing or obscuring personally identifiable information (PII) from a text or dataset. Such data typically includes names, locations, social security/national insurance numbers, contact details, and the like. The process can be approached in a number of ways, but the output usually falls into one of two camps:

a. the masking of PII with labels (“my name is Jane” becomes “my name is <NAME>”)
b. the replacement of PII with dummy data (“my name is Jane” becomes “my name is John”)

We will focus on the first of these. Here is a fuller example of what an output looks like:

Original text:

00:00:02 Speaker 1: hi john, it’s nice to see you again. how was your weekend? do anything special?

00:00:06 Speaker 2: yep, all good thanks. i was with my sister in derby. We saw, you know, that james bond film. what’s it called? then got a couple of drinks at the pitcher and piano, back in nottingham.

00:00:18 Speaker 1: that’s close to your flat, right?

00:00:25 Speaker 2: yeah, about five minutes away. i live on parliament street, remember?

00:00:39 Speaker 1: of course, i remember. you moved last year after you left your parents’ place.

00:00:39 Speaker 2: yeah, it was my sister’s birthday on sunday, susie, the older one. i told you last time about that new job she got. sainsbury’s, the one by victoria centre.

De-identified text:

00:00:02 Speaker 1: hi PER, it’s nice to see you again. how was your weekend? do anything special?

00:00:06 Speaker 2: yep, all good thanks. i was with my sister in LOC. We saw, you know, that MISC film. what’s it called? then got a couple of drinks at the pitcher and piano, back in LOC.

00:00:18 Speaker 1: that’s close to your flat, right?

00:00:25 Speaker 2: yeah, about five minutes away. i live on LOC, remember?

00:00:39 Speaker 1: of course, i remember. you moved last year after you left your parents’ place.

00:00:39 Speaker 2: yeah, it was my sister’s birthday on sunday, PER, the older one. i told you last time about that new job she got. ORG, the one by LOC.

So how can you achieve this?

Colleagues may already be familiar with NLM Scrubber or similar pre-packaged tools. In our case, we have used an open-source model from the hugging face community, a repository of pre-trained models for natural language processing. (The output above is based on this particular model).

With a few lines of Python code, you can produce de-identified text in under a minute.

ANONYMISE YOUR OWN DATA. If you would like to replicate these results, you can find the code here: https://uniofnottm.sharepoint.com/sites/DigitalResearch/SitePages/Automated-an.aspx

Some caveats

Machine-learning algorithms contain bias stemming from the data on which the model has been trained. These biases can lead to errors like false positives (a non-PII word is identified as PII) or false negatives (PII is not identified, and therefore not anonymised). In the case of the model described above, the creators specifically note that:

This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains. Furthermore, the model occasionally tags subword tokens as entities and post-processing of results may be necessary to handle those cases.

Outputs reliant on pre-trained models should always be checked for errors.

Next steps

The process described above requires basic knowledge of running Python scripts. Out next step is to investigate ways of packaging the process in a way that requires no coding knowledge.

If you are interested in talking to us about this work and getting involved, please feel free to contact one of the team.

Further reading

Berg, H., Henriksson, A., Fors, U., Dalianis, H. (2021) ‘De-identification of clinical text for secondary use: research issues‘. HEALTHINF. pp.592-99.

Johnson, A,. Bulgarelli, L., Pollard, T. (2020) ‘Deidentification of free-text medical records using pre-trained bidirectional transformers‘. CHIL ’20 Proceedings. pp. 214-21.

Infographic: The Future of Privacy Forum, A Visual Guide to Practical Data De-identification.

The post Automated anonymisation of texts and transcripts appeared first on Digital Research.

In this blog, we talk about how we created UoN Research Link and some of the technical and design decisions taken.

What is UoN Research Link?

UoN Research Link lets you search for University of Nottingham colleagues according to research interests and expertise. It can help researchers identify collaborators from across the University’s campuses. You can search for any topic of interest and it will return a list of colleagues working in related areas, together with a summary of each researcher’s interests.

The underlying data

Our first challenge was to create a database of researcher interests. Not wanting to burden colleagues with supplying the data manually, we decided to use eStaff profile pages, currently the fullest set of information about research expertise at the University.

Not all information on the profile pages is relevant to UoN Research Link. For example, we ignore contact details, teaching expertise, and publications. The Research Summary and Expertise Summary sections are the main ones we use. Colleagues without an eStaff profile page or with no information in the aforementioned two sections will not appear in UoN Research Link.

The data is cleaned and formatted, for example by removing duplicate profiles (some colleagues are affiliated with more than one School or Research Group) and deleting hyperlinks, special characters (like bullet points), professional titles and degrees, as well as common words like ‘university’ or ‘research’. This is done using the programming language R, which has good packages for text manipulation.

The connection between UoN Research Link and eStaff profiles is not live (i.e. changes to eStaff profiles won’t immediately carry over to UoN Research Link), but the data collection process will be re-run every three months over the lifetime of UoN Research Link. This means that changes to research expertise or the arrival of new research colleagues will be captured, albeit only every quarter.

Generating keywords

The next challenge was to present researchers’ interests succinctly. Keywords are good for this, but few colleagues have separately listed keywords on their profile pages. Again, we did not want to burden colleagues by asking them to supply keywords, so we generate these automatically.

We tested different methods for generating keywords. We tried extracting the most frequently occurring words in a given profile, but that returned mostly banal terms. The same approach but after removing the 1000 most common English words was also disappointing. Models based on machine learning held more promise. Both BERT transformer-based models and topic modelling NLP models such as Gensim worked well and generated sensible keywords distanced from each other (it can be unhelpful when all of a researcher’s keywords are very similar). However, we found that both these approaches were inflexible in the number of words used as keywords. We wanted an approach that would produce both single words and collocations (and parse these accurately). The keyword ‘synthetic organic chemistry’ is more helpful than just ‘chemistry’ or, worse, the unintelligible ‘synthetic organic’. In our case, the model that produced the best results was taken from the Python Keyphrase Extraction toolkit.

Each profile in the database is thus supplemented with up to six keywords, generated automatically. These are displayed under colleagues’ names when they come up in a search, providing users with a quick overview of a researcher’s interests and expertise.

Despite its accuracy, the keyphrase generator still occasionally produces sub-optimal keywords (it struggles on hyphened terms, for example), and we realise that colleagues may also want to provide their own keywords. The site administrators can therefore manually update any researcher’s keywords. Colleagues can visit the Contact page to arrange this.

Suggesting related researchers

We also wanted to identify researchers with similar profiles. Colleagues may be working in related areas and not be aware of this. Researchers new to the University may wish to seek out connections. Prospective post-graduate students might be looking for complementary supervisors.

To achieve the ‘Related Researchers’ function, we use a proxy measure, namely the similarity of one profile text to all others. Using the R package quanteda, we were able to calculate this. Quanteda was used to tokenise the corpus and generate a document-feature matrix. It also has handy functions for removing stop words, word stemming and other NLP processes. We use the cosine similarity measure to compare researcher’s expertise texts, although other measures are available (Jaccard, Sørensen-Dice, etc.). Here is an example output:

Name              School      similarity  
Gosling, Simon    Geography      1     
Grundmann, Reiner Sociology      0.378
Jones, Matthew    Geography      0.338 
Dugdale, Steve    Geography      0.295 
Panizzo, Virginia Geography      0.268

It’s worth emphasising that only the profile texts are being compared here – other contextual information is not considered. We also set a threshold on the similarity measure so only colleagues with reasonably similar profile texts appear.

The entire process for formatting the data, generating the keywords, and calculating the similarity measures represents about 1200 lines of code (90% in R, 10% in Python). This is a work in progress. By separating the database generation from the user portal, we can continue to refine the code over the lifetime of UoN Research Link.

Creating the tool and user testing

Before engaging a development partner to make the website, we worked with the University’s User Experience (UX) team. They helped us think about how to present the data and how users navigate databases successfully. This led to the introduction of various filters as well as the option to search not only by research topic, but also by researcher name. You can still see one of the clickable mock-ups we created here. Our development partner then refined the look and feel, particularly around the aesthetics of the site. Below, for example, are some ideas around toggle buttons that didn’t make the final cut:

Our partner also introduced the ‘try searching related topics’ option (which proposes only topics that actually occur in the database).

Two aspects of this work phase were especially positive and impactful:

1) Adopting an agile approach allowed us to make iterative changes as the site development progressed. This meant that issues could be addressed quickly and workloads could be managed. It allowed us to finish ahead of schedule and with no major setbacks.

2) User testing of an early prototype threw many of our assumptions into relief. The user-testing exercise was absolutely invaluable and made the tool much more intuitive and accessible than we had originally designed.

Working with the development partner and with colleagues from different parts of the University during the creation of UoN Research Link was a pleasure.

So how does it all work?

In the background, it’s quite basic. A user can enter any topic as a search term. The site will then look through the expertise text and keywords of every researcher in the database. If it finds a match for the search-for term or terms, then that researcher will be displayed.  Each relevant researcher is shown on a tile that contains their keywords. Clicking on ‘Related researchers’ at the bottom right of the tile will reveal colleagues with similar expertise profile texts.

What got left out

Technical, financial, and time constraints meant we had to pare down or sacrifice some hoped-for aspects of the tool. For example, we had considered options for visually representing connections between researchers in a network graph. Ultimately this proved too ambitious (although the idea survives in the Research Link logo).

Presenting search results in a logical order also proved more challenging than anticipated. We explored options such as prioritising those researchers for whom the searched-for term appears more frequently in their profile text. But since these calculations would need to happen within the website or would rely on a huge database of all possible permutations of searched terms, it would have greatly slowed down the tool. In the end, we opted for a random-order presentation of search hits.

We had also wanted to include colleagues based at the Malaysia and China campuses, but were unable to obtain the data in time.

Update 07.08.2023: UoN Research Link now includes colleagues based in China and Malaysia

A word of thanks

The development of UoN Research Link was supported by the University’s ESRC Impact Acceleration Account.

Click here to visit UoN Research Link: https://researchlink.nottingham.ac.uk/

The post Creating UoN Research Link appeared first on Digital Research.