Understanding risk with AI is about understanding probabilities

One of the most common misunderstandings about large language models (LLMs) is the idea that if you upload a document such as an interview transcript, confidential leak or sensitive dataset to an AI tool like ChatGPT, it might reveal that document to another user.

There are no publicly documented cases of this happening.

The reason for this is the same reason why AI shouldn’t be treated as a search engine: large language models like GPT-5 don’t work by retrieving documents for users — they work by separating documents into parts of language (‘tokens’ in the jargon), adding those to a vast ocean of other tokens, and training on a database of relationships between those.

When you ask a question like “Who is the CEO of Apple?” a genAI tool doesn’t retrieve information from a document. Instead it breaks down your question into tokens like “CEO” and “Apple” and “Who”, calculates the most statistically likely meaning of those things (e.g. is that apple probably a fruit or a company), and then the most statistically likely sequence of words which might form a meaningful answer to that question.

(In this example, “Tim” and “Cook” are likely to be more strongly associated with “CEO” and “Apple” in its training data than other tokens, and so they will be formed into a response: “Tim Cook”.)

If information you provide in a prompt is used in an LLM’s training (see below), then, it will be a) broken down into tokens and b) those tokens’ relationships with other tokens will be competing with lots of other token relationships to be the ‘most likely’ answer to any question.

So when do those tokens and relationships become a risk?

Probabilities are shaped by training

There are two main steps in a large language model’s training: the most important is pretraining: this is when the model is trained using an enormous collection of language. If your information ends up being included in that collection, it is a drop in the ocean of those trillions of words.

There will also be a delay before it is included in responses, because each model is trained a number of months before it is release — this is called the knowledge cutoff.

ChatGPT’s latest model, GPT-5.6, has a knowledge cutoff of five months ago, which means it doesn’t contain any information after mid-February. If Apple’s CEO has changed since then, that won’t be reflected in its core training data. You can see each model’s cutoff dates here.

The other key step in a model’s training is called reinforcement learning from human feedback (RLHF).

Whenever ChatGPT presents you with two possible responses and asks to choose which one you prefer, that’s you helping to train the model by reinforcing a preference.

AI companies pay humans to train the model too (there are human rights and exploitation concerns around this). Those people train the model by looking at a small sample of conversations and selecting the better responses.

Your information might be selected for this sample (out of millions of conversations every day). If that happens it will be incorporated into a model’s training with a shorter time lag, and weighted more heavily. For example, even where the majority of pretraining data says the Apple CEO is Tim Cook, if a human has ranked a different, more up-to-date, response as more or less accurate, then it can outweigh that other training.

The chances of your information either being incorporated into pretraining data, or selected for reinforcement training, will depend to some extent on policy.

Check the policies

Free consumer accounts with ChatGPT, Gemini, Copilot and Claude opt users in by default to agreeing to allow their conversations to be used for training. The chances of a conversation being used is still tiny, but you can reduce it further by opting out (you might also consider opting out of AI training on platforms like Adobe and AWS where you might use sensitive data).

Enterprise and business accounts and APIs tend to opt users out by default.

Data retention is another factor to consider. Some developers retain chats indefinitely, and even ‘incognito’ or ‘temporary’ chats are likely to be retained for at least 30 days. Some accounts will have a zero data retention policy — PC Tech Magazine has a breakdown of different providers’ policies and processes for this.

Your chats are more likely to be selected for reinforcement training — and retained for longer — if they are ‘flagged’ for training for some reason. The most obvious reason that a chat might be flagged is for safety reasons, or because you have given feedback on a response in some way.

Chats can be flagged for training even if you have opted out of training on a free account, as a recent update to Anthropic’s privacy policy revealed:

“Conversations that Anthropic’s systems flag for safety review can still be used to train its models, regardless of a user’s stated preference. Anthropic’s policy does not define what triggers a safety flag, nor does it commit to notifying users when one occurs.”

The less information about an entity, the higher the risk

If your information was used to train a large language model — either through pretraining or reinforcement training — there is still no guarantee that it will find its way into a response to a user’s prompt.

What will make this more likely, however, is if your information relates to an entity or a question that the LLM ‘knows’ less about (i.e. has less training data).

Here are two scenarios to illustrate:

In Scenario A, a journalist uploads a sensitive document about Microsoft’s hiring policies to Gemini and it ends up being part of Gemini’s training data. All the ‘tokens’ (words and parts of words) from that document are already in its training, and this document changes the strengths of their relationships slightly so some have a stronger relationship with tokens like ‘Microsoft’ or concepts like ‘bias’.

At various points Gemini users will ask a question about Microsoft’s hiring policies. Because the sensitive document is one of many documents about Microsoft and hiring that Gemini has been trained on, its information is very unlikely to make a meaningful difference to most responses.

In Scenario B, on the other hand, a journalist uploads a list of the names of victims of a known predator. In this case, some of the names are quite unique and Gemini has either very little or no training data related to those. As a result the information in the document (the relationships between words) will be much more heavily weighted. Instead of being a drop in this particular ocean, the document constitutes a much larger part of it.

Now when users ask a question about one of those people, Gemini has much less training to draw on — and is therefore more likely to replicate the particular patterns of words (the ‘fact pattern’) in the sensitive document than other patterns it might predict.

Remember, however: it is still not retrieving a sentence from a document — it is only predicting one based on mathematical relationships.

Notably, if they asked about the known predator in general the chances of those names being generated by the AI tool is lower, because there are more, and stronger, connections to other facts, which might be more heavily weighted (depending on the question).

The complicating factor, of course, is that the user has no way of knowing whether the ‘fact’ in either scenario is true. When asked questions where very little information exists, AI tools are much more likely to hallucinate (hence court cases where AI tools have defamed individuals). And because the sensitive material is not online in either scenario, the chatbot cannot point to it as a source for the user to check with. The user has the suggestion of a fact, but not the proof.

When the risk calculation is different: sharing chats, internal access, hacks and law

Outside of user interactions with the AI tool, there are a number of other risks to consider.

The first is a simple one: if you choose to share the chat at any point this creates a risk that it becomes public.

Last year an investigation by Henk Van Ess‘s Digital Digging newsletter uncovered over 100,000 ChatGPT conversations which were findable with advanced search techniques. Last month he found another 90,000 shared conversations from multiple chatbots. The problem isn’t the AI tool — it’s the user sharing their chat. As Henk writes: “It looks like people thought “share” means send. But it means publish.”

It’s also worth mentioning that some engineering, support, and safety/abuse teams at AI companies may have authorisation to access AI conversations. As with reinforcement training, this is more likely if content is flagged for specific reasons, to provide account support, or for legal reasons, and in the context of millions of prompts each day. As with any online service, there is also the potential for an employee to misuse their access.

The same security risks that apply to online behaviour in general apply here, too: if you are using unsecured or public wifi to access an AI tool, it could be intercepted. The provider’s servers could be hacked or breached, as could your account with the provider if, for example, your password is weak. Your employer’s network and your own computer may also be hacked or accessed.

A more specific security risk is extraction attacks: these are specific attempts to extract training data from large language models (documented in literature listed here).

Finally, there are legal orders, which may be more important to consider than technical risks. The most famous of these is the court ruling that forced OpenAI to give millions of user chats to news organisations in a copyright case, but chats are increasingly showing up as evidence in criminal and civil cases, with one legal website advising “to prepare for a steady increase of government and litigation requests for Gen AI data”. In the US, court rulings have indicated that chats with AI tools “were not protected by attorney-client privilege or the work product doctrine” (see Munir v Secretary of State for the Home Department for a similar point in the UK).

Some of these requests will be specific to a particular user, but a warrant in October 2025 revealed that they can also relate to a particular prompt: what is known as “reverse” search warrants.

What can you do about it? Anonymisation and switching to local models

One way to reduce the risk involved in uploading sensitive information to an AI chatbot is to anonymise it. This involves either removing information that identifies individuals, or replacing it with unique IDs that you can use to re-identify them outside of the chat.

It’s not just names that might identify individuals: any collection of details might be connected with an individual (jigsaw identification) — remember that it’s these relationships that large language models are designed to calculate. Some technical solutions and libraries already exist for this.

It’s also worth considering whether uploading the information is necessary at all: just providing a description of the document or data may be enough to get a useful response, especially if you are focusing on a process (journey prompting) rather than an end result.

If the risk of using an AI service is too high, another option is to use a large language model locally — i.e. on your own computer rather than a server belonging to OpenAI, Google or another provider.

A number of tools make it relatively easy to run AI models locally, including LM Studio, AnythingLLM and Ollama. These allow you to choose from a range of models, download them to your computer, (make sure you use the local models as some also offer ‘cloud’ models), and then run prompts against them.

You will need a computer with enough processing power and RAM (CanIRun.ai provides a breakdown of which models you may be able to run based on detecting your computer’s specs), and the larger the model, the more power you will need. Prompts will typically run more slowly than on mainstream AI chatbots, too.

Make a threat model

A useful process to map all this out is to create a threat model that asks four simple questions:

1. What information do you not want other people to know? (This can be anything from passwords to contacts’ details, data and documents)
2. Why might someone want that information? Who?
3. What can they do to get it?
4. What might happen if they do?

Using the information in this post to map out realistic threats will make any subsequent steps much more manageable — and less anxiety-provoking.

This is a work in progress. Contributions, suggestions and updates are welcome.

Thanks to Laura Isotalo who raised a question around security and suggested writing about it after my talk on FOI management with AI at Dataharvest this year.

Short-form vertical video is quickly becoming a key format for telling all types of news stories — and data journalism is no exception.

As a data journalist, I wanted to know how these outlets use this new and evolving digital video format to create “data shorts”: short-form social media videos in a vertical, reel-style format, that analyses or reports on specific datasets or newly released data.

I analysed 50 data shorts from the TikTok accounts of five major UK news publications — the BBC, ITV News, Guardian, Times and Financial Times — to examine how these digital news outlets approach the format. I looked at the genre of story they covered, the angles they focused on, their length, and their use of animation, to help inform my guidelines.

Tip 1: Keep your data shorts under 90 seconds

My analysed data shorts averaged a runtime of just under 84 seconds.

The BBC ranked lowest for average video length, at just under a minute, while the Financial Times had the highest at 105 seconds.

Notably, none of the publications had an average video length over two minutes, despite TikTok allowing longer uploads up to 60 minutes.

By keeping their runtimes short and using an average speaking rate of 100–150 words per minute, data shorts act as video equivalents to the traditional News in Brief (NIB) format — short news articles usually summarised in a single paragraph — rather than conventional 300–500-word news stories.

Based on the average runtime, a script for a data short should be around 200 words, presenting  a condensed overview rather than deep-dives into causes and effects which would require a more time-demanding feature-length format.

By condensing the runtime of a data short, journalists can also get to the story’s point more quickly, maintaining audience engagement with the video by removing unnecessary details, pauses or words that may increase the chances of viewers scrolling to the next video in their timeline.

Tip 2: Maintain a human connection and get in front of the camera

Journalists had a presence in all 50 shorts.

43 out of the 50 shorts were shot in the piece-to-camera format, physically establishing a human relationship between a journalist and the data story — and all the videos featured a journalist’s voice-over.

Featuring the voice and face of a presenter in a data short is important, as vertical video platforms like TikTok are personality-driven and built on relationships with audiences.

By being present, journalists and presenters act as a point of engagement, recognition, and credibility with their audience, building trust faster and establishing themselves as a reliable news source and industry presence.

Tip 3: Locations reinforce your authority

Where you film your piece-to-camera is also important.

All the data shorts with pieces-to-camera I analysed were filmed in a recording studio or a newsroom setting, except one, which was recorded on location.

By recording in professional environments, you further reinforce your reliability as a news source, creating an innate sense of trust with your audience by visually communicating your professionalism.

Tip 4: Show don’t tell

Data visualisations are commonly used in data journalism — and data shorts are no exception, with 33 of the 50 TikTok videos analysed including some form of data visualisation.

Line charts (used 15 times) were the most common form of visualisation, suggesting that change stories such as this video by the FT’s John Burn-Murdoch were one of the most common angles chosen.

Bar charts — used primarily to show ranking comparisons — were the second-most popular form of visualisation.

Choropleth maps (appearing four times) were the third most prevalent visualisation, used in political and economic stories to visualise statistical differences between geographical areas.

More complex visualisations, like dot plots and stacked area charts, appeared only once each throughout my analysis, suggesting that intricate graphs may be too complicated to use effectively in short-form vertical-video formats.

Only eight of the 33 data shorts that included visualisations used more than one type of chart, suggesting that the short time frame may further restrict the amount of analysis achievable through visualisations.

Tip 5: Animate your graphs to maintain momentum

Digitally animated graphics are a visual storytelling technique frequently used in short-form vertical video to engage viewers.

29 of the 33 data visualisations in my analysis featured animation, either zooming in on specific data, revealing data over time or highlighting specific elements through using colour.

This proved effective in adding further storytelling to the graphs, while allowing the video to elaborate on specific datapoints within the visualisation.

Animated text was equally as common, primarily used to emphasise statistical information or to draw attention to particular words or names that were important to the story. An example can be seen in the Times’ 2026 data short on the increase in UK expatriates, which uses text animations to highlight the rising figures, as well as the factors driving the increase in people leaving the UK.

Tip 6: Don’t be afraid to innovate and adapt

Although the analysis documented the most common practices used by five major news brands, there are no set-in-stone methods for creating a data short — and my analysis also highlighted some newly emerging storytelling techniques.

Two of the videos, for example, used satellite imagery as data visualisations. Robert Simmon and Niels de Hoog have used this approach in the past to show how geographical areas have changed over time, and on TikTok both The Guardian and the BBC experimented with satellite data in a similar way, creating effective, visually-led data shorts that show real geographical change rather than simply describing it. Both videos keep the runtime of these data shorts under the two-minute mark, and also allow time to give context on how and why these changes are occurring.

The Guardian frequently implemented more elaborate animations and graphics within their work, giving many of their data shorts greater visual spectacle and flair than those produced by other publications.

Don’t be afraid to experiment and innovate with your data shorts to push the medium further.

If you have produced or work in the field of data shorts or vertical video journalism, I would love to get in contact with you in the comments of this post, through email or on LinkedIn.

A version of this post with further analysis and more detail about methodology was first published on Medium.

The latest “wife of” headline — ITV News’s report on the actor Lisa Faulkner revealing that she has undergone surgery after a cancer diagnosis — is an opportunity to get journalism students exploring how different forms of bias might shape news reporting — and not just the obvious ones.

An opening question might be why a reporter or editor would put another person’s name at the front of the headline “John Torode’s wife Lisa Faulkner reveals breast cancer diagnosis”.

A male-dominated newsroom might be one answer. Another might be a newsroom that is more likely to watch a middle-class cooking programme (Masterchef) than a working-class soap opera (Eastenders). It might also be driven by search analytics: four times as many searches were made for Torode in the last 12 months than Faulkner.

Search analytics are closely related to bias around news values: why, for example, might reporting lead on “Taylor Swift’s boyfriend” or “Nicole Kidman’s husband” instead? How do news values come into play with Amal Clooney or Meghan Markle or The Chicago Tribune leading on the bronze medal-winning “Wife of a Bears’ lineman“? And how do they intersect with other forces?

Increasingly, there will also be the potential for algorithmic bias: might the headline have been AI-generated or -selected? What training data might have informed that? Could A/B testing have shaped it?

A role may be played by cognitive bias too: the availability heuristic means people are more likely to connect new information with what happened most recently and was most widely reported (e.g. John Torode’s sacking, responsible for the spike in searches at the start of that chart above).

It’s important to emphasise that no one escapes the trap of cognitive biases, either: an opportunity for reflection around confirmation bias can do this. Compare the initial reaction to the headline (did it ‘confirm’ existing assumptions?) with the potential for a more complex picture with multiple social and cultural forces at work.

The point here is not what the ‘real’ cause of the headline was: it’s that confirmation bias prevents us from exploring counter hypotheses or less simple explanations before drawing that conclusion.

Each of those biases is a force to name and address, in both critical and practical terms. We can ask what forces act to introduce bias into reporting, and at what levels (individual, team, culture, technology, audience, institution, business model)? What is considered best practice in this area? How should journalists balance chasing search traffic with their duty to avoid prejudice, to be objective and accurate? How do we design prompts to avoid (or identify) bias? How do we recruit or source in ways that reduce the opportunities for bias? And how do we slow down to prevent cognitive biases driving our behaviour?

On one recent project I decided to put together a methodology that made the process less stressful, faster and more accurate. Here’s how it works.

Problem 1: Responses are inconsistent

The main problem journalists face with responses to a bulk FOI request is their inconsistency. This can be reduced by specifying the year used (financial or calendar), the format of the response (spreadsheet) and even providing a template, but inevitably some authorities will ignore either or both.

On top of that, authorities will often use slightly different language to refer to the same thing, or different levels of categories. And some

With so many different inconsistencies, it helps to break those down and make a list that you can turn into a series of steps based on priority. For example:

• Inconsistent filetype: we first need to extract data into one format (CSV)
• Inconsistent shape: …then we need to reshape data (wide to long, all fields)
• Inconsistent naming: …then we need to standardise categories or other fields
• Inconsistent timescale: …then we need to make an editorial decision whether to use all responses and, if needed, how to report the use of different timescales

Problem 2: You need to choose a focus

That inconsistency will apply across every question that you asked in the FOI — so before you spend any time making the responses consistent, it makes sense to decide which parts of the responses you’re actually going to need.

This is often shaped by how many questions were answered by authorities, and to what level of detail. In the FOI project I was working on, for example, the request asked for details of crimes in hospitals, but not all forces provided totals that combined crime category and hospital names and outcomes. Our story would have to focus on either categories, or hospitals, or outcomes.

To choose a focus you need to know how many responses answer each question — you need an audit that does the following:

• Browse the FOIs, or a representative sample
• Identify the range of information covered
• Are there any questions which some bodies refused to answer?
• Or levels of detail they couldn’t provide?
• Identify any differences (e.g. financial year vs calendar year)
• Decide what takes priority where multiple tables are given (e.g. offence category, not hospital or outcome)

You can do this manually, but this is also a task for which AI is well suited: a classification and summarisation task to provide an overview of a large document set. Using the Pulitzer Center’s risk assessment this task also sits in AI’s “sweet spot” because it’s not audience-facing and you don’t need high accuracy.

Specifically, it’s well suited to NotebookLM, an AI tool for working with documents using the Gemini LLM.

Those documents will need to be in PDF or Word format, or Google Sheets, and there’s a limit of 50 documents (you can combined PDFs with Adobe Acrobat and other online tools). An alternative is Google Pinpoint.

A prompt template to audit your responses might look like this:

OBJECTIVE: You are a journalist looking to combine data from multiple FOI responses into a single table that can be analysed to identify trends over time and compare categories or bodies.
TASK: Audit these responses and produce a table identifying the range of information covered for each authority.
Each column should identify Y/N if they provided that information.
Include a column indicating what type of year was used (e.g. financial vs calendar)
and a column indicating whether data is provided for each incident, or per year, month, quarter, another period.
If the response includes any warnings or caveats, quote these and the page number in a caveats column.

The resulting table should give you an idea of the coverage of the FOI responses, including which questions might have the most comprehensive responses, and which questions will have patchier data.

You can now decide on a focus which isn’t going to lead to frustration and having to start over.

Problem 3: You need to extract the data

Now you can start extracting that target data from the FOI requests. For data in Word documents and email tables, or single-sheet XLSX files, copy and paste the data into a spreadsheet and save it as a CSV.

It’s tempting to reach straight for AI tools to tackle trickier formats, but there two tools to try first that are faster, more accurate, more explainable, less energy intensive, and avoid deskilling.

• Try Open Refine to combine sheets from XLSX — or search DuckDuckGo to find other solutions like this 
• Try Tabula first for PDF extraction

You should also conduct a security and privacy assessment before reaching for AI: on the whole information provided under FOI laws should not raise issues around security or privacy, but the organisation may have made a mistake, or there could be issues with the particular nature of the information provided that mean you would not want a large language model to be learning from it.

If you need to use AI to extract data from an XLSX file where it has been provided in different sheets (for example one for each year), here’s a template prompt you can adapt:

This XLS file contains a sheet for each financial year. 
Each sheet has three tables.
Extract the second table, in columns D-E, from each sheet
Create a CSV containing the combined tables with the following columns:
Offence Code | Total | Year from | Year to
Ignore the first two sheets which do not relate to financial years

The prompt should provide context for what is in the sheets, name specific columns or rows, specify the structure and the output, and tell it what not to do.

PDF extraction is more problematic, because there’s an explainability challenge (it’s hard to explain how it extracted the data). I’m going to suggest a way to address this in a moment, but if you have to use AI for PDF extraction, here’s a template prompt for that:

Attached is a PDF. 
Extract ONLY the table on pages 2 and 3 which shows crime totals by category and year. 
If any line breaks or hyphenations cause split labels, reconstruct using nearest-neighbor/line-merge logic and re-validate.
Export as a CSV.
Check you have grabbed rows at the ends and beginnings of pages. 
Check that figures in the extracted CSV, when totalled match totals in the PDF
If any uncertainty remains (e.g., an OCR-confused label), 
include a “flags” list with the exact text span and your best-guess normalization.

Again, this names specific columns and rows, and specifies output. It addresses particular problems with PDF extraction: line breaks where column or row titles are split, provides space for uncertainty (AI is ultimately a machine for guessing) and includes a validation step.

That reduces some of the most common risks in using AI for this — but explainability remains a problem: how do we know what it did? And how can we use that to help us verify the results?

A better approach than is to ask an AI tool help us to extract the data — by giving us some code that we can run ourselves in Google Colab (an app in Google Drive, so it requires no installation).

The template prompt for this is long, so I’m not going to paste it here. Instead you can find it on the GitHub repo for my Dataharvest talk on this. But it has a few key elements to highlight:

• Set a clear objective including any prioritisation of data to extract
• Break down the task step by step
• Describe the PDF and the specific pages and tables you want to target, including the structure. Say whether the PDF has gridlines and any text that appears before or after it.
• Provide any extra details

One thing to highlight in the prompt: it should specify that code “be commented so that a non-coder can understand what is happening” (explainability again).

Once the AI tool has provided you with some code, create a Colab notebook and copy and paste that code into the empty code block, then run it by pressing the ‘play’ button to the left.

Once a code block has run, look for an ‘Upload’ button appearing underneath the code block, where you can upload the PDF

Create extra code blocks for further code by clicking the button marked ‘+ Code’ at the top of the notebook (the same button will also appear above or below a block when you hover).

Once you’ve done this for one response, you can adapt the same template prompt for other responses, adjusting any details on the PDF description etc.

For extra confidence (thanks to someone at Dataharvest for suggesting this), you can ask an AI tool to generate code again, but in a different language (you can run R in Colab). This prompt would be the same with one change:

Look at the attached PDF and suggest code in R that will work in Colab notebook (explain how I run R in Colab).

Problem 4: You need to check it’s accurate

After you’ve extracted the data from a response, you need to check that it’s extracted the data correctly. I combined three methods to do this:

1. “Eyeball” tests: manual checks for known risks, vulnerabilities and issues
2. Analysis tests: calculations such as totals and value counts to cross-check
3. AI tests: document comparison and repeat of above tests

Particular things to check manually (“eyeball” tests) include where the PDF has blank or merged cells (values sometimes get shifted here), where text runs across multiple lines (sometimes only the first line is extracted), and the first and last rows on each page.

Analysis tests are where you don’t compare values directly, but add up all your numbers to see if it matches a total in the original. Any discrepancy is extraction should result in a different figure.

I’ve created a template checklist which lists all the tests I could identify. Using a checklist makes it possible to systematically go through and tick each test (some will not be applicable, e.g. where you don’t have totals in the original PDF).

The AI tests just repeat the above, adding an extra layer on top of the manual checks, which can be useful in identifying human error in your own checks. A prompt template for AI validation is also on the repo (again, it’s too long to paste here).

Problem 5: You need to get the responses into one dataset

Now you have a collection of CSVs with data extracted from FOI responses, and checked against the originals. The next step is to combine those into a single dataset you can question.

That means deciding on a single structure. It will need to be a structure that can accommodate the variety of responses, and that allows you to ask different questions (e.g. pivot tables). Some tips for that structure include:

• Have columns for filename and authority for traceability/combination
• Have ‘year from’/‘year to’ and store the year as a value, not a column name (the data should be long, not wide)
• Have a column that measures the number of events (e.g. crimes) — or two columns for ‘number from’ and ‘number to’ if organisations are likely to give a range (<5 can be encoded as from 1 to 5, for example)*
• Have a category column that classifies the type of event (e.g. ‘theft’)
• You might have sub-category column as well, or columns for category code, outcome, location, etc.
• Allow columns to have ‘NOT SPECIFIED’ where bodies didn’t provide that level of detail (or combined detail)

Once you’ve identified a structure, you’re going to need to reshape some of your data to fit into that. This is another task that can be done well by AI (there are manual alternatives if you are able to code), but again it is better to ask AI to generate and comment code that you can run, rather than performing the reshaping itself.

A prompt template for reshaping data in this way is available here. It sets an objective, describes the target structure, and describes the task step by step.

You can validate reshaped data against the original using the same checklist as before (or you may decide to only validate at this point). In particular, where data has been reshaped from a pivot table-type response, you can validate it against the original data by generating pivot tables to replicate the original shape for comparison.

Problem 6: You need to clean inconsistent categories or entities

It’s likely that different authorities will have used different language to refer to the same entities or similar categories. This is a problem if you want to tell a story about how many events there were in a particular category (or connected to an entity of interest), or which category or entity ranked top or bottom.

Fixing this problem is best left until you have all the responses in a single dataset, as this makes it easier to identify clusters, and means you only have to clean one dataset, rather than each one.

Once you have a single dataset, generate a list of all the unique categories or entities, and have a quick look to get a feel for the problems it presents. The key here isn’t to see this as a single cleaning job — you’ll need to break it up into multiple cleaning stages, each of which tackles a different problem.

Here are six likely stages you might need to clean for, in the following likely order:

1. Mixed data (for example, category and code, or crime category and Act/Section): this might be fixed by using the spreadsheet’s Text to columns tool. This is probably the first stage of cleaning to perform, as it will isolate strings of text for further cleaning.
2. Acronyms (for example some authorities using GBH while others use “Grievous bodily harm”): use an IF or EXACT function combined with UPPER or REGEXMATCH (Google Sheets only) to identify all-caps entries.
3. Upper/lower case inconsistency: this can be fixed by using a text formatting function like PROPER. However, this will also turn acronyms like GBH to “Gbh” so you will need to fix acronyms first, or filter them out while cleaning everything to proper or lower case.
4. Extra/analogous characters: some responses may use quotation marks, spaces, apostrophes, commas, and currency symbols where others don’t. Or some may be “and” while others use “&”. Using Edit > Find and replace can deal with some of this, as can the SUBSTITUTE function. The TRIM function can get rid of spaces at the start and end, and REGEXREPLACE can replace certain patterns of text.
5. Slight variations (e.g. one authority uses “Criminal damage and arson” while another users “Criminal damage and arson offences” or “Starbucks” and “Starbucks Ltd”): Open Refine’s cluster and edit tool is very powerful for tackling this, and you can also sort the categories to bring similar ones together to edit them manually
6. Different category level (e.g. some use the top level categories, others sub-category, and others lower level categories): this is the trickiest to deal with, and best left until you’ve cleaned for all the problems above. You will need a lookup dataset with categories at different levels, and then a lookup function to make matches and establish category levels.

Obtaining reference data on categories is extremely useful in general: it helps give you a target for your own data. If there are 12 official top level categories, for example, then you know that you need to get your list into those 12 categories (by assigning sub categories to their parent category, for example)

For crime the Home Office Crime Recording Rules for frontline officers, staff provides a very useful lookup table listing four different levels of category for crime (from specific offences to offence categories, sub classes and classes) as well as their Home Office code, any of which might be used in responses.

You can use AI tools to assist with ‘auditing’ your data along these lines too. A template prompt is suggested here. This also provides code that you can use to run the audit yourself, and role prompting to highlight any potential problems.

The output from that prompt can be used to sort the data for each stage of cleaning. Make sure that you retain the original list of ‘dirty’ terms alongside the progressively cleaned versions. This will give you a lookup table you can use to translate from inconsistent categories to a consistent list.

A major benefit of this approach is that the results are valuable for any future FOI project, too. That lookup from dirty to clean categories can be used and reused.

Finally, the project is ready to answer questions accurately

At this point you’ve broken down and tackled each of the distinctive problems that mass FOI projects involve:

• You’ve extracted information from responses in different formats, and put them in a single format (CSV)
• You’ve converted information in different shapes (wide, long) and put them in a single shape
• You’ve validated that extracted information to check that data is consistent with the source, and is not missing
• You’ve combined data from different responses into a single file
• You’ve standardised the language used in different responses so that answers to questions will be accurate and consistent

Now you should finally have a dataset that can answer questions.

Giving the whole problem to AI agents doesn’t work

One final note: I tried agentic AI tools — ChatGPT’s Codex and Claude Code — on two of these problems, to see how their performance compared with the more hybrid human+AI approach. Both produced plausible outputs that could easily have been mistaken for the data trapped in the FOI responses, but…

When given the task of getting a folder of FOI responses from police forces into a consistent shape, the resulting CSV was missing half of incidents and one third of forces. Most forces were missing incidents.

The same problems were seen when Codex and Claude Code were given the challenge of cleaning up categories with a target table of official categories. Codex only managed to classify 35% of the categories correctly, while Claude Code managed 53%.

More detailed prompting (perhaps drawing on the experiences above) might produce better results, but the results aren’t really the challenge here: the key challenge is how to know that you have all the results, or the correct matches. For that, there still needs to be some manual validation step, especially focused on typical problems with PDF extraction.

This is a work in progress. If you have other tips, or a collection of FOI responses to test with this, let me know.

*Thanks to Martin Rosenbaum for suggesting using two columns for ‘value from’ and ‘value to’ where organisations do not provide a single number.

Every bad example of charts in videos forgets this. And every good example uses two essential techniques: keeping things simple, and adding motion.

Remove to improve: edit your chart ruthlessly

Viewers looking at an on-screen chart have to take in at least three types of information in a short space of time: shapes (bars, slices, lines, etc.), colours, and text (titles, subtitles, labels, annotations and legends).

The more you can reduce the time needed to interpret each of those, the quicker the viewer can understand the chart.

The best place to start is reducing the number of data points to those that are absolutely essential. This will reduce the number of shapes the viewer has to absorb.

If your story is about ranking the worst or best areas for a problem, for example, a top ten is too much for video. Cut back to just the top three or five.

If you are comparing change over time with a multiple line chart, experiment to find out what the smallest number of lines is that you can use and it still work.

If your story is about parts of a whole then, again, you don’t need to break down each part: your pie chart might simply show two slices: the part you are focusing on, and the rest of the whole.

Next, reduce the number of colours: this is good practice generally, but on video it’s especially important: make the element that is the focus of your story (the bar, line or slice that the story is about) a strong colour. Then, make all other elements a neutral or non-colour such as grey or pale blue. This tells the viewer immediately which bar, line or slice they should be looking at.

Good titles remove the need to read labels or axes

Now, reinforce the story with text: edit the title of the chart so that it tells the story, succinctly. For example, a bar chart showing the worst places for pollution that is titled “Worst cities for pollution” or “Pollution in each city” creates unnecessary work for your viewer. A title that says “Scunthorpe is the most polluted city in the UK” tells them precisely what the chart is doing — and what city the highlighted bar relates to, saving them the work of trying to find the label for that bar.

Similarly a title that says “Crime has dropped 10% in the last three years” or “40% of donations to the party came from Company X” allow a viewer to understand a chart without having to check labels or axes compare slices or lines.

You should also remove unnecessary text. For example there’s no need for both a legend and direct labelling of bars: direct labelling is quicker and easier to understand. Likewise, it might be easier to directly label values rather than place them on an axis. Subheadings are likely to be a luxury you can do without. Use trial and error to see how what can be removed or moved — make sure you test drafts on someone unfamiliar with the chart.

You may be able to remove further elements from the chart if they are being narrated in the video, or if animation will add them later.

Direct attention with motion

Colour and text are just two techniques for directing attention in data visualisation — but in video there’s a third: motion.

Here are just some of the methods you can use to add motion to a chart:

Zoom: zoom in to a focal data point to direct attention, or to zoom out to put a data point into the bigger picture. Zooming might involve expanding or shrinking the scale to illustrate how a particular data point affects it.

Reveal: instead of the chart appearing fully-drawn, it can be revealed in stages. For example, we might start with the whole amount, and then reveal specific parts (slices) of that amount, one by one. Or we might start with a scatterplot, then add a trend line, then add a highlighted area, and so on. Typically this is accompanied by narration that relates to each reveal.

Wipe: often used with a line chart. The wipe reveals the line from left to right. Because a line chart shows change over time this is particularly effective as we see that line change over time. A vertical wipe can also be used with bar charts and histograms to create the effect of the bar(s) growing upwards or outwards. Wipes can even be used to reveal titles or labels to ‘read’ those out.

Highlight or fade: an element can be highlighted — or the rest of the chart quickly faded — to direct attention to that particular element or series of elements. Conversely, a focal element can be faded back or de-highlighted to indicate that the focus is about to shift to the whole, or to another part (which might then be highlighted in turn).

Removal: instead of revealing elements, you can hide them. So instead of starting with simplicity and gradually adding more detail, we start with complexity and strip it back to focus on something in particular.

Scroll or pan: a more complex chart might need to be revealed vertically, as we scroll down to specific elements of interest. The Economist’s Epstein Files video on TikTok provides one example. Or you might pan across a wide chart horizontally to emphasise a scale.

Counter: often used for scale stories that focus on a single number, this technique uses a counter to draw attention to the final figure that it arrives at.

Annotation: extra text is added on screen to direct attention to a specific element, often with an arrow or circle to connect the text with that element. This is especially useful with scatterplots where the story might need to focus on different data points (you can see an example in this Pudding video at 1’03).

Timelapse: bar chart races are a particularly popular example of animating the transition between multiple ‘snapshots’ of a chart at different points in time, and the same principle can be applied to maps and other charts (Joey Chedarchuk collects a number of examples in this Twitter thread).

Presenter interaction: a common way to add motion with charts on TV is for a presenter to interact with it: pointing to different parts of it, or ‘operating’ the chart via touchscreen, clicker or tablet. A classic example is Hans Rosling’s 200 years in 4 minutes.

A newer form of this on TikTok is for a presenter to appear on top of a particular part the chart and then move to a different point to match the focus of the narration.

Do you know of any other key techniques for using charts in video storytelling? Let me know in the comments or at linkedin.com/in/paulbradshawuk

Some essential reading by Agnes Stenbom Swedling explores how news organisations integrate AI into their workflows and the idea of the “human in the loop“. Many newsrooms, she points out, “are not optimised for what humans do best”, and so far the introduction of AI hasn’t involved a critical consideration of whether we want to embed those features in new systems, or rethink them:

“What is being built – incrementally, often unintentionally – is a form of machine-centric hybridisation. Workflows are optimised for what machines do well: speed, scale, pattern recognition, cost efficiency. Humans are then positioned around those systems, adapting their tasks, roles, and decision-making to fit the logics of machines. 

“The consequence is a subtle but significant inversion: rather than engaging in uniquely human activities, work is reorganised to fit machine-driven processes. And once that inversion is embedded at the infrastructural level, it becomes increasingly difficult to reverse.”

The writer Cory Doctorow has a term for those humans: reverse centaurs, “a person who has been conscripted to assist a machine.”

A recent issue of the FT newsletter The AI Shift documents this among software developers: “the most creative activities of the job are being replaced by the tedium of checking machine output,” writes one.

In contrast, a plain old “centaur” in tech terms is someone who enlists technology to assist them. Florent Daudens highlights one example of this recently in a Toronto Star investigation into Ontario’s strong-mayor powers. “The method,” he writes, “deserves just as much attention and shows what happens when you apply editorial judgment at scale. The team first made their editorial judgment explicit [and] a custom AI tool then applied those criteria across thousands of documents. 

“The reporter hadn’t been replaced. They’d been amplified. By an agent built around their judgment, not against it.”

What these examples point to is that machine-centric hybridisation is not inevitable. At the same time as that dark future looms (if many of us aren’t already living it), AI also opens up a counter-intuitive opportunity to make work more human.

After all, one of the qualities that often makes work feel dehumanising is when it becomes overly routine. Many newsroom workflows are built on such habits: check the news diary and forward planning, check the emails, check the wires, rewrite a press release.

A part-automated workflow introduces opportunities to step back from that routine because:

• We have to design prompts and strategies, and engage with agents in ways that force us to make more explicit judgements about values
• We spend less time on (re)producing, monitoring and chasing, and more time on imagining, choosing, checking, reviewing and editing.

The AI Shift newsletter documents this too. One developer explains that their work now focuses more on “architectural decisions (I.e. how do we best assemble all of those small chunks of code to do something complicated?)

“This aspect of development is often overlooked by non-technical leaders because, yes, in the short term, you *can* mash a load of LLM-generated code together and be somewhat confident it’ll do the thing you’ve asked it to do… but you run out of runway quickly as your product becomes more complex. It’s also a non-starter for anything that could end with a lawsuit.”

And journalism involves enough legal risk and complexity for all of the above to apply to newsrooms too.

For investigative work, those points are central. Surface-level scandals often stem from deeper systemic problems. So what tools do we have for recognising those patterns?

Donella Meadows’s classic book Thinking in Systems offers one: “system traps” — patterns that explain how systems get stuck, break down, or behave in ways nobody intends. They are “traps” because attempts to escape them often backfire.

In this post I’ll explain each trap, what it looks like in the wild, and how to use it as a lens for story ideas.

1. ‘Fixes that fail’: policy resistance

Why wasn’t the war on drugs ever won? Why do healthcare reforms so often backfire? Why are targets to reduce emissions missed? These are classic policy resistance stories, where different parts of a system pull in opposite directions.

Policy resistance is perhaps the most visible ‘system failure’ — but recognising it means we know what questions to ask, and what to look for.

• What to look for: Reforms that don’t work, or make things worse. Outcomes that don’t improve. Missed targets, broken promises.
• Questions to ask: Who has a different goal within the system? Is one part of the system cancelling out another? Are there places where the problem is being tackled successfully?

In environmental reporting, for example, stories regularly emerge from conflicts between policies aimed at reducing emissions and policies aimed at increasing production, consumption or employment.

One of the simplest ways to identify and reveal “fixes that fail” is the ‘lack of change‘ story: once a policy has been in place for a long enough period, investigate if it seems to be working. Use data to spot where a system is resisting its own policy.

Look for resistance or opposition to change: The Washington Post, for example, “examined three historic firsts in policing reforms” to reveal reforms “stifled by entrenched cultures, systemic dysfunction, shifts in leadership and swings in public mood”.

That resistance can be external, too, such as local communities resisting green energy plans due to concerns over property prices or local wildlife

Anniversaries can provide useful hooks to review attempts at reform, as in ProPublica’s feature on the stalling of policing reform “More Than Two Years After George Floyd’s Murder Sparked a Movement”.

Another approach is to look for unintended consequences. The increase in orphans and orphanages in Romania in the 1980s and 90s is one example (the unintended consequence of a policy whose intention was to increase birth rates); communities scarred by the war on drugs is another.

2. Tragedy of the commons 

This trap appears when shared resources — fisheries, roads, emergency services — are used by individuals in ways that degrade the whole.

• What to look for: Unregulated or poorly monitored access to a shared resource. Growing use without growing responsibility.
• Questions to ask: Who uses the resource? Who monitors it? Are there rules — and do they work?

A potential story on this system failure might draw on political scientist Elinor Ostrom‘s eight pressure points to look in the following areas:

• What are the boundaries? Who counts as a “user” or stakeholder?
• Is the system being monitored? If so, by whom?
• Who has decision-making power — and is it legitimate?
• Are rules being broken or gamed, and if so, how are they enforced?
• What does the conflict resolution process look like — or is it missing?
• Are local voices recognised — or excluded?

Science‘s investigation into fisheries, for example, “tested how accurate estimates of fish stocks actually are” (monitoring). Daniel Wizenberg’s reporting on cruise ship pollution reports on ports considering bans and restrictions (punishments) and “a divide among residents about the benefits and drawbacks of tourism” (the balance of costs and benefits).

Green to Grey: How Europe Is Squandering the Little Nature It Has Left fills a gap in monitoring, identifies the ineffectiveness of non-binding EU goals, and quotes local voices that were ignored.

But this trap doesn’t just apply to environmental resources: a story about the increasing demands being placed on A&E services might similarly focus on boundaries (e.g. patients using A&E instead of other services) and relationships. Similar areas could be looked at in a story about housing stock being rented out to tourists. Antibiotic overuse is another example.

Jason Del Rey’s story on internal Amazon research warning that the company was “running out of people to hire” is a tragedy of the commons, and the “attention economy” is one way to describe another commons that is being exhausted.

3. Drift to low performance (aka eroding goals/boiling frog syndrome)

A typical “drift to low performance” trap is a negative feedback loop, where systems slowly get worse, people adjust — and low expectations (‘eroding goals‘) become the new normal.

A good example of this being made explicit is the BBC story Harm at risk of being normalised in maternity care.

In education, the Bureau of Investigative Journalism’s investigation into the “broken” special needs system identified how thresholds had been raised to “make it more difficult for children to receive support”, while grade inflation at universities is another symptom of standards being lowered.

In environmental journalism, you might look for targets being weakened, barriers to protection, or a lack of enforcement.

• What to look for: Declining standards, redefined targets, morale collapse.
• Questions to ask: Have goals or benchmarks been quietly lowered? Are people adapting to failure instead of fixing it? Are regulators turning a blind eye?

4. Escalation (arms races and races to the bottom)

Escalation traps form when organisations try to outdo each other, creating a feedback loop as each responds to the other. Examples include arms races, military conflicts, political polarisation, and cat-and-mouse relationships (e.g. spammers versus email filters).

‘Races to the bottom‘ are an escalation trap. Examples include negative political campaigns, increasing deregulation and lowering taxes, and price wars — where competing companies make products increasingly cheaper to undercut rivals.

Escalation of even apparently positive behaviours can also become an escalation trap: Meadows gives the example of morality leading to sanctimoniousness, Puritanism and intolerance. Other examples might include healthy eating becoming orthorexia, and healthy scepticism turning into conspiracism.

Many investigations into social media algorithms are stories about escalation traps: the Wall Street Journal’s Facebook Files, for example, shows how the company’s desire for growth led it to ignore warnings that algorithm changes were making “Facebook, and those who used it, angrier”, while NRK’s investigation into TikTok showed “how TikTok’s algorithms drag children into a world of extreme exercise”. Algorithm Watch’s investigation revealed the Instagram algorithms that encourage “showing skin”: users themselves are often caught in an arms race too.

Political arms races include parties trying to outdo each other on being “tough on crime” or anti-immigration. Stories on the escalation trap might also focus on how inflationary forces are playing out, such as tech companies or sports clubs trying to outspend each other on talent.

• What to look for: reactions and counter-reactions. Unhealthy or extreme behaviours being reinforced. Cat-and-mouse games between rule-breakers and enforcers.
• Questions to ask: What concerns are being raised? What’s driving the escalation? Who benefits? Is there a way out? How might ‘unilateral disarmament’ be facilitated (e.g. through regulation, agreement or redesign)?

5. Success to the successful/competitive exclusion

The rich getting richer and the poor getting poorer are well known examples of the competitive exclusion system trap. This is another feedback loop, where ‘winners’ in a system have an unfair advantage when it comes to competing for new opportunities or resources, leading to a vicious circle.

Cory Doctorow’s book Enshittification provides an endless list of examples of this system trap from multiple fields, along with the playbook that companies follow (including chickenisation and the giant teddy bear tactic), and the forces that lead to it.

Education is an area where this trap shows itself regularly: look for evidence that the gap between private and state school pupils going to university is widening, for example, how the use of private tutoring “deepens inequality”, or the “geographic exclusion” of poorer children from higher performing schools.

Tech is another: in 2015 Reuters revealed that Amazon used its access to seller data to design its own private‑label products and “manipulat[ed] search results to boost its own product lines in India”, while in 2015 The Wall Street Journal used FOI to reveal “evidence that Google’s algorithm was demoting the search results of competing services while placing its own higher on the search results page”.

Focusing on supplier vulnerability can help expose where dominant companies may be exploiting their power. The Guardian‘s coverage of the UK supermarket sector describes multiple examples, while its reporting notably focuses on the rules designed to prevent such exploitation, and the regulator’s (lack of) power to impose effective punishment.

Broader scrutiny of regulation (complaints, investigations, reports, data), tax laws (such as taxes on inheritance but also the use of “sweetheart tax deals for companies like Uber“), and anti-monopoly laws— how often they are enforced, the effectiveness (or not) of punishments, how companies avoid those, and the need for new rules — provides another potential basis of reporting.

In sport, for example, journalists have investigated how “Abramovich’s Chelsea may have secretly bypassed Financial Fair Play (FFP) rules governing world football, helping propel the club to the pinnacle of the game, at the expense of its rivals”.

Other forces to consider include first-mover advantage, the “competency trap” (when people don’t switch to alternatives because they are used to a particular product), and ‘network effects‘.

• What to look for: Self-reinforcing inequality (e.g. funding that rewards institutions with better resources, reviews or results), whistleblowers, network effects, regulatory effectiveness, lobbying for regulatory change, tax deals and subsidies, ‘bundling’ of products
• Questions to ask: Who controls key resources? Are newer players being excluded? What is the experience of those at the bottom of the supply chain? What complaints are being raised with regulators? What reports are produced by regulators internally?

6. Treating symptoms not causes: shifting the burden to the intervenor

Examples of “shifting the burden to the intervenor” include the use of subsidies or price fixes to prop up failing industries or maintain access to cheap fuel or credit, or practices that will be damaging in the long term (e.g. overuse of fertiliser, drilling for more oil) instead of more sustainable options.

The Bureau of Investigative Journalism‘s investigation into the housing crisis, for example, highlights the problems caused by subsidies that ultimately flow to landlords, instead of regulating rents or building affordable homes. And the Marshall Project is just one of a number of organisations to have investigated how jails are used as a quick ‘fix’ for drug and mental health problems instead of treatment, or police being used to deal with public health failures.

Outsourcing is a common example of shifting the burden, as it often results in public bodies losing the ability to provide services themselves and becoming dependent on private agencies (who may prioritise costs over outcomes). When there are safety concerns over those private agencies, or they fail to deliver, it creates further problems.

This system trap is often also called “addiction”, because the ‘trap’ is that these are short term ‘fixes’ offering immediate relief but creating a cycle that’s increasingly hard to break without pain or disruption (addiction itself is just one example of this trap).

The first episode of the FACTA.eu investigation Non è un’agricoltura per piccoli, for example, quotes owners of small farms who have difficulty accessing subsidies and report “a widespread feeling of being faced with a trap rather than a support”.

• What to look for: Band-aid policies, outsourcing, bailouts, pressures in the system spreading to other parts (e.g. from health to police), recurring crises, companies promoting ‘quick fixes’ to deeper problems (e.g. AI), “addiction” to or dependence on subsidies or emergency measures
• Questions to ask: What deeper problem is being avoided? Who’s intervening? Are outcomes improving—or simply being masked? What skills or oversight has been lost as a result of outsourcing? What happens when the ‘contractor ‘intervenor’ fails? Who profits, and who pays when it goes wrong? How effective is regulation of outsourcing, privatisation or services ‘filling the gap’ for others? What is being done to tackle the root causes?

7. Gaming the system: rule beating

In this trap people and organisations follow the rules but violate the spirit of the law. The most widely reported example of this is tax avoidance: strategies for avoiding tax which are legal but widely perceived to be unethical (illegal strategies are called tax evasion).

Rule beating can be a good subject for factchecks or explainers, such as Channel 4 News’s factcheck of why Amazon UK pays so little tax and Ethical Consumer’s explainer on how much that costs citizens in lost taxes. Reporting can also focus on reactions to legal but unethical behaviour from experts and campaigners, including calls for loopholes to be fixed.

Deeper investigations can focus on:

• Uncovering the use of loopholes as in Sky News’s investigation into recruitment companies paying workers “by third-party umbrella companies … what were technically loans”.
• Using laws to avoid scrutiny or accountability, such as the doctrine of qualified immunity investigated in Reuters’ series Shielded, or SLAPPs.
• Revealing unusual behaviour, such as iWatchAfrica revealing that “Facebook had not paid any direct taxes in Ghana and Nigeria since it began operations over a decade ago”
• The scale of such behaviour (“One in six UK public procurement contracts had tax haven link” is one example; LuxLeaks is another. Il Post’s story on the use of agency staff in hospitals identifies its scale through the use of a different procurement code, but also identifies other tactics used to get around the law)
• Unfair variation, as in NRK’s report on foreign-owned wind power companies paying less corporate tax than Norwegian-owned ones.

These approaches can be adapted to other systems where rule beating can be found: in 2023 the Freedom of the Press Foundation reported that “intelligence and law enforcement agencies have been using data brokers as a loophole to get around the warrant requirement and other legal restrictions on accessing information for their investigations”, while in online gambling, The Investigative Post revealed that “Some betting apps promote fantasy sports formats to avoid classification as gambling. In states where online gambling is banned—like California—offshore casinos step in.”

• What to look for: Stagnant or reversed progress, exploited loopholes, odd behaviour at the end of the financial year, unexpected numbers, rules failing to keep up with new technologies, calls for rules to be changed
• Questions to ask: Who benefits from the current rules? Who loses out? Are rules being circumvented? Why does it matter if rules are circumvented? Have rules kept up to date?

8. Seeking the wrong goal

The final system trap is where a person or organisation is motivated by the wrong goal, often because of the metrics that are used for rewards or punishment. As a result, what should be the real goal is undermined (“What gets measured gets managed”) and the measure ceases to become meaningful.

For example, treatment targets may drive hospitals to discharge patients early, when it is not in the best interests of those patients. Police may misclassify crimes to improve their crime figures, and schools might teach students how to pass exams, rather than educating them.

The Stafford Hospital scandal is one high-profile example: “appalling” conditions at the hospital, exposed by local reporter Shaun Lintern, led to an inquiry which noted discussions at the hospital’s board “were dominated by finance, target and achieving foundation trust status. There is little evidence that poor standards of nursing care were identified and discussed.”

In education journalists have exposed the practice of “off-rolling” poor-performing pupils to boost results, while in crime the Tampa Bay Times exposed how police boosted statistics for rape cases despite not having “identified a suspect, assigned a detective or even confirmed that a crime had occurred”.

Sometimes the measure doesn’t even have to be a target: Tim Harford’s book Messy dedicates a chapter to various examples of incentives and measures skewing behaviour, such as “how the APGAR measure went from a metric for newborn babies to the target that drove (at least in part) the rise of C-sections in the United States.”

Identifying how an organisation’s performance is measured, particularly where it determines funding, can be a key step in identifying the ‘wrong goal’ trap in action.

• What to look for: How organisations’ or individuals’ performance is measured (particularly where it determines funding or promotion). Perverse incentives, metric-driven behaviour, goal distortion, signs of ‘gaming’ the system. In data: special/vague categories (which might be used for misclassification), failing to record data, or changing classifications. Board reports that indicate ‘wrong’ priorities.
• Questions to ask: Are the chosen goals actually improving outcomes? Is effort being rewarded instead of results? How should goals be changed?

I’m on the lookout for more examples of investigations based on system traps – if you know of one please let me know in the comments or on LinkedIn.

Making a map for a story is cool — but what if you could make a map for every reader? Or if you’re working on a project involving teams in different regions or countries, what if you could give each one of those teams a map centred on their own patch?

Normally you would have to manually move the map to centre it on a key city, and then export an image. Then do it again and again and again for every area.

Luckily, QGIS has the ability to run code. And this is a great excuse to start using it.

Start by creating your base map

Before getting into the code, however, you need a map for the code to move around

Create a new project in QGIS and import any shape file(s) that you want to show on your map, along with a base map underneath that. I’ve created a tutorial here which imports flood defences and then adds an OpenStreetMap base layer underneath those.

Tweak the design of the map until you’re happy, and then zoom into one of the locations on your map (for example, a key city). We need to use this to create a test ‘area map’ so we have an idea of the scale and proportions we can use when we automate things.

Once zoomed in, create an image layout by selecting Project > New Print Layout…. I’ve created a tutorial for that, too, which you can follow here. This allows you to identify what scale to use when you start to add code.

PS: you can make the map entirely in code itself without doing any of this, but I find it quicker to work with the map on screen first so the results can be checked and tweaked more quickly.

Download the template Python script and open in QGIS

I’ve written a Python script that you can import into QGIS to generate multiple images from our map. You can see the code here. To download it, click the three dots in the upper right corner and select Download. I’ve created a tutorial here which walks through the steps.

The code does the following:

1. Specifies a folder to store all the images
2. Stores a list of lat-longs
3. Creates a transformer to convert those to the projection used by your shape files (the coordinate reference system, or CRS)
4. Sets a map size and scale
5. Loops through each location in the list, zooms to that scale, and adds a label with the location and scale
6. Sets a filename that includes the name of the location
7. Exports an image with that name (for each location in the list)

Download it to the same folder as your QGIS project and open it in QGIS by going to Plugins > Python console as outlined in the tutorial.

Run the script — and customise it

Once you’ve downloaded the script and opened it in QGIS, you can check if it works.

Click the green ‘Run’ button to run the script. As it runs the Console area on the left of the Python Console will display any print messages that are in the script to make it easier to track where the script is (like running script and imported libraries).

The two key things to keep an eye on are whether it exports to the right location, and whether the list of locations works.

The code includes a line which sets a target directory on your computer:

out_dir = os.path.join(os.path.expanduser("~"), "Downloads", "qgis_images")

If it works, you should find the images in your Downloads folder, in a folder called qgis_images.

But it might not work on your computer. If it doesn’t – if you get an error – or if you just want to store the images somewhere else, you can do that by uncommenting the following line of code (i.e. removing the hash character) and changing it to describe a path on your computer:

# out_prefix = "/Users/paul/Downloads/testqgis/images/"

You can edit the code directly in the editor window in QGIS (the part of the Python Console where you can see the code) – make sure that you save it once you’ve made any changes.

The list of locations is in this block of code:

listofdicstocsv = [
    {'lat': 50.844441271809465, 'long': -0.2985307978506628, 'officialname': 'Adur District Council'}, 
    {'lat': 54.71108952940033, 'long': -3.2472347297148736, 'officialname': 'Allerdale Borough Council'}
]

This should work with the flood defences data, but if you are using a different map you will need different lat-longs and place names. You can edit these manually for now with your own test lat-longs and names if needed.

Expand the list

Once you’ve got it working with two locations, it’s time to update the code so that it uses a full list of locations. Automating a process like this is called parameterisation, because it involves generating multiple outputs based on one or more parameters (in this case the lat and long and name).

In the tutorial we are using a list of major cities in the UK. You might need a similar list for another country, or it might be centre points of different regions, or the locations of key events.

Whatever the locations are, you’ll need to convert that list into a Python list of dictionaries.

A dictionary looks like this:

{'lat': 50.844441271809465, 'long': -0.2985307978506628, 'officialname': 'Adur District Council'}

The curly brackets are the giveaway: those mark the start and end of a dictionary. Inside them is a series of pairs, which are equivalent to a label (lat, long, officialname), or key, followed by a value (50.8, -0.29, ‘Adur District Council). Each of these pairs (called key-value pairs) is separated by a comma.

A dictionary is like a row in a table: the keys are the column headings and stay the same each time, while the values are different in each row. So when you have a list of dictionaries, it’s basically a list of rows. In other words, it is just another way of storing a table.

So we are going to need a table of locations (the ones we want to create maps for). We can then convert that table to a list of dictionaries (a list of table rows).

You might be tempted to paste your table or attach a CSV in a prompt and ask an AI tool to convert that into a list of Python dictionaries. But it’s important when using AI to assess whether the task justifies the environmental impact and consider non-AI alternatives that achieve the same results.

For example, the website CSV2JSON will perform this task (JSON is just a list of dictionaries). It will take a CSV file and provide it as a list of dictionaries that you can copy and paste into your Python script. I’ve also written a tutorial on other options to achieve the same results with a spreadsheet formula.

Once you’ve got your list of dictionaries, you can replace the list in the template Python script that starts and ends with square brackets here:

[
    {'lat': 50.844441271809465, 'long': -0.2985307978506628, 'officialname': 'Adur District Council'}, 
    {'lat': 54.71108952940033, 'long': -3.2472347297148736, 'officialname': 'Allerdale Borough Council'}
]

Note that the list starts and ends with square brackets, and each dictionary in that list is separated by a comma.

Once your list is pasted, you’ll probably have a big chunk of code (or a very long line of code) with all those locations.

Testing the expanded list

You can now test the code works with more locations by saving the updated code and running it again.

But you might not want to export hundreds of images straight away, as this will take a while.

Instead, you can try a slice of the list by adding square brackets when you run your loop to specify a limited range of items, like this:

for rec in listofdicstocsv[10:20]:

That [10:20] means just loop through items at positions 10 to 19 (it stops just before the 20th item) in the list listofdictstocsv. You can obviously change those numbers to specify a different range.

Once it runs fine with the range specified, you can then look at the results and decide whether you want to export all the images, or tweak your code further to a different scale — or multiple scales…

Exporting more than one scale

As you export multiple images you might notice that not all locations are the same. Some locations are self-contained and suit a smaller scale (e.g. more rural cities or a London borough), while others are large conurbations (e.g. Birmingham) that need a larger scale to show all the different parts.

Now we need to adapt the code to export at more than one scale.

You can see an example of this adaptation in a second Python script. This adds code that does the following:

1. Creates a list of scales to use
2. Adds an extra loop inside the one that loops through each location, which loops through each scale
3. Zooms to that location at each scale, and exports an image for that scale (with an appropriate label and filename)

The key block of code to adapt is this one:

MAP_SCALES = [
    175000,   # original scale
    120000    # zoomed-in version
]

You can replace those values with your own preferred scales, and it should work.

But what about other adaptations — and trouble-shooting? This might be where you turn to AI…

Using AI to adapt QGIS Python code further

Large language models (LLMs) are particularly well suited to helping with coding and coding problems — but that comes with the risk of deskilling, or denying you the opportunity to learn. To manage that risk, try a prompt design which begins with this template:

You are my mentor, a data journalist with over a decade's experience in the field. 
You have advanced statistical knowledge as well as a healthy scepticism when dealing with both data and human sources. 
I will ask you for help with some coding - you are happy to guide me, but you don't want me to become deskilled and too reliant on you. 
Your advice will always be designed to force me to think for myself, learn new skills and concepts, and practise those.
When you provide code add comments explaining each step as if to a person with no coding experience. 
I prefer simple code that requires multiple lines to complex code in one or two lines.
Flag any assumptions you are making about the data or my question. 
Warn me if the question does not contain enough information to answer accurately.
Warn me if the result could be misleading without additional context.

Responses using this template are likely to help you better understand the code that is being produced, as well as identifying things to consider (such as the risks of mixing layers with different projections) and pushing back if you’re leaving important information out.

To adapt the Python code, you might then attach the script with a prompt like this:

Attached is some Python that generates images for maps centred at hundreds of locations. Adapt this so that it now generates those images at two different scales. Comment the code to highlight the section of code where it does this, so that I can tweak it and try different zoom levels.

Other ways to address deskilling include parallel prompting, journey prompts and destination-journey prompting.

You can find the slides from my Dataharvest workshop, including Spanish, Italian, Portuguese, French and Dutch versions, along with walkthroughs, materials and useful links, in this GitHub repo.

Q: Is the continuous growth of AI-generated content online a danger for journalism?

It is certainly a problem yes, in three ways: it makes reporting harder, it makes it harder to support journalism financially, and it makes it harder for audiences to trust your reporting.

It makes reporting harder because a journalist has to deal not only with a lot more misinformation but also a lot more information full stop.

It makes the business model more difficult because there’s more competition from AI-generated slop.

And that’s compounded by the fact that audiences are less likely to trust your reporting — either because they might dismiss it as AI generated, or because they might have already begun to believe AI misinformation that your reporting contradicts.

Q: Is the industry equipped to handle it?

I don’t think any industry is equipped to handle how AI is changing the information environment, and I think it will take some time for society to adjust to a world of unlimited cheap and unreliable information.

But journalism is probably in a better position than most industries to handle that change because it has rules and workflows that a lot of other industries do not.

For example, journalists are trained to research both sides of the story and seek a right of reply, and to find more than one source for key facts. A high value is put on speaking to real people and capturing raw footage of events. Reporters are encouraged to be sceptical and apply verification and factchecking skills, and their stories go through editors who are supposed to look at the story dispassionately. These are not qualities that all industries have.

Also, the industry has been exploring applications of artificial intelligence for over a decade, so there’s technical knowledge there too which many organisations will lack.

The biggest challenge is the collapse of the financial basis for journalism and I think there will have to come a point where more public or charitable funding is used to support it.

Q: When AI can fabricate convincing news content at scale, who is ultimately responsible for the integrity of what we read and watch online?

Primarily it’s whoever created that content. When the printing press came along someone might have asked who is responsible for these books that are being published? And of course we know the answer now is the authors and the publishers.

But technology companies are also responsible, because AI is not a neutral tool like a printing press.

ChatGPT is a collection of recipes for predicting text, and so OpenAI, which owns ChatGPT, has responsibility for the way it has written those recipes because it can change them. It has changed them, and does change them, regularly.

If you try to create racist or other offensive content in ChatGPT or Gemini or Claude, for example, it will generally refuse to do so. If you create images in Gemini a watermark is added, so it can be identified as AI generated. If you ask Claude for medical advice it will recommend consulting a health professional. These are editorial choices by the technology company.

So while individual crimes involving fabrication might be tackled by identifying and charging the creator, the ultimate responsibility for any systemic social harm will lie with technology companies who haven’t considered key risks and taken steps to prevent that. And there will need to be regulation and enforcement to ensure that happens.

Q: What does this mean for the public’s trust in journalism?

We have always had to earn that trust, and that has become more and more important as competition for the public’s attention has increased.

Part of that trust is about building understanding: not only explaining how we actually do journalism, and why we do it that way, but also listening to our audiences to understand what they need and why.

In order for the public to trust journalism they have to see how it is distinct from other sources of power, and holds that power to account. They have to see how it is on their side, rather than looking down on them.

Q: When fabricated content goes viral, what happens to people’s ability to make informed decisions?

I think we already had a problem with people being able to make informed decisions, as it became easier to find information that confirmed a person’s existing beliefs rather than information that challenged those.

Ultimately the more misinformation there is, the more likely it is that people are either making decisions based on flawed information, or unable to make decisions because they can’t be 100% sure a certain piece of information is true.

I think we are going to learn some quite painful lessons in the next few years because most people think they won’t fall for this stuff, and yet most people do. I am an intelligent person who trains people in verification and factchecking, and I have retweeted information that is not true.

There are two key things that we need to remember: firstly, that you will fall for misinformation at some point, and probably already have, that is certain.

Secondly, slow down and look for counter-evidence.

Q: For an ordinary person watching the news or scrolling their feed, what practical steps can they take right now to check the veracity of what they are seeing?

There’s a great piece of research in psychology that finds when we look at an image of a crowd of faces, we pay attention to the angry faces more: our eyes spend more time on those.

Our brains are designed to pay more attention to threats, and social media algorithms have learned this, so they prioritise material that triggers anger and fear because we spend more time looking at it, and are more likely to react to it.

So the first practical step people can take is to slow down. When we come across new information two parts of our brain processes it: the first part is fast and instinctive. It makes a decision whether to pay attention and what to do with the information. That’s the part that shares or likes or comments on a chat message or video.

Don’t share. Don’t like. Don’t react.

The key thing is to let the information move through to a second phase, the part of the brain that actually processes that information rationally.

Now, ask yourself if that update is triggering some core emotion. Is it confirming something you believed? Is it making you angry or afraid? Why? The short answer is: because it keeps you on that site or app for longer. But is it true?

The main way to check veracity isn’t to look for clues that something is fake. An image or video might be real but from a different time or place. Something might be factually true but misleading because it’s missing context. Trusted friends and authority figures will share this stuff, so don’t use that as a primary signal.

The main way to check veracity is to look for evidence that challenges it.

For example you can use a website called TinEye to see where an image has appeared before (you can also freeze frame videos and take screenshots). You can put the information into Google with the word ‘hoax’ or ‘factcheck’ to see if it’s already been debunked.

You can try to identify the original source, rather than whoever passed it on to you. That might give you clues about their independence, expertise or agenda.

Q: And people can use AI tools to verify as well?

Using AI to verify material is very dangerous. We’ve seen a big increase in the last few years of people using AI tools to check if an image or a video is real, and very often they get it wrong.

Part of the reason is that people ask the wrong question. If you ask AI “is this fake?” then you have to remember tools like ChatGPT and Grok are sycophantic, they are designed to please, so they will say “yes” even if they’re not certain. They are also overconfident.

The key thing with AI in general is to always ask a neutral question. Ask it what evidence there might be to help you identify whether something is true or false. Ask it what methods you can use.

AI is best used to challenge you and open up other ideas and techniques. Don’t use it to confirm things because it will just tell you what you want to hear.

Never rely on AI’s answer as the end of a process – it can only ever point you to next steps. Remember that it is not a factual tool, it is a language prediction tool. So it can predict the language of a piece of advice around verifying something, but it cannot know if something is real or fake, because that’s not what AI is designed to do.

From stories about political speech and song lyrics, to street names and social media chatter, data journalists now have a wide range of examples of text-as-data to draw inspiration and guidance from, while tools such as Pinpoint and NotebookLM are making text analysis easier than ever.

I compiled a list of over 200 pieces of data journalism where text or documents were used as sources. Quantification techniques ranged from counting the frequency of a single word and using Google’s ngram viewer, to machine learning and topic modelling.

Looking at those articles it’s clear that, once quantified, journalists tell the same stories about text as any other piece of data: using the seven most common angles.

But how those angles are used — and how often — is where it gets interesting…

Text-based data journalism most often uses an exploratory feature format

While most data stories about numerical data focused on scale or change, the data stories about text I looked at were overwhelmingly dominated by exploratory formats.

There are two obvious possible explanations for this. First, analysing text normally requires more time and skill than working with numerical data. That would be hard to justify for a simpler news article revealing scale or change.

Second, text is often rich and complex, lending itself more to exploration. The analysis itself will often require explanation too, especially if it involves classification.

The Pudding‘s scrollytell data feature on speeches in Parliament is a good example of an exploratory approach: text from over a million speeches is classified in two ways: by the gender of the speaker, and by the topic it relates to. It is quantified in terms of the percentage of time that speakers spend on each topic (code is shared in a GitHub repo).

The story explores the data through a number of themes: the economy, welfare, education, and so on. Although the focus is largely on variation between men and women, this is secondary to the primary exploratory angle.

The same thematic approach is adopted by South African data journalism website The Outlier in #SONA2022: How does it compare to the last 5 speeches by Ramaphosa? and by Sueddeutsche Zeitung in their feature on why Russians perceive the West as a threat.

Another way to structure an exploratory feature about text is by breaking the feature up around different questions. This is the approach taken by Quartz in The hidden structure of the Apple keynote (“Who’s on stage?”, “Who’s the funniest?” and “When is the unveil?”)

Stories about changes in language

Text can act as a proxy for cultural attitudes and fashions, or political focus, so it’s not surprising that many stories based on data analysis focus on changes in society or power that language can reveal.

Text from social media and forums can be analysed to answer questions about whether the mood of populations is changing (The war in Ukraine has made Russian social-media users glum) or if people are getting ruder (Foul-mouthed mothers are causing problems for Mumsnet).

Song lyrics can be analysed to identify trends in negativity and complexity while, over a longer timescale, book context can be analysed to conclude, as Jon Burn-Murdoch one Twitter thread, that “western society is shifting away from a culture of progress, and towards one of caution, worry and risk-aversion”.

Changes in political speech are an important indicator not only of the priorities of those in power, but also their relationships with each other and their role in setting the tone of national conversation.

One Guardian analysis, for example, leads on How rightwing rhetoric has risen sharply in the UK parliament, while the award-winning USA Today feature Hope’ is out, ‘fight’ is in: Does tweeting divide Congress, or simply echo its division?” reports that “Language has become more divided and emotional” based on an analysis of more than 2.8 million tweets posted by members of Congress (the original article is no longer online but parts can be seen here).

More simply, the Press Association used the official record of Parliament to calculate that mentions of “Brexit” had gone From nine mentions a year to 9,000.

A more investigative use of change is provided by the Washington Post’s investigation into claims that “USAID’s IG removed critical details from public reports”. By obtaining draft versions of audits reporters were able to identify what changes were made before publication: “more than 400 negative references were removed from the audits between the draft and final versions”

Ranking the most used words and phrases

“Doncaster is the least sexist place in Britain when it comes to street names,” the Doncaster Free Press reported in 2020. The newspaper hadn’t done the analysis: a PR firm had scraped the names of over 200,000 streets and classified them along gender lines. They had also ranked the places with the biggest gap between male and female proportions, and the most common names (Victoria and John).

Ranking can be a quick way to get a story out of a corpus of text — if you can identify a pattern to extract data from. ABC News in Australia, for example, analysed over 1,000 job descriptions to reveal “some of the most common words and phrases advertisers used to attract interns”, while The size of things: an ngram experiment “used Google Books’ Ngram dataset to find the most popular size analogies in English books”.

It turns out that “size of a pea” was the most common.

Even exploratory features can focus on exploring different rankings: the Pudding feature Sing My Name moves through sections ranking the most popular names in songs, which songs have the most repeat mentions of the same name, and what songs contain the most names, among others.

The scale of an issue — revealed in language

Text analysis can allow journalists to reveal the scale of a problem which is not visible in more traditional statistics.

Most of the text-based stories I’ve worked on at the BBC fall into this category: there are no official statistics on the outcomes of police misconduct cases, but I was able to analyse police watchdog reports for a story that revealed “Half of police employees who committed gross misconduct were not dismissed”.

Along similar lines, classifying text in festival line-ups led to the story Music festivals: Only 13% of UK headliners in 2022 are female and declarations of interest were analysed to establish that “One in five MPs continue to employ a member of their family”.

Social media analysis might focus on the scale of a problem on particular platforms: 3 million tweets were analysed for the story Scale of abuse of politicians on Twitter revealed and similar techniques were used by Amnesty and by the Turing Institute to establish the scale of abuse of particular groups.

Scale is a useful fallback angle if you are quantifying text, because no one else will have analysed the text before. The Guardian’s investigation into extremist Facebook groups, for example, leads on the finding that the network “exposes hundreds of thousands of Britons to racist language, conspiracy and disinformation” (my emphasis), while USA Today analysed campaign rally speech transcripts for Trump used words like invasion, killer to discuss immigrants 500 times.

It can also be a useful approach for factchecking or putting a news event into context, as in Der Spiegel’s story How much violence is in Rammstein’s lyrics? following allegations of sexual assault against the band’s singer (the investigation was later dropped)

Women versus men and other variation stories

Variation stories rely on an expectation of fairness, equality or parity. This limits the opportunities for this angle, but it can work especially well where language reveals implicit biases in society that are not quantified anywhere else.

The LA Times‘s scrollytell There are more women than ever in Star Wars. Men still do most of the talking is just one of a number of pieces of data journalism looking at gender variation using text. Others include The Gender Divide in Star Wars Scripts, Meghan gets twice as many negative headlines as positive, analysis finds, The New York Times’s The Words Men and Women Use When They Write About Love, and The Pudding’s The physical traits that define men & women in literature:

“Do authors really mention particular body parts more for men than for women? Are women’s bodies described using different adjectives than those attributed to men?”

Variation along lines of race is revealed in Football commentary racially biased, study finds and HuffPost UK’s The Met Police Are More Likely To Publish Your Mugshot If You’re Black, which classified press releases on criminal sentencing based on mentions of ethnicity, and compared the proportions to full data on sentencing by ethnicity.

Examples of political and ideological variation can be found in The Economist’s Twitter’s algorithm does not seem to silence conservatives and Paris Match’s political speech comparison Macron-Le Pen: Their silences speak volumes.

The Washington Post’s Almost all news coverage of the Barcelona attack mentioned terrorism. Very little coverage of Charlottesville did: “Even before we did our study,” the story notes, “research showed disproportionately high media coverage of terrorism committed by Muslims — even though right-wing extremist groups have committed more attacks”.

Given that databases of media coverage (such as Nexis) provide an accessible source of text data, this feels like an area ripe for more analysis.

‘Meta’ data stories are all methodologies

It is increasingly rare to find ‘meta’ data journalism angles: stories about a lack of data, poor data, or ‘Get the data‘ articles that share data for others to analyse. And the exclusive nature of text data makes it even more unlikely.

But the complexity of quantifying text for analysis means that there is sometimes a need to tell the story-behind-the-story about the methodologies that were employed: How Sky News investigated X’s algorithm for political bias, along with The Guardian’s How we measured the rise of populist rhetoric, and Reading the post-riot posts: how we traced far-right radicalisation across 51,000 Facebook messages are typical examples, while Behind the Guardian’s analysis of 100 years of MPs’ language on immigration shows another approach.

Connections, similarities, and correlations revealed by text analysis

Relationship angles were the least common in the examples of text-based data journalism I found. The few stories that did adopt this focus, however, pointed to at least three possible types of relationships that could be used for stories: correlations, similarities and connections.

A correlation angle can be seen in The Economist’s Why beer snobs guzzle lagers they claim to dislike, which visualises the relationship between word frequencies and user ratings.

The example points to the types of sources that might provide material for similar relationship stories: where text appears alongside a numerical measure. 

Reviews are just one such category of text data. Social media updates (which appear alongside numbers of likes, shares or views, and timestamps) and cultural texts such as books, songs and scripts (figures on sales, streams and views) have the same qualities. 

An alternative approach is to quantify text yourself, and combine it with other data: The Guardian used this to establish a relationship between Donald Trump’s use of a teleprompter and populism, while Paris Match used it to establish a lack of relationship between search interest and candidates’ mentions of an issue.

Similarities formed the focus of the network graph-led A Map of Lexical Distances Between Europe’s Languages and Lexical Distance Among Languages of Europe 2015. The methodology points to how the ‘distance’ between words can be quantified in order to identify patterns.

Connections are the focus for Mathematicians mapped out every “Game of Thrones” relationship to find the main character, which quantifies text by classifying two characters being mentioned in the same sentence as a connection. Although this is a ranking story (the focus is on identifying the ‘main’ character), the analysis could equally have been used to tell a story about relationships, and the same method could be adapted for any text where entities (people, companies, locations) are mentioned. Paris Match, for example, used co-occurrence analysis to identify what words candidates associated with ‘system’.

Connections encoded in large language models provide another potential source: Economist data journalist, Sondre Solstad, uses this to look at the connotations of baby names.

The Epstein files provide further ideas for working with text: The Economist’s Inside Epstein’s Network, for example, also chooses a ranking angle revealing the “500 people who appear most frequently”, and while The Guardian headlined their story as “Jeffrey Epstein’s elite relationships visualised”, what is specifically visualised is the relationships between correspondence and time. What is missing from coverage generally are the clusters, cliques and bridges that can be generated by the Epstein Document Network Explorer.

Leads from analysis of text

The Epstein files also provide an example of how data-driven approaches to text can provide useful story leads. The New York Times, for example, used AI to help “identify clusters, patterns and themes,” although it may be that this is being underused. “The brunt of the work,” they admit, “is being done by a team of editors and reporters across beats and bureaus who have been preparing to dig into the files long before they were published.”

Once you have collected a corpus of text, exploratory analysis can surface potential stories you may not have considered.

When I worked on the BBC story DBS background checks mean NHS staff ‘paying to work’, for example, the angle came as a result of identifying phrases which appeared multiple times in NHS job ads (we then went on to use the data to establish the scale of this). And once I’d collected documents on police misconduct, topic modelling allowed me to see common themes across a large number of documents, any one of which could have been an avenue for further reporting.

I’d love to know of any examples where text analysis has been used to identify story leads. If you’ve been involved in a text analysis project, please let me know in the comments or on LinkedIn.